Databricks Runtime 16.4 LTS (Beta)

The following release notes provide information about Databricks Runtime 16.4 LTS, powered by Apache Spark 3.5.2.

Databricks released this beta version in April 2025.

Behavioral changes

Delta tables: "DESCRIBE DETAIL table" will now show the clusterByAuto status of the table

DESCRIBE DETAIL {table} will now show the clusterByAuto status of the table (true or false) next to the current clustering columns. For more details on clusterByAuto, see:Automatic liquid clustering.

Fix to respect options for data source cached plans

This update ensures table reads respect options set for all data source plans when cached, not just the first cached table read.

Previously, data source table reads cached the first plan but failed to account for different options in subsequent queries.

For example, the following query:

spark.sql("CREATE TABLE t(a string, b string) USING CSV".stripMargin)
spark.sql("INSERT INTO TABLE t VALUES ('a;b', 'c')")

spark.sql("SELECT * FROM t").show()
spark.sql("SELECT * FROM t WITH ('delimiter' = ';')")

would produce this output:

+----+----+
|col1|col2|
+----+----+
| a;b| c  |
+----+----+

+----+----+
|col1|col2|
+----+----+
| a;b| c  |
+----+----+

With this fix, it now returns the expected output:

+----+----+
|col1|col2|
+----+----+
| a;b| c  |
+----+----+

+----+----+
|col1|col2|
+----+----+
| a  | b,c|
+----+----+

If your workloads have dependencies on the previous incorrect behavior, you may see different results after this change.

Fix for grouping on aliased integer literals broke for certain operations

Grouping expressions on an aliased integer literal was previously broken for certain operations like MERGE INTO. For example, this expression would return GROUP_BY_POS_OUT_OF_RANGE since the value (val) would be replaced with 202001:

merge into t
using
(select 202001 as val, count(current_date) as total_count group by val) on 1=1
when not matched then insert (id, name)  values (val, total_count)

This has been fixed. To mitigate the issue in your existing queries, make sure that the constants you are using are not equal to the column position that must be in the grouping expressions.

Moved redaction rule from analyzer to optimizer

Previously, DataFrames could create tables that contained redacted values when valid SECRET SQL functions were used. This change removes redaction when saving DataFrames with valid secret access to a table, and the redaction rule has moved from the analyzer to the optimizer.

variant_get and get_json_object now consider leading spaces in paths in Apache Spark

Prior to this change, leading whitespaces and tabs in paths in the variant_get and get_json_object expressions were being ignored with Photon disabled. For example, select get_json_object('{" key": "value"}', '$[' key']') would not be effective in extracting the value of " key". However, users will be able to extract such keys.

Enable flag to disallow disabling source materialization for MERGE operations

Previously, users could disable source materialization in MERGE by setting merge.materializeSource to none. With the new flag enabled, this will be forbidden and cause an error. Databricks plans to enable the flag only for customers who haven't used this configuration flag before, so no customer should notice any change in behavior.

Move partition metadata log enablement anchor to table

The partition metadata log feature has been changed so that once a table is created with spark.databricks.nonDelta.partitionLog.enabled = true, you can anchor it to a table so a cluster doesn't set spark.databricks.nonDelta.partitionLog.enabled = true for all tables processed by the cluster.

Update snowflake-jdbc 3.16.1 -> 3.22.0

Updated the dependency snowflake-jdbc from 3.16.1 to 3.22.0. This may impact users if they directly use the l3.16.1 version of the library.

Downgrade Json4s from 4.0.7 to 3.7.0-M11 in DBR 16.2 and 16.3

Customers cannot use databricks-connect 16.1+ and Apache Spark™ 3.5.x together in the same application because of significant discrepancies in API behavior between Json4s version 3.7.0-M11 and version 4.0.7. To address this, Azure Databricks has downgraded Json4s to 3.7.0-M11.

New features and improvements

New listagg and string_agg functions

Starting with this release you can use the listagg or string_agg functions to aggregate STRING and BINARY values within a group. See string_agg for more details.

Library upgrades

• Upgraded Python libraries:
• Upgraded R libraries:
• Upgraded Java libraries:
  • io.delta.delta-sharing-client_2.12 from 1.2.3 to 1.2.4
  • org.json4s.json4s-ast_2.12 from 4.0.7 to 3.7.0-M11
  • org.json4s.json4s-core_2.12 from 4.0.7 to 3.7.0-M11
  • org.json4s.json4s-jackson_2.12 from 4.0.7 to 3.7.0-M11
  • org.json4s.json4s-scalap_2.12 from 4.0.7 to 3.7.0-M11

Apache Spark

Databricks Runtime 16.4 LTS includes Apache Spark 3.5.2. This release includes all Spark fixes and improvements included in Databricks Runtime 16.3, as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-51762] Fix Resolution of Views in Single-Pass Analyzer When Bridging is Enabled
• [SPARK-51716] Support serializing Variant to XML
• [SPARK-51503] Support Variant type in XML scan
• [SPARK-47895] Revert "[SQL] group by alias should be idempotent"
• [SPARK-51752] Enable rCTE referencing from within a CTE
• [SPARK-51747] [SQL] Data source cached plan should respect options
• [SPARK-51758] Apply late record filtering based on watermark only if timeMode is passed as EventTime to the transformWithState operator
• [SPARK-51576] Prohibit ANSI cast from array<variant, containsNull = false> to array<_, containsNull = false>
• [SPARK-51776] Fix logging in single-pass Analyzer
• [SPARK-51717] Fix SST mismatch corruption that can happen for second snapshot created for a new query
• [SPARK-51751] Fix multiple rCTEs for one WITH statement that reference each other
• [SPARK-51712] [SQL] Swallow non-fatal Throwables when resolving tables/views in spark.catalog.listTables()
• [SPARK-51646] Fix propagating collation in views with default collation
• [SPARK-51690] Change the protocol of ListState.put()/get()/appendList() from Arrow to simple custom protocol
• [SPARK-51732] Apply rpad on attributes with same ExprId if they need to be deduplicated
• [SPARK-51724] RocksDB StateStore's lineage manager should be synchronized
• [SPARK-51720] Add Cross Join as legal in recursion of Recursive CTE
• [SPARK-51682] State Store Checkpoint V2 should handle offset log ahead of commit log correctly
• [SPARK-51721] Change default value of ANALYZER_SINGLE_PASS_RESOLVER_RELATION_BRIDGING_ENABLED flag
• [SPARK-51685] Excessive Info logging from RocksDb operations causing too big executor stderr files
• [SPARK-51645] Fix CREATE OR REPLACE TABLE ... DEFAULT COLLATION ... query
• [SPARK-51298] Support variant in CSV scan
• [SPARK-51271] Add filter pushdown API to Python Data Sources
• [SPARK-51667] [PYTHON] Disable Nagle's algorithm (via TCP_NODELAY = true) in TWS + PySpark for python with regards to state server
• [SPARK-51668] Report metrics for failed writes to V2 data sources
• [SPARK-51187] Implement the graceful deprecation of incorrect config introduced in SPARK-49699
• [SPARK-51700] Fix incorrect logging when no files are eligible for deletion in RocksDBFileManager
• [SPARK-51655] Fix metric collection in UnionLoopExec and add test
• [SPARK-51657] [SC-192703] UTF8_BINARY default table collation shown by default in Desc As JSON (v1)
• [SPARK-51206] [CONNECT] Move Arrow conversion helpers out of Spark Connect
• [SPARK-51566] Python UDF traceback improvement
• [SPARK-51675] Fix col family creation after opening local DB to avoid snapshot creation, if not necessary
• [SPARK-50892] Add UnionLoopExec, physical operator for recursion, to perform execution of recursive queries
• [SPARK-51670] Refactor Intersect and Except to follow Union example to reuse in single-pass Analyzer
• [SPARK-47895] [SQL] group by alias should be idempotent
• [SPARK-51614] [SQL] Introduce ResolveUnresolvedHaving rule in the Analyzer
• [SPARK-51662] Make OrcFileFormat comparable
• [SPARK-51374] Revert "[SC-190311][core] Switch to Using java.util.Map in Logging APIs"
• [SPARK-51652] Refactor SetOperation computation out to reuse it in the single-pass Analyzer
• [SPARK-51622] Titling sections on ExecutionPage
• [SPARK-51586] Initialize input partitions independent of columnar support in continuous mode
• [SPARK-51070] Use scala.collection.Set instead of Set in ValidateExternalType
• [SPARK-51559] Make max broadcast table size configurable
• [SPARK-51496] CaseInsensitiveStringMap comparison should ignore case
• [SPARK-50416] A more portable terminal / pipe test needed for bin/load-spark-env.sh
• [SPARK-51509] [UI] Make Spark Master Environment page support filters
• [SPARK-51374] Switch to Using java.util.Map in Logging APIs
• [SPARK-50820] DSv2: Conditional nullification of metadata columns in DML
• [SPARK-48922] Avoid redundant array transform of identical expression for map type
• [SPARK-51446] Improve the codecNameMap for the compression codec
• [SPARK-51341] Cancel time task with suitable way.
• [SPARK-51452] Improve Thread dump table search
• [SPARK-51444] Remove the unreachable if branch from TaskSchedulerImpl#statusUpdate
• [SPARK-51401] Change ExplainUtils.generateFieldString to directly call QueryPlan.generateFieldString
• [SPARK-49507] Fix the case issue after enabling metastorePartitionPruningFastFallback
• [SPARK-51506] [SS] Do not enforce users to implement close() in TransformWithStateInPandas
• [SPARK-51624] Propagate GetStructField metadata in CreateNamedStruct.dataType
• [SPARK-51201] Make Partitioning Hints support byte and short values
• [SPARK-51186] Add StreamingPythonRunnerInitializationException to PySpark base exception
• [SPARK-50286] Correctly propagate SQL options to WriteBuilder
• [SPARK-51616] Run CollationTypeCasts before ResolveAliases and ResolveAggregateFunctions
• [SPARK-51023] Log remote address on RPC exception
• [SPARK-51087] [CONNECT] Raise a warning when memory-profiler is not installed for memory profiling
• [SPARK-51073] Remove Unstable from SparkSessionExtensionsProvider trait
• [SPARK-51471] RatePerMicroBatchStream - classify the ASSERT error when offset/timestamp in startOffset is larger than the endOffset
• [SPARK-51062] Fix assertSchemaEqual to compare decimal precision and scale
• [SPARK-51612] Display Spark confs set at view creation in Desc As Json
• [SPARK-51573] Fix Streaming State Checkpoint v2 checkpointInfo race condition
• [SPARK-51252] Add instance metrics for last uploaded snapshot version in HDFS State Stores
• [SPARK-51552] [SQL] Disallow temporary variables in persisted views when under identifier
• [SPARK-51625] Command in CTE relations should trigger inline
• [SPARK-51584] Add rule that pushes Project through Offset and Suite that tests it
• [SPARK-51593] Refactor QueryExecutionMetering instantiation
• [SPARK-49082] Support widening Date to TimestampNTZ in Avro reader
• [SPARK-50220] Support listagg in PySpark
• [SPARK-51581] [SQL] Use nonEmpty/isEmpty for empty check for explicit Iterable
• [SPARK-42746] Fix in error class classification for new exceptions with listagg
• [SPARK-51580] Throw proper user facing error message when lambda function is out of place in HigherOrderFunction
• [SPARK-51589] Fix small bug failing to check for aggregate functions in |> SELECT
• [SPARK-50821] Upgrade Py4J from 0.10.9.8 to 0.10.9.9
• [SPARK-42746] Implement LISTAGG function
• [SPARK-48399] Teradata: ByteType should map to BYTEINT instead of BYTE(binary)
• [SPARK-43221] Host local block fetching should use a block status of a block stored on disk
• [SPARK-51467] Make tables of the environment page filterable
• [SPARK-51569] Don't reinstantiate InSubquery in InTypeCoercion if there are no type changes
• [SPARK-51280] Improve RESPONSE_ALREADY_RECEIVED error class
• [SPARK-51525] Collation field for Desc As JSON StringType
• [SPARK-51565] Support SQL parameters in window frame clause
• [SPARK-51544] Add only unique and necessary metadata columns
• [SPARK-49349] Improve error message for LCA with Generate
• [SPARK-51418] Fix DataSource PARTITON TABLE w/ Hive type incompatible partition columns
• [SPARK-51269] Simplify AvroCompressionCodec by removing defaultCompressionLevel
• [SPARK-50618] [SQL] Make DataFrameReader and DataStreamReader leverage the analyzer more
• [SPARK-51097] Re-introduce RocksDB state store's last uploaded snapshot version instance metrics
• [SPARK-51321] Add rpad and lpad support for PostgresDialect and MsSQLServerDialect expression pushdown
• [SPARK-51428] Reassign Aliases for collated expression trees deterministically
• [SPARK-51528] Don't compare hash codes directly in ProtobufCatalystDataConversionSuite
• [SPARK-51443] Fix singleVariantColumn in DSv2 and readStream.
• [SPARK-51397] Fix maintenance pool shutdown handling issue causing long test times
• [SPARK-51522] Disable lazy union children test for single-pass Analyzer
• [SPARK-51079] Support large variable types in pandas UDF, createDataFrame and toPandas with Arrow
• [SPARK-51112] [CONNECT] Avoid using pyarrow's to_pandas on an empty table
• [SPARK-50855] Follow up on [SC-190361] Fix post-merge tests failure in isolated code paths
• [SPARK-51438] Make CatalystDataToProtobuf and ProtobufDataToCatalyst properly comparable and hashable
• [SPARK-50855] Spark Connect Support for TransformWithState In Scala
• [SPARK-51208] ColumnDefinition.toV1Column should preserve EXISTS_DEFAULT resolution
• [SPARK-51468] Revert "From json/xml should not change collations in the given schema"
• [SPARK-51453] [SC-190922] [SQL] AssertTrue uses toPrettySQL instead of simpleString
• [SPARK-51409] Add error classification in the changelog writer creation path
• [SPARK-51440] classify the NPE when null topic field value is in kafka message data and there is no topic option
• [SPARK-51425] Add client API to set custom operation_id
• [SPARK-50652] Add checks to RocksDB V2 backward compatibility
• [SPARK-49164] Fix not NullSafeEqual in predicate of SQL query in JDBC Relation
• [SPARK-50880] Add a new visitBinaryComparison method to V2ExpressionSQLBuilder
• [SPARK-50838] Performs additional checks inside recursive CTEs to throw an error if forbidden case is encountered
• [SPARK-51307] locationUri in CatalogStorageFormat shall be decoded for display
• [SPARK-51079] Support large variable types in pandas UDF, createDataFrame and toPandas with Arrow
• [SPARK-50792] Format binary data as a binary literal in JDBC.
• [SPARK-50596] Upgrade Py4J from 0.10.9.7 to 0.10.9.8

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

• Operating System: Ubuntu 24.04.2 LTS
• Java: Zulu17.54+21-CA
• Scala: 2.12.15
• Python: 3.12.3
• R: 4.4.0
• Delta Lake: 3.3.0

Installed Python libraries

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2025-03-20:

Installed Java and Scala libraries (Scala 2.12 cluster version)