Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Streaming real-time mode has the following limitation:
ARBITRARY_STATEFUL_OPERATIONS_NOT_SUPPORTED
The arbitrary stateful streaming operator, <op>, is not currently supported. Please see the Real-Time Mode User Guide for a list of supported operators.
ASYNC_PROGRESS_TRACKING_NOT_SUPPORTED
Async progress tracking is not supported in real-time mode. Set option asyncProgressTrackingEnabled to false and retry your query.
ASYNC_STATE_CHECKPOINT_NOT_SUPPORTED
Async state checkpointing is not supported in real-time mode. Set spark.databricks.streaming.statefulOperator.asyncCheckpoint.enabled to false and retry your query.
BATCH_UNION_NOT_SUPPORTED
Real-time mode does not support union with batch data sources. Ensure that the union streaming query only contains streaming data sources.
CHECKPOINT_V2_REQUIRED
The real-time mode requires checkpoint v2. Set spark.sql.streaming.stateStore.checkpointFormatVersion to 2 and retry your query.
CONCURRENT_STAGE_DAG_SCHEDULER_REQUIRED
The real-time mode requires the concurrent stage DAG scheduler. Restart your spark cluster and set spark.databricks.dagScheduler.type to ConcurrentStageDAGScheduler.
DROP_DUPLICATES_WITHIN_WATERMARK_NOT_SUPPORTED
dropDuplicatesWithinWatermark is not supported. To work around this limitation, you can use dropDuplicates instead. Remember to pass your event-time column to dropDuplicates so that state can be evicted.
EVENT_TIME_BASED_TIMERS_IN_TRANSFORM_WITH_STATE_NOT_SUPPORTED
Event time based timers in transformWithState operator are not supported yet. If you would like to use the transformWithState operator, please specify processing time in your transformWithState operator and retry your query.
IDENTICAL_SOURCES_IN_UNION_NOT_SUPPORTED
Real-time mode does not support union on two or more identical streaming data sources in a single query. This includes scenarios such as referencing the same source DataFrame more than once, or using two data sources with identical configurations for some sources. For Kafka, avoid reusing the same DataFrame and create different ones. For Kinesis, in addition to duplicating data frames, ensure that each DataFrame sets a unique value for the "consumerName" option. Sources provided in the query: <sources>
INPUT_STREAM_NOT_SUPPORTED
The input stream <className> is not supported. See the Real-Time Mode User Guide for a list of supported input streams.
MULTI_SHUFFLE_MANAGER_REQUIRED
The real-time mode requires the multi-shuffle manager. Set spark.shuffle.manager to org.apache.spark.shuffle.streaming.MultiShuffleManager and restart your spark cluster.
OPERATOR_OR_SINK_NOT_IN_ALLOWLIST
The <errorType>(s): <message> not in the <errorType> allowlist for Real-Time Mode. To bypass this check, set spark.databricks.streaming.realTimeMode.allowlistCheck to false. By changing this, you agree to run the query at your own risk.
OUTPUT_MODE_NOT_SUPPORTED
The output mode <outputMode> is not supported. To work around this limitation, set the output mode to Update. In the future, <outputMode> may be supported.
ROCKSDB_STATE_STORE_PROVIDER_REQUIRED
The real-time mode requires the RocksDB state store provider or providers that extends it. Set spark.sql.streaming.stateStore.providerClass to org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider and retry your query.
SESSION_WINDOWS_NOT_SUPPORTED
Session windows are not yet supported. The only windowed aggregations allowed are over tumbling and sliding windows.
SPECULATIVE_EXECUTION_NOT_SUPPORTED
Speculative execution is not supported in real-time mode. Set spark.speculation to false and restart your spark cluster.
STATEFUL_OPERATORS_BEFORE_UNION_NOT_SUPPORTED
Streaming queries in real-time mode cannot include stateful operators (e.g. aggregate, deduplicate, transformWithState) before a union. Please restructure your query to apply the union before any stateful operations.
STREAM_STREAM_JOIN_NOT_SUPPORTED
No type of streaming joins are supported yet.