Pipeline to wait until spark cluster is available

Question

Pipeline to wait until spark cluster is available

Kumar, Arun 336

Hello,

We have a very small node cluster and due to multiple spark notebooks running at the same time the notebooks or pipeline fails with 6002 timedout error. Some notebooks do not start at all and fails. Is there a way we can queue the notebooks or pipelines until the spark cluster or node is available?

Thanks,
Arun

Chandra Boorla 12,085 Reputation points Microsoft External Staff

2025-04-23T16:01:57.2066667+00:00

@Kumar, Arun

You're correct, when multiple Spark notebooks are triggered at the same time in a small Spark pool, they may fail with the 6002 - timed out while waiting for cluster error if there aren't enough available nodes to serve all sessions concurrently.

Azure Synapse Spark pools don’t currently support built-in queuing for notebook executions. However, here are a few options to help you manage this situation:

Workaround Options:

Add Retry Logic with Until Activity (ADF) - You can build a custom loop using the Until activity in your pipeline to wait until the cluster is ready. For example:

Try to run a lightweight notebook or status check script.

If it fails due to unavailability, wait (e.g., 1-2 minutes), then retry.

Break out of the loop once the notebook runs successfully or the cluster is ready.

Stagger Notebook Executions with Wait Activity - If you're launching multiple notebooks, consider spacing them out using Wait activities. This won't queue them dynamically, but it helps avoid overloading the pool at once.

Use Azure Monitor + Logic Apps - You can set up alerts on your Spark pool (e.g., when CPU or session count is low) and trigger Logic Apps or pipelines based on availability. This creates an automated, resource-aware trigger system.

Consider Scaling the Spark Pool - If possible, review your Spark pool size or consider enabling dynamic allocation (if applicable) to automatically add resources when demand increases.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Chandra Boorla 12,085 Reputation points Microsoft External Staff

2025-04-24T18:05:06.5733333+00:00

@Kumar, Arun

Just checking in to see, did any of the suggestions above help resolve the issue with your Spark cluster and notebook timeouts? As your feedback is valuable and can assist others in the community facing similar issues.

If you have found a resolution to your issue, we would appreciate it if you could share it in the thread to benefit others.

Thank you.
Chandra Boorla 12,085 Reputation points Microsoft External Staff

2025-04-25T17:16:51.25+00:00

@Kumar, Arun

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

2 answers

Your answer

Chandra Boorla 12,085 Reputation points Microsoft External Staff

2025-04-23T16:01:57.2066667+00:00

@Kumar, Arun

You're correct, when multiple Spark notebooks are triggered at the same time in a small Spark pool, they may fail with the 6002 - timed out while waiting for cluster error if there aren't enough available nodes to serve all sessions concurrently.

Azure Synapse Spark pools don’t currently support built-in queuing for notebook executions. However, here are a few options to help you manage this situation:

Workaround Options:

Add Retry Logic with Until Activity (ADF) - You can build a custom loop using the Until activity in your pipeline to wait until the cluster is ready. For example:

Try to run a lightweight notebook or status check script.

If it fails due to unavailability, wait (e.g., 1-2 minutes), then retry.

Break out of the loop once the notebook runs successfully or the cluster is ready.

Stagger Notebook Executions with Wait Activity - If you're launching multiple notebooks, consider spacing them out using Wait activities. This won't queue them dynamically, but it helps avoid overloading the pool at once.

Use Azure Monitor + Logic Apps - You can set up alerts on your Spark pool (e.g., when CPU or session count is low) and trigger Logic Apps or pipelines based on availability. This creates an automated, resource-aware trigger system.

Consider Scaling the Spark Pool - If possible, review your Spark pool size or consider enabling dynamic allocation (if applicable) to automatically add resources when demand increases.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Chandra Boorla 12,085 Reputation points Microsoft External Staff

2025-04-24T18:05:06.5733333+00:00

@Kumar, Arun

Just checking in to see, did any of the suggestions above help resolve the issue with your Spark cluster and notebook timeouts? As your feedback is valuable and can assist others in the community facing similar issues.

If you have found a resolution to your issue, we would appreciate it if you could share it in the thread to benefit others.

Thank you.
Chandra Boorla 12,085 Reputation points Microsoft External Staff

2025-04-25T17:16:51.25+00:00

@Kumar, Arun

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Kumar, Arun 336

Thanks Chandra. Yes you can close the ticket

Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-28T10:53:02.73+00:00

Hello@Kumar, Arun,**

Error:**

6002 - Timed out while waiting for cluster Pipeline error: "Activity timed out"

Notebooks get stuck at "Submitting" with the message "Notebook execution is currently in Starting state." Eventually, the error appears: "CLUSTER_CREATION_TIMED_OUT: Livy session has failed. Session state: Error. Error code: CLUSTER_CREATION_TIMED_OUT."

Resolution:

Deleting the Spark pool and recreating it (even with the same name) did not resolve the issue. However, scaling up the number of nodes and then scaling back down allowed the pipeline to run successfully without any further issues.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-29T11:00:47.91+00:00

Hello@Kumar, Arun,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-30T11:46:15.9266667+00:00

Hello@Kumar, Arun, We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Kumar, Arun 336 Reputation points

2025-04-30T11:56:01.3366667+00:00

Yes the solution should work. We have created three separate pols and testing in progress

Answer 2

Glad to hear that you have resolved the issue. As you mentioned, you applied a workaround by using three different pools.

Just to give info from experience ERROR : The "6002 - timed out while waiting for cluster" error in Spark typically occurs when a session or job is unable to start within the expected time because the cluster resources are not yet available.

This usually happens when There are not enough nodes or executors available to serve new sessions.

The cluster is scaling up but has not provisioned resources fast enough.

Other workloads are currently occupying the available capacity, causing delays for new jobs.

If multiple Spark notebooks are triggered simultaneously on a small Spark pool,

They may fail with a 6002 timeout error while waiting for the cluster, due to insufficient available nodes to handle all sessions at once.

In such scenarios review and scale your Spark pool — If feasible, consider increasing the size of your Spark pool or enabling dynamic allocation to automatically provision additional resources as demand grows.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Share via

Pipeline to wait until spark cluster is available

2 answers

Your answer