Databricks driver worker node connectivity issues inside workflow jobs

@Sergey Strelnikov

The error you're encountering DRIVER_UNREACHABLE (SERVICE_FAULT) typically means that the Databricks cluster was unable to communicate with the Spark driver node. After 30 failed attempts, the system automatically terminated the cluster to avoid further resource consumption.

This is often due to one of the following:

• A temporary network connectivity issue
• An overloaded or under-provisioned driver node
• Issues with the cloud infrastructure (e.g., VM instability)
• An unexpected Databricks runtime or job-level fault

Here are a few steps to help you troubleshoot:

Transient or Intermittent issue

• This could be the result of a transient or intermittent issue in the underlying infrastructure. These can happen occasionally and are often resolved by simply re-running the job.

Cluster Health & Logs

• Check the Event Log and Driver logs to identify early signs of failure or resource contention.
• Consider enabling cluster log delivery for deeper diagnostics.

Network Issues

• If you’re using Databricks Connect or working in a VNet-injected workspace, verify there are no DNS or firewall issues preventing communication between nodes.
• Ensure your workspace and cluster configurations are correctly set.

Driver Resource Overload

• The driver may be under-provisioned for the workload. Try increasing the driver node size or reducing concurrent job execution.
• Avoid running too many simultaneous streaming queries or large actions in a single notebook or workflow.

Cluster Configuration

• Make sure the Databricks Runtime version is compatible with your code and dependencies.
• Review autoscaling settings and consider using a more stable or pinned node type if you're seeing frequent cluster instability.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

@Sergey Strelnikov

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.