Databricks Cluster Start-up Fails on TEST Workspace – NPIP_TUNNEL_SETUP_FAILURE

Question

Databricks Cluster Start-up Fails on TEST Workspace – NPIP_TUNNEL_SETUP_FAILURE

Maroš Urminský 0

Hello,

We are experiencing an issue with our Databricks TEST workspace where clusters fail to start. The error message indicates an NPIP tunnel setup failure.

All our workspaces (TEST, QA, and PROD) are configured with No Public IP access enabled and share identical network policies and topology. Each workspace uses private endpoint connections to a private DNS zone located within our corporate network.

The QA and PROD workspaces function without issues. However, the TEST workspace fails during cluster start-up with the following error:

{
  "reason": {
    "code": "NPIP_TUNNEL_SETUP_FAILURE",
    "type": "SERVICE_FAULT",
    "parameters": {
      "databricks_error_message": "NPIP tunnel setup failure during launch. Please try again later and contact Databricks if the problem persists. \nInstance bootstrap failed command: WaitForNgrokTunnel\nFailure message (may be truncated): Timed out waiting for ngrok tunnel to be up"
    }
  },
  "add_node_failure_details": {
    "failure_count": 1,
    "resource_type": "container",
    "will_retry": false
  }
}

Environment Details:

Region: switzerlandnorth

NSGs: Associated with both public and private databricks subnets, allowing all required inbound and outbound traffic.

Outbound network access: FULL_ACCESS enabled.

No additional firewall is configured.

Troubleshooting Performed:

Recreated compute cluster.

Deleted all cluster instances, waited 30 minutes, and then created a new cluster.

Verified NSG rules and configuration profiles.

Compared network and workspace configurations with working QA and PROD environments.

Could you please investigate this issue and advise further?

Best regards, Maros

Chandra Boorla 12,125 Reputation points Microsoft External Staff Moderator

2025-04-07T17:26:43.31+00:00

@Maroš Urminský

The error you're encountering – NPIP_TUNNEL_SETUP_FAILURE – indicates that the Databricks control plane is unable to establish a secure tunnel to the cluster node during startup. This tunnel is essential in No Public IP (NPIP) configurations, where cluster nodes do not receive public IP addresses.

Given that your QA and PROD workspaces are functioning correctly with the same network setup, it’s likely that something specific to the TEST workspace or its underlying virtual network is interfering with the tunnel setup.

A common cause for this error is DNS resolution issues, particularly in environments using private endpoints and custom DNS zones.

I recommend reviewing the DHCP configuration and DNS settings for the virtual network (VNet) associated with the TEST workspace. Specifically, please check:

Whether the VNet is using either the Azure-provided DNS or a custom DNS server that can resolve Databricks-related domains such as *.cloud.databricks.com.

If the TEST subnet is correctly integrated with the private DNS zone, and that required records (such as for internal ngrok endpoints) are resolving as expected.

Whether there are any custom DHCP options or firewall rules that might block DNS or outbound HTTPS traffic needed for ngrok tunnel initialization.

You can validate this by deploying a temporary VM in the TEST subnet and attempting to resolve and reach the required domains. Compare the behavior with VMs in your QA/PROD subnets to spot discrepancies.

DNS resolution and outbound HTTPS access are critical for establishing the NPIP tunnel, and any misconfiguration here could lead to the failure you're observing.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Chandra Boorla 12,125 Reputation points Microsoft External Staff Moderator

2025-04-08T17:21:58.89+00:00

@Maroš Urminský

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 12,125 Reputation points Microsoft External Staff Moderator

2025-04-09T00:22:38.0366667+00:00

@Maroš Urminský

Following up to see if the above suggestion was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Maroš Urminský 0 Reputation points

2025-04-09T06:46:48.4+00:00
Dear @Chandra Boorla , Thank you for your suggestions. We spent a full day troubleshooting the issue, but unfortunately the problem persists.

We have double-checked the configuration across our TEST, QA, and PROD environments and found no discrepancies in terms of:

NSG rules

Subnet and VNet setup

Private endpoint definitions

Outbound internet access (FULL_ACCESS)

As part of the investigation, we removed and recreated the private endpoints and corresponding DNS records. However, the issue with the TEST workspace remains unresolved.

Furthermore, we conducted an experiment by creating a new DEV workspace within the same VNet as TEST, using the same subnets, and the cluster starts without issue. This strongly suggests that the VNet, subnets, and private DNS zone linking are not the root cause.

Our current hypothesis is that the issue may be tied to the history of the TEST workspace setup:

Originally, the Private DNS zone (privatelink.azuredatabricks.net) was located in the TEST subscription, with a link to the corporate office VNet.

Later, for consistency, we moved the Private DNS zone to the office subscription and linked it to all workspaces (QA, PROD, TEST).

It's possible there’s a residual association or misconfigured link from the earlier setup that isn’t visible in the Azure portal but is still affecting DNS resolution or endpoint routing.

Additionally, we suspect that this issue could be due to cached configuration or metadata on the Databricks control plane, tied to the TEST workspace’s previous setup.

We would appreciate any guidance on how to identify and clean up such “ghost” configurations — or if you can verify this from backend diagnostics on your side.

Thank you for your support.
Rukmini 1,796 Reputation points Microsoft External Staff Moderator

2025-04-11T01:55:08.46+00:00

Maroš Urminský, Did you try to recreate the TEST workspace and checked?
JAYA SHANKAR G S 2,435 Reputation points Microsoft External Staff Moderator

2025-04-11T08:34:25.46+00:00
Hello @Maroš Urminský ,

Can you try some additional steps and considerations to help identify and resolve the issue:

Clear DNS Cache: Ensure that the DNS cache is cleared on the machines that are trying to resolve the private endpoints. This can be done using the following command on the relevant machines:

%sh ipconfig /flushdns

Check for Residual DNS Records: Verify that there are no residual DNS records in the private DNS zone that might be causing conflicts. You can use Azure CLI or PowerShell to list and remove any unnecessary records.

az network private-dns record-set list --resource-group <ResourceGroupName> --zone-name privatelink.azuredatabricks.net Get-AzPrivateDnsRecordSet -ResourceGroupName <ResourceGroupName> -ZoneName privatelink.azuredatabricks.net

Reference: Enable secure cluster connectivity

Thank you
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-04-14T10:53:52.03+00:00

Hello @Maroš Urminský , we haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-04-15T05:04:30.7666667+00:00

Hello @Maroš Urminský ,We haven’t heard from you on the last response and was just checking back to see if the above suggestion have resolved your issue. If it answers your query, do click thumpsup. And, if you have any further query do let us know.
Maroš Urminský 0 Reputation points

2025-04-15T06:42:46.57+00:00

Hello,

we tried flushing dns, but the problem is between cluster VM and databricks control pane. We are unable to connect to the managed VM unless it connects to the databricks workspace.

We are going to have a look on the residual DNS records, that might not be visible in the portal directly. I just have to wait for IT admin availability.

Re-creation of TEST workspace is the last resolution that we would like to avoid.

I will get back to you with the results
AnnuKumari-MSFT 34,541 Reputation points Microsoft Employee Moderator

2025-04-21T14:31:04.33+00:00

Hi Maroš Urminský ,

Thankyou for sharing the approach you are going to attempt. Kindly let us know if that helped resolve your issue? Thanks
Chris Bale 0 Reputation points

2025-04-21T14:58:30.2+00:00

Hey Maros, since QA and PROD are fine, it could be a DNS or private endpoint issue specific to TEST. Try revalidating the private DNS link and restarting the endpoint. If that doesn’t help, best to contact Databricks support with logs they can check backend issues.

Your answer

Chandra Boorla 12,125 Reputation points Microsoft External Staff Moderator

2025-04-08T17:21:58.89+00:00

@Maroš Urminský

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 12,125 Reputation points Microsoft External Staff Moderator

2025-04-09T00:22:38.0366667+00:00

@Maroš Urminský

Following up to see if the above suggestion was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Maroš Urminský 0 Reputation points

2025-04-09T06:46:48.4+00:00

Dear @Chandra Boorla , Thank you for your suggestions. We spent a full day troubleshooting the issue, but unfortunately the problem persists.

We have double-checked the configuration across our TEST, QA, and PROD environments and found no discrepancies in terms of:

NSG rules

Subnet and VNet setup

Private endpoint definitions

Outbound internet access (FULL_ACCESS)

As part of the investigation, we removed and recreated the private endpoints and corresponding DNS records. However, the issue with the TEST workspace remains unresolved.

Furthermore, we conducted an experiment by creating a new DEV workspace within the same VNet as TEST, using the same subnets, and the cluster starts without issue. This strongly suggests that the VNet, subnets, and private DNS zone linking are not the root cause.

Our current hypothesis is that the issue may be tied to the history of the TEST workspace setup:

Originally, the Private DNS zone (privatelink.azuredatabricks.net) was located in the TEST subscription, with a link to the corporate office VNet.

Later, for consistency, we moved the Private DNS zone to the office subscription and linked it to all workspaces (QA, PROD, TEST).

It's possible there’s a residual association or misconfigured link from the earlier setup that isn’t visible in the Azure portal but is still affecting DNS resolution or endpoint routing.

Additionally, we suspect that this issue could be due to cached configuration or metadata on the Databricks control plane, tied to the TEST workspace’s previous setup.

We would appreciate any guidance on how to identify and clean up such “ghost” configurations — or if you can verify this from backend diagnostics on your side.

Thank you for your support.
Rukmini 1,796 Reputation points Microsoft External Staff Moderator

2025-04-11T01:55:08.46+00:00

Maroš Urminský, Did you try to recreate the TEST workspace and checked?
JAYA SHANKAR G S 2,435 Reputation points Microsoft External Staff Moderator

2025-04-11T08:34:25.46+00:00

Hello @Maroš Urminský ,

Can you try some additional steps and considerations to help identify and resolve the issue:

Clear DNS Cache: Ensure that the DNS cache is cleared on the machines that are trying to resolve the private endpoints. This can be done using the following command on the relevant machines:

%sh ipconfig /flushdns

Check for Residual DNS Records: Verify that there are no residual DNS records in the private DNS zone that might be causing conflicts. You can use Azure CLI or PowerShell to list and remove any unnecessary records.

az network private-dns record-set list --resource-group <ResourceGroupName> --zone-name privatelink.azuredatabricks.net Get-AzPrivateDnsRecordSet -ResourceGroupName <ResourceGroupName> -ZoneName privatelink.azuredatabricks.net

Reference: Enable secure cluster connectivity

Thank you
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-04-14T10:53:52.03+00:00

Hello @Maroš Urminský , we haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-04-15T05:04:30.7666667+00:00

Hello @Maroš Urminský ,We haven’t heard from you on the last response and was just checking back to see if the above suggestion have resolved your issue. If it answers your query, do click thumpsup. And, if you have any further query do let us know.
Maroš Urminský 0 Reputation points

2025-04-15T06:42:46.57+00:00

Hello,

we tried flushing dns, but the problem is between cluster VM and databricks control pane. We are unable to connect to the managed VM unless it connects to the databricks workspace.

We are going to have a look on the residual DNS records, that might not be visible in the portal directly. I just have to wait for IT admin availability.

Re-creation of TEST workspace is the last resolution that we would like to avoid.

I will get back to you with the results
AnnuKumari-MSFT 34,541 Reputation points Microsoft Employee Moderator

2025-04-21T14:31:04.33+00:00

Hi Maroš Urminský ,

Thankyou for sharing the approach you are going to attempt. Kindly let us know if that helped resolve your issue? Thanks
Chris Bale 0 Reputation points

2025-04-21T14:58:30.2+00:00

Hey Maros, since QA and PROD are fine, it could be a DNS or private endpoint issue specific to TEST. Try revalidating the private DNS link and restarting the endpoint. If that doesn’t help, best to contact Databricks support with logs they can check backend issues.

Share via

Databricks Cluster Start-up Fails on TEST Workspace – NPIP_TUNNEL_SETUP_FAILURE

Your answer