2,413 questions with Azure Databricks tags

Sort by: Updated
0 answers

If i choose no to the question "Deploy Azure Databricks workspace in your own Virtual Network (Vnet)" is it possible to later change the workspace to be deployed into my own Vnet

If i choose no to the question "Deploy Azure Databricks workspace in your own Virtual Network (Vnet)" is it possible to later change the workspace to be deployed into my own Vnet Thanks

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-05-04T10:20:39.4333333+00:00
Admin, BenV 0 Reputation points
0 answers

Databricks driver worker node connectivity issues inside workflow jobs

Databricks driver worker node connectivity issues inside workflow jobs. Run failed with error message  Cluster '0428-144208-eeb122m2' was terminated. Reason: DRIVER_UNREACHABLE (SERVICE_FAULT). Parameters: databricks_error_message:Terminating cluster due…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-05-02T17:56:50.4266667+00:00
Sergey Strelnikov 0 Reputation points
commented 2025-05-02T18:38:22.1233333+00:00
Chandra Boorla 12,085 Reputation points Microsoft External Staff
1 answer

Databricks Cluster no access to on-prem server, while serverless has access

In Azure Databricks, when using serverless we have connectivity access to our on-prem server, but when using cluster it'll encounter TimeoutError: [Errno 110] Connection timed out. Our Databricks contact, suggested below. Can you help expand with more…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-30T17:00:30.1266667+00:00
Alex Young 0 Reputation points
commented 2025-05-02T16:58:05.22+00:00
Chandra Boorla 12,085 Reputation points Microsoft External Staff
0 answers

Databricks Spark SQL Query Error from Sigma

MDB SQL ERROR.txt We need some help troubleshooting and issues we are having. One of our queries is not running in sigma, but the error is coming from the Databricks side. Here is the error message below. Databricks: execution error: failed to execute…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-24T23:34:36.28+00:00
Keith Wong 0 Reputation points
commented 2025-05-02T08:47:42.42+00:00
Ganesh Gurram 6,460 Reputation points Microsoft External Staff
0 answers

How to run pysftp connection using serverless compute in Azure Databricks

Hello Community, The code below ran successfully using all purpose cluster, but when I switched to serverless compute, I ran into exception: connection timed out. Usually it would take around 2 mins and then failed. I searched online about this issue and…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-30T02:50:20.1633333+00:00
Quan Sun 21 Reputation points
edited a comment 2025-05-02T04:29:43.3333333+00:00
phemanth 15,320 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

Unable to Create Databricks Cluster: "Error: Cloud provide resource Stockout " on Azure for Students

I'm getting the same error when I try to create a cluster in Databricks to run Spark jobs. As suggested, I deleted the Databricks, deployed in a different location, and tried. But I'm still facing the same issue. Please help me create a VM with…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-29T02:17:49.9366667+00:00
Chandravallika Murarisetty 20 Reputation points
accepted 2025-05-01T13:11:06.4833333+00:00
Chandravallika Murarisetty 20 Reputation points
2 answers

Design Pattern Options for Data Ingestion into Databricks

Dear MS Azure Databrick Community, Currently, we are in the process of formulating several design pattern options for the ingestion of data (both structured and semi-structured) into Databricks.  We have come up with the following options, and we would…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2023-09-13T22:07:59.9333333+00:00
Suleman Butt 1 Reputation point
answered 2025-05-01T06:19:24.01+00:00
Mahipal Nehra 0 Reputation points
0 answers

Not able to set up unity catalogue

Absolutely! Here's a detailed and professional description you can use to raise a support ticket with Azure, explaining: The core issue Environment details Steps you've taken Microsoft documentation and community workarounds already attempted…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-23T18:37:21.93+00:00
prem kaushik 0 Reputation points
commented 2025-04-30T11:44:53.1066667+00:00
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

How to select performance mode for Azure Databricks Delta Live Tables

Hi everyone, Based on the official link, For triggered pipelines, you can select the serverless compute performance mode using the Performance optimized setting in the pipeline scheduler. When this setting is disabled, the pipeline uses standard…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-28T01:08:12.8266667+00:00
Quan Sun 21 Reputation points
accepted 2025-04-30T03:27:19.8666667+00:00
Quan Sun 21 Reputation points
1 answer One of the answers was accepted by the question author.

how to stop databricks using powershell code

Hi Everyone, I would like to execute a runbook that checks the total costs and shuts down Azure databricks when the costs exceed a certain threshold. I think this can be done using powershell code but I am not sure. Can someone please offer some advice…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-28T16:03:44.38+00:00
Winnie Yap 20 Reputation points
accepted 2025-04-29T12:13:49.5833333+00:00
Winnie Yap 20 Reputation points
1 answer

Target system temporarily unavailable, Please try again later. [Databricks][DatabricksJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: HTTP retry after response received with no Retry-After header, error: HTTP Response

Our SAAS Connector through Serverless SQL Warehouse is getting this error trying to write to Azure Databricks table: Target system temporarily unavailable, Please try again later. [Databricks][DatabricksJDBCDriver] Communication link failure. Failed to…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-25T00:13:07.9466667+00:00
MDB Admin 0 Reputation points
commented 2025-04-29T06:29:23.56+00:00
Smaran Thoomu 22,840 Reputation points Microsoft External Staff
0 answers

Model Deployment Error – “Must specify a chain type in config” while serving LLM-based Chatbot via Unity Catalog

Hello Community, I’m encountering an issue while deploying a simple LLM-based chatbot on Azure Databricks using Unity Catalog. The model works perfectly within the notebook — the retriever, prompt chain, and response generation all function correctly…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-22T06:26:47.56+00:00
Muhammad Salman Khan 0 Reputation points
commented 2025-04-28T04:57:48.27+00:00
phemanth 15,320 Reputation points Microsoft External Staff
1 answer

and have either the "Workspace access" or the "Databricks SQL access" entitlement, which is required to access this page.

I have created my personal Azure account using a gmail and a databricks resource group. I then created a workspace in us west 2. When i try to lauch the workspace i get this error - " ... and have either the "Workspace access" or the…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-23T18:55:37.0733333+00:00
Uma Lakshman 0 Reputation points
edited a comment 2025-04-25T18:35:44.3633333+00:00
Chandra Boorla 12,085 Reputation points Microsoft External Staff
1 answer

Spark_Ambiguous_Executor_MaxExecutorFailures

Hi I'm running a scheduled multiple run notebook using the below configuration, but I keep getting the below error DAG = { "activities": [ { "name": "Notebook", "path":…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2024-05-06T08:06:10.67+00:00
Seelan 0 Reputation points
commented 2025-04-25T11:45:22.56+00:00
Lindbo Josefine 0 Reputation points
1 answer

how can i setup iceberg in databric

I have a simple pay-as-you-go for learning purposes. I am trying to simulate Adobe Clickstream in Azure Databricks and furthermore trying to create an Iceberg table under Unity Catalog. Tried to utilize assisted AI to accomplish goal to no avail. I…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-22T15:39:29.0433333+00:00
Kirk Anderson 0 Reputation points
commented 2025-04-24T16:11:38.7933333+00:00
Chandra Boorla 12,085 Reputation points Microsoft External Staff
1 answer

Upon connecting to sqlserver from Databricks getting error: Can't open lib 'ODBC Driver 17 for SQL Server'

In our Databricks notebook: We have done: %python Install the ODBC Driver 17 for SQL Server %pip install pyodbc %pip install --upgrade 'databricks-connect>=7.3' Then trying to connect to SQL Server using: conn = pyodbc.connect( …

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-14T13:39:56.4666667+00:00
Immaneni, Nanda 20 Reputation points
commented 2025-04-22T11:23:52.6833333+00:00
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff
0 answers

Missing Genie - Upload File Feature in Preview Section

Despite having admin privileges for both the workspace and Genie Workspace in Azure Databricks we are unable to see the "Genie - Upload File" feature under the Preview section, even though the documentation indicates it should be available. We…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-16T05:57:38.7766667+00:00
Bangaru Bhavya Sree 0 Reputation points
commented 2025-04-22T06:45:12.7533333+00:00
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff
0 answers

One of our Databricks instances constantly has "BOOT STRAP FAILURE"

We have multiple Databricks instances across multiple subsciptions, one of our instances in one subscription constantly has provisioning failures, timing out while waiting for VM's to either provision or attach storage. This is in US West 2, not sure if…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-16T16:43:49.3166667+00:00
Steven Morris 0 Reputation points
commented 2025-04-21T16:04:44.0666667+00:00
Chandra Boorla 12,085 Reputation points Microsoft External Staff
1 answer

Use of Azure Databricks CLI

We have Azure Databricks configured in Azure platform and we want to use Datasbricks CLI. It will be installed on user devices and allows them to work with Databricks. Generally when someone has access to a Databricks workspace, has been authenticated by…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-17T09:09:58.0266667+00:00
paromita ghosh 0 Reputation points
commented 2025-04-21T15:10:21.28+00:00
Smaran Thoomu 22,840 Reputation points Microsoft External Staff
0 answers

Databricks Cluster Start-up Fails on TEST Workspace – NPIP_TUNNEL_SETUP_FAILURE

Hello, We are experiencing an issue with our Databricks TEST workspace where clusters fail to start. The error message indicates an NPIP tunnel setup failure. All our workspaces (TEST, QA, and PROD) are configured with No Public IP access enabled and…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,413 questions
asked 2025-04-07T13:38:47.85+00:00
Maroš Urminský 0 Reputation points
commented 2025-04-21T14:58:30.2+00:00
Chris Bale 0 Reputation points