Hello Winnie Yap,
Thanks for posting your question in the Microsoft Q&A forum.
- You can combine Azure Cost Management budgets, Azure Automation runbooks, and the Databricks PowerShell module.
- Set Up a Cost Budget and Trigger: In Azure Cost Management, create a budget for your subscription or resource group, and configure an action group that triggers when your budget threshold is reached.
- Create an Azure Automation Runbook: In the Azure Portal, create an Azure Automation account if you don’t already have one.
- Install the Databricks PowerShell Module: Use the
DatabricksPS
module to manage Databricks resources via API - Write the PowerShell Script to Stop Databricks Clusters: Here’s a sample script you can use in your runbook. You’ll need your Databricks workspace URL and a personal access token
$accessToken = "<Your_Databricks_PAT>" $apiUrl = "https://<your-region>.azuredatabricks.net" Set-DatabricksEnvironment -AccessToken $accessToken -ApiRootUrl $apiUrl $clusters = Get-DatabricksCluster | Where-Object { $_.state -eq "RUNNING" } foreach ($cluster in $clusters) { Stop-DatabricksCluster -ClusterId $cluster.cluster_id }
- Automate the Trigger: When your cost budget threshold is reached, Azure will trigger your Automation runbook via the action group and webhook you set up in step 1
Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful