How to export Artifacts ARM Template for a Azure Synapse Workspace?

Question

How to export Artifacts ARM Template for a Azure Synapse Workspace?

Napsty Dev 20

Is there a possible way to export an "Artifacts ARM Template" from a specific resource?

I have been trying to implement an Azure devops CI/CD Pipeline that migrates artifacts (Pipeline, dataflows and datasets) between Synpase workspaces (i.e. dev, qa, prod) across multiple workspaces without using any Git configuration.

Screenshot 2025-04-21 at 12.17.32 PM

I have already set up this CI/CD system with Git configuration, where I use both TemplateForWorkspace.json and ParametersTemplateForWorkspace.json to deploy into a different Synapse workspace. These files are auto-generated by Synapse’s workspace publish branch feature.

So far, I haven’t been able to replicate these two files, which are absolutely necessary to perform a deployment. I am attempting this because I prefer to use only Synapse Live mode, without enabling Git integration.

The only thing I've managed to obtain so far is a template fetched via an HTTPS request using Azure's REST API. However, this ARM template does NOT include any artifacts, meaning that even if deployed, the Synapse workspace won’t contain the original resource content/artifacts.

ARM Template fetched via HTTPS request:

{
    "template": {
        "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
        "contentVersion": "1.0.0.0",
        "parameters": {
            "vulnerabilityAssessments_Default_storageContainerPath": {
                "type": "SecureString"
            },
            "workspaces_synapse_dev_napsty_name": {
                "defaultValue": "synapse-dev-napsty",
                "type": "String"
            },
            "storageAccounts_napdatalake_externalid": {
                "defaultValue": "/subscriptions/8be7d968-ce5d-402f-84f0-ba21765d77c1/resourceGroups/devrg/providers/Microsoft.Storage/storageAccounts/napdatalake",
                "type": "String"
            }
        },
        "variables": {},
        "resources": [
            {
                "type": "Microsoft.Synapse/workspaces",
                "apiVersion": "2021-06-01",
                "name": "[parameters('workspaces_synapse_dev_napsty_name')]",
                "location": "australiaeast",
                "identity": {
                    "type": "SystemAssigned"
                },
                "properties": {
                    "defaultDataLakeStorage": {
                        "resourceId": "[parameters('storageAccounts_napdatalake_externalid')]",
                        "createManagedPrivateEndpoint": false,
                        "accountUrl": "https://napdatalake.dfs.core.windows.net",
                        "filesystem": "napdatalakefs"
                    },
                    "encryption": {},
                    "managedResourceGroupName": "synapseworkspace-managedrg-22275c84-71ba-412c-ac0c-90b1a6f93b4b",
                    "sqlAdministratorLogin": "sqladminuser",
                    "privateEndpointConnections": [],
                    "workspaceRepositoryConfiguration": {
                        "accountName": "napstyy0227",
                        "collaborationBranch": "dev",
                        "lastCommitId": "d2c9d624ec8fd345ef2f8acc84ab64ecfd6c3d79",
                        "projectName": "AzureSynapse",
                        "repositoryName": "AzureSynapse",
                        "rootFolder": "/synapse",
                        "tenantId": "63ec89f0-780b-4349-8a65-b8c44dddc8dd",
                        "type": "WorkspaceVSTSConfiguration"
                    },
                    "publicNetworkAccess": "Enabled",
                    "cspWorkspaceAdminProperties": {
                        "initialWorkspaceAdminObjectId": "94888bea-0df0-4bda-8c43-988d03fb0323"
                    },
                    "azureADOnlyAuthentication": false,
                    "trustedServiceBypassEnabled": false
                }
            },
            {
                "type": "Microsoft.Synapse/workspaces/auditingSettings",
                "apiVersion": "2021-06-01",
                "name": "[concat(parameters('workspaces_synapse_dev_napsty_name'), '/Default')]",
                "dependsOn": [
                    "[resourceId('Microsoft.Synapse/workspaces', parameters('workspaces_synapse_dev_napsty_name'))]"
                ],
                "properties": {
                    "retentionDays": 0,
                    "auditActionsAndGroups": [],
                    "isStorageSecondaryKeyInUse": false,
                    "isAzureMonitorTargetEnabled": false,
                    "state": "Disabled",
                    "storageAccountSubscriptionId": "00000000-0000-0000-0000-000000000000"
                }
            },
            {
                "type": "Microsoft.Synapse/workspaces/azureADOnlyAuthentications",
                "apiVersion": "2021-06-01",
                "name": "[concat(parameters('workspaces_synapse_dev_napsty_name'), '/default')]",
                "dependsOn": [
                    "[resourceId('Microsoft.Synapse/workspaces', parameters('workspaces_synapse_dev_napsty_name'))]"
                ],
                "properties": {
                    "azureADOnlyAuthentication": false
                }
            },
            {
                "type": "Microsoft.Synapse/workspaces/dedicatedSQLminimalTlsSettings",
                "apiVersion": "2021-06-01",
                "name": "[concat(parameters('workspaces_synapse_dev_napsty_name'), '/default')]",
                "location": "australiaeast",
                "dependsOn": [
                    "[resourceId('Microsoft.Synapse/workspaces', parameters('workspaces_synapse_dev_napsty_name'))]"
                ],
                "properties": {
                    "minimalTlsVersion": "1.2"
                }
            },
            {
                "type": "Microsoft.Synapse/workspaces/extendedAuditingSettings",
                "apiVersion": "2021-06-01",
                "name": "[concat(parameters('workspaces_synapse_dev_napsty_name'), '/Default')]",
                "dependsOn": [
                    "[resourceId('Microsoft.Synapse/workspaces', parameters('workspaces_synapse_dev_napsty_name'))]"
                ],
                "properties": {
                    "retentionDays": 0,
                    "auditActionsAndGroups": [],
                    "isStorageSecondaryKeyInUse": false,
                    "isAzureMonitorTargetEnabled": false,
                    "state": "Disabled",
                    "storageAccountSubscriptionId": "00000000-0000-0000-0000-000000000000"
                }
            },
            {
                "type": "Microsoft.Synapse/workspaces/firewallRules",
                "apiVersion": "2021-06-01",
                "name": "[concat(parameters('workspaces_synapse_dev_napsty_name'), '/allowAll')]",
                "dependsOn": [
                    "[resourceId('Microsoft.Synapse/workspaces', parameters('workspaces_synapse_dev_napsty_name'))]"
                ],
                "properties": {
                    "startIpAddress": "0.0.0.0",
                    "endIpAddress": "255.255.255.255"
                }
            },
            {
                "type": "Microsoft.Synapse/workspaces/integrationruntimes",
                "apiVersion": "2021-06-01",
                "name": "[concat(parameters('workspaces_synapse_dev_napsty_name'), '/AutoResolveIntegrationRuntime')]",
                "dependsOn": [
                    "[resourceId('Microsoft.Synapse/workspaces', parameters('workspaces_synapse_dev_napsty_name'))]"
                ],
                "properties": {
                    "type": "Managed",
                    "typeProperties": {
                        "computeProperties": {
                            "location": "AutoResolve"
                        }
                    }
                }
            },
            {
                "type": "Microsoft.Synapse/workspaces/securityAlertPolicies",
                "apiVersion": "2021-06-01",
                "name": "[concat(parameters('workspaces_synapse_dev_napsty_name'), '/Default')]",
                "dependsOn": [
                    "[resourceId('Microsoft.Synapse/workspaces', parameters('workspaces_synapse_dev_napsty_name'))]"
                ],
                "properties": {
                    "state": "Disabled",
                    "disabledAlerts": [
                        ""
                    ],
                    "emailAddresses": [
                        ""
                    ],
                    "emailAccountAdmins": false,
                    "retentionDays": 0
                }
            },
            {
                "type": "Microsoft.Synapse/workspaces/vulnerabilityAssessments",
                "apiVersion": "2021-06-01",
                "name": "[concat(parameters('workspaces_synapse_dev_napsty_name'), '/Default')]",
                "dependsOn": [
                    "[resourceId('Microsoft.Synapse/workspaces', parameters('workspaces_synapse_dev_napsty_name'))]"
                ],
                "properties": {
                    "recurringScans": {
                        "isEnabled": false,
                        "emailSubscriptionAdmins": true
                    },
                    "storageContainerPath": "[parameters('vulnerabilityAssessments_Default_storageContainerPath')]"
                }
            }
        ]
    }
}

Accepted answer

0 additional answers

Your answer

Answer 1

J N S S Kasyap 1,715 Microsoft External Staff

Hi @Napsty Dev

In Azure Synapse, without Git integration, there is currently no built-in way to export all workspace artifacts (pipelines, datasets, data flows, etc.) as a full ARM template like the TemplateForWorkspace.json you get from a publish branch.

The Export ARM template option (via REST API or Azure Portal) only exports infrastructure-level settings (workspace configuration, firewall rules, etc.), not artifacts.

Another alternative is to use Synapse REST APIs to extract individual artifacts like pipelines, datasets, and notebooks from your live workspace. This approach is more manual and involves calling different endpoints for each artifact type, then storing the JSON definitions yourself. While this method doesn't give you a full ARM template, you could potentially script a process to package these definitions and deploy them using the Synapse SDK or management APIs. However, it lacks the automation and structure of the Git-based publish process and is not recommended for large or complex projects unless Git integration is not an option.

Using Git integration also enables version control, collaboration, rollback, and audit capabilities, which are essential for managing complex data solutions in a team environment. While alternatives like REST API extraction exist, they lack the automation, structure, and full artifact coverage that Git + Publish provides. If you're aiming for reliability, scalability, and maintainability in your CI/CD process, Git integration is the most robust and production-ready choice.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Napsty Dev 20 Reputation points

2025-04-21T20:53:10.39+00:00

Hi @J N S S Kasyap ,Thank you so much for your prompt and detailed response — I really appreciate the clarity and depth you provided.

Just to clarify: I’ve already implemented Git integration and successfully used the publish branch mechanism to generate and deploy the TemplateForWorkspace.json and ParametersTemplateForWorkspace.json files in other CI/CD scenarios. However, for this particular case, I’m specifically required to avoid Git integration and work solely with the Live mode in Synapse.

That’s why your recommendation to extract artifacts individually using the Synapse REST API is especially valuable in this context. Would you happen to have any official documentation or examples that detail how to script or automate the extraction of pipelines, datasets, dataflows, etc., via REST calls?

Once again, thanks for your quick response and helpful insights — I’ve upvoted your answer to help others in a similar situation.

Best regards,

Napsty Dev
J N S S Kasyap 1,715 Reputation points Microsoft External Staff

2025-04-22T08:21:57.4266667+00:00
@Napsty Dev

To export Azure Synapse pipeline artifacts as an ARM template without using Git integration, you can utilize the Synapse REST APIs to programmatically extract individual artifacts such as pipelines, datasets, and data flows. For instance, to retrieve a list of data flows, you can send a GET request to the endpoint:

Common endpoints include:

GET https://<workspace>.dev.azuresynapse.net/pipelines?api-version=2021-06-01 for pipelines.

GET https://<workspace>.dev.azuresynapse.net/datasets?api-version=2021-06-01 for datasets.

GET https://<workspace>.dev.azuresynapse.net/dataflows?api-version=2021-06-01 for data flows.

https://learn.microsoft.com/en-us/rest/api/synapse/

To automate this process:

Authenticate using an Azure AD token or service principal.

Use a scripting language like Python or PowerShell to iterate over these endpoints.

Save the retrieved JSON definitions to your preferred storage.

While this method allows for the extraction of individual artifacts, it requires manual handling and scripting to manage dependencies and deployment sequences.

Note that this method extracts individual artifact definitions, not a single comprehensive ARM template like TemplateForWorkspace.json generated via Git integration. Combining these definitions into a full ARM template requires manual scripting to handle dependencies and deployment sequences.

I hope this info helpful. Please do let us know if you have any further queries.
Napsty Dev 20 Reputation points

2025-04-24T18:56:28.6166667+00:00
Thank you @J N S S Kasyap for the detailed response!

I’ve already tested the Synapse REST API and successfully pulled the main artifacts like pipelines, datasets, and dataflows using the respective endpoints.

However, I’ve noticed that there are also secondary artifacts—not only under the "Develop" section (such as SQL scripts and Notebooks), but also within categories like pipelines, datasets, and dataflows—that are not returned through these API calls.

For example, when I run the following request:

GET https://synapse-dev-napsty.dev.azuresynapse.net/dataflows?api-version=2021-06-01

I get the following response:

{ "value": [] }

Despite having visible SQL Scripts in Synapse Live mode.

This leads me to think that some artifacts, possibly those that aren't exactly a pipeline, a dataset or a dataflow, are not exposed via the current public API endpoints. I’m trying to find a way to extract all artifacts, both primary and secondary, that are required to rebuild the workspace in a CI/CD scenario—without using Git integration.

Is there any known method (API or alternative) to programmatically extract all Synapse artifacts, including those secondary ones that may not be listed via the standard endpoints?
J N S S Kasyap 1,715 Reputation points Microsoft External Staff

2025-04-28T16:55:51.29+00:00

@Napsty Dev
Apologies for the delayed response.

In Azure Synapse Analytics, primary artifacts such as pipelines, datasets, and dataflows can be accessed easily through the public Synapse REST APIs. However, secondary artifacts like SQL scripts, KQL scripts, notebooks, and Spark job definitions are not exposed via these standard APIs.
These secondary artifacts are treated as development workspace resources and are not part of the published Synapse metadata. Instead, they are internally stored in a hidden Azure Data Lake Storage Gen2 container associated with the Synapse workspace.

Because of this design, there is currently no official API method to extract all artifacts, including secondary ones, directly. The recommended and supported method to capture artifacts, both primary and secondary, is to enable Git integration in Synapse.

Git integration ensures that all artifacts are saved into a repository, allowing for easy export and CI/CD setup.
Without Git, the only alternative is an unofficial workaround that involves accessing the Synapse internal storage account directly and extracting the files manually, which is risky, unsupported, and prone to breaking if Microsoft changes Synapse’s internal structure. Therefore, for a reliable and future-proof solution, setting up Git integration remains the best approach to fully extract and manage all Synapse workspace artifacts.

I hope this info helps, But any feel free to reach out in the comments I am happy to help further.

Napsty Dev 20

Thanks @J N S S Kasyap for the detailed explanation — that clears things up.

Let’s suppose I’ve decided to step away from the workaround solution without Git. In fact, I’ve already implemented a full CI/CD deployment system using an artifacts transformer and a Deployment pipeline within Azure DevOps (sharing both code snippets below).

Artifacts Transformer Pipeline:

trigger: 
  branches:
    include:
      - dev

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: artifactsTransformer-config

steps:
  - checkout: self
    persistCredentials: true
    fetchDepth: 0

  - task: Bash@3
    displayName: 'Copy and replace names in files'
    env: 
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    inputs:
      targetType: 'inline'
      script: |
        git config --global user.email "$(gitUserEmail)"
        git config --global user.name "$(gitUserName)"
        git remote set-url origin "$(repoUrl)"
        git checkout dev
        git pull origin dev
        rm -rf $(qaFolder)
        cp -r $(devFolder) $(qaFolder)
        find $(qaFolder) -type f -exec sed -i "s/$(devFolder)/$(qaFolder)/g" {} +
        git add $(qaFolder)
        git commit -m "$(commitMessage)"
        git push origin dev
        git fetch origin
        git checkout qa
        git reset --hard origin/dev
        git push --force origin qa

Deployment Pipeline:

trigger:
  branches:
    include:
      - qa

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: goat-config

steps:
  - task: Synapse workspace deployment@2
    inputs:
      operation: 'deploy'
      TemplateFile: '$(Build.SourcesDirectory)$(templateFilePath)'
      ParametersFile: '$(Build.SourcesDirectory)$(parametersFilePath)'
      azureSubscription: '$(azureSubscription)'
      ResourceGroupName: '$(resourceGroupName)'
      TargetWorkspaceName: '$(workspaceName)'
      DeleteArtifactsNotInTemplate: true
      DeployManagedPrivateEndpoints: false
      FailOnMissingOverrides: false
      Environment: 'prod'
      npmpackage: 'prod'

This is the first time I’ve done something like this — do you agree with my setup? What would you improve if not? I’d really appreciate your thoughts or best practices for setting up a Git-integrated Synapse workflow between workspaces :)

Also, what about the scenario where other teammates need to switch to the Git-integrated branch? Their live Synapse workspace will stop showing the resources, right?

Do they need to migrate everything from the live Synapse workspace to the Git branch as a one-time setup and if so, how to do it? Is this the necessary and recommended way to proceed?

Thanks in advance!

Share via

How to export Artifacts ARM Template for a Azure Synapse Workspace?

Is there a possible way to export an "Artifacts ARM Template" from a specific resource?

0 additional answers

Your answer