There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under subscription

Question

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under subscription

Matthew Ethan Tam 0

I am getting a few errors. Firstly, when I trigger a pipeline, I get the error in the title:

Operation on target file failed: There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under subscription 'xxxx', ActivityId: 'xxxx'.

My synapse workspace is in South Africa North, and I don't have any other pipelines running, and I tried to set concurrency to 1 and it didn't change anything.

Secondly, my Data flow debug doesn't work, it wont allow me to enable it, I have tried all time periods, I get the error

Failed to setup debug session. Session ID:

and when I click on the details of the error I get:

Debug session is already terminated.

I am using AutoResolveIntegrationRuntime as my IR, but oddly I can see the format and get a preview of the files in my storage account, and I have tested the connection of the IR to my storage account and it successfully can access the storage account, and all my managed private endpoints have been approved and configured correctly

Smaran Thoomu 22,840 Reputation points Microsoft External Staff

2025-04-15T11:03:08.58+00:00

@Matthew Ethan Tam We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-16T12:11:59.2633333+00:00

Hello @Matthew Ethan Tam

Type=Microsoft.DataTransfer.Execution.Core.ExecutionException,Message=There

are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime ‘AutoResolveIntegrationRuntime’.

Azure Data Factory enforces certain limits on the number of objects you can create within a single data factory, beyond just quota increases. For example, Microsoft currently sets the total maximum number of objects—such as pipelines, datasets, linked services, and integration runtimes—at 5,000. Additionally, there are more specific limits within each category. For instance, a single pipeline can contain up to 40 activities.

Possible workaround:

One potential solution—depending on availability—is to host our own Self-hosted Integration Runtime (SHIR), which doesn’t have the same limitations as the default runtime.

We can also consider scaling our pipelines either functionally or horizontally across multiple integration runtimes. Horizontally scaling could involve programmatically triggering pipelines built with reusable components, while functional scaling might involve assigning different integration runtimes to different areas of the business. Designing our pipelines with reusability in mind from the beginning makes this much easier and enhances scalability.

Another quick workaround—though not a long-term solution for a growing environment—is to stagger pipeline execution times. If all processes are currently running simultaneously, spreading them out (e.g., some in the early morning and others after business hours) can help reduce runtime pressure.

Regarding the

Failed to setup debug session. Session ID:

It might be due to a caching issue. Try refreshing the Data Factory and then attempt the action again.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-17T07:41:37.34+00:00

@Matthew Ethan Tam
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-21T11:04:24.2+00:00

Hello @Matthew Ethan Tam We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Smaran Thoomu 22,840 Reputation points Microsoft External Staff

2025-04-15T11:03:08.58+00:00

@Matthew Ethan Tam We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-16T12:11:59.2633333+00:00

Hello @Matthew Ethan Tam

Type=Microsoft.DataTransfer.Execution.Core.ExecutionException,Message=There

are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime ‘AutoResolveIntegrationRuntime’.

Azure Data Factory enforces certain limits on the number of objects you can create within a single data factory, beyond just quota increases. For example, Microsoft currently sets the total maximum number of objects—such as pipelines, datasets, linked services, and integration runtimes—at 5,000. Additionally, there are more specific limits within each category. For instance, a single pipeline can contain up to 40 activities.

Possible workaround:

One potential solution—depending on availability—is to host our own Self-hosted Integration Runtime (SHIR), which doesn’t have the same limitations as the default runtime.

We can also consider scaling our pipelines either functionally or horizontally across multiple integration runtimes. Horizontally scaling could involve programmatically triggering pipelines built with reusable components, while functional scaling might involve assigning different integration runtimes to different areas of the business. Designing our pipelines with reusability in mind from the beginning makes this much easier and enhances scalability.

Another quick workaround—though not a long-term solution for a growing environment—is to stagger pipeline execution times. If all processes are currently running simultaneously, spreading them out (e.g., some in the early morning and others after business hours) can help reduce runtime pressure.

Regarding the

Failed to setup debug session. Session ID:

It might be due to a caching issue. Try refreshing the Data Factory and then attempt the action again.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-17T07:41:37.34+00:00

@Matthew Ethan Tam
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-21T11:04:24.2+00:00

Hello @Matthew Ethan Tam We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Hi @Matthew Ethan Tam
Regarding the throttling error:

"There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under subscription…"

Even though you have set the pipeline concurrency to 1, the throttling is at the subscription level, not just pipeline-level concurrency.
This means there could be:

Other dataflows, activities, or services in your subscription consuming capacity.
Mapping Data Flows is a heavy operation and is subject to vCore limits of your Azure Integration Runtime (IR). Even if no other pipelines are running, the capacity of your subscription region may be constrained, especially in high-demand regions like South Africa North.

Suggested Steps:

Review quota limits for Data Flows in Synapse or ADF and consider raising a support ticket to increase them or let us know.
Instead of AutoResolve IR, consider setting up a dedicated Azure IR with reserved capacity to avoid shared throttling.

Regarding the Data Flow Debug issue:

"Failed to setup debug session. Session ID: Debug session is already terminated."

This is often related to the same underlying resource capacity issue. When the region or subscription is experiencing throttling, debug clusters can fail to start or terminate prematurely.

Suggested Steps:

Check quota limits for the debug cluster in Synapse (especially if you’re using AutoResolve IR, which depends on shared compute).
Try switching to a different Integration Runtime if possible.

Given both symptoms (throttling and debug failure), this looks like a regional capacity constraint or subscription-level quota issue. I recommend:

Reviewing quotas.
Testing with a dedicated IR.
It is advisable to investigate the capacity and quota limits within your subscription and region.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

AnnuKumari-MSFT 34,466 Reputation points Microsoft Employee

2025-04-28T06:18:18.3+00:00

Hi @Matthew Ethan Tam ,

I understand you are getting the throttling error:

"There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under subscription…"

It usually comes up when You've reached the integration runtime's capacity limit, or , running a large amount of data flow by using the same integration runtime at the same time might also cause this issue.

However, you have already stated that you are only running one pipeline at a time.

Kindly try starting a debug cluster with custom IR by increasing the compute size and creating IR with large cluster and TTL time for dataflow , also kindly try Dynamically size data flow compute at runtime

Instead of debug, try triggering the dataflow and see if it works.

Hope it helps. Kindly accept the answer by clicking as Accept answer button. Thankyou
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-29T11:04:54.8166667+00:00

Hello @Matthew Ethan Tam We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 5 Reputation points Microsoft External Staff

2025-04-30T11:48:26.0466667+00:00

Hello @Matthew Ethan Tam We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under subscription

1 answer

Your answer