I am building a simple pipeline to copy data from an on prem to ADLS gen2. I am using a copy activity with redshift as a source inside a for each loop for a given table array.

Question

I am building a simple pipeline to copy data from an on prem to ADLS gen2. I am using a copy activity with redshift as a source inside a for each loop for a given table array.

Tushar Singh 0

The error is the following -

Failure happened on 'Source' side. ErrorCode=UserErrorUnclassifiedError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Odbc Operation Failed.,Source=Microsoft.DataTransfer.ClientLibrary.Odbc.OdbcConnector,''Type=Microsoft.DataTransfer.ClientLibrary.Odbc.Interop.OdbcException,Message=ERROR [57014] [Microsoft][Amazon Redshift] (30) Error occurred while trying to execute a query: [SQLState 57014] ERROR: Query (361084429) cancelled on user's request

,Source=Microsoft.DataTransfer.ClientLibrary.Odbc.Wrapper,'

Clarifications - I am doing the sequential activity for the loop because of SHIR limited memory, and it all runs successfully except the last run. But for the same table when I run an individual copy activity then it runs without any error.

phemanth 15,490 Reputation points Microsoft External Staff Moderator

2025-05-07T04:17:32.6766667+00:00

@Tushar Singh We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Tushar Singh 0 Reputation points

2025-05-07T09:19:36.2166667+00:00

@phemanth Yes the issue is resolved. We couldn't change the timeout configurations since it is managed by a third party. So we just changes the compression technique to snappy instead of gzip which is much faster and increased the integration units to 128 which reduced the query run times. Thanks for the response.
phemanth 15,490 Reputation points Microsoft External Staff Moderator

2025-05-08T04:02:34.9833333+00:00

@Tushar Singh Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

2 answers

Your answer

phemanth 15,490 Reputation points Microsoft External Staff Moderator

2025-05-07T04:17:32.6766667+00:00

@Tushar Singh We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Tushar Singh 0 Reputation points

2025-05-07T09:19:36.2166667+00:00

@phemanth Yes the issue is resolved. We couldn't change the timeout configurations since it is managed by a third party. So we just changes the compression technique to snappy instead of gzip which is much faster and increased the integration units to 128 which reduced the query run times. Thanks for the response.
phemanth 15,490 Reputation points Microsoft External Staff Moderator

2025-05-08T04:02:34.9833333+00:00

@Tushar Singh Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

@Tushar Singh

It looks like you're experiencing an issue with your Azure Data Factory (ADF) pipeline, specifically when copying data from Amazon Redshift to Azure Data Lake Storage (ADLS) Gen2. The error indicates that the query is being canceled, which can sometimes happen due to timeouts or execution limits.

To troubleshoot this issue:

Check Query Timeout Settings: It’s possible that the query you're attempting to execute is hitting a timeout limit. Make sure to review the query settings in Redshift and consider increasing the timeout if necessary.

Monitor Resource Utilization: Since you've mentioned that you're using a Self-hosted Integration Runtime (SHIR) with limited memory, ensure that there are sufficient resources available. Monitor the memory and CPU usage during the execution of your pipeline.

Log Redshift Execution: Try to log the executed query in Redshift to see if it provides any additional insights into why it might be canceling. If the query takes too long to execute, you might need to optimize it.

Sequential Processing: You mentioned you're running a sequential activity due to memory constraints. Ensure that the previous iterations are indeed completing successfully before the next one starts. This might help in establishing some pace and avoiding memory overload.

Debugging: If running individual copy activities works without a hitch, try isolating the last iteration to see if specific data in that table is causing issues. You could perform a simpler query or a sample operation just to confirm if it’s related to the data itself.

If these steps don't resolve the issue, here are some follow-up questions that may help narrow it down:

What is the size of the data you are attempting to copy in the last iteration?
Are there any specific queries or parameters you are using in this particular copy activity that differ from the others?
What timeout settings are currently configured in your Redshift instance?
Have you looked at Redshift logs to check for any other errors or clues related to the canceled query?
Are there any network latency issues that might affect the integration between SHIR and Redshift?

phemanth 15,490 Reputation points Microsoft External Staff Moderator

2025-05-09T05:35:59.18+00:00

@Tushar Singh Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Tushar Singh 0

The following solution worked for us given the constraints we had:

Changed the compression technique to snappy instead of gzip. It is much faster compressor than gzip
Increase the integration units count to reduce query run time.

Share via

I am building a simple pipeline to copy data from an on prem to ADLS gen2. I am using a copy activity with redshift as a source inside a for each loop for a given table array.

2 answers

Your answer