11,464 questions with Azure Data Factory tags

Sort by: Updated
0 answers

Pipeline runs issue

There are two linked service connection failures in Azure Data Factory. Since we are currently using Version 1 and planning to upgrade to Version 2, is there anything we should be cautious about before making the change? Do you have any guidelines we can…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-05-02T23:41:50.41+00:00
Stephen 1 Reputation point
commented 2025-05-03T00:47:33.4466667+00:00
J N S S Kasyap 1,705 Reputation points Microsoft External Staff
0 answers

When using the Jira Connector in Azure ADF, how can I save the table list in the dataset?

Hi ... I'm using the Jira Connector linked service in a dataset. The connection is successful, and I can see a list of available tables in the "Table" dataset field. Is there a way to extract this list of tables (to save for looping via…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-05-02T20:59:47.8866667+00:00
Gue, David 0 Reputation points
commented 2025-05-02T21:38:53.1833333+00:00
Chandra Boorla 12,080 Reputation points Microsoft External Staff
0 answers

ADF copy activity from ADLS to delta lake fails due to invalid configuration value detected

Hi, I have a copy activity in ADF to copy data from ADLS to Delta lake. I created linked services with authentication method as Access token. The test connection for linked service is successful. But when I try to execute my pipeline I am getting an…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-30T11:56:25.1533333+00:00
Aswathi Kallinkeel 0 Reputation points
commented 2025-05-02T20:45:19.6166667+00:00
Smaran Thoomu 22,840 Reputation points Microsoft External Staff
1 answer

Automating ADF Unit testing. How to automate the ADF pipelines unit testing. Deployment pipelines are in place.

Hello Microsoft Support team - Am writing this to know the best way to automate the Azure Data Factory pipeline Unit testing automation. As part of our integration project we have 100 plus ADF pipelines and we are in the phase of developing them and as…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-21T14:20:34.5833333+00:00
Firthouse M G 60 Reputation points
commented 2025-05-02T20:37:44.47+00:00
Smaran Thoomu 22,840 Reputation points Microsoft External Staff
0 answers

Data Factory Using Copy data activity with Azure Data Lake Storage Gen 2 & CSV Dataset

Hello: Have been experimenting with DataFactory and the copy data activity. My simple use case is as follows: Files are loaded into a folder under ADLS Gen (Blob Container) say once a week. When received a) copy the files to an AsReceived…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-05-02T19:00:55.7166667+00:00
KBD 80 Reputation points
commented 2025-05-02T19:27:53.13+00:00
J N S S Kasyap 1,705 Reputation points Microsoft External Staff
0 answers

How to fix an established connection was aborted by the server error while making SFTP connection using Azure Synapse?

Hello Team, While we are trying to connect SFTP using SSH public key authentication we are getting following error. Established connection was aborted by the server let me share the steps which we followed so far to connect the sftp. [1] Open PuttyGen…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,318 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2023-02-22T19:41:13.66+00:00
Nikunj Patel 61 Reputation points
commented 2025-05-02T18:02:28.6566667+00:00
JKU 0 Reputation points
1 answer

How can one trigger a fabric data pipeline via ADF

I am new to the ADF world.can someone guide me on how can one trigger a fabric pipeline via ADF in most secure way.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-30T17:13:16.9066667+00:00
prarthana bhat 0 Reputation points
commented 2025-05-02T17:56:29.5933333+00:00
J N S S Kasyap 1,705 Reputation points Microsoft External Staff
2 answers

ADF issues when extracting Snowflake data into Azure Blob Storage

Hi all - I have a pipeline created where I am copying data from Snowflake into an Azure Blob Storage authenticated via an SAS token. I have successfully run the pipeline in the past, however when I added new source datasets I am getting the following…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-29T17:17:06.04+00:00
Yi Han Wong 0 Reputation points
commented 2025-05-02T16:57:20.8033333+00:00
Chandra Boorla 12,080 Reputation points Microsoft External Staff
2 answers

I am trying to pull Sharepoint list into SDP using Pipeline but getting error

I am trying to pull Sharepoint list into SDP using Pipeline but getting error : List Name : 333 Backlog Tracker Requesting response from path 333 Backlog Tracker and query $top=10 failed with status code NotFound and message <?xml…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-25T16:02:48.5566667+00:00
Lohot, Abhishek 0 Reputation points
commented 2025-05-02T16:49:27.0633333+00:00
Sina Salam 19,616 Reputation points
0 answers

I can connect to Jira via ADF's Jira Connector, but I can't pull any data.

Hi, I can connect to Jira via ADF's Jira Connector, but I can't pull any data - I get an error: "Error code # 9602 [Microsoft][JIRA] (62) API Connection Failed. Forbidden Request. HTTP Response code: 401". However, when creating the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-05-01T20:57:44.3566667+00:00
Gue, David 0 Reputation points
edited a comment 2025-05-02T14:53:47.6966667+00:00
Gue, David 0 Reputation points
1 answer

Azure Data Factory - No "body" option for GET method into copy data's REST source

I have a list of parameters for the REST API call. The method I am trying to call API is Copy Data with REST source, method is GET and there is no body section there. So where do I exactly specify these parameters? Header is clearly not meant for this…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2023-06-22T13:50:11.2066667+00:00
Gigi Giorgobiani 5 Reputation points
answered 2025-05-02T14:48:31.95+00:00
JASON WELCH 0 Reputation points
0 answers

I'm trying to do an ADF file consolidator pipeline with dynamic column headers

I am trying to merge multiple files but lets just say its 2. File A with 5 columns (Address Line 1, 2, 3, 4, 5) File B with 7 columns (Address Line 1, 2, 3, 4, 5, Telephone Number, Mobile Number) I expect the output file to have all 7 columns but…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-05-02T12:40:42.5366667+00:00
James Southall 0 Reputation points
commented 2025-05-02T13:17:55.47+00:00
James Southall 0 Reputation points
1 answer

CDC for Gen1 Storage Account

I want to setup a CDC in ADF which automatically checks for changes in a Gen1 Storage Account and pushes the data into Gen2 Storage Account. From the Docs, I see this can be done through a Data Flow, but that needs to have a pipeline which in turn needs…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-29T10:43:02.49+00:00
Sanya Pandey 0 Reputation points Microsoft Employee
commented 2025-05-02T11:10:47.7966667+00:00
Venkat Reddy Navari 1,585 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

Azure Data Factory - fetch and freeze mechanism during pipeline run?

Dear community, Context: I have a Azure Data Factory pipeline linked to Databricks workspace. There are a build and a release CI/CD Azure devops pipelines that automatically update the Databricks notebooks in the Databricks workspace. The CI/CD build…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-29T12:07:38.28+00:00
Shad300 20 Reputation points
commented 2025-05-02T10:55:54.6066667+00:00
Venkat Reddy Navari 1,585 Reputation points Microsoft External Staff
1 answer

Azure Data Factory external call POST request body empty

In Azure Data Factory I am using the "External Call" action to generate a POST request to an external API: ContactColumns call(mapColumn( firstname, lastname ), skipDuplicateMapInputs: true, skipDuplicateMapOutputs:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-28T12:25:34.8933333+00:00
TDay 0 Reputation points
commented 2025-05-02T09:29:15.2333333+00:00
TDay 0 Reputation points
1 answer

Keeping a counter and increment it via ADF

Hello, I currently have the below data and for the column of Days Active, it is a daily counter that keeps incrementing. I currently employ the methos of nullifying the column and them executing a stored proc. but this process takes around 60 mins to be…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-28T14:41:44.9+00:00
sam nick 306 Reputation points
answered 2025-05-02T09:01:35.2566667+00:00
Aswin 392 Reputation points
0 answers

Running Redshift Stored procedure on azure data factory and powerbi

Our vendor just migrated from MSSQL server to redshift. We are attempting to migrate powerbi and data factory; however, we cannot figure out how to "call" stored procedures on redshift.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-30T17:26:27.68+00:00
Tyler Eggett 0 Reputation points
commented 2025-05-02T07:43:27.35+00:00
Smaran Thoomu 22,840 Reputation points Microsoft External Staff
0 answers

Intermittent connectivity issue between Azure Data Factory and Blob Storage — 'remote name could not be resolved'

Hi everyone, For about a week now, I’ve been experiencing an intermittent issue when running a pipeline in Azure Data Factory that writes data to an Azure Blob Storage container. The pipeline sometimes fails with the following error: Operation on target…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-05-01T12:27:07.1366667+00:00
Bangal Camara Kaba 0 Reputation points
commented 2025-05-02T07:43:05.01+00:00
Ganesh Gurram 6,460 Reputation points Microsoft External Staff
1 answer

Azure Data Factory Publish Error After Deleting Unused Pipelines, Dataflows, and Datasets

So i have deleted unused data pipelines, dataflows, and datasets which roughly totals to about 200-300 items. I have also ensured that no existing data pipeline and dataflows are connected to said deleted items. When i go to publish the updates, it…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-04-29T07:04:44.92+00:00
RizkyM 0 Reputation points
commented 2025-05-02T05:28:14.2866667+00:00
phemanth 15,320 Reputation points Microsoft External Staff
1 answer

How to fix Integration Runtime throttling for small Mapping Dataflow?

Hi, I have a Data Pipeline with two activities: a lookup which reads a JSON file and passes the contents to a For Each which runs a data flow. The data flow has 4 transformations: a source (CDM), Identify columns, Alter row and then a sink which writes…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,464 questions
asked 2025-05-01T07:30:07.1533333+00:00
Rhys Doyle 0 Reputation points
edited an answer 2025-05-02T05:17:12.4833333+00:00
phemanth 15,320 Reputation points Microsoft External Staff