11,476 questions with Azure Data Factory tags

Sort by: Updated
2 answers

I am building a simple pipeline to copy data from an on prem to ADLS gen2. I am using a copy activity with redshift as a source inside a for each loop for a given table array.

The error is the following - Failure happened on 'Source' side. ErrorCode=UserErrorUnclassifiedError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Odbc Operation…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-06T07:27:43.5466667+00:00
Tushar Singh 0 Reputation points
commented 2025-05-08T04:02:34.9833333+00:00
phemanth 15,485 Reputation points Microsoft External Staff
0 answers

How do I set up an ADLS Gen 2 Linked Service in Azure Data Factory with a SAS URI?

Hi, I am able to set up a linked service to a blob with no problem, and I am actually able to set up a linked service to this current ADLS gen 2 container by treating it as a blob, but when I try to treat it as ADLS gen 2 via the "set up linked…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-07T16:58:30.8+00:00
Neal Lockhart 0 Reputation points
commented 2025-05-08T03:46:41.91+00:00
J N S S Kasyap 1,960 Reputation points Microsoft External Staff
1 answer

Azure Function Forbidden Access to Key Vault despite Appropriate IAM Permission and Firewall Settings

I am encountering an issue where my Azure Function is being called from Azure Data Factory (ADF) and needs to access an Azure Key Vault. Setup: The Azure Function uses a System Assigned Managed Identity. It has been granted Key Vault Secrets User role…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-07T16:18:49.57+00:00
Phillip Blackwell 0 Reputation points
commented 2025-05-07T20:36:44.9433333+00:00
Chandra Boorla 12,110 Reputation points Microsoft External Staff
1 answer

When using the Jira Connector in Azure ADF, how can I save the table list in the dataset?

Hi ... I'm using the Jira Connector linked service in a dataset. The connection is successful, and I can see a list of available tables in the "Table" dataset field. Is there a way to extract this list of tables (to save for looping via…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-02T20:59:47.8866667+00:00
Gue, David 30 Reputation points
commented 2025-05-07T20:02:36.05+00:00
Gue, David 30 Reputation points
2 answers One of the answers was accepted by the question author.

Pipeline runs issue

There are two linked service connection failures in Azure Data Factory. Since we are currently using Version 1 and planning to upgrade to Version 2, is there anything we should be cautious about before making the change? Do you have any guidelines we can…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-02T23:41:50.41+00:00
Stephen 6 Reputation points
accepted 2025-05-07T17:08:41.3133333+00:00
Stephen 6 Reputation points
0 answers

Azure Data Factory - Internal Server Error

Hi, I'm using Azure Data Factory. I'm unable to run pipeline since yesterday. I've the following error: { "code": "InternalServerError", "message": null, "target":…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-07T13:04:52.7933333+00:00
Mickael GERVAIS (Externe) 0 Reputation points
commented 2025-05-07T16:59:32.8333333+00:00
Chandra Boorla 12,110 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

Google Big Query not working on self hosted integration runtime

Google Big Query is not working on self hosted integration runtime v5.48.9106.2. When I run a query it returns the following error: Method not found: 'Microsoft.DI.MSDISDKConnectorInterfaces.DataObjects.Condition…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-05T11:29:38.4033333+00:00
Jonty 25 Reputation points
accepted 2025-05-07T13:21:14.5033333+00:00
Jonty 25 Reputation points
0 answers

Im unable to get the dataset information from power bi workspace using adf

Hi Im unable to get the dataset or workspace information from power bi using adf web activity error message : Invoking endpoint failed with HttpStatusCode - '404 : NotFound', message - 'The requested endpoint(url) does not exist on the server. Please…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-07T11:52:03.9333333+00:00
Radha Jahnavi Pusarapu 0 Reputation points
commented 2025-05-07T12:40:48.6166667+00:00
Venkat Reddy Navari 1,585 Reputation points Microsoft External Staff
0 answers

I'm trying to do an ADF file consolidator pipeline with dynamic column headers

I am trying to merge multiple files but lets just say its 2. File A with 5 columns (Address Line 1, 2, 3, 4, 5) File B with 7 columns (Address Line 1, 2, 3, 4, 5, Telephone Number, Mobile Number) I expect the output file to have all 7 columns but…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-02T12:40:42.5366667+00:00
James Southall 0 Reputation points
commented 2025-05-07T09:55:58.3133333+00:00
Venkat Reddy Navari 1,585 Reputation points Microsoft External Staff
0 answers

ADF Resource Manager ContinuationToken not working

I'm using the ADF Resource Manager as per this documentation. I have managed to make the first Post API call to return the first 100 results as well as the continuationToken, but there is no documentation on how to use this to retrieve the next page of…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-06T10:38:12.4833333+00:00
Tommy Tam 0 Reputation points
commented 2025-05-07T09:50:16.7233333+00:00
Venkat Reddy Navari 1,585 Reputation points Microsoft External Staff
1 answer

Unable to retrieve previous month's data from Azure Marketplace Cost API

In ADF, at a web activity: I am using this API: GET https://management.azure.com/{scope}/providers/Microsoft.Consumption/marketplaces?api-version=2024-08-01 With scope as: /providers/Microsoft.Billing/billingAccounts/{billingAccountId} I want to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-05T13:07:07.0166667+00:00
Vishesh Agarwal 0 Reputation points
commented 2025-05-07T09:25:32.22+00:00
Smaran Thoomu 22,930 Reputation points Microsoft External Staff
1 answer

How to Fix commas(,) being read as decimal(.) on ADF while reading from SAP BW via MDX

While doing a preview data from SAP BW source, the commas (,) are being considered as decimal(.). However, on the SAP Analyzer through excel this isn't happening. Example, 1,995 on SAP gets converted to 1.995 on ADF.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-05T06:46:14.83+00:00
Raghav Haralalka - sID 0 Reputation points
commented 2025-05-07T09:24:02.0566667+00:00
Smaran Thoomu 22,930 Reputation points Microsoft External Staff
0 answers

Pipeline Copy Data Activity Timing Out

I have an ODBC copy data activity that is erroring and the suggestion is to just retry. the table is not the biggest table, and bigger tables successfully copy over.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-06T13:08:56.6133333+00:00
Ashley Sturgeon 0 Reputation points
commented 2025-05-07T09:22:12.2566667+00:00
Smaran Thoomu 22,930 Reputation points Microsoft External Staff
1 answer

Automating ADF Unit testing. How to automate the ADF pipelines unit testing. Deployment pipelines are in place.

Hello Microsoft Support team - Am writing this to know the best way to automate the Azure Data Factory pipeline Unit testing automation. As part of our integration project we have 100 plus ADF pipelines and we are in the phase of developing them and as…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-04-21T14:20:34.5833333+00:00
Firthouse M G 80 Reputation points
commented 2025-05-07T09:21:22.64+00:00
Smaran Thoomu 22,930 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

SFTP error when connecting from Azure Data Factory

When testing a connection from Azure Data Factory to an SFTP server which I created on a storage account, I receive the following error when creating the linked service: Meet network issue when connect to Sftp server…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-06T15:20:08.19+00:00
Koen van Wielink 0 Reputation points
edited a comment 2025-05-07T09:18:30.58+00:00
Koen van Wielink 0 Reputation points
0 answers

Can I access Oracle Database @Azure hosted on a different tenant using my Azure Data Factory pipeline hosted on my tenant using Azure PrivateLink?

I need to connect my Azure Data Factory pipeline configured on my tenant, to Oracle Database @ Azure hosted on a different tenant. I would prefer not to use Azure Integration and Self-Hosted Integration runtime. I am referring to the below mentioned…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-07T06:07:47.8833333+00:00
Sunil Menon 60 Reputation points
commented 2025-05-07T06:54:31.57+00:00
Sunil Menon 60 Reputation points
0 answers

Running Redshift Stored procedure on azure data factory and powerbi

Our vendor just migrated from MSSQL server to redshift. We are attempting to migrate powerbi and data factory; however, we cannot figure out how to "call" stored procedures on redshift.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-04-30T17:26:27.68+00:00
Tyler Eggett 0 Reputation points
commented 2025-05-07T05:36:41.2833333+00:00
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff
1 answer

Linked Service to on prem SQL server works everywhere except in data flow

I have a lined service to an on prem sql server through a VPN. Test connection works. The linked service works through pipelines as well. However when trying to use it in a Dataflow I get the following error. Only one valid authentication should be…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-04-30T12:35:57.73+00:00
Brian Daddino 0 Reputation points
commented 2025-05-06T12:43:52.3233333+00:00
Smaran Thoomu 22,930 Reputation points Microsoft External Staff
1 answer

How to fix Integration Runtime throttling for small Mapping Dataflow?

Hi, I have a Data Pipeline with two activities: a lookup which reads a JSON file and passes the contents to a For Each which runs a data flow. The data flow has 4 transformations: a source (CDM), Identify columns, Alter row and then a sink which writes…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-01T07:30:07.1533333+00:00
Rhys Doyle 0 Reputation points
commented 2025-05-06T05:13:41.0033333+00:00
phemanth 15,485 Reputation points Microsoft External Staff
2 answers

Convert CSV to Excel file

Hi Team, I am exploring a solution to convert CSV files to Excel format using Azure Data Factory (ADF). The requirement is to read CSV files stored in ADLS Gen2, convert them into Excel format, and then write the resulting Excel files back to the same…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,476 questions
asked 2025-05-01T05:48:09.29+00:00
Jaganath Kumar 110 Reputation points
commented 2025-05-06T04:25:19.12+00:00
phemanth 15,485 Reputation points Microsoft External Staff