Pipeline runs issue
There are two linked service connection failures in Azure Data Factory. Since we are currently using Version 1 and planning to upgrade to Version 2, is there anything we should be cautious about before making the change? Do you have any guidelines we can…
Azure Data Factory
When using the Jira Connector in Azure ADF, how can I save the table list in the dataset?
Hi ... I'm using the Jira Connector linked service in a dataset. The connection is successful, and I can see a list of available tables in the "Table" dataset field. Is there a way to extract this list of tables (to save for looping via…
Azure Data Factory
ADF copy activity from ADLS to delta lake fails due to invalid configuration value detected
Hi, I have a copy activity in ADF to copy data from ADLS to Delta lake. I created linked services with authentication method as Access token. The test connection for linked service is successful. But when I try to execute my pipeline I am getting an…
Azure Data Factory
Automating ADF Unit testing. How to automate the ADF pipelines unit testing. Deployment pipelines are in place.
Hello Microsoft Support team - Am writing this to know the best way to automate the Azure Data Factory pipeline Unit testing automation. As part of our integration project we have 100 plus ADF pipelines and we are in the phase of developing them and as…
Azure Data Factory
Data Factory Using Copy data activity with Azure Data Lake Storage Gen 2 & CSV Dataset
Hello: Have been experimenting with DataFactory and the copy data activity. My simple use case is as follows: Files are loaded into a folder under ADLS Gen (Blob Container) say once a week. When received a) copy the files to an AsReceived…
Azure Data Factory
How to fix an established connection was aborted by the server error while making SFTP connection using Azure Synapse?
Hello Team, While we are trying to connect SFTP using SSH public key authentication we are getting following error. Established connection was aborted by the server let me share the steps which we followed so far to connect the sftp. [1] Open PuttyGen…
Azure Synapse Analytics
Azure Data Factory
How can one trigger a fabric data pipeline via ADF
I am new to the ADF world.can someone guide me on how can one trigger a fabric pipeline via ADF in most secure way.
Azure Data Factory
ADF issues when extracting Snowflake data into Azure Blob Storage
Hi all - I have a pipeline created where I am copying data from Snowflake into an Azure Blob Storage authenticated via an SAS token. I have successfully run the pipeline in the past, however when I added new source datasets I am getting the following…
Azure Data Factory
I am trying to pull Sharepoint list into SDP using Pipeline but getting error
I am trying to pull Sharepoint list into SDP using Pipeline but getting error : List Name : 333 Backlog Tracker Requesting response from path 333 Backlog Tracker and query $top=10 failed with status code NotFound and message <?xml…
Azure Data Factory
I can connect to Jira via ADF's Jira Connector, but I can't pull any data.
Hi, I can connect to Jira via ADF's Jira Connector, but I can't pull any data - I get an error: "Error code # 9602 [Microsoft][JIRA] (62) API Connection Failed. Forbidden Request. HTTP Response code: 401". However, when creating the…
Azure Data Factory
Azure Data Factory - No "body" option for GET method into copy data's REST source
I have a list of parameters for the REST API call. The method I am trying to call API is Copy Data with REST source, method is GET and there is no body section there. So where do I exactly specify these parameters? Header is clearly not meant for this…
Azure Data Factory
I'm trying to do an ADF file consolidator pipeline with dynamic column headers
I am trying to merge multiple files but lets just say its 2. File A with 5 columns (Address Line 1, 2, 3, 4, 5) File B with 7 columns (Address Line 1, 2, 3, 4, 5, Telephone Number, Mobile Number) I expect the output file to have all 7 columns but…
Azure Data Factory
CDC for Gen1 Storage Account
I want to setup a CDC in ADF which automatically checks for changes in a Gen1 Storage Account and pushes the data into Gen2 Storage Account. From the Docs, I see this can be done through a Data Flow, but that needs to have a pipeline which in turn needs…
Azure Data Factory
Azure Data Factory - fetch and freeze mechanism during pipeline run?
Dear community, Context: I have a Azure Data Factory pipeline linked to Databricks workspace. There are a build and a release CI/CD Azure devops pipelines that automatically update the Databricks notebooks in the Databricks workspace. The CI/CD build…
Azure Data Factory
Azure Data Factory external call POST request body empty
In Azure Data Factory I am using the "External Call" action to generate a POST request to an external API: ContactColumns call(mapColumn( firstname, lastname ), skipDuplicateMapInputs: true, skipDuplicateMapOutputs:…
Azure Data Factory
Keeping a counter and increment it via ADF
Hello, I currently have the below data and for the column of Days Active, it is a daily counter that keeps incrementing. I currently employ the methos of nullifying the column and them executing a stored proc. but this process takes around 60 mins to be…
Azure Data Factory
Running Redshift Stored procedure on azure data factory and powerbi
Our vendor just migrated from MSSQL server to redshift. We are attempting to migrate powerbi and data factory; however, we cannot figure out how to "call" stored procedures on redshift.
Azure Data Factory
Intermittent connectivity issue between Azure Data Factory and Blob Storage — 'remote name could not be resolved'
Hi everyone, For about a week now, I’ve been experiencing an intermittent issue when running a pipeline in Azure Data Factory that writes data to an Azure Blob Storage container. The pipeline sometimes fails with the following error: Operation on target…
Azure Data Factory
Azure Data Factory Publish Error After Deleting Unused Pipelines, Dataflows, and Datasets
So i have deleted unused data pipelines, dataflows, and datasets which roughly totals to about 200-300 items. I have also ensured that no existing data pipeline and dataflows are connected to said deleted items. When i go to publish the updates, it…
Azure Data Factory
How to fix Integration Runtime throttling for small Mapping Dataflow?
Hi, I have a Data Pipeline with two activities: a lookup which reads a JSON file and passes the contents to a For Each which runs a data flow. The data flow has 4 transformations: a source (CDM), Identify columns, Alter row and then a sink which writes…