I am building a simple pipeline to copy data from an on prem to ADLS gen2. I am using a copy activity with redshift as a source inside a for each loop for a given table array.
The error is the following - Failure happened on 'Source' side. ErrorCode=UserErrorUnclassifiedError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Odbc Operation…
Azure Data Factory
How do I set up an ADLS Gen 2 Linked Service in Azure Data Factory with a SAS URI?
Hi, I am able to set up a linked service to a blob with no problem, and I am actually able to set up a linked service to this current ADLS gen 2 container by treating it as a blob, but when I try to treat it as ADLS gen 2 via the "set up linked…
Azure Data Factory
Azure Function Forbidden Access to Key Vault despite Appropriate IAM Permission and Firewall Settings
I am encountering an issue where my Azure Function is being called from Azure Data Factory (ADF) and needs to access an Azure Key Vault. Setup: The Azure Function uses a System Assigned Managed Identity. It has been granted Key Vault Secrets User role…
Azure Data Factory
When using the Jira Connector in Azure ADF, how can I save the table list in the dataset?
Hi ... I'm using the Jira Connector linked service in a dataset. The connection is successful, and I can see a list of available tables in the "Table" dataset field. Is there a way to extract this list of tables (to save for looping via…
Azure Data Factory
Pipeline runs issue
There are two linked service connection failures in Azure Data Factory. Since we are currently using Version 1 and planning to upgrade to Version 2, is there anything we should be cautious about before making the change? Do you have any guidelines we can…
Azure Data Factory
Azure Data Factory - Internal Server Error
Hi, I'm using Azure Data Factory. I'm unable to run pipeline since yesterday. I've the following error: { "code": "InternalServerError", "message": null, "target":…
Azure Data Factory
Google Big Query not working on self hosted integration runtime
Google Big Query is not working on self hosted integration runtime v5.48.9106.2. When I run a query it returns the following error: Method not found: 'Microsoft.DI.MSDISDKConnectorInterfaces.DataObjects.Condition…
Azure Data Factory
Im unable to get the dataset information from power bi workspace using adf
Hi Im unable to get the dataset or workspace information from power bi using adf web activity error message : Invoking endpoint failed with HttpStatusCode - '404 : NotFound', message - 'The requested endpoint(url) does not exist on the server. Please…
Azure Data Factory
I'm trying to do an ADF file consolidator pipeline with dynamic column headers
I am trying to merge multiple files but lets just say its 2. File A with 5 columns (Address Line 1, 2, 3, 4, 5) File B with 7 columns (Address Line 1, 2, 3, 4, 5, Telephone Number, Mobile Number) I expect the output file to have all 7 columns but…
Azure Data Factory
ADF Resource Manager ContinuationToken not working
I'm using the ADF Resource Manager as per this documentation. I have managed to make the first Post API call to return the first 100 results as well as the continuationToken, but there is no documentation on how to use this to retrieve the next page of…
Azure Data Factory
Unable to retrieve previous month's data from Azure Marketplace Cost API
In ADF, at a web activity: I am using this API: GET https://management.azure.com/{scope}/providers/Microsoft.Consumption/marketplaces?api-version=2024-08-01 With scope as: /providers/Microsoft.Billing/billingAccounts/{billingAccountId} I want to…
Azure Data Factory
How to Fix commas(,) being read as decimal(.) on ADF while reading from SAP BW via MDX
While doing a preview data from SAP BW source, the commas (,) are being considered as decimal(.). However, on the SAP Analyzer through excel this isn't happening. Example, 1,995 on SAP gets converted to 1.995 on ADF.
Azure Data Factory
Pipeline Copy Data Activity Timing Out
I have an ODBC copy data activity that is erroring and the suggestion is to just retry. the table is not the biggest table, and bigger tables successfully copy over.
Azure Data Factory
Automating ADF Unit testing. How to automate the ADF pipelines unit testing. Deployment pipelines are in place.
Hello Microsoft Support team - Am writing this to know the best way to automate the Azure Data Factory pipeline Unit testing automation. As part of our integration project we have 100 plus ADF pipelines and we are in the phase of developing them and as…
Azure Data Factory
SFTP error when connecting from Azure Data Factory
When testing a connection from Azure Data Factory to an SFTP server which I created on a storage account, I receive the following error when creating the linked service: Meet network issue when connect to Sftp server…
Azure Data Factory
Can I access Oracle Database @Azure hosted on a different tenant using my Azure Data Factory pipeline hosted on my tenant using Azure PrivateLink?
I need to connect my Azure Data Factory pipeline configured on my tenant, to Oracle Database @ Azure hosted on a different tenant. I would prefer not to use Azure Integration and Self-Hosted Integration runtime. I am referring to the below mentioned…
Azure Data Factory
Running Redshift Stored procedure on azure data factory and powerbi
Our vendor just migrated from MSSQL server to redshift. We are attempting to migrate powerbi and data factory; however, we cannot figure out how to "call" stored procedures on redshift.
Azure Data Factory
Linked Service to on prem SQL server works everywhere except in data flow
I have a lined service to an on prem sql server through a VPN. Test connection works. The linked service works through pipelines as well. However when trying to use it in a Dataflow I get the following error. Only one valid authentication should be…
Azure Data Factory
How to fix Integration Runtime throttling for small Mapping Dataflow?
Hi, I have a Data Pipeline with two activities: a lookup which reads a JSON file and passes the contents to a For Each which runs a data flow. The data flow has 4 transformations: a source (CDM), Identify columns, Alter row and then a sink which writes…
Azure Data Factory
Convert CSV to Excel file
Hi Team, I am exploring a solution to convert CSV files to Excel format using Azure Data Factory (ADF). The requirement is to read CSV files stored in ADLS Gen2, convert them into Excel format, and then write the resulting Excel files back to the same…