Azure Data Factory Publish Error After Deleting Unused Pipelines, Dataflows, and Datasets

Question

Azure Data Factory Publish Error After Deleting Unused Pipelines, Dataflows, and Datasets

RizkyM 0

So i have deleted unused data pipelines, dataflows, and datasets which roughly totals to about 200-300 items. I have also ensured that no existing data pipeline and dataflows are connected to said deleted items.

When i go to publish the updates, it always ends with an error that indicates a missing dataset/dataflow which refers to one of the deleted items like below.

Missing Dataset

If i check the detail, it gives no assistance at all.

Detail Error

I have further checked every json file in the ADF and confirmed that there are no dataset/dataflow that still refers to the deleted items. It is as if ADF is trying to find a ghost dataset/dataflow.

Should I try Overwrite live mode using the master branch or are there any alternative solutions?

Thanks

RizkyM 0 Reputation points

2025-04-29T09:34:50.4566667+00:00

I will try the methods you have mentioned and will be commenting the result after 2 weeks, thank you. So this comment should not be counted as resolved yet.
phemanth 15,320 Reputation points Microsoft External Staff

2025-05-02T05:28:14.2866667+00:00

@RizkyM Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

RizkyM 0 Reputation points

2025-04-29T09:34:50.4566667+00:00

I will try the methods you have mentioned and will be commenting the result after 2 weeks, thank you. So this comment should not be counted as resolved yet.
phemanth 15,320 Reputation points Microsoft External Staff

2025-05-02T05:28:14.2866667+00:00

@RizkyM Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

@RizkyM

It sounds like you're encountering an issue with Azure Data Factory (ADF) after deleting unused items.

Please check below steps and confirm us

As you mentioned, trying to overwrite the live mode using the master branch can help. This will sync the live environment with your Git repository, potentially resolving any discrepancies
Sometimes, references to deleted items might still exist in hidden or less obvious places. Double-check all linked services, triggers, and parameters to ensure there are no lingering references.
Clear the cache and restart the ADF service. This can sometimes resolve issues related to stale data.
Go to the Git configuration page in ADF and re-sync the content. This can help resolve conflicts between the Data Factory mode and Git mode
If the above steps don't work, you might need to manually clean up the ADF environment. This involves checking and removing any orphaned references in the JSON files and ensuring all dependencies are correctly updated.

Share via

Azure Data Factory Publish Error After Deleting Unused Pipelines, Dataflows, and Datasets

1 answer

Your answer