Azure Data Explorer ingests duplicated records from Event Hub

Question

Azure Data Explorer ingests duplicated records from Event Hub

J.P. Zhang 25 Microsoft Employee

We have data ingestions set up between an event hub and Azure Data Explorer. There are abnormally high duplicated records in Azure Data Explorer database tables. I understand that the ingestion delivery is at least once. having dups are sort of expected. But we are seeing very high dups at a rate of 50 dups out of 160 records. What could be the issue here? What configuration changes can be made to mitigate this?

Sander van de Velde | MVP 36,146 Reputation points MVP

2025-03-14T22:30:36.6+00:00

Hello @J.P. Zhang

welcome to this moderated Azure community forum.

These numbers are a bit strange.

Can you share so information about the region(s) these resources are living in?

How many partitions are there? Did you provide the Azure Data Explorer data connection its own consumer group?
phemanth 15,320 Reputation points Microsoft External Staff

2025-03-17T07:03:54.52+00:00

@J.P. Zhang We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J.P. Zhang 25 Reputation points Microsoft Employee

2025-03-17T16:23:36.7466667+00:00

@Sander van de Velde, thank you for responding to my questions. The Event Hubs and Data Explorer cluster are in West US 3. There are 32 partitions in the Event Hubs. Yes, the data connection has its own consumer group.

I checked all data connections. They looked right. I deleted all data connections and recreated them. After that, the test runs no longer show duplicates. Before that I was a bit reluctant to delete and recreate the data connections because I have two Data Explorer clusters and multiple DBs and tables and quite a few event hubs.
Sander van de Velde | MVP 36,146 Reputation points MVP

2025-03-17T18:17:25.79+00:00

Hello @J.P. Zhang,

good to hear the issue seems to be resolved.

The 32 partitions feels a bit odd. This is the maximum (shared over all eventhubs per namespace) for both the basic and standard tier but you are fine with premium and dedicated if you have deployed on one of those tiers.
J.P. Zhang 25 Reputation points Microsoft Employee

2025-03-17T19:50:35.7866667+00:00

Our data volume is fairly high therefore we have 32 partitions.

Accepted answer

0 additional answers

Your answer

Sander van de Velde | MVP 36,146 Reputation points MVP

2025-03-14T22:30:36.6+00:00

Hello @J.P. Zhang

welcome to this moderated Azure community forum.

These numbers are a bit strange.

Can you share so information about the region(s) these resources are living in?

How many partitions are there? Did you provide the Azure Data Explorer data connection its own consumer group?
phemanth 15,320 Reputation points Microsoft External Staff

2025-03-17T07:03:54.52+00:00

@J.P. Zhang We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J.P. Zhang 25 Reputation points Microsoft Employee

2025-03-17T16:23:36.7466667+00:00

@Sander van de Velde, thank you for responding to my questions. The Event Hubs and Data Explorer cluster are in West US 3. There are 32 partitions in the Event Hubs. Yes, the data connection has its own consumer group.

I checked all data connections. They looked right. I deleted all data connections and recreated them. After that, the test runs no longer show duplicates. Before that I was a bit reluctant to delete and recreate the data connections because I have two Data Explorer clusters and multiple DBs and tables and quite a few event hubs.
Sander van de Velde | MVP 36,146 Reputation points MVP

2025-03-17T18:17:25.79+00:00

Hello @J.P. Zhang,

good to hear the issue seems to be resolved.

The 32 partitions feels a bit odd. This is the maximum (shared over all eventhubs per namespace) for both the basic and standard tier but you are fine with premium and dedicated if you have deployed on one of those tiers.
J.P. Zhang 25 Reputation points Microsoft Employee

2025-03-17T19:50:35.7866667+00:00

Our data volume is fairly high therefore we have 32 partitions.

Answer 1

Hello @J.P. Zhang,

welcome to this moderated Azure community forum.

The data ingestions via Data connections set up between an event hub and Azure Data Explorer showed an abnormally high rate of duplicated records in Azure Data Explorer database tables (50 dups out of 160 records).

The solution chosen here was deleting all data connections and recreated them again.

After that, the test runs no longer show duplicates.

Azure Eventhubs have an option to disable sending:

User's image

If the duration of disabling sending data is less than the retention time set of the eventhub, not data is lost, only delayed.

This does not touch any schema or data already ingested in the Azure Data Explorer.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Share via

Azure Data Explorer ingests duplicated records from Event Hub

0 additional answers

Your answer