How to make Eventhub Consumer resilient?

Question

How to make Eventhub Consumer resilient?

Shruti 40

I am building a consumer application(Spring boot application running on AKS) that consumes and processes events from Azure Event Hub topics (Enterprise solution hosted on a different subscription) and updates data in a database.

Note: Eventhub is currently the chosen solution, so we can't change it.
Currently, the consumer application is able to consume events and process them in the happy path, but in case of failures, I have only implemented a retry and skip logic.

I want the solution to be more resiliant in a way that I can store messages that I have consumed but not processed (issues with updating the database) or in case of not being able to process the incoming event (serialisation errors) . My application is idempotent and has a way to handle duplicate or out of sequence messages based on timestamps.

Or any such circumstances.

I need a pattern like DLQ to automatically route failed messages to a queue.

I have seen patterns where I can use the try-catch method to route messages to another event hub or service bus queue, but I am not sure if it will handle all edge cases like De-serialisation issues.

Can I have a Service Bus between my Eventhub and AKS application?

Eventhub -> Service Bus -> Consumer Application

The Service Bus will have a built-in DLQ. I can use it with Azure Functions or Logic apps to deal with failed messages.

We consume around around 4-5k messages per day across 4 different eventhub topics.

What are some other cost-friendly options?

Accepted answer

0 additional answers

Your answer

Answer 1

Smaran Thoomu 23,095 Microsoft External Staff Moderator

Hi @Shruti

To address your concerns about making your Event Hub consumer more resilient, here are specific recommendations:

Handling unprocessed messages due to database failures:

Since your application is idempotent, you can store unprocessed messages in a retry queue (e.g., an Azure Service Bus queue or Blob Storage).
A background job (Azure Function or Logic App) can retry processing messages periodically.

Handling deserialization errors

If an event cannot be deserialized, route it to a Dead Letter Queue (DLQ).
You can create a separate Event Hub or Service Bus queue to store these failed events for debugging and reprocessing.

Using Service Bus between Event Hub and the Consumer (Event Hub → Service Bus → AKS Consumer)

Yes, this is a valid approach. Service Bus has a built-in DLQ** and supports retries.
However, adding Service Bus increases cost and complexity. If buffering is not required, consider logging failed messages instead.

Cost-friendly alternatives

If your main concern is failed messages, a separate Event Hub DLQ or Blob Storage is more cost-effective than Service Bus.
If failures are rare, you can log messages in Cosmos DB or Blob Storage and retry them later.

Suggestions:

If event buffering is needed → Service Bus works well.
If you just need a DLQ → A separate Event Hub or Blob Storage is more cost-effective.
If failures are rare → Logging failed events for later analysis is the cheapest option.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Shruti 40 Reputation points

2025-03-26T15:14:29.46+00:00

We do need a way to automatically reprocess the failed messages. Failures are not frequent as per recent analysis, but multiple downstream processes are involved based on this data update, so we cannot afford any major delays or missed updates to the data.

We currently do have a log and alert mechanism, but that's not sufficient to handle production flows. We need a DLQ like solution to further add automation for alerting and reprocessing.

There are multiple consumers for the Event Hub topics, everyone in different azure subscriptions. So by that logic, if we opt for option 2, using a separate eventhub or blob storage, should this be per consumer and hosted on the same subscription on the consumer?

Also, will blob storage / eventhub solution work with Deserialisation issues? We faced this recently when producer changed the serialisation method, and we couldn't deserialise without updating the code on our end, we need to be able to save messages for such instances.
Smaran Thoomu 23,095 Reputation points Microsoft External Staff Moderator

2025-03-26T15:50:37.4133333+00:00
@Shruti Thanks for the clarification! Since you need automated reprocessing and alerting for failed messages, a DLQ-based approach is the best fit. Here’s how you can refine your setup:

Where to host the DLQ (Per Consumer or Shared?)

Since multiple consumers exist across different subscriptions, each consumer should manage its own DLQ in their respective subscription.

If a centralized DLQ is needed, it must be in a subscription accessible to all consumers, but this adds cross-subscription complexity.

Handling Deserialization Issues

Blob Storage or Event Hub as a DLQ can store failed messages, but they won’t directly help with deserialization issues.

The best practice is to store both the raw message and metadata (e.g., schema version, timestamp, producer info) to aid reprocessing.

If schema changes are frequent, consider Schema Registry in Event Hubs to manage versioning and prevent breaking changes.

Recommended DLQ Approach

Option1: Azure Service Bus DLQ (Preferred for Automation)

Messages that fail processing (e.g., DB update failure, serialization issue) automatically go to the DLQ.

Use Azure Functions or Logic Apps to reprocess failed messages.

Alerts can be set up via Azure Monitor or Application Insights.

Option 2: Event Hub / Blob Storage as DLQ

Each consumer stores failed messages in their own Event Hub or Blob Storage for manual reprocessing.

Suitable if failures are rare and manual intervention is acceptable.

Suggested Approach

If automation is key, Service Bus with a DLQ is the best option.

If you prefer a lightweight approach, Event Hub/Blob Storage per consumer can work, but you'll need additional logic to handle reprocessing and alerts.

Consider implementing Schema Registry to prevent deserialization issues in the future.

Let me know your thoughts!
Shruti 40 Reputation points

2025-03-26T15:57:26.1166667+00:00

Thank you @Smaran Thoomu . This helps get more clarity on designing the solution.
I suppose the Service Bus - DLQ based approach seems to be the best fit for the use case, although expensive than using BLOB Storage. I will discuss with my team on the SLA and cost requirements. But your insights are really helpful!
Smaran Thoomu 23,095 Reputation points Microsoft External Staff Moderator

2025-03-26T22:53:53.22+00:00

Shruti Glad to hear that the insights were helpful! The Service Bus-DLQ approach provides built-in automation for reprocessing and alerting, making it a strong choice for reliability. Do let us know if you need further guidance in evaluating cost vs. SLA considerations.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know. Thanks!
Ganesh Gurram 6,630 Reputation points Microsoft External Staff Moderator

2025-03-28T09:05:26.4+00:00

@Shruti

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

How to make Eventhub Consumer resilient?

0 additional answers

Your answer