Azure OpenAI Batch Jobs are getting stuck in Validating stage

Question

Azure OpenAI Batch Jobs are getting stuck in Validating stage

Seena Mary Augusty 0 Microsoft Employee

There are so many Azure Open AI batch jobs stuck in validating stage starting from April 11, 7.16 AM PDT and there isn't a single batch reached completion. We had raised a similar issue on March 18 and was resolved by using a new deployment model, but this time no new deployment had helped to resolve this issue. Any new batch is getting stuck under validating stage. User's image

Here are the details of my setup:

Endpoint: https://live-signals-copilot.openai.azure.com/
Pricing Tier: Global-Batch
Region: East US
Resource Configuration:
Name: Live-Signals-Copilot
Subscription: Customer Understanding
Subscription ID: 62e7b93a-1bbe-492b-b06a-3a93f3d269fd
- Resource Group: Clarity

Could you help us resolve this issue as soon as possible. Our production endpoints are reliant on these batch jobs.

Seena Mary Augusty 0 Reputation points Microsoft Employee

2025-04-14T17:10:55.4433333+00:00

It is back to normal now, is there a way to do something if these kind of issue occurs again, I think it is 4th incident for us
santoshkc 14,415 Reputation points Microsoft External Staff

2025-04-15T14:07:09+00:00

Hi @Seena Mary Augusty,

It looks like your Azure OpenAI batch jobs have been getting stuck in the "Validating" stage since April 11, and none of them have completed since then. A similar issue was reported in March and temporarily resolved by switching to a new deployment, but this time that workaround hasn’t helped.

This seems to be a known problem that other users have faced recently too, it may be tied to something specific in your setup or region (East US).

In the meantime, a few things might help: first, try cancelling any batch jobs that are stuck and re-submitting them, sometimes this clears up whatever is blocking validation. Also, make sure your input files are well-formatted and meet the required batch limits. Check that sufficient quota is enabled for your deployment, as that can help avoid capacity-related issues.

Thank you for your understanding!
santoshkc 14,415 Reputation points Microsoft External Staff

2025-04-16T11:16:37.58+00:00

Hi @Seena Mary Augusty,

Just checking if the earlier update helped clarify the issue with batch jobs stuck in "Validating." Let us know if there's any progress or if further input is needed from our side.
Seena Mary Augusty 0 Reputation points Microsoft Employee

2025-04-16T16:59:54.4+00:00

Hi @santoshkc The issue was resolved on April 13, but from yesterday it got stuck again in validating stage
santoshkc 14,415 Reputation points Microsoft External Staff

2025-04-17T14:17:58.5633333+00:00

Hi @Seena Mary Augusty,

Thanks for the update. Since the issue was previously resolved on April 13 but has reappeared again, it seems like there may be a backend regression or an intermittent problem with Azure OpenAI’s batch job system.

In the meantime, there are a few steps you can try. First, cancel any batch jobs that are currently stuck in the "Validating" stage and attempt to resubmit them. This has occasionally helped clear jobs that are blocked in the queue. Second, review the model version you're currently using in your deployment. If it's an older version like, try updating to a newer version and then resubmit the batch jobs. Additionally, double-check that your input files follow the required format and that you're not exceeding any batch size or token limits.

I hope this helps! Thank you.
Seena Mary Augusty 0 Reputation points Microsoft Employee

2025-04-29T21:17:44.9533333+00:00

Hello @santoshkc thank you so much for the steps, I have tried the steps you mentioned but the batches are either getting stuck or extremely slow, is there a livesite link to azure open ai where we could raise tickets.
Sometimes it run after 3 days

Thank you so much for the help again
Seena Mary Augusty 0 Reputation points Microsoft Employee

2025-05-01T21:30:57.77+00:00

Hi @santoshkc we would like to understand the root cause, Is there an escalation process or can you help us by getting on a call to further investigate the root cause since we are using it in our production, thank you so much again

Your answer

Seena Mary Augusty 0 Reputation points Microsoft Employee

2025-04-14T17:10:55.4433333+00:00

It is back to normal now, is there a way to do something if these kind of issue occurs again, I think it is 4th incident for us
santoshkc 14,415 Reputation points Microsoft External Staff

2025-04-15T14:07:09+00:00

Hi @Seena Mary Augusty,

It looks like your Azure OpenAI batch jobs have been getting stuck in the "Validating" stage since April 11, and none of them have completed since then. A similar issue was reported in March and temporarily resolved by switching to a new deployment, but this time that workaround hasn’t helped.

This seems to be a known problem that other users have faced recently too, it may be tied to something specific in your setup or region (East US).

In the meantime, a few things might help: first, try cancelling any batch jobs that are stuck and re-submitting them, sometimes this clears up whatever is blocking validation. Also, make sure your input files are well-formatted and meet the required batch limits. Check that sufficient quota is enabled for your deployment, as that can help avoid capacity-related issues.

Thank you for your understanding!
santoshkc 14,415 Reputation points Microsoft External Staff

2025-04-16T11:16:37.58+00:00

Hi @Seena Mary Augusty,

Just checking if the earlier update helped clarify the issue with batch jobs stuck in "Validating." Let us know if there's any progress or if further input is needed from our side.
Seena Mary Augusty 0 Reputation points Microsoft Employee

2025-04-16T16:59:54.4+00:00

Hi @santoshkc The issue was resolved on April 13, but from yesterday it got stuck again in validating stage
santoshkc 14,415 Reputation points Microsoft External Staff

2025-04-17T14:17:58.5633333+00:00

Hi @Seena Mary Augusty,

Thanks for the update. Since the issue was previously resolved on April 13 but has reappeared again, it seems like there may be a backend regression or an intermittent problem with Azure OpenAI’s batch job system.

In the meantime, there are a few steps you can try. First, cancel any batch jobs that are currently stuck in the "Validating" stage and attempt to resubmit them. This has occasionally helped clear jobs that are blocked in the queue. Second, review the model version you're currently using in your deployment. If it's an older version like, try updating to a newer version and then resubmit the batch jobs. Additionally, double-check that your input files follow the required format and that you're not exceeding any batch size or token limits.

I hope this helps! Thank you.
Seena Mary Augusty 0 Reputation points Microsoft Employee

2025-04-29T21:17:44.9533333+00:00

Hello @santoshkc thank you so much for the steps, I have tried the steps you mentioned but the batches are either getting stuck or extremely slow, is there a livesite link to azure open ai where we could raise tickets.
Sometimes it run after 3 days

Thank you so much for the help again
Seena Mary Augusty 0 Reputation points Microsoft Employee

2025-05-01T21:30:57.77+00:00

Hi @santoshkc we would like to understand the root cause, Is there an escalation process or can you help us by getting on a call to further investigate the root cause since we are using it in our production, thank you so much again

Share via

Azure OpenAI Batch Jobs are getting stuck in Validating stage

Your answer