OpenAI Fine Tuning status stuck on "Running" Since days

Hi Neil Sanghavi,

Thank you for sharing the issue and the screenshot. From your explanation and the job timeline in the screenshot, it appears that the fine-tuning process on Azure OpenAI is stuck in the "Training started" phase since April 25, 2025, 2:40 PM, and hasn't progressed for several days. Here's a detailed breakdown of the problem and how to troubleshoot and resolve it:

Reasons for Job Getting Stuck:

1. Backend capacity issue – Sometimes Azure regions face GPU shortages or job bottlenecks.

2. Large training data – If your dataset is too large, it might hang due to resource limits or preprocessing bugs.

3. Service bug or unresponsive job handler – Rare but possible.

4. Incorrect data format – Might pass initial validation but cause failure during actual training.

5. No auto-timeout or stale job cleanup – Your job might need a manual trigger or support intervention.

Steps to Troubleshoot and Fix:

1.Check Job Status via Azure CLI or REST API

Use the Azure CLI to check current status and logs:

az openai fine-tune job show --resource-group <your-rg> --deployment-id <deployment-id> --job-id <job-id>

2.Verify Dataset Format

Make sure your JSONL file meets the OpenAI fine-tuning requirements:

·       Must contain lines like: {"prompt": "some input", "completion": "some output"}

·       Prompt should end with a delimiter like \n\n###\n\n

3.Reduce Data Size Temporarily

If possible, try fine-tuning with a smaller subset of your data (~10–50 samples) to rule out volume-related delays.

4.Check Azure Region and Quota

·       Go to Azure Portal → Azure OpenAI resource → Usage + quotas tab.

·       Ensure you have fine-tuning quota and available GPU capacity in your region.

5. Cancel and Retry Job

You can cancel the job and retry:

az openai fine-tune job cancel --resource-group <your-rg> --deployment-id <deployment-id> --job-id <job-id>

Hope this helps, do let me know if you have any further queries.

Thank you!