A node in a pool is stuck in rebooting state since last 4 days

Question

A node in a pool is stuck in rebooting state since last 4 days

Tanmay Sharma 0

A node (tvmps_8f4c7b22047eb71169466eb201a4437548f517ea99832a76a7dc3a564307646c_d) in a pool called OSAthresholdS_v2 is stuck since 19th March. We did a RDP in the machine and see no data for the task it is trying to run. We have also disabled the job which the node had taken just remove possible dependency on the node but still the node is stuck and is not rebooting. The deallocation of the node from the pool also does not work, this is really a frustrating thing. Following are few things we already did just to save time of the support engineer to:

Remove the Node from the Pool
Check for Configuration Issues
Reboot the Node

Geethasri.V 65 Reputation points Microsoft External Staff

2025-03-24T18:40:14.9866667+00:00
Hi Tanmay Sharma,It looks like the node is stuck in a rebooting state and is unresponsive, preventing removal from the pool. This can happen due to various reasons, such as OS-level failures, Azure Batcl service inconsistencies, or underlying infrastructure issues. Since you have already attempted removing the node, checking configurations, and rebooting, I recommend the following additional troubleshooting steps to recover the node.

Check Node status:

az batch pool node list --pool-id OSAthresholds_v2 --query "[?id=='tvmps_ 8f4c7b22047eb71169466eb201a4437548f517ea99832a76a7dc3a564307646c_d ' ]"

Reimage the Node:

az batch node reimage --pool-id OSAthresholds_v2 --node-id tvmps_ 8fAc7b22047eb71169466eb201a4437548f517ea99832a76a7dc3a564307646C_d

Delete the Node (if reimage fails):

az batch node delete --pool-id OSAthresholds_v2 --node-id tvmps_ 8f4c7b22047eb71169466eb201a4437548f517ea99832a76a7dc3a564307646c_d

Check Logs via RDP:

Windows: Event Viewer → System/Application Logs

Linux: journalctt -xe

Please refer the below documentation:

Manage Batch Nodes: https://learn.microsoft.com/en-us/azure/batch/nodes-and-pools

If you have any further queries, please let us know we are glad to help you.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Geethasri.V 65 Reputation points Microsoft External Staff

2025-03-25T05:43:19.28+00:00

Hi Tanmay Sharma,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Tanmay Sharma 0 Reputation points

2025-03-25T12:20:59.49+00:00
Hi,
Thanks for the reply.

Node status is "rebooting"

reimage does support when node is in rebooting state.

deletion of node is not possible via the cli when the pool is in auto scale mode
Geethasri.V 65 Reputation points Microsoft External Staff

2025-03-28T07:22:28.27+00:00

Hi Tanmay Sharma,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Geethasri.V 65 Reputation points Microsoft External Staff

2025-03-28T11:04:52.5866667+00:00

Hi Tanmay Sharma,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.

1 answer

Your answer

Geethasri.V 65 Reputation points Microsoft External Staff

2025-03-24T18:40:14.9866667+00:00

Hi Tanmay Sharma,It looks like the node is stuck in a rebooting state and is unresponsive, preventing removal from the pool. This can happen due to various reasons, such as OS-level failures, Azure Batcl service inconsistencies, or underlying infrastructure issues. Since you have already attempted removing the node, checking configurations, and rebooting, I recommend the following additional troubleshooting steps to recover the node.

Check Node status:

az batch pool node list --pool-id OSAthresholds_v2 --query "[?id=='tvmps_ 8f4c7b22047eb71169466eb201a4437548f517ea99832a76a7dc3a564307646c_d ' ]"

Reimage the Node:

az batch node reimage --pool-id OSAthresholds_v2 --node-id tvmps_ 8fAc7b22047eb71169466eb201a4437548f517ea99832a76a7dc3a564307646C_d

Delete the Node (if reimage fails):

az batch node delete --pool-id OSAthresholds_v2 --node-id tvmps_ 8f4c7b22047eb71169466eb201a4437548f517ea99832a76a7dc3a564307646c_d

Check Logs via RDP:

Windows: Event Viewer → System/Application Logs

Linux: journalctt -xe

Please refer the below documentation:

Manage Batch Nodes: https://learn.microsoft.com/en-us/azure/batch/nodes-and-pools

If you have any further queries, please let us know we are glad to help you.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Geethasri.V 65 Reputation points Microsoft External Staff

2025-03-25T05:43:19.28+00:00

Hi Tanmay Sharma,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Tanmay Sharma 0 Reputation points

2025-03-25T12:20:59.49+00:00

Hi,
Thanks for the reply.

Node status is "rebooting"

reimage does support when node is in rebooting state.

deletion of node is not possible via the cli when the pool is in auto scale mode
Geethasri.V 65 Reputation points Microsoft External Staff

2025-03-28T07:22:28.27+00:00

Hi Tanmay Sharma,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Geethasri.V 65 Reputation points Microsoft External Staff

2025-03-28T11:04:52.5866667+00:00

Hi Tanmay Sharma,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.

Answer 1

Hi Tanmay Sharma,

Since the node is stuck in the "rebooting" state and cannot be reimaged or deleted due to auto-scaling, try these steps:

Disable Auto-Scaling Temporarily:

az batch pool autoscale disable --pool-id 
OSAthresholdS_v2

After disabling auto-scale, try deleting the node:

az batch node delete --pool-id OSAthresholdS_v2 --node-id tvmps_8f4c7b22047eb71169466eb201a4437548f517ea99832a76a7dc3a564307646c_d

Once removed, re-enable auto-scaling if needed.

Manually Resize the Pool (Alternative to Deletion):

az batch pool resize --pool-id OSAthresholdS_v2 --target-dedicated-nodes <new_count>

This forces Azure to replace the problematic node.

Use Azure REST API for Forceful Removal: If CLI deletion still fails, try using the Batch Node Removal API: https://learn.microsoft.com/en-us/rest/api/batchservice/pool/remove-nodes?view=rest-batchservice-2024-07-01&tabs=HTTP

Refer to the below document:

Disable Auto-Scaling: https://learn.microsoft.com/en-us/azure/batch/batch-automatic-scaling#disable-autoscale

If you have any further queries, please let us know we are glad to help you.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.

Share via

A node in a pool is stuck in rebooting state since last 4 days

1 answer

Your answer