NVidia drivers not installed by extension on Windows 2019 Scale Set NV6ads A10 v5

Question

NVidia drivers not installed by extension on Windows 2019 Scale Set NV6ads A10 v5

Henrik Aldermo 56

Related to https://learn.microsoft.com/en-us/answers/questions/2180437/how-to-fix-no-gpu-issue-with-nc-series-on-azure-vm

Ping @kobulloc-MSFT

On an Azure Virtual Machine Scale set running Windows Server 2019 on Standard_NV6ads_A10_v5, the Nvidia GPU Driver Extension fails, and no graphic drivers are installed.

Manual installation of NVidia drivers is successful. Does not affect Windows Server 2022.

VM has reported a failure when processing extension 'NvidiaGpuDriverWindows' (publisher 'Microsoft.HpcCompute' and type 'NvidiaGpuDriverWindows'). Error message: 'Enable Extension'

Log on VM (excerpt):

Execution Error:

Number of Tries: 1

Command C:\Packages\Plugins\Microsoft.HpcCompute.NvidiaGpuDriverWindows\1.10.0.4\run.cmd of Microsoft.HpcCompute.NvidiaGpuDriverWindows has exited with Exit code: 0

Plugin (name: Microsoft.HpcCompute.NvidiaGpuDriverWindows, version: 1.10.0.4) completed successfully.

Mounika Reddy Anumandla 5,055 Reputation points Microsoft External Staff Moderator

2025-03-18T09:24:55.8566667+00:00

Hi Henrik Aldermo,

The extension (Microsoft.HpcCompute.NvidiaGpuDriverWindows) always installs the latest GRID driver. The extension might be trying to install a driver that is incompatible with Windows Server 2019.

Windows Server 2019 does support the GPU but may require a specific driver version that the extension doesn’t install correctly. For links to all previous Nvidia GRID driver versions, visit GitHub.

Since Windows Server 2022 is officially supported, the extension installs the correct GRID 17.5 driver without issues.

The note section mentions

For Azure NVads A10 v5 VMs we recommend customers to always be on the latest driver version. The latest NVIDIA major driver branch(n) is only backward compatbile with the previous major branch(n-1). For eg, vGPU 17.x is backward compatible with vGPU 16.x only. Any VMs still runnig n-2 or lower may see driver failures when the latest drive branch is rolled out to Azure hosts. NVs_v3 VMs only support vGPU 16 or lower driver version. Windows server 2019 support is only available till vGPU 16.x.

https://learn.microsoft.com/en-us/azure/virtual-machines/windows/n-series-driver-setup

Since manual installation works, you can disable the extension and manually install the driver every time a VM is deployed.
https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-windows#known-issues

Hope this helps!
Henrik Aldermo 56 Reputation points

2025-03-18T10:04:16.8766667+00:00
Hi @Mounika Reddy Anumandla and thanks for the reply. I agree the underlying issue seems to be driver incompatibility with Windows 2019 and that installing an older driver would be a solution.

However, in my case the affected system is a virtual machine scale set, so installing drivers manually on every new instance is not viable.

I found a working solution by specifying the driver version in the NVidia Driver Extension, as documented in the "Known Issues" section here: https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-windows#known-issues

For a Virtual Machine Scale set, use the following:

az vmss extension set --resource-group MyResourceGroup --vmss-name MyVmss --name NvidiaGpuDriverWindows --publisher Microsoft.HpcCompute --settings "{'driverVersion':'538.46'}"

To view the settings:

az vmss extension list --resource-group MyResourceGroup --vmss-name MyVmss

Accepted answer

0 additional answers

Your answer

Mounika Reddy Anumandla 5,055 Reputation points Microsoft External Staff Moderator

2025-03-18T09:24:55.8566667+00:00

Hi Henrik Aldermo,

The extension (Microsoft.HpcCompute.NvidiaGpuDriverWindows) always installs the latest GRID driver. The extension might be trying to install a driver that is incompatible with Windows Server 2019.

Windows Server 2019 does support the GPU but may require a specific driver version that the extension doesn’t install correctly. For links to all previous Nvidia GRID driver versions, visit GitHub.

Since Windows Server 2022 is officially supported, the extension installs the correct GRID 17.5 driver without issues.

The note section mentions

For Azure NVads A10 v5 VMs we recommend customers to always be on the latest driver version. The latest NVIDIA major driver branch(n) is only backward compatbile with the previous major branch(n-1). For eg, vGPU 17.x is backward compatible with vGPU 16.x only. Any VMs still runnig n-2 or lower may see driver failures when the latest drive branch is rolled out to Azure hosts. NVs_v3 VMs only support vGPU 16 or lower driver version. Windows server 2019 support is only available till vGPU 16.x.

https://learn.microsoft.com/en-us/azure/virtual-machines/windows/n-series-driver-setup

Since manual installation works, you can disable the extension and manually install the driver every time a VM is deployed.
https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-windows#known-issues

Hope this helps!
Henrik Aldermo 56 Reputation points

2025-03-18T10:04:16.8766667+00:00

Hi @Mounika Reddy Anumandla and thanks for the reply. I agree the underlying issue seems to be driver incompatibility with Windows 2019 and that installing an older driver would be a solution.

However, in my case the affected system is a virtual machine scale set, so installing drivers manually on every new instance is not viable.

I found a working solution by specifying the driver version in the NVidia Driver Extension, as documented in the "Known Issues" section here: https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-windows#known-issues

For a Virtual Machine Scale set, use the following:

az vmss extension set --resource-group MyResourceGroup --vmss-name MyVmss --name NvidiaGpuDriverWindows --publisher Microsoft.HpcCompute --settings "{'driverVersion':'538.46'}"

To view the settings:

az vmss extension list --resource-group MyResourceGroup --vmss-name MyVmss

Answer 1

Hi Henrik Aldermo,

Glad the issue is resolved for you. I will have this answer promoted by reposting it. This is in the attempt to help others looking for a solution for a similar issue.

The extension (Microsoft.HpcCompute.NvidiaGpuDriverWindows) always installs the latest GRID driver. The extension might be trying to install a driver that is incompatible with Windows Server 2019.

Windows Server 2019 does support the GPU but may require a specific driver version that the extension doesn’t install correctly. For links to all previous Nvidia GRID driver versions, visit GitHub.

Since Windows Server 2022 is officially supported, the extension installs the correct GRID 17.5 driver without issues.

However, in your case the affected system is a virtual machine scale set, so installing drivers manually on every new instance is not viable.

A working solution by specifying the driver version in the NVidia Driver Extension, as documented in the "Known Issues" section here: https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-windows#known-issues

For a Virtual Machine Scale set, use the following:

az vmss extension set --resource-group MyResourceGroup --vmss-name MyVmss --name NvidiaGpuDriverWindows --publisher Microsoft.HpcCompute --settings "{'driverVersion':'538.46'}"

To view the settings:

az vmss extension list --resource-group MyResourceGroup --vmss-name MyVmss

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Share via

NVidia drivers not installed by extension on Windows 2019 Scale Set NV6ads A10 v5

0 additional answers

Your answer