nvidia-smi command could communicate with nvidia driver (Microsoft azure N series)

Hi Nilesh Maharjan,
Thanks for patience while we are reviewing your question.

Based on the query, I understand your concern why it happens at that time it might be issue when secure boot not enabled properly, I suggest that they check if Secure Boot is enabled correctly. This may be the cause of the problem. If Secure Boot is enabled properly, they can try restarting the VM in the Azure portal.

For more little bit details, during the installation of the NVIDIA GPU drivers, they will need to enroll the signing key because it belongs to a kernel module. When Secure Boot is enabled, they have to import the key, which is similar to the Machine Owner Key (MOK). After importing the MOK, they should reboot the VM

For more information, please refer to the documents and forum below. If you're already familiar with them, that's great
Ubuntu
Installing the NVIDIA Driver
How to install nvidia driver with secure boot enabled?

For the reboot loop with Secure Boot disabled Restarting the VM from the Azure portal can sometimes help resolve this. If the reboot loop continues, this could indicate a configuration issue.

Disclaimer: For third party links Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there

If you need any further assistance or additional information, please don't hesitate to let us know. We're here to support you and are always ready to help with anything to provide support whenever you need it.

Problem Statement:
I was getting the message NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. whenever I ran nvidia-smi. The error was appearing because of a corrupted installation due to Secure Boot.

When you try to turn off Secure Boot through the terminal, it won't be turned off. Also, you won't be prompted to reenter the password.

Solution:

1. Turn off Secure Boot from the Azure platform, not the terminal. (Your VM page => Settings => Configration => Uncheck "Enable secure boot")
2. Remove your corrupted driver installations

      sudo apt remove --purge '^nvidia-.*'
      sudo apt autoremove
      sudo apt clean
   

3. Install the driver after a sudo update.

      sudo apt update
      sudo apt install -y nvidia-driver-570
      sudo reboot
   

4. Run nvidia-smi to see your properly installed driver details.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 NVL                Off |   00000001:00:00.0 Off |                    0 |
| N/A   40C    P0             64W /  400W |       0MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+