Edit

Share via

Azure VMware Solution Software-Defined Data Center (SDDC) maintenance best practices

  • Article

Azure VMware Solution undertakes periodic maintenance of the private cloud. This maintenance includes security patches, minor and major updates to VMware software stack. This page describes the host monitoring, remediation, and recommended best practices that help keep the private cloud ready for maintenance.

Host maintenance and lifecycle management

One benefit of Azure VMware Solution private clouds is that the platform is maintained for you. Microsoft is responsible for the lifecycle management of VMware software (ESXi, vCenter Server, and vSAN) and NSX appliances. Microsoft is also responsible for bootstrapping the network configuration, like creating the Tier-0 gateway and enabling North-South routing. You’re responsible for the NSX SDN configuration: network segments, distributed firewall rules, Tier 1 gateways, and load balancers.

Note

A T0 gateway is created and configured as part of a private cloud deployment. Any modification to that logical router or the NSX edge node VMs could affect connectivity to your private cloud and should be avoided.

Microsoft is responsible for applying any patches, updates, or upgrades to ESXi, vCenter Server, vSAN, and NSX in your private cloud. The impact of patches, updates, and upgrades on ESXi, vCenter Server, and NSX has the following considerations:

  • ESXi - There's no impact to workloads running in your private cloud. Access to vCenter Server and NSX isn't blocked during this time. During this time, we recommend you don't plan other activities like: scaling up private cloud, scheduling or initiating active HCX migrations, making HCX configuration changes, and so on, in your private cloud.

  • vCenter Server - There's no impact to workloads running in your private cloud. During this time, vCenter Server is unavailable and you can't manage VMs (stop, start, create, or delete). We recommend you don't plan other activities like scaling up private cloud, creating new networks, and so on, in your private cloud. When you use VMware Site Recovery Manager or vSphere Replication user interfaces, we recommend you don't do either of the actions: configure vSphere Replication, and configure or execute site recovery plans during the vCenter Server upgrade.

  • NSX - The workload is impacted. When a particular host is being upgraded, the VMs on that host might lose connectivity from 2 seconds to 1 minute with any of the following symptoms:

    • Ping errors

    • Packet loss

    • Error messages (for example, Destination Host Unreachable and Net unreachable)

    During this upgrade window, all access to the NSX management plane is blocked. You can't make configuration changes to the NSX environment for the duration. Your workloads continue to run as normal, subject to the upgrade impact previously detailed.

    During the upgrade time, we recommend you don't plan other activities like, scaling up private cloud, and so on, in your private cloud. Other activities can prevent the upgrade from starting or could have adverse impacts on the upgrade and the environment.

You're notified through Azure Service Health that includes the timeline of the upgrade. This notification also provides details on the upgraded component, its effect on workloads, private cloud access, and other Azure services. You can reschedule an upgrade as needed.

Software updates include:

  • Patches - Security patches or bug fixes released by VMware

  • Updates - Minor version change of a VMware stack component

  • Upgrades - Major version change of a VMware stack component

Note

Microsoft tests a critical security patch as soon as it becomes available from VMware.

Documented VMware workarounds are implemented in lieu of installing a corresponding patch until the next scheduled updates are deployed.

Host monitoring and remediation

Azure VMware Solution continuously monitors the health of both the VMware components and underlay. When Azure VMware Solution detects a failure, it takes action to repair the failed components. When Azure VMware Solution detects a degradation or failure on an Azure VMware Solution node, it triggers the host remediation process.

Host remediation involves replacing the faulty node with a new healthy node in the cluster. Then, when possible, the faulty host is placed in VMware vSphere maintenance mode. VMware vSphere vMotion moves the VMs off the faulty host to other available servers in the cluster, potentially allowing zero downtime for live migration of workloads. If the faulty host can't be placed in maintenance mode, the host is removed from the cluster. Before the faulty host is removed, the customer workloads are migrated to a newly added host.

Tip

Customer communication: An email is sent to the customer's email address before the replacement is initiated and again after the replacement is successful.

To receive emails related to host replacement, you need to be added to any of the following Azure Role-Based Access Control (RBAC) roles in the subscription: 'ServiceAdmin', 'CoAdmin', 'Owner', 'Contributor'.

Azure VMware Solution monitors the following conditions on the host:

  • Processor status
  • Memory status
  • Connection and power state
  • Hardware fan status
  • Network connectivity loss
  • Hardware system board status
  • Errors occurred on one or more disks of a vSAN host
  • Hardware voltage
  • Hardware temperature status
  • Hardware power status
  • Storage status
  • Connection failure

Maintenance Operations Best Practices

The following actions are always recommended for ensuring host maintenance operations are carried out successfully:

  • vSAN storage utilization: To maintain Service Level Agreement (SLA), ensure that your vSphere cluster's storage space utilization remains below 75%. If the utilization exceeds 75%, upgrades may take longer than expected or fail entirely. If your storage utilization exceeds 75%, consider adding a node to expand the cluster and prevent potential downtime during upgrades.
  • Distributed Resource Scheduler (DRS) rules: DRS VM-VM anti-affinity rules must be configured in a way to have at least (N+1) hosts in the cluster, where N is the number of VMs part of DRS rule.
  • Failures To Tolerate (FTT) violation: To prevent data loss, change VMs configured with a vSAN storage policy for Failures to Tolerate (FTT) of 0 to a vSAN storage policy compliant with Microsoft SLA (FTT=1 for up to five hosts in a cluster and FTT=2 for six or more hosts in a cluster) and ensure host maintenance operations can carried out seamlessly.
  • Remove VM CD-ROM mounts: VMs mounted with "Emulate mode" CD-ROMs block host maintenance. Ensure CD-ROMs are mounted in "Passthrough mode".
  • Serial/parallel port or external device: If you're using an image file (ISO, FLP, etc.), ensure that it's accessible from all ESXi hosts in the cluster. Store the files on a datastore that are shared between all ESXi Servers that participate in the vMotion of the virtual machine. For more information, see Broadcom KB article.
  • Orphaned VMs: In the case of an orphaned virtual machine, the Virtual Machine (VM) needs to be either re-registered if possible (if it hasn't been deleted) or removed from inventory. For more information, see Broadcom KB article.
  • SCSI shared controller: When using SCSI bus sharing use with bus type as "Physical" for VMs. VMs connected to Virtual SCSCI controllers will be powered-off. For more information, see Broadcom KB article.
  • Third-party VMs & applications: For third-party VMs & applications:
    • Ensure that third-party solutions deployed on Azure VMware Solution are compliant and don't interfere with maintenance operations.
    • Ensure that the VM isn’t installed with a VM-Host "Must run" DRS rule. Additionally, verify that these applications are compatible with upcoming versions of the VMware stack.
    • Consult with your solution vendor and update in advance if necessary to maintain compatibility post-upgrade.

Alert Codes and Remediation Table

Error Code Error Details Recommended Action
EPC_CDROM_EMULATEMODE This error is encountered when CD-ROM on the Virtual Machine uses emulate mode, whose ISO image isn't accessible Follow this KB article for the removal of any CDROM mounted on customer's workload Virtual Machines in emulate mode or detach ISO. It's recommended to use "Passthrough mode" for mounting any CD-ROM.
EPC_DRSOVERRIDERULE This error is encountered when there's a Virtual Machine with DRS Override set to "Disabled" mode. VM shouldn't block vMotion while putting host into maintenance. Set Partially Automated DRS rules for the VM. Refer to this document to know more about VM placement policies.
EPC_SCSIDEVICE_SHARINGMODE This error is encountered when a Virtual Machine is configured to use a device that prevents a maintenance operation: A device that is a SCSI controller which is engaged in bus-sharing Follow this KB article for the removal of any SCSI controller engaged in bus-sharing attached to VMs
EPC_DATASTORE_INACCESSIBLE This error is encountered when any external Datastore attached to AVS Private Cloud becomes inaccessible Follow this article for the removal of any stale Datastore attached to cluster
EPC_NWADAPTER_STALE This error is encountered when connected Network interface on the Virtual Machine uses network adapter which becomes inaccessible Follow this KB article for the removal of any stale N/W adapters attached to Virtual Machines
EPC_SERIAL_PORT This error is encountered when a Virtual Machine’s serial port is connected to a device that can't be accessed on the destination host. If you're using an image file (ISO, FLP, and so on), ensure that it's accessible from all ESXi servers on the cluster. Store the files on a data store that is shared between all ESXi servers that participate in vMotion of the virtual machine. Refer to this KB article from Broadcom for more information.
EPC_HARDWARE_DEVICE This error is encountered when a Virtual Machine’s parallel Port/USB Device is connected to a device can't be accessed on the destination host. If you're using an image file (ISO, FLP, and so on), ensure that it's accessible from all ESXi servers of the cluster. Store the files on a data store that is shared between all ESXi servers that participate in the vMotion of the virtual machine. Refer to this KB article from Broadcom for more information.
EPC_INVALIDVM / EPC_ORPHANVM This error is encountered when there's an orphaned or Invalid VM in the inventory Ensure all your Virtual Machines are accessible to the vCenter. Refer to this KB article for more information

Note

Azure VMware Solution tenant admins must not edit or delete the previously defined VMware vCenter Server alarms because they're managed by the Azure VMware Solution control plane on vCenter Server. These alarms are used by Azure VMware Solution monitoring to trigger the Azure VMware Solution host remediation process.

Next steps

Now that you've covered Azure VMware Solution private cloud maintenance best practices, you might want to learn about: