Manage emergency access to a Bare Metal Machine using the az networkcloud cluster bmckeyset

There are rare situations where a user needs to investigate & resolve issues with a Bare Metal Machine and all other ways using Azure are exhausted. Operator Nexus provides the az networkcloud cluster bmckeyset command so users can manage SSH access to the baseboard management controller (BMC) on these bare metal machines. On keyset creation, users are validated against Microsoft Entra ID for proper authorization by cross referencing the User Principal Name provided for a user against the supplied Azure Group ID --azure-group-id <Entra Group ID>.

Users in a keyset are validated every four hours, and also when any changes are made to any keyset. Each user's status is then set to "Active" or "Invalid." Invalid users remain in the keyset but their keys are removed from all hosts and they aren't allowed access. Reasons for a user being invalid are:

• The user's User Principal Name isn't specified
• The user's User Principal Name isn't a member of the given Microsoft Entra group
• The given Microsoft Entra group doesn't exist (in which case all users in the keyset are invalid)
• The keyset is expired (in which case all users in the keyset are invalid)

The keyset and each individual user also have detailed status messages communicating other information:

• The keyset's detailedStatusMessage tells you whether the keyset is expired, and other information about problems encountered while updating the keyset across the cluster.
• The user's statusMessage tells you whether the user is active or invalid, and a list of machines that aren't yet updated to the user's latest active/invalid state. In each case, causes of problems are included if known.

When the command runs, it executes on each Bare Metal Machine in the Cluster with an active Kubernetes node. There's a reconciliation process that runs periodically that retries the command on any Bare Metal Machine that wasn't available at the time of the original command. Also, any Bare Metal Machine that returns to the cluster via an az networkcloud baremetalmachine actionreimage or az networkcloud baremetalmachine actionreplace command (see BareMetal functions) sends a signal causing any active keysets to be sent to the machine as soon as it returns to the cluster. Multiple commands execute in the order received.

Prerequisites

• Install the latest version of the appropriate CLI extensions.
• The on-premises Cluster must have connectivity to Azure.
• Get the Resource Group name for the Cluster resource.
• The process applies keysets to all running bare metal machines.
• The users added must be part of a Microsoft Entra group. For more information, see How to Manage Groups.
• To restrict access for managing keysets, create a custom role. For more information, see Azure Custom Roles. In this instance, add or exclude permissions for Microsoft.NetworkCloud/clusters/bmcKeySets. The options are /read, /write, and /delete.

Limitations

BMC Keyset User limitations

While the BMCs support a maximum number of 16 users, 5 are reserved for system use leaving 11 for BMC Keyset Users. BMC Keyset Users are defined on a per Cluster basis and applied to each Bare Metal Machine. Attempts to add more than 11 users results in an error. Delete a user before adding another one when 11 already exist.

Here's a mapping of the BMC slots to users.

Microsoft Entra group user limitations

Using a Microsoft Entra group ID with greater than 5,000 users isn't recommended. Reconciling a large number of users can result in time-outs, blocking access and causing sign-in issues.

Creating a BMC Keyset

The bmckeyset create command creates SSH access to the Bare Metal Machine in a Cluster for a group of users.

The command syntax is:

az networkcloud cluster bmckeyset create \
  --name <BMC Keyset Name> \
  --extended-location name=<Extended Location ARM ID> \
    type="CustomLocation" \
  --location <Azure Region> \
  --azure-group-id <Azure AAD Group ID> \
  --expiration <Expiration Timestamp> \
  --privilege-level <"Administrator" or "ReadOnly"> \
  --user-list '[{"description":"<User List Description>","azureUserName":"<User Name>",\
    "sshPublicKey":{"keyData":"<SSH Public Key>"}, \
    "userPrincipalName":""}]', \
  --tags key1=<Key Value> key2=<Key Value> \
  --cluster-name <Cluster Name> \
  --resource-group <Resource Group Name>

Create Arguments

  --azure-group-id                            [Required] : The object ID of Azure Active Directory
                                                           group that all users in the list must
                                                           be in for access to be granted. Users
                                                           that are not in the group do not have
                                                           access.
  --bmc-key-set-name --name -n                [Required] : The name of the BMC key set.
  --cluster-name                              [Required] : The name of the cluster.
  --expiration                                [Required] : The date and time after which the users
                                                           in this key set are removed from
                                                           the BMCs. The maximum expiration date is a
                                                           year from creation date. Format is
                                                           "YYYY-MM-DDTHH:MM:SS.000Z".
  --extended-location                         [Required] : The extended location of the cluster
                                                           associated with the resource.
    Usage: --extended-location name=XX type=XX
      name: Required. The resource ID of the extended location on which the resource is created.
      type: Required. The extended location type: "CustomLocation".
  --privilege-level                           [Required] : The access level allowed for the users
                                                           in this key set.  Allowed values:
                                                           "Administrator" or "ReadOnly".
  --resource-group -g                         [Required] : Name of resource group. Optional if
                                                           configuring the default group using `az
                                                           configure --defaults group=<name>`.
  --user-list                                 [Required] : The unique list of permitted users.
    Usage: --user-list azure-user-name=XX description=XX key-data=XX
      azure-user-name: Required. User name used to login to the server.
      description: The free-form description for this user.
      key-data: Required. The public ssh key of the user.
      userPrincipalName: Required. The User Principal Name of the User.

      Multiple users can be specified by using more than one --user-list argument.
  --tags                                                 : Space-separated tags: key[=value]
                                                           [key[=value] ...]. Use '' to clear
                                                           existing tags.
  --location -l                                          : Azure Region. Values from: `az account
                                                           list-locations`. You can configure the
                                                           default location using `az configure
                                                           --defaults location=<location>`.
  --no-wait                                              : Do not wait for the long-running
                                                           operation to finish.

Global Azure CLI arguments (applicable to all commands)

  --debug                                                : Increase logging verbosity to show all
                                                           debug logs.
  --help -h                                              : Show this help message and exit.
  --only-show-errors                                     : Only show errors, suppressing warnings.
  --output -o                                            : Output format.  Allowed values: json,
                                                           jsonc, none, table, tsv, yaml, yamlc.
                                                           Default: json.
  --query                                                : JMESPath query string. See
                                                           http://jmespath.org/ for more
                                                           information and examples.
  --subscription                              [Required] : Name or ID of subscription. Optional if
                                                           configuring the default subscription
                                                           using `az account set -s NAME_OR_ID`.
  --verbose                                              : Increase logging verbosity. Use --debug
                                                           for full debug logs.

This example creates a new keyset with two users that have standard access from two jump hosts.

az networkcloud cluster bmckeyset create \
  --name "bmcKeySetName" \
  --extended-location name="/subscriptions/subscriptionId/resourceGroups/resourceGroupName/providers/Microsoft.ExtendedLocation/customLocations/clusterExtendedLocationName" \
    type="CustomLocation" \
  --location "location" \
  --azure-group-id "f110271b-XXXX-4163-9b99-214d91660f0e" \
  --expiration "2023-12-31T23:59:59.008Z" \
  --privilege-level "Standard" \
  --user-list '[{"description":"Needs access for troubleshooting as a part of the support team",\
  "azureUserName":"userABC","sshPublicKey":{"keyData":"ssh-rsa  AAtsE3njSONzDYRIZv/WLjVuMfrUSByHp+jfaaOLHTIIB4fJvo6dQUZxE20w2iDHV3tEkmnTo84eba97VMueQD6OzJPEyWZMRpz8UYWOd0IXeRqiFu1lawNblZhwNT/ojNZfpB3af/YDzwQCZgTcTRyNNhL4o/blKUmug0daSsSXISTRnIDpcf5qytjs1XoyYyJMvzLL59mhAyb3p/cD+Y3/s3WhAx+l0XOKpzXnblrv9d3q4c2tWmm/SyFqthaqd0= admin@vm"}},\
  {"description":"Needs access for troubleshooting as a part of the support team",\
  "azureUserName":"userXYZ","sshPublicKey":{"keyData":"ssh-rsa  AAtsE3njSONzDYRIZv/WLjVuMfrUSByHp+jfaaOLHTIIB4fJvo6dQUZxE20w2iDHV3tEkmnTo84eba97VMueQD6OzJPEyWZMRpz8UYWOd0IXeRqiFu1lawNblZhwNT/ojNZfpB3af/YDzwQCZgTcTRyNNhL4o/blKUmug0daSsSXTSTRnIDpcf5qytjs1XoyYyJMvzLL59mhAyb3p/cD+Y3/s3WhAx+l0XOKpzXnblrv9d3q4c2tWmm/SyFqthaqd0= admin@vm"}}]' \
  --tags key1="myvalue1" key2="myvalue2" \
  --cluster-name "clusterName" \
  --resource-group "resourceGroupName"

For assistance in creating the --user-list structure, see Azure CLI Shorthand.

Deleting a BMC Keyset

The bmckeyset delete command removes SSH access to the BMC for a group of users. All members of the group lose SSH access to any of the BMCs in the Cluster.

The command syntax is:

az networkcloud cluster bmckeyset delete \
  --name <BMC Keyset Name> \
  --cluster-name <Cluster Name> \
  --resource-group <Resource Group Name> \

Delete Arguments

  --bmc-key-set-name --name -n                [Required] : The name of the BMC key set to be deleted.
  --cluster-name                              [Required] : The name of the cluster.
  --resource-group -g                         [Required] : Name of resource group. Optional if configuring the
                                                           default group using `az configure --defaults
                                                           group=<name>`.
  --no-wait                                              : Do not wait for the long-running operation to finish.
  --yes -y                                               : Do not prompt for confirmation.

This example removes the "bmcKeysetName" keyset group in the "clusterName" Cluster.

az networkcloud cluster bmckeyset delete \
  --name "bmcKeySetName" \
  --cluster-name "clusterName" \
  --resource-group "resourceGroupName" \

Updating a BMC Keyset

The bmckeyset update command allows users to make changes to an existing keyset group.

The command syntax is:

az networkcloud cluster bmckeyset update \
  --name <BMC Keyset Name> \
  --privilege-level <"Standard" or "Superuser"> \
  --user-list '[{"description":"<User List Description>","azureUserName":"<User Name>",\
    "sshPublicKey":{"keyData":"<SSH Public Key>"}, \
    "userPrincipalName":""}]', \
  --tags key1=<Key Value> key2=<Key Value> \
  --cluster-name <Cluster Name> \
  --resource-group <Resource Group Name>

Update Arguments

  --bmc-key-set-name --name -n                [Required] : The name of the BMC key set.
  --cluster-name                              [Required] : The name of the cluster.
  --expiration                                [Required] : The date and time after which the users
                                                           in this key set are removed from
                                                           the BMCs. The maximum expiration date is a
                                                           year from creation date. Format is
                                                           "YYYY-MM-DDTHH:MM:SS.000Z".
  --privilege-level                                      : The access level allowed for the users
                                                           in this key set.  Allowed values:
                                                           "Administrator" or "ReadOnly".
  --user-list                                            : The unique list of permitted users.
    Usage: --user-list azure-user-name=XX description=XX key-data=XX
      azure-user-name: Required. User name used to login to the server.
      description: The free-form description for this user.
      key-data: Required. The public SSH key of the user.
      userPrincipalName: Required. The User Principal Name of the User.

      Multiple users can be specified by using more than one --user-list argument.
  --resource-group -g                         [Required] : Name of resource group. Optional if
                                                           configuring the default group using `az
                                                           configure --defaults group=<name>`.
  --tags                                                 : Space-separated tags: key[=value]
                                                           [key[=value] ...]. Use '' to clear
                                                           existing tags.
  --no-wait                                              : Do not wait for the long-running
                                                           operation to finish.

This example adds two new users to the "bmcKeySetName" group and changes the expiry time for the group.

az networkcloud cluster bmckeyset update \
  --name "bmcKeySetName" \
  --expiration "2023-12-31T23:59:59.008Z" \
  --user-list '[{"description":"Needs access for troubleshooting as a part of the support team",\
    "azureUserName":"userDEF", \
    "sshPublicKey":{"keyData":"ssh-rsa  AAtsE3njSONzDYRIZv/WLjVuMfrUSByHp+jfaaOLHTIIB4fJvo6dQUZxE20w2iDHV3tEkmnTo84eba97VMueQD6OzJPEyWZMRpz8UYWOd0IXeRqiFu1lawNblZhwNT/ojNZfpB3af/YDzwQCZgTcTRyNNhL4o/blKUmug0daSsSXISTRnIDpcf5qytjs1XoyYyJMvzLL59mhAyb3p/cD+Y3/s3WhAx+l0XOKpzXnblrv9d3q4c2tWmm/SyFqthaqd0= admin@vm"}, \
    "userPrincipalName":"[email protected]"}] \
  --cluster-name "clusterName" \
  --resource-group "resourceGroupName"

Listing BMC Keysets

The bmckeyset list command allows users to see the existing keyset groups in a Cluster.

The command syntax is:

az networkcloud cluster bmckeyset list \
  --cluster-name <Cluster Name> \
  --resource-group <Resource Group Name>

List Arguments

  --cluster-name                              [Required] : The name of the cluster.
  --resource-group -g                         [Required] : Name of resource group. Optional if
                                                             configuring the default group using `az
                                                             configure --defaults group=<name>`.

Show BMC Keyset Details

The bmckeyset show command allows users to see the details of an existing keyset group in a Cluster.

The command syntax is:

az networkcloud cluster bmckeyset show \
  --cluster-name <Cluster Name> \
  --resource-group <Resource Group Name>

Show Arguments

  --bmc-key-set-name --name -n                [Required] : The name of the BMC key set.
  --cluster-name                              [Required] : The name of the cluster.
  --resource-group -g                         [Required] : Name of resource group. You can
                                                           configure the default group using `az
                                                           configure --defaults group=<name>`.