Flux in a Hub $ Spoke Architecture - How to access remote AKS Cluster with Azure RBAC enabled and local_user access disabled

Hi @Akshay kumar Mandha ,

I noticed that only the source-controller and kustomize-controller have the expected labels and annotations related to workload identity in place. However, the helm-controller does not, despite being deployed in the same way. I set up all controllers using Terraform, and the setup process was identical across them.

Here are the steps I’ve already taken to troubleshoot:

Verified that all federated identity credentials are correctly configured in the UI for the user-assigned identity.

Double-checked the azurerm_federated_identity_credential resources for all three controllers (source-controller, kustomize-controller, and helm-controller) and confirmed they are being created using a loop over the same list.

Inspected the deployed annotations and labels on the controllers — only helm-controller is missing them.

Confirmed that kubelogin is not installed on all controllers (which might be related).

Here is a snippet of my Terraform configuration:

flux_service_accounts  = ["source-controller", "helm-controller", "kustomize-controller"]
resource "azurerm_federated_identity_credential" "fluxControllerFederation" {
  for_each = toset(var.flux_service_accounts)
  name     = "flux_${each.key}_k8s"
  ...
  subject  = "system:serviceaccount:flux-system:${each.value}"
}

and the Flux extension configuration:

resource "azapi_resource" "fluxExtensions" {
  ...
  configurationSettings = {
    "workloadIdentity.enable"        = "true"
    "workloadIdentity.azureClientId" = azurerm_user_assigned_identity.fluxSourceControllerIdentity.client_id
  }
}

Hi Hering, Martin
Based on you latest comment missing helm controller missing. you can inspect the helm-controller service account configuration using the following command:
kubectl get serviceaccount helm-controller -n flux-system -o yaml

Please refer the below Kubernetes document for command line

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server

Confirm that the Azure client ID is correctly linked for helm-controller
Check the Azure Client ID from Flux Extension:

Run the following command to retrieve the Azure Client ID configured for Flux

az k8s-extension show \
  --resource-group <YOUR_RESOURCE_GROUP> \
  --cluster-name <YOUR_CLUSTER_NAME> \
  --name flux

Look for the workloadIdentity.azureClientId value in the output. This is the client ID used by Flux
Verify the Federated Identity Credential:

Run the following command to list the federated identity credentials associated with the user-assigned identity

az identity federated-credential list \
  --identity-name <YOUR_USER_ASSIGNED_IDENTITY_NAME> \
  --resource-group <YOUR_RESOURCE_GROUP>

Please refer the below commands line documents
az connectedk8s show
az identity federated-credential

To set up kubelogin, first install the Azure Kubernetes CLI by running the command az aks install-cli. Once that’s done, you’ll use the command kubelogin convert-kubeconfig to update your kubeconfig file so it can handle authentication methods like workload identity. You’ll need to provide some specific information, like your Microsoft Entra tenant ID (--tenant-id), the app’s client ID (--client-id), and the server ID (--server-id).
Please refer the below document for more information
How authentication works
If you have any further query, please let me know

Hi Akshay kumar Mandha,

Thanks for your quick response.

I have a quick follow-up question regarding the Helm controller:

• Is it really necessary to manually configure the service account for the Helm controller?
• Is there no way to automate this setup, similar to how it works with the Source and Kustomize controllers?
• Why does it work out of the box for Source and Kustomize controllers, but not for the Helm controller,m when setting this up via terraform?

I'll try out the approach you suggested and will get back to you once I've tested it.

Thanks again and best regards,

Martin

Hi Hering, Martin,
Thanks for response back to us
updating on the testing things if you have any other query, please let me know

Hi Akshay kumar Mandha,

I tested it, but it wouldn’t work. I configured the Helm Controller the same way—using the appropriate service account and adding the label azure.workload.identity/use: "true". I also updated the service account to enable workload identity. After restarting the Helm Controller deployment, the Helm Controller pod did not inherit these values.

I don’t understand why it works out of the box for the Source and Kustomize controllers but not for the Helm Controller. Is there a way to make this work with my Terraform setup, or is this a known issue?

I also found this discussion on GitHub: https://github.com/fluxcd/flux2/discussions/4153

Best regards

Martin

Hi Hering, Martin,
Thank you for your reply and for trying this out. I truly appreciate your efforts. Have you reviewed the Kustomize and Helm controller logs? Did you notice any other errors? If there's any additional information you may have missed, please feel free to share it so I can work on this more effectively. If the information you’ve already provided covers everything, that would be great

Hi Hering, Martin,
Just I wanted to check on my previous comment, please review it once and let me know

Hi Hering, Martin,
If you have any update on my previous response, please let me know if you have any other query

Hi Akshay kumar Mandha, i will run some additional testing tonight. I will let you know my results as soon as possible.

Beste regards

Martin