Container instance with managed identity is not working with storage account and other Azure services

Question

Container instance with managed identity is not working with storage account and other Azure services

Serhii Divnych 45

Hi. I'm facing issues with container instances for 2 subscriptions in East US. Code deployed to container instance is using managed identity access to storage account, and to Azure AI services multi-service account. Beginning from 9th of March managed identity access stoped working, and container instances are not working getting below errors from logs

      Execute threw an unhandled exception.
      Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Service request failed.
      Status: 500 (Internal Server Error)
      
      Content:
      Received invalid token. Please try again.
      
      
      Headers:
      X-Content-Type-Options: REDACTED
      Date: Wed, 12 Mar 2025 11:53:22 GMT
      Content-Type: text/plain; charset=utf-8
      Content-Length: 42
      
      See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
       ---> Azure.RequestFailedException: Service request failed.
      Status: 500 (Internal Server Error)
      
      Content:
      Received invalid token. Please try again.
      
      
      Headers:
      X-Content-Type-Options: REDACTED
      Date: Wed, 12 Mar 2025 11:53:22 GMT
      Content-Type: text/plain; charset=utf-8
      Content-Length: 42
      
         at Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, HttpMessage message, CancellationToken cancellationToken)
         at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, HttpMessage message, CancellationToken cancellationToken)

Suresh Estharakula 400 Reputation points Microsoft External Staff

2025-03-12T15:07:17.3833333+00:00
Hi Serhii Divnych,

The error message suggests that the ManagedIdentityCredential is encountering authentication failures when attempting to access Azure services, leading to a 500 Internal Server Error. This generally indicates a problem with either the managed identity configuration or the service itself.

Based on our understanding of your issue, we recommend that you refer to the following methods for troubleshooting:

The managed identity might not be correctly assigned to the Azure Container Instance.

There could be misconfigurations present within the Azure environment or the identity settings.

Service Availability: The Azure service you are attempting to access may be facing issues or experiencing downtime.

It is essential to verify that the managed identity possesses the required permissions to access the Azure Storage Account and other services. Additionally, review the Role-Based Access Control (RBAC) settings.

If you have access to the container, you can test the managed identity endpoint using the following command:

curl 'http://169.254.169.254/metadata/identity/oauth2/token?resource=https://management.core.windows.net&api-version=2018-02-01' -H "Metadata: true"

Make sure that the managed identity has the appropriate permissions to access the Azure Storage Account or other services. Common roles include:Storage Blob Data Contributor/Storage Account Contributor

Assign a role to a managed identity using Azure CLI:

Assign the Storage Blob Data Contributor role to the managed identity

az role assignment create --assignee <managed-identity-client-id> --role "Storage Blob Data Contributor" --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>

Please look into this document for more information:

https://learn.microsoft.com/en-us/azure/container-instances/container-instances-managed-identity?source=recommendations

https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-assign-app-role-managed-identity?pivots=identity-mi-app-role-powershell

https://learn.microsoft.com/en-us/entra/identity-platform/multi-service-web-app-access-storage?tabs=azure-portal%2Cprogramming-language-csharp

If you found it helpful, could you kindly click the “upvote” on the post.

If you have any further queries, please let us know we are glad to help you.
Toni Hoang 90 Reputation points

2025-03-12T22:21:46.7233333+00:00

I have the same problem. When both System Assigned and User Assigned are used, I can get tokens directly only, when I provide the User assigned client id with the client_id parameter. Trying to using the System Assigned identity and removing client_id parameter gives me 500. After deleting the User assigned identity it works again, but then again I lost the User Assigned identity.
Brijan Elwadhi 10 Reputation points

2025-03-13T01:46:38.3566667+00:00

We are facing the exact same issue
Rasmus Nyeland 0 Reputation points

2025-03-13T08:26:26.8866667+00:00

We are facing the issue and tried what you are listing below, to mitigate this we switched to Toni Hoang approach as well
Serhii Divnych 45 Reputation points

2025-03-13T08:49:51.1033333+00:00

Toni Hoang workaround works for me as well
Suresh Estharakula 400 Reputation points Microsoft External Staff

2025-03-13T16:53:19.6566667+00:00
Hi Serhii Divnych,

The error message indicates "Received invalid token," which could suggest a problem related to token expiration or management. Ensure that your application is properly managing token refreshes or retries in the event of a failed authentication request. While the Managed Identity SDKs generally take care of this, it is advisable to review your code's logic to confirm that retry policies are implemented.

It only happens when both System Assigned and User Assigned Identity are active. As a temporary workaround removing the User Assigned Identity seems to make it work again.

It appears that you are encountering a problem with the managed identity of your Azure container instance, specifically when utilizing both System Assigned and User Assigned managed identities simultaneously. Based on your description, it seems that while you are able to obtain tokens by providing the client ID of the User Assigned identity, attempting to use the System Assigned identity without including the client ID results in a 500 error.

The occurrence of a 500 error along with an invalid token may suggest that the System Assigned Managed Identity is not being properly recognized when both identities are utilized. This issue could be associated with the method by which tokens are obtained through the SDK. In situations where both identities are activated, there can occasionally be ambiguity in the resolution of tokens.

az account get-access-token --identity
Toni Hoang 90 Reputation points

2025-03-13T17:13:54.37+00:00

In my case the system was running fine for more than a year. It just changed yesterday morning and we only noticed in production. So it is not a configuration error.
Saarivirta, Jussi 20 Reputation points

2025-03-14T09:31:11.8066667+00:00

We've ran into this as well. Our services have been running fine for over a year.

This is definitely caused by something that Microsoft has changed.

Edit: to clarify, our services are running in West Europe.
Yuto Takei 0 Reputation points MVP

2025-03-17T06:48:01.5233333+00:00

I also saw the issues in two different subscriptions in Japan East as of Mar 14. We also experienced the same issue and the workaround from Toni Hoang worked for me.
Suresh Estharakula 400 Reputation points Microsoft External Staff

2025-03-18T13:47:10.55+00:00

Hi Serhii Divnych,

Just checking in to see if the below answer provided by Toni Hoang helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Tom Gielow 5 Reputation points

2025-04-09T19:02:47.89+00:00

This issue appeared about a month ago and it is still an issue as stated above. When both system and user identities are assigned, call to get identity fails with 500: internal server error.

I confirmed with Microsoft that this is a known bug, and the workaround is to only use user managed identity until the fix is rolled out. I did not get any timeframe on that.

Accepted answer

0 additional answers

Your answer

Toni Hoang 90 Reputation points

2025-03-12T22:21:46.7233333+00:00

I have the same problem. When both System Assigned and User Assigned are used, I can get tokens directly only, when I provide the User assigned client id with the client_id parameter. Trying to using the System Assigned identity and removing client_id parameter gives me 500. After deleting the User assigned identity it works again, but then again I lost the User Assigned identity.
Brijan Elwadhi 10 Reputation points

2025-03-13T01:46:38.3566667+00:00

We are facing the exact same issue
Rasmus Nyeland 0 Reputation points

2025-03-13T08:26:26.8866667+00:00

We are facing the issue and tried what you are listing below, to mitigate this we switched to Toni Hoang approach as well
Serhii Divnych 45 Reputation points

2025-03-13T08:49:51.1033333+00:00

Toni Hoang workaround works for me as well
Suresh Estharakula 400 Reputation points Microsoft External Staff

2025-03-13T16:53:19.6566667+00:00

Hi Serhii Divnych,

The error message indicates "Received invalid token," which could suggest a problem related to token expiration or management. Ensure that your application is properly managing token refreshes or retries in the event of a failed authentication request. While the Managed Identity SDKs generally take care of this, it is advisable to review your code's logic to confirm that retry policies are implemented.

It only happens when both System Assigned and User Assigned Identity are active. As a temporary workaround removing the User Assigned Identity seems to make it work again.

It appears that you are encountering a problem with the managed identity of your Azure container instance, specifically when utilizing both System Assigned and User Assigned managed identities simultaneously. Based on your description, it seems that while you are able to obtain tokens by providing the client ID of the User Assigned identity, attempting to use the System Assigned identity without including the client ID results in a 500 error.

The occurrence of a 500 error along with an invalid token may suggest that the System Assigned Managed Identity is not being properly recognized when both identities are utilized. This issue could be associated with the method by which tokens are obtained through the SDK. In situations where both identities are activated, there can occasionally be ambiguity in the resolution of tokens.

az account get-access-token --identity
Toni Hoang 90 Reputation points

2025-03-13T17:13:54.37+00:00

In my case the system was running fine for more than a year. It just changed yesterday morning and we only noticed in production. So it is not a configuration error.
Saarivirta, Jussi 20 Reputation points

2025-03-14T09:31:11.8066667+00:00

We've ran into this as well. Our services have been running fine for over a year.

This is definitely caused by something that Microsoft has changed.

Edit: to clarify, our services are running in West Europe.
Yuto Takei 0 Reputation points MVP

2025-03-17T06:48:01.5233333+00:00

I also saw the issues in two different subscriptions in Japan East as of Mar 14. We also experienced the same issue and the workaround from Toni Hoang worked for me.
Suresh Estharakula 400 Reputation points Microsoft External Staff

2025-03-18T13:47:10.55+00:00

Hi Serhii Divnych,

Just checking in to see if the below answer provided by Toni Hoang helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Tom Gielow 5 Reputation points

2025-04-09T19:02:47.89+00:00

This issue appeared about a month ago and it is still an issue as stated above. When both system and user identities are assigned, call to get identity fails with 500: internal server error.

I confirmed with Microsoft that this is a known bug, and the workaround is to only use user managed identity until the fix is rolled out. I did not get any timeframe on that.

Answer 1

Toni Hoang 90

To have it as an answer: it seems it only happens when both System Assigned and User Assigned Identity are active. As a temporary workaround removing the User Assigned Identy seems to make it work again.

Tom Gielow 5 Reputation points

2025-04-10T07:32:26.3266667+00:00

I have also confirmed this works when just using user assigned identity

Share via

Container instance with managed identity is not working with storage account and other Azure services

0 additional answers

Your answer