Orphaned ACR Resources Preventing Container Apps from Launching

We consolidated our Images into a single Container Registry on Azure and ran into some issues with orphaned resources that are preventing some container apps from launching

I redacted the names of the apps:

• resourcegroup is the RG name, and resourcegroupacr is just the name of the acr prefixed with our old resourcegroup name
• oldacr1 is our old ACR that we can't remove for some reason
• appname is just the name of the app we're to launch a revision of.

I might have been a bit to hasty in deleting the old container registry resourcegroupacr.azurecr.io and it looks like some resources were tied to it. I should have confirmed these fired up before deleting the old registries, as some just hang and didn't launch after I moved over the revisions to point at the new ACR and images. Now it looks like we have some references to container registries that I can't get rid of.

The registries I tried to delete were oldacr1.azurecr.io and resourcegroupacr.azurecr.io. resourcegroupacr.azurecr.io is gone, but I get errors deleting oldacr1.azurecr.io. As you can see in the following errors, the container app still references the deleted registry.

The error we get when re-deploying the Container App revision with the proper tag and Authentication selected is:

Failed to deploy new revision: PasswordSecretRef 'reg-pswd-8d81bbc3-8d27' defined for registry server 'oldacr1.azurecr.io' not found

I'm not selecting oldacr1.azurecr.io as my repo (this is the one I wish I could delete), so this is a strange error. I checked to see if we had

I confirmed this with this command:

az containerapp show \                                                
--name resourcegroup-appname \  
--resource-group resourcegroup \  
--query "properties.configuration.registries"

Output - Ideally only want newacr.azurecr.io:

[
  {
    "identity": "/subscriptions/xxx/resourceGroups/resourcegroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-resourcegroup-secretuser",
    "passwordSecretRef": "",
    "server": "resourcegroupacr.azurecr.io",
    "username": ""
  },
  {
    "identity": "/subscriptions/xxx/resourcegroups/resourcegroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-resourcegroup-secretuser",
    "passwordSecretRef": "",
    "server": "newacr.azurecr.io",
    "username": ""
  },
  {
    "identity": "",
    "passwordSecretRef": "reg-pswd-8d81bbc3-8d27",
    "server": "oldacr1.azurecr.io",
    "username": "oldacr1"
  }
]

I tried to clean up the resources explicitly by setting the registries with AzureCLI and an update to the container app.

az containerapp update \
  --name resourcegroup-appname \
  --resource-group resourcegroup \
  --set configuration.registries="[]"

Running ths 'show' command again, I still see the same output (the three registries)

az containerapp show \                                                
--name resourcegroup-appname \  
--resource-group resourcegroup \  
--query "properties.configuration.registries"

I even tried to update it specifically to use the single container app registry that we actually want.

az containerapp update \       
  --name resourcegroup-appname \
  --resource-group resourcegroup \
  --set configuration.registries="[{'server':'newacr.azurecr.io','identity':'/subscriptions/xxx/resourcegroups/resourcegroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-resourcegroup-secretuser'}]"

Show will indicate the same registries are attached.

I'm out of things to try. Deleting and recreating our container apps isn't sustainable, and I'd like to avoid this as strongly as possible.

Any help would be appreciated. Thank you in advance!