Issue with azure arc enabled server configured with private link

Question

Issue with azure arc enabled server configured with private link

Mido5 10

We are experiencing an issue across multiple Azure Arc-enabled servers (various configurations and customers , differents dns server ) that are using Azure Arc Private Link. Since April 1st, the following URL:

https://francecentral-gas.guestconfiguration.azure.com

has been returning a 404 error Nginx instead of the expected 200 response.

No changes were made on these virtual machines, and the issue appears to affect only environments using Private Link. We have compared with another customer using public Azure Arc endpoints, and everything works fine in that setup.

Is anyone else facing this issue or aware of a resolution?

Luke Murray 11,351 Reputation points MVP

2025-04-05T05:14:18.4633333+00:00

Hi, I am not aware of the issue, however do you have a Firewall in Azure and user defined routes configured? If so make sure network policy is enabled. I've seen it do some weird things with Arc private endpoint traffic if not used.
Mido5 10 Reputation points

2025-04-05T07:33:50.3033333+00:00

Hi Luke ,

We have observed that since April 1st, numerous Azure Arc-enabled servers across different tenants and configurations are unable to perform assessments with Azure Update Manager ( the issue started the same time for all servers , howerver they are not on the infrastructure and no change was made ) . Attempts to reinstall the extension remain stuck in the "Creating" state. Upon reviewing the logs, we identified that the issue is widespread across servers with varying configurations, all beginning simultaneously. It appears that private endpoints can no longer reach the URL francecentral-gas.guestconfiguration.azure.com. In contrast, servers configured to use public endpoint function correctly.

As mentioned, the configurations vary significantly across tenants, but they all share one common factor: they are using Azure Arc with Private Link.
Christopher Stephens 0 Reputation points

2025-04-07T19:06:06.28+00:00

Same problem here, only different URLs (as shown below), accessed via Arc private link scopes/private endpoints. All 404 from nginx as it's trying to retrieve extensions and their updates from the guestconfiguration URLs, hence the hanging "Updating", "Creating", "Deleting" messages on extensions affected by this problem. Logs show it started around 5:00p CST 4/2/25 for centralus. I have an escalated ticket open with Azure support on this at the moment.

https://agentserviceapi.guestconfiguration.azure.com/
https://centralus-gas.guestconfiguration.azure.com
https://eastus2-gas.guestconfiguration.azure.com/
https://northcentralus-gas.guestconfiguration.azure.com/
Srinivasa Reddy Jaggavarapu 700 Reputation points Microsoft External Staff

2025-04-09T06:44:31.88+00:00

Hi Mido5,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Christopher Stephens 0 Reputation points

2025-04-09T11:22:21.2333333+00:00

Support reached back out to me yesterday ~4:00p CST advising the problem was indeed on the Azure side and was fixed. All my Arc machines across all three regions they’re in magically started working again. No change required on my side.
Srinivasa Reddy Jaggavarapu 700 Reputation points Microsoft External Staff

2025-04-10T08:14:33.5433333+00:00

Hi Mido5,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Srinivasa Reddy Jaggavarapu 700 Reputation points Microsoft External Staff

2025-04-11T02:07:47.0533333+00:00

Hi Mido5,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Luke Murray 11,351 Reputation points MVP

2025-04-05T05:14:18.4633333+00:00

Hi, I am not aware of the issue, however do you have a Firewall in Azure and user defined routes configured? If so make sure network policy is enabled. I've seen it do some weird things with Arc private endpoint traffic if not used.
Mido5 10 Reputation points

2025-04-05T07:33:50.3033333+00:00

Hi Luke ,

We have observed that since April 1st, numerous Azure Arc-enabled servers across different tenants and configurations are unable to perform assessments with Azure Update Manager ( the issue started the same time for all servers , howerver they are not on the infrastructure and no change was made ) . Attempts to reinstall the extension remain stuck in the "Creating" state. Upon reviewing the logs, we identified that the issue is widespread across servers with varying configurations, all beginning simultaneously. It appears that private endpoints can no longer reach the URL francecentral-gas.guestconfiguration.azure.com. In contrast, servers configured to use public endpoint function correctly.

As mentioned, the configurations vary significantly across tenants, but they all share one common factor: they are using Azure Arc with Private Link.
Christopher Stephens 0 Reputation points

2025-04-07T19:06:06.28+00:00

Same problem here, only different URLs (as shown below), accessed via Arc private link scopes/private endpoints. All 404 from nginx as it's trying to retrieve extensions and their updates from the guestconfiguration URLs, hence the hanging "Updating", "Creating", "Deleting" messages on extensions affected by this problem. Logs show it started around 5:00p CST 4/2/25 for centralus. I have an escalated ticket open with Azure support on this at the moment.

https://agentserviceapi.guestconfiguration.azure.com/
https://centralus-gas.guestconfiguration.azure.com
https://eastus2-gas.guestconfiguration.azure.com/
https://northcentralus-gas.guestconfiguration.azure.com/
Srinivasa Reddy Jaggavarapu 700 Reputation points Microsoft External Staff

2025-04-09T06:44:31.88+00:00

Hi Mido5,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Christopher Stephens 0 Reputation points

2025-04-09T11:22:21.2333333+00:00

Support reached back out to me yesterday ~4:00p CST advising the problem was indeed on the Azure side and was fixed. All my Arc machines across all three regions they’re in magically started working again. No change required on my side.
Srinivasa Reddy Jaggavarapu 700 Reputation points Microsoft External Staff

2025-04-10T08:14:33.5433333+00:00

Hi Mido5,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Srinivasa Reddy Jaggavarapu 700 Reputation points Microsoft External Staff

2025-04-11T02:07:47.0533333+00:00

Hi Mido5,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Hi Mido5,

As per your description, this problem probably results from modifications to the backend configuration of Azure or DNS resolution affecting Private Link endpoints. The Nginx 404 error indicates requests are making it to the endpoint but are not being properly routed or resolved to the desired service.

Verify that DNS settings for Private Link endpoints are correctly configured. Azure Private Link requires specific DNS configurations to resolve private endpoint IP addresses to Azure services. Reference: https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns

Verify the Guest Configuration service for Azure Arc is correctly registered in the Azure region you use. PowerShell command to confirm:

Get-AzProviderFeature -ProviderNamespace "Microsoft.HybridCompute" -FeatureName "GuestConfiguration"

Use tools like curl, Invoke-WebRequest, or Azure Network Watcher to test the connectivity from the affected servers to the Private Link endpoint. If the DNS resolves correctly, but the 404 persists, it indicates a routing issue at the Azure service level.

These steps will help you to identify and resolve the issues affecting your Azure Arc-enabled servers configured with Private Link.

Refer to the following Microsoft documentation links to cross check and validate the recommended steps:
https://learn.microsoft.com/en-us/azure/azure-arc/servers/private-link-security#troubleshooting
https://learn.microsoft.com/en-us/azure/update-manager/troubleshoot?tabs=azure-machines#unable-to-generate-periodic-assessment-for-arc-enabled-servers

Hope the above provided information help you resolve the issue, if you have any further concerns or queries, please feel free to reach out to us.

Share via

Issue with azure arc enabled server configured with private link

1 answer

Your answer