All four of our Managed Redis instances reject connections

Question

All four of our Managed Redis instances reject connections

Jesper Lindholm 25

(This is about Azure Managed Redis; there is no child tag for it, so I have to tag it as Azure Cache for Redis.)

Yesterday at separate times, all four of our Managed Redis instances stopped accepting connections. It is impossible to connect to them. According to metrics, the client connections dipped to zero:

User's image

...and data points for Used memory and Total Keys are completely missing: User's image

This happened on all of our instances, separately, within an hour or so. All are hosted in West Europe.

I do not wish to post the resource names here.

Attempts to scale to another size failed, likely because the scaling process can't access the database either.

And the Export data part of the portal does not list any storage accounts, despite there being several in the same resource group as the instance, to which I have access: User's image

Then again, considering that this happened to all four of our instances independently and over time, I have slim hope that importing the data in a new instance would work.

Preview or not preview, this failure and the lack of action for over 20 hours is completely unacceptable.

There is no note on Azure status nor on Azure Service Health.

I would have filed a support ticket, but am unable to do so because of the reseller through which we acquire Azure services. This policy also seems very ill-advised – how are we supposed to get in touch with the technical support running Managed Redis when something like this happens?

Pending a postmortem explaining what went wrong, I would not be able to recommend Azure Managed Redis to the simplest and least consequential of toy examples.

Mallaiah Sangi 655 Reputation points Microsoft External Staff Moderator

2025-04-21T11:44:24.4866667+00:00
Hi Jesper Lindholm

Greetings!

We apologize for the inconvenience caused.

As we understand that, you are unable to connect Managed Redis instances.

Here some common reason for a cache disconnects.

Client-side causes

The client application was redeployed.

The client application did a scaling operation.

Web Apps might cause a cache to disconnect during autoscaling.

The networking layer on the client side changed.

Transient errors occurred in the client or in the network nodes between the client and the server.

The bandwidth threshold limits were reached.

CPU bound operations took too long to complete.

Server-side causes

The Azure Managed Redis service started a fail-over from the primary node to the replica node.

Azure was patching the instance where the cache was deployed during a Redis server update or general VM maintenance. Note: Please let me know if any above changes happened. If possible, kindly share the error log details to assist with deeper analysis. I hope this information helps. Please do let us know if you have any further queries.
Jesper Lindholm 25 Reputation points

2025-04-21T13:09:37.0733333+00:00

Thanks for the reply.

Seeming as how metrics have stopped being reported for "Used memory" and "Total keys", it appears that the Redis instance is no longer running at all. (See above.) So I am waiting for the Managed Redis service to recover. I am not privy to any details whatsoever that I can find, and do not have access to any restart/stop/start commands. The logs show nothing except that my attempt at scaling one of the four instances failed, presumably from a timeout; since scaling was not attempted at three of the four instances, it can't possibly have been the trigger.

It's possible that patching the instance or a similar backend action was the trigger. However, since this is a managed service, and I am now looking at 24 hours of four unrelated instances being unavailable, I do not know what I am supposed to do with this information; I hope that this is being escalated and that someone is looking into it.

I realize that this could be sped up with resource names. I do not wish to share the exact names here - I would like to be able to do so with Microsoft Support, but as mentioned we are unable to create support tickets. Two of the instances are located in the subscription ID * and two in the subscription ID * The email address connected to the account that I am commenting from has access to these resources in our Entra setup.
Mallaiah Sangi 655 Reputation points Microsoft External Staff Moderator

2025-04-21T13:51:31.29+00:00

Hi Jesper Lindholm

Kindly request you to please share the details requested over the private message for further investigation.
Jesper Lindholm 25 Reputation points

2025-04-21T14:51:47.58+00:00

Have done so now - thanks!
Mallaiah Sangi 655 Reputation points Microsoft External Staff Moderator

2025-04-22T09:16:15.4133333+00:00

Hi Jesper Lindholm

Thank you for providing the requested details.

We are attempting to create a support plan ticket for backend assistance using your subscription ID. However, we are unable to retrieve the support plan details. Could you please confirm which subscription ID has an active support plan so we can proceed with the support request?
Jesper Lindholm 25 Reputation points

2025-04-22T09:33:50.7133333+00:00

My guess is that, since we are unable to create support tickets, there is no active support plan, since we use Azure through a reseller.

Thanks for your patience and assistance.

After 42 hours of downtime, concurrent with metrics that indicate downtime that could have been acted on within minutes, after experiencing data loss and being unable to authenticate when the services came back up, and after being unable to even use the built-in export functionality, we are giving up on this service and migrating to Redis Cloud.

Hopefully the product team will see this as an example to drill down into the many things that went wrong, or many further customers will be affected in the future.

Your answer

Mallaiah Sangi 655 Reputation points Microsoft External Staff Moderator

2025-04-21T11:44:24.4866667+00:00

Hi Jesper Lindholm

Greetings!

We apologize for the inconvenience caused.

As we understand that, you are unable to connect Managed Redis instances.

Here some common reason for a cache disconnects.

Client-side causes

The client application was redeployed.

The client application did a scaling operation.

Web Apps might cause a cache to disconnect during autoscaling.

The networking layer on the client side changed.

Transient errors occurred in the client or in the network nodes between the client and the server.

The bandwidth threshold limits were reached.

CPU bound operations took too long to complete.

Server-side causes

The Azure Managed Redis service started a fail-over from the primary node to the replica node.

Azure was patching the instance where the cache was deployed during a Redis server update or general VM maintenance. Note: Please let me know if any above changes happened. If possible, kindly share the error log details to assist with deeper analysis. I hope this information helps. Please do let us know if you have any further queries.
Jesper Lindholm 25 Reputation points

2025-04-21T13:09:37.0733333+00:00

Thanks for the reply.

Seeming as how metrics have stopped being reported for "Used memory" and "Total keys", it appears that the Redis instance is no longer running at all. (See above.) So I am waiting for the Managed Redis service to recover. I am not privy to any details whatsoever that I can find, and do not have access to any restart/stop/start commands. The logs show nothing except that my attempt at scaling one of the four instances failed, presumably from a timeout; since scaling was not attempted at three of the four instances, it can't possibly have been the trigger.

It's possible that patching the instance or a similar backend action was the trigger. However, since this is a managed service, and I am now looking at 24 hours of four unrelated instances being unavailable, I do not know what I am supposed to do with this information; I hope that this is being escalated and that someone is looking into it.

I realize that this could be sped up with resource names. I do not wish to share the exact names here - I would like to be able to do so with Microsoft Support, but as mentioned we are unable to create support tickets. Two of the instances are located in the subscription ID * and two in the subscription ID * The email address connected to the account that I am commenting from has access to these resources in our Entra setup.
Mallaiah Sangi 655 Reputation points Microsoft External Staff Moderator

2025-04-21T13:51:31.29+00:00

Hi Jesper Lindholm

Kindly request you to please share the details requested over the private message for further investigation.
Jesper Lindholm 25 Reputation points

2025-04-21T14:51:47.58+00:00

Have done so now - thanks!
Mallaiah Sangi 655 Reputation points Microsoft External Staff Moderator

2025-04-22T09:16:15.4133333+00:00

Hi Jesper Lindholm

Thank you for providing the requested details.

We are attempting to create a support plan ticket for backend assistance using your subscription ID. However, we are unable to retrieve the support plan details. Could you please confirm which subscription ID has an active support plan so we can proceed with the support request?
Jesper Lindholm 25 Reputation points

2025-04-22T09:33:50.7133333+00:00

My guess is that, since we are unable to create support tickets, there is no active support plan, since we use Azure through a reseller.

Thanks for your patience and assistance.

After 42 hours of downtime, concurrent with metrics that indicate downtime that could have been acted on within minutes, after experiencing data loss and being unable to authenticate when the services came back up, and after being unable to even use the built-in export functionality, we are giving up on this service and migrating to Redis Cloud.

Hopefully the product team will see this as an example to drill down into the many things that went wrong, or many further customers will be affected in the future.

Share via

All four of our Managed Redis instances reject connections

Your answer