KEDA scale-to-zero issues
KEDA HTTP Add-on (v0.10.0) Issues in AKS
Problem Description
We're experiencing issues with the KEDA HTTP Add-on (v0.10.0) in our AKS cluster. The main problem manifests in two ways:
- When routing requests through the KEDA HTTP interceptor, we consistently get 502 Bad Gateway errors after about 22 seconds. The interesting part is that direct connections to our target pods work fine, but the interceptor seems to be dropping the connection.
- Despite configuring our HTTPScaledObject with
min: 0
, the HPA that KEDA creates always setsminReplicas: 1
. This prevents us from achieving true scale-to-zero functionality.
Environment
We're running this on an AKS cluster with GPU nodes, using KEDA v2.17.0 with the HTTP Add-on v0.10.0. Our setup includes an Nginx Ingress Controller and a GPU-based deployment that exposes port 5000.
Current Setup
Our request path looks like this:
Ingress -> ExternalName Service -> KEDA Interceptor Service -> Target Deployment
We've configured the HTTPScaledObject with min: 0
and max: 5
, and set various timeout values (900s for interceptor and Nginx) to handle long-running requests.
What We've Found
When investigating the 502 errors, we discovered that while Nginx can successfully connect to the Interceptor Service, the requests never make it to our target pods. The interceptor pods don't show any logs of receiving these requests, and our target pods remain untouched. However, if we bypass the interceptor and connect directly to our target pods, everything works as expected.
For the scale-to-zero issue, we've confirmed that the HTTPScaledObject is correctly configured with min: 0
, but the HPA that KEDA creates always sets minReplicas: 1
. This seems to be a consistent behavior regardless of our configuration.
Questions
- Is the 22-second timeout a known limitation or bug in the KEDA HTTP Add-on?
- Why does the request path break down between Nginx and the Interceptor Service?
- Is there a specific configuration needed to enable proper request routing through the Interceptor?
- Why does the HPA ignore the min: 0 setting from the HTTPScaledObject?
- Are there any known issues with the KEDA HTTP Add-on v0.10.0 that could explain these behaviors?