The HPA Does Nothing Until You Are 10% Past Target

A HorizontalPodAutoscaler set to averageUtilization: 80 does not hold your workload at 80%. It holds it anywhere between 72% and 88% and calls that done — and while it is waiting for new pods to start, it will refuse to add more no matter how hot the running ones get.

The HorizontalPodAutoscaler, HPA for short, is a controller that runs every 15 seconds and writes one number: the replica count on your Deployment's scale subresource. It has no model of your traffic and no memory of what it did last time. Each run it divides the average CPU utilization of the pods it has metrics for by your target, and multiplies. Everything people find surprising about it comes from the two tests it applies to that ratio before it acts.

The panel below runs ten minutes of a real HPA loop against a workload whose demand triples over five minutes. Four pods to start, a target of 80% of the CPU request, every default the Kubernetes control plane ships. The log prints the ratio at every sync and the reason for what it did.

Leave everything alone and read the log from 1:45 onwards. At 1:45 the average is 78% and the ratio is 0.975 — inside the band, no change. At 2:00 it is 84%, ratio 1.050 — still inside the band, still no change. The workload is running a twentieth over the number you configured and the controller has decided that is close enough. Only at 2:15, at 90% and a ratio of 1.125, does it add a pod.

That is the first test, and it is why 294 of the 600 seconds are spent above the target you set. Drag tolerance to 0.30 — the far end, but a value people really do set to stop flapping — and the same ten minutes gives 490 seconds above target, a peak of 126%, and eight replicas instead of nine, from two scaling actions instead of five. Drag it to 0 and you get 248 seconds and a peak of 102%. The tolerance is not a smoothing filter. It is the width of the region in which the HPA is not an autoscaler.

The ratio, and the test that throws it away

The documented formula is one line:

desiredReplicas = ceil[ currentReplicas * ( currentMetricValue / desiredMetricValue ) ]

currentMetricValue for a resource metric is not CPU seconds and not a percentage of the node. It is the average, across the pods the controller has metrics for, of each pod's CPU usage as a fraction of that pod's CPU request. A container with no CPU request has no denominator, and the HPA reports FailedGetResourceMetric rather than guessing — see requests, limits and what actually enforces them for why the request is the number that matters here and the limit is not.

Then comes the test. In replica_calculator.go:

if tolerances.isWithin(usageRatio) {
    // return the current replicas if the change would be too small
    return currentReplicas, utilization, rawUtilization, timestamp, nil
}

isWithin is |1 - ratio| ≤ tolerance, and the tolerance comes from --horizontal-pod-autoscaler-tolerance on the kube-controller-manager, default 0.1. With a target of 80% that makes the band 72% to 88%. Nothing inside it produces an action, an event, or a log line. The HPA object still reports 90%/80% in kubectl get hpa while doing nothing, which is the exact display that gets filed as a bug: issue 78761, HPA doesn't scale down to minReplicas even though metric is under target, collected 113 comments and 89 reactions over three years and was closed as not planned.

The band is a fixed fraction, so its cost grows with the deployment. KEP-4951, the enhancement that made the tolerance configurable per HPA, motivates itself in one sentence: for large deployments, a 10% tolerance translates into very significant resources (i.e. hundreds of pods). On a 1,000-replica deployment the band is 100 replicas wide. Since Kubernetes 1.35 the HPAConfigurableTolerance feature gate is beta and on by default, so you can write the narrower band directly on the object:

behavior:
  scaleUp:
    tolerance: 0.02
  scaleDown:
    tolerance: 0.05

Asymmetry is the point. A tight scale-up tolerance makes the controller react to a small overshoot; a loose scale-down tolerance stops it giving the capacity back the moment the ratio dips. Setting both to zero is available in the panel and is not a good idea — you get an action on every 15-second sync, and every action starts pods that will not report for the next minute.

The pods it refuses to count

Switch the traffic to spike and read the log at 1:30. Twelve replicas exist, four of them are reporting a metric, and those four are at 240% of request. The controller does nothing — and it does nothing again at 1:45, when eight are reporting at 120%. The reason it gives both times is arithmetic, not policy.

Before computing anything, groupPods splits the pod list. Pods that are deleting or Failed are ignored. Pods that are Pending, or unready, or still inside --horizontal-pod-autoscaler-cpu-initialization-period (five minutes, during which a CPU metric is not trusted even if metrics-server has one), or unready and inside --horizontal-pod-autoscaler-initial-readiness-delay (30 seconds) are set aside. The ratio is computed over what is left. Then, if that ratio came out above 1.0 — if the answer was going to be a scale-up — the set-aside pods are put back at a usage of zero, and the ratio is computed again. The documentation calls this conservatively assuming the not-yet-ready pods are consuming 0% of the desired metric, further dampening the magnitude of a scale up.

It does not dampen the magnitude. Look at what the log prints: ratio 1.000 is inside the band — no change. It would still have asked for 12. That is not a coincidence of this simulation, it is algebra. The final line of the calculator is ceil(newUsageRatio × len(metrics)), and newUsageRatio is total usage divided by total requests divided by the target. When every pod in a Deployment has the same request — which is what a Deployment means — adding a pod at zero usage adds nothing to the numerator and exactly cancels in the denominator. The number the controller would ask for is unchanged. What the substitution changes is whether the second tolerance test lets it ask at all.

You can check this. Untick count metric-less pods and re-read the ramp. The same nine replicas, the same five scaling actions, 293 seconds above target instead of 294, and 4,219 pod-seconds instead of 4,159 — one pod, one minute earlier, across the whole ten minutes. The counters swap round: nineteen dead-band holds and ten metric-less holds become twenty-three dead-band holds and none. The substitution is a gate, and it is a gate that mostly agrees with the gate already standing in front of it.

The other direction is the one that costs money, and it is not symmetric. When the ratio comes out below 1.0, the pods without a metric are not put back at the target. They are put back at 100% of their CPU request:

// on a scale-down, treat missing pods as using 100% (all) of the resource request
// or the utilization target for targets higher than 100%
fallbackUtilization := int64(max(100, targetUtilization))
for podName := range missingPods {
    metrics[podName] = metricsclient.PodMetric{Value: requests[podName] * fallbackUtilization / 100}
}

Read that carefully against the documentation, which says the controller assumes the missing pods are consuming 100% of the desired value in case of a scale down. The desired value sounds like your target, and it is not: max(100, targetUtilization) is 100 for every target at or under 100%, which is every target anyone writes, and the 100 is a percentage of the pod's request. The documented sentence is exact only for the other code path — calcPlainMetricReplicas, which serves averageValue and custom metrics, really does substitute targetUsage. For the averageUtilization target this lesson is about, a pod with no metric is assumed to be flat out, not merely at target — and the algebra bites harder than the symmetry suggests. Working it through: ceil(newUsageRatio × len(metrics)) expands to ceil(ratio × reporting + missing ÷ target). The pods without a metric enter that sum as missing ÷ target, not as missing — the single ceil at the end is applied to the whole total, so the inflation is 1 ÷ target per silent pod before rounding: 1.25 at a target of 80%, two at a target of 50%. A rolling update with 5 pods mid-restart therefore asks for six or seven replicas above what the load justifies, which is the mechanism behind issue 72775, Pod HPA creates extra pods during deployment rolling update with no load — reported as a bug, open for three years, and working as designed.

Watch it happen. Set traffic to spike, drag scaleDown stabilizationWindowSeconds to 0 so the window is not what is holding the count up, and drag seconds until a new pod reports to 240. Read the log at 4:00, the sync after the spike ends. Four reporting pods sit at 60% of request, a ratio of 0.750, and every part of that says scale down. The eight pods with no metric yet are counted at 100% of request, the ratio comes back as 1.083 — over 1.0, on the other side of the dead band — and the controller holds twelve replicas for a workload that needs three. It repeats that decision five syncs running.

Note what would have happened if the fallback really were the target. The recomputed ratio would be 0.917: still inside the band, still no change, but pointing down rather than up. The difference between assuming a silent pod is at target and assuming it is saturated is the difference between a controller that is nearly ready to shrink and one that thinks it is nearly ready to grow.

The control that actually moves the number

Put the traffic back on ramp and leave the tolerance alone. Drag seconds until a new pod reports from 45 to 240.

• 5 seconds: 215 seconds above target, peak 93%.
• 45 seconds: 294 seconds above target, peak 108%.
• 120 seconds: 369 seconds above target, peak 138%.
• 240 seconds: 489 seconds above target, peak 180%.

The replica trajectory is identical in all four — four pods rising to nine, five scaling actions, the same 4,159 pod-seconds of capacity. Only the pain changes, and it more than doubles. No autoscaler setting is in that list. The tolerance moved the peak from 108% to 102% at its most aggressive; pod startup time moved it from 93% to 180%.

The reason is structural. The average is taken over pods that are Ready, and a pod that is not Ready is not in the Service's endpoints either — so it takes no traffic while it is starting, which keeps the running pods at exactly the utilization that caused the scale-up in the first place. The HPA then spends the next several syncs looking at a number that cannot improve yet. Every second of container image pull, JVM warm-up, connection-pool fill and readiness-probe initialDelaySeconds is a second the autoscaler spends blind.

This is also the boundary where the whole mechanism gives up. If your pods need two minutes to become useful and your traffic doubles in thirty seconds, no value of tolerance, behavior or target will save the request that arrives at second forty. The HPA is a reactive controller with a dead time equal to your startup time. The fixes that work are the ones that shorten it or pre-empt it: a smaller image, a readiness probe that is honest rather than padded, or capacity you provisioned before you needed it.

The five minutes after the spike

Switch traffic to spike. Demand goes to four times normal at 1:00 and back to normal at 4:00. Watch what the replica chart does after 4:00: nothing, for five minutes.

The default behavior.scaleDown.stabilizationWindowSeconds is 300, and the rule is that the applied replica count is the maximum recommendation seen inside that window. The controller stores a recommendation on every sync, including the syncs where the tolerance stopped it doing anything, so the window does not start counting when the spike ends — it starts counting from the last sync that still recommended the high number. In the panel that is the 3:45 sync — the last one before the traffic fell — and the drop lands at 9:00, five minutes and one sync later.

Priced in the panel's own units: 6,184 pod-seconds with the default window, 3,784 with the window set to zero. The difference is exactly 2,400 pod-seconds — eight surplus pods held for the full five minutes. Whether that is worth paying depends entirely on whether the spike comes back, which is the trade the window exists to make. What it is not is a bug, and it is not the HPA is stuck. Set the window to 150 seconds and the same run costs 4,984 pod-seconds; set it to 600 and the ten minutes ends with all twelve replicas still running.

Note also which pods leave. The controller only writes a number; the ReplicaSet decides who dies, and its ordering prefers unready and newest-first pods — see Deployments and ReplicaSets for why that matters during a rollout. And the pods the HPA adds are only useful once the scheduler can place them; if it cannot, they sit Pending and the autoscaler's job passes to the cluster autoscaler.

Checking it in a running system

Read the conditions, not the events. kubectl describe hpa <name> prints three conditions and they answer three different questions. AbleToScale false means the controller cannot reach the scale subresource at all. ScalingActive false with reason FailedGetResourceMetric means the metric is missing — usually a container in the pod with no CPU request, which is enough to disqualify the whole pod. ScalingLimited true means the number it wanted was clipped, and the message names the clipper: TooManyReplicas, TooFewReplicas, or ScaleDownStabilized. That last one is the five-minute window, stated in plain words, and it is the single most useful string on the page.

Compute the band before you tune anything. Take the target from kubectl get hpa and multiply by 0.9 and 1.1. If the current utilization is inside that interval, the HPA is behaving exactly as written and no amount of restarting metrics-server will change it. If you want it narrower, the field is spec.behavior.scaleUp.tolerance on Kubernetes 1.35 and later, and the cluster-wide fallback is --horizontal-pod-autoscaler-tolerance.

Measure the dead time directly. The number that dominated every result above is the gap between a pod being created and a pod reporting a usable metric. Take a pod's status.startTime, the lastTransitionTime of its Ready condition, and the timestamp of its first kubectl top pod reading. In practice:

kubectl get pod -o jsonpath='{.status.startTime} {range .status.conditions[?(@.type=="Ready")]}{.lastTransitionTime}{end}'

If that gap is over a minute, that is your autoscaler's reaction time and nothing in the HPA spec can shorten it.

Watch it decide, once. kubectl get hpa <name> -w alongside kubectl get events --field-selector involvedObject.name=<name> -w for one real traffic event is worth more than any dashboard. You are looking for the interval between a SuccessfulRescale event and the next one. If it is a multiple of 15 seconds with nothing in between while utilization is visibly high, you are watching the two gates in this lesson, in order.

Two things follow from here. The pods an HPA creates have to land somewhere, and when they do not, the cluster autoscaler is the component that decides whether to buy a node — with a completely different notion of what "not enough capacity" means. And when the cluster is full and something has to give, the ordering is set by pod priority and preemption, not by which workload scaled up first. If a container in your pod has no CPU request at all, start instead with QoS classes — the HPA and the kubelet are both reading the same missing field.