CPU Requests, Limits and CFS Throttling

A CPU limit does not slow your container down. It stops it. The limit is converted into a budget that refills every 100 milliseconds, and twelve threads working against a 1.2 CPU limit spend that entire budget in ten milliseconds. For the remaining ninety, the container does not run at all.

Nothing in the Kubernetes documentation says this. It says limits are "enforced by CPU throttling" and that a container "may not use more CPU than is specified in its cpu limit", which reads like a governor on an engine. The kubelet actually writes cpu.cfs_quota_us and cpu.cfs_period_us into the container's cgroup — a single cpu.max file on cgroup v2 — and the Linux CFS bandwidth controller enforces them the only way it can: by dequeuing every runnable thread in the group the instant the quota reaches zero, and leaving them dequeued until the period timer fires.

The consequence is that throttling is a function of how fast you spend CPU, not how much. Below is the mechanism. The pod is serving a steady 30 requests per second and averages roughly a third of its limit for the whole run. Drag worker threads from 1 up to 16 and watch that average sit still while the throttling readouts go from nothing to a quarter of every period.

At one thread the pod is never throttled, and cannot be: a single runnable thread burns at most 100 ms of CPU in a 100 ms period, which is under a 1.2 CPU quota by construction. At sixteen threads a quarter of all periods contain a stall, the longest of them 83 ms, and the pod is stopped for 100 ms out of every second. Average CPU goes from 30% of the limit to 31%. It cannot do anything else: the pod did exactly the same amount of work, and the only thing that changed is that it did it in twelve milliseconds instead of a hundred and forty-four.

Requests and limits are two unrelated mechanisms

Drag resources.requests.cpu across its whole range. The throttling readouts do not change by a single millisecond. That is not a simplification in the model; it is the actual division of labour, and it is visible in twelve lines of the kubelet.

ResourceConfigForPod takes the CPU request and converts it with MilliCPUToShares — milliCPU × 1024 / 1000, floored at 2 and capped at 262144 — and writes it to cpu.shares. It takes the CPU limit and converts it with MilliCPUToQuota — milliCPU × period / 1000, floored at 1000 µs — and writes it to cpu.cfs_quota_us. Two different fields, two different kernel knobs, no interaction.

• Shares are a weight, and only bind under contention. They decide what fraction of a busy CPU you get relative to your neighbours. On an idle node, a pod with the floor of 2 shares runs exactly as fast as a pod with the ceiling of 262,144.
• Quota is absolute, and binds always. It applies whether the node is saturated or completely idle. This is the asymmetry people find counterintuitive: in the default setup above the pod is stopped for 83 ms at a stretch while using 0.37 of the node's 16 cores, with nothing else competing for any of them.

The request is also the only one of the two the scheduler reads. It fits pods to nodes by summing requests against allocatable capacity; the limit is not part of that arithmetic at all, which is why raising a limit never moves a pod and why a node can be 100% "requested" while its CPUs idle. Whether the two numbers are equal is what decides the QoS class, and therefore eviction order.

On cgroup v2 the names change and the weight conversion becomes lossy. Shares become cpu.weight on a 1–10000 scale, and the kubelet maps them with 1 + (shares − 2) × 9999 / 262142. A 300m request becomes 307 shares becomes weight 12; a 3000m request becomes 3072 shares becomes weight 118. Quota and period are packed into a single cpu.max file as "$MAX $PERIOD", with the literal string max meaning unlimited. The panel above shows both encodings. Bandwidth control underneath is the same scheduler code either way, so every mechanism in this lesson behaves identically on v1 and v2 — what actually changed is the resolution of the weight, which now has ten thousand steps where shares had a quarter of a million, so two pods whose requests differ slightly can land on the same weight.

The period is the unit, and the average hides it

Set threads to 16 and step through the three traffic shapes. All three deliver 30 requests per second and all three average the same CPU. Smooth arrivals never throttle. Clumps of five never throttle either. Clumps of twelve throttle a quarter of all periods. Every dashboard most teams have would show these as the same workload.

The reason is arithmetic on a 100 ms window. A batch of twelve requests at 12 ms each is 144 ms of CPU. With one worker thread that is 144 ms of wall clock spread over two periods, 72 ms in each, comfortably under a 120 ms quota. With sixteen threads it is 144 ms of CPU demanded inside about twelve milliseconds, from a pool holding 120. The pod is not doing more work; it is doing the same work inside one enforcement window instead of two.

That is why a Prometheus query averaging over five minutes cannot predict throttling. It is averaging three thousand independent enforcement decisions into one number. The quantity that matters is the peak demand within any single 100 ms window, and there is no metric for it — container_cpu_usage_seconds_total is a counter scraped every 15 or 30 seconds in a typical setup, four orders of magnitude coarser than the thing being enforced.

Watch the longest single stall readout as you raise the thread count. It goes from nothing at all to 83 ms. Nothing in the request rate, the work per request, or the average utilisation changed. What changed is how much of the period was left when the pool hit zero.

The latency readouts are the part worth staring at. Each request needs 12 ms of CPU. At sixteen threads the p99 is 96 ms, so roughly 84 of those milliseconds are the pod being stopped rather than the pod working. At eight threads the median is 12 ms — the true service time — and the p99 is still 101 ms, because half the requests slip through before the pool empties and the other half wait out the rest of the period. A tail that is eight times the median, on a service that is 30% busy, is the fingerprint.

Every fix has a boundary

Raise the limit. Drag it from 1200m to 1800m and throttling goes to zero. This works, and it is what most teams do. What it costs is not capacity — the limit was never reserved, so nothing else on the node gets less — it is the loss of the limit as a guard rail, and a request-to-limit ratio that keeps widening until the limit is decorative. It also does nothing for the next batch that is twice as large.

Move to a bigger node. Raise node cores from 16 to 64 with threads at 16. Every readout is byte-identical. The quota belongs to the cgroup, not to the machine, and the pod already had more CPUs than it had runnable threads. This is the misdiagnosis the throttling metric invites: it reads like a starvation signal, so people buy CPUs, and the number does not move by a percentage point.

Now drag node cores the other way, down to 4. Throttling falls, from 25% of periods to 20%, and the longest stall drops from 83 ms to 64 ms; at 2 cores it is 10% and 36 ms. A smaller node protects the pod from its own thread pool by making it physically unable to spend quota that fast. That is the mechanism running backwards, and it is the reason the bigger node is not always merely neutral: whether it hurts depends entirely on whether your runtime sizes its thread pool from the machine or from the quota.

Both major runtimes now read the quota, and this is recent enough to be worth checking rather than assuming. The JVM has computed Runtime.availableProcessors() as cpu_quota / cpu_period since JDK 10, backported to 8u191, under -XX:+UseContainerSupport, which is on by default. Go was the laggard: until 1.25, GOMAXPROCS defaulted to the visible CPU count and ignored the cgroup entirely. Go 1.25 changed the default to the minimum of the logical CPU count, the affinity mask and the cgroup CPU limit, rounded up to a whole number and floored at 2, and it re-reads the limit periodically while the process runs.

So the thread pool follows the node in four cases, and they are common enough that the misdiagnosis survives: when there is no limit for the runtime to read; when the limit is larger than the node, which makes the node the binding constraint again; when someone has pinned GOMAXPROCS or -XX:ActiveProcessorCount by hand; and when a Go module still declares go 1.24 or earlier in go.mod, because GODEBUG defaults track the declared language version and an older declaration turns containermaxprocs back off. In any of those, a 64-core node really does raise the thread count for you and the pod burns the same budget four times faster than before.

Shorten the period. Drag cpu.cfs_period_us down to 10000. The longest stall collapses from 83 ms to 5 ms, which is exactly what it should do: the longest you can be stopped is bounded by the period. But the throttled-periods readout goes up, from 25% to 31%, and the median latency goes from 37 ms to 110 ms. The dead time did not go anywhere; it was sliced into a hundred small stalls instead of ten large ones, and now every request waits through several of them instead of one request in four waiting through one. Watch p99 while you do it: 96 ms to 117 ms. Bounding the length of one stall is not the same as bounding the latency of a request that has to sit through a dozen.

Go one step further, to 5000 µs, and the very first line of the log is now a period that stalled while consuming well under its quota. That is the second effect, and it is the floor under this whole approach. The kernel does not hand quota to threads, it hands it to run queues, in sched_cfs_bandwidth_slice_us chunks — 5 ms by default. At a 5000 µs period a 1.2 CPU limit is a 6 ms pool, so the first run queue takes 5 ms of it and the second gets 1, and the other fourteen threads stall while the pool is technically not exhausted. Note that the longest stall now reads 0 ms even though a third of periods are throttled: the pod is never fully stopped, it is permanently running one thread wide instead of twelve.

What sets that floor is the quota, not the period — the period only reaches it by shrinking the quota with it. Once the quota is down to a handful of 5 ms slices, whichever run queues ask first get the CPU and the rest wait, and you can reach the same state without touching the period at all: put it back to 100000 and drag the limit to 800m, and starved periods show up in the log against a 80 ms pool. Two things to know before reaching for the period anyway. It is a kubelet flag, --cpu-cfs-quota-period, so it applies to every container on the node rather than to the pod you are trying to fix. And setting it to anything other than the default 100 ms requires the CustomCPUCFSQuotaPeriod feature gate.

Remove the limit. Uncheck the enforcement box: throttling goes to zero and both p50 and p99 collapse to 12 ms, the actual service time. There are two ways to get here and they are not the same lever — omitting limits.cpu from the pod spec is a per-workload decision, while the kubelet's --cpu-cfs-quota=false turns enforcement off for every container on the node. This is what the "stop using CPU limits" argument is about, and the simulation makes the case honestly, because a limitless pod on an idle node genuinely is faster. What the simulation does not model is the neighbours. With no limit, the only thing bounding your pod is cpu.shares from your request, which binds only when the node is contended — so the failure mode moves from "my pod stalls predictably" to "my pod's bad day is now everyone's bad day," and it arrives during the incident rather than before it. That trade is a judgement call about your blast radius, not a fact.

Size the pool to the limit. The fix that addresses the mechanism rather than the symptom is to stop the runtime creating more runnable threads than the quota can feed. On a current JVM or a Go 1.25 module you already have this for the scheduler's own worker threads, which is most of the reason the problem is less common than it was; what is left to you is everything the runtime does not size for you — your own bounded executors and worker pools, and anything reading nproc. Set threads back to 1 against the 1200m limit and throttling goes to zero while the average CPU stays exactly where it was. Its boundary is visible in the latency readouts: the same change takes the median from 37 ms to 81 ms and the p99 from 96 ms to 141 ms, because you have traded parallelism for predictability and every request now queues behind the ones in front of it. You removed the stalls by making the pod slow enough not to earn them. For a pod that can genuinely use twelve cores in a burst, that is a real loss, and the honest answer there is exclusive cores or no limit at all.

The kernel bug, and why it is probably not your bug

The bug needs a small quota and a lot of threads, so set the limit to 300m, work per request to 3 ms and threads to 16 — a sub-CPU pod still sitting at a third of its limit. Now tick kernel older than 5.4 on and off. Utilisation does not move. Throttled periods go from 33% to 43%, the median latency goes from 5 ms to 28 ms, and the log reports 58 ms of granted quota expiring unused against a budget of 30 ms per period. The pod was charged for CPU time it was never allowed to run.

On kernels from 4.18 up to 5.4, runtime handed to a per-CPU run queue carried an expiry stamp tied to the period. A highly-threaded process on a many-core machine would scatter 5 ms slices across dozens of run queues, use a fraction of each, and then have the remainder invalidated at the period boundary — so it hit throttling while its own accounting said it had never spent its quota. That is the part the simulation cannot show you honestly: it holds a thread on one run queue for as long as it runs, so its expiry losses come from the handful of queues in play rather than from dozens, and the real bug was much worse than the toggle above makes it look. What the toggle does reproduce is the shape — throttling that rises while utilisation does not.

"Before 5.4" is the usual shorthand and it is too generous. The expiry logic dated to 2014, but a conditional added in v3.16 meant slices in practice never expired; it only started biting when commit 512ac999d275 fixed that conditional in v4.18. Dave Chiluk's de53fd7aedb1 then removed slice expiration altogether. His commit message reports "almost 30x performance improvement" on a synthetic test of 10 ms of quota per 100 ms on an 80 CPU machine, and bounds the remaining overshoot at min_cfs_rq_runtime, 1 ms per CPU. It landed in 5.4 — not 5.3 — released November 2019. So the window in which this is your bug is 4.18 to 5.4, and uname -r settles it.

That fix has a visible consequence people mistake for a second bug. Because a run queue now keeps its unused millisecond across the boundary, a period can consume slightly more than its quota — the commit says so in as many words, that limits "no longer strictly apply per period" but remain accurate over longer timeframes. You can see it in the log at the default settings: three of the seven periods listed report consuming 130 ms of a 120 ms quota. Over any window longer than a period this washes out, which is why nobody notices — but it does mean a per-period reading of cpu.stat can show you slightly over your own ceiling without anything being wrong.

Mostly, though, this is a diagnostic dead end. Any cluster on a supported kernel has the fix, and people still reach for it: there is an open Kubernetes issue titled "CPU Throttling on Linux kernel 5.4.0-1029-aws", which is to say, on a kernel that contains the patch. The test usually offered — "the bug throttles you below your quota" — is not sufficient on its own, and the simulation shows why: untick the kernel box at the settings above and the log's second line is still a period that stalled after consuming 11 ms of its 30 ms budget, on a kernel with the fix, purely because a 30 ms pool is six 5 ms slices and sixteen threads want them. Being throttled under quota tells you your quota is only a few slices wide. It tells you about your kernel only in combination with its version.

The kernel did later gain a way to bank unused quota — cpu.cfs_burst_us on v1, cpu.max.burst on v2, both defaulting to 0 — but there is no field for it in the pod spec, so reaching it means a container-runtime annotation rather than YAML.

Checking it on a real pod

Read the enforcement directly. Inside the container on cgroup v2:

• cat /sys/fs/cgroup/cpu.max → 120000 100000. Quota then period, microseconds. max 100000 means no limit.
• cat /sys/fs/cgroup/cpu.stat → nr_periods, nr_throttled, throttled_usec. On cgroup v1 the files are cpu/cpu.cfs_quota_us, cpu/cpu.cfs_period_us and cpu/cpu.stat, and the last field is throttled_time in nanoseconds.

In Prometheus, the ratio everyone graphs is the fraction of periods that contained a stall:

rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])

That number alone is what sends teams after bigger nodes. The one that actually tells you which failure you have is the mean stall length:

rate(container_cpu_cfs_throttled_seconds_total[5m]) / rate(container_cpu_cfs_throttled_periods_total[5m])

If that comes out near a tenth of a second, you are losing whole periods and your tail latency is quota dead-time, not work. If it comes out at a millisecond or two, you are brushing the ceiling at the very end of each period and the limit is roughly right. Compare it against container_cpu_usage_seconds_total over the same window: high throttling with usage far below the limit means the demand is concentrated, and the lever is the thread pool, not the limit. High throttling with usage pinned at the limit means the limit is simply too small.

One thing not to do: point an autoscaler at it. The HPA's utilisation target is a percentage of the request, and throttling is not an input to it at all. What the HPA reads therefore depends entirely on a number that has nothing to do with the stalls: at the simulation's defaults, 0.36 cores against a 300m request is 120% and the autoscaler adds replicas hard, even though nothing about another replica shortens the 83 ms one pod spends stopped inside its own quota. Widen the request to 1 CPU and leave everything else alone and the identically throttled pod reads 36% and never scales at all. Neither number is a measurement of the thing going wrong. And note that all of this is specific to CPU — it is compressible, so the kernel can stop you and resume you. Ask for too much memory and there is nothing to throttle, which is why the memory limit kills the container instead.

Next: what Guaranteed actually guarantees, and the resource whose limit is not compressible at all, memory and OOMKill.