DeepConcepts

Kubernetes / node / resource management / cgroups

cgroup v2: The Limit That Kills You Is Four Levels Above Your Container

The misconception

That migrating a node to cgroup v2 renames files and changes nothing — memory.limit_in_bytes becomes memory.max, cpu.shares becomes cpu.weight, same behaviour. Two structural changes bite. First, the single hierarchy means your container's memory limit is only the innermost of three or four limits it is charged against, and when the failing one is kubepods.slice — which the kubelet limits by default — the OOM domain becomes every pod on the node, the badness denominator becomes node allocatable rather than your limit, and oom_score_adj stops cancelling out and starts outweighing resident memory by three orders of magnitude, so the pod that dies is the Burstable pod with the smallest memory request rather than the pod that allocated. Second, since Kubernetes 1.28 the kubelet sets memory.oom.group=1 on container cgroups, so the kernel kills every process in the victim's container instead of the single fattest one.

16 min

Your container has a memory limit. It is not the limit that kills it. On cgroup v2 every page your process touches is charged against your container and your pod and the slice that holds every pod on the node, in that order, and the first one of those to reach its ceiling is the one that runs the out-of-memory (OOM) killer — the kernel routine that picks a process and kills it when a memory request cannot be satisfied. Which one it is decides who dies — and it is routinely a container three levels away from the one that asked for memory.

A control group, or cgroup, is a kernel object that accounts and limits the resources of the processes inside it. cgroup v1 gave every resource its own separate tree: one hierarchy for memory, another for CPU, another for pids, and a process could sit in unrelated places in each. cgroup v2 collapses all of them into a single hierarchy — this is what "unified" means — and it is now effectively the only option: Kubernetes deprecated cgroup v1 in v1.35, and from that release the kubelet refuses to start on a v1 node unless an administrator sets failCgroupV1 to false.

The migration is usually described as a rename. memory.limit_in_bytes becomes memory.max, cpu.shares becomes cpu.weight, cpu.cfs_quota_us and cpu.cfs_period_us collapse into one cpu.max file. That part is true and it is the least interesting thing about it. The change that shows up in your incident channel is structural: because there is one tree, your container is now four levels deep in it, and every level above you is a limit you did not set and an OOM domain you are a candidate in.

Below is that tree, running. Six pods on one 16 GiB node. Every one of them was admitted by the scheduler, because the scheduler adds up requests, and the requests come to 5,376 MiB. The limits come to 21,632 MiB, on a slice the kubelet has capped at 15,360. Your pod is web, it is using 320 MiB of a 1,024 MiB limit, and it is going to have a bad time. Start with worker processes in your container: drag it from 3 down to 1.

your pod web
the node
the neighbours

Also fixed on the node: redis, Guaranteed at 2048Mi and using 1900; and your pod's otel-agent companion container, 128Mi request and limit, one process, using 96. It is an ordinary second entry in spec.containers, not a native sidecar — see the note in the request section for why that matters.

Three minutes of wall clock, stepped at one second, on a node with 16 GiB of RAM. Each second every container tries to charge what it needs. The charge walks up the tree exactly as page_counter_try_charge() does, and the first ancestor whose memory.max would be exceeded becomes mem_over_limit: reclaim runs in that cgroup's subtree, and if reclaim cannot free enough, the OOM killer scans that cgroup's subtree. Sizes and growth rates are illustrative; the arithmetic on top of them — the walk, the reclaim target, the badness formula, the group kill — is the kernel's own.

first container killed
cgroup that hit its limit
your pod at that moment
your container killed
your page cache reclaimed
OOM kills in 180 s
The charge walk — your container's ancestors, at the first kill

One row per level of the single hierarchy. The bar is memory.current against memory.max. the level the charge actually failed at · admitted the charge · no limit set, so it can never fail.

Badness ranking inside the OOM domain

oom_badness() ranks processes, not containers: points are the task's resident pages plus oom_score_adj × totalpages / 1000, and for a cgroup OOM totalpages is that cgroup's own memory.max. oom_score_adj is a per-process integer from −1000 to 1000 that the kubelet writes from the pod's Quality of Service (QoS) class — the Guaranteed / Burstable / BestEffort label in the second column, which Kubernetes assigns from how the pod's requests compare to its limits. Each row is the largest process in a container, modelled as an equal share of the container's anonymous memory. The model leaves out page tables and swap entries, which the kernel also counts and which are under a percent here.

kubepods.slice memory.current — one bar per second

charges admitted · a charge failed and reclaim covered it · reclaim could not, and something was killed.

At three worker processes your container is never killed, and it is second in the queue. Read the badness board: your server, holding 300 MiB, scores 15,352 points. The checkout-b container holding 3,482 MiB — eleven times as much, and the container that actually asked for the memory — scores 15,278 and survives. Drag worker processes to 1 and your container starts dying: four kills, and 735 MiB of your page cache thrown away on the way there. Drag it to 12 and the kills stop. You did not change how much memory your pod uses by a single byte.

One tree, four limits, and only one of them is yours

The rows in the tree panel are not a diagram. They are the actual directory structure under /sys/fs/cgroup on a node using the systemd cgroup driver, and each of them holds a real memory.max file that the kubelet writes.

  • /, the root cgroup. No memory.max file exists here — you cannot limit the machine below itself. Physical memory is the ceiling, and reaching it means the global OOM killer rather than a cgroup one. The simulation draws it as a 16,384 MiB limit so you can watch the failure move up to it.
  • kubepods.slice. Everything Kubernetes runs. The kubelet writes memory.max here equal to node capacity minus kubeReserved minus systemReserved, and it does this by default: enforceNodeAllocatable defaults to [pods]. On the simulated node that is 16,384 − 1,024 = 15,360 MiB. The kubelet writes this file whether or not you enable enforcement — turning enforcement off raises the number to full node capacity rather than removing it, which the last section returns to.
  • kubepods-burstable.slice. One per QoS class — Guaranteed pods live directly under kubepods.slice, Burstable and BestEffort get their own slice. The kubelet sets cpu.weight here but leaves memory.max at max unless you turn on the alpha QOSReserved feature gate and pass --qos-reserved. This is why the row is grey and can never fail.
  • The pod cgroup. memory.max is the sum of the containers' memory limits — and only if every container declares one. Miss a limit on one sidecar and the whole pod cgroup goes unlimited.
  • The container cgroup. Your limits.memory. The only number in the whole tree that appears in your YAML.

The nesting is not a Kubernetes preference. cgroup v2 has a rule the kernel documentation calls the no-internal-process constraint: a non-root cgroup may distribute resources to its children only if it has no processes of its own. So the moment the kubelet wants a limit on "all pods" it must create kubepods.slice as a pure container of children and push every process one level deeper. Controllers themselves are enabled top-down through cgroup.subtree_control, and the kernel enforces that a child can only enable controllers its parent has already enabled. That is the whole reason your container is at depth four rather than depth one.

And this is where the rename story breaks. On cgroup v1 the memory controller had its own hierarchy and so did the CPU controller, and while memory limits were hierarchical there too, the tree they were hierarchical in was a different tree from the one deciding CPU. On v2 there is one tree, so the level that stops your memory allocation is the same level that owns your CPU weight, your pids.max, and your I/O weight. One structure, one set of ancestors, one place for all of them to go wrong.

What the charge actually does

When a process first touches a page, the kernel charges it to the process's memory cgroup by calling page_counter_try_charge(). That function's own comment says what it does: "try to hierarchically charge pages… Returns true on success, or false and @fail if the counter or one of its ancestors has hit its configured limit." The loop is literally for (c = counter; c; c = c->parent). Every ancestor is incremented; if any of them would go over its max, the increment is rolled back and a pointer to that counter is returned.

try_charge_memcg() then takes that pointer and names it mem_over_limit. Everything after this point uses mem_over_limit and not your container:

  • try_to_free_mem_cgroup_pages(mem_over_limit, …) — reclaim scans the failing cgroup's whole subtree.
  • mem_cgroup_oom(mem_over_limit, …) — the OOM killer's candidate set is the failing cgroup's whole subtree.
  • oc->totalpages = mem_cgroup_get_max(oc->memcg) — the denominator in the badness formula is the failing cgroup's limit.

Three consequences, and each one is a different thing going wrong. Take them in order.

totalpages is the whole trick

oom_badness() is four lines of arithmetic:

points = rss + swapents + pagetables; points += oom_score_adj × (totalpages / 1000)

oom_score_adj is a per-process integer from −1000 to 1000 that the kubelet sets from the pod's QoS class: −997 for Guaranteed, 1000 for BestEffort, and for Burstable 1000 − 1000 × containerMemoryRequest / nodeCapacity, clamped into 3…999. Across that whole range it is a bias on the ranking, not a veto. The one value that is a veto is −1000, and Kubernetes never writes it: oom_badness() returns LONG_MIN the moment adj == OOM_SCORE_ADJ_MIN, and the cgroup v2 documentation says such tasks "are treated as an exception and are never killed" even under a group kill. The kubelet's most protective value is −997, which is a very large bias and still a bias.

The important part is that totalpages is the OOMing cgroup's own limit, so the same oom_score_adj is worth a completely different number of megabytes depending on which cgroup failed. Move server: memory in use to 1,200 MiB, above its 1,024 MiB limit, and watch the board collapse to a single row: the OOM domain is now your own container, totalpages is 1,024 MiB, and one point of oom_score_adj buys 1.0 MiB. Put it back to 320 and the domain is kubepods.slice, totalpages is 15,360 MiB, and one point buys 15.4 MiB. Fifteen times the leverage, from a number nobody changed.

That factor is what produces the default run's result. At the first kill:

  • batch/worker — BestEffort, 2,639 MiB in two processes, so 1,319 MiB in its largest. oom_score_adj 1000, worth +15,360 MiB. Total 16,679. Killed.
  • web/server — that is you — Burstable, 300 MiB in three processes, 100 MiB in the largest. oom_score_adj 993, worth +15,252 MiB. Total 15,352. Second in line.
  • checkout-b/app — the container that actually asked for the memory. 3,482 MiB in four processes, 870 MiB in the largest. oom_score_adj 938, worth +14,408 MiB. Total 15,278. Fourth. Survives.

Your 300 MiB container outranks a 3,482 MiB one. Two things did that. The smaller one is the request: 128Mi against 1024Mi is 55 points of oom_score_adj, and at this level 55 points is 845 MiB. The larger one is process count, and it is the one nobody expects.

The kernel ranks processes; Kubernetes bills containers

select_bad_process() calls mem_cgroup_scan_tasks(oc->memcg, oom_evaluate_task, oc). Tasks, not cgroups. A container that forks four workers presents four candidates of a quarter the size each, and each of them is scored on its own rss. The kubelet's own source says so, in a comment above the Burstable formula: "Note that this is a heuristic, it won't work if a container has many small processes."

Drag worker processes in your container and watch the board reorder while every byte of memory stays where it was:

  • 1 process — your largest task is the whole 300 MiB. 15,552 points, still second, but now the scores above and below you shift enough that your container is killed four times in the run and gives up 735 MiB of page cache.
  • 3 processes — 100 MiB each, 15,352 points, second and never killed.
  • 12 processes — 25 MiB each, 15,277 points. You drop below checkout-b/app to fourth and stay there.

This is a genuinely perverse incentive and it is worth saying plainly: on a node where the binding limit is kubepods.slice, splitting your heap across more processes makes you harder to kill, and it does so without freeing a single page. It also degrades gracefully in the wrong direction — see the group-kill section below, where more processes means a bigger blast radius when you are picked anyway.

Guaranteed is a 30 GiB handicap, at one level only

Set server: requests.memory equal to server: limits.memory — both at 1024Mi — and the pod becomes Guaranteed, because otel-agent already has matching request and limit. Your oom_score_adj goes from 993 to −997 and your score goes from +15,352 to −15,214: last but one, below a redis holding 1,894 MiB. The swing is 1,990 points at 15.36 MiB each, which is 30,566 MiB of head start on a node that only has 16,384.

At the node level, therefore, QoS is not a tie-breaker. It is decisive, and Guaranteed pods are effectively unkillable by a neighbour's allocation. Now push server: memory in use past your own limit while still Guaranteed. The domain becomes your container, every process in it carries the same −997, the constant cancels out of a ranking that only compares processes inside one container, and you die exactly as fast as you would have at any other QoS. That asymmetry is the single most useful thing to know here: Guaranteed protects you from your neighbours, never from your own total.

The part with no metric: reclaim

Long before anything is killed, the failing charge does something quieter. try_to_free_mem_cgroup_pages(mem_over_limit, …) walks the failing cgroup's subtree looking for reclaimable pages, and the cheapest reclaimable pages on a node are clean file-backed pages — page cache. It does not care which container faulted them in, and it certainly does not care which container asked for the memory that triggered the reclaim.

At the simulation's defaults your web pod is never killed and never comes near its own limit, and it still gives up 213 MiB of page cache over three minutes. It never held more than 141 MiB at once: that 213 is a running total, the pod being stripped, refilling, and being stripped again, and it ends the run at zero cache. The your page cache reclaimed readout is that running total, and the reason it is worth staring at is that no Kubernetes signal reports it. Your container_memory_working_set_bytes goes down. Your limit is untouched. Your restart count is zero. Your p99 doubles, because every read that used to hit cache now hits the disk.

Drag share of each pod that is page cache across its range and watch the two effects trade off:

  • 0% — nothing to reclaim, so every failed charge goes straight to the OOM killer. First kill at t+75 s, 19 kills in the run, and your pod loses only 41 MiB.
  • 35% — first kill at t+80 s, 18 kills, 213 MiB of your cache gone.
  • 80% — first kill at t+93 s, only 10 kills, and your pod surrenders 1,321 MiB of cache to buy that quiet.

Page cache is the shock absorber, and the bill is paid by whoever has the most of it, not by whoever caused the pressure. That is the honest reading of the memory.stat counters: pgscan and pgsteal climbing on a container that is doing nothing wrong.

cgroup v2 does give you an instrument for this that v1 never had. Pressure Stall Information (PSI) exposes memory.pressure in every cgroup, reporting the share of wall-clock time in which tasks were stalled waiting on memory — some for at least one task, full for all of them. The kernel raises it around exactly this reclaim path: try_charge_memcg() wraps the reclaim call in psi_memstall_enter() and psi_memstall_leave(). Reading memory.pressure on kubepods.slice is the only cheap way to see the node-level charge failures at all.

memory.high, and why Guaranteed pods do not get one

cgroup v2 also adds memory.high, which the kernel documentation describes as a "memory usage throttle limit": crossing it puts the cgroup under heavy reclaim pressure and stalls the allocator, but it "never invokes the OOM killer." It is the middle setting that cgroup v1's memory controller did not have.

Kubernetes reaches it through the MemoryQoS feature gate, which was alpha and off from v1.22 and becomes beta and on by default in v1.37. When it is on, the kubelet computes memory.high = floor((requests.memory + factor × (limits.memory − requests.memory)) / pageSize) × pageSize from memoryThrottlingFactor. It can also write the protection side of the pair — memory.min for Guaranteed pods, memory.low for Burstable ones, both taken from the request — but only if you ask: that branch is gated on memoryReservationPolicy == TieredReservation, and the kubelet defaults the policy to None, whose else-branch writes literal 0 to both files. Through v1.36, then, turning MemoryQoS on gets you the throttle — memoryThrottlingFactor defaults to 0.9 — and none of the protection. Check both settings on your own kubelet rather than assuming the gate did what its name suggests. Read the guard on the throttle itself in kuberuntime_container_linux.go, too: memoryRequest != memoryLimitSpec || memoryRequest == 0. A Guaranteed container has request equal to limit, so it is skipped. The safest QoS class is the one with no throttle between healthy and dead.

Group kill: the 1.28 change nobody announced to your team

memory.oom.group is a cgroup v2 file with no cgroup v1 equivalent. The kernel documentation says that when it is set, "all tasks belonging to the cgroup or to its descendants … are killed together or not at all." Its kernel default is 0.

Kubernetes overrides that default. Pull request #117793, "use the cgroup aware OOM killer if available", landed in the v1.28 milestone and writes "memory.oom.group": "1" into the container's Unified map on every cgroup v2 node. Its release note is one sentence: processes within the cgroup "will be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup." On cgroup v1 one worker died and the container carried on, usually still passing its readiness probe. On v2 from 1.28, the container is gone.

Untick memory.oom.group on containers and the run changes shape completely: 84 kill events instead of 18. Each single-process kill frees a quarter of a container, the supervisor respawns the worker, the worker refills, and the node is back where it was seconds later. The container never restarts, so kubectl get pods shows RESTARTS 0 throughout and the pod stays Ready while it is being dismembered. That is the cgroup v1 behaviour people describe as gentler, and it is why the group kill was introduced.

It is also why the escape hatch exists. Kubernetes v1.32 added the singleProcessOOMKill kubelet option, whose documentation reads: "if true, will prevent the memory.oom.group flag from being set for container cgroups in cgroups v2… the behavior aligns with the behavior of cgroups v1." Its default is false, and on cgroup v1 nodes only null or true is even accepted. It took two attempts and ten months: pull request #122813 proposed it in January 2024, ran to forty-eight comments, and was closed unmerged that May; #126096 reproposed the same option and merged in November 2024, for v1.32. That is a reasonable proxy for how much of a behaviour change 1.28 actually shipped.

One detail that decides the blast radius. mem_cgroup_get_oom_group() walks from the victim's cgroup up to the OOMing cgroup and picks the highest ancestor with oom.group set. Kubernetes sets it on container cgroups and nowhere else. So when kubepods.slice is the OOM domain, the walk goes container → pod → QoS slice → kubepods.slice, finds the flag only at the bottom, and kills exactly one container. Your other containers survive, your pod stays Running, and the only evidence is RESTARTS ticking on one container. Issue #124253 is people asking for the flag at the pod level instead; it is not there.

Where each fix stops working

Raise your memory limit. Drag server: limits.memory from 1024Mi to 8192Mi and compare the board. It is byte-for-byte identical: same order, same points, same victim. limits.memory is not an input to oom_badness() at any level above your own container, and at the node level it is not an input at all. What it does change is the pod cgroup's memory.max, which climbs to 8,320 MiB and is now even further from being the binding constraint. You have bought protection against exactly one failure mode — your own container overrunning — and none against the one the simulation is showing you.

Raise your memory request. This one does something, and the amount is computable. Each point of oom_score_adj is worth totalpages/1000, and each 1000th of node capacity of request is worth one point. On a 16 GiB node that is 16.4 MiB of request per point and 15.36 MiB of head start per point at the kubepods.slice level: a 1 GiB request buys 62 points, or 952 MiB. Take the request from 128Mi to 2048Mi with the limit at 8192Mi and your score drops from 15,352 to 13,540 and you fall below your own otel-agent, which did not move. Note the denominator, though: it is nodeCapacity, not allocatable and not anything you control. The same 1 GiB request on a 64 GiB node buys 15 points instead of 62, so this lever gets weaker exactly as you move to the larger nodes where the node-level limit is most likely to bind.

One carve-out, because it changes this answer entirely. If otel-agent were a native sidecar — an entry in spec.initContainers carrying restartPolicy: Always, which is what Kubernetes now means by the word — GetContainerOOMScoreAdjust would take a different branch for it. A native sidecar's score is clamped down to the score implied by the smallest memory request among the pod's regular containers, so raising your own request would drag the sidecar's protection along with it instead of leaving it behind. The simulation models the ordinary case, a second entry in spec.containers, where the two are scored independently. If your observability agent is a native sidecar, expect a tie rather than the overtake shown here.

Go Guaranteed. The strongest available move at the node level, worth 30,566 MiB of ranking as shown above, and it costs you the whole point of the Burstable class: you now pay for your peak all the time, and the scheduler reserves your limit rather than your average. It is also worth nothing inside your own container, and — with MemoryQoS on — it removes your memory.high cushion.

Turn off node-allocatable enforcement. Untick enforceNodeAllocatable: [pods] and watch the kubepods.slice row: its memory.max does not disappear, it goes from 15,360 to 16,384 MiB. This surprises people, so it is worth reading the kubelet's own reason for it. enforceNodeAllocatableCgroups() runs whether or not you asked for enforcement, and the comment above the line says why: "We need to update limits on node allocatable cgroup no matter what because default cpu shares on cgroups are low and can cause cpu starvation." The only thing your setting changes is which number gets written — nodeAllocatable := cm.internalCapacity, the machine's full memory, instead of capacity minus the reserves.

A limit equal to the whole machine is not the same as no limit, but it is close, and the difference is who else is on the machine. The root cgroup holds the pods and the node's daemons; kubepods.slice holds only the pods. Both now have a ceiling of 16,384 MiB, so the root reaches it first by exactly the amount the daemons are using, and the charge walk — which tests kubepods.slice before the root — sails through the slice and fails at the root. That is what the tree shows: kubepods.slice at 15,422 of 16,384, the root at 16,190 of 16,384, and the failure marked on the root. Drag kube- + system-reserved to 0 with enforcement still off and the failure snaps back down to kubepods.slice, because with no daemons holding anything the two rows carry identical usage and the lower one is tested first.

So the rescue is much smaller than it looks. The first kill moves from t+80 s to t+82 s. totalpages goes from 15,360 to 16,384, which makes oom_score_adj slightly more powerful, not less. The same container dies. What actually changes is the reclaim scope, and the last line of the decision log reports it. With enforcement on, the node's own daemons give up 0 MiB of page cache all run, because system.slice is not in the failing cgroup's subtree. With it off they give up 256 MiB — all of it — because now they are. Enforcement does not decide who dies. It decides whether sshd, containerd and the kubelet are inside the blast radius of your pods' memory pressure.

The two candidate rows the root cgroup adds explain why the daemons do not simply die instead: kubelet.service scores −16,163 because the kubelet sets its own oom_score_adj to −999, and system.slice scores 28 — its 563 MiB spread over twenty daemons, with an oom_score_adj of 0 adding nothing — against pods scoring five figures. Every pod on the node is a more attractive victim than sshd by roughly the whole size of the machine. That is oom_score_adj doing its job.

Reduce kube-reserved. Drag kube- + system-reserved to 0 and kubepods.slice gains 1,024 MiB: the first kill moves from t+80 s to t+92 s and the run has 7 kills instead of 18. Drag it to 2048Mi and the first kill arrives at t+69 s with 22. Every megabyte you reserve for system daemons is a megabyte the pods provably cannot have, and the Kubernetes documentation describes this enforcement purely as "evicting pods whenever the overall usage across all pods exceeds 'Allocatable'" — which is true and incomplete. Eviction is the kubelet's reaction, on its own polling interval. The cgroup limit underneath it is the kernel's reaction, and it happens in the page fault.

Set a pod-level limit. spec.resources.limits.memory — Kubernetes Enhancement Proposal (KEP) 2837, beta and on by default since v1.34 — writes the pod cgroup's memory.max directly instead of summing containers. Set pod-level limits.memory to 512Mi and server: memory in use to 700, and a fourth OOM domain appears: the pod cgroup, with two candidates and totalpages of 512 MiB. Now one point of oom_score_adj is worth 0.5 MiB, both containers carry the same 993, so the adjustment cancels and the ranking is pure resident size: server's largest process at 152 MiB scores 660 against otel-agent's 67 MiB at 576. Eight kills of your own container in three minutes, from a limit no container in the pod ever exceeded.

Get a bigger node. This one genuinely works, and it is worth understanding why it is not a trick: the sum of limits stops exceeding the slice. It is also the only fix on this list that costs money rather than resilience, which is presumably why it is the one everyone reaches for.

Checking it on a real node

First, confirm which version you are on. On the node:

  • stat -fc %T /sys/fs/cgroup/cgroup2fs for v2, tmpfs for v1. cgroup v2 support has been stable since Kubernetes v1.25 and cgroup v1 is deprecated as of v1.35, where the kubelet refuses to start on it unless failCgroupV1 is set to false.

Then read the ceiling that is not in anybody's YAML. Every path below assumes the systemd cgroup driver, which is what puts .slice on the end of every directory name, and which kubeadm-provisioned and most managed clusters use. It is not the kubelet's built-in default — that is still cgroupfs, which lays the same cgroups out without the suffixes: /sys/fs/cgroup/kubepods/ rather than /sys/fs/cgroup/kubepods.slice/. Check which one you have with kubectl get --raw /api/v1/nodes/<node>/proxy/configz and look for cgroupDriver before copying any path from here.

  • cat /sys/fs/cgroup/kubepods.slice/memory.max and memory.current. Compare the first against kubectl get node -o jsonpath='{.status.allocatable.memory}': the cgroup number is larger, by exactly your evictionHard: memory.available, because the scheduler's Allocatable subtracts the eviction threshold and the cgroup limit does not.
  • cat /sys/fs/cgroup/kubepods.slice/memory.eventslow, high, max, oom, oom_kill, oom_group_kill. A non-zero max here is the single most under-read number on a Kubernetes node: it counts the times a charge failed at the node level and had to reclaim. It rises long before oom_kill does.
  • cat /sys/fs/cgroup/kubepods.slice/memory.pressuresome avg10=… full avg10=…. full above a few percent means every task on the node was stalled on memory at once.

When something is killed, the kernel log settles it in one line. The victim dump prints oom_memcg= — the cgroup that hit its limit — and task_memcg= — the cgroup the victim was in:

checkout invoked oom-killer: gfp_mask=0x1100cca, order=0, oom_score_adj=938
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-9f3….scope,
  mems_allowed=0,oom_memcg=/kubepods.slice,
  task_memcg=/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb71….slice/cri-containerd-2ae….scope,
  task=worker,pid=41207,uid=0
Tasks in /kubepods.slice/…/cri-containerd-2ae….scope are going to be killed due to memory.oom.group set
Memory cgroup out of memory: Killed process 41207 (worker) total-vm:1584312kB, anon-rss:1350784kB,
  file-rss:12608kB, shmem-rss:0kB, UID:0 pgtables:2784kB oom_score_adj:1000

Four things to read off it, in order of usefulness:

  • If oom_memcg and task_memcg disagree, you were killed for somebody else's allocation. Here oom_memcg=/kubepods.slice and the task was in a BestEffort pod's container: the node-level limit failed and the kernel picked the highest-badness process on the whole node. No amount of tuning that container's limit would have changed the outcome.
  • The first line names the invoker, not the victim. checkout invoked oom-killer … oom_score_adj=938 against a victim with oom_score_adj:1000 — two different processes in two different pods. When they match, it is an ordinary container-limit OOM and the container's own limit is the thing to fix.
  • The Tasks in … are going to be killed due to memory.oom.group set line is the 1.28 behaviour, printed by mem_cgroup_print_oom_group(). Its absence on a v2 node means somebody set singleProcessOOMKill: true.
  • oom_score_adj on the victim line tells you its QoS. 1000 is BestEffort, −997 is Guaranteed, anything in between is Burstable and the number decodes to a request: request ≈ (1000 − adj) × nodeCapacity / 1000.

In Prometheus, cAdvisor exports the node-level cgroup with an empty pod label, so the ceiling and the usage are both graphable:

container_memory_working_set_bytes{id="/kubepods.slice"} / container_spec_memory_limit_bytes{id="/kubepods.slice"}

Alert on that ratio, not on per-pod usage. A node can sit at 96% of kubepods.slice with every single pod under 40% of its own limit, and every per-pod dashboard you own will look healthy right up to the kill. The corresponding cgroup file, and the one that will not lie to you about scrape intervals, is the max counter in memory.events.

Finally, the two structural questions worth asking of any pod spec before you ship it. Does every container declare limits.memory? If not the pod cgroup has no limit at all, and the next ceiling above your container is the node's. And how many processes does your container run? On a node whose binding limit is kubepods.slice that number is a survival parameter, and on the CPU side of the same tree it is the parameter that decides how fast you burn a quota. It is the same number pulling in two directions, which is a fair summary of what the unified hierarchy did to capacity planning.

A Burstable pod with requests.memory: 256Mi, limits.memory: 4Gi is OOMKilled while using 900 MiB. The kernel log shows oom_memcg=/kubepods.slice and task_memcg=/kubepods.slice/kubepods-burstable.slice/…. What changes the outcome?

Next, the two limits that live in this same tree and behave nothing like each other: the memory limit that kills the container and the CPU limit that only stops it.

Why this concept is on the site

Topics are chosen from places engineers visibly get stuck, and the sources are kept with the lesson so the claim is checkable.