HN Debrief

Fixing a kubelet memory leak in Kubernetes 1.36

  • Infrastructure
  • Open Source
  • Programming

The post is a debugging writeup from the engineer who found a kubelet memory leak in Kubernetes 1.36.0, 1.36.1, and 1.36.2. Kubelet is the per-node agent that manages pods, so a leak there grows on every affected node. The bug came from Go context cancellation logic in the pod sync path. A cancel function was stored and never called, which kept memory alive across repeated syncs. The author recommends checking `kubectl version`, inspecting each node’s `process_resident_memory_bytes`, and using a kubelet restart as a stopgap until 1.36.3 ships.

If you run Kubernetes 1.36.x, check kubelet resident memory on every node now instead of assuming liveness probes or basic health checks will catch this. More broadly, treat control-plane and node-agent resource regressions as something to monitor explicitly after upgrades, especially on small nodes where leaks surface first by evicting workloads, not by killing the leaking daemon.

Discussion mood

Positive toward the writeup and the fix, with unease about how a leak in a core kubelet loop slipped through and how poorly current Go tooling exposes this kind of context misuse.

Key insights

  1. 01

    Go context leaks evade normal linting

    Once a cancel function escapes the function where it was created, static analysis loses the thread. That is why `go vet` can miss a real leak even when it checks for missing `cancel()` calls. The sharper point is that Go lacks a good way to express ownership for stored root contexts and goroutines, so correct intent and buggy retention can look too similar to a linter. One comment adds that contexts themselves are now garbage-collectible, and that a stronger leak detector might be possible if cancel functions held weak references, likely as an opt-in debug tool similar to the race detector.

    Do not assume `go vet` coverage means your context lifecycle is safe in long-lived services. Add code review checks or custom linting around stored cancel functions and goroutine ownership, especially in loops and manager code.

      Attribution:
    • cyberax #1 #2
  2. 02

    The failure shows up in pods first

    Kubelet can stay alive, answer queries, and keep passing crude health checks while still consuming more memory every day. On many Linux setups it is also protected from the OOM killer with a very negative `oom_score_adj`, so the node sacrifices application pods before it kills kubelet. Combined with a leak rate that may take weeks to become obvious, this explains why routine release checks and basic process monitoring can miss it.

    Track kubelet memory as a first-class post-upgrade metric and compare before versus after releases. If nodes run tight on RAM, add alerts on kubelet growth before pod evictions start masking the root cause.

      Attribution:
    • compumike #1 #2
    • nijave #1

Against the grain

  1. 01

    Health checks and auto-restarts are still useful

    Even though this specific leak leaves kubelet responsive, the suggestion to health check and restart the process points to a broader operational pattern. Node agents are privileged, long-lived daemons, and automated restarts can still cap damage from slow resource regressions that escape functional checks.

    If you operate large fleets, consider bounded auto-remediation for kubelet and similar daemons. Pair it with memory-based triggers, not just liveness endpoints.

      Attribution:
    • __turbobrew__ #1

In plain english

cancel function ↩
A Go function returned when creating a cancelable context, which should be called to release resources and signal cancellation.
context ↩
In Go, an object that carries cancellation, deadlines, and request-scoped values across function calls and goroutines.
garbage-collectible ↩
Able to be automatically reclaimed by the programming language runtime when nothing still references it.
Go ↩
A programming language widely used for infrastructure software, including Kubernetes.
go vet ↩
A Go static analysis tool that looks for suspicious code patterns that often indicate bugs.
kubelet ↩
The Kubernetes agent that runs on each node and is responsible for starting, stopping, and monitoring pods on that machine.
Kubernetes ↩
An open source system for deploying and managing containerized applications across clusters of machines.
OOM killer ↩
The Linux mechanism that terminates processes when the system runs out of memory.
OOM score ↩
A Linux priority value used by the OOM killer to decide which process to terminate first under memory pressure.
pod ↩
The basic Kubernetes unit for running one or more containers together on a node.
process_resident_memory_bytes ↩
A metric that reports how much physical RAM a process is currently occupying.
race detector ↩
A Go debugging tool that finds data races by adding runtime instrumentation during program execution.

Reference links

Original writeup and related docs