Tracing a memory leak bug in PID 1 and contributing an upstream fix: a Linux support story

Some memory leaks are straightforward to detect. A process allocates memory, forgets to free it, and grows until something breaks. This was not that kind of leak.

This is the story of how Canonical Support helped a global retail organization trace the cause for an unusual memory leak originating in PID 1, the very first process started by the kernel during the system boot sequence. The memory footprint was 10 times higher than expected. By investigating the issue across three separate system layers – a misconfigured storage orchestrator, a kernel race condition, and glibc’s allocator – our team was able to identify the source and fast-track a patch.

The symptom that made no sense

The customer was the innovation and engineering arm of a global retail company. They were responsible for driving the company’s retail transformation – operating a fleet of servers running as part of a Ceph storage cluster. Across several of these cluster nodes, they began observing a pattern that made no immediate sense: /sbin/init (a symlink to systemd, which was running as PID 1) was consuming over 50% of available system memory. On servers with 125 GB of RAM, that meant PID 1 alone was holding approximately 70 GB, when only 8 GB were expected. The systems were experiencing out-of-memory kills, with the kernel forcibly terminating processes to free up available memory to survive.

Nothing in the obvious diagnostic places explained it. Memory consumption continued to grow, and was never released. The investigation didn’t point to any obvious application-level cause. The customer had ruled out the workloads running on the servers as being the issue. The problem was somewhere deeper, below the application layer.

Starting the investigation: getting the right artifact

The support team at Canonical’s first move was to request sos-reports from the affected nodes, the standard starting point for system-level investigation. But sos-report output, comprehensive as it is, didn’t contain anyth…