My First Kubernetes Deployment Prepared for a Billion Users

I run a support team. I read Kubernetes output all day in tickets and escalations, but until last week I had never operated a cluster myself. So I did the responsible thing: installed kind on my Mac, deployed a mock customer instance of Nautobot with the official helm chart, and named it Weyland-Yutani, because if a fake company is going to misbehave it should at least be canonically evil.

The plan was an evening of kubectl practice. What I got was a crash loop that survived four fixes, a kernel log full of executions, and eventually a single line of C code from 2011 explaining all of it.

The crash

Ninety seconds after helm install, the web pod was in CrashLoopBackOff. Postgres was fine. Redis was fine. The celery workers were fine. Only the web frontend kept dying, about five seconds after each boot, with a log that looked completely healthy right up to the end:

Nautobot 3.2 initialized!
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 16, cores: 2)
spawned uWSGI worker 2 (pid: 17, cores: 2)
spawned uWSGI worker 3 (pid: 18, cores: 2)
spawned uWSGI http 1 (pid: 21)
stream closed: EOF

Nautobot 3.2 initialized!
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 16, cores: 2)
spawned uWSGI worker 2 (pid: 17, cores: 2)
spawned uWSGI worker 3 (pid: 18, cores: 2)
spawned uWSGI http 1 (pid: 21)
stream closed: EOF

kubectl describe pod gave the cause of death: OOMKilled, exit code 137. The kernel’s memory killer. Fine, I thought. I gave it a small memory limit, Nautobot wants more. Raise the limit.

Dead theories

Here is my actual run ledger from that evening. I’m including it because everyone publishes their clean diagnosis and nobody publishes the attempts that didn’t survive, and those are where the lessons are.

Every theory I had died on contact. It wasn’t undersized: the process ate whatever ceiling I gave it, at roughly a gigabyte per second. It wasn’t request handling: it died with probes off and no traffic. It wasn’t the version. Meanwhile the celery workers, running the exact same image, idled at 220MB like model citizens.

The pattern that finally registered: this thing didn’t leak over time. It detonated at boot, every time, and it always died at exactly whatever the limit was. That is not the shape of an application bug. That is the shape of one enormous allocation hitting a wall.

Asking the kernel

The pod’s own logs end at the moment of death, but the node remembers. dmesg on the kind node had recorded every kill, and the autopsies all shared two odd details. First, only one process in the uWSGI family was huge; the master and the three workers sat at ~200MB each. Second, the doomed process showed the same virtual size every time:

Killed process 3335436 (nautobot-server)

total-vm:8649528kB, anon-rss:3319376kB

Killed process 3335436 (nautobot-server)

total-vm:8649528kB, anon-rss:3319376kB

8,649,528 kB of virtual memory. Every run. Whether it died at 2GB or 5.6GB of resident memory, the process had asked for the same 8.6GB. Hold that number.

Sampling the process tree during a boot caught the culprit’s position: it was the last-spawned child. Not a worker. The HTTP router, spawned uWSGI http 1, the little frontend process that accepts connections and hands them to the Python workers.

The bug

It turns out this is a known uWSGI behavior with a small trail of open issues behind it (unbit/uwsgi#2299, which was even filed from a kind cluster; #2586, closed as a duplicate; linkding#946 hit the same wall).

At boot, the router builds a table to track connections: one pointer slot for every file descriptor the process could ever hold. The code is one line (corerouter.c):

ucr->cr_table = uwsgi_malloc(sizeof(struct corerouter_session *) * uwsgi.max_fd);

ucr->cr_table = uwsgi_malloc(sizeof(struct corerouter_session *) * uwsgi.max_fd);

and then a loop NULLs every entry, which forces the kernel to actually commit every page instead of leaving the allocation theoretical.

On a normal server, max_fd is about a million and this table costs 8MB. Nobody notices it exists. But container runtimes stack their limits, and modern systemd-based stacks can hand a container LimitNOFILE=infinity, which the kernel reports as 1,073,741,816 (2^30, less a few the kernel keeps for itself). Eight bytes times 1,073,741,816 is 8,589,934,528 bytes. My mysterious recurring 8.6GB.

My first Kubernetes deployment looked at its environment, was told it might someday hold a billion simultaneous connections, and earnestly allocated a connection table for all of them. For a demo instance whose realistic peak load was me, clicking.

uWSGI even confesses in its first log lines, if you know to look:

detected max file descriptor number: 1073741816

detected max file descriptor number: 1073741816

The wider ecosystem has been slowly cleaning this up. containerd removed LimitNOFILE=infinity from its service unit in 2023 (containerd#8924) after a series of regressions like this one. But there are a lot of layers between a Mac and a pod, and it only takes one of them re-introducing “unlimited” for the billion-slot table to come back. Mine came through a Colima VM.

The fix

One line in uwsgi.ini:

max-fd = 1048576

max-fd = 1048576

That caps the detected limit at a sane million, shrinking the table from 8.6GB to 8MB. The Nautobot helm chart supports supplying your own uwsgi.ini, so the whole fix is a values override. After it: the pod booted in 40 seconds, passed its health checks, and settled at 240MB resident. My original 2Gi memory limit, the one I’d spent the evening apologizing for, had been right all along.

Worth knowing: the chart’s default uwsgi.ini configures both an http and an https listener, which means two router processes, each of which wants its own table. max-fd bounds both.

Sending the fix upstream

A fix that lives in my values file protects exactly one laptop. The chart’s default still had no guard, and the repo’s own issue tracker showed I wasn’t the first to walk into it: helm-charts#433 was someone’s proof-of-concept hitting the identical wall in 2024, closed without a chart change after the reporter diagnosed it themselves.

So the finding became issue #760 and PR #761: a nautobot.uwsgi.maxFd option modeled on the chart’s existing harakiri pattern, default off so nothing changes for anyone who doesn’t opt in.

The PR taught one last lesson. The chart had no test coverage for its uwsgi rendering, so I wrote some. The first test promptly failed, because the test harness feeds values in as floats and my template line rendered the number in scientific notation: max-fd = 1.048576e+06, which uWSGI would have refused to parse. One int coercion later it passed. The test I wrote to prove my fix worked caught a bug in my fix before any human reviewer saw it. Tests earn their keep fast.

What I actually learned

The exit code tells you who, not why. 137 means SIGKILL; paired with OOMKilled it means the kernel did it. But why has layers: my pod died from its own cgroup limit sometimes and from node-level pressure other times, and dmesg distinguishes them (CONSTRAINT_MEMCG vs global_oom) when kubectl can’t.

“Raised the limit, still OOMKilled” means look down a layer. Once I’d proven the workload ate any ceiling, the problem was by definition not the ceiling.

The trigger can live outside every layer you can see. This bug is not in the application, the image, the chart, the pod spec, or Kubernetes. It’s in a ulimit inherited invisibly through a chain of runtimes. All my early experiments varied things inside the visible boundary, which is exactly why they all produced confusing answers.

Old software plus new defaults is a bug factory. uWSGI predates containers-with-unbounded-limits and has been in maintenance mode since 2022. The code was reasonable when it was written. The environment changed underneath it.

And the check costs nothing. If you run uWSGI in containers, grep your startup logs for detected max file descriptor number right now. If you see ten digits, you’re paying for a table you will never use, and you’re one memory-limit tightening away from meeting the kernel’s OOM killer yourself.

Weyland-Yutani is healthy now, and if #761 merges, the next person’s mock customer never gets sick at all. Building Better Worlds, one correctly sized file descriptor table at a time.