A service logs an out of memory error. You give it more memory. The error was never about memory, and now you have spent RAM you did not have on a problem you did not fix.
I did this one to myself. Here is the whole thing.
[ERROR] Worker (pid:14630) was sent SIGKILL! Perhaps out of memory?Seven of those in a day. On a container with 1GB of RAM.
That looks like a finished sentence. Small container, out of memory error, seven times a day. So I doubled it to 2GB and rebooted.
Then I checked, and none of it held up.
Every one of those SIGKILL lines had a partner:
[CRITICAL] WORKER TIMEOUT (pid:14630)
[ERROR] Worker (pid:14630) was sent SIGKILL! Perhaps out of memory?The timeout came first. gunicorn killed the worker because it stopped answering, not because it ran out of memory.
gunicorn prints Perhaps out of memory? for any worker it has to SIGKILL. It does not
check. It is a hint in the message text, written by a developer guessing at the most common cause.
I read a guess as a diagnosis.
The kernel is the only thing that knows if it killed something for memory.
Check with command journalctl -k --since '7 days ago' | grep -iE 'oom|killed process'
Zero results. Seven days.
An actual out of memory kill always leaves a kernel record. It prints the process it chose, the score, and a table of what was using memory. If that is not there, the kernel did not do it.
No kernel OOM line means no OOM. Full stop.
I assumed the 15 minute sync job was the memory hog.
Run it by hand and watch:
free -m | sed -n 2p
time docker exec container /srv/app cron run_jobs
free -m | sed -n 2p12 seconds. Memory went from 383MB to 385MB.
2MB. On a 2048MB container.
The job I suspected was not using memory at all.
I had claimed the dead workers explained why my inventory sync kept under delivering hosts.
The sync does not run through gunicorn. It runs as its own process out of the container's crond.
A dead web worker cannot touch it. The two things were never connected. I linked them because they were both in the same container and both looked broken.
Check that the component you are blaming is even in the path of the thing that is failing.
gunicorn defaults to a 30 second timeout. Two workers, no timeout set in the config.
Almost no HTTP requests were being logged at all, so these were not slow requests. They were workers that missed a heartbeat.
The host is a Proxmox box running 42 guests. Under load a worker stalls past 30 seconds, the master kills it, and a replacement boots 45 seconds later.
The container was fine. The hypervisor was busy.
This one has caught me twice now.
Run uptime inside an LXC container and you get a load average around 30. That looks
like your container is on fire.
It is not. On most kernels /proc/loadavg is not namespaced, so every container reports
the host's load as its own.
Run uptime on the hypervisor and compare. If the numbers match, you are reading the
host's load, and it tells you nothing about the container.
Same trap applies to PageTables in /proc/meminfo. If a metric looks identical across
containers doing completely different jobs, it is leaking from the host.
Nothing.
The impact is a web admin page that blips a few times a day. The sync was never affected.
Raising the gunicorn timeout would hide host contention rather than fix it, and the config lives inside the container image, so an update would wipe it.
I left the extra RAM in place as headroom and wrote down that the diagnosis was wrong.
Not every alert needs a change. Some need an accurate note.
Error messages contain two different things. Facts about what happened, and the author's guess at why.
WORKER TIMEOUT is a fact. Perhaps out of memory? is a guess, and it even
has a question mark on it.
Before you act on the why, go find a source that measured it. The kernel log, real usage numbers, a timing run.
Adding resources makes almost any symptom quieter for a while. That is what makes it such an easy wrong answer.