A routine check came back with 36 services critical and one host down, minutes after the fleet had been completely clean. The host was never in trouble. Working that out took three separate misreadings of the same number, in one sitting, and the correction is identical every time.
Every one of those 36 entries said the same thing.
(Service Check Timed Out)That included the checks on the monitoring host itself.
Thirty-six unrelated services do not fail at once. A monitoring path degrades and everything behind it times out together. Those look identical on a dashboard and they are not the same event.
It drained on its own while being watched. 36, then 34, then 24, then 17, then 5, then 0. Hosts down went back to zero with it.
A self-clearing wall of timeouts is a capacity blip in the checking path. Treat it as one incident, not 36.
The first look showed this.
load average: 51.40Fifty-one sounds like a fire. It was called CPU saturation on the spot.
The box has 32 cores.
That is 1.6 per core at the peak. The monitoring system was calling it fine the entire time, because it divides by core count before it decides:
CPU load;0;15 min load: 38.14, 15 min load per core: 1.19 (32 cores)Get the core count first with command nproc. Divide, then react.
A load average with no core count next to it is not a measurement. It is a number.
Next look, load was 45. And the CPU was 67.9% idle.
Both were true at the same second.
Linux load counts processes in uninterruptible sleep as well as runnable ones. That means disk and network waits land in the same figure as real CPU demand. High load beside an idle CPU means blocking, not CPU pressure.
So stop guessing and count the blocked processes. List processes in D state with command ps -eo state,pid,comm | awk '$1 ~ /D/'
There were exactly two. Both were ZFS kernel threads. The pool was healthy and the I/O was unremarkable.
Two blocked kernel threads is not an incident. Had it been forty, sitting on one disk, that is a different article and a worse day.
Then a container reported this.
load average: 49.28It has nothing like the workload to produce that.
It did not produce it. /proc/loadavg is not namespaced. An LXC container reads
the host's figure and reports it as its own. Its real state was 27% idle.
This is worth internalising because it corrupts everything downstream. Per-container dashboards, alert thresholds and capacity planning all inherit a number that belongs to a different machine. Every container on a busy host looks like it is on fire at the same moment, which is also the moment you are least able to think clearly about it.
Check where you actually are before trusting it:
systemd-detect-virt --container # prints lxc, docker, or none
cat /proc/loadavg # inside a container, likely the host'sInside a container, get pressure from the cgroup instead, with command cat /sys/fs/cgroup/cpu.stat and look at the throttled counters. That is yours. The load average is not.
All three mistakes come from reading a number before establishing what it describes.
/proc.