Monitoring / Checkmk

Actually Monitor Docker Containers and Non-Systemd Hosts with Checkmk

A request to "alert me if a game server goes down" turned into a much bigger find: a fleet-wide notification rule that had been silently discarding every single service-level alert. Fixing that came first — then closing two real monitoring gaps: Docker containers, which Checkmk doesn't watch out of the box, and a non-systemd (Alpine/OpenRC) host that needed its monitoring agent built from scratch.

Check the thing you'd never think to check: does the alert actually fire?

A check showing the correct state in the GUI is not proof a human would ever be told about it. Before adding any new monitoring, it's worth confirming notifications for existing checks actually work — this environment's one global notification rule had a host-level condition that correctly matched up/down/unreachable transitions, but its service-level condition was an empty list. Every WARN, CRIT, and recovery notification for every service on every monitored host had been silently discarded the entire time that rule existed — only whole-host outages ever generated an email.

Checkmk's own notification log makes this unambiguous once you actually read it instead of just checking the rule's config in the GUI:

-> does not match: Event type 'rw' not handled by this rule. Allowed are:

"Allowed are:" trailing off to nothing is Checkmk telling you, per event, that the rule matches zero service transitions. The fix itself was simple once found — populate the condition with the right event codes (Checkmk uses single letters: r=recovery/OK, w=warn, c=crit, u=unknown, each optionally wildcard-prefixed with ? to mean "from any state"). The real lesson is the verification method: don't trust a config diff or the GUI's "test notification" feature (which only dry-runs — logged as "would notify," never "notifying"). Tail the real notification log and wait for a genuine, naturally-occurring state change to prove delivery end to end.

Docker containers: invisible to Checkmk by default

If your game server (or any app) runs in Docker, Checkmk's standard agent has no idea it exists. A crashed or exited container looks identical to a healthy one from the host's perspective — CPU/memory/filesystem checks on the host itself won't move. Checkmk does ship an official plugin for this, mk_docker.py, but it isn't part of the base agent install and has a few real gotchas:

apt-get install -y python3-docker   # the plugin's only dependency
curl -s https://your-checkmk-site/check_mk/agents/plugins/mk_docker.py \
  -o /usr/lib/check_mk_agent/plugins/mk_docker.py
chmod 0755 /usr/lib/check_mk_agent/plugins/mk_docker.py

The config file's section header is case-sensitive, and nothing warns you. /etc/check_mk/docker.cfg needs a literal uppercase [DOCKER] section — lowercase [docker] is silently treated as no config at all, since Python's configparser is case-sensitive and the plugin's own logic is effectively "use the DOCKER section if any sections exist, else fall back to defaults." No error, no log line — it just quietly runs on hardcoded defaults.

[DOCKER]
base_url = unix://var/run/docker.sock
container_id = combined

Testing the plugin by running it directly gives you misleading results. It reads its config path from the MK_CONFDIR environment variable, which only the real agent wrapper sets. Run it standalone for a quick sanity check and it silently reads zero config files, falling back to every default — which looks exactly like "my config isn't working" even when the config file itself is perfectly fine. Always test through the real agent (check_mk_agent or cmk-agent-ctl dump), not the plugin script in isolation.

Use container_id = combined, not the default. Docker containers become separate Checkmk hosts via piggyback data, not extra services on the node — and the plugin's default naming (a bare short container ID like b7926c8ea2f3) is unusable as a hostname, while the alternative bare-name mode risks real collisions if you run the same compose stack (or the same sidecar container name, like a shared tunnel client) on multiple hosts. combined mode prefixes the container name with the node's own hostname, giving unique, readable names like myhost_mycontainer.

One more step easy to miss: since each container becomes its own piggyback host, you have to create that host object yourself before Checkmk will discover any per-container checks — piggyback data for a host that doesn't exist yet is silently dropped, not auto-created.

A non-systemd host needs its monitoring built from scratch

Most Linux monitoring setups (Checkmk included) lean hard on systemd — service summaries, socket activation, the works. A minimal, non-systemd distro (Alpine Linux with OpenRC, in this case) breaks almost every assumption:

No official package, no scp, no systemd socket activation, and even the base ps command isn't compatible. Every one of these needed a real workaround, not a config tweak:

No .deb/.rpm: installed the generic Linux agent script plus the standalone cmk-agent-ctl binary directly from the monitoring site's own agent bundle — the same approach needed for any distro without a packaged install. The binary itself (Rust, statically linked) ran fine on musl libc with zero special handling.
No scp/SFTP server: a minimal SSH daemon (Dropbear, common on embedded/appliance-style Linux images) often doesn't bundle an SFTP server, so scp fails outright. Piping through a plain SSH command instead works fine: ssh host "cat > file" < local-file.
No systemd socket activation, but the agent still needs a local unix socket: socat stood in for it — one line, listening on the exact socket path the agent controller expects and executing the agent script on each connection, wired up as a plain OpenRC service instead of a systemd socket unit.
A "system user with no home directory" breaks the agent controller silently. It reads its own config from a path under the invoking user's home directory, resolved from the password database — no home directory means no config path, which surfaces only as a bare No such file or directory with no indication of which file. Give the service account a real home directory even if your instinct says it shouldn't need one.
The stock agent's process list silently came back empty. It shells out to ps with GNU-specific column flags that a minimal ps implementation (BusyBox, in this case) doesn't understand — the command just fails, with no error surfaced anywhere, and the process section comes back with a header and zero rows. Installing a real, full-featured ps (Alpine's procps package, in this case) at the same path fixed it with no changes to the agent script itself.

With no systemd-unit-summary check to lean on for "is my actual application still running," the fallback is a specific process-match rule — Checkmk can discover and monitor an arbitrary process by matching its command line, with thresholds for how many instances should exist (useful for catching both "it crashed" and "it somehow started twice"). Match on the actual top-level process, not any worker/child processes it spawns — those can come and go constantly under normal operation and would just be alert noise.

Verify with real state, not rule configuration

For every check added here, the last step was pulling the check's actual live state and output back from the monitoring API — not just confirming the rule existed or looked right. A rule that's present but never discovered, or discovered but never notifying (see the very first section of this page), looks completely fine right up until the moment you actually need it.