Monitoring / Checkmk

Actually Monitor Docker Containers and Non-Systemd Hosts with Checkmk

A request to "alert me if a game server goes down" turned into a much bigger find. A fleet-wide notification rule had been silently discarding every service-level alert. Fixing that came first. Then two real monitoring gaps. Docker containers, which Checkmk does not watch out of the box, and a non-systemd Alpine and OpenRC host that needed its monitoring agent built from scratch.

Check the thing you'd never think to check: does the alert actually fire?

BE WARNED: a check showing the correct state in the GUI is no proof a human would ever be told about it. Confirm notifications for existing checks work before you add any new monitoring. This environment's one global notification rule had a host-level condition that correctly matched up, down and unreachable transitions. Its service-level condition was an empty list. Every WARN, CRIT and recovery notification for every service on every monitored host had been silently discarded the whole time that rule existed. Only whole-host outages ever generated an email.

Checkmk Notifications overview showing 0 notified events for services, and a global rule matching only host DOWN/UNREACHABLE events, not any service event type — **Checkmk → Setup → Notifications.** The bug hiding in plain sight. "Notified events for services … 0" in the overview, and the rule's conditions showing a host match but no service match.

Checkmk's own notification log makes this unambiguous, once you read it instead of checking the rule's config in the GUI:

-> does not match: Event type 'rw' not handled by this rule. Allowed are:

"Allowed are:" trailing off to nothing is Checkmk telling you, per event, that the rule matches zero service transitions. The fix was simple once found. Populate the condition with the right event codes. Checkmk uses single letters. r is recovery or OK, w is warn, c is crit, u is unknown, each optionally prefixed with ? to mean "from any state". The real lesson is the verification method. Do not trust a config diff. Do not trust the GUI's "test notification" feature either, since it only dry-runs, logged as "would notify" and never "notifying". Tail the real notification log and wait for a naturally occurring state change to prove delivery end to end.

Docker containers: invisible to Checkmk by default

If your game server runs in Docker, Checkmk's standard agent has no idea it exists. Same for any other app. A crashed or exited container looks identical to a healthy one from the host's side. CPU, memory and filesystem checks on the host will not move. Checkmk does ship an official plugin for this, mk_docker.py. It is not part of the base agent install, and it has a few real gotchas:

apt-get install -y python3-docker   # the plugin's only dependency
curl -s https://your-checkmk-site/check_mk/agents/plugins/mk_docker.py \
  -o /usr/lib/check_mk_agent/plugins/mk_docker.py
chmod 0755 /usr/lib/check_mk_agent/plugins/mk_docker.py

BE WARNED: the config file's section header is case-sensitive, and nothing warns you. /etc/check_mk/docker.cfg needs a literal uppercase [DOCKER] section. Lowercase [docker] is treated as no config at all. Python's configparser is case-sensitive, and the plugin's logic is effectively "use the DOCKER section if any sections exist, else fall back to defaults". No error, no log line. It quietly runs on hardcoded defaults.

[DOCKER]
base_url = unix://var/run/docker.sock
container_id = combined

BE WARNED: testing the plugin by running it directly gives you misleading results. It reads its config path from the MK_CONFDIR environment variable, which only the real agent wrapper sets. Run it standalone for a quick sanity check and it reads zero config files, falling back to every default. That looks exactly like "my config isn't working" even when the config file is perfectly fine. Always test through the real agent, check_mk_agent or cmk-agent-ctl dump, not the plugin script in isolation.

Use container_id = combined, not the default. Docker containers become separate Checkmk hosts via piggyback data, not extra services on the node. The plugin's default naming is a bare short container ID like b7926c8ea2f3, which is unusable as a hostname. The alternative bare-name mode risks real collisions if you run the same compose stack on multiple hosts, or the same sidecar container name, like a shared tunnel client. combined mode prefixes the container name with the node's own hostname, giving unique, readable names like myhost_mycontainer.

One more step is easy to miss. Each container becomes its own piggyback host, so you have to create that host object yourself before Checkmk discovers any per-container checks. Piggyback data for a host that does not exist yet is silently dropped, not auto-created.

A non-systemd host needs its monitoring built from scratch

Most Linux monitoring setups lean hard on systemd, Checkmk included. Service summaries, socket activation, the works. A minimal non-systemd distro breaks almost every assumption. Alpine Linux with OpenRC, in this case:

BE WARNED: no official package, no scp, no systemd socket activation, and even the base ps command is not compatible. Every one of these needed a real workaround, not a config tweak:

No .deb or .rpm. Install the generic Linux agent script plus the standalone cmk-agent-ctl binary straight from the monitoring site's own agent bundle. Same approach for any distro without a packaged install. The binary is Rust and statically linked, and it ran fine on musl libc with zero special handling.
No scp or SFTP server. A minimal SSH daemon often does not bundle an SFTP server, so scp fails outright. Dropbear, common on embedded and appliance-style Linux images, is the usual one. Pipe through a plain SSH command instead: ssh host "cat > file" < local-file.
No systemd socket activation, and the agent still needs a local unix socket. socat stood in for it. One line, listening on the exact socket path the agent controller expects, executing the agent script on each connection, wired up as a plain OpenRC service instead of a systemd socket unit.
A system user with no home directory breaks the agent controller silently. It reads its own config from a path under the invoking user's home directory, resolved from the password database. No home directory means no config path. That surfaces as a bare No such file or directory with no indication of which file. Give the service account a real home directory even if your instinct says it should not need one.
The stock agent's process list came back empty. It shells out to ps with GNU-specific column flags a minimal ps does not understand. BusyBox, in this case. The command just fails, no error surfaces anywhere, and the process section comes back with a header and zero rows. Installing a full-featured ps at the same path fixed it with no changes to the agent script. Alpine's procps package, here.

With no systemd-unit-summary check to lean on for "is my application still running", the fallback is a process-match rule. Checkmk can discover and monitor an arbitrary process by matching its command line, with thresholds for how many instances should exist. That catches both "it crashed" and "it somehow started twice". Match on the top-level process, not the worker or child processes it spawns. Those come and go constantly under normal operation, and they would just be alert noise.

Verify with real state, not rule configuration

For every check added here, the last step was pulling the check's live state and output back from the monitoring API. Not confirming the rule existed, or that it looked right. A rule that is present but never discovered looks completely fine right up until you need it. So does one that is discovered but never notifying, per the first section of this page.