Self-Hosting / Monitoring

Checkmk — Reviewed After Running It as a Real Homelab's Monitoring Backbone

Checkmk has been the nervous system of this homelab for months. Every VM, container, game server and NAS reports through it. It is what pages me when something breaks. This is the honest version of that experience. What it does well at real scale, and several sharp, non-obvious ways it can report "everything's fine" while something important silently is not.

What it's actually monitoring

This is not a lab install poked at for a weekend. It covers dozens of real hosts across a mixed fleet. Proxmox VMs and LXCs, a NAS, cloud droplets, Docker-based game servers. Mostly via real installed agents, not piggyback data forwarded through a hypervisor integration. That distinction decides how much of the rest applies to your setup. What follows is about running Checkmk as infrastructure you depend on, not a dashboard you glance at occasionally.

What it gets right

Once you learn the reliable commands, and there is more on that below, Checkmk is solid at the core job. Agent-based discovery finds real, meaningful services on its own. Systemd units, disk, memory, network, application-specific checks. No hand-written config for each one. The REST API is real and usable for automation. Host creation, bulk changes, querying state. It has sharp edges, covered below. Piggyback data is a genuinely clever pattern once you understand it. One host reports monitoring data on behalf of another, which is how a Proxmox host reports each of its VMs' resource stats. It comes with a real tradeoff, further down.

The big one: its own API can tell you a change applied when it didn't

BE WARNED: this is the single most important thing to know before you trust Checkmk in production. The REST API's activate-changes endpoint can return "Activation status: Complete" while the pending-changes list still shows dozens of unapplied changes. In one real case, 31 changes across 14 hosts, some dating back two days. The changes were real, new hosts and host edits, and they never reached the live monitoring config. The API insisted they had.

One fix has proven reliable. Skip the REST API's activation endpoint and run the CLI reload over SSH as the site user, with cmk -O. That regenerates the config and reloads the core every time. If a newly added host is not showing its real services, or you are not sure recent changes landed, try this first. It is faster than debugging why the API says something untrue.

A correctly-showing check is not the same as someone getting alerted

This is the sharpest lesson from running Checkmk at real scale. It happened twice, in two different ways:

A single misconfigured global notification rule silently discarded every service-level alert, fleet-wide, for as long as the rule existed. The rule's host-event condition was correctly populated. Its service-event condition was empty, so it matched zero service state transitions, ever, across every host. Every CRIT, every WARN, every recovery on any service anywhere in the fleet produced a real check result and zero notification. Host-level down and unreachable alerts still worked, which is exactly why it went unnoticed for so long. The failure mode hides behind "a service degrades while the host itself stays up".
A notification rule created through the REST API stored its SMTP password in a format the notification engine's runtime code does not recognize. The API accepted the config as valid. Schema validation passed. Every real send attempt then failed silently with an unhelpful list index out of range error, visible only by reading the notification engine's own log file. The built-in "test notifications" feature did not catch it either, since it only dry-runs and never sends.

BE WARNED: the only real verification for "will this alert someone" is watching a live event fire through the notification log and confirming it dispatched. Not reading a rule back from the API and confirming it looks right. Not trusting that a check showing the right state in the GUI says anything about who gets told. Those are three separate claims. Checkmk will happily let two of them be true while the third, the one that matters, silently is not.

A false alarm worth knowing about specifically: LXC memory "page tables"

Monitor Linux containers rather than just full VMs and you will eventually see a memory check go critical on "page tables" with a value that looks physically impossible. It exceeds the container's entire RAM allocation. That is a false positive, not a real problem. On some kernel and container configurations the PageTables value in /proc/meminfo is not namespaced per-container, so every container's agent reads and reports the host's own page-table total as its own. Do not adjust the threshold. Checkmk's default is fine for a real host or VM. Tell Checkmk to ignore that one sub-metric for the affected container hosts, and leave the rest of the memory check intact. The rest still reflects real pressure.

Piggyback coverage is not the same as real coverage

It is tempting to treat a hypervisor's piggyback integration as good enough monitoring and skip installing a real agent on each guest. Proxmox's special agent forwards basic stats for every VM it hosts. That leaves you a thin baseline of a few generic services. Meanwhile the host running the hypervisor can end up with barely any coverage of its own, if nobody thinks to give it a proper agent too. Piggyback is a good supplement. It is how you get per-VM resource stats without touching every guest. It is not a substitute for a real installed agent wherever you care about depth. Application processes, systemd unit health, real disk and network detail.

The classic core doesn't scale itself. You have to notice and resize

On Checkmk's free classic edition you are running the Nagios core underneath. It spawns a process per active check rather than using a shared model. That is fine at small scale and falls over hard once it is not. Growing from mostly-piggyback monitoring to real per-host agents across several dozen hosts in a short window pushed one monitoring container's load average from normal into the 30s, on far fewer cores than that. High enough to segfault the whole monitoring stack. Not one component. The entire site, web UI, core, everything, went down together. Here is the diagnostic tell. If load average is very high while individual processes' CPU usage all look low and idle, that is scheduling contention from too many small processes, not one runaway workload. The fix is more cores, not hunting for "the" slow check.

Verdict

Checkmk earns real trust for the core job. Learn to distrust its own "success" claims and verify independently, and it is a capable, flexible monitoring platform that scales to a real mixed fleet without becoming unmanageable. It will not warn you about its own sharp edges. An API that can lie about whether a change landed. A notification pipeline that can silently discard every alert while every underlying check keeps working perfectly. A classic-core deployment that needs someone to notice it has outgrown its sizing before it announces that all at once. None of these are exotic edge cases. They are exactly what bites a real, growing homelab, because nothing about them looks broken from the GUI. Worth the investment if you will verify the parts that matter. A real notification firing. pending_changes actually clearing. Not the dashboard at face value.