Self-Hosting / Monitoring

Checkmk — Reviewed After Running It as a Real Homelab's Monitoring Backbone

Checkmk has been the actual nervous system of this homelab for months — every VM, container, game server, and NAS reports through it, and it's what pages me when something breaks. This is the honest version of that experience: what it genuinely does well at real scale, and several sharp, non-obvious ways it can report "everything's fine" while something important silently isn't.

What it's actually monitoring

This isn't a lab install poked at for a weekend — it's covering dozens of real hosts across a mixed fleet: Proxmox VMs and LXCs, a NAS, cloud droplets, Docker-based game servers, and more, mostly via real installed agents rather than just piggyback data forwarded through a hypervisor integration. That distinction matters for how much of the following actually applies to a given setup — a lot of what follows is specifically about running Checkmk as infrastructure you depend on, not a dashboard you glance at occasionally.

What it gets right

Once you learn the reliable commands (more on that below), Checkmk is genuinely solid at the core job: agent-based discovery finds real, meaningful services automatically — systemd units, disk/memory/network, application-specific checks — without hand-writing config for each one. The REST API is real and usable for automation (host creation, bulk changes, querying state), even though it has some sharp edges covered below. Piggyback data — one host reporting monitoring data on behalf of another, the mechanism behind a Proxmox host reporting each of its VMs' resource stats — is a genuinely clever pattern once you understand it, though it comes with a real tradeoff explained further down.

The big one: its own API can tell you a change applied when it didn't

This is the single most important thing to know before trusting Checkmk in production. The REST API's activate-changes endpoint can return "Activation status: Complete" while the pending-changes list still shows dozens of genuinely unapplied changes — in one real case, 31 changes across 14 different hosts, some dating back two days. The changes were real (new hosts, host edits) and never actually reached the live monitoring config despite the API insisting they had.

The fix that's actually proven reliable: skip the REST API's activation endpoint entirely and run the CLI reload directly over SSH as the site user — cmk -O. This genuinely regenerates the config and reloads the core every time. If a newly-added host isn't showing its real services, or you're not sure whether recent changes actually landed, this is the first thing to try — faster than debugging why the API says something that isn't true.

A correctly-showing check is not the same as someone getting alerted

This is the sharpest, most generalizable lesson from running this at real scale, and it happened twice in two different ways:

A single misconfigured global notification rule silently discarded every service-level alert, fleet-wide, for as long as the rule existed. The rule's host-event condition was correctly populated, but its service-event condition was empty — meaning it matched zero service state transitions, ever, across every host. Every CRIT, every WARN, every recovery on any service anywhere in the fleet produced a real check result and zero notification. Host-level down/unreachable alerts still worked, which is exactly why this went unnoticed for so long — the failure mode specifically hides behind "a service degrades while the host itself stays up."
A notification rule created through the REST API stored its SMTP password in a format the notification engine's runtime code doesn't actually recognize. The API accepted the config as valid — schema validation passed — but every real send attempt failed silently with an unhelpful list index out of range error, visible only by reading the notification engine's own log file directly. The built-in "test notifications" feature didn't catch this either, since it only dry-runs and never actually sends.

The only real verification for "will this actually alert someone" is watching a live event fire through the notification log and confirming it dispatched — not reading a rule back from the API and confirming it looks correctly configured, and not trusting that a check showing the right state in the GUI means anything about who gets told. These are three separate claims, and Checkmk will happily let two of them be true while the third — the one that actually matters — silently isn't.

A false alarm worth knowing about specifically: LXC memory "page tables"

If you're monitoring Linux containers (not just full VMs) and see a memory check go critical on "page tables" with a value that looks physically impossible — exceeding the container's entire RAM allocation — it's very likely a false positive, not a real problem. On at least some kernel/container configurations, the PageTables value in /proc/meminfo isn't properly namespaced per-container, so every container's agent ends up reading and reporting the host's own page-table total as if it were its own. The fix isn't adjusting the threshold (Checkmk's default is fine for a real host or VM) — it's telling Checkmk to ignore that specific sub-metric for affected container hosts specifically, while leaving the rest of the memory check (which usually still reflects real, meaningful pressure) intact.

Piggyback coverage is not the same as real coverage

It's tempting to treat a hypervisor's piggyback integration (Proxmox's special agent forwarding basic stats for every VM it hosts) as "good enough" monitoring and skip installing a real agent on each guest. In practice this leaves you with a thin baseline — a handful of generic services — while the host itself running the hypervisor can end up with barely any real coverage of its own if nobody thinks to give it a proper agent too. Piggyback is a good supplement (it's how you get per-VM resource stats without touching every guest), not a substitute for a real installed agent wherever you actually care about depth — application processes, systemd unit health, real disk/network detail.

The classic core doesn't scale itself — you have to notice and resize

If you're on Checkmk's free/classic edition, you're running the Nagios core underneath, which spawns a process per active check rather than a more efficient shared model. That's fine at a small scale and can fall over hard once it isn't: growing from mostly-piggyback monitoring to real per-host agents across several dozen hosts in a short window pushed one setup's monitoring container's load average from normal into the 30s (on far fewer cores than that), high enough to eventually segfault the whole monitoring stack — not just one component, the entire site (web UI, core, everything) going down together. The diagnostic tell worth knowing: if load average is very high but individual processes' CPU usage all look low and idle, that's scheduling contention from too many small processes, not one runaway workload — the fix is more cores, not finding "the" slow check.

Verdict

Checkmk earns real trust for the core job — once you know to distrust its own "success" claims and verify independently, it's a genuinely capable, flexible monitoring platform that scales to a real mixed fleet without becoming unmanageable. But it will not proactively warn you about several of its own sharp edges: an API that can lie about whether a change landed, a notification pipeline that can silently discard every alert while every underlying check keeps working perfectly, and a classic-core deployment that needs someone to notice it's outgrown its original sizing before it does that noticeably, all at once. None of these are exotic edge cases — they're exactly the kind of thing that bites a real, growing homelab specifically because nothing about them looks broken from the GUI. Worth the investment if you're willing to actually verify the parts that matter (a real notification firing, `pending_changes` actually clearing) rather than trusting the dashboard at face value.