Backups / Monitoring

Diagnose a Silently Broken btrfs Backup (and Make Sure It Can't Happen Again)

A workstation's off-site backup had been failing every single night for five nights straight — with zero alerting anywhere. Here's how it was found by accident, the obvious suspect that turned out to be completely unrelated, the actual root cause, the fix, and how it got real monitoring afterward so this class of bug can't hide again.

The setup

A common two-layer btrfs backup pattern: local snapshots on every system change (via snapper), plus a nightly incremental btrfs send / btrfs receive off-box to a NAS (via btrbk), run as a systemd oneshot service on a timer.

volume /mnt/btrfs-root
  subvolume @
    snapshot_name     root
    target            nas.example.com:/volume1/backups
  subvolume @home
    snapshot_name     home
    target            nas.example.com:/volume1/backups

Found by accident, not by alerting

A systemd oneshot service sitting in failed state produces zero proactive signal on its own. Nothing surfaces it in normal use — no email, no desktop notification, nothing. This one had been broken for 5 consecutive nights before an unrelated question ("should X also be backed up?") prompted a check of what was actually covered, which is what turned up systemctl status btrbk.service reporting failed.

The obvious suspect that wasn't it

This machine had a known-flaky secondary NVMe drive that had dropped to an unresponsive state before (twice), and had done so again around the same time — the natural first suspect. Before assuming anything, the actual physical layout got checked directly rather than guessed at:

findmnt /mnt/btrfs-root
lsblk -o NAME,SIZE,MODEL,MOUNTPOINT

The backed-up filesystem lived entirely on a completely different, healthy drive. Kernel/journal logs for the flaky drive showed only errors scoped to that device's own health checks — no PCIe-bus-level symptoms, nothing touching the actual backup source. The dropped drive was not the cause — confirmed, not assumed, before moving on.

Lesson: when there's a known-flaky component in the system, check that it's actually in the causal path before treating it as the explanation — "this thing is broken and something else broke around the same time" isn't the same as "this thing caused that." A five-minute device/mount trace settled it definitively instead of chasing a red herring.

The actual root cause: a version mismatch, not corruption

The real error, from the service's own log:

ERROR: Failed to send/receive subvolume: .../root.20260718T0300 [.../root.20260717T0937] -> nas:.../root.20260718T0300
ERROR: ... attribute 12 requested but not present.
ERROR: Error while resuming backups, aborting

The sending side was running a modern btrfs-progs (v7.x); the receiving NAS was running an ancient, vendor-bundled version (v4.x) that isn't independently upgradable. Something in that night's incremental delta produced a send-stream command using a newer timestamp attribute the old receiver's parser simply didn't understand, aborting the transfer partway through.

The real damage: because the backup tool needs the previous successful target-side snapshot as the parent for the next incremental, one broken night permanently blocked every night after it. The service wasn't randomly failing each night — it was retrying the exact same doomed transfer every single time, never even attempting the newer snapshots waiting behind it.

A partial, corrupted subvolume from the interrupted transfer was still sitting on the NAS target, confirmed via:

btrfs subvolume show /volume1/backups/root.20260718T0300

It had a Received UUID already assigned — proof the stream got partway through writing before it died, not that it failed instantly.

The fix: route around the incompatible delta, don't fight it

Rather than trying to identify the exact file operation that triggered the incompatible stream command, the pragmatic fix was a one-time full, non-incremental resend of the current state — full sends use a simpler stream encoding than incrementals and bypass the specific broken delta entirely:

btrfs send /mnt/btrfs-root/.snapshots/root.LATEST | \
  ssh nas.example.com "btrfs receive /volume1/backups/"

Verified success by checking the receiving end got a real Received UUID with the readonly flag set — not just "the command didn't error." Then cleanup: deleted the corrupted partial subvolume on the NAS, and deleted the handful of local snapshots that had never successfully synced and were now superseded by the fresh baseline, so the backup tool wouldn't keep trying (and failing) to walk through them on future runs. A manual run afterward completed with zero errors — the incremental chain was healthy again going forward.

Lesson: when an incremental send/receive pipeline breaks partway through and then keeps retrying identically forever, suspect a version/protocol mismatch between the two ends before suspecting data corruption. A forced full resync is a legitimate, low-risk way to route around one incompatible delta without needing to reverse-engineer the exact triggering operation — you don't have to fully understand a bug to route around it safely.

Closing the actual gap: this needed real monitoring, not just a fix

Fixing the immediate failure doesn't prevent the next silent one. The real fix was adding this host to the existing Checkmk monitoring stack, so a failed systemd unit generates a real alert instead of waiting for someone to notice.

This particular workstation runs a distro with no native .deb/.rpm packaging, so the agent went on manually from Checkmk's own generic Linux artifacts rather than a packaged install — every modern Checkmk site serves these directly, no special access needed:

# the plain agent data-collector script, and the TLS registration/transport binary
curl -s https://checkmk.example.com/site/check_mk/agents/check_mk_agent.linux -o /usr/bin/check_mk_agent
curl -s https://checkmk.example.com/site/check_mk/agents/linux/cmk-agent-ctl.gz | gunzip > /usr/bin/cmk-agent-ctl
chmod 0755 /usr/bin/check_mk_agent /usr/bin/cmk-agent-ctl

# systemd units: a socket-activated agent + a TLS controller daemon on 6556/tcp
# (copy the exact unit files from any already-working packaged install on the
# same Checkmk version with `systemctl cat check-mk-agent.socket [email protected]
# cmk-agent-ctl-daemon.service` — they're portable, no distro-specific content)

cmk-agent-ctl register --hostname my-workstation --server checkmk.example.com \
  --site mysite --user automation --password <secret> --trust-cert

Gotcha: a local firewall can silently block the agent even though everything looks correctly installed and running. The controller daemon listens on 6556/tcp same as any packaged install, but if this is the only host in the fleet running its own firewall (most VMs/containers rely on network-level segmentation instead), that port needs an explicit allow rule — otherwise the monitoring server's fetch just times out with no useful error on either side. Scope the rule to the monitoring server specifically rather than opening the port broadly:

ufw allow from <checkmk-server-ip> to any port 6556 proto tcp

You probably don't need a dedicated check — the default summary already does this

The instinct is to configure a specific, individually-named check for the one service you care about. Before doing that: Checkmk's stock "Systemd Service Summary" check — discovered automatically on every Linux host, zero configuration required — already treats any unit entering failed state as critical by default, and names the specific failed unit(s) in its output. A oneshot backup service's normal healthy resting state is inactive, not active, so this doesn't false-positive between runs — only a genuine failed state trips it.

Lesson: check what your monitoring tool already does by default before building something custom. A dedicated per-service check attempt here hit an internal rule-matching quirk that wasn't worth chasing down, and turned out to be unnecessary anyway — the fleet-wide summary check already delivered exactly the alerting behavior needed, with the specific failed unit named right in the output.