Backups / Monitoring

btrbk btrfs Backups Silently Stopped: How to Diagnose and Monitor Them

A workstation's off-site backup had failed every night for five nights straight, with zero alerting anywhere. Here is how it was found by accident, the obvious suspect that turned out to be unrelated, the real root cause, the fix, and the monitoring added afterward so this class of bug cannot hide again.

The setup

A common two-layer btrfs backup pattern. Local snapshots on every system change, via snapper. Plus a nightly incremental btrfs send and btrfs receive off-box to a NAS, via btrbk, run as a systemd oneshot service on a timer.

volume /mnt/btrfs-root
  subvolume @
    snapshot_name     root
    target            nas.example.com:/volume1/backups
  subvolume @home
    snapshot_name     home
    target            nas.example.com:/volume1/backups

Found by accident, not by alerting

BE WARNED: a systemd oneshot service sitting in failed state produces zero proactive signal on its own. Nothing surfaces it in normal use. No email, no desktop notification, nothing. This one had been broken for 5 consecutive nights. An unrelated question, "should X also be backed up?", prompted a check of what was actually covered. That is what turned up systemctl status btrbk.service reporting failed.

The obvious suspect that wasn't it

This machine has a known-flaky secondary NVMe drive. It had dropped to an unresponsive state twice before, and had done so again around the same time. The natural first suspect. Before assuming anything, I checked the physical layout directly rather than guessing:

findmnt /mnt/btrfs-root
lsblk -o NAME,SIZE,MODEL,MOUNTPOINT

The backed-up filesystem lived entirely on a different, healthy drive. Kernel and journal logs for the flaky drive showed errors scoped to that device's own health checks. No PCIe-bus-level symptoms. Nothing touching the backup source. The dropped drive was not the cause. Confirmed, not assumed, before moving on.

Lesson: when there is a known-flaky component in the system, check it is in the causal path before treating it as the explanation. "This thing is broken and something else broke around the same time" is not "this thing caused that". A five-minute device and mount trace settled it instead of chasing a red herring.

The actual root cause: a version mismatch, not corruption

The real error, from the service's own log:

ERROR: Failed to send/receive subvolume: .../root.20260718T0300 [.../root.20260717T0937] -> nas:.../root.20260718T0300
ERROR: ... attribute 12 requested but not present.
ERROR: Error while resuming backups, aborting

The sending side ran a modern btrfs-progs, v7.x. The receiving NAS ran an ancient vendor-bundled v4.x that is not independently upgradable. Something in that night's incremental delta produced a send-stream command using a newer timestamp attribute the old receiver's parser did not understand. That aborted the transfer partway through.

BE WARNED, here is the real damage. The backup tool needs the previous successful target-side snapshot as the parent for the next incremental, so one broken night permanently blocked every night after it. The service was not failing randomly each night. It retried the exact same doomed transfer every time, never attempting the newer snapshots waiting behind it.

A partial, corrupted subvolume from the interrupted transfer was still sitting on the NAS target. Confirmed with:

btrfs subvolume show /volume1/backups/root.20260718T0300

It already had a Received UUID assigned. Proof the stream got partway through writing before it died, rather than failing instantly.

The fix: route around the incompatible delta, don't fight it

I did not try to identify the exact file operation that triggered the incompatible stream command. The pragmatic fix was a one-time full, non-incremental resend of the current state. Full sends use a simpler stream encoding than incrementals, and bypass the broken delta entirely:

btrfs send /mnt/btrfs-root/.snapshots/root.LATEST | \
  ssh nas.example.com "btrfs receive /volume1/backups/"

Verify success by checking the receiving end got a real Received UUID with the readonly flag set. Not by "the command didn't error". Then clean up. Delete the corrupted partial subvolume on the NAS. Delete the handful of local snapshots that never synced and are now superseded by the fresh baseline, so the backup tool does not keep trying and failing to walk through them on future runs. A manual run afterward completed with zero errors. The incremental chain was healthy again.

Lesson: when an incremental send and receive pipeline breaks partway through and then retries identically forever, suspect a version or protocol mismatch between the two ends before data corruption. A forced full resync is a legitimate, low-risk way to route around one incompatible delta. You do not have to reverse-engineer the triggering operation. You do not have to fully understand a bug to route around it safely.

Closing the actual gap: this needed real monitoring, not just a fix

Fixing the immediate failure does not prevent the next silent one. The real fix was adding this host to the existing Checkmk monitoring stack, so a failed systemd unit generates a real alert instead of waiting for someone to notice.

This workstation runs a distro with no native .deb or .rpm packaging, so the agent went on by hand from Checkmk's own generic Linux artifacts. Every modern Checkmk site serves these directly, no special access needed:

# the plain agent data-collector script, and the TLS registration/transport binary
curl -s https://checkmk.example.com/site/check_mk/agents/check_mk_agent.linux -o /usr/bin/check_mk_agent
curl -s https://checkmk.example.com/site/check_mk/agents/linux/cmk-agent-ctl.gz | gunzip > /usr/bin/cmk-agent-ctl
chmod 0755 /usr/bin/check_mk_agent /usr/bin/cmk-agent-ctl

# systemd units: a socket-activated agent plus a TLS controller daemon on 6556/tcp.
# Copy the exact unit files from any already-working packaged install on the same
# Checkmk version with `systemctl cat check-mk-agent.socket [email protected]
# cmk-agent-ctl-daemon.service`. They are portable, with no distro-specific content.

cmk-agent-ctl register --hostname my-workstation --server checkmk.example.com \
  --site mysite --user automation --password <secret> --trust-cert

BE WARNED: a local firewall silently blocks the agent even though everything looks correctly installed and running. The controller daemon listens on 6556/tcp, same as any packaged install. If this is the only host in the fleet running its own firewall, and most VMs and containers rely on network-level segmentation instead, that port needs an explicit allow rule. Without one the monitoring server's fetch times out with no useful error on either side. Scope the rule to the monitoring server rather than opening the port broadly:

ufw allow from <checkmk-server-ip> to any port 6556 proto tcp

You probably don't need a dedicated check. The default summary already does this

The instinct is to configure a specific, individually named check for the one service you care about. Look at what you already have first. Checkmk's stock "Systemd Service Summary" check is discovered automatically on every Linux host with zero configuration. It already treats any unit entering failed state as critical, and names the failed units in its output. A oneshot backup service's healthy resting state is inactive, not active, so it does not false-positive between runs. Only a genuine failed state trips it.

Checkmk service list showing Systemd Service Summary: Total 148, Disabled 18, Failed 0 — **Checkmk → Monitor → All services, filtered to one host.** The stock "Systemd Service Summary" check, healthy here at "Failed: 0". This is what caught the backup service the previous time it silently failed.

Lesson: check what your monitoring tool already does by default before you build something custom. A dedicated per-service check here hit an internal rule-matching quirk not worth chasing down. It was unnecessary anyway. The fleet-wide summary check already delivered the exact alerting behavior needed, with the failed unit named right in the output.