A Guest Watchdog That Restarts Stopped VMs, and the Hold File That Stops It Fighting You

A game server stopped and stayed stopped for roughly 40 hours. Monitoring caught it, emailed once at 02:47, and then went quiet because nothing had changed. Nobody read that email. Nothing ever tried to start the thing again. Two separate gaps, and only one of them is a monitoring problem.

onboot solves a different problem than you think

Proxmox's onboot flag starts a guest when the host boots. That is the only case it covers. A guest that stops for any other reason, a crash, an out-of-memory kill, an update that never came back, someone stopping it by hand and getting distracted, stays stopped indefinitely and nothing in the stack disagrees.

So while you are here, check that the flag is even set. Two of the most load-bearing guests in this fleet turned out to have it unset, including the one carrying the media server, a VLAN bridge and several timers. Neither would have survived a host reboot:

for id in $(pct list | awk 'NR>1{print $1}'); do
  printf '%s ct onboot=%s\n' "$id" "$(pct config $id | awk '/^onboot:/{print $2}')"
done
for id in $(qm list | awk 'NR>1{print $1}'); do
  printf '%s vm onboot=%s\n' "$id" "$(qm config $id | awk '/^onboot:/{print $2}')"
done

Let onboot define what "should be running" means

The hard part of a watchdog is not restarting things. It is knowing what deserves restarting, without keeping a hand-maintained list that goes stale the first time you build something.

Reuse onboot: 1 as the answer. It already means "this is meant to be up", you already set it, and it makes membership self-maintaining. Setting the flag opts a guest in. Unsetting it opts out. Every deliberately-stopped guest in this fleet, retired projects and half-built experiments, has no flag and is never touched, without anybody having to enumerate them.

The hold file

touch /etc/guest-watchdog/hold     # before any planned maintenance
rm    /etc/guest-watchdog/hold     # after

A file is the right shape for this. It survives the timer restarting, it survives your SSH session dropping, and anyone can see the current state with ls rather than by reading a script.

Three conditions where doing nothing is correct

• The guest holds a lock. A lock: line in the config means a backup, snapshot or migration is in progress. Starting it now corrupts the operation. Skip it.
• A backup run is active. Skip the entire cycle while vzdump is running, rather than checking guests one at a time, because a guest can enter the locked state between your check and your action.
• The hold file exists. As above.

Rate limit it, or it becomes the outage

A guest that fails immediately on start will be restarted forever by a naive loop. Every five minutes, against storage and a hypervisor that have other jobs. Cap attempts and then stop:

MAX_ATTEMPTS=3      # within a 6 hour window, then give up and escalate

Giving up is the feature. Three failed starts is not a transient blip, it is a broken guest, and the correct response is a person rather than a fourth attempt.

Make it act, and let your existing monitoring alert

The temptation is to have the watchdog email you. Resist it, because that means a second notification path with its own credential, its own failure modes and nothing watching it.

Instead have the watchdog write state, and expose that state as a check in the monitoring you already run. A local check on the host is enough:

0 Guest_Watchdog - all 34 monitored guests running
1 Guest_Watchdog - restarted palworld (attempt 1/3)
2 Guest_Watchdog - dune down, 3/3 attempts exhausted

Three states worth distinguishing. OK is everything running. WARN is something was automatically restarted, which you still want to know about even though it is fixed, because a guest that keeps needing restarts is a real problem being hidden by automation. CRIT is down and not recovering.

Also alert on the watchdog dying

A watchdog that stops running looks exactly like a watchdog with nothing to do. Both produce silence. Have the check go CRIT when the state file is older than a few cycles:

if (( $(date +%s) - $(stat -c %Y "$STATE") > 1800 )); then
  echo "2 Guest_Watchdog - state file stale, the timer is not running"
fi

Thirty minutes against a five-minute timer. Generous enough to survive a slow cycle, short enough to notice the same day.

Test all three paths before you trust it

Not just the happy one. Watch a healthy cycle do nothing. Then deliberately stop a guest you do not care about, confirm the watchdog starts it, and put it back. Then set the hold file, stop something, and confirm nothing happens. The third test is the one people skip, and it is the one that matters at 2am during a maintenance window.

When this isn't your problem

• You run a real cluster with HA enabled. Proxmox HA restarts guests when a node fails, and does it properly. It does not cover a guest stopping on a healthy node, which is the more common case, so a watchdog still earns its place. See why two nodes cannot survive one rebooting.
• Your services are containers under a supervisor. Docker's restart: unless-stopped and systemd's Restart=on-failure already do this one level down. The gap is the layer above them, the VM or container they live inside.
• You want to know before it self-heals. Then keep the alerting and skip the restarting. Just be honest that this is a choice to accept downtime, and see the check that can never pass for how monitoring quietly stops telling you things.