"Just cron a nightly update" is the easy 80% of this. The other 20% — a backup that's actually verified before anything changes, a health check that isn't fooled by a command lying about its own exit code, and per-game update mechanisms that range from "one Docker pull" to "a whole Kubernetes operator stack" — is where a fleet of game servers actually gets left on stale, occasionally broken builds indefinitely. This is a real per-game breakdown of what that took, including two bugs a naive "no update available" check would have hidden forever.
The prompt for this wasn't a game server — it was a self-hosted password manager whose browser extension silently stopped syncing. Root cause: the extension auto-updates itself in the Chrome/Firefox store, and a new client version required a server release that had shipped the day before. The server was one version behind, with zero update automation anywhere in the fleet — everything only ever got patched manually, on no particular schedule, which is exactly how a client/server compatibility gap like that goes unnoticed until something breaks. Game servers have the same exposure, just with a friendlier failure mode (usually "can't connect" instead of "silently missing data").
Every service, no matter how it's actually updated, goes through the same four steps, in this order, every time:
Everything else — Docker vs. a raw binary, a simple systemd service vs. a whole Kubernetes-style operator stack — is just what step 1, 3, and 4 look like for that specific game.
Most self-hosted game servers ship as a Docker image with a floating :latest tag. For these,
the whole check step is comparing the running container's image ID before and after a docker compose
pull — if the ID didn't change, nothing downloaded, and it's safe to stop right there:
CURRENT_ID=$(docker compose images -q <service>)
docker compose pull --quiet
NEW_ID=$(docker compose images -q <service>)
[ "$CURRENT_ID" = "$NEW_ID" ] && exit 0 # nothing changed, don't even take a backupApply is just docker compose up -d to recreate the container on the new image. Health check
is whatever makes sense for that game — an HTTP endpoint if it has one, or a direct port/process check if
it doesn't.
Not every self-hosted game ships a Docker image. Some only distribute a Steam-hosted dedicated server binary meant to be installed directly with SteamCMD, running as a plain systemd service on its own VM — genuinely common for games that added dedicated-server support without ever officially supporting Linux containers. This is the tier that's easy to leave out of an update automation project entirely, since it doesn't fit the "diff a Docker image ID" pattern at all — and then it just doesn't get automated, quietly, possibly for a long time, with nothing anywhere actually saying so.
The fix is the same four-step pattern, just with different plumbing:
steamcmd +login anonymous +app_info_print <appid> +quit, compared against the
installed manifest file SteamCMD already maintains locally
(steamapps/appmanifest_<appid>.acf).+app_update <appid> validate), start the service again.Since this runs on its own VM rather than a container on a shared host, it needs an SSH path in instead of a local exec — a dedicated automation key, authorized only for that one VM, reusing whatever standing sudo access already exists there rather than provisioning a new account just for this.
Some live-service games go a step further and distribute their entire dedicated-server stack as its own lightweight Kubernetes deployment — a database, a message queue, a gateway, and the actual game-world processes, all managed by the publisher's own custom operator rather than a single process. Same SteamCMD-based check step works fine here too — the update itself is still a real Steam depot underneath. Apply and health-check look different:
update && start — this silently skips the required start step after
every single real update, forever, without a single visible error anywhere. It only surfaces if
you deliberately force a real update through the automation end-to-end and watch what actually happens,
not just trust a dry "no update available" check that never exercises the apply path at all. The fix is
to decouple the two steps (update; start, semicolon not &&) and trust
only the independent health check afterward — never a command's own exit code when you don't fully control
what that exit code actually means.
The operator-managed server above already had its own standalone daily update cron before any of this — and it had never actually worked, the entire time it existed. The original theory (documented at the time) was an intermittent failure in the anonymous Steam login used for the version check. That theory was wrong.
The generalizable lesson: when a script that's supposed to fail loudly on any error instead "just doesn't seem to do anything," check the authentication/connectivity layer directly before theorizing about whatever application sits above it. A permission error and an application-level failure can look identical from a distance if the script itself doesn't distinguish them — and it's much faster to rule out "can this even connect" first than to debug the wrong layer for months.
Checking whether a given game server can broadcast a warning message to connected players before an update-triggered restart is worth doing directly rather than assuming — it genuinely varies. Some expose a real in-game console/broadcast command; several common ones expose no console or RCON interface whatsoever, meaning there's no way to warn players in-game at all, full stop.
Rather than build a different notification mechanism per game (or silently skip it for the ones with no console access), one shared voice-chat server already sitting at the center of the community works as a single broadcast channel for every game at once — a short script posts a warning message to it over its own HTTP query API a few minutes before any update actually restarts anything, reused identically across every server in the fleet regardless of what that particular game can or can't do on its own.
Every service in this fleet was confirmed via a live manual trigger before being trusted to run unattended — but "confirmed" meant different things depending on whether the server actually had a real pending update at the time. A dry run that correctly detects "already current" and exits cleanly proves the check step works. It proves nothing about the apply or health-check steps, since they never actually ran. Both real bugs above — the silently-skipped restart and the SSH key that was never authorized — would have passed a dry-run-only verification with a straight face. Catching them took deliberately forcing a real update through the full pipeline at least once per service (temporarily pointing the "installed version" check at an artificially old value is enough to force this safely) and watching every step actually execute, not just trusting that a clean exit code meant the whole thing worked end to end.