Monitoring / Automation

Automatic Updates for Docker Game Servers, Gated on a Verified Backup

"Just cron a nightly update" is the easy 80% of this. The other 20% is where a fleet of game servers gets left on stale, occasionally broken builds indefinitely. A backup verified before anything changes. A health check that is not fooled by a command lying about its own exit code. Per-game update mechanisms ranging from one Docker pull to a whole Kubernetes operator stack. This is the per-game breakdown of what that took, including two bugs a naive "no update available" check would have hidden forever.

Why bother automating this at all

The prompt for this was not a game server. It was a self-hosted password manager whose browser extension silently stopped syncing. The extension auto-updates itself in the Chrome and Firefox stores, and a new client version required a server release that had shipped the day before. The server was one version behind, with zero update automation anywhere in the fleet. Everything got patched by hand, on no particular schedule. That is exactly how a client and server compatibility gap goes unnoticed until something breaks. Game servers have the same exposure with a friendlier failure mode. Usually "can't connect" instead of "silently missing data".

The core pattern, regardless of what's underneath

Every service goes through the same four steps, in this order, every time. However it is actually updated:

Check whether an update is available. Compare a "what's installed" value against a "what's live" value. If they match, stop here. No backup, no restart, no risk, for the common case where nothing changed.
Back up, always, before touching anything. Abort the whole run if the backup itself fails. No update is worth a bad snapshot being the only thing between "minor annoyance" and "lost world".
Apply the update.
Health-check independently afterward. Do not trust the apply step's own exit code, for reasons below. Check directly that the service is up and doing its job.

Everything else is just what steps 1, 3 and 4 look like for that specific game. Docker or a raw binary. A simple systemd service or a whole Kubernetes-style operator stack.

Docker-based servers: the easy tier

Most self-hosted game servers ship as a Docker image with a floating :latest tag. For those, the whole check step is comparing the running container's image ID before and after a docker compose pull. If the ID did not change, nothing downloaded, and it is safe to stop right there:

CURRENT_ID=$(docker compose images -q <service>)
docker compose pull --quiet
NEW_ID=$(docker compose images -q <service>)
[ "$CURRENT_ID" = "$NEW_ID" ] && exit 0   # nothing changed, don't even take a backup

Apply is docker compose up -d to recreate the container on the new image. Health check is whatever makes sense for that game. An HTTP endpoint if it has one, a direct port or process check if it does not.

Multi-container stacks need every service pulled, not just the one you care about. A photo-management or file-sync stack with a separate worker container needs both to move together. Otherwise you get an API server on a new protocol version talking to a worker that does not speak it yet. Pulling every service in the compose file is safe even for the ones that never change. A database or cache container pinned to a specific tag and digest is a no-op pull every time.

Raw-binary and VM-based servers: the tier that gets skipped

Not every self-hosted game ships a Docker image. Some only distribute a Steam-hosted dedicated server binary, installed with SteamCMD and running as a plain systemd service on its own VM. That is common for games that added dedicated-server support without ever officially supporting Linux containers. This tier is easy to leave out of an update automation project, because it does not fit the "diff a Docker image ID" pattern at all. So it quietly does not get automated, possibly for a long time, with nothing anywhere saying so.

The fix is the same four-step pattern, just with different plumbing:

Check. SteamCMD queries an app's live build ID without installing anything. steamcmd +login anonymous +app_info_print <appid> +quit, compared against the installed manifest file SteamCMD already maintains locally at steamapps/appmanifest_<appid>.acf.
Apply. Stop the systemd service, re-run SteamCMD's own update command +app_update <appid> validate, start the service again.
Health check. Confirm the systemd unit is active and the game's real network port is listening. Not that the stop and start commands did not error.

This runs on its own VM rather than a container on a shared host, so it needs an SSH path in instead of a local exec. A dedicated automation key, authorized only for that one VM, reusing whatever standing sudo access already exists there rather than provisioning a new account for this.

The harder end of that same tier: an operator-managed server

Some live-service games distribute their entire dedicated-server stack as its own lightweight Kubernetes deployment. A database, a message queue, a gateway and the game-world processes, all managed by the publisher's custom operator rather than one process. The same SteamCMD-based check step works fine, because the update is still a real Steam depot underneath. Apply and health-check look different:

Apply. The publisher's own management script pulls new container images for every internal component and reloads them into the local cluster. It usually needs a separate, explicit "start" step afterward. An update alone leaves the world stopped, expecting a human or an automation to bring it back up as a distinct action.
Health check. The only real signal is the game-world pods reporting ready with zero restarts. Not the supporting database, queue and gateway pods, which look "running" even when the world never came back.

BE WARNED: a management script reporting failure does not always mean it failed. One publisher's own update script exits non-zero even on a fully successful update, because of a couple of harmless "symlink already exists" warnings it does not suppress internally. Chain it the obvious way, as update && start, and it silently skips the required start step after every single real update, forever, with no visible error anywhere. It only surfaces if you force a real update through the automation end to end and watch what happens. A dry "no update available" check never exercises the apply path at all. The fix is to decouple the two steps, running update then start as separate statements, and to trust only the independent health check afterward. Never a command's own exit code, when you do not control what that exit code means.

The bug that predates this project, and the lesson worth generalizing

The operator-managed server above already had its own standalone daily update cron before any of this. It had never worked, the entire time it existed. The theory documented at the time was an intermittent failure in the anonymous Steam login used for the version check. That theory was wrong.

BE WARNED, here is the real cause. The automation's own SSH key had never been authorized on the target VM. Every cron run failed on a plain authentication error before the script reached the update-check step at all, because it was wrapped in a fail-fast mode with no partial execution. Once the key was authorized, the identical version-check command succeeded cleanly and instantly, every time, with no flakiness. By the time this was caught the server was several days stale on a real build. Silently, with a "working" cron job logging nothing unusual every night.

The lesson generalizes. When a script that is supposed to fail loudly on any error instead "just does not seem to do anything", check the authentication and connectivity layer directly before you theorize about the application above it. A permission error and an application-level failure look identical from a distance when the script does not distinguish them. Ruling out "can this even connect" is much faster than debugging the wrong layer for months.

Warn players before a restart, without building six different notification systems

Check directly whether a given game server can broadcast a warning to connected players before an update-triggered restart. Do not assume. It genuinely varies. Some expose a real in-game console or broadcast command. Several common ones expose no console or RCON interface at all, so there is no way to warn players in-game. Full stop.

Do not build a different notification mechanism per game, and do not silently skip the ones with no console access. One shared voice-chat server already sitting at the centre of the community works as a single broadcast channel for every game at once. A short script posts a warning to it over its own HTTP query API a few minutes before any update restarts anything. The same script, reused across every server in the fleet, whatever that particular game can or cannot do on its own.

Verify with a real forced update, not just a clean dry run

Every service in this fleet was confirmed with a live manual trigger before it was trusted to run unattended. "Confirmed" meant different things depending on whether the server had a real pending update at the time. A dry run that correctly detects "already current" and exits cleanly proves the check step works. It proves nothing about apply or health-check, since neither ran. Both real bugs above would have passed a dry-run-only verification with a straight face. The silently skipped restart and the SSH key that was never authorized. Catching them took forcing a real update through the full pipeline at least once per service, and watching every step execute. Point the "installed version" check at an artificially old value and you can force that safely. A clean exit code is not proof the whole thing worked end to end.