Proxmox Backup Server Fingerprint Mismatch After a Certificate Renewal

A backup server's certificate renewed itself at two in the morning, exactly as it was designed to. Within seconds, every backup client on the fleet stopped working — the whole-fleet job covering around forty virtual machines and containers, plus five independent per-host jobs on separate machines. Nothing alerted. It was found two days later by accident, because a release checklist happened to require taking a backup by hand.

Why backup clients pin a fingerprint at all

Proxmox Backup Server — and plenty of other tools that talk to a server over TLS on a private network — asks clients to record the SHA-256 fingerprint of the server's certificate. Once recorded, the client refuses to talk to anything presenting a different certificate.

This is a good idea, and it exists for a real reason. A homelab backup server usually has a certificate from a private CA, or a self-signed one. Normal certificate validation depends on a chain of trust the client doesn't have, so the options are "verify nothing" or "verify this exact certificate". Pinning is the second one. It means a machine on your network that manages to answer on the backup server's address can't quietly collect your backups, because it can't present the certificate you pinned.

The trade is that the pin is a copy. Every client has its own. And a certificate that renews is, by definition, a different certificate.

What it looks like when it happens

The client-side error is clear enough on its own:

WARNING: certificate fingerprint does not match expected fingerprint!
expected:    b1:8b:9d:46:19:9c:94:b2:...
certificate validation failed - Certificate fingerprint was not confirmed.
Error: error trying to connect: error:0A000086:SSL routines:
  tls_post_process_server_certificate:certificate verify failed

On the hypervisor, where the backup server is configured as a storage target rather than as a client script, the same problem wears a different hat and is much easier to misread:

could not activate storage 'pbs': pbs: error fetching datastores -
  fingerprint 'AA:C7:72:...' not verified, abort!

That one reads like a storage or network fault. It isn't. It's the same pin, in a different file.

The first trap: the machines that look fine

The renewal happened at 02:11. Two hosts had backup timers that fired at 23:41 and 23:46 the night before — before the renewal. Their most recent run was a success. Their last-backup timestamp was hours old, not days. By every check that asks "did the last backup work?", they were healthy.

They were not healthy. They were scheduled to fail on their next run, and would have, silently, that night. If you go looking after an event like this, check the stored pin on every client, not just the result of the last run. The last run tells you about the past; the pin tells you about tonight.

The second trap: your logs are in different timezones

The backup server logged in local time. The cloud hosts logged in UTC. Lining those timestamps up naively made it look like the certificate had rotated several times across two days, which pointed at a completely wrong theory — that something was over-renewing and the real bug was in the renewal schedule.

It was one renewal. The hosts that appeared to "succeed after the rotation" had simply run before it, in a timezone that made them look later. Before building a theory out of a timeline, confirm every timestamp in it is in the same zone. date -u on each box costs nothing.

Resolving it

1. Get the new fingerprint from the server itself

This is the important step, and it is easy to get subtly wrong. The obvious move is to ask the network what fingerprint it's seeing now and paste that in. Don't. The entire purpose of the pin is to protect you in the case where what's answering on that address isn't your server. Taking the new value from whatever answers defeats the check completely.

Read it from the machine, over a channel you already trust:

# on the backup server itself
proxmox-backup-manager cert info | grep -i fingerprint

# or straight from the certificate file
openssl x509 -in /etc/proxmox-backup/proxy.pem -noout -fingerprint -sha256 -dates

Then, optionally, compare it against what a client sees over the network. If those two agree, you have a renewal. If they disagree, you have something much more interesting, and you should stop and find out why.

2. Find every copy of the old pin

They will not all be in the same kind of file. In one fleet the pin lived in five places with three different shapes:

• The hypervisor's storage definition — a fingerprint line in /etc/pve/storage.cfg. This is the one that covers every guest, so it's the one that matters most and the one whose error message looks least like a certificate problem.
• Per-host environment files for standalone backup scripts, typically PBS_FINGERPRINT=....
• A script that exported it instead: export PBS_FINGERPRINT='...'. This one is worth calling out, because a search-and-replace anchored to the start of the line (s/^PBS_FINGERPRINT=.*/.../) silently matches nothing here, reports success, and leaves the job broken. Match on the old fingerprint value itself rather than the variable name, and always print the line back afterwards to confirm the change landed.

A blunt way to find them all, before you start editing:

grep -rn "b1:8b:9d" /etc /root /usr/local/bin 2>/dev/null

3. Back up the config, change it, then prove it works

Copy each file to a .bak- before editing. Then — and this is the part people skip — run each job for real and read the output. A pin that's been corrected still isn't a backup. On one host the corrected job still failed, because the anchored-sed problem above meant nothing had actually changed; the only reason that was caught is that the job was run and the log read rather than assumed.

systemctl start <your-backup>.service
journalctl -u <your-backup>.service --since "5 minutes ago" | grep -E "Duration|Error|Finished"

On the hypervisor, confirm the storage comes back before waiting on the scheduled job:

pvesm status --storage pbs

Preventing it

The real fix: alert on the outcome, not the exit code

The certificate is not the interesting part of this story. Certificates renew; that's their job. The interesting part is that a scheduled job failed every night and nothing said anything. A systemd timer that runs a service which exits non-zero is completely invisible unless something is watching for it.

The strongest version of this check doesn't watch the job at all — it watches the result. Monitor the age of the newest snapshot for each backup group and alert when it exceeds the interval it's supposed to run at. That catches this failure, and it also catches the failures an exit-code check can't see:

• A job that stopped being scheduled at all, because a timer got disabled or a host was rebuilt.
• A job that "succeeds" while backing up nothing, because its source path moved.
• A new machine nobody added to the backup job in the first place.

The datastore already knows all of this. Snapshot directories are timestamped, so the check is little more than "newest entry per group, compared against now".

The cheap version, if you want something today

Add an OnFailure= drop-in to each backup unit, pointing at a service that emails you. It's a few lines per host and it converts a silent failure into a loud one:

# /etc/systemd/system/your-backup.service.d/onfailure.conf
[Unit]
OnFailure=backup-alert@%n.service

It's weaker than the snapshot-age check — it can't tell you about a job that never ran — but it would have caught this one on night one instead of day three.

Remove the pin problem entirely

If your backup server's certificate comes from your own internal CA, the durable fix is to stop pinning fingerprints and instead install the CA's root certificate in each client's trust store. You keep the security property you wanted — clients still refuse to talk to anything that isn't presenting a certificate your CA issued — but renewals stop being events, because the new certificate is signed by the same root the client already trusts.

If you must keep pinning, then treat the fingerprint as configuration that has to be distributed, not as something typed once per host. A renewal hook on the backup server that pushes the new value out, or the pin managed by whatever config tooling you already run, both work. What doesn't work is five hand-edited copies and a good memory.

Audit the coverage while you're in there

Fixing this meant listing every backup group and its newest snapshot, which immediately surfaced something unrelated: four machines existed on the hypervisor that were in no backup job at all. Not failing — simply never included, because the job's list of IDs was written once and machines created later were never added to it.

That's worth doing as a deliberate exercise, separately from any incident. Compare the list of guests that exist against the list your backup job actually names, and expect the two to have drifted:

# what exists
{ qm list | awk 'NR>1{print $1}'; pct list | awk 'NR>1{print $1}'; } | sort -n

# what the job covers
grep vmid /etc/pve/jobs.cfg

A backup job with a hand-maintained ID list is a backup job that quietly stops covering your newest machines. If your tooling supports "all guests except X", that failure mode disappears.