Docker / Disk

Your Auto-Updater Never Removes What It Replaces

A calendar server was nine days from filling its disk. The application data was tiny. Four gigabytes of the disk was old container images that nothing was using. The auto-updater was not broken. It was doing exactly what it was told, which is to pull the new image and never mention the old one again.

What it looked like

The disk check had a trend on it, which is the only reason this was caught before it mattered.

/dev/mapper/pve-vm--211--disk--0   7.8G  5.9G  1.5G  80% /
trend per 1 day: +167 MiB (+2.10%)
Time left until disk full: 9 days 2 hours

The first instinct is to go looking at the application. Wrong instinct. Nothing about a calendar grows at 167 MiB a day.

Ask the container runtime what it is holding, with command docker system df

Images    26 total    1 active    4.265GB    RECLAIMABLE 4.089GB (95%)

Twenty-six images. One of them in use. Ninety-five percent of that space reclaimable.

The active image had been pulled 20 hours earlier, so the updater was working. It pulls, it restarts the service, it leaves the previous image on disk. Every single time.

It is a design gap, not a quirk of one service

This is the part worth taking away, because fixing one host fixes nothing.

Check whether anything in your update configuration prunes at all:

grep -l 'prune' /etc/backup-update/*.conf

No matches, across 12 configurations.

So every host doing automatic container updates was accumulating. They were just at different points on the same curve, and the one with the smallest disk hit the wall first.

Host	Images	Reclaimable	Disk used
calendar	26	4.089 GB (95%)	80%, 1.5G free
photos	7	2.026 GB	59%, 7.7G free
passwords	3	59 MB	17%
git	2	0 B	7%

Read that table as one host photographed at four different ages. The one at the top is not special. It is just further along.

BE WARNED: A HOST WITH A BIG DISK IS NOT SAFE, IT IS SLOW. The accumulation rate is set by how often you update, not by how much room you have. A generous disk buys you months of not noticing, and then the same wall on a host you have long since stopped thinking about.

Reclaim it now

Remove images no container is using, with command docker image prune -a

Confirm what you got back with command docker system df

Do that on the host that is nearly full first. Then do it everywhere, because you already know from the table above that everywhere is the same.

Be deliberate about -a. Plain docker image prune only removes dangling layers and will barely touch this. The whole problem here is tagged images that are complete, valid and simply not in use any more.

Then stop it coming back

A one-off prune is a chore you will forget. Put it in the thing that caused it.

Add a prune to the end of your update routine, after the new container is confirmed healthy. Order matters. Prune before the health check and a failed update has nothing to roll back to.

Keep one generation if you roll back by hand. Prune by age rather than everything unused:

docker image prune -a --filter "until=168h"

That keeps a week of history and still bounds the growth.

Then monitor the thing that actually predicts the outage. Disk percentage is a lagging indicator and it is flat right up until it is not. A trend line, which is what caught this with nine days to spare, is worth more than a threshold.

When this isn't your problem

You update by hand, rarely. Then you accumulate slowly enough that it never becomes an incident. Still worth one prune.
Your updater already prunes. Some do. Verify it rather than assume it, because the check is one command and the failure is a full disk.
Your images are tiny. A handful of small images will not fill anything. The hosts that get bitten run big images on small disks, which describes most appliance style containers.
The growth is not images. Confirm with docker system df before pruning. Logs and volumes fill disks too, and prune will not touch either.