Docker / Debugging

Config Changes That Silently Never Applied: Two Real Ways It Happens

You edit a config file. You reload the service. The command exits cleanly. Nothing about the running behaviour changes, and you go looking for a bug in your config. Here are two mechanisms that produce that exact experience, both hit in the same fleet, both leaving the previous configuration serving happily while every signal says you succeeded.

One: a reload that talks to an API you turned off

Caddy's caddy reload does not signal the process. It makes an HTTP request to Caddy's admin API on localhost:2019. Perfectly reasonable, until the config contains:

{
	admin off
}

Now there is no admin API to talk to, and reload can never work on that host, no matter how correct your Caddyfile is:

sudo systemctl reload caddy
# dial tcp [::1]:2019: connect: connection refused

This one at least fails loudly and leaves the previous config serving, so you get an error and no outage. That is a much better failure mode than the silent variant below. The correct action is a restart, with validation first:

caddy validate --config /etc/caddy/Caddyfile
sudo systemctl restart caddy

BE WARNED: the silent version of this exists too. On a host where the admin API is enabled, caddy reload can log "config is unchanged" and skip applying your new block, while the file on disk is definitely different and caddy validate parses it fine. Same outcome, no error at all. If a reload does not take effect, stop retrying the reload and restart the process.

Two: a single-file bind mount and an inode-replacing write

This is the nastier one, because nothing fails anywhere.

Docker bind mounts are resolved at container start. Mount a directory and the container sees whatever is in it, always. Mount a single file and the container is bound to that file's inode, not its path.

volumes:
  - /opt/caddy/Caddyfile:/etc/caddy/Caddyfile   # single file: bound to the inode

Now consider how ordinary tools write files. sed -i does not edit in place despite the name. It writes a temporary file and renames it over the target, which produces a new inode. So do mv, most editors, and any "write new file, then replace" pattern.

Your host path now points at the new inode with your changes. The container is still holding the old one. Both files exist, both are on disk, and the container will read the stale one until it is recreated. No error, ever.

# prove it: the two should be identical, and are not
md5sum /opt/caddy/Caddyfile
docker exec caddy md5sum /etc/caddy/Caddyfile

The fix is to write through the existing inode rather than replacing it:

# truncate and rewrite in place. Same inode, container sees it
cat /tmp/Caddyfile.new > /opt/caddy/Caddyfile

# or with sed, keeping the original inode
sed 's/old/new/' /opt/caddy/Caddyfile | sponge /opt/caddy/Caddyfile

Better: bind-mount the directory, not the file. Single-file bind mounts are the root of this entire class of problem. Mount /opt/caddy at /etc/caddy and every write method works, because path resolution happens per-access rather than once at container start.

The habit that catches both

Never verify a config change by looking at the file you just edited. That file is the input. Ask the running process what it is using:

sshd has sshd -T, which prints the fully resolved config including every drop-in.
A container can be asked directly: docker exec <name> cat <path>, then compare hashes with the host.
Anything behind a reload can be tested by its behaviour. Request the route you added. Look for the header you set.

The general form is that a config file, a validated config file, and the config a process is actually running are three different things, and only the third one matters.

Two more places the same shape shows up

Drop-in ordering. Your file is read, and another file read earlier already set the same key. sshd takes the first value it sees, which is why a hardening file has to sort before cloud-init's, not after.
Cached evaluation. The config applied fine and the object it should affect is carrying a stale cached result from before your rule existed. See your rule is correct, enabled, and does nothing.

When this isn't your problem

You restart rather than reload. A restart re-reads everything and recreates mounts, so both mechanisms here disappear. It costs you a moment of downtime and buys certainty.
Config is baked into the image. No bind mount, no drift, and a rebuild is the only way to change anything, which is its own kind of honest.
The change did take effect. Then stop reading and go debug the config itself, which is a much more ordinary problem.