💀SON OF A TECH
← Back to Wiki
Deployment / Debugging

Your Deploy Pipeline May Be Publishing to a Host Nobody Visits

A site went three days without publishing anything, while every deploy printed its success message and exited zero. Nothing was broken. The bare repo received each push, the hook ran, and it wrote correct current files to disk. It wrote them to a machine that no longer served the domain. The live page's last-modified header had been frozen since the day the site moved hosts.

Share on X

Why every component can be healthy and the result still wrong

Push-to-deploy is a chain, and each link only knows about the next one:

  1. You push to a bare repo. It accepts the objects. Success.
  2. Its post-receive hook fires and checks the work tree out. Success.
  3. Files land in a docroot, correct and current. Success.
  4. A web server on that host serves that docroot. Success.
  5. DNS for the domain points at a different host, which has its own copy and no sync.

Nothing in steps one to four has any way to detect step five. The exit code is telling you the truth about a question you did not mean to ask.

BE WARNED: a host migration is exactly when this appears, and it appears late. The move itself is done carefully and the site works afterwards, because the content is already there. The gap only shows up on the next content change, which might be days or weeks later, by which time nobody connects the two events. Whoever moves the DNS and whoever pushes the next commit are often the same person on different days.

The check: ask the public URL, not the pipeline

One header answers it. Push something, then:

curl -sI https://example.com/ | grep -i last-modified

A timestamp older than your push means you published nothing, whatever the deploy said. This is the whole check, and it belongs at the end of every deploy you care about.

To find out where the content is actually coming from, ask each candidate host directly with an explicit Host header, which bypasses DNS entirely:

# does this specific host serve current content for that name
curl -sI --resolve example.com:443:<host-ip> https://example.com/ | grep -i last-modified

# and where does the world think the domain lives
dig +short example.com

When one host returns your new content and the public URL does not, you have found the split.

Diffing the two trees needs a stable sort

To compare what is live against what is in the repo, list both and diff. There is a trap in the obvious version:

ssh live-host  'cd /srv/site && find . -type f' | LC_ALL=C sort > /tmp/live
ssh build-host 'cd /srv/site && find . -type f' | LC_ALL=C sort > /tmp/repo
LC_ALL=C comm -23 /tmp/live /tmp/repo   # live but not in the repo

comm requires input sorted in the same collation it compares with. Sort in one locale and compare in another and it reports nonsense, typically listing the same files as unique to both sides, which reads like catastrophic drift and is an artefact. Force LC_ALL=C on the sorts and on comm itself.

Guard the sync against deleting things it does not own

Once you do wire up a real sync, rsync --delete is how the destination stops drifting. It is also how you lose files nobody realised were only ever on the live host.

The diff above found a Search Console verification file that existed on the live host and in the serving docroot, but was in version control nowhere. It survived by accident. A clean re-checkout would have removed it from the source, and the next --delete sync would have silently un-verified the domain.

# anything listed here is live content a --delete would drop
git status --porcelain --untracked-files=all | grep '^??'

Run that in the serving docroot before trusting any sync. Then commit what should be tracked, and explicitly exclude what the destination legitimately owns. Generated status files and anything written by a process on the live host belong in the exclude list, not in the repo.

Guard against a truncated source too. Have the sync refuse to run if the source tree has implausibly few files. A checkout that failed halfway, combined with --delete, turns a broken build into an empty website. A one-line sanity check costs nothing and removes the worst outcome.

Make the deploy prove itself

The durable fix is that the pipeline verifies its own claim. Put the public-URL check in the hook, after the sync, and fail loudly when it does not move:

before=$(curl -sI https://example.com/ | awk -F': ' '/[Ll]ast-[Mm]odified/{print $2}')
rsync -a --delete ...
sleep 2
after=$(curl -sI https://example.com/ | awk -F': ' '/[Ll]ast-[Mm]odified/{print $2}')
[ "$before" = "$after" ] && echo "WARNING: live site did not change" >&2

When this isn't your problem