PKI / Networking

When Let's Encrypt Can't Reach You Any More, a Private CA Is the Right Answer

A site moved to a new host. One piece stayed behind, and the new front end proxies a single path back to it. That backend still had a Let's Encrypt certificate, and it could never renew again. HTTP-01 and TLS-ALPN-01 both require the hostname to resolve at the machine being validated, and it now resolves somewhere else entirely. There was a hard expiry date on the calendar and a contact form that would break on it. The fix was not a cleverer ACME challenge.

Why all three challenge types were dead

Work through them, because the answer is structural rather than a configuration problem:

HTTP-01 serves a token at /.well-known/acme-challenge/ on the hostname being validated. That hostname now resolves to the new front end, so the token would be requested from a machine that does not have it.
TLS-ALPN-01 has the same problem one layer down. The validator connects to whatever the name resolves to, which is not the host holding the private key.
DNS-01 is the only one that does not care where the name points. It needs an API credential for the zone, scoped and stored on the backend, which is a new standing secret on a machine that did not need one before.

DNS-01 would have worked. That is worth saying plainly, because it is the answer most people reach for. It also means minting a token, storing it, rotating it, and accepting that a renewal now depends on a third-party API being up and a credential still being valid.

Ask what the certificate is actually for

Here is the question that changes the answer. Who is the audience for this certificate?

Not a browser. Not the public. This certificate exists so that one server you own can verify another server you own before proxying a request to it. It is a machine-to-machine hop across your own infrastructure, and no human ever sees it.

A public CA exists to vouch for strangers to strangers. There are no strangers in this hop. Using one here means both ends depend on an external service, a public trust store, and a challenge mechanism that only works while DNS happens to be arranged a particular way. Every one of those is a dependency bought for no benefit.

An internal CA removes the dependency rather than trading it. That is the actual argument. DNS-01 swaps "needs the hostname to point here" for "needs a cloud API token to keep working". A private CA swaps it for nothing external at all. If you do not already run one, the step-ca guide covers standing one up.

It ends up stricter, not looser

This surprises people, so it is worth being precise. Before the change, the front end verified the backend's certificate against the public trust store, meaning any publicly trusted CA could have issued a certificate for that name and been accepted.

After the change it trusts exactly one CA, yours:

backend.example.com {
	tls_trusted_ca_certs /etc/ssl/internal-ca-root.pem
}

That is a smaller trust set than a public CA gives you, not a bigger one. "Self-signed is less secure" is a reasonable instinct for a public website and exactly backwards for a hop between two machines you control.

Cut over with no window, using a combined bundle

Do not swap the certificate and the trust setting at the same time. Widen trust first, move the certificate, then narrow trust. Nothing is ever untrusted at any point:

Trust both. Point the front end at a bundle containing your internal CA root and the system's public roots. The old public certificate still validates, and a new internal one would too.
Issue and install the internal certificate on the backend, then reload it. Verify the proxied path still returns 200.
Tighten the bundle to your CA alone. Verify again.

Verify at each step rather than at the end. If step three breaks something, you want to know it was step three.

Pin the issuer explicitly, or your proxy will undo this

BE WARNED: a proxy with automatic HTTPS will quietly take the name back. Caddy manages any hostname it sees, and managing it means Let's Encrypt, which is the exact thing that cannot work here. The site block needs an explicit tls directive naming your certificate files. That line is load-bearing. Without it the automation reasserts itself on the next reload and starts failing ACME orders for a name it can never validate.

Test the renewal path by forcing it

A renewal script that has never renewed anything is a hypothesis. Delete the certificate to force the "no existing certificate, issue one" branch, then watch the whole loop run:

# force the issue path, then confirm the full chain end to end
step ca certificate backend.example.com /tmp/t.crt /tmp/t.key --force
curl -s -o /dev/null -w '%{http_code}\n' https://frontend.example.com/api/health

Check the other sites on the same proxy afterwards too. A reload touches all of them, and a renewal that quietly breaks an unrelated site is a worse outcome than the expiry you were preventing.

When this isn't your problem

A browser sees the certificate. Then you need a public CA, full stop. Internal CAs require installing a root on every client, which is fine for your own machines and unacceptable for visitors.
The hostname still resolves to the host. HTTP-01 works. Leave it alone.
You already have a scoped DNS token. DNS-01 is a perfectly good answer and less work than standing up a CA. The argument above only wins when you would be creating that dependency from scratch.