💀SON OF A TECH
← Back to Wiki
Security

Your SSH Hardening Test Passed and the Server Still Takes Passwords

An audit across a fleet found SSH password authentication still enabled on 32 of 34 containers, plus the hypervisor itself. That was the easy part. The harder part was that the checks written to confirm the fix reported success three separate times against hosts that were not fixed. A check that cannot fail is not a check.

A sweep is not a default

The fleet had been hardened once, months earlier, and it had held for exactly as long as no new hosts were built. Every machine created afterwards came back with the distro default, and nothing noticed, because nothing was looking.

That is the shape of the problem. A one-off pass fixes the hosts that exist on the day it runs. If hardening is not part of how a host gets built, the fleet drifts back one new machine at a time, and the drift is invisible because the hosts that were fixed stay fixed.

Trap one: an AEAD cipher skips MAC negotiation entirely

The obvious way to prove a server no longer offers a weak MAC is to ask for one and watch it refuse. Try it with command ssh -o MACs=hmac-sha1 root@host true.

It connects. Not because the server still offers hmac-sha1, but because the client and server negotiated an AEAD cipher such as chacha20-poly1305 or an AES-GCM mode. AEAD ciphers authenticate the data as part of the cipher, so there is no separate MAC to negotiate, and your -o MACs= is ignored.

The test passes against a hardened server. It also passes against one that offers every weak MAC there is. It is measuring nothing.

Force a non-AEAD cipher so a MAC actually has to be chosen. Run ssh -o Ciphers=aes256-ctr -o MACs=hmac-sha1 root@host true and a hardened server answers plainly.

Unable to negotiate with 10.0.0.5 port 22: no matching MAC found.
Their offer: hmac-sha2-256,hmac-sha2-512,[email protected],[email protected]

That is the result worth trusting. It names what the server offers, so it is evidence rather than an exit code.

Trap two: an empty answer is not the same as no service

Auditing a fleet usually means asking every host what its running config is. The natural tool is sshd -T, which prints the effective configuration.

On one host it printed nothing at all. The audit script treated an empty answer as "this host has no SSH server" and moved on.

It had one. It was listening on port 22, accepting passwords, and it had been doing so the whole time. sshd -T validates the config before printing it, and validation fails when the privilege separation directory /run/sshd is missing, which happens easily enough in a container. It exits non-zero and prints nothing useful.

BE WARNED: absence of output is not evidence of absence. This is the failure mode that hides hosts from an audit rather than flagging them, which makes it worse than an error. The host disappears from the report and the total still looks complete. Confirm what is actually listening with command ss -lnt before concluding a host has no SSH server.

Trap three: a guard that fails open

Turning off password authentication on a host with no SSH key installed does not harden it. It locks you out of it. So the script that does the work should refuse to touch a host with no key.

The guard counted keys and bailed if the count was zero.

KEYS=$(grep -cE '^(ssh|ecdsa)-' /root/.ssh/authorized_keys 2>/dev/null || echo 0)
if [ "$KEYS" -eq 0 ]; then
  echo "REFUSED no root key present"
  exit 2
fi

On a host with no authorized_keys file, that refuses correctly on some systems and hardens the host anyway on others. grep -c on a missing file prints 0 and also exits non-zero. The || echo 0 then fires as well, so $KEYS becomes two lines reading 0 and 0.

The integer comparison cannot parse that. It prints an error to stderr and evaluates false. The if does not run, the script continues, and the host is hardened with no way back in.

The visible symptom is one line of noise in the output.

[: 0
0: integer expected

Easy to scroll past. It appeared directly above the word OK.

Count in a way that cannot produce a non-integer, and take the first line only.

KEYS=0
for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
  [ -f "$f" ] || continue
  n=$(grep -cE '^(ssh|ecdsa)-' "$f" 2>/dev/null | head -1)
  KEYS=$(( KEYS + ${n:-0} ))
done

Then prove the guard works. Build a throwaway host with no keys on it, run the script, and confirm it refuses and changes nothing. A guard you have never seen refuse is a guard you are assuming.

What to actually verify

Every one of these passed a check that read as green. The fix in all three cases was to test the behaviour rather than the configuration.

Then make it a build step. A drift check that walks every host on a timer and reports anything still accepting passwords is what stops the sweep being needed again in three months. Report it rather than auto-fixing it, and make it critical rather than a warning, because a warning nobody is paged for is the same as no check at all.

Share on X