A Monitoring Check That Can Never Pass Is Worse Than No Check

A host sat CRIT in the monitoring dashboard. Not intermittently, permanently, and it had done for as long as anyone could remember. The check was a plain ICMP ping across a VLAN boundary that had been deliberately isolated from the monitoring server's network. It could never have succeeded. It was not measuring anything. It was training everyone who looked at that dashboard to ignore the colour red, and underneath it two genuine failures had been sitting unnoticed.

Why this is a security problem, not a tidiness one

The instinct is to file a permanently-red check under cosmetic and get to it eventually. That undersells it.

A monitoring system's value is entirely in the contrast between normal and abnormal. Every check that is red for a reason nobody intends to fix erodes that contrast. People stop reading the dashboard and start remembering which reds are "the usual ones", and that memory is the thing that fails. It fails silently, it fails under pressure, and it fails for the new person who has no idea which reds are load-bearing.

Point the host-check at something that can actually answer

ICMP is the default host-check almost everywhere, and it is a poor fit the moment your network has deliberate boundaries in it. If the monitoring server reaches a host through an agent, a proxy or a jump host, then the agent is what proves liveness. Ping proves that a path you intentionally closed is still closed.

In Checkmk that is a host_check_commands rule scoped to the host, using the agent's own service instead of ICMP:

# liveness comes from the agent check, not from ping
host_check_commands = [
    (("service", "Check_MK"), [], ["printer-host"], {}),
]

The equivalent exists everywhere. Nagios and Icinga let you set check_command per host, and Zabbix has agent availability as a first-class concept. The principle is what matters: the host-check should test the path you actually use to reach the host.

Ask whether the check is possible before you debug it

The useful question when something is permanently red is not "why is this failing". It is "what would have to be true for this to pass, and do I want that to be true?"

Here the answer was: VLAN 1 would need to reach VLAN 9 by ICMP. That isolation was built on purpose, for good reasons, and rerouting monitoring through a bridge host was the whole point of the design. Making the check pass would have meant undoing the security control it was accidentally testing.

Once you phrase it that way the fix is obvious and the alternative is unthinkable. Phrase it as "why is ping failing" and you will spend an hour on firewall rules trying to break your own segmentation.

Go looking for the rest of them

Impossible checks cluster around exactly the places where your network got more sophisticated. Isolated VLANs, hosts behind NAT, cloud instances with no inbound ICMP, anything monitored through a proxy or a tunnel. Sweep for checks that have been in the same non-OK state for longer than any real incident would last:

# anything CRIT or UNKNOWN for more than a week is a candidate
cmk --list-hosts | while read h; do
  cmk -D "$h" | grep -q 'ping' && echo "$h uses ICMP host-check"
done

For each one, decide deliberately: fix it, repoint it at something answerable, or remove it. Those are the only three acceptable outcomes. "Leave it red and remember" is not on the list.

Deleting a check is a legitimate fix

If a check cannot pass and you do not want it to pass, delete it. That is not giving up, it is being honest about what you are monitoring. A dashboard with nine green checks and no entry for a thing you deliberately cannot see is more truthful than one with nine green and one permanent red that everybody has learned to scroll past.

Write down why it was removed, so the next person does not helpfully add it back.

When this isn't your problem

• The red is real. Obviously. Fix the host.
• It is genuinely intermittent. A flapping check is a different problem with different fixes, usually thresholds or retries rather than a wrong check type.
• You already alert only on transitions. Then a permanent CRIT is quieter than described here. It still suppresses the services beneath it, so it is still worth fixing.