Running an inventory export by hand deleted two live hosts from monitoring. One was a public game server with players on it. Both were flagged offline in the inventory system. Neither was offline. Nothing alerted, because a host that has been deleted cannot go critical.
The setup is ordinary. An inventory system holds every host. A sync tool exports those hosts into the monitoring system. A filter drops retired hosts so they stop reappearing after you delete them.
That filter was added months earlier. It worked.
The export also runs a cleanup step. Cleanup deletes any monitored host that is missing from the export.
Read those two behaviours together.
The filter removes a host from the export. Cleanup then deletes it from monitoring.
So an offline flag does not only stop a retired host coming back. It removes the monitoring for anything that carries it.
Both deleted hosts were flagged offline. Neither was.
game-server status: running onboot: 1 port 22 reachable
internal-site status: runningThe game server was public and serving players while its monitoring was being removed.
Nobody set those flags to cause harm. Someone marked them offline during an old cleanup pass, long before the filter existed. The field was never corrected. It sat there as a harmless label until the day it became a delete switch.
That is the real lesson. The flag did not change. What changed was what reads it.
This is the part that makes it dangerous.
Both hosts returned 404 from the monitoring config API. Gone.
Both were still live in the running core. Still checked. Still green on the dashboard.
The reason is staging. This monitoring system queues configuration changes and applies them on activation. There were 312 unactivated changes sitting in that queue. The two deletions were in there with them.
Nothing breaks until somebody activates. Then the checks stop, quietly, and the dashboard has fewer hosts on it than it did yesterday.
Deleting monitoring produces no alert anywhere. There is no state transition to notify on. The host does not go critical. It stops existing.
Do this before you activate anything. The window is the only reason a clean repair is possible.
The config API has already forgotten the host. The running core has not. Query the core and you get the real pre-deletion values instead of your best reconstruction.
lq "GET hosts
Filter: name = game-server
Columns: name address custom_variables"That returns the address and the full tag set:
game-server 10.0.0.70 cmk-agent ip-v4-only lan no-snmp piggyback prod tcp
internal-site 10.0.0.50 cmk-agent ip-v4-only lan no-snmp piggyback prod tcpRecreate each host through the REST API with those exact attributes. Not similar ones. The same ones. A host recreated with a guessed tag set looks fine and monitors the wrong things.
Then fix the inventory status that caused it. Set both back to active, or the next export does it all again.
Then activate, and confirm the states came back:
internal-site;0 game-server;0One repair is not a fix.
Ask which other objects carry the same flag. In this case ten more were flagged offline. None of them currently had a monitored host, so none could be deleted by the same path.
That is luck. It is not a design. Check it rather than assume it, and check it again after anyone adds a host.
Any time one system's filter feeds another system's delete, you have this.
Ask two questions of any sync you run.
Does the export have a cleanup, prune or reconcile step? Read the documentation. Do not assume that "export" only writes.
What is the blast radius if the source data is wrong? If the honest answer is "it deletes production monitoring", put a dry run in front of it and read the output.