A filter rule sat there for six days, present, enabled and correct. It worked for seven objects and did nothing at all for two others whose data was byte-for-byte identical. No error, no log line, no difference in the config. The cause was not the rule. It was that the rule's result had been cached before the rule existed, and nothing would ever invalidate it.
The setup was a sync tool pushing an inventory system's hosts into a monitoring system. Retired hosts were supposed to be filtered out by a rule matching a status label. Nine hosts carried the identical label:
labels.status = [["value","offline"],["label","Offline"]]Seven of them were correctly ignored. Two kept reappearing in the monitoring system after every manual deletion. The only difference lived in the tool's own cache:
| Host | Cached filter result |
|---|---|
| the seven that worked | {"ignore_host": true} |
| the two that did not | {} |
Rules are evaluated once per object and the result stored. The invalidation logic is the interesting part, and it is a completely reasonable thing to write:
if self.labels.get(key) != value:
updates.append(...)
...
self.labels = label_dict
self.cache = {} # only reached when a label actually changedThe cache clears when an input changes. That is efficient and correct for the normal case.
Now consider an object that was already in the target state before the rule was written. Its label was set to
offline months ago. It has a cached "no match" from back when no rule existed to match it. And
nothing will ever change its labels again, because it is retired and static.
Worth being precise, because it changes the fix. Any object that enters the target state from now on has its input change, which clears its cache, which lets the rule apply on the next run. The system is not broken going forward.
What you have is a fixed, finite backlog of objects that were already in that state on the day the rule was created. Sweep it once and the problem is genuinely over.
This is where the first attempt went wrong. An earlier session had cleared the cache for the three objects it was actively chasing. Two more were in the same state, were not in that set, and silently kept their stale result for another six days.
Write the query as "everything matching the condition whose cached result disagrees", never as a list of names you are currently annoyed about:
# the shape that matters: condition matches, cached outcome does not
db.host.find({
"labels.status": "offline",
"cache.checkmk_filter.ignore_host": {"$ne": true}
})That returns the complete set by construction. Clear the cache on all of them in one pass.
A log line saying a rule matched is weaker evidence than the outcome you actually want. Delete the affected objects from the destination, run the sync fresh, and confirm nothing recreates them:
# delete downstream, run the real export, then confirm they stayed gone
cmk -l | grep -E 'host-a|host-b' # expect no outputWhile in there, look for cache entries whose source object no longer exists at all. This deployment had two of those, left behind by dynamically created hosts that were deleted upstream and never purged locally. They were independently trying to recreate themselves, which is the same class of problem from the other direction.
Anything that caches a computed result keyed on an object has this failure mode, and the question to ask is always the same: what invalidates this, and does that event ever happen for an object that is already finished?
If the answer is "an input change invalidates it", then introducing a new rule is a change the cache cannot see, and you owe it a manual sweep.