Automation / Monitoring

Automate Device Inventory & Monitoring

Manually adding every new host or VM to your monitoring system gets old fast. Here's a pipeline that keeps a network controller, a hypervisor, an inventory system, and a monitoring stack in sync automatically — and several subtle failure modes that can silently break each leg without anyone noticing.

The pipeline

Network Controller --[sync plugin]--> NetBox --[sync tool]--> Monitoring System
Hypervisor         --[sync plugin]--/

The idea: your network controller (a UniFi setup, in this case) already knows about every device on your network, and your hypervisor (Proxmox, here) already knows about every VM and container. NetBox becomes the single source of truth for both, synced automatically from each side. A sync tool (CMDBsyncer, in this case) then pushes NetBox's combined device/VM list into your monitoring system (Checkmk), so new devices and new VMs just show up monitored without anyone touching any of the three systems by hand.

Client devices: inventory only, never monitored

It's tempting to sync every device your network controller sees — including phones, laptops, and other inherently ephemeral clients — straight into active monitoring. Don't. Actively alerting on up/down state for devices that are supposed to come and go constantly (a phone leaving the house, a laptop sleeping) recreates exactly the kind of notification-storm problem monitoring is supposed to prevent — legitimate, constant presence changes aren't incidents. Sync client devices into your inventory system for asset visibility, and stop there; never let them become monitored hosts.

The silent failure mode: all-or-nothing batch syncs

The real bug that bit this pipeline for weeks: the sync tool's "export new hosts to monitoring" step bulk-creates every pending host in a single API request. One invalid device name (a switch with spaces in its name — not valid in the monitoring system's hostname format) silently aborted the entire batch, every sync cycle, for an unknown period. The real consequence: six genuine hosts had been missing from monitoring since each was built — not a one-off miss, a standing gap that looked like everything was fine because the sync job itself reported success.

Lesson: if a new host doesn't appear in your monitoring system after a sync that reports as successful, check for other pending hosts with invalid names first — a single bad entry in a batch export can silently block every other host queued alongside it, and the sync job itself won't necessarily surface that as an error.

A second layer of the same bug: stale cached identity

Fixing the invalid hostname at the source (renaming the device) may not be enough on its own. If your sync tool tracks hosts by an internal database keyed on hostname string rather than a stable device ID, a rename can look like a brand-new device to the sync tool — leaving the old, invalid-named record orphaned in its internal state, still getting included in every export batch. Check for (and remove) stale duplicate records in the sync tool's own storage, not just the source system, before assuming a rename actually fixed anything.

An even subtler variant: if the field you're fixing is itself populated by an upstream integration (e.g. NetBox re-importing device names from the network controller every 15 minutes), a fix made downstream can get silently reverted on the very next sync cycle. Before treating any field as the thing to fix, check whether it's actually sourced from somewhere further upstream that will just overwrite your fix — the durable fix has to happen at the real source.

Watch for missing required attributes on newly synced hosts

Even after a bulk-create succeeds, newly synced hosts may fail service discovery if your monitoring system expects an explicit IP address attribute and there's no internal DNS to resolve the hostname otherwise. Compare a working host's configuration against a newly failing one's — a missing attribute like this is easy to spot once you know to look for it, but easy to miss if you assume the sync itself "worked."

Extending the pipeline to your hypervisor

Adding a plugin that syncs your hypervisor's VMs/containers into the same inventory system (rather than a one-time manual import) closes a gap that's easy to overlook: manually-imported inventory records go stale the moment anything changes on the hypervisor. A live sync plugin (in this case, one that talks to Proxmox's API and writes back into NetBox) keeps that from drifting — but getting the first sync to actually complete surfaced three separate, genuinely instructive bugs.

localhost means "myself," not "my neighbor," even for two containers on the same physical host. If your inventory system and your sync plugin's backend service run as two separate containers that aren't on the same Docker network/Compose project, pointing either one at the other's localhost silently resolves to itself instead — and fails in a way that looks like a networking or auth problem, not an addressing mistake. Two sibling containers that aren't explicitly networked together need to address each other by the host's real IP, never localhost, in both directions — check both sides' config, since only fixing one still leaves the other broken.

Regenerating an API token can silently drop the permission grant tied to the old token instance. If a sync plugin's stored credential is unrecoverable and needs regenerating, don't assume the new token inherits the old one's permissions — on at least one hypervisor's token system, removing and re-adding a token wipes its role/ACL assignment, and the new token then fails with a permissions error that looks unrelated to the rotation you just did. Re-grant the role explicitly after any token regeneration, and verify it actually took before moving on.

A stricter downstream consumer can expose inconsistent data your inventory system let through earlier. A device record can end up with a "primary IP" set that isn't actually assigned to any network interface on that device — most inventory tools' UI wouldn't let you create that state through the normal form, but a scripted/API-driven import can slip past that validation. It sits there looking fine until a stricter integration (like a hypervisor sync plugin building dependent objects against that same device) hits the inconsistency and refuses to proceed. The fix is to actually create the missing interface and assign the IP to it properly — not to relax the stricter tool's validation to match the looser one.

Don't trust your monitoring system's "success" response either

An API call that reports "activation complete" doesn't guarantee anything was actually applied. On at least one monitoring system, the REST endpoint for activating pending configuration changes returned a clean "complete" status while a genuine backlog of changes — in one real case, over 30 changes spanning multiple days — sat unapplied underneath it. Newly discovered services for a host can look like they "should" be there and just aren't, with nothing in the API response hinting that anything is wrong.

Fix: check your monitoring system's own "pending changes" list before trusting an activation response, or — if the CLI/admin tooling has a lower-level reload command (most do, since it's what the API itself calls under the hood) — just run that directly instead of going through the API. On one instance this cleared an entire multi-day backlog across more than a dozen hosts in a single run, none of which the REST API had ever managed to actually apply despite reporting success on every attempt.