Automation / Monitoring

Sync NetBox to Checkmk Automatically with CMDBsyncer

Manually adding every new host or VM to your monitoring system gets old fast. This is a pipeline that keeps a network controller, a hypervisor, an inventory system and a monitoring stack in sync automatically. It also covers the subtle failure modes that break each leg without anyone noticing.

The pipeline

Network Controller --[sync plugin]--> NetBox --[sync tool]--> Monitoring System
Hypervisor         --[sync plugin]--/

Your network controller already knows about every device on your network. A UniFi setup, in this case. Your hypervisor already knows about every VM and container. Proxmox, here. NetBox becomes the single source of truth for both, synced automatically from each side. A sync tool (CMDBsyncer) then pushes NetBox's combined device and VM list into your monitoring system (Checkmk). New devices and new VMs show up monitored without anyone touching any of the three systems by hand.

NetBox device inventory list showing device name, status, role, manufacturer, and type columns — **NetBox → Devices.** Source of truth for every physical device, synced in from the network controller. IP addresses redacted here.

Client devices: inventory only, never monitored

It is tempting to sync every device your network controller sees straight into active monitoring. Phones, laptops, every other ephemeral client. Do not. Alerting on up/down state for devices that are supposed to come and go recreates the notification storm monitoring exists to prevent. A phone leaving the house is not an incident. A laptop going to sleep is not an incident. Sync client devices into your inventory system for asset visibility and stop there. Never let them become monitored hosts.

The silent failure mode: all-or-nothing batch syncs

BE WARNED: the sync tool's "export new hosts to monitoring" step bulk-creates every pending host in a single API request. One invalid device name silently aborted the entire batch, every sync cycle, for an unknown period. The name was a switch with spaces in it, which the monitoring system's hostname format does not allow. Six genuine hosts had been missing from monitoring since each was built. Not a one-off miss. A standing gap that looked fine, because the sync job itself reported success.

Lesson: if a new host does not appear in your monitoring system after a sync that reports success, check for other pending hosts with invalid names first. One bad entry in a batch export blocks every other host queued alongside it, and the sync job will not necessarily surface that as an error.

A second layer of the same bug: stale cached identity

Fixing the invalid hostname at the source may not be enough. Renaming the device is only half of it. If your sync tool tracks hosts by an internal database keyed on the hostname string rather than a stable device ID, a rename looks like a brand-new device. The old, invalid-named record sits orphaned in its internal state, still going into every export batch. Check the sync tool's own storage for stale duplicate records, not just the source system, before you assume a rename fixed anything.

An even subtler variant. If the field you are fixing is itself populated by an upstream integration, a fix made downstream gets silently reverted on the very next sync cycle. NetBox re-importing device names from the network controller every 15 minutes, for example. Before you treat any field as the thing to fix, check whether it is sourced further upstream and will just overwrite you. The durable fix has to happen at the real source.

Watch for missing required attributes on newly synced hosts

Even after a bulk-create succeeds, newly synced hosts can fail service discovery. Your monitoring system may expect an explicit IP address attribute, with no internal DNS to resolve the hostname otherwise. Compare a working host's configuration against a newly failing one. A missing attribute like this is easy to spot once you know to look, and easy to miss if you assume the sync "worked".

Checkmk All Hosts view showing host state and OK/warning/unknown/critical service counts per host — **Checkmk → Monitor → All hosts.** The other end of the pipeline. Hosts landing automatically, with live service state per host. One redacted row was showing a raw IP instead of a hostname.

Extending the pipeline to your hypervisor

Add a plugin that syncs your hypervisor's VMs and containers into the same inventory system, rather than doing a one-time manual import. That closes a gap which is easy to overlook. Manually imported inventory records go stale the moment anything changes on the hypervisor. A live sync plugin keeps that from drifting. In this case one that talks to Proxmox's API and writes back into NetBox. Getting the first sync to complete surfaced three separate bugs, all of them instructive.

BE WARNED: localhost means "myself", not "my neighbor", even for two containers on the same physical host. If your inventory system and your sync plugin's backend run as two separate containers that are not on the same Docker network or Compose project, pointing either at the other's localhost resolves to itself. It then fails in a way that looks like a networking or auth problem, not an addressing mistake. Two sibling containers that are not explicitly networked together have to address each other by the host's real IP, in both directions. Check both sides. Fixing one leaves the other broken.

BE WARNED: regenerating an API token can silently drop the permission grant tied to the old token instance. If a sync plugin's stored credential is unrecoverable and needs regenerating, do not assume the new token inherits the old one's permissions. On at least one hypervisor's token system, removing and re-adding a token wipes its role and ACL assignment. The new token then fails with a permissions error that looks unrelated to the rotation you just did. Re-grant the role explicitly after any token regeneration, and verify it took before you move on.

A stricter downstream consumer can expose inconsistent data your inventory system let through earlier. A device record can end up with a "primary IP" that is not assigned to any network interface on that device. Most inventory tools' UI will not let you create that state through the normal form, but a scripted or API-driven import slips past the validation. It sits there looking fine until a stricter integration hits the inconsistency and refuses to proceed. A hypervisor sync plugin building dependent objects against that same device, for one. Create the missing interface and assign the IP to it properly. Do not relax the stricter tool's validation to match the looser one.

Don't trust your monitoring system's "success" response either

BE WARNED: an API call that reports "activation complete" does not guarantee anything was applied. On at least one monitoring system, the REST endpoint for activating pending configuration changes returned a clean "complete" status while a real backlog sat unapplied underneath it. In one case over 30 changes spanning multiple days. Newly discovered services for a host look like they should be there and just are not, with nothing in the API response hinting at a problem.

Fix: check your monitoring system's own "pending changes" list before you trust an activation response. Better, if the CLI or admin tooling has a lower-level reload command, run that directly instead of going through the API. Most do, since it is what the API calls under the hood. On one instance that cleared an entire multi-day backlog across more than a dozen hosts in a single run. The REST API had never applied any of it, despite reporting success every time.