Virtualization / Automation

Debug a VM Autoscaler Stuck in an Infinite Rebuild Loop

An autoscaling platform that clones a fresh VM, waits for it to check in, then destroys and retries if it doesn't — is genuinely convenient, right up until the clone can never succeed and the platform just keeps trying forever. Here's how to catch the real cause before the evidence gets deleted, and a separate but related lesson about stopping that same churn from starving another VM's disk.

Recognize the pattern: exact intervals, identical errors

Autoscalers that provision on demand normally show some organic variation — a scale-up here, a scale-down there, timed around actual load. A stuck loop looks different: the same four steps (stop → destroy → clone → start) repeating at an almost perfectly fixed interval, with the identical error message every single time. That regularity is the tell — it means the same timeout is firing on every attempt, not that load genuinely keeps changing.

Don't dismiss it as "normal churn" just because you've seen benign autoscaling activity from the same system before. A real case: an autoscaler had previously shown legitimate, expected VM churn as load changed. When it later got stuck destroying and recreating the exact same VM every 10-11 minutes for over 90 minutes straight, it was easy to assume it was more of the same — it wasn't. The fixed interval and identical failure on every attempt were the giveaway that this was a real, unrecoverable bug, not normal scaling behavior.

Catch the evidence before it gets destroyed

The hardest part of debugging this kind of loop is that the orchestrator deletes the failing VM as part of its own retry logic — by the time you notice the pattern, the evidence is usually already gone. The fix is to watch for the next clone and inspect it live, before its timeout expires:

Get the new instance's IP as soon as it's cloned (a hypervisor's guest-agent query works even without SSH access to an ephemeral VM you don't have keys for).
Check what's actually running inside it — process list, container list, and any install/provisioning log the platform's own bootstrap script writes.
If there's a per-instance provisioning log, read its last line specifically — a script that hangs partway through will have its last real output sitting right at the point it got stuck.

The actual cause: an unattended script hit an interactive prompt

In this case, the provisioning script's last line of output was the full text of a software license agreement, cut off mid-prompt. The install script it called had an interactive read -p "...accept? (y/n)" step with no default and no way to answer it non-interactively — and the automatically-generated startup script calling it never passed the install tool's own "accept license" flag. Every single fresh clone hung there forever: the actual application never started, so the orchestrator's health check always timed out and destroyed the VM before trying again.

Check for a duplicate, broken command block too. The stored startup script in this case turned out to have the install command written twice — the platform's own admin panel had somehow ended up with a real, correct invocation followed by a second, clearly-broken one (wrong target hostname, a literal placeholder token string never filled in). Neither one would have worked without the missing "accept license" flag, but the duplicate block was a separate, additional bug worth cleaning up on its own.

Fix: add the install tool's non-interactive license-acceptance flag to the stored provisioning script, and remove any duplicate/broken command blocks while you're in there. Verify by watching the next clone specifically — the instance already in flight when you find the bug is still running the old broken script (most platforms bake the script in at clone time, not read it live), so it isn't a valid test of your fix.

The orchestrator may cache its own config in memory. Fixing the stored value alone wasn't enough here — the manager process had been running for days and kept using the old value until it was restarted. If a config change doesn't seem to take effect on the next provisioning attempt, check whether the service reads it live or only at its own startup.

Separately: stop the churn from starving another VM's disk

Once the loop itself is fixed, a second, related problem can remain: if the template being cloned and the destination for every new clone both live on the same physical storage as another active VM, every legitimate clone still generates a real, sizeable read-and-write burst competing with that VM's own disk activity. This is exactly the same class of problem as an NFS bind-mount going unbacked-up — two workloads unknowingly sharing one physical resource.

The fix doesn't have to be moving everything off the shared pool. Reads and writes have very different costs on a mirrored disk pair: a write must complete on every member of the mirror, while a read can be served by whichever member is free. If the platform lets you separate where the source template lives from where new clones' disks actually land, you can:

Leave the template's reads on the original shared pool — comparatively cheap, and mirrors can spread read load across members.
Point new clone target disks at a separate, otherwise-idle drive instead — removing the write half of the contention entirely.

This resolved real, measured contention: a live clone cycle was shown (via the hypervisor's own storage metrics) to spike write bandwidth on the shared pool to more than ten times its normal baseline, at the exact moment another VM on that same pool needed to write its own data.

Verifying it actually worked

Don't trust "the loop stopped" alone — force a fresh test rather than waiting and hoping:

Manually destroy a currently-healthy instance to trigger the platform's own normal provisioning path for a replacement, rather than waiting for the next natural scaling event.
Confirm the new instance actually completes its full startup (not just that a VM exists) and starts reporting in on schedule.
Check the new instance's actual disk location against your intended storage target, not just that the platform's config field shows the value you expect — a cached or stale config can silently keep using the old location even after you've changed it.