← Back to Reviews
Self-Hosting / Retrospective

Kasm Workspaces — A Real Retrospective on Why We Walked Away

Kasm Workspaces ran in this homelab for weeks. Browser-based virtual desktops, with Proxmox auto-provisioning new agent VMs on demand. Real testing, real bugs found and fixed, real autoscaling working correctly at the end. Then it got retired anyway, and not because it broke. Here is the honest arc, including three separate root causes hiding behind what looked like the same bug recurring.

Share on X

See our full install and permission-setup guide for the mechanics of getting Kasm's autoscaler talking to Proxmox in the first place. This piece is about what happened after it was running.

The same symptom, three genuinely different causes

Kasm's autoscaler kept doing the same thing. Clone a new agent VM, boot it, wait for it to check in, never hear from it, destroy it, repeat. Forever, every 10 to 11 minutes. The temptation on a recurring bug like this is to assume same symptom, same cause, check the last fix. That assumption was wrong every time it recurred.

  1. First cause. A missing non-interactive flag. The clone script never passed an EULA acceptance flag, so every fresh VM hung forever at an interactive install prompt with no terminal to answer it. Found by catching a clone before it got destroyed and reading its install log. The log's last line was the EULA text, cut off mid-prompt.
  2. Second, a separate issue found while investigating the first. Real disk contention. The agent template and every clone's disk lived on the same physical storage as an unrelated, latency-sensitive game server. Every clone and destroy cycle did a tens-of-gigabytes read and write burst on that shared storage, measurably competing with the game server's own disk activity. Confirmed with real I/O throughput monitoring during a live cycle. Fixed without moving the whole template. Only the setting controlling where new clones' disks land changed, to a separate idle drive, while template reads stayed on the original storage. Reads are cheap to share. Writes are not.
  3. Third recurrence, same day. A stale integration token stored in Kasm's own database, confirmed by a direct API call comparison against a known-good token. Fixing this one is where a separate, real mistake happened. See below.
BE WARNED: never assume a recurring bug is the bug you already fixed. Same user-visible symptom, three unrelated root causes. Re-diagnose from scratch every time. Especially on autoscaling and orchestration systems, where "it destroys and retries" masks almost any underlying failure equally well.

A real mistake made while fixing the third cause

Writing the new, valid token straight into Kasm's Postgres database with a plain SQL UPDATE seemed like the obvious fix. It was not. That column was encrypted at the application layer, not just at rest, and writing plaintext into it broke every future read silently. The visible result was an entire configuration section of the admin UI going empty, looking exactly like deleted data. Nothing was deleted. The row was permanently unreadable without the app's own encryption key, which you cannot recover after the fact. The full writeup of this mistake, and how to avoid it, is its own guide: never write directly to an app-encrypted database column.

Recovery used a same-morning backup restore. That immediately surfaced a second, unrelated problem. The restored VM's disk filled up, crashing the app's own database, which the app's UI reported as a licensing error. Nothing about licensing was wrong. The error message was misleading about the real cause, a full disk. Remember that any time a self-hosted app reports something licensing-shaped or auth-shaped. Check basic resource health before you trust the app's explanation of itself.

What was actually good about it

By the end real autoscaling was working. Agent VMs provisioned on demand, connected correctly, and torn down cleanly when no longer needed. The core idea works once the integration points are correct. Browser-based virtual desktops with real elastic capacity instead of a fixed always-on pool. Those integration points are the Proxmox token, the clone script's flags, and storage placement. None of the bugs above were fundamental to the platform. Every one was an integration-layer issue specific to wiring it into this Proxmox setup.

Why it got retired anyway

Not because of any of the above. Everything listed here was fixed and working. The deployment was retired because Kasm got adopted at work instead, which made a second, separately maintained homelab instance redundant rather than valuable. The VMs were stopped, not destroyed. Every piece of monitoring and inventory was cleanly deregistered. A real "we could pick this back up quickly if we needed to, but we do not right now" retirement, not an abandonment out of frustration.

Verdict

Kasm Workspaces earned a positive verdict on its technical merits. The autoscaling model works. Every bug hit along the way was an integration issue with this environment, not a flaw in the product. The real takeaway from the whole arc is not about Kasm. A recurring bug with an identical symptom is no proof of an identical cause. And "the fix" for one incident can introduce a separate, subtler problem when you reach for a shortcut instead of the tool designed for the job. The shortcut here was raw SQL against a database I did not fully understand.