Self-Hosting / Virtualization

Kasm Workspaces + Proxmox Autoscale: Findings & Install Guide

Kasm Workspaces is a self-hosted containerized desktop/browser streaming platform — browser-based VDI. Pointed at Proxmox VE, it can automatically clone a template VM on demand as session load increases ("autoscale") instead of running a fixed pool of always-on agents. Getting there took several rounds of real trial and error — here's the whole install path plus every gotcha found along the way.

Proxmox-side setup

# Resource pool for Kasm-managed VMs
pveum pool add kasm-workspaces --comment "Kasm Autoscale Pool"

# Dedicated user + API token
pveum user add kasm@pve --comment "Kasm Workspaces Provisioner"
pveum passwd kasm@pve
pveum user token add kasm@pve kasm-token --privsep 0

Permissions needed are broader than they first appear. Custom privilege lists kept hitting invalid-privilege-name errors (e.g. VM.Monitor isn't a real privilege on current Proxmox) — Proxmox's built-in roles turned out simpler and more reliable than hand-rolling a custom one:

pveum aclmod /pool/kasm-workspaces -user kasm@pve -role PVEVMAdmin
pveum aclmod /pool/kasm-workspaces -user kasm@pve -role PVEDatastoreUser
pveum aclmod / -user kasm@pve -role PVEPoolAdmin

This is the minimum permission set found through live trial and error — each error below only surfaced once the previous one was fixed, at successive stages of the actual clone operation:

Error	Root cause	Fix
`No Resource Pools found matching name(<pool>)`	Template VM was never added as a member of the pool — creating the pool doesn't auto-populate it	`pvesh set /pools/<pool> -vms <template-vmid>`
`No Storage Pools found matching name(<storage>)`	Token had zero visibility into node-level storage	`pveum aclmod / -user kasm@pve -role PVEDatastoreAdmin`
`403 Forbidden ... SDN.Use`	No SDN permission granted at all	`PVESDNUser`, plus explicit grants at the zone and zone/bridge paths — root-level alone didn't fully propagate
`Error uploading startup script`	The "Startup Script Path" field expects a directory, and appends its own filename	Use a bare directory path (e.g. `/tmp/`), never a filename

Diagnostic technique used throughout: test the exact same API call the Kasm manager makes, directly via curl with the token, to isolate whether a problem is Proxmox-side permissions or a Kasm-side bug — and read the manager's real traceback from container logs (docker logs kasm_manager --since 10m), since the UI's error banner is too generic to debug from.

Kasm Workspaces Autoscale Configurations list showing a Proxmox-backed Docker Agent autoscale config, enabled — **Kasm → Admin → Infrastructure → Pools → All AutoScale Configs.** The Autoscale Configuration tying the pool above to a Proxmox provider config — this is what actually triggers clone/destroy cycles.

Template VM

Docker + qemu-guest-agent installed on the base OS.
Gotcha: qemu-guest-agent.service can show inactive (dead) if the virtio-serial device isn't present — caused by the QEMU Guest Agent option not being enabled on the VM in Proxmox. Enabling it requires a full VM restart, not just a guest reboot.

Clean the machine-id before templating, so clones don't share identical DHCP/network identity:

sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
sudo shutdown -h now

Convert to template (irreversible — becomes clone-only): qm template <vmid>.
Must be explicitly added to the resource pool — not automatic even after templating.

Startup script — the real gotchas

#!/bin/bash
systemctl enable --now docker
systemctl enable --now qemu-guest-agent

cd /tmp
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_<version>.tar.gz
tar -xf kasm_release_<version>.tar.gz

MY_IP=$(hostname -I | cut -d' ' -f1)

bash kasm_release/install.sh --role agent \
  --accept-eula \
  --manager-hostname <manager-ip> \
  --manager-token <token-from-Settings-Global> \
  --public-hostname "$MY_IP"

Gotcha — $1-style shell syntax gets misread as a Kasm template variable, not literal shell. awk '{print $1}' in a startup script broke every single provisioning attempt with Exception during provisioning ... : 'print $1'. Use cut -d' ' -f1 instead — no $N positional syntax for the templating engine to misfire on.

Gotcha — --accept-eula is mandatory. Without it, install.sh hangs forever at an interactive EULA prompt with no TTY to answer it. The agent software never starts, so the manager's health check times out and destroys the VM — then retries forever. This produces the exact same symptom as several other, completely unrelated bugs below. Don't assume a recurrence is the same root cause as last time just because the symptom looks identical — re-diagnose from the actual logs each time.

Diagnosing a stuck clone/destroy loop — general method

The manager destroys and retries any VM that doesn't check in within its timeout, logging generic messages (Timed out trying to establish agent connection, has not checked in. Destroying!). These don't tell you why — three completely different root causes produced this exact symptom in testing (a bad startup script, a stale API token, and an unrelated network flake). Every time, the fix was to catch a live clone before it gets destroyed and inspect it directly, using Proxmox's own guest-agent channel — works even with no SSH key on the ephemeral clone:

qm guest exec <clone-vmid> -- uptime
qm guest exec <clone-vmid> -- sudo docker ps --format '{{.Names}}: {{.Status}}'
qm guest exec <clone-vmid> -- sudo sh -c 'tail -40 /tmp/kasm_install_*.log'

Zero running containers and no install log at all → the startup script never executed (usually a Proxmox permissions/token problem at the guest-agent-exec layer). An install log that exists but stops partway through → a real failure inside install.sh itself; its last lines are the actual error.

A token-specific diagnostic technique: to tell "this credential is simply invalid" apart from "this credential is valid but under-permissioned," curl the exact API call directly with the suspect token and compare the HTTP status against the same call with a different, known-good token. 401 means the credential itself is bad/stale; 403 means it's valid but lacks a specific permission. This is decisive where reading permission grants alone can be ambiguous.

Autoscale can churn even when clones succeed

If Standby Cores/Memory are set high relative to a single agent's own footprint, the autoscaler's math can perpetually conclude "one more agent is needed" even with an already-healthy agent running — continuous clone/destroy cycling that looks identical to a broken-clone loop but has nothing to do with clone success or failure. Setting Standby Cores/Memory to 0 (provision only on actual session demand) is the most reliable way to stop churn while debugging anything else, independent of whatever else might also be wrong.

Storage contention

If the template and every dynamic clone both live on the same physical storage pool as another I/O-sensitive workload, every clone/destroy cycle does a full disk-image read+write burst that measurably competes for I/O. The clone-target storage (the AutoScale Config's "Storage Pool Name") is a separate setting from the template's own storage location — moving just the clone target elsewhere (leaving the template's reads on the original pool, since reads are far cheaper than writes) can resolve contention without migrating the whole template.

Critical: never write directly to an app-encrypted database column via SQL

Kasm encrypts sensitive config fields (like a stored cloud-provider API token) at the application layer, not just at rest — a pattern common across self-hosted apps for anything credential-shaped. Writing a plaintext value directly into such a column via raw SQL silently breaks the app's ability to decrypt it on every subsequent read.

Symptom if this happens: the app throws a decrypt error reading the row back. In the UI this can look completely unrelated to encryption — pages that depend on that row can go empty or error out, looking exactly like the underlying data was deleted, when it's actually just unreadable.

Before writing to any DB column directly, check whether the application encrypts that specific field. If it does, use the application's own UI/API instead — plain, unencrypted columns on the very same table remain safe to edit directly via SQL. If a column does get broken this way with no way to obtain the app's encryption key: delete the row if the foreign key allows it safely (ideally SET NULL, not CASCADE) and re-enter the config through the UI, or better — restore the affected component from a backup taken before the mistake, which brings back the intact row in one shot.

A misleading error worth knowing about: if the app or its database runs out of disk space, some self-hosted apps surface this as a confusing, unrelated-looking error. In one case here, Kasm's own Admin UI reported a licensing error that had nothing to do with licensing — the real cause was Postgres unable to write its lock file (disk full). A self-hosted app's own error message about licensing/auth/config is not proof of its stated cause — check basic resource health first. docker system df -v to see what's actually consuming space, then docker image prune -f (dangling/untagged images only — never removes a tagged image still in use).

Config caching gotcha

The manager process appears to cache provider/autoscale config at container startup, not read it fresh on every provisioning attempt. After any direct database change to this config, restart the manager container before expecting the change to take effect — confirmed necessary more than once in testing. Don't assume a DB-level fix is live just because the write succeeded.

Lesson: a stuck-loop symptom that looks identical to a previous incident isn't proof it's the same bug. Three different root causes produced the exact same clone→destroy→repeat symptom here — the only reliable path was catching a live clone before it died and reading its actual logs, every single time, rather than pattern-matching against what fixed it last time.