Self-Hosting / Virtualization

Kasm Workspaces + Proxmox Autoscale: Findings & Install Guide

Kasm Workspaces is a self-hosted containerized desktop and browser streaming platform. Browser-based VDI. Point it at Proxmox VE and it clones a template VM on demand as session load rises, instead of running a fixed pool of always-on agents. That is autoscale. Getting there took several rounds of real trial and error. Here is the whole install path plus every gotcha found along the way.

Proxmox-side setup

# Resource pool for Kasm-managed VMs
pveum pool add kasm-workspaces --comment "Kasm Autoscale Pool"

# Dedicated user + API token
pveum user add kasm@pve --comment "Kasm Workspaces Provisioner"
pveum passwd kasm@pve
pveum user token add kasm@pve kasm-token --privsep 0

The permissions needed are broader than they first appear. Custom privilege lists kept hitting invalid-privilege-name errors. VM.Monitor is not a real privilege on current Proxmox, for one. Proxmox's built-in roles turned out simpler and more reliable than hand-rolling a custom one:

pveum aclmod /pool/kasm-workspaces -user kasm@pve -role PVEVMAdmin
pveum aclmod /pool/kasm-workspaces -user kasm@pve -role PVEDatastoreUser
pveum aclmod / -user kasm@pve -role PVEPoolAdmin

This is the minimum permission set, found through live trial and error. Each error below only surfaced once the previous one was fixed, at successive stages of the real clone operation:

Error	Root cause	Fix
`No Resource Pools found matching name(<pool>)`	Template VM was never added as a member of the pool. Creating the pool does not auto-populate it	`pvesh set /pools/<pool> -vms <template-vmid>`
`No Storage Pools found matching name(<storage>)`	Token had zero visibility into node-level storage	`pveum aclmod / -user kasm@pve -role PVEDatastoreAdmin`
`403 Forbidden ... SDN.Use`	No SDN permission granted at all	`PVESDNUser`, plus explicit grants at the zone and zone/bridge paths. Root-level alone did not fully propagate
`Error uploading startup script`	The "Startup Script Path" field expects a directory, and appends its own filename	Use a bare directory path like `/tmp/`, never a filename

Diagnostic technique used throughout. Run the exact API call the Kasm manager makes, directly via curl with the token. That isolates whether the problem is Proxmox-side permissions or a Kasm-side bug. Then read the manager's real traceback from container logs with docker logs kasm_manager --since 10m. The UI's error banner is too generic to debug from.

Kasm Workspaces Autoscale Configurations list showing a Proxmox-backed Docker Agent autoscale config, enabled — **Kasm → Admin → Infrastructure → Pools → All AutoScale Configs.** The Autoscale Configuration tying the pool above to a Proxmox provider config. This is what triggers the clone and destroy cycles.

Template VM

Docker + qemu-guest-agent installed on the base OS.
BE WARNED: qemu-guest-agent.service shows inactive (dead) if the virtio-serial device is missing. That happens when the QEMU Guest Agent option is not enabled on the VM in Proxmox. Enabling it needs a full VM restart, not just a guest reboot.

Clean the machine-id before templating, so clones don't share identical DHCP/network identity:

sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
sudo shutdown -h now

Convert to template with qm template <vmid>. This is irreversible. The VM becomes clone-only.
Add it to the resource pool explicitly. Templating does not do it for you.

Startup script: the real gotchas

#!/bin/bash
systemctl enable --now docker
systemctl enable --now qemu-guest-agent

cd /tmp
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_<version>.tar.gz
tar -xf kasm_release_<version>.tar.gz

MY_IP=$(hostname -I | cut -d' ' -f1)

bash kasm_release/install.sh --role agent \
  --accept-eula \
  --manager-hostname <manager-ip> \
  --manager-token <token-from-Settings-Global> \
  --public-hostname "$MY_IP"

BE WARNED: $1-style shell syntax gets read as a Kasm template variable, not literal shell. awk '{print $1}' in a startup script broke every provisioning attempt with Exception during provisioning ... : 'print $1'. Use cut -d' ' -f1 instead. No $N positional syntax for the templating engine to misfire on.

BE WARNED: --accept-eula is mandatory. Without it install.sh hangs forever at an interactive EULA prompt, with no TTY to answer it. The agent software never starts. The manager's health check times out, destroys the VM, and retries forever. That is the exact same symptom as several unrelated bugs below. Do not assume a recurrence has the same root cause as last time just because the symptom looks identical. Re-diagnose from the logs every time.

Diagnosing a stuck clone and destroy loop

The manager destroys and retries any VM that does not check in within its timeout. It logs generic messages. Timed out trying to establish agent connection. has not checked in. Destroying! Neither tells you why. Three completely different root causes produced this exact symptom in testing. A bad startup script. A stale API token. An unrelated network flake. Every time, the fix was to catch a live clone before it gets destroyed and inspect it directly, through Proxmox's own guest-agent channel. That works even with no SSH key on the ephemeral clone:

qm guest exec <clone-vmid> -- uptime
qm guest exec <clone-vmid> -- sudo docker ps --format '{{.Names}}: {{.Status}}'
qm guest exec <clone-vmid> -- sudo sh -c 'tail -40 /tmp/kasm_install_*.log'

Zero running containers and no install log at all means the startup script never executed. That is usually a Proxmox permissions or token problem at the guest-agent-exec layer. An install log that exists but stops partway through means a real failure inside install.sh. Its last lines are the error.

A token-specific diagnostic technique. To tell "this credential is invalid" apart from "this credential is valid but under-permissioned", curl the exact API call with the suspect token and compare the HTTP status against the same call with a known-good one. 401 means the credential is bad or stale. 403 means it is valid and lacks a specific permission. That is decisive where reading permission grants alone is ambiguous.

Autoscale can churn even when clones succeed

Set Standby Cores and Memory high relative to a single agent's own footprint and the autoscaler's math concludes "one more agent is needed" forever, even with a healthy agent already running. That is continuous clone and destroy cycling. It looks identical to a broken-clone loop and has nothing to do with clone success or failure. Set Standby Cores and Memory to 0 so it provisions only on real session demand. That is the most reliable way to stop churn while you debug anything else, whatever else is also wrong.

Storage contention

If the template and every dynamic clone live on the same physical storage pool as another I/O-sensitive workload, every clone and destroy cycle does a full disk-image read and write burst that competes for I/O. The clone-target storage, the AutoScale Config's "Storage Pool Name", is a separate setting from the template's own storage location. Move just the clone target elsewhere and the contention resolves without migrating the whole template. The template's reads stay on the original pool, and reads are far cheaper than writes.

Critical: never write directly to an app-encrypted database column via SQL

Kasm encrypts sensitive config fields at the application layer, not just at rest. A stored cloud-provider API token, for one. Self-hosted apps do this for anything credential-shaped. Write a plaintext value into such a column via raw SQL and you silently break the app's ability to decrypt it on every subsequent read.

BE WARNED, here is the symptom. The app throws a decrypt error reading the row back. In the UI that can look completely unrelated to encryption. Pages depending on that row go empty or error out, looking exactly like the data was deleted. It is just unreadable.

Before you write to any DB column directly, check whether the application encrypts that field. If it does, use the application's own UI or API. Plain, unencrypted columns on the same table stay safe to edit via SQL. If a column does get broken this way and you cannot get the app's encryption key, you have two options. Delete the row, if the foreign key allows it safely, ideally SET NULL rather than CASCADE, then re-enter the config through the UI. Or better, restore the affected component from a backup taken before the mistake. That brings back the intact row in one shot.

A misleading error worth knowing about. If the app or its database runs out of disk space, some self-hosted apps surface that as a confusing, unrelated error. Here, Kasm's own Admin UI reported a licensing error that had nothing to do with licensing. The real cause was Postgres unable to write its lock file, because the disk was full. A self-hosted app's error message about licensing, auth or config is not proof of its stated cause. Check basic resource health first. Run docker system df -v to see what is consuming space, then docker image prune -f. That takes dangling and untagged images only. It never removes a tagged image still in use.

Config caching gotcha

The manager process caches provider and autoscale config at container startup. It does not read it fresh on every provisioning attempt. After any direct database change to that config, restart the manager container before you expect the change to take effect. Confirmed necessary more than once in testing. Do not assume a DB-level fix is live just because the write succeeded.

Lesson: a stuck-loop symptom that looks identical to a previous incident is no proof it is the same bug. Three different root causes produced the exact same clone, destroy, repeat symptom here. The only reliable path was catching a live clone before it died and reading its logs. Every single time, rather than pattern-matching against what fixed it last time.