Workspace storage¶
Every workspace owns exactly two persistent artifacts, created with the workspace and kept for its whole life:
- the root overlay: a qcow2 copy-on-write file over the
workspace's base image. Everything the guest writes to the root
filesystem (an
apt install, an/etcedit) lands in the overlay; the shared base stays pristine, so every workspace on that image starts from an identical, unmodified root. - the /home volume: an ext4 filesystem attached as a second
virtio-blk disk the guest mounts at
/home. User data lives here, outside the root, so it survives even a full factory reset.
Both artifacts have lifetimes separate from the VM's: stop
powers the machine off and keeps the data; start boots the
same overlay and volume again. Only delete (and factory reset,
for the overlay alone) removes them.
Local backend layout¶
<state_dir>/
├── vms/<workspace_id>/
│ ├── root.qcow2 # the root overlay (plus the VM's runtime files)
│ └── seed.img # the cidata seed (user_data workspaces only)
└── volumes/
└── <workspace_id>.ext4 # the /home volume (sparse ext4)
The overlay is created with
qemu-img create -f qcow2 -F raw -b <base rootfs>: a one-level
chain, never nested overlays. Its backing file is the rootfs
recorded at workspace create, so a workspace always runs the
image version it was created with: importing a newer image
changes what new workspaces boot, never existing ones.
(Rebasing an overlay onto a newer base exists as a qemu-img
operation and may become an explicit "upgrade workspace image"
action later; it never happens implicitly.)
The home volume is a sparse ext4 file. An idle volume costs its metadata, not its nominal size, and it grows as the guest writes.
The seed exists exactly when the workspace was created with
user_data. It is a kilobytes-small iso9660 image labeled
cidata, built with mkisofs (genisoimage) and attached to
the VM read-only. It is installed mode 0600 because the
payload can embed tokens; its staging directory is 0700, and
the database file that records the payload is 0600. It lives
in the workspace's vm directory, so it survives
stop/start and dies with the workspace. Factory reset
keeps it (it is immutable create-time input, so the reset
workspace re-provisions from it), and docs/images.md
describes what runs the payload.
A start heals missing artifacts: if the overlay or the volume file is absent (a crash mid-create, or a workspace row created before this scheme existed), the daemon recreates it before the VM boots. Data that exists is never touched.
Creation is atomic per artifact. Each file is built under a
private temporary name and installed with one rename, and a
failed create rolls back what it made. A missing tool, ENOSPC,
or a partial write leaves nothing at the final path, so the
retry starts from a clean pair instead of booting a blank
disk. A daemon crash mid-create can still leave a finished
artifact (or its *.tmp scratch) behind; the removal helpers
sweep the scratch files, and the next create reports any
leftover instead of adopting it.
Workspace create never reuses an existing artifact. A file found where a new workspace's overlay or volume would live fails the create with a named error. That file is the debris of a previous workspace of the same id, and adopting it silently would hand the new workspace the old one's data. Clear the files and create again.
Sizing¶
Both sizes are set at workspace create and configurable three
ways. Per request: root_mib / home_mib on workspace create.
Per daemon: MSKSD_ROOT_MIB, default 10240, and
MSKSD_HOME_MIB, default 20480. Per image: the catalog's
manifest carries the cmdline the guest boots, which pairs with
the sizes. The overlay's virtual size never drops below its
base image's size: a smaller disk would truncate the base
filesystem.
Create is not the only time sizes move: POST
/api/v1/workspaces/{id}/resize (msks resize) revisits them
on a stopped workspace:
- the /home volume grows or shrinks. The daemon first
quiets the filesystem's journal with
e2fsck -fy. A volume a hard stop left dirty is corrected here; corrections are that tool's job, not an error. Thenresize2fsmoves the filesystem and the file follows. A grow truncates the file up first. A shrink truncates down only after the filesystem agreed.resize2fsrefuses to shrink below the blocks in use, and that refusal reaches the caller as a named409: free data in the workspace or shrink less. - the root overlay grows only, measured against the
overlay's actual virtual size (create clamps it to the base
image's size, so it can sit above the row). Its partition
table and root filesystem belong to the guest, and the
boot's cloud-init
growpartfills the larger device on the next start for free. Shrinking it is a factory reset, not a resize.
A resize carries the home-volume moves' guards (free lifecycle
statuses, the placement check, the move-lock against a
concurrent boot). The artifact files are the truth each side
moves. The grow gate reads the overlay's actual virtual size;
create clamps it to the base image, and a corrupt overlay (one
qemu-img probes as raw or empty) answers a named 503 instead
of being truncated. The home move classifies grow-versus-shrink
by the volume file, so a volume imported above its row's size
reconciles through the same request.
The row follows each side as it moves: msks ls --json and
msks storage report the new ceiling immediately, and the
bytes move at once on the host. A workspace whose artifact file
is absent simply records the new size, and the next start's
artifact heal builds the blank artifact at it. A request giving
sizes the files already have answers an idempotent 200 that
moves nothing. A completed resize is announced on the events
channel (workspace.resized, with the new sizes and topology).
Capacity reporting¶
msks storage answers the operator's three capacity questions,
each from its own source of truth:
$ msks storage
state disk used 23.4G of 40G free 16.6G pressure ok
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━┓
┃ workspace ┃ root cost/ceiling ┃ home cost/ceiling ┃ cost ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━┩
│ ws4 │ 3.1G / 10G │ 812M / 2G │ 3.9G │
└───────────┴───────────────────┴───────────────────┴──────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━┓
┃ image ┃ imported ┃ cost ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━┩
│ debian:13 │ 2026-09-21 12:03 │ 3G │
│ debian:13 │ 2026-08-02 05:11 │ 3G │
└───────────┴───────────────────┴──────┘
- The budget line is the daemon's own
statvfson the state disk: the number that predicts what a full state disk causes (guest-side I/O errors on every workspace).pressuregives the condition:warnpastMSKSD_STORAGE_WARN_PCT(default 90) percent used,criticalat or belowMSKSD_STORAGE_FLOOR_MIB(default 512) free,okotherwise. - Cost is the disk blocks an artifact occupies: the sparse
files' real footprint, never their nominal size. An overlay's
cost sits a little above the root filesystem usage the
guest's own
dfreports (qcow2 bookkeeping rides along) and stays below its ceiling until the guest fills it. - Ceiling is the size fixed at create (
root_mib/home_mib): the quota the guest'sdfshows its user. - Imported is when the archive came in, local time to the
minute. Rebuilding an image and importing it again adds a
second entry under the same reference; the time tells the
entries apart, and the short hash
msks image lsshows identifies the onemsks image rmremoves. Re-importing the same archive refreshes the time.
The fullness a workspace user feels is their own df inside
the workspace; the daemon never guesses at it from outside a
running VM (the ext4 journal lags the host's view of it).
The watcher probes the state disk every MSKSD_EVENT_POLL_S
and publishes a storage.pressure event on each change (a
steady condition announces once, with the daemon's first known
pressure announced as a baseline), with a named log line at
warn and critical. Below the floor, workspace creates,
image imports, and home-volume imports answer 507 giving the
floor and the reclaim path. msks storage gives the consumers;
msks rm and msks image rm remove them. Lowering
MSKSD_STORAGE_FLOOR_MIB admits writes at less headroom, if
that is the deployment's choice. Raising the floor refuses
more writes, not fewer.
Imports are sized when the size is knowable. An image import
must fit the floor plus twice the archive's bytes: the retained
copy plus its unpacked cache. A home-volume import must fit the
floor plus the upload's Content-Length when the client sent
one. A chunked upload carries no length and gets the floor
alone. Existing workspaces keep running below the floor; their
own writes can still fill the disk, so a warn line is the cue
to reclaim before they do.
GET /api/v1/storage serves the same document the CLI renders,
and msks storage <id> narrows the workspace table to one
workspace.
Factory reset¶
POST /api/v1/workspaces/{id}/reset deletes only the overlay.
The next start boots the pristine base image again (installed
packages and every other root change are gone) while /home
keeps its data. The VM is stopped first (the overlay is the
running root device); a wedged VM is killed, exactly as
delete does.
Because the overlay is the root filesystem, reset also clears
any provisioned state the guest keeps there (cloud-init's
run-once markers live under /var/lib/cloud), so future
provisioning re-runs on the next boot.
Home volume export and import¶
The home volume travels through the daemon's authenticated listener as byte streams: the interim mechanism for backup, migration to another daemon, and seeding a fresh workspace with data:
GET /api/v1/workspaces/{id}/home → the volume file's bytes, streamed
PUT /api/v1/workspaces/{id}/home → replaces the volume from the body
msks home export / msks home import drive both endpoints
from the CLI (docs/cli.md); a token holder may also call them
with any HTTP client.
The body is the volume file's bytes verbatim: an ext4
image, labeled msks-home when msksd made it. An import
installs what it receives: a hand-made ext4 image (or one
exported from another workspace) lands exactly as uploaded. The
ext-family magic sits ~1 KiB into the image, and the daemon
checks it as the body's first bytes arrive; a wrong file is
refused with 400 at kilobyte cost, and the workspace's
existing volume is untouched. The check cannot judge
completeness: the daemon installs the bytes that arrive, so
verify a hand-made image (e2fsck -n) before importing it. A
transfer that dies mid-body is the disconnect case and installs
nothing.
Sparseness survives the round trip: the import writes in 1 MiB
windows and turns every all-zero window back into a sparse
hole, so a blank 2 GiB volume re-imports at its data's cost,
not its nominal size. A volume exported over a slow link
composes with any compressor (msks home export ws - | gzip >
ws.ext4.gz) because - streams the bytes to stdout.
Both endpoints answer 409 unless the workspace's row says its
VM is down. created, stopped, and absent move freely.
starting, running, and paused keep the volume: an export
under a guest mid-write is a torn image, and an import into a
mounted device would be overwritten or lost. unknown refuses
too. The seam reports it when a live VMM stopped answering its
API, so a possibly-live VM gets the volume's protection. Stop
the workspace first (msks stop). Under the move's lock the
daemon also asks the seam itself: a row can lag a boot by one
watcher poll interval, and the live VMM's answer refuses where
the row would pass.
A move, a boot, and a delete serialize per workspace on one
lock. A boot that arrives during a move waits for it to finish
and boots the volume the move left. A move that arrives after a
boot sees the running row and answers 409. A delete orders
against an import instead of racing it. A waiter gives up after
MSKSD_MOVE_WAIT_TIMEOUT_S (120 s default) and answers a named
409 (a volume move is in flight) rather than hanging; a
stalled export reader holds its lock as long as its connection
lives. A foreign host answers the placement 409 every
artifact route shares.
An import whose filesystem size differs from the workspace's
recorded home_mib mounts fine either way: an ext4 filesystem
smaller than its device is legal. home_mib stays the size a
blank rebuilt volume gets, and growing an imported filesystem
is a guest-side resize2fs. The install is atomic: a failed or
cut-off upload leaves the existing volume in place. Every
completed move is announced on the events channel as
home.exported / home.imported, with the byte count. An
export announces when its stream reaches the end, so a
cancelled download publishes nothing.
Placement and single-attach¶
The workspace row records the host that owns its artifacts.
On the local backend that is always the msksd host that created
the workspace. A start, stop, reset, or delete through a daemon
on another host fails with 409 giving where the artifacts
live. Booting a workspace with an empty /home, marking a
running VM stopped, or dropping the row out from under a live
VM are all worse than the refusal. The host name is the
daemon's MSKSD_HOST_NAME (default: the host's hostname at
daemon start); pin it explicitly when the hostname is not
stable (laptops, containers) so a rename does not strand every
workspace. Deleting a workspace whose artifacts live elsewhere
happens from that host: this one refuses rather than orphan the
files. A workspace runs on at most one host at a time; the
row's placement is the authority locally, and the volume's
single-writer semantics make a double attach impossible.
Image pinning¶
A catalog image with live workspaces cannot be removed
(DELETE /api/v1/images/{hash} answers 409 giving the
workspace): each workspace's overlay backs that exact image
file, and its kernel and initrd are read from the image's cache
on every boot. Deleting the workspace releases the pin.
Environment variables¶
| Variable | Default | Meaning |
|---|---|---|
MSKSD_ROOT_MIB |
10240 |
Default overlay (root) size for new workspaces, MiB. |
MSKSD_HOME_MIB |
20480 |
Default /home volume size, MiB. |
MSKSD_STORAGE_WARN_PCT |
90 |
State-disk percentage used that moves pressure to warn. |
MSKSD_STORAGE_FLOOR_MIB |
512 |
Free state-disk MiB below which pressure is critical and writes answer 507. |
MSKSD_QEMU_IMG |
qemu-img |
The qemu-img binary that creates overlays. |
MSKSD_MKFS_EXT4 |
mkfs.ext4 |
The mkfs that formats /home volumes. |
MSKSD_RESIZE2FS |
resize2fs |
The resize2fs that moves /home volumes. |
MSKSD_E2FSCK |
e2fsck |
The e2fsck that quiets a volume before a resize. |
MSKSD_MKISOFS |
mkisofs |
The mkisofs (genisoimage) that builds cidata seed disks. |
MSKSD_HOST_NAME |
the hostname | The host recorded as owning locally-created artifacts. |
MSKSD_SHUTDOWN_TIMEOUT_S |
20 |
How long stop waits for the guest's clean poweroff before the fallback kill; a stop answers within this bound. |
MSKSD_MOVE_WAIT_TIMEOUT_S |
120 |
How long a boot, delete, or volume move waits for the workspace's other volume move before answering a named 409. |