Workspace images¶
A workspace image is one file, a container-image tar: the layout
podman load and skopeo already understand — carrying a kernel, an
initrd, a rootfs disk, and a manifest that describes all three. This
chapter covers the image contract, how to build one, how to check
one, how to register it with a running daemon, and how workspaces
select an image.
msksd boots the kernel directly and attaches the rootfs as a virtio
disk. Nothing ever runs the image as a container; the container-image
packaging exists so the file is inspectable and loadable by stock
tooling (tar tf, podman load, skopeo copy) and can travel
through container registries unchanged.
The image contract¶
The tar follows the containerDisk convention: standard container
bookkeeping (manifest.json, an image config, repositories)
wrapped around a single uncompressed layer whose root carries the
workspace boot files:
workspace-<name>-<version>.tar
├── manifest.json # container bookkeeping
├── repositories
└── <image-id>/
├── VERSION, json # container bookkeeping
└── layer.tar # uncompressed; the containerDisk:
├── boot/vmlinuz # bzImage kernel
├── boot/initrd.img # initramfs
├── disk/rootfs.ext4 # raw ext4 root filesystem
└── disk/image.json # the manifest (below)
disk/image.json is what msksd reads; schema 2:
| Field | Meaning |
|---|---|
schema |
2 |
arch |
The guest's architecture, uname machine names — x86_64 or aarch64. |
| The shipped builders always emit it. Import compares it to the | |
| host's own machine and refuses a mismatch naming both sides — a | |
| foreign-arch kernel would hang wordlessly at boot. An image without | |
| the field declares nothing and imports as before | |
name |
Catalog name, e.g. debian |
version |
Catalog version, e.g. 13.6; numeric segments sort correctly |
cmdline |
Kernel command line for workspace boots |
kernel_version |
e.g. 6.12.107+deb13-amd64 (informational) |
kernel_format |
bzImage on x86_64, Image on aarch64 (informational) |
capabilities |
Optional capability object; provisioner names the seed consumer (below) |
The manifest is self-describing: importing the archive needs nothing beside the archive itself.
What the host provides¶
Every workspace artifact the guest touches is owned by msksd
at runtime. The root and home volumes and the cidata seed
disks are created under the state dir (MSKSD_STATE_DIR, the
one relocation variable). The console reaches the guest over
the virtio-console port of the VMM msksd launched. Both
deployment shapes (the NixOS module and the dev daemon)
deliver this without host-side scripting. The
host itself provides four things, all configuration:
/dev/kvm, reachable through thekvmgroup.- Two ambient capabilities for msksd's service user:
CAP_NET_ADMIN(taps, addresses, nftables) andCAP_NET_BIND_SERVICE(the DHCP and DNS listeners). - The egress kernel modules:
tunand the nftables/NAT set, loaded at boot (the NixOS module'sboot.kernelModules). net.ipv4.ip_forward=1, verified by the daemon at start when egress is armed; the module ships it as a sysctl.
The daemon e2e smoke (test_smoke/test_daemon_e2e.py) drives this
whole surface against a real msksd process: the image boots,
serves its console, and reaches the outside exactly as this
section describes.
What a guest must provide¶
The rootfs and kernel together decide whether a workspace actually boots. A guest image must:
- Boot a kernel with direct-kernel boot support. cloud-hypervisor
loads the kernel and initrd itself and passes
cmdline; the guest never runs its own bootloader. Stock Debian/Ubuntu kernels work (a bzImage on x86_64, an Image on aarch64). - Serve a root getty on the serial console.
msks consolebridges the client to the guest's serial device through the VMM's serial socket, so the guest needs an autologin root getty on its serial console. The shipped images runserial-getty@on the arch's console device —ttyS0on x86_64,ttyAMA0on aarch64 — with an autologin drop-in. The kernel console (the same device, named on the cmdline) shares the port: kernel messages interleave with the session, which is exactly what a failsafe console wants. Authentication is the daemon's TLS + token listener, and the stream is raw bytes with no in-band protocol. The session keeps the getty's own terminal geometry for its life. A workspace user's shell with the client's terminal size rides ssh through the forward seam. - Register the virtiofs filesystem driver when shares
attach. Host directories reach a workspace as virtiofs
mounts (#667), so the guest's kernel must know the filesystem —
built in (both pinned Debian flavours:
CONFIG_VIRTIO_FS=y) or loadable (Fedora's image ships the module in its runtime closure and loads it at boot).msks image check'svirtiofsrow probes the driver; a workspace that attaches shares on an image without it reports the named gap on the guest's console and cloud-init log at start, and the mounts never come up. - Take an address over DHCP when a NIC is present.
Workspaces are networked by default: the VM boots with a
virtio-net NIC, and the daemon's DHCP service offers the
address, gateway, and its own resolver. Debian's kernel
ships
virtio_net, and the shipped overlay loads it at boot. The overlay carries a systemd-networkd.networkunit (Name=en* eth*,DHCP=yes) plus resolved's stub resolv.conf, so the guest configures whatever NIC appears. The offered resolver becomes resolved's per-link upstream:/etc/resolv.confnames the127.0.0.53stub, and the lease's resolver in effect is read fromresolvectl dns(the git-out smoke's probe). A workspace created with"egress": falsepresents no NIC: nothing matches the unit and networkd stays idle — the same image serves both postures. The shipped base carries no developer toolchain: git, curl, and uv all land over the egress NIC when a seed or the operator installs them. - Answer the ACPI power button.
stoppresses the power button (vm.power-button) and waits; the guest's own handler runs the clean shutdown that flushes its disks — systemd-logind does this by default, an acpid rule works too. A guest that ignores the button is killed at the deadline, and whatever its page cache still held is lost. - Tolerate a read-write root. The daemon boots each workspace's
root read-write through a per-workspace qcow2 overlay, so the
base image stays pristine under copy-on-write, and attaches an
ext4 volume at
/home. Mount it by the volume labelmsks-home(the daemon formats the volume with that label) so the mount stays on the right device whatever the disk order is;nofailkeeps boots moving when the volume is absent. A udev rule asks systemd for the mount again when the labeled disk appears, so a boot whose device job expired while udev was still probing mounts the volume moments after the label lands — every boot the image ships, restarted workspaces included (the daemon's identity seed carries the same retry for first boots on images that predate the rule). - Grant passwordless sudo to the
wheelgroup. Both images ship the conventional admin groupwheelwhose members administer the VM. Debian does it through a%wheeldropin in/etc/sudoers.d, and the image creates the group because Debian's own tree carries none. NixOS does it through a declarative sudo rule. The identity seed creates a named login user, gives it the workspace user's shell, and joins it to the group. The seed never writes sudo configuration, so a guest that is rebuilt keeps exactly the sudo policy its configuration declares. - Run cloud-init against the cidata seed disk. A
workspace created with
user_dataor a seeded ssh identity boots with a third, read-only virtio disk: a small iso9660 filesystem labeledcidata, exactly cloud-init's NoCloud seed layout. Its root carriesuser-data(the operator payload, composed beside the identity's seeding script when a key was supplied) andmeta-data. The meta-data holdsinstance-id, keyed off the workspace id, andlocal-hostname, the workspace's creation name (the minted id when the workspace was created without one). The hostname lands at first boot, and a factory reset re-provisions from the seed with it; a stop/start never re-provisions (instance-id never changes). A launch heal that rebuilds a lost seed carries the hostname as well, but a guest that booted before keeps the hostname it already has. cloud-init sets the hostname per instance, and its per-boot update leaves a hostname it never set untouched, so only a factory reset moves an older workspace's./etc/hostscarries the workspace's name as well: the Debian image turns on cloud-init'smanage_etc_hostsin the mode that adds the conventional127.0.1.1record naming the host, in place, on every boot — the file's other entries stay in place. A guest first booted before whose healed seed now carries the name gets the record naming the creation name while its kernel hostname keeps the older one; the module reads the seed, not the kernel. Name resolution does not wait on the record: both images' resolvers answer the local hostname on their own, via themyhostnameNSS module (Debian'sgenericcloudbase shipslibnss-myhostname, and it is part of the NixOS base system). The record is the plain files-source entry a tool reading/etc/hostsdirectly still finds. The image ships cloud-init (the Debiangenericcloudbase does), and dropins pin the behavior the msks contract needs.datasource_list: [ NoCloud, None ]makes the seed disk answer immediately, with no EC2 or OpenStack probing and no network timeouts.network: {config: disabled}leaves the image's own networkd unit owning whatever NIC appears, with cloud-init's netplan rendering out of the way. Both payload forms run: cloud-config YAML and#!scripts.
cloud-init's run-once state (the /var/lib/cloud cache) lives on
the workspace's root overlay, so stop/start never
re-provisions and a factory reset does (the reset drops the
overlay). Any distro cloud image that ships cloud-init (Debian's
generic and genericcloud, Ubuntu, Fedora, ...) imports as-is
and declares "capabilities": {"provisioner": "cloud-init"}.
Building an image¶
Architectures¶
The guests build for x86_64 and aarch64. The bare flavor names
follow the building host's uname -m, so msks-build-guest on an
aarch64 dev host IS the arm64 build; the -x86_64 and -aarch64
suffixes pin an arch explicitly (msks-build-guest nixos-aarch64),
and the Debian and Fedora flavors are fetch-and-pack, so their
pinned attrs build on any host (the NixOS closure wants its native
evaluator — the arm64 busybox guard in the initrd build refuses a
cross-eval with the cause named).
The arm64 guests run 16K-page kernels — Debian's own arm64-16k
flavour, and a kernel patch on the NixOS side — because a flat
guest's page size is guest-internal (the image boots on any arm64
host) while nested KVM inside the guest initializes only where the
guest's page size matches a 16K host's (#635's Stage-2 scoping):
one image covers both. The console differs by arch — the cmdline
names ttyS0 on x86_64 and ttyAMA0 on aarch64, where
cloud-hypervisor presents a PL011 UART — and the manifest records
the arch, the kernel format, and the cmdline together, so the
archive stays self-describing.
The shipped builder¶
builds the default image (workspace-debian-13.6.tar) from a
date-pinned, sha512-verified Debian trixie genericcloud qcow2 —
the cloud-init-bearing variant of Debian's cloud image family, so
cloud-init and its python3 runtime arrive with the base. The build converts the qcow2 to raw, extracts the root
filesystem, and applies the msks overlay and boot diet. It
repacks the result as a deterministic ext4 (mke2fs -d under
fakeroot; the intermediate rides the nix store as one opaque
tarball, never as a tree). It then wraps the kernel, initrd,
rootfs, and a generated image.json into the container-image
tar with byte-stable tar flags (--sort=name --mtime=@1
--numeric-owner). Identical rebuilds hash identically, so the same
image deduplicates across hosts.
The output lands under .devenv/state/guest/ (GUEST_DIR
relocates it); scripts/build-guest.sh and
nix/guest-debian.nix document every step and are the reference for
what an image build does.
Both images bake the agent toolchain: pinned Node, the
pinned pi coding agent, and the pinned herdr terminal
workspace manager (herdr.dev, the release's static binary,
with its license). They also bake pinned Claude Code (the npm
wrapper plus its linux-x64 native binary, staged in npm's
global layout with the wrapper's own post-install linking done
at build time) and the pi model-discovery extension in
/etc/skel and root's home. The internal docs/llm.md
describes what the extension does at pi startup. The staging differs per image. Debian lands the official
Node tarball and the pinned packages under /usr/local.
NixOS rides nixpkgs' own Node and the same pinned packages
through the system profile. Claude Code's native binary is
loader-patched to the closure's glibc, because a stock NixOS
ships no usable loader where the published interpreter points.
The extension is planted by tmpfiles copy-once rules. The pins move with an image rebuild: agentNodeTarball in
nix/guest-debian.nix; piTarball, npmDepsHash (and, on a
pi bump, the table in nix/pi-shrinkwrap-integrity.json),
agentHerdrBinary and its license pin, and the two
agentClaude* tarballs in nix/agent-toolchain.nix. The
build stays pure derivations: pi's dependency closure is
prefetched against its shrinkwrap, with the five integrity
gaps the published lock leaves closed by hash, and installed
offline.
A workspace that already booted keeps the toolchain it booted
with; a rebuilt image serves the new pins to the next workspace.
pi's own tool dependencies ride the image too. pi resolves
fd and rg from PATH, at interactive startup and again at
first tool use, and downloads each from GitHub releases when
it finds neither. Without them, a fresh workspace's first
agent start would wait on that download, behind the egress
interceptor and the GitHub API quota every workspace behind
one address shares. The Debian image stages Debian's own fd-find and ripgrep
debs (fdFindDeb and ripgrepDeb in nix/guest-debian.nix,
pinned by pool URL and checksum like the kernel and rsync
debs) with the same linkage guard rsync gets. fd-find's binary
lands under /usr/lib/cargo/bin with /usr/bin/fdfind a
symlink, exactly the layout apt install fd-find leaves, and
pi accepts the fdfind name. The Fedora image stages the
distro's own fd-find and ripgrep rpm (fdFindRpm and
ripgrepRpm in nix/guest-fedora.nix, pinned from the
compose's Everything repo, with fd's libjemalloc dependency
pinned beside them) under the same guard; Fedora names the
binary fd, the name pi probes first. The NixOS image
ships nixpkgs' own fd and ripgrep on the profile PATH the
toolchain rides. With both present, a first pi start needs no
downloads.
The tools themselves decide some of their own freshness.
Claude Code checks for updates on startup and installs a newer
self into the user's home when it finds one; its
autoUpdates setting and the DISABLE_AUTOUPDATER
environment variable are the switches. herdr's herdr update
does the same on demand. A workspace
user can therefore run a newer tool than the image pin — the pin
governs what a fresh workspace starts with, not what a user's home
accumulates. A no-egress workspace sees these checks fail quietly.
builds the second catalog image (workspace-nixos-<version>.tar)
from a NixOS system evaluated against the same pinned nixpkgs the
development shell uses — the guest's cloud-init, sshd, and rsync
are built by nixpkgs instead of fetched as Debian artifacts. The root filesystem is the whole system closure packed
into a fresh ext4 (the same mke2fs -d under fakeroot; no cloud
image exists to extract), and every store path resolves from the
image itself. The archive carries the same image.json schema with
the same declared capabilities (cloud-init): the
daemon serves it with nothing keyed off the image's name. The
output lands under .devenv/state/guest-nixos/
(GUEST_NIXOS_DIR relocates it); nix/guest-nixos.nix
documents every step. The image ships the same agent toolchain as
the Debian one: nixpkgs' own Node and the shared pins
(nix/agent-toolchain.nix) ride the system profile — the
loader-patched Claude Code and the tmpfiles-planted extension
staging described above.
Rebuilding the system inside a NixOS workspace¶
A NixOS workspace is nixos-rebuild-ready. nix runs as on any NixOS host: the daemon, the build
users, and the CLI ship enabled. The image bakes everything a
rebuild evaluates against:
- The store database. Every store path the image ships
is registered valid: the build loads the closure's
registration into
/nix/var/nix/db/db.sqlite. The guest's nix therefore answers reference queries, and a rebuild reuses the shipped closure instead of refetching or rebuilding it. - The pinned nixpkgs source. The same revision the image was
built from rides the store as root's channel profile, so
<nixpkgs>resolves with no network. Anix-channel --updatemoves to the rolling channel deliberately — that is the stock NixOS posture, and the pin governs what a fresh workspace starts with. /etc/nixos/configuration.nix. It importsnix/guest-nixos-configuration.nix, the very module the image build evaluated, with the module's whole import chain (its package files, the shrinkwrap pair) shipped beside it at the same relative paths. The image build evaluates the shipped chain exactly as a rebuild does, so a configuration a rebuild cannot evaluate fails the build, not a workspace's first boot. A rebuild re-evaluates the shipped configuration. Edit the module or add settings inconfiguration.nixand run:
-
The nix search path. The same four entries ship in
/etc/nix/nix.confand in every login shell'sNIX_PATH:nixpkgs,nixos-config, thenixos-systementrypoint the rebuild tool prefers, and the channels profile.<nixpkgs>therefore resolves identically for the root rebuild path and for the workspace user at the console. -
The system profile as the boot's init. The kernel cmdline names
init=/nix/var/nix/profiles/system/init— the activation-maintained indirection anixos-rebuild switchre-points — so a rebuilt system survives a workspace stop/start with no daemon involvement. The VMM always boots the image's frozen kernel and initrd, so a rebuilt system's new kernel takes effect only through a new image; userspace changes apply fully on reboot.
A rebuild that changes nothing but re-activates keeps every msks
contract item — the console getty, the sshd posture, cloud-init
seed handling, the agent toolchain on PATH — because the
configuration it evaluates is the one that shipped. nix works the same way for the unprivileged workspace user
logged in on the console. nix-build '<nixpkgs>' -A <pkg>
builds from the baked pin through the daemon;
nix-env -iA nixpkgs.<pkg> installs into the user's own
profile; nix-shell -p <pkg> builds and enters an
environment. Fetches the shipped store lacks come from
cache.nixos.org through the workspace's egress path. The image ships one background consumer of this readiness.
The msks-interceptor-ca oneshot folds the daemon's
interception CA (staged by the identity seed under
/etc/msks, one CA for every workspace) into the system
trust bundle. The module reads the staged certificate at
evaluation time; the unit runs one nixos-rebuild switch per
fresh certificate after cloud-final; a hash marker keeps
later boots no-ops. The image build itself refuses a host that
carries /etc/msks/interceptor-ca.crt (the module's assertion
fails the build), since that evaluation would bake a foreign CA
into every guest. A workspace with egress fetches substitutes from
cache.nixos.org through the workspace's egress path like any
other traffic. A no-egress workspace rebuilds against what is
already in the store; new packages that need a fetch fail with
nix's own error until egress is granted. Both images stay the same size class as before —
the nix closure, the validity database, and the nixpkgs source
grow the NixOS rootfs by roughly a gigabyte uncompressed.
Each image build also executes every staged launcher and
fails on one that does not run. The Debian build runs node,
pi, claude, fdfind, and rg through the tree's own
dynamic loader and libraries, the same world the guest execs
them in, because the build sandbox carries no /usr/bin/env
or /lib64. herdr, a static binary, runs as-is. pi's and
npm's published #!/usr/bin/env node shebangs, which that
sandbox cannot resolve, are asserted byte-exact instead. The NixOS build runs the profile
binaries directly. The first image that shipped the toolchain passed every
presence check while pi could not start: the npm build had
rewritten cli.js's shebang to a build-time Nix store node no
guest carries. test -x on a symlink says nothing about the
interpreter behind it. The launcher executions turn
that class of breakage into a build failure.
builds the third catalog image (workspace-fedora-44.tar, #651)
from Fedora's date-pinned, sha256-verified Generic cloud
qcow2 — the same fetch-and-pack shape as the Debian build, with
Fedora's own divergences adapted at each step:
- btrfs, not ext4. The cloud image's root filesystem is a
btrfs subvolume set (
root,boot,home,var); the build extracts it withbtrfs restore— unprivileged, no mount — flattens the set into the plain tree a direct-boot guest wants, and packs a fresh ext4 from it (the samemke2fs -dunder fakeroot). The extraction runs inside one fakeroot session, andbtrfs restore -mrestores the distro's own owner/group/mode — the setuid bits included — into the session's faked view; the session writes that view out as a manifest and the pack stage re-applies it, because a faked view cannot outlive its session and untarring it under a new one would land setuid binaries genuinely unreadable (mke2fs -dreads file contents through the real modes). - The kernel rides inside the image. Fedora publishes no
separate kernel artifact:
/usr/lib/modules/<kver>/vmlinuzis the kernel, its config, and the module tree in one pin. The minimal initramfs carries zero modules — Fedora's kernel builds ext4, virtio_blk, and crc32c in — so busybox mounts the root and switch_roots into systemd. - SELinux stays off. The build reconstructs the filesystem
without SELinux xattrs (
btrfs restoredrops them, and the unprivileged pack cannot write them), so every file would boot unlabeled and enforcing policy freezes PID 1 before the first getty; the boot cmdline carriesselinux=0. The workspace's isolation boundary is the VM itself — the Debian and NixOS images carry no MAC either. - NetworkManager owns the NIC. Fedora's image runs NetworkManager, whose default policy takes a DHCP address on any ethernet it finds, so an egress workspace's NIC configures itself with no msks-side unit (the Debian overlay's networkd enablement is the divergence there) and cloud-init's network rendering stays off the same way.
- The console prompt is pinned to the shared shape. Fedora's
default bash prompt is
[root@host ~]#; the build appends thePS1that presents the sameroot@host:~#every family's console surface answers with. - Boot diet. chronyd and the zram generator's config drop (the generator would stall each boot ~45s on a zram device no workspace VM presents), and the module closure ships the virtiofs module Fedora builds separately from the Debian pair.
The output lands under .devenv/state/guest-fedora/
(GUEST_FEDORA_DIR relocates it); nix/guest-fedora.nix
documents every step. The image stages the same agent toolchain
as the Debian one (the official Node tarball and the shared pins
under /usr/local) and the same fd/rg staging described above,
from the distro's own rpm.
Building your own¶
Any rootfs that satisfies the contract above can become a workspace image. The outline, using a distro's own cloud image as the source:
- Fetch the source image and verify it against its published checksum.
- Produce a raw ext4 of the guest root filesystem
(
qemu-img convert+ partition extraction, or unpack the cloud image's root archive directly). - Enable an autologin root getty on
/dev/hvc0; make sure thevirtio_consolemodule is present so the device exists at boot. - Bring up networking. Workspaces boot with a virtio-net NIC by default, so the image must be able to configure one. How it does so is the distro's choice:
- a kernel with the
virtio_netdriver present (module or built-in; on most distros udev autoloads the module when the device appears); - a DHCP client that configures whatever NIC appears and honors the offered address, gateway, and resolver — networkd, dhcpcd, or anything else that speaks DHCP;
- name resolution must follow the resolver DHCP names (however
the distro wires
/etc/resolv.conf); - a boot with no NIC (a workspace created with
"egress": false) must still reach a usable login — the same image serves both postures.
The shipped image does this with systemd-networkd + resolved; any equivalent stack works.
- Ship cloud-init, configured for the cidata seed. Starting from
a distro cloud image (Debian
generic/genericcloud, Ubuntu, Fedora — they carry cloud-init and its python runtime) gives you this for free; otherwise install the distro's cloud-init package. Two dropins under/etc/cloud/cloud.cfg.d/pin the msks contract: datasource_list: [ NoCloud, None ]— the workspace's seed disk answers immediately; nothing probes EC2 or OpenStack sources or waits on the network;network: {config: disabled}— the network configuration from step 4 owns the NIC; cloud-init's renderer would only fight it.
The guest kernel needs the isofs module present to mount the
seed (every stock distro kernel carries it). Declare the consumer
in the manifest (step 6) so listings show what eats the seed. An
image whose guest runs no cloud-init still accepts user_data at
create, but nothing executes it — the daemon cannot tell.
- Write
disk/image.jsondescribing your kernel and cmdline, and declare"capabilities": {"provisioner": "cloud-init"}. - Lay out
boot/anddisk/as the layer tree and wrap it:
tar --sort=name --mtime='@1' --owner=0 --group=0 --numeric-owner \
-C layer-root -cf workspace-mine-1.0.layer.tar .
# then wrap layer.tar in a container-image tar, or simply:
podman import workspace-mine-1.0.layer.tar workspace-mine:1.0
podman save -o workspace-mine-1.0.tar workspace-mine:1.0
podman import/save produce a valid outer layout; msksd accepts
both compressed and uncompressed layers (uncompressed layers keep
import and in-place inspection cheaper).
Checking an image¶
The daemon verifies layout and manifest at import, but a tar that
parses can still carry a guest that does not boot — the layout is
the only thing the daemon can see. msks image check <archive> is the gate an image author runs
before publishing. It boots the archive exactly the way a
workspace boots (same driver, overlay, home volume, seed disk,
and console path, against a throwaway state dir under /tmp)
and reports each contract point by name:
$ msks image check workspace-mine-1.0.tar
PASS archive imported mine:1.0 (1d6a5e782fc0)
PASS boot guest answered the console in 3.4s (kernel 6.12.107+deb13-amd64)
PASS console root autologin getty on the console port
PASS virtiofs the virtiofs filesystem driver answers (mount -t virtiofs works)
PASS user-data seed payload ran on first boot
PASS acpi-shutdown clean shutdown within 120s
PASS root-rw root is writable and the write survived a stop/start (overlay)
PASS home-label /home mounted by label msks-home and its write survived a stop/start (volume)
Any host with /dev/kvm runs it — nothing else from msks is
needed (no daemon, no state). The points map one-to-one onto "What a guest must provide"
above. The checks cover the archive layout and manifest, a
guest whose console getty answers, the virtiofs driver a share
mounts through (#674 — an image without it fails its own row
with the named gap, before a workspace's share attaches as a
mount unit that never comes up), a root that accepts writes
through the overlay, and /home mounted by its label (both
surviving a stop/start cycle). They also cover the seed
payload running on first boot (checked when the manifest
declares a provisioner; a provisioner-less image reports the
point skipped with that reason) and the ACPI power button
producing a clean shutdown. A broken point fails its row, the exit code is 1,
and the summary names the first failure; --keep preserves the
throwaway state dir (serial logs included) for inspection.
--egress adds the networking point — the guest must take a
global address over DHCP when the checker boots it with a NIC
through the daemon's own net stack. That leg needs root (taps, nftables, the DHCP and DNS
listeners on their privileged ports) and an egress-capable
default route (--uplink names another interface). The
checker sets net.ipv4.ip_forward to 1 for the pass and
restores what it found; a checker killed outright leaves the
sysctl at 1. Run the leg on a host that is not serving
the real daemon: both bind the same privileged DHCP and DNS
ports. The no-NIC half of the posture is
the core pass itself: it boots without a NIC and requires a usable
login. The write probes (root-rw, home-label, user-data)
run over the root console.
Registering an image¶
Import makes the daemon unpack the boot files once into a per-hash
cache and record the image in the catalog. The msks image commands
drive this surface from the CLI — msks image import <path>, msks
image ls, msks image rm <ref>, msks image info <ref>, and msks
image default <ref> (see
docs/cli.md); the raw HTTP form:
curl -X POST https://192.168.77.2:8660/api/v1/images \
-H "authorization: Bearer $TOKEN" \
-H "content-type: application/json" \
-d '{"source": "/path/on/the/daemon/workspace-mine-1.0.tar"}'
The source is a path on the daemon's filesystem (the daemon
reads them from its state dir), or — — an https:// URL the
daemon downloads itself:
msks image import https://images.example.com/workspace-mine-1.0.tar
curl -X POST .../api/v1/images -d '{"source": "https://images.example.com/workspace-mine-1.0.tar"}'
A URL download lands in the catalog's staging area under a
size ceiling and a deadline (MSKSD_IMAGE_IMPORT_MAX_MIB,
MSKSD_IMAGE_IMPORT_TIMEOUT_S; see docs/config.md). It
verifies TLS against the system roots, refuses a redirect that
leaves https, and imports from the downloaded copy, so the
recorded hash always reflects the fetched bytes. The ceiling also clamps to the
bytes the storage floor protects, so the download itself cannot
spend the state disk's headroom. The fetch runs with the calling token's authority. A
deployment hands its API token to operators it trusts to
import images, and the URL form reaches whatever network the
daemon can reach. That is the same trust a daemon-side path
import already carries. Either way the daemon
copies privately and hashes that copy, so a source file changing
underneath the import cannot desync the recorded hash from the
imported content, and importing the same content twice is idempotent.
An import registers the archive under an operator-chosen pair
when name/version ride the request — msks image
import <path> --name mine --version 1.0 on the CLI, or the keys
beside source over HTTP. Either key alone moves; the other
falls back to the archive's own manifest. The override changes
the registered record — the listing, image info, and
name:version reference resolution. The archive bytes, the
content hash, and the archive's own image.json stay exactly as
they were; the manifest's pair stays recorded as the row's
origin beside the registered pair, so a usable image's
provenance reads from the catalog alone.
A cataloged row renames in place under the same rules:
msks image rename debian:13 --name mine --version 1.0
curl -X PATCH .../api/v1/images/<hash> \
-H "authorization: Bearer $TOKEN" -d '{"name": "mine", "version": "1.0"}'
Either key alone keeps the other at its registered value. A rename validates the new pair the same way an import
override does: non-empty, and free of : and @ (the
reference forms key on them). It refuses, with a named error
that names the holder by hash and by its origin when an
override or a rename moved it there, a pair another row
already holds. It keeps the origin pair and changes nothing
else: the bytes, the hash, the listed import time, and
workspaces already booting the image stay put. The refusal holds
for overrides and renames; a plain import of a rebuilt archive
under a pair a row already holds stays allowed (the long-standing
shape — name@hash and bare-hash forms resolve the ambiguity).
Re-importing unchanged bytes carries the row's current
registration forward; re-importing the same bytes with a new
override renames the row the same way a rename does.
Rules worth knowing:
- The first image imported becomes the default — the one a bare
workspace create uses — and keeps the designation while more
images arrive;
POST /api/v1/images/defaultre-designates at any time:
# designate by any reference form create's image field takes
curl -X POST .../api/v1/images/default \
-H "authorization: Bearer $TOKEN" -d '{"ref": "debian:13.6"}'
# clear the designation; the answer reports the fallback
# (the sole catalog entry, or none)
curl -X DELETE .../api/v1/images/default \
-H "authorization: Bearer $TOKEN"
- The designation is the catalog's
defaultpointer file. It survives restarts, andmsks image ls/image infomark the designated row. With it cleared, a bare create falls back to the sole catalog entry; on a catalog holding several images it answers a named error until one is designated again. MSKSD_DEFAULT_IMAGE(set by the dev daemon from its state dir's image) imports and designates at first boot; on later boots the daemon only re-checks the hash, not a full re-import.MSKSD_SEED_IMAGESimports at every start, warm when the archive is unchanged and never designated; the dev daemon sets it to the NixOS guest's archive whenevermsks-build-guest nixoshas built one. A rebuilt guest is therefore what a bare-name create boots, without a manual re-import.- Listing shows every registered image with its hash, name,
version, kernel facts, which one is default, and when each
entered the catalog (the
importedfield, ISO 8601 UTC). A renamed row additionally carries its origin pair (origin_name/origin_version, equal toname/versionfor an unrenamed row):
-
Storage: one image costs roughly twice its rootfs size on the daemon's state dir (the retained archive plus the unpacked boot cache), carried beside the workspace overlays and volumes —
docs/storage.mdhas the capacity model. -
An image with workspaces still booting it cannot be removed (
DELETE /api/v1/images/{hash}answers 409 naming the workspace); once its workspaces are gone, deletion drops the cache and the retained archive.
Using an image¶
Workspace create selects an image by reference:
# name:version — exact
curl -X POST .../api/v1/workspaces -d '{"id": "ws1", "image": "debian:13.6"}'
# bare name — resolves to the newest registered version; the
# most recently imported row wins when two builds carry the same
# leading numbers (e.g. a NixOS version's trailing store hash
# differs between builds)
{"id": "ws1", "image": "debian"}
# name@hash — pins identity AND content
{"id": "ws1", "image": "debian@<sha256>"}
# bare hash — content-addressed
{"id": "ws1", "image": "<sha256>"}
# no image at all — the designated default
{"id": "ws1"}
Versions order numerically (13.10 sorts after 13.9). A malformed
reference (debian@not-a-hash) is a named 400 rather than a
miss. Explicit kernel/rootfs fields still win over the image —
the image is the convenient path, not a mandate.
First-boot provisioning (user_data)¶
A workspace created with user_data gets a customization payload
that runs on its first boot — EC2-style, delivered on the workspace's
own seed disk:
# a script (any image with a provisioner; the leading #! is what
# makes it one)
msks create ws --user-data provision.sh --start
# cloud-config (needs a cloud-init image)
curl -X POST .../api/v1/workspaces -d '{
"id": "ws", "image": "mycloud:1.0",
"user_data": "#cloud-config\nusers:\n- name: alice\n..."
}'
The rules worth knowing:
- Create-time and immutable. The payload is part of the
workspace's identity;
user_datais accepted only at create. It is capped at 64 Ki characters (pydantic's 422 names the bound), an empty payload is a 400, and any mutation attempt answers a named 405 (delete and recreate to change it). cloud-init keys its run-once semantics off the workspace, so changing it after the fact would silently do nothing anyway. - The seed is per-workspace state. It is built at create (a
few hundred KiB of iso9660 overhead regardless of payload size,
cidata-labeled), attached read-only as the third disk, survivesstop/startand factory reset, and is deleted with the workspace. It can embed tokens, so the daemon stores it mode 0600 under the workspace's own directory — and creates its database file (which records the payload on the workspace's row) 0600 too. Listing endpoints andmsks ls --jsonecho the payload back over the same TLS + token channel as the console. - Both payload forms run. cloud-init executes
#!scripts fromuser_dataand applies cloud-config documents alike; declare the image'scapabilities.provisioner: cloud-initso operators and tooling can see what consumes the seed. - The host's timezone rides the seed (#633). A create on a
host with a timezone other than UTC captures it, and the seed's
script sets the guest's clock to it on first boot —
dateand logs inside the guest read the operator's wall clock with no operator action. The capture is create-time and immutable like the rest of the seed: a host timezone change later leaves existing workspaces on the zone their seed baked, and only new creates see the new zone. A host on UTC (or whose clock file reads as nothing recognizable) seeds no timezone — the guest keeps the image's own default. - A seedless boot needs a quiet create. A workspace created without a payload, a key, mounts, or a timezone to carry boots with two disks and no added cost: cloud-init finds no seed, applies nothing, and the interactive budget is unchanged (~3.0s start→shell).
- Failure posture is cloud-init's. The console getty
starts before cloud-init runs, so the interactive budget is
unaffected (the shipped image measures ~3.0s
start-to-shell). Payloads run in cloud-init's final stage:
a slow or hanging payload delays boot-complete, not the
shell.
cloud-init status --wait(or the serial log) says when provisioning finished; a factory reset re-runs the payload from the same seed.
The published image channel¶
Every release publishes the built guest images to a
hash-keyed channel on this repository's GitHub releases. Each
image lives on its own release tagged
images/<hash-prefix>, as a zstd-compressed archive
(<flavor>.tar.zst) whose sha256 the catalog records.
Importing the compressed form hashes exactly the shipped
bytes, and the daemon's import accepts the form directly (the
archive reader detects it). An unchanged
derivation publishes nothing: the workflow compares the archive
derivation's evaluation against the previous manifest and skips
the build entirely, and a channel slot that already exists is
never rewritten.
The version release carries images.json, the manifest that
names every published image — one row per flavor and arch —
ref, description, kernel, arch, size, hash, url,
and the source facts (build, built_from) — so fetching a
published image is one command. Pick the row whose arch field
matches the host's uname -m; the import then accepts it
without ceremony (and refuses a row for another architecture
with the same named error any other foreign-arch import gets):
# the URL from the manifest row
msks image import \
https://github.com/mcdonc/msks/releases/download/images/<hash>/debian.tar.zst
The description the listing and image info print is baked into
the image itself: the build writes one line into the layer's
image.json naming the distro, kernel, and shipped stack, and it
rides the import into any catalog. msks image ls carries it
(capped for the row), msks image info prints it at full length,
and the manifest rows carry it for the release channel.
The manifest answers from the moment the release exists: the release workflow attaches the previous release's manifest when it creates the version release, and replaces it with the new one when the image builds finish. While the builds run, the manifest names the previous images and an import pulls those; a release that changes no guest derivation publishes a manifest identical to the one it replaces.