jschoubben
e88b448145
Snapshot 9.9s -> 0.13s. Restore 10.4s -> 0.80s. Three snapshots sharing 1.36 GB instead of costing 4.8 GB. The projected four-machine reset cycle falls from ~90s, unbounded at worst, to ~15s dominated by a boot that cannot be avoided. ADR 0029's inner-loop argument holds with copy-on-write and did not without it. The consistency matters as much as the speed: three consecutive snapshots took 0.13, 0.12 and 0.13 seconds, against a dir second snapshot that never finished. One honest counter-observation recorded: launching onto the fresh copy-on-write pool was slower, 20.2s against 14.3s, because the image had to be unpacked into a pool that had never seen it. Paid once per pool, and dwarfed by what snapshotting saves, but it went the other way. Doing the measurement produced the answer to how the lab installs on a clean machine, because both failure modes appeared while doing it. Installed is not available: the daemon was present with units disabled and no group. Issue 007. Available is not adequate, and this is worse: with the storage tooling absent everything worked and snapshots were seventy-six times slower. Nothing failed, nothing warned. That is a variant the mesh has not catalogued — its usual failure is reported success and did nothing; this is reported success and did it seventy-six times slower, which no error surface catches because nothing is wrong. So the lab verifies CAPABILITY, never installation, and refuses to run degraded rather than warning — a warning about a slow inner loop is read once and ignored forever. Prerequisites may arrive from a mesh module or from the lab's own bootstrap, and the second path is required rather than convenient: a lab installable only by a mesh cannot host the development of the mesh that installs it. The lab is the second thing installed by hand, after the node host, and for the same reason: something has to be first, and pretending otherwise produces a circularity papered over by a script nobody exercises.
61 lines
3.5 KiB
Markdown
61 lines
3.5 KiB
Markdown
---
|
|
status: active
|
|
initiated: 2026-08-24
|
|
touches:
|
|
- 03-DESIGN/01-to-be/03-scenario-lifecycle.md
|
|
- 02-DECISIONS/0016-a-lab-node-is-a-virtual-machine.md
|
|
- 02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md
|
|
became: []
|
|
---
|
|
|
|
# 010 — What the lab's inner loop actually costs
|
|
|
|
## What is being investigated
|
|
|
|
[The lifecycle design](../../03-DESIGN/01-to-be/03-scenario-lifecycle.md) closes on an open
|
|
question that is measurable rather than arguable:
|
|
|
|
> **What a snapshot costs.** Whole-scenario snapshots of several machines are the operation the
|
|
> inner loop repeats most, so their cost sets the loop's speed. If restoring is slow, the loop
|
|
> is slow, and everything above is theory.
|
|
|
|
Measured on a workstation, 2026-08-24. Numbers in [`measurements.md`](measurements.md).
|
|
|
|
## Why it matters
|
|
|
|
[ADR 0029](../../02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md) makes the
|
|
bootstrap scenario the inner development loop for tiers 0 and 1 — the argument being that
|
|
raising a node from nothing stops being the least-exercised path and becomes the most-exercised
|
|
one. **That argument is only true if raising and resetting are cheap.** A loop that costs
|
|
minutes is a loop people avoid, and the least-exercised path stays least-exercised.
|
|
|
|
## Status
|
|
|
|
Measured, and the finding is a blocker rather than a data point.
|
|
|
|
**A snapshot on the current host is a full copy of the machine's disk.** 1.6 GB and ten seconds
|
|
for one small virtual machine at best — and over two minutes when observed a second time. Cost
|
|
scales with the number of machines and the size of their disks, not with what changed.
|
|
|
|
**With copy-on-write it is 0.13 seconds and costs the delta.** Verified, not assumed. The
|
|
projected four-machine reset cycle falls from roughly ninety seconds to roughly fifteen, of
|
|
which almost all is a boot that cannot be avoided.
|
|
|
|
The cause is not virtual machines and not incus. It is that the host offers incus exactly one
|
|
storage driver, `dir`, which has no copy-on-write and therefore no cheap snapshot. The kernel
|
|
supports btrfs; the userspace tool that would let incus use it is simply not installed.
|
|
|
|
So the question *"is the lab's inner loop fast enough"* currently answers itself the wrong way,
|
|
for a reason that is one declared package away from being fixed — and declaring packages is
|
|
something the mesh already does.
|
|
|
|
## Open questions
|
|
|
|
| Question | Why it matters |
|
|
|---|---|
|
|
| ~~How much does a copy-on-write pool actually improve it?~~ **Measured: snapshot 9.9 s → 0.13 s, restore 10.4 s → 0.80 s, three snapshots sharing 1.36 GB rather than costing 4.8 GB.** The projected four-machine cycle falls from ~90 s to ~15 s. | Answered. The inner-loop argument holds *with* copy-on-write and did not without it. |
|
|
| How does the lab install its own prerequisites on a clean machine? | The lab needs a virtualisation daemon, copy-on-write tooling and a pool before it can do anything — and it cannot depend on the mesh for them, since it is where the mesh is built. |
|
|
| Why was the second snapshot more than twelve times slower than the first? | If snapshot cost is unpredictable rather than merely high, that is worse — a loop with a variable multi-minute step is one nobody trusts. |
|
|
| Does a scenario snapshot need the machines stopped? | Stateless snapshots of a running virtual machine capture the disk but not memory. Whether a mesh restored that way is coherent is not established. |
|
|
| What is the cost at scenario scale — four machines rather than one? | Only single-machine numbers were taken. If the operation is serial, four machines is four times the wait. |
|