jschoubben
5e83ac2c22
Jochen: a normal application has 3-5 ADRs, maybe 10 for a large one, and we are at 65. Fair, and the cause is mine -- I recorded every FINDING as a decision rather than every fork in the road. Two merges, both cases where one decision had been split across many records because it was taken over several days rather than at once. 0019 absorbs ten records about how this repository works: what it is and that it is public, the folder flow, the two design layers, the issue front door, status in frontmatter, playbooks, the naming rule, the product name. Those were never ten decisions -- they were one, seen from ten angles as the repository took shape. 0016 absorbs the five about the lab: a node is a virtual machine, a router is scenery, a scenario declares the underlay, a scenario is a closed address space, and the two scenario classes. Same pattern -- one design, split by the order it was worked out in. The consolidated 0019 also raises the bar for what earns a record, since that is what produced 65: a record is warranted when there is a genuine fork -- a direction reversed, an alternative that will be proposed again, something contested. A finding is not a decision, and a bug is certainly not. Everything else belongs in the design document where the reasoning is actually read. The checker earned its place here. Deleting nine records left 13 dangling links across the repository and it named every one, including in AGENTS.md. Nothing was found by reading. Remaining clusters worth the same treatment: the host (8 records), delivery (5), modules (6), connectivity (4), substrate and control plane (4). That would be 52 down to roughly 30.
7.0 KiB
layer, status, code, updated, decisions
| layer | status | code | updated | decisions | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| as-is | implemented |
|
2026-08-25 |
|
The lab, as it stands
The first piece of the new shape that exists. It is the only repository outside the monorepo so far, and unlike everything else planned it ships to nobody: it runs on a workstation, raises virtual machines, and throws them away.
Written from the implementation. Where intent and implementation disagree, the implementation is what is recorded here and the disagreement is stated.
What it does
A scenario is a YAML file declaring an underlay — segments, the gateways between them, and
machines placed on them. raise materialises it on incus; destroy removes it. In between,
exec runs a command inside a machine, and snapshot / restore capture and return the whole
scenario as one state.
| segments as isolated links | works |
| machines, multi-homed or detached | works |
| declared addresses, both families | works |
| segment MTU | works |
| gateways, NAT, masquerade | works |
published: ports, as DNAT through the gateway's address |
works |
mapping_ttl: as a conntrack timeout, read back after setting |
works |
forwardable: false — outbound only |
works |
policy: between segments, asymmetric |
works |
inbound: deny as a host firewall, read back after applying |
works |
| several public networks, routed through a transit router | works |
diagram — the scenario drawn, from the declaration or from the hypervisor |
works |
place: |
refused at raise |
| the lab's own certificate authority | not built |
What it does not do, and why that matters
place: is refused. A scenario can declare that a node host is placed on a machine; the
lab names the gap and refuses rather than raising a scenario that silently lacks what it
declared. Nothing can be placed because tier 0 does not exist yet.
The consequence is worth stating plainly rather than leaving to be inferred: the lab raises empty machines. It reproduces a network faithfully and puts nothing on it. It is infrastructure whose consumer has not been built, and it stays that way until tier 0 does.
The certificate story is designed and absent.
01-end-to-end-testing.md specifies the lab running
its own ACME issuer on the public segment, preserving production's two-CA split. None of that
is built.
What shipped differently from the design
The drawing was never designed. diagram renders a scenario as draw.io, from the
declaration or from the running instance, and it exists because it was asked for during the
build. It has tests and a decision record (ADR 0035,
proposed) but no document in the to-be layer. It is recorded here because it runs, not because
it was planned.
A router is tagged as a machine as well as a router. The design speaks of routers and
machines as distinct. In the implementation a router carries user.mesh-lab.machine too,
because destroy finds an instance's resources with one query and a router that carried only
router= was left behind — holding its networks open, so destroy reported removing zero
segments.
The router image is built once and cached. A scenario is a closed address space, so a
router has no route to a package repository and cannot install nftables at raise time. The
image is prepared once, with temporary connectivity. That is the only step in the whole lab
that needs the workstation to be online.
The rules that turned out to be load-bearing
Public segments must use documentation ranges (RFC 5737, RFC 3849), refused by the validator before anything is raised. Research 004 found why: the mesh decides public-versus-private by matching the address, so a private range on a segment meant to be routable makes the mesh silently never form.
A scenario is a closed address space. The workstation has no route in, so two instances
raised from one declaration hold the same addresses and never meet. Reachability is therefore
asked from inside — exec on one machine, testing another. The workstation's opinion would
be a different question with a misleadingly similar answer.
One public address is one gateway. Two gateway declarations sharing an address are one box, and their address lists union. Before this, gateways were grouped on their exact address list, and a household declaring a v6 address on one of its two segments became two router containers holding one address on one segment — which resolved to whichever answered ARP last.
What it costs
Measured on a workstation, not asserted:
| one machine | two machines | two machines and a router | |
|---|---|---|---|
| raise, to usable | 12.5 s | 14.6 s | 32 s |
| snapshot | 0.14 s | 0.28 s | — |
| restore, to usable again | 10.5 s | 11.6 s | — |
Machines boot concurrently, so a second machine costs seconds rather than doubling the wait. Nearly all the remaining time is boot.
These numbers depend entirely on a copy-on-write storage pool. On dir the same snapshot takes
9.9 s and a full copy of the disk, and a second did not finish in two minutes — so the lab's
check refuses rather than warns. A machine without copy-on-write runs scenarios correctly
and snapshots roughly 76× slower, which does not make the lab slow, it makes it unused.
How it is checked
npm run check — typecheck over source and tests, then the offline suite, then integration
against a real hypervisor. Mocking the hypervisor is forbidden
(ADR 0034, proposed): a test that fakes
the system under integration asserts that the fake behaves as expected.
Integration tests skip with a reason on a machine that cannot raise scenarios, rather than passing green having checked nothing.
Two things the suite does not yet do, recorded because their absence is invisible:
- It raises two of the five scenarios. Both faults found so far — two gateways holding one address, and a gateway drawn across an unrelated network — lived in scenarios nothing ever built. They were found by looking at pictures, not by running tests.
- Nothing opens the generated draw.io file. The tests assert on the XML and check the stencil names against draw.io's own library, but no test has ever opened one.
References
01-to-be/02-scenario-declaration.md— what a scenario declares.01-to-be/03-scenario-lifecycle.md— what happens to one.01-to-be/04-lab-installation.md— what the workstation needs.- Research 004 — the topology, and the address-range constraint the whole thing turns on.
- Research 010 — where the measured costs come from.