novox/mesh-lab
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
Files
5c91c0ecd288967d1cdc4c8b9b749b104d169a20
mesh-lab/README.md
T
jschoubben 5c91c0ecd2 Scenarios: egress where images are needed, and images: cut to what is ours 
Every scenario that places a container runtime gives each of its machines
`egress: true` — a node that runs modules pulls images from the internet, which is what
a node does. The underlay-only scenarios (bootstrap-single, behind-nat,
segmented-and-unforwardable, the-ordinary-shape, two-on-a-segment) stay sealed on
purpose: an extra NIC would change the very reachability they are asserting about.

`images:` keeps only the mesh's own — 55 third-party entries leave whole-mesh-full
alone, and the machine fetches them itself by the digest its module.json already pins.
The whole-mesh beds also say per machine which of ours they get: novox the substrate
control plane and its own fifteen runtimes, ace its twenty-four, the two workstations
one each. That is not a lab economy. An operator's workstation holds the images its own
modules need, and giving these two the union would put some thirty gigabytes onto a
thirty-gigabyte disk.

bootstrap-with-registry.yml is deleted. It existed only to demonstrate the lab's
registry, nothing referenced it, and there is nothing left for it to demonstrate.

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
2026-09-10 23:16:23 +02:00

365 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# mesh-lab
The lab: a disposable Novox Mesh on one machine.
It ships to nobody. It runs on a workstation, raises virtual machines, puts things inside
them, and throws them away.
## Why it exists first
The node host takes over a machine's packages, services and network. It cannot be developed
against a machine anyone needs — so the place to develop it has to exist before it does.
That makes this repository **phase 0** of the migration, ahead of every tier it will later
test.
## Two classes of scenario
| | **Bootstrap** | **Full** |
|---|---|---|
| Contains | virtual machines, the node host, a pinned substrate bundle | a complete mesh: forge, control plane, delivery, modules |
| Verdict from | what the host reports about the state it reconciled | a pipeline result ending in verify |
| Exercises | tiers 0 and 1 | tier 2 and above, and modules |
| Exists to | **develop the mesh** | **test what runs on it** |
The bootstrap scenario is a **strict subset** — same virtualisation, same networking, same
lifecycle, stopping before a control plane exists. The full scenario is reached by putting more
inside the machines, not by building a second thing.
## Using it
```
mesh-lab check can this machine run scenarios at all
mesh-lab validate scenarios/x.yml parse and check, raising nothing
mesh-lab raise scenarios/x.yml materialise it, wait until the machines are USABLE
mesh-lab list instances currently standing
mesh-lab exec <instance> <machine> -- <cmd...>
mesh-lab snapshot <instance> <label>
mesh-lab restore <instance> <label>
mesh-lab destroy <instance>
mesh-lab diagram scenarios/x.yml draw what the scenario asks for
mesh-lab diagram --live <instance> draw what is actually standing
```
`check` refuses rather than warns. A machine without copy-on-write storage runs scenarios
correctly and snapshots roughly 76× slower — which does not make the lab slow, it makes it
unused, and a warning about that is read once and ignored forever.
If the incus socket is not reachable as your user — the group was granted to a session that
already existed — set `MESH_LAB_INCUS="sudo -n incus"`.
## What a scenario declares
The **underlay**: what a hosting provider and a home router would provide, and nothing the
mesh is responsible for.
```yaml
segments:
hosting: # one public network
kind: public
cidr: [192.0.2.0/24, "2001:db8:a::/48"]
isp-home: # another, unrelated — routed to it, never bridged
kind: public
cidr: [198.51.100.0/24, "2001:db8:b::/48"]
home:
kind: private
cidr: [192.168.1.0/24, "2001:db8:b:1::/64"]
mtu: 1492
gateway:
to: isp-home
address: [198.51.100.7] # what the world sees this network as
nat: [v4] # v4 translated, v6 routed
forwardable: true
mapping_ttl: 120s
machines:
home-server:
at: { segment: home, address: [192.168.1.135, "2001:db8:b:1::135"] }
published: [{ port: 443, on: home }]
inbound: allow
```
It declares **nothing** about overlay addresses, hubs, peering, names or certificates. Those
are what the mesh does, and a scenario that supplied them would be certifying its own work.
Public segments must use documentation ranges (RFC 5737, RFC 3849) and the validator refuses
anything else **before raising**. That is not pedantry: the mesh decides public-versus-private
by matching the address, so a private range on a segment meant to be routable makes the mesh
silently never form — no error, nothing to notice.
## Reaching in
Everything goes through incus, never over IP. A scenario is a closed address space, so two
instances raised from one declaration hold the same addresses and never meet — and the
workstation has no route into either.
So a reachability question is asked **from inside**: *can this machine reach that one* is
`exec` on the first, testing the second. The workstation's opinion would be a different
question with a misleadingly similar answer.
## What is implemented, and what is not
The declaration model is complete — it is the design's shape, and validating against it is
useful before any of it can be raised. **The runtime is not**, and the gap is refused rather
than ignored:
| | |
|---|---|
| segments as isolated links | **works** |
| machines, multi-homed or detached | **works** |
| declared addresses, both families | **works** |
| segment MTU | **works** |
| raise · exec · snapshot · restore · destroy · list | **works** |
| gateways, NAT, masquerade | **works** |
| `published:` ports (DNAT through the gateway's address) | **works** |
| `mapping_ttl:` (conntrack timeout) | **works**, and verified after setting — a declared expiry that silently did not apply would be the fault this catches |
| `forwardable: false` | **works** — outbound only, no DNAT, unsolicited inbound dropped |
| `policy:` between segments | **works**, asymmetric |
| `inbound: deny` | **works** — host firewall, read back after applying |
| several public networks, routed not bridged | **works** — a transit router, never a shared bridge |
| `place: [host]` | **works** — tier 0 is placed and asked what the machine is |
| `place:` anything above tier 0 | **refused, by name** — those tiers do not exist yet |
`raise` refuses a scenario declaring anything in the lower half, naming every gap. It does not
raise a mesh that silently lacks what it declared — that is the fault this lab exists to catch
(`novox/hq` 04-ISSUES/003: a firewall key declared in five manifests and read by no code, so a
manifest appears to restrict a port and restricts nothing).
`bootstrap-single.yml` places the host. The rest raise an underlay and put nothing on it,
which is still correct for what they test.
Placing needs a built host binary — set `MESH_LAB_HOST_BINARY` to one. It is an explicit path
rather than a search on purpose: the declaration design leaves *where `place:` gets its
artifacts from* open, and guessing would harden into the answer by accident.
## Pointing a run at the repositories
**Every variable is an explicit path, and none of them has a default.** A test whose artifact was
not pointed at *skips* — it does not fail — so an unset variable is a green run that proved
nothing. That is `novox/hq` 04-ISSUES/005 exactly, and it has now been rediscovered twice, so it
is written down here rather than reconstructed a third time.
```sh
export MESH_LAB_HOST_BINARY=<mesh-host>/mesh-host
export MESH_LAB_BUNDLE=<mesh-host>/examples/substrate-first-node.lock
export MESH_LAB_MODULES=<mesh-control>/examples/modules
export MESH_LAB_BUILDER=<mesh-control>/build/mesh-builder # build/, which is git-ignored
# Built with `go build -o <path> ./examples/<name>` in mesh-control.
export MESH_LAB_PROVISIONER=<somewhere>/postgres-provisioner
export MESH_LAB_OBJECTSTORE_PROVISIONER=<somewhere>/objectstore-provisioner
export MESH_LAB_ROUTE_PROXY=<somewhere>/route-proxy
```
`MESH_LAB_HOST_BINARY` and `MESH_LAB_MODULES` do double duty: the repository each sits in is what
`suite` rebuilds and what the receipt claims. Point the run at a repository and it is built and
claimed; leave it out and it is neither.
Check before running a long suite — it says which of these are missing rather than skipping
quietly:
```sh
node --experimental-strip-types src/cli.ts check
```
If it says the daemon is not reachable, the group grant postdates the shell. `newgrp` fixes it,
but a heredoc into `newgrp` runs the suite as a child of a shell that then exits — start it with
`setsid nohup … &` inside the heredoc, or the run dies with the shell that launched it.
## When something takes too long
```sh
export MESH_LAB_LOG=info # or debug, or trace
export MESH_LAB_LOG_FILE=/tmp/lab.log # unset writes to stderr
```
| level | what it adds |
|---|---|
| `info` | each step of a raise, with how long the previous one took; anything that failed; **and a line every 15s naming whatever is still running** |
| `debug` | every command the lab runs — incus, docker, and anything local — with its duration, and the stderr of anything that failed |
| `trace` | what those commands printed |
**The heartbeat is the point.** A stall is a command that started and has not finished, and the
only thing separating it from ordinary work is how long it has been going — which nothing can tell
you unless something is still counting. At `info` a run says `… docker load -i /tmp/image.tar —
still running after 45s` while it happens, rather than nothing until it gives up.
Written with `appendFileSync`, so unlike a redirected stdout it cannot lag behind the run. It is
off unless asked for.
**A redirected log lags, so do not diagnose a stall from it.** Node block-buffers stdout when it
is a file rather than a terminal, so `> run.log` can sit unchanged for minutes while the run is
working normally. On 2026-09-01 that was read as a stall twice, once after a real stall had just
been fixed — the most expensive kind of false signal, because it argues the fix did not work. Ask
the machines instead:
```sh
incus list -c ns
incus exec <instance>-anchor -- systemctl is-active docker
```
*"I cannot see progress" is not evidence of no progress.*
## Measured on a workstation
| | one machine | two machines | two machines + a router |
|---|---|---|---|
| raise, to usable | 12.5 s | 14.6 s | 32 s |
| snapshot | 0.14 s | 0.28 s | — |
| restore, to usable again | 10.5 s | 11.6 s | — |
A router adds seconds, not a boot: it is a container, because it is scenery rather than
something under test (`novox/hq` ADR 0016).
**Verified by running**, not asserted — a machine at `192.168.1.135` behind a household
gateway, reached from a machine on a routable address:
```
home-server -> anchor 0% loss, through masquerade
anchor -> 192.168.1.135 (private, direct) unreachable ✓
anchor -> 192.0.2.50:8080 (the GATEWAY) HTTP 200
home -> devices (policy allow) reachable ✓
devices -> home (policy deny) blocked ✓
roamer behind unforwardable NAT -> anchor reachable ✓ (outbound only)
anchor -> roamer unreachable ✓
workstation with inbound: deny, dialling out reachable ✓ (defended, not disconnected)
home-server -> workstation refused ✓
```
The third line is the case research 004 says only exists in production.
**Routed, never bridged**, proven rather than asserted — ping TTL across the full topology:
```
within one segment ttl=64 no hops
across two unrelated public networks ttl=62 gateway + transit
multicast between public networks 0 replies
```
A flat "internet" would have shown ttl=64 and answered multicast, which would have let a node
discover a peer it could never reach in production — and report success.
Machines boot concurrently, so a second machine costs seconds rather than doubling the wait.
Nearly all of the remaining time is boot, which cannot be avoided.
These numbers depend entirely on a copy-on-write pool. On `dir` the same snapshot takes 9.9 s
and a full copy of the disk, and a second one did not finish in two minutes — which is why
`check` refuses rather than warns.
## Drawing one
```
mesh-lab diagram scenarios/the-ordinary-shape.yml what the declaration asks for
mesh-lab diagram --live <instance> what the hypervisor actually holds
```
Both produce draw.io files, laid out the same way — public networks at the top, each private
one below the network it sits behind. Drawing both sources through one layout is the point: a
difference between what was asked for and what exists becomes a difference you can *see*.
Two kinds of symbol, and the split matters:
- the **shape** says what a resource is, and is fixed per kind — a server is always the server
shape, a gateway always the router shape, whatever else is true about it;
- the **badges** say what is true about that particular one, and come entirely from metadata:
`N` translated, `F` forwarding (green yes, red no), `T` mappings expire, `D` refuses inbound,
`C` container, `VM` virtual machine, `▶` running. Each carries the full sentence as a
tooltip, because a one-letter code with no explanation is a private language.
Badges exist because the interesting properties of a network are exactly the ones with no
visual consequence. An address that is translated looks identical to one that is not, until
traffic proves otherwise.
The live drawing reads **only** the hypervisor — the same tags `destroy` uses — and never
re-opens the scenario file. A picture built from the declaration and labelled *as raised*
would report the request as though it were the result, which is the whole failure the pairing
exists to expose. So `raise` records what it applied: a segment's kind and ranges on the link,
a gateway's translation, forwardability and mapping expiry on the gateway, and `inbound: deny`
on the machine.
**Every behavioural tag is written after the thing works, never before.** A tag written when
the resource is created would restate the request; a failed raise leaves its wreckage standing
on purpose, so a picture of that wreckage would badge translation the gateway was never
configured to do. The gateway is tagged after its ruleset is applied, and the machine after
the read-back proves its firewall loaded.
That pairing has already earned itself. Drawn side by side, the live picture showed every
virtual machine holding no addresses at all: a container's interface carries the device's
name, a virtual machine names its own, and joining them by name silently dropped one whole
class of machine. The two pictures disagreed, so the bug was visible in seconds.
## Where the reasoning lives
Design and decisions are in [`novox/hq`](https://git.novox.be/novox/hq), not here:
- `03-DESIGN/01-to-be/02-scenario-declaration.md` — what a scenario declares
- `03-DESIGN/01-to-be/03-scenario-lifecycle.md` — what happens to one
- `02-DECISIONS/0031-the-lab-provides-the-underlay.md`
- `02-DECISIONS/0032-a-scenario-is-an-isolated-address-space.md`
This repository carries implementation. It does not carry decisions.
## Development
No build step — Node strips the types.
```
npm test the declaration layer and the diagram, offline
npm run test:integration real scenarios against a real hypervisor
npm run typecheck source and tests both — a test that does not compile is a test
that silently never ran
npm run check typecheck + both suites — this is the gate
npm run last-run when this machine last ran the suite, and whether that still counts
```
**The integration suite rebuilds what it tests, and leaves a receipt saying it ran.**
It needs a hypervisor, so it cannot run on every push — which means it runs when somebody
remembers, and *remembering is not a mechanism*. The harness this replaces had not built for two
and a half months and nothing said so (`novox/hq` 04-ISSUES/005). So:
- **Before the run**, the host binary, the control-plane image and the builder are rebuilt from
source. The last two both parse manifests; building one and not the other is how a rename gets
tested against an eleven-hour-old binary.
- **After the run**, a receipt is written to XDG state — outside the repository, because the
question is *has this machine run it*, and a receipt in git would be a claim about everybody's
machine made by whoever committed last.
- `last-run` judges it and exits non-zero when it no longer counts: old, failed, taken against
commits the repositories have moved past, taken against a tree with uncommitted work, or a run
that never included the end-to-end file.
**A receipt that says nothing about something is not a receipt that clears it.**
`suite <paths>` runs something narrower, and the receipt records that it did — a green run of the
unit tests must not be readable as coverage of the pipeline. `--no-build` skips the rebuild, for
iterating on a test rather than on the code under it.
**A test names the decision it defends** (`novox/hq` ADR 0017). A decision with no test is one
that will quietly stop being true, and nobody learns that from a document:
| Test | Defends |
|---|---|
| the lab provides the underlay and nothing of the overlay | ADR 0016 |
| the workstation has no route into the scenario | ADR 0016 |
| a router is a container while machines are virtual machines | ADR 0016 |
| raise waits for *usable*, not for the call to return | the lifecycle design |
| snapshots are whole-scenario | the lifecycle design |
| a public range that is not documentation space is refused | the declaration design |
| a scenario declaring what cannot be materialised is refused | the declaration design |
| the live diagram distinguishes scenery from a node | ADR 0016 |
| the live diagram draws what exists, never what was asked for | the diagram design |
| a picture nobody can open is not a picture | the diagram design |
| a run that raised no machines is not end-to-end coverage | 04-ISSUES/005 |
| a run whose result could not be read writes nothing | 04-ISSUES/005 |
| the control plane's image and builder are always built together | 04-ISSUES/005 |
| every repository the receipt claims was built by the run | 04-ISSUES/005 |
| a run against uncommitted work does not cover the commit | 04-ISSUES/005 |
**Mocking the hypervisor is forbidden.** A fake would assert that the fake behaves as expected,
which is the shape of test this project exists to stop shipping. Integration tests skip with a
reason on a machine that cannot raise scenarios, rather than passing green having checked
nothing.
That suite earned itself on its first run: it found that a snapshot of a running machine could
miss a file written seconds earlier — not stale, **absent** — because the write was still in
the guest's page cache. The design had listed that as an open question. The test answered it,
and `snapshot` now flushes first.
Reference in New Issue View Git Blame Copy Permalink