mesh-lab/README.md

# mesh-lab

The lab: a disposable Novox Mesh on one machine.

It ships to nobody. It runs on a workstation, raises virtual machines, puts things inside
them, and throws them away.

## Why it exists first

The node host takes over a machine's packages, services and network. It cannot be developed
against a machine anyone needs — so the place to develop it has to exist before it does.

That makes this repository **phase 0** of the migration, ahead of every tier it will later
test.

## Two classes of scenario

|  | **Bootstrap** | **Full** |
|---|---|---|
| Contains | virtual machines, the node host, a pinned foundation bundle | a complete mesh: forge, control plane, delivery, modules |
| Verdict from | what the host reports about the state it reconciled | a pipeline result ending in verify |
| Exercises | tiers 0 and 1 | tier 2 and above, and modules |
| Exists to | **develop the mesh** | **test what runs on it** |

The bootstrap scenario is a **strict subset** — same virtualisation, same networking, same
lifecycle, stopping before a control plane exists. The full scenario is reached by putting more
inside the machines, not by building a second thing.

## Using it

```
mesh-lab check                       can this machine run scenarios at all
mesh-lab validate scenarios/x.yml    parse and check, raising nothing
mesh-lab raise scenarios/x.yml       materialise it, wait until the machines are USABLE
mesh-lab list                        instances currently standing
mesh-lab exec <instance> <machine> -- <cmd...>
mesh-lab snapshot <instance> <label>
mesh-lab restore <instance> <label>
mesh-lab destroy <instance>
mesh-lab diagram scenarios/x.yml     draw what the scenario asks for
mesh-lab diagram --live <instance>   draw what is actually standing
```

`check` refuses rather than warns. A machine without copy-on-write storage runs scenarios
correctly and snapshots roughly 76× slower — which does not make the lab slow, it makes it
unused, and a warning about that is read once and ignored forever.

If the incus socket is not reachable as your user — the group was granted to a session that
already existed — set `MESH_LAB_INCUS="sudo -n incus"`.

## What a scenario declares

The **underlay**: what a hosting provider and a home router would provide, and nothing the
mesh is responsible for.

```yaml
segments:
  hosting:                          # one public network
    kind: public
    cidr: [192.0.2.0/24, "2001:db8:a::/48"]
  isp-home:                         # another, unrelated — routed to it, never bridged
    kind: public
    cidr: [198.51.100.0/24, "2001:db8:b::/48"]
  home:
    kind: private
    cidr: [192.168.1.0/24, "2001:db8:b:1::/64"]
    mtu: 1492
    gateway:
      to: isp-home
      address: [198.51.100.7]       # what the world sees this network as
      nat: [v4]                     # v4 translated, v6 routed
      forwardable: true
      mapping_ttl: 120s
machines:
  home-server:
    at: { segment: home, address: [192.168.1.135, "2001:db8:b:1::135"] }
    published: [{ port: 443, on: home }]
    inbound: allow
```

It declares **nothing** about overlay addresses, hubs, peering, names or certificates. Those
are what the mesh does, and a scenario that supplied them would be certifying its own work.

Public segments must use documentation ranges (RFC 5737, RFC 3849) and the validator refuses
anything else **before raising**. That is not pedantry: the mesh decides public-versus-private
by matching the address, so a private range on a segment meant to be routable makes the mesh
silently never form — no error, nothing to notice.

## Reaching in

Everything goes through incus, never over IP. A scenario is a closed address space, so two
instances raised from one declaration hold the same addresses and never meet — and the
workstation has no route into either.

So a reachability question is asked **from inside**: *can this machine reach that one* is
`exec` on the first, testing the second. The workstation's opinion would be a different
question with a misleadingly similar answer.

## What is implemented, and what is not

The declaration model is complete — it is the design's shape, and validating against it is
useful before any of it can be raised. **The runtime is not**, and the gap is refused rather
than ignored:

| | |
|---|---|
| segments as isolated links | **works** |
| machines, multi-homed or detached | **works** |
| declared addresses, both families | **works** |
| segment MTU | **works** |
| raise · exec · snapshot · restore · destroy · list | **works** |
| gateways, NAT, masquerade | **works** |
| `published:` ports (DNAT through the gateway's address) | **works** |
| `mapping_ttl:` (conntrack timeout) | **works**, and verified after setting — a declared expiry that silently did not apply would be the fault this catches |
| `forwardable: false` | **works** — outbound only, no DNAT, unsolicited inbound dropped |
| `policy:` between segments | **works**, asymmetric |
| `inbound: deny` | **works** — host firewall, read back after applying |
| several public networks, routed not bridged | **works** — a transit router, never a shared bridge |
| `place: [host]` | **works** — tier 0 is placed and asked what the machine is |
| `place:` anything above tier 0 | **refused, by name** — those tiers do not exist yet |

`raise` refuses a scenario declaring anything in the lower half, naming every gap. It does not
raise a mesh that silently lacks what it declared — that is the fault this lab exists to catch
(`novox/hq` 04-ISSUES/003: a firewall key declared in five manifests and read by no code, so a
manifest appears to restrict a port and restricts nothing).

`bootstrap-single.yml` places the host. The rest raise an underlay and put nothing on it,
which is still correct for what they test.

Placing needs a built host binary — set `MESH_LAB_HOST_BINARY` to one. It is an explicit path
rather than a search on purpose: the declaration design leaves *where `place:` gets its
artifacts from* open, and guessing would harden into the answer by accident.

## Pointing a run at the repositories

**Every variable is an explicit path, and none of them has a default.** A test whose artifact was
not pointed at *skips* — it does not fail — so an unset variable is a green run that proved
nothing. That is `novox/hq` 04-ISSUES/005 exactly, and it has now been rediscovered twice, so it
is written down here rather than reconstructed a third time.

```sh
export MESH_LAB_HOST_BINARY=<mesh-host>/mesh-host
export MESH_LAB_BUNDLE=<mesh-host>/examples/foundation-first-node.lock
export MESH_LAB_MODULES=<mesh-controller>/examples/modules
export MESH_LAB_BUILDER=<mesh-controller>/build/mesh-builder   # build/, which is git-ignored

# The installer. `suite` builds it with mesh-host's `make bootstrap`, which embeds a `docker save`
# of the control-plane image — so it is built AFTER that image, in the same run, or it carries a
# stale one sealed inside a binary where nothing would ever notice. The whole-mesh bed raises its
# anchor by RUNNING this, rather than by applying a foundation bundle itself (novox/hq ADR 0067).
export MESH_LAB_BOOTSTRAP_BINARY=<mesh-host>/mesh-bootstrap
export MESH_LAB_CONTROL_IMAGE=mesh-controller:development   # optional; what it carries

# A checkout of the mesh's catalogue. The installer reads the registry's and the builder's
# manifests from a copy of it ON THE MACHINE, because at genesis there is no forge, no build
# machine and — until the registry is up — nothing serving anything. The control plane's manifest
# is not among them: it comes out of the build the installer runs (novox/hq ADR 0069).
export MESH_LAB_CATALOG=<mesh-catalog>/modules

# Built with `go build -o <path> ./examples/<name>` in mesh-controller.
export MESH_LAB_PROVISIONER=<somewhere>/postgres-provisioner
export MESH_LAB_OBJECTSTORE_PROVISIONER=<somewhere>/objectstore-provisioner
export MESH_LAB_ROUTE_PROXY=<somewhere>/route-proxy
```

`MESH_LAB_HOST_BINARY` and `MESH_LAB_MODULES` do double duty: the repository each sits in is what
`suite` rebuilds and what the receipt claims. Point the run at a repository and it is built and
claimed; leave it out and it is neither.

Check before running a long suite — it says which of these are missing rather than skipping
quietly:

```sh
node --experimental-strip-types src/cli.ts check
```

If it says the daemon is not reachable, the group grant postdates the shell. `newgrp` fixes it,
but a heredoc into `newgrp` runs the suite as a child of a shell that then exits — start it with
`setsid nohup … &` inside the heredoc, or the run dies with the shell that launched it.

## When something takes too long

```sh
export MESH_LAB_LOG=info                 # or debug, or trace
export MESH_LAB_LOG_FILE=/tmp/lab.log    # unset writes to stderr
```

| level | what it adds |
|---|---|
| `info` | each step of a raise, with how long the previous one took; anything that failed; **and a line every 15s naming whatever is still running** |
| `debug` | every command the lab runs — incus, docker, and anything local — with its duration, and the stderr of anything that failed |
| `trace` | what those commands printed |

**The heartbeat is the point.** A stall is a command that started and has not finished, and the
only thing separating it from ordinary work is how long it has been going — which nothing can tell
you unless something is still counting. At `info` a run says `… docker load -i /tmp/image.tar —
still running after 45s` while it happens, rather than nothing until it gives up.

Written with `appendFileSync`, so unlike a redirected stdout it cannot lag behind the run. It is
off unless asked for.

**A redirected log lags, so do not diagnose a stall from it.** Node block-buffers stdout when it
is a file rather than a terminal, so `> run.log` can sit unchanged for minutes while the run is
working normally. On 2026-09-01 that was read as a stall twice, once after a real stall had just
been fixed — the most expensive kind of false signal, because it argues the fix did not work. Ask
the machines instead:

```sh
incus list -c ns
incus exec <instance>-anchor -- systemctl is-active docker
```

*"I cannot see progress" is not evidence of no progress.*

## Measured on a workstation

| | one machine | two machines | two machines + a router |
|---|---|---|---|
| raise, to usable | 12.5 s | 14.6 s | 32 s |
| snapshot | 0.14 s | 0.28 s | — |
| restore, to usable again | 10.5 s | 11.6 s | — |

A router adds seconds, not a boot: it is a container, because it is scenery rather than
something under test (`novox/hq` ADR 0016).

**Verified by running**, not asserted — a machine at `192.168.1.135` behind a household
gateway, reached from a machine on a routable address:

```
home-server -> anchor                        0% loss, through masquerade
anchor -> 192.168.1.135  (private, direct)   unreachable ✓
anchor -> 192.0.2.50:8080 (the GATEWAY)      HTTP 200
home -> devices  (policy allow)              reachable ✓
devices -> home  (policy deny)               blocked ✓
roamer behind unforwardable NAT -> anchor    reachable ✓  (outbound only)
anchor -> roamer                             unreachable ✓
workstation with inbound: deny, dialling out reachable ✓  (defended, not disconnected)
home-server -> workstation                   refused ✓
```

The third line is the case research 004 says only exists in production.

**Routed, never bridged**, proven rather than asserted — ping TTL across the full topology:

```
within one segment                           ttl=64   no hops
across two unrelated public networks         ttl=62   gateway + transit
multicast between public networks            0 replies
```

A flat "internet" would have shown ttl=64 and answered multicast, which would have let a node
discover a peer it could never reach in production — and report success.

Machines boot concurrently, so a second machine costs seconds rather than doubling the wait.
Nearly all of the remaining time is boot, which cannot be avoided.

These numbers depend entirely on a copy-on-write pool. On `dir` the same snapshot takes 9.9 s
and a full copy of the disk, and a second one did not finish in two minutes — which is why
`check` refuses rather than warns.

## Drawing one

```
mesh-lab diagram scenarios/the-ordinary-shape.yml   what the declaration asks for
mesh-lab diagram --live <instance>                  what the hypervisor actually holds
```

Both produce draw.io files, laid out the same way — public networks at the top, each private
one below the network it sits behind. Drawing both sources through one layout is the point: a
difference between what was asked for and what exists becomes a difference you can *see*.

Two kinds of symbol, and the split matters:

- the **shape** says what a resource is, and is fixed per kind — a server is always the server
  shape, a gateway always the router shape, whatever else is true about it;
- the **badges** say what is true about that particular one, and come entirely from metadata:
  `N` translated, `F` forwarding (green yes, red no), `T` mappings expire, `D` refuses inbound,
  `C` container, `VM` virtual machine, `▶` running. Each carries the full sentence as a
  tooltip, because a one-letter code with no explanation is a private language.

Badges exist because the interesting properties of a network are exactly the ones with no
visual consequence. An address that is translated looks identical to one that is not, until
traffic proves otherwise.

The live drawing reads **only** the hypervisor — the same tags `destroy` uses — and never
re-opens the scenario file. A picture built from the declaration and labelled *as raised*
would report the request as though it were the result, which is the whole failure the pairing
exists to expose. So `raise` records what it applied: a segment's kind and ranges on the link,
a gateway's translation, forwardability and mapping expiry on the gateway, and `inbound: deny`
on the machine.

**Every behavioural tag is written after the thing works, never before.** A tag written when
the resource is created would restate the request; a failed raise leaves its wreckage standing
on purpose, so a picture of that wreckage would badge translation the gateway was never
configured to do. The gateway is tagged after its ruleset is applied, and the machine after
the read-back proves its firewall loaded.

That pairing has already earned itself. Drawn side by side, the live picture showed every
virtual machine holding no addresses at all: a container's interface carries the device's
name, a virtual machine names its own, and joining them by name silently dropped one whole
class of machine. The two pictures disagreed, so the bug was visible in seconds.

## Where the reasoning lives

Design and decisions are in [`novox/hq`](https://git.novox.be/novox/hq), not here:

- `03-DESIGN/01-to-be/02-scenario-declaration.md` — what a scenario declares
- `03-DESIGN/01-to-be/03-scenario-lifecycle.md` — what happens to one
- `02-DECISIONS/0031-the-lab-provides-the-underlay.md`
- `02-DECISIONS/0032-a-scenario-is-an-isolated-address-space.md`

This repository carries implementation. It does not carry decisions.

## Development

No build step — Node strips the types.

```
npm test                   the declaration layer and the diagram, offline
npm run test:integration   real scenarios against a real hypervisor
npm run typecheck          source and tests both — a test that does not compile is a test
                           that silently never ran
npm run check              typecheck + both suites — this is the gate
npm run last-run           when this machine last ran the suite, and whether that still counts
```

**The integration suite rebuilds what it tests, and leaves a receipt saying it ran.**

It needs a hypervisor, so it cannot run on every push — which means it runs when somebody
remembers, and *remembering is not a mechanism*. The harness this replaces had not built for two
and a half months and nothing said so (`novox/hq` 04-ISSUES/005). So:

- **Before the run**, the host binary, the control-plane image and the builder are rebuilt from
  source. The last two both parse manifests; building one and not the other is how a rename gets
  tested against an eleven-hour-old binary.
- **After the run**, a receipt is written to XDG state — outside the repository, because the
  question is *has this machine run it*, and a receipt in git would be a claim about everybody's
  machine made by whoever committed last.
- `last-run` judges it and exits non-zero when it no longer counts: old, failed, taken against
  commits the repositories have moved past, taken against a tree with uncommitted work, or a run
  that never included the end-to-end file.
  **A receipt that says nothing about something is not a receipt that clears it.**

`suite <paths>` runs something narrower, and the receipt records that it did — a green run of the
unit tests must not be readable as coverage of the pipeline. `--no-build` skips the rebuild, for
iterating on a test rather than on the code under it.

**A test names the decision it defends** (`novox/hq` ADR 0017). A decision with no test is one
that will quietly stop being true, and nobody learns that from a document:

| Test | Defends |
|---|---|
| the lab provides the underlay and nothing of the overlay | ADR 0016 |
| the workstation has no route into the scenario | ADR 0016 |
| a router is a container while machines are virtual machines | ADR 0016 |
| raise waits for *usable*, not for the call to return | the lifecycle design |
| snapshots are whole-scenario | the lifecycle design |
| a public range that is not documentation space is refused | the declaration design |
| a scenario declaring what cannot be materialised is refused | the declaration design |
| the live diagram distinguishes scenery from a node | ADR 0016 |
| the live diagram draws what exists, never what was asked for | the diagram design |
| a picture nobody can open is not a picture | the diagram design |
| a run that raised no machines is not end-to-end coverage | 04-ISSUES/005 |
| a run whose result could not be read writes nothing | 04-ISSUES/005 |
| the control plane's image and builder are always built together | 04-ISSUES/005 |
| every repository the receipt claims was built by the run | 04-ISSUES/005 |
| a run against uncommitted work does not cover the commit | 04-ISSUES/005 |

**Mocking the hypervisor is forbidden.** A fake would assert that the fake behaves as expected,
which is the shape of test this project exists to stop shipping. Integration tests skip with a
reason on a machine that cannot raise scenarios, rather than passing green having checked
nothing.

That suite earned itself on its first run: it found that a snapshot of a running machine could
miss a file written seconds earlier — not stale, **absent** — because the write was still in
the guest's page cache. The design had listed that as an open question. The test answered it,
and `snapshot` now flushes first.