The scenario model, the lifecycle, and what the lab actually costs #6

2026-08-23T22:40:48Z

jschoubben commented

2026-08-23 22:40:48 +00:00

Ten commits. The lab design taken from a sketch to something measured.

The scenario declaration grew a real network model. forwarded: [443] was the tell — it implied a NAT rule while never saying from which address, and the address is the whole point. Now: three positions a machine can be in, keyed on forwardability rather than ownership (carrier-grade NAT is your own connection and still unforwardable); gateways carrying the address the world sees them as; IPv6, where a machine is typically unforwardable on v4 and directly attached on v6 at once, making reachability a property of (machine, family).

Two modelling errors caught in review, both of which would have produced labs that pass while production fails:

A segment with no gateway was read as the internet, making an isolated network inexpressible.
Every public address sat in one /24, which would have made ARP adjacency, non-decrementing TTL and crossing multicast true in the lab. The as-is layer already records that mesh names are deliberately not multicast names — a flat "internet" would let a node discover a peer it could never reach in production, and report success.

ADR 0031 — the lab provides the underlay, the mesh builds the overlay. A scenario that pre-builds peering certifies its own work.
ADR 0032 — a scenario is a closed address space, so the lab never reaches in over IP. Two scenarios carrying the same prefix would otherwise put one's traffic in the other.

Research 010 — measured, not argued. dir snapshots: 9.9 s and 1.6 GB, second one unfinished at 120 s. btrfs: 0.13 s, three sharing 1.36 GB. Restore 10.4 s → 0.80 s. The four-machine reset cycle falls from ~90 s to ~15 s, nearly all boot. ADR 0029's inner-loop argument holds with copy-on-write and did not without it. Fix shipped as hal PR #973 and verified on this node.

That measurement also surfaced a failure shape the mesh hasn't catalogued: not reported success and did nothing, but reported success and did it 76× slower — which no error surface catches because nothing is wrong.

Ten commits. The lab design taken from a sketch to something measured. **The scenario declaration** grew a real network model. `forwarded: [443]` was the tell — it implied a NAT rule while never saying from which address, and the address is the whole point. Now: three positions a machine can be in, keyed on **forwardability rather than ownership** (carrier-grade NAT is your own connection and still unforwardable); gateways carrying the address the world sees them as; IPv6, where a machine is typically *unforwardable on v4 and directly attached on v6 at once*, making reachability a property of **(machine, family)**. **Two modelling errors caught in review**, both of which would have produced labs that pass while production fails: - A segment with no gateway was read as *the internet*, making an isolated network inexpressible. - Every public address sat in one `/24`, which would have made ARP adjacency, non-decrementing TTL and **crossing multicast** true in the lab. The as-is layer already records that mesh names are deliberately not multicast names — a flat "internet" would let a node discover a peer it could never reach in production, and report success. **ADR 0031** — the lab provides the underlay, the mesh builds the overlay. A scenario that pre-builds peering certifies its own work. **ADR 0032** — a scenario is a closed address space, so the lab never reaches in over IP. Two scenarios carrying the same prefix would otherwise put one's traffic in the other. **Research 010 — measured, not argued.** `dir` snapshots: 9.9 s and 1.6 GB, second one unfinished at 120 s. btrfs: 0.13 s, three sharing 1.36 GB. Restore 10.4 s → 0.80 s. The four-machine reset cycle falls from ~90 s to ~15 s, nearly all boot. ADR 0029's inner-loop argument holds *with* copy-on-write and did not without it. Fix shipped as hal PR #973 and verified on this node. That measurement also surfaced a failure shape the mesh hasn't catalogued: not *reported success and did nothing*, but **reported success and did it 76× slower** — which no error surface catches because nothing is wrong.

jschoubben referenced this issue from a commit

2026-08-23 22:40:54 +00:00

Merge pull request 'The scenario model, the lifecycle, and what the lab actually costs' (#6) from design/scenario-underlay-detail into main

jschoubben closed this pull request

2026-08-23 22:40:55 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/hq#6