novox/hq
3
0
Fork 0
Code Issues 5 Pull Requests 4 Releases Wiki Activity

The scenario model, the lifecycle, and what the lab actually costs #6

Merged
jschoubben merged 9 commits from design/scenario-underlay-detail into main 2026-08-23 22:40:55 +00:00
Conversation 0 Commits 9 Files Changed 7
+122 -45
1 changed files with 122 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit e65e5809dc - Show all commits
03-DESIGN/01-to-be/02-scenario-declaration.md
+122 -45
View File
@@ -17,31 +17,60 @@ everything in the lab hangs off, so it is worth getting small.
It states what a hosting provider and a home router would provide, and nothing the mesh is It states what a hosting provider and a home router would provide, and nothing the mesh is
responsible for ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)). responsible for ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
## Three positions a machine can be in
The underlay's whole job is to reproduce **where a machine sits relative to the internet**,
because that is what the mesh has to cope with and what only production currently exercises.
There are three positions, and they are genuinely different:
| Position | Reachable from outside | Address | Example |
|---|---|---|---|
| **Directly attached** | yes, at its own address | fixed, its own | a hosted server |
| **Behind a gateway you control** | only through a forwarded port, at the *gateway's* address | private, plus the gateway's public one | a machine at home |
| **Behind a gateway you don't control** | **no** | private, and it changes | a laptop on someone else's network |
The third is the hard one and the reason this matters. A machine there can dial out and nothing
more: it cannot be published, its apparent address belongs to somebody else's router, and that
address changes when it moves. Every assumption a mesh makes about reachability breaks there
first.
A declaration has to be able to say all three, and to move a machine between them.
## The shape ## The shape
```yaml ```yaml
scenario: published-behind-nat scenario: roaming-and-published
segments: segments:
wan: internet:
cidr: 203.0.113.0/24 # RFC 5737 — never routes on the real internet cidr: 203.0.113.0/24 # the simulated public internet, RFC 5737
lan: home:
cidr: 192.168.1.0/24 cidr: 192.168.1.0/24
behind: wan # NAT; the lab materialises a router gateway:
to: internet
address: 203.0.113.50 # what the world sees this network as
nat: true
elsewhere: # a network we do not control
cidr: 198.51.100.0/24
gateway:
to: internet
address: 203.0.113.80
nat: true
machines: machines:
anchor: anchor:
segment: wan at: { segment: internet, address: 203.0.113.10 }
address: 203.0.113.10
home-server: home-server:
segment: lan at: { segment: home, address: 192.168.1.135 }
address: 192.168.1.135 published:
forwarded: [443] # reachable from wan through the router - { port: 443, on: home } # DNAT: 203.0.113.50:443 → 192.168.1.135:443
workstation: workstation:
segment: lan at: { segment: home, address: 192.168.1.250 }
address: 192.168.1.250
laptop: laptop:
segment: detached # reachable by nothing until attached at: { segment: home, address: 192.168.1.98 }
place: place:
all: [host] all: [host]
@@ -50,41 +79,87 @@ place:
snapshot: raised snapshot: raised
``` ```
That is a complete bootstrap scenario. Nothing in it mentions the overlay, a hub, peering, ## What each part means, precisely
names or certificates — all of which are outcomes to be observed.
## The four parts **`segments`** — a broadcast domain with an address range. A segment with no `gateway:` *is*
the internet as far as the scenario is concerned. A segment with one sits behind it.
**`segments`** — the networks that exist. `behind:` declares NAT, and is the only place a **`gateway:`** — how a segment reaches its parent, and this is where the previous version was
router comes from: the lab materialises one without being asked, because NAT has to run too thin. It carries three facts, and all three are load-bearing:
somewhere. This is the one implicit machine in an otherwise explicit declaration.
**`machines`** — what sits where. A machine has a segment and an address, and that is nearly - `to:` — the parent segment.
all. `forwarded:` opens a port through the router, which is what makes *published but behind - `address:` — **the address the outside world sees this network as.** For a household
NAT* reproducible — the case that exists only in production today. `segment: detached` is a connection this is the public address the ISP hands out. It is not decoration: it is what a
machine on no network, which is how a roaming node is expressed at rest. peer records as the endpoint when a machine here dials out, and what a public name for a
published machine here resolves to.
- `nat:` — whether addresses are translated. `true` gives the ordinary household case: many
private machines behind one public address. `false` describes a routed range, where machines
keep their own addresses and the gateway only forwards.
**`place`** — what goes inside. `all:` applies to every machine; a machine name overrides for The lab materialises a machine to be the gateway. That is the one implicit machine in an
that machine. This is the only part that differs between the two scenario classes. otherwise explicit declaration, and it exists because NAT has to run somewhere.
**`snapshot`** — names the state once placement finishes, so a run can return to it without **`machines[].at`** — segment and address. That pair alone determines which of the three
raising everything again. Snapshots are what make repetition cheap, and cheap repetition is positions a machine is in: on a gateway-less segment it is directly attached; on a segment with
what makes the bootstrap path the inner development loop rather than a ceremony. a gateway it is behind one.
**`machines[].published`** — a destination-NAT rule on a named gateway, stated as an outcome
rather than a port list. `{ port: 443, on: home }` means the `home` gateway forwards its own
`203.0.113.50:443` to this machine's `443`. The resulting public endpoint is derivable, which is
the point: a scenario never writes an endpoint down, and the mesh has to discover it.
A machine may be published on **any gateway between it and the internet** — which is how *"our
LAN also has a public IP"* is expressed, and why `on:` names the gateway rather than being
implied. It cannot be published at all on a gateway the scenario models as foreign; attempting
it is a declaration error, because that is precisely the constraint being reproduced.
**`at: detached`** — on no segment. A machine that exists and can reach nothing.
## Moving a machine is a lifecycle operation
`at:` states where a machine *starts*. Moving it is something a run does:
```
move laptop → { segment: elsewhere, address: 198.51.100.23 }
move laptop → detached
move laptop → { segment: home, address: 192.168.1.98 }
```
This is the roaming case made testable, and it is the one that finds the interesting faults.
The same machine, the same identity, three positions in one run: at home where its peers can
reach it directly, on a foreign network where it can only dial out and its apparent address
belongs to a router it does not control, and asleep.
Whether the overlay survives that, re-forms, and is noticed to have changed endpoint is
**observed**, never arranged
([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
## Why the addresses are load-bearing ## Why the addresses are load-bearing
The routable segment uses RFC 5737 documentation space, and this is not a stylistic choice. The internet segment uses RFC 5737 documentation space, and this is not a stylistic choice.
The mesh decides *public versus private* by matching the address. A private range on the The mesh decides *public versus private* by matching the address. A private range on the
segment meant to be routable makes the hub test as unreachable, and **the mesh silently never segment meant to be routable makes a would-be hub test as unreachable, and **the mesh silently
forms** — no error, no failed step, just a mesh that does not exist. Research 004 calls this never forms** — no error, no failed step, just a mesh that does not exist. Research 004 calls
the single most important fact in its analysis. this the single most important fact in its analysis.
So the format should make this hard to get wrong rather than merely documented: a segment RFC 5737 reserves three ranges, which is exactly enough for the topology above:
without `behind:` is a routable segment, and an address in it that is not documentation space
is a declaration error, refused before anything is raised. That is | Range | Used for |
|---|---|
| `203.0.113.0/24` | the internet segment itself — directly attached machines, and gateway addresses |
| `198.51.100.0/24` | a foreign network, so a roaming machine's apparent address is plainly not ours |
| `192.0.2.0/24` | spare — a second foreign network, or a second site |
Private segments use RFC 1918 and can be **byte-identical to production**, because those
addresses mean the same thing everywhere. Only the public side is substituted, and only because
it must be.
The format should make getting this wrong hard rather than merely documented: a segment without
a `gateway:` is a public segment, and an address in it — including a gateway's `address:` — that
is not documentation space is a declaration error, refused before anything is raised. That is
[ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) applied to a configuration [ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) applied to a configuration
file — the failure it prevents is silent, so the check has to be loud. file: the failure it prevents is silent, so the check has to be loud.
## The same declaration serves both classes ## The same declaration serves both classes
@@ -127,15 +202,17 @@ not first.
## Open ## Open
- **`user` and `edge` profiles have no scenario.** A lab machine is always privileged, so the - **`user` and `edge` profiles have no scenario.** A lab machine is always privileged, so the
two profiles that exist for unprivileged and phone-like participation cannot currently be two profiles that exist for unprivileged and phone-like participation cannot be exercised.
exercised. Either the lab grows a way to run the host unprivileged, or those profiles are Either the lab grows a way to run the host unprivileged, or those profiles are developed
developed against something that is not a virtual machine. against something that is not a virtual machine. This is the largest gap.
- **Attaching and detaching during a run.** `segment: detached` covers a machine at rest;
moving one between segments while a scenario is live is what makes a roaming node
interesting, and that is lifecycle rather than declaration.
- **Where `place:` gets its artifacts from.** Before the mesh is self-hosting these come from - **Where `place:` gets its artifacts from.** Before the mesh is self-hosting these come from
outside; afterwards from the mesh itself. The declaration should not have to care, which outside; afterwards from the mesh itself. The declaration should not have to care, which
suggests a named source rather than a path. suggests a named source rather than a path.
- **Multiple scenarios at once.** Each needs its own segments and addresses. Whether the - **Multiple scenarios at once.** Each needs its own segments and addresses, and the shape above
declaration carries absolute addresses, as above, or a template the lab allocates from, writes addresses absolutely. Whether a scenario carries literal addresses or a template the
decides whether two scenarios can run side by side. lab allocates from decides whether two can run side by side — and there are only three
documentation ranges to go round.
- **Gateway behaviour beyond forwarding.** A real household gateway also has a NAT table with
timeouts, and connection tracking that drops idle flows. Whether a scenario can express *"the
gateway forgets a mapping after N seconds"* decides whether keepalive behaviour is testable
or merely hoped for.