The question is not whether the model covers our mesh but whether it can express any mesh. Audited against the axes a deployment varies along, with the standard being every property that changes how the mesh BEHAVES rather than every property a network has — bandwidth does not change correctness, MTU does. One real bug, now fixed. A segment with no gateway was read as the internet, which made an isolated network inexpressible: a LAN with no route out would have been treated as public and forced onto documentation addresses. Segments now state kind: public or private, and a private segment with no gateway is an island. A mesh spanning a site with no internet is a real topology. One modelling error, now corrected. The three positions were framed by ownership — a gateway you control versus one you do not. The axis is forwardability. Carrier-grade NAT is your own connection and is still unforwardable, so it belongs with the café network. Gateways gain forwardable:, independent of nat:, and publishing through an unforwardable one is a declaration error because that is the constraint being reproduced. Three genuine gaps recorded in priority order. Address family: cidr is implicitly v4, and a v6-only node is not exotic — a mesh that assumes v4 fails there completely rather than partially, which makes this a second world rather than a refinement. Expiring NAT mappings: without them keepalive behaviour is hoped for rather than tested, and for a mesh mostly behind NAT that is the fault that shows up after an idle night. MTU: tunnels fragment, and a smaller-MTU path establishes a connection that then silently drops large packets — the exact shape this effort exists to stop shipping. Latency and loss are deliberately out: they change performance, not correctness, and modelling them makes a network simulator rather than a fixture. Also adds a NAT primer, because the three positions are consequences of it and the document should not assume the reader already knows why a mesh dials outward and never inward.
322 lines
16 KiB
Markdown
322 lines
16 KiB
Markdown
---
|
|
layer: to-be
|
|
status: designed
|
|
code: [mesh-lab]
|
|
updated: 2026-08-23
|
|
decisions:
|
|
- 02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md
|
|
- 02-DECISIONS/0031-the-lab-provides-the-underlay.md
|
|
- 02-DECISIONS/0016-a-lab-node-is-a-virtual-machine.md
|
|
---
|
|
|
|
# The scenario declaration
|
|
|
|
A scenario is a **declaration of an underlay**, plus what to put on it. It is the interface
|
|
everything in the lab hangs off, so it is worth getting small.
|
|
|
|
It states what a hosting provider and a home router would provide, and nothing the mesh is
|
|
responsible for ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
|
|
|
|
## What NAT does, and why the design turns on it
|
|
|
|
A household or office has **one** address the outside world can see, and **many** machines
|
|
behind it. Network address translation is what reconciles those.
|
|
|
|
When a machine inside dials out, the gateway rewrites the packet's source from the private
|
|
address to the public one, **remembers the mapping**, and rewrites the replies on the way back.
|
|
Four consequences follow, and every one of them shapes this design:
|
|
|
|
1. **Outbound works; inbound does not.** A mapping exists only because something inside started
|
|
a conversation. Nothing outside can start one — there is no mapping to look up, and no way
|
|
to know which internal machine was meant.
|
|
2. **A forwarded port is a permanent mapping made by hand**, in the inbound direction:
|
|
*anything arriving at the public address on 443 goes to this machine.* That is the only way
|
|
a machine behind NAT becomes reachable, and it requires control of the gateway.
|
|
3. **Mappings expire.** A gateway forgets one that goes unused. This is why anything holding a
|
|
connection through NAT sends keepalives, and why a mesh that does not is fine until it is
|
|
idle.
|
|
4. **From outside, every machine behind the gateway looks like one address.** Identity and
|
|
address stop corresponding.
|
|
|
|
This is why the mesh dials outward and never inward
|
|
([ADR 0001](../../02-DECISIONS/0001-nodes-communicate-over-a-broker.md)), why a hub exists at
|
|
all, and why a node's endpoint is something a peer **learns** from arriving packets rather than
|
|
something anyone configures.
|
|
|
|
**Carrier-grade NAT** is the same mechanism applied by an ISP: your own gateway gets a private
|
|
address too, and the public one is shared with strangers. Nothing can be forwarded, because the
|
|
rule would have to live on equipment you do not own. Common on mobile connections and
|
|
increasingly on fixed ones.
|
|
|
|
## Three positions a machine can be in
|
|
|
|
The underlay's whole job is to reproduce **where a machine sits relative to the internet**,
|
|
because that is what the mesh has to cope with and what only production currently exercises.
|
|
There are three positions, and they are genuinely different:
|
|
|
|
| Position | Reachable from outside | Apparent address | Example |
|
|
|---|---|---|---|
|
|
| **Attached** | yes, at its own address | its own | a hosted server |
|
|
| **Behind a forwardable gateway** | only through a forwarded port, at the *gateway's* address | the gateway's | a machine at home |
|
|
| **Behind an unforwardable gateway** | **no** | someone else's, and it changes | a laptop on a café network; anything behind carrier-grade NAT |
|
|
|
|
The axis is **forwardability, not ownership** — which is worth stating because the obvious
|
|
framing gets it wrong. Carrier-grade NAT is *your* connection and is still unforwardable, so it
|
|
belongs in the third row alongside the café. What the mesh has to cope with is whether an
|
|
inbound mapping can be made, not who owns the equipment.
|
|
|
|
The third position is the hard one. A machine there can dial out and nothing more: it cannot be
|
|
published, its apparent address belongs to a router it does not control, and that address
|
|
changes when it moves. Every assumption a mesh makes about reachability breaks there first.
|
|
|
|
A declaration has to be able to say all three, and to move a machine between them.
|
|
|
|
## The shape
|
|
|
|
```yaml
|
|
scenario: roaming-and-published
|
|
|
|
segments:
|
|
internet:
|
|
kind: public # stands in for the internet — RFC 5737 addresses
|
|
cidr: 203.0.113.0/24
|
|
home:
|
|
kind: private
|
|
cidr: 192.168.1.0/24
|
|
gateway:
|
|
to: internet
|
|
address: 203.0.113.50 # what the world sees this network as
|
|
nat: true
|
|
forwardable: true # we control it, so ports can be opened
|
|
elsewhere: # a network we do not control
|
|
kind: private
|
|
cidr: 198.51.100.0/24
|
|
gateway:
|
|
to: internet
|
|
address: 203.0.113.80
|
|
nat: true
|
|
forwardable: false # café wifi, or carrier-grade NAT
|
|
|
|
machines:
|
|
anchor:
|
|
at: { segment: internet, address: 203.0.113.10 }
|
|
|
|
home-server:
|
|
at: { segment: home, address: 192.168.1.135 }
|
|
published:
|
|
- { port: 443, on: home } # DNAT: 203.0.113.50:443 → 192.168.1.135:443
|
|
|
|
workstation:
|
|
at: { segment: home, address: 192.168.1.250 }
|
|
|
|
laptop:
|
|
at: { segment: home, address: 192.168.1.98 }
|
|
|
|
place:
|
|
all: [host]
|
|
anchor: [substrate]
|
|
|
|
snapshot: raised
|
|
```
|
|
|
|
## What each part means, precisely
|
|
|
|
**`segments`** — a broadcast domain with an address range, and a `kind:`.
|
|
|
|
`kind: public` marks the segment that stands in for the internet. `kind: private` is everything
|
|
else. This is stated rather than inferred, and the earlier version inferred it — *a segment with
|
|
no gateway is the internet* — which made an **isolated network inexpressible**: a LAN with no
|
|
route out is a private segment with no gateway, and would have been read as the internet and
|
|
forced to use documentation addresses. A mesh spanning a site with no internet access is a real
|
|
topology, and the model has to be able to say it.
|
|
|
|
**`gateway:`** — how a segment reaches its parent, and this is where the previous version was
|
|
too thin. It carries three facts, and all three are load-bearing:
|
|
|
|
- `to:` — the parent segment.
|
|
- `address:` — **the address the outside world sees this network as.** For a household
|
|
connection this is the public address the ISP hands out. It is not decoration: it is what a
|
|
peer records as the endpoint when a machine here dials out, and what a public name for a
|
|
published machine here resolves to.
|
|
- `nat:` — whether addresses are translated. `true` gives the ordinary household case: many
|
|
private machines behind one public address. `false` describes a routed range, where machines
|
|
keep their own addresses and the gateway only forwards.
|
|
- `forwardable:` — whether an inbound mapping can be created. Independent of `nat:`, and the
|
|
field that separates a home gateway from carrier-grade NAT. Publishing through a gateway with
|
|
`forwardable: false` is a declaration error, because that is exactly the constraint being
|
|
reproduced.
|
|
|
|
The lab materialises a machine to be the gateway. That is the one implicit machine in an
|
|
otherwise explicit declaration, and it exists because NAT has to run somewhere.
|
|
|
|
**`machines[].at`** — segment and address. That pair alone determines which of the three
|
|
positions a machine is in: on a gateway-less segment it is directly attached; on a segment with
|
|
a gateway it is behind one.
|
|
|
|
**`machines[].published`** — a destination-NAT rule on a named gateway, stated as an outcome
|
|
rather than a port list. `{ port: 443, on: home }` means the `home` gateway forwards its own
|
|
`203.0.113.50:443` to this machine's `443`. The resulting public endpoint is derivable, which is
|
|
the point: a scenario never writes an endpoint down, and the mesh has to discover it.
|
|
|
|
A machine may be published on **any gateway between it and the internet** — which is how *"our
|
|
LAN also has a public IP"* is expressed, and why `on:` names the gateway rather than being
|
|
implied. It cannot be published at all on a gateway the scenario models as foreign; attempting
|
|
it is a declaration error, because that is precisely the constraint being reproduced.
|
|
|
|
**`at: detached`** — on no segment. A machine that exists and can reach nothing.
|
|
|
|
## Moving a machine is a lifecycle operation
|
|
|
|
`at:` states where a machine *starts*. Moving it is something a run does:
|
|
|
|
```
|
|
move laptop → { segment: elsewhere, address: 198.51.100.23 }
|
|
move laptop → detached
|
|
move laptop → { segment: home, address: 192.168.1.98 }
|
|
```
|
|
|
|
This is the roaming case made testable, and it is the one that finds the interesting faults.
|
|
The same machine, the same identity, three positions in one run: at home where its peers can
|
|
reach it directly, on a foreign network where it can only dial out and its apparent address
|
|
belongs to a router it does not control, and asleep.
|
|
|
|
Whether the overlay survives that, re-forms, and is noticed to have changed endpoint is
|
|
**observed**, never arranged
|
|
([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
|
|
|
|
## Why the addresses are load-bearing
|
|
|
|
The internet segment uses RFC 5737 documentation space, and this is not a stylistic choice.
|
|
|
|
The mesh decides *public versus private* by matching the address. A private range on the
|
|
segment meant to be routable makes a would-be hub test as unreachable, and **the mesh silently
|
|
never forms** — no error, no failed step, just a mesh that does not exist. Research 004 calls
|
|
this the single most important fact in its analysis.
|
|
|
|
RFC 5737 reserves three ranges, which is exactly enough for the topology above:
|
|
|
|
| Range | Used for |
|
|
|---|---|
|
|
| `203.0.113.0/24` | the internet segment itself — directly attached machines, and gateway addresses |
|
|
| `198.51.100.0/24` | a foreign network, so a roaming machine's apparent address is plainly not ours |
|
|
| `192.0.2.0/24` | spare — a second foreign network, or a second site |
|
|
|
|
Private segments use RFC 1918 and can be **byte-identical to production**, because those
|
|
addresses mean the same thing everywhere. Only the public side is substituted, and only because
|
|
it must be.
|
|
|
|
The format should make getting this wrong hard rather than merely documented: a segment without
|
|
a `gateway:` is a public segment, and an address in it — including a gateway's `address:` — that
|
|
is not documentation space is a declaration error, refused before anything is raised. That is
|
|
[ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) applied to a configuration
|
|
file: the failure it prevents is silent, so the check has to be loud.
|
|
|
|
## The same declaration serves both classes
|
|
|
|
The bootstrap and full scenarios differ **only in `place:`**
|
|
([ADR 0029](../../02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md)). Everything
|
|
about the underlay is identical, which is what makes one a strict subset of the other rather
|
|
than a fork.
|
|
|
|
```yaml
|
|
# bootstrap — tiers 0 and 1
|
|
place:
|
|
all: [host]
|
|
anchor: [substrate]
|
|
|
|
# full — adds a control plane, a forge, and a module under test
|
|
place:
|
|
all: [host]
|
|
anchor: [substrate, control, forge]
|
|
module: a-web-service
|
|
assert:
|
|
- the service answers on its published name
|
|
- the certificate presented is valid for that name
|
|
```
|
|
|
|
`module:` and `assert:` are meaningless in a bootstrap scenario and absent from one. A
|
|
bootstrap scenario's verdict comes from what the host reports about the state it reconciled,
|
|
not from an assertion runner — which is why assertion execution is second in the build order,
|
|
not first.
|
|
|
|
## What a scenario deliberately cannot say
|
|
|
|
- **Overlay addresses, the hub, peer configuration.** Outcomes, not inputs
|
|
([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
|
|
- **What a machine is in mesh terms** — server or workstation, its site, its names. Mesh
|
|
configuration, established by the mesh.
|
|
- **A host's capability profile.** Detected, never declared.
|
|
- **Steps.** A scenario is a desired state. Anything expressed as an ordered list of actions
|
|
belongs in the lifecycle, not the declaration.
|
|
|
|
## Is this general? — the axes a setup can vary along
|
|
|
|
The question that matters is not *does this cover our mesh*, but **can it express any mesh**.
|
|
Audited against the axes a real deployment varies along, the answer is *most, deliberately not
|
|
all, and three genuine gaps*.
|
|
|
|
The standard applied is not "every property a network has". It is **every property that changes
|
|
how the mesh behaves**. Bandwidth does not change correctness; MTU does.
|
|
|
|
| Axis | Values | Expressible | |
|
|
|---|---|---|---|
|
|
| **Reachability** | attached · forwardable gateway · unforwardable gateway · isolated | yes | the core of the model |
|
|
| **Address stability** | static · dynamic · changes mid-run | **partly** | a machine can be *moved*, but an address that changes under it cannot be stated |
|
|
| **Gateway depth** | direct · one gateway · nested gateways | **partly** | `to:` chains, so nesting exists; `published:` names one gateway, so forwarding through two does not |
|
|
| **Address family** | IPv4 · IPv6 · dual-stack | **no** | `cidr:` is implicitly v4. A v6-only node is a real topology and cannot be written |
|
|
| **Interfaces per machine** | one · several | **no** | `at:` is singular. A multi-homed node — on a LAN and a WAN at once — is inexpressible |
|
|
| **Path properties** | MTU · latency · loss | **no** | MTU matters: tunnels fragment, and a lower-MTU path is a classic silent failure |
|
|
| **Reachability policy** | symmetric · asymmetric | **no** | a firewall dropping inbound while outbound works is different from NAT and behaves differently |
|
|
| **Gateway state** | permanent · expiring mappings | **no** | mappings time out; whether keepalives work is untestable without it |
|
|
| **Overlapping ranges** | distinct · two sites both on `192.168.1.0/24` | yes | two segments may carry the same range — common, and it breaks routing |
|
|
| **Segment count** | one · many · isolated island | yes | after the `kind:` fix above |
|
|
|
|
### What this says
|
|
|
|
**Three gaps are real and should be closed**, in this order:
|
|
|
|
1. **Address family.** A v6-only or dual-stack node is not exotic, and a mesh that assumes v4
|
|
fails there completely rather than partially. This is the largest gap.
|
|
2. **Expiring NAT mappings.** Without it, keepalive behaviour is hoped for rather than tested —
|
|
and for a mesh where most nodes sit behind NAT, that is the failure mode most likely to
|
|
appear only after everything has been idle overnight.
|
|
3. **MTU.** Tunnels fragment. A path with a smaller MTU produces a connection that establishes
|
|
and then silently drops large packets, which is exactly the shape of fault this whole effort
|
|
exists to stop shipping.
|
|
|
|
**Two are deliberately out of scope** unless something argues otherwise: latency and loss.
|
|
They change performance, not correctness, and a scenario that models them is a network
|
|
simulator rather than a fixture.
|
|
|
|
**Two are partial and probably fine for now:** nested forwarding and mid-run address change.
|
|
Both are expressible with small extensions when something needs them, and neither blocks the
|
|
bootstrap scenario.
|
|
|
|
### The honest summary
|
|
|
|
The model covers **where a machine sits**, which is what the mesh's reachability logic turns
|
|
on, and it now covers it completely. It does not yet cover **what the path between machines is
|
|
like**, and one of those — address family — is not a refinement but a second world the mesh
|
|
would have to work in.
|
|
|
|
None of this blocks phase 0. A bootstrap scenario is one machine and a pinned bundle, and needs
|
|
none of it. But the gaps should be closed before the lab is trusted to say a mesh *works*,
|
|
because today it could only say it works over IPv4, on an unconstrained path, against gateways
|
|
that never forget.
|
|
|
|
## Open
|
|
|
|
- **`user` and `edge` profiles have no scenario.** A lab machine is always privileged, so the
|
|
two profiles that exist for unprivileged and phone-like participation cannot be exercised.
|
|
Either the lab grows a way to run the host unprivileged, or those profiles are developed
|
|
against something that is not a virtual machine. This is the largest gap.
|
|
- **Where `place:` gets its artifacts from.** Before the mesh is self-hosting these come from
|
|
outside; afterwards from the mesh itself. The declaration should not have to care, which
|
|
suggests a named source rather than a path.
|
|
- **Multiple scenarios at once.** Each needs its own segments and addresses, and the shape above
|
|
writes addresses absolutely. Whether a scenario carries literal addresses or a template the
|
|
lab allocates from decides whether two can run side by side — and there are only three
|
|
documentation ranges to go round.
|
|
- **The three gaps from the audit above** — address family, expiring NAT mappings, MTU — in
|
|
that order. The first is the one that is a second world rather than a refinement.
|