diff --git a/03-DESIGN/01-to-be/02-scenario-declaration.md b/03-DESIGN/01-to-be/02-scenario-declaration.md index 84e40af..38f6fd0 100644 --- a/03-DESIGN/01-to-be/02-scenario-declaration.md +++ b/03-DESIGN/01-to-be/02-scenario-declaration.md @@ -17,22 +17,57 @@ everything in the lab hangs off, so it is worth getting small. It states what a hosting provider and a home router would provide, and nothing the mesh is responsible for ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)). +## What NAT does, and why the design turns on it + +A household or office has **one** address the outside world can see, and **many** machines +behind it. Network address translation is what reconciles those. + +When a machine inside dials out, the gateway rewrites the packet's source from the private +address to the public one, **remembers the mapping**, and rewrites the replies on the way back. +Four consequences follow, and every one of them shapes this design: + +1. **Outbound works; inbound does not.** A mapping exists only because something inside started + a conversation. Nothing outside can start one — there is no mapping to look up, and no way + to know which internal machine was meant. +2. **A forwarded port is a permanent mapping made by hand**, in the inbound direction: + *anything arriving at the public address on 443 goes to this machine.* That is the only way + a machine behind NAT becomes reachable, and it requires control of the gateway. +3. **Mappings expire.** A gateway forgets one that goes unused. This is why anything holding a + connection through NAT sends keepalives, and why a mesh that does not is fine until it is + idle. +4. **From outside, every machine behind the gateway looks like one address.** Identity and + address stop corresponding. + +This is why the mesh dials outward and never inward +([ADR 0001](../../02-DECISIONS/0001-nodes-communicate-over-a-broker.md)), why a hub exists at +all, and why a node's endpoint is something a peer **learns** from arriving packets rather than +something anyone configures. + +**Carrier-grade NAT** is the same mechanism applied by an ISP: your own gateway gets a private +address too, and the public one is shared with strangers. Nothing can be forwarded, because the +rule would have to live on equipment you do not own. Common on mobile connections and +increasingly on fixed ones. + ## Three positions a machine can be in The underlay's whole job is to reproduce **where a machine sits relative to the internet**, because that is what the mesh has to cope with and what only production currently exercises. There are three positions, and they are genuinely different: -| Position | Reachable from outside | Address | Example | +| Position | Reachable from outside | Apparent address | Example | |---|---|---|---| -| **Directly attached** | yes, at its own address | fixed, its own | a hosted server | -| **Behind a gateway you control** | only through a forwarded port, at the *gateway's* address | private, plus the gateway's public one | a machine at home | -| **Behind a gateway you don't control** | **no** | private, and it changes | a laptop on someone else's network | +| **Attached** | yes, at its own address | its own | a hosted server | +| **Behind a forwardable gateway** | only through a forwarded port, at the *gateway's* address | the gateway's | a machine at home | +| **Behind an unforwardable gateway** | **no** | someone else's, and it changes | a laptop on a café network; anything behind carrier-grade NAT | -The third is the hard one and the reason this matters. A machine there can dial out and nothing -more: it cannot be published, its apparent address belongs to somebody else's router, and that -address changes when it moves. Every assumption a mesh makes about reachability breaks there -first. +The axis is **forwardability, not ownership** — which is worth stating because the obvious +framing gets it wrong. Carrier-grade NAT is *your* connection and is still unforwardable, so it +belongs in the third row alongside the café. What the mesh has to cope with is whether an +inbound mapping can be made, not who owns the equipment. + +The third position is the hard one. A machine there can dial out and nothing more: it cannot be +published, its apparent address belongs to a router it does not control, and that address +changes when it moves. Every assumption a mesh makes about reachability breaks there first. A declaration has to be able to say all three, and to move a machine between them. @@ -43,19 +78,24 @@ scenario: roaming-and-published segments: internet: - cidr: 203.0.113.0/24 # the simulated public internet, RFC 5737 + kind: public # stands in for the internet — RFC 5737 addresses + cidr: 203.0.113.0/24 home: + kind: private cidr: 192.168.1.0/24 gateway: to: internet address: 203.0.113.50 # what the world sees this network as nat: true + forwardable: true # we control it, so ports can be opened elsewhere: # a network we do not control + kind: private cidr: 198.51.100.0/24 gateway: to: internet address: 203.0.113.80 nat: true + forwardable: false # café wifi, or carrier-grade NAT machines: anchor: @@ -81,8 +121,14 @@ snapshot: raised ## What each part means, precisely -**`segments`** — a broadcast domain with an address range. A segment with no `gateway:` *is* -the internet as far as the scenario is concerned. A segment with one sits behind it. +**`segments`** — a broadcast domain with an address range, and a `kind:`. + +`kind: public` marks the segment that stands in for the internet. `kind: private` is everything +else. This is stated rather than inferred, and the earlier version inferred it — *a segment with +no gateway is the internet* — which made an **isolated network inexpressible**: a LAN with no +route out is a private segment with no gateway, and would have been read as the internet and +forced to use documentation addresses. A mesh spanning a site with no internet access is a real +topology, and the model has to be able to say it. **`gateway:`** — how a segment reaches its parent, and this is where the previous version was too thin. It carries three facts, and all three are load-bearing: @@ -95,6 +141,10 @@ too thin. It carries three facts, and all three are load-bearing: - `nat:` — whether addresses are translated. `true` gives the ordinary household case: many private machines behind one public address. `false` describes a routed range, where machines keep their own addresses and the gateway only forwards. +- `forwardable:` — whether an inbound mapping can be created. Independent of `nat:`, and the + field that separates a home gateway from carrier-grade NAT. Publishing through a gateway with + `forwardable: false` is a declaration error, because that is exactly the constraint being + reproduced. The lab materialises a machine to be the gateway. That is the one implicit machine in an otherwise explicit declaration, and it exists because NAT has to run somewhere. @@ -199,6 +249,61 @@ not first. - **Steps.** A scenario is a desired state. Anything expressed as an ordered list of actions belongs in the lifecycle, not the declaration. +## Is this general? — the axes a setup can vary along + +The question that matters is not *does this cover our mesh*, but **can it express any mesh**. +Audited against the axes a real deployment varies along, the answer is *most, deliberately not +all, and three genuine gaps*. + +The standard applied is not "every property a network has". It is **every property that changes +how the mesh behaves**. Bandwidth does not change correctness; MTU does. + +| Axis | Values | Expressible | | +|---|---|---|---| +| **Reachability** | attached · forwardable gateway · unforwardable gateway · isolated | yes | the core of the model | +| **Address stability** | static · dynamic · changes mid-run | **partly** | a machine can be *moved*, but an address that changes under it cannot be stated | +| **Gateway depth** | direct · one gateway · nested gateways | **partly** | `to:` chains, so nesting exists; `published:` names one gateway, so forwarding through two does not | +| **Address family** | IPv4 · IPv6 · dual-stack | **no** | `cidr:` is implicitly v4. A v6-only node is a real topology and cannot be written | +| **Interfaces per machine** | one · several | **no** | `at:` is singular. A multi-homed node — on a LAN and a WAN at once — is inexpressible | +| **Path properties** | MTU · latency · loss | **no** | MTU matters: tunnels fragment, and a lower-MTU path is a classic silent failure | +| **Reachability policy** | symmetric · asymmetric | **no** | a firewall dropping inbound while outbound works is different from NAT and behaves differently | +| **Gateway state** | permanent · expiring mappings | **no** | mappings time out; whether keepalives work is untestable without it | +| **Overlapping ranges** | distinct · two sites both on `192.168.1.0/24` | yes | two segments may carry the same range — common, and it breaks routing | +| **Segment count** | one · many · isolated island | yes | after the `kind:` fix above | + +### What this says + +**Three gaps are real and should be closed**, in this order: + +1. **Address family.** A v6-only or dual-stack node is not exotic, and a mesh that assumes v4 + fails there completely rather than partially. This is the largest gap. +2. **Expiring NAT mappings.** Without it, keepalive behaviour is hoped for rather than tested — + and for a mesh where most nodes sit behind NAT, that is the failure mode most likely to + appear only after everything has been idle overnight. +3. **MTU.** Tunnels fragment. A path with a smaller MTU produces a connection that establishes + and then silently drops large packets, which is exactly the shape of fault this whole effort + exists to stop shipping. + +**Two are deliberately out of scope** unless something argues otherwise: latency and loss. +They change performance, not correctness, and a scenario that models them is a network +simulator rather than a fixture. + +**Two are partial and probably fine for now:** nested forwarding and mid-run address change. +Both are expressible with small extensions when something needs them, and neither blocks the +bootstrap scenario. + +### The honest summary + +The model covers **where a machine sits**, which is what the mesh's reachability logic turns +on, and it now covers it completely. It does not yet cover **what the path between machines is +like**, and one of those — address family — is not a refinement but a second world the mesh +would have to work in. + +None of this blocks phase 0. A bootstrap scenario is one machine and a pinned bundle, and needs +none of it. But the gaps should be closed before the lab is trusted to say a mesh *works*, +because today it could only say it works over IPv4, on an unconstrained path, against gateways +that never forget. + ## Open - **`user` and `edge` profiles have no scenario.** A lab machine is always privileged, so the @@ -212,7 +317,5 @@ not first. writes addresses absolutely. Whether a scenario carries literal addresses or a template the lab allocates from decides whether two can run side by side — and there are only three documentation ranges to go round. -- **Gateway behaviour beyond forwarding.** A real household gateway also has a NAT table with - timeouts, and connection tracking that drops idle flows. Whether a scenario can express *"the - gateway forgets a mapping after N seconds"* decides whether keepalive behaviour is testable - or merely hoped for. +- **The three gaps from the audit above** — address family, expiring NAT mappings, MTU — in + that order. The first is the one that is a second world rather than a refinement.