diff --git a/03-DESIGN/01-to-be/02-scenario-declaration.md b/03-DESIGN/01-to-be/02-scenario-declaration.md index 38f6fd0..a818906 100644 --- a/03-DESIGN/01-to-be/02-scenario-declaration.md +++ b/03-DESIGN/01-to-be/02-scenario-declaration.md @@ -71,6 +71,32 @@ changes when it moves. Every assumption a mesh makes about reachability breaks t A declaration has to be able to say all three, and to move a machine between them. +## Reachability is per address family, not per machine + +Adding IPv6 is not a field. It changes the position model, and the reason is worth stating +before the syntax. + +**IPv6 usually has no NAT.** A machine behind a household gateway can hold a *globally routable* +v6 address while its v4 address is private and unforwardable. The same machine, at the same +moment, is in **two different positions at once**: + +| | IPv4 | IPv6 | +|---|---|---| +| a typical machine at home | behind an unforwardable-or-forwardable gateway | **attached**, directly reachable | +| a machine on mobile data | behind carrier-grade NAT | often attached, sometimes absent entirely | +| a machine on an older network | attached or behind NAT | **no address at all** | + +So the three positions apply **per family**, and a machine's reachability is a property of +*(machine, family)* rather than of the machine. A mesh that treats reachability as one fact per +node will reach a peer over one family, fail over the other, and report whichever it tried. + +That has a direct consequence for what the lab is for: *"can these two nodes reach each other"* +stops being a yes/no question. It is asked once per family, and the interesting answers are the +asymmetric ones. + +The v6 documentation prefix is `2001:db8::/32` (RFC 3849) — the exact counterpart of the RFC +5737 rule, and load-bearing for the same reason. + ## The shape ```yaml @@ -78,39 +104,52 @@ scenario: roaming-and-published segments: internet: - kind: public # stands in for the internet — RFC 5737 addresses - cidr: 203.0.113.0/24 + kind: public # RFC 5737 for v4, RFC 3849 for v6 + cidr: [203.0.113.0/24, 2001:db8::/32] + home: kind: private - cidr: 192.168.1.0/24 + cidr: [192.168.1.0/24, 2001:db8:1::/64] + mtu: 1500 gateway: to: internet - address: 203.0.113.50 # what the world sees this network as - nat: true - forwardable: true # we control it, so ports can be opened + address: 203.0.113.50 # what the world sees this network as, on v4 + nat: [v4] # v4 is translated; v6 is routed, not translated + forwardable: true + mapping_ttl: 120s # an unused inbound mapping is forgotten after this + elsewhere: # a network we do not control kind: private - cidr: 198.51.100.0/24 + cidr: [198.51.100.0/24] # v4 only — no v6 offered here at all + mtu: 1400 # a tunnelled path, smaller than standard gateway: to: internet address: 203.0.113.80 - nat: true + nat: [v4] forwardable: false # café wifi, or carrier-grade NAT + mapping_ttl: 30s # aggressive, as carrier NAT tends to be machines: anchor: - at: { segment: internet, address: 203.0.113.10 } + at: { segment: internet, address: [203.0.113.10, 2001:db8::10] } home-server: - at: { segment: home, address: 192.168.1.135 } + at: { segment: home, address: [192.168.1.135, 2001:db8:1::135] } published: - - { port: 443, on: home } # DNAT: 203.0.113.50:443 → 192.168.1.135:443 + - { port: 443, on: home } # v4 only: 203.0.113.50:443 → 192.168.1.135:443 + inbound: allow # v6 is routable here, so this decides whether it is reachable workstation: - at: { segment: home, address: 192.168.1.250 } + at: { segment: home, address: [192.168.1.250, 2001:db8:1::250] } + inbound: deny # a host firewall: dials out, accepts nothing laptop: - at: { segment: home, address: 192.168.1.98 } + at: { segment: home, address: [192.168.1.98, 2001:db8:1::98] } + + border: # a machine on two segments at once + at: + - { segment: home, address: [192.168.1.2] } + - { segment: internet, address: [203.0.113.60] } place: all: [host] @@ -138,20 +177,40 @@ too thin. It carries three facts, and all three are load-bearing: connection this is the public address the ISP hands out. It is not decoration: it is what a peer records as the endpoint when a machine here dials out, and what a public name for a published machine here resolves to. -- `nat:` — whether addresses are translated. `true` gives the ordinary household case: many - private machines behind one public address. `false` describes a routed range, where machines - keep their own addresses and the gateway only forwards. +- `nat:` — **which families are translated**, as a list. `[v4]` is the ordinary modern case: + v4 translated, v6 routed. `[v4, v6]` describes a gateway that translates both, which exists + and is worth being able to reproduce. `[]` is a routed range, where machines keep their own + addresses and the gateway only forwards. - `forwardable:` — whether an inbound mapping can be created. Independent of `nat:`, and the field that separates a home gateway from carrier-grade NAT. Publishing through a gateway with `forwardable: false` is a declaration error, because that is exactly the constraint being reproduced. +- `mapping_ttl:` — how long an unused inbound mapping survives. This is what makes keepalive + behaviour testable: a mesh that holds a connection through NAT without refreshing it works + perfectly until the far side goes quiet for longer than this. Aggressive values reproduce + carrier NAT; omitting it means mappings never expire, which no real gateway does. + +**`segments[].mtu`** — the largest packet the segment carries, defaulting to 1500. Lower values +reproduce tunnelled and PPPoE paths. This matters because an overlay adds its own header: a +tunnel over a 1400-byte path establishes a connection and then silently drops large packets, +which is the shape of fault this whole effort exists to stop shipping. + +**`machines[].inbound`** — `allow` or `deny`, a host firewall. Distinct from NAT and behaves +differently: a machine can be perfectly routable and still refuse everything unsolicited, which +is the normal state of a v6-addressed machine. Without this, v6 addressing would imply +reachability, and it does not. The lab materialises a machine to be the gateway. That is the one implicit machine in an otherwise explicit declaration, and it exists because NAT has to run somewhere. -**`machines[].at`** — segment and address. That pair alone determines which of the three -positions a machine is in: on a gateway-less segment it is directly attached; on a segment with -a gateway it is behind one. +**`machines[].at`** — segment and addresses, or a **list** of them for a machine on several +segments at once. Multi-homing is not exotic: it is what a border machine is, and what any node +with both a LAN and a WAN interface is. Each entry carries the addresses that machine holds on +that segment, one per family. + +Position follows from the pair, per family: on a `kind: public` segment a machine is attached; +on a private one it is behind that segment's gateway, unless the gateway does not translate +that family — in which case it is attached on that family and behind a gateway on the other. **`machines[].published`** — a destination-NAT rule on a named gateway, stated as an outcome rather than a port list. `{ port: 443, on: home }` means the `home` gateway forwards its own @@ -252,57 +311,57 @@ not first. ## Is this general? — the axes a setup can vary along The question that matters is not *does this cover our mesh*, but **can it express any mesh**. -Audited against the axes a real deployment varies along, the answer is *most, deliberately not -all, and three genuine gaps*. The standard applied is not "every property a network has". It is **every property that changes how the mesh behaves**. Bandwidth does not change correctness; MTU does. | Axis | Values | Expressible | | |---|---|---|---| -| **Reachability** | attached · forwardable gateway · unforwardable gateway · isolated | yes | the core of the model | -| **Address stability** | static · dynamic · changes mid-run | **partly** | a machine can be *moved*, but an address that changes under it cannot be stated | -| **Gateway depth** | direct · one gateway · nested gateways | **partly** | `to:` chains, so nesting exists; `published:` names one gateway, so forwarding through two does not | -| **Address family** | IPv4 · IPv6 · dual-stack | **no** | `cidr:` is implicitly v4. A v6-only node is a real topology and cannot be written | -| **Interfaces per machine** | one · several | **no** | `at:` is singular. A multi-homed node — on a LAN and a WAN at once — is inexpressible | -| **Path properties** | MTU · latency · loss | **no** | MTU matters: tunnels fragment, and a lower-MTU path is a classic silent failure | -| **Reachability policy** | symmetric · asymmetric | **no** | a firewall dropping inbound while outbound works is different from NAT and behaves differently | -| **Gateway state** | permanent · expiring mappings | **no** | mappings time out; whether keepalives work is untestable without it | -| **Overlapping ranges** | distinct · two sites both on `192.168.1.0/24` | yes | two segments may carry the same range — common, and it breaks routing | -| **Segment count** | one · many · isolated island | yes | after the `kind:` fix above | +| **Reachability** | attached · forwardable gateway · unforwardable gateway · isolated | yes | the core of the model, and **per family** | +| **Address family** | IPv4 · IPv6 · dual-stack · neither | yes | `cidr:` and `address:` take both; `nat:` names which families are translated | +| **Interfaces per machine** | one · several | yes | `at:` takes a list | +| **Reachability policy** | symmetric · asymmetric | yes | `inbound:` — a routable machine that refuses everything | +| **Gateway state** | permanent · expiring mappings | yes | `mapping_ttl:` | +| **Path MTU** | standard · reduced | yes | `segments[].mtu` | +| **Overlapping ranges** | distinct · two sites both on `192.168.1.0/24` | yes | segments may carry the same range | +| **Segment count** | one · many · isolated island | yes | `kind:` distinguishes an island from the internet | +| **Gateway depth** | direct · one gateway · nested | **partly** | `to:` chains, so nesting exists; `published:` names one gateway, so forwarding through two does not | +| **Address stability** | static · dynamic · changes mid-run | **partly** | a machine can be *moved*; an address changing under it in place cannot be stated | +| **Path quality** | latency · loss · bandwidth | **no**, deliberately | changes performance, not correctness — modelling it makes a network simulator, not a fixture | -### What this says +### What closing the gaps changed -**Three gaps are real and should be closed**, in this order: +**Address family was not a field.** It changed the position model: a machine behind a household +gateway is typically *unforwardable on v4 and directly attached on v6, simultaneously*. So +reachability is a property of *(machine, family)*, and *"can these two nodes reach each other"* +is no longer a yes/no question — it is asked once per family, and the asymmetric answers are the +interesting ones. That distinction did not exist in the model an hour ago and would have been +discovered by a mesh failing over one family while reporting the other. -1. **Address family.** A v6-only or dual-stack node is not exotic, and a mesh that assumes v4 - fails there completely rather than partially. This is the largest gap. -2. **Expiring NAT mappings.** Without it, keepalive behaviour is hoped for rather than tested — - and for a mesh where most nodes sit behind NAT, that is the failure mode most likely to - appear only after everything has been idle overnight. -3. **MTU.** Tunnels fragment. A path with a smaller MTU produces a connection that establishes - and then silently drops large packets, which is exactly the shape of fault this whole effort - exists to stop shipping. +**`inbound:` became necessary because of v6.** With NAT, unreachability was implied by the +topology. With a globally routable v6 address, a machine is reachable unless something refuses — +so refusing has to be sayable, or v6 addressing would silently imply reachability. -**Two are deliberately out of scope** unless something argues otherwise: latency and loss. -They change performance, not correctness, and a scenario that models them is a network -simulator rather than a fixture. +**`nat:` became a list rather than a boolean** for the same reason: a real gateway translates v4 +and routes v6, and a boolean cannot say that. -**Two are partial and probably fine for now:** nested forwarding and mid-run address change. -Both are expressible with small extensions when something needs them, and neither blocks the -bootstrap scenario. +### What remains open, and whether it matters + +Two partial axes, both extensible when something needs them, neither blocking: **nested +forwarding** and **an address changing in place**. A machine can already be moved, which covers +the roaming case; what is missing is a lease expiring underneath a machine that stays put. + +One deliberate exclusion: **path quality**. Latency and loss change how fast the mesh is, not +whether it is correct. If a timeout turns out to be load-bearing that judgement should be +revisited — and it would be revisited by a real failure, which is the right trigger. ### The honest summary -The model covers **where a machine sits**, which is what the mesh's reachability logic turns -on, and it now covers it completely. It does not yet cover **what the path between machines is -like**, and one of those — address family — is not a refinement but a second world the mesh -would have to work in. +The model now covers **where a machine sits** and **what the path between machines is like**, +across both address families, which together are what the mesh's reachability logic turns on. -None of this blocks phase 0. A bootstrap scenario is one machine and a pinned bundle, and needs -none of it. But the gaps should be closed before the lab is trusted to say a mesh *works*, -because today it could only say it works over IPv4, on an unconstrained path, against gateways -that never forget. +What it does not model is *change over time* beyond moving a machine, and *degradation* short of +failure. Both are absences chosen rather than overlooked. ## Open @@ -317,5 +376,6 @@ that never forget. writes addresses absolutely. Whether a scenario carries literal addresses or a template the lab allocates from decides whether two can run side by side — and there are only three documentation ranges to go round. -- **The three gaps from the audit above** — address family, expiring NAT mappings, MTU — in - that order. The first is the one that is a second world rather than a refinement. +- **Nested forwarding** — `published:` names one gateway, so a machine behind two cannot be + published through both. +- **An address changing in place**, as a DHCP lease expiring under a machine that has not moved.