diff --git a/03-DESIGN/01-to-be/02-scenario-declaration.md b/03-DESIGN/01-to-be/02-scenario-declaration.md index 5789e4f..695268d 100644 --- a/03-DESIGN/01-to-be/02-scenario-declaration.md +++ b/03-DESIGN/01-to-be/02-scenario-declaration.md @@ -17,6 +17,54 @@ everything in the lab hangs off, so it is worth getting small. It states what a hosting provider and a home router would provide, and nothing the mesh is responsible for ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)). +## Public networks are unrelated, and routed rather than bridged + +The internet is not a network. It is a very large number of unrelated networks that route to +each other, and a machine in one is **many hops** from a machine in another with no shared +broadcast domain between them. + +So a scenario does not have *an* internet segment. It has **one public segment per public +network**, each with its own unrelated prefix, and the lab wires them together **through a +router, never onto a shared bridge**. + +That distinction is load-bearing, and putting several public addresses in one prefix would +quietly make four things true that are false in reality: + +| If public addresses share a segment | Reality | +|---|---| +| machines resolve each other by ARP and talk directly | they are routed, many hops apart | +| TTL never decrements | every hop decrements it | +| broadcast and multicast reach across | neither crosses a router | +| any two are adjacent | adjacency is the exception, not the rule | + +The third is not hypothetical here. The mesh has already been bitten by multicast name +resolution — [`00-as-is/01-mesh-and-transport.md`](../00-as-is/01-mesh-and-transport.md) +records that mesh names are deliberately not multicast names, after a delay and a +one-node-only failure mode. A lab where "the internet" is one broadcast domain would let a node +discover a peer by multicast that it could never discover in production, and report success. + +**The rule: `kind: public` segments are routed to one another and never bridged.** It is a +property of how the lab wires them, not a field anyone sets, because there is no correct +scenario in which two public networks are adjacent. + +### Which addresses to use + +RFC 5737 reserves three ranges and RFC 3849 reserves one v6 prefix. Each public network takes +its own, and they are chosen to look nothing like each other — because in reality they would +not: + +| Public network | v4 | v6 | +|---|---|---| +| a hosting provider | `192.0.2.0/24` | `2001:db8:a::/48` | +| a household ISP | `198.51.100.0/24` | `2001:db8:b::/48` | +| a mobile or foreign network | `203.0.113.0/24` | `2001:db8:c::/48` | + +Three is enough for the topologies that matter, and a fourth public network can subnet one of +them — an ISP handing out `198.51.100.0/25` and `198.51.100.128/25` to two customers is exactly +what really happens. + +Private segments still use RFC 1918 and stay byte-identical to production. + ## What NAT does, and why the design turns on it A household or office has **one** address the outside world can see, and **many** machines @@ -103,53 +151,57 @@ The v6 documentation prefix is `2001:db8::/32` (RFC 3849) — the exact counterp scenario: roaming-and-published segments: - internet: - kind: public # RFC 5737 for v4, RFC 3849 for v6 - cidr: [203.0.113.0/24, 2001:db8::/32] + hosting: # one public network + kind: public + cidr: [192.0.2.0/24, 2001:db8:a::/48] + + isp-home: # another, unrelated — routed to it, not bridged + kind: public + cidr: [198.51.100.0/24, 2001:db8:b::/48] home: kind: private - cidr: [192.168.1.0/24, 2001:db8:1::/64] + cidr: [192.168.1.0/24, 2001:db8:b:1::/64] mtu: 1500 gateway: - to: internet - address: 203.0.113.50 # what the world sees this network as, on v4 + to: isp-home + address: [198.51.100.7, 2001:db8:b::7] # what the world sees this network as nat: [v4] # v4 is translated; v6 is routed, not translated forwardable: true mapping_ttl: 120s # an unused inbound mapping is forgotten after this - elsewhere: # a network we do not control + cafe: # a network we do not control kind: private - cidr: [198.51.100.0/24] # v4 only — no v6 offered here at all + cidr: [10.50.0.0/16] # v4 only — no v6 offered here at all mtu: 1400 # a tunnelled path, smaller than standard gateway: - to: internet - address: 203.0.113.80 + to: hosting # it reaches the world via a different public network + address: [192.0.2.200] nat: [v4] forwardable: false # café wifi, or carrier-grade NAT mapping_ttl: 30s # aggressive, as carrier NAT tends to be machines: anchor: - at: { segment: internet, address: [203.0.113.10, 2001:db8::10] } + at: { segment: hosting, address: [192.0.2.10, 2001:db8:a::10] } home-server: - at: { segment: home, address: [192.168.1.135, 2001:db8:1::135] } + at: { segment: home, address: [192.168.1.135, 2001:db8:b:1::135] } published: - - { port: 443, on: home } # v4 only: 203.0.113.50:443 → 192.168.1.135:443 + - { port: 443, on: home } # v4 only: 198.51.100.7:443 → 192.168.1.135:443 inbound: allow # v6 is routable here, so this decides whether it is reachable workstation: - at: { segment: home, address: [192.168.1.250, 2001:db8:1::250] } + at: { segment: home, address: [192.168.1.250, 2001:db8:b:1::250] } inbound: deny # a host firewall: dials out, accepts nothing laptop: - at: { segment: home, address: [192.168.1.98, 2001:db8:1::98] } + at: { segment: home, address: [192.168.1.98, 2001:db8:b:1::98] } border: # a machine on two segments at once at: - - { segment: home, address: [192.168.1.2] } - - { segment: internet, address: [203.0.113.60] } + - { segment: home, address: [192.168.1.2] } + - { segment: isp-home, address: [198.51.100.60] } place: all: [host] @@ -160,13 +212,14 @@ snapshot: raised ## What each part means, precisely -**`segments`** — a broadcast domain with an address range, and a `kind:`. +**`segments`** — a broadcast domain with an address range, and a `kind:`. A segment is a single +broadcast domain, which is precisely why several public networks cannot be one segment. -`kind: public` marks the segment that stands in for the internet. `kind: private` is everything -else. This is stated rather than inferred, and the earlier version inferred it — *a segment with +`kind: public` marks a segment that stands in for a public network — and there is normally more +than one, unrelated to each other. `kind: private` is everything else. This is stated rather than inferred, and the earlier version inferred it — *a segment with no gateway is the internet* — which made an **isolated network inexpressible**: a LAN with no -route out is a private segment with no gateway, and would have been read as the internet and -forced to use documentation addresses. A mesh spanning a site with no internet access is a real +route out is a private segment with no gateway, and would have been read as public and forced +to use documentation addresses. A mesh spanning a site with no internet access is a real topology, and the model has to be able to say it. **`gateway:`** — how a segment reaches its parent, and this is where the previous version was @@ -217,7 +270,7 @@ rather than a port list. `{ port: 443, on: home }` means the `home` gateway forw `203.0.113.50:443` to this machine's `443`. The resulting public endpoint is derivable, which is the point: a scenario never writes an endpoint down, and the mesh has to discover it. -A machine may be published on **any gateway between it and the internet** — which is how *"our +A machine may be published on **any gateway between it and a public network** — which is how *"our LAN also has a public IP"* is expressed, and why `on:` names the gateway rather than being implied. It cannot be published at all on a gateway the scenario models as foreign; attempting it is a declaration error, because that is precisely the constraint being reproduced. @@ -252,13 +305,8 @@ segment meant to be routable makes a would-be hub test as unreachable, and **the never forms** — no error, no failed step, just a mesh that does not exist. Research 004 calls this the single most important fact in its analysis. -RFC 5737 reserves three ranges, which is exactly enough for the topology above: - -| Range | Used for | -|---|---| -| `203.0.113.0/24` | the internet segment itself — directly attached machines, and gateway addresses | -| `198.51.100.0/24` | a foreign network, so a roaming machine's apparent address is plainly not ours | -| `192.0.2.0/24` | spare — a second foreign network, or a second site | +Which range goes where is covered above, under *public networks are unrelated*: one range per +public network, chosen to look nothing like each other. Private segments use RFC 1918 and can be **byte-identical to production**, because those addresses mean the same thing everywhere. Only the public side is substituted, and only because @@ -352,11 +400,11 @@ segments: trusted: kind: private cidr: [192.168.1.0/24] - gateway: { to: internet, address: 203.0.113.50, nat: [v4], forwardable: true } + gateway: { to: isp-home, address: [198.51.100.7], nat: [v4], forwardable: true } devices: kind: private cidr: [192.168.30.0/24] - gateway: { to: internet, address: 203.0.113.50, nat: [v4], forwardable: true } + gateway: { to: isp-home, address: [198.51.100.7], nat: [v4], forwardable: true } ``` Identical gateway declarations mean **one gateway machine**, not two. The lab materialises a @@ -398,18 +446,29 @@ out completely, with every field the model has. scenario: the-ordinary-shape segments: - internet: + # ---- three unrelated public networks. Routed to each other, never bridged. ---- + + hosting: # where the always-on machine lives kind: public - cidr: [203.0.113.0/24, 2001:db8::/32] - mtu: 1500 + cidr: [192.0.2.0/24, 2001:db8:a::/48] + + isp-home: # the household's uplink + kind: public + cidr: [198.51.100.0/24, 2001:db8:b::/48] + + isp-mobile: # wherever the roaming machine happens to be + kind: public + cidr: [203.0.113.0/24, 2001:db8:c::/48] + + # ---- private networks behind them ---- home: # the household network kind: private - cidr: [192.168.1.0/24, 2001:db8:1::/64] + cidr: [192.168.1.0/24, 2001:db8:b:1::/64] mtu: 1492 # PPPoE on the uplink; 1500 if the line is not PPPoE gateway: - to: internet - address: 203.0.113.50 # the address the ISP hands the household + to: isp-home + address: [198.51.100.7, 2001:db8:b::7] nat: [v4] # v4 translated, v6 routed — the modern default forwardable: true # the household router is ours to configure mapping_ttl: 120s @@ -418,21 +477,21 @@ segments: kind: private cidr: [192.168.30.0/24] gateway: - to: internet - address: 203.0.113.50 # identical → the SAME gateway machine + to: isp-home + address: [198.51.100.7, 2001:db8:b::7] # identical → the SAME gateway machine nat: [v4] forwardable: true mapping_ttl: 120s - elsewhere: # wherever the roaming machine happens to be + cafe: # a network we do not control kind: private - cidr: [198.51.100.0/24] + cidr: [10.50.0.0/16] # RFC 1918 — someone else's private range mtu: 1400 gateway: - to: internet - address: 203.0.113.80 + to: isp-mobile + address: [203.0.113.129] nat: [v4] - forwardable: false # someone else's network, or carrier-grade NAT + forwardable: false # carrier-grade NAT, or simply not ours mapping_ttl: 30s policy: @@ -441,21 +500,21 @@ policy: machines: anchor: # routable, nothing in front of it - at: { segment: internet, address: [203.0.113.10, 2001:db8::10] } + at: { segment: hosting, address: [192.0.2.10, 2001:db8:a::10] } inbound: allow home-server: # publicly named, behind the household connection - at: { segment: home, address: [192.168.1.135, 2001:db8:1::135] } + at: { segment: home, address: [192.168.1.135, 2001:db8:b:1::135] } published: - { port: 443, on: home } # v4 reaches it only through the forward inbound: allow # and v6 reaches it directly, so this matters workstation: # on the household network, not published - at: { segment: home, address: [192.168.1.250, 2001:db8:1::250] } + at: { segment: home, address: [192.168.1.250, 2001:db8:b:1::250] } inbound: deny laptop: # starts at home; moves during the run - at: { segment: home, address: [192.168.1.98, 2001:db8:1::98] } + at: { segment: home, address: [192.168.1.98, 2001:db8:b:1::98] } inbound: deny place: @@ -489,13 +548,18 @@ it is observed. ### Running it ``` -move laptop → { segment: elsewhere, address: [198.51.100.23] } # it leaves the house -move laptop → detached # it sleeps -move laptop → { segment: home, address: [192.168.1.98, 2001:db8:1::98] } +move laptop → { segment: cafe, address: [10.50.3.23] } # it leaves the house +move laptop → detached # it sleeps +move laptop → { segment: home, address: [192.168.1.98, 2001:db8:b:1::98] } ``` -One identity, four positions, one run. The `mapping_ttl: 30s` on `elsewhere` means a connection -held without refreshing dies while it is out — which is the point of putting a number there. +One identity, four positions, one run. The `mapping_ttl: 30s` on `cafe` means a connection held +without refreshing dies while it is out — which is the point of putting a number there. + +Note that the laptop's three positions are on **three different public networks**: at home it +appears as `198.51.100.7`, at the café as `203.0.113.129`, and the machine it is trying to reach +is on a fourth. None of them are adjacent, all of them are routed. That is the situation the +mesh actually faces. ## Is this general? — the axes a setup can vary along @@ -515,7 +579,8 @@ how the mesh behaves**. Bandwidth does not change correctness; MTU does. | **Gateway state** | permanent · expiring mappings | yes | `mapping_ttl:` | | **Path MTU** | standard · reduced | yes | `segments[].mtu` | | **Overlapping ranges** | distinct · two sites both on `192.168.1.0/24` | yes | segments may carry the same range | -| **Segment count** | one · many · isolated island | yes | `kind:` distinguishes an island from the internet | +| **Segment count** | one · many · isolated island | yes | `kind:` distinguishes an island from a public network | +| **Adjacency** | same segment · routed · unrelated public networks | yes | public segments are routed, never bridged — so no two are adjacent unless declared so | | **Gateway depth** | direct · one gateway · nested | **partly** | `to:` chains, so nesting exists; `published:` names one gateway, so forwarding through two does not | | **Address stability** | static · dynamic · changes mid-run | **partly** | a machine can be *moved*; an address changing under it in place cannot be stated | | **Path quality** | latency · loss · bandwidth | **no**, deliberately | changes performance, not correctness — modelling it makes a network simulator, not a fixture |