novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Public networks are unrelated, and routed rather than bridged

Browse Source 
Caught in review: every public address sat in one /24, which made the
three of them look like one network. They are not. The internet is a very
large number of unrelated networks routing to each other, and a machine in
one is many hops from a machine in another with no shared broadcast domain
between them.

Putting them in one prefix would have quietly made four false things true
in the lab: machines resolving each other by ARP and talking directly, TTL
never decrementing, broadcast and multicast crossing between them, and any
two being adjacent.

The third is not hypothetical. The as-is layer records that mesh names are
deliberately not multicast names, after a delay and a one-node-only failure
mode. A lab where the internet is one broadcast domain would let a node
discover a peer by multicast that it could never discover in production,
and report success — the exact false green this effort exists to prevent.

So a scenario has one public segment per public NETWORK, each with its own
unrelated prefix, wired together through a router and never onto a shared
bridge. That is a property of how the lab wires them rather than a field
anyone sets, because no correct scenario has two public networks adjacent.

Addresses now spread across all three RFC 5737 ranges plus RFC 3849 /48s,
chosen to look nothing like each other, and a foreign private network uses
someone else's RFC 1918 range rather than a documentation one.

All three examples in the document rewritten, since two of them still
showed a single flat internet segment and contradicted the new rule.
This commit is contained in:
jschoubben
2026-08-23 23:49:38 +02:00
parent a873088140
commit e88a6df924
1 changed files with 120 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files
03-DESIGN/01-to-be/02-scenario-declaration.md
+118 -53
View File
@@ -17,6 +17,54 @@ everything in the lab hangs off, so it is worth getting small.
It states what a hosting provider and a home router would provide, and nothing the mesh is It states what a hosting provider and a home router would provide, and nothing the mesh is
responsible for ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)). responsible for ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
## Public networks are unrelated, and routed rather than bridged
The internet is not a network. It is a very large number of unrelated networks that route to
each other, and a machine in one is **many hops** from a machine in another with no shared
broadcast domain between them.
So a scenario does not have *an* internet segment. It has **one public segment per public
network**, each with its own unrelated prefix, and the lab wires them together **through a
router, never onto a shared bridge**.
That distinction is load-bearing, and putting several public addresses in one prefix would
quietly make four things true that are false in reality:
| If public addresses share a segment | Reality |
|---|---|
| machines resolve each other by ARP and talk directly | they are routed, many hops apart |
| TTL never decrements | every hop decrements it |
| broadcast and multicast reach across | neither crosses a router |
| any two are adjacent | adjacency is the exception, not the rule |
The third is not hypothetical here. The mesh has already been bitten by multicast name
resolution — [`00-as-is/01-mesh-and-transport.md`](../00-as-is/01-mesh-and-transport.md)
records that mesh names are deliberately not multicast names, after a delay and a
one-node-only failure mode. A lab where "the internet" is one broadcast domain would let a node
discover a peer by multicast that it could never discover in production, and report success.
**The rule: `kind: public` segments are routed to one another and never bridged.** It is a
property of how the lab wires them, not a field anyone sets, because there is no correct
scenario in which two public networks are adjacent.
### Which addresses to use
RFC 5737 reserves three ranges and RFC 3849 reserves one v6 prefix. Each public network takes
its own, and they are chosen to look nothing like each other — because in reality they would
not:
| Public network | v4 | v6 |
|---|---|---|
| a hosting provider | `192.0.2.0/24` | `2001:db8:a::/48` |
| a household ISP | `198.51.100.0/24` | `2001:db8:b::/48` |
| a mobile or foreign network | `203.0.113.0/24` | `2001:db8:c::/48` |
Three is enough for the topologies that matter, and a fourth public network can subnet one of
them — an ISP handing out `198.51.100.0/25` and `198.51.100.128/25` to two customers is exactly
what really happens.
Private segments still use RFC 1918 and stay byte-identical to production.
## What NAT does, and why the design turns on it ## What NAT does, and why the design turns on it
A household or office has **one** address the outside world can see, and **many** machines A household or office has **one** address the outside world can see, and **many** machines
@@ -103,53 +151,57 @@ The v6 documentation prefix is `2001:db8::/32` (RFC 3849) — the exact counterp
scenario: roaming-and-published scenario: roaming-and-published
segments: segments:
internet: hosting: # one public network
kind: public # RFC 5737 for v4, RFC 3849 for v6 kind: public
cidr: [203.0.113.0/24, 2001:db8::/32] cidr: [192.0.2.0/24, 2001:db8:a::/48]
isp-home: # another, unrelated — routed to it, not bridged
kind: public
cidr: [198.51.100.0/24, 2001:db8:b::/48]
home: home:
kind: private kind: private
cidr: [192.168.1.0/24, 2001:db8:1::/64] cidr: [192.168.1.0/24, 2001:db8:b:1::/64]
mtu: 1500 mtu: 1500
gateway: gateway:
to: internet to: isp-home
address: 203.0.113.50 # what the world sees this network as, on v4 address: [198.51.100.7, 2001:db8:b::7] # what the world sees this network as
nat: [v4] # v4 is translated; v6 is routed, not translated nat: [v4] # v4 is translated; v6 is routed, not translated
forwardable: true forwardable: true
mapping_ttl: 120s # an unused inbound mapping is forgotten after this mapping_ttl: 120s # an unused inbound mapping is forgotten after this
elsewhere: # a network we do not control cafe: # a network we do not control
kind: private kind: private
cidr: [198.51.100.0/24] # v4 only — no v6 offered here at all cidr: [10.50.0.0/16] # v4 only — no v6 offered here at all
mtu: 1400 # a tunnelled path, smaller than standard mtu: 1400 # a tunnelled path, smaller than standard
gateway: gateway:
to: internet to: hosting # it reaches the world via a different public network
address: 203.0.113.80 address: [192.0.2.200]
nat: [v4] nat: [v4]
forwardable: false # café wifi, or carrier-grade NAT forwardable: false # café wifi, or carrier-grade NAT
mapping_ttl: 30s # aggressive, as carrier NAT tends to be mapping_ttl: 30s # aggressive, as carrier NAT tends to be
machines: machines:
anchor: anchor:
at: { segment: internet, address: [203.0.113.10, 2001:db8::10] } at: { segment: hosting, address: [192.0.2.10, 2001:db8:a::10] }
home-server: home-server:
at: { segment: home, address: [192.168.1.135, 2001:db8:1::135] } at: { segment: home, address: [192.168.1.135, 2001:db8:b:1::135] }
published: published:
- { port: 443, on: home } # v4 only: 203.0.113.50:443 → 192.168.1.135:443 - { port: 443, on: home } # v4 only: 198.51.100.7:443 → 192.168.1.135:443
inbound: allow # v6 is routable here, so this decides whether it is reachable inbound: allow # v6 is routable here, so this decides whether it is reachable
workstation: workstation:
at: { segment: home, address: [192.168.1.250, 2001:db8:1::250] } at: { segment: home, address: [192.168.1.250, 2001:db8:b:1::250] }
inbound: deny # a host firewall: dials out, accepts nothing inbound: deny # a host firewall: dials out, accepts nothing
laptop: laptop:
at: { segment: home, address: [192.168.1.98, 2001:db8:1::98] } at: { segment: home, address: [192.168.1.98, 2001:db8:b:1::98] }
border: # a machine on two segments at once border: # a machine on two segments at once
at: at:
- { segment: home, address: [192.168.1.2] } - { segment: home, address: [192.168.1.2] }
- { segment: internet, address: [203.0.113.60] } - { segment: isp-home, address: [198.51.100.60] }
place: place:
all: [host] all: [host]
@@ -160,13 +212,14 @@ snapshot: raised
## What each part means, precisely ## What each part means, precisely
**`segments`** — a broadcast domain with an address range, and a `kind:`. **`segments`** — a broadcast domain with an address range, and a `kind:`. A segment is a single
broadcast domain, which is precisely why several public networks cannot be one segment.
`kind: public` marks the segment that stands in for the internet. `kind: private` is everything `kind: public` marks a segment that stands in for a public network — and there is normally more
else. This is stated rather than inferred, and the earlier version inferred it — *a segment with than one, unrelated to each other. `kind: private` is everything else. This is stated rather than inferred, and the earlier version inferred it — *a segment with
no gateway is the internet* — which made an **isolated network inexpressible**: a LAN with no no gateway is the internet* — which made an **isolated network inexpressible**: a LAN with no
route out is a private segment with no gateway, and would have been read as the internet and route out is a private segment with no gateway, and would have been read as public and forced
forced to use documentation addresses. A mesh spanning a site with no internet access is a real to use documentation addresses. A mesh spanning a site with no internet access is a real
topology, and the model has to be able to say it. topology, and the model has to be able to say it.
**`gateway:`** — how a segment reaches its parent, and this is where the previous version was **`gateway:`** — how a segment reaches its parent, and this is where the previous version was
@@ -217,7 +270,7 @@ rather than a port list. `{ port: 443, on: home }` means the `home` gateway forw
`203.0.113.50:443` to this machine's `443`. The resulting public endpoint is derivable, which is `203.0.113.50:443` to this machine's `443`. The resulting public endpoint is derivable, which is
the point: a scenario never writes an endpoint down, and the mesh has to discover it. the point: a scenario never writes an endpoint down, and the mesh has to discover it.
A machine may be published on **any gateway between it and the internet** — which is how *"our A machine may be published on **any gateway between it and a public network** — which is how *"our
LAN also has a public IP"* is expressed, and why `on:` names the gateway rather than being LAN also has a public IP"* is expressed, and why `on:` names the gateway rather than being
implied. It cannot be published at all on a gateway the scenario models as foreign; attempting implied. It cannot be published at all on a gateway the scenario models as foreign; attempting
it is a declaration error, because that is precisely the constraint being reproduced. it is a declaration error, because that is precisely the constraint being reproduced.
@@ -252,13 +305,8 @@ segment meant to be routable makes a would-be hub test as unreachable, and **the
never forms** — no error, no failed step, just a mesh that does not exist. Research 004 calls never forms** — no error, no failed step, just a mesh that does not exist. Research 004 calls
this the single most important fact in its analysis. this the single most important fact in its analysis.
RFC 5737 reserves three ranges, which is exactly enough for the topology above: Which range goes where is covered above, under *public networks are unrelated*: one range per
public network, chosen to look nothing like each other.
| Range | Used for |
|---|---|
| `203.0.113.0/24` | the internet segment itself — directly attached machines, and gateway addresses |
| `198.51.100.0/24` | a foreign network, so a roaming machine's apparent address is plainly not ours |
| `192.0.2.0/24` | spare — a second foreign network, or a second site |
Private segments use RFC 1918 and can be **byte-identical to production**, because those Private segments use RFC 1918 and can be **byte-identical to production**, because those
addresses mean the same thing everywhere. Only the public side is substituted, and only because addresses mean the same thing everywhere. Only the public side is substituted, and only because
@@ -352,11 +400,11 @@ segments:
trusted: trusted:
kind: private kind: private
cidr: [192.168.1.0/24] cidr: [192.168.1.0/24]
gateway: { to: internet, address: 203.0.113.50, nat: [v4], forwardable: true } gateway: { to: isp-home, address: [198.51.100.7], nat: [v4], forwardable: true }
devices: devices:
kind: private kind: private
cidr: [192.168.30.0/24] cidr: [192.168.30.0/24]
gateway: { to: internet, address: 203.0.113.50, nat: [v4], forwardable: true } gateway: { to: isp-home, address: [198.51.100.7], nat: [v4], forwardable: true }
``` ```
Identical gateway declarations mean **one gateway machine**, not two. The lab materialises a Identical gateway declarations mean **one gateway machine**, not two. The lab materialises a
@@ -398,18 +446,29 @@ out completely, with every field the model has.
scenario: the-ordinary-shape scenario: the-ordinary-shape
segments: segments:
internet: # ---- three unrelated public networks. Routed to each other, never bridged. ----
hosting: # where the always-on machine lives
kind: public kind: public
cidr: [203.0.113.0/24, 2001:db8::/32] cidr: [192.0.2.0/24, 2001:db8:a::/48]
mtu: 1500
isp-home: # the household's uplink
kind: public
cidr: [198.51.100.0/24, 2001:db8:b::/48]
isp-mobile: # wherever the roaming machine happens to be
kind: public
cidr: [203.0.113.0/24, 2001:db8:c::/48]
# ---- private networks behind them ----
home: # the household network home: # the household network
kind: private kind: private
cidr: [192.168.1.0/24, 2001:db8:1::/64] cidr: [192.168.1.0/24, 2001:db8:b:1::/64]
mtu: 1492 # PPPoE on the uplink; 1500 if the line is not PPPoE mtu: 1492 # PPPoE on the uplink; 1500 if the line is not PPPoE
gateway: gateway:
to: internet to: isp-home
address: 203.0.113.50 # the address the ISP hands the household address: [198.51.100.7, 2001:db8:b::7]
nat: [v4] # v4 translated, v6 routed — the modern default nat: [v4] # v4 translated, v6 routed — the modern default
forwardable: true # the household router is ours to configure forwardable: true # the household router is ours to configure
mapping_ttl: 120s mapping_ttl: 120s
@@ -418,21 +477,21 @@ segments:
kind: private kind: private
cidr: [192.168.30.0/24] cidr: [192.168.30.0/24]
gateway: gateway:
to: internet to: isp-home
address: 203.0.113.50 # identical → the SAME gateway machine address: [198.51.100.7, 2001:db8:b::7] # identical → the SAME gateway machine
nat: [v4] nat: [v4]
forwardable: true forwardable: true
mapping_ttl: 120s mapping_ttl: 120s
elsewhere: # wherever the roaming machine happens to be cafe: # a network we do not control
kind: private kind: private
cidr: [198.51.100.0/24] cidr: [10.50.0.0/16] # RFC 1918 — someone else's private range
mtu: 1400 mtu: 1400
gateway: gateway:
to: internet to: isp-mobile
address: 203.0.113.80 address: [203.0.113.129]
nat: [v4] nat: [v4]
forwardable: false # someone else's network, or carrier-grade NAT forwardable: false # carrier-grade NAT, or simply not ours
mapping_ttl: 30s mapping_ttl: 30s
policy: policy:
@@ -441,21 +500,21 @@ policy:
machines: machines:
anchor: # routable, nothing in front of it anchor: # routable, nothing in front of it
at: { segment: internet, address: [203.0.113.10, 2001:db8::10] } at: { segment: hosting, address: [192.0.2.10, 2001:db8:a::10] }
inbound: allow inbound: allow
home-server: # publicly named, behind the household connection home-server: # publicly named, behind the household connection
at: { segment: home, address: [192.168.1.135, 2001:db8:1::135] } at: { segment: home, address: [192.168.1.135, 2001:db8:b:1::135] }
published: published:
- { port: 443, on: home } # v4 reaches it only through the forward - { port: 443, on: home } # v4 reaches it only through the forward
inbound: allow # and v6 reaches it directly, so this matters inbound: allow # and v6 reaches it directly, so this matters
workstation: # on the household network, not published workstation: # on the household network, not published
at: { segment: home, address: [192.168.1.250, 2001:db8:1::250] } at: { segment: home, address: [192.168.1.250, 2001:db8:b:1::250] }
inbound: deny inbound: deny
laptop: # starts at home; moves during the run laptop: # starts at home; moves during the run
at: { segment: home, address: [192.168.1.98, 2001:db8:1::98] } at: { segment: home, address: [192.168.1.98, 2001:db8:b:1::98] }
inbound: deny inbound: deny
place: place:
@@ -489,13 +548,18 @@ it is observed.
### Running it ### Running it
``` ```
move laptop → { segment: elsewhere, address: [198.51.100.23] } # it leaves the house move laptop → { segment: cafe, address: [10.50.3.23] } # it leaves the house
move laptop → detached # it sleeps move laptop → detached # it sleeps
move laptop → { segment: home, address: [192.168.1.98, 2001:db8:1::98] } move laptop → { segment: home, address: [192.168.1.98, 2001:db8:b:1::98] }
``` ```
One identity, four positions, one run. The `mapping_ttl: 30s` on `elsewhere` means a connection One identity, four positions, one run. The `mapping_ttl: 30s` on `cafe` means a connection held
held without refreshing dies while it is out — which is the point of putting a number there. without refreshing dies while it is out — which is the point of putting a number there.
Note that the laptop's three positions are on **three different public networks**: at home it
appears as `198.51.100.7`, at the café as `203.0.113.129`, and the machine it is trying to reach
is on a fourth. None of them are adjacent, all of them are routed. That is the situation the
mesh actually faces.
## Is this general? — the axes a setup can vary along ## Is this general? — the axes a setup can vary along
@@ -515,7 +579,8 @@ how the mesh behaves**. Bandwidth does not change correctness; MTU does.
| **Gateway state** | permanent · expiring mappings | yes | `mapping_ttl:` | | **Gateway state** | permanent · expiring mappings | yes | `mapping_ttl:` |
| **Path MTU** | standard · reduced | yes | `segments[].mtu` | | **Path MTU** | standard · reduced | yes | `segments[].mtu` |
| **Overlapping ranges** | distinct · two sites both on `192.168.1.0/24` | yes | segments may carry the same range | | **Overlapping ranges** | distinct · two sites both on `192.168.1.0/24` | yes | segments may carry the same range |
| **Segment count** | one · many · isolated island | yes | `kind:` distinguishes an island from the internet | | **Segment count** | one · many · isolated island | yes | `kind:` distinguishes an island from a public network |
| **Adjacency** | same segment · routed · unrelated public networks | yes | public segments are routed, never bridged — so no two are adjacent unless declared so |
| **Gateway depth** | direct · one gateway · nested | **partly** | `to:` chains, so nesting exists; `published:` names one gateway, so forwarding through two does not | | **Gateway depth** | direct · one gateway · nested | **partly** | `to:` chains, so nesting exists; `published:` names one gateway, so forwarding through two does not |
| **Address stability** | static · dynamic · changes mid-run | **partly** | a machine can be *moved*; an address changing under it in place cannot be stated | | **Address stability** | static · dynamic · changes mid-run | **partly** | a machine can be *moved*; an address changing under it in place cannot be stated |
| **Path quality** | latency · loss · bandwidth | **no**, deliberately | changes performance, not correctness — modelling it makes a network simulator, not a fixture | | **Path quality** | latency · loss · bandwidth | **no**, deliberately | changes performance, not correctness — modelling it makes a network simulator, not a fixture |