jschoubben
4e80820e2f
Written as one document because the five are one design. They share inputs, they must agree, and every one of them today is computed in a different place by a different module from a different copy of the same facts. The through-line is that none of the five can be answered by a machine alone, so all five are decided centrally and delivered as `file` resources. That costs no new host vocabulary and removes both remaining direct database connections from nodes -- wireguard and traefik are the only two, and both are connectivity. Three decisions fall out, all proposed: 0050 -- reachability is declared, not inferred from an address. The RFC1918 regex is wrong for carrier-grade NAT (100.64/10 tests as public, so an endpoint is written to an address nothing can reach), wrong for IPv6, and wrong for a routable address behind a closed firewall. The lab needing TEST-NET-3 to satisfy the regex is the same bug from the other side. Also kills hub election by address prefix, which fails silently and makes renumbering an outage. 0051 -- the enrolment token carries where the mesh is and how to recognise it. Closes two circles with one mechanism: verifying the mesh needed the CA, and obtaining the CA meant trusting whoever handed it over; and a node had to reach the mesh before it could resolve any mesh name. An address plus a fingerprint, carried out of band, resolves both -- and closes the CA question 0049 deferred. 0052 -- a filter rule names its source. `scope:` is declared in five manifests, is part of no rule type, and is referenced by no code, so those manifests appear to restrict ports and restrict nothing. Removed rather than implemented; the general fix is refusing unknown keys, which the host already does and manifests do not. Also corrects two claims in 0049 asserting wireguard was already handled. Research 006 says both modules still reach upward; neither is.
120 lines
6.6 KiB
Markdown
120 lines
6.6 KiB
Markdown
---
|
|
status: proposed
|
|
date: 2026-08-27
|
|
deciders: jochen
|
|
reconstructed: false
|
|
extends: 0031-the-lab-provides-the-underlay.md
|
|
---
|
|
|
|
# 50. Reachability is declared, not inferred from an address
|
|
|
|
## Context
|
|
|
|
The overlay's peer graph is computed on each node by the `wireguard` module, which decides —
|
|
per pair — whether to write an `Endpoint` for a peer. The rule
|
|
([research 004](../01-RESEARCH/004-lab-network/analysis.md)) is a regular expression:
|
|
|
|
```js
|
|
const isPrivate = (a) => /^(10\.|127\.|192\.168\.|172\.(1[6-9]|2\d|3[01])\.)/.test(a);
|
|
```
|
|
|
|
Not private ⇒ assumed reachable ⇒ an `Endpoint` is written. Private ⇒ no endpoint, and the peer
|
|
must initiate.
|
|
|
|
The module's own comments record what this cost to arrive at: testing `profile === "server"`
|
|
was tried and was wrong, because a home-hosted node **is** a server and is not publicly
|
|
reachable — *"role does not imply reachability; the address does."*
|
|
|
|
**That lesson is right and the implementation of it is not.** The address is evidence of
|
|
reachability; it is not the fact itself, and the gap between the two has already caused failures
|
|
and will cause more:
|
|
|
|
| Address | The regex says | Actually |
|
|
|---|---|---|
|
|
| `100.64.0.0/10` — carrier-grade NAT ([RFC 6598](https://www.rfc-editor.org/rfc/rfc6598)) | **public** | **not reachable.** An endpoint is written to an address nothing can reach. |
|
|
| any IPv6 address, including `fd00::/8` unique-local | **public** | unique-local is not reachable; the regex tests v4 shapes only |
|
|
| a routable address behind a closed firewall | public | not reachable |
|
|
| `10.200.0.0/24` standing in for a public segment | private | reachable — this is the lab bug |
|
|
|
|
The CGNAT row is the serious one. A node on a carrier-grade NAT address presents exactly the
|
|
failure already recorded for the hairpin case: *1.77 MiB sent, 0 B received, no handshake.* The
|
|
mesh silently never forms, and it presents as a WireGuard fault rather than an addressing one.
|
|
|
|
The lab row is the same bug seen from the other side. [Research 004](../01-RESEARCH/004-lab-network/00-overview.md)
|
|
calls the required substitution *"the single most important fact in this document"* — a
|
|
simulated public segment must use TEST-NET-3, or nothing can ever initiate. **A test environment
|
|
having to choose its addresses to satisfy a regex is the regex telling us it is not a fact.**
|
|
|
|
There is a second inference in the same code, and it is worse because nothing records it:
|
|
|
|
> **Hub election is by convention.** The hub is the node whose `profile='server'` *and* whose
|
|
> overlay address begins `10.10.0.1`. A lab must assign that address to the node it intends as
|
|
> hub **or there will be no hub** — and nothing says so.
|
|
|
|
An election decided by the first four characters of an address is not an election. It fails
|
|
silently, it cannot be queried, and it makes a renumbering into an outage.
|
|
|
|
## Considered options
|
|
|
|
1. **Fix the regex.** Add CGNAT, add IPv6, add the ranges as they are discovered. Rejected: the
|
|
list is unbounded, and each addition is written after the outage that revealed it. The
|
|
firewall case cannot be fixed at all — no address shape encodes it.
|
|
2. **Probe for reachability and cache the answer.** Attractive, and wrong as the *primary*
|
|
source: at the moment the graph is computed a node may be legitimately down, and a probe
|
|
cannot distinguish *unreachable* from *asleep* ([ADR 0036](0036-a-node-is-a-managed-machine.md)).
|
|
Deriving topology from a liveness check makes the overlay flap with the network.
|
|
3. **Declare it, and let observation contradict it.** Chosen.
|
|
|
|
## Decision
|
|
|
|
**A node's reachability is a declared fact on its record, not an inference from its address.**
|
|
|
|
Two facts, and the mesh stores both:
|
|
|
|
- **an endpoint, or none** — where peers may reach this node, if anywhere. Absent means *this
|
|
node initiates and is never dialled*, which is the safe default and the common case.
|
|
- **its role in the overlay** — whether it is a hub. **Declared, never derived from an address.**
|
|
|
|
**The address remains evidence and stops being the fact.** When a node's observed endpoint
|
|
disagrees with its declared one, that is a **reportable condition**, not a silent correction —
|
|
the same discipline as [ADR 0035](0035-a-picture-is-read-from-what-runs.md): a picture is read
|
|
from the system, and where the reading disagrees with the intent, the disagreement is the
|
|
finding.
|
|
|
|
**The peer graph is computed by the control plane**, from these declared facts, and delivered to
|
|
each node as configuration. It is not computed on the node, which is
|
|
[ADR 0037](0037-the-host-applies-it-does-not-decide.md) and is what removes `wireguard`'s direct
|
|
database connection ([ADR 0049](0049-a-route-is-a-grant.md) does the same for the proxy).
|
|
|
|
**What does not change** is the rule the comment was defending. Role still does not imply
|
|
reachability — a home-hosted node is still a server that cannot be dialled. This record keeps
|
|
that lesson and stops encoding it as a pattern match.
|
|
|
|
## Consequences
|
|
|
|
- **CGNAT and IPv6 nodes become expressible**, which today they are not. Neither needs a code
|
|
change to support; they need a field that says what is true.
|
|
- **The lab stops needing its substitution.** TEST-NET-3 remains the right choice for a
|
|
documentation range, but the scenario now says *this segment is reachable* rather than relying
|
|
on an address shape to imply it. The constraint research 004 calls its most important fact
|
|
becomes an ordinary declaration, and the validator's enforcement of it becomes unnecessary
|
|
rather than load-bearing.
|
|
- **Hub election becomes queryable and renumbering becomes safe.** Both follow from the same
|
|
change and neither is possible today.
|
|
- **Two facts can now disagree, and something must say so.** Declared-versus-observed is a new
|
|
reportable condition and a new way to be wrong — a node declared reachable that is not will
|
|
fail exactly as it does today until somebody looks. The gain is that *looking* is now possible;
|
|
the regex offered nothing to compare against.
|
|
- **Somebody must set these when a node joins.** [ADR 0038](0038-a-node-joins-by-linking-first.md)
|
|
has the mesh finish the job after the link, so this is one more thing it finishes — and the
|
|
honest default (no endpoint, not a hub) is correct for every node except the ones somebody
|
|
deliberately publishes.
|
|
|
|
## References
|
|
|
|
- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) — the regex, the hub convention, and
|
|
the hairpin failure.
|
|
- [ADR 0037](0037-the-host-applies-it-does-not-decide.md) — why the graph is computed centrally.
|
|
- [ADR 0035](0035-a-picture-is-read-from-what-runs.md) — declared versus observed.
|
|
- [`08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md) — the design this serves.
|