diff --git a/02-DECISIONS/0049-a-route-is-a-grant.md b/02-DECISIONS/0049-a-route-is-a-grant.md index 05871da..f0ac34e 100644 --- a/02-DECISIONS/0049-a-route-is-a-grant.md +++ b/02-DECISIONS/0049-a-route-is-a-grant.md @@ -95,8 +95,9 @@ shapes built. The proxy becomes a `container` with `file` configuration, which t knows how to apply and read back. And it removes a database credential from every node, which is half of what -[ADR 0039](0039-the-link-is-the-security-boundary.md) is for. The same move was already made for -`wireguard`; this applies it to the other offender. +[ADR 0039](0039-the-link-is-the-security-boundary.md) is for. +[ADR 0037](0037-the-host-applies-it-does-not-decide.md) decided this in principle; **neither +offender has actually been changed**, and this applies it to one of the two. ### A node without a public address is routed through one that has @@ -117,8 +118,10 @@ behind NAT* — as the case the lab exists to get right. - **Traefik's upward dependency is removed, and the module mostly disappears.** It computed its own configuration; now it is an image plus files somebody else derived. Of the 426 lines research 006 counted, what survives is a declaration. -- **One of two remaining database credentials leaves the node.** `wireguard` was the other, and - it is already handled — so this closes the set that ADR 0039 identified. +- **One of the two direct database connections goes.** `wireguard` is the other and is *not* + yet handled — [ADR 0050](0050-reachability-is-a-property-of-the-address.md) is what handles it, + and only both together close the set ADR 0039 identified. Until then a node still holds the + credential, so this record alone changes the design and not the exposure. - **Certificate issuance becomes a control-plane responsibility with a real constraint**: an ACME challenge can only be answered at a publicly reachable address, so issuance happens through a public node regardless of where the workload runs. The lab already runs its own @@ -130,6 +133,9 @@ behind NAT* — as the case the lab exists to get right. mesh, which needs something a joining node can verify before it trusts anything. **Internal identity and public exposure are two certificate stories and this record only closes the second.** Conflating them is what made the gap hard to see. + **Since closed** by [ADR 0051](0051-the-enrolment-token-carries-the-mesh.md): the joining node + verifies the control plane against a fingerprint carried in its enrolment token, so nothing + needs the CA before membership. - **Nothing says how a route is revoked** when a module is unassigned. The grant model implies it — removing a consumer drops what it was granted ([ADR 0045](0045-a-context-owns-its-store.md)) — but a stale public name pointing at nothing is diff --git a/02-DECISIONS/0050-reachability-is-a-property-of-the-address.md b/02-DECISIONS/0050-reachability-is-a-property-of-the-address.md new file mode 100644 index 0000000..1169ed3 --- /dev/null +++ b/02-DECISIONS/0050-reachability-is-a-property-of-the-address.md @@ -0,0 +1,119 @@ +--- +status: proposed +date: 2026-08-27 +deciders: jochen +reconstructed: false +extends: 0031-the-lab-provides-the-underlay.md +--- + +# 50. Reachability is declared, not inferred from an address + +## Context + +The overlay's peer graph is computed on each node by the `wireguard` module, which decides — +per pair — whether to write an `Endpoint` for a peer. The rule +([research 004](../01-RESEARCH/004-lab-network/analysis.md)) is a regular expression: + +```js +const isPrivate = (a) => /^(10\.|127\.|192\.168\.|172\.(1[6-9]|2\d|3[01])\.)/.test(a); +``` + +Not private ⇒ assumed reachable ⇒ an `Endpoint` is written. Private ⇒ no endpoint, and the peer +must initiate. + +The module's own comments record what this cost to arrive at: testing `profile === "server"` +was tried and was wrong, because a home-hosted node **is** a server and is not publicly +reachable — *"role does not imply reachability; the address does."* + +**That lesson is right and the implementation of it is not.** The address is evidence of +reachability; it is not the fact itself, and the gap between the two has already caused failures +and will cause more: + +| Address | The regex says | Actually | +|---|---|---| +| `100.64.0.0/10` — carrier-grade NAT ([RFC 6598](https://www.rfc-editor.org/rfc/rfc6598)) | **public** | **not reachable.** An endpoint is written to an address nothing can reach. | +| any IPv6 address, including `fd00::/8` unique-local | **public** | unique-local is not reachable; the regex tests v4 shapes only | +| a routable address behind a closed firewall | public | not reachable | +| `10.200.0.0/24` standing in for a public segment | private | reachable — this is the lab bug | + +The CGNAT row is the serious one. A node on a carrier-grade NAT address presents exactly the +failure already recorded for the hairpin case: *1.77 MiB sent, 0 B received, no handshake.* The +mesh silently never forms, and it presents as a WireGuard fault rather than an addressing one. + +The lab row is the same bug seen from the other side. [Research 004](../01-RESEARCH/004-lab-network/00-overview.md) +calls the required substitution *"the single most important fact in this document"* — a +simulated public segment must use TEST-NET-3, or nothing can ever initiate. **A test environment +having to choose its addresses to satisfy a regex is the regex telling us it is not a fact.** + +There is a second inference in the same code, and it is worse because nothing records it: + +> **Hub election is by convention.** The hub is the node whose `profile='server'` *and* whose +> overlay address begins `10.10.0.1`. A lab must assign that address to the node it intends as +> hub **or there will be no hub** — and nothing says so. + +An election decided by the first four characters of an address is not an election. It fails +silently, it cannot be queried, and it makes a renumbering into an outage. + +## Considered options + +1. **Fix the regex.** Add CGNAT, add IPv6, add the ranges as they are discovered. Rejected: the + list is unbounded, and each addition is written after the outage that revealed it. The + firewall case cannot be fixed at all — no address shape encodes it. +2. **Probe for reachability and cache the answer.** Attractive, and wrong as the *primary* + source: at the moment the graph is computed a node may be legitimately down, and a probe + cannot distinguish *unreachable* from *asleep* ([ADR 0036](0036-a-node-is-a-managed-machine.md)). + Deriving topology from a liveness check makes the overlay flap with the network. +3. **Declare it, and let observation contradict it.** Chosen. + +## Decision + +**A node's reachability is a declared fact on its record, not an inference from its address.** + +Two facts, and the mesh stores both: + +- **an endpoint, or none** — where peers may reach this node, if anywhere. Absent means *this + node initiates and is never dialled*, which is the safe default and the common case. +- **its role in the overlay** — whether it is a hub. **Declared, never derived from an address.** + +**The address remains evidence and stops being the fact.** When a node's observed endpoint +disagrees with its declared one, that is a **reportable condition**, not a silent correction — +the same discipline as [ADR 0035](0035-a-picture-is-read-from-what-runs.md): a picture is read +from the system, and where the reading disagrees with the intent, the disagreement is the +finding. + +**The peer graph is computed by the control plane**, from these declared facts, and delivered to +each node as configuration. It is not computed on the node, which is +[ADR 0037](0037-the-host-applies-it-does-not-decide.md) and is what removes `wireguard`'s direct +database connection ([ADR 0049](0049-a-route-is-a-grant.md) does the same for the proxy). + +**What does not change** is the rule the comment was defending. Role still does not imply +reachability — a home-hosted node is still a server that cannot be dialled. This record keeps +that lesson and stops encoding it as a pattern match. + +## Consequences + +- **CGNAT and IPv6 nodes become expressible**, which today they are not. Neither needs a code + change to support; they need a field that says what is true. +- **The lab stops needing its substitution.** TEST-NET-3 remains the right choice for a + documentation range, but the scenario now says *this segment is reachable* rather than relying + on an address shape to imply it. The constraint research 004 calls its most important fact + becomes an ordinary declaration, and the validator's enforcement of it becomes unnecessary + rather than load-bearing. +- **Hub election becomes queryable and renumbering becomes safe.** Both follow from the same + change and neither is possible today. +- **Two facts can now disagree, and something must say so.** Declared-versus-observed is a new + reportable condition and a new way to be wrong — a node declared reachable that is not will + fail exactly as it does today until somebody looks. The gain is that *looking* is now possible; + the regex offered nothing to compare against. +- **Somebody must set these when a node joins.** [ADR 0038](0038-a-node-joins-by-linking-first.md) + has the mesh finish the job after the link, so this is one more thing it finishes — and the + honest default (no endpoint, not a hub) is correct for every node except the ones somebody + deliberately publishes. + +## References + +- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) — the regex, the hub convention, and + the hairpin failure. +- [ADR 0037](0037-the-host-applies-it-does-not-decide.md) — why the graph is computed centrally. +- [ADR 0035](0035-a-picture-is-read-from-what-runs.md) — declared versus observed. +- [`08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md) — the design this serves. diff --git a/02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md b/02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md new file mode 100644 index 0000000..8c0b5e7 --- /dev/null +++ b/02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md @@ -0,0 +1,117 @@ +--- +status: proposed +date: 2026-08-27 +deciders: jochen +reconstructed: false +extends: 0039-the-link-is-the-security-boundary.md +--- + +# 51. The enrolment token carries where the mesh is and how to recognise it + +## Context + +[ADR 0039](0039-the-link-is-the-security-boundary.md) requires **mutual** authority: the node +proves it may join, and **the control plane proves it is the mesh**. It states why one-way is not +enough — the host applies whatever the link delivers, so *an attacker who can answer a joining +node's first call owns the machine.* + +It does not say **how** the control plane proves itself, and +[ADR 0049](0049-a-route-is-a-grant.md) deferred the question again, noting only that internal +identity and public exposure are two different certificate stories. + +Left unanswered it produces a circle. Verifying the mesh needs the mesh's CA. Obtaining the CA +means trusting whatever hands it over — which is the thing being verified. + +There is a second circle in the same place, and today they are solved by the same unfortunate +mechanism. A node must reach the mesh before the mesh has configured it, so it cannot yet +resolve any mesh name. [Research 004](../01-RESEARCH/004-lab-network/analysis.md) records the +workaround: + +> `dnsmasq-app` generates `.internal` names on each node and writes an `/etc/hosts` block **as a +> floor underneath, because a node must reach the mesh DB before its own DNS exists.** + +and, separately, that a joining node must have *"registry database and object-store host and +credentials, plus an npm token"* placed on disk beforehand — the credentials +[ADR 0039](0039-the-link-is-the-security-boundary.md) exists to remove. + +**Both circles are the same shape: a node needs some fact about the mesh before it has any +trustworthy way to obtain one.** Whatever supplies that fact must arrive by a path other than the +mesh. + +## Considered options + +1. **Ship the CA with the host binary.** Then the binary is mesh-specific, which + [ADR 0041](0041-the-host-depends-on-nothing.md) and + [ADR 0046](0046-the-installer-fetches-what-it-pins.md) both work to avoid, and rotating the + CA means rebuilding and redistributing the host everywhere. Rejected. +2. **Trust on first use, plainly.** Accept whatever answers the first call and pin it. Rejected: + it is exactly the attack ADR 0039 names, and the first call is the one moment the node has no + way to tell. +3. **A public certificate authority for the control plane's own endpoint.** Workable, and it + makes joining depend on public DNS and public issuance for a link that is otherwise entirely + the mesh's business. Rejected as a dependency, not as a technique — a mesh whose nodes cannot + join because an unrelated public authority is having a bad day has bought nothing. +4. **The token carries it.** Chosen. + +## Decision + +**The enrolment token carries three things**, and it is the only thing a joining node needs: + +| | | | +|---|---|---| +| **where** | an **address**, not a name | there is no resolution yet, and this is why none is needed | +| **who** | the fingerprint of the identity to expect | what makes the mesh provable rather than assumed | +| **the right to join** | the one-time secret ADR 0039 already specifies | useless once used, useless after it expires | + +The token is issued by the mesh for one enrolment and **carried out of band** — by the person +adopting the machine. That is what breaks both circles: its authenticity comes from the channel +it travelled, not from anything the node can check afterwards. + +**This is trust-on-first-use with the first use moved out of band**, which is the difference +between a pin and a guess. The node does not accept whatever answers; it accepts the one thing +it was told to expect, before it spoke to anything. + +### What this settles + +**The mesh CA is not a bootstrap concern.** It is how `.internal` names are certified once a node +is a member, and nothing needs it earlier. The open item +[ADR 0049](0049-a-route-is-a-grant.md) left — *what the control plane presents to a node that +trusts nothing yet* — is closed: it presents the identity whose fingerprint the token named. + +**Nothing needs name resolution before the link exists**, because the token carries an address. +The `/etc/hosts` floor exists to solve a problem that stops existing, and it should go rather than +be carried forward — a fallback nothing needs is a path nothing tests. + +**Nothing is placed on disk beforehand except the token.** No database credential, no object-store +credential, no registry token. That is ADR 0039's central claim finally made true at the one +moment it was still false, and it is the difference between *a node holds only its own identity* +being a design statement and being a fact. + +## Consequences + +- **The token becomes security-critical in a way it was not**, because it now carries the pin. + Tampering with it in transit substitutes the mesh. That is a real exposure and it is strictly + better than the alternative: without a pin there is nothing to tamper *with*, and the node + trusts the first answer unconditionally. The exposure moves to a channel a person controls and + can verify, from one nobody could. +- **Token delivery is now a designed step, not an incidental one.** It is short-lived and + single-use, so interception is bounded — but how it reaches a machine is part of adoption and + needs saying. [Research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md) is + where that belongs. +- **Rotating the control plane's identity invalidates outstanding tokens**, which is correct and + needs to fail legibly. A node presenting a token with a stale fingerprint must be told that, + not left to time out. +- **The address in the token can go stale.** If the control plane moves, unissued tokens point + somewhere wrong. Tokens are short-lived, which bounds it; moving the control plane is + [`06`](../03-DESIGN/01-to-be/06-the-control-plane.md)'s undesigned territory regardless. +- **A rejoining node is an ordinary case, not a special one.** A node that has lost its identity + gets a new token. There is no recovery path to design because there is no long-lived secret to + recover. + +## References + +- [ADR 0039](0039-the-link-is-the-security-boundary.md) — the mutual authority this implements. +- [ADR 0038](0038-a-node-joins-by-linking-first.md) — the join this is the first step of. +- [ADR 0049](0049-a-route-is-a-grant.md) — the open item this closes. +- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) — the `/etc/hosts` floor and the + credentials placed beforehand. diff --git a/02-DECISIONS/0052-a-filter-rule-names-its-source.md b/02-DECISIONS/0052-a-filter-rule-names-its-source.md new file mode 100644 index 0000000..ee2582f --- /dev/null +++ b/02-DECISIONS/0052-a-filter-rule-names-its-source.md @@ -0,0 +1,71 @@ +--- +status: proposed +date: 2026-08-27 +deciders: jochen +reconstructed: false +extends: 0043-a-declaration-is-an-ordered-list-of-owned-resources.md +--- + +# 52. A filter rule names its source, or it is not a rule + +## Context + +[Research 004](../01-RESEARCH/004-lab-network/analysis.md) found this while looking for something +else: + +> Several manifests declare `scope: public` on firewall rules — `wireguard`, `traefik`, `gitea`, +> `mailu`, `qbittorrent`. It is **not part of the rule type** and is **referenced by no code** in +> the firewall path. Real scoping is done with `from:`. + +**So five manifests appear to restrict a port and restrict nothing.** Anyone reading them — +including whoever wrote the next one by copying — sees an access control that does not exist. + +Research 004 already names the shape: it is `how-we-build`'s *an unenforced rule is +indistinguishable from a wrong one, and costs more, because people believe it.* This is that +rule, in the firewall, on the modules most worth restricting. + +The mechanism that let it happen is worth more than the instance. `scope:` was accepted because +**unknown keys were ignored**. Nothing rejected it, nothing warned, and it spread by copying for +long enough to reach five manifests. + +## Decision + +**A rule names its source.** `from:` is the only way to scope a rule, and a rule without one is +open — which it must therefore say plainly rather than imply otherwise. + +**`scope:` is removed, not implemented.** Giving it meaning would leave two ways to express one +thing, and a manifest carrying both would need a precedence rule nobody would remember. The five +manifests are corrected to `from:` where they meant to restrict something, and left open where +they did not — and finding out which is which is part of the work, not a formality. + +**An unknown key is refused.** This is the general fix and the reason to bother: + +> A manifest carrying a key the schema does not define is **rejected**, naming the key. + +The host already works this way — its declaration parser sets `DisallowUnknownFields` and +collects every problem into one refusal +([ADR 0043](0043-a-declaration-is-an-ordered-list-of-owned-resources.md)). **Manifests are the +layer where that discipline is missing**, and `scope:` is what missing looks like: not a wrong +value, an invented one, silently accepted for months. + +## Consequences + +- **This class of fiction stops at validation** rather than at an audit. A misspelled `form:`, + an invented `scope:`, a key from a different schema — each fails on the manifest that + introduces it, once, instead of spreading. +- **Existing manifests will fail validation**, and some of those failures will be keys somebody + believed were doing something. That is the finding, not the cost — but it means the refusal + cannot be switched on without reading every manifest first. +- **The firewall becomes reviewable.** Today a rule's real effect is only visible by knowing + which keys are fictional. Afterwards the manifest says what happens. +- **Five manifests need a decision each**, and `wireguard` and `traefik` genuinely are open to + the world — they must be, which the manifest should state rather than appear to deny. +- **It does not make the rules correct**, only honest. A rule that says `from:` anywhere is + legitimately open; this record ensures it says so. + +## References + +- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) §7 — the finding. +- [`how-we-build.md`](../00-META/how-we-build.md) — the rule about unenforced rules. +- [ADR 0043](0043-a-declaration-is-an-ordered-list-of-owned-resources.md) — unknown-is-refused, + where it already holds. diff --git a/03-DESIGN/01-to-be/06-the-control-plane.md b/03-DESIGN/01-to-be/06-the-control-plane.md index 4b7b66f..b1019e3 100644 --- a/03-DESIGN/01-to-be/06-the-control-plane.md +++ b/03-DESIGN/01-to-be/06-the-control-plane.md @@ -49,7 +49,7 @@ Ten contexts and one interface, from the skeleton | **record** | the event log every other context integrates through | | **inventory** | nodes, modules, assignments, versions | | **config** | settings, secrets, and deriving them onto nodes | -| **connectivity** | overlay, resolution, exposure, filtering, certificates — it *decides* routes; the proxy on a node applies them ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)) | +| **connectivity** | overlay, resolution, exposure, filtering, certificates — **specified in full in [`08-connectivity.md`](08-connectivity.md)** | | **provisioning** | resource grants between modules | | **delivery** | source to artifact to node | | **observability** | health, logs, metrics, alerts | diff --git a/03-DESIGN/01-to-be/08-connectivity.md b/03-DESIGN/01-to-be/08-connectivity.md new file mode 100644 index 0000000..6bf8c74 --- /dev/null +++ b/03-DESIGN/01-to-be/08-connectivity.md @@ -0,0 +1,233 @@ +--- +layer: to-be +status: designed +code: [] +updated: 2026-08-27 +decisions: + - 02-DECISIONS/0037-the-host-applies-it-does-not-decide.md + - 02-DECISIONS/0039-the-link-is-the-security-boundary.md + - 02-DECISIONS/0049-a-route-is-a-grant.md + - 02-DECISIONS/0050-reachability-is-a-property-of-the-address.md + - 02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md + - 02-DECISIONS/0052-a-filter-rule-names-its-source.md +--- + +# Connectivity + +One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most +moving parts: **overlay, resolution, exposure, filtering, certificates.** + +It is written as a whole because the five are one design. They share inputs, they must agree, and +every one of them today is computed in a different place by a different module from a different +copy of the same facts. + +## Why it is control-plane work + +Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one +node* — to each responsibility: + +| | needs to know | whose | +|---|---|---| +| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane | +| **resolution** — which name is which node | **every node** | control plane | +| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)) | control plane | +| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies | +| **certificates** — who may present which name | which name belongs to which node | control plane | + +**Not one of the five can be answered by a machine on its own.** That is the whole reason this is +a context rather than a set of node-local modules — and it is exactly what the current +arrangement gets wrong, by computing all five on the node from a direct database connection. + +## The shape: decided centrally, delivered as files + +Every one of the five resolves the same way, and it is worth stating once rather than five times: + +> **The connectivity context computes the configuration. It arrives over the link as `file` +> resources. The service reads files and knows nothing about the mesh.** + +This costs **no new host vocabulary**. `file`, `directory`, `service` and `container` already +exist; WireGuard, the resolver and the proxy are all *a container or a package, plus files*. + +It is also what removes the last two upward dependencies. +[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules +opening a direct connection to the control plane's database — `wireguard` and `traefik` — and +they are the reason every node permanently holds a credential to it +([ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)). Both are connectivity +modules. **Closing this context closes that set.** + +## The order it comes up in + +The one thing to get right, because everything else depends on it: + +``` +0 the node has an underlay address the machine's own — DHCP, or a provider gave it one +1 the node dials the mesh OVER THE UNDERLAY, at the address in its token +2 it proves itself, and is proved to the link exists (ADR 0039, ADR 0051) +3 the mesh grants it an identity and an overlay address +4 the overlay comes up peer graph delivered as files +5 names resolve resolver config delivered as files +6 filtering is applied derived from what is assigned here +7 routes and certificates once this node has something to expose +``` + +**Step 1 runs on the underlay and never on the overlay.** This is the circularity that must not +be created: the overlay is configured by the mesh, so a link that required the overlay could +never be established on a new node. The link stays on the underlay permanently — it is +outbound-only and carries its own identity, so it needs nothing the overlay provides. + +**Nothing before step 3 can resolve a mesh name**, which is why the token carries an *address* +([ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)). Today this is +patched with an `/etc/hosts` floor written underneath the resolver; under this design there is +nothing to patch. + +## 1 — The overlay + +**What is decided:** the peer graph. For every node: its overlay address, which peers it holds, +which of those it may dial, and which must dial it. + +**Inputs, all declared:** + +- **reachability** — an endpoint, or none + ([ADR 0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md)). Not + inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and + wrong for a routable address behind a closed firewall. +- **site** — where the machine physically is, or nothing if it roams. +- **role** — hub or not, **declared**. Today it is inferred from an address prefix, which means a + renumbering is an outage and nothing can be asked which node is the hub. + +**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the +public key is published to the mesh. This is already true and it is already right — it is +[ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)'s *a node holds its own +identity* applied to the overlay, and it means the control plane computes a graph it cannot +itself impersonate. + +**Shape: a hub, with direct peering between co-located nodes.** + +| | | +|---|---| +| two nodes at the same site | peer **directly**, host-routed, with a keepalive | +| everything else | routes through the **hub** | +| a node with no site — it roams | **hub only** | + +**Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a +preference: there is no failover.** A more specific route to a dead endpoint blackholes; it does +not fall back to the general one. So a node whose location changes gets exactly one path, because +two paths would mean one of them silently swallowing traffic. + +**What the host receives:** an interface configuration and a peer list, as files. It does not +compute them, and after this it holds no credential to the mesh's database. + +## 2 — Resolution + +**Two name spaces, and they do not mix:** + +| | resolves to | certified by | +|---|---|---| +| **internal names** | overlay addresses | the **mesh CA** | +| **public names** | whatever the outside world must reach | a **public authority** | + +A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact +used by things outside the mesh — and the separation carries two lessons that were learned +expensively enough to be worth restating: + +- **Mesh names are not multicast names.** A name resolved by local multicast discovery introduces + a delay and a failure mode that appears on one node and not others — the worst shape a fault + can have. +- **A node must not pin its own public name locally.** The duplicate record breaks resolution of + that name for everything else that needs it. + +**What the host receives:** the resolver's configuration, as files, listing every peer's internal +name and overlay address. + +**What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh +database before its own DNS existed; with [ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md) +nothing needs a name before the link, and a fallback nothing needs is a path nothing tests. + +## 3 — Exposure + +Settled by [ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md); summarised here because +this is where it belongs. + +**A route is a grant.** A module that must be reachable declares it needs one; the proxy provides +it and hands back the public name. Ordinary +[ADR 0044](../../02-DECISIONS/0044-a-module-declares-presence-instantiation-and-exclusion.md) +vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a +name rather than supplying nothing and receiving credentials. + +**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is +the case is a mesh-level fact, which is the fourth reason exposure is control-plane work. + +## 4 — Filtering + +**Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a +consequence of what runs on it and who must reach it, not an independent declaration to keep in +step by hand. + +**A rule names its source** ([ADR 0052](../../02-DECISIONS/0052-a-filter-rule-names-its-source.md)). +A rule with no source is open, and must say so rather than appear to restrict something. `scope:` +is removed rather than implemented: five manifests carry it today, it is referenced by no code, +and it is the clearest instance in the repository of *an unenforced rule is indistinguishable +from a wrong one, and costs more, because people believe it.* + +**Unknown keys are refused** — the discipline the host's declaration parser already has +([ADR 0043](../../02-DECISIONS/0043-a-declaration-is-an-ordered-list-of-owned-resources.md)), and +the one manifests lack. `scope:` survived because nothing rejected it. + +## 5 — Certificates + +**Two authorities, kept separate on purpose.** + +| | issued by | for | +|---|---|---| +| **public names** | a public ACME authority | anything outside the mesh reaches | +| **internal names** | the **mesh CA** | node-to-node, over the overlay | + +**The split is not collapsed, including in the lab.** A single-CA lab would hide any bug living +in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA +unchanged ([research 004](../../01-RESEARCH/004-lab-network/00-overview.md)). + +**Public issuance requires genuine public reachability.** The HTTP-01 challenge must be answered +at the name being certified, so issuance happens through a publicly reachable node regardless of +where the workload runs — the same asymmetry as exposure, for the same reason. + +**The issuer must be configurable.** Today it is not: the proxy sets no `caServer` and therefore +defaults to the public authority's *production* endpoint. Two consequences, and the second is +worse than the lab problem that found it — every certificate experiment on a real node consumes +production issuance quota, and a retry loop can exhaust it for a week. + +**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the +fingerprint in its token ([ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)), +so nothing needs the CA before membership. It certifies internal names afterwards, and that is +all it does. + +## What this removes + +The list is worth having in one place, because it is most of the argument: + +- **The last two direct database connections from nodes** — `wireguard` and `traefik`, the only + two, both connectivity. +- **Therefore the database credential on every node**, and the object-store credential beside it. + [ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)'s central claim becomes + true rather than aspirational. +- **The `/etc/hosts` floor**, and the bootstrap circularity it patched. +- **Hub election by address prefix**, and the silent no-hub failure when nobody knew the + convention. +- **The RFC1918 inference**, and the lab substitution that existed to satisfy it. +- **`scope:`**, and the class of manifest key that means nothing. + +## Open + +- **What happens when the hub is down.** WireGuard has no failover, and the hub is a single point + through which every non-co-located pair routes. This design does not add a second hub and does + not pretend the first one is redundant. It is the same shape as + [`06`](06-the-control-plane.md)'s *how many run, and what a node does without one*, and it + should be answered with it rather than separately. +- **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from + an address, but no procedure exists, and a graph delivered node by node has an ordering problem + while it is half-applied. +- **Revoking a route** when a module is unassigned ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)). + A stale public name pointing at nothing fails more visibly than a stale grant. +- **IPv6.** [ADR 0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md) makes + it expressible; nothing here says the overlay or the resolver handle it. +- **Reporting declared-versus-observed.** ADR 0050 makes the disagreement detectable and does not + say who looks or what they are told. diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index a8800c7..201d13f 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -16,13 +16,15 @@ document is written and this one's status becomes `implemented`. | [`04-lab-installation.md`](04-lab-installation.md) | Getting the lab onto a clean machine, and why it verifies capability rather than installation | [ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) | | [`05-the-node-host.md`](05-the-node-host.md) | Tier 0 — the one thing installed by hand, and the only thing that changes a machine | [ADR 0037](../../02-DECISIONS/0037-the-host-applies-it-does-not-decide.md) | | [`06-the-control-plane.md`](06-the-control-plane.md) | Tier 2 — what the term means, and the test for what belongs in it | [ADR 0037](../../02-DECISIONS/0037-the-host-applies-it-does-not-decide.md) | -| [`07-the-substrate.md`](07-the-substrate.md) | Tier 1 — what the control plane consumes and cannot grant itself | [ADR 0038](../../02-DECISIONS/0038-a-node-joins-by-linking-first.md) | +| [`07-the-substrate.md`](07-the-substrate.md) | Tier 1 — what the control plane consumes and cannot grant itself | [ADR 0038](../../02-DECISIONS/0038-a-node-joins-by-linking-first.md), [0048](../../02-DECISIONS/0048-the-substrate-is-named.md) | +| [`08-connectivity.md`](08-connectivity.md) | One context in full — overlay, resolution, exposure, filtering, certificates | [ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md), [0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md), [0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md) | ## Not yet written -- **The eight bounded contexts.** [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md) - decides the decomposition; the per-context specifications do not exist yet. The work - breakdown says in what order they are needed. +- **The remaining contexts.** [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md) + decides the decomposition; `connectivity` is the first written in full + ([`08`](08-connectivity.md)) and the others do not exist yet. The work breakdown says in what + order they are needed. - **Domain grouping outside the core.** [ADR 0017](../../02-DECISIONS/0017-modules-outside-the-core-are-grouped-by-domain.md) settles the principle and explicitly does not settle the domain list. That is a research effort, not a design document, until it concludes.