From 331cb94c6e652f3e953212d16d67ba36e9852527 Mon Sep 17 00:00:00 2001 From: jochens Date: Fri, 2 Oct 2026 13:13:15 +0200 Subject: [PATCH 1/2] ADR 0169 (proposed): a machine joins through the tunnel, and the bus is never public The bus was public only so a new machine could enrol before it had a tunnel. The machine now makes its tunnel key first, the token is issued for it and makes it a peer of the hub, and enrolment happens over the tunnel. --- ...-the-tunnel-and-the-bus-is-never-public.md | 89 +++++++++++++++++++ 02-DECISIONS/README.md | 1 + 2 files changed, 90 insertions(+) create mode 100644 02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md diff --git a/02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md b/02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md new file mode 100644 index 0000000..3cc43bf --- /dev/null +++ b/02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md @@ -0,0 +1,89 @@ +--- +topic: the mesh +status: proposed +date: 2026-10-02 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0004-a-node-and-how-it-joins.md +--- + +# 169. A machine joins through the tunnel, and the bus is never public + +## Context + +The bus is the one channel every machine depends on: enrolment, every declaration, every tool. The +`nats` module declares it reachable from the mesh only. The controller still opens it to the whole +internet on the machine that runs it, as a *foundation* port that no module declares and nothing may +close ([issue 051](../04-ISSUES/051-the-mesh-cannot-update-what-it-depends-on/00-report.md)). +The reason is joining. [ADR 0004](0004-a-node-and-how-it-joins.md) has a new machine enrol over the bus +**before** it has a tunnel. [ADR 0007](0007-connectivity.md) states it as a requirement: the node +running the broker must be reachable from wherever nodes are, at a stable address. + +So the bus listens on the internet permanently, for an event that happens a few times a year. A +sweep of every machine on 2026-10-02 found no client using the public path. Every connection arrives +over the tunnel or from the machine itself. The join token does not use it either: it carries the +controller's configured broker address, a mesh name with the old broker's port. + +ADR 0004 already says what a joining machine needs: *an identity, an address, and one peer to reach*. +The tunnel can be that peer, if the hub knows the new machine's key before the machine first knocks. +WireGuard answers nothing to a key it does not know, which is why the tunnel's own port is safe to +leave open where the bus's is not. + +## Considered Options + +1. **Keep the bus public.** It is authenticated and encrypted, but every exposure of it, and of the + server behind it, is exposure of the one thing everything depends on. +2. **Open the bus publicly only while a join token is live.** Small, and the hub is open only during a + join window. But the window is real, the rule is about time rather than about who may reach the + bus, and the opening and closing are pushes that can fail between them. +3. **The controller makes the new machine's tunnel key and puts it in the token.** One step for the + operator, but the private half leaves a machine it does not belong to. ADR 0004 refuses that for + every key a node holds. +4. **The machine makes its key first, and the token is issued for it.** The machine prints the public + half of its tunnel key. The operator issues the token for that key. The controller gives the + machine its address and adds it as a peer on the hub. The token carries the hub's tunnel endpoint + and key, the machine's address, and the bus's address on the private network. The machine brings + up its tunnel and enrols over it. + +## Decision + +**Option 4.** + +- **A machine makes its own tunnel key before it has a token**, and prints the public half. The private + half never leaves it, as ADR 0004 says of every key a node holds. +- **A token is issued for a tunnel key.** Issuing it assigns the machine's address on the private + network, records the key, and makes the machine a peer of the hub. The hub is sent that before the + token is shown, so the tunnel answers the moment the machine first uses it. +- **The token carries the one peer.** It adds the hub's tunnel endpoint and public key and the + machine's own address. **Where** becomes the bus's address on the private network, which needs no + name resolution. +- **The machine joins through the tunnel.** It brings the tunnel up from the token alone, then enrols + over it exactly as before. The enrolment checks that the key it is offered is the one the token was + issued for. +- **The bus is never public.** It is no longer a foundation port. Its reach is what the `nats` module + declares: the mesh. The tunnel's port stays open, as the one way in. + +This changes three things earlier records say. ADR 0004's *where* is the bus's private address, and the +token carries the peer. ADR 0007's requirement that the broker be reachable from wherever nodes are +becomes: **the hub's tunnel is**. Issue 051's broker port stops being a foundation port. + +## Consequences + +- Joining is two commands on the new machine, with the token issued between them. A token issued for + the wrong key gives a tunnel that never answers, and the machine says so rather than timing out at + the bus. +- An unused token leaves a peer on the hub until it expires. Expiry removes it, the same way it voids + the secret. +- A machine already in the mesh is unaffected: it reaches the bus over its tunnel today. +- The genesis machine, the first one, raises the bus on itself and needs no tunnel to reach it. + +## How this is checked + +| Rule | Checked by | +|---|---| +| A token is refused without a tunnel key, and carries the hub's peer and the machine's address | a controller test | +| Issuing a token makes the machine a peer of the hub before the token is shown | a controller test over the hub's composed tunnel | +| An expired, unused token's peer is gone from the hub | a controller test | +| Enrolment refuses a tunnel key other than the one the token was issued for | a controller test | +| No machine's filter opens the bus to anywhere | a controller test over the composed filter, and the live sweep from outside the mesh | +| A new machine joins from outside the hub's network with the bus closed to it | the lab, then by hand | diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index d1b924a..556dcb3 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -179,6 +179,7 @@ python3 00-META/checks/index.py fail if stale - **0163** — [Taking a module over is a comparison: what it compares, what it refuses, and what it carries](0163-taking-a-module-over-is-a-comparison.md) - **0167** — [A membership carries what its module receives, and who the mesh is](0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md) - **0168** — [A converged machine is filtered by the mesh alone, and the host says what else refuses](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md) +- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md) *(proposed)* ### Its tiers, from the bottom up -- 2.54.0 From 79642251a1ef456f2e85b846b3d8f039e7613f28 Mon Sep 17 00:00:00 2001 From: jochens Date: Fri, 2 Oct 2026 13:17:32 +0200 Subject: [PATCH 2/2] ADR 0169 accepted; design 08's join order starts with the tunnel --- ...-the-tunnel-and-the-bus-is-never-public.md | 2 +- 02-DECISIONS/README.md | 2 +- 03-DESIGN/01-to-be/08-connectivity.md | 23 +++++++++++++++++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md b/02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md index 3cc43bf..3f69b72 100644 --- a/02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md +++ b/02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md @@ -1,6 +1,6 @@ --- topic: the mesh -status: proposed +status: accepted date: 2026-10-02 deciders: jochen reconstructed: false diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 556dcb3..ee87d97 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -179,7 +179,7 @@ python3 00-META/checks/index.py fail if stale - **0163** — [Taking a module over is a comparison: what it compares, what it refuses, and what it carries](0163-taking-a-module-over-is-a-comparison.md) - **0167** — [A membership carries what its module receives, and who the mesh is](0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md) - **0168** — [A converged machine is filtered by the mesh alone, and the host says what else refuses](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md) -- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md) *(proposed)* +- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/08-connectivity.md b/03-DESIGN/01-to-be/08-connectivity.md index 78830f5..21fb8a9 100644 --- a/03-DESIGN/01-to-be/08-connectivity.md +++ b/03-DESIGN/01-to-be/08-connectivity.md @@ -9,6 +9,7 @@ code: - mesh-host internal/apply (the service that reflects a rule set) updated: 2026-10-02 decisions: + - 02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md - 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md - 02-DECISIONS/0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md - 02-DECISIONS/0148-the-meshs-names-are-resolved-not-copied-into-containers.md @@ -173,6 +174,28 @@ the broker's node must be dialable by every node, at a stable address, and so mu reachable; on one network it does not. A mesh whose nodes are all behind NAT cannot be raised, and a broker node whose address moves invalidates every token issued for it. +*2026-10-02.* **The order changes at step 1: the tunnel comes first, from the token** +([ADR 0169](../../02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)). +The circularity above is real, and it is broken differently. The overlay is configured by the mesh, +except for the one peer a joining machine needs, and the token carries that peer. So the sequence +becomes: + +``` +0 the node has an underlay address the machine's own +1 the node makes its tunnel key before any token; it prints the public half +2 a token is issued for that key its address assigned, and the hub sent it as a peer +3 the tunnel comes up to the hub from the token alone: the hub's endpoint and key, its address +4 the node dials the bus OVER THE TUNNEL, at the bus's private address +5 it proves itself, and is proved to enrolment, checking the key is the one the token named +6 the rest of the overlay the whole peer set, delivered as files +7 names, filtering, routes as before +``` + +The link no longer stays on the underlay. The bus is reached over the tunnel by every machine, +including one that is joining, so it is never opened to the internet. The precondition becomes: **the +hub's tunnel must be dialable by every node, at a stable address.** That port answers nothing to a +key it does not know. + **Whether the link should later move onto the overlay, with the underlay as fallback, is [open](../../02-DECISIONS/0007-connectivity.md).** It is a decision rather than a derivation: the gain is which network carries bytes, not what an attacker can reach, since the link is already -- 2.54.0