Merge pull request 'ADR 0169 (proposed): a machine joins through the tunnel, and the bus is never public' (#284) from jschoubben/a-machine-joins-through-the-tunnel into main

2026-10-02 11:17:42 +00:00
parent 78351560f7 79642251a1
commit aa5d9f1045
3 changed files with 113 additions and 0 deletions
@@ -0,0 +1,89 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-02
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0004-a-node-and-how-it-joins.md
+---
+
+# 169. A machine joins through the tunnel, and the bus is never public
+
+## Context
+
+The bus is the one channel every machine depends on: enrolment, every declaration, every tool. The
+`nats` module declares it reachable from the mesh only. The controller still opens it to the whole
+internet on the machine that runs it, as a *foundation* port that no module declares and nothing may
+close ([issue 051](../04-ISSUES/051-the-mesh-cannot-update-what-it-depends-on/00-report.md)).
+The reason is joining. [ADR 0004](0004-a-node-and-how-it-joins.md) has a new machine enrol over the bus
+**before** it has a tunnel. [ADR 0007](0007-connectivity.md) states it as a requirement: the node
+running the broker must be reachable from wherever nodes are, at a stable address.
+
+So the bus listens on the internet permanently, for an event that happens a few times a year. A
+sweep of every machine on 2026-10-02 found no client using the public path. Every connection arrives
+over the tunnel or from the machine itself. The join token does not use it either: it carries the
+controller's configured broker address, a mesh name with the old broker's port.
+
+ADR 0004 already says what a joining machine needs: *an identity, an address, and one peer to reach*.
+The tunnel can be that peer, if the hub knows the new machine's key before the machine first knocks.
+WireGuard answers nothing to a key it does not know, which is why the tunnel's own port is safe to
+leave open where the bus's is not.
+
+## Considered Options
+
+1. **Keep the bus public.** It is authenticated and encrypted, but every exposure of it, and of the
+   server behind it, is exposure of the one thing everything depends on.
+2. **Open the bus publicly only while a join token is live.** Small, and the hub is open only during a
+   join window. But the window is real, the rule is about time rather than about who may reach the
+   bus, and the opening and closing are pushes that can fail between them.
+3. **The controller makes the new machine's tunnel key and puts it in the token.** One step for the
+   operator, but the private half leaves a machine it does not belong to. ADR 0004 refuses that for
+   every key a node holds.
+4. **The machine makes its key first, and the token is issued for it.** The machine prints the public
+   half of its tunnel key. The operator issues the token for that key. The controller gives the
+   machine its address and adds it as a peer on the hub. The token carries the hub's tunnel endpoint
+   and key, the machine's address, and the bus's address on the private network. The machine brings
+   up its tunnel and enrols over it.
+
+## Decision
+
+**Option 4.**
+
+- **A machine makes its own tunnel key before it has a token**, and prints the public half. The private
+  half never leaves it, as ADR 0004 says of every key a node holds.
+- **A token is issued for a tunnel key.** Issuing it assigns the machine's address on the private
+  network, records the key, and makes the machine a peer of the hub. The hub is sent that before the
+  token is shown, so the tunnel answers the moment the machine first uses it.
+- **The token carries the one peer.** It adds the hub's tunnel endpoint and public key and the
+  machine's own address. **Where** becomes the bus's address on the private network, which needs no
+  name resolution.
+- **The machine joins through the tunnel.** It brings the tunnel up from the token alone, then enrols
+  over it exactly as before. The enrolment checks that the key it is offered is the one the token was
+  issued for.
+- **The bus is never public.** It is no longer a foundation port. Its reach is what the `nats` module
+  declares: the mesh. The tunnel's port stays open, as the one way in.
+
+This changes three things earlier records say. ADR 0004's *where* is the bus's private address, and the
+token carries the peer. ADR 0007's requirement that the broker be reachable from wherever nodes are
+becomes: **the hub's tunnel is**. Issue 051's broker port stops being a foundation port.
+
+## Consequences
+
+- Joining is two commands on the new machine, with the token issued between them. A token issued for
+  the wrong key gives a tunnel that never answers, and the machine says so rather than timing out at
+  the bus.
+- An unused token leaves a peer on the hub until it expires. Expiry removes it, the same way it voids
+  the secret.
+- A machine already in the mesh is unaffected: it reaches the bus over its tunnel today.
+- The genesis machine, the first one, raises the bus on itself and needs no tunnel to reach it.
+
+## How this is checked
+
+| Rule | Checked by |
+|---|---|
+| A token is refused without a tunnel key, and carries the hub's peer and the machine's address | a controller test |
+| Issuing a token makes the machine a peer of the hub before the token is shown | a controller test over the hub's composed tunnel |
+| An expired, unused token's peer is gone from the hub | a controller test |
+| Enrolment refuses a tunnel key other than the one the token was issued for | a controller test |
+| No machine's filter opens the bus to anywhere | a controller test over the composed filter, and the live sweep from outside the mesh |
+| A new machine joins from outside the hub's network with the bus closed to it | the lab, then by hand |
@@ -179,6 +179,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0163** — [Taking a module over is a comparison: what it compares, what it refuses, and what it carries](0163-taking-a-module-over-is-a-comparison.md)
 - **0167** — [A membership carries what its module receives, and who the mesh is](0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md)
 - **0168** — [A converged machine is filtered by the mesh alone, and the host says what else refuses](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)
+- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)

 ### Its tiers, from the bottom up

@@ -9,6 +9,7 @@ code:
  - mesh-host internal/apply (the service that reflects a rule set)
 updated: 2026-10-02
 decisions:
+  - 02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md
  - 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md
  - 02-DECISIONS/0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md
  - 02-DECISIONS/0148-the-meshs-names-are-resolved-not-copied-into-containers.md
@@ -173,6 +174,28 @@ the broker's node must be dialable by every node, at a stable address, and so mu
 reachable; on one network it does not. A mesh whose nodes are all behind NAT cannot be raised, and
 a broker node whose address moves invalidates every token issued for it.

+*2026-10-02.* **The order changes at step 1: the tunnel comes first, from the token**
+([ADR 0169](../../02-DECISIONS/0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)).
+The circularity above is real, and it is broken differently. The overlay is configured by the mesh,
+except for the one peer a joining machine needs, and the token carries that peer. So the sequence
+becomes:
+
+```
+0  the node has an underlay address     the machine's own
+1  the node makes its tunnel key        before any token; it prints the public half
+2  a token is issued for that key       its address assigned, and the hub sent it as a peer
+3  the tunnel comes up to the hub       from the token alone: the hub's endpoint and key, its address
+4  the node dials the bus               OVER THE TUNNEL, at the bus's private address
+5  it proves itself, and is proved to   enrolment, checking the key is the one the token named
+6  the rest of the overlay              the whole peer set, delivered as files
+7  names, filtering, routes             as before
+```
+
+The link no longer stays on the underlay. The bus is reached over the tunnel by every machine,
+including one that is joining, so it is never opened to the internet. The precondition becomes: **the
+hub's tunnel must be dialable by every node, at a stable address.** That port answers nothing to a
+key it does not know.
+
 **Whether the link should later move onto the overlay, with the underlay as fallback, is
 [open](../../02-DECISIONS/0007-connectivity.md).** It is a decision rather than a derivation: the
 gain is which network carries bytes, not what an attacker can reach, since the link is already