jschoubben
cb2117f1c4
Two birth-address outages and a registry that would have been the third; a container that keeps a stale environment after its file changes; a host command that applied a converged declaration to an adopted node; the hub and the vault without seats. And the decision the operator made under it all: the hub adopts the predecessor's tunnel in place, key and peers and range and port.
7.0 KiB
topic, status, date, deciders, reconstructed, extends
| topic | status | date | deciders | reconstructed | extends |
|---|---|---|---|---|---|
| the mesh | accepted | 2026-09-23 | jochen | false | 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md |
105. The mesh adopts the predecessor's tunnel in place
Context
The control-node is adopted and its store and broker have been moved onto the ports the predecessor served them on, so the predecessor's other machines keep reaching them. They reach them over the predecessor's tunnel: a WireGuard interface on the control-node with three peers, an address range, and a port the hosting provider already lets through. The mesh's own private network runs beside it on a second interface, a second range and a second port — one the provider does not let through, so no other machine can join the mesh (research 012, the runbook's measurement of the upstream filter).
ADR 0100 says the mesh's range must not overlap a tunnel the predecessor still runs. That was written for coexistence. It leaves the migration with two tunnels for as long as any predecessor machine exists, and the second one unreachable.
The mesh already adopts in place where the thing found is the thing it would have raised: the store and the broker were taken over as modules, keyed on the container that was running (ADR 0078). A WireGuard interface is the same shape: a private key, a listening port, a list of peers by public key, an address. The mesh's is not different in kind from the predecessor's; it is a second one.
The operator's instruction: take over the tunnel interface, same range — everything stays the same.
Considered Options
- Two tunnels until the last predecessor machine is gone. Rejected: the mesh's stays unreachable from outside, so no machine can join, so the last predecessor machine is never gone.
- Move the mesh's tunnel onto the predecessor's port with the mesh's own key and range. The peers' packets arrive and are dropped — WireGuard authenticates by key, and the mesh's key is not the one they know. Every other machine loses its tunnel until it enrols, and it enrols over a bus it reaches through that tunnel. Rejected.
- Adopt the predecessor's tunnel in place: its private key, its peers, its range, its port. Adopted.
Decision
On an adopted node that is the hub, the private network takes over the tunnel it finds. The mesh's interface is raised with the found interface's private key, on its port, with its address and range, and every peer the found interface had — public key, allowed address — carried into the mesh's peer list as a peer not yet enrolled. The found interface is stopped, never flushed; its configuration stays on disk, kept like any held file.
Nothing a peer knows changes. A predecessor machine keeps the same server key, the same endpoint, the same address and the same route; it cannot tell the tunnel changed hands. When that machine enrols, it keeps its address: the mesh assigns an enrolling node the address the tunnel already had for its key, and only a node with no such address is given a fresh one from the range.
The mesh's own addresses are the range's. The controller composes every node's private address from the tunnel it holds, so adopting the predecessor's range moves the mesh's addresses with it — the hub's, and every binding, hosts-file entry and endpoint derived from it. Those are readers of the setting; they follow it, per ADR 0100's rule for ports. A reader that does not follow is issue 102.
This narrows ADR 0100. Its rule that the range must not overlap a tunnel the predecessor still runs applies to a node that is not adopting the tunnel: where the found tunnel is left running beside the mesh's, the ranges must differ. Where it is adopted, there is one tunnel and one range.
The guard's question answers itself. ADR 0103 admits the mesh's own ports only from the private network's interface. With one tunnel, the predecessor's peers arrive on it, and nothing needs to be admitted from an interface the mesh does not own.
Consequences
- The order of a migration changes. The hub's tunnel can be taken as soon as the node is adopted, before any service — and should be, because it is what lets other machines join. The runbook's step order is amended.
- The hub takes the provider's open port for free. The port the predecessor's tunnel used is by definition one the provider passes.
- A peer's identity precedes its enrolment. The mesh holds public keys and addresses for machines it has no record of. They are peers of the tunnel, not nodes of the mesh, until they enrol; the registry must be able to say both.
- What got harder: the private key of the found interface is read from the machine and becomes the mesh's — the one case where the mesh takes a credential it did not mint. It is sealed like any own secret from then on, and the found configuration file is kept, not copied further.
- Two documents currently say the opposite: ADR 0100's non-overlap rule (narrowed above) and the runbook's port plan, which is amended with this record.
How it is checked
A lab bed prepares a hub the way the predecessor leaves one: a WireGuard interface with a key, a port, a range and two peers, each peer a second machine that reaches a service on the hub through the tunnel. Then:
- Adopted, the tunnel changes hands and the peers notice nothing: the found interface is down and its file is on disk; the mesh's interface is up with the found key, port and address; each peer's service call succeeds before, during and after, with no reconfiguration on the peer.
- A peer enrols and keeps its address: the machine joins the mesh over the tunnel it already has, and its node address is the one the tunnel held for it.
- A new machine gets a fresh address from the same range, and reaches both the hub and the enrolled peer.
- Nothing derived from the address is stale: every binding, hosts entry and endpoint the controller composes says the adopted range, before and after a push.
Unit tests hold the controller to reading the hub's address and range from the adopted tunnel, assigning an enrolling node the address its key already had, and refusing to hand out an address the tunnel already holds; and the host to raising the mesh's interface with the found key and peers and stopping the found interface without flushing it.