jschoubben
f660620637
Implemented in mesh-controller #49 and mesh-host #24. One proposal was rejected on the record's own terms: converging the hub is not made to wait on other machines' migrations.
147 lines
9.1 KiB
Markdown
147 lines
9.1 KiB
Markdown
---
|
|
topic: the mesh
|
|
status: accepted
|
|
date: 2026-09-23
|
|
deciders: jochen
|
|
reconstructed: false
|
|
extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
|
---
|
|
|
|
# 105. The mesh adopts the predecessor's tunnel in place
|
|
|
|
## Context
|
|
|
|
The control-node is adopted and its store and broker have been moved onto the ports the
|
|
predecessor served them on, so the predecessor's other machines keep reaching them. They reach
|
|
them **over the predecessor's tunnel**: a WireGuard interface on the control-node with three
|
|
peers, an address range, and a port the hosting provider already lets through. The mesh's own
|
|
private network runs beside it on a second interface, a second range and a second port — one the
|
|
provider does not let through, so no other machine can join the mesh
|
|
([research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md), the runbook's
|
|
measurement of the upstream filter).
|
|
|
|
[ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) says the mesh's range *must not
|
|
overlap a tunnel the predecessor still runs*. That was written for coexistence. It leaves the
|
|
migration with two tunnels for as long as any predecessor machine exists, and the second one
|
|
unreachable.
|
|
|
|
The mesh already adopts in place where the thing found is the thing it would have raised: the
|
|
store and the broker were taken over as modules, keyed on the container that was running
|
|
([ADR 0078](0078-the-store-and-broker-are-modules.md)). A WireGuard interface is the same shape: a
|
|
private key, a listening port, a list of peers by public key, an address. The mesh's is not
|
|
different in kind from the predecessor's; it is a second one.
|
|
|
|
The operator's instruction: take over the tunnel interface, same range — everything stays the same.
|
|
|
|
## Considered Options
|
|
|
|
1. **Two tunnels until the last predecessor machine is gone.** Rejected: the mesh's stays
|
|
unreachable from outside, so no machine can join, so the last predecessor machine is never gone.
|
|
2. **Move the mesh's tunnel onto the predecessor's port with the mesh's own key and range.** The
|
|
peers' packets arrive and are dropped — WireGuard authenticates by key, and the mesh's key is not
|
|
the one they know. Every other machine loses its tunnel until it enrols, and it enrols over a bus
|
|
it reaches through that tunnel. Rejected.
|
|
3. **Adopt the predecessor's tunnel in place: its private key, its peers, its range, its port.**
|
|
Adopted.
|
|
|
|
## Decision
|
|
|
|
**On an adopted node that is the hub, the private network takes over the tunnel it finds.** The
|
|
mesh's interface is raised with the found interface's **private key**, on its **port**, with its
|
|
**address and range**, and every **peer** the found interface had — public key, allowed address —
|
|
carried into the mesh's peer list as a peer not yet enrolled. The found interface is stopped, never
|
|
flushed; its configuration stays on disk, kept like any held file.
|
|
|
|
**Nothing a peer knows changes.** A predecessor machine keeps the same server key, the same
|
|
endpoint, the same address and the same route; it cannot tell the tunnel changed hands. When that
|
|
machine enrols, it keeps its address: the mesh assigns an enrolling node the address the tunnel
|
|
already had for its key, and only a node with no such address is given a fresh one from the range.
|
|
|
|
**The mesh's own addresses are the range's.** The controller composes every node's private address
|
|
from the tunnel it holds, so adopting the predecessor's range moves the mesh's addresses with it —
|
|
the hub's, and every binding, hosts-file entry and endpoint derived from it. Those are readers of
|
|
the setting; they follow it, per ADR 0100's rule for ports. A reader that does not follow is
|
|
[issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md).
|
|
|
|
**This narrows ADR 0100.** Its rule that the range must not overlap a tunnel the predecessor still
|
|
runs applies to a node that is *not* adopting the tunnel: where the found tunnel is left running
|
|
beside the mesh's, the ranges must differ. Where it is adopted, there is one tunnel and one range.
|
|
|
|
**The guard's question answers itself.** [ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
|
|
admits the mesh's own ports only from the private network's interface. With one tunnel, the
|
|
predecessor's peers arrive on it, and nothing needs to be admitted from an interface the mesh does
|
|
not own.
|
|
|
|
## Consequences
|
|
|
|
- **The order of a migration changes.** The hub's tunnel can be taken as soon as the node is
|
|
adopted, before any service — and should be, because it is what lets other machines join. The
|
|
runbook's step order is amended.
|
|
- **The hub takes the provider's open port for free.** The port the predecessor's tunnel used is
|
|
by definition one the provider passes.
|
|
- **A peer's identity precedes its enrolment.** The mesh holds public keys and addresses for
|
|
machines it has no record of. They are peers of the tunnel, not nodes of the mesh, until they
|
|
enrol; the registry must be able to say both.
|
|
- **What got harder:** the private key of the found interface is read from the machine and becomes
|
|
the mesh's — the one case where the mesh takes a credential it did not mint. It is sealed like
|
|
any own secret from then on, and the found configuration file is kept, not copied further.
|
|
- Two documents currently say the opposite: ADR 0100's non-overlap rule (narrowed above) and the
|
|
runbook's port plan, which is amended with this record.
|
|
|
|
## How it is checked
|
|
|
|
A lab bed prepares a hub the way the predecessor leaves one: a WireGuard interface with a key, a
|
|
port, a range and two peers, each peer a second machine that reaches a service on the hub through
|
|
the tunnel. Then:
|
|
|
|
- **Adopted, the tunnel changes hands and the peers notice nothing**: the found interface is down
|
|
and its file is on disk; the mesh's interface is up with the found key, port and address; each
|
|
peer's service call succeeds before, during and after, with no reconfiguration on the peer.
|
|
- **A peer enrols and keeps its address**: the machine joins the mesh over the tunnel it already
|
|
has, and its node address is the one the tunnel held for it.
|
|
- **A new machine gets a fresh address from the same range**, and reaches both the hub and the
|
|
enrolled peer.
|
|
- **Nothing derived from the address is stale**: every binding, hosts entry and endpoint the
|
|
controller composes says the adopted range, before and after a push.
|
|
|
|
Unit tests hold the controller to reading the hub's address and range from the adopted tunnel,
|
|
assigning an enrolling node the address its key already had, and refusing to hand out an address
|
|
the tunnel already holds; and the host to raising the mesh's interface with the found key and
|
|
peers and stopping the found interface without flushing it.
|
|
|
|
## What review settled that this record did not
|
|
|
|
*Added 2026-09-24, from the review of the implementation. A decision record is not edited to change
|
|
its meaning; this says what was decided under it.*
|
|
|
|
- **A spoke's view of its hub is not a peer the mesh carries.** A predecessor gives a spoke the
|
|
whole subnet through the hub, so the spoke's found tunnel names one peer routed a range rather
|
|
than an address. Only the hub's peers are ever carried; a spoke presenting its own is skipped,
|
|
not refused — a machine enrols with what it found, and what it found is its route home.
|
|
- **The range and the carried peers outlive the flip.** They follow from the hub having taken the
|
|
tunnel over — its key being the tunnel's — and not from the node being adopted. A converged hub
|
|
keeps the range it adopted and the addresses it is holding, and `AssignAddress` keeps excluding
|
|
them.
|
|
- **Converging the hub is not refused while a carried peer has not enrolled.** Proposed in review
|
|
and rejected on the record's own terms: *a node converges when its migration is done*, and the
|
|
other machines' migrations are not this node's. With the range and the peers surviving the flip
|
|
there is nothing left for the refusal to protect, and it would have made one machine's converge
|
|
wait on every other machine.
|
|
- **A takeover whose placement disagrees with the tunnel is refused before it is composed** — an
|
|
address or a port that is not the tunnel's would stop the found interface and raise the mesh's
|
|
somewhere the peers are not, while reporting success.
|
|
- **The host says three things, not two**: the found interface still up, the mesh's up in its place,
|
|
or — the state worth naming — the found one down and the mesh's not up, which is the only one
|
|
where the peers reach nothing.
|
|
- **A hub that enrolled before this existed keeps its identity.** Re-enrolling would have remade
|
|
every credential in the mesh, because the hub provides the store and the broker. Instead the
|
|
machine takes the tunnel's key as its overlay key and says so in a message signed with the
|
|
identity it already has, so a forged report cannot move a node's key.
|
|
|
|
## References
|
|
|
|
- [ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md),
|
|
[ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
|
|
- [issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
|
|
- [research 012 — the minimum viable node](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md)
|