13 KiB
topic, status, date, deciders, reconstructed, extends
| topic | status | date | deciders | reconstructed | extends |
|---|---|---|---|---|---|
| the mesh | accepted | 2026-09-22 | jochen | false | 02-DECISIONS/0078-the-store-and-broker-are-modules.md |
100. A node in use is adopted before it is converged
Context
The mesh replaces a predecessor mesh that is running, on the same machines, with the services people use. The control-node is the machine that already carries the predecessor's broker and build pipeline. The operator proposed the migration's shape: stop the predecessor's control on a machine, bring the mesh up there in an adoption mode that keeps the machine's configuration in force, migrate its modules one at a time, move to the next machine, and flip adoption off when every machine is done.
Measured on the control-node (research 012, migrating a node that is in use): 60 predecessor containers; the predecessor's control in user-level daemons separate from its services; its firewall active, with 54 incoming and 52 forwarding rules allowing each served port explicitly; 12 files its configuration sync writes, 3 of them system files.
Four things in the mesh as it stands break that shape:
- The base ruleset closes the machine. Genesis loads a drop-by-default table (ADR 0088). Every base chain at a hook runs in priority order; an accept ends only its own chain and a drop in any is final — whether the other firewall's chains are nftables or legacy iptables. So the predecessor's allowed ports would close at genesis.
- The host replaces what it finds. A declared file is written whatever is at its path, except a file declared create-once. A declared container replaces a running one of the same name. So assigning a module the predecessor also runs replaces the predecessor's service and its files at once.
- Some foundation ports are held. The foundation's store and bus ports are free — the predecessor publishes its own elsewhere — but the registry's port and the broker's management port are held, and on a control-node that is the private network's hub, so is the private network's port. The foundation's ports are fixed in the installer's bundle and the catalogue's manifests, so a collision surfaces as a container that fails to bind, and a port changed at genesis would be changed back when the foundation is adopted as modules.
- Published container ports are forwarded, not received. The foundation publishes its ports on every interface; a predecessor firewall that filters only incoming traffic never sees them. The base ruleset is what keeps the store unreachable from outside, and it is the thing that cannot be loaded.
The mesh already adopts in one place: the foundation's store and broker are taken over in place as modules, keyed on the container that is already running (ADR 0078). The research this record draws on already settled the conflict rule for adoption: on conflict, what is on the machine stays.
Considered Options
- Cut each machine over in one go — stop the predecessor's store, broker, registry and proxy, raise the foundation in their place. Rejected: every predecessor service on the machine is down until it has migrated, the predecessor's other machines lose their broker, and the rollback is restarting the predecessor — a recovery, not a step.
- Put the control-node on a separate machine. Rejected: the control-node is decided.
- Converge on joining, as today. Rejected: the base ruleset closes the machine at genesis, and the predecessor's services and files are replaced as soon as any module naming them is assigned.
- Only make the foundation's ports configurable. Rejected as insufficient: it answers the bind collisions and neither the firewall nor the files.
- Treat assigning a module as migrating it. Considered and rejected on review: it makes the rule that keeps found files never fire — the host only ever sees files of assigned modules — and it makes an assignment on an adopted node an outage rather than a preparation.
- Adoption as a mode per node; each module taken explicitly; the node converged by an explicit flip. Adopted.
Decision
A node is adopted or converged, and the controller records which. The operator says so: at genesis for the control-node, and in the enrolment token for the others. The controller is authoritative, and every declaration it sends says whether the node is adopted and which modules have been taken on it. A node stays adopted until the operator converges it. An adopted node is said to be adopted wherever the mesh reports a node's state.
A converged genesis refuses a machine in use. Raised without saying adopted on a machine with an active firewall or services listening that are not the mesh's, genesis refuses and names what it found — a forgotten flag must not close a working machine.
Before a node is adopted, its predecessor's control is stopped by the operator — the daemons that write its configuration. Its services keep running on what they have.
Found means present with no record. A file at a declared path, or a container at a declared name, that the host's store has no record of writing is found. A file the host wrote in an earlier life of the node is not found; its record says so.
On an adopted node, what is found is kept until its module is taken. The host keeps a found file and a found container as they are, records the file's original content before anything else happens to it, and reports each as held. Assigning a module on an adopted node prepares it: what the module declares that is not found is created; what is found is held. Taking a module on a node is its cutover — the operator's act, done when that module's data has moved — and from then on the module's resources converge on that node like any other. A held file that changes while held is reported as changed by something else, not reverted: that is how a predecessor still writing is caught. A held file whose module is unassigned is left where it is, never removed.
The firewall found on the machine stays in force. The mesh loads no table of its own on an adopted node — neither genesis's base ruleset nor the filter module's derived one. What the mesh needs is declared as openings: a resource that says a port is reachable, from where, on the incoming path or the forwarded path — a published container port is forwarded. The host converges an opening through the found firewall in that firewall's own terms, marks it as the mesh's, and removes only what it marked; it re-checks each opening on every reconcile, so a reload or a reboot of the found firewall does not lose it. An opening is a state, not a command, which is what lets it travel over the link. The host reports which firewall it found, and a machine with a kind no host speaks is refused adoption rather than adopted with nothing protecting the mesh's ports.
The mesh protects its own ports itself. On an adopted node its foundation's ports get openings from the private network and marked refusals from anywhere else, on the forwarded path — so the store is unreachable from outside whether or not the found firewall filters forwarded traffic.
The foundation's ports are the node's. Every port the foundation binds is an input to genesis, checked free before anything is raised, refused with the name of what holds it. The ports given become that node's settings for the foundation's modules — the catalogue's numbers are only their defaults — and every place that uses them reads them from there: the modules' containers, the filter, the base ruleset, the private network's endpoint, and the addresses consumers are given. The private network's address range must not overlap a tunnel the predecessor still runs; genesis checks that too.
Converging a node is one act, previewed. The preview lists what is reachable on the machine now — every listening socket and every published container port — and for each whether an assigned module declares it or it will close. The flip then takes every module not yet taken, loads the mesh's derived filter, and retires the found firewall by disabling it, never by flushing: the container runtime's rules and the found firewall's own configuration stay on disk. Returning a converged node to adopted re-enables the firewall it retired. A node converges when its migration is done; the mesh is migrated when every node has converged.
The order is the operator's: the control-node first, adopted, its modules assigned and taken one at a time; then each other machine, adopted, migrated, converged in turn. The predecessor's pipeline runs on the control-node, so its updates stop for every machine while the migration runs; that is accepted.
Consequences
Each step says what it changes before it changes it. Adopting a node changes nothing that serves; assigning a module adds what is not there; taking a module replaces one service; the flip replaces the firewall, after naming every port it will close. Two steps are not undone by the mesh: taking a module replaces the predecessor's container, and the kept original of a file is recorded but not yet restored by any act of the mesh (research 012 leaves where it lives open).
What got harder:
- The mesh must speak a firewall it did not install, on both the incoming and the forwarded path. One kind is found on the machines measured; another is refused until a host speaks it.
- The host gains a guard it did not have — keep what you found — and its report must say which files and containers it holds, or an adopted node reads as converged.
- The declaration gains a node's mode and its taken modules, and a resource, the opening.
- Genesis grows inputs, and they outlive genesis. The foundation's ports stop being constants; every reader of them reads the node's settings.
- A node can sit adopted indefinitely. Nothing forces the flip; the mesh's status says which nodes are adopted, so one left behind is visible.
- This narrows ADR 0088 for adopted nodes: the base ruleset is not loaded on a node raised adopted, and its duty — the store never reachable from outside — passes to the mesh's own openings and refusals on the forwarded path.
How it is checked
A lab bed prepares a machine the way the predecessor leaves one: its firewall allowing a served port and denying the rest, a service container listening on that port under a name a catalogue module also uses, a file at a path that module declares, a stand-in for the predecessor's control that would rewrite that file, and a container holding the registry's port. Then:
- Genesis converged on it refuses and names the firewall and the listener.
- Genesis adopted, with the registry's port held, refuses and names the holder; with another port given, the foundation comes up — and adopting the foundation as modules leaves it on that port.
- Nothing that serves changed: the service is reachable from a second machine, the file is byte for byte what it was, and the found firewall's rules differ only by rules marked as the mesh's.
- The store is unreachable from outside — probed from a machine off the private network — and reachable over it.
- The mesh works through the found firewall, and keeps working after it is reloaded and after the machine reboots: the second machine enrols, and the openings are there again.
- A predecessor still writing is caught: with the stand-in left running, the held file's change is reported and not reverted.
- Assigning prepares, taking cuts over: the module assigned holds the found container and file; taken, it replaces them and its port is opened.
- Converging previews, then changes: the preview names the service's port and a published port no firewall rule mentions; after the flip the mesh's derived filter is loaded, the found firewall is disabled with its configuration still on disk, the declared port is open and the undeclared one closed. Returned to adopted, the found firewall is enabled again.
Unit tests hold the host to keeping a found file and container on an adopted node, converging them once taken, never removing a held file, and reporting a held file that changed; genesis to refusing a held port and a converged raise on a machine in use; the controller to carrying the mode and the taken modules in every declaration.