15 KiB
topic, status, date, deciders, reconstructed, extends
| topic | status | date | deciders | reconstructed | extends |
|---|---|---|---|---|---|
| the mesh | accepted | 2026-09-22 | jochen | false | 02-DECISIONS/0078-the-store-and-broker-are-modules.md |
100. A node in use is adopted before it is converged
Context
The mesh replaces a predecessor mesh that is running, on the same machines, with the services people use. The control-node is the machine that already carries the predecessor's broker and build pipeline. The operator proposed the migration's shape: stop the predecessor's control on a machine, bring the mesh up there in an adoption mode that keeps the machine's configuration in force, migrate its modules one at a time, move to the next machine, and flip adoption off when every machine is done.
Measured on the control-node (research 012, migrating a node that is in use): 60 predecessor containers; the predecessor's control in user-level daemons separate from its services; its firewall active, with 54 incoming and 52 forwarding rules allowing each served port explicitly; 12 files its configuration sync writes, 3 of them system files.
Four things in the mesh as it stands break that shape:
- The base ruleset closes the machine. Genesis loads a drop-by-default table (ADR 0088). Every base chain at a hook runs in priority order; an accept ends only its own chain and a drop in any is final — whether the other firewall's chains are nftables or legacy iptables. So the predecessor's allowed ports would close at genesis.
- The host replaces what it finds. A declared file is written whatever is at its path, except a file declared create-once. A declared container replaces a running one of the same name. So assigning a module the predecessor also runs replaces the predecessor's service and its files at once.
- Some foundation ports are held. The foundation's store and bus ports are free — the predecessor publishes its own elsewhere — but the registry's port and the broker's management port are held, and on a control-node that is the private network's hub, so is the private network's port. The foundation's ports are fixed in the installer's bundle and the catalogue's manifests, so a collision surfaces as a container that fails to bind, and a port changed at genesis would be changed back when the foundation is adopted as modules.
- Published container ports are forwarded, not received. The foundation publishes its ports on every interface; a predecessor firewall that filters only incoming traffic never sees them. The base ruleset is what keeps the store unreachable from outside, and it is the thing that cannot be loaded.
The mesh already adopts in one place: the foundation's store and broker are taken over in place as modules, keyed on the container that is already running (ADR 0078). The research this record draws on already settled the conflict rule for adoption: on conflict, what is on the machine stays.
Considered Options
- Cut each machine over in one go — stop the predecessor's store, broker, registry and proxy, raise the foundation in their place. Rejected: every predecessor service on the machine is down until it has migrated, the predecessor's other machines lose their broker, and the rollback is restarting the predecessor — a recovery, not a step.
- Put the control-node on a separate machine. Rejected: the control-node is decided.
- Converge on joining, as today. Rejected: the base ruleset closes the machine at genesis, and the predecessor's services and files are replaced as soon as any module naming them is assigned.
- Only make the foundation's ports configurable. Rejected as insufficient: it answers the bind collisions and neither the firewall nor the files.
- Treat assigning a module as migrating it. Considered and rejected on review: it makes the rule that keeps found files never fire — the host only ever sees files of assigned modules — and it makes an assignment on an adopted node an outage rather than a preparation.
- Adoption as a mode per node; each module taken explicitly; the node converged by an explicit flip. Adopted.
Decision
A node is adopted or converged, and the controller records which. The operator says so: at genesis for the control-node, and in the enrolment token for the others. The controller is authoritative, and every declaration it sends says whether the node is adopted and which modules have been taken on it. A node stays adopted until the operator converges it. An adopted node is said to be adopted wherever the mesh reports a node's state.
A converged genesis refuses a machine in use. A machine is in use when a container is running on it or a port is listening that is neither ssh nor one of the operating system's own services. Raised without saying adopted on such a machine, genesis refuses and names what it found — a forgotten flag must not close a working machine.
Before a node is adopted, its predecessor's control is stopped by the operator — the daemons that write its configuration. Its services keep running on what they have.
Found means present with no record. A file at a declared path, or a container at a declared name, that the host's store has no record of writing is found. A file the host wrote in an earlier life of the node is not found; its record says so.
On an adopted node, what is found is kept until its module is taken. The host keeps a found file and a found container as they are, records the file's original content before anything else happens to it, and reports each as held. Assigning a module on an adopted node prepares it: what the module declares that is not found is created; what is found is held. Taking a module on a node is its cutover — the operator's act, done when that module's data has moved — and from then on the module's resources converge on that node like any other. What is held is never removed, even when its module is unassigned, and a held file or container that changes while held — a file rewritten, a container stopped or replaced — is reported as changed by something else, not reverted or restarted: that is how a predecessor still writing is caught.
The firewall found on the machine stays in force. The mesh loads no table on an adopted node
that drops by default or that accepts — neither genesis's base ruleset nor the filter module's
derived one. What the mesh needs reachable is declared as openings: a resource that says a port
is reachable, from where, on the incoming path or the forwarded path — a published container port is
forwarded. The controller derives them from the same inputs as the filter: the listens of the
modules assigned there, the private network's hub port, and the foundation's ports. The host
converges an opening through the found firewall in that firewall's own terms, marks it as the
mesh's, and removes only what it marked; it re-checks each opening on every reconcile, so a reload
or a reboot of the found firewall does not lose it for longer than one reconcile. An opening is a
state, not a command, which is what lets it travel over the link. The host reports which firewall
it found. A machine with no firewall needs no openings; a machine with a kind no host speaks is
refused adoption.
The mesh guards its own ports itself, in a table of its own that only refuses. It accepts by default and holds nothing but refusals, so it cannot close anything the machine serves — a drop in any chain is final and an accept in it would change nothing — and it is the mesh's, so the found firewall reloading does not touch it. It refuses the store's port and the broker's management port from anywhere but the private network, matched on the port the packet was sent to, before the container runtime redirects it. The bus and the registry stay reachable from anywhere, as a node enrols over the bus and pulls from the registry before it has a private-network address (ADR 0088); so does the private network's hub port. The store is unreachable from outside whatever the found firewall does, and on a machine with none.
The foundation's ports are the node's. Every port the foundation binds is an input to genesis, checked free before anything is raised, refused with the name of what holds it. The ports given become that node's settings for the foundation's modules — the catalogue's numbers are only their defaults — and every place that uses them reads them from there: the modules' containers, the filter, the base ruleset, the private network's endpoint, and the addresses consumers are given. The private network's address range must not overlap a tunnel the predecessor still runs; genesis checks that too.
Converging a node is one act, previewed. It refuses while an assigned module still holds a found container: each service is taken on its own, when its data has moved, never by the flip. The preview lists what is reachable on the machine now — every listening socket and every published container port — and for each whether an assigned module declares it or it will close, and every module the flip will take, with what each will replace. The flip then takes those modules, loads the mesh's derived filter in place of its refusal-only table, and retires the found firewall by disabling it, never by flushing: the container runtime's rules and the found firewall's own configuration stay on disk. Returning a converged node to adopted unloads the derived filter, restores the refusal-only table and enables the found firewall again; what was taken stays taken. A node converges when its migration is done; the mesh is migrated when every node has converged.
The order is the operator's: the control-node first, adopted, its modules assigned and taken one at a time; then each other machine, adopted, migrated, converged in turn. The predecessor's pipeline runs on the control-node, so its updates stop for every machine while the migration runs; that is accepted.
Consequences
Each step says what it changes before it changes it. Adopting a node changes nothing that serves; assigning a module adds what is not there; taking a module replaces one service; the flip replaces the firewall, after naming every port it will close. Two steps are not undone by the mesh: taking a module replaces the predecessor's container, and the kept original of a file is recorded but not yet restored by any act of the mesh (research 012 leaves where it lives open).
What got harder:
- The mesh must speak a firewall it did not install, on both the incoming and the forwarded path. One kind is found on the machines measured; another is refused until a host speaks it. The mesh's own guard does not depend on it: that table is the mesh's.
- The host gains a guard it did not have — keep what you found — and its report must say which files and containers it holds, or an adopted node reads as converged.
- The declaration gains a node's mode and its taken modules, and a resource, the opening.
- Genesis grows inputs, and they outlive genesis. The foundation's ports stop being constants; every reader of them reads the node's settings.
- A node can sit adopted indefinitely. Nothing forces the flip; the mesh's status says which nodes are adopted, so one left behind is visible.
- This narrows ADR 0088 for adopted nodes: the base ruleset is not loaded on a node raised adopted, and its duty — the store never reachable from outside — passes to a table of the mesh's that only refuses.
How it is checked
A lab bed prepares a machine the way the predecessor leaves one: its firewall allowing a served port and denying the rest, a service container listening on that port under a name a catalogue module also uses, a file at a path that module declares, a stand-in for the predecessor's control that would rewrite that file, and a container holding the registry's port. Then:
- Genesis converged on it refuses and names the running container and the listener.
- Genesis adopted, with the registry's port held, refuses and names the holder; with another port given, the foundation comes up — and adopting the foundation as modules leaves it on that port.
- Nothing that serves changed: the service is reachable from a second machine, the file is byte for byte what it was, and the found firewall's rules differ only by rules marked as the mesh's.
- The store is unreachable from outside — probed from a machine off the private network, and again after the found firewall is reloaded — and reachable over it; the bus is reachable from a machine that has not yet enrolled.
- The mesh works through the found firewall, and keeps working after it is reloaded and after the machine reboots: the second machine enrols, and the openings are there again.
- A predecessor still writing is caught: with the stand-in left running, the held file's change is reported and not reverted.
- Assigning prepares, taking cuts over: the module assigned holds the found container and file; taken, it replaces them and its port is opened.
- Converging previews, then changes: it refuses while the service's module holds its found container; once that module is taken, the preview names the service's port and a published port no firewall rule mentions, and the modules it will take; after the flip the mesh's derived filter is loaded, the found firewall is disabled with its configuration still on disk, the declared port is open and the undeclared one closed. Returned to adopted, the found firewall is enabled again and the derived filter is gone.
Unit tests hold the host to keeping a found file and container on an adopted node, converging them once taken, never removing what it holds, and reporting a held file or container that changed; genesis to refusing a held port and a converged raise on a machine in use; the controller to carrying the mode and the taken modules in every declaration, deriving the openings, and refusing a flip while a found container is held.