Migrating a node that is in use — adoption as a mode, not a moment

2026-09-22. Measured on the machine that will be the control-node, which is running the predecessor mesh today. Nothing below names it; the counts are its own.

The question

The mesh replaces a predecessor mesh that is running, on the same machines, with the services people use. The control-node is decided: it is the machine that already carries the predecessor's broker and build pipeline. So the question is not where the mesh starts but how a machine running the predecessor becomes a node of the mesh without its services noticing — and then how each of the other machines follows.

The operator's proposal

Proposed by the operator, 2026-09-22, and the shape this document tests:

Stop the predecessor's control on a machine — its daemons that write configuration: the network and firewall configuration above all, which decide what is reachable and what is blocked. Its services keep running; only the control over their configuration stops.
Bring the mesh up on that machine in adoption mode. It takes custody of those files and keeps what it finds in force.
Migrate the modules one at a time, data preserved, per the cutover procedure.
Move to the next machine and repeat — adopted first, keeping its local configuration, then migrated.
When every machine is migrated, flip adoption mode, and the mesh takes full control of the configuration it has been holding.

This is the research above made concrete. It keeps the conflict rule already decided here — on conflict, what is on the machine stays — and gives the middle state, adopted, a length: not a one-time import before generating starts, but a mode that lasts for as long as the machine is being migrated, ended by an explicit act.

What the machine actually looks like

Measured, read-only:


Containers running	60, all the predecessor's services and their stores
The predecessor's control	user-level daemons, separate from the services; none of the 20 running system services is the predecessor's control
Firewall	the predecessor's, active: 54 incoming rules and 52 forwarding rules, each served port allowed explicitly — how a default-deny firewall reads
Files the predecessor's configuration sync writes	12, of which 3 are system files (an ssh server drop-in, the package manager's configuration, one service's configuration); the rest are the operator's shell and agent files
Per-service configuration	environment and composition files per service, written by the predecessor's service tooling

Stopping the predecessor's control stops nothing that serves. The services are containers and system units that run without it; what stops is the rewriting of their configuration. Nothing changes on the machine until something else writes.

What collides, measured rather than assumed

An earlier note assumed the mesh's foundation could not stand beside the predecessor because both want the store's and the broker's standard ports. The measurement says otherwise. The predecessor publishes its own store on a non-standard port and its broker on another; the standard ports the foundation binds for its store and its bus are free.

What does collide:

The mesh wants	Held by	When
the registry's port	the predecessor's registry	at genesis — the foundation raises a registry
the broker's management port, on loopback	the predecessor's broker	at genesis
the private network's port	the predecessor's own tunnel	at genesis on a control-node that is the private network's hub; otherwise when the node is placed on it
the web ports	the predecessor's reverse proxy	when the route proxy is assigned
the resolver's port	a resolver the predecessor runs	when a resolver module is assigned

On the control-node, which is the private network's hub, three of these fall at genesis and cannot be deferred; two only when a particular module is taken, the moment its predecessor stops anyway. Today the foundation's ports are fixed: written in the installer's bundle and in the catalogue's manifests, so a collision is found when a container fails to bind, not before — and a port changed at genesis would be changed back when the foundation is adopted as modules, since the applier recreates a container whose declared spec differs. The private network's address range must also stay clear of the range the predecessor's tunnel uses; on the machines measured they are distinct.

What would break if the mesh came up as it is today

The firewall. Genesis loads a base ruleset — drop anything undeclared — in its own table (ADR 0088). The predecessor's firewall is a different table. The kernel runs every base chain registered at the same hook, in priority order: an accept ends only its own chain and the packet goes on to the next, and a drop in any is final — whether the other firewall's chains are nftables or legacy iptables. So the mesh's base ruleset would drop everything the predecessor's firewall allows and the mesh has not declared — every web, mail and database port in the table above — the moment genesis ran.

The files. The host writes a declared file whatever it finds at the path, reporting it as updated. A file the predecessor left — the ssh server drop-in, the resolver's configuration — is replaced the first time a module declaring that path is assigned, before that module's service has moved.

The container names. The applier keys a container on its name. A catalogue module whose container carries the same name as the predecessor's service it replaces takes that container over the moment it is assigned: today, assigning a module is migrating it, never a preparation.

The published ports. The foundation publishes its ports on every interface, and a published container port reaches the container through the forwarded path, not the incoming one. A firewall that filters only incoming traffic never sees it. The base ruleset is what keeps the store unreachable from outside today — and it is the thing that cannot be loaded on this machine.

The pipeline freezes while the control-node migrates

The predecessor's build pipeline and its coordinator run on the control-node. Stopping its control there stops the predecessor's updates for every machine it manages, until the migration is done. Their services keep running; they receive nothing new. That is the price of the proposal and it is worth stating, not a reason against it: the migration is the period in which the predecessor is being replaced, and it does not need to keep changing.

What adoption mode has to mean

For the proposal to hold, adopted must be a state the mesh records per node, not an intention, and each thing the mesh would otherwise take must say what it does in that state:

What is found is kept until its module is taken. Found is precise: present at a declared path or name with no record in the host's store. A found file or container is held, its original recorded, until the operator takes the module on that node — the cutover, done when the module's data has moved. Assigning prepares; taking migrates. Without the distinction the rule never fires: the host only ever sees what assigned modules declare.
The firewall found on the machine stays in force. The mesh loads no table on an adopted node that drops by default or holds an accept. What it needs open it declares as openings the host converges through the found firewall, on the incoming and the forwarded path, marked as the mesh's and re-checked on every reconcile so a reload or reboot does not lose them. An accept in a table of its own would not help: the found firewall's drop would still be final. What a table of its own can do is refuse, and a refusal is final too — so the mesh guards the store and the broker's management port from everyone but the private network and the machine itself, in a table that only refuses, ahead of the container runtime's redirect — which the found firewall does not do and cannot undo. The bus, the registry and the hub's port stay open to anywhere: a node enrols before it has a private-network address.
The foundation's ports are the node's to give — set at genesis, checked free, and kept as that node's settings, read everywhere they are used, so adopting the foundation as modules does not move them back.
The flip is per node and previewed from what is actually reachable — listening sockets and published ports, not the found firewall's allow list, which does not see what a container runtime forwards.

Options weighed

Option	Verdict
Cut the machine over in one go: stop the predecessor's store, broker, registry and proxy, raise the foundation in their place	Rejected. Every predecessor service goes down until it has migrated, and the predecessor's other machines lose their broker. The rollback is restarting the predecessor, which is a recovery, not a step.
A separate machine as the control-node	Rejected. Contradicts the decision that the control-node is the machine that already carries the predecessor's broker and pipeline.
Converge on joining, as the mesh does today	Rejected. The base ruleset closes every predecessor port at genesis, and found files are replaced before their services move.
Make only the foundation's ports configurable and otherwise converge	Rejected as insufficient. It solves the bind collisions and none of the firewall or file ones.
Treat assigning a module as migrating it	Rejected on review. The rule that keeps found files would never fire — the host only sees what assigned modules declare — and an assignment on an adopted node would be an outage rather than a preparation.
Adoption as a mode, per node, ended by an explicit flip	The proposal. It is the conflict rule already decided here, given a duration.

What this leaves open

Answered by the decision record this feeds — ADR 0100 — only for the migration's needs. The general questions above stay open: whether a module can say which settings are load-bearing, what shape a briefing takes, how a flagged conflict is reconciled. One is newly sharp: the mesh opening ports through a firewall it did not install needs to speak that firewall. There is one kind on the machines measured; a machine with another is not covered until someone writes for it.

10 KiB Raw Permalink Blame History