136 lines
8.9 KiB
Markdown
136 lines
8.9 KiB
Markdown
# Migrating a node that is in use — adoption as a mode, not a moment
|
|
|
|
*2026-09-22. Measured on the machine that will be the control-node, which is running the
|
|
predecessor mesh today. Nothing below names it; the counts and port numbers are its own.*
|
|
|
|
## The question
|
|
|
|
The mesh replaces a predecessor mesh that is running, on the same machines, with the services
|
|
people use. The control-node is decided: it is the machine that already carries the predecessor's
|
|
broker and build pipeline. So the question is not *where* the mesh starts but **how a machine
|
|
running the predecessor becomes a node of the mesh without its services noticing** — and then how
|
|
each of the other machines follows.
|
|
|
|
## The operator's proposal
|
|
|
|
Proposed by the operator, 2026-09-22, and the shape this document tests:
|
|
|
|
1. **Stop the predecessor's control on a machine** — its daemons that write configuration: the
|
|
network and firewall configuration above all, which decide what is reachable and what is
|
|
blocked. Its services keep running; only the control over their configuration stops.
|
|
2. **Bring the mesh up on that machine in adoption mode.** It takes custody of those files and
|
|
keeps what it finds in force.
|
|
3. **Migrate the modules one at a time**, data preserved, per the cutover procedure.
|
|
4. **Move to the next machine and repeat** — adopted first, keeping its local configuration, then
|
|
migrated.
|
|
5. **When every machine is migrated, flip adoption mode**, and the mesh takes full control of the
|
|
configuration it has been holding.
|
|
|
|
This is the research above made concrete. It keeps the conflict rule already decided here — *on
|
|
conflict, what is on the machine stays* — and gives the middle state, *adopted*, a length: not a
|
|
one-time import before generating starts, but a mode that lasts for as long as the machine is
|
|
being migrated, ended by an explicit act.
|
|
|
|
## What the machine actually looks like
|
|
|
|
Measured, read-only:
|
|
|
|
| | |
|
|
|---|---|
|
|
| Containers running | 60, all the predecessor's services and their stores |
|
|
| The predecessor's control | user-level daemons, separate from the services; none of the 20 running system services is the predecessor's control |
|
|
| Firewall | the predecessor's, active: 54 incoming rules and 52 forwarding rules, each served port allowed explicitly — how a default-deny firewall reads |
|
|
| Files the predecessor's configuration sync writes | 12, of which 3 are system files (an ssh server drop-in, the package manager's configuration, one service's configuration); the rest are the operator's shell and agent files |
|
|
| Per-service configuration | environment and composition files per service, written by the predecessor's service tooling |
|
|
|
|
**Stopping the predecessor's control stops nothing that serves.** The services are containers and
|
|
system units that run without it; what stops is the rewriting of their configuration. Nothing
|
|
changes on the machine until something else writes.
|
|
|
|
## What collides, measured rather than assumed
|
|
|
|
An earlier note assumed the mesh's foundation could not stand beside the predecessor because both
|
|
want the store's and the broker's standard ports. **The measurement says otherwise.** The
|
|
predecessor publishes its own store on a non-standard port and its broker on another; the standard
|
|
ports the foundation binds for its store and its bus are free.
|
|
|
|
What does collide:
|
|
|
|
| The mesh wants | Held by | When |
|
|
|---|---|---|
|
|
| the registry's port | the predecessor's registry | at genesis — the foundation raises a registry |
|
|
| the broker's management port, on loopback | the predecessor's broker | at genesis |
|
|
| the private network's port | the predecessor's own tunnel | when the node is placed on the overlay |
|
|
| the web ports | the predecessor's reverse proxy | when the route proxy is assigned |
|
|
| the resolver's port | a resolver the predecessor runs | when a resolver module is assigned |
|
|
|
|
Three of these fall at genesis and cannot be deferred; two only when a particular module is
|
|
assigned, which is the moment that module migrates and its predecessor stops anyway. Today the
|
|
foundation's ports are **fixed**: written in the installer's bundle and in the catalogue's
|
|
manifests, so a collision is found when a container fails to bind, not before.
|
|
|
|
## What would break if the mesh came up as it is today
|
|
|
|
**The firewall.** Genesis loads a base ruleset — drop anything undeclared — in its own table
|
|
([ADR 0088](../../02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md)). The
|
|
predecessor's firewall is a different table. The kernel runs every table hooked at the same point
|
|
in turn: an accept in one lets the packet go on to the next, and a drop in any is final. So the
|
|
mesh's base ruleset would drop everything the predecessor's firewall allows and the mesh has not
|
|
declared — every web, mail and database port in the table above — the moment genesis ran.
|
|
|
|
**The files.** The host writes a declared file whatever it finds at the path, reporting it as
|
|
updated. A file the predecessor left — the ssh server drop-in, the resolver's configuration — is
|
|
replaced the first time a module declaring that path is assigned, before that module's service
|
|
has moved.
|
|
|
|
**The container names.** The applier keys a container on its name. A catalogue module whose
|
|
container carries the same name as the predecessor's service it replaces takes that container over
|
|
the moment it is assigned — which is exactly the cutover moment, so this is the migration's step
|
|
and not a fault, but it means *assigning a module is migrating it*, never a rehearsal.
|
|
|
|
## The pipeline freezes while the control-node migrates
|
|
|
|
The predecessor's build pipeline and its coordinator run on the control-node. Stopping its control
|
|
there stops the predecessor's updates for **every** machine it manages, until the migration is
|
|
done. Their services keep running; they receive nothing new. That is the price of the proposal and
|
|
it is worth stating, not a reason against it: the migration is the period in which the predecessor
|
|
is being replaced, and it does not need to keep changing.
|
|
|
|
## What adoption mode has to mean
|
|
|
|
For the proposal to hold, *adopted* must be a **state the mesh records per node**, not an
|
|
intention, and each thing the mesh would otherwise take must say what it does in that state:
|
|
|
|
- **The firewall found on the machine stays in force.** The mesh does not load its own table on an
|
|
adopted node. What the mesh needs open — its foundation, and each module as it migrates — it opens
|
|
*through the found firewall*, as rules it marks as its own, and it removes only what it added.
|
|
An accept in a table of its own would not help: the found firewall's drop would still be final.
|
|
- **A file found at a path the mesh declares is kept**, recorded with what it contained, and
|
|
converged only when the module that declares it migrates. Node-level files no module owns — the
|
|
hosts file, the resolver, the ssh drop-in — converge at the flip.
|
|
- **The foundation's ports are the node's to give.** Every port the foundation binds is set at
|
|
genesis rather than written into a manifest, and genesis checks each one is free before it
|
|
raises anything, naming what holds it.
|
|
- **The flip is per node and previewed.** Converging a node replaces the found firewall with the
|
|
mesh's derived one in one step, after saying which currently open ports would close — every one
|
|
of them either declared by an assigned module or deliberately dropped.
|
|
|
|
## Options weighed
|
|
|
|
| Option | Verdict |
|
|
|---|---|
|
|
| Cut the machine over in one go: stop the predecessor's store, broker, registry and proxy, raise the foundation in their place | Rejected. Every predecessor service goes down until it has migrated, and the predecessor's other machines lose their broker. The rollback is restarting the predecessor, which is a recovery, not a step. |
|
|
| A separate machine as the control-node | Rejected. Contradicts the decision that the control-node is the machine that already carries the predecessor's broker and pipeline. |
|
|
| Converge on joining, as the mesh does today | Rejected. The base ruleset closes every predecessor port at genesis, and found files are replaced before their services move. |
|
|
| Make only the foundation's ports configurable and otherwise converge | Rejected as insufficient. It solves the bind collisions and none of the firewall or file ones. |
|
|
| **Adoption as a mode, per node, ended by an explicit flip** | The proposal. It is the conflict rule already decided here, given a duration. |
|
|
|
|
## What this leaves open
|
|
|
|
Answered by the decision record this feeds — [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)
|
|
— only for the migration's needs. The general questions above stay open: whether a module can say
|
|
which settings are load-bearing, what shape a briefing takes, how a flagged conflict is reconciled.
|
|
One is newly sharp: the mesh opening ports through a firewall it did not install needs to speak that
|
|
firewall. There is one kind on the machines measured; a machine with another is not covered until
|
|
someone writes for it.
|