hq/02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md

---
topic: the mesh
status: accepted
date: 2026-09-22
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0078-the-store-and-broker-are-modules.md
---

# 100. A node in use is adopted before it is converged

## Context

The mesh replaces a predecessor mesh that is running, on the same machines, with the services
people use. The control-node is the machine that already carries the predecessor's broker and
build pipeline. The operator proposed the migration's shape: stop the predecessor's control on a
machine, bring the mesh up there in an adoption mode that keeps the machine's configuration in
force, migrate its modules one at a time, move to the next machine, and flip adoption off when every
machine is done.

Measured on the control-node ([research 012,
*migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)):
60 predecessor containers; the predecessor's control in user-level daemons separate from its
services; its firewall active, with 54 incoming and 52 forwarding rules allowing each served port
explicitly; 12 files its configuration sync writes, 3 of them system files.

Four things in the mesh as it stands break that shape:

1. **The base ruleset closes the machine.** Genesis loads a drop-by-default table
   ([ADR 0088](0088-the-foundation-filters-before-anything-listens.md)). Every base chain at a hook
   runs in priority order; an accept ends only its own chain and a drop in any is final — whether
   the other firewall's chains are nftables or legacy iptables. So the predecessor's allowed ports
   would close at genesis.
2. **The host replaces what it finds.** A declared file is written whatever is at its path, except
   a file declared create-once. A declared container replaces a running one of the same name. So
   assigning a module the predecessor also runs replaces the predecessor's service and its files at
   once.
3. **Some foundation ports are held.** The foundation's store and bus ports are free — the
   predecessor publishes its own elsewhere — but the registry's port and the broker's management
   port are held, and on a control-node that is the private network's hub, so is the private
   network's port. The foundation's ports are fixed in the installer's bundle and the catalogue's
   manifests, so a collision surfaces as a container that fails to bind, and a port changed at
   genesis would be changed back when the foundation is adopted as modules.
4. **Published container ports are forwarded, not received.** The foundation publishes its ports
   on every interface; a predecessor firewall that filters only incoming traffic never sees them.
   The base ruleset is what keeps the store unreachable from outside, and it is the thing that
   cannot be loaded.

The mesh already adopts in one place: the foundation's store and broker are taken over in place as
modules, keyed on the container that is already running
([ADR 0078](0078-the-store-and-broker-are-modules.md)). The research this record draws on already
settled the conflict rule for adoption: *on conflict, what is on the machine stays*.

## Considered Options

1. **Cut each machine over in one go** — stop the predecessor's store, broker, registry and proxy,
   raise the foundation in their place. Rejected: every predecessor service on the machine is down
   until it has migrated, the predecessor's other machines lose their broker, and the rollback is
   restarting the predecessor — a recovery, not a step.
2. **Put the control-node on a separate machine.** Rejected: the control-node is decided.
3. **Converge on joining, as today.** Rejected: the base ruleset closes the machine at genesis, and
   the predecessor's services and files are replaced as soon as any module naming them is assigned.
4. **Only make the foundation's ports configurable.** Rejected as insufficient: it answers the bind
   collisions and neither the firewall nor the files.
5. **Treat assigning a module as migrating it.** Considered and rejected on review: it makes the
   rule that keeps found files never fire — the host only ever sees files of assigned modules — and
   it makes an assignment on an adopted node an outage rather than a preparation.
6. **Adoption as a mode per node; each module taken explicitly; the node converged by an explicit
   flip.** Adopted.

## Decision

**A node is adopted or converged, and the controller records which.** The operator says so: at
genesis for the control-node, and in the enrolment token for the others. The controller is
authoritative, and every declaration it sends says whether the node is adopted and which modules
have been taken on it. A node stays adopted until the operator converges it. An adopted node is
said to be adopted wherever the mesh reports a node's state.

**A converged genesis refuses a machine in use.** A machine is in use when a container is running
on it, or a port is listening on an address other than loopback that is not ssh's. Raised without
saying adopted on such a machine, genesis refuses and names every container and listener it
counted — a forgotten flag must not close a working machine.

**Before a node is adopted, its predecessor's control is stopped by the operator** — the daemons
that write its configuration. Its services keep running on what they have.

**Found means present with no record.** A file at a declared path, or a container at a declared
name, that the host's store has no record of writing is *found*. A file the host wrote in an
earlier life of the node is not found; its record says so.

**On an adopted node, what is found is kept until its module is taken.** The host keeps a found
file and a found container as they are, records the file's original content before anything else
happens to it, and reports each as held. Assigning a module on an adopted node prepares it: what
the module declares that is not found is created; what is found is held. **Taking a module** on a
node is its cutover — the operator's act, done when that module's data has moved — and from then
on the module's resources converge on that node like any other. What is held is never removed,
even when its module is unassigned, and a held file or container that changes while held — a file
rewritten, a container stopped or replaced — is reported as changed by something else, not reverted
or restarted: that is how a predecessor still writing is caught.

**The firewall found on the machine stays in force.** The mesh loads no table on an adopted node
that drops by default or holds an accept — neither genesis's base ruleset nor the filter module's
derived one. What the mesh needs reachable is declared as **openings**: a resource that says a port
is reachable, from where, on the incoming path or the forwarded path — a published container port is
forwarded. The controller derives them from the same inputs as the filter, each from where the
filter would admit it: the `listens` of the modules assigned there, the private network's hub port
and the bus and the registry from anywhere, the store's port and the broker's management port from
the private network. The host
converges an opening through the found firewall in that firewall's own terms, marks it as the
mesh's, and removes only what it marked; it re-checks each opening on every reconcile, so a reload
or a reboot of the found firewall does not lose it for longer than one reconcile. An opening is a
state, not a command, which is what lets it travel over the link. The host reports which firewall
it found. A machine with no firewall needs no openings; a machine with a kind no host speaks is
refused adoption.

**The mesh guards its own ports itself, in a table of its own that only refuses.** It passes
everything by default and holds nothing but refusals, and the two ports it refuses are the
foundation's own, checked free at genesis, so it cannot close anything the machine serves; it is
the mesh's, so the found firewall reloading does not touch it. It refuses the store's port and the
broker's management port except from the private network and from the machine itself — its
loopback and the container runtime's own networks, known by the interface a packet arrives on and
never by its source address alone — at the prerouting hook, ahead of the runtime's
destination translation, so it matches the port the packet was sent to, for both address
families. The bus and the registry stay reachable from anywhere, as a node
enrols over the bus and pulls from the registry before it has a private-network address
([ADR 0088](0088-the-foundation-filters-before-anything-listens.md)); so does the private network's
hub port. The store is unreachable from outside whatever the found firewall does, and on a machine
with none.

**The foundation's ports are the node's.** Every port the foundation binds is an input to genesis,
checked free before anything is raised, refused with the name of what holds it. The ports given
become that node's settings for the foundation's modules — the catalogue's numbers are only their
defaults — and every place that uses them reads them from there: the modules' containers, the
filter, the base ruleset, the private network's endpoint, and the addresses consumers are given.
The private network's address range must not overlap a tunnel the predecessor still runs; genesis
checks that too.

**Converging a node is one act, previewed.** It refuses while an assigned module still holds a
found container: each service is taken on its own, when its data has moved, never by the flip. The
preview lists what is reachable on the machine now — every listening socket and every published
container port — and for each whether an assigned module declares it or it will close, and every
module the flip will take, with the held files each will replace. The flip then takes those modules, loads the
mesh's derived filter in place of its refusal-only table, and retires the found firewall by
disabling it, never by flushing: the container runtime's rules and the found firewall's own
configuration stay on disk. Returning a converged node to adopted unloads the derived filter,
restores the refusal-only table, enables the found firewall again and converges the openings
through it once more; what was taken stays taken. A
node converges when its migration is done; the mesh is migrated when every node has converged.

**The order is the operator's:** the control-node first, adopted, its modules assigned and taken
one at a time; then each other machine, adopted, migrated, converged in turn. The predecessor's
pipeline runs on the control-node, so its updates stop for every machine while the migration runs;
that is accepted.

## Consequences

Each step says what it changes before it changes it. Adopting a node changes nothing that serves;
assigning a module adds what is not there; taking a module replaces one service; the flip replaces
the firewall, after naming every port it will close. Two steps are not undone by the mesh: taking a
module replaces the predecessor's container, and the kept original of a file is recorded but not
yet restored by any act of the mesh
([research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md) leaves where it lives
open).

What got harder:

- **The mesh must speak a firewall it did not install**, on both the incoming and the forwarded
  path. One kind is found on the machines measured; another is refused until a host speaks it. The
  mesh's own guard does not depend on it: that table is the mesh's.
- **The host gains a guard it did not have** — keep what you found — and its report must say
  which files and containers it holds, or an adopted node reads as converged.
- **The declaration gains a node's mode and its taken modules**, and a resource, the opening.
- **Genesis grows inputs, and they outlive genesis.** The foundation's ports stop being constants;
  every reader of them reads the node's settings.
- **A node can sit adopted indefinitely.** Nothing forces the flip; the mesh's status says which
  nodes are adopted, so one left behind is visible.
- **This narrows [ADR 0088](0088-the-foundation-filters-before-anything-listens.md) for adopted
  nodes**: the base ruleset is not loaded on a node raised adopted, and its duty — the store never
  reachable from outside — passes to a table of the mesh's that only refuses.

## How it is checked

A lab bed prepares a machine the way the predecessor leaves one: its firewall allowing a served
port and denying the rest, a service container listening on that port under a name a catalogue
module also uses, a file at a path that module declares, a stand-in for the predecessor's control
that would rewrite that file, and a container holding the registry's port. Then:

- **Genesis converged** on it refuses and names every container and listener it counted.
- **Genesis adopted, with the registry's port held**, refuses and names the holder; with another
  port given, the foundation comes up — and adopting the foundation as modules leaves it on that
  port.
- **Nothing that serves changed**: the service is reachable from a second machine, the file is byte
  for byte what it was, and the found firewall's rules differ only by rules marked as the mesh's.
- **The store is unreachable from outside** — probed from a machine off the private network, and
  again after the found firewall is reloaded — and reachable over it and from a container on the
  node itself; the bus is reachable from a machine that has not yet enrolled.
- **The mesh works through the found firewall, and keeps working after it is reloaded and after the
  machine reboots**: the second machine enrols, and the openings are there again.
- **A predecessor still writing is caught**: with the stand-in left running, the held file's change
  is reported and not reverted.
- **Assigning prepares, taking cuts over**: the module assigned holds the found container and file;
  taken, it replaces them and its port is opened.
- **Converging previews, then changes**: it refuses while the service's module holds its found
  container; once that module is taken, the preview names the service's port and a published port no
  firewall rule mentions, and the modules it will take; after the flip the mesh's derived filter is
  loaded, the found firewall is disabled with its configuration still on disk, the declared port is
  open and the undeclared one closed. Returned to adopted, the found firewall is enabled again and
  the derived filter is gone.

Unit tests hold the host to keeping a found file and container on an adopted node, converging them
once taken, never removing what it holds, and reporting a held file or container that changed;
genesis to refusing a held port and a converged raise on a machine in use; the controller to
carrying the mode and the taken modules in every declaration, deriving the openings, and refusing
a flip while a found container is held.

## References

- [research 012 — the minimum viable node, and adopting what is already there](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md),
  and its document [*migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)
- [ADR 0005](0005-the-node-host.md), [ADR 0011](0011-managed-files-are-generated-never-edited.md),
  [ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0088](0088-the-foundation-filters-before-anything-listens.md)