196 lines
13 KiB
Markdown
196 lines
13 KiB
Markdown
---
|
|
topic: the mesh
|
|
status: accepted
|
|
date: 2026-09-22
|
|
deciders: jochen
|
|
reconstructed: false
|
|
extends: 02-DECISIONS/0078-the-store-and-broker-are-modules.md
|
|
---
|
|
|
|
# 100. A node in use is adopted before it is converged
|
|
|
|
## Context
|
|
|
|
The mesh replaces a predecessor mesh that is running, on the same machines, with the services
|
|
people use. The control-node is the machine that already carries the predecessor's broker and
|
|
build pipeline. The operator proposed the migration's shape: stop the predecessor's control on a
|
|
machine, bring the mesh up there in an adoption mode that keeps the machine's configuration in
|
|
force, migrate its modules one at a time, move to the next machine, and flip adoption off when every
|
|
machine is done.
|
|
|
|
Measured on the control-node ([research 012,
|
|
*migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)):
|
|
60 predecessor containers; the predecessor's control in user-level daemons separate from its
|
|
services; its firewall active, with 54 incoming and 52 forwarding rules allowing each served port
|
|
explicitly; 12 files its configuration sync writes, 3 of them system files.
|
|
|
|
Four things in the mesh as it stands break that shape:
|
|
|
|
1. **The base ruleset closes the machine.** Genesis loads a drop-by-default table
|
|
([ADR 0088](0088-the-foundation-filters-before-anything-listens.md)). Every base chain at a hook
|
|
runs in priority order; an accept ends only its own chain and a drop in any is final — whether
|
|
the other firewall's chains are nftables or legacy iptables. So the predecessor's allowed ports
|
|
would close at genesis.
|
|
2. **The host replaces what it finds.** A declared file is written whatever is at its path, except
|
|
a file declared create-once. A declared container replaces a running one of the same name. So
|
|
assigning a module the predecessor also runs replaces the predecessor's service and its files at
|
|
once.
|
|
3. **Some foundation ports are held.** The foundation's store and bus ports are free — the
|
|
predecessor publishes its own elsewhere — but the registry's port and the broker's management
|
|
port are held, and on a control-node that is the private network's hub, so is the private
|
|
network's port. The foundation's ports are fixed in the installer's bundle and the catalogue's
|
|
manifests, so a collision surfaces as a container that fails to bind, and a port changed at
|
|
genesis would be changed back when the foundation is adopted as modules.
|
|
4. **Published container ports are forwarded, not received.** The foundation publishes its ports
|
|
on every interface; a predecessor firewall that filters only incoming traffic never sees them.
|
|
The base ruleset is what keeps the store unreachable from outside, and it is the thing that
|
|
cannot be loaded.
|
|
|
|
The mesh already adopts in one place: the foundation's store and broker are taken over in place as
|
|
modules, keyed on the container that is already running
|
|
([ADR 0078](0078-the-store-and-broker-are-modules.md)). The research this record draws on already
|
|
settled the conflict rule for adoption: *on conflict, what is on the machine stays*.
|
|
|
|
## Considered Options
|
|
|
|
1. **Cut each machine over in one go** — stop the predecessor's store, broker, registry and proxy,
|
|
raise the foundation in their place. Rejected: every predecessor service on the machine is down
|
|
until it has migrated, the predecessor's other machines lose their broker, and the rollback is
|
|
restarting the predecessor — a recovery, not a step.
|
|
2. **Put the control-node on a separate machine.** Rejected: the control-node is decided.
|
|
3. **Converge on joining, as today.** Rejected: the base ruleset closes the machine at genesis, and
|
|
the predecessor's services and files are replaced as soon as any module naming them is assigned.
|
|
4. **Only make the foundation's ports configurable.** Rejected as insufficient: it answers the bind
|
|
collisions and neither the firewall nor the files.
|
|
5. **Treat assigning a module as migrating it.** Considered and rejected on review: it makes the
|
|
rule that keeps found files never fire — the host only ever sees files of assigned modules — and
|
|
it makes an assignment on an adopted node an outage rather than a preparation.
|
|
6. **Adoption as a mode per node; each module taken explicitly; the node converged by an explicit
|
|
flip.** Adopted.
|
|
|
|
## Decision
|
|
|
|
**A node is adopted or converged, and the controller records which.** The operator says so: at
|
|
genesis for the control-node, and in the enrolment token for the others. The controller is
|
|
authoritative, and every declaration it sends says whether the node is adopted and which modules
|
|
have been taken on it. A node stays adopted until the operator converges it. An adopted node is
|
|
said to be adopted wherever the mesh reports a node's state.
|
|
|
|
**A converged genesis refuses a machine in use.** Raised without saying adopted on a machine with
|
|
an active firewall or services listening that are not the mesh's, genesis refuses and names what
|
|
it found — a forgotten flag must not close a working machine.
|
|
|
|
**Before a node is adopted, its predecessor's control is stopped by the operator** — the daemons
|
|
that write its configuration. Its services keep running on what they have.
|
|
|
|
**Found means present with no record.** A file at a declared path, or a container at a declared
|
|
name, that the host's store has no record of writing is *found*. A file the host wrote in an
|
|
earlier life of the node is not found; its record says so.
|
|
|
|
**On an adopted node, what is found is kept until its module is taken.** The host keeps a found
|
|
file and a found container as they are, records the file's original content before anything else
|
|
happens to it, and reports each as held. Assigning a module on an adopted node prepares it: what
|
|
the module declares that is not found is created; what is found is held. **Taking a module** on a
|
|
node is its cutover — the operator's act, done when that module's data has moved — and from then
|
|
on the module's resources converge on that node like any other. A held file that changes while
|
|
held is reported as changed by something else, not reverted: that is how a predecessor still
|
|
writing is caught. A held file whose module is unassigned is left where it is, never removed.
|
|
|
|
**The firewall found on the machine stays in force.** The mesh loads no table of its own on an
|
|
adopted node — neither genesis's base ruleset nor the filter module's derived one. What the mesh
|
|
needs is declared as **openings**: a resource that says a port is reachable, from where, on the
|
|
incoming path or the forwarded path — a published container port is forwarded. The host converges
|
|
an opening through the found firewall in that firewall's own terms, marks it as the mesh's, and
|
|
removes only what it marked; it re-checks each opening on every reconcile, so a reload or a reboot
|
|
of the found firewall does not lose it. An opening is a state, not a command, which is what lets it
|
|
travel over the link. The host reports which firewall it found, and a machine with a kind no host
|
|
speaks is refused adoption rather than adopted with nothing protecting the mesh's ports.
|
|
|
|
**The mesh protects its own ports itself.** On an adopted node its foundation's ports get openings
|
|
from the private network and marked refusals from anywhere else, on the forwarded path — so the
|
|
store is unreachable from outside whether or not the found firewall filters forwarded traffic.
|
|
|
|
**The foundation's ports are the node's.** Every port the foundation binds is an input to genesis,
|
|
checked free before anything is raised, refused with the name of what holds it. The ports given
|
|
become that node's settings for the foundation's modules — the catalogue's numbers are only their
|
|
defaults — and every place that uses them reads them from there: the modules' containers, the
|
|
filter, the base ruleset, the private network's endpoint, and the addresses consumers are given.
|
|
The private network's address range must not overlap a tunnel the predecessor still runs; genesis
|
|
checks that too.
|
|
|
|
**Converging a node is one act, previewed.** The preview lists what is reachable on the machine now
|
|
— every listening socket and every published container port — and for each whether an assigned
|
|
module declares it or it will close. The flip then takes every module not yet taken, loads the
|
|
mesh's derived filter, and retires the found firewall by disabling it, never by flushing: the
|
|
container runtime's rules and the found firewall's own configuration stay on disk. Returning a
|
|
converged node to adopted re-enables the firewall it retired. A node converges when its migration
|
|
is done; the mesh is migrated when every node has converged.
|
|
|
|
**The order is the operator's:** the control-node first, adopted, its modules assigned and taken
|
|
one at a time; then each other machine, adopted, migrated, converged in turn. The predecessor's
|
|
pipeline runs on the control-node, so its updates stop for every machine while the migration runs;
|
|
that is accepted.
|
|
|
|
## Consequences
|
|
|
|
Each step says what it changes before it changes it. Adopting a node changes nothing that serves;
|
|
assigning a module adds what is not there; taking a module replaces one service; the flip replaces
|
|
the firewall, after naming every port it will close. Two steps are not undone by the mesh: taking a
|
|
module replaces the predecessor's container, and the kept original of a file is recorded but not
|
|
yet restored by any act of the mesh
|
|
([research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md) leaves where it lives
|
|
open).
|
|
|
|
What got harder:
|
|
|
|
- **The mesh must speak a firewall it did not install**, on both the incoming and the forwarded
|
|
path. One kind is found on the machines measured; another is refused until a host speaks it.
|
|
- **The host gains a guard it did not have** — keep what you found — and its report must say
|
|
which files and containers it holds, or an adopted node reads as converged.
|
|
- **The declaration gains a node's mode and its taken modules**, and a resource, the opening.
|
|
- **Genesis grows inputs, and they outlive genesis.** The foundation's ports stop being constants;
|
|
every reader of them reads the node's settings.
|
|
- **A node can sit adopted indefinitely.** Nothing forces the flip; the mesh's status says which
|
|
nodes are adopted, so one left behind is visible.
|
|
- **This narrows [ADR 0088](0088-the-foundation-filters-before-anything-listens.md) for adopted
|
|
nodes**: the base ruleset is not loaded on a node raised adopted, and its duty — the store never
|
|
reachable from outside — passes to the mesh's own openings and refusals on the forwarded path.
|
|
|
|
## How it is checked
|
|
|
|
A lab bed prepares a machine the way the predecessor leaves one: its firewall allowing a served
|
|
port and denying the rest, a service container listening on that port under a name a catalogue
|
|
module also uses, a file at a path that module declares, a stand-in for the predecessor's control
|
|
that would rewrite that file, and a container holding the registry's port. Then:
|
|
|
|
- **Genesis converged** on it refuses and names the firewall and the listener.
|
|
- **Genesis adopted, with the registry's port held**, refuses and names the holder; with another
|
|
port given, the foundation comes up — and adopting the foundation as modules leaves it on that
|
|
port.
|
|
- **Nothing that serves changed**: the service is reachable from a second machine, the file is byte
|
|
for byte what it was, and the found firewall's rules differ only by rules marked as the mesh's.
|
|
- **The store is unreachable from outside** — probed from a machine off the private network — and
|
|
reachable over it.
|
|
- **The mesh works through the found firewall, and keeps working after it is reloaded and after the
|
|
machine reboots**: the second machine enrols, and the openings are there again.
|
|
- **A predecessor still writing is caught**: with the stand-in left running, the held file's change
|
|
is reported and not reverted.
|
|
- **Assigning prepares, taking cuts over**: the module assigned holds the found container and file;
|
|
taken, it replaces them and its port is opened.
|
|
- **Converging previews, then changes**: the preview names the service's port and a published port
|
|
no firewall rule mentions; after the flip the mesh's derived filter is loaded, the found firewall
|
|
is disabled with its configuration still on disk, the declared port is open and the undeclared one
|
|
closed. Returned to adopted, the found firewall is enabled again.
|
|
|
|
Unit tests hold the host to keeping a found file and container on an adopted node, converging them
|
|
once taken, never removing a held file, and reporting a held file that changed; genesis to refusing
|
|
a held port and a converged raise on a machine in use; the controller to carrying the mode and the
|
|
taken modules in every declaration.
|
|
|
|
## References
|
|
|
|
- [research 012 — the minimum viable node, and adopting what is already there](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md),
|
|
and its document [*migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)
|
|
- [ADR 0005](0005-the-node-host.md), [ADR 0011](0011-managed-files-are-generated-never-edited.md),
|
|
[ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0088](0088-the-foundation-filters-before-anything-listens.md)
|