novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Research 012: migrating a node that is in use, measured on the control-node; ADR 0100 proposed — a node in use is adopted before it is converged

Browse Source
This commit is contained in:
jschoubben
2026-09-22 16:16:28 +02:00
parent 21baf397a8
commit 5fc3cbde4c
4 changed files with 295 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-RESEARCH/012-the-minimum-viable-node/00-overview.md
+14
View File
@@ -7,6 +7,9 @@ touches:
- 03-DESIGN/01-to-be/05-the-node-host.md
- 03-DESIGN/00-as-is/05-runtime-and-installation.md
- 01-RESEARCH/011-the-module-graph/00-overview.md
- 02-DECISIONS/0078-the-store-and-broker-are-modules.md
- 02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md
- 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
---
# 012 — The minimum viable node, and adopting what is already there
@@ -180,6 +183,16 @@ same makes a node where something the mesh needed never happened indistinguishab
where a log level differed. Whether a failed line still lets adoption complete is therefore
reopened by adding severity, and is not decided here.
## Adoption as a mode, for the migration
*2026-09-22.* The migration from the predecessor mesh gave the middle state a length. A machine
running the predecessor is **adopted** when the mesh comes up on it — the predecessor's control
stopped, the machine's firewall and files kept in force, the mesh opening what it needs through
them — and stays adopted while its modules migrate one at a time, until the operator **converges**
it. Measured on the control-node, and weighed against the alternatives, in
[*migrating a node that is in use*](migrating-a-node-in-use.md); proposed as
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md).
## Open questions
| Question | Why it is open |
@@ -191,6 +204,7 @@ reopened by adding severity, and is not decided here.
| Can a module say which of its settings are load-bearing? | The question that dissolves the conflict rule rather than choosing a side. A setting the module *requires* cannot be kept from the machine without producing something installed and broken; a setting it merely *prefers* should always yield. Until a module can say which is which, adoption is defaulting in the dark. Belongs with the graph. |
| Does a `failed` line still let adoption complete? | *Flags inform, they do not block* was decided about conflicts, where the mesh chose and the machine works. A failure is *we could not*, which is different in kind — and treating them alike hides the worse one behind the commoner one. |
| How is a flagged conflict reconciled, and by whom? | The briefing hands it to a session. What that session is empowered to change, and whether the resolution is recorded so the next adoption does not re-raise it, is undecided. |
| ~~How long is a machine adopted?~~ | **Proposed** ([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)) — for as long as it is being migrated: a mode per node, recorded, ended by an explicit and previewed flip. |
| Where does the kept original live, and for how long? | Whether it is recorded in the node's state so adoption is visibly reversible, and whether it is returned when the mesh stops managing the thing. |
| What shape is a briefing? | Structured enough to be acted on, prose enough to be read. It is the first thing a session on a new node sees, which makes it an interface rather than a log. |
| Does owning a package mean owning its version? | Owning configuration and owning the package are different scopes. The second means the mesh decides which version is installed, and that decision then has to survive the machine's own package manager updating it. |
01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md
+135
View File
@@ -0,0 +1,135 @@
# Migrating a node that is in use — adoption as a mode, not a moment
*2026-09-22. Measured on the machine that will be the control-node, which is running the
predecessor mesh today. Nothing below names it; the counts and port numbers are its own.*
## The question
The mesh replaces a predecessor mesh that is running, on the same machines, with the services
people use. The control-node is decided: it is the machine that already carries the predecessor's
broker and build pipeline. So the question is not *where* the mesh starts but **how a machine
running the predecessor becomes a node of the mesh without its services noticing** — and then how
each of the other machines follows.
## The operator's proposal
Proposed by the operator, 2026-09-22, and the shape this document tests:
1. **Stop the predecessor's control on a machine** — its daemons that write configuration: the
network and firewall configuration above all, which decide what is reachable and what is
blocked. Its services keep running; only the control over their configuration stops.
2. **Bring the mesh up on that machine in adoption mode.** It takes custody of those files and
keeps what it finds in force.
3. **Migrate the modules one at a time**, data preserved, per the cutover procedure.
4. **Move to the next machine and repeat** — adopted first, keeping its local configuration, then
migrated.
5. **When every machine is migrated, flip adoption mode**, and the mesh takes full control of the
configuration it has been holding.
This is the research above made concrete. It keeps the conflict rule already decided here — *on
conflict, what is on the machine stays* — and gives the middle state, *adopted*, a length: not a
one-time import before generating starts, but a mode that lasts for as long as the machine is
being migrated, ended by an explicit act.
## What the machine actually looks like
Measured, read-only:
| | |
|---|---|
| Containers running | 60, all the predecessor's services and their stores |
| The predecessor's control | user-level daemons, separate from the services; none of the 20 running system services is the predecessor's control |
| Firewall | the predecessor's, active: 54 incoming rules and 52 forwarding rules, each served port allowed explicitly — how a default-deny firewall reads |
| Files the predecessor's configuration sync writes | 12, of which 3 are system files (an ssh server drop-in, the package manager's configuration, one service's configuration); the rest are the operator's shell and agent files |
| Per-service configuration | environment and composition files per service, written by the predecessor's service tooling |
**Stopping the predecessor's control stops nothing that serves.** The services are containers and
system units that run without it; what stops is the rewriting of their configuration. Nothing
changes on the machine until something else writes.
## What collides, measured rather than assumed
An earlier note assumed the mesh's foundation could not stand beside the predecessor because both
want the store's and the broker's standard ports. **The measurement says otherwise.** The
predecessor publishes its own store on a non-standard port and its broker on another; the standard
ports the foundation binds for its store and its bus are free.
What does collide:
| The mesh wants | Held by | When |
|---|---|---|
| the registry's port | the predecessor's registry | at genesis — the foundation raises a registry |
| the broker's management port, on loopback | the predecessor's broker | at genesis |
| the private network's port | the predecessor's own tunnel | when the node is placed on the overlay |
| the web ports | the predecessor's reverse proxy | when the route proxy is assigned |
| the resolver's port | a resolver the predecessor runs | when a resolver module is assigned |
Three of these fall at genesis and cannot be deferred; two only when a particular module is
assigned, which is the moment that module migrates and its predecessor stops anyway. Today the
foundation's ports are **fixed**: written in the installer's bundle and in the catalogue's
manifests, so a collision is found when a container fails to bind, not before.
## What would break if the mesh came up as it is today
**The firewall.** Genesis loads a base ruleset — drop anything undeclared — in its own table
([ADR 0088](../../02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md)). The
predecessor's firewall is a different table. The kernel runs every table hooked at the same point
in turn: an accept in one lets the packet go on to the next, and a drop in any is final. So the
mesh's base ruleset would drop everything the predecessor's firewall allows and the mesh has not
declared — every web, mail and database port in the table above — the moment genesis ran.
**The files.** The host writes a declared file whatever it finds at the path, reporting it as
updated. A file the predecessor left — the ssh server drop-in, the resolver's configuration — is
replaced the first time a module declaring that path is assigned, before that module's service
has moved.
**The container names.** The applier keys a container on its name. A catalogue module whose
container carries the same name as the predecessor's service it replaces takes that container over
the moment it is assigned — which is exactly the cutover moment, so this is the migration's step
and not a fault, but it means *assigning a module is migrating it*, never a rehearsal.
## The pipeline freezes while the control-node migrates
The predecessor's build pipeline and its coordinator run on the control-node. Stopping its control
there stops the predecessor's updates for **every** machine it manages, until the migration is
done. Their services keep running; they receive nothing new. That is the price of the proposal and
it is worth stating, not a reason against it: the migration is the period in which the predecessor
is being replaced, and it does not need to keep changing.
## What adoption mode has to mean
For the proposal to hold, *adopted* must be a **state the mesh records per node**, not an
intention, and each thing the mesh would otherwise take must say what it does in that state:
- **The firewall found on the machine stays in force.** The mesh does not load its own table on an
adopted node. What the mesh needs open — its foundation, and each module as it migrates — it opens
*through the found firewall*, as rules it marks as its own, and it removes only what it added.
An accept in a table of its own would not help: the found firewall's drop would still be final.
- **A file found at a path the mesh declares is kept**, recorded with what it contained, and
converged only when the module that declares it migrates. Node-level files no module owns — the
hosts file, the resolver, the ssh drop-in — converge at the flip.
- **The foundation's ports are the node's to give.** Every port the foundation binds is set at
genesis rather than written into a manifest, and genesis checks each one is free before it
raises anything, naming what holds it.
- **The flip is per node and previewed.** Converging a node replaces the found firewall with the
mesh's derived one in one step, after saying which currently open ports would close — every one
of them either declared by an assigned module or deliberately dropped.
## Options weighed
| Option | Verdict |
|---|---|
| Cut the machine over in one go: stop the predecessor's store, broker, registry and proxy, raise the foundation in their place | Rejected. Every predecessor service goes down until it has migrated, and the predecessor's other machines lose their broker. The rollback is restarting the predecessor, which is a recovery, not a step. |
| A separate machine as the control-node | Rejected. Contradicts the decision that the control-node is the machine that already carries the predecessor's broker and pipeline. |
| Converge on joining, as the mesh does today | Rejected. The base ruleset closes every predecessor port at genesis, and found files are replaced before their services move. |
| Make only the foundation's ports configurable and otherwise converge | Rejected as insufficient. It solves the bind collisions and none of the firewall or file ones. |
| **Adoption as a mode, per node, ended by an explicit flip** | The proposal. It is the conflict rule already decided here, given a duration. |
## What this leaves open
Answered by the decision record this feeds — [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)
— only for the migration's needs. The general questions above stay open: whether a module can say
which settings are load-bearing, what shape a briefing takes, how a flagged conflict is reconciled.
One is newly sharp: the mesh opening ports through a firewall it did not install needs to speak that
firewall. There is one kind on the machines measured; a machine with another is not covered until
someone writes for it.
02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
+145
View File
@@ -0,0 +1,145 @@
---
topic: the mesh
status: proposed
date: 2026-09-22
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0078-the-store-and-broker-are-modules.md
---
# 100. A node in use is adopted before it is converged
## Context
The mesh replaces a predecessor mesh that is running, on the same machines, with the services
people use. The control-node is the machine that already carries the predecessor's broker and
build pipeline. The operator proposed the migration's shape: stop the predecessor's control on a
machine, bring the mesh up there in an adoption mode that keeps the machine's configuration in
force, migrate its modules one at a time, move to the next machine, and flip adoption off when every
machine is done.
Measured on the control-node ([research 012,
*migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)):
60 predecessor containers; the predecessor's control in user-level daemons separate from its
services; its firewall active, with 54 incoming and 52 forwarding rules allowing each served port
explicitly; 12 files its configuration sync writes, 3 of them system files.
Three things in the mesh as it stands break that shape:
1. **The base ruleset closes the machine.** Genesis loads a drop-by-default table
([ADR 0088](0088-the-foundation-filters-before-anything-listens.md)). Every table at a hook is
run in turn and a drop in any is final, so the predecessor's allowed ports — web, mail, stores —
would close at genesis.
2. **The host replaces what it finds.** A declared file is written whatever is at its path. A
file the predecessor left is replaced as soon as any module declaring its path is assigned.
3. **Some foundation ports are held.** The foundation's store and bus ports are free — the
predecessor publishes its own elsewhere — but the registry's port and the broker's management
port are held by the predecessor's, and the private network's port by its tunnel. The
foundation's ports are fixed in the installer's bundle and the catalogue's manifests, so the
collision surfaces as a container that fails to bind.
The mesh already adopts in one place: the foundation's store and broker are taken over in place as
modules, keyed on the container that is already running
([ADR 0078](0078-the-store-and-broker-are-modules.md)). The research this record draws on already
settled the conflict rule for adoption: *on conflict, what is on the machine stays*.
## Considered Options
1. **Cut each machine over in one go** — stop the predecessor's store, broker, registry and proxy,
raise the foundation in their place. Rejected: every predecessor service on the machine is down
until it has migrated, the predecessor's other machines lose their broker, and the rollback is
restarting the predecessor — a recovery, not a step.
2. **Put the control-node on a separate machine.** Rejected: the control-node is decided.
3. **Converge on joining, as today.** Rejected: the base ruleset closes the machine at genesis, and
found files are replaced before their services move.
4. **Only make the foundation's ports configurable.** Rejected as insufficient: it answers the
bind collisions and neither the firewall nor the files.
5. **Adoption as a mode per node, ended by an explicit flip.** Adopted.
## Decision
**A node is either adopted or converged, and the mesh records which.** A node joins adopted when
its machine is in use; the operator says so at genesis for the control-node and at enrolment for
the others. It stays adopted until the operator converges it. Adopted is not a one-time import
before generating starts: it lasts for as long as the machine is being migrated.
**Before a node is adopted, the predecessor's control on it is stopped** — the daemons that write
its configuration. Its services keep running on the configuration they have.
**On an adopted node:**
- **The firewall found on the machine stays in force.** The mesh does not load its own table
there — neither genesis's base ruleset nor the filter module's derived one. What the mesh needs
reachable — its foundation's ports and, as each module migrates, that module's declared
`listens` — it opens *through the found firewall*, as rules marked as the mesh's. It removes only
rules it marked, and edits nothing else.
- **A file found at a path the mesh declares is kept.** The host records what it found there and
does not replace it while the node is adopted. A module's files converge when that module is
assigned, because assigning a module is migrating it. Files no module owns — the hosts file, the
resolver, the ssh drop-in — converge at the flip.
- **The foundation takes the ports it is given.** Every port the foundation binds is an input to
genesis rather than a constant, and genesis checks each is free before raising anything,
refusing with the name of what holds it.
**Converging a node is one act, previewed.** It first lists every port the found firewall allows
and says, for each, whether an assigned module declares it or it will close; then it loads the
mesh's derived filter, retires the found firewall, and converges the files kept at adoption. A
node converges when its migration is done. The mesh is migrated when every node has converged.
**The order is the operator's:** the control-node first, adopted, its modules migrated one at a
time; then each other machine, adopted, migrated, converged in turn. The predecessor's pipeline
runs on the control-node, so its updates stop for every machine while the migration runs; that is
accepted.
## Consequences
The migration becomes a sequence of reversible steps. A module moved is one service changed; a node
adopted is a node on which nothing changed; the flip is the one act that changes what is reachable,
and it says beforehand what it will change.
What got harder:
- **The mesh must speak a firewall it did not install.** Opening a port through the found
firewall means writing to it in its own terms. One kind is found on the machines measured; a
machine with another is not covered until someone writes for it.
- **The host gains a guard it did not have** — *do not replace what you found* — and its report
has to say which files it is holding rather than converging, or an adopted node looks converged.
That is the companion to the host's rule of never touching what it did not create
([ADR 0005](0005-the-node-host.md)) that the research asked for.
- **Genesis grows inputs.** The foundation's ports stop being constants in the bundle and the
catalogue; a mesh that never needed to move them now carries the choice.
- **A node can sit adopted indefinitely.** Nothing forces the flip, and an adopted node filters
with a firewall the mesh does not derive; the mesh's own status has to say which nodes are
adopted, so that one left behind is visible.
- **This narrows [ADR 0088](0088-the-foundation-filters-before-anything-listens.md) for adopted
nodes**: the base ruleset is not loaded on a node that is adopted at genesis, because the
machine's own firewall already filters and a second, stricter table would close it.
## How it is checked
A lab bed prepares a machine the way the predecessor leaves one: its firewall allowing a served
port and denying the rest, a service listening on that port, a file at a path a mesh module
declares, and a container holding the registry's port. Then:
- **Genesis adopted, with the registry's port held**, refuses and names what holds it; with
another port given, the foundation comes up.
- **Nothing on the machine changed**: the service is still reachable from a second machine, the
found file is byte for byte what it was, and the found firewall's rules differ only by rules
marked as the mesh's.
- **The mesh works through the found firewall**: the second machine enrols over the bus the mesh
opened there.
- **A module assigned migrates only itself**: its port is opened, its files converge, the found
file it does not own is untouched.
- **Converging previews, then changes**: the preview names the service's port as closing unless
declared; after the flip the mesh's derived filter is loaded, the found firewall is retired, the
declared port is open and the undeclared one is closed.
Unit tests hold the host to keeping a found file on an adopted node and replacing it on a converged
one, and genesis to refusing a held port.
## References
- [research 012 — the minimum viable node, and adopting what is already there](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md),
and its document [*migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)
- [ADR 0005](0005-the-node-host.md), [ADR 0011](0011-managed-files-are-generated-never-edited.md),
[ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0088](0088-the-foundation-filters-before-anything-listens.md)
02-DECISIONS/README.md
+1
View File
@@ -88,6 +88,7 @@ python3 00-META/checks/index.py fail if stale
- **0083** — [One push leaves the mesh consistent](0083-one-push-leaves-the-mesh-consistent.md)
- **0088** — [The foundation filters before anything listens](0088-the-foundation-filters-before-anything-listens.md)
- **0090** — [A failure that repeats is said to be stuck](0090-a-failure-that-repeats-is-said-to-be-stuck.md)
- **0100** — [A node in use is adopted before it is converged](0100-a-node-in-use-is-adopted-before-it-is-converged.md) *(proposed)*
### Its tiers, from the bottom up