ADR 0100 (proposed): a node in use is adopted before it is converged #74

Merged
jschoubben merged 6 commits from feat/adoption-mode into main 2026-09-22 14:35:32 +00:00
7 changed files with 112 additions and 74 deletions
Showing only changes of commit f3152d827f - Show all commits
@@ -116,12 +116,15 @@ intention, and each thing the mesh would otherwise take must say what it does in
recorded, until the operator **takes** the module on that node — the cutover, done when the
module's data has moved. Assigning prepares; taking migrates. Without the distinction the rule
never fires: the host only ever sees what assigned modules declare.
- **The firewall found on the machine stays in force.** The mesh does not load its own table on an
adopted node. What it needs open it declares as openings the host converges *through the found
firewall*, on the incoming and the forwarded path, marked as the mesh's and re-checked on every
reconcile so a reload or reboot does not lose them. An accept in a table of its own would not
help: the found firewall's drop would still be final. And the mesh protects its own ports itself,
on the forwarded path, since the found firewall may not.
- **The firewall found on the machine stays in force.** The mesh loads no table on an adopted node
that drops by default or accepts. What it needs open it declares as openings the host converges
*through the found firewall*, on the incoming and the forwarded path, marked as the mesh's and
re-checked on every reconcile so a reload or reboot does not lose them. An accept in a table of
its own would not help: the found firewall's drop would still be final. What a table of its own
*can* do is refuse, and a refusal is final too — so the mesh guards the store and the broker's
management port from outside the private network in a table that only refuses, which the found
firewall may not do and cannot undo. The bus, the registry and the hub's port stay open to
anywhere: a node enrols before it has a private-network address.
- **The foundation's ports are the node's to give** — set at genesis, checked free, and kept as
that node's settings, read everywhere they are used, so adopting the foundation as modules does
not move them back.
@@ -76,9 +76,10 @@ authoritative, and every declaration it sends says whether the node is adopted a
have been taken on it. A node stays adopted until the operator converges it. An adopted node is
said to be adopted wherever the mesh reports a node's state.
**A converged genesis refuses a machine in use.** Raised without saying adopted on a machine with
an active firewall or services listening that are not the mesh's, genesis refuses and names what
it found — a forgotten flag must not close a working machine.
**A converged genesis refuses a machine in use.** A machine is in use when a container is running
on it or a port is listening that is neither ssh nor one of the operating system's own services.
Raised without saying adopted on such a machine, genesis refuses and names what it found — a
forgotten flag must not close a working machine.
**Before a node is adopted, its predecessor's control is stopped by the operator** — the daemons
that write its configuration. Its services keep running on what they have.
@@ -92,23 +93,34 @@ file and a found container as they are, records the file's original content befo
happens to it, and reports each as held. Assigning a module on an adopted node prepares it: what
the module declares that is not found is created; what is found is held. **Taking a module** on a
node is its cutover — the operator's act, done when that module's data has moved — and from then
on the module's resources converge on that node like any other. A held file that changes while
held is reported as changed by something else, not reverted: that is how a predecessor still
writing is caught. A held file whose module is unassigned is left where it is, never removed.
on the module's resources converge on that node like any other. What is held is never removed,
even when its module is unassigned, and a held file or container that changes while held — a file
rewritten, a container stopped or replaced — is reported as changed by something else, not reverted
or restarted: that is how a predecessor still writing is caught.
**The firewall found on the machine stays in force.** The mesh loads no table of its own on an
adopted node — neither genesis's base ruleset nor the filter module's derived one. What the mesh
needs is declared as **openings**: a resource that says a port is reachable, from where, on the
incoming path or the forwarded path — a published container port is forwarded. The host converges
an opening through the found firewall in that firewall's own terms, marks it as the mesh's, and
removes only what it marked; it re-checks each opening on every reconcile, so a reload or a reboot
of the found firewall does not lose it. An opening is a state, not a command, which is what lets it
travel over the link. The host reports which firewall it found, and a machine with a kind no host
speaks is refused adoption rather than adopted with nothing protecting the mesh's ports.
**The firewall found on the machine stays in force.** The mesh loads no table on an adopted node
that drops by default or that accepts — neither genesis's base ruleset nor the filter module's
derived one. What the mesh needs reachable is declared as **openings**: a resource that says a port
is reachable, from where, on the incoming path or the forwarded path — a published container port is
forwarded. The controller derives them from the same inputs as the filter: the `listens` of the
modules assigned there, the private network's hub port, and the foundation's ports. The host
converges an opening through the found firewall in that firewall's own terms, marks it as the
mesh's, and removes only what it marked; it re-checks each opening on every reconcile, so a reload
or a reboot of the found firewall does not lose it for longer than one reconcile. An opening is a
state, not a command, which is what lets it travel over the link. The host reports which firewall
it found. A machine with no firewall needs no openings; a machine with a kind no host speaks is
refused adoption.
**The mesh protects its own ports itself.** On an adopted node its foundation's ports get openings
from the private network and marked refusals from anywhere else, on the forwarded path — so the
store is unreachable from outside whether or not the found firewall filters forwarded traffic.
**The mesh guards its own ports itself, in a table of its own that only refuses.** It accepts by
default and holds nothing but refusals, so it cannot close anything the machine serves — a drop in
any chain is final and an accept in it would change nothing — and it is the mesh's, so the found
firewall reloading does not touch it. It refuses the store's port and the broker's management port
from anywhere but the private network, matched on the port the packet was sent to, before the
container runtime redirects it. The bus and the registry stay reachable from anywhere, as a node
enrols over the bus and pulls from the registry before it has a private-network address
([ADR 0088](0088-the-foundation-filters-before-anything-listens.md)); so does the private network's
hub port. The store is unreachable from outside whatever the found firewall does, and on a machine
with none.
**The foundation's ports are the node's.** Every port the foundation binds is an input to genesis,
checked free before anything is raised, refused with the name of what holds it. The ports given
@@ -118,13 +130,16 @@ filter, the base ruleset, the private network's endpoint, and the addresses cons
The private network's address range must not overlap a tunnel the predecessor still runs; genesis
checks that too.
**Converging a node is one act, previewed.** The preview lists what is reachable on the machine now
— every listening socket and every published container port — and for each whether an assigned
module declares it or it will close. The flip then takes every module not yet taken, loads the
mesh's derived filter, and retires the found firewall by disabling it, never by flushing: the
container runtime's rules and the found firewall's own configuration stay on disk. Returning a
converged node to adopted re-enables the firewall it retired. A node converges when its migration
is done; the mesh is migrated when every node has converged.
**Converging a node is one act, previewed.** It refuses while an assigned module still holds a
found container: each service is taken on its own, when its data has moved, never by the flip. The
preview lists what is reachable on the machine now — every listening socket and every published
container port — and for each whether an assigned module declares it or it will close, and every
module the flip will take, with what each will replace. The flip then takes those modules, loads the
mesh's derived filter in place of its refusal-only table, and retires the found firewall by
disabling it, never by flushing: the container runtime's rules and the found firewall's own
configuration stay on disk. Returning a converged node to adopted unloads the derived filter,
restores the refusal-only table and enables the found firewall again; what was taken stays taken. A
node converges when its migration is done; the mesh is migrated when every node has converged.
**The order is the operator's:** the control-node first, adopted, its modules assigned and taken
one at a time; then each other machine, adopted, migrated, converged in turn. The predecessor's
@@ -144,7 +159,8 @@ open).
What got harder:
- **The mesh must speak a firewall it did not install**, on both the incoming and the forwarded
path. One kind is found on the machines measured; another is refused until a host speaks it.
path. One kind is found on the machines measured; another is refused until a host speaks it. The
mesh's own guard does not depend on it: that table is the mesh's.
- **The host gains a guard it did not have** — keep what you found — and its report must say
which files and containers it holds, or an adopted node reads as converged.
- **The declaration gains a node's mode and its taken modules**, and a resource, the opening.
@@ -154,7 +170,7 @@ What got harder:
nodes are adopted, so one left behind is visible.
- **This narrows [ADR 0088](0088-the-foundation-filters-before-anything-listens.md) for adopted
nodes**: the base ruleset is not loaded on a node raised adopted, and its duty — the store never
reachable from outside — passes to the mesh's own openings and refusals on the forwarded path.
reachable from outside — passes to a table of the mesh's that only refuses.
## How it is checked
@@ -163,29 +179,33 @@ port and denying the rest, a service container listening on that port under a na
module also uses, a file at a path that module declares, a stand-in for the predecessor's control
that would rewrite that file, and a container holding the registry's port. Then:
- **Genesis converged** on it refuses and names the firewall and the listener.
- **Genesis converged** on it refuses and names the running container and the listener.
- **Genesis adopted, with the registry's port held**, refuses and names the holder; with another
port given, the foundation comes up — and adopting the foundation as modules leaves it on that
port.
- **Nothing that serves changed**: the service is reachable from a second machine, the file is byte
for byte what it was, and the found firewall's rules differ only by rules marked as the mesh's.
- **The store is unreachable from outside** — probed from a machine off the private network — and
reachable over it.
- **The store is unreachable from outside** — probed from a machine off the private network, and
again after the found firewall is reloaded — and reachable over it; the bus is reachable from a
machine that has not yet enrolled.
- **The mesh works through the found firewall, and keeps working after it is reloaded and after the
machine reboots**: the second machine enrols, and the openings are there again.
- **A predecessor still writing is caught**: with the stand-in left running, the held file's change
is reported and not reverted.
- **Assigning prepares, taking cuts over**: the module assigned holds the found container and file;
taken, it replaces them and its port is opened.
- **Converging previews, then changes**: the preview names the service's port and a published port
no firewall rule mentions; after the flip the mesh's derived filter is loaded, the found firewall
is disabled with its configuration still on disk, the declared port is open and the undeclared one
closed. Returned to adopted, the found firewall is enabled again.
- **Converging previews, then changes**: it refuses while the service's module holds its found
container; once that module is taken, the preview names the service's port and a published port no
firewall rule mentions, and the modules it will take; after the flip the mesh's derived filter is
loaded, the found firewall is disabled with its configuration still on disk, the declared port is
open and the undeclared one closed. Returned to adopted, the found firewall is enabled again and
the derived filter is gone.
Unit tests hold the host to keeping a found file and container on an adopted node, converging them
once taken, never removing a held file, and reporting a held file that changed; genesis to refusing
a held port and a converged raise on a machine in use; the controller to carrying the mode and the
taken modules in every declaration.
once taken, never removing what it holds, and reporting a held file or container that changed;
genesis to refusing a held port and a converged raise on a machine in use; the controller to
carrying the mode and the taken modules in every declaration, deriving the openings, and refusing
a flip while a found container is held.
## References
+8 -2
View File
@@ -139,8 +139,9 @@ name, that the host's store has no record of writing. On an adopted node the hos
found for any module not yet taken: it records a found file's original content before anything
else, and it reports the file or container as held — a report that says what it holds, so an
adopted node never reads as converged. Once the module is taken, its resources converge like any
other. A held file that changes while held is reported as changed by something else, not
reverted; a held file is never removed, even when its module is unassigned. The host also
other. What is held is never removed, even when its module is unassigned, and a held file or
container that changes while held is reported as changed by something else, not reverted or
restarted. The host also
converges a new resource, the **opening** — a port made reachable through the firewall it found
([08-connectivity](08-connectivity.md)) — and reports which firewall it found. This is the
companion the host's ownership rule needed: *never touch what you did not create, unless adoption
@@ -212,6 +213,11 @@ ready; the current bundle simply does not. **All of them are built:**
| `container` | **built** | pinned by digest ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)); identified by a label carrying a digest of the declaration that made it, because a runtime normalises what it is given and that is indistinguishable from drift |
| `action` | **built** | bundle-only ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)); verify is mandatory and is the idempotency check as well as the read-back |
One more shape is decided and not yet built: **`opening`**, on adopted nodes only
([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)) — a port
made reachable, from where, on the incoming or forwarded path, through the firewall found on the
machine, marked as the mesh's and re-checked on every reconcile.
**A service says what it must reflect, and that is declared state rather than a command.**
`restart-on` names files whose change means the unit must be restarted — because a running service
does not re-read its configuration, and replacing a file, finding the service already running and
+4 -2
View File
@@ -178,8 +178,10 @@ port never answers, the bus's does.
machine already serving under a predecessor has a firewall of its own, and a second, stricter
table would close everything it serves. There the base ruleset is not loaded and the filter module
is not assigned until the node converges; the foundation's ports are opened through the found
firewall from the private network and refused from anywhere else on the forwarded path, which
keeps the same promise — the store's port never answers from outside — by other means. The
firewall, and a table of the mesh's that only refuses keeps the store's port and the broker's
management port from anyone off the private network — the same promise, the store's port never
answering from outside, kept by other means. The bus and the registry stay reachable from anywhere,
as they are here, because a node enrols and pulls before it has a private-network address. The
foundation's ports themselves are the node's, given at genesis and kept as its settings. **Checked**
by the adoption bed, which makes the same outside probe.
+25 -17
View File
@@ -537,24 +537,32 @@ is why the check reads packets.*
### On an adopted node
*2026-09-22, [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md).* **The firewall found on the machine stays in force**, and the mesh
loads no table of its own there — neither genesis's base ruleset nor the derived one. Every base
chain at a hook runs and a drop in any is final, whatever the other firewall is written in, so a
second, stricter table would close every port the machine serves. What the mesh needs reachable
it declares as **openings**: a port, from where, on the incoming path or the forwarded path — a
loads no table there that drops by default or that accepts — neither genesis's base ruleset nor
the derived one. Every base chain at a hook runs and a drop in any is final, whatever the other
firewall is written in, so a second, stricter table would close every port the machine serves, and
an accept in one would open nothing the found firewall drops. What the mesh needs reachable it
declares as **openings**: a port, from where, on the incoming path or the forwarded path — a
published container port is forwarded, and a firewall that filters only incoming traffic never
sees it. The host converges each opening through the found firewall in that firewall's own terms,
marks it as the mesh's, removes only what it marked, and re-checks every opening on each reconcile
so a reload or a reboot does not lose it. An opening is state, not a command, so it travels over
the link like any other resource. **The mesh protects its own ports itself**: its foundation's ports
are opened from the private network and refused from anywhere else on the forwarded path, so the
store stays unreachable from outside whether or not the found firewall filters forwarded traffic.
One kind of found firewall is spoken; a machine with another is refused adoption rather than
adopted unprotected. Converging the node previews what is reachable now — listening sockets and
published ports — and what will close, then loads the derived filter and disables the found
firewall without flushing it. *How it is checked:* the adoption bed asserts the found firewall's
rules differ only by the mesh's marked rules, that the store is unreachable from off the private
network, that a second machine enrols through the openings before and after a reload and a reboot,
and that after the flip the declared port is open and the undeclared one closed.
sees it. The controller derives them from what the filter would be derived from: the assigned
modules' `listens`, the hub's port, the foundation's ports. The host converges each opening through
the found firewall in that firewall's own terms, marks it as the mesh's, removes only what it
marked, and re-checks every opening on each reconcile so a reload or a reboot does not lose it for
longer than one reconcile. An opening is state, not a command, so it travels over the link like any
other resource. **The mesh guards its own ports in a table of its own that only refuses** —
accepting by default and holding nothing but refusals, so it cannot close what the machine serves,
and the found firewall's reload does not touch it. It refuses the store's port and the broker's
management port from outside the private network, matched on the port the packet was sent to; the
bus, the registry and the hub's port stay reachable from anywhere, as a node enrols and pulls
before it has a private-network address. One kind of found firewall is spoken; a machine with none
needs no openings, and a machine with another kind is refused adoption. Converging the node refuses
while a found container is still held; otherwise it previews what is reachable now — listening
sockets and published ports — what will close and which modules it will take, then loads the
derived filter in place of the refusal-only table and disables the found firewall without flushing
it. *How it is checked:* the adoption bed asserts the found firewall's rules differ only by the
mesh's marked rules, that the store is unreachable from off the private network before and after
the found firewall reloads, that a machine not yet enrolled reaches the bus, that a second machine
enrols through the openings before and after a reload and a reboot, and that after the flip the
declared port is open and the undeclared one closed.
## 5 — Certificates
+4 -5
View File
@@ -391,12 +391,11 @@ should be and sends it; the host applies the difference and removes what is no l
The host removes what it *made* and leaves what it merely *configured*. Uninstalling a container
runtime because a declaration changed would stop every container on the node.
---
*On an adopted node* ([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)),
what the host is holding was found, not made, so nothing held is ever removed: a held file whose
module is unassigned stays where it is.
what the host is holding was found, not made, so nothing held is ever removed: a held file or
container whose module is unassigned stays where it is.
---
## enrolled ⇄ disconnected
+5 -5
View File
@@ -110,11 +110,11 @@ ruleset, the private network's endpoint, the addresses consumers are given — r
the control-node measured, the store's and the bus's usual ports were free and the registry's, the
broker's management port and, as the private network's hub, its port were held. Genesis also
checks the private network's range does not overlap a tunnel the predecessor runs. Genesis adopted
**does not load the base ruleset**: the machine's own firewall already filters, and the mesh opens
its foundation's ports through it and refuses them from outside itself
([08-connectivity](08-connectivity.md)). **A converged genesis refuses a machine in use** — an
active firewall or listeners that are not the mesh's — so a forgotten flag cannot close a working
machine. *How it is checked:* the adoption bed raises genesis converged on a machine in use and
**does not load the base ruleset**: the machine's own firewall already filters; the mesh opens its
foundation's ports through it and keeps the store from outside with a table of its own that only
refuses ([08-connectivity](08-connectivity.md)). **A converged genesis refuses a machine in use** —
a container running, or a port listening that is neither ssh nor the operating system's own — so a
forgotten flag cannot close a working machine. *How it is checked:* the adoption bed raises genesis converged on a machine in use and
asserts the refusal, then adopted with the registry's port held and asserts the refusal names its
holder, then with another port given asserts the foundation comes up, stays on that port once
adopted as modules, and the machine's service is still reachable.