jschoubben
605c9fd441
Two corrections and one new decision, all from Jochen catching things. Pushed, not polled. I described updates as landing "on the next reconcile", which reads as polling and is not the design. A declaration arrives as a message on a link that is already open; the host applies it then. Polling over an existing connection would be slower to land AND constant traffic to learn nothing. The timer is for drift and nothing else, and it cannot be replaced by an event for a definitional reason: drift is change the mesh did not make -- somebody edited a managed file, a distribution upgrade replaced a config -- so nothing will ever publish a message about it. Only looking finds it. Separated the heartbeat from the reconcile timer, which I had been conflating. They point in opposite directions and answer different questions: the timer looks at the machine and asks whether it still matches; the heartbeat reports upward and is what makes silence mean something. A node with nothing to do sends nothing, and without a heartbeat that is indistinguishable from a node that stopped. 0059 -- a host that cannot start is rolled back by the service manager. I had left this open on the grounds that recovery meant the host judging its own health. That objection does not survive being asked properly: a keepalive is something else judging the host. The watchdog must be local, because nothing dials a node and a host that cannot start cannot report -- so it is the service manager, which is already there. The failure it prevents is sharper than "the node is down": a host that will not start looks exactly like a machine somebody switched off, which is the one condition this design has deliberately decided not to alarm on. So a bad release reaches every node, each goes quiet, and the mesh reports a fleet of sleeping laptops. Confirmed means started and completed one reconcile -- deliberately not "the link is up", or a laptop on a train would roll itself back. The rollback is a script shipped by the package, not a host subcommand, because a binary that will not start cannot be its own recovery. It rolls back once: a second failure means the machine is the problem, not the binary. Also refined the records checker, which produced a false positive: a proposed record may extend another proposed one, because decisions are drafted in chains and the alternative is marking things accepted to satisfy a check. An accepted document resting on a proposed record still fails, and that was verified. 0057, 0058 and 0059 are all proposed.
9.3 KiB
status, date, deciders, reconstructed, extends
| status | date | deciders | reconstructed | extends |
|---|---|---|---|---|
| proposed | 2026-08-27 | jochen | false | 0041-the-host-depends-on-nothing.md |
57. The host is a root service, installed as a package, that never manages itself
Context
05-the-node-host.md describes what the host does
and never says what it is at runtime. Searched: the words daemon, long-running, interval,
poll and heartbeat appear in none of it, nor in
ADR 0038 or
ADR 0039.
What exists today is a command that runs and exits — mesh-host apply FILE. What the design
requires is a process holding an outbound link to the control plane. Nobody wrote down that
these are different things, so several questions have no answer: does it reconcile on a timer,
what happens when the link drops, and who installs the unit that starts it — given that the host
is the thing that installs units.
Decision
It runs on every node, and that is what a node is
ADR 0036 defines a node as a managed machine. The host is what makes it managed, so a machine without one is not a node with a missing component; it is not a node. There is no partial mode, no agentless node, and no second way in.
It is a root service
Root, because there is no useful subset of its job that is not privileged: it writes under
/etc, installs packages, manages units, and runs containers. A host that dropped privilege
could apply almost nothing, and the almost is where the confusion would live.
A service rather than a command, because it holds the link, and something must survive a reboot to hold it. The command-line entry points remain — they are how a person inspects and rescues a machine — but the ordinary case is a unit that is always up.
It never manages its own unit
The host's own service file is not a resource the host applies. The temptation is obvious — it manages units, and its own unit is a unit — and it ends with a host stopping itself half way through an apply, leaving a machine in a state nothing is running to fix.
So the boundary is: the installation owns the host; the host owns everything else. A declaration that names the host's own unit is refused rather than obeyed.
It is installed as a package, and a tarball is the floor
Two mechanisms, and the second is not a fallback for the first failing — it is what makes the first possible.
package — pacman -S nox-mesh-host |
the ordinary path. Carries the binary, the unit file, the state directory, and an upgrade path |
tarball — curl … | tar -xz |
the floor. One static binary, no repository, no distribution assumed |
Why a package rather than only a binary. ADR 0041 says copying the binary onto a machine is the whole installation, and that remains true of the binary. But a unit file, a state directory and an upgrade path are real, and something has to own them. A package that installs one statically linked binary plus a unit file adds no runtime dependency — 0041 is about what must already be present for the host to work, not about how the bytes arrived.
Why the tarball must keep working. The package lives in a repository, and the mesh's own repository is hosted on the mesh. A first node cannot fetch from a mesh that does not exist yet, and neither can a node whose mesh is down — which is exactly when somebody is trying to fix it. Any path that requires the mesh to install the thing that joins the mesh is a circle, so the tarball is the path that is never allowed to acquire a dependency.
The host may replace its own binary; it may not stop its own unit
The first draft of this record said the mesh must not upgrade the host at all. That was too broad, and it conflated two different acts.
Replacing the binary is safe. Unix keeps the running executable's inode open, so a package upgrade writes a new file and the running process continues on the old one, undisturbed. Stopping the unit is what is unsafe — that is the host killing itself part-way through an apply, leaving a machine with nothing running to finish or fix it.
So the host may apply a package naming itself. What it must never do is ask the service
manager to restart it.
The restart happens by exiting, not by asking. When the host notices its own executable has
been replaced, it finishes the apply it is in, reports what it did, and exits cleanly. The
supervisor's Restart=always starts it again, on the new binary. Nothing stops the host; the
host stops, having finished.
Three conditions, and they are the whole safety argument:
- after the apply completes and its outcomes are recorded — never mid-way;
- only when the executable actually changed, which Linux reports plainly: a replaced
/proc/self/exereads as the old path marked deleted; - exit zero, so a restart is what a supervisor does next rather than a failure it backs off from.
This makes a fleet-wide host upgrade an ordinary declaration, which the first draft gave up.
Changes are pushed. The timer is for drift, and only for drift
The host does not poll for work. A new declaration arrives as a message on the link, and the host applies it then (ADR 0001). Polling for updates over a connection that already exists would be strictly worse in both directions: slower to land, and constant traffic to learn nothing.
Four triggers, and only one of them is a clock:
| Trigger | Kind | Why |
|---|---|---|
| a declaration arrives | pushed | the ordinary path — this is how changes land |
| start | event | the machine may have changed while nothing was running |
| reconnect | event | declarations may have been missed while disconnected |
| a timer | periodic | drift, and nothing else |
The timer cannot be replaced by an event, and the reason is definitional. Drift is change the mesh did not make — somebody edited a managed file, a distribution upgrade replaced a config, a container was stopped by hand. Nothing will ever send a message about it, because the thing that did it is not part of the mesh. A local periodic check is the only way to see it at all.
Without it, owned reports what the host applied rather than what is there, which is
ADR 0035 violated by omission.
Ten minutes, configurable. The check is cheap: it asks the package database, the service manager and the container runtime about resources the host already knows it owns.
It reports upward on a heartbeat
Separate from reconciling, and easy to conflate with it: the node tells the mesh what it is — its running version, what it holds, what it last applied — on link, after every apply, and periodically while idle.
The heartbeat is what makes silence mean something. Without it, the mesh cannot distinguish a node that is fine and has had nothing to do from one that stopped. With it, last heard from is a fact per node, and ADR 0059 is what handles the case where the node cannot report at all.
Consequences
- Adoption becomes two concrete steps, which is the point of writing this down: install the package, then hand it a token. Nothing else.
- The host gains a mode it does not have, and it is the larger half of stage 3. Today every entry point runs and exits.
- Refusing to manage its own unit needs enforcing, not just stating. A declaration naming the host's unit must be refused by name, and that refusal is a test.
- A host that cannot reach the mesh keeps reconciling from its store, which is ADR 0036 made operational rather than aspirational: a disconnected node is not merely tolerated, it is actively holding its machine in the last state it was told to hold.
- The timer makes drift visible and also makes it loud. A resource the host cannot apply
will now fail every ten minutes rather than once. That is correct and it needs somewhere to go
other than a log nobody reads — which is
observability's, and it does not exist yet. - A host upgrade is an ordinary declaration, which is worth the care it needs: the exit path is the only place the host deliberately stops, and a bug there is a node that restarts in a loop or never comes back. It wants a test that the host does not exit when its binary is unchanged, as much as one that it does when it changed.
- A version-skewed fleet is now normal and needs saying. Nodes restart onto the new binary at whatever moment their apply finishes, so "the fleet is upgraded" is a range rather than an instant. What a node reports as its version must be the running one, not the installed one, or the mesh will believe an upgrade landed before it took effect.