Merge pull request 'ADR 0141: the host delivers its own successor' (#173) from decision/0141-the-host-delivers-its-own-successor into main

2026-09-28 22:22:28 +00:00
parent 61dc90e508 a815433214
commit c262de3833
4 changed files with 174 additions and 2 deletions
@@ -0,0 +1,128 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0005-the-node-host.md
+---
+
+# 141. The host delivers its own successor, and versions live side by side
+
+## Context
+
+[Issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md). A
+merge builds every changed module and the control plane — which is itself a module — and the result
+reaches the machines running it with nobody asking. The host is the exception: it is not a build
+target, no declaration delivers it, and every machine in this mesh runs a byte-identical binary that
+somebody built on a workstation and copied out.
+
+The half that *recovers* from a bad host exists. `internal/upgrade` can tell that the executable this
+process started from was replaced on disk, and it records which version last completed a reconcile.
+The launcher counts consecutive failed starts, calls a rollback at the limit, and treats a clean exit
+as the host standing aside so that the next loop runs whatever is on disk now. That supervision is
+complete and correct.
+
+Two things make it dead code:
+
+- **`Replaced()` is called by nothing but its own tests.** Nothing tells the running host that a
+  successor is waiting.
+- **The rollback resolves a version through the machine's package manager** — `pacman -U` from the
+  package cache. No machine here has the host installed as a package, so the recovery cannot run on
+  any of them; and being written in one package manager's terms, it cannot run on two of the three
+  operating systems the host is built for — [ADR 0005](0005-the-node-host.md) builds one binary per
+  operating system, pinned at link time.
+
+**The record already points at the answer.** What is kept is a *version*, not a path. Keeping a
+version is only useful to something that can choose between versions present on the machine, which is
+what the package manager was being asked to do. The versions can simply be on disk.
+
+## Considered Options
+
+1. **Deliver the host as a package, as the rollback assumes.** Rejected: it needs a package built and
+   a repository trusted per operating system, three of each, and the existing `package` resource
+   asserts presence and deliberately never a version — "version is the package manager's business and
+   the mesh does not hold a second opinion about it" — so it cannot ask for a particular host anyway.
+   Heaviest of the three and the only one that is different on every machine.
+2. **Write the new binary over the running one.** Rejected on a fact: a running executable cannot be
+   truncated, and `archive` opens what it unpacks with `O_TRUNC`. It could be made to write and
+   rename, which is better hygiene and worth doing for its own sake, but it buys nothing here that
+   option 3 does not, and it leaves rollback with nowhere to go back to.
+3. **Versions side by side; the newest retires the old.** Adopted.
+
+## Decision
+
+**A host version is delivered as an archive into a directory named for it, and never over a running
+one.** The declaration names it like any other archive — fetched by digest, the digest checked before
+anything is unpacked. Nothing new travels, no new resource kind, and no change to how archives are
+applied, because the path being written is not the path being executed.
+
+**The launcher starts the most recently delivered version.** That is what "the newest" means: the
+version whose directory arrived last. It reads no pointer and follows no link — the mesh creates no
+links ([ADR 0012](0012-the-mesh-creates-no-symlinks.md)) — and the version is in the path, so nothing
+has to be told what is running.
+
+**The running host stands aside for a successor, and only between reconciles.** Finding a newer
+version delivered, it finishes the reconcile it is in and exits cleanly. The launcher already reads a
+clean exit as exactly this and starts what is on disk now. A host that stood aside mid-apply is the
+half-configured machine this project exists to prevent, so the check happens at the boundary and
+nowhere else.
+
+**A version that completes a reconcile records itself, and retires what came before it.** The
+known-good record is written as it is today. Then versions older than the one before the running one
+are removed: the running version and its predecessor are kept, which is exactly what a rollback
+needs, and nothing else accumulates.
+
+**Rollback starts the previous version instead of reinstalling a package.** At the failure limit the
+launcher pins the known-good version and starts that, once. The second failure is still a different
+diagnosis — the previously working version does not run either, so it is the machine and not the
+binary — and the halt is unchanged. No package manager, no package cache, and the same script on every
+operating system.
+
+**A machine says which host version it is running,** on the report it already sends, beside the other
+facts it states about itself. Without it nothing can say a machine is behind, so "every machine
+current with its source" cannot include the host.
+
+## Consequences
+
+- **The host becomes a build target and a module** — a module whose resource is the next host, applied
+  by the host that is running. The bootstrap is not circular because the two are different versions in
+  different directories.
+- **Rollback becomes usable on every machine**, having been usable on none. It also stops being
+  written in one operating system's terms.
+- **One copy by hand remains, once.** The first host that understands versioned directories cannot be
+  fetched by a host that does not. That copy is the last, and it is the honest cost of the change
+  rather than a step in the design.
+- **Two versions occupy disk instead of one.** About nine megabytes. The predecessor is the price of a
+  rollback that does not depend on a cache somebody else may clean.
+- **What got harder:** a host must now be able to find its own successor and to judge when it is safe
+  to stand aside. Both are between reconciles, which is the only moment the host is not mid-change.
+- **A machine that is never told a newer version keeps running what it has**, indefinitely and
+  visibly, because its report says which version that is.
+
+## How it is checked
+
+- **A delivered version is run, and the old one is not.** A bed delivers a second version to a machine
+  running the first; the host exits between reconciles, the launcher starts the new one, and the
+  machine reports the new version. This fails against the previous behaviour, where nothing notices a
+  delivered version at all.
+- **It stands aside between reconciles and never inside one.** Asserted by delivering a version while
+  an apply is in flight: the apply completes, and the exit follows it.
+- **A version that will not start is rolled back to its predecessor, once**, and the second failure
+  halts with the machine named rather than the binary — asserted with no package manager involved.
+- **A completed reconcile retires what is older than the predecessor**, and never the predecessor
+  itself, because that is what a rollback needs. Asserted on the directory afterwards.
+- **The report names the running version**, asserted end to end rather than on the function that reads
+  it, since the point is that the control plane can tell a machine is behind.
+- **The launcher picks the newest delivered version** with no pointer file and no link, asserted by
+  delivering two and checking which runs.
+
+## References
+
+- [ADR 0005](0005-the-node-host.md) — the host, and what its supervision is for
+- [ADR 0010](0010-delivery.md) — a declaration is owned resources; this adds no kind to it
+- [ADR 0012](0012-the-mesh-creates-no-symlinks.md) — why the version is in the path
+- [ADR 0005](0005-the-node-host.md), *it is built per operating system* — why a rollback written in
+  one package manager's terms was wrong for two of three
+- [issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md) — the
+  measurement
@@ -221,6 +221,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0138** — [An assignment binds an endpoint and says how far it reaches](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)
 - **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md)  *(superseded)*
 - **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
+- **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)

 ### How it is built

@@ -2,8 +2,9 @@
 layer: to-be
 status: in-progress
 code: [mesh-host]
-updated: 2026-09-22
+updated: 2026-09-29
 decisions:
+  - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
  - 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
  - 02-DECISIONS/0102-the-mesh-writes-into-a-shared-file-never-over-it.md
  - 02-DECISIONS/0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md
@@ -423,3 +424,45 @@ ignored an instruction and "applied" would be a lie. Applying stays one at a tim
 is not. **Checked** by the link's unit tests on the drain, and by the genesis bed's settle wait,
 which counts on a node catching up to the newest declaration rather than the oldest.

+## The host delivers its own successor
+
+*2026-09-29, from a change to the host that could reach no machine —
+[issue 142](../../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md),
+settled by [ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md).*
+
+A merge builds every changed module and the control plane, and the result reaches the machines running
+it with nobody asking. The host was the exception: not a build target, named by no declaration, and
+identical on every machine because somebody had copied it there.
+
+The supervision needed for this was already right. A clean exit from the host means it has stood aside,
+and the launcher's next turn runs whatever is on disk. Consecutive failed starts are counted, a
+rollback happens at the limit, and a second failure halts with the machine named rather than the binary.
+What was missing was smaller than it looked: nothing told the running host a successor was waiting, and
+the rollback resolved its known-good *version* through one operating system's package manager, which no
+machine here used.
+
+Keeping a version rather than a path was the clue. That is only useful to something that can choose
+between versions present on the machine — so the versions live side by side:
+
+- **A version arrives as an archive, in a directory named for it.** The ordinary resource, fetched by
+  digest and checked before anything is unpacked. The path written is never the path being executed, so
+  replacing a running binary — which the kernel refuses — never comes up.
+- **The launcher starts the most recently delivered version**, reading no pointer and following no
+  link, because the version is in the path.
+- **The running host stands aside between reconciles and never inside one.** Standing aside mid-apply is
+  the half-configured machine this document exists to prevent.
+- **A version that completes a reconcile records itself and retires what is older than its
+  predecessor.** The predecessor stays, because that is what a rollback needs.
+- **Rollback starts that predecessor** instead of reinstalling a package: no package manager, no cache
+  somebody else may clean, and the same script on every operating system.
+- **A machine says which host version it runs**, on the report it already sends, so being behind is
+  answerable at all.
+
+One copy by hand remains, once: the first host that understands versioned directories cannot be fetched
+by a host that does not.
+
+*How it is checked* is stated with the decision — a second version delivered to a running machine is
+run and reported; the exit follows an in-flight apply rather than interrupting it; a version that will
+not start is rolled back once and the second failure halts; a completed reconcile retires what is older
+than the predecessor and never the predecessor; and the newest of two delivered versions is the one
+that runs.
@@ -6,7 +6,7 @@ located-in:
  - mesh-host cmd/mesh-host
  - mesh-controller (no build source for the host; no resource delivers it)
 fixed-by:
-amended-design:
+amended-design: 03-DESIGN/01-to-be/05-the-node-host.md
 ---

 # 142 — The host is the one thing the mesh does not deliver