Merge pull request 'ADRs 0184 and 0185: a service is still running a moment later; a control plane behind its row serves what it can' (#298) from fix/a-service-asked-to-run-is-still-running into main

2026-10-02 16:25:44 +00:00
parent 7f72f3b79a 131a5e4714
commit a9f91fdd0c
5 changed files with 196 additions and 1 deletions
@@ -0,0 +1,75 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-02
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0005-the-node-host.md
+---
+
+# 184. A service the mesh asked to run is still running a moment later
+
+## Context
+
+The host already refuses to take a service manager's word for it. Three places in one function read
+a unit back after acting on it, each with a comment saying why: *a service manager accepting a
+command says the transaction was accepted, not that the unit is running — one that starts and
+immediately dies satisfies it.* The intent was right and the implementation did not reach it.
+
+On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the
+files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at
+that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public
+machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said
+so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing.
+
+The read-back races the failure. A service manager returns when it has started the process; a daemon
+that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One
+look sees `activating` or `active` whatever the process is about to do, and *the host reports success
+for a machine that is already wrong* — the one shape of failure this host exists to refuse
+([ADR 0005](0005-the-node-host.md)).
+
+A command the module declares — *test the configuration before restarting* — was considered and
+rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification
+command is a command: a declaration that carried one would be remote execution over the bus,
+arriving as root on every machine, which is a far larger door than the fault it closes. The host
+does not need one. It already knows what it asked for.
+
+## Decision
+
+**1. A unit the host has just asked to run is read twice**, with a pause between the reads long
+enough for a daemon that refuses its configuration to have exited. Not running at the second look is
+a failure of that resource, named with the unit and the state it is in — the same failure the single
+read was always meant to catch.
+
+**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both
+looks and is accepted, exactly as before. What the second look catches is a unit that *was* running
+and is not any more. A service asked to be stopped is not waited on at all.
+
+**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking
+the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and
+no command reaches a machine that did not already come from a built artifact.
+
+## Consequences
+
+- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is
+  bounded by the number of services that changed in that apply, which is usually none.
+- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the
+  resolver — now fails its apply when the composition is bad, instead of reporting success onto a
+  dead daemon. `status` names the machine, which is how the operator finds out.
+- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s
+  manifest check is what refuses the one that caused this, at merge time; this record is what makes
+  the *next* one visible within a minute rather than invisible until something asks the daemon a
+  question.
+
+## How this is checked
+
+| Rule | Checked by |
+|---|---|
+| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does |
+| A unit still starting is accepted at both looks | a host test |
+| A service asked to be stopped is not waited on | a host test |
+
+## References
+
+- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
+- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md)
@@ -0,0 +1,73 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-02
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
+---
+
+# 185. A control plane behind its seat's row serves what it can
+
+## Context
+
+The mesh's own verbs are the controller seat's tools, and the seat's row is the store's
+([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). A control plane reads the
+row at start and installs a handler per verb; a verb the row carries that the binary cannot run was
+refused at start rather than at the first call, so that a disagreement between the row and the
+binary was said early. The refusal aborted the start.
+
+On 2026-10-02 a merge added one verb. The new control plane started, widened the row, and ran. A
+push a few seconds later recreated its container at the previous image — a stale declaration from
+an overlapping wave, [issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) —
+and the older binary read a row naming a word it had never heard. It refused to start, and kept
+refusing. The mesh had no voice for ten minutes: no verb answered, no node could be pushed, no build
+was dispatched, and `status` said nothing because `status` is one of the verbs that had stopped
+being served. The way back was a person running the binary by hand outside its service, because the
+push that would have replaced it is itself a verb of the control plane that was down.
+
+The check was right about the fact and wrong about the cost. A row ahead of a binary is the ordinary
+state of a roll-out: the row is widened by whichever control plane starts first, and a mesh with one
+control plane sees that gap on every merge that adds a verb. Making it fatal turned a transient into
+an outage with no path out that did not need a human.
+
+## Decision
+
+**1. A control plane serves the verbs it can run and does not refuse to start for the ones it
+cannot.** The row remains the authority on what the seat serves; this is only about what this binary
+does when it is behind the row.
+
+**2. A verb it cannot run answers the reason.** Not silence and not a missing subject: a caller gets
+a sentence naming the verb, saying this control plane cannot run it and that it is a verb of a newer
+build. A verb that is simply absent from the row is still not served at all — that is the row
+deciding, which is unchanged.
+
+**3. It says so once at start**, naming every verb of the row it cannot run, so the gap is visible
+in the log of the thing that has it rather than only at the moment somebody calls one.
+
+**4. A mesh with no controller seat at all is still a refusal.** That is not a version gap, it is a
+mesh that has not been seeded, and nothing this control plane does would be meaningful.
+
+## Consequences
+
+- An overlapping roll-out costs the verbs the newer build added, for as long as the older binary is
+  in place. Everything else — every push, every build, every read — keeps working, and the ordinary
+  machinery that notices a machine is behind is what puts the newer binary back.
+- The log gains one line on a control plane that is behind, and nothing on one that is not.
+- Issue 201's other half remains: the push that sent a stale declaration is a race worth closing on
+  its own terms. This record makes that race survivable rather than fatal, which is the difference
+  between a transient and an outage, and is deliberately the cheaper half.
+
+## How this is checked
+
+| Rule | Checked by |
+|---|---|
+| A row carrying a verb this build cannot run still serves every verb it can, and names the one it cannot | a controller test over a widened row |
+| The unknown verb answers a sentence naming itself and saying this build is behind | the same test |
+| A mesh with no controller seat is refused | the existing start-up path |
+
+## References
+
+- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
+- [Issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)
+- [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)
@@ -184,6 +184,8 @@ python3 00-META/checks/index.py            fail if stale
 - **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md)
 - **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
 - **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
+- **0184** — [A service the mesh asked to run is still running a moment later](0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md)
+- **0185** — [A control plane behind its seat's row serves what it can](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)

 ### Its tiers, from the bottom up

@@ -4,6 +4,7 @@ status: in-progress
 code: [mesh-host]
 updated: 2026-10-02
 decisions:
+  - 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md
  - 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md
  - 02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md
  - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
@@ -498,3 +499,17 @@ run and reported; the exit follows an in-flight apply rather than interrupting i
 not start is rolled back once and the second failure halts; a completed reconcile retires what is older
 than the predecessor and never the predecessor; and the newest of two delivered versions is the one
 that runs.
+
+## A service is still running a moment later, 2026-10-02
+
+[ADR 0184](../../02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md).
+The host has always read a unit back after acting on it, because a service manager accepting a
+command says the transaction was accepted and nothing about the process. The read raced the failure:
+a daemon that refuses the configuration the mesh just wrote exits a fraction of a second after the
+manager returns, and one look sees it alive. So the host looks twice, with a pause between, and a
+unit that was running and is not any more fails its resource by name. A unit still coming up reads
+as running at both looks and is accepted; a service asked to stop is not waited on.
+
+No command for this reaches a machine. A module declaring *how to test my configuration* was weighed
+and refused: the link carries no actions, and a verification command is one. The host is checking
+the state it was told to establish, which is what it is for. *How it is checked:* ADR 0184's table.
@@ -3,7 +3,7 @@ status: open
 opened: 2026-10-02
 located-in:
  - mesh-controller
-fixed-by:
+fixed-by: 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md
 amended-design:
 ---

@@ -56,3 +56,33 @@ one, wait for `node show` on the control node to report the new controller befor

 - [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
 - mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`
+
+## Half of it is closed, 2026-10-02
+
+[ADR 0185](../../02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) takes
+the outage out of it: a control plane behind its seat's row now serves every verb it can run, says
+which it cannot, and answers the reason when one of those is called. The same race today would cost
+the verbs the newer build added, for as long as the older binary is in place, and the ordinary
+"this machine is behind" machinery would put the newer one back without a hand.
+
+**The race itself is still open**, and this report stays open for it. What was established while
+closing the other half, so the next reader does not redo it:
+
+- Composing and sending are serialised per machine by a session advisory lock in the store, so two
+  control planes cannot compose one machine's declaration at the same time. The stale content did
+  not come from two concurrent composes.
+- A container's image is resolved into the module's manifest when it is *built*, and a push composes
+  from the catalogue as it is at that moment, under the hold. So a compose that ran after the build
+  was taken in could not have named the older image.
+- The declaration's sequence orders arrival and nothing else (the numbering of
+  [issue 107](../107-a-declaration-carries-no-order/00-report.md)); it cannot tell a later send
+  carrying earlier content from a later send carrying later content. The host refuses a declaration
+  numbered below the last it applied, and both of these were above it.
+- The machine's own journal shows the two applies ten seconds apart and which replaced what; it does
+  not record which image each declaration named, which is the one fact that would settle it. A host
+  that recorded the digest it was told, per apply, would have answered this in a minute.
+
+So the trigger is not yet pinned, and guessing at the push path is the most expensive place in the
+mesh to guess. The fix the report first suggested — a push never sending a control plane a digest
+older than the one that machine reports running — closes the class without needing the trigger, and
+is now a correctness nicety rather than the difference between a working mesh and a dead one.