From 329a24fdae52a0846c45c4967f498def584f8880 Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 2 Oct 2026 18:16:24 +0200 Subject: [PATCH 1/2] ADRs 0184 and 0185: a service is still running a moment later; a control plane behind its row serves what it can; issue 201 half closed --- ...-to-run-is-still-running-a-moment-later.md | 75 +++++++++++++++++++ ...behind-its-seats-row-serves-what-it-can.md | 73 ++++++++++++++++++ 02-DECISIONS/README.md | 2 + 03-DESIGN/01-to-be/05-the-node-host.md | 15 ++++ .../00-report.md | 18 ++++- 5 files changed, 182 insertions(+), 1 deletion(-) create mode 100644 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md create mode 100644 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md diff --git a/02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md b/02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md new file mode 100644 index 0000000..7a955c4 --- /dev/null +++ b/02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md @@ -0,0 +1,75 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-02 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0005-the-node-host.md +--- + +# 184. A service the mesh asked to run is still running a moment later + +## Context + +The host already refuses to take a service manager's word for it. Three places in one function read +a unit back after acting on it, each with a comment saying why: *a service manager accepting a +command says the transaction was accepted, not that the unit is running — one that starts and +immediately dies satisfies it.* The intent was right and the implementation did not reach it. + +On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the +files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at +that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public +machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said +so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing. + +The read-back races the failure. A service manager returns when it has started the process; a daemon +that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One +look sees `activating` or `active` whatever the process is about to do, and *the host reports success +for a machine that is already wrong* — the one shape of failure this host exists to refuse +([ADR 0005](0005-the-node-host.md)). + +A command the module declares — *test the configuration before restarting* — was considered and +rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification +command is a command: a declaration that carried one would be remote execution over the bus, +arriving as root on every machine, which is a far larger door than the fault it closes. The host +does not need one. It already knows what it asked for. + +## Decision + +**1. A unit the host has just asked to run is read twice**, with a pause between the reads long +enough for a daemon that refuses its configuration to have exited. Not running at the second look is +a failure of that resource, named with the unit and the state it is in — the same failure the single +read was always meant to catch. + +**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both +looks and is accepted, exactly as before. What the second look catches is a unit that *was* running +and is not any more. A service asked to be stopped is not waited on at all. + +**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking +the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and +no command reaches a machine that did not already come from a built artifact. + +## Consequences + +- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is + bounded by the number of services that changed in that apply, which is usually none. +- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the + resolver — now fails its apply when the composition is bad, instead of reporting success onto a + dead daemon. `status` names the machine, which is how the operator finds out. +- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s + manifest check is what refuses the one that caused this, at merge time; this record is what makes + the *next* one visible within a minute rather than invisible until something asks the daemon a + question. + +## How this is checked + +| Rule | Checked by | +|---|---| +| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does | +| A unit still starting is accepted at both looks | a host test | +| A service asked to be stopped is not waited on | a host test | + +## References + +- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) +- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md) diff --git a/02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md b/02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md new file mode 100644 index 0000000..fed69e8 --- /dev/null +++ b/02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md @@ -0,0 +1,73 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-02 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md +--- + +# 185. A control plane behind its seat's row serves what it can + +## Context + +The mesh's own verbs are the controller seat's tools, and the seat's row is the store's +([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). A control plane reads the +row at start and installs a handler per verb; a verb the row carries that the binary cannot run was +refused at start rather than at the first call, so that a disagreement between the row and the +binary was said early. The refusal aborted the start. + +On 2026-10-02 a merge added one verb. The new control plane started, widened the row, and ran. A +push a few seconds later recreated its container at the previous image — a stale declaration from +an overlapping wave, [issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) — +and the older binary read a row naming a word it had never heard. It refused to start, and kept +refusing. The mesh had no voice for ten minutes: no verb answered, no node could be pushed, no build +was dispatched, and `status` said nothing because `status` is one of the verbs that had stopped +being served. The way back was a person running the binary by hand outside its service, because the +push that would have replaced it is itself a verb of the control plane that was down. + +The check was right about the fact and wrong about the cost. A row ahead of a binary is the ordinary +state of a roll-out: the row is widened by whichever control plane starts first, and a mesh with one +control plane sees that gap on every merge that adds a verb. Making it fatal turned a transient into +an outage with no path out that did not need a human. + +## Decision + +**1. A control plane serves the verbs it can run and does not refuse to start for the ones it +cannot.** The row remains the authority on what the seat serves; this is only about what this binary +does when it is behind the row. + +**2. A verb it cannot run answers the reason.** Not silence and not a missing subject: a caller gets +a sentence naming the verb, saying this control plane cannot run it and that it is a verb of a newer +build. A verb that is simply absent from the row is still not served at all — that is the row +deciding, which is unchanged. + +**3. It says so once at start**, naming every verb of the row it cannot run, so the gap is visible +in the log of the thing that has it rather than only at the moment somebody calls one. + +**4. A mesh with no controller seat at all is still a refusal.** That is not a version gap, it is a +mesh that has not been seeded, and nothing this control plane does would be meaningful. + +## Consequences + +- An overlapping roll-out costs the verbs the newer build added, for as long as the older binary is + in place. Everything else — every push, every build, every read — keeps working, and the ordinary + machinery that notices a machine is behind is what puts the newer binary back. +- The log gains one line on a control plane that is behind, and nothing on one that is not. +- Issue 201's other half remains: the push that sent a stale declaration is a race worth closing on + its own terms. This record makes that race survivable rather than fatal, which is the difference + between a transient and an outage, and is deliberately the cheaper half. + +## How this is checked + +| Rule | Checked by | +|---|---| +| A row carrying a verb this build cannot run still serves every verb it can, and names the one it cannot | a controller test over a widened row | +| The unknown verb answers a sentence naming itself and saying this build is behind | the same test | +| A mesh with no controller seat is refused | the existing start-up path | + +## References + +- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) +- [Issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) +- [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 1a57948..8f3bbbd 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -184,6 +184,8 @@ python3 00-META/checks/index.py fail if stale - **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md) - **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) - **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md) +- **0184** — [A service the mesh asked to run is still running a moment later](0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md) +- **0185** — [A control plane behind its seat's row serves what it can](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/05-the-node-host.md b/03-DESIGN/01-to-be/05-the-node-host.md index 36da208..499d344 100644 --- a/03-DESIGN/01-to-be/05-the-node-host.md +++ b/03-DESIGN/01-to-be/05-the-node-host.md @@ -4,6 +4,7 @@ status: in-progress code: [mesh-host] updated: 2026-10-02 decisions: + - 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md - 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md - 02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md @@ -498,3 +499,17 @@ run and reported; the exit follows an in-flight apply rather than interrupting i not start is rolled back once and the second failure halts; a completed reconcile retires what is older than the predecessor and never the predecessor; and the newest of two delivered versions is the one that runs. + +## A service is still running a moment later, 2026-10-02 + +[ADR 0184](../../02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md). +The host has always read a unit back after acting on it, because a service manager accepting a +command says the transaction was accepted and nothing about the process. The read raced the failure: +a daemon that refuses the configuration the mesh just wrote exits a fraction of a second after the +manager returns, and one look sees it alive. So the host looks twice, with a pause between, and a +unit that was running and is not any more fails its resource by name. A unit still coming up reads +as running at both looks and is accepted; a service asked to stop is not waited on. + +No command for this reaches a machine. A module declaring *how to test my configuration* was weighed +and refused: the link carries no actions, and a verification command is one. The host is checking +the state it was told to establish, which is what it is for. *How it is checked:* ADR 0184's table. diff --git a/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md b/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md index 0daa672..417d803 100644 --- a/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md +++ b/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md @@ -3,7 +3,7 @@ status: open opened: 2026-10-02 located-in: - mesh-controller -fixed-by: +fixed-by: 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md amended-design: --- @@ -56,3 +56,19 @@ one, wait for `node show` on the control node to report the new controller befor - [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) - mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go` + +## Half of it is closed, 2026-10-02 + +[ADR 0185](../../02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) takes +the outage out of it: a control plane behind its seat's row now serves every verb it can run, says +which it cannot, and answers the reason when one of those is called. The same race today would cost +the verbs the newer build added, for as long as the older binary is in place, and the ordinary +"this machine is behind" machinery would put the newer one back without a hand. + +**The race itself is still open**, and this report stays open for it. A push sends the declaration +it composed; two waves overlapping on one machine can leave the later send carrying the earlier +content, and the declaration's sequence orders the sends rather than what is in them +(the numbering of [issue 107](../107-a-declaration-carries-no-order/00-report.md) orders arrival, not freshness). The +smaller of the two fixes the report first suggested — a push never sending a control plane a digest +older than the one it is running — is still the one to build, and is now a correctness nicety rather +than the difference between a working mesh and a dead one. -- 2.54.0 From 131a5e4714072de869abe07fa638a708f7a84b42 Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 2 Oct 2026 18:19:12 +0200 Subject: [PATCH 2/2] Issue 201: what was established about the race while closing the outage half --- .../00-report.md | 28 ++++++++++++++----- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md b/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md index 417d803..53469f7 100644 --- a/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md +++ b/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md @@ -65,10 +65,24 @@ which it cannot, and answers the reason when one of those is called. The same ra the verbs the newer build added, for as long as the older binary is in place, and the ordinary "this machine is behind" machinery would put the newer one back without a hand. -**The race itself is still open**, and this report stays open for it. A push sends the declaration -it composed; two waves overlapping on one machine can leave the later send carrying the earlier -content, and the declaration's sequence orders the sends rather than what is in them -(the numbering of [issue 107](../107-a-declaration-carries-no-order/00-report.md) orders arrival, not freshness). The -smaller of the two fixes the report first suggested — a push never sending a control plane a digest -older than the one it is running — is still the one to build, and is now a correctness nicety rather -than the difference between a working mesh and a dead one. +**The race itself is still open**, and this report stays open for it. What was established while +closing the other half, so the next reader does not redo it: + +- Composing and sending are serialised per machine by a session advisory lock in the store, so two + control planes cannot compose one machine's declaration at the same time. The stale content did + not come from two concurrent composes. +- A container's image is resolved into the module's manifest when it is *built*, and a push composes + from the catalogue as it is at that moment, under the hold. So a compose that ran after the build + was taken in could not have named the older image. +- The declaration's sequence orders arrival and nothing else (the numbering of + [issue 107](../107-a-declaration-carries-no-order/00-report.md)); it cannot tell a later send + carrying earlier content from a later send carrying later content. The host refuses a declaration + numbered below the last it applied, and both of these were above it. +- The machine's own journal shows the two applies ten seconds apart and which replaced what; it does + not record which image each declaration named, which is the one fact that would settle it. A host + that recorded the digest it was told, per apply, would have answered this in a minute. + +So the trigger is not yet pinned, and guessing at the push path is the most expensive place in the +mesh to guess. The fix the report first suggested — a push never sending a control plane a digest +older than the one that machine reports running — closes the class without needing the trigger, and +is now a correctness nicety rather than the difference between a working mesh and a dead one. -- 2.54.0