diff --git a/02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md b/02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md new file mode 100644 index 0000000..7a955c4 --- /dev/null +++ b/02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md @@ -0,0 +1,75 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-02 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0005-the-node-host.md +--- + +# 184. A service the mesh asked to run is still running a moment later + +## Context + +The host already refuses to take a service manager's word for it. Three places in one function read +a unit back after acting on it, each with a comment saying why: *a service manager accepting a +command says the transaction was accepted, not that the unit is running — one that starts and +immediately dies satisfies it.* The intent was right and the implementation did not reach it. + +On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the +files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at +that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public +machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said +so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing. + +The read-back races the failure. A service manager returns when it has started the process; a daemon +that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One +look sees `activating` or `active` whatever the process is about to do, and *the host reports success +for a machine that is already wrong* — the one shape of failure this host exists to refuse +([ADR 0005](0005-the-node-host.md)). + +A command the module declares — *test the configuration before restarting* — was considered and +rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification +command is a command: a declaration that carried one would be remote execution over the bus, +arriving as root on every machine, which is a far larger door than the fault it closes. The host +does not need one. It already knows what it asked for. + +## Decision + +**1. A unit the host has just asked to run is read twice**, with a pause between the reads long +enough for a daemon that refuses its configuration to have exited. Not running at the second look is +a failure of that resource, named with the unit and the state it is in — the same failure the single +read was always meant to catch. + +**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both +looks and is accepted, exactly as before. What the second look catches is a unit that *was* running +and is not any more. A service asked to be stopped is not waited on at all. + +**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking +the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and +no command reaches a machine that did not already come from a built artifact. + +## Consequences + +- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is + bounded by the number of services that changed in that apply, which is usually none. +- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the + resolver — now fails its apply when the composition is bad, instead of reporting success onto a + dead daemon. `status` names the machine, which is how the operator finds out. +- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s + manifest check is what refuses the one that caused this, at merge time; this record is what makes + the *next* one visible within a minute rather than invisible until something asks the daemon a + question. + +## How this is checked + +| Rule | Checked by | +|---|---| +| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does | +| A unit still starting is accepted at both looks | a host test | +| A service asked to be stopped is not waited on | a host test | + +## References + +- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) +- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md) diff --git a/02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md b/02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md new file mode 100644 index 0000000..fed69e8 --- /dev/null +++ b/02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md @@ -0,0 +1,73 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-02 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md +--- + +# 185. A control plane behind its seat's row serves what it can + +## Context + +The mesh's own verbs are the controller seat's tools, and the seat's row is the store's +([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). A control plane reads the +row at start and installs a handler per verb; a verb the row carries that the binary cannot run was +refused at start rather than at the first call, so that a disagreement between the row and the +binary was said early. The refusal aborted the start. + +On 2026-10-02 a merge added one verb. The new control plane started, widened the row, and ran. A +push a few seconds later recreated its container at the previous image — a stale declaration from +an overlapping wave, [issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) — +and the older binary read a row naming a word it had never heard. It refused to start, and kept +refusing. The mesh had no voice for ten minutes: no verb answered, no node could be pushed, no build +was dispatched, and `status` said nothing because `status` is one of the verbs that had stopped +being served. The way back was a person running the binary by hand outside its service, because the +push that would have replaced it is itself a verb of the control plane that was down. + +The check was right about the fact and wrong about the cost. A row ahead of a binary is the ordinary +state of a roll-out: the row is widened by whichever control plane starts first, and a mesh with one +control plane sees that gap on every merge that adds a verb. Making it fatal turned a transient into +an outage with no path out that did not need a human. + +## Decision + +**1. A control plane serves the verbs it can run and does not refuse to start for the ones it +cannot.** The row remains the authority on what the seat serves; this is only about what this binary +does when it is behind the row. + +**2. A verb it cannot run answers the reason.** Not silence and not a missing subject: a caller gets +a sentence naming the verb, saying this control plane cannot run it and that it is a verb of a newer +build. A verb that is simply absent from the row is still not served at all — that is the row +deciding, which is unchanged. + +**3. It says so once at start**, naming every verb of the row it cannot run, so the gap is visible +in the log of the thing that has it rather than only at the moment somebody calls one. + +**4. A mesh with no controller seat at all is still a refusal.** That is not a version gap, it is a +mesh that has not been seeded, and nothing this control plane does would be meaningful. + +## Consequences + +- An overlapping roll-out costs the verbs the newer build added, for as long as the older binary is + in place. Everything else — every push, every build, every read — keeps working, and the ordinary + machinery that notices a machine is behind is what puts the newer binary back. +- The log gains one line on a control plane that is behind, and nothing on one that is not. +- Issue 201's other half remains: the push that sent a stale declaration is a race worth closing on + its own terms. This record makes that race survivable rather than fatal, which is the difference + between a transient and an outage, and is deliberately the cheaper half. + +## How this is checked + +| Rule | Checked by | +|---|---| +| A row carrying a verb this build cannot run still serves every verb it can, and names the one it cannot | a controller test over a widened row | +| The unknown verb answers a sentence naming itself and saying this build is behind | the same test | +| A mesh with no controller seat is refused | the existing start-up path | + +## References + +- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) +- [Issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) +- [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 1a57948..8f3bbbd 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -184,6 +184,8 @@ python3 00-META/checks/index.py fail if stale - **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md) - **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) - **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md) +- **0184** — [A service the mesh asked to run is still running a moment later](0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md) +- **0185** — [A control plane behind its seat's row serves what it can](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/05-the-node-host.md b/03-DESIGN/01-to-be/05-the-node-host.md index 36da208..499d344 100644 --- a/03-DESIGN/01-to-be/05-the-node-host.md +++ b/03-DESIGN/01-to-be/05-the-node-host.md @@ -4,6 +4,7 @@ status: in-progress code: [mesh-host] updated: 2026-10-02 decisions: + - 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md - 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md - 02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md @@ -498,3 +499,17 @@ run and reported; the exit follows an in-flight apply rather than interrupting i not start is rolled back once and the second failure halts; a completed reconcile retires what is older than the predecessor and never the predecessor; and the newest of two delivered versions is the one that runs. + +## A service is still running a moment later, 2026-10-02 + +[ADR 0184](../../02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md). +The host has always read a unit back after acting on it, because a service manager accepting a +command says the transaction was accepted and nothing about the process. The read raced the failure: +a daemon that refuses the configuration the mesh just wrote exits a fraction of a second after the +manager returns, and one look sees it alive. So the host looks twice, with a pause between, and a +unit that was running and is not any more fails its resource by name. A unit still coming up reads +as running at both looks and is accepted; a service asked to stop is not waited on. + +No command for this reaches a machine. A module declaring *how to test my configuration* was weighed +and refused: the link carries no actions, and a verification command is one. The host is checking +the state it was told to establish, which is what it is for. *How it is checked:* ADR 0184's table. diff --git a/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md b/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md index 0daa672..417d803 100644 --- a/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md +++ b/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md @@ -3,7 +3,7 @@ status: open opened: 2026-10-02 located-in: - mesh-controller -fixed-by: +fixed-by: 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md amended-design: --- @@ -56,3 +56,19 @@ one, wait for `node show` on the control node to report the new controller befor - [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) - mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go` + +## Half of it is closed, 2026-10-02 + +[ADR 0185](../../02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) takes +the outage out of it: a control plane behind its seat's row now serves every verb it can run, says +which it cannot, and answers the reason when one of those is called. The same race today would cost +the verbs the newer build added, for as long as the older binary is in place, and the ordinary +"this machine is behind" machinery would put the newer one back without a hand. + +**The race itself is still open**, and this report stays open for it. A push sends the declaration +it composed; two waves overlapping on one machine can leave the later send carrying the earlier +content, and the declaration's sequence orders the sends rather than what is in them +(the numbering of [issue 107](../107-a-declaration-carries-no-order/00-report.md) orders arrival, not freshness). The +smaller of the two fixes the report first suggested — a push never sending a control plane a digest +older than the one it is running — is still the one to build, and is now a correctness nicety rather +than the difference between a working mesh and a dead one.