From e6f0e7e359d5de6cff932288bc44604d45340019 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 20:58:44 +0200 Subject: [PATCH] Issue 336: a send held for the bus's planned step raises no condition, so the operator is never asked For 28 minutes on 2026-10-08 every send to the control-node waited for a bus upgrade only a person can start, and nothing but a walk's note said so. Located in the controller: the refusal is kept as the walk's note, S3 fires only past a 30-minute bound with generic words, and probe DB reads only a running step. --- .../00-report.md | 96 +++++++++++++++++++ .../01-diagnosis.md | 60 ++++++++++++ 2 files changed, 156 insertions(+) create mode 100644 04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md create mode 100644 04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md diff --git a/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md new file mode 100644 index 00000000..d1b04adf --- /dev/null +++ b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md @@ -0,0 +1,96 @@ +--- +status: located +opened: 2026-10-08 +located-in: [mesh-controller cmd/mesh-controller (push.go sendToEach and bus_step.go busHeld, the hold; signals.go S3 and bus_step.go probe DB, the conditions that do not say it; merge_gate.go and delivery.go, which do not say it before the merge)] +fixed-by: +amended-design: +--- + +# 336. A send held for the bus's planned step raises nothing, and the operator is never asked + +## Symptom + +On 2026-10-08 (times UTC) a catalogue merge, the work of +[issue 327](../327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md), built a +new build of the bus (the `nats` module) for the control-node. A new bus build is a **planned step** +([ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md), +[to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §8): only a person's +`mesh-controller.bus upgrade` moves it, and no other send reaches the bus's machine while the new build +waits. + +From the merge on, every send to the control-node was held. `mesh-controller.plans` said, for example: + +> tier 0: sending mesh-host to laptop, control-node, workstation after tier 0: a new bus build waits for its +> planned step: sending control-node would replace the bus (nats 88135ad0 → 32307bd1), which is a planned +> step ... — tried again for 27m45s + +The same refusal held the controller's own walk and the release of builds kept back for a gate. + +For those 28 minutes `mesh-controller.conditions` listed **no condition** naming the hold or saying it +waited for a person. The operator was not told. The hold was found only because a walk looked stuck, and +the operator asked: "where would you be waiting then? Where is my approve button?" + +The operator then gave the word, the bus upgrade started at 18:52 and the bus answered healthy at 18:54. +The held sends then went through. + +## Why it is an issue + +The design makes the bus wait for a person, and that part worked: nothing replaced the bus unasked. But a +wait that only a person can end is said nowhere a person looks. The refusal is kept as the note on each +held walk, which is read only by somebody already asking `plans`. Rule 4 of +[ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) +(no failure is silent) is met in the letter and missed in effect: the mesh held every send to its +most important machine and said nothing until asked. + +A precedent exists for a wait on a person being said at once and not as lateness: +[ADR 0254](../../02-DECISIONS/0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md) +says a new login the operator owes as its own condition, `relogin-needed`. The bus's planned step has no +such condition. + +The diagnosis is in [01-diagnosis.md](01-diagnosis.md). + +## Impact + +- Every send to the control-node stops: the controller's own walk, the node-engine's, every module's walk + that includes the control-node, and gated builds waiting for release. Other machines in the same tier + wait with it. +- The operator learns of it only by noticing a stuck walk, which can take hours when nobody is watching. +- The only condition that could have fired, a tier past its bound (watchdog S3, "stalled"), fires no + sooner than 30 minutes in, and its words ("stuck halfway ... longer than usual. Nothing is lost") name + neither the bus nor the verb a person runs. It never fired in this case, because the step was taken + after 28 minutes. +- Nothing before the merge says that merging the change needs a bus upgrade by a person. The merge gate + and the delivery plan were green, and the merge silently blocked every later send to the control-node. + +## How it is checked + +| Rule | Check | +|---|---| +| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a walk wants to send raises a condition on the next watchdog tick, before any tier bound | +| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held walks or modules, the bus build from and to, and `mesh-controller.bus upgrade` | +| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a walk whose only refusal is the bus's planned step | +| it clears when the step starts | the test ends with a bus step started: the condition clears, `bus-maintenance` opens | +| the merge says it before it happens | the merge gate's test: a change that moves the bus's build reports that a bus upgrade by a person is needed and which machine waits until then | + +## Core or not + +**Core.** It is located in the controller, so it resolves with a replay in mesh-lab's replays register, +proved to fail on the commit before the fix and to pass on it (`replay:`), or with `replay-none:` saying +why none is possible. + +## Suggested fix + +1. **A condition for the wait.** While a pending bus build would move on its machine and at least one + send has been refused for it, raise a condition in the bus scope (a kind such as `bus-step-waiting`, + with the operator as its resolver). Its headline says, in plain words, what waits ("sends to the + control-node: the controller, the node-engine, …"), behind what (the bus build, from and to), since + when, and the verb that ends it (`mesh-controller.bus upgrade --why … --reversible|--irreversible`). It + is raised on the first refusal, not after a bound. A walk held only by it is excluded from S3, as a + walk held by a paused build seat is. +2. **An ask, once asks exist.** When the decision flow of ADR 0259 (an ask is a message on the bus, any + channel answers it; on an open pull request, not yet merged) is built, the condition becomes an ask + to the operator: "start the bus upgrade now?", whose yes runs the step with the operator's word + recorded as the hand act. +3. **Said before the merge.** The merge gate, or the delivery plan, says when a change moves the bus's + build: "merging this needs a bus upgrade by a person; until then nothing reaches the control-node." + Whether that is a warning or a hold for the operator's word is for the fix's decision. diff --git a/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md new file mode 100644 index 00000000..70d78006 --- /dev/null +++ b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md @@ -0,0 +1,60 @@ +# 336 — Diagnosis + +## 2026-10-08 + +**The records first.** `records_search` for "a new bus build waits for its planned step" found nothing. +A search for "planned step" found the decisions and designs that define the step (ADR 0221, 0227, 0236, +to-be 45 §8) and issue 280, none of which says how a wait for it is told to a person. + +**Where the words come from.** In mesh-controller, `errBusWaits` in `bus_step.go` is "a new bus build waits +for its planned step". `sendToEach` in `push.go` returns it, wrapped with `busHeld`'s reason, for any send +not marked as the bus step whose machine list includes a machine where the pending bus build would move. +The send is refused whole. `busHeld` (`bus_step.go`) gives the words "sending X would replace the bus +(nats from → to), which is a planned step: `bus upgrade --why …` …". So the remedy is in the words, but +only in the error. + +**Where the error goes.** In `release_plan.go`, the walk's step loop catches the error, keeps it as the +plan's note ("tier N: … — tried again"), saves the plan when the note changed, prints it to the +controller's log once, and tries again on the next tick. Nothing raises a condition from it. That is why +`plans` showed it and `conditions` did not. + +**Which watchdog could have said it.** Only S3 (`watchPlans`, `signals.go`), "plan tier progress", raises +`plan..stalled` when a walk has been in one tier past its bound. The bound is +`max(30 min, 3 × the p90 of that repository's measured tiers)` (`tierAtLeast = planWaitBound = 30 min`, +`watchdogs.go` `boundsOf`). So: + +- it fires no sooner than 30 minutes after the tier was entered; the hold here was ended by hand at about + 28 minutes, so it never fired; +- its words are generic ("Delivery of X is stuck halfway … longer than usual. Nothing is lost, and the + mesh keeps it where it is."). Its summary carries the plan's note, so the bus's reason would have + appeared there, but the headline a person reads names neither the bus nor the verb, and its resolver is + not the operator; +- a walk held by a paused build seat is excluded from S3 (`pausedWaiting`), and a walk waiting for its + delivery's word has its own watchdog (S16). A walk held for the bus step has neither. + +**What the bus step does raise.** Probe DB (`probeBusStep`, `bus_step.go`) raises `bus-maintenance` only +while a step runs, and `bus-upgrade-failed` when one ended unhealthy. It reads only the latest bus step. +Before a step is started there is nothing for it to read, so a pending bus build raises nothing. + +**Before the merge.** `delivery.go` lists the bus among the modules whose walk starts at the merge +(`onTheControllersPath`), and the merge gate (`merge_gate.go`) composes the machines with the change. Neither +reads `pendingBus` or says that the change will hold the bus's machine until a person runs the step. So a +merge that moves the bus's build passes green and blocks every later send to the control-node. + +**Ruled out.** + +- *A wrong hold.* The bus build did change (88135ad0 → 32307bd1); the build was not the same source rebuilt + with a new digest, which issue 280 already stops from being read as a new bus. The hold was the design + working. +- *The step itself.* Started at 18:52, it answered healthy at 18:54, and the held sends went through, so + the step and its end are not part of this issue. + +**A question for the fix (a guess, not checked).** Since +[issue 295](../295-a-recorded-build-was-carried-by-a-plans-send/00-report.md), every send but a person's +push composes a recorded module at the build its machine was last sent (`recorded_kept.go`). The comment in +`sendToEach` says the bus's send is refused whole because a send "cannot leave the bus behind and carry the +rest" (ADR 0221 option 2), which was written before that. I think a send could now keep the bus at its +running build and carry the rest, so a pending bus step would stop holding every other module on the +control-node. That needs checking against the bus's user list, which [issue 249](../249-a-modules-new-state-is-refused-until-a-push-the-merge-did-not-make/00-report.md) sends first in the same +declaration, before it is proposed as a fix. It does not replace the condition: the step still waits for a +person, and the person must be told.