novox/hq
6
0
Fork 0
Code Issues Pull Requests 5 Releases Wiki Activity

Merge pull request 'Issue 336: a send held for the bus's planned step raises no condition, so the operator is never asked' (#228) from issues/336-a-send-held-for-the-bus-step-raises-nothing into main

Browse Source
This commit was merged in pull request #228.
This commit is contained in:
mesh-admin
2026-10-09 07:23:41 +00:00
parent a5656b9cce e6f0e7e359
commit 9744ed7b9d
2 changed files with 156 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md
+96
View File
@@ -0,0 +1,96 @@
---
status: located
opened: 2026-10-08
located-in: [mesh-controller cmd/mesh-controller (push.go sendToEach and bus_step.go busHeld, the hold; signals.go S3 and bus_step.go probe DB, the conditions that do not say it; merge_gate.go and delivery.go, which do not say it before the merge)]
fixed-by:
amended-design:
---
# 336. A send held for the bus's planned step raises nothing, and the operator is never asked
## Symptom
On 2026-10-08 (times UTC) a catalogue merge, the work of
[issue 327](../327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md), built a
new build of the bus (the `nats` module) for the control-node. A new bus build is a **planned step**
([ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md),
[to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §8): only a person's
`mesh-controller.bus upgrade` moves it, and no other send reaches the bus's machine while the new build
waits.
From the merge on, every send to the control-node was held. `mesh-controller.plans` said, for example:
> tier 0: sending mesh-host to laptop, control-node, workstation after tier 0: a new bus build waits for its
> planned step: sending control-node would replace the bus (nats 88135ad0 → 32307bd1), which is a planned
> step ... — tried again for 27m45s
The same refusal held the controller's own walk and the release of builds kept back for a gate.
For those 28 minutes `mesh-controller.conditions` listed **no condition** naming the hold or saying it
waited for a person. The operator was not told. The hold was found only because a walk looked stuck, and
the operator asked: "where would you be waiting then? Where is my approve button?"
The operator then gave the word, the bus upgrade started at 18:52 and the bus answered healthy at 18:54.
The held sends then went through.
## Why it is an issue
The design makes the bus wait for a person, and that part worked: nothing replaced the bus unasked. But a
wait that only a person can end is said nowhere a person looks. The refusal is kept as the note on each
held walk, which is read only by somebody already asking `plans`. Rule 4 of
[ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md)
(no failure is silent) is met in the letter and missed in effect: the mesh held every send to its
most important machine and said nothing until asked.
A precedent exists for a wait on a person being said at once and not as lateness:
[ADR 0254](../../02-DECISIONS/0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md)
says a new login the operator owes as its own condition, `relogin-needed`. The bus's planned step has no
such condition.
The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
## Impact
- Every send to the control-node stops: the controller's own walk, the node-engine's, every module's walk
that includes the control-node, and gated builds waiting for release. Other machines in the same tier
wait with it.
- The operator learns of it only by noticing a stuck walk, which can take hours when nobody is watching.
- The only condition that could have fired, a tier past its bound (watchdog S3, "stalled"), fires no
sooner than 30 minutes in, and its words ("stuck halfway ... longer than usual. Nothing is lost") name
neither the bus nor the verb a person runs. It never fired in this case, because the step was taken
after 28 minutes.
- Nothing before the merge says that merging the change needs a bus upgrade by a person. The merge gate
and the delivery plan were green, and the merge silently blocked every later send to the control-node.
## How it is checked
| Rule | Check |
|---|---|
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a walk wants to send raises a condition on the next watchdog tick, before any tier bound |
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held walks or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a walk whose only refusal is the bus's planned step |
| it clears when the step starts | the test ends with a bus step started: the condition clears, `bus-maintenance` opens |
| the merge says it before it happens | the merge gate's test: a change that moves the bus's build reports that a bus upgrade by a person is needed and which machine waits until then |
## Core or not
**Core.** It is located in the controller, so it resolves with a replay in mesh-lab's replays register,
proved to fail on the commit before the fix and to pass on it (`replay:`), or with `replay-none:` saying
why none is possible.
## Suggested fix
1. **A condition for the wait.** While a pending bus build would move on its machine and at least one
send has been refused for it, raise a condition in the bus scope (a kind such as `bus-step-waiting`,
with the operator as its resolver). Its headline says, in plain words, what waits ("sends to the
control-node: the controller, the node-engine, …"), behind what (the bus build, from and to), since
when, and the verb that ends it (`mesh-controller.bus upgrade --why … --reversible|--irreversible`). It
is raised on the first refusal, not after a bound. A walk held only by it is excluded from S3, as a
walk held by a paused build seat is.
2. **An ask, once asks exist.** When the decision flow of ADR 0259 (an ask is a message on the bus, any
channel answers it; on an open pull request, not yet merged) is built, the condition becomes an ask
to the operator: "start the bus upgrade now?", whose yes runs the step with the operator's word
recorded as the hand act.
3. **Said before the merge.** The merge gate, or the delivery plan, says when a change moves the bus's
build: "merging this needs a bus upgrade by a person; until then nothing reaches the control-node."
Whether that is a warning or a hold for the operator's word is for the fix's decision.
04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md
+60
View File
@@ -0,0 +1,60 @@
# 336 — Diagnosis
## 2026-10-08
**The records first.** `records_search` for "a new bus build waits for its planned step" found nothing.
A search for "planned step" found the decisions and designs that define the step (ADR 0221, 0227, 0236,
to-be 45 §8) and issue 280, none of which says how a wait for it is told to a person.
**Where the words come from.** In mesh-controller, `errBusWaits` in `bus_step.go` is "a new bus build waits
for its planned step". `sendToEach` in `push.go` returns it, wrapped with `busHeld`'s reason, for any send
not marked as the bus step whose machine list includes a machine where the pending bus build would move.
The send is refused whole. `busHeld` (`bus_step.go`) gives the words "sending X would replace the bus
(nats from → to), which is a planned step: `bus upgrade --why …` …". So the remedy is in the words, but
only in the error.
**Where the error goes.** In `release_plan.go`, the walk's step loop catches the error, keeps it as the
plan's note ("tier N: … — tried again"), saves the plan when the note changed, prints it to the
controller's log once, and tries again on the next tick. Nothing raises a condition from it. That is why
`plans` showed it and `conditions` did not.
**Which watchdog could have said it.** Only S3 (`watchPlans`, `signals.go`), "plan tier progress", raises
`plan.<id>.stalled` when a walk has been in one tier past its bound. The bound is
`max(30 min, 3 × the p90 of that repository's measured tiers)` (`tierAtLeast = planWaitBound = 30 min`,
`watchdogs.go` `boundsOf`). So:
- it fires no sooner than 30 minutes after the tier was entered; the hold here was ended by hand at about
28 minutes, so it never fired;
- its words are generic ("Delivery of X is stuck halfway … longer than usual. Nothing is lost, and the
mesh keeps it where it is."). Its summary carries the plan's note, so the bus's reason would have
appeared there, but the headline a person reads names neither the bus nor the verb, and its resolver is
not the operator;
- a walk held by a paused build seat is excluded from S3 (`pausedWaiting`), and a walk waiting for its
delivery's word has its own watchdog (S16). A walk held for the bus step has neither.
**What the bus step does raise.** Probe DB (`probeBusStep`, `bus_step.go`) raises `bus-maintenance` only
while a step runs, and `bus-upgrade-failed` when one ended unhealthy. It reads only the latest bus step.
Before a step is started there is nothing for it to read, so a pending bus build raises nothing.
**Before the merge.** `delivery.go` lists the bus among the modules whose walk starts at the merge
(`onTheControllersPath`), and the merge gate (`merge_gate.go`) composes the machines with the change. Neither
reads `pendingBus` or says that the change will hold the bus's machine until a person runs the step. So a
merge that moves the bus's build passes green and blocks every later send to the control-node.
**Ruled out.**
- *A wrong hold.* The bus build did change (88135ad0 → 32307bd1); the build was not the same source rebuilt
with a new digest, which issue 280 already stops from being read as a new bus. The hold was the design
working.
- *The step itself.* Started at 18:52, it answered healthy at 18:54, and the held sends went through, so
the step and its end are not part of this issue.
**A question for the fix (a guess, not checked).** Since
[issue 295](../295-a-recorded-build-was-carried-by-a-plans-send/00-report.md), every send but a person's
push composes a recorded module at the build its machine was last sent (`recorded_kept.go`). The comment in
`sendToEach` says the bus's send is refused whole because a send "cannot leave the bus behind and carry the
rest" (ADR 0221 option 2), which was written before that. I think a send could now keep the bus at its
running build and carry the rest, so a pending bus step would stop holding every other module on the
control-node. That needs checking against the bus's user list, which [issue 249](../249-a-modules-new-state-is-refused-until-a-push-the-merge-did-not-make/00-report.md) sends first in the same
declaration, before it is proposed as a fix. It does not replace the condition: the step still waits for a
person, and the person must be told.