The check of 2026-11-02 22:35 on a loaded shanks failed two buildrun tests on wall-clock bounds of 60 ms, the
second failure of this kind tonight (the first: mesh-sdk's busretry tests, f02f142).
Quick fix: give the two buildrun asker tests bounds of seconds.
Thorough fix (chosen, because the kind recurs across every module of the repository): every test that raced a
wall clock now waits on an event, steps a clock it is given, or has a bound only a failing test reaches.
- buildrun asker tests: the steps' own times stand for the clock (bound an hour, starts and phases written in the
past), no goroutine paces phases with sleeps.
- builder: the decline bound reads Builder.Now; the decline test steps it; the stream-wait test waits for five
check reads instead of 100 ms.
- checker: the lost-ask try bounds an untaken ask at 200 ms only where that is what ends it; the program says when a
merge turn changes, and the integration test waits on that line instead of 300 ms.
- teller: the bed makes the teller's own ends bucket (a 10 s stall at start the tests raced by 0.5 s), waits for its
warrant subscription instead of 500 ms pauses, and the package-level now is restored with t.Cleanup.
- watchman: the recount, run in its own goroutine, is waited for instead of 10 ms.
- polls of 2-5 s become 10 s (unit) and 20 s (integration).
Under a throttled cgroup (half a core shared with two busy loops) beside the node-engine's suite, the buildrun
tests failed 5 of 20 runs before and passed 30 of 30 after; no test added.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
buildrun leaves mesh-sdk for the builder's own Go module (ADR 0593 §1); the
checker, assigner, indexer and watchman require it by a replace to the sibling
directory. Issue 1930's bound was already on mesh-sdk's main and moves with it.
The builder copies the imports a build module answers to the done key and the
log's end entry, where the indexer keeps them (§2; D7).
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check checking e7888024a826 on 769815ac0fea | novox/mesh-delivery#103 in group fix/lost-check
Re-review of fix/lost-check: writing an empty list at the end kept one key
for every verdict key ever checked. Tests: every row of the lost-ask table
proves the key gone at the end; one row more (deaf after its build module
replaced it, asked again by hand); counts unchanged.
Review of fix/lost-check: an ask no node took yet has no step key, so
asking again then would run the check twice (issue 1791's kind). The
checker keeps the asks each running check put in a new state asks, and the
tool refuses unless every one of them has a step key. Asking again under the
same ask id is not done: the stream drops a second put of an id within its
de-duplication window and keeps both past it, so the waiting copy is never
replaced.
Tests: the hand row of the lost-ask table refuses while its ask waits;
counts unchanged.
The checker's HeardEnds moved to mesh-sdk buildrun (fix/lost-check,
6aeb401): both modules pin it, and the builder, which waits on the same
Asker, now reads an ask's step at each step end of its build module too.
checker_ask_again and checker_status's lost stay in the checker.
Tests: one row in the builder's ask table (its watch deaf, read at a
nudge, done with no new ask); counts unchanged.
mesh-catalog #553's verdict stayed running after its step mvsgthyjed5786
passed on shanks at 19:09:31Z: the watch of that one step key, opened at
18:46:05Z, heard neither its start nor its end and never closed, so the
wait had no bound running and nothing would ever judge it.
- HeardEnds stands between the asker and each ask's watch: at every step
end of the same build module (one watch per steps state), at a
reconnect, the asks waited on read their keys now and are judged; a
watch that closes is opened again, at most 3 times.
- A step the build module's next process ended (RestartReason, a delivery
replacing it) counts as a lost ask: asked again under the claim, at most
LostAskTries, then failed as not heard, judged again when the nodes change.
- checker_ask_again {group, why} ends a running verdict as asked again by
hand; the checker hears its verdicts, ends the wait and claims it anew.
Refused while a step of its pull requests runs, or for a verdict that
is not running.
- checker_status names such checks under lost, and each step's what.
Tests: one table replaces the given-up test (rows for given up, replaced,
deaf and nudged, deaf and asked by hand); counts unchanged (147 unit, 10
integration in the checker).
The waiter moved to mesh-sdk buildrun (fix/734-lost-ask, d96b9fd6), where every asker of the build seats shares it.
The builder hears DEAD_LETTERS for the go, node and image seats and asks a given-up part of a build again at once, with
the seat's words as why, instead of waiting for the check bound. The checker keeps what it does with a lost ask (a try
ends as error, a check is asked again) and its verdict words.
A check ask handed out five times during a bus reload was given up into DEAD_LETTERS and no build node wrote a step for it,
so try-1793017340003993788 stayed running for ever, held one of the four try places, and checker_try of its head kept
answering its id. The checker now hears the copies DEAD_LETTERS keeps of the build seats' given-up asks (an ordered
consumer from its start; the grant is mesh-sdk/mesh-controller DeadLetterReads, branch fix/734-lost-ask) and ends the
step it waits for: a try ends as error saying why, a check is asked again under its claim, up to three times, then fails
as not heard. A try whose ask no node takes within the step bound ends the same way (a check's backlog stays no fault,
ADR 0525 §9). checker_try_end ends a try by hand, freeing its place and stopping its judging.
jschoubben