The operator, 2026-11-02: "A failed check shows only its first fault". Group feat/1126-removal's
check failed four times, each verdict naming one fault, so three faults took three runs of 20-40 minutes.
The checker no longer returns at a member's first failure: every member is asked, every level's
failures and every member's refused step or conflict are joined in judging order, the manifests are
judged and the composition is read also when the tests failed, and a failed composition is added after
them. The status line says how many failed and names the first, cut so "; and N more" stays whole.
checker_verdicts and checker_pull answer the whole list (failed). Needs mesh-sdk's buildstep change
(every level runs after a failure) pinned at the group's head.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
buildrun leaves mesh-sdk for the builder's own Go module (ADR 0593 §1); the
checker, assigner, indexer and watchman require it by a replace to the sibling
directory. Issue 1930's bound was already on mesh-sdk's main and moves with it.
The builder copies the imports a build module answers to the done key and the
log's end entry, where the indexer keeps them (§2; D7).
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
Review of fix/lost-check: an ask no node took yet has no step key, so
asking again then would run the check twice (issue 1791's kind). The
checker keeps the asks each running check put in a new state asks, and the
tool refuses unless every one of them has a step key. Asking again under the
same ask id is not done: the stream drops a second put of an id within its
de-duplication window and keeps both past it, so the waiting copy is never
replaced.
Tests: the hand row of the lost-ask table refuses while its ask waits;
counts unchanged.
mesh-catalog #553's verdict stayed running after its step mvsgthyjed5786
passed on shanks at 19:09:31Z: the watch of that one step key, opened at
18:46:05Z, heard neither its start nor its end and never closed, so the
wait had no bound running and nothing would ever judge it.
- HeardEnds stands between the asker and each ask's watch: at every step
end of the same build module (one watch per steps state), at a
reconnect, the asks waited on read their keys now and are judged; a
watch that closes is opened again, at most 3 times.
- A step the build module's next process ended (RestartReason, a delivery
replacing it) counts as a lost ask: asked again under the claim, at most
LostAskTries, then failed as not heard, judged again when the nodes change.
- checker_ask_again {group, why} ends a running verdict as asked again by
hand; the checker hears its verdicts, ends the wait and claims it anew.
Refused while a step of its pull requests runs, or for a verdict that
is not running.
- checker_status names such checks under lost, and each step's what.
Tests: one table replaces the given-up test (rows for given up, replaced,
deaf and nudged, deaf and asked by hand); counts unchanged (147 unit, 10
integration in the checker).
Only tries a build node holds count toward the 4 at once; tries waiting for a node are bounded
apart at 8, and the refusal names both counts (issue 1890: at 19:53 four tries queued behind 44
checks for up to 47 minutes refused every further try).
The checker wrote a try's asks only at its end, so a 31-minute try could not be told queued from
running (issue 1890). The try's record now names each ask as it is put, then the node that took
it, its state and the phase its step says, written on the step key's own changes. build_usage
names a running try's asks and takes a build module's capacity.<node> (configured bounded by its
node's build CPU, issue 1806's follow-up) when it is current with mesh-config.
mesh-grants moves to the same mesh-sdk (3203358c), so its shadow user list grants the checker and the builder the
DEAD_LETTERS reads the controller's list will, and the take-over is not blocked by a difference. The checker and the
builder hear DEAD_LETTERS through buildrun.DeadLetterHearing: in the background, again at every reconnect and every ask
waited for while deaf, and keep whether they hear it in their new state hearing, which checker_status and builder_stuck
say as dead_letters. checker_try says that a try ended by the not-taken bound leaves its ask in the stream.
Not done as asked: a module health tool. The catalogue refuses a tool check on a module with no runtime of its own and no
other check beside it (ADR 0227 rule 8, ADR 0366 §1), so the deaf state is said in the status tools and the log.
A check ask handed out five times during a bus reload was given up into DEAD_LETTERS and no build node wrote a step for it,
so try-1793017340003993788 stayed running for ever, held one of the four try places, and checker_try of its head kept
answering its id. The checker now hears the copies DEAD_LETTERS keeps of the build seats' given-up asks (an ordered
consumer from its start; the grant is mesh-sdk/mesh-controller DeadLetterReads, branch fix/734-lost-ask) and ends the
step it waits for: a try ends as error saying why, a check is asked again under its claim, up to three times, then fails
as not heard. A try whose ask no node takes within the step bound ends the same way (a check's backlog stays no fault,
ADR 0525 §9). checker_try_end ends a try by hand, freeing its place and stopping its judging.
builder_stuck, checker_kept, checker_verdicts, checker_status, checker_previews, checker_pull and checker_try listed a
bucket of thousands of keys and read each value (builder_stuck also its history) one round trip at a time before
filtering, so they timed out at 30 s. Now keys are narrowed by what the key carries first, history is read only for rows
kept, values are read sixteen at a time, at most 1000 a call with the answer saying it was cut, steps newest first, a new
try reads only the newest 64 tries, and each tool logs the values read and the time taken. Same kind as the view's fix
(PR 52); the helper is copied, not put in mesh-sdk, as a few lines do not earn a group delivery.
mesh/check checking 4ffe5f4d5243 on 85aca46348a0 | novox/mesh-delivery#25 in group feat/fastcheck
A new pull request waited 25 to 45 minutes behind every open pull request's re-judge after a
merge. The semaphore becomes a ranked admission (ADR 0536 §4); checker_status names the
waiting checks by kind.
mesh/check pass e76202876032 on b714eab995a5 | unit 422 run, 0 kept | integration 467 run, 0 kept | over bound: integration (slowest integration mesh-…
The checker's own places are released when a step ends or its wait passes the step bound; checker_status read every
'running' key in the build modules' steps buckets as a step in flight, so keys left by stopped, restarted or unassigned
build modules looked like full nodes. It now lists them under left_running with why (past the bound, module not on the
node, module not running), and leaves hq-only groups out of the unjudged candidates.
mesh-controller's try asked build-agent, which leaves the checking at R4e. A try is now a key in the checker's state
tries: the checker judges the head merged onto its trunk as it judges a pull request's head, through the same
judge, places and holds, writes its end in the key and owes the forge nothing (no status, no preview, no verdict).
One try per head, at most four at once; hq and a repository whose trunk the gate does not name are refused with
"no level of <repo> is checked by the checker".
jschoubben