A delivery restarted the checker and the builder before the grants part created their new state hearing;
OptionalBucket answered nil and the hearing was never kept. They, and the builder's reasks and queues and the
indexer's takeins, now open their own states through mesh-sdk process.AwaitBucket (mesh-sdk
fix/1790-state-appears d515c71), which binds on the bus's STREAM.CREATED advisory and writes the latest kept
value. mesh-grants takes the same SDK, whose composer grants that advisory.
The operator's question (how much are the build nodes used, is ace eligible) took six calls over four tools.
build_usage reads it from the bus's states: seats held (have), capacity (mesh-config resolved), pauses, the
build modules' steps by kind with left-running keys, the last hour and day with total and median time and time
at capacity, the running tries, and the backlog the view program reads from the seats' streams. A try now names
the asks it put, and the checker announces its tries readable, so the view tells a try's asks from a check's.
mesh-grants moves to the same mesh-sdk (3203358c), so its shadow user list grants the checker and the builder the
DEAD_LETTERS reads the controller's list will, and the take-over is not blocked by a difference. The checker and the
builder hear DEAD_LETTERS through buildrun.DeadLetterHearing: in the background, again at every reconnect and every ask
waited for while deaf, and keep whether they hear it in their new state hearing, which checker_status and builder_stuck
say as dead_letters. checker_try says that a try ended by the not-taken bound leaves its ask in the stream.
Not done as asked: a module health tool. The catalogue refuses a tool check on a module with no runtime of its own and no
other check beside it (ADR 0227 rule 8, ADR 0366 §1), so the deaf state is said in the status tools and the log.
The waiter moved to mesh-sdk buildrun (fix/734-lost-ask, d96b9fd6), where every asker of the build seats shares it.
The builder hears DEAD_LETTERS for the go, node and image seats and asks a given-up part of a build again at once, with
the seat's words as why, instead of waiting for the check bound. The checker keeps what it does with a lost ask (a try
ends as error, a check is asked again) and its verdict words.
A check ask handed out five times during a bus reload was given up into DEAD_LETTERS and no build node wrote a step for it,
so try-1793017340003993788 stayed running for ever, held one of the four try places, and checker_try of its head kept
answering its id. The checker now hears the copies DEAD_LETTERS keeps of the build seats' given-up asks (an ordered
consumer from its start; the grant is mesh-sdk/mesh-controller DeadLetterReads, branch fix/734-lost-ask) and ends the
step it waits for: a try ends as error saying why, a check is asked again under its claim, up to three times, then fails
as not heard. A try whose ask no node takes within the step bound ends the same way (a check's backlog stays no fault,
ADR 0525 §9). checker_try_end ends a try by hand, freeing its place and stopping its judging.
A check that landed on ace passed with mesh-build-node's integration tests skipped. The build module now leaves such a
package unmet; the checker keeps the passes and asks the levels from the first unmet one with the need named, putting the
ask again under a new id when a node lacking it declines it. Re-pins mesh-sdk feat/890-placement.
mesh/check failed: ace: the check of novox/mesh-catalog#492 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
mesh/check failed: ace: the check of novox/mesh-delivery#65 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
The checker stopped reading a level's test results at the first that did
not pass and kept only its first failure, so a check named one fault per
round. It now reads and counts every result and writes every failed test
with its first line into the verdict's Failures and the try's failed
list (at most 50, then how many more); Failed keeps the first one's
words whole (issue 1901). Pins mesh-sdk fix/1921-every-failure.
builder_stuck, checker_kept, checker_verdicts, checker_status, checker_previews, checker_pull and checker_try listed a
bucket of thousands of keys and read each value (builder_stuck also its history) one round trip at a time before
filtering, so they timed out at 30 s. Now keys are narrowed by what the key carries first, history is read only for rows
kept, values are read sixteen at a time, at most 1000 a call with the answer saying it was cut, steps newest first, a new
try reads only the newest 64 tries, and each tool logs the values read and the time taken. Same kind as the view's fix
(PR 52); the helper is copied, not put in mesh-sdk, as a few lines do not earn a group delivery.
mesh/check pass 4628ba26b41a on 73be3df28cd4 | unit 167 run, 443 kept | integration 177 run, 509 kept | composition passed on 3 machines
The open helpers were copied into seven parts; code shared between
modules lives in mesh-sdk, so they are mesh-sdk process's now and the
parts are pinned at fix/1945-optional-buckets (e0d1de04), moved to its
main once it merges, one delivery group with mesh-sdk.
Met twice on 2026-10-25: the builder of #48 exited on a refused read of
the gate's source and crash-looped, and its new reasks bucket would have
done the same. A grant reaches the bus with a delivery, so a refusal or a
timeout at start is often only a grant not there yet; exiting for it
stopped the whole part until a hand step on the bus.
Every startup open in the parts is now essential or optional. Essential
ones (a part's own state and the input it exists for) still stop the
start, with an error naming the bucket and its grant. Optional ones are
said and left: watches the bus refuses too. The builder builds trunk keys
without the gate's source, which it opens again at the next candidate
that asks a trunk head; toolchains and previews open again at their next
event, as a missing bucket did. builder_ask_again says that asking a
running step again builds it twice.
The operator, 2026-10-25: only if a module has a new version in commit x is a new value written naming commit x. A merge
wrote every module of the repository at the new commit, fingerprint unchanged (about 186 keys per mesh-catalog merge), and
the controller's take-in took each in again: twice tonight it flooded. Now a merge that gives a module the value it holds
but for the commit writes nothing, and the value keeps the commit its fingerprint first appeared at. The kind: a write
that differs only in where it was computed. The other instance, a source revision re-stamping Src on every module at its
repository's head after any merge anywhere, now stamps Src once. The trunk head stays the gate's source.main.
mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…
On 2026-10-25 20:56 assigning mesh-build-node to ace again brought ace's report of 2026-10-23 (v26.12.0) back into the
agreement before the module ran there; the indexer blanked every module's fingerprint to "builders disagree" and the
controller's take-in refused modules already computed and built.
- The toolchain gate (indexer, builder, checker; still one file in three) leaves a report out, said once, until its node
writes it since its current assignment: the assignment's own time, or the revision that opened it.
- The indexer keeps a module's held fingerprint while a language has no agreed toolchain, when its kept inputs with the
toolchains it was computed with give it again; only a new fingerprint (a merge, a base, an agreed toolchain moved)
needs the agreement.
mesh/delivery-group group fix/1901-package-cause held: a member is held: novox/mesh-catalog@155124b59af4 (merged without a passing check: only a person decides…
mesh/check pass dd9138310c48 on 51444a834173 | unit 2015 run, 2883 kept, 5 skipped | integration 2037 run, 2978 kept, 9 skipped | over bound: unit, in…
The build module on main sends a timeout, exit or kill as only 'FAIL <pkg>':
dropping summary lines left empty words. They now say the package failed with
no cause sent. Causes match at column 0 only, summary lines only in go's exact
forms, and the words count the cause, the marker and the newlines within
MaxWords.
An older build module's output for a race after the tests' PASS is now worded
by the race's heading, access and frame, not by go test's PASS or the report's
separator line (1901/AC5).
A try of mesh-controller's status read 'cmd/mesh-controller: PASS': the build
module sent a package failure whose output began with go test's PASS. The
checker now drops go test's summary lines, takes the first panic, fatal error
or signal line as the cause when the build module told none apart (read before
the words are cut), says the tests passed and the program failed after them
when nothing else is left, and cuts a cause to 1 KB so the words keep their
bound (1901/AC5).
The assigner reads mesh-controller.builds again (the composer's runningCommits reads the build record of every open
assignment), puts its compose asks on node-build-go and says its conditions on operator-channel.
mesh-build-node was unassigned from ace on 2026-10-23 and its ace.typescript key stayed in
mesh-build-node_toolchains at v26.12.0: nothing deletes it, so the indexer, the builder and the
checker took it for a builder and every Node fingerprint said 'builders disagree'. The three now
read mesh-controller_assignments and pass a toolchain report on only while its build module has an
open assignment on its node; builders that run different versions still disagree.
jschoubben
mesh-tools