mesh/check pass 41f25d3129fd on 73be3df28cd4 | unit 21 run, 579 kept | integration 21 run, 655 kept | composition passed on shanks
On 2026-10-25 node_haves and assigner_wants timed out at 30 s four times, even narrowed to one node and one module,
while assigner_want answered at once. Each listed every key of its bucket and then read every value one after the
other, filtering only afterwards; every value is a round trip from the tool through the runtime to the bus, which
opens the bucket and gets the key, so a few hundred keys outlasted the call. Now a node and a module read that one
key directly, a node or a module alone filters the keys before any value is read, and the values are read sixteen
at a time, at most 1000 a call, the answer saying when it was cut. indexer_modules, indexer_candidates and
builder_builds read their values the same parallel way. The kind: a read tool that reads a whole bucket key by key;
every list tool of the view is covered.
mesh/check pass 4628ba26b41a on 73be3df28cd4 | unit 167 run, 443 kept | integration 177 run, 509 kept | composition passed on 3 machines
The open helpers were copied into seven parts; code shared between
modules lives in mesh-sdk, so they are mesh-sdk process's now and the
parts are pinned at fix/1945-optional-buckets (e0d1de04), moved to its
main once it merges, one delivery group with mesh-sdk.
Met twice on 2026-10-25: the builder of #48 exited on a refused read of
the gate's source and crash-looped, and its new reasks bucket would have
done the same. A grant reaches the bus with a delivery, so a refusal or a
timeout at start is often only a grant not there yet; exiting for it
stopped the whole part until a hand step on the bus.
Every startup open in the parts is now essential or optional. Essential
ones (a part's own state and the input it exists for) still stop the
start, with an error naming the bucket and its grant. Optional ones are
said and left: watches the bus refuses too. The builder builds trunk keys
without the gate's source, which it opens again at the next candidate
that asks a trunk head; toolchains and previews open again at their next
event, as a missing bucket did. builder_ask_again says that asking a
running step again builds it twice.
A new assignment whose trunk build was not done fell through the rules
with nothing said: no want, no neither-list entry, no log line, and the
rule holding it back was computed and dropped. The pass now logs the rule
when a node's reason appears or changes and when it has a want again, one
line per change. A node whose want is removed and that does not run the
module gets the reason in that removed want's words, so assigner_want
answers why; a node with no key gets no removed want, since its node agent
would then remove the module and stop holding its resources back.
A build ask its seat's stream gave up on, dropped or ended with no step was
waited for with no end, and on restart taken over and waited for again: no
build node would ever answer it, and nothing could ask it again (trunk key
of mesh-build-node at 4e7b118e, building since 19:06; seven keys at 18:44).
At the step bound, a context's bound and no timer of its own, the builder
reads what the bus says of an ask with no step: gone from its stream, or
first in line while a node took a later ask. Such an ask is asked again
under a new id the key names, logged and said as build.asked-again, at most
three askings, then the key fails with why. The bus's max-deliveries
advisory would tell it sooner, but the builder is granted neither it nor the
workers' information, and the controller is not changed here.
builder_stuck reads every waiting key with its asks, askings and steps;
builder_ask_again writes the new reasks state, which the builder watches
and answers.
mesh/check failed: integration: mesh-delivery-assigner/cmd/mesh-delivery-assigner.TestTheAssignerStartsAndSaysWhatItKeeps: the assigner said {V:1 Modu…
mesh/check pass ac82187b852c on 82180bffd2c6 | unit 167 run, 432 kept | integration 177 run, 498 kept | composition passed on 2 machines
The test predates the fix that keeps a held fingerprint while builders disagree (#47, 3e7f852) and still asserted the
blanking that fix removed, so it fails on main 4efe6cf as well. It now asserts the fingerprint is kept after n2's
assignment, and that a merge moving avahi during the disagreement gets none until the builders agree.
mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…
Candidate.Base was never written non-zero (review round 2): the builder's base lookup would have found no trunk for any
live candidate and built no preview at all. The indexer now writes, whenever it writes a candidate's fingerprints, the gate's
source revision whose trunk heads it read the previews onto; that head of the pull request's repository is the preview's
trunk. No mesh-sdk change: the field and its meaning were there. The builder's test now takes the candidate the indexer's
own test writes (testdata/candidate-written.json, kept equal to it), not a hand-made one.
The builder took a candidate build's merge base from the module's value commit. Now that a value keeps the commit its
fingerprint first appeared at, that commit can be many merges behind the trunk, so a preview would be built onto a stale
trunk (review round 1). The base is now the head of the pull request's repository in the gate's source revision the
candidate names (Candidate.Base): the builder reads mesh-delivery-gate.source, watched with its history (a direct get of
one revision is not what a read grants), and a candidate heard before its revision is built once the revision is heard.
The operator, 2026-10-25: only if a module has a new version in commit x is a new value written naming commit x. A merge
wrote every module of the repository at the new commit, fingerprint unchanged (about 186 keys per mesh-catalog merge), and
the controller's take-in took each in again: twice tonight it flooded. Now a merge that gives a module the value it holds
but for the commit writes nothing, and the value keeps the commit its fingerprint first appeared at. The kind: a write
that differs only in where it was computed. The other instance, a source revision re-stamping Src on every module at its
repository's head after any merge anywhere, now stamps Src once. The trunk head stays the gate's source.main.
mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…
On 2026-10-25 20:56 assigning mesh-build-node to ace again brought ace's report of 2026-10-23 (v26.12.0) back into the
agreement before the module ran there; the indexer blanked every module's fingerprint to "builders disagree" and the
controller's take-in refused modules already computed and built.
- The toolchain gate (indexer, builder, checker; still one file in three) leaves a report out, said once, until its node
writes it since its current assignment: the assignment's own time, or the revision that opened it.
- The indexer keeps a module's held fingerprint while a language has no agreed toolchain, when its kept inputs with the
toolchains it was computed with give it again; only a new fingerprint (a merge, a base, an agreed toolchain moved)
needs the agreement.
mesh/delivery-group group fix/1945-credential-when-row-without-seal held: a member is held: novox/mesh-controller@fbf14b68dbab (merged without a passing check:…
mesh/check pass 61196c99e6a4 on 5ba6881f9fd3 | unit 163 run, 421 kept | integration 173 run, 487 kept | composition passed on ace
Rule 6 read only whether a grant of the module's fingerprint existed. A grant
written for the module's other nodes then let a node whose account was never
issued or sealed be wanted: on 2026-10-25 at 20:04 ace's mesh-build-go was
wanted with no ace.mesh-build-go on the bus, and crash-looped 20 times for
its missing credential. A grant that names accounts now grants a node only
when it names that node's own or a process's account; one that names none
grants every node, as before.
Research 072: the grants piece (memberships and credentials heard from placement).
mesh/delivery-group group fix/1945-kept-compositions-follow-config held: a member is held: novox/mesh-controller@ff3cc47027e0 (merged without a passing check: …
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestTheDeclarationsStreamGoesOnceEveryNodeRunsTheNewEngine: with …
The controller now composes a module's kept compositions again when its config
changes, so going back keeps today's config. A want held back at an earlier
fingerprint kept the digest it was first written with, and the node applied
the old revision from the bucket's history: on g14 a rollback of mesh-build-go
put its capacity back to 0. The want now names the new digest, asked through
the record hold as any move of a record module is. A want held at F by the
rules still waits for them, as a new fingerprint does (ADR 0539 §5).
Research 072: milestone 2, the composer reacting to the change of an input.
mesh/check pass 91d041b62a1a on ec76ab77c058 | unit 163 run, 421 kept | integration 173 run, 487 kept | over bound: integration (slowest integration m…
systemd-resolved, held on shanks by the operator under rule 2, was raised
as an urgent "on no list" condition and reminded every hour, although its
hold is already said and answered on the held list. A node now goes on
neither list only when it gets no want and the module is not held, by a
hold read at its fingerprint or one this pass raises. The first pass after
a start clears the neither condition of every assigned node not on neither
list, not only those with a want, so a neither raised before the restart
for a held module (or a move waiting for its record ask) is cleared too.
mesh/delivery-group group fix/1901-package-cause held: a member is held: novox/mesh-catalog@155124b59af4 (merged without a passing check: only a person decides…
mesh/check pass dd9138310c48 on 51444a834173 | unit 2015 run, 2883 kept, 5 skipped | integration 2037 run, 2978 kept, 9 skipped | over bound: unit, in…
The build module on main sends a timeout, exit or kill as only 'FAIL <pkg>':
dropping summary lines left empty words. They now say the package failed with
no cause sent. Causes match at column 0 only, summary lines only in go's exact
forms, and the words count the cause, the marker and the newlines within
MaxWords.
An older build module's output for a race after the tests' PASS is now worded
by the race's heading, access and frame, not by go test's PASS or the report's
separator line (1901/AC5).
mesh/check pass 975f0ed65460 on 653da54c0da3 | unit 154 run, 419 kept | integration 164 run, 485 kept | composition passed on ace
Phase guard, same kind as the failed-health hold. allRunHealthy counted
every have whatever its phase, so a node with a removed, held or still
moving have kept a hold up for ever: it counts only phase running (healthy
or not) and failed (not healthy). The failed-health hold already ignores
removed and held haves (phase running or failed only).
Stale conditions (B1). The neither and held conditions this process said
lived only in its memory, so one raised by a previous process stayed urgent
for ever once the module had its want or was no longer held. The first pass
after a start clears, once, the neither condition of every assigned module
with a want and not on neither list, and the held condition of every
module's newest fingerprint that is not held; a clear of a condition that is
not open is quiet at the router. A hold raised again forgets that it was
cleared, so a later release clears it again.
Record ask (review note 3). mustAsk wrote no ask when the want before was
removed, so the first-want path could write a recreates or moves-data
composition of a policy-record module with no ask. A removed want is now
no want, and the first want after it goes through the record ask.
Tests first, each seen failing: a hold lifted over removed, held and moving
haves; the record ask after a removed want; a fresh pass clearing what a
previous process raised, once.
On novox at 15:27:36 a have reading "health failed, phase applying" at the
controller's new fingerprint held its canary two seconds before the new
build was healthy: Wants held on a failed health whatever the phase. A
failed health while the node backs up, applies or checks is a step of the
move, not its outcome, so it holds nothing now.
Every other Health read in the assigner (rule 3's healthy, rule 5's
providedHealthy, the world's lastHealthy) already asks for phase running.
Test from the live row first, seen failing before the fix.
A try of mesh-controller's status read 'cmd/mesh-controller: PASS': the build
module sent a package failure whose output began with go test's PASS. The
checker now drops go test's summary lines, takes the first panic, fatal error
or signal line as the cause when the build module told none apart (read before
the words are cut), says the tests passed and the program failed after them
when nothing else is left, and cuts a cause to 1 KB so the words keep their
bound (1901/AC5).
An old-form build (not src2:) has no composition, so no want can ever name
what such a node runs. Design limit, decided by the lead (lead-47): its first
want names the newest F, by every rule as usual - the canary (rule 3: this
node is the canary, or the canary runs F healthy), everything F needs (built,
not held, granted, composed) and the providers (rule 5). Otherwise nothing is
written and the neither condition stays. The canary stays in force, and
old-form modules (clipmenu and screen-lock on g14) leave neither list by
themselves, without a hand-written want.
The code already behaved so (an old-form have falls through to Holds); the
case is now named in Wants, and proven: a unit test for both sides of the
canary, the providers and the grant, and the integration test gives g14 F
and clears its neither condition once the canary ace runs F healthy.
A first want at the newest fingerprint passed over the canary for every module right after the switch. It now names what the node
runs: F when it runs F (by the rules without which there is nothing to want), or an older fingerprint of the new form whose
composition and grant exist, held back from F by the rule; otherwise nothing is written and the node stays on neither list. A new
assignment, with no have, waits for the rules and raises no condition (1945/AC19-AC21).
The node's own composition gives up every module with a fingerprint of the new form, so such a module is applied only through its
want. The assigner wrote no want while a rule held it back and the node had none, so clipmenu, screen-lock, i3status-rust and every
module requiring x11-display lay on neither list on g14 and shanks: xorg's health is unknown (rule 5) and the canary never ran them
healthy (rule 3), and their files were removed and never made again.
A node that runs a module and has no want for it now gets its first want by the rules without which there is nothing to want (built,
not held, granted, composed); the canary and providers rules order every move after it. Any module still on neither list is raised
as an urgent condition naming module, node and rule, written to the journal, and cleared once it has a want (1945/AC19-AC21).
A hold answered keep on the phone could not be changed: its ask ended with the
answer, nothing listed it, and a kept hold is never lifted by itself. The view
lists every hold with its node, cause and answer (mesh-delivery-cli holds) and
answers the standing hold ask with hold-release or hold-keep through its one
write; the teller takes a later view answer that changes a hold's option into
the answer of that asking, never an older one.
A hold written again after a lift and a restart reused its first ask id, which
the router drops; the hold's time is now its asking. Seven holds could take the
router's three places for the teller and keep a merge Approve from being asked;
the teller keeps at most two assigner asks open. A release or keep answered for
one hold carried over to a later hold of the same fingerprint; an answer now
counts only for the asking its warrant names.
Seven modules stayed held on every machine and the operator was never asked:
the teller only filled assigner asks from the view, a single failed check held
a fingerprint for good, and the held record kept no cause. The teller now asks
every assigner ask as it asks merge asks and writes the phone's answer under
the ask's key; the assigner lifts a hold whose fingerprint runs healthy on
every node that runs the module, says so once and takes its ask back; the held
record names health, phase, when and the node's words.
jschoubben
mesh-tools