mesh/check checking f007ca42279a on 4b36cc29a6ae | novox/mesh-delivery#81 in group fix/1496-start-health
Review of 50c4674: the bar on new wants naming F while a node fails it alone
also caught a node that runs F with no live want, the issue-1945 recovery path:
it went on neither list and its node-engine gave the module up. The bar now
holds only a node that neither wants nor runs F.
ADR 0557 version 9: a failure that is one node's holds nothing and sets nothing
back, but the version reaches no node that does not want it yet until it is
over. A node whose want does not name F gets no new want naming it (rule 2, in
words naming the failing nodes); wants already written stay.
Review round 2 of 16e4e61: gating the first-pass clear on a failed have left a
condition an earlier process raised standing for good once its node recovered
while no assigner watched. The first pass now clears it for every module and
every node the presence bucket knows, a quiet clear at the router for one that
is not open.
Review of 2ee7589: a node that sleeps after running F healthy kept its stale
have as a witness, so F stayed unheld while it failed on the online nodes; and a
failed-alone condition was never cleared once its node left the module.
A witness must now be online by the watchman's presence. F failing on a second
node is no longer one node's failure: the hold and the set-back stand (ADR 0557
version 6, §2). A remembered failed-alone key whose node is no longer one of the
module's nodes is cleared once; the first pass after a start clears it for every
node the presence bucket knows, for a module with a failed have.
On ace, 2026-10-26 21:19, a newly assigned mesh-build-node failed its first
health check at 8a98386f (later 31be2f24) while shanks, g14 and novox had run it
healthy for an hour. Being first by name, ace was the canary, and the assigner
held the version and moved the three healthy nodes back.
A failed health check of the newest fingerprint now holds it only when no other
node's have names it healthy (ADR 0557 §2 keeps ADR 0429 §6's hold and set-back
for that case). Otherwise the failure is that node's (ADR 0557 §1): nothing is
held or set back, the node keeps its want, and the assigner raises the node's
condition once (failed-alone.<node>.<module>), naming the node, the module, the
fingerprint and the have's words, and clears it once when it ends; the first
pass after a start clears what a previous process may have raised.
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 2e0013ec2500 on df8b9ac3b1b5 | unit 1245…
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 2493aa65299e on df8b9ac3b1b5 | unit 1245…
One commit for the branch feat/1126-nodes-compose of mesh-delivery: its 16 commits squashed so its history does not carry the 17.8 MB compiled mesh-delivery-assigner a commit added by mistake and a later one removed (issue 1577). The tree equals the former head ff6c038d. The operator allowed this rewrite of the feature branch on 2026-10-26.
Review round 1: while an operator's put runs through its steps it owns the want, so the clear said
'the node does not run it' of a node that runs the bus.
A bus whose move only the operator's put may make (ADR 0546 §8) got no want and nothing in the outcome:
nats on novox, running an old-form build, was on no list while the pass cleared its condition and logged
that it had a want again. The bar now names its holding, puts a node without a live want on neither list
(its condition needs the operator: assigner_put), and a neither condition is cleared as 'has a want again'
only when a live want exists.
mesh/check pass 769dab43a24d on 2f697216eee4 | unit 168 run, 488 kept | integration 178 run, 554 kept | composition passed on ace
The held record is the memory that a fingerprint failed (ADR 0408 section 4); a revert returns the
trunk to exactly it, and without the key rule 3 would move the canary onto the failed build with
nothing asked. A stale hold ask now clears only the condition (raised again on a revisit), and the
hold is asked again under a new asking. Also read the single want key for a record ask.
Review round 1: a record ask is about the composition of any fingerprint a node may still want
(trunk, want, have), not only the trunk's, so deleting it looped with the wants rule that asks it
again and the operator was never asked. It is now stale only when the module left the trunk or no
read composition of those fingerprints has its digest. A hold or restore ask taken away as stale
also takes its held record and clears its held condition.
An ask outlived its fingerprint: a lift left the restore asks of its fingerprint on every node,
and no pass removed an ask of a fingerprint that is no longer the module's trunk fingerprint,
so the operator kept being asked about a module that runs the trunk healthy. Each pass now
takes those asks away (the teller then withdraws the router ask), and a lift takes the restore
asks of its fingerprint. Nothing is taken away while the trunk or a composition is unread.
mesh/check pass 4628ba26b41a on 73be3df28cd4 | unit 167 run, 443 kept | integration 177 run, 509 kept | composition passed on 3 machines
The open helpers were copied into seven parts; code shared between
modules lives in mesh-sdk, so they are mesh-sdk process's now and the
parts are pinned at fix/1945-optional-buckets (e0d1de04), moved to its
main once it merges, one delivery group with mesh-sdk.
Met twice on 2026-10-25: the builder of #48 exited on a refused read of
the gate's source and crash-looped, and its new reasks bucket would have
done the same. A grant reaches the bus with a delivery, so a refusal or a
timeout at start is often only a grant not there yet; exiting for it
stopped the whole part until a hand step on the bus.
Every startup open in the parts is now essential or optional. Essential
ones (a part's own state and the input it exists for) still stop the
start, with an error naming the bucket and its grant. Optional ones are
said and left: watches the bus refuses too. The builder builds trunk keys
without the gate's source, which it opens again at the next candidate
that asks a trunk head; toolchains and previews open again at their next
event, as a missing bucket did. builder_ask_again says that asking a
running step again builds it twice.
A new assignment whose trunk build was not done fell through the rules
with nothing said: no want, no neither-list entry, no log line, and the
rule holding it back was computed and dropped. The pass now logs the rule
when a node's reason appears or changes and when it has a want again, one
line per change. A node whose want is removed and that does not run the
module gets the reason in that removed want's words, so assigner_want
answers why; a node with no key gets no removed want, since its node agent
would then remove the module and stop holding its resources back.
mesh/check failed: integration: mesh-delivery-assigner/cmd/mesh-delivery-assigner.TestTheAssignerStartsAndSaysWhatItKeeps: the assigner said {V:1 Modu…
mesh/delivery-group group fix/1945-credential-when-row-without-seal held: a member is held: novox/mesh-controller@fbf14b68dbab (merged without a passing check:…
mesh/check pass 61196c99e6a4 on 5ba6881f9fd3 | unit 163 run, 421 kept | integration 173 run, 487 kept | composition passed on ace
Rule 6 read only whether a grant of the module's fingerprint existed. A grant
written for the module's other nodes then let a node whose account was never
issued or sealed be wanted: on 2026-10-25 at 20:04 ace's mesh-build-go was
wanted with no ace.mesh-build-go on the bus, and crash-looped 20 times for
its missing credential. A grant that names accounts now grants a node only
when it names that node's own or a process's account; one that names none
grants every node, as before.
Research 072: the grants piece (memberships and credentials heard from placement).
mesh/delivery-group group fix/1945-kept-compositions-follow-config held: a member is held: novox/mesh-controller@ff3cc47027e0 (merged without a passing check: …
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestTheDeclarationsStreamGoesOnceEveryNodeRunsTheNewEngine: with …
The controller now composes a module's kept compositions again when its config
changes, so going back keeps today's config. A want held back at an earlier
fingerprint kept the digest it was first written with, and the node applied
the old revision from the bucket's history: on g14 a rollback of mesh-build-go
put its capacity back to 0. The want now names the new digest, asked through
the record hold as any move of a record module is. A want held at F by the
rules still waits for them, as a new fingerprint does (ADR 0539 §5).
Research 072: milestone 2, the composer reacting to the change of an input.
mesh/check pass 91d041b62a1a on ec76ab77c058 | unit 163 run, 421 kept | integration 173 run, 487 kept | over bound: integration (slowest integration m…
systemd-resolved, held on shanks by the operator under rule 2, was raised
as an urgent "on no list" condition and reminded every hour, although its
hold is already said and answered on the held list. A node now goes on
neither list only when it gets no want and the module is not held, by a
hold read at its fingerprint or one this pass raises. The first pass after
a start clears the neither condition of every assigned node not on neither
list, not only those with a want, so a neither raised before the restart
for a held module (or a move waiting for its record ask) is cleared too.
mesh/check pass 975f0ed65460 on 653da54c0da3 | unit 154 run, 419 kept | integration 164 run, 485 kept | composition passed on ace
Phase guard, same kind as the failed-health hold. allRunHealthy counted
every have whatever its phase, so a node with a removed, held or still
moving have kept a hold up for ever: it counts only phase running (healthy
or not) and failed (not healthy). The failed-health hold already ignores
removed and held haves (phase running or failed only).
Stale conditions (B1). The neither and held conditions this process said
lived only in its memory, so one raised by a previous process stayed urgent
for ever once the module had its want or was no longer held. The first pass
after a start clears, once, the neither condition of every assigned module
with a want and not on neither list, and the held condition of every
module's newest fingerprint that is not held; a clear of a condition that is
not open is quiet at the router. A hold raised again forgets that it was
cleared, so a later release clears it again.
Record ask (review note 3). mustAsk wrote no ask when the want before was
removed, so the first-want path could write a recreates or moves-data
composition of a policy-record module with no ask. A removed want is now
no want, and the first want after it goes through the record ask.
Tests first, each seen failing: a hold lifted over removed, held and moving
haves; the record ask after a removed want; a fresh pass clearing what a
previous process raised, once.
On novox at 15:27:36 a have reading "health failed, phase applying" at the
controller's new fingerprint held its canary two seconds before the new
build was healthy: Wants held on a failed health whatever the phase. A
failed health while the node backs up, applies or checks is a step of the
move, not its outcome, so it holds nothing now.
Every other Health read in the assigner (rule 3's healthy, rule 5's
providedHealthy, the world's lastHealthy) already asks for phase running.
Test from the live row first, seen failing before the fix.
An old-form build (not src2:) has no composition, so no want can ever name
what such a node runs. Design limit, decided by the lead (lead-47): its first
want names the newest F, by every rule as usual - the canary (rule 3: this
node is the canary, or the canary runs F healthy), everything F needs (built,
not held, granted, composed) and the providers (rule 5). Otherwise nothing is
written and the neither condition stays. The canary stays in force, and
old-form modules (clipmenu and screen-lock on g14) leave neither list by
themselves, without a hand-written want.
The code already behaved so (an old-form have falls through to Holds); the
case is now named in Wants, and proven: a unit test for both sides of the
canary, the providers and the grant, and the integration test gives g14 F
and clears its neither condition once the canary ace runs F healthy.
A first want at the newest fingerprint passed over the canary for every module right after the switch. It now names what the node
runs: F when it runs F (by the rules without which there is nothing to want), or an older fingerprint of the new form whose
composition and grant exist, held back from F by the rule; otherwise nothing is written and the node stays on neither list. A new
assignment, with no have, waits for the rules and raises no condition (1945/AC19-AC21).
The node's own composition gives up every module with a fingerprint of the new form, so such a module is applied only through its
want. The assigner wrote no want while a rule held it back and the node had none, so clipmenu, screen-lock, i3status-rust and every
module requiring x11-display lay on neither list on g14 and shanks: xorg's health is unknown (rule 5) and the canary never ran them
healthy (rule 3), and their files were removed and never made again.
A node that runs a module and has no want for it now gets its first want by the rules without which there is nothing to want (built,
not held, granted, composed); the canary and providers rules order every move after it. Any module still on neither list is raised
as an urgent condition naming module, node and rule, written to the journal, and cleared once it has a want (1945/AC19-AC21).
A hold written again after a lift and a restart reused its first ask id, which
the router drops; the hold's time is now its asking. Seven holds could take the
router's three places for the teller and keep a merge Approve from being asked;
the teller keeps at most two assigner asks open. A release or keep answered for
one hold carried over to a later hold of the same fingerprint; an answer now
counts only for the asking its warrant names.
Seven modules stayed held on every machine and the operator was never asked:
the teller only filled assigner asks from the view, a single failed check held
a fingerprint for good, and the held record kept no cause. The teller now asks
every assigner ask as it asks merge asks and writes the phone's answer under
the ask's key; the assigner lifts a hold whose fingerprint runs healthy on
every node that runs the module, says so once and takes its ask back; the held
record names health, phase, when and the node's words.
The assigner reads mesh-controller.builds again (the composer's runningCommits reads the build record of every open
assignment), puts its compose asks on node-build-go and says its conditions on operator-channel.
mesh-tools
jschoubben