mesh/check checking 33131dcbfe77 on 2ae7c53376e8 | novox/mesh-host#181 in group fix/build-not-done-holds-one-module
At 19:30 on 2026-10-30 node-tools' want moved to a build not yet done; once it landed, ace, g14 and shanks stayed
refused until some other input moved (g14 and shanks at 19:40, ace after a restart), since the composer watched no
build.
Quick fix: compose again on a timer while anything is held. Thorough fix: the composer watches the builder's builds
(within the node's compose reads, NodeComposeReads) and makes a pass when a done build of a module the node wants, at
its want's or the trunk's build, lands; nothing else moves it. With mesh-sdk 9685990a (pinned here) a running module
whose build is not done is held, keeping what runs, so the pass that waited composes it as soon as it lands. Chose
thorough: event-driven, no poll.
mesh/check checking 4055e0094b48 on ee71ba4dea81 | novox/mesh-host#180 in group fix/one-bad-item-holds-one-module
At 19:13 on 2026-10-30 power's bar item (mesh-catalog ca8d6f33) made frombus.Compose refuse the whole node, and the
composer marked every module not composed (32 of 81 failed on shanks). The composer already holds what Compose names
held (NotComposedEach, issue 1126); the SDK now names a refused manifest's module held instead of refusing the node.
Quick fix: none in mesh-host; thorough fix is the SDK's input (one rule for Compose, Answer and PlansHeld). The test
replays the captured power manifest and fails on the old pin with the live words.
mesh/check checking b9c24a7b42cb on 737bce9c9448 | novox/mesh-host#179 in group fix/sweep-2026-10-30
mesh-vault (mesh-catalog f3b13de2) answers a move it refuses with `refused: {reason, held}` and `why`, and
no read-back. The engine read only the read-back, so every such row failed "answered with no read-back"
and the controller asked it again on every pass.
Quick fix: treat any answer without a read-back as done. Thorough fix (chosen): read `refused`. For
holds-another-value the vault's value is the live one: the move answers done, superseded, at the held
generation with the held record in its words, so the controller records the row and stops asking; the
journal says it once per secret. Any other reason fails in the vault's own words.
Every engine's node-seat subscriptions were refused at 12:10 by a user list composed without them. nats
asks a refused subscription again only on a new connection, and a user-list reload makes none, so the
seat answered nobody all afternoon and the messenger held every Telegram ask. The endpoints also lacked
the kind and scope discovery reads, so the console said nothing in the mesh is called node-engine.
Quick fix: restart the engines. Thorough fix (chosen, the second time a stale user list left an engine
deaf): a refused subscription of any kind is said and the link is opened again after a pause (1 min,
doubling to 30 min), so every subscription the engine makes is asked again; the seat's endpoints carry
kind, seat, scope, module, description and schema like every tool runner's.
Every account resource marked root never goes into NodeHealth.Root; the statement's contract into Contract. Relogin,
root and found are judged each on its own, so an account may be both. Pins mesh-sdk f3677eb. Test: the report's row in
TestAHealthStatementIsSaidOnTheLinkOpenNow carries the agent account's verdict and the contract.
delivery.NetworkPart.Targets (mesh-sdk 6eae076) is filled where the report is made, by the controller's own rule:
- the hub's node for "hub";
- each resolver's node by its address on the private network;
- none when the part points at nothing, at this machine, or at an address no other mesh machine has.
The names come from the placements the node's own composer holds (Decided.Address, Decided.Hub). A spoke's tunnel names
only the hub, so the tunnel cannot name a resolver on another spoke. A node that does not compose its own names no
target.
Test: rows in TestAHealthStatementIsSaidOnTheLinkOpenNow (hub, two resolvers, a VPN's resolver, its own, a healthy
part).
delivery.NodeHealth and its parts replace this package's copies. The message's network and units parts are aliases of
the same types, so its JSON is unchanged. The engine keeps the derivation from its statement and the write on a change.
Test: the key's bytes, decoded and encoded again by the sdk, are the same bytes.
Until now the six machine alerts came from a health message passing to the controller. The report is now kept in the
engine's declared state `health`, key `<node>`, created by the grants part as `acts` is. Only this node's engine
writes it, after comparing it on every look and only when what it says changed; it is read first after a restart.
It carries:
- the network parts (machine-network; network-rewritten through the resolver file part's writer and owner);
- the tunnel's peers, each seen within five minutes or not (network-unreachable);
- the failed units no module owns (machine-units);
- the accounts marked relogin needed (relogin-needed);
- the directories used as found (directory-used-as-found).
The streaks and the time of the look are left out, so a look alone writes nothing. A peer is reported seen or not,
never its handshake time, so a renewed handshake writes nothing.
The layout is internal/link's (NodeHealth) until it moves into mesh-sdk. Pins mesh-sdk 58a8324.
Tests:
- rows in TestOnTheHubAnyFreshPeerIsAHealthyTunnel: the peers, and a change only when one is seen or lost;
- a case in TestAHealthStatementIsSaidOnTheLinkOpenNow: what the report takes, and one write per change across a
restart;
- the manifest's states in TestTheManifestConsumesTheBackupEvent.
The provisioner's client fills a handed module's file but for each `${secret:<provision>}` (or a whole `secrets` file
that is the placeholder alone).
- The node-engine reads the provision's key `<node>.<consumer>.<provision>[.<local>]` in mesh-provisioner's
`provisions`. The key must be `provided` and carry the vault's reference.
- It asks the vault's `own` for that pair credential sealed to this machine, at least at the reference's generation,
and opens it with the node's sealing key.
- It writes the file as declared (its owner), 0600 when the declaration gives no mode.
- Anything short of that refuses the write, naming the provision and why.
A module the node composes itself already has its `secrets` files written so: its composition carries the vault's
value from `own`, written 0600 with the declared owner.
Pins mesh-sdk 4a96793 (go/provisions). Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise.
The reviewer showed how 76e214a still lets the 07:54 outage happen at the re-landed engine's first start on novox.
applied/ is empty there, so loadApplied loads nothing. The kept declaration retires, because every placed module has a
have. Nothing then carries the node's own composition. The guard misses nats, which has no want there, so its
nats.conf is guessed as nats's and removed.
- At the start, when the store has no node's own composition, the node's own runs as the kept declaration has it, and
the store keeps it now. The kept declaration retires only once the store holds `node`.
- What this engine applied runs until loadHad reads the composition a differing have names. The bucket no longer keeps
the one novox's have names, so the stored one stays.
- In the node's own apply, a resource of another module goes only when that module's want says removed
(Scope.RemovedByWant). A module the node's own composition carried has no want, so Placed never named it.
Cost: the node's own apply cannot drop one of its own dotted resources without such a want. It is refused whole and
said, never removed silently.
Tests, with no new test functions:
- A first-start case in the agent's Everything test (empty store, then a start with the declaration retired). It fails
on 76e214a.
- Two rows in the guard's table: no want refuses; a want saying removed removes.
- A case for the store in the authority test.
On novox on 2026-10-29 at 07:54 UTC the kept declaration was retired and the node's own composition was in no bucket.
The composer's first pass reads what runs before the agent reads any have (the agent waits for the composer's caught,
sent after that pass), so the node's own was composed with its folded modules running nothing, and the bus server and
four held modules were removed.
- Quick fix: read the node's own have in loadHad too. It still waits for the composer's first pass, and still finds
nothing when the bucket no longer keeps the composition the have names.
- Thorough fix, chosen: the engine keeps each composition it applied (the node's own too) in `applied/` beside its
state, 0600. The agent reads it at Run's start, before any watch, as what runs ahead of the bucket and the kept
declaration. A have naming another running composition wins. An apply keeps; a removal forgets.
Test: a case in TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest (no new test function). It fails on L3's
engine 4f9227a (0 compositions at the first own composition) and passes here.
On novox at 01:16 on 2026-10-29, after the node began composing its own modules, mesh-vault's apply failed while the
vault ran: its own composition carries the export as it runs (mesh-sdk frombus KeptRuns) by a digest whose object the
controller's sweep (mesh-controller cmd/mesh-controller/declaration_sweep.go, sweepNode and sweepDeclarationFiles)
had removed, since the sweep keeps only what the controller's compositions in composed-<node> and the wants name.
The node may only read its bucket (ADR 0486 §2), so the node keeps what it had: a file here that verifies, or an
object fetched, goes to <state dir>/objects under the object's name, and a kept one is used before the bus is asked.
An object unused for 14 days is let go. A fetch that still fails says the keep never held it, before its cause.
Quick fix weighed: re-put the object from the controller's store. Thorough (chosen): the node never depends on the
bucket for content it once had, for every module, since the sweep and a node composing its own can always disagree.
The whole-apply test gains three rows that fail on the old code; the test count stays.
The kept-declaration fallback built a composition without Data, so after a restart a data module (mongodb, mssql
on novox) was judged to move its data against its first composition and the operator was asked a false yes. Now
every such composition takes Data from the manifest (agent.Env.HoldsData, the composer's DataItems), and a module
whose manifest cannot be read is said to be unjudgeable rather than taken to hold none. The composition a have
names comes before the kept declaration in the judgement too.
loadHad takes a have only in the phases baseline takes as running (running, held, a composer's mark): one backing
up, applying, checking, failed or waiting names a composition not applied (review of 031e106). Two rows added to
the same table test; the count stays.
A held module on a node composing its own never gets a composition, so the agent never took it and the filling
judged its files by the kept declaration, older than power's three new drop-ins: refused from every engine start
on ace, g14 and shanks until some module settled (2026-10-29 00:09 UTC). The second meeting of issue 924.
Quick fix: knock the filling once the haves are read at the start. Thorough fix (chosen, the kind recurs for every
file added since the switch): what runs of such a module is the composition its have names, from the bucket or
its revisions, ahead of the kept declaration; the filling is knocked on every change of what runs, not only on a
settle. One unit test replaced by a table of four cases; the count stays.
mesh-sdk's composer no longer holds a module behind a held one it reads from; the selfcompose test now expects
proxy handed, composed without de-spiegel's routes, and de-spiegel told alone. Needs mesh-sdk pinned at the fix.
mesh/check failed: ace: the check of novox/mesh-host#168 failed: the pin of git.novox.be/novox/mesh-sdk/go at commit d4c61ee8ffd0 is on no pull reques…
The answer names its proposal and the revision of the node's own candidate key; a node with no key in a proposal is not asked. The watch for its own placement key is tried again until the seat's state exists, so the node switches when its key is written.
mesh/check being carried to bf667a602d21 on 58b4f8b6bc07 | from 9a8b3870bad0
A reboot left the have running, so a unit that failed to start at boot was never started again and nothing said so. Baseline now asks the machine; the judge's look calls Agent.Reconcile once per down edge, leaving held and unwanted modules alone.
mesh/check pass 24c23a1d066f on a548a61e86d7 | unit 2667 run, 590 kept, 36 skipped | integration 2676 run, 633 kept, 29 skipped | over bound: unit (sl…
frombus.Compose now holds a module refused alone (mesh-sdk 0de3006, the rule
its answer follows too): composed as what runs of it, with what depends on
how it is held held too. The composer hands every other module, keeps what it
handed of a held one, and tells the agent which modules it holds and why;
agent.NotComposedEach marks those alone and clears the rest. Only a refusal
of no single module still marks every module (NotComposed).
heldOut also holds out, of the node's own composition as it runs, the
resources of each module held alone, until a composition of its own is
handed: the node's own apply never removes a module leaving the fold with its
names unset (review of the apply, round 1; issue 1945 by a new path).
Pins mesh-sdk fix/1126-apply-culprit at 0de3006 (a group).
mesh/check pass bc28606d4566 on 690aa662a837 | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
A skipped or held file comes back on every apply; counting it would reload the
service manager on every apply of a node with a gated or held module. The same
allowlist as MovesRoot: created, updated, corrected, removed.
A drop-in that no service reflects, like the power module's bound on
reboot.target, was written but never loaded: the node-engine reloaded units only
for a file named in a service's restart-on, so the next reboot still ran the
target as loaded. Any file written or removed under a system unit directory, or
a drop-in directory below one, now gets one daemon-reload per apply.
mesh/check pass 8bd0b5db15a3 on b4fd2cc210bf | unit 1639 run, 4201 kept, 46 skipped | integration 1674 run, 4215 kept, 47 skipped | over bound: unit, …
Each answer's line now reads 'answering placement candidate G (revision N; woken by X)', X being its first answer,
a retry after a read the bus did not answer, the candidate, or the inputs that changed (placement, records, modules,
values, haves, wants). The single input counter becomes one per input, which also decides whether a group is answered
again, as before.
Quick fix weighed: a counter word in the line. Thorough fix chosen: one count per input, so the line names exactly what
moved since the group's last answer.
mesh/check being carried to 1a021052fe07 on b4fd2cc210bf | from 7a2af2376405
Review round 1 measured it on systemd 262.1: a finished one-shot nothing references is
unloaded, and loaded again it reads inactive, ExecMainStatus=0 and a start timestamp of
0, as one that never ran does. The previous commit's rule read such a unit stopped, so
a one-shot declared running would be started, read stopped and fail on every apply, and
the hold, the take-over and the giving back would judge it wrongly.
ServiceState keeps main's reading. ServiceAtRest and declaredState stay: they alone
satisfy a stopped declaration for a one-shot at rest, which is the kernel cleanup's
case. A one-shot declared running that must be seen to have run lingers
(RemainAfterExit) or is enabled at boot. Unit tests: the arch table has 5 rows, the
apply test 1; 1262 in all.
The kernel module's boot-time cleanup, linux-modules-cleanup.service (Type=oneshot,
declared stopped and enabled at boot), failed its delivery on ace with 'was asked to
be stopped and is running': ServiceState read every inactive one-shot with
ExecMainStatus=0 as running, and systemd says 0 for one that never ran too. Stopping
an inactive unit changes nothing, so the read back said running again; after a boot
that ran it, the same declaration would fail every apply.
ServiceState now reads ExecMainStartTimestampMonotonic: 0 means never ran, read as
stopped (a package's one-shot declared running is then started, and the hold no
longer counts it as in use). The service apply reads a one-shot at rest as stopped
to a declaration that it be stopped (ServiceAtRest), whatever its last run did.
Quick fix weighed: declare the cleanup running in the catalogue. Thorough chosen:
the reading was wrong for every one-shot, and this is its second fault of the kind
(the firewall's was the first). Unit tests: three arch one-shot tests merged into one
table (7 rows), one apply test (2 rows); 1263 to 1262.
Met twice on 2026-10-28: shanks kept 'want.shanks.brother could not be read ... no
responders' as the own-secret step's last word, and ace kept 'the vault's own: not yet
... no responders' as its answer to seed-7, each until another input moved. Every bus
read of the composer (manifests, builds, archives, served values, the vault's own and
credentials, the have of what runs) is noted (BusFailed); an answer or a pass that met
one is made again after a growing bound (5 s to 5 min), and a pass is not handed when
the have of its sealed values did not answer. The own-secret step tells a read or an
ask that did not answer (Retry, taken again) from a key absent or the vault's word.
On shanks probe/1126-seed-7 was answered at revision 84 every 13-17 s: each look rewrote
the haves of its unhealthy modules with new words, and the shadow took any write of its
own module haves, placement, modules or config values as a change. Each answer loaded
the bus, the health tools timed out, the words moved again. Now only what an answer is
composed from wakes it (what runs, folded modules, a changed value), and each group is
answered again only when its keys' revisions or the node's inputs moved.
A new step, internal/ownresync, after each shadow answer: for each own secret masked as another value, while the node
applies the controller's compositions and runs the one its want names, it asks the vault's move-in with
replace_running: true and the running value sealed to the call's key, once per value, and says what it asked and what
the vault answered. The answer written to the node's have key carries values_differ. Pins the group's mesh-sdk head
1062b33 (go.sum is filled once that head is pushed).
A box is reused by slot (resource, field) after the composition, from what runs or the
have's slots; a have written before slots is reused within its module alone. Config
values are laid by mesh-sdk's one rule (never a default). The shadow answers the
newest candidate first and hands each answer as it lands, says when an answer starts,
every minute what it waits on, and when it ends, and fits an answer to the node's key.
Needs mesh-sdk fix/1126-seed-same (16b8d8a1).
jschoubben