mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
The answer names its proposal and the revision of the node's own candidate key; a node with no key in a proposal is not asked. The watch for its own placement key is tried again until the seat's state exists, so the node switches when its key is written.
mesh/check being carried to bf667a602d21 on 58b4f8b6bc07 | from 9a8b3870bad0
A reboot left the have running, so a unit that failed to start at boot was never started again and nothing said so. Baseline now asks the machine; the judge's look calls Agent.Reconcile once per down edge, leaving held and unwanted modules alone.
mesh/check pass 24c23a1d066f on a548a61e86d7 | unit 2667 run, 590 kept, 36 skipped | integration 2676 run, 633 kept, 29 skipped | over bound: unit (sl…
frombus.Compose now holds a module refused alone (mesh-sdk 0de3006, the rule
its answer follows too): composed as what runs of it, with what depends on
how it is held held too. The composer hands every other module, keeps what it
handed of a held one, and tells the agent which modules it holds and why;
agent.NotComposedEach marks those alone and clears the rest. Only a refusal
of no single module still marks every module (NotComposed).
heldOut also holds out, of the node's own composition as it runs, the
resources of each module held alone, until a composition of its own is
handed: the node's own apply never removes a module leaving the fold with its
names unset (review of the apply, round 1; issue 1945 by a new path).
Pins mesh-sdk fix/1126-apply-culprit at 0de3006 (a group).
mesh/check pass bc28606d4566 on 690aa662a837 | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
A skipped or held file comes back on every apply; counting it would reload the
service manager on every apply of a node with a gated or held module. The same
allowlist as MovesRoot: created, updated, corrected, removed.
A drop-in that no service reflects, like the power module's bound on
reboot.target, was written but never loaded: the node-engine reloaded units only
for a file named in a service's restart-on, so the next reboot still ran the
target as loaded. Any file written or removed under a system unit directory, or
a drop-in directory below one, now gets one daemon-reload per apply.
mesh/check pass 8bd0b5db15a3 on b4fd2cc210bf | unit 1639 run, 4201 kept, 46 skipped | integration 1674 run, 4215 kept, 47 skipped | over bound: unit, …
Each answer's line now reads 'answering placement candidate G (revision N; woken by X)', X being its first answer,
a retry after a read the bus did not answer, the candidate, or the inputs that changed (placement, records, modules,
values, haves, wants). The single input counter becomes one per input, which also decides whether a group is answered
again, as before.
Quick fix weighed: a counter word in the line. Thorough fix chosen: one count per input, so the line names exactly what
moved since the group's last answer.
mesh/check being carried to 1a021052fe07 on b4fd2cc210bf | from 7a2af2376405
Review round 1 measured it on systemd 262.1: a finished one-shot nothing references is
unloaded, and loaded again it reads inactive, ExecMainStatus=0 and a start timestamp of
0, as one that never ran does. The previous commit's rule read such a unit stopped, so
a one-shot declared running would be started, read stopped and fail on every apply, and
the hold, the take-over and the giving back would judge it wrongly.
ServiceState keeps main's reading. ServiceAtRest and declaredState stay: they alone
satisfy a stopped declaration for a one-shot at rest, which is the kernel cleanup's
case. A one-shot declared running that must be seen to have run lingers
(RemainAfterExit) or is enabled at boot. Unit tests: the arch table has 5 rows, the
apply test 1; 1262 in all.
The kernel module's boot-time cleanup, linux-modules-cleanup.service (Type=oneshot,
declared stopped and enabled at boot), failed its delivery on ace with 'was asked to
be stopped and is running': ServiceState read every inactive one-shot with
ExecMainStatus=0 as running, and systemd says 0 for one that never ran too. Stopping
an inactive unit changes nothing, so the read back said running again; after a boot
that ran it, the same declaration would fail every apply.
ServiceState now reads ExecMainStartTimestampMonotonic: 0 means never ran, read as
stopped (a package's one-shot declared running is then started, and the hold no
longer counts it as in use). The service apply reads a one-shot at rest as stopped
to a declaration that it be stopped (ServiceAtRest), whatever its last run did.
Quick fix weighed: declare the cleanup running in the catalogue. Thorough chosen:
the reading was wrong for every one-shot, and this is its second fault of the kind
(the firewall's was the first). Unit tests: three arch one-shot tests merged into one
table (7 rows), one apply test (2 rows); 1263 to 1262.
Met twice on 2026-10-28: shanks kept 'want.shanks.brother could not be read ... no
responders' as the own-secret step's last word, and ace kept 'the vault's own: not yet
... no responders' as its answer to seed-7, each until another input moved. Every bus
read of the composer (manifests, builds, archives, served values, the vault's own and
credentials, the have of what runs) is noted (BusFailed); an answer or a pass that met
one is made again after a growing bound (5 s to 5 min), and a pass is not handed when
the have of its sealed values did not answer. The own-secret step tells a read or an
ask that did not answer (Retry, taken again) from a key absent or the vault's word.
On shanks probe/1126-seed-7 was answered at revision 84 every 13-17 s: each look rewrote
the haves of its unhealthy modules with new words, and the shadow took any write of its
own module haves, placement, modules or config values as a change. Each answer loaded
the bus, the health tools timed out, the words moved again. Now only what an answer is
composed from wakes it (what runs, folded modules, a changed value), and each group is
answered again only when its keys' revisions or the node's inputs moved.
A new step, internal/ownresync, after each shadow answer: for each own secret masked as another value, while the node
applies the controller's compositions and runs the one its want names, it asks the vault's move-in with
replace_running: true and the running value sealed to the call's key, once per value, and says what it asked and what
the vault answered. The answer written to the node's have key carries values_differ. Pins the group's mesh-sdk head
1062b33 (go.sum is filled once that head is pushed).
A box is reused by slot (resource, field) after the composition, from what runs or the
have's slots; a have written before slots is reused within its module alone. Config
values are laid by mesh-sdk's one rule (never a default). The shadow answers the
newest candidate first and hands each answer as it lands, says when an answer starts,
every minute what it waits on, and when it ends, and fits an answer to the node's key.
Needs mesh-sdk fix/1126-seed-same (16b8d8a1).
mesh/check pass 047237c911a4 on 449cda1b003e | unit 2980 run, 270 kept, 36 skipped | integration 2999 run, 302 kept, 29 skipped | over bound: unit (sl…
ace, g14 and shanks answered nothing to seed #531 and said nothing either. The
shadow now says, after a minute not ready, which inputs it has not read, and
answers each group it knows refused for that; an answer running over a minute
is said while it runs and noted in the answer. The answer's differing resources
carry both bodies, masked, the node saying whether two sealed values open to the
same value (Composer.sameValue: only the comparison leaves the node).
mesh/check pass a6e53a485813 on 199f6b48b1df | unit 2992 run, 1008 kept, 36 skipped | integration 3025 run, 1102 kept, 29 skipped | over bound: unit (…
NotComposed marked from a read have whatever its phase: an apply that failed,
naming the want's build and digest, became a composer's mark, which baseline
then took for what runs, and never applied again. It now marks only a have that
runs, says nothing, or is a composer's mark already. And its loop, whose reads
can take seconds, checks the run's context at each module, so a run that ended
writes no have.
Live 2026-10-28: the own run's last pass, failing on its run's end, marked
every module of the controller-mode agent that replaced it (ace 60, shanks 63),
and the marks dropped the build and digest because the have read failed in the
bus's reload pause. Now: the refusal hook is bound to the run's agent and ctx;
a pass whose context ended returns nothing; JetStream not enabled or no
responders are tried again after 1, 2 and 4 s before any mark; NotComposed reads
the have again while the bus does not answer and marks nothing it cannot read;
baseline takes a have a composer marked as what runs, so a mark it did not make
clears; each run's shadow lives with the run.
mesh/check checking 463626143567 on bb15aab50eda | novox/mesh-host#154 in group fix/1126-shadow-all
selfComposer took a KVHistory error for 'not placed': at any restart a timeout
or a bus not ready flipped a self-composing node to the controller's
compositions for a whole run. selfcompose.Placed chooses only on a read that
succeeds (a missing placement state is one), and tries again otherwise, said
once. The shadow's answer to a head deleting the placement reads the
controller's composition at each want in the node's bucket and compares it with
what runs; a want with none refuses, without returning.
The seed's revert (mesh-catalog #526) was refused on 'would stop running node',
while a self-composing node would have refused its placement's absence, kept
what runs and marked every have failed until a restart. endWhenUnplaced ends the
agent's run when the key goes, so it begins again on the controller's
compositions, as at a start without a key; the composer refuses nothing while
unplaced; the node's have says whose compositions it applies (Composes); the
shadow answers a deleted placement Returns, verdict same.
novox refused its own composition and its answer on de-spiegel, a module it runs
folded into its own composition at a build only the controller holds. The
composer now gives frombus what the node runs (Scope.Running), so such a module
is carried as it runs. The shadow's answer names every refusal (mesh-sdk). A
templates archive read is bounded at 30 s: nothing else held an unanswered store.
H8 found it: a module the node could not compose yet got a failed have naming
no fingerprint, which counted as run inside the node's own composition, so the
next composition took it from its manifest past its build.
The node read a behind module's source manifest digest and composed every module
from the indexer's raw manifest; frombus now takes the build itself (Scope.KeptAt,
a fingerprint). The reader gives frombus the builder's key, a templates archive
fetched as every archive is (apply.Fetch) once per digest and checked against it,
and another node's served keys one by one, as the node's grant allows them.
The fold's watch of every have key ('>') was refused (permissions violation on
CONSUMER.CREATE KV_node-engine_have), and every self-composing agent stopped
after the seed. The composer now watches '*.node' (records) and '<node>.>' (its
own haves), as buscompose grants a node; the shadow watches '<node>.>' too. The
selfcompose tests check every watch and get against PermissionsFor(KindNode),
and the agent's integration tests run the agent as the node under that grant.
mesh/check checking 69b845c0f7b2 on 0d22776b20ef | novox/mesh-host#152 in group fix/1126-fold
A fresh agent composes before it has applied anything (Running empty), so
InOwn of Running folded only mesh-wireguard and the switch would drop nats.
The composer now watches the node's haves: a placed module whose have names no
new-form fingerprint is folded; one with a new-form have whose want is removed
is not folded. The shadow uses the same set, over the candidate's placement.
Pins mesh-sdk 05b67831, whose compositions fold a module no new-form build
names into the node's own. The shadow composed a placed module it runs inside
its own composition (nats, photos) at the trunk's build, as if the candidate
added it; shadowBuilds now leaves those to the fold, and both paths pass
frombus.InOwn of what runs.
mesh/check checking b225c7bb4cf5 on be94d2689e2f | novox/mesh-host#150 in group fix/1496-start-health
On ace, 2026-10-26 21:19, mesh-build-node's cache directory was there before the
mesh and used as found. The health statement says such a directory unhealthy, and
the have folded it into the module's health as failed, with no words, while the
module's own check answered healthy; the assigner read it as its canary failing
and held the version on all four nodes.
The directory waits for the operator's hand-over and keeps its own condition,
raised from the same statement; it no longer counts in the module's have. A have
whose health is failed or waits and whose apply said nothing carries the words of
the resource that made it so, so the next one is readable.
mesh/check failed: unit: ..test: ✔ a transient empty list for four passes retires nothing; five stable passes retire, data kept (290.408814ms) | 58de6…
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | cdd3f3fd4f63 on ff9386dcccc2 | unit 1245…
mesh/check failed: ace: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the grou…
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the g…
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk f1d7c7377dc4, which is neither on mesh-sdk's trunk nor the g…
mesh/check pass 65cd7b47873a on bba9081ca127 | carried from 94f9aa298cb8, mesh-sdk 21252194d530 | unit 971 run, 423 kept, 32 skipped | integration 976…
A process no witness judges had its directory removed before the new bundle
was unpacked and the unit restarted, so the old process ran on in a deleted
working directory: mesh-build-go's `go version` probe failed with "getwd: no
such file or directory" and refused three checks on ace on 2026-10-26, each in
the second a new build of it arrived.
Every process is now replaced the way a judged one is: its build is renamed
under .witness/<name>/retired/ and deleted only once nothing runs from it.
"Runs from it" now counts a process's working directory as well as its image,
so the children a daemon starts keep the directory too. Forget keeps a retired
build that still drains, and the witness's sweep covers every process it keeps
a directory for, judged or not.
Two unit-file tests are folded into one, so the unit count goes down by one.
The share belongs to the part that enforces it: build-cpu-share, default 50,
set per node through the node's settings layer and read by the composer for
the build seats' runtimes. Pin mesh-sdk at fix/build-cpu-yields' head.
Review round 1: one quota per unit let three build modules take three shares.
The engine now writes meshyield.slice, top-level so its idle weight binds
against the operator's session, with the node's share as its CPUQuota; each
yielding unit joins it at nice 19. Comments say the IO controls bind only under
bfq or io.cost, not under the none scheduler NVMe disks use.
mesh/check pass 43f2d6020e11 on 94f9aa298cb8 | carried from 7f04d1204765, mesh-catalog a4f5df797b68, mesh-sdk b4cd95e7c4a4 | unit 972 run, 423 kept, 3…
On the bus node the bus module has no want and has left the declaration, so no
pass of this engine held its fact and the follower's user list was kept in
memory and never written: the vault and the grants part read no hashes and no
key. A module's user list now reaches its key on every pass, merged into what
the key says with every other field kept; a missing key is said once, never
made up (ADR 0547 §8).
A build seat's runtime took 1650% of shanks' 24 processors and slowed the
operator's desktop. The unit of a process declared with yields now carries
CPUWeight=idle, IOWeight=1, Nice=19, the lowest best-effort IO level, and
CPUQuota for its share of the processors this machine has.
Merges fix/1945-adopt-hashes 1a190090. A new follower (one per link) waits for the grants part's key to catch up before
its first write, so a restart after take-over never writes the controller's list first. A refused or ended watch is
tried again on link.Backoff, the have says why it is not watched, and the key last heard, kept beside the node's state,
holds meanwhile. While the grants part's list is followed, the bus module's composition is applied with the list on the
machine in place of the controller's. The grants list is never handed to the vault. Pins mesh-sdk c2a410bb.
On the node whose compositions carry the bus's list, the engine also watches mesh-grants' key user-list.sealed. While it
says follow grants, the object it names is read by name, opened with this machine's sealing key, checked against the
key's digest and the guard, and written to the same file at 0600 only when its digest differs; the bus reloads it
itself. The have says which list is followed and this machine's sealing key, which the grants part seals to.
The follower's test waits for its watch to catch up before a later put, which raced under load. Pins mesh-sdk 175a5567.
The vault holds no value for the controller's own account, the node accounts and the enrolled devices, so it cannot
hash them, and take-over waits on their hashes. The bus node's engine is the one part that has them, in the list the
bus loaded. Its follower of the user list now puts every loaded user's bcrypt hash in the bus module's have, sealed to
the vault's own key (mesh-sdk delivery.AccountsHanded), so of every reader of the have bucket the vault alone opens
it; the clear part says only the list's digest, the key's fingerprint and how many users. The vault's key is heard from
its answer to node-key, which the engine already asks at start; a first or changed key looks at the list again. It is
sealed again only when the list on the machine or the vault's key changes; a refused list hands the list kept on the
machine. The vault takes from it only the users it holds no value for.
Also: the user list follower's test sent its later puts before the watch had handed the keys there before it, so two
of its cases failed on main whenever the goroutine started late; the feed now says when the watch began.
mesh/check pass 99285cb080db on 1195cf45971e | unit 303 run, 1080 kept, 5 skipped | integration 306 run, 1082 kept, 4 skipped | over bound: unit (slow…
The bus node's engine writes the bus user list (ADR 0547 bridge); each write makes the bus reload its accounts, and
for about three seconds every JetStream request meets no responders. An apply that failed then was kept failed until a
key changed (mesh-vault on novox, 2026-10-25 23:04:24), holding every module that needs the secret seat. A transient
bus failure is now retried with a backoff (2 s, 5 s, 15 s, 60 s, then every 5 min), the have saying retrying meanwhile;
after the engine writes the user list, the transiently failed modules are tried again once the reload is over. A have
write that met it is written again the same way. shorten keeps the head and the tail, the cause.
mesh/check pass 5e71ab6ae612 on d75c66dc7e27 | unit 303 run, 1077 kept, 5 skipped | integration 306 run, 1079 kept, 4 skipped | over bound: unit, inte…
B1: the controller sends the user list by digest (over 64 KiB), so its users are counted from the list as resolved, not
from the composition's inline content, which is empty then.
R1: a composition signed for another node or module than its key names is no source.
R2: a list without the controller's own user, or one that drops more than half the users of the file on the machine,
is not written; the have says why and the file on the machine is kept.
The list reached the bus only through a want of the bus module, which nobody writes (ADR 0539 §15): on 2026-10-25 it
stood from 11:23 to 20:50 and was put back by hand. The node agent now follows its composition bucket for the newest
signed composition carrying the user list, writes that file alone through the apply's file step when it differs
(mode 0600, read back, recorded as the module's), moves nothing that reads it (the bus module's entrypoint reloads the
server in place), never goes back to an older list, looks again after every apply of the bus module, and says source,
revision, wanted and written digests, users and any error in the bus module's have (mesh-sdk UserListWritten).
Read with mesh-delivery-view.node_haves (module nats); the bus's own nats_user_can says in_force once it reloaded.
mesh/check pass 338797a52703 on d75c66dc7e27 | unit 2301 run, 3410 kept, 10 skipped | integration 2326 run, 3431 kept, 13 skipped | over bound: unit, …
A write judged by the declaration before the apply settled could record its
refusal just after Knock found nothing owed: the original fault in a small
window. Knock now holds the apply's lock as a write does, reads no declaration
while nothing is owed, and its comment names Settled as its caller.
ace's filling refused mesh-build-go's files as not named as filled 32 s before
its want arrived; after the apply that named them nothing asked again, and the
runtime crash-looped on its missing broker credential. The filling now keeps
what it refused or failed and, when the node agent says a module's apply or
removal settled (agent.Env.Settled, called once what runs holds it), touches its
write socket if the declaration now names such a path, names the provisioner's
account, or a write had failed; the client watches that socket and fills every
module again. Said in the journal each time.
mesh/check pass b32a27c115a2 on 0dfbe1a179f2 | unit 292 run, 1060 kept, 5 skipped | integration 295 run, 1062 kept, 4 skipped | over bound: unit (slow…
The facts look inside an apply runs while the module is still applying and judges nothing; the
running write then left xorg running and unknown until the five-minute statement, holding every
x11-display consumer behind it. The running write now nudges the health loop, which looks and says
at once. A declaration's report gets the same motion and composition check, and a removed module
leaves nothing marked unjudged (review of fix 15).
A have said applying and failed for the controller on novox at 15:27:36, from a look at the
container being replaced, two seconds before its launch said healthy; the assigner held the
canary on it and set novox back. A look now judges only a module that is not in motion, and a
module whose composition moved carries unknown until a look after the move.
xorg's have on g14 and shanks kept the unknown an engine before 958ccbd wrote: the
baseline believed the have and never handed the module to the session, so no look
rewrote its health, and the assigner held every module requiring x11-display back.
mesh/check pass 1bdbf47ffa83 on 0dfbe1a179f2 | unit 302 run, 1044 kept, 5 skipped | integration 305 run, 1046 kept, 4 skipped | over bound: unit (slow…
A unit or container of the node's own that shares only its name with a
module's resource is not the module's: dropped, its record became an orphan
the apply removes, stopping the module's unit (1945/AC17).
The node's own composition and a module's both declared the same files, and
the joined declaration was refused whole ("2 resources declare X"): the
node's own copy is now left out when joining and at the node's own apply, and
said once (1945/AC17). A module still running from the kept declaration now
carries its filled paths, so the filling no longer refuses them as not named
as filled (1945/AC18).
jschoubben