The node key <node>.node is a write's commit: it names each placed module with its value's digest, the
module keys carry the value they replace until it lands, so a reader never composes half a write. Every
reader goes through PlacementKeys/PlacementsOf, which read the legacy placement.<node> too while the
change is delivered.
mesh/check checking 748543c35f7f on 712ec9481b40 | novox/mesh-sdk#147 in group fix/engine-seat-any-verb
The grants part composed the engine's subscriptions from catalogue.EngineVerbs at its own mesh-sdk pin, so mesh-host #184's optional verb composition was refused on every node and every engine reopened its link every four minutes. Quick fix: re-pin the grants part. Thorough fix (this): the seat's subjects on the engine's node, any verb, as a module answers mesh.mod.<module>.tool.>; callers stay granted per verb.
mesh/check checking 8ed3b872e0ba on dec3bfe7a8cb | novox/mesh-sdk#145 in group fix/held-keeps-contributions
A module held for its own secret keeps its contributions (leaveOwn). When its grant to the provider was not issued
yet, the provider's grants file carried "foreseen: made by the next send" for it, and the applying path passed over
stand-ins of a module already held. Such a stand-in now leaves that provision's contribution out of the held module.
A provider waiting for a yes said why only in its have's words; nobody could read the composition that waits beside
what runs. The engine's read verb composition answers that, and ChangedResources lists what a move changes, a
received file naming the consumers that join or leave.
A module held for an own secret never made (nextcloud's admin on novox, 2026-10-30) was composed as a bare stub, so
its contributions left postgres's grants and keycloak's clients, and their helpers would have retired its login.
The culprit now tries leaving out its own secrets alone before its whole offer.
JudgeMove said a changed received-, bound- or secret- file as "where its data is kept"; it now names the consumers
that join or leave. The judgement stays a move: its yes is what kept nextcloud's login.
Quick fix: leave out own secrets alone before the whole offer. Thorough fix (not here): carry a held module's last
contributions from what runs into its providers' received files whatever part of it fails to compose.
The assigner's rule 8 waited for a key for every `requires`, while the plans named only the needs of handed modules:
every new build of grafana, dnsmasq, mesh-knowledge, mesh-mail and the builders waited for a key nobody would make,
and grafana, not handed, had no plan, so no route key and no route on route-proxy.
catalogue.Keyed is the rule (a need a module answers is keyed when its consumer is handed or the provider's helper
answers it), catalogue.Awaited the same rule from manifests for the assigner, and a module not handed now has a plan
for what a helper answers of it. Handed modules' plans are unchanged.
Quick fix: drop the missing-key check from rule 8. Thorough fix (chosen): one rule in mesh-sdk all three parts use,
since the kind (three parts deciding the same set apart) already drifted once and hid failures as waits.
mesh/check checking eff81f1220e2 on b74e23f973ab | novox/mesh-sdk#142 in group fix/held-module-and-assigner-contract
On 2026-10-30 at 21:04 mesh-self-check (ace) and mesh-conditions, mesh-healers (novox) were placed before
their bus accounts were issued. Each was held for "needs a secret called broker and none was made"; its bare
stub then lost its own secrets and was refused by ownRuntimes ("runs as its own account and declares no own
secret broker"), a refusal no culprit takes, so 74 modules on ace and 79 on novox failed. A held runtime's stub
met the same kind earlier today ("its build produced 0 bundle(s)").
Quick fix: let the bare stub keep its broker own secret, for this one rule.
Thorough fix (chosen, the fourth of the kind today): a held module's stand-in (Stubbed) has no runtime
composed at all, so no rule about its runtime, bundle or broker can refuse it: ownRuntimes passes it over and
runtimeProcess composes none for a held runtime (what runs of either is carried, as before). A module of its
own account left without its runtime by a held runtime is refused with a typed NoRuntimeToRunError, which makes
the held runtime the culprit, so those modules are held after it and the node composes the rest.
mesh/check pass 4e9621edd664 on 14de8dadb4ce | unit 2719 run, 602 kept, 36 skipped | integration 2731 run, 645 kept, 29 skipped | over bound: unit, in…
The proposal on ace said only "mesh-self-check runs as its own account and declares no own secret broker": the words
of its bare stub, held for "has no bus credential", which no one saw. The second hidden first reason of the day, after
"0 bundle(s)".
Quick fix: log the held map in composeEach. Thorough fix: firstReason leads a refusal no culprit is found for with the
first reason of every held module it names ("<m> is held: <why>; its stand-in is then refused: <err>"), for Compose and
the node's Answer alike; the hold's own why already keeps its first words. Chose thorough.
On 2026-10-30 the placement proposal p-20261030t192428-77dabc was refused on ace and novox: "has no bus credential:
nothing was issued for novox.mesh-conditions" (and mesh-self-check, mesh-healers). The vault mints a bus account from
the grants part's wanted list once the placement is written on the operator's Approve (ADR 0561 §1: "From the lease
write, the vault mints every new bus account and the controller mints none"; ADR 0567 §5: "A placement is written on
the operator's Approve"), so a shadow composition can never hold it, and no new module with its own account could be
placed.
Quick fix: skip the check for the three modules. Thorough fix: the Foreseeing mode foresees an unissued bus account like
any own secret the next step makes; the issue-203 refusal (busCredentialIssued), the controller's send's, is gone, and
the Reading mode still leaves out what was never issued. Chose thorough: it holds for every module with its own account.
mesh/check checking 9685990a15ed on 14de8dadb4ce | novox/mesh-sdk#140 in group fix/build-not-done-holds-one-module
At 19:30 on 2026-10-30 node-tools' trunk fingerprint moved to a build not yet done; fromBuilds then composed it from
the indexer's manifest, which names no bundle, and ace, g14 and shanks refused every module ("its build produced 0
bundle(s)") until another input moved after the build landed.
Quick fix: skip the runtime's check while its build is not done. Thorough fix: Compose reads each placed module at the
build its want names (as Scope.KeptAt), and a module that runs whose build is not done is held, keeping what runs (a
held runtime is composed as what runs of it, its running bundle), named with why; a module nothing runs of still
composes from the indexer's manifest as before. Input, PlansHeld and Answer read the trunk's builds as before. Chose
thorough: no running module is ever composed from a source manifest that names no build.
The culprit rule held a module whose stub made the composition fail otherwise. The runtime's stub carries no bundle,
so it always fails otherwise: any refusal naming node-tools (or a module whose name holds it) held the runtime, and
the node was then refused whole for the runtime's "0 bundle(s)".
Quick fix: never take node-tools as a culprit. Thorough fix: blames, the rule as one function, takes the runtime only
where its stub composes. Chose thorough: a runtime whose stub composes may still be held alone.
At 19:30 on 2026-10-30 ace, g14 and shanks refused every module with "node-tools … its build produced 0 bundle(s)":
the refusal of node-tools' stand-in, while why node-tools could not be composed was dropped with the hold.
Quick fix: log held in mesh-host before marking. Thorough fix: Compose's node-wide error carries "(held before:
<module>: <why>; …)", so the engine's journal, every have's detail and any caller see the cause. Chose thorough.
mesh/check checking 754edfae6924 on 16932c93d672 | novox/mesh-sdk#139 in group fix/one-bad-item-holds-one-module
On 2026-10-30 at 19:13 power's bar item (mesh-catalog ca8d6f33) was refused by ParseManifest, and frombus's input
returned that refusal for the whole read: every node-engine marked all its modules failed and no update could land.
The third of a kind that day, after the bus user list and the provisioner's plans (PlansHeld).
Quick fix: skip a manifest that does not parse in input's catalogue loop. Thorough fix: input follows the rule a build
that cannot be read already follows there (refuse + stub), so Compose, Answer and PlansHeld all hold the one module:
held with why on a node it is placed on, read for its offer alone (its unjudged stub, without items) elsewhere. A
manifest the bus does not answer for still ends the read (transient, waited out by the caller), and Input (refuse
unset, the check's whole composition) still refuses. Chose thorough: one place serves all three callers.
mesh/check checking a37181261c24 on 4c1384faefa0 | novox/mesh-sdk#138 in group fix/sweep-2026-10-30
The review found that busretry could ask a delivered call again: Conn.request turned a callee's error answer into
errors.New(words), and Undelivered matched the words "no responders" and "jetstream not enabled", so a callee that
answered so during a blackout ran its work twice. A callee's answer is now *busretry.Answered, which Undelivered
never counts: only the transport's own no responders makes a call undelivered. The other busretry sites read or
write through JetStream's API (State, Ask) or the runtime's state verbs, where the words are the transport's.
i3status-rust on g14 and shanks was issued nothing to read ("was issued: none", 148 times): ReadsGranted derived a
holder's reads from contributions' data alone, and design 57 L3 moved power's draw block to an item
(mesh-catalog ca8d6f33: shows: state, state: power.draw). ReadsGranted now reads a module's items of a kind the
holder's seat receives too, their attributes in a contribution's data shape, always at the machine's own key
(ADR 0260), never another's. Both callers, the controller's busrecords and mesh-grants' records.go, get it
through ReadsGranted with no change of their own.
mesh-config reads mesh-controller.lease to take the config over only once the lease is handed away (ADR 0561 §3,
its takeover.go leaseAway), and was refused it as a reader the composer does not name.
The seat-invoke rule refused mesh-config, mesh-healers and messenger calling the controller's verbs, and
mesh-self-check calling the watchman's seat, over a catalogue that holds no manifest of their holders: the
controller answers its seat itself, and the watchman lives in mesh-delivery. mesh-control's
TestModuleCheckPassesTheCatalogue (the captured mesh-catalog) failed on it. A mesh seat is held by the mesh or by a
module of any repository, so one repository's catalogue cannot say it has no holder: for a mesh seat only the verb
is judged; a seat a module declares is still judged for a claimant. Over the captured catalogue, the live
mesh-catalog alone, and mesh-catalog, mesh-control, mesh-delivery and mesh-host together: no refusal.
mesh-grants' refused.go keptEntry wrote a user of the accounts block from grants it keeps by copying
ComposeAccounts' lines; a change to one would leave the other writing a list the bus reads differently.
buscompose.AccountsEntry(user, passwordHash, perms) is now the one writer, and ComposeAccounts uses it. No change
to what is written.
At each mesh-grants restart and user-list reload (16:37:24, 16:55:15) nats-server 2.11's reloadAuthorization left
the mesh's account without its imports, $JS.API.> among them, for a few seconds: requests had no responders,
JetStream calls said it was not enabled, and a build-test result on ace was lost.
Quick fix, chosen as asked; the thorough fix is a reload that keeps the account's imports, in the bus's holder.
The package busretry asks a request again, backing off within about 30 s, only while its error says it reached no
one (no responders, JetStream not enabled, no response from the stream, or either in words): nothing was done, so
even a create or an update is safe to ask again. A timeout or any answer is never asked again. After the bound it
is said on standard error and returned. Used by process.State's requests, process.Conn's calls (asked again only
while $JS.API.INFO says the bus itself answers nothing, so a callee that is not running still fails at once) and
asks, and the bundle's state requests through the runtime.
The provisioner's watches of gitea.bound, mesh-vault.bound and postgres.bound failed every minute for an hour on
the control-node ("the current values ... did not arrive in 10s") and 1257 changes were refused as "names no
watch of this bundle". Both sides were at fault. The SDK answered a change only once its handler returned, so a
slow handler (Rejudge waits on the lock its composition holds) counted against the runtime's 10 s bound on the
current values; and when the runtime gave up, the SDK forgot the id at once. The runtime (mesh-tools node-tools
internal/bus/state.go StateWatch) stops its watcher on that bound but still drains and delivers the values already
buffered, to an id nobody holds.
Quick fix: a longer runtime bound. Thorough fix here: each state and provision watch has a follower that takes a
change in order and answers at once, its handler on its own goroutine; Watch still returns only once the current
values were handled, and a handler's error is said on standard error. A watch the runtime refuses is said given up
and kept by id, so a change still handed to it is refused saying so. KeptState.Follow asks again with backoff until
taken, the loop four modules wrote for themselves. A log watch keeps its answer after the handler: that is its
back-pressure.
messenger invoked seat:node-engine.root-free while nothing answered it, and Telegram held every ask that offers an
approval, silently. Nothing judged a seat:<seat>.<verb> invoke beyond its spelling: not that the seat exists, that
it promises the verb, nor that anything holds it.
Quick fix: messenger asks the controller's root-free (mesh-catalog bc271896) and the engine announces its seat as
discovery reads it (mesh-host 7ecf7ba). Thorough fix, chosen because an invoke nothing answers fails only at the
call, silently for whoever waits on it: CatalogueProblems refuses every seat verb a module or one of its processes
invokes whose seat no module declares and the mesh does not define, whose seat does not promise the verb, or whose
seat no module claims (a seat every node's engine holds, ADR 0563 §1, is held). Over some manifests alone a seat
whose holder was not given is a note (IsUndeclaredSeat). Over every manifest of mesh-catalog, mesh-control and
mesh-delivery today it refuses nothing.
Since 13:25 the vault, the assigner, the watchman, the grants part and the artifact store's tools were refused
mesh-controller.lease, and the vault mesh-grants.accounts, though the composer names them those states' readers
(ADR 0547 clause 9): the controller splits reads with its own copy of the reader-less ReadsAnnounced, which knows
no named readers, and Users only moved reads from granted to refused, never back.
Quick fix: the controller calls ReadsAnnouncedBy with the module. Thorough fix, chosen because the kind was met
on four owners and the judgement had two copies: Users judges every read again for its reader against the
announcements in Records (withAnnouncedReads), whatever split the caller made, and the reader-less
ReadsAnnounced is removed so no caller can judge without the reader. A rule over manifests was not added:
ADR 0495 clauses 1 and 5 keep readability out of the manifest and replaced the merge gate's reads check by each
module's announcement test. gitea.bound and disk-load.readings stay refused: their owners announce them not
readable, which their own code fixes.
mesh-grants' sealed grants and the controller's merge record each met "message size exceeds maximum" (JetStream
10054) only after the put, and a writer that retries looped on a value that can never land; an update over the
limit was even answered as a revision mismatch, which every compare-and-set writer retries.
Quick fix: cap the two values in their modules. Thorough fix, chosen because the kind was met twice and every
module writes through the SDK: the package kvlimit checks a value against its bucket's largest message (read from
its stream and kept) before the put, naming the bucket, the key, the size and the limit, and turns the bus's own
refusal into the same *TooLarge (errors.Is kvlimit.ErrTooLarge), so no writer retries it. The SDK's own writers
use it: process.State (Put, Create, CreateTTL, Update), process.AwaitedBucket, the runtime state of a bundle
(stdio KeptState, against the module state's 256 KiB) and mesh-hand-compose. A value that can outgrow its bucket
is split over keys or kept in the object store with its name in the key.
PlansHeld holds a module whose build cannot be taken in or whose plan refuses, by composeEach's rule, and composes
every other module's plan; sealedFor gives a file the stand-in its own secret's file has (baserow on ace); shipsABundle
reads a built manifest's Bundles (nats on novox).
mesh/check being carried to af66ccb8335a on 1d9de5dbb63b | from 42988a734b3d | novox/mesh-sdk#137 in group fix/provisioner-reads-every-plan
The client V3 watches the seat's plans whole to bind a provider's consumers (ADR 0373); the grant gave each machine
only its own keys, so its watch was refused and no files were placed. Quick fix, chosen in a recovery: plans join
the buckets a client reads whole (references, never secret values). Thorough fix, queued: the client fills consumers
from the seat's provision keys (ADR 0570 clause 3) and plans go back to its own machine's keys.
mesh/check checking 6e7820395952 on 42988a734b3d | novox/mesh-sdk#136 in group fix/runner-carries-no-engine-state
Second meeting of the engine-state-in-runner kind (mesh-host_acts this
morning, mesh-host_health now); thorough: the runner never carries the
engine module. One predicate, Composed, keeps the engine out of the
placed loop (no user, no process, nothing carried); CarriedByRunner is
the one test of what the runner carries. The engine's grants stay the
node account's, its state created through Records.Engine.
mesh/check checking 9379e9a96ad4 on d0625eea3d11 | novox/mesh-sdk#135 in group fix/credential-forms-one-list
Second meeting of the credential-form list kind: the provisioner client's
own copy lacked "none" and refused every plan naming it, as the
controller's had before. Thorough: one list in mesh-sdk,
catalogue.CredentialForms and IsCredentialForm; provisions' forms are now
the catalogue's constants.
mesh/check checking 83cf97b690f5 on 0c99e847a2ae | novox/mesh-sdk#134 in group fix/memberships-shape
The grants part's old build wrote its wrapper for every membership, a held-back one too, with the unpublished
composition inside; the new part never rewrites such a key, so the decoder took the held-back membership as given.
It now reads a wrapper only when its published says published or unchanged. The codec's rows join the round-trip
test (mesh-sdk is over its unit budget). Review round 1 of issue 774's fix.
The grants part wrote its own wrapper into mesh-grants_memberships while the tool runner read the value as a bare
membership: novox's tool runner served no tool. EncodeGrantsMembership writes the bare membership;
DecodeGrantsMembership and BareGrantsMembership read it, still read the old wrapper for the transition, and refuse a
value naming no node or module instead of answering an empty membership.
Issue 774 (the bus login list, L3). Quick fix: write the bare membership in the grants part alone. Thorough, chosen:
one codec in mesh-sdk both sides use, because a hand-built test fixture agreed with the mistake and hid it.
NodeHealth.Root: each such account's verdict (healthy, unhealthy with its way to root, unknown while the setuid search
runs), the module and resource declaring it. NodeHealth.Contract: the engine's statement contract, so a reader tells an
engine older than the judging. Test: a row in TestTheControlRecordsReadTheirWritersValues, read back byte-equal.
NetworkPart.Targets: the hub's machine for "hub", the machine whose tunnel address is each resolver's; empty where the
part is the machine's own. Filled by the node-engine from its tunnel's peers. Test: a row in
TestTheControlRecordsReadTheirWritersValues, read back byte-equal.
Quick fix chosen (in askrun.go, the one place every build module's check ask runs): own-test packages already ran first, but the rest ran after their failure (issue 1921). 1921 still holds past them.
rev-held2: ToolsRunning took a carried module's bundles from the runtime's MESH_TOOL_MODULES alone, and shanks had
already applied the drop of systemd-resolved, so its runs stayed empty and the drop permanent. Now every bundle
archive that runs at its module's BundlePath is a running bundle; one the runtime loads nothing of takes its loads and
words from the module's build (its built bundles, else its build's artifacts, by catalogue.BundleLoads, the rule a
build's resolution keeps, now one function). The runtime process and each archive are matched by their owning module
too, not only by id and path. A row added: the runtime already dropped it.
novox's mongodb and mssql asked a yes for moving their data on a toolchain move: the running side, read back from the
kept declaration, carries no data flag, and judge.go returned moves-data on the flag before it compared the data
directories and files, so a flag mismatch both raised a false ask and hid a real move. Now moves-data only when a
data item (a directory or file within the data places, or a process of a module with data) changes; a flag-only
difference is neither, with a WARNING in its words. One row changed and one added in TestJudgeMove.
After the take-over of placement on 2026-10-29, shanks's runtime stopped serving the folded systemd-resolved (the
node-resolver seat's verbs answer "nothing serves") and novox's the folded nats. Neither has a want: each runs inside
its node's own composition at a build of the old form (Scope.InOwn), and is composed as Carried of the indexer's
manifest. That manifest is the catalogue's, which no build resolved, so it has no bundles (catalogue/build.go: Bundles
are set only by a build's resolution), and the runtime's MESH_TOOL_MODULES and MESH_TOOL_ENV are made from bundles
(catalogue/runtime.go). The controller composed them from the resolved manifest its store kept.
The kind: a module composed as what runs of it (Stub, Carried: held, or folded at an old build) contributes to the
others from its manifest, not from what runs. For a held module with a new build it is the other way round: its new
bundles are served, which were never delivered.
- The quick fix: read the old-form build's resolved manifest for a folded module. One instance; a held module still
hands its new bundles.
- The thorough fix, chosen: compose.ToolsRunning reads what the node's runtime serves as it runs (its process's
MESH_TOOL_MODULES and MESH_TOOL_ENV, each bundle's archive), frombus gives it as Node.RunsTools, and every stubbed
module of the machine's runtime (held or carried) contributes those bundles, whatever its manifest says.
Tests: two rows (folded, held) added to TestCompositionsAreTheDeclarationSplitByModule; no new test function.
mesh/check pass dbcaa29a1a38 on 394fec8a63c1 | unit 3318 run, 1951 kept, 38 skipped | integration 3994 run, 2125 kept, 38 skipped | over bound: unit, …
On novox on 2026-10-29 de-spiegel, held for an unset config value, held mailu, route-proxy and nftables ("it reads
what de-spiegel contributes or listens on"), and mailu held postgres. mesh-sdk #128 (issue 1126) stopped the hold
spreading to consumers and collectors but kept dependents(): the provider of what a held module contributes to, and
the filter where it listens, were held behind it, transitively.
- Quick fix: drop dependents() alone. The filter and the proxy would then read a held module's new manifest, which
may not be what runs (review of the apply, round 2).
- Thorough fix, chosen: no module is held behind another, in the apply and in the answer (dependents, heldBehind and
the held-behind path of heldAsToday removed). A held module offers what it last offered: its contributions and
listens from its manifest at the build its own composition runs at (ranOf), else the manifest it is composed from.
Only the part of its offer that does not compose is left out, by provision (culprit): de-spiegel's routes leave
route-proxy without them, while mailu keeps its smtp account; the hold names it.
Tests: rows changed in the existing answer test (de-spiegel held, mailu and proxy composed; site held, the filter
composed with what runs; leave-out by provision). No new test function.
mesh/check pass c2999a4e13cc on d4c61ee8ffd0 | unit 3054 run, 234 kept, 36 skipped | integration 3075 run, 266 kept, 29 skipped | over bound: unit (sl…
novox's seed p-20261028t231141-ec84b4 refused 11 modules "what it composes of ..., held, is not what runs of them":
de-spiegel's hold spread through mailu to postgres, nftables and route-proxy, and from postgres to every consumer, and
gitea, held so, refused the node: "its credential of postgres-database is handed to the provisioner and the module is
not: nothing would fill it". Three rounds each moved the error. The kind: a held module still took part in the
composition as a dependence, so its hold spread.
- The quick fix: let heldAsToday also follow dependsOnHeld's words, and skip the handed refusal for gitea. It moves the
error once more and leaves every node's applying path holding consumers.
- The thorough fix, chosen: a held module composes only as what runs of it. composeEach no longer compares the
composition with each held module as its stub (dependsOnHeld removed): a consumer or collector composes against what
runs of the held module like any other module, and a real difference is still a named difference in the answer, a
module's own refusal still its refusal. A stub (compose.Stub, catalogue.Manifest.Stubbed) leaves the credentials and
own secrets it would hand the provisioner out of its plan as they run, never refused for want of a plan. Only what a
held module's new manifest says to others in their own files is still held behind it (dependents: the provider of
what it contributes to, the filter where it listens), which heldAsToday already follows.
Tests: rows added to existing tests (a held provider with a binding consumer, which the old code held in the seed's
words; a held feeder of a collector; a held module with a handed credential and own secret). No new test function.
jschoubben