mesh/check checking fdb8deea4c9e on a68b39f7b512 | novox/mesh-catalog#577 in group fix/merge-ask-ends-on-merge
endStaleAsks skipped every merged delivery, so a pull request merged without the
Approve kept its merge ask in the gate's asks until it lapsed: on 2026-10-31 two
such asks held two of the router's three places for the teller and a placement
ask was refused for want of room. Every look now takes away a standing ask none of
whose deliveries is still open, read from what the gate keeps, so a failed take
away is retried. A settle asked again for another set takes the earlier settle ask
away for the same reason.
mesh/check being carried to b60138990b34 on dcf3950f7b07 | from 0e78909a75b9
Quick fix: the plan reader keeps the machine's one placement key for the answer, as it keeps each module's assignment (issue 1126, 2026-10-28); the second time one answer's reads per credential held own past the node-engine's 15 s.
Thorough fix (not here): the node-engine asks own only when a secret or a credential it composes from changed, and the assigner writes no want that is unchanged.
mesh/check being carried to 843c1c337cdd on cb91a4b9c292 | from mesh-sdk b74e23f973ab | novox/mesh-catalog#572 in group fix/one-rule-for-provisions
A module not handed now has a plan for what a provider's helper answers of it, so grafana's route is keyed and
route-proxy's helper hears it; handed modules' plans are unchanged. Deliver after #570, whose client reads a plan with
no templates.
On 2026-10-30 mesh-catalog #571 joined fix/held-module-and-assigner-contract at 21:59; the group stayed
checking, so mesh/delivery-group kept saying "3 of 3 member(s) ready" for a group of four, and #571 had
no mesh/delivery-group status until the state changed.
Quick fix: compare the why too. Thorough fix: compare the why and the member ids (each names its head),
so a push that keeps the count is said as well. Chose thorough: same size, covers the push case.
mesh/check checking b5475cfa5c3d on 395a37dddab9 | novox/mesh-catalog#571 in group fix/held-module-and-assigner-contract
mesh-healers slept 2 s before its first subscribe to the keeper's conditions; its whole test delivered
condition-raised once the lease was read, often before that, and the event was dropped: "never: the would-act
written" in about two runs of three on main, failing the group check. Eight modules had the same shape.
Quick fix: sleep after a failed subscribe in mesh-healers alone.
Thorough fix (chosen, the kind is in eight modules): every loop that asks the runtime for events or a watch
waits for stdio.Served() and subscribes at once, sleeping only after a refusal (mesh-healers, mesh-config,
mesh-mail, mesh-issues, mesh-provisioner, mesh-watcher, mesh-delivery-gate); mesh-knowledge, whose pinned
mesh-sdk has no Served, subscribes first and sleeps after a refusal. The healers' whole test delivers the
event only once the bundle has subscribed it, as the runtime's consumer holds it until then.
mesh/check being carried to 7522961ae3b0 on cb91a4b9c292 | from 5d9cead336ab
A plan with no templates archive made the client fetch an empty source and fail the module's report. The next
mesh-provisioner composes such plans (a module not handed gets one for what a provider's helper answers, its route), so
this reaches the nodes first.
Quick fix: none (a failed report per routed module). Thorough fix (chosen): a plan with no archive is read as provisions
alone by the one client every node runs.
mesh/delivery-group group fix/held-module-and-assigner-contract rejected: composed together: the checker's verdict at the members' heads (c82f18b5e6bb): integr…
mesh/check failed: integration: mesh-healers/cmd/mesh-healers.TestTheHealersWholeActOnTheBusAndBrake: never: the would-act written | b8ce4f0c0ac5 on 3…
The sweep of 2026-10-30 20:10: <node>.claude-code.gitea-read and <node>.mesh-build-node.npm-package-registry failed on
all four nodes with "gitea does not serve": gitea's provider helper answers only forge-account and refuses every other
bind into gitea.bound, and the provisioner took any entry of a module declaring `bound` as the outcome. And
novox.gitea.route, novox.keycloak.route and <node>.claude-code.mcp-endpoint stood asked: route-proxy and node-tools keep
entries the manifest did not read.
Quick fix: add route-proxy.bound and node-tools.bound to the reads. Thorough fix: judge each offer by who answers it
(answered-by: helper reads the entries, any other offer of a module that names a helper is judged by its have), and say
on gitea's forge-account that the helper answers it. Chose both: the reads alone leave gitea's two provisions failed.
mesh/check checking b4b22aac2060 on 0695ae25dbd2 | novox/mesh-catalog#567 in group fix/one-bad-item-holds-one-module
PlansHeld reads the same frombus input that refused every node on 2026-10-30 at 19:13; at 754edfae it holds the
module whose manifest the mesh refuses, with why, and composes every other module's plan. Quick and thorough fix are
the SDK's (one rule in frombus input); this is the pin.
mesh-provisioner, mesh-config, mesh-issues and mesh-healers need KeptState.Follow; mesh-build-go, -image and -node the
retry of a state write the bus delivered to no one; claude-code asks.Condition; mesh-vault the credential forms list.
Each passes go vet, go vet -tags integration and go test on this pin, built with no go.work.
683fd547 and b7f87822 took every `broker-<id>` own secret for a bus account. mesh-controller's broker-address is the
bus's address (MESH_BROKER_ADDRESS_FILE, read by broker.FromEnvironment to issue enrolment tokens and credentials):
c1 would have refused to keep it, and it was counted among the "owed bus-account hashes" every minute.
A name is now a bus account when it is declared: `broker` or `provisioner-broker`, or the own secret of an account the
grants part wants, which it composes from the manifests' bus processes (what catalogue.ToolsBusSecret names). A
`broker-<id>` holding an account the controller issued before its want was heard is still hashed. Anything else is
kept as given, never refused, never minted over, never hashed and never owed. The mint path replaced only wanted
names already, so the address was never at risk of replacement there; the test now proves it.
Quick fix: except broker-address by name. Thorough fix: the declaration decides, for every name. Chose thorough.
Table rows: broker-address kept as given, not replaced, owed nothing; broker-nats-tools replaced by a minted account
once wanted and refused a plain value after. They fail on the previous code.
mesh-provisioner, mesh-config, mesh-issues and mesh-healers each wrote a watch-again loop (`watched`). mesh-sdk
61b67d61 adds KeptState.Follow, which asks again with backoff and says each give-up, and its watches now answer a
change at once and run the handler beside the runtime (the provisioner's "names no watch of this bundle"). The four
modules use it; their loops are gone. Built and tested against the mesh-sdk group head through a go.work that is not
committed: their mesh-sdk pins move to that head with the group.
The client dated each report with the time it was said, and says a failure again whenever its words change, so the
controller's apply-failing alarm, which reads that time, saw a new failure: on novox the vault's refusal replaced a
bus refusal at 16:56 and the alarm cleared for five minutes while every module still failed.
Quick fix: the controller keeps a raised alarm while any failure stands (done there). Thorough fix: the report says
failing-since, kept from the module's last report (this run's, else the applied bucket's) while it stays failed,
whatever its reason says; any other status ends it. Chose both: the controller can now judge the failure's age
rather than its newest words, which is its change to make.
move-in's refusal "the vault already holds another value for ..." was an error's text, which the node-engine and the
controller could only quote. It is now an answer: `refused` with the reason holds-another-value and the held value's
record (generation, changed by, changed at, origin; never the value), `why` in words, and no read-back, so an
engine that does not read `refused` still fails as before ("answered with no read-back"). keepLocked's refusal is a
typed error (HeldAnother) whose text is unchanged for every other path.
Quick fix: a stable prefix in the text. Thorough fix: the fields. Chose thorough, as asked; mesh-host's
moveIntoTheVault has to read `refused` to use them.
The three random values kept before the vault judged them (own/novox/nats/broker-nats-tools,
own/novox/mesh-controller/broker-address and broker-management) are held, so minting refused them ("held already")
and their users stay off the bus. A value that is no account has no hash and logs in nowhere, so replacing it
breaks no login the grants part composed. When the grants part wants the user (every want is heard at the vault's
start), the vault now mints an account in its place, judged again under the store's lock, keeping the value it
replaces beside it. A held account is still never minted anew (ADR 0561 §4).
Quick fix: let an operator rotate the three by hand. Thorough fix: the mint path replaces any such value, so no
other one stays behind. Chose thorough.
own/novox/nats/broker-nats-tools (moved in today) and mesh-controller's broker-address and broker-management (moved in
from the controller's store) are random values, not the account JSON a bus user is hashed from, so the hash step says
"not JSON" and the grants part leaves those users off the bus. The controller made them (fixed there, c421629f); the
vault kept them, since nothing it keeps judged a covered name's value.
Quick fix: refuse a plain value in move-in. Thorough fix: keepLocked, which every way into the store passes, refuses
for any covered name (broker, provisioner-broker, broker-<binary>, which catalogue.ToolsBusSecret names) a value that
is no account, saying "not issued". Chose thorough: one rule where every value is kept.
gitea asked an owed status again every 30 s forever, even on 403 and 404 (jschoubben/hal and jschoubben/novadock, 119
times each in the sweep), and a key the checker deleted did not end a loop already running.
Quick fix: return from askAgain on 403/404. Thorough fix: the same, said once with its cause; a delete of the key
marks its revision seen, so any running ask again ends before its next write; a newer revision still asks anew.
The checker no longer owes statuses outside the mesh (mesh-delivery), which removes the trigger. Not done: raising
the module's health, since gitea's TypeScript SDK has no seat publish for a condition (mesh-sdk).
pair.go kept its own copy of the forms a plan may name; the copy in the provisioner client went in #564, after it
refused plans naming `none` while the vault's copy took them. Quick fix and thorough fix are the same here: the vault's
constants are mesh-sdk's (catalogue.CredentialForm*), and a form is judged by catalogue.IsCredentialForm, so a new
form is added in one place. No new test: behaviour is unchanged, and the existing form tests pass on the shared list.
mesh-vault's mesh-sdk pin moves to 9379e9a9, the provisioner client's, the first with the list.
i3status-rust on g14 and shanks cannot watch power.draw ("was issued: none", 148 times): its read was granted from
the bar contribution's data (shows: state, options.state; catalogue.ReadsGranted), and design 57 L3 moved the block
to an item, whose attributes said nothing of it, while ReadsGranted reads contributions alone.
Quick fix: `reads: power.draw` on i3status-rust, which grants every machine's draw instead of its own key (ADR 0260).
Thorough fix: the item says its read (shows: state, state: power.draw), and mesh-sdk's ReadsGranted grants a held
item's read as it grants a contribution's, keyed to the machine. Chose thorough; its mesh-sdk half is another
repository's, and until it lands the fault stays.
Grafana on ace answers 401 to the vault's admin password, every 30 s since at least 2026-10-05. I think the cause is
the adoption: Grafana keeps the admin password it was first given in its database and applies
GF_SECURITY_ADMIN_PASSWORD__FILE only when it creates it; ace's Grafana came with its database before the mesh minted
own/ace/grafana/admin (2026-09-30, generation 1, unchanged since), so the minted value never took (keycloak's
admin, hq issue 179, is the same kind).
Quick fix: none in code; the password in Grafana's database must become the vault's (grafana cli admin
reset-admin-password in the container), an act on ace no tool offers yet. Thorough fix: a step that converges the
admin password at every start, which needs a container entrypoint the node-engine's declaration does not carry.
This commit does what the catalogue can now: the refusal is said once with its cause and what ends it, asked again
hourly, and its end is said.
The sweep took the node seat node-engine's root-free for one nothing answers; the engines had been refused their
subscriptions by the bus since 12:10 (fixed in mesh-host), so the address was right. This puts it back and keeps
what the previous commit added: every failed ask said with its error, a fresh word not thrown away, and the failure
as the held ask's reason.
The file gitea-read.secret was declared with no owner, so it was root's at 0600; claude-code's tools run in the
node's runtime as the operator's account and could not read it ("permission denied", ~870 times a machine), said
only in the journal. A file a bundle reads is the runtime account's only when a word of the bundle names it
(catalogue givenTo); this one was found through MESH_CLAUDE_CODE_STATE alone.
Quick fix: name the file with the bundle word MESH_CLAUDE_CODE_FORGE, which the reader now uses. Thorough fix: the
same, plus a failure to place the credential raises a warning condition of the module on the operator-channel seat,
once per reason, cleared when it places again. Chose thorough: the fault ran unseen for days. A change of the
credential now restarts the node's runtime, as for every file a bundle word names.
claude-code's mesh-sdk pin moves to 2da0de75 (main's), the first with asks.Condition.
The controller grants a read only from the owner's announcement (ADR 0495 clause 3). gitea announced `bound`
unreadable, so mesh-provisioner bound no forge account; disk-load announced `readings` unreadable, refused to restic,
nfs-server and mounts; restic announced `good` unreadable, which the self-check reads. Nothing in the catalogue
compared readers with announcements since the gate's manifest check went (clause 5).
Quick fix: mark those three states readable. Thorough fix: also checks/readable, which reads every owner's Go and
TypeScript announcement and refuses any `reads` of a state it does not announce readable, so the next owner to
announce cannot repeat it. Chose thorough: the same kind hit three owners at once.
Reads of other repositories' owners (mesh-controller.lease, mesh-grants.accounts) are not judged here.
The messenger asked seat:node-engine.root-free@<machine>, which nothing on the mesh answers (the engine's acts are
left out of its declared state, mesh-host #178; ADR 0571 §11's self-check probe is not held either), and deleted
its kept word on each failure without a log line: every ask from an agent or a merge was held, silently.
Quick fix: point RootFreeVerb back at mesh-controller.root-free. Thorough fix: the same, plus every failed ask is
logged with its error, the failure is kept beside the word and becomes the held ask's reason, a fresh word is no
longer thrown away by one failed ask (an answer is still judged only on a word asked then), and a held ask whose
reason changes is logged. Chose thorough: the silence is what hid the fault for a day.
novox since 15:36 (nats's built manifest) and ace since 15:37 (baserow's config.json) had no plans at all, so the
vault refused every secret their clients asked. The holder now composes with frombus.PlansHeld: a module whose build
or plan refuses is held, its why kept in the new state refused and read by provisioner_plans, its last plan and its
provisions left as they were, and every other module on the node composed and written. Pins mesh-sdk 59311407.
mesh/check checking 67dcf252ecfa on 82ac613b4ed0 | novox/mesh-catalog#564 in group fix/credential-forms-one-list
Since 13:55 the client refused every plan with credential-form none
(gitea's forge-account, the artifact store), and g14 and shanks placed no
files for mesh-build-go, mesh-build-node, mesh-build-image or ssh-client.
Second meeting of the credential-form list kind: the controller's copy was
fixed alone before. Thorough: one list in mesh-sdk; this client drops its
copy and pins mesh-sdk 9379e9a. A none entry is never a holder's issued
value: its bound values come from the provider's entry, and a secret named
for it fails loudly instead of waiting on a vault that mints nothing.
mesh/delivery-group group feat/774-l3-reland rejected: composed together: the checker's verdict at the members' heads (b48c90fea31f): integration: mesh-self-ch…
mesh/check failed: integration: mesh-self-check/cmd/mesh-self-check.TestTheSelfCheckWholeRunsOnTheBus: the run answered {Run:doctor-1793351737-1 At:20…
One commit with exactly the tree of the reviewed head 35b4509c0a, on main 12c2574c9d, so two compiled programs added and removed on the branch leave no trace in history (issue 1577).
mesh/check pass 96a37f58ca76 on 030f9ee4e780 | unit 1173 run, 3272 kept, 13 skipped | integration 1212 run, 3274 kept, 22 skipped | over bound: unit, …
The controller's declared-config mirror never held de-spiegel's keys, and Resolve skipped
an assigned module it held no declarations of, silently: de-spiegel stayed held on novox
with its values set. Declarations now come from the indexer's trunk manifest with the
wanted build's laid over it per node; which module runs where from the seat
mesh-placement alone. A placed module with no readable manifest is listed as unresolved
with why, logged, and keeps its last resolved values.
mesh/check pass add6f143ca23 on 158348ede789 | unit 1601 run, 5122 kept, 47 skipped | integration 1637 run, 5215 kept, 49 skipped | over bound: unit, …
On 2026-10-29 the group feat/774-vault-mints-l2 was approved and never handed: hq#495 kept a Decline of an earlier ask beside the Approve of the next, readyAsks skipped it and handTurn returned nil for every member, with no log line. The Approve and a new ask now clear an earlier Decline, a run's answer names its ask (answered_ask), an approved merge that waits for nothing and is not handed logs why once and is a stalled line, and delivery_turns lists no unwritten turn while the check gates no merge.
jschoubben
mesh-tools