mesh-provisioner, mesh-config, mesh-issues and mesh-healers each wrote a watch-again loop (`watched`). mesh-sdk
61b67d61 adds KeptState.Follow, which asks again with backoff and says each give-up, and its watches now answer a
change at once and run the handler beside the runtime (the provisioner's "names no watch of this bundle"). The four
modules use it; their loops are gone. Built and tested against the mesh-sdk group head through a go.work that is not
committed: their mesh-sdk pins move to that head with the group.
The client dated each report with the time it was said, and says a failure again whenever its words change, so the
controller's apply-failing alarm, which reads that time, saw a new failure: on novox the vault's refusal replaced a
bus refusal at 16:56 and the alarm cleared for five minutes while every module still failed.
Quick fix: the controller keeps a raised alarm while any failure stands (done there). Thorough fix: the report says
failing-since, kept from the module's last report (this run's, else the applied bucket's) while it stays failed,
whatever its reason says; any other status ends it. Chose both: the controller can now judge the failure's age
rather than its newest words, which is its change to make.
move-in's refusal "the vault already holds another value for ..." was an error's text, which the node-engine and the
controller could only quote. It is now an answer: `refused` with the reason holds-another-value and the held value's
record (generation, changed by, changed at, origin; never the value), `why` in words, and no read-back, so an
engine that does not read `refused` still fails as before ("answered with no read-back"). keepLocked's refusal is a
typed error (HeldAnother) whose text is unchanged for every other path.
Quick fix: a stable prefix in the text. Thorough fix: the fields. Chose thorough, as asked; mesh-host's
moveIntoTheVault has to read `refused` to use them.
The three random values kept before the vault judged them (own/novox/nats/broker-nats-tools,
own/novox/mesh-controller/broker-address and broker-management) are held, so minting refused them ("held already")
and their users stay off the bus. A value that is no account has no hash and logs in nowhere, so replacing it
breaks no login the grants part composed. When the grants part wants the user (every want is heard at the vault's
start), the vault now mints an account in its place, judged again under the store's lock, keeping the value it
replaces beside it. A held account is still never minted anew (ADR 0561 §4).
Quick fix: let an operator rotate the three by hand. Thorough fix: the mint path replaces any such value, so no
other one stays behind. Chose thorough.
own/novox/nats/broker-nats-tools (moved in today) and mesh-controller's broker-address and broker-management (moved in
from the controller's store) are random values, not the account JSON a bus user is hashed from, so the hash step says
"not JSON" and the grants part leaves those users off the bus. The controller made them (fixed there, c421629f); the
vault kept them, since nothing it keeps judged a covered name's value.
Quick fix: refuse a plain value in move-in. Thorough fix: keepLocked, which every way into the store passes, refuses
for any covered name (broker, provisioner-broker, broker-<binary>, which catalogue.ToolsBusSecret names) a value that
is no account, saying "not issued". Chose thorough: one rule where every value is kept.
gitea asked an owed status again every 30 s forever, even on 403 and 404 (jschoubben/hal and jschoubben/novadock, 119
times each in the sweep), and a key the checker deleted did not end a loop already running.
Quick fix: return from askAgain on 403/404. Thorough fix: the same, said once with its cause; a delete of the key
marks its revision seen, so any running ask again ends before its next write; a newer revision still asks anew.
The checker no longer owes statuses outside the mesh (mesh-delivery), which removes the trigger. Not done: raising
the module's health, since gitea's TypeScript SDK has no seat publish for a condition (mesh-sdk).
pair.go kept its own copy of the forms a plan may name; the copy in the provisioner client went in #564, after it
refused plans naming `none` while the vault's copy took them. Quick fix and thorough fix are the same here: the vault's
constants are mesh-sdk's (catalogue.CredentialForm*), and a form is judged by catalogue.IsCredentialForm, so a new
form is added in one place. No new test: behaviour is unchanged, and the existing form tests pass on the shared list.
mesh-vault's mesh-sdk pin moves to 9379e9a9, the provisioner client's, the first with the list.
i3status-rust on g14 and shanks cannot watch power.draw ("was issued: none", 148 times): its read was granted from
the bar contribution's data (shows: state, options.state; catalogue.ReadsGranted), and design 57 L3 moved the block
to an item, whose attributes said nothing of it, while ReadsGranted reads contributions alone.
Quick fix: `reads: power.draw` on i3status-rust, which grants every machine's draw instead of its own key (ADR 0260).
Thorough fix: the item says its read (shows: state, state: power.draw), and mesh-sdk's ReadsGranted grants a held
item's read as it grants a contribution's, keyed to the machine. Chose thorough; its mesh-sdk half is another
repository's, and until it lands the fault stays.
Grafana on ace answers 401 to the vault's admin password, every 30 s since at least 2026-10-05. I think the cause is
the adoption: Grafana keeps the admin password it was first given in its database and applies
GF_SECURITY_ADMIN_PASSWORD__FILE only when it creates it; ace's Grafana came with its database before the mesh minted
own/ace/grafana/admin (2026-09-30, generation 1, unchanged since), so the minted value never took (keycloak's
admin, hq issue 179, is the same kind).
Quick fix: none in code; the password in Grafana's database must become the vault's (grafana cli admin
reset-admin-password in the container), an act on ace no tool offers yet. Thorough fix: a step that converges the
admin password at every start, which needs a container entrypoint the node-engine's declaration does not carry.
This commit does what the catalogue can now: the refusal is said once with its cause and what ends it, asked again
hourly, and its end is said.
The sweep took the node seat node-engine's root-free for one nothing answers; the engines had been refused their
subscriptions by the bus since 12:10 (fixed in mesh-host), so the address was right. This puts it back and keeps
what the previous commit added: every failed ask said with its error, a fresh word not thrown away, and the failure
as the held ask's reason.
The file gitea-read.secret was declared with no owner, so it was root's at 0600; claude-code's tools run in the
node's runtime as the operator's account and could not read it ("permission denied", ~870 times a machine), said
only in the journal. A file a bundle reads is the runtime account's only when a word of the bundle names it
(catalogue givenTo); this one was found through MESH_CLAUDE_CODE_STATE alone.
Quick fix: name the file with the bundle word MESH_CLAUDE_CODE_FORGE, which the reader now uses. Thorough fix: the
same, plus a failure to place the credential raises a warning condition of the module on the operator-channel seat,
once per reason, cleared when it places again. Chose thorough: the fault ran unseen for days. A change of the
credential now restarts the node's runtime, as for every file a bundle word names.
claude-code's mesh-sdk pin moves to 2da0de75 (main's), the first with asks.Condition.
The controller grants a read only from the owner's announcement (ADR 0495 clause 3). gitea announced `bound`
unreadable, so mesh-provisioner bound no forge account; disk-load announced `readings` unreadable, refused to restic,
nfs-server and mounts; restic announced `good` unreadable, which the self-check reads. Nothing in the catalogue
compared readers with announcements since the gate's manifest check went (clause 5).
Quick fix: mark those three states readable. Thorough fix: also checks/readable, which reads every owner's Go and
TypeScript announcement and refuses any `reads` of a state it does not announce readable, so the next owner to
announce cannot repeat it. Chose thorough: the same kind hit three owners at once.
Reads of other repositories' owners (mesh-controller.lease, mesh-grants.accounts) are not judged here.
The messenger asked seat:node-engine.root-free@<machine>, which nothing on the mesh answers (the engine's acts are
left out of its declared state, mesh-host #178; ADR 0571 §11's self-check probe is not held either), and deleted
its kept word on each failure without a log line: every ask from an agent or a merge was held, silently.
Quick fix: point RootFreeVerb back at mesh-controller.root-free. Thorough fix: the same, plus every failed ask is
logged with its error, the failure is kept beside the word and becomes the held ask's reason, a fresh word is no
longer thrown away by one failed ask (an answer is still judged only on a word asked then), and a held ask whose
reason changes is logged. Chose thorough: the silence is what hid the fault for a day.
novox since 15:36 (nats's built manifest) and ace since 15:37 (baserow's config.json) had no plans at all, so the
vault refused every secret their clients asked. The holder now composes with frombus.PlansHeld: a module whose build
or plan refuses is held, its why kept in the new state refused and read by provisioner_plans, its last plan and its
provisions left as they were, and every other module on the node composed and written. Pins mesh-sdk 59311407.
mesh/check checking 67dcf252ecfa on 82ac613b4ed0 | novox/mesh-catalog#564 in group fix/credential-forms-one-list
Since 13:55 the client refused every plan with credential-form none
(gitea's forge-account, the artifact store), and g14 and shanks placed no
files for mesh-build-go, mesh-build-node, mesh-build-image or ssh-client.
Second meeting of the credential-form list kind: the controller's copy was
fixed alone before. Thorough: one list in mesh-sdk; this client drops its
copy and pins mesh-sdk 9379e9a. A none entry is never a holder's issued
value: its bound values come from the provider's entry, and a secret named
for it fails loudly instead of waiting on a vault that mints nothing.
mesh/check being carried to d6a069ae4959 on 1aa22efb0d78 | from 12c2574c9d54
On 2026-10-29 the group feat/774-vault-mints-l2 was approved and never handed: hq#495 kept a Decline of an earlier ask beside the Approve of the next, readyAsks skipped it and handTurn returned nil for every member, with no log line. The Approve and a new ask now clear an earlier Decline, a run's answer names its ask (answered_ask), an approved merge that waits for nothing and is not handed logs why once and is a stalled line, and delivery_turns lists no unwritten turn while the check gates no merge.
mesh/delivery-group group feat/774-l3-reland rejected: composed together: the checker's verdict at the members' heads (b48c90fea31f): integration: mesh-self-ch…
mesh/check failed: integration: mesh-self-check/cmd/mesh-self-check.TestTheSelfCheckWholeRunsOnTheBus: the run answered {Run:doctor-1793351737-1 At:20…
One commit with exactly the tree of the reviewed head 35b4509c0a, on main 12c2574c9d, so two compiled programs added and removed on the branch leave no trace in history (issue 1577).
mesh/check pass 96a37f58ca76 on 030f9ee4e780 | unit 1173 run, 3272 kept, 13 skipped | integration 1212 run, 3274 kept, 22 skipped | over bound: unit, …
The controller's declared-config mirror never held de-spiegel's keys, and Resolve skipped
an assigned module it held no declarations of, silently: de-spiegel stayed held on novox
with its values set. Declarations now come from the indexer's trunk manifest with the
wanted build's laid over it per node; which module runs where from the seat
mesh-placement alone. A placed module with no readable manifest is listed as unresolved
with why, logged, and keeps its last resolved values.
mesh/check pass add6f143ca23 on 158348ede789 | unit 1601 run, 5122 kept, 47 skipped | integration 1637 run, 5215 kept, 49 skipped | over bound: unit, …
On 2026-10-29 the group feat/774-vault-mints-l2 was approved and never handed: hq#495 kept a Decline of an earlier ask beside the Approve of the next, readyAsks skipped it and handTurn returned nil for every member, with no log line. The Approve and a new ask now clear an earlier Decline, a run's answer names its ask (answered_ask), an approved merge that waits for nothing and is not handed logs why once and is a stalled line, and delivery_turns lists no unwritten turn while the check gates no merge.
jschoubben
mesh-tools