The provisioner holds only a bad module, the vault keeps bus accounts by declaration, and silent faults from the release sweep speak up (issue 774)
#565
Merged
jschoubbenmerged 18 commits from fix/sweep-2026-10-30 into main2026-10-30 18:12:53 +00:00
mesh-provisioner, mesh-config, mesh-issues and mesh-healers need KeptState.Follow; mesh-build-go, -image and -node the
retry of a state write the bus delivered to no one; claude-code asks.Condition; mesh-vault the credential forms list.
Each passes go vet, go vet -tags integration and go test on this pin, built with no go.work.
683fd547 and b7f87822 took every `broker-<id>` own secret for a bus account. mesh-controller's broker-address is the
bus's address (MESH_BROKER_ADDRESS_FILE, read by broker.FromEnvironment to issue enrolment tokens and credentials):
c1 would have refused to keep it, and it was counted among the "owed bus-account hashes" every minute.
A name is now a bus account when it is declared: `broker` or `provisioner-broker`, or the own secret of an account the
grants part wants, which it composes from the manifests' bus processes (what catalogue.ToolsBusSecret names). A
`broker-<id>` holding an account the controller issued before its want was heard is still hashed. Anything else is
kept as given, never refused, never minted over, never hashed and never owed. The mint path replaced only wanted
names already, so the address was never at risk of replacement there; the test now proves it.
Quick fix: except broker-address by name. Thorough fix: the declaration decides, for every name. Chose thorough.
Table rows: broker-address kept as given, not replaced, owed nothing; broker-nats-tools replaced by a minted account
once wanted and refused a plain value after. They fail on the previous code.
mesh-provisioner, mesh-config, mesh-issues and mesh-healers each wrote a watch-again loop (`watched`). mesh-sdk
61b67d61 adds KeptState.Follow, which asks again with backoff and says each give-up, and its watches now answer a
change at once and run the handler beside the runtime (the provisioner's "names no watch of this bundle"). The four
modules use it; their loops are gone. Built and tested against the mesh-sdk group head through a go.work that is not
committed: their mesh-sdk pins move to that head with the group.
The client dated each report with the time it was said, and says a failure again whenever its words change, so the
controller's apply-failing alarm, which reads that time, saw a new failure: on novox the vault's refusal replaced a
bus refusal at 16:56 and the alarm cleared for five minutes while every module still failed.
Quick fix: the controller keeps a raised alarm while any failure stands (done there). Thorough fix: the report says
failing-since, kept from the module's last report (this run's, else the applied bucket's) while it stays failed,
whatever its reason says; any other status ends it. Chose both: the controller can now judge the failure's age
rather than its newest words, which is its change to make.
move-in's refusal "the vault already holds another value for ..." was an error's text, which the node-engine and the
controller could only quote. It is now an answer: `refused` with the reason holds-another-value and the held value's
record (generation, changed by, changed at, origin; never the value), `why` in words, and no read-back, so an
engine that does not read `refused` still fails as before ("answered with no read-back"). keepLocked's refusal is a
typed error (HeldAnother) whose text is unchanged for every other path.
Quick fix: a stable prefix in the text. Thorough fix: the fields. Chose thorough, as asked; mesh-host's
moveIntoTheVault has to read `refused` to use them.
The three random values kept before the vault judged them (own/novox/nats/broker-nats-tools,
own/novox/mesh-controller/broker-address and broker-management) are held, so minting refused them ("held already")
and their users stay off the bus. A value that is no account has no hash and logs in nowhere, so replacing it
breaks no login the grants part composed. When the grants part wants the user (every want is heard at the vault's
start), the vault now mints an account in its place, judged again under the store's lock, keeping the value it
replaces beside it. A held account is still never minted anew (ADR 0561 §4).
Quick fix: let an operator rotate the three by hand. Thorough fix: the mint path replaces any such value, so no
other one stays behind. Chose thorough.
own/novox/nats/broker-nats-tools (moved in today) and mesh-controller's broker-address and broker-management (moved in
from the controller's store) are random values, not the account JSON a bus user is hashed from, so the hash step says
"not JSON" and the grants part leaves those users off the bus. The controller made them (fixed there, c421629f); the
vault kept them, since nothing it keeps judged a covered name's value.
Quick fix: refuse a plain value in move-in. Thorough fix: keepLocked, which every way into the store passes, refuses
for any covered name (broker, provisioner-broker, broker-<binary>, which catalogue.ToolsBusSecret names) a value that
is no account, saying "not issued". Chose thorough: one rule where every value is kept.
gitea asked an owed status again every 30 s forever, even on 403 and 404 (jschoubben/hal and jschoubben/novadock, 119
times each in the sweep), and a key the checker deleted did not end a loop already running.
Quick fix: return from askAgain on 403/404. Thorough fix: the same, said once with its cause; a delete of the key
marks its revision seen, so any running ask again ends before its next write; a newer revision still asks anew.
The checker no longer owes statuses outside the mesh (mesh-delivery), which removes the trigger. Not done: raising
the module's health, since gitea's TypeScript SDK has no seat publish for a condition (mesh-sdk).
pair.go kept its own copy of the forms a plan may name; the copy in the provisioner client went in #564, after it
refused plans naming `none` while the vault's copy took them. Quick fix and thorough fix are the same here: the vault's
constants are mesh-sdk's (catalogue.CredentialForm*), and a form is judged by catalogue.IsCredentialForm, so a new
form is added in one place. No new test: behaviour is unchanged, and the existing form tests pass on the shared list.
mesh-vault's mesh-sdk pin moves to 9379e9a9, the provisioner client's, the first with the list.
i3status-rust on g14 and shanks cannot watch power.draw ("was issued: none", 148 times): its read was granted from
the bar contribution's data (shows: state, options.state; catalogue.ReadsGranted), and design 57 L3 moved the block
to an item, whose attributes said nothing of it, while ReadsGranted reads contributions alone.
Quick fix: `reads: power.draw` on i3status-rust, which grants every machine's draw instead of its own key (ADR 0260).
Thorough fix: the item says its read (shows: state, state: power.draw), and mesh-sdk's ReadsGranted grants a held
item's read as it grants a contribution's, keyed to the machine. Chose thorough; its mesh-sdk half is another
repository's, and until it lands the fault stays.
Grafana on ace answers 401 to the vault's admin password, every 30 s since at least 2026-10-05. I think the cause is
the adoption: Grafana keeps the admin password it was first given in its database and applies
GF_SECURITY_ADMIN_PASSWORD__FILE only when it creates it; ace's Grafana came with its database before the mesh minted
own/ace/grafana/admin (2026-09-30, generation 1, unchanged since), so the minted value never took (keycloak's
admin, hq issue 179, is the same kind).
Quick fix: none in code; the password in Grafana's database must become the vault's (grafana cli admin
reset-admin-password in the container), an act on ace no tool offers yet. Thorough fix: a step that converges the
admin password at every start, which needs a container entrypoint the node-engine's declaration does not carry.
This commit does what the catalogue can now: the refusal is said once with its cause and what ends it, asked again
hourly, and its end is said.
The sweep took the node seat node-engine's root-free for one nothing answers; the engines had been refused their
subscriptions by the bus since 12:10 (fixed in mesh-host), so the address was right. This puts it back and keeps
what the previous commit added: every failed ask said with its error, a fresh word not thrown away, and the failure
as the held ask's reason.
The file gitea-read.secret was declared with no owner, so it was root's at 0600; claude-code's tools run in the
node's runtime as the operator's account and could not read it ("permission denied", ~870 times a machine), said
only in the journal. A file a bundle reads is the runtime account's only when a word of the bundle names it
(catalogue givenTo); this one was found through MESH_CLAUDE_CODE_STATE alone.
Quick fix: name the file with the bundle word MESH_CLAUDE_CODE_FORGE, which the reader now uses. Thorough fix: the
same, plus a failure to place the credential raises a warning condition of the module on the operator-channel seat,
once per reason, cleared when it places again. Chose thorough: the fault ran unseen for days. A change of the
credential now restarts the node's runtime, as for every file a bundle word names.
claude-code's mesh-sdk pin moves to 2da0de75 (main's), the first with asks.Condition.
The controller grants a read only from the owner's announcement (ADR 0495 clause 3). gitea announced `bound`
unreadable, so mesh-provisioner bound no forge account; disk-load announced `readings` unreadable, refused to restic,
nfs-server and mounts; restic announced `good` unreadable, which the self-check reads. Nothing in the catalogue
compared readers with announcements since the gate's manifest check went (clause 5).
Quick fix: mark those three states readable. Thorough fix: also checks/readable, which reads every owner's Go and
TypeScript announcement and refuses any `reads` of a state it does not announce readable, so the next owner to
announce cannot repeat it. Chose thorough: the same kind hit three owners at once.
Reads of other repositories' owners (mesh-controller.lease, mesh-grants.accounts) are not judged here.
The messenger asked seat:node-engine.root-free@<machine>, which nothing on the mesh answers (the engine's acts are
left out of its declared state, mesh-host #178; ADR 0571 §11's self-check probe is not held either), and deleted
its kept word on each failure without a log line: every ask from an agent or a merge was held, silently.
Quick fix: point RootFreeVerb back at mesh-controller.root-free. Thorough fix: the same, plus every failed ask is
logged with its error, the failure is kept beside the word and becomes the held ask's reason, a fresh word is no
longer thrown away by one failed ask (an answer is still judged only on a word asked then), and a held ask whose
reason changes is logged. Chose thorough: the silence is what hid the fault for a day.
novox since 15:36 (nats's built manifest) and ace since 15:37 (baserow's config.json) had no plans at all, so the
vault refused every secret their clients asked. The holder now composes with frombus.PlansHeld: a module whose build
or plan refuses is held, its why kept in the new state refused and read by provisioner_plans, its last plan and its
provisions left as they were, and every other module on the node composed and written. Pins mesh-sdk 59311407.
jschoubben