The provisioner holds only a bad module, the vault keeps bus accounts by declaration, and silent faults from the release sweep speak up (issue 774) #565

2026-10-30T17:45:38Z

mesh-tools commented

2026-10-30 17:45:38 +00:00

mesh-provisioner composes each module on its own; a held module keeps its last plan and is named in state refused.
messenger logs a failed root-free ask and keeps a fresh word; a held ask names its real reason.
Readable announcements: gitea bound, disk-load readings, restic good; new check checks/readable refuses a read the owner's code does not announce.
claude-code reads its forge credential through a bundle word; a failure raises a condition.
gitea stops asking again on 403/404 (hal, novadock); grafana logs its 401's cause once and asks hourly (the admin password reset is a separate act).
mesh-vault: a bus account only when declared (never by broker- prefix; broker-address kept); a held non-account for a wanted account is replaced by a minted one; the move-in refusal is structured; one credential forms list.
mesh-provisioner-client reports failing-since.
power bar item says state: power.draw; mesh-provisioner, mesh-config, mesh-issues and mesh-healers use KeptState.Follow.
Removes a committed mesh-vault binary. Nine modules pinned to mesh-sdk a3718126.

Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch fix/sweep-2026-10-30 in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24).
Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix.

Reviewed-by: rev-sweep@shanks at 2455445dab

- **mesh-provisioner** composes each module on its own; a held module keeps its last plan and is named in state `refused`. - **messenger** logs a failed root-free ask and keeps a fresh word; a held ask names its real reason. - **Readable announcements**: gitea `bound`, disk-load `readings`, restic `good`; new check `checks/readable` refuses a read the owner's code does not announce. - **claude-code** reads its forge credential through a bundle word; a failure raises a condition. - **gitea** stops asking again on 403/404 (hal, novadock); **grafana** logs its 401's cause once and asks hourly (the admin password reset is a separate act). - **mesh-vault**: a bus account only when declared (never by `broker-` prefix; broker-address kept); a held non-account for a wanted account is replaced by a minted one; the move-in refusal is structured; one credential forms list. - **mesh-provisioner-client** reports `failing-since`. - **power** bar item says `state: power.draw`; mesh-provisioner, mesh-config, mesh-issues and mesh-healers use `KeptState.Follow`. - Removes a committed mesh-vault binary. Nine modules pinned to mesh-sdk a3718126. --- Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch `fix/sweep-2026-10-30` in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24). Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix. Reviewed-by: rev-sweep@shanks at 2455445dabee79a34b67bc20c9dea1e754d88497

mesh-tools added 18 commits 2026-10-30 17:45:38 +00:00

One bad module must not drop its node: the provisioner composes each module's plan on its own (issue 774) 6d67c22bf3

novox since 15:36 (nats's built manifest) and ace since 15:37 (baserow's config.json) had no plans at all, so the
vault refused every secret their clients asked. The holder now composes with frombus.PlansHeld: a module whose build
or plan refuses is held, its why kept in the new state refused and read by provisioner_plans, its last plan and its
provisions left as they were, and every other module on the node composed and written. Pins mesh-sdk 59311407.

Ask the root-free word of the controller's verb, which answers, and say every failed ask, so Telegram carries asks that offer an approval again (issue 774) bc27189627

The messenger asked seat:node-engine.root-free@<machine>, which nothing on the mesh answers (the engine's acts are
left out of its declared state, mesh-host #178; ADR 0571 §11's self-check probe is not held either), and deleted
its kept word on each failure without a log line: every ask from an agent or a merge was held, silently.

Quick fix: point RootFreeVerb back at mesh-controller.root-free. Thorough fix: the same, plus every failed ask is
logged with its error, the failure is kept beside the word and becomes the held ask's reason, a fresh word is no
longer thrown away by one failed ask (an answer is still judged only on a word asked then), and a held ask whose
reason changes is logged. Chose thorough: the silence is what hid the fault for a day.

Announce every state another module reads as readable, and refuse a catalogue that does not (issue 774) aec2ed3c24

The controller grants a read only from the owner's announcement (ADR 0495 clause 3). gitea announced `bound`
unreadable, so mesh-provisioner bound no forge account; disk-load announced `readings` unreadable, refused to restic,
nfs-server and mounts; restic announced `good` unreadable, which the self-check reads. Nothing in the catalogue
compared readers with announcements since the gate's manifest check went (clause 5).

Quick fix: mark those three states readable. Thorough fix: also checks/readable, which reads every owner's Go and
TypeScript announcement and refuses any `reads` of a state it does not announce readable, so the next owner to
announce cannot repeat it. Chose thorough: the same kind hit three owners at once.
Reads of other repositories' owners (mesh-controller.lease, mesh-grants.accounts) are not judged here.

Give claude-code's runtime its forge credential file, and say a credential it cannot place as a condition (issue 774) 7531c3257d

The file gitea-read.secret was declared with no owner, so it was root's at 0600; claude-code's tools run in the
node's runtime as the operator's account and could not read it ("permission denied", ~870 times a machine), said
only in the journal. A file a bundle reads is the runtime account's only when a word of the bundle names it
(catalogue givenTo); this one was found through MESH_CLAUDE_CODE_STATE alone.

Quick fix: name the file with the bundle word MESH_CLAUDE_CODE_FORGE, which the reader now uses. Thorough fix: the
same, plus a failure to place the credential raises a warning condition of the module on the operator-channel seat,
once per reason, cleared when it places again. Chose thorough: the fault ran unseen for days. A change of the
credential now restarts the node's runtime, as for every file a bundle word names.
claude-code's mesh-sdk pin moves to 2da0de75 (main's), the first with asks.Condition.

Ask each machine's engine for the root-free word again: the engines were deaf, not absent (issue 774) 2d28a52d61

The sweep took the node seat node-engine's root-free for one nothing answers; the engines had been refused their
subscriptions by the bus since 12:10 (fixed in mesh-host), so the address was right. This puts it back and keeps
what the previous commit added: every failed ask said with its error, a fresh word not thrown away, and the failure
as the held ask's reason.

Say grafana's refused admin password once with its cause, and ask again hourly, not every 30 s (issue 774) 0e6ca453be

Grafana on ace answers 401 to the vault's admin password, every 30 s since at least 2026-10-05. I think the cause is
the adoption: Grafana keeps the admin password it was first given in its database and applies
GF_SECURITY_ADMIN_PASSWORD__FILE only when it creates it; ace's Grafana came with its database before the mesh minted
own/ace/grafana/admin (2026-09-30, generation 1, unchanged since), so the minted value never took (keycloak's
admin, hq issue 179, is the same kind).

Quick fix: none in code; the password in Grafana's database must become the vault's (grafana cli admin
reset-admin-password in the container), an act on ace no tool offers yet. Thorough fix: a step that converges the
admin password at every start, which needs a container entrypoint the node-engine's declaration does not carry.
This commit does what the catalogue can now: the refusal is said once with its cause and what ends it, asked again
hourly, and its end is said.

Say on power's draw item which state its bar block reads, so the bar's holder can be granted it (issue 774) ca8d6f33a5

i3status-rust on g14 and shanks cannot watch power.draw ("was issued: none", 148 times): its read was granted from
the bar contribution's data (shows: state, options.state; catalogue.ReadsGranted), and design 57 L3 moved the block
to an item, whose attributes said nothing of it, while ReadsGranted reads contributions alone.

Quick fix: `reads: power.draw` on i3status-rust, which grants every machine's draw instead of its own key (ADR 0260).
Thorough fix: the item says its read (shows: state, state: power.draw), and mesh-sdk's ReadsGranted grants a held
item's read as it grants a contribution's, keyed to the machine. Chose thorough; its mesh-sdk half is another
repository's, and until it lands the fault stays.

Read the vault's credential forms from mesh-sdk's one list, as the provisioner client does (issue 774) 2066021fca

pair.go kept its own copy of the forms a plan may name; the copy in the provisioner client went in #564, after it
refused plans naming `none` while the vault's copy took them. Quick fix and thorough fix are the same here: the vault's
constants are mesh-sdk's (catalogue.CredentialForm*), and a form is judged by catalogue.IsCredentialForm, so a new
form is added in one place. No new test: behaviour is unchanged, and the existing form tests pass on the shared list.
mesh-vault's mesh-sdk pin moves to 9379e9a9, the provisioner client's, the first with the list.

Stop asking the forge again for a mesh/check it refuses for good, and end the ask when its key is deleted (issue 774) f52545c4ba

gitea asked an owed status again every 30 s forever, even on 403 and 404 (jschoubben/hal and jschoubben/novadock, 119
times each in the sweep), and a key the checker deleted did not end a loop already running.

Quick fix: return from askAgain on 403/404. Thorough fix: the same, said once with its cause; a delete of the key
marks its revision seen, so any running ask again ends before its next write; a newer revision still asks anew.
The checker no longer owes statuses outside the mesh (mesh-delivery), which removes the trigger. Not done: raising
the module's health, since gitea's TypeScript SDK has no seat publish for a condition (mesh-sdk).

Keep a bus account in the vault only as an account: a random value for a broker name is not issued (issue 774) 683fd547b2

own/novox/nats/broker-nats-tools (moved in today) and mesh-controller's broker-address and broker-management (moved in
from the controller's store) are random values, not the account JSON a bus user is hashed from, so the hash step says
"not JSON" and the grants part leaves those users off the bus. The controller made them (fixed there, c421629f); the
vault kept them, since nothing it keeps judged a covered name's value.

Quick fix: refuse a plain value in move-in. Thorough fix: keepLocked, which every way into the store passes, refuses
for any covered name (broker, provisioner-broker, broker-<binary>, which catalogue.ToolsBusSecret names) a value that
is no account, saying "not issued". Chose thorough: one rule where every value is kept.

Replace a held bus-account value that is no account by a minted account when its user is wanted (issue 774) b7f878226e

The three random values kept before the vault judged them (own/novox/nats/broker-nats-tools,
own/novox/mesh-controller/broker-address and broker-management) are held, so minting refused them ("held already")
and their users stay off the bus. A value that is no account has no hash and logs in nowhere, so replacing it
breaks no login the grants part composed. When the grants part wants the user (every want is heard at the vault's
start), the vault now mints an account in its place, judged again under the store's lock, keeping the value it
replaces beside it. A held account is still never minted anew (ADR 0561 §4).

Quick fix: let an operator rotate the three by hand. Thorough fix: the mint path replaces any such value, so no
other one stays behind. Chose thorough.

Answer a move the vault refuses for holding another value with fields, not words alone (issue 774) f3b13de20d

move-in's refusal "the vault already holds another value for ..." was an error's text, which the node-engine and the
controller could only quote. It is now an answer: `refused` with the reason holds-another-value and the held value's
record (generation, changed by, changed at, origin; never the value), `why` in words, and no read-back, so an
engine that does not read `refused` still fails as before ("answered with no read-back"). keepLocked's refusal is a
typed error (HeldAnother) whose text is unchanged for every other path.

Quick fix: a stable prefix in the text. Thorough fix: the fields. Chose thorough, as asked; mesh-host's
moveIntoTheVault has to read `refused` to use them.

Say on a failed provisioning report when the failure began, kept while only its words change (issue 774) 3aae6c0890

The client dated each report with the time it was said, and says a failure again whenever its words change, so the
controller's apply-failing alarm, which reads that time, saw a new failure: on novox the vault's refusal replaced a
bus refusal at 16:56 and the alarm cleared for five minutes while every module still failed.

Quick fix: the controller keeps a raised alarm while any failure stands (done there). Thorough fix: the report says
failing-since, kept from the module's last report (this run's, else the applied bucket's) while it stays failed,
whatever its reason says; any other status ends it. Chose both: the controller can now judge the failure's age
rather than its newest words, which is its change to make.

Follow the runtime's watches through mesh-sdk's KeptState.Follow, not four loops of their own (issue 774) 21c86c819c

mesh-provisioner, mesh-config, mesh-issues and mesh-healers each wrote a watch-again loop (`watched`). mesh-sdk
61b67d61 adds KeptState.Follow, which asks again with backoff and says each give-up, and its watches now answer a
change at once and run the handler beside the runtime (the provisioner's "names no watch of this bundle"). The four
modules use it; their loops are gone. Built and tested against the mesh-sdk group head through a go.work that is not
committed: their mesh-sdk pins move to that head with the group.

Judge a bus account by its declaration, never by a broker- prefix, so the controller's address is kept (issue 774) 434214e615

683fd547 and b7f87822 took every `broker-<id>` own secret for a bus account. mesh-controller's broker-address is the
bus's address (MESH_BROKER_ADDRESS_FILE, read by broker.FromEnvironment to issue enrolment tokens and credentials):
c1 would have refused to keep it, and it was counted among the "owed bus-account hashes" every minute.

A name is now a bus account when it is declared: `broker` or `provisioner-broker`, or the own secret of an account the
grants part wants, which it composes from the manifests' bus processes (what catalogue.ToolsBusSecret names). A
`broker-<id>` holding an account the controller issued before its want was heard is still hashed. Anything else is
kept as given, never refused, never minted over, never hashed and never owed. The mint path replaced only wanted
names already, so the address was never at risk of replacement there; the test now proves it.

Quick fix: except broker-address by name. Thorough fix: the declaration decides, for every name. Chose thorough.
Table rows: broker-address kept as given, not replaced, owed nothing; broker-nats-tools replaced by a minted account
once wanted and refused a plain value after. They fail on the previous code.

Pin nine modules to the fix group's mesh-sdk, 206f794e, which carries Follow, the bus retry and the forms list (issue 774) 647c5c3c06

mesh-provisioner, mesh-config, mesh-issues and mesh-healers need KeptState.Follow; mesh-build-go, -image and -node the
retry of a state write the bus delivered to no one; claude-code asks.Condition; mesh-vault the credential forms list.
Each passes go vet, go vet -tags integration and go test on this pin, built with no go.work.

Take the compiled mesh-vault program out of the tree and ignore it, since build output never belongs in the catalogue (issue 774) bc433b8304

Pin mesh-sdk a3718126, whose bus retry never asks again a call its callee answered (issue 774)

mesh/delivery merged: source.main moved to novox/mesh-catalog@7e905a66944e

Details

mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged

Details

mesh/check checking 2455445dabee on 72cee59acc32 | novox/mesh-catalog#565 in group fix/sweep-2026-10-30

2455445dab

mesh-tools commented

2026-10-30 17:45:53 +00:00

Delivery novox/mesh-catalog@2455445dabee — merged since 2026-10-30T18:12:56Z

Source — source.main moved to novox/mesh-catalog@7e905a6694. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Group fix/sweep-2026-10-30: novox/mesh-catalog@2455445dab, novox/mesh-delivery@e0b56c83f9, novox/mesh-host@b9c24a7b42, novox/mesh-sdk@a37181261c, novox/mesh-tools@316a5a2fc3

composed together: fail — the checker's verdict at the members' heads (1e1a3f6ab540): shanks

Transitions

2026-10-30 17:45 (new) → proposed (announced): novox/mesh-catalog#565's head announced
2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-30 18:12 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@2455445dabee` — **merged** since 2026-10-30T18:12:56Z **Source** — source.main moved to novox/mesh-catalog@7e905a66944e. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Group** `fix/sweep-2026-10-30`: novox/mesh-catalog@2455445dabee, novox/mesh-delivery@e0b56c83f94c, novox/mesh-host@b9c24a7b42cb, novox/mesh-sdk@a37181261c24, novox/mesh-tools@316a5a2fc317 - composed together: fail — the checker's verdict at the members' heads (1e1a3f6ab540): shanks **Transitions** - 2026-10-30 17:45 (new) → proposed (announced): novox/mesh-catalog#565's head announced - 2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-30 18:12 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools commented

2026-10-30 18:06:16 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 4e2b8f32ceea is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

jschoubben merged commit 7e905a6694 into main

2026-10-30 18:12:53 +00:00

jschoubben deleted branch fix/sweep-2026-10-30

2026-10-30 18:12:54 +00:00

jschoubben referenced this issue from a commit

2026-10-30 18:15:55 +00:00

Revert power's bar item attributes shows and state: every node-engine refuses them (placed by bar, place), so no machine could compose any module after #565 (issue 774)

mesh-tools referenced this pull request

2026-10-30 18:16:04 +00:00

Revert power's bar item attributes, which every node-engine refuses, so machines compose again (issue 774) #566

jschoubben referenced this issue from a commit

2026-10-30 18:30:23 +00:00