One bad module no longer drops its machine, too-large values and bus blackouts fail loud or are waited out, watches are never forgotten (issue 774) #138

2026-10-30T17:45:37Z

mesh-tools commented

2026-10-30 17:45:37 +00:00

Fixes found by the live sweep after design 47's release, each as a kind:

One bad module no longer drops its machine (59311407): plans are composed module by module (PlansHeld); a foreseen own secret fills a file; a built bus manifest keeps its bundle.
A value too big for its bucket is refused before the put (go/kvlimit), naming bucket, key, size and limit; never retried.
Reads are judged for their reader (Users re-judges through ReadsAnnouncedBy); the reader-less ReadsAnnounced is gone; mesh-config named a reader of the controller's lease (ADR 0561 §3).
A seat verb nothing serves is refused at validation (seat_invokes.go); a mesh seat is judged by its verb only.
A watch is never silently forgotten: changes are answered at once, handlers run in order on their own (follower.go); KeptState.Follow asks again with backoff.
A bus blackout at a user-list reload is waited out (go/busretry, about 30 s), only for requests that reached no one; a callee's answer is never asked again.
buscompose.AccountsEntry exported; ReadsGranted grants a held bar item's read at the machine's own key (ADR 0260).

Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch fix/sweep-2026-10-30 in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24).
Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix.

Reviewed-by: rev-sweep@shanks at a37181261c

Fixes found by the live sweep after design 47's release, each as a kind: - **One bad module no longer drops its machine** (59311407): plans are composed module by module (`PlansHeld`); a foreseen own secret fills a file; a built bus manifest keeps its bundle. - **A value too big for its bucket is refused before the put** (`go/kvlimit`), naming bucket, key, size and limit; never retried. - **Reads are judged for their reader** (`Users` re-judges through `ReadsAnnouncedBy`); the reader-less `ReadsAnnounced` is gone; mesh-config named a reader of the controller's lease (ADR 0561 §3). - **A seat verb nothing serves is refused** at validation (`seat_invokes.go`); a mesh seat is judged by its verb only. - **A watch is never silently forgotten**: changes are answered at once, handlers run in order on their own (`follower.go`); `KeptState.Follow` asks again with backoff. - **A bus blackout at a user-list reload is waited out** (`go/busretry`, about 30 s), only for requests that reached no one; a callee's answer is never asked again. - `buscompose.AccountsEntry` exported; `ReadsGranted` grants a held bar item's read at the machine's own key (ADR 0260). --- Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch `fix/sweep-2026-10-30` in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24). Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix. Reviewed-by: rev-sweep@shanks at a37181261c24a15b8092c06e11cce63e0e5f8ade

mesh-tools added 11 commits 2026-10-30 17:45:37 +00:00

One bad module must not drop its node: plans are composed module by module, a foreseen own secret fills a file, a built bus manifest keeps its bundle (issue 774) 59311407e2

PlansHeld holds a module whose build cannot be taken in or whose plan refuses, by composeEach's rule, and composes
every other module's plan; sealedFor gives a file the stand-in its own secret's file has (baserow on ace); shipsABundle
reads a built manifest's Bundles (nats on novox).

Refuse a key-value write too large for its bucket before it is put (issue 774) 7b02f63daf

mesh-grants' sealed grants and the controller's merge record each met "message size exceeds maximum" (JetStream
10054) only after the put, and a writer that retries looped on a value that can never land; an update over the
limit was even answered as a revision mismatch, which every compare-and-set writer retries.

Quick fix: cap the two values in their modules. Thorough fix, chosen because the kind was met twice and every
module writes through the SDK: the package kvlimit checks a value against its bucket's largest message (read from
its stream and kept) before the put, naming the bucket, the key, the size and the limit, and turns the bus's own
refusal into the same *TooLarge (errors.Is kvlimit.ErrTooLarge), so no writer retries it. The SDK's own writers
use it: process.State (Put, Create, CreateTTL, Update), process.AwaitedBucket, the runtime state of a bundle
(stdio KeptState, against the module state's 256 KiB) and mesh-hand-compose. A value that can outgrow its bucket
is split over keys or kept in the object store with its name in the key.

Judge every read for its reader where the bus's users are composed (issue 774) e703e9b0e2

Since 13:25 the vault, the assigner, the watchman, the grants part and the artifact store's tools were refused
mesh-controller.lease, and the vault mesh-grants.accounts, though the composer names them those states' readers
(ADR 0547 clause 9): the controller splits reads with its own copy of the reader-less ReadsAnnounced, which knows
no named readers, and Users only moved reads from granted to refused, never back.

Quick fix: the controller calls ReadsAnnouncedBy with the module. Thorough fix, chosen because the kind was met
on four owners and the judgement had two copies: Users judges every read again for its reader against the
announcements in Records (withAnnouncedReads), whatever split the caller made, and the reader-less
ReadsAnnounced is removed so no caller can judge without the reader. A rule over manifests was not added:
ADR 0495 clauses 1 and 5 keep readability out of the manifest and replaced the merge gate's reads check by each
module's announcement test. gitea.bound and disk-load.readings stay refused: their owners announce them not
readable, which their own code fixes.

Refuse a catalogue where a module invokes a seat verb nothing answers (issue 774) 7115b7bf64

messenger invoked seat:node-engine.root-free while nothing answered it, and Telegram held every ask that offers an
approval, silently. Nothing judged a seat:<seat>.<verb> invoke beyond its spelling: not that the seat exists, that
it promises the verb, nor that anything holds it.

Quick fix: messenger asks the controller's root-free (mesh-catalog bc271896) and the engine announces its seat as
discovery reads it (mesh-host 7ecf7ba). Thorough fix, chosen because an invoke nothing answers fails only at the
call, silently for whoever waits on it: CatalogueProblems refuses every seat verb a module or one of its processes
invokes whose seat no module declares and the mesh does not define, whose seat does not promise the verb, or whose
seat no module claims (a seat every node's engine holds, ADR 0563 §1, is held). Over some manifests alone a seat
whose holder was not given is a note (IsUndeclaredSeat). Over every manifest of mesh-catalog, mesh-control and
mesh-delivery today it refuses nothing.

Run a watch's handler beside the runtime, and give a refused watch up loudly (issue 774) 1dc5d8cb3d

The provisioner's watches of gitea.bound, mesh-vault.bound and postgres.bound failed every minute for an hour on
the control-node ("the current values ... did not arrive in 10s") and 1257 changes were refused as "names no
watch of this bundle". Both sides were at fault. The SDK answered a change only once its handler returned, so a
slow handler (Rejudge waits on the lock its composition holds) counted against the runtime's 10 s bound on the
current values; and when the runtime gave up, the SDK forgot the id at once. The runtime (mesh-tools node-tools
internal/bus/state.go StateWatch) stops its watcher on that bound but still drains and delivers the values already
buffered, to an id nobody holds.

Quick fix: a longer runtime bound. Thorough fix here: each state and provision watch has a follower that takes a
change in order and answers at once, its handler on its own goroutine; Watch still returns only once the current
values were handled, and a handler's error is said on standard error. A watch the runtime refuses is said given up
and kept by id, so a change still handed to it is refused saying so. KeptState.Follow asks again with backoff until
taken, the loop four modules wrote for themselves. A log watch keeps its answer after the handler: that is its
back-pressure.

Wait out a bus that answers nothing for a few seconds instead of losing the write (issue 774) 44dbb22ea7

At each mesh-grants restart and user-list reload (16:37:24, 16:55:15) nats-server 2.11's reloadAuthorization left
the mesh's account without its imports, $JS.API.> among them, for a few seconds: requests had no responders,
JetStream calls said it was not enabled, and a build-test result on ace was lost.

Quick fix, chosen as asked; the thorough fix is a reload that keeps the account's imports, in the bus's holder.
The package busretry asks a request again, backing off within about 30 s, only while its error says it reached no
one (no responders, JetStream not enabled, no response from the stream, or either in words): nothing was done, so
even a create or an update is safe to ask again. A timeout or any answer is never asked again. After the bound it
is said on standard error and returned. Used by process.State's requests, process.Conn's calls (asked again only
while $JS.API.INFO says the bus itself answers nothing, so a callee that is not running still fails at once) and
asks, and the bundle's state requests through the runtime.

Export the accounts block's user entry, so a kept user is written in ComposeAccounts' own lines (issue 774) 61b67d61e2

mesh-grants' refused.go keptEntry wrote a user of the accounts block from grants it keeps by copying
ComposeAccounts' lines; a change to one would leave the other writing a list the bus reads differently.
buscompose.AccountsEntry(user, passwordHash, perms) is now the one writer, and ComposeAccounts uses it. No change
to what is written.

Judge only the verbs of a mesh seat a module invokes, not its holder (issue 774) 942f906222

The seat-invoke rule refused mesh-config, mesh-healers and messenger calling the controller's verbs, and
mesh-self-check calling the watchman's seat, over a catalogue that holds no manifest of their holders: the
controller answers its seat itself, and the watchman lives in mesh-delivery. mesh-control's
TestModuleCheckPassesTheCatalogue (the captured mesh-catalog) failed on it. A mesh seat is held by the mesh or by a
module of any repository, so one repository's catalogue cannot say it has no holder: for a mesh seat only the verb
is judged; a seat a module declares is still judged for a claimant. Over the captured catalogue, the live
mesh-catalog alone, and mesh-catalog, mesh-control, mesh-delivery and mesh-host together: no refusal.

Name mesh-config among the controller's lease readers (issue 774) ef1ec1c935

mesh-config reads mesh-controller.lease to take the config over only once the lease is handed away (ADR 0561 §3,
its takeover.go leaseAway), and was refused it as a reader the composer does not name.

Grant a bar's holder the state a contributor's item shows, at its own machine's key (issue 774) 206f794eb3

i3status-rust on g14 and shanks was issued nothing to read ("was issued: none", 148 times): ReadsGranted derived a
holder's reads from contributions' data alone, and design 57 L3 moved power's draw block to an item
(mesh-catalog ca8d6f33: shows: state, state: power.draw). ReadsGranted now reads a module's items of a kind the
holder's seat receives too, their attributes in a contribution's data shape, always at the machine's own key
(ADR 0260), never another's. Both callers, the controller's busrecords and mesh-grants' records.go, get it
through ReadsGranted with no change of their own.

Never ask a call again that its callee answered, whatever its words (issue 774)

mesh/delivery merged: source.main moved to novox/mesh-sdk@4c1384faefa0

Details

mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged

Details

mesh/check checking a37181261c24 on 4c1384faefa0 | novox/mesh-sdk#138 in group fix/sweep-2026-10-30

a37181261c

The review found that busretry could ask a delivered call again: Conn.request turned a callee's error answer into
errors.New(words), and Undelivered matched the words "no responders" and "jetstream not enabled", so a callee that
answered so during a blackout ran its work twice. A callee's answer is now *busretry.Answered, which Undelivered
never counts: only the transport's own no responders makes a call undelivered. The other busretry sites read or
write through JetStream's API (State, Ask) or the runtime's state verbs, where the words are the transport's.

mesh-tools commented

2026-10-30 17:45:50 +00:00

Delivery novox/mesh-sdk@a37181261c24 — merged since 2026-10-30T18:12:50Z

Source — source.main moved to novox/mesh-sdk@4c1384faef. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Group fix/sweep-2026-10-30: novox/mesh-catalog@2455445dab, novox/mesh-delivery@e0b56c83f9, novox/mesh-host@b9c24a7b42, novox/mesh-sdk@a37181261c, novox/mesh-tools@316a5a2fc3

composed together: fail — the checker's verdict at the members' heads (1e1a3f6ab540): shanks

Transitions

2026-10-30 17:45 (new) → proposed (announced): novox/mesh-sdk#138's head announced
2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-30 18:12 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-sdk@a37181261c24` — **merged** since 2026-10-30T18:12:50Z **Source** — source.main moved to novox/mesh-sdk@4c1384faefa0. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Group** `fix/sweep-2026-10-30`: novox/mesh-catalog@2455445dabee, novox/mesh-delivery@e0b56c83f94c, novox/mesh-host@b9c24a7b42cb, novox/mesh-sdk@a37181261c24, novox/mesh-tools@316a5a2fc317 - composed together: fail — the checker's verdict at the members' heads (1e1a3f6ab540): shanks **Transitions** - 2026-10-30 17:45 (new) → proposed (announced): novox/mesh-sdk#138's head announced - 2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-30 18:12 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools commented

2026-10-30 18:06:17 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 4e2b8f32ceea is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

jschoubben merged commit 4c1384faef into main

2026-10-30 18:12:47 +00:00

jschoubben deleted branch fix/sweep-2026-10-30

2026-10-30 18:12:48 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-sdk#138