The grants part writes its list whole however large, one refused principal no longer freezes it, and stale watchman keys close (issue 774) #117

2026-10-30T17:45:38Z

mesh-tools commented

2026-10-30 17:45:38 +00:00

mesh-grants: sealed grants are one key per user with an index written last (the one value outgrew the bucket, 10054, and froze every new user list); one refused principal keeps its in-force grants and the rest of the list is written, raised as mesh-grants.principal-refused; modules the mesh provides are not "missing a grant".
checker: a candidate outside the mesh (no trunk, no module) gets no verdict and owes no status (the 403s on hal and novadock).
watchman: a stuck key whose watched key aged out or was deleted is closed at start (the 26 stuck-key conditions).
All nine modules pinned to mesh-sdk a3718126 (kvlimit, busretry, AccountsEntry).
Not fixed here: the bus blackout itself is nats-server's reload; its thorough fix (logins answered by the grants part) is research after this group.

Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch fix/sweep-2026-10-30 in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24).
Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix.

Reviewed-by: rev-sweep@shanks at e0b56c83f9

- **mesh-grants**: sealed grants are one key per user with an index written last (the one value outgrew the bucket, 10054, and froze every new user list); one refused principal keeps its in-force grants and the rest of the list is written, raised as `mesh-grants.principal-refused`; modules the mesh provides are not "missing a grant". - **checker**: a candidate outside the mesh (no trunk, no module) gets no verdict and owes no status (the 403s on hal and novadock). - **watchman**: a stuck key whose watched key aged out or was deleted is closed at start (the 26 stuck-key conditions). - All nine modules pinned to mesh-sdk a3718126 (kvlimit, busretry, AccountsEntry). Not fixed here: the bus blackout itself is nats-server's reload; its thorough fix (logins answered by the grants part) is research after this group. --- Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch `fix/sweep-2026-10-30` in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24). Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix. Reviewed-by: rev-sweep@shanks at e0b56c83f94c62ad98e3c072a52a68cea1dbff73

mesh-tools added 10 commits 2026-10-30 17:45:39 +00:00

Keep each sealed list's permissions one key per user, so no value outgrows the bucket (issue 774) c1b28cc4fe

The sealed-grants value held every user of the last two lists in one value; at 61 users it was refused with
"message size exceeds maximum" (JetStream 10054, 16:37 and 16:55 on 2026-10-30), so the permissions of the
list in force were never kept and a grant judged against them waited.

Quick fix: keep only the newest list in the one value (halves it; it still grows with the mesh).
Thorough fix, chosen because the value grows with every placed module and the kind was met twice: one key per
user and list, sealed-grants.<digest hex>.<user>, and an index sealed-lists naming the kept lists and their users,
written only once every user's key is there; the third list's keys are taken away, and the old one-value key read
as a fallback and removed. mesh-sdk's kvlimit guard (7b02f63) refuses an oversized value before the put through
process.State.Put, which the part's own store (cmd/mesh-grants ownStore.Put) writes through once pinned.

Compose the user list principal by principal, so one refused principal never freezes it (issue 774) 031cfb8277

Met twice on 2026-10-30 (the engine's acts, then mesh-host_health): the composer refused one principal as a
second writer, and that refusal ended the whole composition, so the list in force stood for every principal
while changes piled up; the next list that composed moved 52 users at once (16:37:28).

Quick fix: leave the refused principal out and write the rest (locks that principal out of the bus).
Thorough fix, chosen because the kind was met twice and it hides every other change: ComposeEach composes each
principal on its own; a refused one keeps its grants of the list in force (the part's sealed grants), written as
the composer writes an entry; one the list in force does not carry stays out; a list that would drop one the
list in force carries, whose grants were not kept, is held, said. The refusal is raised as the warning
mesh-grants.principal-refused, by name and why, cleared once it composes, and listed in the pass
(principals_refused). The entry renderer mirrors buscompose.ComposeAccounts's lines and is pinned to them by the
test (the entry the bus loaded is written back byte for byte); it belongs in mesh-sdk as an exported entry writer.

Never name the private network the mesh provides as a module missing its grant (issue 774) 94f555277c

mesh-wireguard is provided by the mesh itself (mesh-sdk delivery.ProvidedByMesh): no repository builds it, so the
indexer holds no fingerprint of it and the assigner never wants it by one; the controller composes it with the
node's own. The grants part named it "no fingerprint of the new form: the indexer holds no value of it" on every
pass, a warning nothing could clear (sweep 2026-10-30 item 6).

Quick fix: drop mesh-wireguard by name from the missing list.
Thorough fix, chosen as cheap and covering every module the mesh may provide later: MissingGrants passes over any
module ProvidedByMesh names. The controller says the same of it in mesh-control (its grants and switch lines).

Judge nothing and owe no status for a candidate outside the mesh (issue 774) ce054dc8dd

The forge carries the operator's own repositories (jschoubben/hal, jschoubben/novadock) beside the mesh's; the
indexer writes their pull requests as candidates, the gate never names their trunk, so the checker said each
"cannot be judged" and owed the status error on mesh/check, which the forge module wrote into repositories the
mesh's forge account may not write: 403, 119 times each (sweep 2026-10-30 item 7).

Quick fix: name the two repositories as not the mesh's in the checker.
Thorough fix, chosen because any repository on the forge can meet it: a candidate every pull request of which is
of a repository the gate's source.main names no trunk of and touches no module is outside the mesh. The checker
writes it no verdict and owes it no status, takes away a status owed before, says it once in the log, and lists it
in checker_status as outside_the_mesh. A new mesh repository's first pull request adds a module, so it is still
said untrunked and owed error (issue 1422). Nothing is outside before the gate's source is read.

Write a kept user through mesh-sdk's AccountsEntry, so its lines are ComposeAccounts' own (issue 774) 788969bcf8

The grants part repeated the composer's entry lines to keep a refused principal's grants of the list in force;
mesh-sdk 61b67d6 exports them as buscompose.AccountsEntry, so the copy goes. The byte-for-byte test of the kept
entry still passes. Needs the mesh-sdk pin at 61b67d6 or later.

Pin mesh-sdk 61b67d61, so mesh-grants writes through the size guard and every delivery module rides out a user-list reload (issue 774) ff58b16cf9

Close a stuck key whose watched key aged out or was deleted unheard, so its condition clears (issue 774) 255f8b2aca

A key that ages out of its bucket (the build seats' asks pass the stream's 168 h) leaves no marker a watch hears,
and one deleted while the watchman was down is not heard either: its stuck key stood unmoved for ever, and 26
stuck-key conditions of asks from 2026-10-21 were open on 10-30.

Quick fix: mark stuck keys older than the stream's maximum age moved.
Thorough fix, chosen because it covers every bucket the watchman mirrors and any way a key goes: at the start's
end, a stuck key not moved whose bucket was read whole at this start (each build seat's steps, the assigner's
wants) and whose key was not read standing is written moved, which clears its finding, and the closing is said
in the log. A bucket that could not be read closes nothing. A key that ages out while the watchman runs is
closed at its next start: a KV age-out emits no event, and no timer is added.

Pin mesh-sdk ef1ec1c9, the fix group's reviewed library head (issue 774) f756731700

Pin mesh-sdk 206f794e, the fix group's final library head (issue 774) d4ac50c4b1

Pin mesh-sdk a3718126, whose bus retry never asks again a call its callee answered (issue 774)

mesh/delivery merged: source.main moved to novox/mesh-delivery@c104e7500d21

Details

mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged

Details

mesh/check checking e0b56c83f94c on 33973d533825 | novox/mesh-delivery#117 in group fix/sweep-2026-10-30

e0b56c83f9

mesh-tools commented

2026-10-30 17:45:48 +00:00

Delivery novox/mesh-delivery@e0b56c83f94c — merged since 2026-10-30T18:13:04Z

Source — source.main moved to novox/mesh-delivery@c104e7500d. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Group fix/sweep-2026-10-30: novox/mesh-catalog@2455445dab, novox/mesh-delivery@e0b56c83f9, novox/mesh-host@b9c24a7b42, novox/mesh-sdk@a37181261c, novox/mesh-tools@316a5a2fc3

composed together: fail — the checker's verdict at the members' heads (1e1a3f6ab540): shanks

Transitions

2026-10-30 17:45 (new) → proposed (announced): novox/mesh-delivery#117's head announced
2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-30 18:13 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-delivery@e0b56c83f94c` — **merged** since 2026-10-30T18:13:04Z **Source** — source.main moved to novox/mesh-delivery@c104e7500d21. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Group** `fix/sweep-2026-10-30`: novox/mesh-catalog@2455445dabee, novox/mesh-delivery@e0b56c83f94c, novox/mesh-host@b9c24a7b42cb, novox/mesh-sdk@a37181261c24, novox/mesh-tools@316a5a2fc317 - composed together: fail — the checker's verdict at the members' heads (1e1a3f6ab540): shanks **Transitions** - 2026-10-30 17:45 (new) → proposed (announced): novox/mesh-delivery#117's head announced - 2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-30 18:13 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools commented

2026-10-30 18:06:17 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 4e2b8f32ceea is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

jschoubben merged commit c104e7500d into main

2026-10-30 18:13:00 +00:00

jschoubben deleted branch fix/sweep-2026-10-30

2026-10-30 18:13:01 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-delivery#117