Grants purges old sealed lists' keys, and once at its start the ones a delete left behind (issue 1966, recovery) #182

2026-11-08T23:22:45Z

mesh-tools commented

2026-11-08 23:22:45 +00:00

Recovery fix (recovery-fixes-merge-and-tell; the operator: "you can deliver stuff yourself", 2026-11-08).

Symptom. Grants' state user-list (bucket mesh-grants_user-list, history 20, fixed cap 64 MiB) held 63,461,986 bytes at 23:08 UTC: 5,968 keys including deleted ones, with 13,228 values kept. The self-check raised D-fill at 94%. Each new list adds about 310 KB, so about 11 lists of room were left. At the cap the bus refuses grants' next sealed list, and deliveries lock again.

Cause. keepSealedGrants takes away the old lists' per-user keys (sealed-grants.<digest>.<user>) with Delete. A NATS key-value delete only adds a marker, so the value stays in the stream.

Fix.

de320684: old lists' keys are purged through the process's own JetStream connection (takeAway; the tools fall back to Delete). A unit row checks that the oldest list's keys are purged and the kept ones untouched. An integration step on a real server under grants' composed account fails with Delete.
126b40e6: once at the process's start, SweepSealed purges every sealed-grants.* key whose last operation is a marker, and every key of a digest the kept index does not name. It never touches a kept list, and purges nothing if the index cannot be read. It logs the counts and the bytes before and after.

Grants checked. A purge publishes on $KV.mesh-grants_user-list.<key> with the roll-up header, which grants already holds. The listing uses an ordered consumer on its own bucket. Nothing was added.

Quick or thorough. This fix is the thorough one for grants. The kind (a KV delete keeps the value in other modules' states too) is noted on issue 1966.

Limits. At 100% the purge itself (a roll-up is a new message) is refused too, so this must run while room is left. The sweep's result is in the log only, with no tool yet (noted on issue 1966).

Checked locally. go vet ./..., go vet -tags integration ./..., go test ./... in modules/mesh-grants, and the one integration subtest against an embedded bus. No mesh-sdk change, so no mass rebuild.

Undo. Revert this merge.

https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

Recovery fix (recovery-fixes-merge-and-tell; the operator: "you can deliver stuff yourself", 2026-11-08). **Symptom.** Grants' state `user-list` (bucket mesh-grants_user-list, history 20, fixed cap 64 MiB) held 63,461,986 bytes at 23:08 UTC: 5,968 keys including deleted ones, with 13,228 values kept. The self-check raised D-fill at 94%. Each new list adds about 310 KB, so about 11 lists of room were left. At the cap the bus refuses grants' next sealed list, and deliveries lock again. **Cause.** `keepSealedGrants` takes away the old lists' per-user keys (`sealed-grants.<digest>.<user>`) with Delete. A NATS key-value delete only adds a marker, so the value stays in the stream. **Fix.** - de320684: old lists' keys are purged through the process's own JetStream connection (`takeAway`; the tools fall back to Delete). A unit row checks that the oldest list's keys are purged and the kept ones untouched. An integration step on a real server under grants' composed account fails with Delete. - 126b40e6: once at the process's start, `SweepSealed` purges every `sealed-grants.*` key whose last operation is a marker, and every key of a digest the kept index does not name. It never touches a kept list, and purges nothing if the index cannot be read. It logs the counts and the bytes before and after. **Grants checked.** A purge publishes on `$KV.mesh-grants_user-list.<key>` with the roll-up header, which grants already holds. The listing uses an ordered consumer on its own bucket. Nothing was added. **Quick or thorough.** This fix is the thorough one for grants. The kind (a KV delete keeps the value in other modules' states too) is noted on issue 1966. **Limits.** At 100% the purge itself (a roll-up is a new message) is refused too, so this must run while room is left. The sweep's result is in the log only, with no tool yet (noted on issue 1966). **Checked locally.** `go vet ./...`, `go vet -tags integration ./...`, `go test ./...` in modules/mesh-grants, and the one integration subtest against an embedded bus. No mesh-sdk change, so no mass rebuild. **Undo.** Revert this merge. https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

mesh-tools added 2 commits 2026-11-08 23:22:46 +00:00

Grants purges the old sealed lists' keys instead of deleting them (issue 1966) de320684f2

A key-value delete only adds a marker: every value of the per-user keys of the
lists older than the two kept stayed in the state user-list's stream, which
reached 94% of its 64 MiB cap on 2026-11-08 (5,968 keys, 13,228 values). At the
cap the bus refuses grants' next write of the sealed list and deliveries lock.
The part now takes those keys away through a purger on the process's own
JetStream connection, a publish to the key's own subject the account already
writes on; the tools, with no purger, still delete.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

Grants purges, once at its start, the sealed keys a delete left behind (issue 1966)

mesh/delivery merged: source.main moved to novox/mesh-delivery@4f9b15a43f05

Details

mesh/check checking 126b40e6f585 on 4f9b15a43f05

126b40e6f5

The markers and the values of the lists taken away before the purge still fill
the state user-list. Before its first pass the process purges every
sealed-grants key whose last operation is a delete or purge marker, and every
user's key of a list the index of kept lists does not name; never a kept
list's key nor another key of the state, and nothing when the index cannot be
read. It logs the counts and the state's bytes before and after.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

mesh-tools commented

2026-11-08 23:22:50 +00:00

Delivery novox/mesh-delivery@126b40e6f585 — merged since 2026-11-08T23:22:51Z

Source — source.main moved to novox/mesh-delivery@4f9b15a43f. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-11-08 23:22 (new) → proposed (announced): novox/mesh-delivery#182's head announced
2026-11-08 23:22 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-11-08 23:22 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-delivery@126b40e6f585` — **merged** since 2026-11-08T23:22:51Z **Source** — source.main moved to novox/mesh-delivery@4f9b15a43f05. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-11-08 23:22 (new) → proposed (announced): novox/mesh-delivery#182's head announced - 2026-11-08 23:22 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-11-08 23:22 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 4f9b15a43f into main

2026-11-08 23:22:50 +00:00

mesh-tools deleted branch fix/1966-user-list-purge

2026-11-08 23:22:50 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-delivery#182