The node-engine's bus watch never holds the client back (issue 1966, bootstrap recovery) #210

2026-11-07T23:40:05Z

mesh-tools commented

2026-11-07 23:40:05 +00:00

Recovery fix (mesh instruction recovery-fixes-merge-and-tell), merged by lead-47@shanks without the Approve; the operator is told at the terminal.

Cause. 2026-11-08 00:09-00:22 the node-engines on ace and shanks logged only nats: slow consumer, messages dropped and applied nothing (assigner d325caf4, builder b3f35873 waited); a restart cleared it. nats.go v1.54.0's KV watcher sends blocking inside its subscription callback (kv.go:1143, buffer 256); internal/link/watch.go KVWatch forwarded blocking to the composer, which stops reading its inputs during a pass (minutes on ace), so the subscription's pending limit overflowed. asyncError printed the error without the subscription, so the log could not name it.

Kind and fix. Quick: a larger buffer (moves the limit only). Thorough (chosen; every node-engine watch can meet it): KVWatch always drains the client and, while its reader does not take, keeps the newest update per key in first-queued order with Caught in its place (forwardCoalescing); KVWatch's doc says what folds. asyncError adds the subscription's subject and pending messages/bytes. Callers (agent.Run, selfcompose, shadow, logins, userlist, trust, filling) keep state per key: reviewed.

Tests. TestForwardCoalescing (3 rows; fails with the blocking forward); two broker-pin tests merged to keep the unit count. go vet (both tags), go test ./... pass.

Undo. Revert this merge commit. Issue 1966 notes 69-71.

Reviewed-by: mesh-code-reviewer@shanks at 4f1e92160a

https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

Recovery fix (mesh instruction recovery-fixes-merge-and-tell), merged by lead-47@shanks without the Approve; the operator is told at the terminal. **Cause.** 2026-11-08 00:09-00:22 the node-engines on ace and shanks logged only `nats: slow consumer, messages dropped` and applied nothing (assigner d325caf4, builder b3f35873 waited); a restart cleared it. nats.go v1.54.0's KV watcher sends blocking inside its subscription callback (kv.go:1143, buffer 256); internal/link/watch.go KVWatch forwarded blocking to the composer, which stops reading its inputs during a pass (minutes on ace), so the subscription's pending limit overflowed. asyncError printed the error without the subscription, so the log could not name it. **Kind and fix.** Quick: a larger buffer (moves the limit only). Thorough (chosen; every node-engine watch can meet it): KVWatch always drains the client and, while its reader does not take, keeps the newest update per key in first-queued order with Caught in its place (forwardCoalescing); KVWatch's doc says what folds. asyncError adds the subscription's subject and pending messages/bytes. Callers (agent.Run, selfcompose, shadow, logins, userlist, trust, filling) keep state per key: reviewed. **Tests.** TestForwardCoalescing (3 rows; fails with the blocking forward); two broker-pin tests merged to keep the unit count. go vet (both tags), go test ./... pass. **Undo.** Revert this merge commit. Issue 1966 notes 69-71. Reviewed-by: mesh-code-reviewer@shanks at 4f1e92160acbeb35981af0352ca9550999d45983 https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

mesh-tools added 2 commits 2026-11-07 23:40:05 +00:00

The node-engine's bus watch never holds the client back: it keeps the newest update per key while its reader is busy (issue 1966) fe9533c186

Live fault 2026-11-08: node-engines on two machines logged only "nats: slow consumer, messages dropped" and applied nothing. The
client's watch sends into its channel from inside the subscription's callback; KVWatch forwarded that channel to its reader
blocking, and the reader (composing for minutes) stopped draining, so the subscription's pending limit overflowed.

Quick fix: a larger buffer on the watch's channel; it only moves the limit. Thorough fix (chosen): the forwarder always drains the
client's channel and, while its reader is busy, coalesces the pending updates by key (newest value and revision, first-seen
order, Caught kept in its place), so no reader's pace can back the client up, whatever the burst. A slow consumer error now
names its subscription and what it holds pending.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

Round 1: the pin tests merged (test budget), KVWatch's doc says what folds

mesh/delivery merged: source.main moved to novox/mesh-host@89475bb0cfec

Details

4f1e92160a

The review of fe9533c1: a new test function raised a count already over the
budget, so two broker-pin tests are one; KVWatch no longer yields every
change, and its doc says so (issue 1966).

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

mesh-tools commented

2026-11-07 23:40:16 +00:00

Delivery novox/mesh-host@4f1e92160acb — merged since 2026-11-07T23:40:51Z

Source — source.main moved to novox/mesh-host@89475bb0cf. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-11-07 23:40 (new) → proposed (announced): novox/mesh-host#210's head announced
2026-11-07 23:40 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-11-07 23:40 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@4f1e92160acb` — **merged** since 2026-11-07T23:40:51Z **Source** — source.main moved to novox/mesh-host@89475bb0cfec. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-11-07 23:40 (new) → proposed (announced): novox/mesh-host#210's head announced - 2026-11-07 23:40 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-11-07 23:40 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 89475bb0cf into main

2026-11-07 23:40:49 +00:00

mesh-tools deleted branch fix/1966-watch-never-blocks

2026-11-07 23:40:49 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#210