node-tools: a consumer deleted or made again while no pull waits is noticed at the next rebind, so a module is not deaf for a heartbeat (issue 1945) #43

Merged
mesh-tools merged 1 commits from fix/bus-consumer-remade-flake into main 2026-10-25 15:22:24 +00:00
Member

Issue 1945. bus TestAConsumerDeletedAndMadeAgainIsHeardAgain failed 9 of 100 runs on main under -race, and was met by two agents today.

Cause, in the code rather than the test: nats.go learns a consumer was deleted only from the answer to a pull pending at that moment. If the deletion lands while no pull waits (right after an ack), reading.Closed() never fires and the loop stays bound to the dead consumer until its heartbeat, about 30 s. In production a module is deaf that long after the controller deletes and remakes a restricted consumer (ADR 0363 §6).

Fix: each rebind tick (the existing RebindEvery, 30 s in production) asks each bound consumer's info by name and compares the server's creation time with the one recorded at bind. Gone (ErrConsumerNotFound) or made at another time → the old loop stops and the consumer is bound again. Any other error leaves the loop as it is. Each request has a 5 s timeout. The runtime's credential already holds $JS.API.CONSUMER.INFO for these streams.

Checked: gofmt, go vet with and without -tags integration, the node-tools unit suite; -race -count=100 -cpu 2: main 9 failures, branch 0. No try: the operator decided no checks run during the recovery (issue 1945 note 27).

Not NATS-native: this still polls for a state change (one info request per bound consumer per existing tick). The event-driven follow-up is noted on issue 1945: carry the consumer's creation time or generation in the membership (HeardIssued), so a remake changes the membership the runtime already follows.

Reviewed-by: reviewer-bus@shanks at 3331719407

Issue 1945. bus TestAConsumerDeletedAndMadeAgainIsHeardAgain failed 9 of 100 runs on main under -race, and was met by two agents today. Cause, in the code rather than the test: nats.go learns a consumer was deleted only from the answer to a pull pending at that moment. If the deletion lands while no pull waits (right after an ack), reading.Closed() never fires and the loop stays bound to the dead consumer until its heartbeat, about 30 s. In production a module is deaf that long after the controller deletes and remakes a restricted consumer (ADR 0363 §6). Fix: each rebind tick (the existing RebindEvery, 30 s in production) asks each bound consumer's info by name and compares the server's creation time with the one recorded at bind. Gone (ErrConsumerNotFound) or made at another time → the old loop stops and the consumer is bound again. Any other error leaves the loop as it is. Each request has a 5 s timeout. The runtime's credential already holds $JS.API.CONSUMER.INFO for these streams. Checked: gofmt, go vet with and without -tags integration, the node-tools unit suite; -race -count=100 -cpu 2: main 9 failures, branch 0. No try: the operator decided no checks run during the recovery (issue 1945 note 27). Not NATS-native: this still polls for a state change (one info request per bound consumer per existing tick). The event-driven follow-up is noted on issue 1945: carry the consumer's creation time or generation in the membership (HeardIssued), so a remake changes the membership the runtime already follows. Reviewed-by: reviewer-bus@shanks at 33317194073a97cec25b3aec72c3acc5c5d3c002
mesh-tools added 1 commit 2026-10-25 15:08:48 +00:00
Ask the bus whether a bound consumer still exists, so a deleted and remade one is rebound within a rebind tick, not after nats.go's heartbeat (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-tools@bb6a0e3192db
mesh/check pass 33317194073a on bb6a0e3192db | unit 115 run, 11 kept, 11 skipped | integration 117 run, 11 kept, 10 skipped | over bound: unit (slowes…
3331719407
nats.go ends a pull loop on a deleted consumer only when a pull is pending at that moment; otherwise the loop waits out its heartbeat. The rebind tick now looks each bound consumer up by name and compares its creation time.
Author
Member

Delivery novox/mesh-tools@33317194073a — merged since 2026-10-25T15:22:27Z

Source — source.main moved to novox/mesh-tools@bb6a0e3192. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

  • 2026-10-25 15:11 (new) → proposed (announced): novox/mesh-tools#43's head announced
  • 2026-10-25 15:16 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail
  • 2026-10-25 15:22 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-tools@33317194073a` — **merged** since 2026-10-25T15:22:27Z **Source** — source.main moved to novox/mesh-tools@bb6a0e3192db. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 15:11 (new) → proposed (announced): novox/mesh-tools#43's head announced - 2026-10-25 15:16 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail - 2026-10-25 15:22 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.
mesh-tools merged commit bb6a0e3192 into main 2026-10-25 15:22:24 +00:00
mesh-tools deleted branch fix/bus-consumer-remade-flake 2026-10-25 15:22:24 +00:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-tools#43