node-tools: a consumer deleted or made again while no pull waits is noticed at the next rebind, so a module is not deaf for a heartbeat (issue 1945)
#43
Merged
mesh-toolsmerged 1 commits from fix/bus-consumer-remade-flake into main2026-10-25 15:22:24 +00:00
Issue 1945. bus TestAConsumerDeletedAndMadeAgainIsHeardAgain failed 9 of 100 runs on main under -race, and was met by two agents today.
Cause, in the code rather than the test: nats.go learns a consumer was deleted only from the answer to a pull pending at that moment. If the deletion lands while no pull waits (right after an ack), reading.Closed() never fires and the loop stays bound to the dead consumer until its heartbeat, about 30 s. In production a module is deaf that long after the controller deletes and remakes a restricted consumer (ADR 0363 §6).
Fix: each rebind tick (the existing RebindEvery, 30 s in production) asks each bound consumer's info by name and compares the server's creation time with the one recorded at bind. Gone (ErrConsumerNotFound) or made at another time → the old loop stops and the consumer is bound again. Any other error leaves the loop as it is. Each request has a 5 s timeout. The runtime's credential already holds $JS.API.CONSUMER.INFO for these streams.
Checked: gofmt, go vet with and without -tags integration, the node-tools unit suite; -race -count=100 -cpu 2: main 9 failures, branch 0. No try: the operator decided no checks run during the recovery (issue 1945 note 27).
Not NATS-native: this still polls for a state change (one info request per bound consumer per existing tick). The event-driven follow-up is noted on issue 1945: carry the consumer's creation time or generation in the membership (HeardIssued), so a remake changes the membership the runtime already follows.
Issue 1945. bus TestAConsumerDeletedAndMadeAgainIsHeardAgain failed 9 of 100 runs on main under -race, and was met by two agents today.
Cause, in the code rather than the test: nats.go learns a consumer was deleted only from the answer to a pull pending at that moment. If the deletion lands while no pull waits (right after an ack), reading.Closed() never fires and the loop stays bound to the dead consumer until its heartbeat, about 30 s. In production a module is deaf that long after the controller deletes and remakes a restricted consumer (ADR 0363 §6).
Fix: each rebind tick (the existing RebindEvery, 30 s in production) asks each bound consumer's info by name and compares the server's creation time with the one recorded at bind. Gone (ErrConsumerNotFound) or made at another time → the old loop stops and the consumer is bound again. Any other error leaves the loop as it is. Each request has a 5 s timeout. The runtime's credential already holds $JS.API.CONSUMER.INFO for these streams.
Checked: gofmt, go vet with and without -tags integration, the node-tools unit suite; -race -count=100 -cpu 2: main 9 failures, branch 0. No try: the operator decided no checks run during the recovery (issue 1945 note 27).
Not NATS-native: this still polls for a state change (one info request per bound consumer per existing tick). The event-driven follow-up is noted on issue 1945: carry the consumer's creation time or generation in the membership (HeardIssued), so a remake changes the membership the runtime already follows.
Reviewed-by: reviewer-bus@shanks at 33317194073a97cec25b3aec72c3acc5c5d3c002
nats.go ends a pull loop on a deleted consumer only when a pull is pending at that moment; otherwise the loop waits out its heartbeat. The rebind tick now looks each bound consumer up by name and compares its creation time.
Deliverynovox/mesh-tools@33317194073a — merged since 2026-10-25T15:22:27Z
Source — source.main moved to novox/mesh-tools@bb6a0e3192. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.
Transitions
2026-10-25 15:11 (new) → proposed (announced): novox/mesh-tools#43's head announced
2026-10-25 15:16 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail
2026-10-25 15:22 ready → merged (merged): on the trunk its modules follow: merged there
The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.
<!-- mesh-delivery:view -->
**Delivery** `novox/mesh-tools@33317194073a` — **merged** since 2026-10-25T15:22:27Z
**Source** — source.main moved to novox/mesh-tools@bb6a0e3192db. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves.
**Transitions**
- 2026-10-25 15:11 (new) → proposed (announced): novox/mesh-tools#43's head announced
- 2026-10-25 15:16 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail
- 2026-10-25 15:22 ready → merged (merged): on the trunk its modules follow: merged there
The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Issue 1945. bus TestAConsumerDeletedAndMadeAgainIsHeardAgain failed 9 of 100 runs on main under -race, and was met by two agents today.
Cause, in the code rather than the test: nats.go learns a consumer was deleted only from the answer to a pull pending at that moment. If the deletion lands while no pull waits (right after an ack), reading.Closed() never fires and the loop stays bound to the dead consumer until its heartbeat, about 30 s. In production a module is deaf that long after the controller deletes and remakes a restricted consumer (ADR 0363 §6).
Fix: each rebind tick (the existing RebindEvery, 30 s in production) asks each bound consumer's info by name and compares the server's creation time with the one recorded at bind. Gone (ErrConsumerNotFound) or made at another time → the old loop stops and the consumer is bound again. Any other error leaves the loop as it is. Each request has a 5 s timeout. The runtime's credential already holds $JS.API.CONSUMER.INFO for these streams.
Checked: gofmt, go vet with and without -tags integration, the node-tools unit suite; -race -count=100 -cpu 2: main 9 failures, branch 0. No try: the operator decided no checks run during the recovery (issue 1945 note 27).
Not NATS-native: this still polls for a state change (one info request per bound consumer per existing tick). The event-driven follow-up is noted on issue 1945: carry the consumer's creation time or generation in the membership (HeardIssued), so a remake changes the membership the runtime already follows.
Reviewed-by: reviewer-bus@shanks at
3331719407Delivery
novox/mesh-tools@33317194073a— merged since 2026-10-25T15:22:27ZSource — source.main moved to novox/mesh-tools@bb6a0e3192. Whether it reached the machines is
mesh-delivery-view's, derived fromsource.main, the wants and the haves.Transitions
The commit's note under
refs/notes/mesh-plankeeps every transition of this change up to its merge:git log --notes=mesh-plan.