The keeper's health reads the consumers it looks for (issue 1966, recovery) #636

2026-11-08T22:55:41Z

mesh-tools commented

2026-11-08 22:55:41 +00:00

Recovery fix (recovery-fixes-merge-and-tell; the operator: "you can deliver stuff yourself", 2026-11-08).

Symptom. The bus keeper on novox is always "unhealthy: what waits on its consumer cannot be read from the bus's monitoring".

Cause. nats_keeper_health looks for the keeper's consumer (novox_nats_keeper on EVENTS) in the monitoring read it shares with the broker verbs. That read asked /jsz?accounts=1&streams=1&config=1, without consumers=1, so keeperPending never found the consumer and always read -1 (cmd/nats-tools/main.go).

Fix. The shared read is the constant heldJsz, now with consumers=1. The broker verbs ignore the extra consumer detail. TestTheHeldReadAsksForTheConsumers holds the read to asking for consumers and checks that keeperPending finds the keeper's consumer in such a read.

Quick or thorough. One place, so the quick fix is the thorough one.

Checked locally. go vet ./... and go test ./cmd/nats-tools/ in modules/nats.

Delivery. This changes only the tools artifact; the bus server's container is not recreated. It rides in the same bus move as #635, which waits for the operator's yes.

Undo. Revert this merge.

https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

Recovery fix (recovery-fixes-merge-and-tell; the operator: "you can deliver stuff yourself", 2026-11-08). **Symptom.** The bus keeper on novox is always "unhealthy: what waits on its consumer cannot be read from the bus's monitoring". **Cause.** `nats_keeper_health` looks for the keeper's consumer (`novox_nats_keeper` on EVENTS) in the monitoring read it shares with the broker verbs. That read asked `/jsz?accounts=1&streams=1&config=1`, without `consumers=1`, so `keeperPending` never found the consumer and always read -1 (cmd/nats-tools/main.go). **Fix.** The shared read is the constant `heldJsz`, now with `consumers=1`. The broker verbs ignore the extra consumer detail. `TestTheHeldReadAsksForTheConsumers` holds the read to asking for consumers and checks that `keeperPending` finds the keeper's consumer in such a read. **Quick or thorough.** One place, so the quick fix is the thorough one. **Checked locally.** `go vet ./...` and `go test ./cmd/nats-tools/` in modules/nats. **Delivery.** This changes only the `tools` artifact; the bus server's container is not recreated. It rides in the same bus move as #635, which waits for the operator's yes. **Undo.** Revert this merge. https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

mesh-tools added 1 commit 2026-11-08 22:55:41 +00:00

The keeper's health reads the consumers it looks for: the shared monitoring read asks for them

mesh/delivery merged: source.main moved to novox/mesh-catalog@08fbc51af26f

Details

mesh/check checking 1ef1b1356044 on 08fbc51af26f

1ef1b13560

nats_keeper_health found the keeper's consumer novox_nats_keeper in /jsz, but the read it shares with the
broker verbs asked /jsz?accounts=1&streams=1&config=1, without consumers=1, so keeperPending always read -1
and the keeper was "unhealthy: what waits on its consumer cannot be read from the bus's monitoring" for
ever (2026-11-08, issue 1966).

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

mesh-tools merged commit 08fbc51af2 into main

2026-11-08 22:55:46 +00:00

mesh-tools commented

2026-11-08 22:55:46 +00:00

Delivery novox/mesh-catalog@1ef1b1356044 — merged since 2026-11-08T22:55:49Z

Source — source.main moved to novox/mesh-catalog@08fbc51af2. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-11-08 22:55 (new) → proposed (announced): novox/mesh-catalog#636's head announced
2026-11-08 22:55 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-11-08 22:55 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@1ef1b1356044` — **merged** since 2026-11-08T22:55:49Z **Source** — source.main moved to novox/mesh-catalog@08fbc51af26f. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-11-08 22:55 (new) → proposed (announced): novox/mesh-catalog#636's head announced - 2026-11-08 22:55 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-11-08 22:55 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools deleted branch fix/keeper-health-consumers

2026-11-08 22:55:47 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#636