From 995c8cb266a7bd83056500ab256aa2bae8b0127c Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 29 Sep 2026 23:56:31 +0200 Subject: [PATCH] Issue 156: moving a consumer's delivery subject stops a running mesh 146's change is correct on a mesh being raised and fatal on one that is running: the server will not move a push consumer's delivery subject while a subscriber is bound, and a node is bound to its declaration consumer the whole time it is up. Reproduced before fixing; an earlier version of the test unsubscribed first and passed against the broken code. The consumer that works is kept and the assertion says so. Not re-made: the nodes are granted the bare subject only, so re-making would have silenced every machine. --- .../00-report.md | 85 +++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 04-ISSUES/156-moving-a-consumers-delivery-subject-stops-the-control-plane/00-report.md diff --git a/04-ISSUES/156-moving-a-consumers-delivery-subject-stops-the-control-plane/00-report.md b/04-ISSUES/156-moving-a-consumers-delivery-subject-stops-the-control-plane/00-report.md new file mode 100644 index 0000000..5642146 --- /dev/null +++ b/04-ISSUES/156-moving-a-consumers-delivery-subject-stops-the-control-plane/00-report.md @@ -0,0 +1,85 @@ +--- +status: resolved +opened: 2026-09-29 +located-in: + - mesh-controller internal/broker/jetstream.go (EnsureConsumer) +fixed-by: mesh-controller fix/156-a-consumer-that-works-is-not-replaced +amended-design: +--- + +# 156 — Moving a consumer's delivery subject stops the control plane, and only on a mesh that is running + +## What was observed + +The control plane crash-looped, every restart ending the same way: + +``` +mesh-controller: asserting how novox hears its declaration: + bringing consumer novox on NODES to match: nats: consumer name already in use +``` + +It came up on the first build of the controller in eight hours. The machines kept running what they +already held — this stops the mesh being *changed*, not the services it placed — and nothing could be +pushed, no report was consumed and no enrolment answered, for as long as it lasted. + +## Why + +[Issue 146](../146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md) put the +stream into a push consumer's delivery subject, because one process holding two consumers of the same +name on two streams was given one subject and acted on every message twice. + +**The server will not move a push consumer's delivery subject while a subscriber is bound to it.** It +refuses with `consumer name already in use` — a message about the name, for a conflict about the +subject, which is why the trail starts in the wrong place. + +A node is bound to its declaration consumer the whole time it is up. That *is* a node listening for +what it should be. So every node consumer in a mesh that is running is one the assertion cannot bring +to match — and the assertion happens before the controller serves, so it never serves. + +The controller's own two consumers moved without trouble, and are on the new subject in the live mesh. +It asserts them before it subscribes, so nothing was bound. + +## Why nothing caught it + +The change was exercised on a mesh being raised, where every consumer is created rather than updated +and nothing is bound to any of them. On that path the code is correct. The test that would have caught +it needs a mesh that is already running: an existing consumer, a subscriber still attached, and then +the assertion. + +Reproduced exactly that way before the fix — same server version, same stream shape, same consumer — +and it fails with the same words as the machine did. An earlier version of the same test unsubscribed +first and passed against the code that was crash-looping on the control node. + +## What it is not + +- Not the change it shipped beside. The merge that triggered this build carried + [issue 152](../152-a-nodes-plan-failure-silently-drops-its-routed-names/00-report.md)'s fix and three + other commits that had never been deployed; this one is 146's. +- Not a version difference. The test server and the mesh's broker are both nats-server v2.10.29. + +## How it was fixed + +The consumer that works is kept, and the assertion says so instead of failing. + +**Not deleted and re-made.** Re-making moves the subject, and a holder may not yet be allowed to +subscribe to the new one: the wider grant travels in the bus's user list, which the control plane +composes and a machine applies minutes later. On the live mesh the nodes are granted `_DELIVER.` +and not `_DELIVER..>` — re-making would have silenced every machine in the mesh, which is worse +than the collision it was fixing and far harder to undo. That was the first fix written here, and the +permission is the reason it was not shipped. + +**Not fatal**, which is what 146's change intended and did not do: the bare subject still delivers, and +collides only where one holder has two consumers of one name. A node has one. + +## How the fix is checked + +Two tests against a real server: a consumer with a subscriber bound keeps its subject, is reported, +and still delivers to that subscriber; a consumer with nothing bound moves, so 146's fix still applies +where the collision actually was. + +## What is left + +The node consumers stay on the bare subject, which is correct and not tidy. Moving them needs the +wider grant to reach every machine first, and then an assertion made while each node is detached — +which is its own piece of work, not a side effect of a restart. Nothing is wrong while they do not +move: one consumer per name per stream cannot collide with itself. -- 2.54.0