Ask the bus again for a subscription it refused (hq issue 222) #47

2026-10-03T23:20:46Z

mesh-admin commented

2026-10-03 23:20:46 +00:00

A module was assigned to a machine, and that machine was pushed. The push also sent the bus's machine, in the same second, but the runtime subscribed to the module's subjects before the bus had reloaded its user list. The bus refused, and the Go NATS client marks a refused subscription as dead. The module ran unreachable until a much later membership happened to re-serve it.

The runtime now installs an error handler. When the bus refuses a subject the runtime answers on, it subscribes again after 2s, 5s, 10s, 20s, 30s, 60s, 60s and 120s. After that it gives up and logs it. A subject the runtime has stopped answering is not retried.

Tests:

TestARefusedSubscriptionIsAskedForAgain covers asking again, answering afterwards, giving up, and not retrying a stopped subject.
End to end, by hand, against a NATS server whose permissions were reloaded while connected: the new runtime was refused, asked again 2 s later, and answered once the grant arrived. The old runtime never answered within 30 s.
The full suite passes, run one package at a time. Run in parallel against one shared bus, the console's request-counting test sees other packages' traffic. That happens on main too.

A module was assigned to a machine, and that machine was pushed. The push also sent the bus's machine, in the same second, but the runtime subscribed to the module's subjects before the bus had reloaded its user list. The bus refused, and the Go NATS client marks a refused subscription as dead. The module ran unreachable until a much later membership happened to re-serve it. The runtime now installs an error handler. When the bus refuses a subject the runtime answers on, it subscribes again after 2s, 5s, 10s, 20s, 30s, 60s, 60s and 120s. After that it gives up and logs it. A subject the runtime has stopped answering is not retried. Tests: - `TestARefusedSubscriptionIsAskedForAgain` covers asking again, answering afterwards, giving up, and not retrying a stopped subject. - End to end, by hand, against a NATS server whose permissions were reloaded while connected: the new runtime was refused, asked again 2 s later, and answered once the grant arrived. The old runtime never answered within 30 s. - The full suite passes, run one package at a time. Run in parallel against one shared bus, the console's request-counting test sees other packages' traffic. That happens on main too.

mesh-admin added 1 commit 2026-10-03 23:20:46 +00:00

Ask the bus again for a subscription it refused (novox/hq issue 222) 85810ccc8c

A push sends the machine that needs a grant and the bus's machine in the same breath, and the
runtime can subscribe the moment before the bus reloads its user list. Refused once, the
subscription stayed dead until some later membership re-served it, and a newly assigned module ran
unreachable. A refused subject the runtime answers is now asked for again for about five minutes.

mesh-admin merged commit 47cdfad754 into main

2026-10-03 23:20:50 +00:00

mesh-admin deleted branch fix/issue-222-a-refused-subscription-is-asked-again

2026-10-03 23:20:50 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-tools#47