Archived
Retire the controller's merges consumer and read merges from the forge bridge's stream, so a held merge can never again queue every merge or look like a deaf controller (issue 774)
The controller-merges consumer (no max-deliver, one at a time) held the de-spiegel merge e25e1beb from 10-29 09:39: keepForgeMerge put an announcement with its Paths into delivery-merges (16 KiB values), got 10054, retried "not now" forever, and 167 merges queued behind it. Every five-minute hold outran the deaf bound, so core.controller.deaf flapped. Merges are mesh-delivery's: the gate merges on the Approve and the forge bridge says each merge on its stream FORGE (pull request, head, merge commit), kept a week. The controller's copy was a leftover, read only at a moved module's registration (ADR 0420 §7, ADR 0432). That read now finds the merge on FORGE; the consumer, its follow, its handler and its hold delays are removed, and every assertion removes the retired consumer from the bus. Kind fix for the watchdog: the event loop records when a message it held back is due again, and deafness counts from then, so no hold longer than the bound (on any consumer) reads as deafness. Quick fix: strip Paths from the kept announcement and add max-deliver with a dead letter. Thorough fix: remove the duplicate duty and make the deaf watchdog hold-aware. Chose thorough: the controller is being deprecated and the merges already have an owner; the hold-vs-deaf confusion is a kind, not one consumer. Research 072 piece: 03-domains (Control shrinks to placement and composition; the delivery parts own what a merge is) and milestone 3 (health signals stop sharing the controller's loop assumptions).
This commit is contained in:
+34
-22
@@ -339,13 +339,36 @@ func Overlaps() []string {
|
||||
// ack subject is derived from (nats.go: `$JS.ACK.<stream>.controller.>`).
|
||||
const ControllerName = "controller"
|
||||
|
||||
// MergesConsumerName is the controller's durable consumer of the forge's merged event on EVENTS, apart from the
|
||||
// events consumer (novox/hq ADR 0407 decision 3, issue 973). A merge is never dropped, so this consumer has no
|
||||
// max-deliver, which would send a merge held five times to DEAD_LETTERS to wait for a person, and no healer
|
||||
// resets it, which would drop the merges it holds. A consumer of its own because both of those are properties of
|
||||
// a consumer, not of a subject.
|
||||
// MergesConsumerName was the controller's durable consumer of the forge's merged event on EVENTS (novox/hq ADR 0407
|
||||
// decision 3, issue 973), with no max-deliver and one message at a time. **Retired** (sweep 2026-10-30, issue 774):
|
||||
// the forge bridge's stream FORGE is the record of the merges, and the controller reads it where it needs a merge. On
|
||||
// 2026-10-29 one announcement too large for the controller's bucket of merges was held on it for over a day, with
|
||||
// 163 merges queued behind it, and the deaf watchdog read the queue as a deaf controller. RetiredConsumers removes it.
|
||||
const MergesConsumerName = "controller-merges"
|
||||
|
||||
// RetiredConsumers are durable consumers the controller once held and holds no more: each assertion removes them,
|
||||
// so a retired consumer cannot keep a queue nobody takes from (research 072, 03-domains: the controller shrinks).
|
||||
var RetiredConsumers = []Consumer{{Name: MergesConsumerName, Stream: EventsStream}}
|
||||
|
||||
// Remover is the part of a JetStream connection that removes a retired consumer.
|
||||
type Remover interface {
|
||||
RemoveConsumer(stream, name string) error
|
||||
}
|
||||
|
||||
// removeRetired removes every retired consumer, when the connection can.
|
||||
func removeRetired(e Ensurer) error {
|
||||
r, ok := e.(Remover)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
for _, c := range RetiredConsumers {
|
||||
if err := r.RemoveConsumer(c.Stream, c.Name); err != nil {
|
||||
return fmt.Errorf("removing the retired consumer %s on %s: %w", c.Name, c.Stream, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ControllerSeat is the role the control plane holds, and ControllerStates are the facts it states
|
||||
// under it (novox/hq ADR 0134).
|
||||
//
|
||||
@@ -513,23 +536,6 @@ func ControllerConsumers() []Consumer {
|
||||
Why: "the controller is the single consumer of what nodes say; explicit ack and no " +
|
||||
"max-deliver, because the store window's bound is the controller's own",
|
||||
},
|
||||
// The forge's merges (novox/hq ADR 0407 decision 3, issue 973): never dropped, so no max-deliver and no
|
||||
// reset; a merge the controller cannot record yet is held with a growing delay until it can. **Asserted before
|
||||
// the events consumer**, which stops filtering merges: the order of this list is the order of assertion.
|
||||
{
|
||||
Name: MergesConsumerName,
|
||||
Stream: "EVENTS",
|
||||
// Within what the controller may subscribe already (`_DELIVER.controller.>`).
|
||||
DeliverSubject: "_DELIVER." + ControllerName + ".merges",
|
||||
Filters: []string{ControllerFollows[3]},
|
||||
Push: true,
|
||||
AckWaitSeconds: 30,
|
||||
MaxAckPending: 1,
|
||||
FromNow: true,
|
||||
TakesOverFrom: ControllerName,
|
||||
Why: "the forge's merges, one at a time; no max-deliver and never reset, because a merge is never " +
|
||||
"dropped and no person releases one",
|
||||
},
|
||||
{
|
||||
Name: ControllerName,
|
||||
Stream: "EVENTS",
|
||||
@@ -581,6 +587,9 @@ func NoticesConsumer() Consumer {
|
||||
// one on DEAD_LETTER_NOTICES, which the keeper of dead letters asserts and retries by itself, so a fault
|
||||
// there never stops the controller serving (novox/hq issue 330).
|
||||
func AssertServingConsumers(e Ensurer) error {
|
||||
if err := removeRetired(e); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, c := range MeshConsumers() {
|
||||
if c.Stream == DeadLetterNoticesStream {
|
||||
continue
|
||||
@@ -603,6 +612,9 @@ type Ensurer interface {
|
||||
// After the streams, necessarily: a consumer on a stream that does not exist is refused, and the
|
||||
// refusal names the stream rather than the order.
|
||||
func AssertMeshConsumers(e Ensurer) error {
|
||||
if err := removeRetired(e); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, c := range MeshConsumers() {
|
||||
if err := e.EnsureConsumer(c); err != nil {
|
||||
return fmt.Errorf("asserting consumer %s on %s: %w", c.Name, c.Stream, err)
|
||||
|
||||
Reference in New Issue
Block a user