The store owns the seat set, so only the control plane may judge a claim #89

2026-09-27T19:59:54Z

jschoubben commented

2026-09-27 19:59:54 +00:00

This is the fix for tonight's live breakage, and the fault is smaller and more interesting than the outage looked.

A claim on a seat was checked inside ParseManifest, against SeatNamed. The build machine parses manifests too, and it has no store — so there, SeatNamed answered from the set compiled into the binary. ADR 0122 moved that set into the store precisely so it would be data, and this left a copy of it deciding things.

When the two disagreed the copy won where it mattered. The store's row says the bus seat answers for amqp. The binary's said mesh-bus. A holder that provides amqp was refused at build time for not providing mesh-bus, so the seat could not be filled, the controller lost the address it composes through that seat, and it crash-looped against a broker that was healthy throughout.

The two checks that read the set — a seat's scope, and what its holder must provide — move to CatalogueProblems, which runs only in the control plane and only after UseSeats has replaced the set with the store's. The parser keeps everything it can judge from a manifest alone, including the mesh-*/node-* reserved-namespace rule, which is a prefix and not a lookup.

A test pins the behaviour rather than the mechanism: the same manifest, two different values in the store, and the answer follows the store both times. It fails if anyone moves the check back.

Full suite green against a real NATS and store. Three existing tests moved their assertion from the parser to registration, which is the rule changing address, not changing.

This is the fix for tonight's live breakage, and the fault is smaller and more interesting than the outage looked. A claim on a seat was checked inside `ParseManifest`, against `SeatNamed`. The build machine parses manifests too, and it has no store — so there, `SeatNamed` answered from the set compiled into the binary. ADR 0122 moved that set into the store precisely so it would be data, and this left a copy of it deciding things. When the two disagreed the copy won where it mattered. The store's row says the bus seat answers for `amqp`. The binary's said `mesh-bus`. A holder that provides `amqp` was refused *at build time* for not providing `mesh-bus`, so the seat could not be filled, the controller lost the address it composes through that seat, and it crash-looped against a broker that was healthy throughout. The two checks that read the set — a seat's scope, and what its holder must provide — move to `CatalogueProblems`, which runs only in the control plane and only after `UseSeats` has replaced the set with the store's. The parser keeps everything it can judge from a manifest alone, including the `mesh-*`/`node-*` reserved-namespace rule, which is a prefix and not a lookup. A test pins the behaviour rather than the mechanism: the same manifest, two different values in the store, and the answer follows the store both times. It fails if anyone moves the check back. Full suite green against a real NATS and store. Three existing tests moved their assertion from the parser to registration, which is the rule changing address, not changing.

jschoubben added 1 commit 2026-09-27 19:59:54 +00:00

The store owns the seat set, so only the control plane may judge a claim 63ca073938

A claim on a seat was checked against `SeatNamed` inside `ParseManifest`, and the
build machine parses manifests too. It has no store, so there it answered from the
set compiled into the binary — a copy of data the control plane owns (ADR 0122).

When the two disagreed, that copy won where it mattered. The store's row said the
bus seat answers for `amqp`; the binary's said `mesh-bus`; and a holder that
provides `amqp` was refused at build time for not providing `mesh-bus`. The seat
went unheld, the controller lost the address it composes through that seat, and the
control plane crash-looped on a bus that was healthy the whole time.

So the two checks that read the set — a seat's scope, and what its holder must
provide — move to CatalogueProblems, which runs only in the control plane and only
after UseSeats has replaced the set with the store's. The parser keeps what it can
judge from the manifest alone, the reserved-namespace rule included.

A test pins it: the same manifest, two different values in the store, and the answer
follows the store both times. It fails if the check moves back.

jschoubben merged commit 4e4481b6f2 into main

2026-09-27 20:00:05 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#89