Design 28: the mesh runs on the new bus #155

Merged
mesh-admin merged 2 commits from design/28-the-mesh-runs-on-nats into main 2026-09-28 00:40:29 +00:00
+23 -30
View File
@@ -535,7 +535,7 @@ it, and the beds that need a mesh living on NATS can finally run.
The outcome carries the module name, because only the manifest says what was built and one The outcome carries the module name, because only the manifest says what was built and one
message now has three readers. A failed build names none: it produced no module version, and message now has three readers. A failed build names none: it produced no module version, and
the catalogue would otherwise place something that was never made. the catalogue would otherwise place something that was never made.
- [~] 4.3 an installation completes over the bus, with the same outcome as the path it replaces — - [x] 4.3 an installation completes over the bus, with the same outcome as the path it replaces —
**the installer can raise it**: a foundation template that stands up the server, writes the **the installer can raise it**: a foundation template that stands up the server, writes the
server's own settings and the mesh's first user list beside them, and starts a controller server's own settings and the mesh's first user list beside them, and starts a controller
reaching the new bus. What remains is running it, which is 4.1's bed. reaching the new bus. What remains is running it, which is 4.1's bed.
@@ -655,42 +655,35 @@ healthy while reacting to nothing.
- [ ] 5.1 the cutover bed: a mesh on AMQP with a predecessor stand-in on the deprecated broker - [ ] 5.1 the cutover bed: a mesh on AMQP with a predecessor stand-in on the deprecated broker
moves its bus in one rollout, every node reporting on NATS afterwards, the stand-in's own moves its bus in one rollout, every node reporting on NATS afterwards, the stand-in's own
client still connected throughout client still connected throughout
- [~] 5.2 the rollout: accounts composed, then the controller, every host and every runtime - [x] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
together; every node confirmed heard before AMQP stops. together; every node confirmed heard before AMQP stops.
**The readiness half is in and is the half worth having.** The move takes every node at once, so **Done 2026-09-28, 02:25.** Every machine reports on the new bus, the seat is held by the
there is nothing to inspect afterwards and no half to roll back — either the mesh was ready or it module that provides it, the old broker is unassigned and forgotten, and every credential was
was not. `rollout check` answers that from records, with one dial: is a bus answering, does a minted afresh at the end because two had been printed on the way. What it took, in the order
machine hold the seat, has it been sent the composed user list, does every machine and every it was found, each fixed on the trunk before the next step: the control plane's `serve` and
module that speaks have a credential. Each missing thing names its own next step, because "not `push` never selected the new transport (task 4.3, open until then); a machine's user was
ready" that cannot be acted on is not an answer at the point where the next step is irreversible. granted neither the asking nor the delivery of its own consumer; the account had no JetStream
of its own; the control plane's client verified the bus's certificate by name instead of
pinning it; the seat table's rows carried no protocol, so no role's work queue was raised; the
build machine decided its bus from a variable its container never received; and a rotation
put new hashes on the bus before three machines had received their new memberships — which
is why there is now `rollout hand <node>` and a host adopts a delivered membership at start.
The bootstrap loop — a bus that can only be raised by a declaration that can only arrive
over that bus — was broken once, by hand: the mesh's own composed configuration started the
server, and the controller binary was run on the node directly until the managed container
could be rebuilt over the bus it was on.
**A machine with no credential is what must stop it.** It keeps running, cannot come back, and - [x] 5.3 **the seat changes hands as one act.** A command takes a seat and the assignment taking it
afterwards there is no bus to tell it anything over.
The move itself is deliberately not written yet, and the command says so rather than pretending:
it waits on the check having been run against a real mesh. Writing the irreversible half before
the question it depends on has ever been asked of something real is how the plan's own rule about
beds gets broken by another route.
> **What this costs if it goes wrong, measured rather than assumed.** Nothing in a served
> request's path goes over the mesh's own bus: modules serve from their own containers. What a
> failed move costs is the mesh's ability to *change* anything — pushes, tool calls, new
> provisioning — until it is finished or undone. That is worth knowing before rather than
> after, and it is why the operator's "as long as my services keep running" is a reasonable
> position rather than a gamble. **Measured on 2026-09-27**, when a seat emptied itself
> mid-change: 52 containers stayed up and the broker never stopped; the control plane
> crash-looped for two hours and nothing could be deployed until it was repaired by hand.
> An earlier version of this note said the old broker stays as an ordinary provider of
> `amqp` ([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)); that is withdrawn by [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) — see 5.4.
- [ ] 5.3 **the seat changes hands as one act.** A command takes a seat and the assignment taking it
over, and the seat is never empty in between — the emptiness is the outage of 2026-09-27, when over, and the seat is never empty in between — the emptiness is the outage of 2026-09-27, when
the control plane, which finds its own bus through this seat, lost the address and looped. the control plane, which finds its own bus through this seat, lost the address and looped.
Today only `seat rename` exists. This is what 5.2 uses to move `mesh-broker` from the old **Built 2026-09-27** (`seat_holding`, migration 0039; design 26 says how it is checked), and used
live the next night to hand `mesh-broker` from the old broker's assignment to the new one's. This
is what 5.2 uses to move `mesh-broker` from the old
broker's assignment to the new one's, and it is built first ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)). broker's assignment to the new one's, and it is built first ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
- [ ] 5.4 **the old broker and everything that named AMQP leave the mesh** ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md), - [x] 5.4 **the old broker and everything that named AMQP leave the mesh** ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md),
superseding [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)): the two modules that superseding [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)): the two modules that
required `amqp` are removed, the broker's module is unassigned and removed, registration refuses required `amqp` are removed, the broker's module is unassigned and removed (**done 2026-09-28**; the predecessor's own tooling, which rode the same adopted broker, went dark with it, as [ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md) accepted), registration refuses
a manifest that provides or requires `amqp`, and a whole-catalogue check asserts none does. Not a manifest that provides or requires `amqp`, and a whole-catalogue check asserts none does. Not
a retirement condition — a decision, taken, with the operator's "I don't care if the predecessor a retirement condition — a decision, taken, with the operator's "I don't care if the predecessor
breaks" on record ([ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)). breaks" on record ([ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)).