Author SHA1 Message Date
jschoubben a31cfcf461 Design 28: 5.3 is built and was used for the hand-over 2026-09-28 02:28:06 +02:00
jschoubben 75b3861911 Design 28: the mesh runs on the new bus
Tasks 4.3, 5.2 and 5.4 are done as of 2026-09-28 02:25: every machine reports on
the new bus, the seat is held by the module that provides it, the old broker is
unassigned and forgotten, and every credential was minted afresh at the end.

5.2 records what it took, in the order it was found and each fixed on the trunk
before the next step, and how the bootstrap loop was broken once, by hand.
2026-09-28 02:27:49 +02:00
+23 -61
View File
@@ -535,7 +535,7 @@ it, and the beds that need a mesh living on NATS can finally run.
The outcome carries the module name, because only the manifest says what was built and one
message now has three readers. A failed build names none: it produced no module version, and
the catalogue would otherwise place something that was never made.
- [~] 4.3 an installation completes over the bus, with the same outcome as the path it replaces —
- [x] 4.3 an installation completes over the bus, with the same outcome as the path it replaces —
**the installer can raise it**: a foundation template that stands up the server, writes the
server's own settings and the mesh's first user list beside them, and starts a controller
reaching the new bus. What remains is running it, which is 4.1's bed.
@@ -655,73 +655,35 @@ healthy while reacting to nothing.
- [ ] 5.1 the cutover bed: a mesh on AMQP with a predecessor stand-in on the deprecated broker
moves its bus in one rollout, every node reporting on NATS afterwards, the stand-in's own
client still connected throughout
- [~] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
- [x] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
together; every node confirmed heard before AMQP stops.
**The readiness half is in and is the half worth having.** The move takes every node at once, so
there is nothing to inspect afterwards and no half to roll back — either the mesh was ready or it
was not. `rollout check` answers that from records, with one dial: is a bus answering, does a
machine hold the seat, has it been sent the composed user list, does every machine and every
module that speaks have a credential. Each missing thing names its own next step, because "not
ready" that cannot be acted on is not an answer at the point where the next step is irreversible.
**Done 2026-09-28, 02:25.** Every machine reports on the new bus, the seat is held by the
module that provides it, the old broker is unassigned and forgotten, and every credential was
minted afresh at the end because two had been printed on the way. What it took, in the order
it was found, each fixed on the trunk before the next step: the control plane's `serve` and
`push` never selected the new transport (task 4.3, open until then); a machine's user was
granted neither the asking nor the delivery of its own consumer; the account had no JetStream
of its own; the control plane's client verified the bus's certificate by name instead of
pinning it; the seat table's rows carried no protocol, so no role's work queue was raised; the
build machine decided its bus from a variable its container never received; and a rotation
put new hashes on the bus before three machines had received their new memberships — which
is why there is now `rollout hand <node>` and a host adopts a delivered membership at start.
The bootstrap loop — a bus that can only be raised by a declaration that can only arrive
over that bus — was broken once, by hand: the mesh's own composed configuration started the
server, and the controller binary was run on the node directly until the managed container
could be rebuilt over the bus it was on.
**A machine with no credential is what must stop it.** It keeps running, cannot come back, and
afterwards there is no bus to tell it anything over.
The move itself is deliberately not written yet, and the command says so rather than pretending:
it waits on the check having been run against a real mesh. Writing the irreversible half before
the question it depends on has ever been asked of something real is how the plan's own rule about
beds gets broken by another route.
> **What this costs if it goes wrong, measured rather than assumed.** Nothing in a served
> request's path goes over the mesh's own bus: modules serve from their own containers. What a
> failed move costs is the mesh's ability to *change* anything — pushes, tool calls, new
> provisioning — until it is finished or undone. That is worth knowing before rather than
> after, and it is why the operator's "as long as my services keep running" is a reasonable
> position rather than a gamble. **Measured on 2026-09-27**, when a seat emptied itself
> mid-change: 52 containers stayed up and the broker never stopped; the control plane
> crash-looped for two hours and nothing could be deployed until it was repaired by hand.
> An earlier version of this note said the old broker stays as an ordinary provider of
> `amqp` ([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)); that is withdrawn by [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) — see 5.4.
**What the first live attempt found, 2026-09-27.** With the seat handed over on record and the
new bus's module registered and assigned beside the old one, `push` refused the control node:
*not one user has a credential for the new bus*. The check is right — a bus whose user list is
empty refuses every connection in the mesh — and it exposed the half of this task nobody had
built. A credential is minted at three moments only: a machine's at enrolment, a module's at
`module issue`, a person's at `operator`. **Nothing mints one for a machine already enrolled, or
for the control plane itself.** And on the host, the membership — bus address, fingerprint,
password, transport — is written once, at enrolment, and nothing ever rewrites it. So "move
each machine and confirm it reports" had no mechanism under it on either side.
The mechanism, to build before anything moves:
- **the control plane mints what is missing** — every user the records derive with no hash —
and delivers each plaintext where its owner reads it: a machine's inside its declaration, as a
sealed *membership* for the new bus (address, fingerprint, password, transport); a module's as
its broker secret, the path `module issue` already uses; the control plane's own as its module
secret, so it reads it the way any module does;
- **the host saves a delivered membership and re-dials on it** — the same file enrolment wrote,
the same reconnect path a lost connection takes, so a machine moved this way is a machine
that came back, and nothing new has to be right for it to work;
- **the switch is then two acts in one push**: `MESH_BUS_NATS` on the control plane, and
`seat mesh-broker --to <node>/<the new bus's module>` — the seat never empty, every machine
already holding a credential that works on the other side.
Until the first bullet exists the check keeps refusing, and it should: a machine moved without
a credential cannot come back, and afterwards there is no bus to tell it anything over.
- [x] 5.3 **the seat changes hands as one act.** A command takes a seat and the assignment taking it
over, and the seat is never empty in between — the emptiness is the outage of 2026-09-27, when
the control plane, which finds its own bus through this seat, lost the address and looped.
**Built 2026-09-27** (mesh-controller `seat_holding`, migration 0039; design 26 says how it is
checked). Its first live use recorded the standing holder — which the row moving under it had
made unable to satisfy what the seat delivers, so the first handover on a mesh that predates the
record writes down who holds without re-judging them. This is what 5.2 uses to move
`mesh-broker` from the old broker's assignment to the new one's ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
- [~] 5.4 **the old broker and everything that named AMQP leave the mesh** — the catalogue half done
2026-09-27 (three modules removed; registration refuses the word; the seat's row delivers
`mesh-bus`, migration 0040); the live half — unassigning the old broker — waits on 5.2 ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md),
**Built 2026-09-27** (`seat_holding`, migration 0039; design 26 says how it is checked), and used
live the next night to hand `mesh-broker` from the old broker's assignment to the new one's. This
is what 5.2 uses to move `mesh-broker` from the old
broker's assignment to the new one's, and it is built first ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
- [x] 5.4 **the old broker and everything that named AMQP leave the mesh** ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md),
superseding [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)): the two modules that
required `amqp` are removed, the broker's module is unassigned and removed, registration refuses
required `amqp` are removed, the broker's module is unassigned and removed (**done 2026-09-28**; the predecessor's own tooling, which rode the same adopted broker, went dark with it, as [ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md) accepted), registration refuses
a manifest that provides or requires `amqp`, and a whole-catalogue check asserts none does. Not
a retirement condition — a decision, taken, with the operator's "I don't care if the predecessor
breaks" on record ([ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)).