|
|
|
@@ -5,7 +5,7 @@ code:
|
|
|
|
|
- mesh-catalog modules/nats
|
|
|
|
|
- mesh-controller internal/catalogue
|
|
|
|
|
- mesh-lab scenarios
|
|
|
|
|
updated: 2026-09-28
|
|
|
|
|
updated: 2026-09-27
|
|
|
|
|
decisions:
|
|
|
|
|
- 02-DECISIONS/0116-the-bus-is-built-in-five-steps.md
|
|
|
|
|
- 02-DECISIONS/0106-the-bus-is-nats.md
|
|
|
|
@@ -535,7 +535,7 @@ it, and the beds that need a mesh living on NATS can finally run.
|
|
|
|
|
The outcome carries the module name, because only the manifest says what was built and one
|
|
|
|
|
message now has three readers. A failed build names none: it produced no module version, and
|
|
|
|
|
the catalogue would otherwise place something that was never made.
|
|
|
|
|
- [x] 4.3 an installation completes over the bus, with the same outcome as the path it replaces —
|
|
|
|
|
- [~] 4.3 an installation completes over the bus, with the same outcome as the path it replaces —
|
|
|
|
|
**the installer can raise it**: a foundation template that stands up the server, writes the
|
|
|
|
|
server's own settings and the mesh's first user list beside them, and starts a controller
|
|
|
|
|
reaching the new bus. What remains is running it, which is 4.1's bed.
|
|
|
|
@@ -655,35 +655,73 @@ healthy while reacting to nothing.
|
|
|
|
|
- [ ] 5.1 the cutover bed: a mesh on AMQP with a predecessor stand-in on the deprecated broker
|
|
|
|
|
moves its bus in one rollout, every node reporting on NATS afterwards, the stand-in's own
|
|
|
|
|
client still connected throughout
|
|
|
|
|
- [x] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
|
|
|
|
|
- [~] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
|
|
|
|
|
together; every node confirmed heard before AMQP stops.
|
|
|
|
|
|
|
|
|
|
**Done 2026-09-28, 02:25.** Every machine reports on the new bus, the seat is held by the
|
|
|
|
|
module that provides it, the old broker is unassigned and forgotten, and every credential was
|
|
|
|
|
minted afresh at the end because two had been printed on the way. What it took, in the order
|
|
|
|
|
it was found, each fixed on the trunk before the next step: the control plane's `serve` and
|
|
|
|
|
`push` never selected the new transport (task 4.3, open until then); a machine's user was
|
|
|
|
|
granted neither the asking nor the delivery of its own consumer; the account had no JetStream
|
|
|
|
|
of its own; the control plane's client verified the bus's certificate by name instead of
|
|
|
|
|
pinning it; the seat table's rows carried no protocol, so no role's work queue was raised; the
|
|
|
|
|
build machine decided its bus from a variable its container never received; and a rotation
|
|
|
|
|
put new hashes on the bus before three machines had received their new memberships — which
|
|
|
|
|
is why there is now `rollout hand <node>` and a host adopts a delivered membership at start.
|
|
|
|
|
The bootstrap loop — a bus that can only be raised by a declaration that can only arrive
|
|
|
|
|
over that bus — was broken once, by hand: the mesh's own composed configuration started the
|
|
|
|
|
server, and the controller binary was run on the node directly until the managed container
|
|
|
|
|
could be rebuilt over the bus it was on.
|
|
|
|
|
**The readiness half is in and is the half worth having.** The move takes every node at once, so
|
|
|
|
|
there is nothing to inspect afterwards and no half to roll back — either the mesh was ready or it
|
|
|
|
|
was not. `rollout check` answers that from records, with one dial: is a bus answering, does a
|
|
|
|
|
machine hold the seat, has it been sent the composed user list, does every machine and every
|
|
|
|
|
module that speaks have a credential. Each missing thing names its own next step, because "not
|
|
|
|
|
ready" that cannot be acted on is not an answer at the point where the next step is irreversible.
|
|
|
|
|
|
|
|
|
|
**A machine with no credential is what must stop it.** It keeps running, cannot come back, and
|
|
|
|
|
afterwards there is no bus to tell it anything over.
|
|
|
|
|
|
|
|
|
|
The move itself is deliberately not written yet, and the command says so rather than pretending:
|
|
|
|
|
it waits on the check having been run against a real mesh. Writing the irreversible half before
|
|
|
|
|
the question it depends on has ever been asked of something real is how the plan's own rule about
|
|
|
|
|
beds gets broken by another route.
|
|
|
|
|
|
|
|
|
|
> **What this costs if it goes wrong, measured rather than assumed.** Nothing in a served
|
|
|
|
|
> request's path goes over the mesh's own bus: modules serve from their own containers. What a
|
|
|
|
|
> failed move costs is the mesh's ability to *change* anything — pushes, tool calls, new
|
|
|
|
|
> provisioning — until it is finished or undone. That is worth knowing before rather than
|
|
|
|
|
> after, and it is why the operator's "as long as my services keep running" is a reasonable
|
|
|
|
|
> position rather than a gamble. **Measured on 2026-09-27**, when a seat emptied itself
|
|
|
|
|
> mid-change: 52 containers stayed up and the broker never stopped; the control plane
|
|
|
|
|
> crash-looped for two hours and nothing could be deployed until it was repaired by hand.
|
|
|
|
|
> An earlier version of this note said the old broker stays as an ordinary provider of
|
|
|
|
|
> `amqp` ([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)); that is withdrawn by [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) — see 5.4.
|
|
|
|
|
|
|
|
|
|
**What the first live attempt found, 2026-09-27.** With the seat handed over on record and the
|
|
|
|
|
new bus's module registered and assigned beside the old one, `push` refused the control node:
|
|
|
|
|
*not one user has a credential for the new bus*. The check is right — a bus whose user list is
|
|
|
|
|
empty refuses every connection in the mesh — and it exposed the half of this task nobody had
|
|
|
|
|
built. A credential is minted at three moments only: a machine's at enrolment, a module's at
|
|
|
|
|
`module issue`, a person's at `operator`. **Nothing mints one for a machine already enrolled, or
|
|
|
|
|
for the control plane itself.** And on the host, the membership — bus address, fingerprint,
|
|
|
|
|
password, transport — is written once, at enrolment, and nothing ever rewrites it. So "move
|
|
|
|
|
each machine and confirm it reports" had no mechanism under it on either side.
|
|
|
|
|
|
|
|
|
|
The mechanism, to build before anything moves:
|
|
|
|
|
- **the control plane mints what is missing** — every user the records derive with no hash —
|
|
|
|
|
and delivers each plaintext where its owner reads it: a machine's inside its declaration, as a
|
|
|
|
|
sealed *membership* for the new bus (address, fingerprint, password, transport); a module's as
|
|
|
|
|
its broker secret, the path `module issue` already uses; the control plane's own as its module
|
|
|
|
|
secret, so it reads it the way any module does;
|
|
|
|
|
- **the host saves a delivered membership and re-dials on it** — the same file enrolment wrote,
|
|
|
|
|
the same reconnect path a lost connection takes, so a machine moved this way is a machine
|
|
|
|
|
that came back, and nothing new has to be right for it to work;
|
|
|
|
|
- **the switch is then two acts in one push**: `MESH_BUS_NATS` on the control plane, and
|
|
|
|
|
`seat mesh-broker --to <node>/<the new bus's module>` — the seat never empty, every machine
|
|
|
|
|
already holding a credential that works on the other side.
|
|
|
|
|
|
|
|
|
|
Until the first bullet exists the check keeps refusing, and it should: a machine moved without
|
|
|
|
|
a credential cannot come back, and afterwards there is no bus to tell it anything over.
|
|
|
|
|
- [x] 5.3 **the seat changes hands as one act.** A command takes a seat and the assignment taking it
|
|
|
|
|
over, and the seat is never empty in between — the emptiness is the outage of 2026-09-27, when
|
|
|
|
|
the control plane, which finds its own bus through this seat, lost the address and looped.
|
|
|
|
|
**Built 2026-09-27** (`seat_holding`, migration 0039; design 26 says how it is checked), and used
|
|
|
|
|
live the next night to hand `mesh-broker` from the old broker's assignment to the new one's. This
|
|
|
|
|
is what 5.2 uses to move `mesh-broker` from the old
|
|
|
|
|
broker's assignment to the new one's, and it is built first ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
|
|
|
|
|
- [x] 5.4 **the old broker and everything that named AMQP leave the mesh** ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md),
|
|
|
|
|
**Built 2026-09-27** (mesh-controller `seat_holding`, migration 0039; design 26 says how it is
|
|
|
|
|
checked). Its first live use recorded the standing holder — which the row moving under it had
|
|
|
|
|
made unable to satisfy what the seat delivers, so the first handover on a mesh that predates the
|
|
|
|
|
record writes down who holds without re-judging them. This is what 5.2 uses to move
|
|
|
|
|
`mesh-broker` from the old broker's assignment to the new one's ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
|
|
|
|
|
- [~] 5.4 **the old broker and everything that named AMQP leave the mesh** — the catalogue half done
|
|
|
|
|
2026-09-27 (three modules removed; registration refuses the word; the seat's row delivers
|
|
|
|
|
`mesh-bus`, migration 0040); the live half — unassigning the old broker — waits on 5.2 ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md),
|
|
|
|
|
superseding [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)): the two modules that
|
|
|
|
|
required `amqp` are removed, the broker's module is unassigned and removed (**done 2026-09-28**; the predecessor's own tooling, which rode the same adopted broker, went dark with it, as [ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md) accepted), registration refuses
|
|
|
|
|
required `amqp` are removed, the broker's module is unassigned and removed, registration refuses
|
|
|
|
|
a manifest that provides or requires `amqp`, and a whole-catalogue check asserts none does. Not
|
|
|
|
|
a retirement condition — a decision, taken, with the operator's "I don't care if the predecessor
|
|
|
|
|
breaks" on record ([ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)).
|
|
|
|
@@ -694,27 +732,8 @@ healthy while reacting to nothing.
|
|
|
|
|
> shutting it down ends the path that reaches this installation's machines from a workstation.
|
|
|
|
|
> The rollout is driven from the node, or before the broker stops — a sequencing constraint on
|
|
|
|
|
> 5.2, not an afterthought.
|
|
|
|
|
- [x] 5.5 **the AMQP transport is deleted from the control plane and the hosts**. One bus, nothing
|
|
|
|
|
to select ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
|
|
|
|
|
|
|
|
|
|
**Done 2026-09-28.** The control plane's old consume loop, build request, tool call, management
|
|
|
|
|
API and account scoping went, and the host's old dialling and enrolment paths with them; a
|
|
|
|
|
membership or token naming any other bus is refused before anything is sent. Nothing selects a
|
|
|
|
|
transport any more: the variable that once did (`MESH_BUS_NATS`) now only names where the
|
|
|
|
|
control plane reads its own bus credential, the way any module reads a secret. **Checked by the
|
|
|
|
|
build**: neither repository's module file names the AMQP client library, so a line that still
|
|
|
|
|
used it would not compile. The store-window guarantee ([issue 083](../../04-ISSUES/083-other-control-messages-are-lost-while-the-store-restarts/00-report.md))
|
|
|
|
|
is tested against a bus-less fake rather than the old transport's memory, which is what let
|
|
|
|
|
that memory go — the one thing it did that the stream does not (superseding a held report) is
|
|
|
|
|
the staleness check on the message itself (design 25 §3).
|
|
|
|
|
|
|
|
|
|
Found on the way: **no build had ever recorded what it stood on.** A recipe reads its base from
|
|
|
|
|
a build argument, so the digest was never in the file the builder derived edges from, and every
|
|
|
|
|
order that says *bases first* — `build --on`, `build --behind`, the merge follow-up of
|
|
|
|
|
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — walked a
|
|
|
|
|
graph with no edges. The builder now reports the bases it was handed, the control plane records
|
|
|
|
|
them by artifact path, and the graph is read from the newest build of each module — a recorded
|
|
|
|
|
manifest carries no `build.on`, so the edge is derived from the build or it does not exist.
|
|
|
|
|
- [ ] 5.5 **the AMQP transport is deleted from the control plane and the hosts**, and the variable
|
|
|
|
|
that selected a transport is refused at start as unknown. One bus, nothing to select ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
|
|
|
|
|
|
|
|
|
|
> **The old 5.4 note is history.** It recorded that a retirement *condition* was wrong from
|
|
|
|
|
> [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) onward, which framed the old broker
|
|
|
|
|