diff --git a/02-DECISIONS/0115-the-bus-is-built-in-five-steps.md b/02-DECISIONS/0115-the-bus-is-built-in-five-steps.md new file mode 100644 index 0000000..2d4c9ed --- /dev/null +++ b/02-DECISIONS/0115-the-bus-is-built-in-five-steps.md @@ -0,0 +1,142 @@ +--- +topic: the mesh +status: accepted +date: 2026-09-26 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0106-the-bus-is-nats.md +--- + +# 115. The bus is built in five steps, and the protocol moves with it + +## Context + +[ADR 0106](0106-the-bus-is-nats.md) decided the bus is NATS and described the change as one thing: +built beside the migration, cut over in one rollout after its core. The architecture was written as +[design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) and revised once after review. What neither +says is how the work is divided, and three gaps follow from that. + +**The whole of the build is one point.** Design 25 §9 numbers four items. The first — "the `nats` +module, the controller's and host's link on NATS, the runtime's client — built and proven in the +lab" — is every line of code the change requires; the other three are the rollout. A step of that +size ends at nothing provable until it ends at everything, which is the failure +[design 22](../03-DESIGN/01-to-be/22-the-work-ahead.md) opens by naming: *a phase that ends at a +claim is a phase that went missing without anything complaining.* + +**There is no adoption path.** Design 25 §5 says the broker "is raised at genesis like the store, +adopted as a module in the same phase" — which describes a mesh being raised from nothing. The mesh +this is for is already running, and a running node gets a foundation module by adoption in place +([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)), not by genesis. §9 goes +straight from "built beside" to "one rollout" and never crosses that gap. + +**The wire changes and the protocol specification does not know it.** Design 25 §8 says a module +sees nothing new. That is true of the SDK's contract — `request`, `handle`, `publish`, `subscribe` +— and false of the wire underneath it. +[ADR 0074](0074-the-wire-is-specified-not-the-types.md) settled that what an SDK implements is a +*specified wire*, checked by fixtures that must match byte for byte, precisely because two +implementations that disagree about an envelope do not fail to compile. That specification is +[design 19](../03-DESIGN/01-to-be/19-the-module-protocol.md), and it is written entirely in AMQP: +exchanges, a durable per-consumer queue named `..events`, a shared `serve.` +queue — sixteen occurrences of an AMQP term across the document. Design 25 does not cite design 19 +anywhere, and design 19 does not cite ADR 0106. So the record that says *what two implementations +may not disagree about* still describes the bus being replaced. + +## Considered Options + +1. **Keep ADR 0106's shape — build it all, cut over once.** Rejected: not for its rollout, which + is right, but because it leaves the build a single step of unknown length with no intermediate + anyone can run. The three gaps above were found by dividing it; they were invisible while it + was one item. +2. **Cut over incrementally by traffic kind** — events to NATS first, then tools, then control, + the mesh on two buses meanwhile. Rejected: ADR 0106 already rejected two buses, and this is + that with extra steps. The store-window guarantee + ([ADR 0083](0083-one-push-leaves-the-mesh-consistent.md)) is exactly the one that cannot cross + a seam, and control is exactly the traffic that carries it. +3. **Five steps, each ending at something provable, the cutover still one rollout.** The build is + divided; the bus still moves once. Adopted. + +## Decision + +**The bus is built in five steps. Each ends at something a lab bed proves, and no step's proof +waits for the one after it. The cutover remains a single rollout** — dividing the build does not +divide the bus. + +**Step 1 — genesis raises the broker.** The `nats` module and a mesh raised on it from nothing. +This is built and proven even though the mesh it is for will never travel this path, because +genesis is where the foundation is *defined*: the only place the mesh comes from nothing, and the +definition every other path is measured against. A genesis path that exists only on paper is one +nobody discovers is wrong until there is a second mesh. + +**Step 2 — adoption puts the broker in the seat.** A mesh already running receives the broker by +adoption in place, and the seat it claims is **`mesh-broker`** — unchanged. +[ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md) named the foundation seats +after the server's *role* rather than the product for exactly this case, and ADR 0106 restated it: +the broker module changes, the seat does not. A seat named after the product would have to be +renamed by every change the seat exists to survive. + +**Step 3 — the protocol gets a NATS binding.** ADR 0074 stands unamended: the mesh defines a module +protocol, an SDK is an implementation of it in one language and nothing more, the protocol is split +per capability, and conformance is executable fixtures per capability rather than prose. What +changes is what the specification specifies. Design 19's wire section is rewritten from exchanges +and queues to subjects and streams; the fixtures are recaptured on NATS; every SDK claims the +capabilities it passes, and a language may arrive with connection and events alone. + +**The SDK gains no conveniences in the process.** [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) +refuses frequent-and-cascading code in the shared library, and ADR 0074 restates it — an SDK is +"not a convenience layer, not a place for helpers to accumulate." A new transport is the moment +that pressure is highest and the reason to hold hardest: the predecessor's shared library is the +cautionary tale ADR 0039 opens with, and it did not become that in one decision. Code shared among +a module's own features stays in that module. + +**Step 4 — the core speaks NATS.** The controller's link, the host's link and the tool runtime's +client, and with them the flows that are today carried by something other than the bus: a build +source's change reaching the builder, an installation, a module's own reports. Each is a +conversion with a named before and after, not a rewrite. Observation — heartbeats, conditions, +key-value state — is [research 017](../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md)'s +and stays there; that effort already reserves it for after the move, and this step does not +pre-empt its design. + +**Step 5 — deployment.** The rollout ADR 0106 decided, unchanged: the controller, every host and +every runtime move together, each node confirmed to have heard before AMQP stops. What this step +adds is that steps 1 to 4 *ship ahead of it* without moving any node's bus — the module exists in +the catalogue, the protocol is specified, the code is written and beds pass, and the running mesh +is still on AMQP throughout. The bus moves on one day, at the end, once. + +## Consequences + +- **Design 25 §9 is replaced by these five steps** and §10's beds are attributed to the step each + proves, so no proof waits for the last step. +- **Design 19 is stale in its wire section from today** and says so in its own frontmatter and + opening until step 3 rewrites it. A specification that describes the bus being replaced is worse + than an absent one, because it reads as current. +- **`nats-broker` is not a seat.** The module is `nats`; the seat is `mesh-broker`. +- **Steps 1 to 4 leave every node on AMQP.** A step can be abandoned, or reordered after step 2, + without a rollback — the cost of being wrong is bounded until step 5. +- **What got harder:** five steps mean five proofs rather than one, and step 3 rewrites a design + other designs cite, so their references are checked when it lands. Dividing the work does not + reduce it. + +## How it is checked + +- **Per step, a bed, and the bed named in design 25 §10 against the step it belongs to.** Step 1: + a mesh raised on NATS from genesis. Step 2: the broker adopted into a mesh already running, and + a second holder of `mesh-broker` refused at resolution. Step 3: the conformance suite passing per + capability, on NATS, for every SDK that claims it — and a module built before the binding serving + its tools unchanged. Step 4: each converted flow proved against the behaviour it replaced. Step 5: + the cutover bed, then the rollout with every node reporting. +- **Step 3 is not done when the code runs.** It is done when the fixtures match byte for byte + across implementations, which is ADR 0074's own test and the only one that catches two SDKs + quietly ignoring each other. +- **A step that cannot name what its bed proves is not a step**, and is divided further before it + is started. + +## References + +- [ADR 0106](0106-the-bus-is-nats.md) — the decision this divides. +- [ADR 0074](0074-the-wire-is-specified-not-the-types.md) — the protocol and its conformance suite; + step 3 is its NATS binding, not a replacement. +- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — why step 3 adds no helpers. +- [ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md) — why the seat is + `mesh-broker`. +- [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) — the adoption step 2 uses. +- [design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md), [design 19](../03-DESIGN/01-to-be/19-the-module-protocol.md) — the two documents this changes. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 6b46026..a54f2ac 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -95,6 +95,7 @@ python3 00-META/checks/index.py fail if stale - **0104** — [A provision may be answered by an adapter to the predecessor](0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md) - **0105** — [The mesh adopts the predecessor's tunnel in place](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md) - **0106** — [The bus is NATS](0106-the-bus-is-nats.md) +- **0115** — [The bus is built in five steps, and the protocol moves with it](0115-the-bus-is-built-in-five-steps.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/19-the-module-protocol.md b/03-DESIGN/01-to-be/19-the-module-protocol.md index 8f0775b..5f37ac8 100644 --- a/03-DESIGN/01-to-be/19-the-module-protocol.md +++ b/03-DESIGN/01-to-be/19-the-module-protocol.md @@ -5,9 +5,11 @@ code: - mesh-sdk src - mesh-tools src/broker-amqp.ts - mesh-controller internal/link -updated: 2026-09-21 +updated: 2026-09-26 decisions: - 02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md + - 02-DECISIONS/0106-the-bus-is-nats.md + - 02-DECISIONS/0115-the-bus-is-built-in-five-steps.md - 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md - 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md - 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md @@ -23,6 +25,27 @@ language and nothing more ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specif This is a specification, so it says what is required rather than how anything is arranged. Where it describes current behaviour that is *not yet* specified-and-conformed, it says so. +> **The wire below is the bus being replaced.** *2026-09-26.* +> [ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md) moved the mesh's bus to NATS. Everything +> in this document that names an exchange, a queue or a routing key — the event exchanges, the +> durable `..events` queue, the shared `serve.` queue — describes the transport +> being retired, and the conformance fixtures were captured against it. +> +> What does **not** change is this document's model, which is the part ADR 0074 decided: a floor +> plus independent capabilities, an SDK that implements what it claims and is legitimate when it +> claims less, identity taken from the sealed credential rather than the environment, at-least-once +> with dedup on `x-event-id`, and conformance as executable fixtures per capability rather than +> prose. The envelope keeps its shape ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)); +> it becomes the message body. +> +> Rewriting the wire sections onto the subjects and streams of +> [design 25](25-the-bus-on-nats.md) §2–§3, and recapturing the fixtures there, is **step 3 of +> [ADR 0115](../../02-DECISIONS/0115-the-bus-is-built-in-five-steps.md)**. Until that lands, read +> the sections below for what two implementations may not disagree *about*, and design 25 for what +> they will disagree about it *on*. A specification that silently described a retired transport +> would be worse than an absent one, because it reads as current — hence this note rather than a +> quiet edit. + ## The shape of it A **floor** every implementation needs, and three **capabilities** that are independent of each diff --git a/03-DESIGN/01-to-be/25-the-bus-on-nats.md b/03-DESIGN/01-to-be/25-the-bus-on-nats.md index 3ffe4f7..beaa57a 100644 --- a/03-DESIGN/01-to-be/25-the-bus-on-nats.md +++ b/03-DESIGN/01-to-be/25-the-bus-on-nats.md @@ -6,9 +6,14 @@ code: - mesh-host internal/link (to be replaced) - mesh-tools src/broker-amqp.ts (to be replaced) - mesh-catalog modules/nats (to be written) -updated: 2026-09-24 + - mesh-sdk src (the protocol's NATS binding, step 3) +updated: 2026-09-26 decisions: - 02-DECISIONS/0106-the-bus-is-nats.md + - 02-DECISIONS/0115-the-bus-is-built-in-five-steps.md + - 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md + - 02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md + - 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md - 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md - 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md - 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md @@ -221,39 +226,94 @@ bridged. It is three things: Nothing is built of this before §10's bed passes; the MCP surface is a thin adapter over (2). -## 8. What a module sees +## 8. What a module sees, and what the wire does -Nothing new. `publish` on an envelope becomes a publish on `mesh.events..`; -`subscribe` with a pattern becomes a durable JetStream consumer on the matching subject filter; -`request`/`handle` become a NATS request and a queue-group subscription on -`mesh.tools..`. The envelope's shape ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)) -is unchanged; it is the message body. A module built today runs on the new runtime without a -rebuild — that is the test of ADR 0039, and it is in §10. +**The contract a module is written against does not change.** `publish` on an envelope becomes a +publish on `mesh.events..`; `subscribe` with a pattern becomes a durable JetStream +consumer on the matching subject filter; `request`/`handle` become a NATS request and a queue-group +subscription on `mesh.tools..`. The envelope's shape +([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)) is unchanged; it is the +message body. A module built today runs on the new runtime without a rebuild — that is the test of +ADR 0039, and it is in §10. -## 9. Moving from the bus the mesh has +**The wire underneath it changes completely, and that is a specification, not an implementation +detail.** Revision, second review ([ADR 0115](../../02-DECISIONS/0115-the-bus-is-built-in-five-steps.md)): +an earlier draft of this section said "nothing new" and stopped there, which read as though the +change were contained inside the runtime. It is not. +[ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md) settled that an SDK is an +implementation of a *specified wire*, checked by fixtures that must match byte for byte — because +two implementations that disagree about an envelope do not fail to compile, they ignore each other +while both keep running. That specification is +[design 19](19-the-module-protocol.md), and it is written in exchanges, a durable per-consumer queue +named `..events`, and a shared `serve.` queue. Every one of those is gone here. -Per ADR 0106: built beside, cut over once, after the core. +So design 19 is rewritten from exchanges and queues to the subjects and streams of §2 and §3, its +fixtures are recaptured on NATS, and each SDK re-claims the capabilities it passes. That is step 3 +of §9, and until it lands design 19 says in its own opening that its wire section describes the bus +being replaced. What is *not* rewritten: ADR 0074's model — protocol split per capability, an SDK +that implements the floor and events alone is legitimate, conformance executable per capability — +and ADR 0039's refusal. A new transport is when the pressure to grow the shared library is highest; +nothing is added to it here. -1. The `nats` module, the controller's and host's link on NATS, the runtime's client — built and - proven in the lab (§10) while the migration continues on AMQP. Modules converted meanwhile - target the sdk contract and are untouched by this. -2. The cutover is one rollout, previewed: the controller assigns `nats` to the hub (raised beside - the AMQP broker on its own ports), composes every node's and module's account into it, then - rolls out the controller, every host and every runtime built for NATS. Each node's host connects - to the new bus as it comes up and reports; the controller confirms every node heard before it - stops listening on AMQP. The predecessor's clients never notice: their broker is the - compatibility module and stays. -3. The AMQP-side mesh accounts are removed from the compatibility broker; it keeps only the - predecessor's users. The bus's port settings follow ADR 0100 like any port. -4. The compatibility broker retires when its retirement condition holds. +## 9. Moving from the bus the mesh has: five steps -What is not done: no dual-bus period for the mesh's own traffic, no bridge, no module rebuilt. +Per ADR 0106 the bus moves once. Per +[ADR 0115](../../02-DECISIONS/0115-the-bus-is-built-in-five-steps.md) the *build* is five steps, +each ending at something §10 proves, so that no part of this waits on the whole of it. Dividing the +build does not divide the bus: steps 1 to 4 leave every node on AMQP, and step 5 is still one +rollout. + +**Step 1 — genesis raises the broker.** The `nats` module of §5, and a mesh raised on it from +nothing. This is built and proven although the mesh it is for will never travel this path: genesis +is where the foundation is *defined*, the one place the mesh comes from nothing, and the definition +every other path is measured against. A genesis path that exists only on paper is one nobody finds +wrong until there is a second mesh. *Ends at: the genesis bed.* + +**Step 2 — adoption puts the broker in its seat.** A mesh already running does not get a foundation +module by being raised again; it adopts one in place +([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)). The +server is raised beside the AMQP broker on its own ports, carrying no mesh traffic yet, and the +`nats` module is adopted onto it. The seat it claims is **`mesh-broker`**, unchanged — the +foundation seats are named after the server's role rather than the product +([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)) for +exactly this case, and a seat named after the product would need renaming by every change the seat +exists to survive. *Ends at: the adoption bed.* + +**Step 3 — the protocol gets its NATS binding.** §8's other half: design 19 rewritten from +exchanges and queues to subjects and streams, its conformance fixtures recaptured on NATS, each SDK +re-claiming the capabilities it passes. ADR 0074's model is unamended and ADR 0039's refusal holds — +no helper layer arrives with the new transport. A language may still arrive in pieces: connection +and events first, tools and provisioning when something needs them. *Ends at: the conformance suite, +per capability, per implementation.* + +**Step 4 — the core speaks NATS.** The controller's link, the host's link, the tool runtime's +client — and with them the flows carried today by something other than the bus: a build source's +change reaching the builder, an installation, a module's own reports. Each is a conversion with a +named before and after. Observation — heartbeats, conditions, key-value state — belongs to +[research 017](../../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md), which already +reserves it for after the move; this step does not pre-empt its design. Modules converted meanwhile +target the sdk contract and are untouched by any of it. *Ends at: each converted flow proved against +the behaviour it replaced.* + +**Step 5 — the rollout.** Unchanged from ADR 0106 and previewed: the controller composes every +node's and module's account into the server standing since step 2, then rolls out the controller, +every host and every runtime built for NATS. Each node's host connects to the new bus as it comes up +and reports; the controller confirms every node heard before it stops listening on AMQP. The +predecessor's clients never notice — their broker is the compatibility module and stays. Then the +AMQP-side mesh accounts are removed from it, leaving only the predecessor's users, and it retires +when its retirement condition holds. The bus's port settings follow ADR 0100 like any port. +*Ends at: the cutover bed, then the rollout itself.* + +What is not done, at any step: no dual-bus period for the mesh's own traffic, no bridge, no module +rebuilt. ## 10. How it is checked -Two lab beds, both required green before any node's bus moves. +**A bed per step, and each is green before the step after it starts** — the division in §9 is only +real if the proofs divide with it. A step that cannot name what its bed proves is not a step, and is +divided further before it is started. -**The bus bed** — a mesh raised on NATS from genesis: +**Step 1 — the genesis bed**, a mesh raised on NATS from nothing: - a node enrols over TLS with a claimed token, and the enrolment user cannot read a declaration; - a push composes; the store is stopped; the push is held (nak with delay), the store returns, the push applies, nothing was lost or duplicated; @@ -273,12 +333,40 @@ Two lab beds, both required green before any node's bus moves. field a consumer's ack has already claimed; - the `nats` container is not recreated when only its composed configuration file changes, and a change to that file is live (a new user can connect, a revoked one cannot) within one - watcher-poll interval, without a restart; -- a module built before this design serves its tools unchanged on the new runtime. + watcher-poll interval, without a restart. -**The cutover bed** — a mesh on AMQP with a predecessor stand-in on the compatibility broker moves -its bus in one rollout; every node reports on NATS afterwards; the stand-in's client on AMQP is -still connected throughout. +**Step 2 — the adoption bed**, a mesh already running that has never had this server: +- the server is raised beside the AMQP broker on its own ports and the `nats` module is adopted + onto it in place, holding the data and the configuration it was raised with; +- the seat it claims is `mesh-broker`, and a second assignment of it anywhere in the mesh is + refused at resolution — *one per mesh*, as ADR 0079 requires; +- every node stays on AMQP throughout and nothing routes to the adopted server. This is the + check that makes steps 3 and 4 safe to run against a live mesh: adoption that quietly carried + traffic would be step 5 arriving early and unrehearsed. + +**Step 3 — conformance, per capability, per implementation:** +- the fixtures of design 19, recaptured on NATS, produced and consumed byte for byte by every SDK + that claims the capability — the test ADR 0074 set, and the only one that catches two + implementations quietly ignoring each other; +- an SDK implementing connection and events alone passes those two and claims nothing more, + rather than failing as a whole; +- a module built before this design serves its tools unchanged on the new runtime — the test of + ADR 0039; +- the shared library gained nothing but the binding: its surface is the protocol and the + primitives, and a helper that arrived with the transport is a review failure, not a detail. + +**Step 4 — each converted flow against the behaviour it replaced:** +- a build source's change reaches the builder over the bus, and the build that follows is the one + the change asked for; +- an installation completes over the bus, with the same outcome the path it replaces produced; +- a module's reports arrive, and a node that was unreachable catches up rather than losing them; +- nothing of research 017's observation work is present — no heartbeat semantics, no condition + store — because that design is not written yet and a flow built ahead of it would have to be + rebuilt. + +**Step 5 — the cutover bed:** a mesh on AMQP with a predecessor stand-in on the compatibility +broker moves its bus in one rollout; every node reports on NATS afterwards; the stand-in's client +on AMQP is still connected throughout. Unit tests hold the controller to composing accounts from `emits`/`consumes` and nothing else, to creating the four streams and asserting them idempotently, and to spending a token exactly once; @@ -287,7 +375,15 @@ to mapping the sdk contract onto subjects exactly as §8 says. ## 11. Open, for the review -**Closed by this revision** (first review, recorded in `MIGRATION-LOG.md`, 2026-09-24): the +**Closed by the second revision** (2026-09-26, [ADR 0115](../../02-DECISIONS/0115-the-bus-is-built-in-five-steps.md)): +the build was one undivided item (§9 is now five steps, each with its own bed in §10); there was no +adoption path for a mesh already running (§9 step 2); and §8 said a module sees "nothing new" +without distinguishing the sdk's contract, which does not change, from the specified wire, which +changes entirely and is design 19's (§8, §9 step 3). Two smaller corrections: the seat is +`mesh-broker` and not the product's name, and the shared library gains no conveniences with the new +transport. + +**Closed by the first revision** (recorded in `MIGRATION-LOG.md`, 2026-09-24): the `reload-on`/container mismatch (§5), the eaten reply subject on a CONTROL-stream message (§2, §6), the missing ack permission (§4), and the un-scoped reply inbox under one account (§4). Each is named where it was wrong, not silently fixed, so a reader comparing against the first version can