PR #133 landed a different 0115 while this branch was open. The bus record is now 0116, with every citation in designs 19, 25, 28 and the index following it. Note: cycle.py and records.py both fail on main as merged, on that record — nothing cites it, and it rests on 0112, which is still proposed. Both pre-date this branch and are left for their own change.
176 lines
11 KiB
Markdown
176 lines
11 KiB
Markdown
---
|
|
topic: the mesh
|
|
status: accepted
|
|
date: 2026-09-26
|
|
deciders: jochen
|
|
reconstructed: false
|
|
extends: 02-DECISIONS/0106-the-bus-is-nats.md
|
|
---
|
|
|
|
# 116. The bus is built in five steps, and the protocol moves with it
|
|
|
|
## Context
|
|
|
|
[ADR 0106](0106-the-bus-is-nats.md) decided the bus is NATS and described the change as one thing:
|
|
built beside the migration, cut over in one rollout after its core. The architecture was written as
|
|
[design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) and revised once after review. What neither
|
|
says is how the work is divided, and three gaps follow from that.
|
|
|
|
**The whole of the build is one point.** Design 25 §9 numbers four items. The first — "the `nats`
|
|
module, the controller's and host's link on NATS, the runtime's client — built and proven in the
|
|
lab" — is every line of code the change requires; the other three are the rollout. A step of that
|
|
size ends at nothing provable until it ends at everything, which is the failure
|
|
[design 22](../03-DESIGN/01-to-be/22-the-work-ahead.md) opens by naming: *a phase that ends at a
|
|
claim is a phase that went missing without anything complaining.*
|
|
|
|
**There is no adoption path.** Design 25 §5 says the broker "is raised at genesis like the store,
|
|
adopted as a module in the same phase" — which describes a mesh being raised from nothing. The mesh
|
|
this is for is already running, and a running node gets a foundation module by adoption in place
|
|
([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)), not by genesis. §9 goes
|
|
straight from "built beside" to "one rollout" and never crosses that gap.
|
|
|
|
**The wire changes and the protocol specification does not know it.** Design 25 §8 says a module
|
|
sees nothing new. That is true of the SDK's contract — `request`, `handle`, `publish`, `subscribe`
|
|
— and false of the wire underneath it.
|
|
[ADR 0074](0074-the-wire-is-specified-not-the-types.md) settled that what an SDK implements is a
|
|
*specified wire*, checked by fixtures that must match byte for byte, precisely because two
|
|
implementations that disagree about an envelope do not fail to compile. That specification is
|
|
[design 19](../03-DESIGN/01-to-be/19-the-module-protocol.md), and it is written entirely in AMQP:
|
|
exchanges, a durable per-consumer queue named `<node>.<module>.events`, a shared `serve.<key>`
|
|
queue — sixteen occurrences of an AMQP term across the document. Design 25 does not cite design 19
|
|
anywhere, and design 19 does not cite ADR 0106. So the record that says *what two implementations
|
|
may not disagree about* still describes the bus being replaced.
|
|
|
|
## Considered Options
|
|
|
|
1. **Keep ADR 0106's shape — build it all, cut over once.** Rejected: not for its rollout, which
|
|
is right, but because it leaves the build a single step of unknown length with no intermediate
|
|
anyone can run. The three gaps above were found by dividing it; they were invisible while it
|
|
was one item.
|
|
2. **Cut over incrementally by traffic kind** — events to NATS first, then tools, then control,
|
|
the mesh on two buses meanwhile. Rejected: ADR 0106 already rejected two buses, and this is
|
|
that with extra steps. The store-window guarantee
|
|
([ADR 0083](0083-one-push-leaves-the-mesh-consistent.md)) is exactly the one that cannot cross
|
|
a seam, and control is exactly the traffic that carries it.
|
|
3. **Five steps, each ending at something provable, the cutover still one rollout.** The build is
|
|
divided; the bus still moves once. Adopted.
|
|
|
|
## Decision
|
|
|
|
**The bus is built in five steps. Each ends at something a lab bed proves, and no step's proof
|
|
waits for the one after it. The cutover remains a single rollout** — dividing the build does not
|
|
divide the bus.
|
|
|
|
**Step 1 — genesis raises the broker.** The `nats` module, and genesis placing it in the
|
|
foundation. This is built and proven even though the mesh it is for will never travel this path,
|
|
because
|
|
genesis is where the foundation is *defined*: the only place the mesh comes from nothing, and the
|
|
definition every other path is measured against. A genesis path that exists only on paper is one
|
|
nobody discovers is wrong until there is a second mesh.
|
|
|
|
**Step 2 — adoption puts the broker in the seat.** A mesh already running receives the broker by
|
|
adoption in place, and the seat it claims is **`mesh-broker`** — unchanged.
|
|
[ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md) named the foundation seats
|
|
after the server's *role* rather than the product for exactly this case, and ADR 0106 restated it:
|
|
the broker module changes, the seat does not. A seat named after the product would have to be
|
|
renamed by every change the seat exists to survive.
|
|
|
|
**Step 3 — the protocol gets a NATS binding.** ADR 0074 stands unamended: the mesh defines a module
|
|
protocol, an SDK is an implementation of it in one language and nothing more, the protocol is split
|
|
per capability, and conformance is executable fixtures per capability rather than prose. What
|
|
changes is what the specification specifies. Design 19's wire section is rewritten from exchanges
|
|
and queues to subjects and streams; the conformance suite is built — first against the bus the
|
|
mesh has, then restated on NATS; every SDK claims the capabilities it passes, and a language may
|
|
arrive with connection and events alone.
|
|
|
|
**The SDK gains no conveniences in the process.** [ADR 0039](0039-what-the-sdk-holds-and-refuses.md)
|
|
refuses frequent-and-cascading code in the shared library, and ADR 0074 restates it — an SDK is
|
|
"not a convenience layer, not a place for helpers to accumulate." A new transport is the moment
|
|
that pressure is highest and the reason to hold hardest: the predecessor's shared library is the
|
|
cautionary tale ADR 0039 opens with, and it did not become that in one decision. Code shared among
|
|
a module's own features stays in that module.
|
|
|
|
**Step 4 — the core speaks NATS.** The controller's link, the host's link and the tool runtime's
|
|
client, and with them the flows that are today carried by something other than the bus: a build
|
|
source's change reaching the builder, an installation, a module's own reports. Each is a
|
|
conversion with a named before and after, not a rewrite. Observation — heartbeats, conditions,
|
|
key-value state — is [research 017](../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md)'s
|
|
and stays there; that effort already reserves it for after the move, and this step does not
|
|
pre-empt its design.
|
|
|
|
**Step 5 — deployment.** The rollout ADR 0106 decided, unchanged: the controller, every host and
|
|
every runtime move together, each node confirmed to have heard before AMQP stops. What this step
|
|
adds is that steps 1 to 4 *ship ahead of it* without moving any node's bus — the module exists in
|
|
the catalogue, the protocol is specified, the code is written and beds pass, and the running mesh
|
|
is still on AMQP throughout. The bus moves on one day, at the end, once.
|
|
|
|
## Consequences
|
|
|
|
- **Design 25 §9 is replaced by these five steps** and §10's beds are attributed to the step each
|
|
proves, so no proof waits for the last step.
|
|
- **Design 19 is stale in its wire section from today** and says so in its own frontmatter and
|
|
opening until step 3 rewrites it. A specification that describes the bus being replaced is worse
|
|
than an absent one, because it reads as current.
|
|
- **`nats-broker` is not a seat.** The module is `nats`; the seat is `mesh-broker`.
|
|
- **Steps 1 to 4 leave every node on AMQP.** A step can be abandoned, or reordered after step 2,
|
|
without a rollback — the cost of being wrong is bounded until step 5.
|
|
- **What got harder:** five steps mean five proofs rather than one, and step 3 rewrites a design
|
|
other designs cite, so their references are checked when it lands. Dividing the work does not
|
|
reduce it.
|
|
|
|
## How it is checked
|
|
|
|
- **Per step, a bed, and the bed named in design 25 §10 against the step it belongs to.** Step 1:
|
|
a mesh raised from nothing has the server standing, its streams asserted and every account
|
|
composed from the manifests — no mesh traffic on it yet. Step 2: the broker adopted into a mesh
|
|
already running, a second holder of `mesh-broker` refused at resolution, and nothing routed to it.
|
|
Step 3: the conformance suite passing per capability, on NATS, for every implementation that
|
|
claims it — and a module built before the binding serving its tools unchanged. Step 4: each
|
|
converted flow proved against the behaviour it replaced, **and the full genesis bed** — a mesh
|
|
enrolling, holding a push, and rolling out an upgrade on NATS. Step 5: the cutover bed, then the
|
|
rollout with every node reporting.
|
|
- **Step 3 is not done when the code runs.** It is done when the fixtures match byte for byte
|
|
across implementations, which is ADR 0074's own test and the only one that catches two SDKs
|
|
quietly ignoring each other.
|
|
- **A step that cannot name what its bed proves is not a step**, and is divided further before it
|
|
is started.
|
|
|
|
## Progressive insights
|
|
|
|
Corrections of fact made in this record after it was accepted, under the rule in
|
|
[`README.md`](README.md). The decision — five steps, each proved, one rollout — is untouched by
|
|
both; each corrects something this record asserted about the *state of the code*, which nobody had
|
|
measured when it was written.
|
|
|
|
> **Progressive insight — 2026-09-26.** *There is no conformance suite to recapture.* This record
|
|
> said step 3's "fixtures are recaptured on NATS", and design 19's wire was described as specified
|
|
> and conformed. Measuring the four repositories found no conformance fixtures in any of them:
|
|
> [design 22](../03-DESIGN/01-to-be/22-the-work-ahead.md)'s Phase 1.2, which would have built the
|
|
> suite, is still open. Step 3 therefore **builds** it, and builds it first against the bus the
|
|
> mesh has — a suite born on the new bus certifies whatever the new bus happens to do. The step's
|
|
> place in the order, and ADR 0074's model it implements, are unchanged.
|
|
|
|
> **Progressive insight — 2026-09-26.** *The full genesis bed belongs to step 4, not step 1.* This
|
|
> record attributed "a mesh raised on NATS from genesis" to step 1, and described that step as "the
|
|
> `nats` module and a mesh raised on it from nothing". A bed that enrols a node, holds a push and
|
|
> rolls out an upgrade needs the controller and the host to speak NATS — which is step 3's
|
|
> implementations and step 4's flows. A step whose proof cannot run is exactly the failure this
|
|
> record was written to prevent, so step 1 now ends at the server standing from genesis, correctly
|
|
> configured and carrying nothing, and the full bed is named under step 4. The five steps, their
|
|
> names, their order and the single rollout are unchanged; only where two beds run has moved.
|
|
>
|
|
> Recorded here rather than superseded because the decision this record makes — that the work is
|
|
> divided and each division is proved — is what *produced* the correction: the dependency was
|
|
> invisible while the build was one item.
|
|
|
|
## References
|
|
|
|
- [ADR 0106](0106-the-bus-is-nats.md) — the decision this divides.
|
|
- [ADR 0074](0074-the-wire-is-specified-not-the-types.md) — the protocol and its conformance suite;
|
|
step 3 is its NATS binding, not a replacement.
|
|
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — why step 3 adds no helpers.
|
|
- [ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md) — why the seat is
|
|
`mesh-broker`.
|
|
- [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) — the adoption step 2 uses.
|
|
- [design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md), [design 19](../03-DESIGN/01-to-be/19-the-module-protocol.md) — the two documents this changes.
|