332 lines
21 KiB
Markdown
332 lines
21 KiB
Markdown
---
|
|
layer: to-be
|
|
status: in-progress
|
|
code:
|
|
- mesh-catalog modules/nats
|
|
- mesh-controller internal/catalogue
|
|
- mesh-lab scenarios
|
|
updated: 2026-09-26
|
|
decisions:
|
|
- 02-DECISIONS/0116-the-bus-is-built-in-five-steps.md
|
|
- 02-DECISIONS/0106-the-bus-is-nats.md
|
|
- 02-DECISIONS/0119-amqp-is-a-provision-not-the-bus.md
|
|
- 02-DECISIONS/0118-a-module-declares-its-own-seats.md
|
|
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
|
|
- 02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md
|
|
- 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
|
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
|
|
---
|
|
|
|
# 28. Building the bus
|
|
|
|
**The work of [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)'s five steps,
|
|
in the order its dependencies allow, with what each ends at.**
|
|
[Design 25](25-the-bus-on-nats.md) is the architecture and stays the authority on *what is built*;
|
|
this document holds only the order, the sizes and the proofs, and it is wrong the moment it
|
|
disagrees with design 25 rather than the other way round.
|
|
|
|
Each step ends at something runnable. A step that cannot name what its bed proves is not a step,
|
|
and is divided further before it is started.
|
|
|
|
## How this is built, and when it is run
|
|
|
|
**Written as code with unit tests, committed per change, and taken to the lab once the pieces that
|
|
would change the outcome are in place.** The mistake this avoids is the one
|
|
[design 22](22-the-work-ahead.md) records: running a long bed against a mesh mid-transformation and
|
|
debugging paths the next step deletes. Where a fault can be reasoned out of the code path, it is —
|
|
reading, not running.
|
|
|
|
So the beds below are acceptance tests at the end of assembled work, not the tool for finding each
|
|
bug, and a step's bed is run when that step is finished rather than while it is being written.
|
|
|
|
## What the work is, measured
|
|
|
|
Counted 2026-09-26, non-test source only. The point of counting is that none of this is unknown
|
|
territory: every piece has a shape already standing beside it.
|
|
|
|
| Piece | Today | Size | Becomes |
|
|
|---|---|---|---|
|
|
| the controller's link | Go, one package | ~1 800 lines | the same package on NATS |
|
|
| the host's link | Go, one package, mirroring the contracts rather than importing them | ~1 000 lines | the same, on NATS |
|
|
| the tool runtime's client | TypeScript, one file | ~390 lines | the same, on NATS |
|
|
| the sdk's messaging surface | TypeScript: messaging, events, tools, contracts, primitives | ~360 lines across five | **unchanged**, see below |
|
|
| the broker module | the adopted AMQP broker: client, tools, provisioner, bootstrap, image, manifest | ~340 lines of module code | the `nats` module, same shape |
|
|
| the beds | 39 lab scenarios, including a broker bed, an adoption bed, a genesis bed and a store-window bed | — | four analogues and one new |
|
|
|
|
**Two measurements are worth stating on their own, because they change what the steps are.**
|
|
|
|
**The sdk speaks no AMQP, and never did.** The word appears in its source three times, in three
|
|
comments; its messaging module says in as many words that it "carries the contract, not a specific
|
|
AMQP client build." [ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md) put the
|
|
client in the runtime, and the payoff is collected here: **no module is rebuilt for this change, and
|
|
the sdk's own diff is three comments.** That is the whole reason a bus can be replaced under a live
|
|
mesh at all.
|
|
|
|
**The wire therefore has three implementations, not two, and no suite pins any of them.**
|
|
[ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md) spoke of "the existing
|
|
two implementations" — Go and TypeScript. Measured, the Go side is *two separate packages* that
|
|
mirror rather than share (the host imports nothing, by
|
|
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), so the count is the controller's link, the
|
|
host's link, and the runtime's client. And a search for conformance fixtures finds none anywhere in
|
|
the four repositories: design 22's Phase 1.2 — the suite — has not been built.
|
|
|
|
> **This corrected the record.** ADR 0116 said step 3's fixtures were *recaptured* on NATS. There
|
|
> is nothing to recapture, so step 3 **builds** the suite, and its first job is to pin the wire the
|
|
> mesh has before changing it — a suite written only against the new bus certifies whatever the new
|
|
> bus happens to do. A fact went stale while the decision stood, which is a **progressive insight**
|
|
> ([`02-DECISIONS/README.md`](../../02-DECISIONS/README.md)): it is marked and dated in ADR 0116
|
|
> itself rather than left to be discovered here.
|
|
|
|
## The order the work actually allows
|
|
|
|
**The five steps are chunks of capability; the build order is not simply 1 to 5, and pretending
|
|
otherwise would put two beds where they cannot run.** Three edges decide it:
|
|
|
|
- **A specification precedes the implementations it governs.** ADR 0074's whole argument is that
|
|
agreement is specified and checked, not hoped for. So the wire's NATS binding is written *before*
|
|
the three implementations are, even though it is step 3 — and its conformance half can only
|
|
*finish* once two implementations exist to disagree.
|
|
- **A mesh cannot be raised on a bus nothing speaks.** A bed that raises a mesh on NATS from
|
|
genesis — enrolling a node, holding a push while the store restarts, rolling out an upgrade —
|
|
needs the controller and the host to speak NATS already. That is the implementations, and they
|
|
arrive with step 3.
|
|
- **Adoption needs the module and nothing else.** Step 2 puts a correctly configured server into a
|
|
running mesh that continues to ignore it, which depends on no link at all.
|
|
|
|
So step 1's bed proves *the server, from genesis, configured* — not a mesh living on it. The full
|
|
genesis bed is step 4's, where it can first run.
|
|
|
|
> **This corrected the record too.** ADR 0116 first attributed "a mesh raised on NATS from genesis"
|
|
> to step 1. That bed cannot run until the links exist, and a step whose proof cannot run is the
|
|
> exact failure the record was written to prevent — so step 1 now ends at the server standing,
|
|
> correctly configured and carrying nothing, and the full bed is named under step 4. The five
|
|
> steps, their names, their order and the single rollout are unchanged; only where two beds run has
|
|
> moved. Marked and dated in ADR 0116 as a progressive insight, with what the record said before.
|
|
|
|
```
|
|
step 1 module, genesis places it ──┐
|
|
step 2 adoption into a running mesh ──┤ neither needs a link
|
|
│
|
|
step 3 the wire specified ──► three implementations ──► the suite
|
|
│
|
|
step 4 the flows, and the full genesis bed
|
|
│
|
|
step 5 the rollout
|
|
```
|
|
|
|
## Step 1 — the module, and genesis raises it
|
|
|
|
> **Revised 2026-09-26** ([ADR 0118](../../02-DECISIONS/0118-a-module-declares-its-own-seats.md),
|
|
> [design 29](29-what-a-module-declares.md)). Tasks 1.3 and 1.4 said the controller composes every
|
|
> account and creates *the four streams* at genesis, from a fixed set. That is only the mesh's own
|
|
> half. A module declares seats with their protocols, so streams are created **at registration**
|
|
> and durable consumers **at assignment** — neither of which has happened at genesis. The fixed
|
|
> foundation set stays here; the derived machinery moves to step 3, where the declaration model it
|
|
> reads from is specified. Tasks 1.1 and 1.2, already done, are untouched by this: the module and
|
|
> its reload mechanism do not care what the configuration says.
|
|
|
|
**Why here.** Everything else needs a server to talk to, and genesis is where the foundation is
|
|
defined. The mesh this is for will never travel this path — it is already running, and takes step 2
|
|
— but genesis is the definition every other path is measured against, and one that exists only on
|
|
paper is wrong until there is a second mesh to find out.
|
|
|
|
- [x] 1.1 the `nats` module: manifest, image, one container, its client, TLS and monitoring ports,
|
|
JetStream on a named volume — the shape of design 25 §5, and the same shape the broker module
|
|
beside it already has
|
|
- [x] 1.2 the composed configuration as a **directory** resource, and the entrypoint that watches
|
|
the one file and signals the server itself — design 25 §5's correction, kept inside the module
|
|
because a container has no reload and a recreate would drop every connection the mesh has
|
|
- [x] 1.3 the controller composes that file: accounts, permissions, TLS, JetStream — a user's
|
|
permissions derived from its declaration and nothing else, over the three namespaces of
|
|
[design 29](29-what-a-module-declares.md) §2, plus its own ack subject and its own inbox
|
|
prefix (design 25 §4)
|
|
- [x] 1.4 the mesh's own streams, created at genesis and asserted idempotently on start, by the
|
|
controller as their only writer — **the mesh's own, not all of them**: a seat's streams are
|
|
created when the module declaring it is registered, and a module's durable consumers when it
|
|
is assigned, so this task is the fixed foundation set and 3.x carries the derived rest
|
|
- [x] 1.5 genesis raises it as foundation, claiming the seat **`mesh-broker`** — the seat is the
|
|
server's role, not the product. **Already true of the controller and needed no change**: it
|
|
resolves the broker by seat ("that is where the broker is, whatever else the topology says")
|
|
and names no broker module anywhere in its source. What remains is naming `nats` instead of
|
|
the AMQP broker where a genesis module set is declared, which is scenario and installer
|
|
configuration — carried with 1.6 rather than before it.
|
|
- [ ] 1.6 the genesis-broker bed — **deferred**: beds are run once, at the end, rather than per
|
|
step (novox/hq design 22's rule, and the operator's instruction). Every claim step 1 makes
|
|
is covered by a unit test or was demonstrated against the real server; what the bed adds is
|
|
the claims that need a mesh.
|
|
|
|
> **Not done here, deliberately.** The controller builds a module's broker credential as an
|
|
> `amqps://` URL and defaults a portless genesis address to 5671. Those are correct until the
|
|
> rollout and must not move: steps 1 to 4 leave every node on AMQP
|
|
> ([ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)), so changing the
|
|
> credential's shape now would break the running bus to serve a bus nothing speaks yet. They
|
|
> change with the links, in step 3.
|
|
|
|
**Done when.** A mesh raised from nothing has the server standing with the streams asserted and
|
|
every account and permission composed from the manifests; a user cannot publish outside its
|
|
`emits`, subscribe outside its `consumes`, ack another user's delivery or subscribe another's inbox
|
|
prefix; the monitoring port is refused from anything but the private network; a change to the
|
|
composed file is live within one watcher interval without a restart, and the container is not
|
|
recreated by it.
|
|
|
|
**The permission checks belong here rather than later** because the server enforces them itself — a
|
|
plain client proves them, no link required — and they are the whole of what ADR 0043 asks for. No
|
|
mesh traffic is on the bus yet; that is step 4's bed, not this one's.
|
|
|
|
## Step 2 — adoption puts it in the seat
|
|
|
|
**Why here.** It needs only step 1's module, it is the path the mesh that exists will actually take,
|
|
and it is what makes steps 3 and 4 safe to develop against a live mesh. A running mesh does not get
|
|
a foundation module by being raised again; it adopts one in place
|
|
([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)).
|
|
|
|
- [ ] 2.1 the server raised beside the existing broker on its own ports, carrying nothing —
|
|
installer-side, from the upstream image
|
|
- [ ] 2.2 the `nats` module assigned, which **recreates the container once, deliberately** (see
|
|
below), keeping its JetStream directory
|
|
- [x] 2.3 the seat claim, and the resolver's refusal of a second holder mesh-wide — **already
|
|
true and now proved**: the refusal is generic to any mesh-scoped seat, and three tests pin
|
|
what matters for this one — a second bus anywhere is refused naming the seat, a *different*
|
|
bus implementation is refused for the same reason (which is what lets the bus be replaced
|
|
at all), and the AMQP broker no longer contends for it, so both run on one mesh
|
|
- [ ] 2.4 the adoption bed — deferred with the other beds
|
|
|
|
> **Adoption here is not a no-op, and pretending it would be is the trap.** The host keeps an
|
|
> existing container only when its spec matches the declaration exactly
|
|
> ([`apply.go`](https://git.novox.be/novox/mesh-host): *existed && before.Spec == want && running*
|
|
> → unchanged; anything else is `rm -f` and recreate). Genesis raises the server from the
|
|
> **upstream** image, because nothing has been built yet; the module declares the **mesh-built**
|
|
> artifact, which carries the entrypoint that reloads configuration in place. Those two specs
|
|
> differ, so assigning the module recreates the container.
|
|
>
|
|
> That is correct, and it is [ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md)'s pivot
|
|
> exactly: raise a temporary thing, then reinstall it as an ordinary module. It is safe **only
|
|
> because it happens while the bus carries nothing** — which is what 2.1 means by "carrying
|
|
> nothing", and why step 2 comes before anything speaks NATS rather than after. One recreate, at
|
|
> the one moment it costs nothing.
|
|
>
|
|
> **After that, never again.** The configuration is a directory mount rather than a file, so
|
|
> rewriting accounts does not change the container's spec and the entrypoint reloads the server in
|
|
> place. That is the whole point of task 1.2, and this is the moment it pays: every later account,
|
|
> permission or person's access change touches a running bus with connections on it.
|
|
|
|
**Done when.** A mesh already running has the server adopted, holding `mesh-broker`; a second
|
|
assignment anywhere is refused at resolution — *one per mesh*; and every node is still on the old
|
|
bus with nothing routed to the new one. **That last check is the point of the step**: adoption that
|
|
quietly carried traffic would be step 5 arriving early and unrehearsed.
|
|
|
|
## Step 3 — the protocol on NATS
|
|
|
|
**Why here.** The implementations cannot be written against an unwritten wire, and this is the step
|
|
that decides what "agreeing" means for everything after it. It is the largest step and the one that
|
|
pays for itself furthest away.
|
|
|
|
- [ ] 3.1 **the suite first, on the bus the mesh has** — fixtures for the envelope and its required
|
|
headers, the contributions file, a served tool call and a grant, capturing what the three
|
|
implementations do *today*. Written first because a suite born on the new bus certifies
|
|
whatever the new bus happens to do
|
|
- [ ] 3.2 design 19 rewritten from exchanges, queues and routing keys to the subjects and streams of
|
|
design 25 §2–§3, per capability, with ADR 0074's model untouched: floor plus capabilities, an
|
|
implementation legitimate when it claims less, identity from the sealed credential, dedup on
|
|
`x-event-id`
|
|
- [ ] 3.3 the fixtures restated on NATS, and the capability each implementation claims
|
|
- [ ] 3.4 the controller's link on NATS
|
|
- [ ] 3.5 the host's link on NATS — mirroring, still importing nothing
|
|
- [ ] 3.6 the tool runtime's client on NATS, behind the unchanged sdk contract
|
|
- [ ] 3.7 the sdk's three stale comments, and nothing else in it
|
|
- [ ] 3.8 **the declaration model** of [design 29](29-what-a-module-declares.md): local names
|
|
derived to subjects, the three namespaces, permissions computed from a declaration, and a
|
|
manifest that contains no subject
|
|
- [x] 3.9 **seats declared by modules** — the manifest now carries `seats` (name, scope,
|
|
accepts/emits/serves, retention) and `uses`, and registration refuses a `mesh-*` name, a
|
|
duplicate declarer, an undeclared `uses` or claim, a seat with no protocol, a scope
|
|
mismatch, and a holder that does not answer what its seat promises. **Still to do**:
|
|
creating a seat's streams at registration and its holder's work-queue consumer at
|
|
assignment, which need the JetStream client wired in.
|
|
|
|
The refusal for an unknown claim *moved* rather than disappeared — the parser cannot judge
|
|
it from one manifest any more, because another module may legitimately declare that seat,
|
|
so it is registration's. The test that encoded the old rule was rewritten rather than
|
|
deleted, and a second one pins the case the parser could not distinguish.
|
|
|
|
**Done**: a seat's work queue is derived and created, and a holder's worker with it. The
|
|
JetStream client behind them is wired and verified against a running server, which also
|
|
completes 1.4's missing half — the pure `Asserter` had no implementation until now.
|
|
- [ ] 3.10 **the ten seat renames**, carried as a migration with a mapping rather than an edit,
|
|
and the beds that name seats moved with them
|
|
|
|
**Done when.** The fixtures are produced and consumed byte for byte by every implementation that
|
|
claims the capability, and a module built before any of this serves its tools unchanged on the new
|
|
runtime. **The step is not done when the code runs** — two implementations that disagree about an
|
|
envelope do not fail to compile, they ignore each other while both keep running, which is the
|
|
failure ADR 0074 exists to catch.
|
|
|
|
**And the shared library gained nothing but the binding.** A new transport is when the pressure to
|
|
add conveniences is highest, and ADR 0039's rule does not bend for it: a helper that arrives with
|
|
the bus is a review failure, not a detail. Code shared among a module's own features stays in that
|
|
module.
|
|
|
|
## Step 4 — the core speaks it
|
|
|
|
**Why here.** The links exist from step 3, so the flows that are not on the bus at all can move onto
|
|
it, and the beds that need a mesh living on NATS can finally run.
|
|
|
|
- [ ] 4.1 **the full genesis bed** — a mesh raised on NATS from nothing and living on it: a node
|
|
enrols over TLS with a claimed token and the enrolment user cannot read a declaration; a push
|
|
is held while the store restarts and applies after, nothing lost or duplicated; a node that
|
|
was away gets exactly the newest declaration and refuses a replayed older one by sequence; an
|
|
upgrade rolls out to two nodes; an event dead-letters after `max-deliver`; and an enrolment
|
|
held by a `nak`-with-delay cycle still reaches the enrolling node, proving the reply travels
|
|
in the payload and not the transport field the consumer's ack has claimed. The server-enforced
|
|
permissions were proved at step 1 and are not re-proved here
|
|
- [ ] 4.2 a build source's change reaches the builder over the bus, and the build that follows is
|
|
the one the change asked for
|
|
- [ ] 4.3 an installation completes over the bus, with the same outcome as the path it replaces
|
|
- [ ] 4.4 a person's client: the account, the client that speaks the bus, and the tool surface over
|
|
it (design 25 §7) — a module's tool invoked from another node and from a person, refused from
|
|
an account that may not
|
|
- [ ] 4.5 reports and catch-up: a node that was unreachable catches up rather than losing them
|
|
|
|
**Done when.** Each converted flow is proved against the behaviour it replaced, and the full genesis
|
|
bed is green. **Observation is not in this step** — heartbeats, conditions and key-value state are
|
|
[research 017](../../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md)'s, that effort already
|
|
reserves them for after the move, and a flow built ahead of its design would be rebuilt.
|
|
|
|
## Step 5 — the rollout
|
|
|
|
**Why here.** It is the only step that moves a node's bus, and it moves every node's at once.
|
|
|
|
- [ ] 5.1 the cutover bed: a mesh on AMQP with a predecessor stand-in on the compatibility broker
|
|
moves its bus in one rollout, every node reporting on NATS afterwards, the stand-in's own
|
|
client still connected throughout
|
|
- [ ] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
|
|
together; every node confirmed heard before AMQP stops
|
|
- [ ] 5.3 the mesh's accounts removed from the compatibility broker, leaving the predecessor's users
|
|
- [ ] 5.4 the compatibility broker retires when its condition holds — no client connected for the
|
|
period the operator sets
|
|
|
|
**Done when.** Every node reports on NATS and the predecessor's clients never noticed.
|
|
|
|
## The through-line
|
|
|
|
The order is dependency, not preference. **Steps 1 to 4 leave every node on AMQP**, so the cost of
|
|
being wrong is bounded until the last step: a step may be abandoned, or reordered after step 2,
|
|
without a rollback. The server stands before anything speaks to it; the wire is specified before it
|
|
is implemented three times; the flows move once there is something to move them onto; and the bus
|
|
itself moves once, at the end, on one day.
|
|
|
|
## What is deliberately not here
|
|
|
|
- **Observation** — research 017's, after the move, by its own design.
|
|
- **Leaf nodes** — design 25 §11 keeps this out of scope and says so; a leaf per machine is a later
|
|
question, noted so it is not forgotten.
|
|
- **The predecessor's world.** It is AMQP, it cannot move, and it does not need to: its broker is
|
|
the compatibility module until its last client is gone.
|
|
|
|
## How this list is kept true
|
|
|
|
A task is ticked when its change is committed, not when it is written. A step is done when its bed
|
|
is green, not when its tasks are ticked. If a step's tasks are all ticked and its bed has not run,
|
|
the step is **in progress** and this document says so — that gap is the thing the whole shape is
|
|
built to make visible.
|