diff --git a/03-DESIGN/01-to-be/25-the-bus-on-nats.md b/03-DESIGN/01-to-be/25-the-bus-on-nats.md new file mode 100644 index 0000000..8ad276e --- /dev/null +++ b/03-DESIGN/01-to-be/25-the-bus-on-nats.md @@ -0,0 +1,223 @@ +--- +layer: to-be +status: proposed +code: + - mesh-controller internal/link (to be replaced) + - mesh-host internal/link (to be replaced) + - mesh-tools src/broker-amqp.ts (to be replaced) + - mesh-catalog modules/nats (to be written) +updated: 2026-09-23 +decisions: + - 02-DECISIONS/0106-the-bus-is-nats.md + - 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md + - 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md + - 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md +--- + +# 25. The bus on NATS + +**Status: proposed — a design to be reviewed before any code.** This is the architecture +[ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md) asks for. It says what rides the bus, +under which subject, with which guarantee, under whose account; how a node joins; how a person +reaches a tool; how the mesh moves from the bus it has to this one; and how each claim is checked. +Prose and diagrams only; no configuration is pasted. + +## 1. What the bus is for + +The bus carries five kinds of traffic today, and this design keeps the five, renaming nothing a +module can see: + +| Traffic | Today | Guarantee it needs | +|---|---|---| +| **control** — a node's report, its heartbeat, a build's outcome, an enrolment | queues `control`, `.upgrades`, `.catchup` | nothing lost while the store restarts; retried; in order per node | +| **declarations** — the controller tells a node what to be | queue `node.` | the node gets the newest; a stale one is never applied | +| **builds** — the controller asks the build machine to build | queue `builds` | at least once, one builder at a time | +| **events** — a module says something happened | topic exchange `mesh.events`, keys `.` | delivered to every consumer that declared it; dead-lettered when it cannot be | +| **tools** — one module or person asks another's tool a question | exchange `mesh.rpc`, per-tool service queues `serve..` | one answer, from one server, or a timeout | + +The sdk's contract — `request`, `handle`, `publish`, `subscribe`, `close` — is the whole surface a +module sees, and it does not change ([ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)). + +## 2. Subjects + +NATS addresses everything by subject. The mesh's subject space is one tree, and every account's +permissions are expressed as which branches of it that account may publish to and subscribe from. + +``` +mesh.control..report a node's report (JetStream: CONTROL) +mesh.control..alive heartbeat (core, no persistence) +mesh.control.enrol an enrolment request (JetStream: CONTROL) +mesh.control.built a build's outcome (JetStream: CONTROL) +mesh.node..declare a declaration for a node (JetStream: NODES, last-per-subject) +mesh.build.request work for the build machine (JetStream: BUILDS, work queue) +mesh.events.. an event (JetStream: EVENTS) +mesh.tools.. a tool invocation (core request/reply) +mesh.ask.. the controller's command api (core request/reply) +``` + +Two things this buys over the exchanges: **request/reply is native** — a tool call is one +`request` on `mesh.tools..` answered by whichever runtime serves it (a queue group per +tool, so several nodes may serve one tool); and **a declaration is last-per-subject** — the NODES +stream keeps only the newest message on `mesh.node..declare`, so a node that was away gets +exactly the current declaration and nothing older. That is the wire-level answer to +[issue 107](../../04-ISSUES/107-a-declaration-carries-no-order/00-report.md): the stream's sequence +*is* the order, and a node that sees sequence n refuses n−1 by construction. + +## 3. Streams, and the guarantees they carry + +Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStream stream: + +| Stream | Subjects | Retention | Why | +|---|---|---|---| +| CONTROL | `mesh.control.>` except `alive` | work queue, one consumer (the controller), explicit ack | the store-window guarantee ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)): the controller `nak`s with a delay while its store is away and the message is redelivered; nothing is dropped | +| NODES | `mesh.node.>` | last per subject | one declaration per node, always the newest | +| BUILDS | `mesh.build.>` | work queue, explicit ack | at least once; a builder that dies mid-build has its message redelivered | +| EVENTS | `mesh.events.>` | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream) | + +Tool calls and heartbeats stay on core NATS: a lost heartbeat is the next heartbeat; a lost tool +call is a timeout the caller already handles. + +Streams and consumers are objects the controller creates at genesis and asserts on start; a module +declares nothing about them. The controller is the only writer of stream definitions. + +## 4. Accounts + +[ADR 0043](../../02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) says a +module's account may publish only what it `emits` and consume only what it `consumes`. NATS +expresses this exactly, per subject, and better than a vhost could: + +- **One NATS account for the mesh.** Accounts in NATS isolate subject spaces entirely; the mesh + is one space, so it is one account. The predecessor's compatibility broker is not on this bus at + all. +- **One user per module per node**, as today, with publish permissions + `mesh.events..` for each emit, `mesh.tools..>` to serve its tools, and + its reply inbox; subscribe permissions for each consumed event's subject and its tool subjects. + Nothing else. A module that tries to publish outside its emits is refused by the server, not by + convention. +- **The controller's user** owns `mesh.control.>`, `mesh.node.>`, `mesh.build.>` and the streams. + **A host's user** may publish its own `mesh.control..>` and subscribe its own + `mesh.node..declare` — and nothing of any other node's. +- **A person's user** (§7) is a module-shaped user with permissions on the tool subjects it may + invoke, issued and revoked by the controller like any account. + +**Accounts are configuration, not API calls.** The controller composes the server's user list and +permissions into a file the host declares; the server reloads on change (`reload-on`, as the mesh +already does for the container runtime's trust). No management API, no credential travelling +through a management call, and the [issue 102](../../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md) +discipline from the first day: an address or a permission is read where it is used, never stored +with a port. Passwords are minted and sealed exactly as today; the file holds bcrypt hashes. + +Alternative considered and not taken: the operator/JWT model (`nsc`), where accounts are signed +tokens resolved by the server. It is the right model for a multi-tenant NATS; the mesh is one +tenant, already has a sealing key and a controller that writes files, and would gain a second +signing hierarchy for nothing. + +## 5. The broker as a module + +`nats` is a catalogue module claiming the seat `mesh-broker` +([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md): the seat +is the server, and the server changes). It declares one container (a single binary; JetStream on a +named volume), its listening ports — client, TLS, and the monitoring endpoint on loopback — a +configuration file the controller composes (accounts, permissions, TLS, JetStream), and a +`reload-on` for that file. Its guard is the same rule as the AMQP broker's: the monitoring port is +refused from anything but the private network. It is raised at genesis like the store, adopted as a +module in the same phase. The predecessor's AMQP broker remains a module of its own, +`lavinmq-compat`, with a single purpose and a retirement condition: no client connected for a +period the operator sets. + +## 6. Joining: the enrolment handshake + +Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS +with the **enrolment user** — a user that may publish `mesh.control.enrol` and subscribe one reply +inbox and nothing else — publishes its request (the claim of the token, its keys, its proof, and +the found tunnel from [ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)), +and waits on the inbox. The controller spends the token, records the node, composes the node's +own user into the server's configuration, and answers with the credentials sealed to the node's +sealing key. The node reconnects as itself. The enrolment user's permissions are what make a +leaked token useless for anything but enrolling: it cannot read a declaration or hear an event. + +## 7. A person's client + +The operator asked for the mesh's tools from a workstation, and for it designed here rather than +bridged. It is three things: + +1. **A person's account**: `operator issue ` on the controller creates a user whose + permissions are the tool subjects it may invoke — `mesh.tools.>` for an administrator, a list + for anyone else — and nothing on control, nodes or builds. It is issued, sealed to the person's + own key, and revoked, like a module's. +2. **A client that speaks the bus**: a small program on the workstation that connects as that user + over TLS, lists tools by asking the catalogue (`mesh.tools.mesh-catalog.catalog_tools`), and + turns each tool into a call — as an MCP server for an agent, and as a command line for a person. + It uses the sdk's `Broker` contract on the NATS runtime, so it is the same code path a module's + tools use, not a second protocol. +3. **Reachability**: the workstation reaches the bus over the private network once it is a node, + or over the predecessor's tunnel before that, on the bus's port; the guard and the openings + treat the bus as they do today. + +Nothing is built of this before §10's bed passes; the MCP surface is a thin adapter over (2). + +## 8. What a module sees + +Nothing new. `publish` on an envelope becomes a publish on `mesh.events..`; +`subscribe` with a pattern becomes a durable JetStream consumer on the matching subject filter; +`request`/`handle` become a NATS request and a queue-group subscription on +`mesh.tools..`. The envelope's shape ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)) +is unchanged; it is the message body. A module built today runs on the new runtime without a +rebuild — that is the test of ADR 0039, and it is in §10. + +## 9. Moving from the bus the mesh has + +Per ADR 0106: built beside, cut over once, after the core. + +1. The `nats` module, the controller's and host's link on NATS, the runtime's client — built and + proven in the lab (§10) while the migration continues on AMQP. Modules converted meanwhile + target the sdk contract and are untouched by this. +2. The cutover is one rollout, previewed: the controller assigns `nats` to the hub (raised beside + the AMQP broker on its own ports), composes every node's and module's account into it, then + rolls out the controller, every host and every runtime built for NATS. Each node's host connects + to the new bus as it comes up and reports; the controller confirms every node heard before it + stops listening on AMQP. The predecessor's clients never notice: their broker is the + compatibility module and stays. +3. The AMQP-side mesh accounts are removed from the compatibility broker; it keeps only the + predecessor's users. The bus's port settings follow ADR 0100 like any port. +4. The compatibility broker retires when its retirement condition holds. + +What is not done: no dual-bus period for the mesh's own traffic, no bridge, no module rebuilt. + +## 10. How it is checked + +Two lab beds, both required green before any node's bus moves. + +**The bus bed** — a mesh raised on NATS from genesis: +- a node enrols over TLS with a claimed token, and the enrolment user cannot read a declaration; +- a push composes; the store is stopped; the push is held (nak with delay), the store returns, the + push applies, nothing was lost or duplicated; +- a node that was away gets exactly the newest declaration, and a replayed older one is refused + by sequence; +- an upgrade rolls out to two nodes; +- a module's tool is invoked from another node and from a person's client, each with an account + that can invoke it, and refused from one that cannot; +- a module's account cannot publish outside its `emits` nor subscribe outside its `consumes` — + refused by the server; +- an event whose consumer keeps failing dead-letters after `max-deliver`; +- a module built before this design serves its tools unchanged on the new runtime. + +**The cutover bed** — a mesh on AMQP with a predecessor stand-in on the compatibility broker moves +its bus in one rollout; every node reports on NATS afterwards; the stand-in's client on AMQP is +still connected throughout. + +Unit tests hold the controller to composing accounts from `emits`/`consumes` and nothing else, to +creating the four streams and asserting them idempotently, and to spending a token exactly once; +the host to connecting as the enrolment user with nothing but enrolment permissions; the runtime +to mapping the sdk contract onto subjects exactly as §8 says. + +## 11. Open, for the review + +- Whether EVENTS should be one stream or one per emitting module (retention per module vs. one + policy). One stream is proposed; the review may disagree. +- The heartbeat interval and the controller's "quiet" threshold on core NATS without persistence — + the same numbers as today are proposed. +- Whether the person's client is a catalogue module (runs on an enrolled workstation node) or a + standalone program (runs anywhere with credentials). Both, in that order, is proposed. +- Leaf nodes: a NATS leaf per machine would make every module's connection local and survive the + hub's restart. Deliberately out of scope; noted so it is not forgotten.