hq/03-DESIGN/01-to-be/25-the-bus-on-nats.md

layer, status, code, updated, decisions

layer	status	code	updated	decisions
to-be	proposed	mesh-controller internal/link (to be replaced) mesh-host internal/link (to be replaced) mesh-tools src/broker-amqp.ts (to be replaced) mesh-catalog modules/nats (to be written)	2026-09-23	02-DECISIONS/0106-the-bus-is-nats.md 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md

25. The bus on NATS

Status: proposed — a design to be reviewed before any code. This is the architecture ADR 0106 asks for. It says what rides the bus, under which subject, with which guarantee, under whose account; how a node joins; how a person reaches a tool; how the mesh moves from the bus it has to this one; and how each claim is checked. Prose and diagrams only; no configuration is pasted.

1. What the bus is for

The bus carries five kinds of traffic today, and this design keeps the five, renaming nothing a module can see:

Traffic	Today	Guarantee it needs
control — a node's report, its heartbeat, a build's outcome, an enrolment	queues control, .upgrades, .catchup	nothing lost while the store restarts; retried; in order per node
declarations — the controller tells a node what to be	queue node.<name>	the node gets the newest; a stale one is never applied
builds — the controller asks the build machine to build	queue builds	at least once, one builder at a time
events — a module says something happened	topic exchange mesh.events, keys <module>.<event>	delivered to every consumer that declared it; dead-lettered when it cannot be
tools — one module or person asks another's tool a question	exchange mesh.rpc, per-tool service queues serve.<module>.<tool>	one answer, from one server, or a timeout

The sdk's contract — request, handle, publish, subscribe, close — is the whole surface a module sees, and it does not change (ADR 0039).

2. Subjects

NATS addresses everything by subject. The mesh's subject space is one tree, and every account's permissions are expressed as which branches of it that account may publish to and subscribe from.

mesh.control.<node>.report        a node's report              (JetStream: CONTROL)
mesh.control.<node>.alive         heartbeat                    (core, no persistence)
mesh.control.enrol                an enrolment request         (JetStream: CONTROL)
mesh.control.built                a build's outcome            (JetStream: CONTROL)
mesh.node.<node>.declare          a declaration for a node     (JetStream: NODES, last-per-subject)
mesh.build.request                work for the build machine   (JetStream: BUILDS, work queue)
mesh.events.<module>.<event>      an event                     (JetStream: EVENTS)
mesh.tools.<module>.<tool>        a tool invocation            (core request/reply)
mesh.ask.<node>.<command>         the controller's command api (core request/reply)

Two things this buys over the exchanges: request/reply is native — a tool call is one request on mesh.tools.<module>.<tool> answered by whichever runtime serves it (a queue group per tool, so several nodes may serve one tool); and a declaration is last-per-subject — the NODES stream keeps only the newest message on mesh.node.<node>.declare, so a node that was away gets exactly the current declaration and nothing older. That is the wire-level answer to issue 107: the stream's sequence is the order, and a node that sees sequence n refuses n−1 by construction.

3. Streams, and the guarantees they carry

Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStream stream:

Stream	Subjects	Retention	Why
CONTROL	mesh.control.> except alive	work queue, one consumer (the controller), explicit ack	the store-window guarantee (ADR 0083): the controller naks with a delay while its store is away and the message is redelivered; nothing is dropped
NODES	mesh.node.>	last per subject	one declaration per node, always the newest
BUILDS	mesh.build.>	work queue, explicit ack	at least once; a builder that dies mid-build has its message redelivered
EVENTS	mesh.events.>	limits (age, size), durable consumer per subscribing module	a subscriber that was down catches up; after max-deliver attempts the advisory feeds mesh.events.dead (its own small stream)

Tool calls and heartbeats stay on core NATS: a lost heartbeat is the next heartbeat; a lost tool call is a timeout the caller already handles.

Streams and consumers are objects the controller creates at genesis and asserts on start; a module declares nothing about them. The controller is the only writer of stream definitions.

4. Accounts

ADR 0043 says a module's account may publish only what it emits and consume only what it consumes. NATS expresses this exactly, per subject, and better than a vhost could:

• One NATS account for the mesh. Accounts in NATS isolate subject spaces entirely; the mesh is one space, so it is one account. The predecessor's compatibility broker is not on this bus at all.
• One user per module per node, as today, with publish permissions mesh.events.<module>.<event> for each emit, mesh.tools.<module>.> to serve its tools, and its reply inbox; subscribe permissions for each consumed event's subject and its tool subjects. Nothing else. A module that tries to publish outside its emits is refused by the server, not by convention.
• The controller's user owns mesh.control.>, mesh.node.>, mesh.build.> and the streams. A host's user may publish its own mesh.control.<node>.> and subscribe its own mesh.node.<node>.declare — and nothing of any other node's.
• A person's user (§7) is a module-shaped user with permissions on the tool subjects it may invoke, issued and revoked by the controller like any account.

Accounts are configuration, not API calls. The controller composes the server's user list and permissions into a file the host declares; the server reloads on change (reload-on, as the mesh already does for the container runtime's trust). No management API, no credential travelling through a management call, and the issue 102 discipline from the first day: an address or a permission is read where it is used, never stored with a port. Passwords are minted and sealed exactly as today; the file holds bcrypt hashes.

Alternative considered and not taken: the operator/JWT model (nsc), where accounts are signed tokens resolved by the server. It is the right model for a multi-tenant NATS; the mesh is one tenant, already has a sealing key and a controller that writes files, and would gain a second signing hierarchy for nothing.

5. The broker as a module

nats is a catalogue module claiming the seat mesh-broker (ADR 0079: the seat is the server, and the server changes). It declares one container (a single binary; JetStream on a named volume), its listening ports — client, TLS, and the monitoring endpoint on loopback — a configuration file the controller composes (accounts, permissions, TLS, JetStream), and a reload-on for that file. Its guard is the same rule as the AMQP broker's: the monitoring port is refused from anything but the private network. It is raised at genesis like the store, adopted as a module in the same phase. The predecessor's AMQP broker remains a module of its own, lavinmq-compat, with a single purpose and a retirement condition: no client connected for a period the operator sets.

6. Joining: the enrolment handshake

Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS with the enrolment user — a user that may publish mesh.control.enrol and subscribe one reply inbox and nothing else — publishes its request (the claim of the token, its keys, its proof, and the found tunnel from ADR 0105), and waits on the inbox. The controller spends the token, records the node, composes the node's own user into the server's configuration, and answers with the credentials sealed to the node's sealing key. The node reconnects as itself. The enrolment user's permissions are what make a leaked token useless for anything but enrolling: it cannot read a declaration or hear an event.

7. A person's client

The operator asked for the mesh's tools from a workstation, and for it designed here rather than bridged. It is three things:

1. A person's account: operator issue <name> on the controller creates a user whose permissions are the tool subjects it may invoke — mesh.tools.> for an administrator, a list for anyone else — and nothing on control, nodes or builds. It is issued, sealed to the person's own key, and revoked, like a module's.
2. A client that speaks the bus: a small program on the workstation that connects as that user over TLS, lists tools by asking the catalogue (mesh.tools.mesh-catalog.catalog_tools), and turns each tool into a call — as an MCP server for an agent, and as a command line for a person. It uses the sdk's Broker contract on the NATS runtime, so it is the same code path a module's tools use, not a second protocol.
3. Reachability: the workstation reaches the bus over the private network once it is a node, or over the predecessor's tunnel before that, on the bus's port; the guard and the openings treat the bus as they do today.

Nothing is built of this before §10's bed passes; the MCP surface is a thin adapter over (2).

8. What a module sees

Nothing new. publish on an envelope becomes a publish on mesh.events.<module>.<event>; subscribe with a pattern becomes a durable JetStream consumer on the matching subject filter; request/handle become a NATS request and a queue-group subscription on mesh.tools.<module>.<tool>. The envelope's shape (ADR 0042) is unchanged; it is the message body. A module built today runs on the new runtime without a rebuild — that is the test of ADR 0039, and it is in §10.

9. Moving from the bus the mesh has

Per ADR 0106: built beside, cut over once, after the core.

1. The nats module, the controller's and host's link on NATS, the runtime's client — built and proven in the lab (§10) while the migration continues on AMQP. Modules converted meanwhile target the sdk contract and are untouched by this.
2. The cutover is one rollout, previewed: the controller assigns nats to the hub (raised beside the AMQP broker on its own ports), composes every node's and module's account into it, then rolls out the controller, every host and every runtime built for NATS. Each node's host connects to the new bus as it comes up and reports; the controller confirms every node heard before it stops listening on AMQP. The predecessor's clients never notice: their broker is the compatibility module and stays.
3. The AMQP-side mesh accounts are removed from the compatibility broker; it keeps only the predecessor's users. The bus's port settings follow ADR 0100 like any port.
4. The compatibility broker retires when its retirement condition holds.

What is not done: no dual-bus period for the mesh's own traffic, no bridge, no module rebuilt.

10. How it is checked

Two lab beds, both required green before any node's bus moves.

The bus bed — a mesh raised on NATS from genesis:

• a node enrols over TLS with a claimed token, and the enrolment user cannot read a declaration;
• a push composes; the store is stopped; the push is held (nak with delay), the store returns, the push applies, nothing was lost or duplicated;
• a node that was away gets exactly the newest declaration, and a replayed older one is refused by sequence;
• an upgrade rolls out to two nodes;
• a module's tool is invoked from another node and from a person's client, each with an account that can invoke it, and refused from one that cannot;
• a module's account cannot publish outside its emits nor subscribe outside its consumes — refused by the server;
• an event whose consumer keeps failing dead-letters after max-deliver;
• a module built before this design serves its tools unchanged on the new runtime.

The cutover bed — a mesh on AMQP with a predecessor stand-in on the compatibility broker moves its bus in one rollout; every node reports on NATS afterwards; the stand-in's client on AMQP is still connected throughout.

Unit tests hold the controller to composing accounts from emits/consumes and nothing else, to creating the four streams and asserting them idempotently, and to spending a token exactly once; the host to connecting as the enrolment user with nothing but enrolment permissions; the runtime to mapping the sdk contract onto subjects exactly as §8 says.

11. Open, for the review

• Whether EVENTS should be one stream or one per emitting module (retention per module vs. one policy). One stream is proposed; the review may disagree.
• The heartbeat interval and the controller's "quiet" threshold on core NATS without persistence — the same numbers as today are proposed.
• Whether the person's client is a catalogue module (runs on an enrolled workstation node) or a standalone program (runs anywhere with credentials). Both, in that order, is proposed.
• Leaf nodes: a NATS leaf per machine would make every module's connection local and survive the hub's restart. Deliberately out of scope; noted so it is not forgotten.