4.6 KiB
topic, status, date, deciders, reconstructed, extends
| topic | status | date | deciders | reconstructed | extends |
|---|---|---|---|---|---|
| the mesh | accepted | 2026-09-23 | jochen | false | 02-DECISIONS/0002-nodes-communicate-over-a-broker.md |
106. The bus is NATS
Context
The mesh's bus is an AMQP broker. Research 014 measured what that costs and what it would take to change: AMQP is spoken in three places of the mesh's own code — the controller's link, the host's link, the tool runtime's client — and in none of the sdk or the modules, because ADR 0039 kept the client out of the sdk for exactly this. The predecessor's world is AMQP and is being retired module by module; it cannot move and does not need to.
The operator's direction: NATS, native — the mesh built on it, not bridged to it — and the tools a person reaches from a workstation designed on it rather than built quickly on what exists.
Considered Options
- Stay on AMQP. Rejected by the operator.
- NATS behind a bridge, the mesh unchanged. Rejected: two buses, every guarantee crossing a seam, and the reason for changing — one native, simple, subject-addressed bus with request/reply and accounts built in — lost at the seam.
- NATS native, built in the lab in parallel with the migration, cut over in one rehearsed rollout after the migration's core is done. Adopted.
Decision
The mesh's bus is NATS. Every link the mesh has — control, node queues, builds, enrolment,
reports, events, tool invocation — is carried on NATS subjects; durability, catch-up and
hold-unacknowledged-and-retry (ADR 0083) are
JetStream's; a module's account is a NATS account with per-subject permissions derived from its
emits and consumes (ADR 0043),
written as configuration the host declares and the server reloads — never through a management
API. The broker module changes; the seat mesh-broker does not
(ADR 0079).
The sdk's broker contract does not change. Modules are written against request, handle,
publish, subscribe; the runtime implements them on NATS; no module speaks a protocol.
The AMQP broker the mesh adopted becomes the predecessor's compatibility broker, kept as a module with one purpose — the predecessor's clients — and retired with the last of them.
It is built beside the migration and cut over after its core. The bus is not changed under a half-migrated node. A lab bed proves the whole path — enrolment, the store window, upgrades, tool invocation — on NATS before any node's bus moves, and the move is one rollout: controller, every host, every runtime together.
What a person reaches the mesh's tools with is designed on NATS, as part of the same design: a person's account, scoped like a module's, and a client that speaks the bus directly.
Consequences
- The architecture — subjects, streams, accounts, the enrolment handshake, the person's client — is written as a to-be design before code, and reviewed.
- Eight records are touched: 0002 and 0033 (the substrate names a broker — now NATS), 0041 and
0042 (events keep their shape; the exchange becomes a subject prefix), 0043 (accounts as
configuration), 0078 (the broker module is
nats), 0083 (JetStream carries the guarantee), 0039 (unchanged, and the reason this is possible). - The lab beds that prove the bus are re-run on NATS; none is skipped.
- Until the cutover, nothing changes on any node.
How it is checked
A lab bed raises a mesh on NATS from genesis and proves: a node enrols over TLS with a claimed
token; a push composes, is held while the store restarts and applies after; an upgrade rolls out;
a module's tools are invoked from another node and from a person's client; an event dead-letters
after its deliveries are exhausted; a module's account cannot publish outside its emits nor
subscribe outside its consumes. Then the cutover bed: a mesh on AMQP with the predecessor's
compatibility broker beside it moves its bus in one rollout with every node reporting afterwards.