novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Design 19: the protocol on NATS

Browse Source 
Task 3.2. ADR 0074's model is untouched — floor plus capabilities, partial
implementations legitimate, identity from the credential, dedup on
x-event-id, conformance as executable fixtures. The transport beneath it is
rewritten: exchanges and queues become subjects and streams.

Statements marked *verified* were checked against a running server while
the runtime's client was written, not reasoned from documentation. Three
of them are things the specification would otherwise have got wrong:

- the payload is the body alone, with metadata in NATS headers; an
  implementation that nested the whole envelope would agree with nobody
- a durable name may not contain a dot, while the ack subject joins two
  names with one — conflating them looks right in a permission list and is
  refused as a consumer name
- a certificate must carry a name the bus is dialled by, because the NATS
  client has no hook to replace hostname verification the way pinning did
  on AMQP

And one limitation lifts: a module may now call another's tool. Issue 049
recorded that a scoped account could not declare the reply queue a caller
needs, and ADR 0095 routed every ask through the control plane because of
it. Per-account inbox prefixes plus allow_responses replace that. ADR 0095
is not reversed — the control plane is still how a person asks — but
module-to-module calling stops being a question about capability and
becomes one about policy, which `uses` already answers.
This commit is contained in:
jschoubben
2026-09-26 23:32:48 +02:00
parent 80456981be
commit d940e14ec8
1 changed files with 99 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
03-DESIGN/01-to-be/19-the-module-protocol.md
+99 -54
View File
@@ -3,12 +3,13 @@ layer: to-be
status: proposed
code:
- mesh-sdk src
- mesh-tools src/broker-amqp.ts
- mesh-tools src/broker-nats.ts (and broker-amqp.ts until the rollout)
- mesh-controller internal/link
updated: 2026-09-26
decisions:
- 02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md
- 02-DECISIONS/0106-the-bus-is-nats.md
- 02-DECISIONS/0120-the-mesh-bus-is-required-not-ambient.md
- 02-DECISIONS/0116-the-bus-is-built-in-five-steps.md
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
@@ -25,26 +26,17 @@ language and nothing more ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specif
This is a specification, so it says what is required rather than how anything is arranged. Where it
describes current behaviour that is *not yet* specified-and-conformed, it says so.
> **The wire below is the bus being replaced.** *2026-09-26.*
> [ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md) moved the mesh's bus to NATS. Everything
> in this document that names an exchange, a queue or a routing key — the event exchanges, the
> durable `<node>.<module>.events` queue, the shared `serve.<key>` queue — describes the transport
> being retired, and the conformance fixtures were captured against it.
> **Rewritten onto NATS, 2026-09-26** (step 3 of
> [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)). What
> [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md) decided is untouched:
> a floor plus independent capabilities, an implementation legitimate when it claims less,
> identity from the sealed credential, at-least-once with dedup on `x-event-id`, and conformance
> as executable fixtures rather than prose. What changed is the transport beneath all of it —
> exchanges and queues became subjects and streams. The envelope keeps its shape
> ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)).
>
> What does **not** change is this document's model, which is the part ADR 0074 decided: a floor
> plus independent capabilities, an SDK that implements what it claims and is legitimate when it
> claims less, identity taken from the sealed credential rather than the environment, at-least-once
> with dedup on `x-event-id`, and conformance as executable fixtures per capability rather than
> prose. The envelope keeps its shape ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md));
> it becomes the message body.
>
> Rewriting the wire sections onto the subjects and streams of
> [design 25](25-the-bus-on-nats.md) §2–§3, and recapturing the fixtures there, is **step 3 of
> [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)**. Until that lands, read
> the sections below for what two implementations may not disagree *about*, and design 25 for what
> they will disagree about it *on*. A specification that silently described a retired transport
> would be worse than an absent one, because it reads as current — hence this note rather than a
> quiet edit.
> Statements here marked *verified* were checked against a running server while the runtime's
> client was written, not reasoned from documentation.
## The shape of it
@@ -70,22 +62,39 @@ document:
| field | is | required |
|---|---|---|
| `url` | an `amqps://` URL carrying the account's user and password | yes |
| `fingerprint` | sha256 of the certificate the broker must present | yes for a scoped account |
| `url` | a `tls://` URL for the bus, with the account's user and password | yes |
| `fingerprint` | sha256 of the certificate the bus must present | yes for a scoped account |
| `node` | the machine this account was issued for | yes for a scoped account |
| `module` | the module this account was issued for | yes for a scoped account |
A plain string rather than a document is a **bootstrap URL** — unscoped, for the moment before a
mesh can issue anything. An implementation accepts both and must not treat the second as ordinary.
**`node` and `module` are not decoration: every subject an implementation touches is derived from
them.** Its own namespace is `mesh.mod.<module>`, its consumer is `<node>_<module>`, its inbox is
its own. So a credential without them is refused rather than guessed at — an implementation that
fell back to an environment variable would let anything on the machine decide which module it is,
which is what the identity rule below exists to prevent.
The credential itself is fetched, never carried in a declaration: a declaration is persisted as
state and a sealed secret in a stream is an archive rather than a moment
([design 29](29-what-a-module-declares.md) §10).
### Connecting
- The connection **pins the fingerprint**. It does not trust a certificate authority, and it does
not skip verification. A broker presenting a different certificate is refused, whatever else is
not skip verification. A bus presenting a different certificate is refused, whatever else is
true of it.
- A scoped account **does not declare exchanges**. The foundation owns them; an account that may
declare one is an account that may create a parallel mesh by typo.
- An implementation **declares its own queue** and nothing else.
- **The certificate must also carry a name the bus is dialled by.** *Verified:* the NATS client
exposes no hook to replace hostname verification, so pinning no longer makes it redundant the
way it did on AMQP — the pin happens before dialling and the library's own name check happens
beside it. A certificate without a matching subject-alternative name is refused at connect, by
a library error rather than by anything the mesh says.
- An implementation **creates nothing on the bus**: not a stream, not a consumer, not a subject.
Streams and durable consumers are the controller's alone ([design 25](25-the-bus-on-nats.md)
§3), and a module's account cannot reach the JetStream API to make one. An implementation binds
the consumer the mesh created for it, and if it is absent that is a mesh that has not finished
assigning the module, not something for the module to fix.
### Identity
@@ -100,26 +109,49 @@ the credential disagree, the credential wins and the variable is overwritten.
## Capability: events
### The exchanges
### The subjects
| exchange | carries |
| subject | carries |
|---|---|
| `mesh.events` | every event |
| `mesh.events.dead` | what could not be handled |
| `mesh.mod.<module>.event.<key>` | an event that module emitted |
| `mesh.seat.<seat>.event.<verb>` | an event the holder of that role emitted |
### The queue
Both are captured by the `EVENTS` stream. **An event's source is enforced rather than claimed**: a
module's account may publish only into its own namespace, so `x-source` cannot disagree with where
the message arrived from.
One **durable** queue per consumer, named `<node>.<module>.events`, with as many bindings as the
module has patterns. Durable because an event emitted while a module is restarting is exactly the
one that must not be lost.
**The `event` token is load-bearing.** A module's namespace also carries its tool calls
(`mesh.mod.<module>.tool.<tool>`), and a stream is defined by a subject filter — without the token
the events stream would capture every tool invocation in the mesh, and a tool call must never be
persisted.
**A message matching two bindings is delivered once**, so an implementation must match the routing
key against its own patterns locally to decide which handlers run. An implementation that ran every
handler whose exchange binding matched would run the wrong one.
### The consumer
One **durable consumer** per module, named `<node>_<module>`, carrying one filter per pattern the
module consumes. Durable because an event emitted while a module is restarting is exactly the one
that must not be lost.
**Created by the controller, bound by the implementation.** A module declares what it reacts to
and never how delivery works, so it does not name its consumer, does not choose its ack policy or
delivery limit, and cannot misconfigure them.
*Verified, and it is a trap:* a durable name **may not contain a dot**, while the subject a
consumer acknowledges on is `$JS.ACK.<stream>.<consumer>.…` — two names joined by one. An
implementation that treats them as a single string reads correctly in a permission list and is
refused as a consumer name. Left wrong, the symptom is every message redelivered forever while
the permissions look right.
**One consumer may carry filters wider than one handler's pattern**, because a module subscribing
twice gets one consumer with both. So an implementation still matches the key against its own
patterns locally to decide which handlers run — and **acknowledges a message no handler wanted**,
or it is redelivered until it expires.
### The envelope
Headers ride as AMQP headers. The body is JSON.
Headers ride as **NATS headers**; the body is JSON, and the body alone. *Verified:* the payload is
the event's `body`, not the whole envelope re-encoded — an implementation that nested the envelope
would pass every one of its own tests and agree with no other, which is the exact failure the
conformance fixtures exist to catch. The key is recovered from the subject, not carried twice.
| header | is | required |
|---|---|---|
@@ -140,6 +172,11 @@ breaking change for everybody.
At-least-once. **Deduplication is on `x-event-id`**, which only the emitter can produce — a
consumer cannot tell a redelivery from a second event any other way.
On NATS the id does double duty: an implementation passes it as the publish's message id, so the
**server** also refuses a duplicate inside its window. That narrows the window in which a
consumer has to deduplicate; it does not remove the requirement, because the window is finite and
a redelivery after it is still a redelivery.
### What is true, checked (2026-09-16)
Go emits all five required headers; the SDK requires exactly those. `x-causation-id` and `x-schema`
@@ -154,22 +191,30 @@ version to declare.
A module's tools are its operator-facing surface.
- A tool is served from a **shared durable queue**, `serve.<key>`. Shared, so several runtimes
serving one tool compete for a call rather than each answering it.
- A call is request and reply. The reply returns through the RPC exchange `mesh.rpc`, keyed by the
caller's own reply queue — **not** through the default exchange, which would let a caller publish
into any queue on the broker.
- A caller needs a **reply queue**, and that is what a module's scoped account may not declare
([issue 049](../../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)).
So a module may serve tools and may not call them.
- **The control plane is the way to ask**
([ADR 0095](../../02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md)):
`ask <module> <tool> [json]` publishes on `mesh.rpc` under `<module>.<tool>` with a private reply
queue bound under its own name, and prints the answer as the module gave it. A module declares
nothing about being asked — serving a tool is being askable through the control plane. A
module-to-module call, if one is wanted, is a grant like any other and a later decision.
*How it is checked:* a tools-only bed asks a served tool through the control plane and asserts
an answer arrived, where a timeout would read differently.
- A tool is served on `mesh.mod.<module>.tool.<tool>`, with a **queue group** — so several
runtimes serving one tool compete for a call rather than each answering it.
- A call is request and reply on **core NATS, never a stream**. A tool call is not persisted: a
lost one is a timeout the caller already handles, and a stream of them would be the mesh's most
voluminous and least valuable traffic competing for retention with the messages that matter.
- The reply goes to the inbox the request carries. A responder may answer it because its account
is granted **`allow_responses`** — one reply to the subject of a message it actually received,
and nothing wider. That is what makes a per-account inbox prefix workable: no user is ever
granted `_INBOX.>`, so without it a responder could not reach the caller at all.
- **A module may now call a tool, which on AMQP it could not.** *Verified:* two modules on
separate connections, one serving and one calling, with an answer returned and a throwing
handler reaching the caller as an error rather than a timeout.
[Issue 049](../../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)
recorded the old limit — a scoped account could not declare the reply queue a caller needs —
and [ADR 0095](../../02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md) routed
every ask through the control plane because of it. **That constraint is gone**, and each
account's own inbox prefix replaces it.
ADR 0095 is not thereby reversed: the control plane remains *a* way to ask, and a person asking
a module should still go through it. What changes is that "a module-to-module call, if one is
wanted, is a later decision" is no longer a question about *capability*. It is a policy
question, and the answer the mesh already has is `uses`: a module declares the seat it calls,
and the permission follows the declaration.
- A module declares nothing about being asked — serving a tool is being askable.
---