Files
hq/03-DESIGN/01-to-be/19-the-module-protocol.md
T
jschoubben d940e14ec8 Design 19: the protocol on NATS
Task 3.2. ADR 0074's model is untouched — floor plus capabilities, partial
implementations legitimate, identity from the credential, dedup on
x-event-id, conformance as executable fixtures. The transport beneath it is
rewritten: exchanges and queues become subjects and streams.

Statements marked *verified* were checked against a running server while
the runtime's client was written, not reasoned from documentation. Three
of them are things the specification would otherwise have got wrong:

- the payload is the body alone, with metadata in NATS headers; an
  implementation that nested the whole envelope would agree with nobody
- a durable name may not contain a dot, while the ack subject joins two
  names with one — conflating them looks right in a permission list and is
  refused as a consumer name
- a certificate must carry a name the bus is dialled by, because the NATS
  client has no hook to replace hostname verification the way pinning did
  on AMQP

And one limitation lifts: a module may now call another's tool. Issue 049
recorded that a scoped account could not declare the reply queue a caller
needs, and ADR 0095 routed every ask through the control plane because of
it. Per-account inbox prefixes plus allow_responses replace that. ADR 0095
is not reversed — the control plane is still how a person asks — but
module-to-module calling stops being a question about capability and
becomes one about policy, which `uses` already answers.
2026-09-26 23:32:48 +02:00

259 lines
14 KiB
Markdown

---
layer: to-be
status: proposed
code:
- mesh-sdk src
- mesh-tools src/broker-nats.ts (and broker-amqp.ts until the rollout)
- mesh-controller internal/link
updated: 2026-09-26
decisions:
- 02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md
- 02-DECISIONS/0106-the-bus-is-nats.md
- 02-DECISIONS/0120-the-mesh-bus-is-required-not-ambient.md
- 02-DECISIONS/0116-the-bus-is-built-in-five-steps.md
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
- 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
- 02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md
- 02-DECISIONS/0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md
---
# The module protocol
**What a module's code and the mesh say to each other.** An SDK is an implementation of this in one
language and nothing more ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md)).
This is a specification, so it says what is required rather than how anything is arranged. Where it
describes current behaviour that is *not yet* specified-and-conformed, it says so.
> **Rewritten onto NATS, 2026-09-26** (step 3 of
> [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)). What
> [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md) decided is untouched:
> a floor plus independent capabilities, an implementation legitimate when it claims less,
> identity from the sealed credential, at-least-once with dedup on `x-event-id`, and conformance
> as executable fixtures rather than prose. What changed is the transport beneath all of it —
> exchanges and queues became subjects and streams. The envelope keeps its shape
> ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)).
>
> Statements here marked *verified* were checked against a running server while the runtime's
> client was written, not reasoned from documentation.
## The shape of it
A **floor** every implementation needs, and three **capabilities** that are independent of each
other. An SDK implements the floor plus whatever capabilities it claims; a module is refused at
build time if it uses a capability its language's SDK does not implement.
| part | a module uses it to |
|---|---|
| **connection** | reach the broker as itself |
| **events** | emit, and react to what others emit |
| **tools** | answer questions asked of it |
| **provisioning** | give a consumer an instance of what it provides |
---
## The floor: connection
### The credential
A module is given a **sealed credential** as a file, and told where by its declaration. The
document:
| field | is | required |
|---|---|---|
| `url` | a `tls://` URL for the bus, with the account's user and password | yes |
| `fingerprint` | sha256 of the certificate the bus must present | yes for a scoped account |
| `node` | the machine this account was issued for | yes for a scoped account |
| `module` | the module this account was issued for | yes for a scoped account |
A plain string rather than a document is a **bootstrap URL** — unscoped, for the moment before a
mesh can issue anything. An implementation accepts both and must not treat the second as ordinary.
**`node` and `module` are not decoration: every subject an implementation touches is derived from
them.** Its own namespace is `mesh.mod.<module>`, its consumer is `<node>_<module>`, its inbox is
its own. So a credential without them is refused rather than guessed at — an implementation that
fell back to an environment variable would let anything on the machine decide which module it is,
which is what the identity rule below exists to prevent.
The credential itself is fetched, never carried in a declaration: a declaration is persisted as
state and a sealed secret in a stream is an archive rather than a moment
([design 29](29-what-a-module-declares.md) §10).
### Connecting
- The connection **pins the fingerprint**. It does not trust a certificate authority, and it does
not skip verification. A bus presenting a different certificate is refused, whatever else is
true of it.
- **The certificate must also carry a name the bus is dialled by.** *Verified:* the NATS client
exposes no hook to replace hostname verification, so pinning no longer makes it redundant the
way it did on AMQP — the pin happens before dialling and the library's own name check happens
beside it. A certificate without a matching subject-alternative name is refused at connect, by
a library error rather than by anything the mesh says.
- An implementation **creates nothing on the bus**: not a stream, not a consumer, not a subject.
Streams and durable consumers are the controller's alone ([design 25](25-the-bus-on-nats.md)
§3), and a module's account cannot reach the JetStream API to make one. An implementation binds
the consumer the mesh created for it, and if it is absent that is a mesh that has not finished
assigning the module, not something for the module to fix.
### Identity
**A module's node and module name come from its credential, never from its environment.**
This is not a convenience. It is what makes what a module emits match what the mesh authorised: an
environment variable can be set by anything on the machine, and a module that took its identity
from one could emit events attributing them to another module. Where an environment variable and
the credential disagree, the credential wins and the variable is overwritten.
---
## Capability: events
### The subjects
| subject | carries |
|---|---|
| `mesh.mod.<module>.event.<key>` | an event that module emitted |
| `mesh.seat.<seat>.event.<verb>` | an event the holder of that role emitted |
Both are captured by the `EVENTS` stream. **An event's source is enforced rather than claimed**: a
module's account may publish only into its own namespace, so `x-source` cannot disagree with where
the message arrived from.
**The `event` token is load-bearing.** A module's namespace also carries its tool calls
(`mesh.mod.<module>.tool.<tool>`), and a stream is defined by a subject filter — without the token
the events stream would capture every tool invocation in the mesh, and a tool call must never be
persisted.
### The consumer
One **durable consumer** per module, named `<node>_<module>`, carrying one filter per pattern the
module consumes. Durable because an event emitted while a module is restarting is exactly the one
that must not be lost.
**Created by the controller, bound by the implementation.** A module declares what it reacts to
and never how delivery works, so it does not name its consumer, does not choose its ack policy or
delivery limit, and cannot misconfigure them.
*Verified, and it is a trap:* a durable name **may not contain a dot**, while the subject a
consumer acknowledges on is `$JS.ACK.<stream>.<consumer>.…` — two names joined by one. An
implementation that treats them as a single string reads correctly in a permission list and is
refused as a consumer name. Left wrong, the symptom is every message redelivered forever while
the permissions look right.
**One consumer may carry filters wider than one handler's pattern**, because a module subscribing
twice gets one consumer with both. So an implementation still matches the key against its own
patterns locally to decide which handlers run — and **acknowledges a message no handler wanted**,
or it is redelivered until it expires.
### The envelope
Headers ride as **NATS headers**; the body is JSON, and the body alone. *Verified:* the payload is
the event's `body`, not the whole envelope re-encoded — an implementation that nested the envelope
would pass every one of its own tests and agree with no other, which is the exact failure the
conformance fixtures exist to catch. The key is recovered from the subject, not carried twice.
| header | is | required |
|---|---|---|
| `x-event-id` | a unique id, made by the emitter | yes |
| `x-source` | the module, context or node that emitted it | yes |
| `x-node` | the machine it was emitted from | yes |
| `x-time` | emit time, RFC-3339 | yes |
| `content-type` | always `application/json` | yes |
| `x-causation-id` | the event or command that caused this one | no |
| `x-schema` | a version of the body's shape | no |
**An unknown `x-` header is ignored, never refused.** An event is observed by parties that need not
all understand every header, and an implementation that refused one would make adding a header a
breaking change for everybody.
### Delivery
At-least-once. **Deduplication is on `x-event-id`**, which only the emitter can produce — a
consumer cannot tell a redelivery from a second event any other way.
On NATS the id does double duty: an implementation passes it as the publish's message id, so the
**server** also refuses a duplicate inside its window. That narrows the window in which a
consumer has to deduplicate; it does not remove the requirement, because the window is finite and
a redelivery after it is still a redelivery.
### What is true, checked (2026-09-16)
Go emits all five required headers; the SDK requires exactly those. `x-causation-id` and `x-schema`
are **optional** — the SDK sets them when a handler has a causation or a schema, and reads them
back; a bare event carrying neither is correct. So the envelope agrees across the two
implementations. `x-schema` is available for versioning a body's shape and is set by whoever has a
version to declare.
---
## Capability: tools
A module's tools are its operator-facing surface.
- A tool is served on `mesh.mod.<module>.tool.<tool>`, with a **queue group** — so several
runtimes serving one tool compete for a call rather than each answering it.
- A call is request and reply on **core NATS, never a stream**. A tool call is not persisted: a
lost one is a timeout the caller already handles, and a stream of them would be the mesh's most
voluminous and least valuable traffic competing for retention with the messages that matter.
- The reply goes to the inbox the request carries. A responder may answer it because its account
is granted **`allow_responses`** — one reply to the subject of a message it actually received,
and nothing wider. That is what makes a per-account inbox prefix workable: no user is ever
granted `_INBOX.>`, so without it a responder could not reach the caller at all.
- **A module may now call a tool, which on AMQP it could not.** *Verified:* two modules on
separate connections, one serving and one calling, with an answer returned and a throwing
handler reaching the caller as an error rather than a timeout.
[Issue 049](../../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)
recorded the old limit — a scoped account could not declare the reply queue a caller needs —
and [ADR 0095](../../02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md) routed
every ask through the control plane because of it. **That constraint is gone**, and each
account's own inbox prefix replaces it.
ADR 0095 is not thereby reversed: the control plane remains *a* way to ask, and a person asking
a module should still go through it. What changes is that "a module-to-module call, if one is
wanted, is a later decision" is no longer a question about *capability*. It is a policy
question, and the answer the mesh already has is `uses`: a module declares the seat it calls,
and the permission follows the declaration.
- A module declares nothing about being asked — serving a tool is being askable.
---
## Capability: provisioning
A provider ships the provisioner that creates instances of what it offers
([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)).
| direction | carries |
|---|---|
| **grant, in** | the provision, who is asking — **a module on a machine**, not a machine — and what the consumer contributed |
| **credential, out** | the fields the provision promises a consumer |
**Who is asking is one thing with two parts.** A machine routinely runs several modules wanting the
same provision, so a grant addressed to a node alone does not name a consumer, and withdrawing one
would take another's away.
### Checked, and it agrees (2026-09-16)
This looked like the sharpest disagreement and was not one. The live wire is the contributions file
— `as`, `secret`, `node`, `at`, `values` — and it is the same on both sides. The types that
disagreed (`Grant`, `Interface` in the SDK's `contracts`) were dead: exported, imported by nothing,
describing fields the wire does not carry. They have been removed. The lesson kept: a type beside
the wire that has drifted from it is worse than none, which is why the wire is specified and
implementations are checked against it rather than trusted to still match a hand-kept shape.
---
## How an implementation is checked
Per capability, against fixtures rather than prose — a specification nobody can run is a document
two implementations drift from while both believe they conform.
| Rule | Checked by |
|---|---|
| The floor is the floor | Every implementation reads the same credential fixture, and refuses one whose fingerprint does not match what the broker presents. |
| Identity comes from the credential | A fixture sets an environment that disagrees with the credential; the emitted event carries the credential's. |
| The envelope is the envelope | An emitted event is compared header by header against a fixture; a missing required header fails, an unknown `x-` header is accepted. |
| Delivery is at-least-once | A fixture delivered twice is handled once. |
| A grant names a consumer | A grant fixture is read by every implementation and yields the same module and the same node. |
| A partial SDK is legitimate | An implementation claiming the floor and events passes those suites and is listed for them; a module using tools in that language is refused at build time with the reason. |