Task 3.2. ADR 0074's model is untouched — floor plus capabilities, partial implementations legitimate, identity from the credential, dedup on x-event-id, conformance as executable fixtures. The transport beneath it is rewritten: exchanges and queues become subjects and streams. Statements marked *verified* were checked against a running server while the runtime's client was written, not reasoned from documentation. Three of them are things the specification would otherwise have got wrong: - the payload is the body alone, with metadata in NATS headers; an implementation that nested the whole envelope would agree with nobody - a durable name may not contain a dot, while the ack subject joins two names with one — conflating them looks right in a permission list and is refused as a consumer name - a certificate must carry a name the bus is dialled by, because the NATS client has no hook to replace hostname verification the way pinning did on AMQP And one limitation lifts: a module may now call another's tool. Issue 049 recorded that a scoped account could not declare the reply queue a caller needs, and ADR 0095 routed every ask through the control plane because of it. Per-account inbox prefixes plus allow_responses replace that. ADR 0095 is not reversed — the control plane is still how a person asks — but module-to-module calling stops being a question about capability and becomes one about policy, which `uses` already answers.
259 lines
14 KiB
Markdown
259 lines
14 KiB
Markdown
---
|
|
layer: to-be
|
|
status: proposed
|
|
code:
|
|
- mesh-sdk src
|
|
- mesh-tools src/broker-nats.ts (and broker-amqp.ts until the rollout)
|
|
- mesh-controller internal/link
|
|
updated: 2026-09-26
|
|
decisions:
|
|
- 02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md
|
|
- 02-DECISIONS/0106-the-bus-is-nats.md
|
|
- 02-DECISIONS/0120-the-mesh-bus-is-required-not-ambient.md
|
|
- 02-DECISIONS/0116-the-bus-is-built-in-five-steps.md
|
|
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
|
|
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
|
|
- 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
|
|
- 02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md
|
|
- 02-DECISIONS/0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md
|
|
---
|
|
|
|
# The module protocol
|
|
|
|
**What a module's code and the mesh say to each other.** An SDK is an implementation of this in one
|
|
language and nothing more ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md)).
|
|
|
|
This is a specification, so it says what is required rather than how anything is arranged. Where it
|
|
describes current behaviour that is *not yet* specified-and-conformed, it says so.
|
|
|
|
> **Rewritten onto NATS, 2026-09-26** (step 3 of
|
|
> [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)). What
|
|
> [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md) decided is untouched:
|
|
> a floor plus independent capabilities, an implementation legitimate when it claims less,
|
|
> identity from the sealed credential, at-least-once with dedup on `x-event-id`, and conformance
|
|
> as executable fixtures rather than prose. What changed is the transport beneath all of it —
|
|
> exchanges and queues became subjects and streams. The envelope keeps its shape
|
|
> ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)).
|
|
>
|
|
> Statements here marked *verified* were checked against a running server while the runtime's
|
|
> client was written, not reasoned from documentation.
|
|
|
|
## The shape of it
|
|
|
|
A **floor** every implementation needs, and three **capabilities** that are independent of each
|
|
other. An SDK implements the floor plus whatever capabilities it claims; a module is refused at
|
|
build time if it uses a capability its language's SDK does not implement.
|
|
|
|
| part | a module uses it to |
|
|
|---|---|
|
|
| **connection** | reach the broker as itself |
|
|
| **events** | emit, and react to what others emit |
|
|
| **tools** | answer questions asked of it |
|
|
| **provisioning** | give a consumer an instance of what it provides |
|
|
|
|
---
|
|
|
|
## The floor: connection
|
|
|
|
### The credential
|
|
|
|
A module is given a **sealed credential** as a file, and told where by its declaration. The
|
|
document:
|
|
|
|
| field | is | required |
|
|
|---|---|---|
|
|
| `url` | a `tls://` URL for the bus, with the account's user and password | yes |
|
|
| `fingerprint` | sha256 of the certificate the bus must present | yes for a scoped account |
|
|
| `node` | the machine this account was issued for | yes for a scoped account |
|
|
| `module` | the module this account was issued for | yes for a scoped account |
|
|
|
|
A plain string rather than a document is a **bootstrap URL** — unscoped, for the moment before a
|
|
mesh can issue anything. An implementation accepts both and must not treat the second as ordinary.
|
|
|
|
**`node` and `module` are not decoration: every subject an implementation touches is derived from
|
|
them.** Its own namespace is `mesh.mod.<module>`, its consumer is `<node>_<module>`, its inbox is
|
|
its own. So a credential without them is refused rather than guessed at — an implementation that
|
|
fell back to an environment variable would let anything on the machine decide which module it is,
|
|
which is what the identity rule below exists to prevent.
|
|
|
|
The credential itself is fetched, never carried in a declaration: a declaration is persisted as
|
|
state and a sealed secret in a stream is an archive rather than a moment
|
|
([design 29](29-what-a-module-declares.md) §10).
|
|
|
|
### Connecting
|
|
|
|
- The connection **pins the fingerprint**. It does not trust a certificate authority, and it does
|
|
not skip verification. A bus presenting a different certificate is refused, whatever else is
|
|
true of it.
|
|
- **The certificate must also carry a name the bus is dialled by.** *Verified:* the NATS client
|
|
exposes no hook to replace hostname verification, so pinning no longer makes it redundant the
|
|
way it did on AMQP — the pin happens before dialling and the library's own name check happens
|
|
beside it. A certificate without a matching subject-alternative name is refused at connect, by
|
|
a library error rather than by anything the mesh says.
|
|
- An implementation **creates nothing on the bus**: not a stream, not a consumer, not a subject.
|
|
Streams and durable consumers are the controller's alone ([design 25](25-the-bus-on-nats.md)
|
|
§3), and a module's account cannot reach the JetStream API to make one. An implementation binds
|
|
the consumer the mesh created for it, and if it is absent that is a mesh that has not finished
|
|
assigning the module, not something for the module to fix.
|
|
|
|
### Identity
|
|
|
|
**A module's node and module name come from its credential, never from its environment.**
|
|
|
|
This is not a convenience. It is what makes what a module emits match what the mesh authorised: an
|
|
environment variable can be set by anything on the machine, and a module that took its identity
|
|
from one could emit events attributing them to another module. Where an environment variable and
|
|
the credential disagree, the credential wins and the variable is overwritten.
|
|
|
|
---
|
|
|
|
## Capability: events
|
|
|
|
### The subjects
|
|
|
|
| subject | carries |
|
|
|---|---|
|
|
| `mesh.mod.<module>.event.<key>` | an event that module emitted |
|
|
| `mesh.seat.<seat>.event.<verb>` | an event the holder of that role emitted |
|
|
|
|
Both are captured by the `EVENTS` stream. **An event's source is enforced rather than claimed**: a
|
|
module's account may publish only into its own namespace, so `x-source` cannot disagree with where
|
|
the message arrived from.
|
|
|
|
**The `event` token is load-bearing.** A module's namespace also carries its tool calls
|
|
(`mesh.mod.<module>.tool.<tool>`), and a stream is defined by a subject filter — without the token
|
|
the events stream would capture every tool invocation in the mesh, and a tool call must never be
|
|
persisted.
|
|
|
|
### The consumer
|
|
|
|
One **durable consumer** per module, named `<node>_<module>`, carrying one filter per pattern the
|
|
module consumes. Durable because an event emitted while a module is restarting is exactly the one
|
|
that must not be lost.
|
|
|
|
**Created by the controller, bound by the implementation.** A module declares what it reacts to
|
|
and never how delivery works, so it does not name its consumer, does not choose its ack policy or
|
|
delivery limit, and cannot misconfigure them.
|
|
|
|
*Verified, and it is a trap:* a durable name **may not contain a dot**, while the subject a
|
|
consumer acknowledges on is `$JS.ACK.<stream>.<consumer>.…` — two names joined by one. An
|
|
implementation that treats them as a single string reads correctly in a permission list and is
|
|
refused as a consumer name. Left wrong, the symptom is every message redelivered forever while
|
|
the permissions look right.
|
|
|
|
**One consumer may carry filters wider than one handler's pattern**, because a module subscribing
|
|
twice gets one consumer with both. So an implementation still matches the key against its own
|
|
patterns locally to decide which handlers run — and **acknowledges a message no handler wanted**,
|
|
or it is redelivered until it expires.
|
|
|
|
### The envelope
|
|
|
|
Headers ride as **NATS headers**; the body is JSON, and the body alone. *Verified:* the payload is
|
|
the event's `body`, not the whole envelope re-encoded — an implementation that nested the envelope
|
|
would pass every one of its own tests and agree with no other, which is the exact failure the
|
|
conformance fixtures exist to catch. The key is recovered from the subject, not carried twice.
|
|
|
|
| header | is | required |
|
|
|---|---|---|
|
|
| `x-event-id` | a unique id, made by the emitter | yes |
|
|
| `x-source` | the module, context or node that emitted it | yes |
|
|
| `x-node` | the machine it was emitted from | yes |
|
|
| `x-time` | emit time, RFC-3339 | yes |
|
|
| `content-type` | always `application/json` | yes |
|
|
| `x-causation-id` | the event or command that caused this one | no |
|
|
| `x-schema` | a version of the body's shape | no |
|
|
|
|
**An unknown `x-` header is ignored, never refused.** An event is observed by parties that need not
|
|
all understand every header, and an implementation that refused one would make adding a header a
|
|
breaking change for everybody.
|
|
|
|
### Delivery
|
|
|
|
At-least-once. **Deduplication is on `x-event-id`**, which only the emitter can produce — a
|
|
consumer cannot tell a redelivery from a second event any other way.
|
|
|
|
On NATS the id does double duty: an implementation passes it as the publish's message id, so the
|
|
**server** also refuses a duplicate inside its window. That narrows the window in which a
|
|
consumer has to deduplicate; it does not remove the requirement, because the window is finite and
|
|
a redelivery after it is still a redelivery.
|
|
|
|
### What is true, checked (2026-09-16)
|
|
|
|
Go emits all five required headers; the SDK requires exactly those. `x-causation-id` and `x-schema`
|
|
are **optional** — the SDK sets them when a handler has a causation or a schema, and reads them
|
|
back; a bare event carrying neither is correct. So the envelope agrees across the two
|
|
implementations. `x-schema` is available for versioning a body's shape and is set by whoever has a
|
|
version to declare.
|
|
|
|
---
|
|
|
|
## Capability: tools
|
|
|
|
A module's tools are its operator-facing surface.
|
|
|
|
- A tool is served on `mesh.mod.<module>.tool.<tool>`, with a **queue group** — so several
|
|
runtimes serving one tool compete for a call rather than each answering it.
|
|
- A call is request and reply on **core NATS, never a stream**. A tool call is not persisted: a
|
|
lost one is a timeout the caller already handles, and a stream of them would be the mesh's most
|
|
voluminous and least valuable traffic competing for retention with the messages that matter.
|
|
- The reply goes to the inbox the request carries. A responder may answer it because its account
|
|
is granted **`allow_responses`** — one reply to the subject of a message it actually received,
|
|
and nothing wider. That is what makes a per-account inbox prefix workable: no user is ever
|
|
granted `_INBOX.>`, so without it a responder could not reach the caller at all.
|
|
- **A module may now call a tool, which on AMQP it could not.** *Verified:* two modules on
|
|
separate connections, one serving and one calling, with an answer returned and a throwing
|
|
handler reaching the caller as an error rather than a timeout.
|
|
[Issue 049](../../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)
|
|
recorded the old limit — a scoped account could not declare the reply queue a caller needs —
|
|
and [ADR 0095](../../02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md) routed
|
|
every ask through the control plane because of it. **That constraint is gone**, and each
|
|
account's own inbox prefix replaces it.
|
|
|
|
ADR 0095 is not thereby reversed: the control plane remains *a* way to ask, and a person asking
|
|
a module should still go through it. What changes is that "a module-to-module call, if one is
|
|
wanted, is a later decision" is no longer a question about *capability*. It is a policy
|
|
question, and the answer the mesh already has is `uses`: a module declares the seat it calls,
|
|
and the permission follows the declaration.
|
|
- A module declares nothing about being asked — serving a tool is being askable.
|
|
|
|
---
|
|
|
|
## Capability: provisioning
|
|
|
|
A provider ships the provisioner that creates instances of what it offers
|
|
([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)).
|
|
|
|
| direction | carries |
|
|
|---|---|
|
|
| **grant, in** | the provision, who is asking — **a module on a machine**, not a machine — and what the consumer contributed |
|
|
| **credential, out** | the fields the provision promises a consumer |
|
|
|
|
**Who is asking is one thing with two parts.** A machine routinely runs several modules wanting the
|
|
same provision, so a grant addressed to a node alone does not name a consumer, and withdrawing one
|
|
would take another's away.
|
|
|
|
### Checked, and it agrees (2026-09-16)
|
|
|
|
This looked like the sharpest disagreement and was not one. The live wire is the contributions file
|
|
— `as`, `secret`, `node`, `at`, `values` — and it is the same on both sides. The types that
|
|
disagreed (`Grant`, `Interface` in the SDK's `contracts`) were dead: exported, imported by nothing,
|
|
describing fields the wire does not carry. They have been removed. The lesson kept: a type beside
|
|
the wire that has drifted from it is worse than none, which is why the wire is specified and
|
|
implementations are checked against it rather than trusted to still match a hand-kept shape.
|
|
|
|
---
|
|
|
|
## How an implementation is checked
|
|
|
|
Per capability, against fixtures rather than prose — a specification nobody can run is a document
|
|
two implementations drift from while both believe they conform.
|
|
|
|
| Rule | Checked by |
|
|
|---|---|
|
|
| The floor is the floor | Every implementation reads the same credential fixture, and refuses one whose fingerprint does not match what the broker presents. |
|
|
| Identity comes from the credential | A fixture sets an environment that disagrees with the credential; the emitted event carries the credential's. |
|
|
| The envelope is the envelope | An emitted event is compared header by header against a fixture; a missing required header fails, an unknown `x-` header is accepted. |
|
|
| Delivery is at-least-once | A fixture delivered twice is handled once. |
|
|
| A grant names a consumer | A grant fixture is read by every implementation and yields the same module and the same node. |
|
|
| A partial SDK is legitimate | An implementation claiming the floor and events passes those suites and is listed for them; a module using tools in that language is refused at build time with the reason. |
|