diff --git a/03-DESIGN/01-to-be/19-the-module-protocol.md b/03-DESIGN/01-to-be/19-the-module-protocol.md index b95b42b..620facd 100644 --- a/03-DESIGN/01-to-be/19-the-module-protocol.md +++ b/03-DESIGN/01-to-be/19-the-module-protocol.md @@ -3,12 +3,13 @@ layer: to-be status: proposed code: - mesh-sdk src - - mesh-tools src/broker-amqp.ts + - mesh-tools src/broker-nats.ts (and broker-amqp.ts until the rollout) - mesh-controller internal/link updated: 2026-09-26 decisions: - 02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md - 02-DECISIONS/0106-the-bus-is-nats.md + - 02-DECISIONS/0120-the-mesh-bus-is-required-not-ambient.md - 02-DECISIONS/0116-the-bus-is-built-in-five-steps.md - 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md - 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md @@ -25,26 +26,17 @@ language and nothing more ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specif This is a specification, so it says what is required rather than how anything is arranged. Where it describes current behaviour that is *not yet* specified-and-conformed, it says so. -> **The wire below is the bus being replaced.** *2026-09-26.* -> [ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md) moved the mesh's bus to NATS. Everything -> in this document that names an exchange, a queue or a routing key — the event exchanges, the -> durable `..events` queue, the shared `serve.` queue — describes the transport -> being retired, and the conformance fixtures were captured against it. +> **Rewritten onto NATS, 2026-09-26** (step 3 of +> [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)). What +> [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md) decided is untouched: +> a floor plus independent capabilities, an implementation legitimate when it claims less, +> identity from the sealed credential, at-least-once with dedup on `x-event-id`, and conformance +> as executable fixtures rather than prose. What changed is the transport beneath all of it — +> exchanges and queues became subjects and streams. The envelope keeps its shape +> ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)). > -> What does **not** change is this document's model, which is the part ADR 0074 decided: a floor -> plus independent capabilities, an SDK that implements what it claims and is legitimate when it -> claims less, identity taken from the sealed credential rather than the environment, at-least-once -> with dedup on `x-event-id`, and conformance as executable fixtures per capability rather than -> prose. The envelope keeps its shape ([ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md)); -> it becomes the message body. -> -> Rewriting the wire sections onto the subjects and streams of -> [design 25](25-the-bus-on-nats.md) §2–§3, and recapturing the fixtures there, is **step 3 of -> [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)**. Until that lands, read -> the sections below for what two implementations may not disagree *about*, and design 25 for what -> they will disagree about it *on*. A specification that silently described a retired transport -> would be worse than an absent one, because it reads as current — hence this note rather than a -> quiet edit. +> Statements here marked *verified* were checked against a running server while the runtime's +> client was written, not reasoned from documentation. ## The shape of it @@ -70,22 +62,39 @@ document: | field | is | required | |---|---|---| -| `url` | an `amqps://` URL carrying the account's user and password | yes | -| `fingerprint` | sha256 of the certificate the broker must present | yes for a scoped account | +| `url` | a `tls://` URL for the bus, with the account's user and password | yes | +| `fingerprint` | sha256 of the certificate the bus must present | yes for a scoped account | | `node` | the machine this account was issued for | yes for a scoped account | | `module` | the module this account was issued for | yes for a scoped account | A plain string rather than a document is a **bootstrap URL** — unscoped, for the moment before a mesh can issue anything. An implementation accepts both and must not treat the second as ordinary. +**`node` and `module` are not decoration: every subject an implementation touches is derived from +them.** Its own namespace is `mesh.mod.`, its consumer is `_`, its inbox is +its own. So a credential without them is refused rather than guessed at — an implementation that +fell back to an environment variable would let anything on the machine decide which module it is, +which is what the identity rule below exists to prevent. + +The credential itself is fetched, never carried in a declaration: a declaration is persisted as +state and a sealed secret in a stream is an archive rather than a moment +([design 29](29-what-a-module-declares.md) §10). + ### Connecting - The connection **pins the fingerprint**. It does not trust a certificate authority, and it does - not skip verification. A broker presenting a different certificate is refused, whatever else is + not skip verification. A bus presenting a different certificate is refused, whatever else is true of it. -- A scoped account **does not declare exchanges**. The foundation owns them; an account that may - declare one is an account that may create a parallel mesh by typo. -- An implementation **declares its own queue** and nothing else. +- **The certificate must also carry a name the bus is dialled by.** *Verified:* the NATS client + exposes no hook to replace hostname verification, so pinning no longer makes it redundant the + way it did on AMQP — the pin happens before dialling and the library's own name check happens + beside it. A certificate without a matching subject-alternative name is refused at connect, by + a library error rather than by anything the mesh says. +- An implementation **creates nothing on the bus**: not a stream, not a consumer, not a subject. + Streams and durable consumers are the controller's alone ([design 25](25-the-bus-on-nats.md) + §3), and a module's account cannot reach the JetStream API to make one. An implementation binds + the consumer the mesh created for it, and if it is absent that is a mesh that has not finished + assigning the module, not something for the module to fix. ### Identity @@ -100,26 +109,49 @@ the credential disagree, the credential wins and the variable is overwritten. ## Capability: events -### The exchanges +### The subjects -| exchange | carries | +| subject | carries | |---|---| -| `mesh.events` | every event | -| `mesh.events.dead` | what could not be handled | +| `mesh.mod..event.` | an event that module emitted | +| `mesh.seat..event.` | an event the holder of that role emitted | -### The queue +Both are captured by the `EVENTS` stream. **An event's source is enforced rather than claimed**: a +module's account may publish only into its own namespace, so `x-source` cannot disagree with where +the message arrived from. -One **durable** queue per consumer, named `..events`, with as many bindings as the -module has patterns. Durable because an event emitted while a module is restarting is exactly the -one that must not be lost. +**The `event` token is load-bearing.** A module's namespace also carries its tool calls +(`mesh.mod..tool.`), and a stream is defined by a subject filter — without the token +the events stream would capture every tool invocation in the mesh, and a tool call must never be +persisted. -**A message matching two bindings is delivered once**, so an implementation must match the routing -key against its own patterns locally to decide which handlers run. An implementation that ran every -handler whose exchange binding matched would run the wrong one. +### The consumer + +One **durable consumer** per module, named `_`, carrying one filter per pattern the +module consumes. Durable because an event emitted while a module is restarting is exactly the one +that must not be lost. + +**Created by the controller, bound by the implementation.** A module declares what it reacts to +and never how delivery works, so it does not name its consumer, does not choose its ack policy or +delivery limit, and cannot misconfigure them. + +*Verified, and it is a trap:* a durable name **may not contain a dot**, while the subject a +consumer acknowledges on is `$JS.ACK...…` — two names joined by one. An +implementation that treats them as a single string reads correctly in a permission list and is +refused as a consumer name. Left wrong, the symptom is every message redelivered forever while +the permissions look right. + +**One consumer may carry filters wider than one handler's pattern**, because a module subscribing +twice gets one consumer with both. So an implementation still matches the key against its own +patterns locally to decide which handlers run — and **acknowledges a message no handler wanted**, +or it is redelivered until it expires. ### The envelope -Headers ride as AMQP headers. The body is JSON. +Headers ride as **NATS headers**; the body is JSON, and the body alone. *Verified:* the payload is +the event's `body`, not the whole envelope re-encoded — an implementation that nested the envelope +would pass every one of its own tests and agree with no other, which is the exact failure the +conformance fixtures exist to catch. The key is recovered from the subject, not carried twice. | header | is | required | |---|---|---| @@ -140,6 +172,11 @@ breaking change for everybody. At-least-once. **Deduplication is on `x-event-id`**, which only the emitter can produce — a consumer cannot tell a redelivery from a second event any other way. +On NATS the id does double duty: an implementation passes it as the publish's message id, so the +**server** also refuses a duplicate inside its window. That narrows the window in which a +consumer has to deduplicate; it does not remove the requirement, because the window is finite and +a redelivery after it is still a redelivery. + ### What is true, checked (2026-09-16) Go emits all five required headers; the SDK requires exactly those. `x-causation-id` and `x-schema` @@ -154,22 +191,30 @@ version to declare. A module's tools are its operator-facing surface. -- A tool is served from a **shared durable queue**, `serve.`. Shared, so several runtimes - serving one tool compete for a call rather than each answering it. -- A call is request and reply. The reply returns through the RPC exchange `mesh.rpc`, keyed by the - caller's own reply queue — **not** through the default exchange, which would let a caller publish - into any queue on the broker. -- A caller needs a **reply queue**, and that is what a module's scoped account may not declare - ([issue 049](../../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)). - So a module may serve tools and may not call them. -- **The control plane is the way to ask** - ([ADR 0095](../../02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md)): - `ask [json]` publishes on `mesh.rpc` under `.` with a private reply - queue bound under its own name, and prints the answer as the module gave it. A module declares - nothing about being asked — serving a tool is being askable through the control plane. A - module-to-module call, if one is wanted, is a grant like any other and a later decision. - *How it is checked:* a tools-only bed asks a served tool through the control plane and asserts - an answer arrived, where a timeout would read differently. +- A tool is served on `mesh.mod..tool.`, with a **queue group** — so several + runtimes serving one tool compete for a call rather than each answering it. +- A call is request and reply on **core NATS, never a stream**. A tool call is not persisted: a + lost one is a timeout the caller already handles, and a stream of them would be the mesh's most + voluminous and least valuable traffic competing for retention with the messages that matter. +- The reply goes to the inbox the request carries. A responder may answer it because its account + is granted **`allow_responses`** — one reply to the subject of a message it actually received, + and nothing wider. That is what makes a per-account inbox prefix workable: no user is ever + granted `_INBOX.>`, so without it a responder could not reach the caller at all. +- **A module may now call a tool, which on AMQP it could not.** *Verified:* two modules on + separate connections, one serving and one calling, with an answer returned and a throwing + handler reaching the caller as an error rather than a timeout. + [Issue 049](../../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md) + recorded the old limit — a scoped account could not declare the reply queue a caller needs — + and [ADR 0095](../../02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md) routed + every ask through the control plane because of it. **That constraint is gone**, and each + account's own inbox prefix replaces it. + + ADR 0095 is not thereby reversed: the control plane remains *a* way to ask, and a person asking + a module should still go through it. What changes is that "a module-to-module call, if one is + wanted, is a later decision" is no longer a question about *capability*. It is a policy + question, and the answer the mesh already has is `uses`: a module declares the seat it calls, + and the permission follows the declaration. +- A module declares nothing about being asked — serving a tool is being askable. ---