diff --git a/03-DESIGN/01-to-be/25-the-bus-on-nats.md b/03-DESIGN/01-to-be/25-the-bus-on-nats.md index 8ad276e..3ffe4f7 100644 --- a/03-DESIGN/01-to-be/25-the-bus-on-nats.md +++ b/03-DESIGN/01-to-be/25-the-bus-on-nats.md @@ -6,7 +6,7 @@ code: - mesh-host internal/link (to be replaced) - mesh-tools src/broker-amqp.ts (to be replaced) - mesh-catalog modules/nats (to be written) -updated: 2026-09-23 +updated: 2026-09-24 decisions: - 02-DECISIONS/0106-the-bus-is-nats.md - 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md @@ -63,6 +63,21 @@ exactly the current declaration and nothing older. That is the wire-level answer [issue 107](../../04-ISSUES/107-a-declaration-carries-no-order/00-report.md): the stream's sequence *is* the order, and a node that sees sequence n refuses n−1 by construction. +**A reply-to travelling through a JetStream stream is carried in the payload, never in the +transport `Reply` field.** Revision, first review: core NATS request/reply sets the requester's +ephemeral inbox as the message's `Reply` field, and a plain responder answers it directly — but a +message a JetStream consumer delivers has already had that field claimed for the consumer's own +ack address (`$JS.ACK.....`), so by the time the controller (§3's CONTROL +consumer) sees the message, `Reply` names where *it* must ack, not where the original caller is +waiting. `mesh.control.enrol` is the case that matters: a synchronous-feeling caller waiting on an +ephemeral inbox, over a subject the store-window guarantee may legitimately delay by several +`nak` cycles — exactly the combination that would otherwise deliver the answer to a caller who +has long since timed out and unsubscribed. So every CONTROL message that expects an answer states +its reply subject as an ordinary field of its own payload; the controller reads it from there and +publishes the answer to it explicitly, never via `Respond()`. Nothing else in this design routes +a reply through a stream — tools and heartbeats stay on core NATS, where `Reply` means what it has +always meant. + ## 3. Streams, and the guarantees they carry Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStream stream: @@ -89,9 +104,29 @@ expresses this exactly, per subject, and better than a vhost could: - **One NATS account for the mesh.** Accounts in NATS isolate subject spaces entirely; the mesh is one space, so it is one account. The predecessor's compatibility broker is not on this bus at all. + + **One account is a choice with a cost, stated plainly on revision:** none of NATS's own + isolation is free here, because there is only the one subject space, for everyone. Two + consequences that a single-account mesh must therefore grant on purpose, not by omission: + + - **A durable consumer needs permission to ack, or it never really consumes.** Acking a + JetStream delivery is a publish to that consumer's own ack-reply address + (`$JS.ACK...>`), a different subject from anything the consumer subscribes. + A module's user is therefore granted publish on `$JS.ACK.EVENTS..>` as well as its + emits — scoped to the one consumer name the controller derives for that module, so a module + can ack only its own deliveries. Without this, first review found, every message it receives + would be redelivered forever: refused by the permission list it already has. + - **A reply inbox needs a subject nothing else can guess or enumerate.** With one account, + inbox privacy is the permission list or it is nothing — there is no second account backing + it up. So no user is ever granted a bare `_INBOX.>`. Each user's inbox subject is derived + from its own identity (`_INBOX...>`, or `_INBOX.person..>`), and its + permissions name only that one prefix, for the reply to any request it makes and nothing + wider. First review found the account note without this and read it as "any user may + subscribe any inbox" — which was accurate against the text as it stood. - **One user per module per node**, as today, with publish permissions - `mesh.events..` for each emit, `mesh.tools..>` to serve its tools, and - its reply inbox; subscribe permissions for each consumed event's subject and its tool subjects. + `mesh.events..` for each emit, `mesh.tools..>` to serve its tools, its + own ack-reply subject for each durable consumer it holds, and its own inbox prefix; subscribe + permissions for each consumed event's subject, its tool subjects, and that same inbox prefix. Nothing else. A module that tries to publish outside its emits is refused by the server, not by convention. - **The controller's user** owns `mesh.control.>`, `mesh.node.>`, `mesh.build.>` and the streams. @@ -101,9 +136,10 @@ expresses this exactly, per subject, and better than a vhost could: invoke, issued and revoked by the controller like any account. **Accounts are configuration, not API calls.** The controller composes the server's user list and -permissions into a file the host declares; the server reloads on change (`reload-on`, as the mesh -already does for the container runtime's trust). No management API, no credential travelling -through a management call, and the [issue 102](../../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md) +permissions into a file the host declares. **How that file reaches the running server is §5's, +not this one's** — revision, first review: an earlier draft said "reloads" and cited a precedent +that does not apply to a container (see §5). No management API, no credential travelling through a +management call, and the [issue 102](../../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md) discipline from the first day: an address or a permission is read where it is used, never stored with a port. Passwords are minted and sealed exactly as today; the file holds bcrypt hashes. @@ -117,24 +153,53 @@ signing hierarchy for nothing. `nats` is a catalogue module claiming the seat `mesh-broker` ([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md): the seat is the server, and the server changes). It declares one container (a single binary; JetStream on a -named volume), its listening ports — client, TLS, and the monitoring endpoint on loopback — a -configuration file the controller composes (accounts, permissions, TLS, JetStream), and a -`reload-on` for that file. Its guard is the same rule as the AMQP broker's: the monitoring port is -refused from anything but the private network. It is raised at genesis like the store, adopted as a -module in the same phase. The predecessor's AMQP broker remains a module of its own, -`lavinmq-compat`, with a single purpose and a retirement condition: no client connected for a -period the operator sets. +named volume), its listening ports — client, TLS, and the monitoring endpoint on loopback — and a +configuration file the controller composes (accounts, permissions, TLS, JetStream). + +**How that file's changes reach the running server, corrected on revision.** First review: the +earlier draft named `reload-on` as the mechanism, citing the container runtime's own trust file as +precedent. `reload-on` is real, but it is a **service** field +([mesh-host declaration.go](https://git.novox.be/novox/mesh-host), `Service.ReloadOn` — +`docker.service` is reloaded via systemd, which is what the cited precedent actually does). A +**container** resource has no reload field at all — only `restart-on`, and a container's +`restart-on` is documented, exactly, to mean *recreate*. Declared as the earlier draft had it, +either the field is silently meaningless on a container resource or — if read as the nearest real +equivalent — every account, permission, or key change recreates the bus's own server: every +connection dropped, every in-flight JetStream ack lost, mid-flight the moment a module is added, +reassigned, or a person's access changes. For the one resource everything else depends on, that +is not an edge case; it is the common case. + +**The fix asks nothing new of the host.** `nats-server` already reloads its own configuration +live on `SIGHUP` — accounts, permissions, everything in §4 — without dropping a connection; this +is the server's own documented capability, not something built for the mesh. So the composed +configuration file is mounted into a **directory** resource, not directly — a directory's contents +are not compared for change the way [issue 103](../../04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md)'s +fix made a directly-mounted file's content, so a rewritten file inside it is not, on its own, a +reason to recreate the container. The image's own entrypoint watches that one file and sends +`nats-server` its own process `SIGHUP` when it changes — self-contained, inside the module, the +same place `modules/gitea/token.ts` keeps its own state rather than asking the host to model it. +The host's only job is what it already does for any directory resource: keep the file's content +current. Nothing is declared as `reload-on` or `restart-on` for this resource at all. + +Its guard is the same rule as the AMQP broker's: the monitoring port is refused from anything but +the private network. It is raised at genesis like the store, adopted as a module in the same +phase. The predecessor's AMQP broker remains a module of its own, `lavinmq-compat`, with a single +purpose and a retirement condition: no client connected for a period the operator sets. ## 6. Joining: the enrolment handshake Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS -with the **enrolment user** — a user that may publish `mesh.control.enrol` and subscribe one reply -inbox and nothing else — publishes its request (the claim of the token, its keys, its proof, and -the found tunnel from [ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)), -and waits on the inbox. The controller spends the token, records the node, composes the node's -own user into the server's configuration, and answers with the credentials sealed to the node's -sealing key. The node reconnects as itself. The enrolment user's permissions are what make a -leaked token useless for anything but enrolling: it cannot read a declaration or hear an event. +with the **enrolment user** — a user that may publish `mesh.control.enrol` and subscribe its own +`_INBOX.enrol..>` and nothing else — publishes its request (the claim of the token, its +keys, its proof, its own reply subject as §2 now requires, and the found tunnel from +[ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)), and waits +on that inbox. The controller spends the token, records the node, composes the node's own user +into the server's configuration, and — reading the reply subject from the request's payload, never +from the transport `Reply` field the CONTROL consumer has already claimed for its own ack — +answers with the credentials sealed to the node's sealing key. The node reconnects as itself. The +enrolment user's permissions are what make a leaked token useless for anything but enrolling: it +cannot read a declaration or hear an event, and it cannot subscribe any inbox but the one its own +token derives. ## 7. A person's client @@ -199,7 +264,16 @@ Two lab beds, both required green before any node's bus moves. that can invoke it, and refused from one that cannot; - a module's account cannot publish outside its `emits` nor subscribe outside its `consumes` — refused by the server; +- a module acks a delivery from its own durable consumer, and is refused acking another module's; +- a user subscribes another module's or person's inbox prefix and is refused by the server, not + by the client's own good behaviour; - an event whose consumer keeps failing dead-letters after `max-deliver`; +- an enrolment request held by a `nak`-with-delay cycle still reaches the enrolling node's inbox + once the controller answers — proving the reply travels in the payload and not the transport + field a consumer's ack has already claimed; +- the `nats` container is not recreated when only its composed configuration file changes, and + a change to that file is live (a new user can connect, a revoked one cannot) within one + watcher-poll interval, without a restart; - a module built before this design serves its tools unchanged on the new runtime. **The cutover bed** — a mesh on AMQP with a predecessor stand-in on the compatibility broker moves @@ -213,6 +287,14 @@ to mapping the sdk contract onto subjects exactly as §8 says. ## 11. Open, for the review +**Closed by this revision** (first review, recorded in `MIGRATION-LOG.md`, 2026-09-24): the +`reload-on`/container mismatch (§5), the eaten reply subject on a CONTROL-stream message (§2, §6), +the missing ack permission (§4), and the un-scoped reply inbox under one account (§4). Each is +named where it was wrong, not silently fixed, so a reader comparing against the first version can +find what changed and why. + +**Still open:** + - Whether EVENTS should be one stream or one per emitting module (retention per module vs. one policy). One stream is proposed; the review may disagree. - The heartbeat interval and the controller's "quiet" threshold on core NATS without persistence — @@ -221,3 +303,10 @@ to mapping the sdk contract onto subjects exactly as §8 says. standalone program (runs anywhere with credentials). Both, in that order, is proposed. - Leaf nodes: a NATS leaf per machine would make every module's connection local and survive the hub's restart. Deliberately out of scope; noted so it is not forgotten. +- **New, from this revision:** the `nats` image's own entrypoint now carries logic (watch a file, + signal a process) that no other module's container needed before. Is a one-file-watcher-and- + `SIGHUP` helper common enough across future modules with the same shape (a service that reloads + on `SIGHUP` but runs in a container) to belong in `mesh-sdk` rather than written once per module + that needs it? [ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)'s test — + *does editing it recompile unrelated modules, and does it change often* — probably says no for + one instance; worth asking again if a second module needs the same shape.