Design 25: address first review's four findings before any code

Fixes, each named where it was wrong:

- reload-on is a service field; a container only has restart-on, which
  recreates. Cited precedent (registry-trust-reload) is a service resource,
  not a container. Fix: nats-server's own SIGHUP reload, triggered by an
  in-image entrypoint watching a directory-mounted config file (issue 103's
  recreate-on-change applies to a directly-mounted file, not a directory's
  contents) -- asks nothing new of the host.
- A JetStream-delivered message's Reply field is already claimed by the
  consumer's own ack address, so a responder using it answers nobody. Fix:
  every CONTROL message needing a reply carries its reply subject in its own
  payload; the controller publishes there explicitly, never via Respond().
  Enrolment is the case this design actually depends on, so it's fixed there
  too, not just noted.
- The listed permissions never granted publish on a durable consumer's own
  ack-reply subject -- a module could receive but never ack, so every
  message redelivers forever. Fixed with a scoped grant per module's own
  consumer.
- One account (a deliberate choice, kept) means inbox privacy is the
  permission list or nothing. The design granted 'its reply inbox' without
  scoping it, which read as any user reaching any inbox. Fixed: each user's
  inbox prefix is derived from its own identity and its permissions name
  only that prefix.

New open question from this revision, not closed: whether the in-image
watch-and-SIGHUP shape belongs in mesh-sdk if a second module ever needs it.

Checks pass (records.py, cycle.py, index.py).
This commit is contained in:
2026-09-24 13:53:53 +02:00
parent f3ad60b98c
commit 6abfec7433
+109 -20
View File
@@ -6,7 +6,7 @@ code:
- mesh-host internal/link (to be replaced) - mesh-host internal/link (to be replaced)
- mesh-tools src/broker-amqp.ts (to be replaced) - mesh-tools src/broker-amqp.ts (to be replaced)
- mesh-catalog modules/nats (to be written) - mesh-catalog modules/nats (to be written)
updated: 2026-09-23 updated: 2026-09-24
decisions: decisions:
- 02-DECISIONS/0106-the-bus-is-nats.md - 02-DECISIONS/0106-the-bus-is-nats.md
- 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md - 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
@@ -63,6 +63,21 @@ exactly the current declaration and nothing older. That is the wire-level answer
[issue 107](../../04-ISSUES/107-a-declaration-carries-no-order/00-report.md): the stream's sequence [issue 107](../../04-ISSUES/107-a-declaration-carries-no-order/00-report.md): the stream's sequence
*is* the order, and a node that sees sequence n refuses n−1 by construction. *is* the order, and a node that sees sequence n refuses n−1 by construction.
**A reply-to travelling through a JetStream stream is carried in the payload, never in the
transport `Reply` field.** Revision, first review: core NATS request/reply sets the requester's
ephemeral inbox as the message's `Reply` field, and a plain responder answers it directly — but a
message a JetStream consumer delivers has already had that field claimed for the consumer's own
ack address (`$JS.ACK.<stream>.<consumer>...`), so by the time the controller (§3's CONTROL
consumer) sees the message, `Reply` names where *it* must ack, not where the original caller is
waiting. `mesh.control.enrol` is the case that matters: a synchronous-feeling caller waiting on an
ephemeral inbox, over a subject the store-window guarantee may legitimately delay by several
`nak` cycles — exactly the combination that would otherwise deliver the answer to a caller who
has long since timed out and unsubscribed. So every CONTROL message that expects an answer states
its reply subject as an ordinary field of its own payload; the controller reads it from there and
publishes the answer to it explicitly, never via `Respond()`. Nothing else in this design routes
a reply through a stream — tools and heartbeats stay on core NATS, where `Reply` means what it has
always meant.
## 3. Streams, and the guarantees they carry ## 3. Streams, and the guarantees they carry
Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStream stream: Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStream stream:
@@ -89,9 +104,29 @@ expresses this exactly, per subject, and better than a vhost could:
- **One NATS account for the mesh.** Accounts in NATS isolate subject spaces entirely; the mesh - **One NATS account for the mesh.** Accounts in NATS isolate subject spaces entirely; the mesh
is one space, so it is one account. The predecessor's compatibility broker is not on this bus at is one space, so it is one account. The predecessor's compatibility broker is not on this bus at
all. all.
**One account is a choice with a cost, stated plainly on revision:** none of NATS's own
isolation is free here, because there is only the one subject space, for everyone. Two
consequences that a single-account mesh must therefore grant on purpose, not by omission:
- **A durable consumer needs permission to ack, or it never really consumes.** Acking a
JetStream delivery is a publish to that consumer's own ack-reply address
(`$JS.ACK.<stream>.<consumer>.>`), a different subject from anything the consumer subscribes.
A module's user is therefore granted publish on `$JS.ACK.EVENTS.<module>.>` as well as its
emits — scoped to the one consumer name the controller derives for that module, so a module
can ack only its own deliveries. Without this, first review found, every message it receives
would be redelivered forever: refused by the permission list it already has.
- **A reply inbox needs a subject nothing else can guess or enumerate.** With one account,
inbox privacy is the permission list or it is nothing — there is no second account backing
it up. So no user is ever granted a bare `_INBOX.>`. Each user's inbox subject is derived
from its own identity (`_INBOX.<module>.<node>.>`, or `_INBOX.person.<name>.>`), and its
permissions name only that one prefix, for the reply to any request it makes and nothing
wider. First review found the account note without this and read it as "any user may
subscribe any inbox" — which was accurate against the text as it stood.
- **One user per module per node**, as today, with publish permissions - **One user per module per node**, as today, with publish permissions
`mesh.events.<module>.<event>` for each emit, `mesh.tools.<module>.>` to serve its tools, and `mesh.events.<module>.<event>` for each emit, `mesh.tools.<module>.>` to serve its tools, its
its reply inbox; subscribe permissions for each consumed event's subject and its tool subjects. own ack-reply subject for each durable consumer it holds, and its own inbox prefix; subscribe
permissions for each consumed event's subject, its tool subjects, and that same inbox prefix.
Nothing else. A module that tries to publish outside its emits is refused by the server, not by Nothing else. A module that tries to publish outside its emits is refused by the server, not by
convention. convention.
- **The controller's user** owns `mesh.control.>`, `mesh.node.>`, `mesh.build.>` and the streams. - **The controller's user** owns `mesh.control.>`, `mesh.node.>`, `mesh.build.>` and the streams.
@@ -101,9 +136,10 @@ expresses this exactly, per subject, and better than a vhost could:
invoke, issued and revoked by the controller like any account. invoke, issued and revoked by the controller like any account.
**Accounts are configuration, not API calls.** The controller composes the server's user list and **Accounts are configuration, not API calls.** The controller composes the server's user list and
permissions into a file the host declares; the server reloads on change (`reload-on`, as the mesh permissions into a file the host declares. **How that file reaches the running server is §5's,
already does for the container runtime's trust). No management API, no credential travelling not this one's** — revision, first review: an earlier draft said "reloads" and cited a precedent
through a management call, and the [issue 102](../../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md) that does not apply to a container (see §5). No management API, no credential travelling through a
management call, and the [issue 102](../../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
discipline from the first day: an address or a permission is read where it is used, never stored discipline from the first day: an address or a permission is read where it is used, never stored
with a port. Passwords are minted and sealed exactly as today; the file holds bcrypt hashes. with a port. Passwords are minted and sealed exactly as today; the file holds bcrypt hashes.
@@ -117,24 +153,53 @@ signing hierarchy for nothing.
`nats` is a catalogue module claiming the seat `mesh-broker` `nats` is a catalogue module claiming the seat `mesh-broker`
([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md): the seat ([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md): the seat
is the server, and the server changes). It declares one container (a single binary; JetStream on a is the server, and the server changes). It declares one container (a single binary; JetStream on a
named volume), its listening ports — client, TLS, and the monitoring endpoint on loopback — a named volume), its listening ports — client, TLS, and the monitoring endpoint on loopback — and a
configuration file the controller composes (accounts, permissions, TLS, JetStream), and a configuration file the controller composes (accounts, permissions, TLS, JetStream).
`reload-on` for that file. Its guard is the same rule as the AMQP broker's: the monitoring port is
refused from anything but the private network. It is raised at genesis like the store, adopted as a **How that file's changes reach the running server, corrected on revision.** First review: the
module in the same phase. The predecessor's AMQP broker remains a module of its own, earlier draft named `reload-on` as the mechanism, citing the container runtime's own trust file as
`lavinmq-compat`, with a single purpose and a retirement condition: no client connected for a precedent. `reload-on` is real, but it is a **service** field
period the operator sets. ([mesh-host declaration.go](https://git.novox.be/novox/mesh-host), `Service.ReloadOn` —
`docker.service` is reloaded via systemd, which is what the cited precedent actually does). A
**container** resource has no reload field at all — only `restart-on`, and a container's
`restart-on` is documented, exactly, to mean *recreate*. Declared as the earlier draft had it,
either the field is silently meaningless on a container resource or — if read as the nearest real
equivalent — every account, permission, or key change recreates the bus's own server: every
connection dropped, every in-flight JetStream ack lost, mid-flight the moment a module is added,
reassigned, or a person's access changes. For the one resource everything else depends on, that
is not an edge case; it is the common case.
**The fix asks nothing new of the host.** `nats-server` already reloads its own configuration
live on `SIGHUP` — accounts, permissions, everything in §4 — without dropping a connection; this
is the server's own documented capability, not something built for the mesh. So the composed
configuration file is mounted into a **directory** resource, not directly — a directory's contents
are not compared for change the way [issue 103](../../04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md)'s
fix made a directly-mounted file's content, so a rewritten file inside it is not, on its own, a
reason to recreate the container. The image's own entrypoint watches that one file and sends
`nats-server` its own process `SIGHUP` when it changes — self-contained, inside the module, the
same place `modules/gitea/token.ts` keeps its own state rather than asking the host to model it.
The host's only job is what it already does for any directory resource: keep the file's content
current. Nothing is declared as `reload-on` or `restart-on` for this resource at all.
Its guard is the same rule as the AMQP broker's: the monitoring port is refused from anything but
the private network. It is raised at genesis like the store, adopted as a module in the same
phase. The predecessor's AMQP broker remains a module of its own, `lavinmq-compat`, with a single
purpose and a retirement condition: no client connected for a period the operator sets.
## 6. Joining: the enrolment handshake ## 6. Joining: the enrolment handshake
Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS
with the **enrolment user** — a user that may publish `mesh.control.enrol` and subscribe one reply with the **enrolment user** — a user that may publish `mesh.control.enrol` and subscribe its own
inbox and nothing else — publishes its request (the claim of the token, its keys, its proof, and `_INBOX.enrol.<token-id>.>` and nothing else — publishes its request (the claim of the token, its
the found tunnel from [ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)), keys, its proof, its own reply subject as §2 now requires, and the found tunnel from
and waits on the inbox. The controller spends the token, records the node, composes the node's [ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)), and waits
own user into the server's configuration, and answers with the credentials sealed to the node's on that inbox. The controller spends the token, records the node, composes the node's own user
sealing key. The node reconnects as itself. The enrolment user's permissions are what make a into the server's configuration, and — reading the reply subject from the request's payload, never
leaked token useless for anything but enrolling: it cannot read a declaration or hear an event. from the transport `Reply` field the CONTROL consumer has already claimed for its own ack —
answers with the credentials sealed to the node's sealing key. The node reconnects as itself. The
enrolment user's permissions are what make a leaked token useless for anything but enrolling: it
cannot read a declaration or hear an event, and it cannot subscribe any inbox but the one its own
token derives.
## 7. A person's client ## 7. A person's client
@@ -199,7 +264,16 @@ Two lab beds, both required green before any node's bus moves.
that can invoke it, and refused from one that cannot; that can invoke it, and refused from one that cannot;
- a module's account cannot publish outside its `emits` nor subscribe outside its `consumes` — - a module's account cannot publish outside its `emits` nor subscribe outside its `consumes` —
refused by the server; refused by the server;
- a module acks a delivery from its own durable consumer, and is refused acking another module's;
- a user subscribes another module's or person's inbox prefix and is refused by the server, not
by the client's own good behaviour;
- an event whose consumer keeps failing dead-letters after `max-deliver`; - an event whose consumer keeps failing dead-letters after `max-deliver`;
- an enrolment request held by a `nak`-with-delay cycle still reaches the enrolling node's inbox
once the controller answers — proving the reply travels in the payload and not the transport
field a consumer's ack has already claimed;
- the `nats` container is not recreated when only its composed configuration file changes, and
a change to that file is live (a new user can connect, a revoked one cannot) within one
watcher-poll interval, without a restart;
- a module built before this design serves its tools unchanged on the new runtime. - a module built before this design serves its tools unchanged on the new runtime.
**The cutover bed** — a mesh on AMQP with a predecessor stand-in on the compatibility broker moves **The cutover bed** — a mesh on AMQP with a predecessor stand-in on the compatibility broker moves
@@ -213,6 +287,14 @@ to mapping the sdk contract onto subjects exactly as §8 says.
## 11. Open, for the review ## 11. Open, for the review
**Closed by this revision** (first review, recorded in `MIGRATION-LOG.md`, 2026-09-24): the
`reload-on`/container mismatch (§5), the eaten reply subject on a CONTROL-stream message (§2, §6),
the missing ack permission (§4), and the un-scoped reply inbox under one account (§4). Each is
named where it was wrong, not silently fixed, so a reader comparing against the first version can
find what changed and why.
**Still open:**
- Whether EVENTS should be one stream or one per emitting module (retention per module vs. one - Whether EVENTS should be one stream or one per emitting module (retention per module vs. one
policy). One stream is proposed; the review may disagree. policy). One stream is proposed; the review may disagree.
- The heartbeat interval and the controller's "quiet" threshold on core NATS without persistence — - The heartbeat interval and the controller's "quiet" threshold on core NATS without persistence —
@@ -221,3 +303,10 @@ to mapping the sdk contract onto subjects exactly as §8 says.
standalone program (runs anywhere with credentials). Both, in that order, is proposed. standalone program (runs anywhere with credentials). Both, in that order, is proposed.
- Leaf nodes: a NATS leaf per machine would make every module's connection local and survive the - Leaf nodes: a NATS leaf per machine would make every module's connection local and survive the
hub's restart. Deliberately out of scope; noted so it is not forgotten. hub's restart. Deliberately out of scope; noted so it is not forgotten.
- **New, from this revision:** the `nats` image's own entrypoint now carries logic (watch a file,
signal a process) that no other module's container needed before. Is a one-file-watcher-and-
`SIGHUP` helper common enough across future modules with the same shape (a service that reloads
on `SIGHUP` but runs in a container) to belong in `mesh-sdk` rather than written once per module
that needs it? [ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)'s test —
*does editing it recompile unrelated modules, and does it change often* — probably says no for
one instance; worth asking again if a second module needs the same shape.