Design 25: address first review's four findings before any code
Fixes, each named where it was wrong: - reload-on is a service field; a container only has restart-on, which recreates. Cited precedent (registry-trust-reload) is a service resource, not a container. Fix: nats-server's own SIGHUP reload, triggered by an in-image entrypoint watching a directory-mounted config file (issue 103's recreate-on-change applies to a directly-mounted file, not a directory's contents) -- asks nothing new of the host. - A JetStream-delivered message's Reply field is already claimed by the consumer's own ack address, so a responder using it answers nobody. Fix: every CONTROL message needing a reply carries its reply subject in its own payload; the controller publishes there explicitly, never via Respond(). Enrolment is the case this design actually depends on, so it's fixed there too, not just noted. - The listed permissions never granted publish on a durable consumer's own ack-reply subject -- a module could receive but never ack, so every message redelivers forever. Fixed with a scoped grant per module's own consumer. - One account (a deliberate choice, kept) means inbox privacy is the permission list or nothing. The design granted 'its reply inbox' without scoping it, which read as any user reaching any inbox. Fixed: each user's inbox prefix is derived from its own identity and its permissions name only that prefix. New open question from this revision, not closed: whether the in-image watch-and-SIGHUP shape belongs in mesh-sdk if a second module ever needs it. Checks pass (records.py, cycle.py, index.py).
This commit is contained in:
@@ -6,7 +6,7 @@ code:
|
|||||||
- mesh-host internal/link (to be replaced)
|
- mesh-host internal/link (to be replaced)
|
||||||
- mesh-tools src/broker-amqp.ts (to be replaced)
|
- mesh-tools src/broker-amqp.ts (to be replaced)
|
||||||
- mesh-catalog modules/nats (to be written)
|
- mesh-catalog modules/nats (to be written)
|
||||||
updated: 2026-09-23
|
updated: 2026-09-24
|
||||||
decisions:
|
decisions:
|
||||||
- 02-DECISIONS/0106-the-bus-is-nats.md
|
- 02-DECISIONS/0106-the-bus-is-nats.md
|
||||||
- 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
|
- 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
|
||||||
@@ -63,6 +63,21 @@ exactly the current declaration and nothing older. That is the wire-level answer
|
|||||||
[issue 107](../../04-ISSUES/107-a-declaration-carries-no-order/00-report.md): the stream's sequence
|
[issue 107](../../04-ISSUES/107-a-declaration-carries-no-order/00-report.md): the stream's sequence
|
||||||
*is* the order, and a node that sees sequence n refuses n−1 by construction.
|
*is* the order, and a node that sees sequence n refuses n−1 by construction.
|
||||||
|
|
||||||
|
**A reply-to travelling through a JetStream stream is carried in the payload, never in the
|
||||||
|
transport `Reply` field.** Revision, first review: core NATS request/reply sets the requester's
|
||||||
|
ephemeral inbox as the message's `Reply` field, and a plain responder answers it directly — but a
|
||||||
|
message a JetStream consumer delivers has already had that field claimed for the consumer's own
|
||||||
|
ack address (`$JS.ACK.<stream>.<consumer>...`), so by the time the controller (§3's CONTROL
|
||||||
|
consumer) sees the message, `Reply` names where *it* must ack, not where the original caller is
|
||||||
|
waiting. `mesh.control.enrol` is the case that matters: a synchronous-feeling caller waiting on an
|
||||||
|
ephemeral inbox, over a subject the store-window guarantee may legitimately delay by several
|
||||||
|
`nak` cycles — exactly the combination that would otherwise deliver the answer to a caller who
|
||||||
|
has long since timed out and unsubscribed. So every CONTROL message that expects an answer states
|
||||||
|
its reply subject as an ordinary field of its own payload; the controller reads it from there and
|
||||||
|
publishes the answer to it explicitly, never via `Respond()`. Nothing else in this design routes
|
||||||
|
a reply through a stream — tools and heartbeats stay on core NATS, where `Reply` means what it has
|
||||||
|
always meant.
|
||||||
|
|
||||||
## 3. Streams, and the guarantees they carry
|
## 3. Streams, and the guarantees they carry
|
||||||
|
|
||||||
Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStream stream:
|
Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStream stream:
|
||||||
@@ -89,9 +104,29 @@ expresses this exactly, per subject, and better than a vhost could:
|
|||||||
- **One NATS account for the mesh.** Accounts in NATS isolate subject spaces entirely; the mesh
|
- **One NATS account for the mesh.** Accounts in NATS isolate subject spaces entirely; the mesh
|
||||||
is one space, so it is one account. The predecessor's compatibility broker is not on this bus at
|
is one space, so it is one account. The predecessor's compatibility broker is not on this bus at
|
||||||
all.
|
all.
|
||||||
|
|
||||||
|
**One account is a choice with a cost, stated plainly on revision:** none of NATS's own
|
||||||
|
isolation is free here, because there is only the one subject space, for everyone. Two
|
||||||
|
consequences that a single-account mesh must therefore grant on purpose, not by omission:
|
||||||
|
|
||||||
|
- **A durable consumer needs permission to ack, or it never really consumes.** Acking a
|
||||||
|
JetStream delivery is a publish to that consumer's own ack-reply address
|
||||||
|
(`$JS.ACK.<stream>.<consumer>.>`), a different subject from anything the consumer subscribes.
|
||||||
|
A module's user is therefore granted publish on `$JS.ACK.EVENTS.<module>.>` as well as its
|
||||||
|
emits — scoped to the one consumer name the controller derives for that module, so a module
|
||||||
|
can ack only its own deliveries. Without this, first review found, every message it receives
|
||||||
|
would be redelivered forever: refused by the permission list it already has.
|
||||||
|
- **A reply inbox needs a subject nothing else can guess or enumerate.** With one account,
|
||||||
|
inbox privacy is the permission list or it is nothing — there is no second account backing
|
||||||
|
it up. So no user is ever granted a bare `_INBOX.>`. Each user's inbox subject is derived
|
||||||
|
from its own identity (`_INBOX.<module>.<node>.>`, or `_INBOX.person.<name>.>`), and its
|
||||||
|
permissions name only that one prefix, for the reply to any request it makes and nothing
|
||||||
|
wider. First review found the account note without this and read it as "any user may
|
||||||
|
subscribe any inbox" — which was accurate against the text as it stood.
|
||||||
- **One user per module per node**, as today, with publish permissions
|
- **One user per module per node**, as today, with publish permissions
|
||||||
`mesh.events.<module>.<event>` for each emit, `mesh.tools.<module>.>` to serve its tools, and
|
`mesh.events.<module>.<event>` for each emit, `mesh.tools.<module>.>` to serve its tools, its
|
||||||
its reply inbox; subscribe permissions for each consumed event's subject and its tool subjects.
|
own ack-reply subject for each durable consumer it holds, and its own inbox prefix; subscribe
|
||||||
|
permissions for each consumed event's subject, its tool subjects, and that same inbox prefix.
|
||||||
Nothing else. A module that tries to publish outside its emits is refused by the server, not by
|
Nothing else. A module that tries to publish outside its emits is refused by the server, not by
|
||||||
convention.
|
convention.
|
||||||
- **The controller's user** owns `mesh.control.>`, `mesh.node.>`, `mesh.build.>` and the streams.
|
- **The controller's user** owns `mesh.control.>`, `mesh.node.>`, `mesh.build.>` and the streams.
|
||||||
@@ -101,9 +136,10 @@ expresses this exactly, per subject, and better than a vhost could:
|
|||||||
invoke, issued and revoked by the controller like any account.
|
invoke, issued and revoked by the controller like any account.
|
||||||
|
|
||||||
**Accounts are configuration, not API calls.** The controller composes the server's user list and
|
**Accounts are configuration, not API calls.** The controller composes the server's user list and
|
||||||
permissions into a file the host declares; the server reloads on change (`reload-on`, as the mesh
|
permissions into a file the host declares. **How that file reaches the running server is §5's,
|
||||||
already does for the container runtime's trust). No management API, no credential travelling
|
not this one's** — revision, first review: an earlier draft said "reloads" and cited a precedent
|
||||||
through a management call, and the [issue 102](../../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
|
that does not apply to a container (see §5). No management API, no credential travelling through a
|
||||||
|
management call, and the [issue 102](../../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
|
||||||
discipline from the first day: an address or a permission is read where it is used, never stored
|
discipline from the first day: an address or a permission is read where it is used, never stored
|
||||||
with a port. Passwords are minted and sealed exactly as today; the file holds bcrypt hashes.
|
with a port. Passwords are minted and sealed exactly as today; the file holds bcrypt hashes.
|
||||||
|
|
||||||
@@ -117,24 +153,53 @@ signing hierarchy for nothing.
|
|||||||
`nats` is a catalogue module claiming the seat `mesh-broker`
|
`nats` is a catalogue module claiming the seat `mesh-broker`
|
||||||
([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md): the seat
|
([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md): the seat
|
||||||
is the server, and the server changes). It declares one container (a single binary; JetStream on a
|
is the server, and the server changes). It declares one container (a single binary; JetStream on a
|
||||||
named volume), its listening ports — client, TLS, and the monitoring endpoint on loopback — a
|
named volume), its listening ports — client, TLS, and the monitoring endpoint on loopback — and a
|
||||||
configuration file the controller composes (accounts, permissions, TLS, JetStream), and a
|
configuration file the controller composes (accounts, permissions, TLS, JetStream).
|
||||||
`reload-on` for that file. Its guard is the same rule as the AMQP broker's: the monitoring port is
|
|
||||||
refused from anything but the private network. It is raised at genesis like the store, adopted as a
|
**How that file's changes reach the running server, corrected on revision.** First review: the
|
||||||
module in the same phase. The predecessor's AMQP broker remains a module of its own,
|
earlier draft named `reload-on` as the mechanism, citing the container runtime's own trust file as
|
||||||
`lavinmq-compat`, with a single purpose and a retirement condition: no client connected for a
|
precedent. `reload-on` is real, but it is a **service** field
|
||||||
period the operator sets.
|
([mesh-host declaration.go](https://git.novox.be/novox/mesh-host), `Service.ReloadOn` —
|
||||||
|
`docker.service` is reloaded via systemd, which is what the cited precedent actually does). A
|
||||||
|
**container** resource has no reload field at all — only `restart-on`, and a container's
|
||||||
|
`restart-on` is documented, exactly, to mean *recreate*. Declared as the earlier draft had it,
|
||||||
|
either the field is silently meaningless on a container resource or — if read as the nearest real
|
||||||
|
equivalent — every account, permission, or key change recreates the bus's own server: every
|
||||||
|
connection dropped, every in-flight JetStream ack lost, mid-flight the moment a module is added,
|
||||||
|
reassigned, or a person's access changes. For the one resource everything else depends on, that
|
||||||
|
is not an edge case; it is the common case.
|
||||||
|
|
||||||
|
**The fix asks nothing new of the host.** `nats-server` already reloads its own configuration
|
||||||
|
live on `SIGHUP` — accounts, permissions, everything in §4 — without dropping a connection; this
|
||||||
|
is the server's own documented capability, not something built for the mesh. So the composed
|
||||||
|
configuration file is mounted into a **directory** resource, not directly — a directory's contents
|
||||||
|
are not compared for change the way [issue 103](../../04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md)'s
|
||||||
|
fix made a directly-mounted file's content, so a rewritten file inside it is not, on its own, a
|
||||||
|
reason to recreate the container. The image's own entrypoint watches that one file and sends
|
||||||
|
`nats-server` its own process `SIGHUP` when it changes — self-contained, inside the module, the
|
||||||
|
same place `modules/gitea/token.ts` keeps its own state rather than asking the host to model it.
|
||||||
|
The host's only job is what it already does for any directory resource: keep the file's content
|
||||||
|
current. Nothing is declared as `reload-on` or `restart-on` for this resource at all.
|
||||||
|
|
||||||
|
Its guard is the same rule as the AMQP broker's: the monitoring port is refused from anything but
|
||||||
|
the private network. It is raised at genesis like the store, adopted as a module in the same
|
||||||
|
phase. The predecessor's AMQP broker remains a module of its own, `lavinmq-compat`, with a single
|
||||||
|
purpose and a retirement condition: no client connected for a period the operator sets.
|
||||||
|
|
||||||
## 6. Joining: the enrolment handshake
|
## 6. Joining: the enrolment handshake
|
||||||
|
|
||||||
Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS
|
Unchanged in shape, changed in transport. A node that has a token connects to the bus over TLS
|
||||||
with the **enrolment user** — a user that may publish `mesh.control.enrol` and subscribe one reply
|
with the **enrolment user** — a user that may publish `mesh.control.enrol` and subscribe its own
|
||||||
inbox and nothing else — publishes its request (the claim of the token, its keys, its proof, and
|
`_INBOX.enrol.<token-id>.>` and nothing else — publishes its request (the claim of the token, its
|
||||||
the found tunnel from [ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)),
|
keys, its proof, its own reply subject as §2 now requires, and the found tunnel from
|
||||||
and waits on the inbox. The controller spends the token, records the node, composes the node's
|
[ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)), and waits
|
||||||
own user into the server's configuration, and answers with the credentials sealed to the node's
|
on that inbox. The controller spends the token, records the node, composes the node's own user
|
||||||
sealing key. The node reconnects as itself. The enrolment user's permissions are what make a
|
into the server's configuration, and — reading the reply subject from the request's payload, never
|
||||||
leaked token useless for anything but enrolling: it cannot read a declaration or hear an event.
|
from the transport `Reply` field the CONTROL consumer has already claimed for its own ack —
|
||||||
|
answers with the credentials sealed to the node's sealing key. The node reconnects as itself. The
|
||||||
|
enrolment user's permissions are what make a leaked token useless for anything but enrolling: it
|
||||||
|
cannot read a declaration or hear an event, and it cannot subscribe any inbox but the one its own
|
||||||
|
token derives.
|
||||||
|
|
||||||
## 7. A person's client
|
## 7. A person's client
|
||||||
|
|
||||||
@@ -199,7 +264,16 @@ Two lab beds, both required green before any node's bus moves.
|
|||||||
that can invoke it, and refused from one that cannot;
|
that can invoke it, and refused from one that cannot;
|
||||||
- a module's account cannot publish outside its `emits` nor subscribe outside its `consumes` —
|
- a module's account cannot publish outside its `emits` nor subscribe outside its `consumes` —
|
||||||
refused by the server;
|
refused by the server;
|
||||||
|
- a module acks a delivery from its own durable consumer, and is refused acking another module's;
|
||||||
|
- a user subscribes another module's or person's inbox prefix and is refused by the server, not
|
||||||
|
by the client's own good behaviour;
|
||||||
- an event whose consumer keeps failing dead-letters after `max-deliver`;
|
- an event whose consumer keeps failing dead-letters after `max-deliver`;
|
||||||
|
- an enrolment request held by a `nak`-with-delay cycle still reaches the enrolling node's inbox
|
||||||
|
once the controller answers — proving the reply travels in the payload and not the transport
|
||||||
|
field a consumer's ack has already claimed;
|
||||||
|
- the `nats` container is not recreated when only its composed configuration file changes, and
|
||||||
|
a change to that file is live (a new user can connect, a revoked one cannot) within one
|
||||||
|
watcher-poll interval, without a restart;
|
||||||
- a module built before this design serves its tools unchanged on the new runtime.
|
- a module built before this design serves its tools unchanged on the new runtime.
|
||||||
|
|
||||||
**The cutover bed** — a mesh on AMQP with a predecessor stand-in on the compatibility broker moves
|
**The cutover bed** — a mesh on AMQP with a predecessor stand-in on the compatibility broker moves
|
||||||
@@ -213,6 +287,14 @@ to mapping the sdk contract onto subjects exactly as §8 says.
|
|||||||
|
|
||||||
## 11. Open, for the review
|
## 11. Open, for the review
|
||||||
|
|
||||||
|
**Closed by this revision** (first review, recorded in `MIGRATION-LOG.md`, 2026-09-24): the
|
||||||
|
`reload-on`/container mismatch (§5), the eaten reply subject on a CONTROL-stream message (§2, §6),
|
||||||
|
the missing ack permission (§4), and the un-scoped reply inbox under one account (§4). Each is
|
||||||
|
named where it was wrong, not silently fixed, so a reader comparing against the first version can
|
||||||
|
find what changed and why.
|
||||||
|
|
||||||
|
**Still open:**
|
||||||
|
|
||||||
- Whether EVENTS should be one stream or one per emitting module (retention per module vs. one
|
- Whether EVENTS should be one stream or one per emitting module (retention per module vs. one
|
||||||
policy). One stream is proposed; the review may disagree.
|
policy). One stream is proposed; the review may disagree.
|
||||||
- The heartbeat interval and the controller's "quiet" threshold on core NATS without persistence —
|
- The heartbeat interval and the controller's "quiet" threshold on core NATS without persistence —
|
||||||
@@ -221,3 +303,10 @@ to mapping the sdk contract onto subjects exactly as §8 says.
|
|||||||
standalone program (runs anywhere with credentials). Both, in that order, is proposed.
|
standalone program (runs anywhere with credentials). Both, in that order, is proposed.
|
||||||
- Leaf nodes: a NATS leaf per machine would make every module's connection local and survive the
|
- Leaf nodes: a NATS leaf per machine would make every module's connection local and survive the
|
||||||
hub's restart. Deliberately out of scope; noted so it is not forgotten.
|
hub's restart. Deliberately out of scope; noted so it is not forgotten.
|
||||||
|
- **New, from this revision:** the `nats` image's own entrypoint now carries logic (watch a file,
|
||||||
|
signal a process) that no other module's container needed before. Is a one-file-watcher-and-
|
||||||
|
`SIGHUP` helper common enough across future modules with the same shape (a service that reloads
|
||||||
|
on `SIGHUP` but runs in a container) to belong in `mesh-sdk` rather than written once per module
|
||||||
|
that needs it? [ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)'s test —
|
||||||
|
*does editing it recompile unrelated modules, and does it change often* — probably says no for
|
||||||
|
one instance; worth asking again if a second module needs the same shape.
|
||||||
|
|||||||
Reference in New Issue
Block a user