153 lines
11 KiB
Markdown
153 lines
11 KiB
Markdown
---
|
|
status: graduated
|
|
initiated: 2026-10-04
|
|
touches: [the bus, what a module declares, the tool runtime, the SDK, the bus grants, 03-DESIGN/01-to-be/25-the-bus-on-nats.md, 03-DESIGN/01-to-be/32-what-a-module-declares.md]
|
|
became: [02-DECISIONS/0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md, 03-DESIGN/01-to-be/32-what-a-module-declares.md, 03-DESIGN/01-to-be/25-the-bus-on-nats.md]
|
|
---
|
|
|
|
# 024 — State a module keeps on the bus
|
|
|
|
## What is investigated
|
|
|
|
A place on the bus where a module's own code keeps **current state** — not history — that every
|
|
machine sees, including a machine that joins after the state was written: put, get, delete, list and
|
|
watch, reached through the node's runtime the way a bundle already publishes, asks and subscribes
|
|
([ADR 0198](../../02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)).
|
|
On NATS that is a key-value bucket. The questions are what a module declares, who creates the
|
|
bucket, what the grants are, what the runtime's verbs are, and what may never be stored.
|
|
|
|
## Why
|
|
|
|
The mesh carries two kinds of module traffic and a third is missing.
|
|
|
|
- **Events** land in the EVENTS stream: limits retention, seven days, ten thousand messages per
|
|
subject, a durable consumer per consuming module that replays what it missed. Never a secret
|
|
([design 32](../../03-DESIGN/01-to-be/32-what-a-module-declares.md) §10).
|
|
- **Requests** are core request/reply — tool calls, a bundle's `mesh/ask` — and are kept nowhere.
|
|
|
|
Neither is *the current value of something*. Two cases from the first module that needs it, the
|
|
operator's agent on a machine ([design 36](../../03-DESIGN/01-to-be/36-the-operators-agent-on-a-machine.md)):
|
|
|
|
1. **An MCP server registered for every machine.** Registering emits an event every machine's copy
|
|
of the module consumes. A machine the module is assigned to *after* the registration has no
|
|
durable consumer yet — the consumer is created at assignment — so it never hears of it. Wanted
|
|
instead: one entry per server, for every machine or for one; every machine reads the whole current
|
|
set when it starts and watches for changes; unregistering is a delete; any machine can list it.
|
|
2. **Which licence a machine is bound to** ([design 39](../../03-DESIGN/01-to-be/39-the-anthropic-licence-manager.md)).
|
|
As events, a machine that was off for a day replays every rotation since and asks for a token
|
|
after each. It needs only the latest binding and its generation. The token itself stays on
|
|
request/reply and is never stored.
|
|
|
|
The design already expects this. [Design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1:
|
|
"conditions and observed state in key-value buckets that anything may watch".
|
|
[Research 017](../017-a-mesh-that-heals-itself/01-the-intended-behaviour.md) wants a provisioner's
|
|
"what I applied" and a rotation's step kept in one rather than in memory. Nothing implements it.
|
|
|
|
## What exists, measured 2026-10-04
|
|
|
|
| | fact | where |
|
|
|---|---|---|
|
|
| streams | five kinds of mesh stream: CONTROL (work queue), NODES and ASSIGNMENTS (last per subject), EVENTS (limits: 7 days, 10 000 per subject), one work queue per seat that accepts | the controller's broker streams |
|
|
| the state relationship | [design 32](../../03-DESIGN/01-to-be/32-what-a-module-declares.md) §4 already names *state* — 1:1, last per subject — and says it is "declared: the mesh's own". Two streams use it, both written by the controller. No module can declare it | design 32, the controller |
|
|
| key-value buckets | none, anywhere | all four code repositories |
|
|
| the runtime's bus verbs | `mesh/publish`, `mesh/ask`, `mesh/subscribe`; delivery back to the bundle is `mesh/event` | the runtime's launcher |
|
|
| the runtime's principal | one bus user per machine carries every assigned module; its grant is the union of theirs. That one module's code does not act as another is the runtime's to keep: it publishes under the module's own name by construction | the controller's grant composition, the runtime's bus |
|
|
| what a bundle is issued | a membership per assignment, last per subject, read directly by the runtime: where it serves, where it emits, what it reaches | ADR 0160 |
|
|
| who creates bus objects | the controller only — mesh streams on every raise, a seat's stream at registration, a module's consumer at assignment. No module reaches the JetStream API | design 25 §3 |
|
|
|
|
### What a key-value bucket needs from a grant, against a real server
|
|
|
|
Measured against nats-server 2.10 with the Go client the runtime already uses, a bucket created by
|
|
an unrestricted user and used by two users holding only the subjects below (`B` is the bucket):
|
|
|
|
| operation | subject published | writer | reader |
|
|
|---|---|---|---|
|
|
| bind to the bucket | `$JS.API.STREAM.INFO.KV_B` | yes | yes |
|
|
| get | `$JS.API.DIRECT.GET.KV_B.>` | yes | yes |
|
|
| put, delete | `$KV.B.>` | yes | **refused** |
|
|
| list keys, watch | `$JS.API.CONSUMER.CREATE.KV_B.>` — an ordered, ephemeral consumer | yes | yes |
|
|
| stop a watch cleanly | `$JS.API.CONSUMER.DELETE.KV_B.>` | yes | yes |
|
|
| answers | its own inbox, which every principal already subscribes | — | — |
|
|
|
|
*Checked again once built, 2026-10-04:* the grants the controller composes for two machines' runtimes —
|
|
one carrying the owner, one only a reader — were loaded into a server as composed, and each operation
|
|
was run as each runtime's user. The owner's did all of them; the reader's read, listed and watched,
|
|
and its put and delete were refused by the server.
|
|
|
|
Three things the measurement showed that reading the documentation would not have:
|
|
|
|
1. **A refused put is not an error to the caller; it is a timeout.** The server reports the
|
|
permission violation asynchronously, on the connection, and the client waits out its deadline
|
|
for an acknowledgement that never comes. So a runtime that relies on the grant alone tells a
|
|
bundle "timed out" for "you may not write this" — it must refuse first, from what the module was
|
|
issued, with the reason.
|
|
2. **A watch's current values include deletions.** A key deleted earlier arrives among the initial
|
|
values as a delete marker, before the end-of-current marker. A bundle asking "what is there now"
|
|
must not be handed those.
|
|
3. **Without the consumer-delete grant, stopping a watch hangs** until its deadline, and the
|
|
ephemeral consumer lingers on the server until it times out by itself.
|
|
|
|
### Whether the events shape is enough instead
|
|
|
|
Honestly compared, because a new primitive is a cost:
|
|
|
|
- **EVENTS cannot be made last-per-subject for some subjects.** Retention is per stream, and
|
|
JetStream refuses a second stream overlapping the first (verified and recorded in design 32 §3).
|
|
A state subject inside `mesh.mod.*.event.>` keeps EVENTS' seven days: a licence binding unchanged
|
|
for a week disappears.
|
|
- **A separate last-per-subject stream per module** is possible — it is exactly what a key-value
|
|
bucket *is* on the server: a stream with one message per subject, a rollup for purge, and direct
|
|
reads. Building it by hand gives up the client's get, list, delete and watch, which are the
|
|
operations both cases need, and would be the mesh writing NATS's own key-value layer again.
|
|
- **Consumers are the wrong reader.** A durable consumer per reading module is created at
|
|
assignment and replays from where it is; state wants "everything current, now, then changes",
|
|
which an ordered ephemeral consumer from the last value per subject gives and a durable does not.
|
|
|
|
So key-value is not a convenience over events; it is the state relationship design 32 already
|
|
names, opened to modules.
|
|
|
|
## Questions, and what this effort proposes
|
|
|
|
1. **What a manifest says.** `state` names the buckets a module owns, by local name — every
|
|
instance of the module may write them and read them. `reads` names another module's bucket as
|
|
`<module>.<name>`, read-only. Names only, never a bucket or subject (design 32 §1). A bucket's
|
|
options — how many past values it keeps, how long a value lives — are the owner's to declare,
|
|
the way a seat declares its own retention (design 32 §3).
|
|
2. **Scope.** One bucket per module per name, mesh-wide. A key may carry a machine by the module's
|
|
own convention (`all.<server>`, `<machine>.<server>`). A bucket per machine was considered and
|
|
not proposed: "list every server for every machine" becomes a walk over buckets, and the grant
|
|
could only narrow writes, which nothing asked for — every instance of the owner already writes.
|
|
3. **Who creates the bucket.** The controller, from the catalogue, on every raise — a bucket exists
|
|
from registration, like a seat's stream, so a reader can watch before the owner is assigned
|
|
anywhere. Never a module.
|
|
4. **The runtime's verbs.** `mesh/state.get`, `mesh/state.put`, `mesh/state.delete`,
|
|
`mesh/state.keys`, `mesh/state.watch`, each naming the bucket as the module named it. A watch
|
|
is answered once the current values are on their way, then each change is delivered to the
|
|
bundle as a `mesh/state` request it answers — current values first (no deletions among them), an
|
|
end-of-current marker, then changes. A child that restarts watches again, as it subscribes
|
|
again. The runtime refuses, with the reason, a bucket the module was not issued, and a write to
|
|
one it only reads.
|
|
5. **Secrets.** None in a bucket, sealed or not: a bucket is a stream (design 32 §10). Sealed values
|
|
are plain base64 and cannot be recognised, so the mechanical check is partial and said to be: the
|
|
runtime refuses a value carrying a field whose name says it is a credential (`password`,
|
|
`secret`, `token`, `authorization`, …), which catches the ordinary mistake and not a determined
|
|
one. For the first consumer this has a concrete consequence: an MCP server registered with an
|
|
authorisation header keeps that header out of the bucket.
|
|
6. **History, lifetime, size.** One value per key unless the owner says more; no expiry unless it
|
|
says one; a value at most 256 KiB and a bucket at most 64 MiB, the mesh's caps rather than a
|
|
module's. **A bucket outlives its module's assignment** — what a module stored is data, and data
|
|
outlives what declared it ([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md));
|
|
unassigning is not cleaning up. A bucket whose declaration is gone is reported, never removed.
|
|
7. **Events or state.** State (above).
|
|
|
|
## The work, once decided
|
|
|
|
1. A decision record, then design 32 (*state* becomes a relationship a module declares) and design
|
|
25 (key-value buckets are part of the bus) amended.
|
|
2. The controller: the manifest's two words and their registration check; buckets asserted on every
|
|
raise; the grants for owners' and readers' runtimes; the buckets issued in each membership.
|
|
3. The runtime: the five verbs, the watch delivery, the refusals; tested against a real server.
|
|
4. The SDK, TypeScript and Go: a small state surface over the verbs.
|
|
5. Proved on a running mesh with one small module, then handed to the operator's agent, whose
|
|
registered servers move from events to a bucket.
|