novox/hq
6
0
Fork 0
Code Issues 5 Pull Requests 3 Releases Wiki Activity

Research 024 and ADR 0201: a module keeps its current state in key-value buckets

Browse Source 
Events miss a machine that joins after them and replay history where only the
latest matters. A module now declares state it owns and reads; the controller
creates the buckets, the runtime serves them on the bundle's channel. Designs 32
and 25 amended; grants measured against a running server.
This commit is contained in:
jochen
2026-10-04 02:36:59 +02:00
parent 8ca09c70d6
commit e1b0bbde91
5 changed files with 322 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
+106
View File
@@ -0,0 +1,106 @@
---
topic: what runs on it
status: accepted
date: 2026-10-04
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md
---
# 201. A module keeps its current state in key-value buckets it declares, and reaches them through the runtime
## Context
A module's code reaches the bus through the node's runtime: it publishes events, subscribes to them
and asks tools ([ADR 0198](0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)).
Events are kept for a week and replayed to a consumer that was away; requests are kept nowhere. What
neither gives is **the current value of something**, seen by every machine, including one that joins
after it was written. The first module to need it — the operator's agent on a machine — registers MCP
servers for every machine as events, and a machine assigned later never hears of them; and it would
replay a week of licence rotations where it needs only the binding that holds now. Research
[024](../01-RESEARCH/024-state-a-module-keeps-on-the-bus/00-overview.md) measured the alternatives and
the grants against a real server.
[Design 32](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §4 already names *state* as one of the
mesh's relationships — 1:1, last per subject — and reserves it to the mesh's own declarations.
[Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 expects key-value buckets on the bus.
## Considered Options
1. **Key-value buckets a module declares, created by the controller, reached through the runtime.**
Chosen.
2. **State as events on EVENTS, read last-per-subject.** Rejected: retention is per stream and EVENTS
keeps seven days, so a value unchanged for a week disappears; a second stream over the same subjects
is refused by the server (design 32 §3). And events give no get, list or delete.
3. **A last-per-subject stream per module, written by hand.** Rejected: it is what a key-value bucket
is on the server, without the client's get, list, delete and watch — the mesh writing NATS's
key-value layer again.
4. **State in a module's own files or database, shared by asking a tool.** Rejected for state every
machine must see: a machine joining later has to know whom to ask and poll, and an owner that is
down answers nothing — the property the bus exists to remove.
## Decision
**1. A module declares its state by name.** `state` names the buckets it owns, by local name; every
instance of the module may write and read them. `reads` names another module's bucket as
`<module>.<name>`, read-only. A bucket's options are its owner's: how many past values a key keeps,
and how long a value lives. A manifest names no bucket, stream or subject (design 32 §1).
**2. One bucket per module per name, mesh-wide.** A key may name a machine by the module's own
convention; the mesh does not scope buckets per machine.
**3. The controller creates the buckets, from the catalogue, on every raise** — from registration,
like a seat's stream, so a reader can watch a bucket whose owner is not yet assigned anywhere. A module
never creates one. The runtime's grant on each bucket is the union of what its carried modules may do:
an owner's instances write and read, a reader's read.
**4. Each assignment is issued its buckets in its membership** ([ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)),
by the name the module uses for each and whether it may write. The runtime serves `mesh/state.get`,
`put`, `delete`, `keys` and `watch` on the bundle's channel from that list, and refuses — with the
reason — a bucket the module was not issued and a write to one it only reads. A watch delivers the
current values first, without deletions, then an end-of-current marker, then every change, each as a
`mesh/state` request the bundle answers.
**5. No secret is stored in a bucket, sealed or not.** A bucket is a stream, and design 32 §10 keeps
every secret off streams. A value that needs a secret names it; the secret travels on request/reply.
**6. The mesh caps size; a bucket outlives its module.** One value per key and no expiry unless the
owner says otherwise; at most 256 KiB a value and 64 MiB a bucket. Unassigning a module leaves its
buckets and what is in them ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)); a bucket
whose declaration is gone from the catalogue is reported, never removed by the mesh.
## Consequences
- A machine that joins reads the current state at once, and every machine sees a change as it
happens, with no consumer created per reader and nothing replayed.
- The runtime's channel has a sixth verb family, and the SDKs a small state surface over it — a
contract, which ADR 0039 admits: it changes when the verbs do, rarely, and every module should be
rebuilt when it does.
- What got harder: the runtime must keep each module to its own buckets, because one principal per
machine carries all of them and the server enforces only the union. A write the server refuses
surfaces to a client as a timeout, not a refusal, so the runtime's own refusal is what a module sees.
- The secrets rule is only partly mechanical. Sealed values cannot be recognised; the runtime refuses
a value with a field whose name says it is a credential, which catches the ordinary mistake and not a
determined one. For the operator's agent this means an MCP server's authorisation header stays out of
its bucket.
- Buckets accumulate as modules come and go; that they are reported rather than removed is the price
of not deleting data.
## How it is checked
| Rule | Checked by |
|---|---|
| A manifest's state names are local, and a read names a bucket its owner declares | the catalogue's registration check, per manifest; a catalogue test that every `reads` whose owner is present names a bucket that owner declares |
| Buckets exist for every declared state | the controller's raise asserts them idempotently; its test over a real bus |
| Owners write, readers only read | the composer's test of the grants, per principal kind; the runtime's refusal test over a real bus |
| A watch hands current values first, without deletions, then changes | the runtime's test over a real bus |
| No credential-named field in a value | the runtime's refusal test |
| Live | one module puts on one machine and another machine's watch sees it; a machine assigned afterwards reads it at start |
## References
- Research [024](../01-RESEARCH/024-state-a-module-keeps-on-the-bus/00-overview.md)
- [Design 32](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §4 and §10, [design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 and §3
- [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md), [ADR 0039](0039-what-the-sdk-holds-and-refuses.md),
[ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md),
[ADR 0198](0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)
02-DECISIONS/README.md
+1
View File
@@ -298,6 +298,7 @@ python3 00-META/checks/index.py fail if stale
- **0195** — [The mesh's tools are found by address, not announced whole](0195-the-meshs-tools-are-found-by-address-not-announced-whole.md)
- **0197** — [Every tool announces itself on the bus, in the NATS services protocol](0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)
- **0198** — [A module's long-running code is launched by the node's runtime, and reaches the bus through it](0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)
- **0201** — [A module keeps its current state in key-value buckets it declares, and reaches them through the runtime](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)
### How it is built