Compare commits

..
Author SHA1 Message Date
jschoubben bc64c5c187 ADR 0201 → 0202, and three issues from the night it shipped
The derived-value record is renumbered a second time: the key-value-buckets
record took 0201 while this waited to merge, as the bundles record took 0188
before it. Both times free when chosen, taken by the time it landed. cycle.py
caught it; three repositories cite this record, so the number matters.

225 — a provisioner has not been able to read its grant secrets since 01:30,
when a module's own code left its container and the files stayed root's. Four
thousand refusals, each worded as patience, and two consumers unserved. Not
from ADR 0202 or 0189, which landed hours later; dates in the report.

226 — the store's sweep stops at the first reference recorded with an address
and collects nothing. A guard that cannot tell 'I will not ask about this'
from 'it would not answer' stops the wrong amount of work.

227 — the photo app's admin client asks for the port the proxy holds. A module
pinned months behind carries everything its branch gained, the first time
anything makes it move.
2026-10-04 04:33:45 +02:00
mesh-admin be4b5777b8 Merge pull request 'Research 024 and ADR 0201: a module keeps its current state in key-value buckets' (#348) from feat/module-state-on-the-bus into main 2026-10-04 01:43:34 +00:00
mesh-admin d882b3568c Merge pull request 'Group 8: ADR 0201 (a provider declares what it derives, issue 124) and ADR 0189 (the store keeps what the records name, issue 108); issue 202' (#305) from feat/the-store-keeps-what-the-records-name into main 2026-10-04 01:30:47 +00:00
jochen 6b4da63261 Research 024: the composed grants, checked against a server once built 2026-10-04 02:50:02 +02:00
jochen e1b0bbde91 Research 024 and ADR 0201: a module keeps its current state in key-value buckets
Events miss a machine that joins after them and replay history where only the
latest matters. A module now declares state it owns and reads; the controller
creates the buckets, the runtime serves them on the bundle's channel. Designs 32
and 25 amended; grants measured against a running server.
2026-10-04 02:36:59 +02:00
11 changed files with 517 additions and 14 deletions
@@ -0,0 +1,152 @@
---
status: graduated
initiated: 2026-10-04
touches: [the bus, what a module declares, the tool runtime, the SDK, the bus grants, 03-DESIGN/01-to-be/25-the-bus-on-nats.md, 03-DESIGN/01-to-be/32-what-a-module-declares.md]
became: [02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md, 03-DESIGN/01-to-be/32-what-a-module-declares.md, 03-DESIGN/01-to-be/25-the-bus-on-nats.md]
---
# 024 — State a module keeps on the bus
## What is investigated
A place on the bus where a module's own code keeps **current state** — not history — that every
machine sees, including a machine that joins after the state was written: put, get, delete, list and
watch, reached through the node's runtime the way a bundle already publishes, asks and subscribes
([ADR 0198](../../02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)).
On NATS that is a key-value bucket. The questions are what a module declares, who creates the
bucket, what the grants are, what the runtime's verbs are, and what may never be stored.
## Why
The mesh carries two kinds of module traffic and a third is missing.
- **Events** land in the EVENTS stream: limits retention, seven days, ten thousand messages per
subject, a durable consumer per consuming module that replays what it missed. Never a secret
([design 32](../../03-DESIGN/01-to-be/32-what-a-module-declares.md) §10).
- **Requests** are core request/reply — tool calls, a bundle's `mesh/ask` — and are kept nowhere.
Neither is *the current value of something*. Two cases from the first module that needs it, the
operator's agent on a machine ([design 36](../../03-DESIGN/01-to-be/36-the-operators-agent-on-a-machine.md)):
1. **An MCP server registered for every machine.** Registering emits an event every machine's copy
of the module consumes. A machine the module is assigned to *after* the registration has no
durable consumer yet — the consumer is created at assignment — so it never hears of it. Wanted
instead: one entry per server, for every machine or for one; every machine reads the whole current
set when it starts and watches for changes; unregistering is a delete; any machine can list it.
2. **Which licence a machine is bound to** ([design 39](../../03-DESIGN/01-to-be/39-the-anthropic-licence-manager.md)).
As events, a machine that was off for a day replays every rotation since and asks for a token
after each. It needs only the latest binding and its generation. The token itself stays on
request/reply and is never stored.
The design already expects this. [Design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1:
"conditions and observed state in key-value buckets that anything may watch".
[Research 017](../017-a-mesh-that-heals-itself/01-the-intended-behaviour.md) wants a provisioner's
"what I applied" and a rotation's step kept in one rather than in memory. Nothing implements it.
## What exists, measured 2026-10-04
| | fact | where |
|---|---|---|
| streams | five kinds of mesh stream: CONTROL (work queue), NODES and ASSIGNMENTS (last per subject), EVENTS (limits: 7 days, 10 000 per subject), one work queue per seat that accepts | the controller's broker streams |
| the state relationship | [design 32](../../03-DESIGN/01-to-be/32-what-a-module-declares.md) §4 already names *state* — 1:1, last per subject — and says it is "declared: the mesh's own". Two streams use it, both written by the controller. No module can declare it | design 32, the controller |
| key-value buckets | none, anywhere | all four code repositories |
| the runtime's bus verbs | `mesh/publish`, `mesh/ask`, `mesh/subscribe`; delivery back to the bundle is `mesh/event` | the runtime's launcher |
| the runtime's principal | one bus user per machine carries every assigned module; its grant is the union of theirs. That one module's code does not act as another is the runtime's to keep: it publishes under the module's own name by construction | the controller's grant composition, the runtime's bus |
| what a bundle is issued | a membership per assignment, last per subject, read directly by the runtime: where it serves, where it emits, what it reaches | ADR 0160 |
| who creates bus objects | the controller only — mesh streams on every raise, a seat's stream at registration, a module's consumer at assignment. No module reaches the JetStream API | design 25 §3 |
### What a key-value bucket needs from a grant, against a real server
Measured against nats-server 2.10 with the Go client the runtime already uses, a bucket created by
an unrestricted user and used by two users holding only the subjects below (`B` is the bucket):
| operation | subject published | writer | reader |
|---|---|---|---|
| bind to the bucket | `$JS.API.STREAM.INFO.KV_B` | yes | yes |
| get | `$JS.API.DIRECT.GET.KV_B.>` | yes | yes |
| put, delete | `$KV.B.>` | yes | **refused** |
| list keys, watch | `$JS.API.CONSUMER.CREATE.KV_B.>` — an ordered, ephemeral consumer | yes | yes |
| stop a watch cleanly | `$JS.API.CONSUMER.DELETE.KV_B.>` | yes | yes |
| answers | its own inbox, which every principal already subscribes | — | — |
*Checked again once built, 2026-10-04:* the grants the controller composes for two machines' runtimes —
one carrying the owner, one only a reader — were loaded into a server as composed, and each operation
was run as each runtime's user. The owner's did all of them; the reader's read, listed and watched,
and its put and delete were refused by the server.
Three things the measurement showed that reading the documentation would not have:
1. **A refused put is not an error to the caller; it is a timeout.** The server reports the
permission violation asynchronously, on the connection, and the client waits out its deadline
for an acknowledgement that never comes. So a runtime that relies on the grant alone tells a
bundle "timed out" for "you may not write this" — it must refuse first, from what the module was
issued, with the reason.
2. **A watch's current values include deletions.** A key deleted earlier arrives among the initial
values as a delete marker, before the end-of-current marker. A bundle asking "what is there now"
must not be handed those.
3. **Without the consumer-delete grant, stopping a watch hangs** until its deadline, and the
ephemeral consumer lingers on the server until it times out by itself.
### Whether the events shape is enough instead
Honestly compared, because a new primitive is a cost:
- **EVENTS cannot be made last-per-subject for some subjects.** Retention is per stream, and
JetStream refuses a second stream overlapping the first (verified and recorded in design 32 §3).
A state subject inside `mesh.mod.*.event.>` keeps EVENTS' seven days: a licence binding unchanged
for a week disappears.
- **A separate last-per-subject stream per module** is possible — it is exactly what a key-value
bucket *is* on the server: a stream with one message per subject, a rollup for purge, and direct
reads. Building it by hand gives up the client's get, list, delete and watch, which are the
operations both cases need, and would be the mesh writing NATS's own key-value layer again.
- **Consumers are the wrong reader.** A durable consumer per reading module is created at
assignment and replays from where it is; state wants "everything current, now, then changes",
which an ordered ephemeral consumer from the last value per subject gives and a durable does not.
So key-value is not a convenience over events; it is the state relationship design 32 already
names, opened to modules.
## Questions, and what this effort proposes
1. **What a manifest says.** `state` names the buckets a module owns, by local name — every
instance of the module may write them and read them. `reads` names another module's bucket as
`<module>.<name>`, read-only. Names only, never a bucket or subject (design 32 §1). A bucket's
options — how many past values it keeps, how long a value lives — are the owner's to declare,
the way a seat declares its own retention (design 32 §3).
2. **Scope.** One bucket per module per name, mesh-wide. A key may carry a machine by the module's
own convention (`all.<server>`, `<machine>.<server>`). A bucket per machine was considered and
not proposed: "list every server for every machine" becomes a walk over buckets, and the grant
could only narrow writes, which nothing asked for — every instance of the owner already writes.
3. **Who creates the bucket.** The controller, from the catalogue, on every raise — a bucket exists
from registration, like a seat's stream, so a reader can watch before the owner is assigned
anywhere. Never a module.
4. **The runtime's verbs.** `mesh/state.get`, `mesh/state.put`, `mesh/state.delete`,
`mesh/state.keys`, `mesh/state.watch`, each naming the bucket as the module named it. A watch
is answered once the current values are on their way, then each change is delivered to the
bundle as a `mesh/state` request it answers — current values first (no deletions among them), an
end-of-current marker, then changes. A child that restarts watches again, as it subscribes
again. The runtime refuses, with the reason, a bucket the module was not issued, and a write to
one it only reads.
5. **Secrets.** None in a bucket, sealed or not: a bucket is a stream (design 32 §10). Sealed values
are plain base64 and cannot be recognised, so the mechanical check is partial and said to be: the
runtime refuses a value carrying a field whose name says it is a credential (`password`,
`secret`, `token`, `authorization`, …), which catches the ordinary mistake and not a determined
one. For the first consumer this has a concrete consequence: an MCP server registered with an
authorisation header keeps that header out of the bucket.
6. **History, lifetime, size.** One value per key unless the owner says more; no expiry unless it
says one; a value at most 256 KiB and a bucket at most 64 MiB, the mesh's caps rather than a
module's. **A bucket outlives its module's assignment** — what a module stored is data, and data
outlives what declared it ([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md));
unassigning is not cleaning up. A bucket whose declaration is gone is reported, never removed.
7. **Events or state.** State (above).
## The work, once decided
1. A decision record, then design 32 (*state* becomes a relationship a module declares) and design
25 (key-value buckets are part of the bus) amended.
2. The controller: the manifest's two words and their registration check; buckets asserted on every
raise; the grants for owners' and readers' runtimes; the buckets issued in each membership.
3. The runtime: the five verbs, the watch delivery, the refusals; tested against a real server.
4. The SDK, TypeScript and Go: a small state surface over the verbs.
5. Proved on a running mesh with one small module, then handed to the operator's agent, whose
registered servers move from events to a bucket.
@@ -0,0 +1,106 @@
---
topic: what runs on it
status: accepted
date: 2026-10-04
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md
---
# 201. A module keeps its current state in key-value buckets it declares, and reaches them through the runtime
## Context
A module's code reaches the bus through the node's runtime: it publishes events, subscribes to them
and asks tools ([ADR 0198](0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)).
Events are kept for a week and replayed to a consumer that was away; requests are kept nowhere. What
neither gives is **the current value of something**, seen by every machine, including one that joins
after it was written. The first module to need it — the operator's agent on a machine — registers MCP
servers for every machine as events, and a machine assigned later never hears of them; and it would
replay a week of licence rotations where it needs only the binding that holds now. Research
[024](../01-RESEARCH/024-state-a-module-keeps-on-the-bus/00-overview.md) measured the alternatives and
the grants against a real server.
[Design 32](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §4 already names *state* as one of the
mesh's relationships — 1:1, last per subject — and reserves it to the mesh's own declarations.
[Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 expects key-value buckets on the bus.
## Considered Options
1. **Key-value buckets a module declares, created by the controller, reached through the runtime.**
Chosen.
2. **State as events on EVENTS, read last-per-subject.** Rejected: retention is per stream and EVENTS
keeps seven days, so a value unchanged for a week disappears; a second stream over the same subjects
is refused by the server (design 32 §3). And events give no get, list or delete.
3. **A last-per-subject stream per module, written by hand.** Rejected: it is what a key-value bucket
is on the server, without the client's get, list, delete and watch — the mesh writing NATS's
key-value layer again.
4. **State in a module's own files or database, shared by asking a tool.** Rejected for state every
machine must see: a machine joining later has to know whom to ask and poll, and an owner that is
down answers nothing — the property the bus exists to remove.
## Decision
**1. A module declares its state by name.** `state` names the buckets it owns, by local name; every
instance of the module may write and read them. `reads` names another module's bucket as
`<module>.<name>`, read-only. A bucket's options are its owner's: how many past values a key keeps,
and how long a value lives. A manifest names no bucket, stream or subject (design 32 §1).
**2. One bucket per module per name, mesh-wide.** A key may name a machine by the module's own
convention; the mesh does not scope buckets per machine.
**3. The controller creates the buckets, from the catalogue, on every raise** — from registration,
like a seat's stream, so a reader can watch a bucket whose owner is not yet assigned anywhere. A module
never creates one. The runtime's grant on each bucket is the union of what its carried modules may do:
an owner's instances write and read, a reader's read.
**4. Each assignment is issued its buckets in its membership** ([ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)),
by the name the module uses for each and whether it may write. The runtime serves `mesh/state.get`,
`put`, `delete`, `keys` and `watch` on the bundle's channel from that list, and refuses — with the
reason — a bucket the module was not issued and a write to one it only reads. A watch delivers the
current values first, without deletions, then an end-of-current marker, then every change, each as a
`mesh/state` request the bundle answers.
**5. No secret is stored in a bucket, sealed or not.** A bucket is a stream, and design 32 §10 keeps
every secret off streams. A value that needs a secret names it; the secret travels on request/reply.
**6. The mesh caps size; a bucket outlives its module.** One value per key and no expiry unless the
owner says otherwise; at most 256 KiB a value and 64 MiB a bucket. Unassigning a module leaves its
buckets and what is in them ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)); a bucket
whose declaration is gone from the catalogue is reported, never removed by the mesh.
## Consequences
- A machine that joins reads the current state at once, and every machine sees a change as it
happens, with no consumer created per reader and nothing replayed.
- The runtime's channel has a sixth verb family, and the SDKs a small state surface over it — a
contract, which ADR 0039 admits: it changes when the verbs do, rarely, and every module should be
rebuilt when it does.
- What got harder: the runtime must keep each module to its own buckets, because one principal per
machine carries all of them and the server enforces only the union. A write the server refuses
surfaces to a client as a timeout, not a refusal, so the runtime's own refusal is what a module sees.
- The secrets rule is only partly mechanical. Sealed values cannot be recognised; the runtime refuses
a value with a field whose name says it is a credential, which catches the ordinary mistake and not a
determined one. For the operator's agent this means an MCP server's authorisation header stays out of
its bucket.
- Buckets accumulate as modules come and go; that they are reported rather than removed is the price
of not deleting data.
## How it is checked
| Rule | Checked by |
|---|---|
| A manifest's state names are local, and a read names a bucket its owner declares | the catalogue's registration check, per manifest; a catalogue test that every `reads` whose owner is present names a bucket that owner declares |
| Buckets exist for every declared state | the controller's raise asserts them idempotently; its test over a real bus |
| Owners write, readers only read | the composer's test of the grants, per principal kind; the runtime's refusal test over a real bus |
| A watch hands current values first, without deletions, then changes | the runtime's test over a real bus |
| No credential-named field in a value | the runtime's refusal test |
| Live | one module puts on one machine and another machine's watch sees it; a machine assigned afterwards reads it at start |
## References
- Research [024](../01-RESEARCH/024-state-a-module-keeps-on-the-bus/00-overview.md)
- [Design 32](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §4 and §10, [design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 and §3
- [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md), [ADR 0039](0039-what-the-sdk-holds-and-refuses.md),
[ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md),
[ADR 0198](0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)
@@ -7,11 +7,14 @@ reconstructed: false
extends: 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
---
# 201. A provider declares what it derives for each consumer, and the mesh tells both ends
# 202. A provider declares what it derives for each consumer, and the mesh tells both ends
> Written as 0188 on 2026-10-02 and renumbered to 0201 on 2026-10-04: the record of the bundles
> refactor took 0188 on main while this one waited in a pull request, and the mesh's own code now
> cites that one. Only the number moved; the decision is the one taken on the 2nd.
> **Written as 0188 on 2026-10-02, renumbered to 0201, and to 0202 on 2026-10-04.** Twice, for the
> same reason twice: the bundles refactor took 0188 while this waited in a pull request, and the
> key-value-buckets record took 0201 while this waited again. Both times the number was free when
> it was chosen and taken by the time this merged. Only the number moved; the decision is the one
> taken on the 2nd. The check that refuses two records sharing a number is what caught it, both
> times — a number is how a record is cited, and three repositories cite this one.
## Context
+2 -1
View File
@@ -190,7 +190,7 @@ python3 00-META/checks/index.py fail if stale
- **0187** — [A dead tracker is not the machine's failure](0187-a-dead-tracker-is-not-the-machines-failure.md)
- **0189** — [The store keeps what the records name, and a maintenance step holds its writers still](0189-the-store-keeps-what-the-records-name.md)
- **0190** — [A seat's work is shared by its holders, and building is the first such role](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md)
- **0201** — [A provider declares what it derives for each consumer, and the mesh tells both ends](0201-a-provider-declares-what-it-derives-for-each-consumer.md)
- **0202** — [A provider declares what it derives for each consumer, and the mesh tells both ends](0202-a-provider-declares-what-it-derives-for-each-consumer.md)
### Its tiers, from the bottom up
@@ -300,6 +300,7 @@ python3 00-META/checks/index.py fail if stale
- **0195** — [The mesh's tools are found by address, not announced whole](0195-the-meshs-tools-are-found-by-address-not-announced-whole.md)
- **0197** — [Every tool announces itself on the bus, in the NATS services protocol](0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)
- **0198** — [A module's long-running code is launched by the node's runtime, and reaches the bus through it](0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)
- **0201** — [A module keeps its current state in key-value buckets it declares, and reaches them through the runtime](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)
### How it is built
+23 -2
View File
@@ -7,8 +7,10 @@ code:
- mesh-tools src/broker-amqp.ts (to be replaced)
- mesh-catalog modules/nats (to be written)
- mesh-sdk src (the protocol's NATS binding, step 3)
updated: 2026-10-02
- mesh-tools node-tools/internal/bus (a module's state, ADR 0201)
updated: 2026-10-04
decisions:
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
@@ -48,6 +50,7 @@ mesh's own state lives, and where what a module may say is decided by what it de
| **events** — a module says something happened | 1:many | delivered to every consumer that declared it; dead-lettered when it cannot be |
| **tools** — a module or a person asks another's tool | request/reply | one answer, from one server, or a timeout |
| **work to a role** — a module submits to a capability without knowing who provides it | job | exactly one holder does it; it queues while nobody does |
| **state** — a module's current value of something, every machine reading it | key-value | the newest per key, kept until replaced or deleted; read whole by a machine that joins later |
The last two rows are the ones worth dwelling on, because they are not messaging in the sense of
carrying bytes from A to B. **A role is addressable**, so a caller names the capability and never
@@ -56,7 +59,8 @@ changing ([ADR 0126](../../02-DECISIONS/0126-a-module-declares-its-own-seats.md)
property of the mesh's architecture that happens to be expressed in subjects.
And more of the mesh lands here as it is built: conditions and observed state in key-value
buckets that anything may watch, the server's own advisories becoming observations like any other
buckets that anything may watch — the first of them a module's own declared state, *2026-10-04*
([ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)) — the server's own advisories becoming observations like any other
([research 017](../../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md)), and a person's
client speaking the bus directly rather than through a surface built over it (§7). None of that
is a message being moved; all of it is the bus being the mesh's centre.
@@ -84,8 +88,16 @@ mesh.seat.<seat>.event.<verb> a role's own event (JetStream: EVENT
mesh.seat.<seat>.tool.<verb> a role's tool (core request/reply)
mesh.ask.<node>.<command> the controller's command api (core request/reply)
mesh.assignment.<node>.<module> an assignment's membership (JetStream: ASSIGNMENTS, last-per-subject)
$KV.<module>_<name>.<key> a module's state (JetStream: a key-value bucket per declared name)
```
**Added 2026-10-04** ([ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)):
the last row is outside `mesh.` on purpose. A key-value bucket is NATS's own construct and lives
under NATS's own prefix, which is what lets the server's key-value layer — direct reads, rollups,
delete markers, watches — do the work instead of the mesh writing it again. The bucket is named for
the module and the local name joined by an underscore, which neither may contain, so two modules can
never derive one bucket.
**Revised 2026-10-01** ([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)): the rows above
for a module's and a seat's tools are the shapes the controller *issues*, not rules a runtime carries.
Every assignment is published a membership — what it serves and where, in which queue, its seat verbs,
@@ -155,6 +167,7 @@ Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStrea
| CONTROL | `mesh.control.>` except `alive` (a build's outcome moved to its seat, ADR 0121) | work queue, one consumer (the controller), explicit ack | the store-window guarantee ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)): the controller `nak`s with a delay while its store is away and the message is redelivered; nothing is dropped |
| NODES | `mesh.node.>` | last per subject | one declaration per node, always the newest |
| EVENTS | `mesh.mod.*.event.>` and `mesh.seat.*.event.>` | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream). *2026-10-01:* a build's whole log is here too, as the build-machine seat's `log.<build id>` events ([ADR 0157](../../02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)) — one subject per build, a week of retention, read back by `builds --log <id>` with a consumer that is gone when the reading is done |
| `KV_<module>_<name>` | `$KV.<module>_<name>.>` | the newest value per key — as many past values as the owner declared — no age unless the owner declared one; a value at most 256 KiB, a bucket at most 64 MiB | a module's state (ADR 0201): one per name in a manifest's `state`, created from the catalogue on every raise, so it exists before its owner runs anywhere; kept when the module is unassigned, because what it holds is data |
Tool calls and heartbeats stay on core NATS: a lost heartbeat is the next heartbeat; a lost tool
call is a timeout the caller already handles.
@@ -234,6 +247,14 @@ expresses this exactly, per subject, and better than a vhost could:
permissions for each consumed event's subject, its tool subjects, and that same inbox prefix.
Nothing else. A module that tries to publish outside its emits is refused by the server, not by
convention.
- **A module's state** (ADR 0201), for whichever principal carries the module — today the machine's
runtime, whose grant is the union of its modules': binding to the bucket, reading a key directly,
and an ordered consumer for listing and watching, created and deleted on the bucket's own stream
and nothing else's; and, for the owner's instances only, publishing under the bucket's own
`$KV.<bucket>.>`. *Measured 2026-10-04 against a running server:* without the consumer-delete
grant a watch cannot be stopped cleanly, and a write the server refuses reaches the writer as a
timeout rather than a refusal — so the runtime refuses first, from the membership, and the grant
is the second line.
- **The controller's user** owns `mesh.control.>`, `mesh.node.>` and the streams, and may submit work
to the seats the mesh's own flows use — a build, for one (ADR 0121).
**A host's user** may publish its own `mesh.control.<node>.>` and subscribe its own
@@ -14,7 +14,7 @@ decisions:
- 02-DECISIONS/0084-which-provider-serves-a-consumer.md
- 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md
- 02-DECISIONS/0038-the-mesh-assigns-the-port.md
- 02-DECISIONS/0201-a-provider-declares-what-it-derives-for-each-consumer.md
- 02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md
---
# 27 — A module requires, the mesh resolves
@@ -208,7 +208,7 @@ the placeholder allows: the definition says which values reach which requirement
does. *How it is checked:* the unit tests named in issue 173, and the plan comparison that closed it.
*A provider says once what it derives for each consumer (2026-10-02,
[ADR 0201](../../02-DECISIONS/0201-a-provider-declares-what-it-derives-for-each-consumer.md),
[ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md),
[issue 124](../../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)):*
where a provider **names the resource** it gives each consumer — a bucket, a database, a vhost — the
name is derived per consumer, and a literal `serves` block could not carry it. A served value may
@@ -221,7 +221,7 @@ its binding's served facts and as `${bound:<provision>:<key>}` in any file it wr
as `derived` on that consumer's entry in its contributions file, so its provisioner is told the name
rather than recomputing it. A consumer that writes the derived value into its own definition instead
of asking for it is refused, naming the placeholder to use. *How it is checked:* the unit tests in
ADR 0201's "how this is checked", each run against the unchanged controller first.
ADR 0202's "how this is checked", each run against the unchanged controller first.
## How a definition reads what was resolved
@@ -11,8 +11,10 @@ code:
- mesh-host internal/apply/apply.go
- mesh-tools src/main.ts
- mesh-catalog modules/mesh-catalog
updated: 2026-10-02
- mesh-tools node-tools/internal/runtime (a module's state, ADR 0201)
updated: 2026-10-04
decisions:
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
@@ -67,6 +69,8 @@ the catalogue, and the mesh would have hundreds of copies of a decision it made
| `tools: status` | queue-group subscription on `mesh.mod.<module>.tool.status` |
| seat `telegram-sender`, `accepts: send` | work-queue consumer on `mesh.seat.telegram-sender.accept.send` |
| `uses: telegram-sender` | publish on that seat's `accept` subjects, and nothing else |
| `state: servers` | a key-value bucket for the module, created by the controller; its instances write and read it |
| `reads: billing.orders` | read and watch billing's `orders` bucket, and nothing else of it |
**Wildcards, and they are the mesh's rather than a bus's.** *Added 2026-09-27, from
[issue 127](../../04-ISSUES/127-a-module-event-derives-a-subject-nothing-publishes/00-report.md).* A
@@ -221,7 +225,7 @@ service" versus "one worker per machine".
| credential | sealed, per consumer | none | none | none | none |
| reply | — | none | none, or an event later | a report | awaited |
| retention | — | age and size | work queue, explicit ack | **last per subject** | none |
| declared | `provides`/`requires` | `emits`/`consumes` | seat `accepts` / `uses` | the mesh's own | `serves` |
| declared | `provides`/`requires` | `emits`/`consumes` | seat `accepts` / `uses` | the mesh's own; a module's `state` / `reads` | `serves` |
**Job** is the one [ADR 0041](../../02-DECISIONS/0041-events-are-a-relationship.md) had no room
for. Its table has events at 1:many and provisions at 1:1; a module submitting work to a service
@@ -236,6 +240,31 @@ last-per-subject retention, and a node that has seen sequence *n* refuses *n−1
That is the wire-level answer to
[issue 107](../../04-ISSUES/107-a-declaration-carries-no-order/00-report.md).
**A module declares state too.** *Added 2026-10-04,
[ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md).*
State was the mesh's alone, and modules had the same need with nowhere to put it: an MCP server
registered for every machine, sent as an event, never reached a machine assigned afterwards — its
consumer did not exist yet when the event passed — and a licence binding sent as events replays a
week of rotations where only the latest matters. So a module names the state it **owns** with
`state`, and another module's it **reads** with `reads: <module>.<name>`. Each is a key-value bucket
the controller creates from the catalogue, mesh-wide, existing from registration so a reader can
watch before the owner runs anywhere ([design 25](25-the-bus-on-nats.md) §3). Every instance of the
owner writes; a reader reads and watches. A key may name a machine by the module's own convention;
the mesh keeps one bucket per name, not one per machine, because "every server, for every machine"
is then one list rather than a walk.
What a module sees is what it named. Its assignment's membership lists its buckets by those names,
with whether it may write, and the runtime answers `get`, `put`, `delete`, `keys` and `watch` for
them on the bundle's channel — refusing, with the reason, a name it was not issued or a write to a
bucket it only reads. A watch hands the current values first, then every change: a bundle that starts
late, or starts again, has the whole of the state before it has any of the news.
The owner says how many past values a key keeps and how long a value lives, as a seat says how long
its backlog survives (§3); the mesh caps a value's size and a bucket's. **A bucket outlives its
module's assignment** — what a module stored is data
([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md)) — and one whose
declaration is gone is reported, never removed by the mesh.
## 5. Seats
A module declares a seat with its protocol, and the mesh enforces one holder at its scope
@@ -454,6 +483,14 @@ sealing key leaks, that stream is an archive rather than a moment. So:
existing discipline — *fetched from it, not carried* — applied to the one payload where carrying
it is worst.
**A key-value bucket is a stream, so the same holds for it.** *Added 2026-10-04,
[ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md).*
No secret is put in a module's state, sealed or not: state is exactly what a machine joining a year
later reads in full. A value that needs a secret names it, and the secret travels on request/reply.
Sealed values are plain text to anything inspecting them, so this is checked only partly — the
runtime refuses a value carrying a field whose name says it is a credential, which catches the
ordinary mistake and not a determined one.
**The bootstrap, which is circular and has a precedent.** The vault makes every secret
([ADR 0113](../../02-DECISIONS/0113-the-vault-makes-every-secret.md)), including the bus's own
passwords. The vault is a module, and a module needs a bus account, whose password the vault
@@ -523,3 +560,9 @@ billing existing under that name.
on, and exactly those two are rebuilt.
- **A stale declaration is refused.** A bed: replay sequence *n−1* after *n*, and the node refuses
it rather than applying it.
- **A module reaches only the state it declared.** The composer's test: an owner's runtime may write
its buckets, a reader's may only read, and nothing else is granted; the runtime's test over a real
bus: a name not issued and a reader's write are refused with the reason.
- **State is current at once.** The runtime's test over a real bus: a watch hands the current values
without deletions, then an end-of-current marker, then changes. Live: a machine assigned after a put
reads it at start.
@@ -2,7 +2,7 @@
status: resolved
opened: 2026-09-26
located-in: [mesh-controller internal/catalogue, mesh-sdk src/provisioner, mesh-catalog modules/minio]
fixed-by: 02-DECISIONS/0201-a-provider-declares-what-it-derives-for-each-consumer.md
fixed-by: 02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md
amended-design: 03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md
---
@@ -64,7 +64,7 @@ compares it to what the provider will actually create. The one wrong instance wa
bucket in its own configuration against the one the provider would create is a check that could
exist today, for any interface, without the mechanism above.
## Answered, 2026-10-02 — [ADR 0201](../../02-DECISIONS/0201-a-provider-declares-what-it-derives-for-each-consumer.md)
## Answered, 2026-10-02 — [ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md)
The channel is the provider's own `serves` block, which may now name the consumer the mesh is
serving: `${consumer:as}` and `${consumer:as:dns}`. The mesh fills it once, where it knows who the
@@ -0,0 +1,75 @@
---
status: open
opened: 2026-10-04
located-in: [mesh-host internal/apply, mesh-controller internal/catalogue]
fixed-by:
amended-design:
---
# 225 — A provisioner cannot read the grant secrets since its code left the container, and every consumer of it is unserved
## What was observed
On the control machine, 2026-10-04, found while looking at why one app was restarting:
```
[mongodb] [provisioner:mongodb-database] mesh_novox_photos: secret not readable yet
(/var/lib/mongodb/grants/novox.photos.secret):
Error: EACCES: permission denied, open '/var/lib/mongodb/grants/novox.photos.secret'
```
**4330 times, every five seconds, since 01:30:20.** The consequence is not a log line: the
provisioner never reads the password, so it never creates the user, so the consumer never
connects —
```
UserNotFound: Could not find user "mesh_novox_photos" for db "admin"
```
— and the app crash-loops. Two consumers on this machine are in that state.
## Why
The grant secrets are what the mesh seals for each consumer and the host unseals beside the
provider's contributions file. They are written `-rw------- root root`, which was right while a
module's own code ran in a container as root.
[ADR 0198](../../02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)
moved a module's long-running code out of its container and under the node's runtime, which runs
as the operator's account. The provisioner is now that account; the secret is still root's. The
timestamps say it exactly: the files are dated 2026-09-26, the first refusal is 01:30:20 on the
day the runtime rolled.
**Nothing reports it.** The machine applies cleanly and reads as current; the provisioner says
`secret not readable yet`, whose wording is for a real and ordinary race on the first pass — the
host has not written the file yet — and which is indistinguishable, in the log, from a permanent
refusal. Four thousand occurrences of a message that means "wait a moment" is the shape to
recognise.
## Why it matters beyond this instance
This is every provider that provisions. The grant secret is the one file the sdk's harness reads
for every consumer, so a provider that cannot read it serves nobody — and says so only in a line
that reads like patience.
It is also the general question the runtime move leaves: **what the mesh seals for a module is
owned for the shape that module's code used to have.** Each module whose code moved is a module
whose files may now be unreadable to it, and ownership is the mesh's to state, not the module's
to work around.
## What this is not
Not caused by [ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md)
or [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md), which landed two
to three hours after the first refusal. Those rebuilt the two affected consumers, which recreated
their containers and made a silent fault visible as a restarting one. The dates are above.
## Open questions
- Who owns a grant secret now — the module's account, as `secrets-owner` already says for a
module's own secrets? Then the host writes it so, and this is a one-line statement in the
declaration rather than a convention.
- Should `secret not readable yet` stop saying "yet" after the first few passes? A message that
is right once and wrong four thousand times is a message that hides its own meaning.
- Which other modules' files did the runtime move leave behind? The sweep is the same question
for every path the mesh writes for a module: directories, bundles, received files.
@@ -0,0 +1,58 @@
---
status: open
opened: 2026-10-04
located-in: [mesh-controller cmd/mesh-controller/collect.go, mesh-controller internal/inventory/collection.go]
fixed-by:
amended-design:
---
# 226 — The store's sweep stops at the first reference recorded with an address, so it collects nothing at all
## What was observed
The first live run of [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md)'s
sweep, 2026-10-04, printed on every build:
```
the artifact store kept 127.0.0.1:5100/mesh-tools/build@sha256:0de48cd3…, so nothing more was
asked of it: 127.0.0.1:5100/mesh-tools/build@sha256:0de48cd3… is not a reference into the
mesh's artifact store
1681 more to collect; the next build asks again
```
Nothing is collected, and nothing ever will be. The store holds 1681 artifacts the mesh no longer
keeps and the feature that exists to remove them is inert.
## Why
Two correct decisions meeting badly.
**A reference recorded before references were kept without an address** is
`127.0.0.1:5100/<path>@sha256:…` rather than `artifact-store://<path>@sha256:…`
([04-ISSUES/102](../102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)).
`LetGo` rightly refuses to compose a delete for a reference whose shape it does not recognise —
that refusal is what keeps the sweep from reaching something that is not the mesh's.
**The sweep stops at the first refusal**, because "a store that refuses one refuses all of them"
— deletion disabled, the store down, the network gone — and pushing through would mean a hundred
identical failures in front of whoever was building something. That reasoning is right for the
store refusing. It is wrong for *this* record being unreadable.
So one old record, early in the oldest-first order, halts the whole sweep for ever.
## Why it matters beyond this instance
**A guard that cannot tell "I will not ask about this" from "it would not answer" stops the wrong
amount of work.** The two deserve opposite responses: skip one, abandon the other. Collapsing
them into "an error" is how a bounded, cautious loop becomes a loop that does nothing — and it
reports the right number while doing it, which is what made it look healthy.
## What a fix has to settle
- A reference the sweep cannot address is **skipped, and the sweep goes on** — it is a fact about
that record, not about the store.
- `Recorded()` already normalises the old form to the kept one, and is what the rest of the mesh
uses for exactly these references. The sweep should normalise before asking rather than refuse.
- Only a refusal *by the store* ends a sweep.
- **How it is checked:** a sweep over records holding one address-recorded reference and one kept
one collects the second; a sweep against a store that refuses stops at the first.
@@ -0,0 +1,44 @@
---
status: open
opened: 2026-10-04
located-in: [mesh-catalog modules/photos]
fixed-by:
amended-design:
---
# 227 — The photo app's admin client asks for port 80, which the reverse proxy holds, so it cannot start
## What was observed
Applying the control machine, 2026-10-04:
```
applying "photos.admin-client": starting container photos-admin-client:
failed to bind host port 0.0.0.0:80/tcp: address already in use
```
Port 80 on that machine belongs to the reverse proxy (`mesh-route-proxy`, confirmed with `ss`),
which is the whole arrangement: the proxy holds the public ports and every module is reached
through it. A module that publishes 80 itself can never start beside it.
Everything else on the machine applied; this one resource fails every pass.
## How it surfaced
`photos` had been pinned at a commit from 2026-09-28 and was rebuilt to `main` on 2026-10-04 —
forced by [ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md)'s
refusal of its transcribed bucket name. The admin client is one of the changes that came with the
rest of `main`. The rebuild did not create the conflict; it delivered it.
**A module pinned months behind carries whatever its branch gained, all at once, the first time
something makes it move.** That is the cost of a pin, and it is paid in full rather than
gradually.
## What a fix has to settle
- Which port the admin client should ask for, or whether it should be reached through the proxy
like everything else and publish nothing.
- Whether a module declaring a port the machine's proxy already holds should be refused when it
is composed, rather than failing on the machine every pass. The mesh assigns ports
([ADR 0038](../../02-DECISIONS/0038-the-mesh-assigns-the-port.md)); a fixed 80 beside a proxy is
a statement it could check.