Compare commits

..
11 changed files with 23 additions and 203 deletions
@@ -2,7 +2,7 @@
status: graduated
initiated: 2026-10-04
touches: [the bus, what a module declares, the tool runtime, the SDK, the bus grants, 03-DESIGN/01-to-be/25-the-bus-on-nats.md, 03-DESIGN/01-to-be/32-what-a-module-declares.md]
became: [02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md, 03-DESIGN/01-to-be/32-what-a-module-declares.md, 03-DESIGN/01-to-be/25-the-bus-on-nats.md]
became: [02-DECISIONS/0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md, 03-DESIGN/01-to-be/32-what-a-module-declares.md, 03-DESIGN/01-to-be/25-the-bus-on-nats.md]
---
# 024 — State a module keeps on the bus
@@ -7,14 +7,11 @@ reconstructed: false
extends: 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
---
# 202. A provider declares what it derives for each consumer, and the mesh tells both ends
# 201. A provider declares what it derives for each consumer, and the mesh tells both ends
> **Written as 0188 on 2026-10-02, renumbered to 0201, and to 0202 on 2026-10-04.** Twice, for the
> same reason twice: the bundles refactor took 0188 while this waited in a pull request, and the
> key-value-buckets record took 0201 while this waited again. Both times the number was free when
> it was chosen and taken by the time this merged. Only the number moved; the decision is the one
> taken on the 2nd. The check that refuses two records sharing a number is what caught it, both
> times — a number is how a record is cited, and three repositories cite this one.
> Written as 0188 on 2026-10-02 and renumbered to 0201 on 2026-10-04: the record of the bundles
> refactor took 0188 on main while this one waited in a pull request, and the mesh's own code now
> cites that one. Only the number moved; the decision is the one taken on the 2nd.
## Context
@@ -7,7 +7,7 @@ reconstructed: false
extends: 02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md
---
# 201. A module keeps its current state in key-value buckets it declares, and reaches them through the runtime
# 202. A module keeps its current state in key-value buckets it declares, and reaches them through the runtime
## Context
+2 -2
View File
@@ -190,7 +190,7 @@ python3 00-META/checks/index.py fail if stale
- **0187** — [A dead tracker is not the machine's failure](0187-a-dead-tracker-is-not-the-machines-failure.md)
- **0189** — [The store keeps what the records name, and a maintenance step holds its writers still](0189-the-store-keeps-what-the-records-name.md)
- **0190** — [A seat's work is shared by its holders, and building is the first such role](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md)
- **0202** — [A provider declares what it derives for each consumer, and the mesh tells both ends](0202-a-provider-declares-what-it-derives-for-each-consumer.md)
- **0201** — [A provider declares what it derives for each consumer, and the mesh tells both ends](0201-a-provider-declares-what-it-derives-for-each-consumer.md)
### Its tiers, from the bottom up
@@ -300,7 +300,7 @@ python3 00-META/checks/index.py fail if stale
- **0195** — [The mesh's tools are found by address, not announced whole](0195-the-meshs-tools-are-found-by-address-not-announced-whole.md)
- **0197** — [Every tool announces itself on the bus, in the NATS services protocol](0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)
- **0198** — [A module's long-running code is launched by the node's runtime, and reaches the bus through it](0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)
- **0201** — [A module keeps its current state in key-value buckets it declares, and reaches them through the runtime](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)
- **0202** — [A module keeps its current state in key-value buckets it declares, and reaches them through the runtime](0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)
### How it is built
+6 -6
View File
@@ -7,10 +7,10 @@ code:
- mesh-tools src/broker-amqp.ts (to be replaced)
- mesh-catalog modules/nats (to be written)
- mesh-sdk src (the protocol's NATS binding, step 3)
- mesh-tools node-tools/internal/bus (a module's state, ADR 0201)
- mesh-tools node-tools/internal/bus (a module's state, ADR 0202)
updated: 2026-10-04
decisions:
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
@@ -60,7 +60,7 @@ property of the mesh's architecture that happens to be expressed in subjects.
And more of the mesh lands here as it is built: conditions and observed state in key-value
buckets that anything may watch — the first of them a module's own declared state, *2026-10-04*
([ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)) — the server's own advisories becoming observations like any other
([ADR 0202](../../02-DECISIONS/0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)) — the server's own advisories becoming observations like any other
([research 017](../../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md)), and a person's
client speaking the bus directly rather than through a surface built over it (§7). None of that
is a message being moved; all of it is the bus being the mesh's centre.
@@ -91,7 +91,7 @@ mesh.assignment.<node>.<module> an assignment's membership (JetStream: ASSIG
$KV.<module>_<name>.<key> a module's state (JetStream: a key-value bucket per declared name)
```
**Added 2026-10-04** ([ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)):
**Added 2026-10-04** ([ADR 0202](../../02-DECISIONS/0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)):
the last row is outside `mesh.` on purpose. A key-value bucket is NATS's own construct and lives
under NATS's own prefix, which is what lets the server's key-value layer — direct reads, rollups,
delete markers, watches — do the work instead of the mesh writing it again. The bucket is named for
@@ -167,7 +167,7 @@ Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStrea
| CONTROL | `mesh.control.>` except `alive` (a build's outcome moved to its seat, ADR 0121) | work queue, one consumer (the controller), explicit ack | the store-window guarantee ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)): the controller `nak`s with a delay while its store is away and the message is redelivered; nothing is dropped |
| NODES | `mesh.node.>` | last per subject | one declaration per node, always the newest |
| EVENTS | `mesh.mod.*.event.>` and `mesh.seat.*.event.>` | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream). *2026-10-01:* a build's whole log is here too, as the build-machine seat's `log.<build id>` events ([ADR 0157](../../02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)) — one subject per build, a week of retention, read back by `builds --log <id>` with a consumer that is gone when the reading is done |
| `KV_<module>_<name>` | `$KV.<module>_<name>.>` | the newest value per key — as many past values as the owner declared — no age unless the owner declared one; a value at most 256 KiB, a bucket at most 64 MiB | a module's state (ADR 0201): one per name in a manifest's `state`, created from the catalogue on every raise, so it exists before its owner runs anywhere; kept when the module is unassigned, because what it holds is data |
| `KV_<module>_<name>` | `$KV.<module>_<name>.>` | the newest value per key — as many past values as the owner declared — no age unless the owner declared one; a value at most 256 KiB, a bucket at most 64 MiB | a module's state (ADR 0202): one per name in a manifest's `state`, created from the catalogue on every raise, so it exists before its owner runs anywhere; kept when the module is unassigned, because what it holds is data |
Tool calls and heartbeats stay on core NATS: a lost heartbeat is the next heartbeat; a lost tool
call is a timeout the caller already handles.
@@ -247,7 +247,7 @@ expresses this exactly, per subject, and better than a vhost could:
permissions for each consumed event's subject, its tool subjects, and that same inbox prefix.
Nothing else. A module that tries to publish outside its emits is refused by the server, not by
convention.
- **A module's state** (ADR 0201), for whichever principal carries the module — today the machine's
- **A module's state** (ADR 0202), for whichever principal carries the module — today the machine's
runtime, whose grant is the union of its modules': binding to the bucket, reading a key directly,
and an ordered consumer for listing and watching, created and deleted on the bucket's own stream
and nothing else's; and, for the owner's instances only, publishing under the bucket's own
@@ -14,7 +14,7 @@ decisions:
- 02-DECISIONS/0084-which-provider-serves-a-consumer.md
- 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md
- 02-DECISIONS/0038-the-mesh-assigns-the-port.md
- 02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md
- 02-DECISIONS/0201-a-provider-declares-what-it-derives-for-each-consumer.md
---
# 27 — A module requires, the mesh resolves
@@ -208,7 +208,7 @@ the placeholder allows: the definition says which values reach which requirement
does. *How it is checked:* the unit tests named in issue 173, and the plan comparison that closed it.
*A provider says once what it derives for each consumer (2026-10-02,
[ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md),
[ADR 0201](../../02-DECISIONS/0201-a-provider-declares-what-it-derives-for-each-consumer.md),
[issue 124](../../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)):*
where a provider **names the resource** it gives each consumer — a bucket, a database, a vhost — the
name is derived per consumer, and a literal `serves` block could not carry it. A served value may
@@ -221,7 +221,7 @@ its binding's served facts and as `${bound:<provision>:<key>}` in any file it wr
as `derived` on that consumer's entry in its contributions file, so its provisioner is told the name
rather than recomputing it. A consumer that writes the derived value into its own definition instead
of asking for it is refused, naming the placeholder to use. *How it is checked:* the unit tests in
ADR 0202's "how this is checked", each run against the unchanged controller first.
ADR 0201's "how this is checked", each run against the unchanged controller first.
## How a definition reads what was resolved
@@ -11,10 +11,10 @@ code:
- mesh-host internal/apply/apply.go
- mesh-tools src/main.ts
- mesh-catalog modules/mesh-catalog
- mesh-tools node-tools/internal/runtime (a module's state, ADR 0201)
- mesh-tools node-tools/internal/runtime (a module's state, ADR 0202)
updated: 2026-10-04
decisions:
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
@@ -241,7 +241,7 @@ That is the wire-level answer to
[issue 107](../../04-ISSUES/107-a-declaration-carries-no-order/00-report.md).
**A module declares state too.** *Added 2026-10-04,
[ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md).*
[ADR 0202](../../02-DECISIONS/0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md).*
State was the mesh's alone, and modules had the same need with nowhere to put it: an MCP server
registered for every machine, sent as an event, never reached a machine assigned afterwards — its
consumer did not exist yet when the event passed — and a licence binding sent as events replays a
@@ -484,7 +484,7 @@ sealing key leaks, that stream is an archive rather than a moment. So:
it is worst.
**A key-value bucket is a stream, so the same holds for it.** *Added 2026-10-04,
[ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md).*
[ADR 0202](../../02-DECISIONS/0202-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md).*
No secret is put in a module's state, sealed or not: state is exactly what a machine joining a year
later reads in full. A value that needs a secret names it, and the secret travels on request/reply.
Sealed values are plain text to anything inspecting them, so this is checked only partly — the
@@ -2,7 +2,7 @@
status: resolved
opened: 2026-09-26
located-in: [mesh-controller internal/catalogue, mesh-sdk src/provisioner, mesh-catalog modules/minio]
fixed-by: 02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md
fixed-by: 02-DECISIONS/0201-a-provider-declares-what-it-derives-for-each-consumer.md
amended-design: 03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md
---
@@ -64,7 +64,7 @@ compares it to what the provider will actually create. The one wrong instance wa
bucket in its own configuration against the one the provider would create is a check that could
exist today, for any interface, without the mechanism above.
## Answered, 2026-10-02 — [ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md)
## Answered, 2026-10-02 — [ADR 0201](../../02-DECISIONS/0201-a-provider-declares-what-it-derives-for-each-consumer.md)
The channel is the provider's own `serves` block, which may now name the consumer the mesh is
serving: `${consumer:as}` and `${consumer:as:dns}`. The mesh fills it once, where it knows who the
@@ -1,75 +0,0 @@
---
status: open
opened: 2026-10-04
located-in: [mesh-host internal/apply, mesh-controller internal/catalogue]
fixed-by:
amended-design:
---
# 225 — A provisioner cannot read the grant secrets since its code left the container, and every consumer of it is unserved
## What was observed
On the control machine, 2026-10-04, found while looking at why one app was restarting:
```
[mongodb] [provisioner:mongodb-database] mesh_novox_photos: secret not readable yet
(/var/lib/mongodb/grants/novox.photos.secret):
Error: EACCES: permission denied, open '/var/lib/mongodb/grants/novox.photos.secret'
```
**4330 times, every five seconds, since 01:30:20.** The consequence is not a log line: the
provisioner never reads the password, so it never creates the user, so the consumer never
connects —
```
UserNotFound: Could not find user "mesh_novox_photos" for db "admin"
```
— and the app crash-loops. Two consumers on this machine are in that state.
## Why
The grant secrets are what the mesh seals for each consumer and the host unseals beside the
provider's contributions file. They are written `-rw------- root root`, which was right while a
module's own code ran in a container as root.
[ADR 0198](../../02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)
moved a module's long-running code out of its container and under the node's runtime, which runs
as the operator's account. The provisioner is now that account; the secret is still root's. The
timestamps say it exactly: the files are dated 2026-09-26, the first refusal is 01:30:20 on the
day the runtime rolled.
**Nothing reports it.** The machine applies cleanly and reads as current; the provisioner says
`secret not readable yet`, whose wording is for a real and ordinary race on the first pass — the
host has not written the file yet — and which is indistinguishable, in the log, from a permanent
refusal. Four thousand occurrences of a message that means "wait a moment" is the shape to
recognise.
## Why it matters beyond this instance
This is every provider that provisions. The grant secret is the one file the sdk's harness reads
for every consumer, so a provider that cannot read it serves nobody — and says so only in a line
that reads like patience.
It is also the general question the runtime move leaves: **what the mesh seals for a module is
owned for the shape that module's code used to have.** Each module whose code moved is a module
whose files may now be unreadable to it, and ownership is the mesh's to state, not the module's
to work around.
## What this is not
Not caused by [ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md)
or [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md), which landed two
to three hours after the first refusal. Those rebuilt the two affected consumers, which recreated
their containers and made a silent fault visible as a restarting one. The dates are above.
## Open questions
- Who owns a grant secret now — the module's account, as `secrets-owner` already says for a
module's own secrets? Then the host writes it so, and this is a one-line statement in the
declaration rather than a convention.
- Should `secret not readable yet` stop saying "yet" after the first few passes? A message that
is right once and wrong four thousand times is a message that hides its own meaning.
- Which other modules' files did the runtime move leave behind? The sweep is the same question
for every path the mesh writes for a module: directories, bundles, received files.
@@ -1,58 +0,0 @@
---
status: open
opened: 2026-10-04
located-in: [mesh-controller cmd/mesh-controller/collect.go, mesh-controller internal/inventory/collection.go]
fixed-by:
amended-design:
---
# 226 — The store's sweep stops at the first reference recorded with an address, so it collects nothing at all
## What was observed
The first live run of [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md)'s
sweep, 2026-10-04, printed on every build:
```
the artifact store kept 127.0.0.1:5100/mesh-tools/build@sha256:0de48cd3…, so nothing more was
asked of it: 127.0.0.1:5100/mesh-tools/build@sha256:0de48cd3… is not a reference into the
mesh's artifact store
1681 more to collect; the next build asks again
```
Nothing is collected, and nothing ever will be. The store holds 1681 artifacts the mesh no longer
keeps and the feature that exists to remove them is inert.
## Why
Two correct decisions meeting badly.
**A reference recorded before references were kept without an address** is
`127.0.0.1:5100/<path>@sha256:…` rather than `artifact-store://<path>@sha256:…`
([04-ISSUES/102](../102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)).
`LetGo` rightly refuses to compose a delete for a reference whose shape it does not recognise —
that refusal is what keeps the sweep from reaching something that is not the mesh's.
**The sweep stops at the first refusal**, because "a store that refuses one refuses all of them"
— deletion disabled, the store down, the network gone — and pushing through would mean a hundred
identical failures in front of whoever was building something. That reasoning is right for the
store refusing. It is wrong for *this* record being unreadable.
So one old record, early in the oldest-first order, halts the whole sweep for ever.
## Why it matters beyond this instance
**A guard that cannot tell "I will not ask about this" from "it would not answer" stops the wrong
amount of work.** The two deserve opposite responses: skip one, abandon the other. Collapsing
them into "an error" is how a bounded, cautious loop becomes a loop that does nothing — and it
reports the right number while doing it, which is what made it look healthy.
## What a fix has to settle
- A reference the sweep cannot address is **skipped, and the sweep goes on** — it is a fact about
that record, not about the store.
- `Recorded()` already normalises the old form to the kept one, and is what the rest of the mesh
uses for exactly these references. The sweep should normalise before asking rather than refuse.
- Only a refusal *by the store* ends a sweep.
- **How it is checked:** a sweep over records holding one address-recorded reference and one kept
one collects the second; a sweep against a store that refuses stops at the first.
@@ -1,44 +0,0 @@
---
status: open
opened: 2026-10-04
located-in: [mesh-catalog modules/photos]
fixed-by:
amended-design:
---
# 227 — The photo app's admin client asks for port 80, which the reverse proxy holds, so it cannot start
## What was observed
Applying the control machine, 2026-10-04:
```
applying "photos.admin-client": starting container photos-admin-client:
failed to bind host port 0.0.0.0:80/tcp: address already in use
```
Port 80 on that machine belongs to the reverse proxy (`mesh-route-proxy`, confirmed with `ss`),
which is the whole arrangement: the proxy holds the public ports and every module is reached
through it. A module that publishes 80 itself can never start beside it.
Everything else on the machine applied; this one resource fails every pass.
## How it surfaced
`photos` had been pinned at a commit from 2026-09-28 and was rebuilt to `main` on 2026-10-04 —
forced by [ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md)'s
refusal of its transcribed bucket name. The admin client is one of the changes that came with the
rest of `main`. The rebuild did not create the conflict; it delivered it.
**A module pinned months behind carries whatever its branch gained, all at once, the first time
something makes it move.** That is the cost of a pin, and it is paid in full rather than
gradually.
## What a fix has to settle
- Which port the admin client should ask for, or whether it should be reached through the proxy
like everything else and publish nothing.
- Whether a module declaring a port the machine's proxy already holds should be refused when it
is composed, rather than failing on the machine every pass. The mesh assigns ports
([ADR 0038](../../02-DECISIONS/0038-the-mesh-assigns-the-port.md)); a fixed 80 beside a proxy is
a statement it could check.