novox/hq
6
0
Fork 0
Code Issues Pull Requests 6 Releases Wiki Activity

Merge pull request 'Research 036: everything moves over NATS — where the mesh still talks around the bus, and what a view onto the bus needs' (#213) from research/036-everything-over-nats into main

Browse Source
This commit was merged in pull request #213.
This commit is contained in:
mesh-admin
2026-10-08 14:12:02 +00:00
parent 9efa785f38 5e9a27a6a3
commit 91272a8e2c
3 changed files with 530 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-RESEARCH/036-everything-over-nats/00-overview.md
+119
View File
@@ -0,0 +1,119 @@
---
status: active
initiated: 2026-10-08
touches:
- the bus
- 03-DESIGN/01-to-be/25-the-bus-on-nats.md
- 03-DESIGN/01-to-be/32-what-a-module-declares.md
- 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
- 03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md
- 03-DESIGN/00-as-is/14-a-modules-state.md
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- the controller's records
- the catalogue's module manifests
became: []
---
# 036 — Everything moves over NATS
## What is investigated
This effort asks two questions:
1. Where do the mesh's modules still talk to each other around the bus?
2. What does a **view** onto the bus need? A view is a web app or a terminal UI built from the bus
alone. *View* is proposed as a glossary word for this.
The operator set three principles on 2026-10-08:
1. **All communication between modules goes over NATS.** A module may keep its own state. Another
module reading its files, calling it over HTTP or a socket, sharing env files, running `docker exec`
in its container, or using ssh is a gap. Only a channel's outside link, an external API, leaves the
bus.
2. **The approval flow, for a person's decisions, is NATS-native and generic.** Another effort is
designing it. This one lists only how it fits ([evidence §6](01-evidence.md#6-how-the-approval-flow-fits)).
3. **Views are built from the bus.** State lives in key-value buckets, with events that say what
changed, and history lives in streams. A browser reaches the bus by NATS WebSocket and an auth
callout. Every module's important state should therefore be watchable on the bus.
## Why
[Design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 already says "the bus is where the mesh
happens", and that conditions and observed state belong in buckets anything may watch. Two parts of
that are now built:
- **Buckets for a module's state** ([ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)).
- **The controller's conditions**, kept on the bus.
This effort measures how far the rest is from the principles, so that the next decisions start from
evidence.
## What was found
The detail is in [01-evidence.md](01-evidence.md), whose gap table is §5. The target state is in
[02-utopia.md](02-utopia.md). These are the gaps, ranked by how much they stand between the mesh and
the three principles:
1. **Most of the controller's state cannot be watched (G2).**
- Plans, builds, artifacts, node facts, assignments, settings, data items, heals, durations and
licences live in the controller's databases.
- They can be read only by calling a verb again and again.
- Settings change without any event, and `settings_history` holds 0 rows.
- Only the conditions follow the bucket-plus-event pattern a view needs, and they are the template
for the rest.
2. **Nothing outside the private network can reach the bus as a person (G1).**
- There is no WebSocket listener and no auth callout.
- There is one account of password users, composed into a static file and reloaded by signal.
- No grant is shaped for a view.
- Until this exists, no view can be built from the bus.
3. **The bus is one server with one copy of everything (G3).**
- There is no cluster and no leaf node, and every stream and bucket has one replica.
- A restart of the control-node stops every tool call, declaration and event in the mesh.
- A laptop away from the private network has no bus at all, even between its own modules.
4. **Modules reach each other through side doors (G5).**
- `artifact-store-tools` runs `docker exec` in the registry's container.
- The backup holder dumps stores by `docker exec`.
- Two modules hold the container runtime's socket.
- forticlient calls the resolver's verbs over a unix socket.
- route-adapter writes into the proxy's directory.
- Eight media modules carry copied code that calls other apps' HTTP APIs.
5. **Some mesh state and addresses live off the bus (G4).**
- The controller's facts snapshot is kept as a tag in the artifact store and read over HTTP.
- The lab is given the forge's address as a hand-written setting.
- What the registry holds is never announced.
- The bytes of artifacts and git repositories are a reasonable named exception. Their index is not.
6. **The bus does not watch itself (G7).** There is no system account and no dead-letter stream,
although design 25 promises one. These live numbers are unexplained and nothing raises them:
- 2 slow consumers;
- 4 688 JetStream API errors;
- 4 407 connections in 39 hours, against 16 open;
- about 40 writes a second to a delivery bucket of 998 keys.
7. **Smaller gaps:**
- Modules' own databases are invisible to views (G6).
- Buckets keep one value and events last a week, so history is short (G8).
- Lists asked of verbs meet the 1 MiB message limit (G9).
- The controller keeps two unused HTTP ports (G10).
**Much already passes the test.** The forge is wrapped by its module, and pull requests arrive as
events. No module calls the vault. No module uses ssh to reach another. Apps using a database or an
API through a provision are acceptable by the principle.
## What it touches
- **The bus's design and server configuration:** accounts, WebSocket, the callout, leaf nodes,
replicas and the system account.
- **The controller's records**, and which of them it mirrors to buckets and events.
- **What a module declares:** state, history streams, and tools in place of side doors.
- **The conversation with the operator**, where the approval flow meets a view.
## Open questions for the next step
1. **Order.** Which comes first: G2 for the controller, which makes views possible, or G1, which
makes them reachable? This effort suggests G2. It is useful to the mesh's own modules without any
browser.
2. **Accounts for people.** Does a person's view get its own NATS account, with imports from the
mesh account, or stay a user in the one account under a narrower grant? Design 25 §4 chose one
account when only modules were users.
3. **Which module serves the auth callout,** and which login it trusts: the mesh's identity provider,
or the proof of to-be 46 §10.
4. **Whether leaf nodes come before or after a cluster.** The laptop argues for leaf nodes first.
01-RESEARCH/036-everything-over-nats/01-evidence.md
+330
View File
@@ -0,0 +1,330 @@
# 036 — Evidence: where the mesh still talks around the bus
Measured on 2026-10-08, read-only. Three sources were used:
- **The code.** The catalogue (116 module manifests and their code), the media catalogue (12), the
controller, the node-engine, the tool runner and the SDK, all at their trunk heads of that day.
- **The running bus.** It was read through the `nats` module's tools (`nats_server`, `nats_streams`,
`nats_buckets`, `nats_users`, `nats_connections`, `nats_subscriptions`, `nats_backlog`).
- **The running store.** It was read through `mesh-store.databases` and `mesh-store.query`, for table
names and row counts only.
No machine was logged into and nothing was changed.
The test applied throughout is the operator's first principle. **A module may keep its own state, but
another module reaches it only over the bus.** A module that reads another's files, calls it over HTTP
or a socket, shares an env file with it, runs `docker exec` in its container or uses ssh to reach it is
a gap. Only a channel's outside link, an external API, leaves the bus.
## 1. What is already on the bus
This section is the baseline, because most of the mesh already passes the test.
| fact | measured |
|---|---|
| server | one NATS server, release 2.11, on the control-node; 16 connections at the time of reading |
| streams | 21: CONTROL (work queue), NODES and ASSIGNMENTS (last per subject), EVENTS (7 days, 10 000 per subject), two seat work queues, and 15 key-value buckets |
| EVENTS | 183 205 messages, 143 MB, 5 260 distinct subjects, 19 durable consumers |
| key-value buckets | 15, owned by 5 modules: the controller (`calls`, `conditions`, `condition-history`, `hand-acts`, `lease`), mesh-delivery (`deliveries`, `groups`), the messenger (`open`, `sent`), claude-code (`config`, `holdings`, `servers`) and the licence manager (`bindings`), plus a `cancelled` bucket per build seat |
| events declared | 36 of 128 modules declare `emits`; 16 declare `consumes` |
| tool calls | every module tool and seat verb is core request/reply, `mesh.mod.<module>.tool.<tool>` and `mesh.seat.<seat>.tool.<verb>`, announced in the NATS services protocol (ADR 0197) |
| the forge's own state | the `gitea` module wraps the forge. Pull requests become events (`pull.merged`, `pull.updated`, `pull.closed`), and commit statuses are its tools. mesh-delivery reaches the forge only through those tools |
| vault | no module calls the vault. The node-engine unseals a secret into a file, and the vault's export travels in its node's declaration |
| ssh between modules | none. The ssh-client and claude-code modules only parse configuration (`ssh -G`) |
The controller's conditions already live on the bus. They are kept in `mesh-controller_conditions`
(802 keys) and `mesh-controller_condition-history` (90 days), and each change is announced as
`condition-raised`, `condition-changed` or `condition-cleared`. **They are the one complete example
of the pattern a view needs**: current state in a bucket, plus an event that says what changed.
## 2. Communication that leaves the bus
### 2.1 Acceptable by the principle
These are an app using a provision the way it would use any database or API, or the mesh talking to
the world outside:
- **Apps on their own provisions** (about 16 modules). Postgres serves baserow, gitea, keycloak, letta,
mailu, n8n, nextcloud and umami. The other provisions in use are mongo, S3, influx, MQTT, OIDC and
local model access, each through `${bound:<provision>:at}` and `:port`.
- **The route data plane.** About 30 modules are forwarded by the proxy over HTTP.
- **Desktop modules on the display server's socket.** Fifteen modules use it, and it never leaves its
machine.
- **Media apps sharing a library.** Nine modules share the media library directories.
- **External APIs** reached by a channel holder (the messenger's chat service) and by the licence
manager (the model vendor).
### 2.2 Gaps
The gaps are numbered for the table in §5.
**Docker into another module's container, and the container runtime's socket (G5).**
- **`artifact-store-tools`** runs `docker exec` inside the registry's container, which belongs to the
`distribution` module, to read what the registry holds.
- **The `nats` module's own tools** read `accounts.conf` with `docker exec … cat`, and fall back to
`docker exec … wget` when the monitoring port does not answer. These calls stay inside the module,
so they are not a gap.
- **The backup holder (restic)** runs each module's declared `dump` command. For postgres, mongodb,
mssql, nats and supabase, that command is a `docker exec` into the module's container. The command
is the owning module's own declaration, so the module still chooses what is read. The channel is
still a side door, though: the backup holder needs the container runtime's whole power over every
container.
- **The node-engine at genesis** reaches the controller with `docker exec <controller> mesh-controller
<subcommand>` and `docker cp`. It seeds gitea's role in the store with `docker exec <store> psql`,
and it reads the store's and the deprecated broker's administrator secrets from their files.
- **Two modules mount the container runtime's socket:** `build-agent` (read-write) and `supabase`
(read-only). The merge gate's check also mounts it into its check container. Anything holding that
socket can act as any other module's container.
**A unix socket and a shared directory between modules (G5).**
- **forticlient** dials the systemd-resolved module's verb socket to route its domains. That is a
module calling another module's verb without the bus.
- **route-adapter** writes route files into the proxy's (traefik's) watched directory.
**Mesh-written code calling another app's HTTP API (G5).**
- **sonarr, radarr, lidarr and bookshelf** carry identical provisioner code that calls nzbget's,
qbittorrent's and jackett's HTTP APIs to check a connection.
- **The plex API** is called from the tools of sonarr, radarr, lidarr, tautulli and ombi.
- **Status endpoints of the other apps** are called by ombi and bazarr, and home-assistant calls
sonarr's, radarr's and lidarr's.
These are eight modules in all. Each check is a question that the called module could answer as a tool
of its own. Today the code that asks it is copied into each caller.
**The artifact store over HTTP (G4).**
- **The controller** speaks the OCI registry's HTTP API to hold, tag, sweep and delete artifacts.
- **The builder** pushes images and archives, and mirrors upstream images.
- **The node-engine** fetches bundles and archives by URL.
- **The docker module** lists the registry, by address, as an insecure registry.
This traffic is bytes by digest, which a 1 MiB message is not shaped for. Two parts of it are state
rather than bytes, though:
- **The facts snapshot** is the controller's composed record of what the mesh is, which the merge check
judges against. It is kept as a document under an OCI tag in the registry, written and read over
HTTP (to-be 45 §9). It is the controller's state, and it lives in another module's store.
- **What the registry holds** is learnt by `docker exec` (above) or over its HTTP API. Nothing
announces it on the bus.
**The forge reached by address (G4).**
- **The builder, the `records` module and the `lab` module** run `git clone` and `git fetch` against
the forge. The lab is given the forge's address as a setting (`${setting:forge}`) rather than a
provision.
- **The node-engine at genesis** uses the forge's admin HTTP API to create its organisation, team and
user.
- **The builder's package registry** is the forge's npm endpoint, given as a binding.
Git objects are bytes, like artifacts. The lab's setting, however, is an address written by hand,
which is what provisions exist to replace.
**The certificate authority over HTTPS.** The proxy and `ca-trust` reach step-ca's ACME directory and
root certificates by `${bound:internal-acme-ca:…}`. ACME is the protocol the proxy speaks, so this
belongs with §2.1. It is listed here because the trust roots are a mesh fact that could be announced
instead.
**The MCP endpoint over loopback HTTP.** claude-code points the agent at the tool runner's loopback
port. The agent speaks MCP over HTTP, and the tool runner is the bridge onto the bus, so this is the
designed entrance (the mesh MCP server) and not a gap.
**Unused HTTP surfaces in the controller (G10).** The controller still listens on two loopback ports:
- An `api` port, whose authentication is a stub that refuses every request.
- A `board` port.
Neither appears in any design as the way to read the mesh. A view built from the bus replaces both.
**Mesh tool code on postgres (G6).**
- **The controller** opens its three databases (`inventory`, `identity`, `licences`). The merge gate
also creates and drops throwaway databases with an administrator credential.
- **Four modules' tool code** opens its own database directly: `mesh-catalog`, `model-usage`,
`claude-licence-manager` and `showcase`.
Each reads only its own database, which the principle allows. The gap is not the connection. It is
that state kept this way is invisible to anything else except through a verb (§3).
`mesh-store.query` runs one read-only statement against any database the store holds. A future view
could be tempted by it. It is an operator's diagnostic, and it would bypass every module's ownership of
its own data.
## 3. State a view could not watch
A view, in the operator's third principle, reads current state from a bucket, learns that it changed
from an event, and reads history from a stream. The table below says which of the mesh's important
state allows that today.
| state | where it lives | event on change | bucket | otherwise |
|---|---|---|---|---|
| conditions | bus only | raised, changed, cleared | `conditions`, `condition-history` | — |
| hand-acts | bus only | **none** | `hand-acts` | — |
| calls the controller answered | bus only | none | `calls` (2 000 kept, 14 days) | — |
| the lease and epoch | bus and `controller_epoch` | none | `lease` | — |
| deliveries and delivery groups | bus only (mesh-delivery) | `transition`, `group` | `deliveries`, `groups` | — |
| what the operator was told | bus only (messenger) | `refused` | `open`, `sent` | — |
| licence bindings | licence manager's database, mirrored | adopted, refused, failing, usage read | `bindings` | — |
| **plans and walks** | `release_plan` (456 rows) | `plan-moved` | **none** | `plans`, `delivery-walks` verbs |
| **builds and artifacts** | `build` (7 193), `build_gate`, `artifact_collected` (2 312) | `built` (from the build seat), `rolled-back` | **none** | `builds`, `artifacts` verbs |
| **nodes: facts, reports, health** | `node`, `node_report`, `node_health`, `tunnel_peer`, `taken`; the facts snapshot in the registry | `applied` and `refused` per report | **none** | `nodes`, `node`, `status` verbs |
| **assignments, modules, seats** | `assignment` (244), `module` (133), `seat`, `seat_holding`, `port_assignment`, `provision_pin`, `binding` | none; a membership per assignment is kept in the ASSIGNMENTS stream | **none** (ASSIGNMENTS is last-per-subject, but has no event) | `modules`, `seats`, `node` verbs |
| **settings** | `settings` (68), `settings_history` (0 rows) | **none** | **none** | `settings` verb |
| **data items and their readings** | `data_item` (124), `data_reading` (602) | none | none | `data` verb |
| **heals** | `heal` (28) | `healer-acted` | none | `healers` verb |
| **durations** | `duration` (14 617) | none | none | `durations` verb |
| **licences, usage, audit** | licence manager's database: `licence`, `usage`, `audit`, `lease`, `offered` | adopted, failing, usage read | only `bindings` | `licences`, `usage` verbs |
| **credentials and grants** | `bus_user` (76), `secret` (143), `module_secret` (121), `enrolment_token`, `person` | `secret-replaced` only | none | `issue`, `rotate`, `token` verbs |
| **the bus's own health** | the server's monitoring port, on loopback | none | none | the `nats` module's tools |
The most watched state is the open conditions, and it is the one example that is complete. The state
a view of the mesh would show first is what each machine runs, how far a delivery has got, and what is
being built. Each of these can be learnt only by calling a verb again and again. A verb's answer is
also bounded by the server's message limit (§4.4).
**Settings change with no trace.** `settings_history` has 0 rows, and no event is published when a
setting changes. A view cannot show who changed a setting or when, and neither can the controller.
**hand-acts have a bucket but no event.** A view can watch the bucket's keys, but other modules cannot
consume a hand-act through the EVENTS stream the way they consume a condition.
## 4. The bus itself
### 4.1 Accounts, grants and credentials
- **Accounts and users.** The bus has one account (`MESH`), and its users hold passwords. The
controller mints each password and seals it to the recipient. It keeps a bcrypt hash and composes
every user, with explicit publish and subscribe lists and `allow_responses` (one answer, one
minute), into `accounts.conf`.
- **Reload.** The node-engine writes that file into the `nats` module's directory. The module's
entrypoint checks the file's digest every five seconds and sends the server SIGHUP when it changes.
- **What is not configured.** The code has no nkeys, no JWT accounts, no operator and no auth callout.
Design 25 §4 rejected the JWT model on purpose for a single tenant.
- **20 bus accounts were live.** There are three per machine (`node.<machine>`, `<machine>.node-tools`
and `<machine>.build-agent`), plus the controller and the control-node's own module users.
- **The grants are wide.** The tool runner on one machine carries the union of every module assigned
there: up to 170 publish and 599 subscribe grants. What keeps one module's code from acting as
another's on the same machine is the tool runner, not the server.
- **People.** A person's bus account exists in the design (design 25 §7), and the `person` table has
0 rows. The operator reaches the bus through the mesh MCP server, which uses the machine's tool
runner account.
**What a view in a browser would need, and is missing:**
- **A WebSocket listener.** None is configured. The server configuration has `port`, `http`, `tls`,
`jetstream` and `include` and nothing else.
- **A way to admit a person** with a short-lived credential and a grant derived from what that person
may see. The NATS mechanism for this is the auth callout, and it is not configured. Without it, each
browser would need a password in the static file, and the file would be rewritten and reloaded for
each login.
- **A grant shape for a view:** read a set of buckets and subscribe to a set of event subjects. No such
grant is composed today. The nearest are the module state grants of ADR 0201.
### 4.2 One server, one copy
- **No cluster, no leaf nodes, no replicas.** The server runs with none of these. No stream or bucket
sets replicas, so every one has a single copy (the server reports `ha_assets: 0`).
- **The control-node is a single point of failure for every link.** All 16 connections cross the
private network to that one machine, at round trips of 33 to 94 ms from the other machines.
- **What stops when it restarts.** Every tool call, declaration, report and event stops. Design 25
§11 names leaf nodes as "deliberately out of scope; noted so it is not forgotten".
- **Backups.** JetStream state is protected by a nightly snapshot of each stream through the snapshot
API (ADR 0235). The live state between two snapshots has one copy, on one disk.
- **The laptop.** It travels and sleeps behind a VPN client. While it is away from the private network
it has no bus at all, even for tool calls between modules on itself.
### 4.3 Observability
- **No system account.** The bus has none, so nothing hears the server's own events about other
users: slow consumers, permission violations, connections and disconnections. The controller hears
only two JetStream advisories, `CONSUMER.MAX_DELIVERIES` and `CONSUMER.DELETED`.
- **Who reads the monitoring port.** The monitoring port (8222) is published on the control-node's
loopback. Only the `nats` module's tools read it, when asked. Nothing turns what it says into a
condition.
Numbers from the live bus that nothing currently raises:
- **2 slow consumers** since the server started 1 day 14 hours earlier.
- **4 688 JetStream API errors** out of 6.08 million calls.
- **4 407 connections** in those 39 hours, against 16 open. The tool runners' connections were 2 to 39
minutes old when read. *I think* a tool runner reconnects often, because its connection age was far
below the server's uptime; the cause is not yet known. Reading the tool runner's journal for
disconnect reasons would find out.
- **About 5.7 million messages** delivered on the subscription to `$KV.mesh-delivery_deliveries.>` in
those 39 hours. That is about 40 a second, for a bucket of 998 keys. *I think* mesh-delivery rewrites
unchanged deliveries on every pass. Comparing each value's revision with its content would confirm
it. Every view that watches the bucket would receive the same stream.
Two further findings:
- **No dead-letter stream.** Design 25 §3 says a message that exceeds `max-deliver` feeds
`mesh.events.dead`, its own small stream. No such stream exists. The advisory is heard by the
controller and kept nowhere else.
- **Stale consumers.** Four durable consumers, one per machine's claude-code, filter on
`claude-licence-manager` events `licence.rotated` and `licence.switched`. No manifest emits either,
and none of the four has delivered a message. The bindings bucket replaced them (ADR 0201). ADR 0230
leaves deleting a consumer to a person.
### 4.4 The message limit
- **The limit.** `max_payload` is the server default, 1 MiB. No configuration sets it.
- **Issue 314.** Answers above the limit were dropped in the sender (issue 314). Both the controller
and the tool runner now page a large answer. The holder keeps the pages for five minutes, and the
caller asks again with a `Mesh-Page` header.
- **Trimming.** Verbs that grow, such as `conditions`, `status`, the self-check and the registry
listings, are also trimmed.
- **Other ceilings.** A module's state value is capped at 256 KiB and a bucket at 64 MiB. A kept call
is capped at 68 KB.
- **What this means for a view.** A view that reads a bucket per key never meets the limit. A view
that asks a verb for a whole list does.
### 4.5 Retention
- **Buckets keep one value.** Every bucket keeps history 1: the newest value per key, and no earlier
ones.
- **Events last a week.** EVENTS keeps 7 days and at most 10 000 messages per subject. Condition
history keeps 90 days and hand-acts keep 90 days.
- **No longer record.** No stream keeps what happened beyond those windows. A view of "what happened
last month" has the store's tables (builds, plans, durations) and nothing on the bus.
## 5. The gap table
The effort column is a rough size: **S** is a pull request or two, **M** is a design section and a few
pull requests across repositories, and **L** is a decision record, a design and work in several
repositories.
| # | gap | where | evidence | impact | suggested direction | effort |
|---|---|---|---|---|---|---|
| G1 | No way for a browser or a person's client to reach the bus | the `nats` module's server configuration; the controller's account composition | no WebSocket listener; no auth callout; one account of static password users; `person` 0 rows | no view onto the bus can be built; every view today would need a backend that reads verbs | a WebSocket listener behind the proxy; an auth callout served by a mesh module that checks the operator's identity provider and issues a short-lived user with a view-shaped grant | L |
| G2 | Most of the controller's state is not watchable | the controller's `inventory` and `licences` databases | §3: plans, builds, artifacts, nodes, assignments, settings, data, heals, durations, licences have no bucket; settings and assignments have no event | a view must poll verbs; nothing else in the mesh can react to a plan, a node's facts or a setting changing; settings changes leave no trace | for each, a bucket of current state written on change and an event naming what changed, the store remaining the controller's record; conditions are the template | M each, L in total |
| G3 | One bus server, one copy of everything | the `nats` module on the control-node | no cluster, no leaf nodes, replicas 1 everywhere, `ha_assets: 0` | a restart of one machine stops every link in the mesh; the laptop away from the private network has no bus at all | a leaf node per machine, so local traffic and tool calls survive the hub; later a three-node cluster with replicated streams for what must not be lost | L |
| G4 | Mesh state and addresses kept off the bus | the facts snapshot in the registry; the forge address as the lab's setting; the registry's contents | §2.2: the snapshot is an OCI tag over HTTP; `${setting:forge}`; the registry is read by `docker exec` or HTTP | the merge check depends on the registry for the controller's state; a hand-written address does not follow the forge | the facts snapshot in a bucket or JetStream object store; the forge as a provision; the registry announcing what it holds as events. The bytes of artifacts and git stay off the bus as a named exception | S–M |
| G5 | Side doors between modules | `artifact-store-tools`, restic's dumps, the node-engine's genesis, `build-agent` and `supabase` on the container runtime's socket, forticlient's socket, route-adapter's files, 8 media modules' HTTP checks | §2.2 | a module acts on another's container, socket or files with no grant the bus can check and nothing announced; code is copied between modules | each owning module serves the act as a tool (`distribution` lists what it holds, each store module serves its dump, the resolver's verbs over the bus, each download client answers "is this reachable with this key"); the socket kept only where building containers is the module's job | M |
| G6 | State behind a module's own database, seen only through its verbs | `claude-licence-manager`, `model-usage`, `mesh-catalog`, `showcase` | each opens its own postgres; licences, usage and audit have no bucket | allowed by the principle, but invisible to a view | the same pattern as G2, per module: a declared bucket (ADR 0201) for the current state a person would look at | S per module |
| G7 | The bus does not watch itself | the `nats` module; the controller's advisories | no system account; no dead-letter stream; 2 slow consumers, 4 688 API errors, 4 407 connections in 39 h, unexplained; four consumers with nothing to consume | a failing or churning link is noticed only when someone asks | a system-account user for the `nats` module that turns the server's own events into conditions; the dead-letter stream design 25 promises; the reconnect rate and the delivery bucket's write rate investigated | S–M |
| G8 | History is short and in the store | every bucket keeps history 1; EVENTS keeps 7 days | §4.5 | a view of what happened beyond a week reads tables, not the bus | a history stream per kind of change that a view needs (deliveries, plans, builds, hand-acts), with its own retention, declared like a bucket | M |
| G9 | Answers are bounded by one message | the server's default `max_payload` | 1 MiB; paging since issue 314 | a view that asks verbs for lists pages; one that reads buckets does not | let views read buckets, not verbs; keep paging for verbs | S |
| G10 | Unused HTTP surfaces | the controller's `api` and `board` ports | `api` refuses every request; neither is named in a design | dead code that suggests a second way in | remove both once a view exists | S |
## 6. How the approval flow fits
Another effort is designing the approval flow: a person's decision, asked and answered on the bus,
generic over what is being approved. It continues to-be 46 §6 (asks) and §10 (asks that authorise)
under ADR 0234 and ADR 0258. This effort does not duplicate it. It notes only where it meets the
three principles:
- **An open ask is current state.** It belongs in a bucket, keyed by the ask, so a view lists what
waits for the operator and watches it change. The messenger's `open` bucket is the precedent.
- **An answer is an event.** The proof it carries is checked by the controller, never by the view
that collected it (ADR 0258: a click proves only that something at the session chose).
- **A view that lets the operator answer** needs G1's person-shaped credential. That credential lets
the controller tell the operator's own answer from anything else that holds the session.
## 7. What this effort did not read
- **The TypeScript of the tool runner** (`node-tools/src`). The Go part was read.
- **Each media app's own code beyond its provisioner and tools.** The apps themselves are upstream
software and are judged by §2.1.
- **Live traffic per subject**, beyond the subscription counters the server keeps.
01-RESEARCH/036-everything-over-nats/02-utopia.md
+81
View File
@@ -0,0 +1,81 @@
# 036 — The target: a mesh that happens on the bus
This is where the gaps in [01-evidence.md](01-evidence.md) point. It is a picture to test proposals
against, not a design. Each paragraph names the gap it answers.
## Every link between modules is a subject
A module owns its state: its files, its database, its container. It is reached only by its tools, its
seat's verbs, its events and its buckets.
- **No module acts on another's container, socket or files.** A store module serves its own dump. The
registry says what it holds. The resolver's verbs are called over the bus like any other seat's. A
download client answers whether it is reachable with a key (G5).
- **Every act is grantable.** Because every act is a subject, the bus can refuse it, and the controller
can say who may perform it.
The bus is not for bulk bytes, and two kinds of traffic stay off it by name:
- **Bulk bytes stay where they are.** The bytes of an artifact and of a git repository, and an app
using its database, keep their own protocols.
- **Who reaches them is decided on the bus.** Where to reach them is a provision, and what exists is
announced on the bus (G4).
## Every important state is watchable
For each thing a person would look at, the owner keeps three things on the bus:
1. **A bucket** of current state, one key per thing: a node, an assignment, a plan, a build, a
delivery, a setting, a licence, an open ask.
2. **An event** for each change, naming the thing and what changed. Another module can consume it, and
a view can redraw from it.
3. **A history stream**, where the past matters, with a retention the owner declares (G8).
The store remains the owner's record where it needs one. The bucket is written from it on change, never
instead of it. The open conditions already work this way. Plans, builds, nodes, assignments, settings
and licences come next (G2, G6). A verb is still how something is asked or done. It stops being the
only way to see.
## Views are built from the bus
A view is a web page or a terminal UI that shows part of the mesh. *View* is proposed here as the
glossary word for this. It has no backend of its own:
- **It connects** to the bus over WebSocket, through the proxy, as the person using it.
- **It reads** the buckets it shows, watches them, and subscribes to the events that explain a change.
- **It acts** by calling the same tools and verbs an agent calls. A decision travels as an answer to
an ask, carrying its proof, and the controller checks it (the approval flow, §6 of the evidence).
The person is admitted by an **auth callout** served by a mesh module. That module checks the person's
login with the mesh's identity provider and issues a short-lived bus user. The user's grant is derived
from the view's declaration: these buckets to read, these events to hear, these tools to call. No
password is written into the server's file for a person, and a person's access ends when the credential
expires (G1).
## The bus survives a machine
Every machine runs a **leaf node**:
- **Local traffic stays local.** A module's connection is local, and tool calls between modules on one
machine keep working when the hub is away, as they must on a laptop behind a VPN client.
- **What must not be lost is replicated.** The control-node's server is one of three in a cluster, and
the declarations, the control stream and the controller's buckets are kept on three disks rather than
one. The nightly snapshot remains the guard against our own mistakes, not against a disk (G3).
## The bus watches itself
- **Server events become conditions.** The `nats` module holds a system-account user and turns the
server's own events into conditions, like any other probe: a slow consumer, a permission refused, a
client reconnecting in a loop, a consumer that nothing feeds.
- **Exhausted messages are kept.** A message delivered as often as allowed lands in a dead-letter stream
that a view can show (G7).
- **Nothing to rescue by paging.** Answers stay within one message because views read keys, not
whole lists (G9).
## What is not wanted
- **No view with a database of its own** that copies the mesh's state. That copy drifts, which is the
thing the mesh exists to prevent.
- **No HTTP API on the controller beside the bus** (G10).
- **No second signing hierarchy** for its own sake. The callout issues users under the account the mesh
already has, unless a separate account for people turns out to be the simpler grant.