From 5e9a27a6a326e241a5d48af0f270c91be6ce02f4 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 15:28:57 +0200 Subject: [PATCH] Research 036: measure where the mesh talks around the bus, so views can be built from it --- .../036-everything-over-nats/00-overview.md | 119 +++++++ .../036-everything-over-nats/01-evidence.md | 330 ++++++++++++++++++ .../036-everything-over-nats/02-utopia.md | 81 +++++ 3 files changed, 530 insertions(+) create mode 100644 01-RESEARCH/036-everything-over-nats/00-overview.md create mode 100644 01-RESEARCH/036-everything-over-nats/01-evidence.md create mode 100644 01-RESEARCH/036-everything-over-nats/02-utopia.md diff --git a/01-RESEARCH/036-everything-over-nats/00-overview.md b/01-RESEARCH/036-everything-over-nats/00-overview.md new file mode 100644 index 00000000..48d1cf8a --- /dev/null +++ b/01-RESEARCH/036-everything-over-nats/00-overview.md @@ -0,0 +1,119 @@ +--- +status: active +initiated: 2026-10-08 +touches: + - the bus + - 03-DESIGN/01-to-be/25-the-bus-on-nats.md + - 03-DESIGN/01-to-be/32-what-a-module-declares.md + - 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md + - 03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md + - 03-DESIGN/00-as-is/14-a-modules-state.md + - 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md + - the controller's records + - the catalogue's module manifests +became: [] +--- + +# 036 — Everything moves over NATS + +## What is investigated + +This effort asks two questions: + +1. Where do the mesh's modules still talk to each other around the bus? +2. What does a **view** onto the bus need? A view is a web app or a terminal UI built from the bus + alone. *View* is proposed as a glossary word for this. + +The operator set three principles on 2026-10-08: + +1. **All communication between modules goes over NATS.** A module may keep its own state. Another + module reading its files, calling it over HTTP or a socket, sharing env files, running `docker exec` + in its container, or using ssh is a gap. Only a channel's outside link, an external API, leaves the + bus. +2. **The approval flow, for a person's decisions, is NATS-native and generic.** Another effort is + designing it. This one lists only how it fits ([evidence §6](01-evidence.md#6-how-the-approval-flow-fits)). +3. **Views are built from the bus.** State lives in key-value buckets, with events that say what + changed, and history lives in streams. A browser reaches the bus by NATS WebSocket and an auth + callout. Every module's important state should therefore be watchable on the bus. + +## Why + +[Design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 already says "the bus is where the mesh +happens", and that conditions and observed state belong in buckets anything may watch. Two parts of +that are now built: + +- **Buckets for a module's state** ([ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)). +- **The controller's conditions**, kept on the bus. + +This effort measures how far the rest is from the principles, so that the next decisions start from +evidence. + +## What was found + +The detail is in [01-evidence.md](01-evidence.md), whose gap table is §5. The target state is in +[02-utopia.md](02-utopia.md). These are the gaps, ranked by how much they stand between the mesh and +the three principles: + +1. **Most of the controller's state cannot be watched (G2).** + - Plans, builds, artifacts, node facts, assignments, settings, data items, heals, durations and + licences live in the controller's databases. + - They can be read only by calling a verb again and again. + - Settings change without any event, and `settings_history` holds 0 rows. + - Only the conditions follow the bucket-plus-event pattern a view needs, and they are the template + for the rest. +2. **Nothing outside the private network can reach the bus as a person (G1).** + - There is no WebSocket listener and no auth callout. + - There is one account of password users, composed into a static file and reloaded by signal. + - No grant is shaped for a view. + - Until this exists, no view can be built from the bus. +3. **The bus is one server with one copy of everything (G3).** + - There is no cluster and no leaf node, and every stream and bucket has one replica. + - A restart of the control-node stops every tool call, declaration and event in the mesh. + - A laptop away from the private network has no bus at all, even between its own modules. +4. **Modules reach each other through side doors (G5).** + - `artifact-store-tools` runs `docker exec` in the registry's container. + - The backup holder dumps stores by `docker exec`. + - Two modules hold the container runtime's socket. + - forticlient calls the resolver's verbs over a unix socket. + - route-adapter writes into the proxy's directory. + - Eight media modules carry copied code that calls other apps' HTTP APIs. +5. **Some mesh state and addresses live off the bus (G4).** + - The controller's facts snapshot is kept as a tag in the artifact store and read over HTTP. + - The lab is given the forge's address as a hand-written setting. + - What the registry holds is never announced. + - The bytes of artifacts and git repositories are a reasonable named exception. Their index is not. +6. **The bus does not watch itself (G7).** There is no system account and no dead-letter stream, + although design 25 promises one. These live numbers are unexplained and nothing raises them: + - 2 slow consumers; + - 4 688 JetStream API errors; + - 4 407 connections in 39 hours, against 16 open; + - about 40 writes a second to a delivery bucket of 998 keys. +7. **Smaller gaps:** + - Modules' own databases are invisible to views (G6). + - Buckets keep one value and events last a week, so history is short (G8). + - Lists asked of verbs meet the 1 MiB message limit (G9). + - The controller keeps two unused HTTP ports (G10). + +**Much already passes the test.** The forge is wrapped by its module, and pull requests arrive as +events. No module calls the vault. No module uses ssh to reach another. Apps using a database or an +API through a provision are acceptable by the principle. + +## What it touches + +- **The bus's design and server configuration:** accounts, WebSocket, the callout, leaf nodes, + replicas and the system account. +- **The controller's records**, and which of them it mirrors to buckets and events. +- **What a module declares:** state, history streams, and tools in place of side doors. +- **The conversation with the operator**, where the approval flow meets a view. + +## Open questions for the next step + +1. **Order.** Which comes first: G2 for the controller, which makes views possible, or G1, which + makes them reachable? This effort suggests G2. It is useful to the mesh's own modules without any + browser. +2. **Accounts for people.** Does a person's view get its own NATS account, with imports from the + mesh account, or stay a user in the one account under a narrower grant? Design 25 §4 chose one + account when only modules were users. +3. **Which module serves the auth callout,** and which login it trusts: the mesh's identity provider, + or the proof of to-be 46 §10. +4. **Whether leaf nodes come before or after a cluster.** The laptop argues for leaf nodes first. diff --git a/01-RESEARCH/036-everything-over-nats/01-evidence.md b/01-RESEARCH/036-everything-over-nats/01-evidence.md new file mode 100644 index 00000000..9e682df3 --- /dev/null +++ b/01-RESEARCH/036-everything-over-nats/01-evidence.md @@ -0,0 +1,330 @@ +# 036 — Evidence: where the mesh still talks around the bus + +Measured on 2026-10-08, read-only. Three sources were used: + +- **The code.** The catalogue (116 module manifests and their code), the media catalogue (12), the + controller, the node-engine, the tool runner and the SDK, all at their trunk heads of that day. +- **The running bus.** It was read through the `nats` module's tools (`nats_server`, `nats_streams`, + `nats_buckets`, `nats_users`, `nats_connections`, `nats_subscriptions`, `nats_backlog`). +- **The running store.** It was read through `mesh-store.databases` and `mesh-store.query`, for table + names and row counts only. + +No machine was logged into and nothing was changed. + +The test applied throughout is the operator's first principle. **A module may keep its own state, but +another module reaches it only over the bus.** A module that reads another's files, calls it over HTTP +or a socket, shares an env file with it, runs `docker exec` in its container or uses ssh to reach it is +a gap. Only a channel's outside link, an external API, leaves the bus. + +## 1. What is already on the bus + +This section is the baseline, because most of the mesh already passes the test. + +| fact | measured | +|---|---| +| server | one NATS server, release 2.11, on the control-node; 16 connections at the time of reading | +| streams | 21: CONTROL (work queue), NODES and ASSIGNMENTS (last per subject), EVENTS (7 days, 10 000 per subject), two seat work queues, and 15 key-value buckets | +| EVENTS | 183 205 messages, 143 MB, 5 260 distinct subjects, 19 durable consumers | +| key-value buckets | 15, owned by 5 modules: the controller (`calls`, `conditions`, `condition-history`, `hand-acts`, `lease`), mesh-delivery (`deliveries`, `groups`), the messenger (`open`, `sent`), claude-code (`config`, `holdings`, `servers`) and the licence manager (`bindings`), plus a `cancelled` bucket per build seat | +| events declared | 36 of 128 modules declare `emits`; 16 declare `consumes` | +| tool calls | every module tool and seat verb is core request/reply, `mesh.mod..tool.` and `mesh.seat..tool.`, announced in the NATS services protocol (ADR 0197) | +| the forge's own state | the `gitea` module wraps the forge. Pull requests become events (`pull.merged`, `pull.updated`, `pull.closed`), and commit statuses are its tools. mesh-delivery reaches the forge only through those tools | +| vault | no module calls the vault. The node-engine unseals a secret into a file, and the vault's export travels in its node's declaration | +| ssh between modules | none. The ssh-client and claude-code modules only parse configuration (`ssh -G`) | + +The controller's conditions already live on the bus. They are kept in `mesh-controller_conditions` +(802 keys) and `mesh-controller_condition-history` (90 days), and each change is announced as +`condition-raised`, `condition-changed` or `condition-cleared`. **They are the one complete example +of the pattern a view needs**: current state in a bucket, plus an event that says what changed. + +## 2. Communication that leaves the bus + +### 2.1 Acceptable by the principle + +These are an app using a provision the way it would use any database or API, or the mesh talking to +the world outside: + +- **Apps on their own provisions** (about 16 modules). Postgres serves baserow, gitea, keycloak, letta, + mailu, n8n, nextcloud and umami. The other provisions in use are mongo, S3, influx, MQTT, OIDC and + local model access, each through `${bound::at}` and `:port`. +- **The route data plane.** About 30 modules are forwarded by the proxy over HTTP. +- **Desktop modules on the display server's socket.** Fifteen modules use it, and it never leaves its + machine. +- **Media apps sharing a library.** Nine modules share the media library directories. +- **External APIs** reached by a channel holder (the messenger's chat service) and by the licence + manager (the model vendor). + +### 2.2 Gaps + +The gaps are numbered for the table in §5. + +**Docker into another module's container, and the container runtime's socket (G5).** + +- **`artifact-store-tools`** runs `docker exec` inside the registry's container, which belongs to the + `distribution` module, to read what the registry holds. +- **The `nats` module's own tools** read `accounts.conf` with `docker exec … cat`, and fall back to + `docker exec … wget` when the monitoring port does not answer. These calls stay inside the module, + so they are not a gap. +- **The backup holder (restic)** runs each module's declared `dump` command. For postgres, mongodb, + mssql, nats and supabase, that command is a `docker exec` into the module's container. The command + is the owning module's own declaration, so the module still chooses what is read. The channel is + still a side door, though: the backup holder needs the container runtime's whole power over every + container. +- **The node-engine at genesis** reaches the controller with `docker exec mesh-controller + ` and `docker cp`. It seeds gitea's role in the store with `docker exec psql`, + and it reads the store's and the deprecated broker's administrator secrets from their files. +- **Two modules mount the container runtime's socket:** `build-agent` (read-write) and `supabase` + (read-only). The merge gate's check also mounts it into its check container. Anything holding that + socket can act as any other module's container. + +**A unix socket and a shared directory between modules (G5).** + +- **forticlient** dials the systemd-resolved module's verb socket to route its domains. That is a + module calling another module's verb without the bus. +- **route-adapter** writes route files into the proxy's (traefik's) watched directory. + +**Mesh-written code calling another app's HTTP API (G5).** + +- **sonarr, radarr, lidarr and bookshelf** carry identical provisioner code that calls nzbget's, + qbittorrent's and jackett's HTTP APIs to check a connection. +- **The plex API** is called from the tools of sonarr, radarr, lidarr, tautulli and ombi. +- **Status endpoints of the other apps** are called by ombi and bazarr, and home-assistant calls + sonarr's, radarr's and lidarr's. + +These are eight modules in all. Each check is a question that the called module could answer as a tool +of its own. Today the code that asks it is copied into each caller. + +**The artifact store over HTTP (G4).** + +- **The controller** speaks the OCI registry's HTTP API to hold, tag, sweep and delete artifacts. +- **The builder** pushes images and archives, and mirrors upstream images. +- **The node-engine** fetches bundles and archives by URL. +- **The docker module** lists the registry, by address, as an insecure registry. + +This traffic is bytes by digest, which a 1 MiB message is not shaped for. Two parts of it are state +rather than bytes, though: + +- **The facts snapshot** is the controller's composed record of what the mesh is, which the merge check + judges against. It is kept as a document under an OCI tag in the registry, written and read over + HTTP (to-be 45 §9). It is the controller's state, and it lives in another module's store. +- **What the registry holds** is learnt by `docker exec` (above) or over its HTTP API. Nothing + announces it on the bus. + +**The forge reached by address (G4).** + +- **The builder, the `records` module and the `lab` module** run `git clone` and `git fetch` against + the forge. The lab is given the forge's address as a setting (`${setting:forge}`) rather than a + provision. +- **The node-engine at genesis** uses the forge's admin HTTP API to create its organisation, team and + user. +- **The builder's package registry** is the forge's npm endpoint, given as a binding. + +Git objects are bytes, like artifacts. The lab's setting, however, is an address written by hand, +which is what provisions exist to replace. + +**The certificate authority over HTTPS.** The proxy and `ca-trust` reach step-ca's ACME directory and +root certificates by `${bound:internal-acme-ca:…}`. ACME is the protocol the proxy speaks, so this +belongs with §2.1. It is listed here because the trust roots are a mesh fact that could be announced +instead. + +**The MCP endpoint over loopback HTTP.** claude-code points the agent at the tool runner's loopback +port. The agent speaks MCP over HTTP, and the tool runner is the bridge onto the bus, so this is the +designed entrance (the mesh MCP server) and not a gap. + +**Unused HTTP surfaces in the controller (G10).** The controller still listens on two loopback ports: + +- An `api` port, whose authentication is a stub that refuses every request. +- A `board` port. + +Neither appears in any design as the way to read the mesh. A view built from the bus replaces both. + +**Mesh tool code on postgres (G6).** + +- **The controller** opens its three databases (`inventory`, `identity`, `licences`). The merge gate + also creates and drops throwaway databases with an administrator credential. +- **Four modules' tool code** opens its own database directly: `mesh-catalog`, `model-usage`, + `claude-licence-manager` and `showcase`. + +Each reads only its own database, which the principle allows. The gap is not the connection. It is +that state kept this way is invisible to anything else except through a verb (§3). + +`mesh-store.query` runs one read-only statement against any database the store holds. A future view +could be tempted by it. It is an operator's diagnostic, and it would bypass every module's ownership of +its own data. + +## 3. State a view could not watch + +A view, in the operator's third principle, reads current state from a bucket, learns that it changed +from an event, and reads history from a stream. The table below says which of the mesh's important +state allows that today. + +| state | where it lives | event on change | bucket | otherwise | +|---|---|---|---|---| +| conditions | bus only | raised, changed, cleared | `conditions`, `condition-history` | — | +| hand-acts | bus only | **none** | `hand-acts` | — | +| calls the controller answered | bus only | none | `calls` (2 000 kept, 14 days) | — | +| the lease and epoch | bus and `controller_epoch` | none | `lease` | — | +| deliveries and delivery groups | bus only (mesh-delivery) | `transition`, `group` | `deliveries`, `groups` | — | +| what the operator was told | bus only (messenger) | `refused` | `open`, `sent` | — | +| licence bindings | licence manager's database, mirrored | adopted, refused, failing, usage read | `bindings` | — | +| **plans and walks** | `release_plan` (456 rows) | `plan-moved` | **none** | `plans`, `delivery-walks` verbs | +| **builds and artifacts** | `build` (7 193), `build_gate`, `artifact_collected` (2 312) | `built` (from the build seat), `rolled-back` | **none** | `builds`, `artifacts` verbs | +| **nodes: facts, reports, health** | `node`, `node_report`, `node_health`, `tunnel_peer`, `taken`; the facts snapshot in the registry | `applied` and `refused` per report | **none** | `nodes`, `node`, `status` verbs | +| **assignments, modules, seats** | `assignment` (244), `module` (133), `seat`, `seat_holding`, `port_assignment`, `provision_pin`, `binding` | none; a membership per assignment is kept in the ASSIGNMENTS stream | **none** (ASSIGNMENTS is last-per-subject, but has no event) | `modules`, `seats`, `node` verbs | +| **settings** | `settings` (68), `settings_history` (0 rows) | **none** | **none** | `settings` verb | +| **data items and their readings** | `data_item` (124), `data_reading` (602) | none | none | `data` verb | +| **heals** | `heal` (28) | `healer-acted` | none | `healers` verb | +| **durations** | `duration` (14 617) | none | none | `durations` verb | +| **licences, usage, audit** | licence manager's database: `licence`, `usage`, `audit`, `lease`, `offered` | adopted, failing, usage read | only `bindings` | `licences`, `usage` verbs | +| **credentials and grants** | `bus_user` (76), `secret` (143), `module_secret` (121), `enrolment_token`, `person` | `secret-replaced` only | none | `issue`, `rotate`, `token` verbs | +| **the bus's own health** | the server's monitoring port, on loopback | none | none | the `nats` module's tools | + +The most watched state is the open conditions, and it is the one example that is complete. The state +a view of the mesh would show first is what each machine runs, how far a delivery has got, and what is +being built. Each of these can be learnt only by calling a verb again and again. A verb's answer is +also bounded by the server's message limit (§4.4). + +**Settings change with no trace.** `settings_history` has 0 rows, and no event is published when a +setting changes. A view cannot show who changed a setting or when, and neither can the controller. + +**hand-acts have a bucket but no event.** A view can watch the bucket's keys, but other modules cannot +consume a hand-act through the EVENTS stream the way they consume a condition. + +## 4. The bus itself + +### 4.1 Accounts, grants and credentials + +- **Accounts and users.** The bus has one account (`MESH`), and its users hold passwords. The + controller mints each password and seals it to the recipient. It keeps a bcrypt hash and composes + every user, with explicit publish and subscribe lists and `allow_responses` (one answer, one + minute), into `accounts.conf`. +- **Reload.** The node-engine writes that file into the `nats` module's directory. The module's + entrypoint checks the file's digest every five seconds and sends the server SIGHUP when it changes. +- **What is not configured.** The code has no nkeys, no JWT accounts, no operator and no auth callout. + Design 25 §4 rejected the JWT model on purpose for a single tenant. +- **20 bus accounts were live.** There are three per machine (`node.`, `.node-tools` + and `.build-agent`), plus the controller and the control-node's own module users. +- **The grants are wide.** The tool runner on one machine carries the union of every module assigned + there: up to 170 publish and 599 subscribe grants. What keeps one module's code from acting as + another's on the same machine is the tool runner, not the server. +- **People.** A person's bus account exists in the design (design 25 §7), and the `person` table has + 0 rows. The operator reaches the bus through the mesh MCP server, which uses the machine's tool + runner account. + +**What a view in a browser would need, and is missing:** + +- **A WebSocket listener.** None is configured. The server configuration has `port`, `http`, `tls`, + `jetstream` and `include` and nothing else. +- **A way to admit a person** with a short-lived credential and a grant derived from what that person + may see. The NATS mechanism for this is the auth callout, and it is not configured. Without it, each + browser would need a password in the static file, and the file would be rewritten and reloaded for + each login. +- **A grant shape for a view:** read a set of buckets and subscribe to a set of event subjects. No such + grant is composed today. The nearest are the module state grants of ADR 0201. + +### 4.2 One server, one copy + +- **No cluster, no leaf nodes, no replicas.** The server runs with none of these. No stream or bucket + sets replicas, so every one has a single copy (the server reports `ha_assets: 0`). +- **The control-node is a single point of failure for every link.** All 16 connections cross the + private network to that one machine, at round trips of 33 to 94 ms from the other machines. +- **What stops when it restarts.** Every tool call, declaration, report and event stops. Design 25 + §11 names leaf nodes as "deliberately out of scope; noted so it is not forgotten". +- **Backups.** JetStream state is protected by a nightly snapshot of each stream through the snapshot + API (ADR 0235). The live state between two snapshots has one copy, on one disk. +- **The laptop.** It travels and sleeps behind a VPN client. While it is away from the private network + it has no bus at all, even for tool calls between modules on itself. + +### 4.3 Observability + +- **No system account.** The bus has none, so nothing hears the server's own events about other + users: slow consumers, permission violations, connections and disconnections. The controller hears + only two JetStream advisories, `CONSUMER.MAX_DELIVERIES` and `CONSUMER.DELETED`. +- **Who reads the monitoring port.** The monitoring port (8222) is published on the control-node's + loopback. Only the `nats` module's tools read it, when asked. Nothing turns what it says into a + condition. + +Numbers from the live bus that nothing currently raises: + +- **2 slow consumers** since the server started 1 day 14 hours earlier. +- **4 688 JetStream API errors** out of 6.08 million calls. +- **4 407 connections** in those 39 hours, against 16 open. The tool runners' connections were 2 to 39 + minutes old when read. *I think* a tool runner reconnects often, because its connection age was far + below the server's uptime; the cause is not yet known. Reading the tool runner's journal for + disconnect reasons would find out. +- **About 5.7 million messages** delivered on the subscription to `$KV.mesh-delivery_deliveries.>` in + those 39 hours. That is about 40 a second, for a bucket of 998 keys. *I think* mesh-delivery rewrites + unchanged deliveries on every pass. Comparing each value's revision with its content would confirm + it. Every view that watches the bucket would receive the same stream. + +Two further findings: + +- **No dead-letter stream.** Design 25 §3 says a message that exceeds `max-deliver` feeds + `mesh.events.dead`, its own small stream. No such stream exists. The advisory is heard by the + controller and kept nowhere else. +- **Stale consumers.** Four durable consumers, one per machine's claude-code, filter on + `claude-licence-manager` events `licence.rotated` and `licence.switched`. No manifest emits either, + and none of the four has delivered a message. The bindings bucket replaced them (ADR 0201). ADR 0230 + leaves deleting a consumer to a person. + +### 4.4 The message limit + +- **The limit.** `max_payload` is the server default, 1 MiB. No configuration sets it. +- **Issue 314.** Answers above the limit were dropped in the sender (issue 314). Both the controller + and the tool runner now page a large answer. The holder keeps the pages for five minutes, and the + caller asks again with a `Mesh-Page` header. +- **Trimming.** Verbs that grow, such as `conditions`, `status`, the self-check and the registry + listings, are also trimmed. +- **Other ceilings.** A module's state value is capped at 256 KiB and a bucket at 64 MiB. A kept call + is capped at 68 KB. +- **What this means for a view.** A view that reads a bucket per key never meets the limit. A view + that asks a verb for a whole list does. + +### 4.5 Retention + +- **Buckets keep one value.** Every bucket keeps history 1: the newest value per key, and no earlier + ones. +- **Events last a week.** EVENTS keeps 7 days and at most 10 000 messages per subject. Condition + history keeps 90 days and hand-acts keep 90 days. +- **No longer record.** No stream keeps what happened beyond those windows. A view of "what happened + last month" has the store's tables (builds, plans, durations) and nothing on the bus. + +## 5. The gap table + +The effort column is a rough size: **S** is a pull request or two, **M** is a design section and a few +pull requests across repositories, and **L** is a decision record, a design and work in several +repositories. + +| # | gap | where | evidence | impact | suggested direction | effort | +|---|---|---|---|---|---|---| +| G1 | No way for a browser or a person's client to reach the bus | the `nats` module's server configuration; the controller's account composition | no WebSocket listener; no auth callout; one account of static password users; `person` 0 rows | no view onto the bus can be built; every view today would need a backend that reads verbs | a WebSocket listener behind the proxy; an auth callout served by a mesh module that checks the operator's identity provider and issues a short-lived user with a view-shaped grant | L | +| G2 | Most of the controller's state is not watchable | the controller's `inventory` and `licences` databases | §3: plans, builds, artifacts, nodes, assignments, settings, data, heals, durations, licences have no bucket; settings and assignments have no event | a view must poll verbs; nothing else in the mesh can react to a plan, a node's facts or a setting changing; settings changes leave no trace | for each, a bucket of current state written on change and an event naming what changed, the store remaining the controller's record; conditions are the template | M each, L in total | +| G3 | One bus server, one copy of everything | the `nats` module on the control-node | no cluster, no leaf nodes, replicas 1 everywhere, `ha_assets: 0` | a restart of one machine stops every link in the mesh; the laptop away from the private network has no bus at all | a leaf node per machine, so local traffic and tool calls survive the hub; later a three-node cluster with replicated streams for what must not be lost | L | +| G4 | Mesh state and addresses kept off the bus | the facts snapshot in the registry; the forge address as the lab's setting; the registry's contents | §2.2: the snapshot is an OCI tag over HTTP; `${setting:forge}`; the registry is read by `docker exec` or HTTP | the merge check depends on the registry for the controller's state; a hand-written address does not follow the forge | the facts snapshot in a bucket or JetStream object store; the forge as a provision; the registry announcing what it holds as events. The bytes of artifacts and git stay off the bus as a named exception | S–M | +| G5 | Side doors between modules | `artifact-store-tools`, restic's dumps, the node-engine's genesis, `build-agent` and `supabase` on the container runtime's socket, forticlient's socket, route-adapter's files, 8 media modules' HTTP checks | §2.2 | a module acts on another's container, socket or files with no grant the bus can check and nothing announced; code is copied between modules | each owning module serves the act as a tool (`distribution` lists what it holds, each store module serves its dump, the resolver's verbs over the bus, each download client answers "is this reachable with this key"); the socket kept only where building containers is the module's job | M | +| G6 | State behind a module's own database, seen only through its verbs | `claude-licence-manager`, `model-usage`, `mesh-catalog`, `showcase` | each opens its own postgres; licences, usage and audit have no bucket | allowed by the principle, but invisible to a view | the same pattern as G2, per module: a declared bucket (ADR 0201) for the current state a person would look at | S per module | +| G7 | The bus does not watch itself | the `nats` module; the controller's advisories | no system account; no dead-letter stream; 2 slow consumers, 4 688 API errors, 4 407 connections in 39 h, unexplained; four consumers with nothing to consume | a failing or churning link is noticed only when someone asks | a system-account user for the `nats` module that turns the server's own events into conditions; the dead-letter stream design 25 promises; the reconnect rate and the delivery bucket's write rate investigated | S–M | +| G8 | History is short and in the store | every bucket keeps history 1; EVENTS keeps 7 days | §4.5 | a view of what happened beyond a week reads tables, not the bus | a history stream per kind of change that a view needs (deliveries, plans, builds, hand-acts), with its own retention, declared like a bucket | M | +| G9 | Answers are bounded by one message | the server's default `max_payload` | 1 MiB; paging since issue 314 | a view that asks verbs for lists pages; one that reads buckets does not | let views read buckets, not verbs; keep paging for verbs | S | +| G10 | Unused HTTP surfaces | the controller's `api` and `board` ports | `api` refuses every request; neither is named in a design | dead code that suggests a second way in | remove both once a view exists | S | + +## 6. How the approval flow fits + +Another effort is designing the approval flow: a person's decision, asked and answered on the bus, +generic over what is being approved. It continues to-be 46 §6 (asks) and §10 (asks that authorise) +under ADR 0234 and ADR 0258. This effort does not duplicate it. It notes only where it meets the +three principles: + +- **An open ask is current state.** It belongs in a bucket, keyed by the ask, so a view lists what + waits for the operator and watches it change. The messenger's `open` bucket is the precedent. +- **An answer is an event.** The proof it carries is checked by the controller, never by the view + that collected it (ADR 0258: a click proves only that something at the session chose). +- **A view that lets the operator answer** needs G1's person-shaped credential. That credential lets + the controller tell the operator's own answer from anything else that holds the session. + +## 7. What this effort did not read + +- **The TypeScript of the tool runner** (`node-tools/src`). The Go part was read. +- **Each media app's own code beyond its provisioner and tools.** The apps themselves are upstream + software and are judged by §2.1. +- **Live traffic per subject**, beyond the subscription counters the server keeps. diff --git a/01-RESEARCH/036-everything-over-nats/02-utopia.md b/01-RESEARCH/036-everything-over-nats/02-utopia.md new file mode 100644 index 00000000..6bebff90 --- /dev/null +++ b/01-RESEARCH/036-everything-over-nats/02-utopia.md @@ -0,0 +1,81 @@ +# 036 — The target: a mesh that happens on the bus + +This is where the gaps in [01-evidence.md](01-evidence.md) point. It is a picture to test proposals +against, not a design. Each paragraph names the gap it answers. + +## Every link between modules is a subject + +A module owns its state: its files, its database, its container. It is reached only by its tools, its +seat's verbs, its events and its buckets. + +- **No module acts on another's container, socket or files.** A store module serves its own dump. The + registry says what it holds. The resolver's verbs are called over the bus like any other seat's. A + download client answers whether it is reachable with a key (G5). +- **Every act is grantable.** Because every act is a subject, the bus can refuse it, and the controller + can say who may perform it. + +The bus is not for bulk bytes, and two kinds of traffic stay off it by name: + +- **Bulk bytes stay where they are.** The bytes of an artifact and of a git repository, and an app + using its database, keep their own protocols. +- **Who reaches them is decided on the bus.** Where to reach them is a provision, and what exists is + announced on the bus (G4). + +## Every important state is watchable + +For each thing a person would look at, the owner keeps three things on the bus: + +1. **A bucket** of current state, one key per thing: a node, an assignment, a plan, a build, a + delivery, a setting, a licence, an open ask. +2. **An event** for each change, naming the thing and what changed. Another module can consume it, and + a view can redraw from it. +3. **A history stream**, where the past matters, with a retention the owner declares (G8). + +The store remains the owner's record where it needs one. The bucket is written from it on change, never +instead of it. The open conditions already work this way. Plans, builds, nodes, assignments, settings +and licences come next (G2, G6). A verb is still how something is asked or done. It stops being the +only way to see. + +## Views are built from the bus + +A view is a web page or a terminal UI that shows part of the mesh. *View* is proposed here as the +glossary word for this. It has no backend of its own: + +- **It connects** to the bus over WebSocket, through the proxy, as the person using it. +- **It reads** the buckets it shows, watches them, and subscribes to the events that explain a change. +- **It acts** by calling the same tools and verbs an agent calls. A decision travels as an answer to + an ask, carrying its proof, and the controller checks it (the approval flow, §6 of the evidence). + +The person is admitted by an **auth callout** served by a mesh module. That module checks the person's +login with the mesh's identity provider and issues a short-lived bus user. The user's grant is derived +from the view's declaration: these buckets to read, these events to hear, these tools to call. No +password is written into the server's file for a person, and a person's access ends when the credential +expires (G1). + +## The bus survives a machine + +Every machine runs a **leaf node**: + +- **Local traffic stays local.** A module's connection is local, and tool calls between modules on one + machine keep working when the hub is away, as they must on a laptop behind a VPN client. +- **What must not be lost is replicated.** The control-node's server is one of three in a cluster, and + the declarations, the control stream and the controller's buckets are kept on three disks rather than + one. The nightly snapshot remains the guard against our own mistakes, not against a disk (G3). + +## The bus watches itself + +- **Server events become conditions.** The `nats` module holds a system-account user and turns the + server's own events into conditions, like any other probe: a slow consumer, a permission refused, a + client reconnecting in a loop, a consumer that nothing feeds. +- **Exhausted messages are kept.** A message delivered as often as allowed lands in a dead-letter stream + that a view can show (G7). +- **Nothing to rescue by paging.** Answers stay within one message because views read keys, not + whole lists (G9). + +## What is not wanted + +- **No view with a database of its own** that copies the mesh's state. That copy drifts, which is the + thing the mesh exists to prevent. +- **No HTTP API on the controller beside the bus** (G10). +- **No second signing hierarchy** for its own sake. The callout issues users under the account the mesh + already has, unless a separate account for people turns out to be the simpler grant.