hq/01-RESEARCH/036-everything-over-nats/01-evidence.md

036 — Evidence: where the mesh still talks around the bus

Measured on 2026-10-08, read-only. Three sources were used:

• The code. The catalogue (116 module manifests and their code), the media catalogue (12), the controller, the node-engine, the tool runner and the SDK, all at their trunk heads of that day.
• The running bus. It was read through the nats module's tools (nats_server, nats_streams, nats_buckets, nats_users, nats_connections, nats_subscriptions, nats_backlog).
• The running store. It was read through mesh-store.databases and mesh-store.query, for table names and row counts only.

No machine was logged into and nothing was changed.

The test applied throughout is the operator's first principle. A module may keep its own state, but another module reaches it only over the bus. A module that reads another's files, calls it over HTTP or a socket, shares an env file with it, runs docker exec in its container or uses ssh to reach it is a gap. Only a channel's outside link, an external API, leaves the bus.

1. What is already on the bus

This section is the baseline, because most of the mesh already passes the test.

fact	measured
server	one NATS server, release 2.11, on the control-node; 16 connections at the time of reading
streams	21: CONTROL (work queue), NODES and ASSIGNMENTS (last per subject), EVENTS (7 days, 10 000 per subject), two seat work queues, and 15 key-value buckets
EVENTS	183 205 messages, 143 MB, 5 260 distinct subjects, 19 durable consumers
key-value buckets	15, owned by 5 modules: the controller (calls, conditions, condition-history, hand-acts, lease), mesh-delivery (deliveries, groups), the messenger (open, sent), claude-code (config, holdings, servers) and the licence manager (bindings), plus a cancelled bucket per build seat
events declared	36 of 128 modules declare emits; 16 declare consumes
tool calls	every module tool and seat verb is core request/reply, mesh.mod.<module>.tool.<tool> and mesh.seat.<seat>.tool.<verb>, announced in the NATS services protocol (ADR 0197)
the forge's own state	the gitea module wraps the forge. Pull requests become events (pull.merged, pull.updated, pull.closed), and commit statuses are its tools. mesh-delivery reaches the forge only through those tools
vault	no module calls the vault. The node-engine unseals a secret into a file, and the vault's export travels in its node's declaration
ssh between modules	none. The ssh-client and claude-code modules only parse configuration (ssh -G)

The controller's conditions already live on the bus. They are kept in mesh-controller_conditions (802 keys) and mesh-controller_condition-history (90 days), and each change is announced as condition-raised, condition-changed or condition-cleared. They are the one complete example of the pattern a view needs: current state in a bucket, plus an event that says what changed.

2. Communication that leaves the bus

2.1 Acceptable by the principle

These are an app using a provision the way it would use any database or API, or the mesh talking to the world outside:

• Apps on their own provisions (about 16 modules). Postgres serves baserow, gitea, keycloak, letta, mailu, n8n, nextcloud and umami. The other provisions in use are mongo, S3, influx, MQTT, OIDC and local model access, each through ${bound:<provision>:at} and :port.
• The route data plane. About 30 modules are forwarded by the proxy over HTTP.
• Desktop modules on the display server's socket. Fifteen modules use it, and it never leaves its machine.
• Media apps sharing a library. Nine modules share the media library directories.
• External APIs reached by a channel holder (the messenger's chat service) and by the licence manager (the model vendor).

2.2 Gaps

The gaps are numbered for the table in §5.

Docker into another module's container, and the container runtime's socket (G5).

• artifact-store-tools runs docker exec inside the registry's container, which belongs to the distribution module, to read what the registry holds.
• The nats module's own tools read accounts.conf with docker exec … cat, and fall back to docker exec … wget when the monitoring port does not answer. These calls stay inside the module, so they are not a gap.
• The backup holder (restic) runs each module's declared dump command. For postgres, mongodb, mssql, nats and supabase, that command is a docker exec into the module's container. The command is the owning module's own declaration, so the module still chooses what is read. The channel is still a side door, though: the backup holder needs the container runtime's whole power over every container.
• The node-engine at genesis reaches the controller with docker exec <controller> mesh-controller <subcommand> and docker cp. It seeds gitea's role in the store with docker exec <store> psql, and it reads the store's and the deprecated broker's administrator secrets from their files.
• Two modules mount the container runtime's socket: build-agent (read-write) and supabase (read-only). The merge gate's check also mounts it into its check container. Anything holding that socket can act as any other module's container.

A unix socket and a shared directory between modules (G5).

• forticlient dials the systemd-resolved module's verb socket to route its domains. That is a module calling another module's verb without the bus.
• route-adapter writes route files into the proxy's (traefik's) watched directory.

Mesh-written code calling another app's HTTP API (G5).

• sonarr, radarr, lidarr and bookshelf carry identical provisioner code that calls nzbget's, qbittorrent's and jackett's HTTP APIs to check a connection.
• The plex API is called from the tools of sonarr, radarr, lidarr, tautulli and ombi.
• Status endpoints of the other apps are called by ombi and bazarr, and home-assistant calls sonarr's, radarr's and lidarr's.

These are eight modules in all. Each check is a question that the called module could answer as a tool of its own. Today the code that asks it is copied into each caller.

The artifact store over HTTP (G4).

• The controller speaks the OCI registry's HTTP API to hold, tag, sweep and delete artifacts.
• The builder pushes images and archives, and mirrors upstream images.
• The node-engine fetches bundles and archives by URL.
• The docker module lists the registry, by address, as an insecure registry.

This traffic is bytes by digest, which a 1 MiB message is not shaped for. Two parts of it are state rather than bytes, though:

• The facts snapshot is the controller's composed record of what the mesh is, which the merge check judges against. It is kept as a document under an OCI tag in the registry, written and read over HTTP (to-be 45 §9). It is the controller's state, and it lives in another module's store.
• What the registry holds is learnt by docker exec (above) or over its HTTP API. Nothing announces it on the bus.

The forge reached by address (G4).

• The builder, the records module and the lab module run git clone and git fetch against the forge. The lab is given the forge's address as a setting (${setting:forge}) rather than a provision.
• The node-engine at genesis uses the forge's admin HTTP API to create its organisation, team and user.
• The builder's package registry is the forge's npm endpoint, given as a binding.

Git objects are bytes, like artifacts. The lab's setting, however, is an address written by hand, which is what provisions exist to replace.

The certificate authority over HTTPS. The proxy and ca-trust reach step-ca's ACME directory and root certificates by ${bound:internal-acme-ca:…}. ACME is the protocol the proxy speaks, so this belongs with §2.1. It is listed here because the trust roots are a mesh fact that could be announced instead.

The MCP endpoint over loopback HTTP. claude-code points the agent at the tool runner's loopback port. The agent speaks MCP over HTTP, and the tool runner is the bridge onto the bus, so this is the designed entrance (the mesh MCP server) and not a gap.

Unused HTTP surfaces in the controller (G10). The controller still listens on two loopback ports:

• An api port, whose authentication is a stub that refuses every request.
• A board port.

Neither appears in any design as the way to read the mesh. A view built from the bus replaces both.

Mesh tool code on postgres (G6).

• The controller opens its three databases (inventory, identity, licences). The merge gate also creates and drops throwaway databases with an administrator credential.
• Four modules' tool code opens its own database directly: mesh-catalog, model-usage, claude-licence-manager and showcase.

Each reads only its own database, which the principle allows. The gap is not the connection. It is that state kept this way is invisible to anything else except through a verb (§3).

mesh-store.query runs one read-only statement against any database the store holds. A future view could be tempted by it. It is an operator's diagnostic, and it would bypass every module's ownership of its own data.

3. State a view could not watch

A view, in the operator's third principle, reads current state from a bucket, learns that it changed from an event, and reads history from a stream. The table below says which of the mesh's important state allows that today.

state	where it lives	event on change	bucket	otherwise
conditions	bus only	raised, changed, cleared	conditions, condition-history	—
hand-acts	bus only	none	hand-acts	—
calls the controller answered	bus only	none	calls (2 000 kept, 14 days)	—
the lease and epoch	bus and controller_epoch	none	lease	—
deliveries and delivery groups	bus only (mesh-delivery)	transition, group	deliveries, groups	—
what the operator was told	bus only (messenger)	refused	open, sent	—
licence bindings	licence manager's database, mirrored	adopted, refused, failing, usage read	bindings	—
plans and walks	release_plan (456 rows)	plan-moved	none	plans, delivery-walks verbs
builds and artifacts	build (7 193), build_gate, artifact_collected (2 312)	built (from the build seat), rolled-back	none	builds, artifacts verbs
nodes: facts, reports, health	node, node_report, node_health, tunnel_peer, taken; the facts snapshot in the registry	applied and refused per report	none	nodes, node, status verbs
assignments, modules, seats	assignment (244), module (133), seat, seat_holding, port_assignment, provision_pin, binding	none; a membership per assignment is kept in the ASSIGNMENTS stream	none (ASSIGNMENTS is last-per-subject, but has no event)	modules, seats, node verbs
settings	settings (68), settings_history (0 rows)	none	none	settings verb
data items and their readings	data_item (124), data_reading (602)	none	none	data verb
heals	heal (28)	healer-acted	none	healers verb
durations	duration (14 617)	none	none	durations verb
licences, usage, audit	licence manager's database: licence, usage, audit, lease, offered	adopted, failing, usage read	only bindings	licences, usage verbs
credentials and grants	bus_user (76), secret (143), module_secret (121), enrolment_token, person	secret-replaced only	none	issue, rotate, token verbs
the bus's own health	the server's monitoring port, on loopback	none	none	the nats module's tools

The most watched state is the open conditions, and it is the one example that is complete. The state a view of the mesh would show first is what each machine runs, how far a delivery has got, and what is being built. Each of these can be learnt only by calling a verb again and again. A verb's answer is also bounded by the server's message limit (§4.4).

Settings change with no trace. settings_history has 0 rows, and no event is published when a setting changes. A view cannot show who changed a setting or when, and neither can the controller.

hand-acts have a bucket but no event. A view can watch the bucket's keys, but other modules cannot consume a hand-act through the EVENTS stream the way they consume a condition.

4. The bus itself

4.1 Accounts, grants and credentials

• Accounts and users. The bus has one account (MESH), and its users hold passwords. The controller mints each password and seals it to the recipient. It keeps a bcrypt hash and composes every user, with explicit publish and subscribe lists and allow_responses (one answer, one minute), into accounts.conf.
• Reload. The node-engine writes that file into the nats module's directory. The module's entrypoint checks the file's digest every five seconds and sends the server SIGHUP when it changes.
• What is not configured. The code has no nkeys, no JWT accounts, no operator and no auth callout. Design 25 §4 rejected the JWT model on purpose for a single tenant.
• 20 bus accounts were live. There are three per machine (node.<machine>, <machine>.node-tools and <machine>.build-agent), plus the controller and the control-node's own module users.
• The grants are wide. The tool runner on one machine carries the union of every module assigned there: up to 170 publish and 599 subscribe grants. What keeps one module's code from acting as another's on the same machine is the tool runner, not the server.
• People. A person's bus account exists in the design (design 25 §7), and the person table has 0 rows. The operator reaches the bus through the mesh MCP server, which uses the machine's tool runner account.

What a view in a browser would need, and is missing:

• A WebSocket listener. None is configured. The server configuration has port, http, tls, jetstream and include and nothing else.
• A way to admit a person with a short-lived credential and a grant derived from what that person may see. The NATS mechanism for this is the auth callout, and it is not configured. Without it, each browser would need a password in the static file, and the file would be rewritten and reloaded for each login.
• A grant shape for a view: read a set of buckets and subscribe to a set of event subjects. No such grant is composed today. The nearest are the module state grants of ADR 0201.

4.2 One server, one copy

• No cluster, no leaf nodes, no replicas. The server runs with none of these. No stream or bucket sets replicas, so every one has a single copy (the server reports ha_assets: 0).
• The control-node is a single point of failure for every link. All 16 connections cross the private network to that one machine, at round trips of 33 to 94 ms from the other machines.
• What stops when it restarts. Every tool call, declaration, report and event stops. Design 25 §11 names leaf nodes as "deliberately out of scope; noted so it is not forgotten".
• Backups. JetStream state is protected by a nightly snapshot of each stream through the snapshot API (ADR 0235). The live state between two snapshots has one copy, on one disk.
• The laptop. It travels and sleeps behind a VPN client. While it is away from the private network it has no bus at all, even for tool calls between modules on itself.

4.3 Observability

• No system account. The bus has none, so nothing hears the server's own events about other users: slow consumers, permission violations, connections and disconnections. The controller hears only two JetStream advisories, CONSUMER.MAX_DELIVERIES and CONSUMER.DELETED.
• Who reads the monitoring port. The monitoring port (8222) is published on the control-node's loopback. Only the nats module's tools read it, when asked. Nothing turns what it says into a condition.

Numbers from the live bus that nothing currently raises:

• 2 slow consumers since the server started 1 day 14 hours earlier.
• 4 688 JetStream API errors out of 6.08 million calls.
• 4 407 connections in those 39 hours, against 16 open. The tool runners' connections were 2 to 39 minutes old when read. I think a tool runner reconnects often, because its connection age was far below the server's uptime; the cause is not yet known. Reading the tool runner's journal for disconnect reasons would find out.
• About 5.7 million messages delivered on the subscription to $KV.mesh-delivery_deliveries.> in those 39 hours. That is about 40 a second, for a bucket of 998 keys. I think mesh-delivery rewrites unchanged deliveries on every pass. Comparing each value's revision with its content would confirm it. Every view that watches the bucket would receive the same stream.

Two further findings:

• No dead-letter stream. Design 25 §3 says a message that exceeds max-deliver feeds mesh.events.dead, its own small stream. No such stream exists. The advisory is heard by the controller and kept nowhere else.
• Stale consumers. Four durable consumers, one per machine's claude-code, filter on claude-licence-manager events licence.rotated and licence.switched. No manifest emits either, and none of the four has delivered a message. The bindings bucket replaced them (ADR 0201). ADR 0230 leaves deleting a consumer to a person.

4.4 The message limit

• The limit. max_payload is the server default, 1 MiB. No configuration sets it.
• Issue 314. Answers above the limit were dropped in the sender (issue 314). Both the controller and the tool runner now page a large answer. The holder keeps the pages for five minutes, and the caller asks again with a Mesh-Page header.
• Trimming. Verbs that grow, such as conditions, status, the self-check and the registry listings, are also trimmed.
• Other ceilings. A module's state value is capped at 256 KiB and a bucket at 64 MiB. A kept call is capped at 68 KB.
• What this means for a view. A view that reads a bucket per key never meets the limit. A view that asks a verb for a whole list does.

4.5 Retention

• Buckets keep one value. Every bucket keeps history 1: the newest value per key, and no earlier ones.
• Events last a week. EVENTS keeps 7 days and at most 10 000 messages per subject. Condition history keeps 90 days and hand-acts keep 90 days.
• No longer record. No stream keeps what happened beyond those windows. A view of "what happened last month" has the store's tables (builds, plans, durations) and nothing on the bus.

5. The gap table

The effort column is a rough size: S is a pull request or two, M is a design section and a few pull requests across repositories, and L is a decision record, a design and work in several repositories.

#	gap	where	evidence	impact	suggested direction	effort
G1	No way for a browser or a person's client to reach the bus	the nats module's server configuration; the controller's account composition	no WebSocket listener; no auth callout; one account of static password users; person 0 rows	no view onto the bus can be built; every view today would need a backend that reads verbs	a WebSocket listener behind the proxy; an auth callout served by a mesh module that checks the operator's identity provider and issues a short-lived user with a view-shaped grant	L
G2	Most of the controller's state is not watchable	the controller's inventory and licences databases	§3: plans, builds, artifacts, nodes, assignments, settings, data, heals, durations, licences have no bucket; settings and assignments have no event	a view must poll verbs; nothing else in the mesh can react to a plan, a node's facts or a setting changing; settings changes leave no trace	for each, a bucket of current state written on change and an event naming what changed, the store remaining the controller's record; conditions are the template	M each, L in total
G3	One bus server, one copy of everything	the nats module on the control-node	no cluster, no leaf nodes, replicas 1 everywhere, ha_assets: 0	a restart of one machine stops every link in the mesh; the laptop away from the private network has no bus at all	a leaf node per machine, so local traffic and tool calls survive the hub; later a three-node cluster with replicated streams for what must not be lost	L
G4	Mesh state and addresses kept off the bus	the facts snapshot in the registry; the forge address as the lab's setting; the registry's contents	§2.2: the snapshot is an OCI tag over HTTP; ${setting:forge}; the registry is read by docker exec or HTTP	the merge check depends on the registry for the controller's state; a hand-written address does not follow the forge	the facts snapshot in a bucket or JetStream object store; the forge as a provision; the registry announcing what it holds as events. The bytes of artifacts and git stay off the bus as a named exception	S–M
G5	Side doors between modules	artifact-store-tools, restic's dumps, the node-engine's genesis, build-agent and supabase on the container runtime's socket, forticlient's socket, route-adapter's files, 8 media modules' HTTP checks	§2.2	a module acts on another's container, socket or files with no grant the bus can check and nothing announced; code is copied between modules	each owning module serves the act as a tool (distribution lists what it holds, each store module serves its dump, the resolver's verbs over the bus, each download client answers "is this reachable with this key"); the socket kept only where building containers is the module's job	M
G6	State behind a module's own database, seen only through its verbs	claude-licence-manager, model-usage, mesh-catalog, showcase	each opens its own postgres; licences, usage and audit have no bucket	allowed by the principle, but invisible to a view	the same pattern as G2, per module: a declared bucket (ADR 0201) for the current state a person would look at	S per module
G7	The bus does not watch itself	the nats module; the controller's advisories	no system account; no dead-letter stream; 2 slow consumers, 4 688 API errors, 4 407 connections in 39 h, unexplained; four consumers with nothing to consume	a failing or churning link is noticed only when someone asks	a system-account user for the nats module that turns the server's own events into conditions; the dead-letter stream design 25 promises; the reconnect rate and the delivery bucket's write rate investigated	S–M
G8	History is short and in the store	every bucket keeps history 1; EVENTS keeps 7 days	§4.5	a view of what happened beyond a week reads tables, not the bus	a history stream per kind of change that a view needs (deliveries, plans, builds, hand-acts), with its own retention, declared like a bucket	M
G9	Answers are bounded by one message	the server's default max_payload	1 MiB; paging since issue 314	a view that asks verbs for lists pages; one that reads buckets does not	let views read buckets, not verbs; keep paging for verbs	S
G10	Unused HTTP surfaces	the controller's api and board ports	api refuses every request; neither is named in a design	dead code that suggests a second way in	remove both once a view exists	S

6. How the approval flow fits

Another effort is designing the approval flow: a person's decision, asked and answered on the bus, generic over what is being approved. It continues to-be 46 §6 (asks) and §10 (asks that authorise) under ADR 0234 and ADR 0258. This effort does not duplicate it. It notes only where it meets the three principles:

• An open ask is current state. It belongs in a bucket, keyed by the ask, so a view lists what waits for the operator and watches it change. The messenger's open bucket is the precedent.
• An answer is an event. The proof it carries is checked by the controller, never by the view that collected it (ADR 0258: a click proves only that something at the session chose).
• A view that lets the operator answer needs G1's person-shaped credential. That credential lets the controller tell the operator's own answer from anything else that holds the session.

7. What this effort did not read

• The TypeScript of the tool runner (node-tools/src). The Go part was read.
• Each media app's own code beyond its provisioner and tools. The apps themselves are upstream software and are judged by §2.1.
• Live traffic per subject, beyond the subscription counters the server keeps.