Files
hq/03-DESIGN/01-to-be/38-building-the-operators-machine.md
T

353 lines
26 KiB
Markdown

---
layer: to-be
status: in-progress
code: [mesh-tools, mesh-controller, mesh-host, mesh-catalog]
updated: 2026-10-03
decisions:
- 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
- 02-DECISIONS/0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md
- 02-DECISIONS/0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md
- 02-DECISIONS/0176-the-login-shell-is-a-node-seat-and-execute-is-its-contract.md
- 02-DECISIONS/0177-a-unit-may-be-user-scoped-and-the-service-manager-is-a-node-seat.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md
- 02-DECISIONS/0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md
- 02-DECISIONS/0192-a-tools-bundle-declares-what-it-is-given-and-the-runtime-hands-it-to-that-bundle-alone.md
- 02-DECISIONS/0193-every-bundle-the-runtime-serves-is-launched-and-the-runtime-knows-no-language.md
- 02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md
---
# 38. Building the operator's machine
**The work of [design 37](37-the-operators-machine.md), broken into packages small enough that each
ends at something a person can see run, in the order their dependencies allow.** Design 37 is the
authority on *what* is built; this document holds only the packages, their order, their sizes and
their proofs, and is wrong the moment it disagrees with 37 rather than the other way round. It is
the shape [design 28](28-building-the-bus.md) gave the bus work, applied here.
## How this is built, and where it is run
**On the live mesh, by the operator's decision.** Every package is written with unit tests and
committed on one branch per repository; its proof runs on the four machines, not in the lab.
[ADR 0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md) already says the live mesh is
the test bed; the operator's words on 2026-10-02 were *skip the lab, it is not too bad if something
is broken*. The cost accepted: a package that breaks the runtime breaks every tool on a node until
the next push, and the controller's own verbs stay reachable through the controller seat whatever
happens to a node's runtime — which is the one thing that must hold, and does by construction
([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)).
Each package names what proves it. A package that cannot name its proof is divided until it can.
## What exists already, measured
Counted 2026-10-02 in the four repositories, non-test source. The point of the count is the same
as design 28's: nothing here is new ground; every package reshapes something standing.
| Piece | Today | Size | Becomes |
|---|---|---|---|
| the tool runtime | TypeScript: loads `MESH_TOOL_MODULES`, serves one module's tools and its claimed seats' verbs; `serve` is the console | ~1 700 lines over six files | loads every assigned module's bundle; `serve` is node tools |
| the host's `process` shape | Go: fetch a bundle by digest, unpack under the mesh's daemons directory, write the unit, run it | 343 lines | **unchanged** — the runtime is one such process |
| the host's `archive` shape | Go: fetch and unpack an artifact at a path | 185 lines | **unchanged** — a module's tools bundle is one such archive |
| the controller's bus principals | Go: one principal per module per node, grants from what it declares | 132 lines | gains one principal per node for the runtime |
| the controller's memberships | Go: one per assignment, the subjects a runtime serves | 143 lines | **unchanged** in shape; the runtime reads several |
| the controller's declaration composer | Go, one file | 2 053 lines | gains the runtime's process, the bundles' archives, two env words |
| the catalogue | 35 manifests build a per-module tool container on the runtime's base image | — | none do; the runtime is a module of its own |
**Two measurements decide the shape.** The host needs no change: a `process` and an `archive` are
what the runtime and a bundle are, and both are applied today. And the runtime already does
nine-tenths of the job — the loop over entrypoints, the seat verbs, the membership subscription —
for one module; the work is to let it do the same for a list.
## The order the work allows
```
WP1 the runtime serves many modules (mesh-tools) ──┐
WP2 the controller composes one runtime a node (mesh-controller) ──┤ independent, test-proven
│
WP3 the runtime is a module; the console is its serving mode (mesh-tools, mesh-catalog)
│
WP4 the first holder moves: the packet filter (mesh-catalog) ── the live proof
│
WP5 the shell, on a server (mesh-catalog) ── the first environment module live
WP6 the service manager, on a workstation (mesh-host #72, mesh-catalog)
│
WP7 the login manager, the display server, the window manager … ── one record per seat, after this document
WP8 settings for the theme knobs ── after issue 168 closes
```
WP1 and WP2 touch different repositories and meet only at the membership's shape, which neither
changes; they are built in parallel. WP3 needs both. WP4 is the first time anything on a machine
changes, and it is the proof of the whole. WP5 and WP6 are the first environment modules; the
packages after them are design 37 §4's candidates and are not broken down here, because each
begins with a decision record this document cannot anticipate.
## WP1 — The runtime serves many modules
*mesh-tools. About a day.*
**What changes.** `serve` takes a list of modules to serve, each with its entrypoints, rather than
one module and one credential. The runtime reads one membership per module from the subjects
[ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)
derives for each, and serves each module's tools on that module's subjects and each held seat's
verbs on the seat's. The filter that drops a registration under any name but the one module goes;
what remains is the rule that a registration under a seat's name is served only where some module
the runtime serves claims that seat. A bundle that throws on import is named in the log and in
what `tools` answers, and the others serve. The runtime reads `MESH_OPERATOR_ACCOUNT` and
`MESH_OPERATOR_HOME` and hands them to every tool's environment.
**What does not change.** The SDK. The broker client. The MCP surface. A module's tool code.
**Proof.** The runtime's test against a real bus: three bundles, one of which throws on import;
five tools and two seat verbs answer on their subjects; `tools` names the failed bundle; a
membership republished mid-run re-subscribes without a restart.
*Built and proven 2026-10-02* (mesh-tools, branch `feat/the-operators-machine`, commit `6390d1d`).
**WP1b — the launcher beside the loader** ([ADR 0188](../../02-DECISIONS/0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md)).
*mesh-tools, mesh-sdk. A day for the skeleton.* A bundle whose entry is not JavaScript is launched
as a child process with the runtime's environment and spoken to over MCP on stdio: `tools/list`
once, `tools/call` per call; a tool named `<seat>.<verb>` is the seat's implementation. A child
that exits is named as a failed bundle and restarted on the next call. The TypeScript import stays
as the shortcut. Beside it, one skeleton SDK per language of the first set — the stdio loop and the
tool-definition type, nothing else — each proven by one bundle in that language answering one tool
in the runtime's test. **Proof.** The runtime's test: a bundle in a second language, launched, its
tool answering on its subject over a real bus; the TypeScript fixture served through the protocol
with the shortcut off answers the same.
## WP2 — The controller composes one runtime per node
*mesh-controller. Two to three days; the largest package.*
**What changes**, in four pieces, each its own commit:
1. **A node principal.** Beside one principal per module per node, one per node of kind
`node-tools`: its serving grants are the union of every assigned module's tool subjects and every
held seat's verbs on that node, its invoking grant is `*`, and it consumes nothing. The
per-module memberships are composed as today; nothing else on the bus learns a new shape.
2. **Bundle delivery.** For every assigned module whose build produced a `bundle`, the node's
declaration gains an `archive` placed under a directory the controller derives, so the host
fetches and unpacks it as it does any artifact. The bundle's digest is what the build recorded.
3. **The runtime's process.** One `process` per node running the runtime from its own bundle
(WP3), `MESH_TOOL_MODULES` composed from the unpacked entrypoints — each as
`<module>=<path>`, and the runtime decides from the file whether it is loaded or launched
(WP1b) — `MESH_OPERATOR_ACCOUNT` and
`MESH_OPERATOR_HOME` from the account fact, `restart-on` naming every bundle so a push that
changes one restarts it. A node with no account composes the runtime without the two words.
4. **The gate.** A manifest declaring `tools` and a container built on the runtime's base image is
refused at registration once the runtime module is registered, naming this record. It is the
mechanism that keeps the old pattern from returning by habit. ADR 0188 widens it, after WP4:
a module whose own code is an image artifact is refused, whatever image it is built on.
*Amended 2026-10-02, at WP3.* The gate refuses the pattern **spreading**, not standing: a module
new to the catalogue in that shape, or one that had already moved to a bundle and returns to it,
is refused; a module the catalogue already holds in that shape — judged from the manifest it
holds and what that module's newest build stood on — is rebuilt without complaint. The day the
runtime arrives some thirty such modules stand, each moves in its own change from WP4 on, and a
gate refusing every rebuild in the meantime would stop the catalogue's pipeline to make a point
this record already makes.
**Proof.** Composition tests: a node with three assigned modules, one holding a seat, yields one
process, three archives, one node principal whose grants are the union, and the same three
memberships as before. The gate's test: the packet-filter manifest as it is today is refused once
the runtime is registered.
## WP3 — The runtime is a module, and the console is its serving mode
*mesh-tools and mesh-catalog. A day.*
**What changes.** mesh-tools gains a `bundle` artifact of itself beside its images, and its manifest
becomes the `node-tools` module: a package for the interpreter, the loopback listener the console
declared, `invokes: *`, and nothing else — the process is the controller's to compose (WP2). In the
catalogue, `mesh-console` is retired as a module and `node-tools` assigned where it was. The
runtime's `serve` keeps answering MCP on loopback; the person's end of it keeps the name *console*
([glossary](../../00-META/glossary.md)).
**Proof.** On every node: the console's container is gone, `node-tools` runs as a unit the host
wrote, `tools/list` on loopback answers as before, and the controller's verbs answer through it.
This is the first live step, and it is reversible by re-assigning `mesh-console`.
*Decided 2026-10-02:* `mesh-tools` keeps its name as the module the TypeScript images come from, and
`node-tools` is a second module in the same repository ([ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md))
rather than a rename — the thirty-five manifests that build `on` `mesh-tools` stay true. Three things
WP3 found that the plan did not say: a TypeScript bundle must carry its dependencies and a
`package.json` naming its files as ES modules, which the toolchain now copies in from its own image;
the runtime's credential must be owned by the account the runtime runs as, which the controller
composes; and `MESH_TOOL_MODULES` is empty on a node where the runtime is the only bundle, which the
runtime accepts. *Built 2026-10-02* (mesh-tools `c46f950`, mesh-controller `ca7e81e` `773b561`
`729a5f9`). *Proven live 2026-10-02/03, on all four machines*: the console's container is gone,
`node-tools` runs as a unit the host wrote, as the operator's account, `tools/list` on each loopback
answers with the same 219 tools as before, and the controller's verbs answer through it; `mesh-console`
retired from the catalogue. Three things the step found are issues
[203](../../04-ISSUES/203-a-fresh-assignment-is-pushed-before-its-credential-exists/00-report.md),
[204](../../04-ISSUES/204-a-controller-handover-re-sent-every-node-a-stale-declaration/00-report.md) and
[205](../../04-ISSUES/205-a-package-resource-fails-against-a-stale-package-database/00-report.md).
## WP4 — The first holder moves: the packet filter
*mesh-catalog. Half a day. The live proof of ADR 0175.*
**What changes.** The nftables module drops its container, its `NET_ADMIN` and its runtime
artifact; its tools bundle stays and its claim stays. Its `remove` and `reload` escalate inside the
tool where they need root, which they have, since the runtime runs as the node's account.
**Proof.** `node-packet-filter.rules@<node>`, `reload` and `remove` answer from the runtime on all
four machines; `docker ps` shows no `mesh-nftables`; `status` is well. Then the fail2ban holder
proposed in an open change follows the same way when it lands.
*Found 2026-10-03, before the step ran:* a bundle imported in-process brings its own copy of the SDK
(WP3's *carry its dependencies*), and the SDK's tool registry is the copy's own — the first module
loaded beside the runtime would have registered its tools where the runtime never looks, and served
nothing, silently. Issue
[209](../../04-ISSUES/209-a-bundles-own-sdk-copy-registers-into-a-registry-the-runtime-never-reads/00-report.md):
the runtime now resolves every bundle's import of the SDK to its own copy, one registry and one
broker per node. Two things the package did not say, settled in the module: the filter's commands
run through `sudo` without a prompt where the runtime is not root, since the operator's account may
escalate as the operator would; and a bundle has no environment of its own, so the tool reads the
filter from the path the manifest's `filtering` names rather than from a variable the container used
to carry, a test holding the two together. Three things a review of the change found: the module's
own bus credential and state directory went with the container, since nothing reads them once the
runtime speaks with the node's (the shell module of WP5 declares neither); the `iptables` package the
image used to carry is now declared on the host; and that the operator's account may escalate without
a prompt is a fact about the machine the mesh neither declares nor checks — true on all four today,
and when it is not, the tool names it by how it failed, which is the only check there is until a
record says where the fact belongs.
*Built 2026-10-03* (mesh-tools `7152148` for issue 209, mesh-catalog `db5e7c8`). *Proven live
2026-10-03, on all four machines*: `node-packet-filter.rules`, `reload` and `remove` answer from the
node's runtime on each — `rules` and the module's own tool list the mesh's table, `reload` loads the
file and answers with the table, `remove` refuses the mesh's own table by name — `docker ps` shows no
`mesh-nftables` on any, the container's credential is gone with it, and `status` is well. One thing
the step found is issue
[210](../../04-ISSUES/210-the-host-re-creates-the-nodes-runtime-on-every-reconcile/00-report.md):
the host re-creates the runtime's process on every reconcile (resolved the same day, mesh-host #80).
*fail2ban followed 2026-10-03* (mesh-catalog `aa5bf7d`), the same shape: container, base images,
credential and state directory gone, the client through `sudo` since the daemon's socket is root's;
proven on all four machines — `status`, `banned` and the module's own `fail2ban_settings` answer from
the runtime, no `mesh-fail2ban` container, the runtime serving both bundles. Two holders moved; of the
thirty-three tool containers the catalogue held, thirty-one remain, and all but these two carried their
module's configuration and secrets in the container's environment, which a bundle does not have — the
question research [020](../../01-RESEARCH/020-what-a-bundled-tool-is-given/00-overview.md) opened
and [ADR 0192](../../02-DECISIONS/0192-a-tools-bundle-declares-what-it-is-given-and-the-runtime-hands-it-to-that-bundle-alone.md)
settled the same day: a tools bundle declares `env` on its artifact, the composer resolves it as a
container's, the runtime hands each bundle its own. That is WP4b below.
## WP4b — Every tool container moves
*mesh-controller, mesh-tools, mesh-catalog. One day. The rest of ADR 0175, under ADR 0192.*
**What changes**, in order: the manifest's tools artifact gains `env` and the catalogue check
refuses a secret's content in it; the composer resolves a bundle's `env` per machine and carries it
beside the bundle's archive, `restart-on` included; the runtime hands each bundle its own
environment — the contributor's argument for an imported bundle, the child's environment for a
launched one — and a test holds two bundles apart. Then the thirty-one remaining tool containers
move in one change: each container's `env` becomes its tools artifact's, mount targets folded into
the host paths they came from, the container, its base images, its Dockerfile and its own bus
credential gone. Last, the registration gate refuses the container shape for every module.
**Proof.** The controller's and the runtime's tests named in ADR 0192; live, every module's tools
answer from the runtime on the machines that run it, `docker ps` shows no tool container on any of
the four, and `status` is well.
*Found 2026-10-03, building it:* of the thirty-one, nine run only tools, three a main of their own,
and twenty import the module's own event handlers and provisioners beside their tools (ADR 0192's
dated note). WP4b moves the tools-only nine; WP4c holds the rest. Two of the nine stay with WP4c as
well — one carries a run-once provisioning step in a second container, one reads an env-file and two
sockets — so seven move here. *Built 2026-10-03:* mesh-sdk #12 (`collectToolsEach`), mesh-tools #33
(each bundle its own environment), mesh-controller #236 and #237 (the words composed, made the
account's to read, and named files restarting the runtime), and the seven modules in one change.
Building it found issue [211](../../04-ISSUES/211-a-bundle-is-built-before-the-toolchain-it-is-compiled-in/00-report.md).
*Proven live 2026-10-03* (mesh-catalog #242, #243): on the one machine that runs them, baserow,
letta, searxng and unifi answer from the runtime with no tool container — each reading its
configuration file as the operator's account — and the runtime serves seventeen tools for six
modules there. confluence, gitlab and jira are assigned nowhere and retire with the predecessor.
Two traps met on the way: a tools bundle whose module declares no `tools` list must say `loads`, or
the composer delivers it nowhere while the build reports success; and a module whose builds are
pinned to an old commit is left out of a merge's plan and must be built from `main` by hand.
## WP4d — Every served bundle is launched; the runtime in Go
*mesh-sdk, mesh-controller, mesh-tools. [ADR 0193](../../02-DECISIONS/0193-every-bundle-the-runtime-serves-is-launched-and-the-runtime-knows-no-language.md).*
**In order.** The SDK's stdio loop serves what a bundle registered under the module it is told it
serves as. The builder writes, beside every TypeScript entrypoint, an executable launcher that
imports it and serves what it registered; the composer names the launcher where it named the
entrypoint. The runtime launches every served entrypoint and imports none; the resolve hook and the
per-registration environment go. Proven live on all four machines. Then the runtime is rewritten in
Go against the same contract — the bus, the memberships and seats, the launcher, the console's MCP
over HTTP — and replaces the TypeScript one, proven the same way.
**Proof.** The tests ADR 0193 names; live, every moved module's tools and both node seats answer from
launched bundles on every machine, and then do again from the Go runtime.
*Built and proven live 2026-10-03.* mesh-sdk #13/#14 (0.1.4, 0.1.5: served as the named module; an
emit travels through the runtime), mesh-controller #239/#240 (a launcher beside every TypeScript
entrypoint; a runtime compiled to a binary runs itself), mesh-host #81 (`./name` is the process's own
binary), mesh-tools #35/#36/#37/#38 (the module named; launch-only; the toolchain requiring 0.1.5; the
runtime in Go). On all four machines node-tools is now the Go binary, launching every served bundle:
both node seats answered from it on every machine and the four moved modules on theirs. Found on the
way: issue [212](../../04-ISSUES/212-a-toolchain-rebuild-keeps-the-sdk-it-cached/00-report.md) (the
toolchain image kept a cached SDK, and the seats' verbs went unanswered on three machines for an hour),
and the controller's plan losing track of its own rebuild when it restarts mid-plan.
## WP4c — The module's own long-running code moves
*Not yet broken down.* Twenty-three containers carry code that is not a tool: event handlers,
provisioners, a step, a main. [ADR 0188](../../02-DECISIONS/0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md)
§3 already says such code is a `process` bundle the host runs. What no record says yet is how that
process is given what its container was: the module's own bus credential and the subscriptions it
consumes with, the words its code reads at import, the packages the image installed (a database's
client), and the service it reaches by a container network name. That begins with a decision record,
after which the twenty-three move and the registration gate refuses the container shape for all.
*Decided 2026-10-03, [ADR 0198](../../02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md):*
the node's runtime launches that code as it launches tools, and is its bus — `mesh/subscribe` and
`mesh/ask` beside `mesh/publish` on the stdio channel, the module's own durable consumer bound by the
runtime and acknowledged only after the child answered. **In order:** the runtime's subscription and
its grants; the SDK's `on` and provisioner bound to the channel; then the modules in three waves — the
provisioners and handlers whose backends are reached on loopback with a system package (postgres,
redis, mosquitto, influxdb, keycloak, umami, cloudflare-dns, grafana, icecast, home-assistant, nodered,
nextcloud, minio), the two whose clients exist on no system (mongodb, mssql: a driver in the bundle),
and last the mesh's own (mesh-catalog, mesh-vault, records, gitea, mailu, audit-logger, lab, and the
three mains).
## WP5 — The shell, on a server first
*mesh-catalog #224, already written. Half a day to assign and prove.*
**Order.** Assign `zsh` to one server; push; `login-shell.execute@<server> command="uptime"`
answers; the account's login shell reads zsh; its `~/.zshrc` carries the mesh's block with the
operator's lines around it. Then the other three nodes. The two things the manifest cannot say
— the `user` shape applying only where the seat is held, and a second shell module installed
beside the holder — are the first follow-up record after this document.
## WP6 — The service manager, on a workstation
*mesh-host #72 merged first; mesh-catalog #224. Half a day.*
**Order.** Merge the host's user-scope change and let it roll. Assign `systemd` everywhere;
`node-service-manager.units@<node> scope=user` answers on a workstation. Then the first user-scoped
unit the mesh sends: the window manager's reload watcher, declared `scope: user` by the window
manager module when WP7 writes it — until then, the host's change is proven by its tests and by
the verb answering.
## What is deliberately not here
- **The graphical stack's seats** (WP7). Each begins with a record naming its holders and verbs,
and the first graphical module asks the resolver a question this document cannot answer for it:
whether a held seat gates another's assignment.
- **Settings for the theme knobs** (WP8). Blocked on the settings record proposed in an open change
and on [issue 168](../../04-ISSUES/168-a-setting-reaches-every-file-and-contribution/00-report.md).
- **Reload without restart.** WP2 restarts the runtime on a bundle change; a reload that keeps the
other modules' tools up during one module's change is a refinement for after WP4 proves the
simple form.
- **Lingering.** A user-scoped unit answers only while the account's manager runs; declaring
lingering for the account is a field on the `user` shape, decided when a server first needs a
user unit.
## How this list is kept true
Each package's proof is run on the live mesh when the package is finished and its line here gains
the date and the commit, the way [ADR 0170](../../02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md)
carries *built and proven live*. A package whose proof fails is not reworded; the failure is
recorded under it and the package stays open. When WP6 is proven, design 37's status moves to
`implemented` for what it covers and this document's to the same.