Compare commits

...
Author SHA1 Message Date
mesh-admin 5938d40dee Merge pull request 'ADRs 0164–0166 and issue 190: the container runtime gets a module, a seat and declared settings' (#271) from decision/docker-module into main 2026-10-02 20:14:58 +00:00
mesh-admin f062672f83 Merge pull request 'Issues 192 and 193: the console reaches a person only by hand; the store's read-only query is not' (#272) from issues/192-193-console-registration-and-store-query into main 2026-10-02 20:14:46 +00:00
jschoubben e4a0c73e2b Merge remote-tracking branch 'origin/main' into issues/192-193-console-registration-and-store-query 2026-10-02 22:14:24 +02:00
jschoubben d1aeee42a4 Regenerate the decision index after merging main 2026-10-02 22:14:21 +02:00
mesh-admin 81d780f973 Merge pull request 'Design 38: WP3 built — node-tools beside mesh-tools, the three things the plan did not say, and the gate refuses spreading not standing' (#304) from design/38-wp3-built-and-the-gate-softened into main 2026-10-02 19:57:16 +00:00
jochen 3e30846e0f Design 38: point at ADR 0069's real file name 2026-10-02 21:46:34 +02:00
jochen b3f18c54c6 Design 38: WP3 built — node-tools beside mesh-tools, the three things the plan did not say, and the gate refuses spreading not standing 2026-10-02 21:46:01 +02:00
mesh-admin e11bf320c9 Merge pull request 'ADR 0188: a module's own code is bundles in any language, and a tools bundle speaks MCP to the runtime; design 38 gains WP1b' (#300) from decision/0187-a-modules-own-code-is-bundles-in-any-language into main 2026-10-02 19:27:39 +00:00
jschoubben da8b4b4ee4 Renumber to ADR 0188: 0187 landed on main first, as the dead-tracker record
Two records shared 0187 (issue 155's collision); the branch landing last
renumbers, and this is it. Only the number changes.
2026-10-02 21:01:02 +02:00
jschoubben c026d5221e Merge remote-tracking branch 'origin/main' into renumber-0187 2026-10-02 21:00:45 +02:00
mesh-admin 983fd412c6 Merge pull request 'ADRs 0180, 0186 and 0187: their live rows, done' (#302) from docs/the-four-fixes-proven-live into main 2026-10-02 18:46:49 +00:00
jschoubben 9ac2493e2c ADRs 0180, 0186 and 0187: their live rows, done — all four machines filtered by the mesh alone, both front ends removed, no machine wrong or behind 2026-10-02 20:46:35 +02:00
mesh-admin 560f25c2c7 Merge pull request 'ADR 0187: a dead tracker is not the machine's failure' (#301) from fix/a-dead-tracker-is-not-the-meshs-failure into main 2026-10-02 18:42:01 +00:00
jschoubben 9e0288128b ADR 0187: a dead tracker is not the machine's failure; design 32 2026-10-02 20:41:30 +02:00
jochen 709240ec1f ADR 0187: a module's own code is bundles in any language, and a tools bundle speaks MCP to the runtime; design 38 gains WP1b
The operator's direction, absent from every record until now: the SDK must not limit who writes a
module; tools and services may be written in any language; one module may ship several bundles
(tools, a seat's implementation, a daemon); skeleton first, a full implementation when the work
requires it. ADR 0175 had the runtime import a bundle, which only JavaScript can be.

0187 makes a tools bundle a process the node's runtime launches and speaks MCP over stdio to —
the vocabulary the runtime already speaks outward — so any language with an MCP library can write
one today and the mesh's SDK per language is thin; the transport stays in the runtime (0039's
refusal, kept). Importing a TypeScript bundle is the shortcut, not the contract. Notes in 0175,
0039 and 0150 say where their mechanism moved; design 38 records WP1 as built and adds WP1b (the
launcher and the skeleton SDKs); the glossary's bundle widens.
2026-10-02 18:56:57 +02:00
mesh-admin d57289e049 Merge pull request 'ADR 0186: a ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang' (#299) from fix/a-ban-list-never-holds-a-neighbour into main 2026-10-02 16:43:32 +00:00
jschoubben d4a2f99ab5 ADR 0186: a ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang; design 31 2026-10-02 18:42:16 +02:00
mesh-admin a9f91fdd0c Merge pull request 'ADRs 0184 and 0185: a service is still running a moment later; a control plane behind its row serves what it can' (#298) from fix/a-service-asked-to-run-is-still-running into main 2026-10-02 16:25:44 +00:00
jschoubben 131a5e4714 Issue 201: what was established about the race while closing the outage half 2026-10-02 18:19:12 +02:00
jschoubben 329a24fdae ADRs 0184 and 0185: a service is still running a moment later; a control plane behind its row serves what it can; issue 201 half closed 2026-10-02 18:16:24 +02:00
mesh-admin 7f72f3b79a Merge pull request 'ADR 0179: the intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail' (#295) from feat/the-intrusion-seat-serves-its-verbs into main 2026-10-02 15:28:48 +00:00
jschoubben 114a71f36f ADR 0179: built and proven live; the one fault the machine found, and the check that refuses it 2026-10-02 17:28:38 +02:00
jschoubben 0f407417f3 Merge main: the ufw record renumbered to 0180, and ADR 0175 retires the per-module tool runtime this one ships 2026-10-02 17:28:23 +02:00
jschoubben d0d5799884 Issue 201: a push recreated the controller at a digest older than the seat row its successor wrote 2026-10-02 17:15:22 +02:00
jschoubben 9ba4de5557 ADR 0179: the intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail; designs 31 and 33 2026-10-02 17:02:49 +02:00
jschoubben 17ca9a262b 0164: a setting names the file it lands in (issue 198's leak between one module's files); 190 notes the fourth machine now has the resolver 2026-10-02 12:23:28 +02:00
jschoubben 967c793eaa Merge remote-tracking branch 'origin/main' into decision/docker-module 2026-10-02 12:23:27 +02:00
jschoubben 27c1db8a86 Review of 0164-0166 and 190: the mesh's own setting words stay settable; changing runtime verbs are not the console's wildcard; migration steps 1-2 are one push; dnsmasq's dns key dates from 09-23 2026-10-02 00:48:06 +02:00
jschoubben 24aeb203f7 Issue 193 resolved: both readers live on every machine, checked by asking each copy who it is 2026-10-02 00:35:03 +02:00
jschoubben afbfd5f29d Issue 193: mssql's variable substitution and shell commands, proven and fixed by mesh-catalog PR 210 2026-10-02 00:25:48 +02:00
jschoubben 696957aa5e Issue 193: proven on a throwaway server, fixed for postgres by mesh-catalog PR 209; mssql has the same hole 2026-10-02 00:09:38 +02:00
jschoubben 3d54fcbb86 Merge remote-tracking branch 'origin/main' into decision/docker-module 2026-10-02 00:02:57 +02:00
jschoubben b13ef1be81 Issues 192 and 193: the console reaches a person only by hand; the store's read-only query is not
192: no provision says where the console is, its port was never assigned, and nothing
owns a person's agent configuration since the predecessor left. 193: the query verb wraps
the caller's text in a read-only transaction the text can end, and its rows come back
keyed by BEGIN.
2026-10-02 00:02:51 +02:00
jschoubben f1941304cc ADRs 0164-0166 and issue 190: the container runtime gets a module, a seat and declared settings
Proposed for the operator's review: settings declared with defaults and cost (0164),
container-runtime as a kernel capability (0165), node-container-runtime seat with the
host creating containers through its holder (0166), and the runtime's file written by
modules that are not its own (190).
2026-10-01 23:13:18 +02:00
24 changed files with 1478 additions and 8 deletions
+1 -1
View File
@@ -108,7 +108,7 @@ term retired here may still appear there, and the mapping above is how to read i
memberships issue; its serving mode on loopback is what was called **the console**
([ADR 0175](../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)).
Replaces **"console"** as the module's name; *console* remains the word for the person's end of it.
- **bundle** — the artifact a module's tools are built into, interpreted or compiled; never an image.
- **bundle** — the artifact a module's own code is built into — its tools, a seat's implementation, a daemon — in any language the mesh has a toolchain for, interpreted or compiled; never an image. One module may declare several ([ADR 0188](../02-DECISIONS/0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md)).
- **kept region** — a marked block in a managed file the mesh writes *into*, where the operator's own
lines survive every push and are given back when the module goes
([ADR 0174](../02-DECISIONS/0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)).
@@ -8,6 +8,8 @@ reconstructed: false
# 39. What the SDK holds, and what it refuses
> **The mechanism changed — 2026-10-02, by [ADR 0188](0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md).** The test of this record — frequent *and* cascading does not belong — stands and now applies to one SDK per language. Where *what it holds* names the broker client and the event consumer, read: the local protocol a tools bundle speaks to the node's runtime; the transport lives in the runtime and in no SDK, which is what keeps a bus change from rebuilding any module in any language.
_Reconciliation note (2026-09-05): supersedes the earlier "repository structure" decision, which the consolidation folded; no standalone record remains to point at, so body references to it now point at the nearest surviving record, [ADR 0015](0015-applications-live-in-their-own-repository.md)._
## Context
@@ -9,6 +9,8 @@ extends: 02-DECISIONS/0047-a-module-runs-its-code-as-its-own-process-with-its-ow
# 150. A module's own code runs as supervised processes under the module's one account
> **Widened — 2026-10-02, by [ADR 0188](0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md).** A module's long-lived process is a bundle in any language the mesh has a toolchain for, run as a unit the host writes; this record never said one language and never meant one, and 0188 says so as the rule.
> **The mechanism changed — 2026-10-02, by [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md).** For a module's *tools*, read that record: one runtime per node, the node's one account, bundles loaded from the memberships. This record still governs a module's long-lived processes — a daemon, a provisioner, a scheduled ingest — and the account invariant for them.
## Context
@@ -0,0 +1,146 @@
---
topic: what runs on it
status: proposed
date: 2026-10-01
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md
---
# 164. A setting is declared with its default, its meaning and what changing it costs
## Context
The operator asked for one thing for every module, with the container runtime as the first case: **one
consistent default configuration for every machine, overridable per assignment, and easy to change
later.** The four machines' runtime configurations were each written by hand and differ — one keeps
running containers through a daemon restart and one does not, their log rotation differs, and each
names its resolver and its trusted registries in its own words.
Most of this was already decided.
[ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md) said the definition is
identity and **defaults**, the assignment's settings are the configuration, *unset is the default*, and
*an unknown setting is refused*. [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md) made
a setting an operator requirement whose contract is "a type and, optionally, a default", answered by
"the assignment's, or the requirement's default, or unresolved". An assignment is a module on a node,
so the node layer of a module's settings already *is* the per-assignment override, and the mesh-wide
layer is the one consistent default a person changes once.
What was built is narrower than what was decided, measured in the controller on the day of deciding:
- **Nothing declares which keys are settable.** A mergeable file's content is its defaults, and every
key of it — and every key not in it — is accepted. Nothing tells a person, or the console, what can be
set, of what type, or what it means.
- **The refusal of an unknown setting is not there for most modules.** The stray-setting report returns
nothing at all for a module with any mergeable file, because such a file "takes any key"
([issue 173](../04-ISSUES/173-a-modules-settings-reach-every-fact-it-contributes/00-report.md) left
files that way on purpose). It reports rather than refuses where it does run.
- **A value in a file that is not JSON can have no default.** `${setting:<key>}`
([ADR 0155](0155-a-definition-names-no-installation-and-how-that-is-checked.md)) is refused when no
layer sets it — right for a mail domain, where a default is the very literal 0155 removes, and wrong
for a tunable like the resolver's upstreams, which the resolver module therefore carries as literals
in its file.
- **A setting reaches every mergeable file its module owns.** The layers are one flat map per module,
laid over each such file. Adding a setting to the resolver module for its own configuration put the
key into the container runtime's file as well — the resolver writes into that file too — and the
runtime refuses keys it does not know. The plan showed it before any push; the runtime's file was
then made to take no settings at all ([issue 198](../04-ISSUES/198-the-lans-dns-server-ran-outside-the-mesh-and-its-filter-closed-it/00-report.md)). Issue 173 stopped settings leaking into
contributions and served facts; between one module's own files the leak remains.
- **What a change costs is said per file, not per key.** A service names the files it is reloaded or
restarted on. The runtime re-reads its trusted registries on a reload and its `dns` key only when it
starts; the resolver module declared a reload, so on two machines the key was written, reloaded,
and never read, and every container got a public resolver for weeks while everything read as
current ([issue 110](../04-ISSUES/110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/01-resolution.md)).
## Considered Options
1. **Leave settings implicit; document each module's keys in its README.** Rejected: a key the mesh
does not know cannot be refused, typed, listed by the console or costed, and a README is a rule
enforced by nothing.
2. **A second mechanism for tunables beside settings** — defaults in a new block, settings untouched.
Rejected: two ways to state one person's value, and design 27 already retires six mechanisms
that grew that way.
3. **Settings declared in the definition, as 0112's operator requirement: a key, a type, a meaning,
optionally a default, and what a change costs.** Adopted.
## Decision
**A module declares every setting it takes.** Each declared setting has a name, a type, one sentence
of meaning, optionally a default, and what a change to it costs. The spelling is design 27's to settle
with the rest of the requirement form; this record decides the content.
**A setting with a default is a tunable; a setting without one is the operator's.** A tunable resolves
to its default when no layer sets it — wherever it is read, a mergeable file or `${setting:<key>}` in a
file of any format. A setting with no default is refused by name when nothing sets it, as 0155 decided;
0155's refusal is narrowed to exactly that case, not changed for it. Whether a value has a default is a
fact about the software (a log size does, a mail domain does not), and the definition states it once.
**The layers stay as they are, and every value says where it came from.** The definition's default,
then the mesh-wide layer, then the node's — later wins, objects merge, lists replace. One consistent
configuration for every machine is the default plus the mesh-wide layer; one machine that differs says
so in its own layer and nothing else. Asked for a module's configuration on a machine, the mesh lists
every declared setting with its effective value and its source: *default*, *mesh*, or *node*.
**Changing later is changing one of three places, and the plan shows its reach before anything moves.**
A new default ships with the module's next version and reaches every assignment that does not override
it; a mesh-wide setting reaches every assignment of the module; a node's reaches one. The plan of a
change names each assignment whose effective value moves.
**A declared setting says where it lands.** Each names the file or files of its module that read it,
and reaches no other: a module that owns two mergeable files no longer has one flat map laid over both.
A file that names no setting takes none.
**A declared setting is the only kind accepted.** Setting a key the module does not declare is refused
when it is set, naming the declared keys, rather than reported when the machine is planned. The mesh's
own words — where a port, a directory or an operator's data is placed, how far an endpoint reaches —
are the mesh's to validate as they are today, and no module declares them. A module
that declares no settings keeps today's behaviour until it does; a catalogue test lists those modules,
and the list shrinks to empty before the implicit form is removed — design 27's rule for every retired
mechanism.
**A setting says what it costs: nothing, a reload, or a restart.** When a file changes, the host
applies the strongest cost among the settings whose values moved in it, so a key the software reads
only at start can no longer be written and never read. A setting that reaches a container's environment
costs that container being recreated, which the host already does when a container's specification
changes; it needs no declaration. A service's `reload-on` and `restart-on` keep
naming the files that are not settings — a generated roster, a credential.
**The container runtime is the first module to declare its settings** and the model for the rest:
its log rotation, keeping containers through a daemon restart, and its resolver are tunables, and
its trusted registries are what the mesh tells it.
## Consequences
- The console can show a module's settings as a form: what can be set, of what type, its default,
and where the current value came from. That is the surface the operator wants for changing a
default later.
- `settings set` can refuse an unknown key, so ADR 0046's rule is enforced where it was only stated.
- The resolver's upstreams, the runtime's log rotation, and other literals a definition carries
because it could not give them a default become declared tunables.
- **What got harder:** every module that takes settings must list them, and a mergeable file no
longer silently accepts a key its author did not foresee. A person who needs one adds it to the
definition, which is a new module version, not a setting.
- Issue 173's open question — a consumer checks nothing against a contract — is unchanged; this record
is the operator half of design 27's contract, not the provider half.
- Not decided here: the spelling (design 27); whether a node's layer may be narrowed to a single key
rather than replaced whole, as `settings set` does today.
## How this is checked
| Rule | Checked by |
|---|---|
| Every setting a module takes is declared | A parser test refusing a setting declaration without a type or meaning; a catalogue test listing modules with mergeable files or `${setting:}` and no declarations, which must be empty before the implicit form is removed |
| A tunable resolves to its default; an operator value without one is refused | Resolution tests: an unset tunable in a JSON file and in a text file both take the default; an unset setting with no default is refused naming it (0155's existing test) |
| A setting reaches only the files it names | A resolution test: a module with two mergeable files and a setting declared for one; the other file's content is unchanged by it (the case of issue 198) |
| An undeclared key is refused when set | A controller test: `settings set` with an undeclared key fails naming the declared keys, and nothing is stored |
| Every effective value names its source | A test listing a module's configuration on a node with one key from each of default, mesh and node |
| A change's reach is shown before it moves | A plan test: changing a mesh-wide setting names every assignment whose effective value moves and no other |
| The strongest cost applies | A host test: a file where a reload-cost key and a restart-cost key both moved restarts; a file where only reload-cost keys moved reloads |
| Live | The container runtime's module lists its settings with their sources on every machine, and a mesh-wide change to its log rotation reaches all four at the next push |
## References
- [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0155](0155-a-definition-names-no-installation-and-how-that-is-checked.md), [ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)
- [Design 27 — a module requires, the mesh resolves](../03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md)
- Issues [110](../04-ISSUES/110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md), [173](../04-ISSUES/173-a-modules-settings-reach-every-fact-it-contributes/00-report.md)
- mesh-controller `internal/catalogue/settings.go` (`settle`, `UnusedSettings`), `internal/catalogue/setting_into.go`
@@ -0,0 +1,105 @@
---
topic: what runs on it
status: proposed
date: 2026-10-01
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0161-what-deserves-a-seat.md
---
# 165. `container-runtime` is what a machine can run; that a runtime is running is its holder's health
## Context
A capability is a requirement a module places on a machine, detected by the host and renewed with
every report ([ADR 0161](0161-what-deserves-a-seat.md)). The host's `container-runtime` asks the
daemon for its version: *a running daemon, not an installed client*. It was made that way by
[issue 007](../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md), where an
installed package was believed to be a working service, and
[design 05](../03-DESIGN/01-to-be/05-the-node-host.md)'s table says the same: *a runtime is
running*. The installer's preflight borrows the same detector to wait for the runtime the
foundation bundle installs, so there is one answer to "is there a runtime here".
The mesh is now to have a module for the runtime itself — its packages, its configuration, its
service — on every machine ([ADR 0166](0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md)).
That module cannot declare `container-runtime` as defined: it would require the very thing it
installs, the cycle [research 011](../01-RESEARCH/011-the-module-graph/cases.md)'s case 12 names
("something the mesh installs that then becomes a node capability"). The operator defined the word
for it: **`container-runtime` means the machine is able, at the kernel level, to install a runtime and
execute containers** — not that one is installed, and not that one is running.
The host already draws this line once. `seat` is hardware, a display server *could* run here;
`graphical-session` is state, one *is* running; the detector's own comment says "assignment needs the
first". A machine without a display has no seat however much software is installed, and a machine
with one has a seat before anything is.
Fifty-four catalogue modules declare `container-runtime` today, counted on the catalogue's main
branch on the day of deciding: every module that delivers a container. Each relies on the current
meaning to keep it off a machine with no running runtime.
## Considered Options
1. **Keep the meaning; let the runtime's module declare nothing.** Rejected: a module that installs
the runtime has requirements on the machine — the kernel features without which installing it is
pointless — and would state none of them. The cycle stays, only hidden.
2. **Two capabilities, "can run" and "is running".** Rejected: the second is made true by assigning a
module, so it is the module's state, not a fact of the machine; a capability the mesh itself
flips by its own assignment is case 12's cycle with an extra name.
3. **The capability is the kernel's; whether a runtime runs is the runtime module's health, and a
module that delivers a container needs the runtime's seat held.** Adopted.
## Decision
**`container-runtime` is detected from what the kernel offers**, as `seat` is: the namespaces a
container needs, a control-group hierarchy the runtime can manage, and an overlay filesystem the
running kernel has or can load. Present when all three are; absent naming the missing one. Nothing is
run and no runtime is asked. The verdict's detail names what was found, not a runtime's version.
**"A runtime is running and answers" is one probe, owned by the host and used twice:** by the
installer's preflight, which waits for the runtime the foundation installs, and as the runtime
module's health. It asks the daemon, as issue 007 requires. The preflight stops borrowing the
capability's detector, and there is still one answer to "is a runtime running here".
**The runtime's module declares `container-runtime`**, with `package-manager`, `service-manager` and
`privileged`, like any module that manages machine software.
**A module that delivers a container needs the runtime seat held on its machine**, and is refused
otherwise, naming the seat and the modules that could hold it — the refusal design 27 already lists
for an unheld seat. That requirement is derived from the container resource and needs no manifest
field ([ADR 0166](0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md)).
The fifty-four existing declarations of the capability stay valid and become redundant; a catalogue
test lists them, and they retire when the list is empty.
**The order is fixed, not preferred.** The detector changes only once the seat requirement is
enforced. In between, a machine with the kernel and no running runtime would read as able to run
every containerised module, which is issue 007 again.
## Consequences
- Design 05's capability table changes its `container-runtime` row from *a runtime is running* to
*the kernel can run containers*, and names the runtime module's health as where "running" is now
asked.
- The node listing stops showing the runtime's version beside the capability. The version moves to
the runtime module's health and its seat's verbs.
- A fresh machine with no runtime reads as able to run one, so it can be assigned the runtime's
module, which is what makes the mesh able to install the runtime instead of the bootstrap alone.
- **What got harder:** "is this machine running containers" is no longer one glance at the profile;
it is the runtime seat's holder and its health. The node's listing should show both side by side.
## How this is checked
| Rule | Checked by |
|---|---|
| The capability is the kernel's | Host detector tests over a fixture `/proc` and `/sys`: all three present → present; each one missing → absent naming it; no runtime binary on the fixture machine changes nothing |
| One probe asks whether a runtime runs | A host test that the preflight and the runtime module's health call the same probe, and that the probe fails against a stopped daemon with an installed client (issue 007's shape) |
| A containerised module needs the runtime seat held | A resolution test: a module with a container resource on a machine whose runtime seat is unheld is refused, naming the seat and its candidate holders |
| The order holds | The host release that changes the detector is gated on the controller release that enforces the seat requirement — stated in both changes' descriptions and checked at review |
| Live | Every machine's profile shows `container-runtime` present with the kernel's features as its detail; a machine with no runtime installed reads present too |
## References
- [ADR 0161](0161-what-deserves-a-seat.md) — the profile renewed by every report; a capability that names a dialect
- [ADR 0166](0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md) — the seat and its holder
- [Issue 007](../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md), [research 011](../01-RESEARCH/011-the-module-graph/cases.md) cases 12–13
- [Design 05 — the node host](../03-DESIGN/01-to-be/05-the-node-host.md)
- mesh-host `internal/profile/detectors.go` (the runtime detector), `internal/profile/seat.go` (the hardware/state split), `internal/bootstrap/preflight.go` (the preflight that borrows it)
@@ -0,0 +1,161 @@
---
topic: what runs on it
status: proposed
date: 2026-10-01
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0161-what-deserves-a-seat.md
---
# 166. The container runtime is a node seat, and the host creates containers through its holder
## Context
Every container the mesh runs on a machine is created by the host, which looks for a runtime
(`docker info`, then `podman info`) and drives that runtime's command line itself: run, inspect,
remove, exec. Research 012 called this "detected rather than declared": the host takes over whatever
runtime it finds. Nothing in the mesh owns the runtime. Its package came from the foundation bundle
or was already on the machine. Its configuration file was written by hand, differs on each of the
four machines, and is also written into by two modules that are not the runtime's
([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)).
Its service is declared by those same two.
The operator set the direction:
- a module for the runtime, on every machine, owning "what is needed to run containers here": its
packages, its configuration and its service;
- that module holds a node seat for the runtime, so a second runtime (podman) can later compete
for the seat;
- the host stops speaking to the runtime directly and uses the seat's holder. The host stays the one
that decides, and the holder becomes the one that executes;
- every container on the machine is in scope, not only the mesh's. A development environment started
by hand, or a test database a tool runs, is legitimate. The host already calls these *strays*: 3,
8 and 25 on three of the machines on the day of deciding;
- the runtime's events and verbs are subjects on the bus, and the mesh's own interface is built on
them. The third-party interface run until now was removed by hand.
[ADR 0161](0161-what-deserves-a-seat.md)'s test for a seat is whether the mesh's own code finds it by
name. Here it does: the host would look up the holder on its own machine. A singular role of a module
held once per machine is a `node-*` seat ([ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md)),
in the controller's seed.
The constraint that decides most of this record is a cycle. The bus runs in containers. On the
broker's machine, the broker's own container is created by the host. A holder's code served from a
container cannot create the container that runs it. On a first machine, before the controller exists,
nothing holds anything.
## Considered Options
1. **The host calls the holder's verbs over the bus.** Rejected: with the broker down, no machine can
create any container, including the broker's. The mesh would be unable to restart its own
transport.
2. **The holder picks a dialect that the host speaks itself, as with the uplink.** Rejected: the
host would still drive the runtime, and the module would drive it too for every other caller.
That is two programs speaking to one daemon, and they come to disagree about the same machine
(the installer's preflight already exists to avoid this).
3. **The holder's code runs as a supervised process on the machine and serves the seat's verbs
twice: locally to the host, on the bus to everyone else.** Adopted.
## Decision
**`node-container-runtime` is a seat of the mesh's own, node-scoped,** in the controller's seed under
this record. It delivers no provision; what it carries is its role's protocol: verbs its holder must
serve ([ADR 0132](0132-a-seat-carries-the-tools-its-holder-must-serve.md)) and events its holder emits
([ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md)). The runtime's module, `docker`, claims
it and is assigned to every machine. A podman module may claim it later; one machine runs one.
**The seat's verbs cover every container on the machine:** list, inspect, logs, stats, start, stop,
restart, create and remove. A mesh-held container is marked by the host's label and says which
assignment holds it. **A container the runtime runs can be root on the machine** — privileged, a host
path mounted, the host's network or process namespace, the runtime's own socket — so a caller other
than the host may not create one that is any of these; only a declaration the mesh composed may ask
for them. And the verbs that change anything are granted by name, never by a wildcard: a grant of
every tool (the console's today) reaches the reading verbs only. Issue 193 is what a verb that trusts
its caller costs. **Creating or removing a mesh-held container is the host's alone.** Any other
caller is refused naming the assignment, because the host would undo it at its next apply. Starting,
stopping or restarting one is allowed, and the answer says the host will restore what its
declaration says. A container the mesh does not hold is the caller's to do anything with.
**The seat's events are the runtime's own** — a container created, started, stopped, died, removed,
its health changed. They are emitted on the seat's subjects, so every holder emits the same events and
no reader depends on which runtime holds the seat. As
[ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)
decides, the subjects are issued by the controller, not composed by the module.
**The holder's code is a supervised process, not a container** ([ADR 0150](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md)).
A runtime cannot be run by the thing it runs. The process serves the seat's verbs on the bus to the
console, to tools and to the mesh's interface. The same verbs are served on a local socket on the
machine, which only the host may use. **The host creates, inspects and removes its containers
through that socket and nothing else.** If the holder does not answer, the host creates nothing. It
says so in its report, naming the seat. It never falls back to the command line.
**A container needs the seat held on its machine.** An assignment that delivers a container on a
machine whose runtime seat is unheld is refused, naming the seat and its candidates
([ADR 0165](0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md)).
Mounting the runtime's socket into a container is granted by the seat, not by the capability. The
socket's path is the holder's to state, because podman's is not docker's.
**The runtime module owns the runtime's configuration.** Its settings are declared with defaults
([ADR 0164](0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md)):
the resolver containers use, the registries it trusts, log rotation, and keeping containers through a
daemon restart. The module is given the resolver's address and the mesh's registry as values; no other
module writes the runtime's file or declares its service.
**The first machine is bootstrapped with the holder, and adopted afterwards.** The foundation bundle
already installs the runtime's package and service. It also carries the holder's process, delivered as
a binary the way the host is ([ADR 0142](0142-the-mesh-delivers-its-own-components-as-binaries.md)).
When the runtime module is assigned, it adopts what the bundle made, as the store and broker modules
adopt theirs ([ADR 0078](0078-the-store-and-broker-are-modules.md)).
## Consequences
- **The migration on the running mesh has a fixed order:**
1. Each machine's hand-written configuration is read, because the module's defaults replace what
differs.
2. In one push per machine: the resolver module and the private network stop writing the
runtime's file ([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)),
and the runtime module is assigned and adopts the runtime, its file and its service. Split in
two, either the controller refuses two modules declaring one path, or a machine is left with
nothing setting `dns` and `live-restore`.
3. The controller seeds the seat and enforces the container requirement.
4. The host releases the version that uses the holder.
5. The host's command-line path is removed in the release after every machine's holder answers.
Until then, the host reports per machine which path it used.
- **Every container the host makes depends on the holder's process.** A crash-looping holder stops
new containers on its machine. Running containers are unaffected. The host's report names the cause.
- The process form of a module's own code must serve tools on the live mesh before this ships. Only the
showcase declares it, and [issue 117](../04-ISSUES/117-a-modules-own-code-is-a-container-and-a-process/01-diagnosis.md)
found the showcase's tools declared in a form nothing runs. The runtime module is the first whose
tools cannot fall back to a container.
- A user interface subscribing to events directly does not exist. Today a reader of events is a module
that consumes them. The mesh's container view is a module, or waits for that path.
- [ADR 0005](0005-the-node-host.md) ("a container runtime is detected, not chosen") and
[ADR 0006](0006-the-substrate-and-the-control-plane.md)'s matching line describe the mechanism this replaces: on
acceptance, each gets a dated note saying the runtime is now a seat's holder, as the decision
records' rule for a moved mechanism requires. Design 05 and design 26 are amended after acceptance.
- The operator's decision to remove the third-party interface by hand needs no mechanism. No
module-retires-module rule is introduced.
- **What got harder:** the host gains a dependency it did not have, and a first machine's bundle gains
a component. The direct path was simpler and is what makes a runtime a black box to the rest of the
mesh.
## How this is checked
| Rule | Checked by |
|---|---|
| The seat is the mesh's own, node-scoped, with its verbs and events | A catalogue test on the default seats; registration refuses a claimant that does not serve every verb (design 33's existing check) |
| Creating or removing a mesh-held container is the host's alone | A test of the runtime module's verbs: create or remove of a container carrying the host's label, from any caller but the host's socket, is refused naming the assignment; the same verbs on an unlabelled container succeed |
| No caller but the host creates a container that is root on the machine | A test of `create` from the bus: privileged, a host path, the host's namespaces and the runtime's socket are each refused; the same request on the host's socket is accepted. A broker test: a grant of every tool does not reach a changing verb |
| The host uses the holder and never the command line | A host test with a fake holder on the local socket: every container operation goes to it, and with the holder absent the apply creates nothing and reports the seat; after step 5, the host carries no command-line runtime code (checked by build: the package is gone) |
| A container needs the seat held | A resolution test refusing a containerised assignment on a machine with the seat unheld, naming the seat |
| Socket mounts are granted by the seat | A catalogue test: a module mounting the runtime's socket on a machine whose holder states a different path is refused |
| No other module writes the runtime's file | The existing collision check, once the private network's computed resources are inside it ([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)) |
| Live | `seats` lists `node-container-runtime` held on every machine; the node listing shows each machine's containers, strays included, from the seat's `list` verb; a container started by hand appears as an event on the bus |
## References
- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md), [ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md), [ADR 0132](0132-a-seat-carries-the-tools-its-holder-must-serve.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md), [ADR 0161](0161-what-deserves-a-seat.md)
- [ADR 0150](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md), [ADR 0142](0142-the-mesh-delivers-its-own-components-as-binaries.md), [ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0005](0005-the-node-host.md)
- [ADR 0164](0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md), [ADR 0165](0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md), [issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)
- [Design 26 — the seats](../03-DESIGN/01-to-be/26-the-seats.md), [design 33 — the tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)
- mesh-host `internal/apply/apply.go` (the runtime lookup and the command line it drives)
@@ -9,6 +9,8 @@ extends: 02-DECISIONS/0150-a-modules-own-code-runs-as-supervised-processes-under
# 175. One tool runtime per node serves every module's tools, on the host side
> **The mechanism changed — 2026-10-02, by [ADR 0188](0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md).** Everything decided here stands: one runtime per node, host-side, every module's tools and every held seat's verbs on the memberships' subjects, root the module's concern, any node calls any tool, the console its serving mode. What moved is how the runtime brings a bundle to life. Decision 3 and the consequence *the node tools runtime needs an interpreter on the machine* read as though a bundle were always interpreted code the runtime imports; a tools bundle is now a process in any language that speaks MCP over stdio to the runtime, and importing a TypeScript bundle is the shortcut, not the contract.
## Context
A module's tools are code the module wrote, one function behind each verb, served on the subjects
@@ -0,0 +1,132 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
---
# 179. The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail
## Context
Read on the control node on 2026-10-02, the day the machines were confirmed filtered by the mesh
alone ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)): the intrusion
prevention watched one door. Its two jails read the ssh daemon's journal and its own log, banned
five failures in ten minutes for ten minutes, and in a day had seen twelve thousand failed logins
from three hundred addresses and banned none of the busiest, which paced themselves at one try every
ten minutes. The mail submission port took a hundred and sixty password guesses in the same day from
thirty-eight addresses with no jail reading it at all; the forge and the public proxy had no jail
either, and the proxy logged nothing a jail could read. Nobody could see the jails without a shell:
the module's three tools existed in code and were served by nothing, and the seat it holds declared
no verbs.
Three things were missing and they are three shapes the mesh already has. The packet filter's seat
serves verbs every holder owes ([ADR 0170](0170-the-firewall-seat-serves-its-verbs.md)); the
intrusion seat serves none. A module's `listens` compose into the machine's filter, and [to-be 31](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
says a module's `jails` compose into the machine's intrusion prevention the same way — the controller
composes them, and no module declares one. And a jail reads a log; a container's output goes to a
file of the runtime's own under a path that changes when the container is recreated, which is why
no jail could read the mail front end, the forge or the proxy, however they logged.
## Decision
**1. The `node-intrusion-prevention` seat serves four verbs**, and a module that claims it serves
all four or is refused the claim, as with every seat:
- `status` — every jail with what it watches, how many addresses it is counting failures against and
holding now, and the totals since it started; one jail's detail when named. Read-only.
- `banned` — every address banned now, with the jail holding it, when it was banned and when the ban
ends. Read-only.
- `ban` — ban one address in one jail now, for that jail's ban time. An operator's act on the live
ban list, which the mesh composes the rules for and never writes itself.
- `unban` — let one address go, from one jail or from every jail.
A holder may serve its own tools beside these; the fail2ban module reads one jail's effective
settings as its own.
**2. A container may log to the journal.** `logging: journald` on a container has the host run it
with the journal as its log driver; the journal keeps the container's name on every line, and
`docker logs` keeps working. Where a container logs is part of its spec, so moving it recreates the
container, and the only place besides the runtime's own file is the journal: a machine's intrusion
prevention reads the journal already, for the ssh daemon, and a container that logs there is read
the same way, by the container's name, whatever the container is called by the runtime this time.
**3. A module with a door declares its jail, and the holder composes them.** What to-be 31 designed
is now the rule: a module whose service authenticates from outside — the mail front end, the forge,
the public proxy — declares in its manifest what a failed attempt looks like in its log and how to
ban on it, naming no node and no path; the module that holds the intrusion seat declares where the
composed jails and filters land, and the mesh writes them on every machine that runs both. A machine
not running the module has no such jail. The holder restarts its daemon on the composed file.
**4. The base is strict, and the mesh's own range is never banned.** Three failures in a day ban for
a day, on every jail unless the jail says otherwise; banned twice in two weeks, by any jail, is
banned for four. The attackers this mesh sees pace themselves under any ten-minute window; a day's
window counts them. A person who mistypes three times from one address is out for a day from that
address, and never from a machine of the mesh, whose range stays in the never-banned list the module
has carried since [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md). The operator
chose this knowing it.
**5. The proxy says a refused name in its log.** A request for a name this mesh does not serve, from
outside, is what a scanner does; the proxy already logged a certificate refused for such a name, and
now logs the plain request too, with the asking address last, as its own jail's filter expects it.
## Consequences
- The seat's row gains four verbs; a mesh that already runs widens its row at the next controller
start. The fail2ban module claims them and gains a runtime — a tool server whose image carries the
fail2ban client, with the daemon's socket shared in from the machine, and nothing else of the
machine. The daemon stays the machine's; what runs in the container is only the client.
- **That runtime is the shape the catalogue has today, and it is on its way out.**
[ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md),
accepted the same day as this record, replaces a tool container per module with one tool runtime
per node on the host side, taking each module's tools as a bundle. Nothing here depends on the
container: the verbs, the client that speaks to the daemon over its socket, and the jails are the
same code under either. This module converts with the packet filter's, whose runtime that record
names, and the socket it needs becomes the node runtime's to reach rather than a mount of its own.
- The host's container vocabulary grows by `logging`; an older host refuses a declaration that carries
it, so the host rolls before the modules. Three containers are recreated once, when their modules
are pushed with the field: the mail front end, the forge and the proxy — each a moment's outage.
- The fail2ban module declares where jails compose (`jailing`) and the directory the filters go in;
the mail, forge and proxy modules each declare one jail reading the journal by their container's
name. The composed jail file is the one resource the daemon restarts on when a module arrives or
leaves a machine.
- The two base jails and the composed ones take the day's window; the ssh jail's ten minutes are
gone. An address banned on the first day of this record stays banned for the day.
- The module's three old tools, served by nothing, are replaced by the seat's four verbs and one
own tool; `fail2ban_status` as a name is gone.
## How this is checked
| Rule | Checked by |
|---|---|
| The seat declares the four verbs; a claim that serves fewer is refused by name | the catalogue's seat tests |
| `status`, `banned`, `ban` and `unban` read and steer the daemon through its client, with the shapes fail2ban 1.1.0 printed live; a non-address and a non-name are refused before anything runs | the module's tests over a fake command runner |
| A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests |
| A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) |
| The proxy logs a refused name with the address last | the proxy's tests |
| A jail's pattern names `<HOST>` once per shape, since two is a duplicate capture group and costs the machine every ban | the catalogue's manifest tests |
| Live | done 2026-10-02: `status` and `banned` answered on both servers through the console; the proxy's jail counted seven refusals on the home server; a documentation address banned in the ssh jail came back with its end time and was released |
## Built and proven live, 2026-10-02
All five rules are in the mesh. The host carries `logging`; the controller's seat row carries the four
verbs and the proxy says a refused name in its log; the fail2ban module holds the seat from a runtime
with the daemon's socket shared in, composes the jails, and the mail front end, the forge and the
proxy each declare one. Through the console on the control node: `status` listed five jails with what
each watches, `banned` listed the nine the long jail holds, and a documentation address banned in the
ssh jail came back with its ban's end time and was released again. On the home server the proxy's jail
had counted seven refusals within minutes of starting.
**One fault, found by the machine and not by a test.** The proxy's pattern matched two shapes of
refusal in one expression and so named `<HOST>` twice. fail2ban expands that placeholder into a named
capture group; two of them is a duplicate group name, and the daemon refuses *its whole configuration*
and exits — both servers kept no bans at all for about ten minutes, every jail and not the one at
fault. The pattern is now one per shape. A manifest check refuses the mistake at merge time, naming
what it would cost, which is the only reason this record can claim the rule rather than the instance.
## References
- [ADR 0170](0170-the-firewall-seat-serves-its-verbs.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
- [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md), [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md), [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md)
@@ -60,7 +60,7 @@ configuration file; the rollback path it described is given up on purpose.
|---|---|
| An absent package is removed when present, left when not, and read back | host tests over a fake package manager |
| An uninstalled front end is recorded as removed and nothing is asked of it | a host test with ufw missing on a converged apply |
| Live | the two machines report ufw gone: `pacman -Q ufw` has no answer, `node show` says removed, `status` is well |
| Live | done 2026-10-02: both machines report ufw gone — `pacman -Q ufw` has no answer, `node show` says *removed*, `status` is well. The home server said *retired* for six hours after the package went, because this record's step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)) and one dead tracker was failing its applies ([ADR 0187](0187-a-dead-tracker-is-not-the-machines-failure.md)) |
## References
@@ -0,0 +1,75 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0005-the-node-host.md
---
# 184. A service the mesh asked to run is still running a moment later
## Context
The host already refuses to take a service manager's word for it. Three places in one function read
a unit back after acting on it, each with a comment saying why: *a service manager accepting a
command says the transaction was accepted, not that the unit is running — one that starts and
immediately dies satisfies it.* The intent was right and the implementation did not reach it.
On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the
files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at
that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public
machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said
so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing.
The read-back races the failure. A service manager returns when it has started the process; a daemon
that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One
look sees `activating` or `active` whatever the process is about to do, and *the host reports success
for a machine that is already wrong* — the one shape of failure this host exists to refuse
([ADR 0005](0005-the-node-host.md)).
A command the module declares — *test the configuration before restarting* — was considered and
rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification
command is a command: a declaration that carried one would be remote execution over the bus,
arriving as root on every machine, which is a far larger door than the fault it closes. The host
does not need one. It already knows what it asked for.
## Decision
**1. A unit the host has just asked to run is read twice**, with a pause between the reads long
enough for a daemon that refuses its configuration to have exited. Not running at the second look is
a failure of that resource, named with the unit and the state it is in — the same failure the single
read was always meant to catch.
**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both
looks and is accepted, exactly as before. What the second look catches is a unit that *was* running
and is not any more. A service asked to be stopped is not waited on at all.
**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking
the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and
no command reaches a machine that did not already come from a built artifact.
## Consequences
- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is
bounded by the number of services that changed in that apply, which is usually none.
- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the
resolver — now fails its apply when the composition is bad, instead of reporting success onto a
dead daemon. `status` names the machine, which is how the operator finds out.
- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s
manifest check is what refuses the one that caused this, at merge time; this record is what makes
the *next* one visible within a minute rather than invisible until something asks the daemon a
question.
## How this is checked
| Rule | Checked by |
|---|---|
| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does |
| A unit still starting is accepted at both looks | a host test |
| A service asked to be stopped is not waited on | a host test |
## References
- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md)
@@ -0,0 +1,73 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
---
# 185. A control plane behind its seat's row serves what it can
## Context
The mesh's own verbs are the controller seat's tools, and the seat's row is the store's
([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). A control plane reads the
row at start and installs a handler per verb; a verb the row carries that the binary cannot run was
refused at start rather than at the first call, so that a disagreement between the row and the
binary was said early. The refusal aborted the start.
On 2026-10-02 a merge added one verb. The new control plane started, widened the row, and ran. A
push a few seconds later recreated its container at the previous image — a stale declaration from
an overlapping wave, [issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) —
and the older binary read a row naming a word it had never heard. It refused to start, and kept
refusing. The mesh had no voice for ten minutes: no verb answered, no node could be pushed, no build
was dispatched, and `status` said nothing because `status` is one of the verbs that had stopped
being served. The way back was a person running the binary by hand outside its service, because the
push that would have replaced it is itself a verb of the control plane that was down.
The check was right about the fact and wrong about the cost. A row ahead of a binary is the ordinary
state of a roll-out: the row is widened by whichever control plane starts first, and a mesh with one
control plane sees that gap on every merge that adds a verb. Making it fatal turned a transient into
an outage with no path out that did not need a human.
## Decision
**1. A control plane serves the verbs it can run and does not refuse to start for the ones it
cannot.** The row remains the authority on what the seat serves; this is only about what this binary
does when it is behind the row.
**2. A verb it cannot run answers the reason.** Not silence and not a missing subject: a caller gets
a sentence naming the verb, saying this control plane cannot run it and that it is a verb of a newer
build. A verb that is simply absent from the row is still not served at all — that is the row
deciding, which is unchanged.
**3. It says so once at start**, naming every verb of the row it cannot run, so the gap is visible
in the log of the thing that has it rather than only at the moment somebody calls one.
**4. A mesh with no controller seat at all is still a refusal.** That is not a version gap, it is a
mesh that has not been seeded, and nothing this control plane does would be meaningful.
## Consequences
- An overlapping roll-out costs the verbs the newer build added, for as long as the older binary is
in place. Everything else — every push, every build, every read — keeps working, and the ordinary
machinery that notices a machine is behind is what puts the newer binary back.
- The log gains one line on a control plane that is behind, and nothing on one that is not.
- Issue 201's other half remains: the push that sent a stale declaration is a race worth closing on
its own terms. This record makes that race survivable rather than fatal, which is the difference
between a transient and an outage, and is deliberately the cheaper half.
## How this is checked
| Rule | Checked by |
|---|---|
| A row carrying a verb this build cannot run still serves every verb it can, and names the one it cannot | a controller test over a widened row |
| The unknown verb answers a sentence naming itself and saying this build is behind | the same test |
| A mesh with no controller seat is refused | the existing start-up path |
## References
- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
- [Issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)
- [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)
@@ -0,0 +1,81 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
---
# 186. A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang
## Context
[ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) gave the
public proxy a jail. Within the hour the home server's ban list held `192.168.1.1` — the house's own
router. The router reflects local traffic, so every client in the building reaches that machine as
the gateway's address; one local request for a name the mesh does not serve, three times in a day,
and the whole house is refused by the machine it was asking. The jails inherited an `ignoreip` of
the loopback and the mesh's own range, which was right when the only jail read the ssh daemon and
the only clients were the mesh's; a jail on a public front door sees the neighbours too.
The same jail broke the other half of [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md).
The home server began reading *NOT the mesh alone: 1 rule set the mesh did not write refuses traffic
here*, and the rule set named was the mesh's own ban chain, written by the mesh's own intrusion
prevention minutes earlier. The host's reader of the legacy filter required every path into a chain
of refusals to come from a built-in chain whose policy accepts, before it would call that chain a
ban. On that machine the chain hangs off the container runtime's user chain as well as the input
chain, and the runtime had set the forward policy to DROP — so the mesh reported its own work as a
foreigner's, on the one machine where the group's exit condition was supposed to hold.
Both faults are one mistake in two places: a rule written about the public internet, applied to
everything that arrives.
## Decision
**1. A ban list never holds a neighbour.** The jails the mesh composes never ban a source on a
private range — the mesh's own range, which was already named rather than written
([ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md)), and every address space
reserved for private use beside it, in both families. A machine behind a router that reflects local
traffic sees its whole building as one address; a ban there is a self-inflicted outage, and the
sources worth banning are not on those ranges in the first place.
**2. The mesh's own bans are its own wherever they hang.** A chain of refusals is a ban list when
every refusal names the sources it refuses and the chain accepts nothing — the rule the host already
applied to the packet filter's own tables, now applied to the legacy filter too, and nothing more.
The policy of the chains that jump into it says nothing about what it is: that policy is already
classified where it belongs, as the container runtime's, and requiring it here counted it twice.
**3. A chain that accepts anything is still not a ban.** That is what keeps a predecessor's
allow-these-and-drop-the-rest chain classified as something an operator must look at, which is the
distinction [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md) exists to draw.
## Consequences
- The composed jails gain the private ranges in their never-ban list. An address already banned
stays banned until it is released; the house's router was released by hand the moment it was found.
- The home server reads *the mesh alone* again, which is group 7's exit condition and was false for
about an hour.
- A machine whose apply fails for an unrelated reason does not revisit its found firewall's record
at all — the step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)).
The home server's record therefore still reads *retired by the mesh* although the front end is
uninstalled, and will correct itself once that machine's own stuck module is fixed. It is a stale
record, not a wrong machine.
- The record number the front end's removal was given moved under it: another session took 0175
while that record was in review, and it is now
[ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md). The citations
the host and the control plane print were pointing at an unrelated record and are corrected here.
## How this is checked
| Rule | Checked by |
|---|---|
| A private source is never banned | the module's jail configuration, read back by `fail2ban.fail2ban_settings` on a machine |
| The mesh's own ban chain reads as a ban behind a dropping forward policy | a host test over the home server's own captured rule set |
| A chain that accepts anything is not a ban | a host test |
| Live | done 2026-10-02: all four machines read *the mesh alone*, the home server counting its own ban chain as a ban; no ban held anywhere is a private address |
## References
- [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
- [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md), [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
@@ -0,0 +1,73 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
---
# 187. A dead tracker is not the machine's failure
## Context
The home server had not applied a declaration cleanly since midday. One run-once step — the one
that writes a media app's download clients and indexers through the app's own API — exited
non-zero, forty-nine times over six hours, for one public tracker that had stopped answering. The
step's own words: the entry was *written*, and the app's test of it then failed with a 400 from the
indexer proxy. The machine reported *not doing what it was told* for the rest of the day.
What that gated matters more than the step. A converged machine retires the firewall it was found
with only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)),
so that machine went on recording its found front end as merely *retired* long after the package
had been uninstalled ([ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)).
A dead public tracker was holding a firewall record hostage, which is not a connection anybody
would design.
The step already knew this was not its business. It had a rule for exactly this: an entry the mesh
only *found and re-pointed*, rather than one it was told to make, whose feed is gone, is said and
left as found — *failing the node's apply on every heartbeat for it reports the mesh as wrong about
a tracker*. The rule was there and matched one shape of the fault. An app can refuse to save such an
entry, and it can save it and then fail its own test; saving validates settings, and the test runs a
live search. The rule caught the first and let the second through.
## Decision
**1. An entry the mesh only found is never the machine's failure.** Whatever shape the app's
refusal takes — it would not save it, or it saved it and its own test fails — an indexer the mesh
found and re-pointed is reported as a notice and left as found. What decides is whose entry it is,
not which sentence the app returned.
**2. What the mesh is answerable for is the plumbing.** That the entry exists, points at this
mesh's indexer proxy, and carries the credential the mesh delivered — which was checked against the
proxy before anything was written. Whether a public tracker answers today is not the mesh's to
promise, and a machine that reports itself broken because one did is lying about itself.
**3. An entry the operator listed is theirs to insist on.** An indexer named in the step's settings
is one the mesh was told to make, and it still fails the step when it cannot be made to work. The
notice says so, and says that listing the indexer is how to turn it back into a failure.
## Consequences
- The home server applies cleanly again, and everything a clean apply gates — its found firewall's
record among it — follows.
- A tracker that dies is a line in a report rather than a machine that reads as broken. An operator
who wants it gone removes the entry or repairs the feed; the mesh says which, every time it runs.
- The four Servarr modules carry one byte-identical copy of this step each
([ADR 0069](0069-a-module-is-a-repository-and-a-path.md)), so the change lands in four places and
a test refuses any drift between them.
- It does not widen to a download client: one the mesh was told to write and cannot is still a
failure, because the mesh chose it and nothing else will fix it.
## How this is checked
| Rule | Checked by |
|---|---|
| A found feed whose tracker answers an error after the entry was written is a notice | the step's tests, with the home server's own message and the app's two validations modelled apart |
| An indexer the settings list is still a failure | the same test |
| The four copies of the step do not drift | the step's own sameness test |
| Live | done 2026-10-02: the home server applies cleanly after six hours of failing, `status` holds no machine wrong or behind, and its found firewall reads *removed* |
## References
- [ADR 0136](0136-a-step-gates-its-module-not-the-machine.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md), [ADR 0069](0069-a-module-is-a-repository-and-a-path.md)
@@ -0,0 +1,127 @@
---
topic: what runs on it
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
---
# 188. A module's own code is bundles in any language, and a tools bundle speaks MCP to the runtime
## Context
[ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md) put one
tool runtime on every node and said a module brings its tools as a bundle. The runtime that exists
is written in TypeScript and brings a bundle to life by **importing it into its own process**, which
only JavaScript can be. The SDK ([ADR 0039](0039-what-the-sdk-holds-and-refuses.md)) is one
TypeScript package. The builder knows three toolchains — TypeScript, Go, Python — and every one of
the 35 catalogue modules with tools wraps them in a container on the runtime's TypeScript image.
Nothing in the records says a module's code may be written in anything else, and nothing refuses a
module that wraps its own code in an image to get around that.
The operator's direction, stated on 2026-10-02 and repeated: *the SDK is the most important part;
we must not limit developers; tools can be written in any possible language — Rust, C, Go,
JavaScript. A service in Go or Rust as a systemd unit must be possible too. One module can deliver
all kinds of bundles: one for its tools, one for a seat's implementation, one for a daemon. Support
the bare minimum first, as a skeleton; a full implementation comes when the work requires it.*
Measured against that: the `bundle` artifact kind already names a language and the `process`
resource already runs a command from an unpacked bundle as a unit the host writes
([ADR 0150](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md)), so a Go
daemon as a native service is possible today and one module in the catalogue does it. What is not
possible is a tool in any language but one, and what is not written is that any of this is the
rule.
## Considered Options
1. **One SDK, one language, as now.** Rejected: it limits who can write a module to one
ecosystem, which the operator declines, and it is what made every module's tools a container
on one image.
2. **A full bus client per language.** Each SDK speaks the bus itself; the runtime only
supervises. Rejected: a transport in every SDK is what ADR 0039 refuses, and a bus change
would then rebuild every module in every language — the cascade, multiplied.
3. **A tools bundle is a process the runtime launches and speaks a small local protocol to,
and that protocol is MCP over stdio.** Chosen. The runtime already speaks MCP outward (the
console); speaking it inward to a child process is the same vocabulary. Every language that
has an MCP server library can write a tools bundle today with no mesh SDK at all, and the
mesh's own SDK for a language is a thin convenience over it. The transport stays in the
runtime, so a bus change rebuilds nothing.
4. **A protocol of the mesh's own design.** Rejected: a second way to describe a tool, its
schema and its call, inventing what MCP already settled, for no gain.
## Decision
**1. A module's own code is bundles, in any language the mesh has a toolchain for, and never an
image.** A `bundle` names its language and what it is for. Images are for third-party software a
module installs — a database, a forge — never for code the module wrote. One module may declare
several bundles: its tools, its implementation of a seat's verbs, a daemon, a step. Each is built
alone and delivered alone, as [ADR 0156](0156-an-artifact-is-what-a-build-produces-and-the-store-is-named-for-its-scope.md)
already has it.
**2. A bundle the runtime serves is a process that speaks MCP over stdio.** The node's runtime
launches it as the bundle names it — an interpreter and a file, or a binary — with the runtime's
environment, asks `tools/list`, and answers each call on the bus by `tools/call`. A tool whose name
is `<seat>.<verb>` is the module's implementation of that seat's verb; any other name is the
module's own tool. Everything the runtime does with what it is told — subjects from the membership,
a held seat's verbs, the `tools` answer, a bundle that fails named and the others serving — stays as
[ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md) and
[ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)
have it. A TypeScript bundle may still be imported into the runtime's own process; that is a
shortcut over the same contract, not a second contract, and a TypeScript bundle written against
the protocol is served the same way as any other.
**3. A bundle that is a service is a `process`**, run by the host as a unit, in whatever language it
is compiled from, exactly as the host's own bundle already is. Nothing new is decided here; it is
said so that it is the rule and not an example.
**4. One thin SDK per language, and the test of ADR 0039 applies to each.** An SDK for a language
holds the MCP-over-stdio loop, the tool-definition type and the few primitives a module's code
needs; it holds no transport, no module's client and nothing volatile. Where a language has a
sound MCP library, the SDK wraps it rather than re-implementing it. The languages are those that
make sense to write a module in; the first set is TypeScript, Go, Python, Rust and C, and the set
grows when a module needs one, not before.
**5. Skeleton first.** Each piece — a toolchain, a launcher, an SDK — exists at the bare minimum
that lets one bundle in that language be built, delivered and answer one tool on the live mesh.
Anything beyond that is added when a module needs it. A skeleton that is not proven by one bundle
answering is not a skeleton; it is a promise.
## Consequences
- The runtime gains a launcher beside its loader. The loader, the memberships, the seats and the
failure handling built for ADR 0175 stand; the launcher is the one new step.
- The builder gains a toolchain per language, each at the skeleton: compile, pack, name the
entrypoint. Rust and C are new; a language that compiles to a binary says its operating system
as a Go bundle already does.
- An existing MCP server in any language is already a valid tools bundle. What the mesh adds is
the subjects, the seats and the memberships around it.
- The gate [to-be 38](../03-DESIGN/01-to-be/38-building-the-operators-machine.md) WP2 adds —
refusing a tools container built on the runtime's image — widens: a module whose own code is
an image artifact is refused at registration, naming this record.
- What got harder: a tools bundle is now a process per module on the node rather than code in
one process, so the runtime supervises children and restarts one that dies. The one-process
shape ADR 0175 counted on for the TypeScript shortcut remains available for it.
- ADR 0039's "what the SDK holds" now reads per language; its refusals are unchanged and are the
reason option 2 was rejected.
## How it is checked
| Rule | Checked by |
|---|---|
| A module's own code is never an image | the catalogue's registration check: a manifest with a `bundle` kind of own code *and* an image artifact built from the module's own directory is refused, naming this record |
| A tools bundle in a language other than TypeScript answers on the bus | the runtime's tests: a bundle written against the protocol in a second language, launched, its tool called over a real bus |
| A TypeScript bundle written against the protocol is served like any other | the same tests, with the TypeScript shortcut off |
| Each SDK is thin | each SDK's own README states what it holds under ADR 0039's test, and its size is in the mesh's records |
| Live | a tool in a compiled language answers from the node's runtime on one machine |
## References
- [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md),
[ADR 0039](0039-what-the-sdk-holds-and-refuses.md),
[ADR 0150](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md),
[ADR 0156](0156-an-artifact-is-what-a-build-produces-and-the-store-is-named-for-its-scope.md),
[ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)
- [To-be 38](../03-DESIGN/01-to-be/38-building-the-operators-machine.md) — the work packages this
record widens
- The Model Context Protocol's stdio transport — the local protocol a tools bundle speaks
+9
View File
@@ -182,7 +182,12 @@ python3 00-META/checks/index.py fail if stale
- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)
- **0170** — [The firewall seat serves its verbs, and a foreign rule set is removed through one of them](0170-the-firewall-seat-serves-its-verbs.md)
- **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md)
- **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
- **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
- **0184** — [A service the mesh asked to run is still running a moment later](0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md)
- **0185** — [A control plane behind its seat's row serves what it can](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)
- **0186** — [A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang](0186-a-ban-list-never-holds-a-neighbour.md)
- **0187** — [A dead tracker is not the machine's failure](0187-a-dead-tracker-is-not-the-machines-failure.md)
### Its tiers, from the bottom up
@@ -273,6 +278,9 @@ python3 00-META/checks/index.py fail if stale
- **0150** — [A module's own code runs as supervised processes under the module's one account](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md)
- **0152** — [The operator's surface is a module the mesh assigns: the console](0152-the-operators-surface-is-a-module-the-console.md)
- **0155** — [A definition names no installation: how that is checked, and the three ways a value that did gets out](0155-a-definition-names-no-installation-and-how-that-is-checked.md)
- **0164** — [A setting is declared with its default, its meaning and what changing it costs](0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md) *(proposed)*
- **0165** — [`container-runtime` is what a machine can run; that a runtime is running is its holder's health](0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md) *(proposed)*
- **0166** — [The container runtime is a node seat, and the host creates containers through its holder](0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md) *(proposed)*
- **0173** — [The operator's machine is the mesh's, and a module is whatever it declares](0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md)
- **0175** — [One tool runtime per node serves every module's tools, on the host side](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
- **0176** — [The login shell is a node seat held by one shell module, and `execute` is its contract](0176-the-login-shell-is-a-node-seat-and-execute-is-its-contract.md)
@@ -280,6 +288,7 @@ python3 00-META/checks/index.py fail if stale
- **0181** — [The operator account is a node fact, and a home is a placement root](0181-the-operator-account-is-a-node-fact-and-a-home-is-a-placement-root.md)
- **0182** — [Inside a home, the mesh owns the directory and the files it places, writes into the tool's own files, and holds everything else as found](0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md)
- **0183** — [The Anthropic licence manager is a module holding a seat; it hands each node's agent its token over the bus, sealed; the controller and the host have no part](0183-the-anthropic-licence-manager-is-a-module-and-hands-tokens-to-the-agent-over-the-bus.md)
- **0188** — [A module's own code is bundles in any language, and a tools bundle speaks MCP to the runtime](0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md)
### How it is built
+15
View File
@@ -4,6 +4,7 @@ status: in-progress
code: [mesh-host]
updated: 2026-10-02
decisions:
- 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md
- 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md
- 02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md
- 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
@@ -498,3 +499,17 @@ run and reported; the exit follows an in-flight apply rather than interrupting i
not start is rolled back once and the second failure halts; a completed reconcile retires what is older
than the predecessor and never the predecessor; and the newest of two delivered versions is the one
that runs.
## A service is still running a moment later, 2026-10-02
[ADR 0184](../../02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md).
The host has always read a unit back after acting on it, because a service manager accepting a
command says the transaction was accepted and nothing about the process. The read raced the failure:
a daemon that refuses the configuration the mesh just wrote exits a fraction of a second after the
manager returns, and one look sees it alive. So the host looks twice, with a pause between, and a
unit that was running and is not any more fails its resource by name. A unit still coming up reads
as running at both looks and is accepted; a service asked to stop is not waited on.
No command for this reaches a machine. A module declaring *how to test my configuration* was weighed
and refused: the link carries no actions, and a verification command is one. The host is checking
the state it was told to establish, which is what it is for. *How it is checked:* ADR 0184's table.
@@ -1,10 +1,15 @@
---
layer: to-be
status: proposed
code: []
updated: 2026-09-27
status: in-progress
code:
- mesh-controller: internal/catalogue/jails_into.go, internal/catalogue/manifest.go (Jail, Jailing)
- mesh-catalog: modules/fail2ban (jailing, the base and the seat's verbs), modules/mailu, modules/route-proxy, modules/gitea (jails)
- mesh-host: internal/declaration/declaration.go (a container's logging)
updated: 2026-10-02
decisions:
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
- 02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md
---
# 31 — A module declares its fail2ban jail, and the mesh composes them per node
@@ -63,3 +68,38 @@ jail, composed from the postgres module's manifest, without anyone editing a nod
beside)
- mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules
(`postgres`, `mssql`, `mailu`) that will declare jails
## Decided and built, 2026-10-02
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
made this the rule and built it. A module declares `jails` — each a name, the `failregex` of a
failed attempt in its log, and the stanza's own keys — and the fail2ban module declares `jailing`:
the one file the stanzas compose into and the directory each filter lands in. The controller gathers
every assigned module's jails per node into those; the holder's daemon restarts on the composed file.
What made it workable was the log. A container's output went to a file of the runtime's own, under
a path that changes when the container is recreated, so no jail could read a container's service
however it logged. A container now declares `logging: journald`, the host runs it with the journal as
its driver, and a jail reads it with `backend = systemd` and a `journalmatch` on the container's
name — the same way the base's ssh jail has always read the ssh daemon. The first three doors: the
mail front end (every login failure on its proxying ports), the forge (a failed authentication
attempt) and the public proxy (a certificate or request for a name the mesh does not serve, which
the proxy now says in its log). The base is strict — three in a day for a day; twice banned in two
weeks for four — and the mesh's own range stays never banned.
The seat the module holds serves `status`, `banned`, `ban` and `unban`, from a runtime that carries
only the fail2ban client with the daemon's socket shared in; the jails are composed, the ban list is
the daemon's, and both are read through the console.
*How it is checked:* ADR 0179's table.
## What the first jails taught, 2026-10-02
[ADR 0186](../../02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md). Within an hour of the
first public jail the home server had banned the house's own router: the router reflects local
traffic, so every client in the building arrives as the gateway's address. The never-ban list now
holds every private range as well as the mesh's own. And the mesh read its own ban chain as a
foreign rule set on that machine, because the chain hangs off the container runtime's user chain and
that machine's forward policy is the runtime's DROP — the reader now calls a chain of source-named
refusals a ban wherever it hangs, as it already did for the packet filter's own tables.
@@ -11,7 +11,7 @@ code:
- mesh-host internal/apply/apply.go
- mesh-tools src/main.ts
- mesh-catalog modules/mesh-catalog
updated: 2026-09-28
updated: 2026-10-02
decisions:
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
@@ -24,6 +24,7 @@ decisions:
- 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
- 02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md
- 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
- 02-DECISIONS/0187-a-dead-tracker-is-not-the-machines-failure.md
- 02-DECISIONS/0134-the-mesh-says-what-it-applied.md
---
@@ -470,6 +471,21 @@ moment the mesh can mint for itself: **the bus's own accounts** (§the bootstrap
needs an account before it can run) and **the vault's own credential**. Any third exception is a
design failure, and naming these two is what makes a third one visible.
## What a step is answerable for, 2026-10-02
[ADR 0187](../../02-DECISIONS/0187-a-dead-tracker-is-not-the-machines-failure.md). A step gates its
module and not the machine ([ADR 0136](../../02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md)),
but a step that exits non-zero still leaves the machine reporting that it is not doing what it was
told — and a clean apply gates other things entirely, the found firewall's retirement among them. So
what a step calls a failure matters beyond the step.
The rule the media step now follows, and the one to copy: a step fails for what the mesh chose and
can fix, and reports what it merely found and cannot. An indexer entry the mesh re-pointed at this
mesh's proxy is plumbing the mesh is answerable for; whether the public tracker behind it answers
today is not. An entry the operator listed is the operator's to insist on, and still fails. Six
hours of a machine reading as broken, for one tracker that had died, is what the distinction costs
when it is missing.
## 11. Open
**Semantic change has no mechanical defence** (§8). Recorded as open rather than solved, because
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-tools]
updated: 2026-10-02
decisions:
- 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md
- 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
@@ -184,6 +185,18 @@ container to declare a capability. Removing a predecessor's rule set is an opera
through the seat, recorded on the bus, instead of a shell on the machine. *How it is checked:* ADR
0169's table.
## The intrusion seat's verbs, 2026-10-02
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md).
The second node-scoped seat to carry verbs: `node-intrusion-prevention` serves `status` (every jail
with what it watches and holds), `banned` (every address held now, with its jail and when the ban
ends), `ban` and `unban` (an operator's act on the live ban list). The fail2ban module serves them
from a runtime that carries only the daemon's client, the socket shared in from the machine — no
capability, no machine network, since the daemon on the machine does the banning. That runtime is the
per-module container [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
retires; the verbs and the client are the same code once the node's own runtime loads them as a bundle.
The module's own tool beside them reads one jail's effective settings. *How it is checked:* ADR 0179's table.
## What this does not settle
- Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a
@@ -11,6 +11,7 @@ decisions:
- 02-DECISIONS/0177-a-unit-may-be-user-scoped-and-the-service-manager-is-a-node-seat.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md
- 02-DECISIONS/0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md
---
# 38. Building the operator's machine
@@ -97,6 +98,19 @@ what `tools` answers, and the others serve. The runtime reads `MESH_OPERATOR_ACC
five tools and two seat verbs answer on their subjects; `tools` names the failed bundle; a
membership republished mid-run re-subscribes without a restart.
*Built and proven 2026-10-02* (mesh-tools, branch `feat/the-operators-machine`, commit `6390d1d`).
**WP1b — the launcher beside the loader** ([ADR 0188](../../02-DECISIONS/0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md)).
*mesh-tools, mesh-sdk. A day for the skeleton.* A bundle whose entry is not JavaScript is launched
as a child process with the runtime's environment and spoken to over MCP on stdio: `tools/list`
once, `tools/call` per call; a tool named `<seat>.<verb>` is the seat's implementation. A child
that exits is named as a failed bundle and restarted on the next call. The TypeScript import stays
as the shortcut. Beside it, one skeleton SDK per language of the first set — the stdio loop and the
tool-definition type, nothing else — each proven by one bundle in that language answering one tool
in the runtime's test. **Proof.** The runtime's test: a bundle in a second language, launched, its
tool answering on its subject over a real bus; the TypeScript fixture served through the protocol
with the shortcut off answers the same.
## WP2 — The controller composes one runtime per node
*mesh-controller. Two to three days; the largest package.*
@@ -111,12 +125,23 @@ membership republished mid-run re-subscribes without a restart.
declaration gains an `archive` placed under a directory the controller derives, so the host
fetches and unpacks it as it does any artifact. The bundle's digest is what the build recorded.
3. **The runtime's process.** One `process` per node running the runtime from its own bundle
(WP3), `MESH_TOOL_MODULES` composed from the unpacked entrypoints, `MESH_OPERATOR_ACCOUNT` and
(WP3), `MESH_TOOL_MODULES` composed from the unpacked entrypoints — each as
`<module>=<path>`, and the runtime decides from the file whether it is loaded or launched
(WP1b) — `MESH_OPERATOR_ACCOUNT` and
`MESH_OPERATOR_HOME` from the account fact, `restart-on` naming every bundle so a push that
changes one restarts it. A node with no account composes the runtime without the two words.
4. **The gate.** A manifest declaring `tools` and a container built on the runtime's base image is
refused at registration once the runtime module is registered, naming this record. It is the
mechanism that keeps the old pattern from returning by habit.
mechanism that keeps the old pattern from returning by habit. ADR 0188 widens it, after WP4:
a module whose own code is an image artifact is refused, whatever image it is built on.
*Amended 2026-10-02, at WP3.* The gate refuses the pattern **spreading**, not standing: a module
new to the catalogue in that shape, or one that had already moved to a bundle and returns to it,
is refused; a module the catalogue already holds in that shape — judged from the manifest it
holds and what that module's newest build stood on — is rebuilt without complaint. The day the
runtime arrives some thirty such modules stand, each moves in its own change from WP4 on, and a
gate refusing every rebuild in the meantime would stop the catalogue's pipeline to make a point
this record already makes.
**Proof.** Composition tests: a node with three assigned modules, one holding a seat, yields one
process, three archives, one node principal whose grants are the union, and the same three
@@ -138,6 +163,16 @@ runtime's `serve` keeps answering MCP on loopback; the person's end of it keeps
wrote, `tools/list` on loopback answers as before, and the controller's verbs answer through it.
This is the first live step, and it is reversible by re-assigning `mesh-console`.
*Decided 2026-10-02:* `mesh-tools` keeps its name as the module the TypeScript images come from, and
`node-tools` is a second module in the same repository ([ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md))
rather than a rename — the thirty-five manifests that build `on` `mesh-tools` stay true. Three things
WP3 found that the plan did not say: a TypeScript bundle must carry its dependencies and a
`package.json` naming its files as ES modules, which the toolchain now copies in from its own image;
the runtime's credential must be owned by the account the runtime runs as, which the controller
composes; and `MESH_TOOL_MODULES` is empty on a node where the runtime is the only bundle, which the
runtime accepts. *Built 2026-10-02* (mesh-tools `c46f950`, mesh-controller `ca7e81e` `773b561`
`729a5f9`); the live proof follows node by node.
## WP4 — The first holder moves: the packet filter
*mesh-catalog. Half a day. The live proof of ADR 0175.*
@@ -0,0 +1,85 @@
---
status: located
opened: 2026-10-01
located-in: [mesh-catalog modules/dnsmasq, mesh-controller internal/overlay/generator.go, mesh-controller internal/catalogue/resolve.go (checkResources)]
fixed-by:
amended-design:
---
# 190 — The container runtime's configuration is written by modules that are not the runtime's
## What was observed
The runtime's configuration file and its service are declared by two parties, neither of which is
the runtime:
- **The resolver module** writes the runtime's `dns` key (the machine's private address) and
`live-restore` into the runtime's file, written into rather than over
([ADR 0102](../../02-DECISIONS/0102-the-mesh-writes-into-a-shared-file-never-over-it.md)). It also
declares the runtime's service, reloaded when that file changes. The `dns` key has been written
since the resolver module was converted from its predecessor on 2026-09-23; `live-restore` and the
service were added on 2026-09-30 while fixing
[issue 110](../110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md),
where containers silently resolved through a public resolver.
- **The private network** writes the runtime's `insecure-registries` into the same file, and declares
the same service reloaded on it, as [ADR 0082](../../02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)
and ADR 0102 decided. The controller generates both resources per machine.
On the three machines that run the resolver module, both declare one path and one unit. Nothing refuses
it. The collision check compares the resources of catalogue modules. The private network is computed,
so its resources are produced when a machine's declaration is composed, and the check never sees them.
The machine without the resolver module shows the other half. Its runtime still has the predecessor's
resolver and `live-restore` off, because the only module that sets them is a DNS server. A machine
gets a correct container runtime only as a side effect of being given a resolver.
> **Later the same day, 2026-10-02.** The resolver module and its sibling for the resolver file were
> assigned to the fourth machine ([issue 198](../198-the-lans-dns-server-ran-outside-the-mesh-and-its-filter-closed-it/00-report.md)),
> so all four now have the resolver writing into the runtime's file, and the predecessor's
> `live-restore: false` there is gone. The same work made the runtime's file, as the resolver declares
> it, take no settings: a setting meant for the resolver's own configuration had reached it. The
> collision and the ownership question above are unchanged.
## Why this is here
The operator ruled it a defect, not a design: **a module does not write another software's
configuration.** The need behind each write is real. Containers must resolve the mesh's names
([ADR 0148](../../02-DECISIONS/0148-the-meshs-names-are-resolved-not-copied-into-containers.md) step 2).
A daemon restart must not stop every container. Every machine on the network must trust the mesh's
registry. But each of these is a fact the runtime must be *given*, and the module that gives it is the
runtime's own. With three writers, nobody can say what the file should contain. Two of the facts are
reloaded when one of them needs a restart (issue 110's first fault). And the moment a module for the
runtime exists, it is refused on every machine with the resolver, or, through the private network's
path, accepted without anyone noticing a collision.
## What resolves it
[ADR 0166](../../02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md)
gives the runtime a module that holds its seat and owns its file and service.
[ADR 0164](../../02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md)
gives that module declared settings with defaults. The fix, once both are accepted:
1. The resolver module drops its runtime file and runtime service. It knows nothing of the runtime.
2. The private network stops generating either resource. ADR 0082's decision stands — being on the
network is what grants the trust, and no module author is involved — and only *who writes it*
moves. The mesh gives the registry to the runtime module as a value. ADR 0082 and ADR 0102 each
get a dated note saying where their mechanism now lives.
3. The runtime module writes `dns`, `live-restore` and `insecure-registries`, each a declared
setting with its cost: `dns` costs a restart, which `live-restore` makes harmless.
4. Steps 1–3 land in one push. A runtime module declaring the file beside a resolver module still
declaring it is refused.
5. The collision check sees a computed module's resources as well, so a second writer cannot come
back through generated code.
## Open questions
- **How the resolver's address reaches the runtime.** Either the resolver seat (`node-dns-resolver`)
delivers an address its holder serves, or the runtime module reads a machine fact and the seat
being held is only a precondition. The first tracks a resolver moving off the private address. The
second needs nothing new.
- **What `dns` defaults to on a machine with no resolver seat held.** Nothing, leaving the runtime's
own behaviour, is the honest default. A public resolver hides exactly the failure issue 110 took a
day to find.
- **The adopted machine's predecessor values.** The runtime module adopting a file with a
hand-written `dns` and `live-restore: false` replaces both. That is intended, and is the one
restart the operator must make on that machine.
@@ -0,0 +1,78 @@
---
status: open
opened: 2026-10-02
located-in: [mesh-catalog modules/mesh-console, mesh-controller cmd/mesh-controller/plan.go (port assignment)]
fixed-by:
amended-design:
---
# 192 — The mesh's tools reach a person only by a registration made by hand
## What was observed
A design session on a workstation had none of the mesh's tools. The console was running on that
machine and answering on its loopback port. It was reached over the bus as the console's account, and
listed every running module's tools and every seat's verbs
([ADR 0152](../../02-DECISIONS/0152-the-operators-surface-is-a-module-the-console.md)). What was missing
was the registration that tells the person's coding agent where the console is. That registration had
been made by hand, once, while migrating the machine, and scoped to the one project directory it was
made in. Every session started anywhere else had no mesh tools. Nothing said so: the agent simply
offered no mesh tools, and the session fell back to a pull-request link for a person to open by hand.
The predecessor did this job itself: it wrote its tool server into the agent's user configuration on
every machine. Migrating removed that entry, as it should have, and no module took the job over.
## Why this is here
Three gaps, each of which would have stopped a module from doing it even if one existed.
**1. The console tells nobody where it is.** Its definition listens on a port and provides nothing.
A module that wanted to point an agent at the console has no requirement it could name, so it
would have to write the address into its own definition as a literal. That is exactly what
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) and
[ADR 0155](../../02-DECISIONS/0155-a-definition-names-no-installation-and-how-that-is-checked.md)
remove.
**2. The console's port is one its definition chose.** The definition names a port, and the mesh
never assigned one: no port assignment exists for the console on any machine. The plan assigns a
machine port only to a port a container publishes through a mapping, "without one the software binds
what it binds". The console runs on the host network with no mapping, but it reads its listening
address from `${port:…}`, so the mesh could move it and does not. That is a module choosing a
machine port, which [ADR 0038](../../02-DECISIONS/0038-the-mesh-assigns-the-port.md) exists to
prevent, through a gap in how the rule is applied rather than a decision against it. A module that
reads its port from the mesh should be assigned one like any other.
**3. Nothing in the mesh owns a person's agent configuration.** No catalogue module writes the agent's
settings, its tool-server registrations, or the rules and skills the predecessor delivered. On the
four machines these are hand-kept, or left over from the predecessor, or missing.
## What a fix looks like (not decided)
- **The console provides its endpoint.** A provision, working name `mesh-tools`, served as the URL on
the machine port the mesh gives it. The console listens only on loopback, so the provider must be on
the consumer's own machine. Co-location already chooses it
([ADR 0084](../../02-DECISIONS/0084-which-provider-serves-a-consumer.md)), and a machine with no
console refuses the consumer, naming the provision.
- **A module for the coding agent requires it** and writes the registration into the agent's
system-wide managed settings. The agent reads tool servers from a `managedMcpServers` key there. That
file is the machine's rather than a user's, so the module owns it whole and no home directory is
named. People keep their own registrations beside it. The agent's separate *exclusive* managed
file is the wrong one: it blocks every registration a person makes and hides the hosted connectors.
The agent's per-user file is rewritten by the agent continuously and sits in a home directory,
which would make its path an operator value. These facts come from the agent's documentation
(managed MCP and managed settings pages), not yet verified on a machine.
- **The same module owns the rest of the agent's configuration** the predecessor delivered: managed
settings and the rules, skills and instructions every session reads. Each declared setting carries
a default (ADR 0164,
proposed on its own branch), so one configuration serves every machine and one machine may differ.
## Open questions
- **Is the agent's configuration one module or several?** Tool registration, managed settings, and
the instruction files have different readers and change at different rates.
- **Whose machine port is the console's?** Should a host-network container that reads its port from
`${port:…}` be assigned one, or should a machine-only listener keep its declared number? The second
needs a decision, because ADR 0038 does not allow it today.
- **Credentials.** The console's authority is the machine's login (ADR 0152). A registration that
reaches it carries no secret today. If the console ever listens beyond loopback, the registration
needs one, from the vault.
@@ -0,0 +1,112 @@
---
status: resolved
opened: 2026-10-02
located-in: [mesh-catalog modules/postgres/client.ts (readOnlyQuery)]
fixed-by: [mesh-catalog PR 209 (postgres), mesh-catalog PR 210 (mssql)]
amended-design:
---
# 193 — The store seat's read-only query is read-only by convention, and its answer is unreadable
## What was observed
Asking the store seat's `query` verb for a count through the console returned this. Rows are each
wrapped in an object under a key named `BEGIN`: the column name, then the value, then the word
`ROLLBACK`. A query returning nothing gave the column name and `ROLLBACK` alone. The answer to
`select count(*) as n from <table>` was:
> `rows: [ {BEGIN: "n"}, {BEGIN: "46"}, {BEGIN: "ROLLBACK"} ]`
A reader can work it out. A program cannot, and a query with two columns loses which value belongs to
which.
## Why this is here
**The cause is the same line that makes the query read-only.** The holder's tool sends
`BEGIN TRANSACTION READ ONLY; <the caller's statement>; ROLLBACK;` to the command-line client as one
string. The client prints a command tag for each of the three statements, and the parser takes the
first line, `BEGIN`, as the header.
**And it is not read-only.** [ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md)
decided the store seat's `query` verb is "one read-only statement against one database". The only
thing enforcing that is the wrapping transaction, and the caller's statement is pasted inside it as
text. A statement that begins by ending the transaction (a commit, then anything) runs whatever
follows it outside the read-only transaction, with the holder's own role, the administrative one that creates every
consumer's role and database. A rule stated in a decision and enforced by string concatenation is enforced by nothing.
*This is read from the code, not tried against the live store, and it should not be tried there.*
The lab bed is where it gets proven.
Every caller with `invokes` on the store seat's `query` can do this. The console has `invokes: ["*"]`,
so that includes anyone logged in on a machine running the console.
## What a fix looks like
- **One statement, refused otherwise.** Send the caller's statement alone, through the client's
single-statement path (the extended protocol takes one statement per call and refuses more). The
read-only property then comes from the session, not from text around the statement.
- **Read-only by role, not by transaction.** Run the verb as a role that can only read, granted
`pg_read_all_data`, not as the administrative role. A statement that escapes every wrapper still cannot write.
- **Rows as rows.** Parse the client's output with the column names it returns, or use a driver
instead of the command-line client, so a row is an object keyed by its columns.
- **The check 0159 lacks:** a test that sends a commit followed by a write and asserts the write is
refused and nothing changed. Another asserts a two-column row comes back keyed by both columns.
## Proven, 2026-10-02
On a throwaway server — the same engine image, no network, reached over a socket — the module's code
from the catalogue's main branch ran `COMMIT; COPY (select 1) TO PROGRAM '<a command>'` and **the
command ran on the database host** as the server's own user. `COMMIT; DROP TABLE t` executed the drop
outside the read-only transaction; the wrapper's own trailing rollback happened to undo it, which a
caller ending their statement with a commit of their own would get past (not tried). Nothing was tried
against the live store.
The fix (mesh-catalog PR 209) runs the caller's statement as a login granted `pg_read_all_data` and
nothing else, read-only by its role and its session, with a password the mesh mints as one of the
module's own secrets; without that password the call is refused rather than run as the admin. On the
same throwaway server every escape above, and `SET ROLE`, `RESET SESSION AUTHORIZATION`, turning
read-only off, creating a table, altering the role and reading a server file, is refused; a plain
select comes back keyed by its columns. One attempt — turning the transaction's read-only off, then
deleting — got past the first layer and was stopped by the second, which is why both exist.
**Not answered by the statement-count fix proposed above.** The command-line client sends one string
in one message, so several statements still arrive together. They are harmless as the reader, and
refusing them is left to whoever moves the module to a driver.
## The same hole, elsewhere — and two worse ones
The `mssql` module wrapped a caller's statement the same way (`BEGIN TRANSACTION; … ROLLBACK;` as its
administrator) for its `mssql_query` tool. Its command-line client added two holes of its own. Both
were proven on a throwaway server, running the client the way the module ran it:
- **It substitutes `$(NAME)` from its environment into the caller's text**, and the administrator's
password is in that environment. Selecting it as a string returned the password.
- **It reads a line beginning `:!!` as a command that starts a program**, in the container that holds
the administrator's password and the module's bus credentials. Its switch for refusing such commands
makes the shipped version ignore the statement entirely, so the switch cannot be the guard.
None of it was reachable on the live mesh, for a reason that is a defect of its own: the runtime image
never installed the client, so every mssql tool failed (`spawn sqlcmd ENOENT`). The fix (mesh-catalog
PR 210) installs the client at a pinned digest and runs the caller's statement as a login that can
connect and read and do nothing else. Substitution is off. The statement must be one line, placed after
the module's own text, so no line of it can begin a command; a line break is refused before the client
starts. On the throwaway server, writes, `xp_cmdshell`, impersonating the administrator, and joining
the administrators' role were all refused, and the variable came back as the literal text.
**The general lesson**, worth more than either module: *a command-line client is an interpreter with
its own syntax, and a caller's text handed to it is a program in that syntax as well as in SQL.* A
module that passes a caller's text to a client has two languages to defend, and a transaction drawn
around the text defends neither.
## Resolved, 2026-10-02
Both pull requests merged, built and pushed to the two machines that run each module. Checked live, on
every copy, by asking each one who it is:
- the store seat's `query`, and postgres's own tool on each machine, answer as the reader login —
not a superuser, in a read-only transaction — with rows keyed by their columns;
- mssql's tool, on each machine, answers as its reader login, outside the administrators' role, and
returns `$(SQLCMDPASSWORD)` as the literal text it is. Its tools work for the first time.
The escapes themselves were tried only on the throwaway servers above; on the live mesh the check is
the identity a statement runs as, which is what makes every escape a statement that the login cannot do.
@@ -0,0 +1,88 @@
---
status: open
opened: 2026-10-02
located-in:
- mesh-controller
fixed-by: 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md
amended-design:
---
# 201 — A push recreated the controller at a digest older than the seat row its successor had written
## What was observed
2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the
controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan
built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The
host's plan then pushed the control node with the controller digest it had recorded when it was made —
the previous build — and recreated the controller container on it. The older binary read the row, found
a verb it could not run, and refused to start:
```
mesh-controller: the mesh-controller seat's row declares "command", which this control plane
cannot run: "command" is not a verb the mesh-controller seat serves
```
A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since
the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the
newer image, outside the service, to push the control node again; the push sent the newer digest and
the controller came up.
## Why it matters beyond this instance
The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md));
a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of
2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that
wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md));
for every other module an older digest is a brief regression a later push corrects. For the controller
it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct
it is the thing that is down. Two plans that overlap will happen again whenever two people merge within
a minute.
## What a fix would have to do
Either of two, and the first is the smaller:
- A push never sends a controller digest older than the one the running controller is — the controller
knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words.
- Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves
what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check
exists to refuse.
Until one is built: do not merge a controller change while another plan is rolling, and after merging
one, wait for `node show` on the control node to report the new controller before merging anything else.
## References
- [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
- mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`
## Half of it is closed, 2026-10-02
[ADR 0185](../../02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) takes
the outage out of it: a control plane behind its seat's row now serves every verb it can run, says
which it cannot, and answers the reason when one of those is called. The same race today would cost
the verbs the newer build added, for as long as the older binary is in place, and the ordinary
"this machine is behind" machinery would put the newer one back without a hand.
**The race itself is still open**, and this report stays open for it. What was established while
closing the other half, so the next reader does not redo it:
- Composing and sending are serialised per machine by a session advisory lock in the store, so two
control planes cannot compose one machine's declaration at the same time. The stale content did
not come from two concurrent composes.
- A container's image is resolved into the module's manifest when it is *built*, and a push composes
from the catalogue as it is at that moment, under the hold. So a compose that ran after the build
was taken in could not have named the older image.
- The declaration's sequence orders arrival and nothing else (the numbering of
[issue 107](../107-a-declaration-carries-no-order/00-report.md)); it cannot tell a later send
carrying earlier content from a later send carrying later content. The host refuses a declaration
numbered below the last it applied, and both of these were above it.
- The machine's own journal shows the two applies ten seconds apart and which replaced what; it does
not record which image each declaration named, which is the one fact that would settle it. A host
that recorded the digest it was told, per apply, would have answered this in a minute.
So the trigger is not yet pinned, and guessing at the push path is the most expensive place in the
mesh to guess. The fix the report first suggested — a push never sending a control plane a digest
older than the one that machine reports running — closes the class without needing the trigger, and
is now a correctness nicety rather than the difference between a working mesh and a dead one.