Compare commits

..
Author SHA1 Message Date
jschoubben 0b9fc90885 ADR 0188: a provider declares what it derives for each consumer, and the mesh tells both ends
Issue 124: a value the mesh's own rule produced reached neither end as a
statement. The object store's provisioner derived each consumer's bucket in
its own code; all three consumers transcribed the rule into their own
definitions, one of them wrong, and each of the three also named the machine
it happens to run on.

A served value may now name the consumer the mesh is serving. Design 27
amended; issue 124 resolved.
2026-10-02 21:24:31 +02:00
mesh-admin 983fd412c6 Merge pull request 'ADRs 0180, 0186 and 0187: their live rows, done' (#302) from docs/the-four-fixes-proven-live into main 2026-10-02 18:46:49 +00:00
jschoubben 9ac2493e2c ADRs 0180, 0186 and 0187: their live rows, done — all four machines filtered by the mesh alone, both front ends removed, no machine wrong or behind 2026-10-02 20:46:35 +02:00
mesh-admin 560f25c2c7 Merge pull request 'ADR 0187: a dead tracker is not the machine's failure' (#301) from fix/a-dead-tracker-is-not-the-meshs-failure into main 2026-10-02 18:42:01 +00:00
jschoubben 9e0288128b ADR 0187: a dead tracker is not the machine's failure; design 32 2026-10-02 20:41:30 +02:00
mesh-admin d57289e049 Merge pull request 'ADR 0186: a ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang' (#299) from fix/a-ban-list-never-holds-a-neighbour into main 2026-10-02 16:43:32 +00:00
jschoubben d4a2f99ab5 ADR 0186: a ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang; design 31 2026-10-02 18:42:16 +02:00
mesh-admin a9f91fdd0c Merge pull request 'ADRs 0184 and 0185: a service is still running a moment later; a control plane behind its row serves what it can' (#298) from fix/a-service-asked-to-run-is-still-running into main 2026-10-02 16:25:44 +00:00
jschoubben 131a5e4714 Issue 201: what was established about the race while closing the outage half 2026-10-02 18:19:12 +02:00
jschoubben 329a24fdae ADRs 0184 and 0185: a service is still running a moment later; a control plane behind its row serves what it can; issue 201 half closed 2026-10-02 18:16:24 +02:00
mesh-admin 7f72f3b79a Merge pull request 'ADR 0179: the intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail' (#295) from feat/the-intrusion-seat-serves-its-verbs into main 2026-10-02 15:28:48 +00:00
jschoubben 114a71f36f ADR 0179: built and proven live; the one fault the machine found, and the check that refuses it 2026-10-02 17:28:38 +02:00
jschoubben 0f407417f3 Merge main: the ufw record renumbered to 0180, and ADR 0175 retires the per-module tool runtime this one ships 2026-10-02 17:28:23 +02:00
jschoubben d0d5799884 Issue 201: a push recreated the controller at a digest older than the seat row its successor wrote 2026-10-02 17:15:22 +02:00
jschoubben 9ba4de5557 ADR 0179: the intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail; designs 31 and 33 2026-10-02 17:02:49 +02:00
16 changed files with 796 additions and 237 deletions
@@ -0,0 +1,132 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
---
# 179. The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail
## Context
Read on the control node on 2026-10-02, the day the machines were confirmed filtered by the mesh
alone ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)): the intrusion
prevention watched one door. Its two jails read the ssh daemon's journal and its own log, banned
five failures in ten minutes for ten minutes, and in a day had seen twelve thousand failed logins
from three hundred addresses and banned none of the busiest, which paced themselves at one try every
ten minutes. The mail submission port took a hundred and sixty password guesses in the same day from
thirty-eight addresses with no jail reading it at all; the forge and the public proxy had no jail
either, and the proxy logged nothing a jail could read. Nobody could see the jails without a shell:
the module's three tools existed in code and were served by nothing, and the seat it holds declared
no verbs.
Three things were missing and they are three shapes the mesh already has. The packet filter's seat
serves verbs every holder owes ([ADR 0170](0170-the-firewall-seat-serves-its-verbs.md)); the
intrusion seat serves none. A module's `listens` compose into the machine's filter, and [to-be 31](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
says a module's `jails` compose into the machine's intrusion prevention the same way — the controller
composes them, and no module declares one. And a jail reads a log; a container's output goes to a
file of the runtime's own under a path that changes when the container is recreated, which is why
no jail could read the mail front end, the forge or the proxy, however they logged.
## Decision
**1. The `node-intrusion-prevention` seat serves four verbs**, and a module that claims it serves
all four or is refused the claim, as with every seat:
- `status` — every jail with what it watches, how many addresses it is counting failures against and
holding now, and the totals since it started; one jail's detail when named. Read-only.
- `banned` — every address banned now, with the jail holding it, when it was banned and when the ban
ends. Read-only.
- `ban` — ban one address in one jail now, for that jail's ban time. An operator's act on the live
ban list, which the mesh composes the rules for and never writes itself.
- `unban` — let one address go, from one jail or from every jail.
A holder may serve its own tools beside these; the fail2ban module reads one jail's effective
settings as its own.
**2. A container may log to the journal.** `logging: journald` on a container has the host run it
with the journal as its log driver; the journal keeps the container's name on every line, and
`docker logs` keeps working. Where a container logs is part of its spec, so moving it recreates the
container, and the only place besides the runtime's own file is the journal: a machine's intrusion
prevention reads the journal already, for the ssh daemon, and a container that logs there is read
the same way, by the container's name, whatever the container is called by the runtime this time.
**3. A module with a door declares its jail, and the holder composes them.** What to-be 31 designed
is now the rule: a module whose service authenticates from outside — the mail front end, the forge,
the public proxy — declares in its manifest what a failed attempt looks like in its log and how to
ban on it, naming no node and no path; the module that holds the intrusion seat declares where the
composed jails and filters land, and the mesh writes them on every machine that runs both. A machine
not running the module has no such jail. The holder restarts its daemon on the composed file.
**4. The base is strict, and the mesh's own range is never banned.** Three failures in a day ban for
a day, on every jail unless the jail says otherwise; banned twice in two weeks, by any jail, is
banned for four. The attackers this mesh sees pace themselves under any ten-minute window; a day's
window counts them. A person who mistypes three times from one address is out for a day from that
address, and never from a machine of the mesh, whose range stays in the never-banned list the module
has carried since [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md). The operator
chose this knowing it.
**5. The proxy says a refused name in its log.** A request for a name this mesh does not serve, from
outside, is what a scanner does; the proxy already logged a certificate refused for such a name, and
now logs the plain request too, with the asking address last, as its own jail's filter expects it.
## Consequences
- The seat's row gains four verbs; a mesh that already runs widens its row at the next controller
start. The fail2ban module claims them and gains a runtime — a tool server whose image carries the
fail2ban client, with the daemon's socket shared in from the machine, and nothing else of the
machine. The daemon stays the machine's; what runs in the container is only the client.
- **That runtime is the shape the catalogue has today, and it is on its way out.**
[ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md),
accepted the same day as this record, replaces a tool container per module with one tool runtime
per node on the host side, taking each module's tools as a bundle. Nothing here depends on the
container: the verbs, the client that speaks to the daemon over its socket, and the jails are the
same code under either. This module converts with the packet filter's, whose runtime that record
names, and the socket it needs becomes the node runtime's to reach rather than a mount of its own.
- The host's container vocabulary grows by `logging`; an older host refuses a declaration that carries
it, so the host rolls before the modules. Three containers are recreated once, when their modules
are pushed with the field: the mail front end, the forge and the proxy — each a moment's outage.
- The fail2ban module declares where jails compose (`jailing`) and the directory the filters go in;
the mail, forge and proxy modules each declare one jail reading the journal by their container's
name. The composed jail file is the one resource the daemon restarts on when a module arrives or
leaves a machine.
- The two base jails and the composed ones take the day's window; the ssh jail's ten minutes are
gone. An address banned on the first day of this record stays banned for the day.
- The module's three old tools, served by nothing, are replaced by the seat's four verbs and one
own tool; `fail2ban_status` as a name is gone.
## How this is checked
| Rule | Checked by |
|---|---|
| The seat declares the four verbs; a claim that serves fewer is refused by name | the catalogue's seat tests |
| `status`, `banned`, `ban` and `unban` read and steer the daemon through its client, with the shapes fail2ban 1.1.0 printed live; a non-address and a non-name are refused before anything runs | the module's tests over a fake command runner |
| A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests |
| A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) |
| The proxy logs a refused name with the address last | the proxy's tests |
| A jail's pattern names `<HOST>` once per shape, since two is a duplicate capture group and costs the machine every ban | the catalogue's manifest tests |
| Live | done 2026-10-02: `status` and `banned` answered on both servers through the console; the proxy's jail counted seven refusals on the home server; a documentation address banned in the ssh jail came back with its end time and was released |
## Built and proven live, 2026-10-02
All five rules are in the mesh. The host carries `logging`; the controller's seat row carries the four
verbs and the proxy says a refused name in its log; the fail2ban module holds the seat from a runtime
with the daemon's socket shared in, composes the jails, and the mail front end, the forge and the
proxy each declare one. Through the console on the control node: `status` listed five jails with what
each watches, `banned` listed the nine the long jail holds, and a documentation address banned in the
ssh jail came back with its ban's end time and was released again. On the home server the proxy's jail
had counted seven refusals within minutes of starting.
**One fault, found by the machine and not by a test.** The proxy's pattern matched two shapes of
refusal in one expression and so named `<HOST>` twice. fail2ban expands that placeholder into a named
capture group; two of them is a duplicate group name, and the daemon refuses *its whole configuration*
and exits — both servers kept no bans at all for about ten minutes, every jail and not the one at
fault. The pattern is now one per shape. A manifest check refuses the mistake at merge time, naming
what it would cost, which is the only reason this record can claim the rule rather than the instance.
## References
- [ADR 0170](0170-the-firewall-seat-serves-its-verbs.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
- [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md), [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md), [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md)
@@ -60,7 +60,7 @@ configuration file; the rollback path it described is given up on purpose.
|---|---|
| An absent package is removed when present, left when not, and read back | host tests over a fake package manager |
| An uninstalled front end is recorded as removed and nothing is asked of it | a host test with ufw missing on a converged apply |
| Live | the two machines report ufw gone: `pacman -Q ufw` has no answer, `node show` says removed, `status` is well |
| Live | done 2026-10-02: both machines report ufw gone — `pacman -Q ufw` has no answer, `node show` says *removed*, `status` is well. The home server said *retired* for six hours after the package went, because this record's step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)) and one dead tracker was failing its applies ([ADR 0187](0187-a-dead-tracker-is-not-the-machines-failure.md)) |
## References
@@ -0,0 +1,75 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0005-the-node-host.md
---
# 184. A service the mesh asked to run is still running a moment later
## Context
The host already refuses to take a service manager's word for it. Three places in one function read
a unit back after acting on it, each with a comment saying why: *a service manager accepting a
command says the transaction was accepted, not that the unit is running — one that starts and
immediately dies satisfies it.* The intent was right and the implementation did not reach it.
On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the
files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at
that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public
machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said
so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing.
The read-back races the failure. A service manager returns when it has started the process; a daemon
that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One
look sees `activating` or `active` whatever the process is about to do, and *the host reports success
for a machine that is already wrong* — the one shape of failure this host exists to refuse
([ADR 0005](0005-the-node-host.md)).
A command the module declares — *test the configuration before restarting* — was considered and
rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification
command is a command: a declaration that carried one would be remote execution over the bus,
arriving as root on every machine, which is a far larger door than the fault it closes. The host
does not need one. It already knows what it asked for.
## Decision
**1. A unit the host has just asked to run is read twice**, with a pause between the reads long
enough for a daemon that refuses its configuration to have exited. Not running at the second look is
a failure of that resource, named with the unit and the state it is in — the same failure the single
read was always meant to catch.
**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both
looks and is accepted, exactly as before. What the second look catches is a unit that *was* running
and is not any more. A service asked to be stopped is not waited on at all.
**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking
the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and
no command reaches a machine that did not already come from a built artifact.
## Consequences
- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is
bounded by the number of services that changed in that apply, which is usually none.
- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the
resolver — now fails its apply when the composition is bad, instead of reporting success onto a
dead daemon. `status` names the machine, which is how the operator finds out.
- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s
manifest check is what refuses the one that caused this, at merge time; this record is what makes
the *next* one visible within a minute rather than invisible until something asks the daemon a
question.
## How this is checked
| Rule | Checked by |
|---|---|
| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does |
| A unit still starting is accepted at both looks | a host test |
| A service asked to be stopped is not waited on | a host test |
## References
- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md)
@@ -0,0 +1,73 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
---
# 185. A control plane behind its seat's row serves what it can
## Context
The mesh's own verbs are the controller seat's tools, and the seat's row is the store's
([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). A control plane reads the
row at start and installs a handler per verb; a verb the row carries that the binary cannot run was
refused at start rather than at the first call, so that a disagreement between the row and the
binary was said early. The refusal aborted the start.
On 2026-10-02 a merge added one verb. The new control plane started, widened the row, and ran. A
push a few seconds later recreated its container at the previous image — a stale declaration from
an overlapping wave, [issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) —
and the older binary read a row naming a word it had never heard. It refused to start, and kept
refusing. The mesh had no voice for ten minutes: no verb answered, no node could be pushed, no build
was dispatched, and `status` said nothing because `status` is one of the verbs that had stopped
being served. The way back was a person running the binary by hand outside its service, because the
push that would have replaced it is itself a verb of the control plane that was down.
The check was right about the fact and wrong about the cost. A row ahead of a binary is the ordinary
state of a roll-out: the row is widened by whichever control plane starts first, and a mesh with one
control plane sees that gap on every merge that adds a verb. Making it fatal turned a transient into
an outage with no path out that did not need a human.
## Decision
**1. A control plane serves the verbs it can run and does not refuse to start for the ones it
cannot.** The row remains the authority on what the seat serves; this is only about what this binary
does when it is behind the row.
**2. A verb it cannot run answers the reason.** Not silence and not a missing subject: a caller gets
a sentence naming the verb, saying this control plane cannot run it and that it is a verb of a newer
build. A verb that is simply absent from the row is still not served at all — that is the row
deciding, which is unchanged.
**3. It says so once at start**, naming every verb of the row it cannot run, so the gap is visible
in the log of the thing that has it rather than only at the moment somebody calls one.
**4. A mesh with no controller seat at all is still a refusal.** That is not a version gap, it is a
mesh that has not been seeded, and nothing this control plane does would be meaningful.
## Consequences
- An overlapping roll-out costs the verbs the newer build added, for as long as the older binary is
in place. Everything else — every push, every build, every read — keeps working, and the ordinary
machinery that notices a machine is behind is what puts the newer binary back.
- The log gains one line on a control plane that is behind, and nothing on one that is not.
- Issue 201's other half remains: the push that sent a stale declaration is a race worth closing on
its own terms. This record makes that race survivable rather than fatal, which is the difference
between a transient and an outage, and is deliberately the cheaper half.
## How this is checked
| Rule | Checked by |
|---|---|
| A row carrying a verb this build cannot run still serves every verb it can, and names the one it cannot | a controller test over a widened row |
| The unknown verb answers a sentence naming itself and saying this build is behind | the same test |
| A mesh with no controller seat is refused | the existing start-up path |
## References
- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
- [Issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)
- [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)
@@ -0,0 +1,81 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
---
# 186. A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang
## Context
[ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) gave the
public proxy a jail. Within the hour the home server's ban list held `192.168.1.1` — the house's own
router. The router reflects local traffic, so every client in the building reaches that machine as
the gateway's address; one local request for a name the mesh does not serve, three times in a day,
and the whole house is refused by the machine it was asking. The jails inherited an `ignoreip` of
the loopback and the mesh's own range, which was right when the only jail read the ssh daemon and
the only clients were the mesh's; a jail on a public front door sees the neighbours too.
The same jail broke the other half of [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md).
The home server began reading *NOT the mesh alone: 1 rule set the mesh did not write refuses traffic
here*, and the rule set named was the mesh's own ban chain, written by the mesh's own intrusion
prevention minutes earlier. The host's reader of the legacy filter required every path into a chain
of refusals to come from a built-in chain whose policy accepts, before it would call that chain a
ban. On that machine the chain hangs off the container runtime's user chain as well as the input
chain, and the runtime had set the forward policy to DROP — so the mesh reported its own work as a
foreigner's, on the one machine where the group's exit condition was supposed to hold.
Both faults are one mistake in two places: a rule written about the public internet, applied to
everything that arrives.
## Decision
**1. A ban list never holds a neighbour.** The jails the mesh composes never ban a source on a
private range — the mesh's own range, which was already named rather than written
([ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md)), and every address space
reserved for private use beside it, in both families. A machine behind a router that reflects local
traffic sees its whole building as one address; a ban there is a self-inflicted outage, and the
sources worth banning are not on those ranges in the first place.
**2. The mesh's own bans are its own wherever they hang.** A chain of refusals is a ban list when
every refusal names the sources it refuses and the chain accepts nothing — the rule the host already
applied to the packet filter's own tables, now applied to the legacy filter too, and nothing more.
The policy of the chains that jump into it says nothing about what it is: that policy is already
classified where it belongs, as the container runtime's, and requiring it here counted it twice.
**3. A chain that accepts anything is still not a ban.** That is what keeps a predecessor's
allow-these-and-drop-the-rest chain classified as something an operator must look at, which is the
distinction [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md) exists to draw.
## Consequences
- The composed jails gain the private ranges in their never-ban list. An address already banned
stays banned until it is released; the house's router was released by hand the moment it was found.
- The home server reads *the mesh alone* again, which is group 7's exit condition and was false for
about an hour.
- A machine whose apply fails for an unrelated reason does not revisit its found firewall's record
at all — the step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)).
The home server's record therefore still reads *retired by the mesh* although the front end is
uninstalled, and will correct itself once that machine's own stuck module is fixed. It is a stale
record, not a wrong machine.
- The record number the front end's removal was given moved under it: another session took 0175
while that record was in review, and it is now
[ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md). The citations
the host and the control plane print were pointing at an unrelated record and are corrected here.
## How this is checked
| Rule | Checked by |
|---|---|
| A private source is never banned | the module's jail configuration, read back by `fail2ban.fail2ban_settings` on a machine |
| The mesh's own ban chain reads as a ban behind a dropping forward policy | a host test over the home server's own captured rule set |
| A chain that accepts anything is not a ban | a host test |
| Live | done 2026-10-02: all four machines read *the mesh alone*, the home server counting its own ban chain as a ban; no ban held anywhere is a private address |
## References
- [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
- [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md), [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
@@ -0,0 +1,73 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
---
# 187. A dead tracker is not the machine's failure
## Context
The home server had not applied a declaration cleanly since midday. One run-once step — the one
that writes a media app's download clients and indexers through the app's own API — exited
non-zero, forty-nine times over six hours, for one public tracker that had stopped answering. The
step's own words: the entry was *written*, and the app's test of it then failed with a 400 from the
indexer proxy. The machine reported *not doing what it was told* for the rest of the day.
What that gated matters more than the step. A converged machine retires the firewall it was found
with only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)),
so that machine went on recording its found front end as merely *retired* long after the package
had been uninstalled ([ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)).
A dead public tracker was holding a firewall record hostage, which is not a connection anybody
would design.
The step already knew this was not its business. It had a rule for exactly this: an entry the mesh
only *found and re-pointed*, rather than one it was told to make, whose feed is gone, is said and
left as found — *failing the node's apply on every heartbeat for it reports the mesh as wrong about
a tracker*. The rule was there and matched one shape of the fault. An app can refuse to save such an
entry, and it can save it and then fail its own test; saving validates settings, and the test runs a
live search. The rule caught the first and let the second through.
## Decision
**1. An entry the mesh only found is never the machine's failure.** Whatever shape the app's
refusal takes — it would not save it, or it saved it and its own test fails — an indexer the mesh
found and re-pointed is reported as a notice and left as found. What decides is whose entry it is,
not which sentence the app returned.
**2. What the mesh is answerable for is the plumbing.** That the entry exists, points at this
mesh's indexer proxy, and carries the credential the mesh delivered — which was checked against the
proxy before anything was written. Whether a public tracker answers today is not the mesh's to
promise, and a machine that reports itself broken because one did is lying about itself.
**3. An entry the operator listed is theirs to insist on.** An indexer named in the step's settings
is one the mesh was told to make, and it still fails the step when it cannot be made to work. The
notice says so, and says that listing the indexer is how to turn it back into a failure.
## Consequences
- The home server applies cleanly again, and everything a clean apply gates — its found firewall's
record among it — follows.
- A tracker that dies is a line in a report rather than a machine that reads as broken. An operator
who wants it gone removes the entry or repairs the feed; the mesh says which, every time it runs.
- The four Servarr modules carry one byte-identical copy of this step each
([ADR 0069](0069-a-module-is-a-repository-and-a-path.md)), so the change lands in four places and
a test refuses any drift between them.
- It does not widen to a download client: one the mesh was told to write and cannot is still a
failure, because the mesh chose it and nothing else will fix it.
## How this is checked
| Rule | Checked by |
|---|---|
| A found feed whose tracker answers an error after the entry was written is a notice | the step's tests, with the home server's own message and the app's two validations modelled apart |
| An indexer the settings list is still a failure | the same test |
| The four copies of the step do not drift | the step's own sameness test |
| Live | done 2026-10-02: the home server applies cleanly after six hours of failing, `status` holds no machine wrong or behind, and its found firewall reads *removed* |
## References
- [ADR 0136](0136-a-step-gates-its-module-not-the-machine.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md), [ADR 0069](0069-a-module-is-a-repository-and-a-path.md)
@@ -0,0 +1,130 @@
---
topic: the mesh
status: accepted
date: 2026-10-02
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
---
# 188. A provider declares what it derives for each consumer, and the mesh tells both ends
## Context
An arrangement between a consumer and a provider is delivered entirely by the mesh. Where the
provider is, which port it answers on, what name the consumer must present, where its password
is — each arrives as a fact the consumer reads from its binding, or as `${bound:…}` filled into a
file before the declaration leaves the control plane. The provider invents none of it and hands
none of it back ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)).
One kind of value escapes that. Where the **provider names the resource** — a bucket, a database,
a vhost — the name is derived from the consumer, per consumer, and the mesh has no way to carry
it. `serves` is a literal block in the provider's definition: the same values for every consumer.
A provisioner's contract takes a provision and returns nothing. So a value the mesh's own rule
produced reaches neither end as a statement; it is recomputed at one end and transcribed at the
other.
The object store is the instance ([issue 124](../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)).
Its provisioner normalises the login the mesh minted into a bucket name and creates, checks and
removes exactly that; the rule lives in twenty lines of the module's own TypeScript. Its three
consumers each write the answer into their own definition by hand. Two transcribed it correctly;
one named a predecessor's bucket, and would have authenticated successfully and been refused on
every object, which reads like a credential fault and is not one.
Even corrected, the transcriptions are wrong in a second way. Each is `mesh-<node>-<slug>`, so
each **names the machine the module happens to run on today** — a definition stating a fact about
one installation, which [ADR 0155](0155-a-definition-names-no-installation-and-how-that-is-checked.md)
forbids and whose check does not catch because the name is not a domain. Move any of the three to
another machine and its configuration points at a bucket its key cannot open.
The shape is not the object store's. A database provisioner that prefixed names, a queue provider
that scoped vhosts, any provider that derives a resource from who is asking: each forces the
consumer to reproduce somebody else's rule and keep it in agreement by hand.
## Decision
**1. A served value may name the consumer the mesh is serving.** A `serves` block, which is
literal today, may interpolate the mesh's own statement of who the consumer is:
- `${consumer:as}` — the identity the mesh minted for this consumer, exactly as the login it is
told to present ([ADR 0049](0049-a-consumers-identity-fits-the-tightest-backend.md));
- `${consumer:as:dns}` — the same identity written as a DNS label.
Nothing else. **The mesh learns no protocol here; it spells its own name in an alphabet it already
knows.** The identity is the mesh's, minted by the mesh, already capped at twenty characters
because of what an S3 access key accepts; `dns` is that same name with its separator written `-`
instead of `_`, which is the whole of the difference between the mesh's identifier alphabet and
the one buckets, vhosts and hostnames use. A provider that needs a prefix or a suffix writes it
around the placeholder, because a served value is a string.
The rejected alternative is **the provider returning values from provisioning** — the natural
channel, since the provider is what derived them. It is rejected for three reasons, in order of
weight. It inverts the delivery the mesh is built on: a grant would carry data the provider wrote
rather than only data the mesh minted, and [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)
removed exactly that second path once already. It makes a consumer's declaration incomplete until
its provider's reconcile loop has run, so a consumer could not be composed before a provider
answered — a bootstrap order the mesh does not have and does not want. And it puts the rule where
nothing can check it: a value that arrives from a running process cannot be refused at resolution,
only discovered wrong later, which is the failure this record exists to end.
**2. The mesh resolves it once, per consumer, and tells both ends from the one resolution.** At the
moment a consumer's declaration is composed, the mesh knows exactly who the consumer is. There, and
only there, the placeholders are filled. The result reaches:
- the **consumer**, as the served facts in its binding file and as `${bound:<provision>:<key>}` in
any file it writes — unchanged mechanisms, carrying one more key;
- the **provider**, as `serves` on that consumer's entry in its contributions file, so the
provisioner is *told* the name rather than recomputing it.
**The provider stops deriving in code and starts declaring.** One statement, filled once, delivered
to both ends: the two cannot disagree, because there is no second computation to disagree with.
**3. A served value stays settled before it is per-consumer.** Settings still compose into `serves`
([ADR 0174](0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)), and the consumer
placeholders are filled after that, so an operator may set a prefix and the mesh still derives the
rest. A `${consumer:…}` naming a fact or an alphabet the mesh does not have is refused when the
definition is parsed, with what it may say.
**4. A consumer may no longer name the resource its provider derives.** With the value delivered,
a literal in a consumer's definition is not merely redundant — it is the one thing that can
disagree with what the provider will actually create. The three object-store consumers lose their
hand-written bucket names in this change.
## Consequences
- One more thing a definition may say, and one less thing a module may be wrong about. The
vocabulary grows by a placeholder; the catalogue loses three literals that named this
installation's control node.
- A provider's naming rule becomes readable in its definition instead of in its source. `minio`'s
`bucketFor` goes; the manifest says `"bucket": "${consumer:as:dns}"` and the provisioner uses
what it is given.
- A provider that already serves consumers keeps serving them: the derived value equals what the
code derived, so no bucket, database or login changes name. This is a change of **who says it**,
not of **what is said**.
- The mesh now holds a rule in another system's alphabet — one rule, `dns`, stated once. A second
alphabet is a decision, not an addition: the cost of each is that the mesh must be right about
somebody else's naming, and that cost is only worth paying where the mesh already mints the name.
## How this is checked
- A served value naming an unknown fact or alphabet is refused at parse, with the list of what it
may say — tested on both halves of the message.
- Resolving a consumer whose provider derives a value puts that value in the consumer's binding
file, in its `${bound:…}` substitutions, and in the provider's contributions entry for that
consumer — one test asserting the three agree, because agreeing is the whole point.
- Two consumers of one provider on one machine get two different derived values, and neither gets
the other's.
- A catalogue-wide test refuses a consumer definition that writes a literal where its provider
derives: the provider's `serves` names the key, so the catalogue can say which definitions
transcribe one.
- `dns` is checked against the identity the mesh actually mints, not against an invented string:
the test derives an identity with `ConsumerIdentity` and asserts the label it becomes.
## References
- [issue 124 — a consumer cannot be told a value its provider derived for it](../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)
- [ADR 0048 — a provider creates the credential the mesh minted, and seals nothing](0048-a-provider-creates-the-credential-the-mesh-minted.md)
- [ADR 0049 — a consumer's identity fits the tightest backend](0049-a-consumers-identity-fits-the-tightest-backend.md)
- [ADR 0174 — a node varies a module through settings and kept regions, never through an edit](0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)
- [ADR 0155 — a definition names no installation, and how that is checked](0155-a-definition-names-no-installation-and-how-that-is-checked.md)
- [design 27 — a module requires, the mesh resolves](../03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md)
+6
View File
@@ -182,7 +182,13 @@ python3 00-META/checks/index.py fail if stale
- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)
- **0170** — [The firewall seat serves its verbs, and a foreign rule set is removed through one of them](0170-the-firewall-seat-serves-its-verbs.md)
- **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md)
- **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
- **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
- **0184** — [A service the mesh asked to run is still running a moment later](0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md)
- **0185** — [A control plane behind its seat's row serves what it can](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)
- **0186** — [A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang](0186-a-ban-list-never-holds-a-neighbour.md)
- **0187** — [A dead tracker is not the machine's failure](0187-a-dead-tracker-is-not-the-machines-failure.md)
- **0188** — [A provider declares what it derives for each consumer, and the mesh tells both ends](0188-a-provider-declares-what-it-derives-for-each-consumer.md)
### Its tiers, from the bottom up
+15
View File
@@ -4,6 +4,7 @@ status: in-progress
code: [mesh-host]
updated: 2026-10-02
decisions:
- 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md
- 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md
- 02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md
- 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
@@ -498,3 +499,17 @@ run and reported; the exit follows an in-flight apply rather than interrupting i
not start is rolled back once and the second failure halts; a completed reconcile retires what is older
than the predecessor and never the predecessor; and the newest of two delivered versions is the one
that runs.
## A service is still running a moment later, 2026-10-02
[ADR 0184](../../02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md).
The host has always read a unit back after acting on it, because a service manager accepting a
command says the transaction was accepted and nothing about the process. The read raced the failure:
a daemon that refuses the configuration the mesh just wrote exits a fraction of a second after the
manager returns, and one look sees it alive. So the host looks twice, with a pause between, and a
unit that was running and is not any more fails its resource by name. A unit still coming up reads
as running at both looks and is accepted; a service asked to stop is not waited on.
No command for this reaches a machine. A module declaring *how to test my configuration* was weighed
and refused: the link carries no actions, and a verification command is one. The host is checking
the state it was told to establish, which is what it is for. *How it is checked:* ADR 0184's table.
@@ -2,7 +2,7 @@
layer: to-be
status: in-progress
code: [mesh-controller internal/catalogue]
updated: 2026-09-30
updated: 2026-10-02
decisions:
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
- 02-DECISIONS/0155-a-definition-names-no-installation-and-how-that-is-checked.md
@@ -14,6 +14,7 @@ decisions:
- 02-DECISIONS/0084-which-provider-serves-a-consumer.md
- 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md
- 02-DECISIONS/0038-the-mesh-assigns-the-port.md
- 02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md
---
# 27 — A module requires, the mesh resolves
@@ -206,6 +207,22 @@ name when nothing sets it. That is the contract half of this design's operator p
the placeholder allows: the definition says which values reach which requirement, and nothing else
does. *How it is checked:* the unit tests named in issue 173, and the plan comparison that closed it.
*A provider says once what it derives for each consumer (2026-10-02,
[ADR 0188](../../02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md),
[issue 124](../../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)):*
where a provider **names the resource** it gives each consumer — a bucket, a database, a vhost — the
name is derived per consumer, and a literal `serves` block could not carry it. A served value may
now name the consumer the mesh is serving: `${consumer:as}`, the identity the mesh minted, and
`${consumer:as:dns}`, that same identity written as a DNS label. Nothing else — **the mesh learns no
protocol here; it spells its own name in an alphabet it already knows.** Settings are laid on first,
so an operator may still set a prefix and the mesh derives the rest. The mesh fills it at the one
moment it knows who the consumer is, and the one filled value reaches both ends: the consumer, as
its binding's served facts and as `${bound:<provision>:<key>}` in any file it writes; the provider,
as `derived` on that consumer's entry in its contributions file, so its provisioner is told the name
rather than recomputing it. A consumer that writes the derived value into its own definition instead
of asking for it is refused, naming the placeholder to use. *How it is checked:* the unit tests in
ADR 0188's "how this is checked", each run against the unchanged controller first.
## How a definition reads what was resolved
**One form, naming a requirement and a field of its contract.** A definition that needs the database's
@@ -214,7 +231,9 @@ name in a configuration file writes the same thing: the requirement's name and t
controller fills it at resolution.
This one form replaces the placeholders that exist today, one per mechanism: bound values, secrets,
ports and machine facts.
ports and machine facts. It subsumes the consumer placeholder too — a value a provider derives is
read by the consumer exactly as any other field of the contract is, and `${consumer:…}` is only
how the *provider* states the rule.
**The seat placeholder stays, for the controller alone.** The controller composes its own
declaration and reaches the store and broker it made before any module existed, so it cannot be
@@ -1,10 +1,15 @@
---
layer: to-be
status: proposed
code: []
updated: 2026-09-27
status: in-progress
code:
- mesh-controller: internal/catalogue/jails_into.go, internal/catalogue/manifest.go (Jail, Jailing)
- mesh-catalog: modules/fail2ban (jailing, the base and the seat's verbs), modules/mailu, modules/route-proxy, modules/gitea (jails)
- mesh-host: internal/declaration/declaration.go (a container's logging)
updated: 2026-10-02
decisions:
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
- 02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md
---
# 31 — A module declares its fail2ban jail, and the mesh composes them per node
@@ -63,3 +68,38 @@ jail, composed from the postgres module's manifest, without anyone editing a nod
beside)
- mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules
(`postgres`, `mssql`, `mailu`) that will declare jails
## Decided and built, 2026-10-02
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
made this the rule and built it. A module declares `jails` — each a name, the `failregex` of a
failed attempt in its log, and the stanza's own keys — and the fail2ban module declares `jailing`:
the one file the stanzas compose into and the directory each filter lands in. The controller gathers
every assigned module's jails per node into those; the holder's daemon restarts on the composed file.
What made it workable was the log. A container's output went to a file of the runtime's own, under
a path that changes when the container is recreated, so no jail could read a container's service
however it logged. A container now declares `logging: journald`, the host runs it with the journal as
its driver, and a jail reads it with `backend = systemd` and a `journalmatch` on the container's
name — the same way the base's ssh jail has always read the ssh daemon. The first three doors: the
mail front end (every login failure on its proxying ports), the forge (a failed authentication
attempt) and the public proxy (a certificate or request for a name the mesh does not serve, which
the proxy now says in its log). The base is strict — three in a day for a day; twice banned in two
weeks for four — and the mesh's own range stays never banned.
The seat the module holds serves `status`, `banned`, `ban` and `unban`, from a runtime that carries
only the fail2ban client with the daemon's socket shared in; the jails are composed, the ban list is
the daemon's, and both are read through the console.
*How it is checked:* ADR 0179's table.
## What the first jails taught, 2026-10-02
[ADR 0186](../../02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md). Within an hour of the
first public jail the home server had banned the house's own router: the router reflects local
traffic, so every client in the building arrives as the gateway's address. The never-ban list now
holds every private range as well as the mesh's own. And the mesh read its own ban chain as a
foreign rule set on that machine, because the chain hangs off the container runtime's user chain and
that machine's forward policy is the runtime's DROP — the reader now calls a chain of source-named
refusals a ban wherever it hangs, as it already did for the packet filter's own tables.
@@ -11,7 +11,7 @@ code:
- mesh-host internal/apply/apply.go
- mesh-tools src/main.ts
- mesh-catalog modules/mesh-catalog
updated: 2026-09-28
updated: 2026-10-02
decisions:
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
@@ -24,6 +24,7 @@ decisions:
- 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
- 02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md
- 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
- 02-DECISIONS/0187-a-dead-tracker-is-not-the-machines-failure.md
- 02-DECISIONS/0134-the-mesh-says-what-it-applied.md
---
@@ -470,6 +471,21 @@ moment the mesh can mint for itself: **the bus's own accounts** (§the bootstrap
needs an account before it can run) and **the vault's own credential**. Any third exception is a
design failure, and naming these two is what makes a third one visible.
## What a step is answerable for, 2026-10-02
[ADR 0187](../../02-DECISIONS/0187-a-dead-tracker-is-not-the-machines-failure.md). A step gates its
module and not the machine ([ADR 0136](../../02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md)),
but a step that exits non-zero still leaves the machine reporting that it is not doing what it was
told — and a clean apply gates other things entirely, the found firewall's retirement among them. So
what a step calls a failure matters beyond the step.
The rule the media step now follows, and the one to copy: a step fails for what the mesh chose and
can fix, and reports what it merely found and cannot. An indexer entry the mesh re-pointed at this
mesh's proxy is plumbing the mesh is answerable for; whether the public tracker behind it answers
today is not. An entry the operator listed is the operator's to insist on, and still fails. Six
hours of a machine reading as broken, for one tracker that had died, is what the distinction costs
when it is missing.
## 11. Open
**Semantic change has no mechanical defence** (§8). Recorded as open rather than solved, because
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-tools]
updated: 2026-10-02
decisions:
- 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md
- 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
@@ -184,6 +185,18 @@ container to declare a capability. Removing a predecessor's rule set is an opera
through the seat, recorded on the bus, instead of a shell on the machine. *How it is checked:* ADR
0169's table.
## The intrusion seat's verbs, 2026-10-02
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md).
The second node-scoped seat to carry verbs: `node-intrusion-prevention` serves `status` (every jail
with what it watches and holds), `banned` (every address held now, with its jail and when the ban
ends), `ban` and `unban` (an operator's act on the live ban list). The fail2ban module serves them
from a runtime that carries only the daemon's client, the socket shared in from the machine — no
capability, no machine network, since the daemon on the machine does the banning. That runtime is the
per-module container [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
retires; the verbs and the client are the same code once the node's own runtime loads them as a bundle.
The module's own tool beside them reads one jail's effective settings. *How it is checked:* ADR 0179's table.
## What this does not settle
- Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a
@@ -1,226 +0,0 @@
---
layer: to-be
status: designed
code: []
updated: 2026-10-02
decisions:
- 02-DECISIONS/0183-the-anthropic-licence-manager-is-a-module-and-hands-tokens-to-the-agent-over-the-bus.md
- 02-DECISIONS/0181-the-operator-account-is-a-node-fact-and-a-home-is-a-placement-root.md
- 02-DECISIONS/0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md
- 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
- 02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md
- 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
---
# 40. Building the operator's agent and its licence manager
**The work of [design 36](36-the-operators-agent-on-a-machine.md) and [design 39](39-the-anthropic-licence-manager.md),
broken into packages that each end at something a person can see run, in the order their
dependencies allow.** The two designs are the authority on *what* is built; this document holds the
packages, their order, their sizes and their proofs, and is wrong the moment it disagrees with them.
It is the shape [design 38](38-building-the-operators-machine.md) gave the operator's machine, applied
to the two modules that make its agent work.
## How this is built, and where it is run
**On the live mesh, by the operator's decision** ([ADR 0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md),
and design 38's words on the same day). Every package is written with unit tests, committed on one
branch per repository ([playbook 07](../../00-META/process/07-feature-branches.md)), and proven on the
machines: the control node first for the manager, one workstation first for the agent, then the rest.
The cost accepted: a broken agent module leaves a workstation's agent without the mesh's instructions
or with a stale token until the next push; the person's own files under the home are never in reach of
the failure, by [ADR 0182](../../02-DECISIONS/0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md).
Each package names what proves it. A package that cannot name its proof is divided until it can.
## What exists already, measured
Counted 2026-10-02 in the repositories and on the machines. Nothing here is new ground; every package
reshapes something standing.
| Piece | Today | Becomes |
|---|---|---|
| the vendor's refresh, the sealed box, the grant file | `anthropic-manager` in the catalogue: a token-endpoint client, a sealed-box primitive, adoption of a grant sealed to a node's key; assigned to nothing | the manager's refresh and adoption, with the lease, the floor and the cadence the predecessor's manager had |
| the credentials write, the refresh-token strip, the identity read | `anthropic-consumer` in the catalogue: tested; assigned to nothing | the agent module's write, unchanged in shape |
| the predecessor's manager and consumer | two modules in the retired system: the lease per licence, the expiry floor, the lineage comparison, the identity guard, three touchpoints with fallbacks, cooldowns on alarms | ported as logic with its tests; nothing of its registry or its bus |
| the host's `process` and `archive` shapes | fetch a bundle by digest and run it supervised; fetch and unpack an artifact | **unchanged** — the manager's daemon is one process; both modules' tools are bundles |
| the manifest's `uses`, `claims`, `invokes seat:<seat>.<verb>`, node-scoped `provides` with `serves: {port}` | all four exist and are used by other modules | **unchanged** — the agent uses the seat and invokes its verbs; the console provides its endpoint |
| the console | a container per node, MCP on loopback, no provision | gains one provision; becomes the node-tools runtime's serving mode under design 38's WP3 |
| the agent's package | present on both workstations from a build the predecessor's helper made; the distribution's repositories do not carry it | declared; satisfied where present, refused where not, until a package repository seat exists |
| the operator account | a column on every node record, **empty on all four** | stated by the operator, before anything home-scoped lands |
**One dependency decides the order.** Both modules serve tools and the agent module's tools write
under `/etc` and, as the operator, under the home. Under [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
tools run in the node's tool runtime, host-side, which design 38 builds in its WP1–WP3. Writing a
per-module tool container for these two modules would be building the pattern that record retires, so
**the live proofs of WP3 to WP5 below wait for design 38's WP3.** Everything before a live proof —
manifests, code, tests — does not, and is written now.
## The order the work allows
```
WP0 the operator states the facts (the live mesh) ── accounts, roles, licences to adopt
WP1 the console provides its endpoint (mesh-catalog) ── small, independent
WP2 the licence manager, built and tested (mesh-catalog) ──┐ independent of each other;
WP3 the agent module, built and tested (mesh-catalog) ──┘ both wait on design 38 WP3 to run
│
WP4 the manager live on the control node (the live mesh) ── three licences adopted, a refresh seen
WP5 the agent live on one workstation (the live mesh) ── the hand-over, the switch, the instructions
WP6 the rest of the nodes, and the predecessor's remains ── adoption from a login, the retirements
```
WP1, WP2 and WP3 touch different directories of one repository and meet only at the seat's name and
the provision's name; they are built in parallel. WP4 is the first time anything on a machine changes.
WP5 is the proof of the whole.
## WP0 — The operator states the facts
*The live mesh. An hour, and it is the operator's.*
The account on each node record, through the controller's node command — none is stated today, and
[ADR 0181](../../02-DECISIONS/0181-the-operator-account-is-a-node-fact-and-a-home-is-a-placement-root.md)
refuses a home-scoped module without one. The role of each node, as the agent module's setting on the
node layer, once the module exists. Which three licences exist and what each is called.
**Proof.** The controller's node command lists an account for every node.
## WP1 — The console provides its endpoint
*mesh-catalog. Half a day.*
**What changes.** The console's manifest gains a node-scoped provision — working name
`console-endpoint`, fixed when the manifest is written — serving the port the machine gave it, as the
local model server already does for its API. Co-location resolves it
([ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md),
[to-be 34](34-the-console.md) as amended). When design 38's WP3 moves the console into the node-tools
module, the provision moves with it; it is a line in a manifest either way.
**Proof.** The controller's plan for a workstation shows a consumer of the provision bound to the
console's port; the same consumer on a machine without the console is refused naming the provision;
the catalogue's tests pass.
## WP2 — The licence manager, built and tested
*mesh-catalog. Two to three days; the largest package.*
**What is written**, as design 39 says:
1. **The manifest.** Claims the mesh-scoped seat `anthropic-licence-manager` with its verbs; requires a
database, a `secret` for the key its grants are encrypted with, and the bus; a `bundle` of tools; a
`process` for the daemon that refreshes, collects usage and notifies, on a schedule; declared
settings for the cadence, the usage threshold and the cooldown, each with a default; `invokes` the
agent module's `apply`.
2. **The store.** Migrations for licences, bindings, usage and audit, with the lease and the
notification slot as columns, numbered and idempotent.
3. **The refresh.** The token-endpoint client and the sealed box from `anthropic-manager`; the plan
(floor, cadence, forced, cannot) and the lease from the predecessor, as pure functions with their
tests; the vendor's reason logged on failure; counted failures, one notification per cooldown.
4. **Adoption.** From a file on the manager's node for the API key; from a sealed grant a node offers;
the identity guard that refuses a mismatch and notifies.
5. **Usage.** The vendor's reading per licence on a schedule, stored raw and normalised
([ADR 0054](../../02-DECISIONS/0054-model-usage-is-recorded-at-two-grains.md)), one notification
per threshold crossing.
6. **The verbs**: `licences`, `bindings`, `bind`, `switch`, `release`, `refresh`, `usage`, `adopt`,
`register`, `current` — the last answering a consumer's token sealed to the key that consumer
registered.
7. **The hand-over**: on rotation or switch, one call to `claude-code.apply@<node>` per bound node,
the token sealed to that node's key, the answer recorded.
**Proof, before anything runs live.** Unit tests: two refresh runs started together rotate one grant
once; a grant with a mismatching identity is refused; a worker bound to a dead licence is refused and
never lent another; every event the daemon emits is free of a token; the hand-over payload opens only
with the registered key. The catalogue's checks: no installation named, no secret in a declared file.
## WP3 — The agent module, built and tested
*mesh-catalog. Two days.*
**What is written**, as design 36 says:
1. **The manifest.** The agent's package; the state directory; the facts file carrying the node's
name, the operator account and its home, the console's bound port, the role and the extra tool
servers from settings; requires the console's endpoint and the bus; `uses` the seat and `invokes`
its `register`, `current` and `adopt`; a `bundle` of tools. **No file resource under a home or
under `/etc`.**
2. **The renderer.** From the facts file and the current binding, the managed settings file (the tool
servers under the entry `mesh`, the attribution trailers, and the key-helper for an API-key binding)
and the managed instruction file (§3 of design 36), written under the agent's managed directory
with the escalation the tool performs for itself; idempotent; re-run when the facts file changes.
3. **The keypair**, made once in the state directory, the public half registered with the seat at
start and at every start.
4. **The consumer side**: `apply` (a rotation applied only if newer within one lineage, a switch applied
regardless, the answer naming the outcome and never a token); the pull at start and near expiry;
the credentials write as the operator, access-token-only, from `anthropic-consumer` with its tests;
the key-helper program for the API key; the offer of a login to the seat, sealed, after reading the
account's identity.
5. **The tools**: `apply`, `licence_status`, `mcp_configure` (validates a server, sets the module's
setting through the controller's settings verb), `render` (re-render now, for a person).
6. **The documentation**: the six predecessor files and the hand-made console entry a person removes
on a workstation that carried the predecessor.
**Proof, before anything runs live.** Unit tests: the renderer writes only the mesh's keys and leaves
every other key of a seeded settings file; the credentials write strips a refresh token and is atomic;
the lineage comparison from the predecessor, with its cases; a login offer carries the identity it read.
The catalogue's checks pass.
## WP4 — The manager live on the control node
*The live mesh. Half a day, after design 38's WP3.*
**Order.** Assign the manager on the control node; push. Adopt the API key from a file there. Adopt
the two subscription grants: a login in a throwaway home on the control node, offered to the seat the
way a node's agent module will. Bind each node's agent to a licence.
**Proof.** Through the console: `anthropic-licence-manager.licences` lists three licences with
identity and expiry; within the cadence, the audit shows a rotation and `licences` shows a later
expiry; a forced `refresh` on one licence is logged with the vendor's answer; the two retired catalogue
modules are still assigned to nothing.
## WP5 — The agent live on one workstation
*The live mesh. Half a day. The proof of the whole.*
**Order.** Set the workstation's role in the module's settings. Record the checksums of everything
under the person's agent directory. Assign the module; push. Remove the six predecessor files and the
hand-made console entry. Start a new session.
**Proof.** The managed directory holds the settings and instruction files, owned by root. Everything
under the person's agent directory is byte-identical to before except the credentials file, which is
owned by the operator, readable by nobody else, and names no refresh token. The new session lists the
mesh's tools under `mesh` once, answers *which node am I* from the instruction file, and makes a model
request. `anthropic-licence-manager.switch` to the second subscription licence changes the token on
the workstation within a minute, and neither the verb's answer nor either module's log holds a token.
Switched to the API-key licence, the credentials file is left as it was and the agent authenticates
through the key-helper. Switched back.
## WP6 — The rest of the nodes, and the predecessor's remains
*The live mesh and mesh-catalog. One day.*
**Order.** Assign the module on the second workstation and on the servers whose account is stated;
remove the predecessor's files on the second workstation. Log in on a workstation under a licence's
account and watch the offer be adopted — and under the wrong account, and watch it refused and
notified. Retire `anthropic-manager` and `anthropic-consumer` from the catalogue. Set designs 36 and
39 to `implemented` for what runs, with the as-is written
([playbook 02](../../00-META/process/02-graduation.md)).
**Proof.** Every node with an account runs the module and `licence_status` answers on each; the
refused login's notification arrived; the catalogue has no module built on the old placement.
## What is deliberately not here
- **The package repository seat** for a distribution that does not carry the agent's package
(design 36 §7). A fresh node refuses the module in the package manager's words until it exists.
- **An automated switch on exhaustion.** The readings are kept from WP4; the policy is a later record.
- **Workers and the mesh's own sessions as consumers.** The manager's bindings and fallbacks know them
from WP2; the consumers themselves do not exist yet ([to-be 15](15-the-agent-session.md),
[ADR 0003](../../02-DECISIONS/0003-agents-are-persistent-employees.md)).
- **Whether a refresh token is single-use.** WP4 may measure it on a licence deliberately refreshed
twice; the design holds either way.
## How this list is kept true
Each package's proof is run when the package is finished and its line here gains the date and the
commit, as design 38 does. A package whose proof fails is not reworded; the failure is recorded under
it and the package stays open. When WP5 is proven, designs 36 and 39 move to `in-progress` with their
owning repository, and when WP6 is proven to `implemented`, with the as-is written.
@@ -1,9 +1,9 @@
---
status: located
status: resolved
opened: 2026-09-26
located-in: [mesh-controller internal/catalogue/declaration.go, mesh-sdk src/provisioner, mesh-catalog modules/minio]
fixed-by:
amended-design:
located-in: [mesh-controller internal/catalogue, mesh-sdk src/provisioner, mesh-catalog modules/minio]
fixed-by: 02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md
amended-design: 03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md
---
# 124 — A consumer cannot be told a value its provider derived for it, so it transcribes one
@@ -63,3 +63,27 @@ compares it to what the provider will actually create. The one wrong instance wa
- What would have caught the wrong instance? A test that resolves a consumer's grant and compares the
bucket in its own configuration against the one the provider would create is a check that could
exist today, for any interface, without the mechanism above.
## Answered, 2026-10-02 — [ADR 0188](../../02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md)
The channel is the provider's own `serves` block, which may now name the consumer the mesh is
serving: `${consumer:as}` and `${consumer:as:dns}`. The mesh fills it once, where it knows who the
consumer is, and delivers the one filled value to both ends — the consumer's binding and its
`${bound:…}` substitutions, and the provider's contributions entry, so a provisioner is told the
name rather than deriving it. Each open question above, answered:
- **Should a provider return values from provisioning?** No. It would make a grant carry data the
provider wrote, make a consumer's declaration wait on its provider's reconcile loop, and put the
rule where nothing can refuse it. The reasoning is in the record.
- **Or should `serves` say a value is derived?** Yes, and the mesh performs the derivation — but it
learns no protocol doing it. The only fact is the identity the mesh itself minted, in one of two
alphabets it already knows.
- **Should a consumer that names the resource be refused?** Yes. A consumer's file that already
contains the value the mesh is about to derive for it is refused at resolution, naming the
placeholder to write instead. That is the check this report asked for, and it is exact rather than
heuristic: a derived value carries the identity minted for this consumer on this machine, which
nothing else would spell out.
minio's `bucketFor` is gone; its manifest serves `"bucket": "${consumer:as:dns}"`. The three
consumers' hand-written bucket names are gone with it — each of them also named the machine the
module happens to run on, which is the second thing wrong with a transcription.
@@ -0,0 +1,88 @@
---
status: open
opened: 2026-10-02
located-in:
- mesh-controller
fixed-by: 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md
amended-design:
---
# 201 — A push recreated the controller at a digest older than the seat row its successor had written
## What was observed
2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the
controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan
built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The
host's plan then pushed the control node with the controller digest it had recorded when it was made —
the previous build — and recreated the controller container on it. The older binary read the row, found
a verb it could not run, and refused to start:
```
mesh-controller: the mesh-controller seat's row declares "command", which this control plane
cannot run: "command" is not a verb the mesh-controller seat serves
```
A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since
the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the
newer image, outside the service, to push the control node again; the push sent the newer digest and
the controller came up.
## Why it matters beyond this instance
The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md));
a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of
2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that
wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md));
for every other module an older digest is a brief regression a later push corrects. For the controller
it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct
it is the thing that is down. Two plans that overlap will happen again whenever two people merge within
a minute.
## What a fix would have to do
Either of two, and the first is the smaller:
- A push never sends a controller digest older than the one the running controller is — the controller
knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words.
- Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves
what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check
exists to refuse.
Until one is built: do not merge a controller change while another plan is rolling, and after merging
one, wait for `node show` on the control node to report the new controller before merging anything else.
## References
- [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
- mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`
## Half of it is closed, 2026-10-02
[ADR 0185](../../02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) takes
the outage out of it: a control plane behind its seat's row now serves every verb it can run, says
which it cannot, and answers the reason when one of those is called. The same race today would cost
the verbs the newer build added, for as long as the older binary is in place, and the ordinary
"this machine is behind" machinery would put the newer one back without a hand.
**The race itself is still open**, and this report stays open for it. What was established while
closing the other half, so the next reader does not redo it:
- Composing and sending are serialised per machine by a session advisory lock in the store, so two
control planes cannot compose one machine's declaration at the same time. The stale content did
not come from two concurrent composes.
- A container's image is resolved into the module's manifest when it is *built*, and a push composes
from the catalogue as it is at that moment, under the hold. So a compose that ran after the build
was taken in could not have named the older image.
- The declaration's sequence orders arrival and nothing else (the numbering of
[issue 107](../107-a-declaration-carries-no-order/00-report.md)); it cannot tell a later send
carrying earlier content from a later send carrying later content. The host refuses a declaration
numbered below the last it applied, and both of these were above it.
- The machine's own journal shows the two applies ten seconds apart and which replaced what; it does
not record which image each declaration named, which is the one fact that would settle it. A host
that recorded the digest it was told, per apply, would have answered this in a minute.
So the trigger is not yet pinned, and guessing at the push path is the most expensive place in the
mesh to guess. The fix the report first suggested — a push never sending a control plane a digest
older than the one that machine reports running — closes the class without needing the trigger, and
is now a correctness nicety rather than the difference between a working mesh and a dead one.