Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0b9fc90885 | ||
|
|
983fd412c6 | ||
|
|
9ac2493e2c | ||
|
|
560f25c2c7 | ||
|
|
9e0288128b | ||
|
|
d57289e049 | ||
|
|
d4a2f99ab5 | ||
|
|
a9f91fdd0c | ||
|
|
131a5e4714 | ||
|
|
329a24fdae | ||
|
|
7f72f3b79a | ||
|
|
114a71f36f | ||
|
|
0f407417f3 | ||
|
|
d0d5799884 | ||
|
|
9ba4de5557 |
+132
@@ -0,0 +1,132 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
---
|
||||
|
||||
# 179. The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail
|
||||
|
||||
## Context
|
||||
|
||||
Read on the control node on 2026-10-02, the day the machines were confirmed filtered by the mesh
|
||||
alone ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)): the intrusion
|
||||
prevention watched one door. Its two jails read the ssh daemon's journal and its own log, banned
|
||||
five failures in ten minutes for ten minutes, and in a day had seen twelve thousand failed logins
|
||||
from three hundred addresses and banned none of the busiest, which paced themselves at one try every
|
||||
ten minutes. The mail submission port took a hundred and sixty password guesses in the same day from
|
||||
thirty-eight addresses with no jail reading it at all; the forge and the public proxy had no jail
|
||||
either, and the proxy logged nothing a jail could read. Nobody could see the jails without a shell:
|
||||
the module's three tools existed in code and were served by nothing, and the seat it holds declared
|
||||
no verbs.
|
||||
|
||||
Three things were missing and they are three shapes the mesh already has. The packet filter's seat
|
||||
serves verbs every holder owes ([ADR 0170](0170-the-firewall-seat-serves-its-verbs.md)); the
|
||||
intrusion seat serves none. A module's `listens` compose into the machine's filter, and [to-be 31](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
|
||||
says a module's `jails` compose into the machine's intrusion prevention the same way — the controller
|
||||
composes them, and no module declares one. And a jail reads a log; a container's output goes to a
|
||||
file of the runtime's own under a path that changes when the container is recreated, which is why
|
||||
no jail could read the mail front end, the forge or the proxy, however they logged.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. The `node-intrusion-prevention` seat serves four verbs**, and a module that claims it serves
|
||||
all four or is refused the claim, as with every seat:
|
||||
|
||||
- `status` — every jail with what it watches, how many addresses it is counting failures against and
|
||||
holding now, and the totals since it started; one jail's detail when named. Read-only.
|
||||
- `banned` — every address banned now, with the jail holding it, when it was banned and when the ban
|
||||
ends. Read-only.
|
||||
- `ban` — ban one address in one jail now, for that jail's ban time. An operator's act on the live
|
||||
ban list, which the mesh composes the rules for and never writes itself.
|
||||
- `unban` — let one address go, from one jail or from every jail.
|
||||
|
||||
A holder may serve its own tools beside these; the fail2ban module reads one jail's effective
|
||||
settings as its own.
|
||||
|
||||
**2. A container may log to the journal.** `logging: journald` on a container has the host run it
|
||||
with the journal as its log driver; the journal keeps the container's name on every line, and
|
||||
`docker logs` keeps working. Where a container logs is part of its spec, so moving it recreates the
|
||||
container, and the only place besides the runtime's own file is the journal: a machine's intrusion
|
||||
prevention reads the journal already, for the ssh daemon, and a container that logs there is read
|
||||
the same way, by the container's name, whatever the container is called by the runtime this time.
|
||||
|
||||
**3. A module with a door declares its jail, and the holder composes them.** What to-be 31 designed
|
||||
is now the rule: a module whose service authenticates from outside — the mail front end, the forge,
|
||||
the public proxy — declares in its manifest what a failed attempt looks like in its log and how to
|
||||
ban on it, naming no node and no path; the module that holds the intrusion seat declares where the
|
||||
composed jails and filters land, and the mesh writes them on every machine that runs both. A machine
|
||||
not running the module has no such jail. The holder restarts its daemon on the composed file.
|
||||
|
||||
**4. The base is strict, and the mesh's own range is never banned.** Three failures in a day ban for
|
||||
a day, on every jail unless the jail says otherwise; banned twice in two weeks, by any jail, is
|
||||
banned for four. The attackers this mesh sees pace themselves under any ten-minute window; a day's
|
||||
window counts them. A person who mistypes three times from one address is out for a day from that
|
||||
address, and never from a machine of the mesh, whose range stays in the never-banned list the module
|
||||
has carried since [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md). The operator
|
||||
chose this knowing it.
|
||||
|
||||
**5. The proxy says a refused name in its log.** A request for a name this mesh does not serve, from
|
||||
outside, is what a scanner does; the proxy already logged a certificate refused for such a name, and
|
||||
now logs the plain request too, with the asking address last, as its own jail's filter expects it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The seat's row gains four verbs; a mesh that already runs widens its row at the next controller
|
||||
start. The fail2ban module claims them and gains a runtime — a tool server whose image carries the
|
||||
fail2ban client, with the daemon's socket shared in from the machine, and nothing else of the
|
||||
machine. The daemon stays the machine's; what runs in the container is only the client.
|
||||
- **That runtime is the shape the catalogue has today, and it is on its way out.**
|
||||
[ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md),
|
||||
accepted the same day as this record, replaces a tool container per module with one tool runtime
|
||||
per node on the host side, taking each module's tools as a bundle. Nothing here depends on the
|
||||
container: the verbs, the client that speaks to the daemon over its socket, and the jails are the
|
||||
same code under either. This module converts with the packet filter's, whose runtime that record
|
||||
names, and the socket it needs becomes the node runtime's to reach rather than a mount of its own.
|
||||
- The host's container vocabulary grows by `logging`; an older host refuses a declaration that carries
|
||||
it, so the host rolls before the modules. Three containers are recreated once, when their modules
|
||||
are pushed with the field: the mail front end, the forge and the proxy — each a moment's outage.
|
||||
- The fail2ban module declares where jails compose (`jailing`) and the directory the filters go in;
|
||||
the mail, forge and proxy modules each declare one jail reading the journal by their container's
|
||||
name. The composed jail file is the one resource the daemon restarts on when a module arrives or
|
||||
leaves a machine.
|
||||
- The two base jails and the composed ones take the day's window; the ssh jail's ten minutes are
|
||||
gone. An address banned on the first day of this record stays banned for the day.
|
||||
- The module's three old tools, served by nothing, are replaced by the seat's four verbs and one
|
||||
own tool; `fail2ban_status` as a name is gone.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The seat declares the four verbs; a claim that serves fewer is refused by name | the catalogue's seat tests |
|
||||
| `status`, `banned`, `ban` and `unban` read and steer the daemon through its client, with the shapes fail2ban 1.1.0 printed live; a non-address and a non-name are refused before anything runs | the module's tests over a fake command runner |
|
||||
| A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests |
|
||||
| A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) |
|
||||
| The proxy logs a refused name with the address last | the proxy's tests |
|
||||
| A jail's pattern names `<HOST>` once per shape, since two is a duplicate capture group and costs the machine every ban | the catalogue's manifest tests |
|
||||
| Live | done 2026-10-02: `status` and `banned` answered on both servers through the console; the proxy's jail counted seven refusals on the home server; a documentation address banned in the ssh jail came back with its end time and was released |
|
||||
|
||||
## Built and proven live, 2026-10-02
|
||||
|
||||
All five rules are in the mesh. The host carries `logging`; the controller's seat row carries the four
|
||||
verbs and the proxy says a refused name in its log; the fail2ban module holds the seat from a runtime
|
||||
with the daemon's socket shared in, composes the jails, and the mail front end, the forge and the
|
||||
proxy each declare one. Through the console on the control node: `status` listed five jails with what
|
||||
each watches, `banned` listed the nine the long jail holds, and a documentation address banned in the
|
||||
ssh jail came back with its ban's end time and was released again. On the home server the proxy's jail
|
||||
had counted seven refusals within minutes of starting.
|
||||
|
||||
**One fault, found by the machine and not by a test.** The proxy's pattern matched two shapes of
|
||||
refusal in one expression and so named `<HOST>` twice. fail2ban expands that placeholder into a named
|
||||
capture group; two of them is a duplicate group name, and the daemon refuses *its whole configuration*
|
||||
and exits — both servers kept no bans at all for about ten minutes, every jail and not the one at
|
||||
fault. The pattern is now one per shape. A manifest check refuses the mistake at merge time, naming
|
||||
what it would cost, which is the only reason this record can claim the rule rather than the instance.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0170](0170-the-firewall-seat-serves-its-verbs.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
- [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md), [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md), [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md)
|
||||
@@ -60,7 +60,7 @@ configuration file; the rollback path it described is given up on purpose.
|
||||
|---|---|
|
||||
| An absent package is removed when present, left when not, and read back | host tests over a fake package manager |
|
||||
| An uninstalled front end is recorded as removed and nothing is asked of it | a host test with ufw missing on a converged apply |
|
||||
| Live | the two machines report ufw gone: `pacman -Q ufw` has no answer, `node show` says removed, `status` is well |
|
||||
| Live | done 2026-10-02: both machines report ufw gone — `pacman -Q ufw` has no answer, `node show` says *removed*, `status` is well. The home server said *retired* for six hours after the package went, because this record's step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)) and one dead tracker was failing its applies ([ADR 0187](0187-a-dead-tracker-is-not-the-machines-failure.md)) |
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 184. A service the mesh asked to run is still running a moment later
|
||||
|
||||
## Context
|
||||
|
||||
The host already refuses to take a service manager's word for it. Three places in one function read
|
||||
a unit back after acting on it, each with a comment saying why: *a service manager accepting a
|
||||
command says the transaction was accepted, not that the unit is running — one that starts and
|
||||
immediately dies satisfies it.* The intent was right and the implementation did not reach it.
|
||||
|
||||
On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the
|
||||
files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at
|
||||
that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public
|
||||
machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said
|
||||
so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing.
|
||||
|
||||
The read-back races the failure. A service manager returns when it has started the process; a daemon
|
||||
that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One
|
||||
look sees `activating` or `active` whatever the process is about to do, and *the host reports success
|
||||
for a machine that is already wrong* — the one shape of failure this host exists to refuse
|
||||
([ADR 0005](0005-the-node-host.md)).
|
||||
|
||||
A command the module declares — *test the configuration before restarting* — was considered and
|
||||
rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification
|
||||
command is a command: a declaration that carried one would be remote execution over the bus,
|
||||
arriving as root on every machine, which is a far larger door than the fault it closes. The host
|
||||
does not need one. It already knows what it asked for.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A unit the host has just asked to run is read twice**, with a pause between the reads long
|
||||
enough for a daemon that refuses its configuration to have exited. Not running at the second look is
|
||||
a failure of that resource, named with the unit and the state it is in — the same failure the single
|
||||
read was always meant to catch.
|
||||
|
||||
**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both
|
||||
looks and is accepted, exactly as before. What the second look catches is a unit that *was* running
|
||||
and is not any more. A service asked to be stopped is not waited on at all.
|
||||
|
||||
**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking
|
||||
the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and
|
||||
no command reaches a machine that did not already come from a built artifact.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is
|
||||
bounded by the number of services that changed in that apply, which is usually none.
|
||||
- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the
|
||||
resolver — now fails its apply when the composition is bad, instead of reporting success onto a
|
||||
dead daemon. `status` names the machine, which is how the operator finds out.
|
||||
- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s
|
||||
manifest check is what refuses the one that caused this, at merge time; this record is what makes
|
||||
the *next* one visible within a minute rather than invisible until something asks the daemon a
|
||||
question.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does |
|
||||
| A unit still starting is accepted at both looks | a host test |
|
||||
| A service asked to be stopped is not waited on | a host test |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
|
||||
---
|
||||
|
||||
# 185. A control plane behind its seat's row serves what it can
|
||||
|
||||
## Context
|
||||
|
||||
The mesh's own verbs are the controller seat's tools, and the seat's row is the store's
|
||||
([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). A control plane reads the
|
||||
row at start and installs a handler per verb; a verb the row carries that the binary cannot run was
|
||||
refused at start rather than at the first call, so that a disagreement between the row and the
|
||||
binary was said early. The refusal aborted the start.
|
||||
|
||||
On 2026-10-02 a merge added one verb. The new control plane started, widened the row, and ran. A
|
||||
push a few seconds later recreated its container at the previous image — a stale declaration from
|
||||
an overlapping wave, [issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) —
|
||||
and the older binary read a row naming a word it had never heard. It refused to start, and kept
|
||||
refusing. The mesh had no voice for ten minutes: no verb answered, no node could be pushed, no build
|
||||
was dispatched, and `status` said nothing because `status` is one of the verbs that had stopped
|
||||
being served. The way back was a person running the binary by hand outside its service, because the
|
||||
push that would have replaced it is itself a verb of the control plane that was down.
|
||||
|
||||
The check was right about the fact and wrong about the cost. A row ahead of a binary is the ordinary
|
||||
state of a roll-out: the row is widened by whichever control plane starts first, and a mesh with one
|
||||
control plane sees that gap on every merge that adds a verb. Making it fatal turned a transient into
|
||||
an outage with no path out that did not need a human.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A control plane serves the verbs it can run and does not refuse to start for the ones it
|
||||
cannot.** The row remains the authority on what the seat serves; this is only about what this binary
|
||||
does when it is behind the row.
|
||||
|
||||
**2. A verb it cannot run answers the reason.** Not silence and not a missing subject: a caller gets
|
||||
a sentence naming the verb, saying this control plane cannot run it and that it is a verb of a newer
|
||||
build. A verb that is simply absent from the row is still not served at all — that is the row
|
||||
deciding, which is unchanged.
|
||||
|
||||
**3. It says so once at start**, naming every verb of the row it cannot run, so the gap is visible
|
||||
in the log of the thing that has it rather than only at the moment somebody calls one.
|
||||
|
||||
**4. A mesh with no controller seat at all is still a refusal.** That is not a version gap, it is a
|
||||
mesh that has not been seeded, and nothing this control plane does would be meaningful.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An overlapping roll-out costs the verbs the newer build added, for as long as the older binary is
|
||||
in place. Everything else — every push, every build, every read — keeps working, and the ordinary
|
||||
machinery that notices a machine is behind is what puts the newer binary back.
|
||||
- The log gains one line on a control plane that is behind, and nothing on one that is not.
|
||||
- Issue 201's other half remains: the push that sent a stale declaration is a race worth closing on
|
||||
its own terms. This record makes that race survivable rather than fatal, which is the difference
|
||||
between a transient and an outage, and is deliberately the cheaper half.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A row carrying a verb this build cannot run still serves every verb it can, and names the one it cannot | a controller test over a widened row |
|
||||
| The unknown verb answers a sentence naming itself and saying this build is behind | the same test |
|
||||
| A mesh with no controller seat is refused | the existing start-up path |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
||||
- [Issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)
|
||||
- [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
---
|
||||
|
||||
# 186. A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) gave the
|
||||
public proxy a jail. Within the hour the home server's ban list held `192.168.1.1` — the house's own
|
||||
router. The router reflects local traffic, so every client in the building reaches that machine as
|
||||
the gateway's address; one local request for a name the mesh does not serve, three times in a day,
|
||||
and the whole house is refused by the machine it was asking. The jails inherited an `ignoreip` of
|
||||
the loopback and the mesh's own range, which was right when the only jail read the ssh daemon and
|
||||
the only clients were the mesh's; a jail on a public front door sees the neighbours too.
|
||||
|
||||
The same jail broke the other half of [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md).
|
||||
The home server began reading *NOT the mesh alone: 1 rule set the mesh did not write refuses traffic
|
||||
here*, and the rule set named was the mesh's own ban chain, written by the mesh's own intrusion
|
||||
prevention minutes earlier. The host's reader of the legacy filter required every path into a chain
|
||||
of refusals to come from a built-in chain whose policy accepts, before it would call that chain a
|
||||
ban. On that machine the chain hangs off the container runtime's user chain as well as the input
|
||||
chain, and the runtime had set the forward policy to DROP — so the mesh reported its own work as a
|
||||
foreigner's, on the one machine where the group's exit condition was supposed to hold.
|
||||
|
||||
Both faults are one mistake in two places: a rule written about the public internet, applied to
|
||||
everything that arrives.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A ban list never holds a neighbour.** The jails the mesh composes never ban a source on a
|
||||
private range — the mesh's own range, which was already named rather than written
|
||||
([ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md)), and every address space
|
||||
reserved for private use beside it, in both families. A machine behind a router that reflects local
|
||||
traffic sees its whole building as one address; a ban there is a self-inflicted outage, and the
|
||||
sources worth banning are not on those ranges in the first place.
|
||||
|
||||
**2. The mesh's own bans are its own wherever they hang.** A chain of refusals is a ban list when
|
||||
every refusal names the sources it refuses and the chain accepts nothing — the rule the host already
|
||||
applied to the packet filter's own tables, now applied to the legacy filter too, and nothing more.
|
||||
The policy of the chains that jump into it says nothing about what it is: that policy is already
|
||||
classified where it belongs, as the container runtime's, and requiring it here counted it twice.
|
||||
|
||||
**3. A chain that accepts anything is still not a ban.** That is what keeps a predecessor's
|
||||
allow-these-and-drop-the-rest chain classified as something an operator must look at, which is the
|
||||
distinction [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md) exists to draw.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The composed jails gain the private ranges in their never-ban list. An address already banned
|
||||
stays banned until it is released; the house's router was released by hand the moment it was found.
|
||||
- The home server reads *the mesh alone* again, which is group 7's exit condition and was false for
|
||||
about an hour.
|
||||
- A machine whose apply fails for an unrelated reason does not revisit its found firewall's record
|
||||
at all — the step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)).
|
||||
The home server's record therefore still reads *retired by the mesh* although the front end is
|
||||
uninstalled, and will correct itself once that machine's own stuck module is fixed. It is a stale
|
||||
record, not a wrong machine.
|
||||
- The record number the front end's removal was given moved under it: another session took 0175
|
||||
while that record was in review, and it is now
|
||||
[ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md). The citations
|
||||
the host and the control plane print were pointing at an unrelated record and are corrected here.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A private source is never banned | the module's jail configuration, read back by `fail2ban.fail2ban_settings` on a machine |
|
||||
| The mesh's own ban chain reads as a ban behind a dropping forward policy | a host test over the home server's own captured rule set |
|
||||
| A chain that accepts anything is not a ban | a host test |
|
||||
| Live | done 2026-10-02: all four machines read *the mesh alone*, the home server counting its own ban chain as a ban; no ban held anywhere is a private address |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
|
||||
- [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md), [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
|
||||
---
|
||||
|
||||
# 187. A dead tracker is not the machine's failure
|
||||
|
||||
## Context
|
||||
|
||||
The home server had not applied a declaration cleanly since midday. One run-once step — the one
|
||||
that writes a media app's download clients and indexers through the app's own API — exited
|
||||
non-zero, forty-nine times over six hours, for one public tracker that had stopped answering. The
|
||||
step's own words: the entry was *written*, and the app's test of it then failed with a 400 from the
|
||||
indexer proxy. The machine reported *not doing what it was told* for the rest of the day.
|
||||
|
||||
What that gated matters more than the step. A converged machine retires the firewall it was found
|
||||
with only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)),
|
||||
so that machine went on recording its found front end as merely *retired* long after the package
|
||||
had been uninstalled ([ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)).
|
||||
A dead public tracker was holding a firewall record hostage, which is not a connection anybody
|
||||
would design.
|
||||
|
||||
The step already knew this was not its business. It had a rule for exactly this: an entry the mesh
|
||||
only *found and re-pointed*, rather than one it was told to make, whose feed is gone, is said and
|
||||
left as found — *failing the node's apply on every heartbeat for it reports the mesh as wrong about
|
||||
a tracker*. The rule was there and matched one shape of the fault. An app can refuse to save such an
|
||||
entry, and it can save it and then fail its own test; saving validates settings, and the test runs a
|
||||
live search. The rule caught the first and let the second through.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. An entry the mesh only found is never the machine's failure.** Whatever shape the app's
|
||||
refusal takes — it would not save it, or it saved it and its own test fails — an indexer the mesh
|
||||
found and re-pointed is reported as a notice and left as found. What decides is whose entry it is,
|
||||
not which sentence the app returned.
|
||||
|
||||
**2. What the mesh is answerable for is the plumbing.** That the entry exists, points at this
|
||||
mesh's indexer proxy, and carries the credential the mesh delivered — which was checked against the
|
||||
proxy before anything was written. Whether a public tracker answers today is not the mesh's to
|
||||
promise, and a machine that reports itself broken because one did is lying about itself.
|
||||
|
||||
**3. An entry the operator listed is theirs to insist on.** An indexer named in the step's settings
|
||||
is one the mesh was told to make, and it still fails the step when it cannot be made to work. The
|
||||
notice says so, and says that listing the indexer is how to turn it back into a failure.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The home server applies cleanly again, and everything a clean apply gates — its found firewall's
|
||||
record among it — follows.
|
||||
- A tracker that dies is a line in a report rather than a machine that reads as broken. An operator
|
||||
who wants it gone removes the entry or repairs the feed; the mesh says which, every time it runs.
|
||||
- The four Servarr modules carry one byte-identical copy of this step each
|
||||
([ADR 0069](0069-a-module-is-a-repository-and-a-path.md)), so the change lands in four places and
|
||||
a test refuses any drift between them.
|
||||
- It does not widen to a download client: one the mesh was told to write and cannot is still a
|
||||
failure, because the mesh chose it and nothing else will fix it.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A found feed whose tracker answers an error after the entry was written is a notice | the step's tests, with the home server's own message and the app's two validations modelled apart |
|
||||
| An indexer the settings list is still a failure | the same test |
|
||||
| The four copies of the step do not drift | the step's own sameness test |
|
||||
| Live | done 2026-10-02: the home server applies cleanly after six hours of failing, `status` holds no machine wrong or behind, and its found firewall reads *removed* |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0136](0136-a-step-gates-its-module-not-the-machine.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md), [ADR 0069](0069-a-module-is-a-repository-and-a-path.md)
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
|
||||
---
|
||||
|
||||
# 188. A provider declares what it derives for each consumer, and the mesh tells both ends
|
||||
|
||||
## Context
|
||||
|
||||
An arrangement between a consumer and a provider is delivered entirely by the mesh. Where the
|
||||
provider is, which port it answers on, what name the consumer must present, where its password
|
||||
is — each arrives as a fact the consumer reads from its binding, or as `${bound:…}` filled into a
|
||||
file before the declaration leaves the control plane. The provider invents none of it and hands
|
||||
none of it back ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)).
|
||||
|
||||
One kind of value escapes that. Where the **provider names the resource** — a bucket, a database,
|
||||
a vhost — the name is derived from the consumer, per consumer, and the mesh has no way to carry
|
||||
it. `serves` is a literal block in the provider's definition: the same values for every consumer.
|
||||
A provisioner's contract takes a provision and returns nothing. So a value the mesh's own rule
|
||||
produced reaches neither end as a statement; it is recomputed at one end and transcribed at the
|
||||
other.
|
||||
|
||||
The object store is the instance ([issue 124](../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)).
|
||||
Its provisioner normalises the login the mesh minted into a bucket name and creates, checks and
|
||||
removes exactly that; the rule lives in twenty lines of the module's own TypeScript. Its three
|
||||
consumers each write the answer into their own definition by hand. Two transcribed it correctly;
|
||||
one named a predecessor's bucket, and would have authenticated successfully and been refused on
|
||||
every object, which reads like a credential fault and is not one.
|
||||
|
||||
Even corrected, the transcriptions are wrong in a second way. Each is `mesh-<node>-<slug>`, so
|
||||
each **names the machine the module happens to run on today** — a definition stating a fact about
|
||||
one installation, which [ADR 0155](0155-a-definition-names-no-installation-and-how-that-is-checked.md)
|
||||
forbids and whose check does not catch because the name is not a domain. Move any of the three to
|
||||
another machine and its configuration points at a bucket its key cannot open.
|
||||
|
||||
The shape is not the object store's. A database provisioner that prefixed names, a queue provider
|
||||
that scoped vhosts, any provider that derives a resource from who is asking: each forces the
|
||||
consumer to reproduce somebody else's rule and keep it in agreement by hand.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A served value may name the consumer the mesh is serving.** A `serves` block, which is
|
||||
literal today, may interpolate the mesh's own statement of who the consumer is:
|
||||
|
||||
- `${consumer:as}` — the identity the mesh minted for this consumer, exactly as the login it is
|
||||
told to present ([ADR 0049](0049-a-consumers-identity-fits-the-tightest-backend.md));
|
||||
- `${consumer:as:dns}` — the same identity written as a DNS label.
|
||||
|
||||
Nothing else. **The mesh learns no protocol here; it spells its own name in an alphabet it already
|
||||
knows.** The identity is the mesh's, minted by the mesh, already capped at twenty characters
|
||||
because of what an S3 access key accepts; `dns` is that same name with its separator written `-`
|
||||
instead of `_`, which is the whole of the difference between the mesh's identifier alphabet and
|
||||
the one buckets, vhosts and hostnames use. A provider that needs a prefix or a suffix writes it
|
||||
around the placeholder, because a served value is a string.
|
||||
|
||||
The rejected alternative is **the provider returning values from provisioning** — the natural
|
||||
channel, since the provider is what derived them. It is rejected for three reasons, in order of
|
||||
weight. It inverts the delivery the mesh is built on: a grant would carry data the provider wrote
|
||||
rather than only data the mesh minted, and [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)
|
||||
removed exactly that second path once already. It makes a consumer's declaration incomplete until
|
||||
its provider's reconcile loop has run, so a consumer could not be composed before a provider
|
||||
answered — a bootstrap order the mesh does not have and does not want. And it puts the rule where
|
||||
nothing can check it: a value that arrives from a running process cannot be refused at resolution,
|
||||
only discovered wrong later, which is the failure this record exists to end.
|
||||
|
||||
**2. The mesh resolves it once, per consumer, and tells both ends from the one resolution.** At the
|
||||
moment a consumer's declaration is composed, the mesh knows exactly who the consumer is. There, and
|
||||
only there, the placeholders are filled. The result reaches:
|
||||
|
||||
- the **consumer**, as the served facts in its binding file and as `${bound:<provision>:<key>}` in
|
||||
any file it writes — unchanged mechanisms, carrying one more key;
|
||||
- the **provider**, as `serves` on that consumer's entry in its contributions file, so the
|
||||
provisioner is *told* the name rather than recomputing it.
|
||||
|
||||
**The provider stops deriving in code and starts declaring.** One statement, filled once, delivered
|
||||
to both ends: the two cannot disagree, because there is no second computation to disagree with.
|
||||
|
||||
**3. A served value stays settled before it is per-consumer.** Settings still compose into `serves`
|
||||
([ADR 0174](0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)), and the consumer
|
||||
placeholders are filled after that, so an operator may set a prefix and the mesh still derives the
|
||||
rest. A `${consumer:…}` naming a fact or an alphabet the mesh does not have is refused when the
|
||||
definition is parsed, with what it may say.
|
||||
|
||||
**4. A consumer may no longer name the resource its provider derives.** With the value delivered,
|
||||
a literal in a consumer's definition is not merely redundant — it is the one thing that can
|
||||
disagree with what the provider will actually create. The three object-store consumers lose their
|
||||
hand-written bucket names in this change.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One more thing a definition may say, and one less thing a module may be wrong about. The
|
||||
vocabulary grows by a placeholder; the catalogue loses three literals that named this
|
||||
installation's control node.
|
||||
- A provider's naming rule becomes readable in its definition instead of in its source. `minio`'s
|
||||
`bucketFor` goes; the manifest says `"bucket": "${consumer:as:dns}"` and the provisioner uses
|
||||
what it is given.
|
||||
- A provider that already serves consumers keeps serving them: the derived value equals what the
|
||||
code derived, so no bucket, database or login changes name. This is a change of **who says it**,
|
||||
not of **what is said**.
|
||||
- The mesh now holds a rule in another system's alphabet — one rule, `dns`, stated once. A second
|
||||
alphabet is a decision, not an addition: the cost of each is that the mesh must be right about
|
||||
somebody else's naming, and that cost is only worth paying where the mesh already mints the name.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- A served value naming an unknown fact or alphabet is refused at parse, with the list of what it
|
||||
may say — tested on both halves of the message.
|
||||
- Resolving a consumer whose provider derives a value puts that value in the consumer's binding
|
||||
file, in its `${bound:…}` substitutions, and in the provider's contributions entry for that
|
||||
consumer — one test asserting the three agree, because agreeing is the whole point.
|
||||
- Two consumers of one provider on one machine get two different derived values, and neither gets
|
||||
the other's.
|
||||
- A catalogue-wide test refuses a consumer definition that writes a literal where its provider
|
||||
derives: the provider's `serves` names the key, so the catalogue can say which definitions
|
||||
transcribe one.
|
||||
- `dns` is checked against the identity the mesh actually mints, not against an invented string:
|
||||
the test derives an identity with `ConsumerIdentity` and asserts the label it becomes.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 124 — a consumer cannot be told a value its provider derived for it](../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)
|
||||
- [ADR 0048 — a provider creates the credential the mesh minted, and seals nothing](0048-a-provider-creates-the-credential-the-mesh-minted.md)
|
||||
- [ADR 0049 — a consumer's identity fits the tightest backend](0049-a-consumers-identity-fits-the-tightest-backend.md)
|
||||
- [ADR 0174 — a node varies a module through settings and kept regions, never through an edit](0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)
|
||||
- [ADR 0155 — a definition names no installation, and how that is checked](0155-a-definition-names-no-installation-and-how-that-is-checked.md)
|
||||
- [design 27 — a module requires, the mesh resolves](../03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md)
|
||||
@@ -182,7 +182,13 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)
|
||||
- **0170** — [The firewall seat serves its verbs, and a foreign rule set is removed through one of them](0170-the-firewall-seat-serves-its-verbs.md)
|
||||
- **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md)
|
||||
- **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
- **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
|
||||
- **0184** — [A service the mesh asked to run is still running a moment later](0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md)
|
||||
- **0185** — [A control plane behind its seat's row serves what it can](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)
|
||||
- **0186** — [A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang](0186-a-ban-list-never-holds-a-neighbour.md)
|
||||
- **0187** — [A dead tracker is not the machine's failure](0187-a-dead-tracker-is-not-the-machines-failure.md)
|
||||
- **0188** — [A provider declares what it derives for each consumer, and the mesh tells both ends](0188-a-provider-declares-what-it-derives-for-each-consumer.md)
|
||||
|
||||
### Its tiers, from the bottom up
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ status: in-progress
|
||||
code: [mesh-host]
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md
|
||||
- 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md
|
||||
- 02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md
|
||||
- 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
|
||||
@@ -498,3 +499,17 @@ run and reported; the exit follows an in-flight apply rather than interrupting i
|
||||
not start is rolled back once and the second failure halts; a completed reconcile retires what is older
|
||||
than the predecessor and never the predecessor; and the newest of two delivered versions is the one
|
||||
that runs.
|
||||
|
||||
## A service is still running a moment later, 2026-10-02
|
||||
|
||||
[ADR 0184](../../02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md).
|
||||
The host has always read a unit back after acting on it, because a service manager accepting a
|
||||
command says the transaction was accepted and nothing about the process. The read raced the failure:
|
||||
a daemon that refuses the configuration the mesh just wrote exits a fraction of a second after the
|
||||
manager returns, and one look sees it alive. So the host looks twice, with a pause between, and a
|
||||
unit that was running and is not any more fails its resource by name. A unit still coming up reads
|
||||
as running at both looks and is accepted; a service asked to stop is not waited on.
|
||||
|
||||
No command for this reaches a machine. A module declaring *how to test my configuration* was weighed
|
||||
and refused: the link carries no actions, and a verification command is one. The host is checking
|
||||
the state it was told to establish, which is what it is for. *How it is checked:* ADR 0184's table.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-controller internal/catalogue]
|
||||
updated: 2026-09-30
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
|
||||
- 02-DECISIONS/0155-a-definition-names-no-installation-and-how-that-is-checked.md
|
||||
@@ -14,6 +14,7 @@ decisions:
|
||||
- 02-DECISIONS/0084-which-provider-serves-a-consumer.md
|
||||
- 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md
|
||||
- 02-DECISIONS/0038-the-mesh-assigns-the-port.md
|
||||
- 02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md
|
||||
---
|
||||
|
||||
# 27 — A module requires, the mesh resolves
|
||||
@@ -206,6 +207,22 @@ name when nothing sets it. That is the contract half of this design's operator p
|
||||
the placeholder allows: the definition says which values reach which requirement, and nothing else
|
||||
does. *How it is checked:* the unit tests named in issue 173, and the plan comparison that closed it.
|
||||
|
||||
*A provider says once what it derives for each consumer (2026-10-02,
|
||||
[ADR 0188](../../02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md),
|
||||
[issue 124](../../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)):*
|
||||
where a provider **names the resource** it gives each consumer — a bucket, a database, a vhost — the
|
||||
name is derived per consumer, and a literal `serves` block could not carry it. A served value may
|
||||
now name the consumer the mesh is serving: `${consumer:as}`, the identity the mesh minted, and
|
||||
`${consumer:as:dns}`, that same identity written as a DNS label. Nothing else — **the mesh learns no
|
||||
protocol here; it spells its own name in an alphabet it already knows.** Settings are laid on first,
|
||||
so an operator may still set a prefix and the mesh derives the rest. The mesh fills it at the one
|
||||
moment it knows who the consumer is, and the one filled value reaches both ends: the consumer, as
|
||||
its binding's served facts and as `${bound:<provision>:<key>}` in any file it writes; the provider,
|
||||
as `derived` on that consumer's entry in its contributions file, so its provisioner is told the name
|
||||
rather than recomputing it. A consumer that writes the derived value into its own definition instead
|
||||
of asking for it is refused, naming the placeholder to use. *How it is checked:* the unit tests in
|
||||
ADR 0188's "how this is checked", each run against the unchanged controller first.
|
||||
|
||||
## How a definition reads what was resolved
|
||||
|
||||
**One form, naming a requirement and a field of its contract.** A definition that needs the database's
|
||||
@@ -214,7 +231,9 @@ name in a configuration file writes the same thing: the requirement's name and t
|
||||
controller fills it at resolution.
|
||||
|
||||
This one form replaces the placeholders that exist today, one per mechanism: bound values, secrets,
|
||||
ports and machine facts.
|
||||
ports and machine facts. It subsumes the consumer placeholder too — a value a provider derives is
|
||||
read by the consumer exactly as any other field of the contract is, and `${consumer:…}` is only
|
||||
how the *provider* states the rule.
|
||||
|
||||
**The seat placeholder stays, for the controller alone.** The controller composes its own
|
||||
declaration and reaches the store and broker it made before any module existed, so it cannot be
|
||||
|
||||
@@ -1,10 +1,15 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code: []
|
||||
updated: 2026-09-27
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller: internal/catalogue/jails_into.go, internal/catalogue/manifest.go (Jail, Jailing)
|
||||
- mesh-catalog: modules/fail2ban (jailing, the base and the seat's verbs), modules/mailu, modules/route-proxy, modules/gitea (jails)
|
||||
- mesh-host: internal/declaration/declaration.go (a container's logging)
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
|
||||
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
- 02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md
|
||||
---
|
||||
|
||||
# 31 — A module declares its fail2ban jail, and the mesh composes them per node
|
||||
@@ -63,3 +68,38 @@ jail, composed from the postgres module's manifest, without anyone editing a nod
|
||||
beside)
|
||||
- mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules
|
||||
(`postgres`, `mssql`, `mailu`) that will declare jails
|
||||
|
||||
## Decided and built, 2026-10-02
|
||||
|
||||
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
made this the rule and built it. A module declares `jails` — each a name, the `failregex` of a
|
||||
failed attempt in its log, and the stanza's own keys — and the fail2ban module declares `jailing`:
|
||||
the one file the stanzas compose into and the directory each filter lands in. The controller gathers
|
||||
every assigned module's jails per node into those; the holder's daemon restarts on the composed file.
|
||||
|
||||
What made it workable was the log. A container's output went to a file of the runtime's own, under
|
||||
a path that changes when the container is recreated, so no jail could read a container's service
|
||||
however it logged. A container now declares `logging: journald`, the host runs it with the journal as
|
||||
its driver, and a jail reads it with `backend = systemd` and a `journalmatch` on the container's
|
||||
name — the same way the base's ssh jail has always read the ssh daemon. The first three doors: the
|
||||
mail front end (every login failure on its proxying ports), the forge (a failed authentication
|
||||
attempt) and the public proxy (a certificate or request for a name the mesh does not serve, which
|
||||
the proxy now says in its log). The base is strict — three in a day for a day; twice banned in two
|
||||
weeks for four — and the mesh's own range stays never banned.
|
||||
|
||||
The seat the module holds serves `status`, `banned`, `ban` and `unban`, from a runtime that carries
|
||||
only the fail2ban client with the daemon's socket shared in; the jails are composed, the ban list is
|
||||
the daemon's, and both are read through the console.
|
||||
|
||||
*How it is checked:* ADR 0179's table.
|
||||
|
||||
## What the first jails taught, 2026-10-02
|
||||
|
||||
[ADR 0186](../../02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md). Within an hour of the
|
||||
first public jail the home server had banned the house's own router: the router reflects local
|
||||
traffic, so every client in the building arrives as the gateway's address. The never-ban list now
|
||||
holds every private range as well as the mesh's own. And the mesh read its own ban chain as a
|
||||
foreign rule set on that machine, because the chain hangs off the container runtime's user chain and
|
||||
that machine's forward policy is the runtime's DROP — the reader now calls a chain of source-named
|
||||
refusals a ban wherever it hangs, as it already did for the packet filter's own tables.
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ code:
|
||||
- mesh-host internal/apply/apply.go
|
||||
- mesh-tools src/main.ts
|
||||
- mesh-catalog modules/mesh-catalog
|
||||
updated: 2026-09-28
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
|
||||
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
|
||||
@@ -24,6 +24,7 @@ decisions:
|
||||
- 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
|
||||
- 02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md
|
||||
- 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
|
||||
- 02-DECISIONS/0187-a-dead-tracker-is-not-the-machines-failure.md
|
||||
- 02-DECISIONS/0134-the-mesh-says-what-it-applied.md
|
||||
---
|
||||
|
||||
@@ -470,6 +471,21 @@ moment the mesh can mint for itself: **the bus's own accounts** (§the bootstrap
|
||||
needs an account before it can run) and **the vault's own credential**. Any third exception is a
|
||||
design failure, and naming these two is what makes a third one visible.
|
||||
|
||||
## What a step is answerable for, 2026-10-02
|
||||
|
||||
[ADR 0187](../../02-DECISIONS/0187-a-dead-tracker-is-not-the-machines-failure.md). A step gates its
|
||||
module and not the machine ([ADR 0136](../../02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md)),
|
||||
but a step that exits non-zero still leaves the machine reporting that it is not doing what it was
|
||||
told — and a clean apply gates other things entirely, the found firewall's retirement among them. So
|
||||
what a step calls a failure matters beyond the step.
|
||||
|
||||
The rule the media step now follows, and the one to copy: a step fails for what the mesh chose and
|
||||
can fix, and reports what it merely found and cannot. An indexer entry the mesh re-pointed at this
|
||||
mesh's proxy is plumbing the mesh is answerable for; whether the public tracker behind it answers
|
||||
today is not. An entry the operator listed is the operator's to insist on, and still fails. Six
|
||||
hours of a machine reading as broken, for one tracker that had died, is what the distinction costs
|
||||
when it is missing.
|
||||
|
||||
## 11. Open
|
||||
|
||||
**Semantic change has no mechanical defence** (§8). Recorded as open rather than solved, because
|
||||
|
||||
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-tools]
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
|
||||
- 02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md
|
||||
- 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
|
||||
@@ -184,6 +185,18 @@ container to declare a capability. Removing a predecessor's rule set is an opera
|
||||
through the seat, recorded on the bus, instead of a shell on the machine. *How it is checked:* ADR
|
||||
0169's table.
|
||||
|
||||
## The intrusion seat's verbs, 2026-10-02
|
||||
|
||||
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md).
|
||||
The second node-scoped seat to carry verbs: `node-intrusion-prevention` serves `status` (every jail
|
||||
with what it watches and holds), `banned` (every address held now, with its jail and when the ban
|
||||
ends), `ban` and `unban` (an operator's act on the live ban list). The fail2ban module serves them
|
||||
from a runtime that carries only the daemon's client, the socket shared in from the machine — no
|
||||
capability, no machine network, since the daemon on the machine does the banning. That runtime is the
|
||||
per-module container [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
retires; the verbs and the client are the same code once the node's own runtime loads them as a bundle.
|
||||
The module's own tool beside them reads one jail's effective settings. *How it is checked:* ADR 0179's table.
|
||||
|
||||
## What this does not settle
|
||||
|
||||
- Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a
|
||||
|
||||
@@ -1,226 +0,0 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0183-the-anthropic-licence-manager-is-a-module-and-hands-tokens-to-the-agent-over-the-bus.md
|
||||
- 02-DECISIONS/0181-the-operator-account-is-a-node-fact-and-a-home-is-a-placement-root.md
|
||||
- 02-DECISIONS/0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md
|
||||
- 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
|
||||
- 02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md
|
||||
- 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# 40. Building the operator's agent and its licence manager
|
||||
|
||||
**The work of [design 36](36-the-operators-agent-on-a-machine.md) and [design 39](39-the-anthropic-licence-manager.md),
|
||||
broken into packages that each end at something a person can see run, in the order their
|
||||
dependencies allow.** The two designs are the authority on *what* is built; this document holds the
|
||||
packages, their order, their sizes and their proofs, and is wrong the moment it disagrees with them.
|
||||
It is the shape [design 38](38-building-the-operators-machine.md) gave the operator's machine, applied
|
||||
to the two modules that make its agent work.
|
||||
|
||||
## How this is built, and where it is run
|
||||
|
||||
**On the live mesh, by the operator's decision** ([ADR 0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md),
|
||||
and design 38's words on the same day). Every package is written with unit tests, committed on one
|
||||
branch per repository ([playbook 07](../../00-META/process/07-feature-branches.md)), and proven on the
|
||||
machines: the control node first for the manager, one workstation first for the agent, then the rest.
|
||||
The cost accepted: a broken agent module leaves a workstation's agent without the mesh's instructions
|
||||
or with a stale token until the next push; the person's own files under the home are never in reach of
|
||||
the failure, by [ADR 0182](../../02-DECISIONS/0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md).
|
||||
|
||||
Each package names what proves it. A package that cannot name its proof is divided until it can.
|
||||
|
||||
## What exists already, measured
|
||||
|
||||
Counted 2026-10-02 in the repositories and on the machines. Nothing here is new ground; every package
|
||||
reshapes something standing.
|
||||
|
||||
| Piece | Today | Becomes |
|
||||
|---|---|---|
|
||||
| the vendor's refresh, the sealed box, the grant file | `anthropic-manager` in the catalogue: a token-endpoint client, a sealed-box primitive, adoption of a grant sealed to a node's key; assigned to nothing | the manager's refresh and adoption, with the lease, the floor and the cadence the predecessor's manager had |
|
||||
| the credentials write, the refresh-token strip, the identity read | `anthropic-consumer` in the catalogue: tested; assigned to nothing | the agent module's write, unchanged in shape |
|
||||
| the predecessor's manager and consumer | two modules in the retired system: the lease per licence, the expiry floor, the lineage comparison, the identity guard, three touchpoints with fallbacks, cooldowns on alarms | ported as logic with its tests; nothing of its registry or its bus |
|
||||
| the host's `process` and `archive` shapes | fetch a bundle by digest and run it supervised; fetch and unpack an artifact | **unchanged** — the manager's daemon is one process; both modules' tools are bundles |
|
||||
| the manifest's `uses`, `claims`, `invokes seat:<seat>.<verb>`, node-scoped `provides` with `serves: {port}` | all four exist and are used by other modules | **unchanged** — the agent uses the seat and invokes its verbs; the console provides its endpoint |
|
||||
| the console | a container per node, MCP on loopback, no provision | gains one provision; becomes the node-tools runtime's serving mode under design 38's WP3 |
|
||||
| the agent's package | present on both workstations from a build the predecessor's helper made; the distribution's repositories do not carry it | declared; satisfied where present, refused where not, until a package repository seat exists |
|
||||
| the operator account | a column on every node record, **empty on all four** | stated by the operator, before anything home-scoped lands |
|
||||
|
||||
**One dependency decides the order.** Both modules serve tools and the agent module's tools write
|
||||
under `/etc` and, as the operator, under the home. Under [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
tools run in the node's tool runtime, host-side, which design 38 builds in its WP1–WP3. Writing a
|
||||
per-module tool container for these two modules would be building the pattern that record retires, so
|
||||
**the live proofs of WP3 to WP5 below wait for design 38's WP3.** Everything before a live proof —
|
||||
manifests, code, tests — does not, and is written now.
|
||||
|
||||
## The order the work allows
|
||||
|
||||
```
|
||||
WP0 the operator states the facts (the live mesh) ── accounts, roles, licences to adopt
|
||||
WP1 the console provides its endpoint (mesh-catalog) ── small, independent
|
||||
WP2 the licence manager, built and tested (mesh-catalog) ──┐ independent of each other;
|
||||
WP3 the agent module, built and tested (mesh-catalog) ──┘ both wait on design 38 WP3 to run
|
||||
│
|
||||
WP4 the manager live on the control node (the live mesh) ── three licences adopted, a refresh seen
|
||||
WP5 the agent live on one workstation (the live mesh) ── the hand-over, the switch, the instructions
|
||||
WP6 the rest of the nodes, and the predecessor's remains ── adoption from a login, the retirements
|
||||
```
|
||||
|
||||
WP1, WP2 and WP3 touch different directories of one repository and meet only at the seat's name and
|
||||
the provision's name; they are built in parallel. WP4 is the first time anything on a machine changes.
|
||||
WP5 is the proof of the whole.
|
||||
|
||||
## WP0 — The operator states the facts
|
||||
|
||||
*The live mesh. An hour, and it is the operator's.*
|
||||
|
||||
The account on each node record, through the controller's node command — none is stated today, and
|
||||
[ADR 0181](../../02-DECISIONS/0181-the-operator-account-is-a-node-fact-and-a-home-is-a-placement-root.md)
|
||||
refuses a home-scoped module without one. The role of each node, as the agent module's setting on the
|
||||
node layer, once the module exists. Which three licences exist and what each is called.
|
||||
|
||||
**Proof.** The controller's node command lists an account for every node.
|
||||
|
||||
## WP1 — The console provides its endpoint
|
||||
|
||||
*mesh-catalog. Half a day.*
|
||||
|
||||
**What changes.** The console's manifest gains a node-scoped provision — working name
|
||||
`console-endpoint`, fixed when the manifest is written — serving the port the machine gave it, as the
|
||||
local model server already does for its API. Co-location resolves it
|
||||
([ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md),
|
||||
[to-be 34](34-the-console.md) as amended). When design 38's WP3 moves the console into the node-tools
|
||||
module, the provision moves with it; it is a line in a manifest either way.
|
||||
|
||||
**Proof.** The controller's plan for a workstation shows a consumer of the provision bound to the
|
||||
console's port; the same consumer on a machine without the console is refused naming the provision;
|
||||
the catalogue's tests pass.
|
||||
|
||||
## WP2 — The licence manager, built and tested
|
||||
|
||||
*mesh-catalog. Two to three days; the largest package.*
|
||||
|
||||
**What is written**, as design 39 says:
|
||||
|
||||
1. **The manifest.** Claims the mesh-scoped seat `anthropic-licence-manager` with its verbs; requires a
|
||||
database, a `secret` for the key its grants are encrypted with, and the bus; a `bundle` of tools; a
|
||||
`process` for the daemon that refreshes, collects usage and notifies, on a schedule; declared
|
||||
settings for the cadence, the usage threshold and the cooldown, each with a default; `invokes` the
|
||||
agent module's `apply`.
|
||||
2. **The store.** Migrations for licences, bindings, usage and audit, with the lease and the
|
||||
notification slot as columns, numbered and idempotent.
|
||||
3. **The refresh.** The token-endpoint client and the sealed box from `anthropic-manager`; the plan
|
||||
(floor, cadence, forced, cannot) and the lease from the predecessor, as pure functions with their
|
||||
tests; the vendor's reason logged on failure; counted failures, one notification per cooldown.
|
||||
4. **Adoption.** From a file on the manager's node for the API key; from a sealed grant a node offers;
|
||||
the identity guard that refuses a mismatch and notifies.
|
||||
5. **Usage.** The vendor's reading per licence on a schedule, stored raw and normalised
|
||||
([ADR 0054](../../02-DECISIONS/0054-model-usage-is-recorded-at-two-grains.md)), one notification
|
||||
per threshold crossing.
|
||||
6. **The verbs**: `licences`, `bindings`, `bind`, `switch`, `release`, `refresh`, `usage`, `adopt`,
|
||||
`register`, `current` — the last answering a consumer's token sealed to the key that consumer
|
||||
registered.
|
||||
7. **The hand-over**: on rotation or switch, one call to `claude-code.apply@<node>` per bound node,
|
||||
the token sealed to that node's key, the answer recorded.
|
||||
|
||||
**Proof, before anything runs live.** Unit tests: two refresh runs started together rotate one grant
|
||||
once; a grant with a mismatching identity is refused; a worker bound to a dead licence is refused and
|
||||
never lent another; every event the daemon emits is free of a token; the hand-over payload opens only
|
||||
with the registered key. The catalogue's checks: no installation named, no secret in a declared file.
|
||||
|
||||
## WP3 — The agent module, built and tested
|
||||
|
||||
*mesh-catalog. Two days.*
|
||||
|
||||
**What is written**, as design 36 says:
|
||||
|
||||
1. **The manifest.** The agent's package; the state directory; the facts file carrying the node's
|
||||
name, the operator account and its home, the console's bound port, the role and the extra tool
|
||||
servers from settings; requires the console's endpoint and the bus; `uses` the seat and `invokes`
|
||||
its `register`, `current` and `adopt`; a `bundle` of tools. **No file resource under a home or
|
||||
under `/etc`.**
|
||||
2. **The renderer.** From the facts file and the current binding, the managed settings file (the tool
|
||||
servers under the entry `mesh`, the attribution trailers, and the key-helper for an API-key binding)
|
||||
and the managed instruction file (§3 of design 36), written under the agent's managed directory
|
||||
with the escalation the tool performs for itself; idempotent; re-run when the facts file changes.
|
||||
3. **The keypair**, made once in the state directory, the public half registered with the seat at
|
||||
start and at every start.
|
||||
4. **The consumer side**: `apply` (a rotation applied only if newer within one lineage, a switch applied
|
||||
regardless, the answer naming the outcome and never a token); the pull at start and near expiry;
|
||||
the credentials write as the operator, access-token-only, from `anthropic-consumer` with its tests;
|
||||
the key-helper program for the API key; the offer of a login to the seat, sealed, after reading the
|
||||
account's identity.
|
||||
5. **The tools**: `apply`, `licence_status`, `mcp_configure` (validates a server, sets the module's
|
||||
setting through the controller's settings verb), `render` (re-render now, for a person).
|
||||
6. **The documentation**: the six predecessor files and the hand-made console entry a person removes
|
||||
on a workstation that carried the predecessor.
|
||||
|
||||
**Proof, before anything runs live.** Unit tests: the renderer writes only the mesh's keys and leaves
|
||||
every other key of a seeded settings file; the credentials write strips a refresh token and is atomic;
|
||||
the lineage comparison from the predecessor, with its cases; a login offer carries the identity it read.
|
||||
The catalogue's checks pass.
|
||||
|
||||
## WP4 — The manager live on the control node
|
||||
|
||||
*The live mesh. Half a day, after design 38's WP3.*
|
||||
|
||||
**Order.** Assign the manager on the control node; push. Adopt the API key from a file there. Adopt
|
||||
the two subscription grants: a login in a throwaway home on the control node, offered to the seat the
|
||||
way a node's agent module will. Bind each node's agent to a licence.
|
||||
|
||||
**Proof.** Through the console: `anthropic-licence-manager.licences` lists three licences with
|
||||
identity and expiry; within the cadence, the audit shows a rotation and `licences` shows a later
|
||||
expiry; a forced `refresh` on one licence is logged with the vendor's answer; the two retired catalogue
|
||||
modules are still assigned to nothing.
|
||||
|
||||
## WP5 — The agent live on one workstation
|
||||
|
||||
*The live mesh. Half a day. The proof of the whole.*
|
||||
|
||||
**Order.** Set the workstation's role in the module's settings. Record the checksums of everything
|
||||
under the person's agent directory. Assign the module; push. Remove the six predecessor files and the
|
||||
hand-made console entry. Start a new session.
|
||||
|
||||
**Proof.** The managed directory holds the settings and instruction files, owned by root. Everything
|
||||
under the person's agent directory is byte-identical to before except the credentials file, which is
|
||||
owned by the operator, readable by nobody else, and names no refresh token. The new session lists the
|
||||
mesh's tools under `mesh` once, answers *which node am I* from the instruction file, and makes a model
|
||||
request. `anthropic-licence-manager.switch` to the second subscription licence changes the token on
|
||||
the workstation within a minute, and neither the verb's answer nor either module's log holds a token.
|
||||
Switched to the API-key licence, the credentials file is left as it was and the agent authenticates
|
||||
through the key-helper. Switched back.
|
||||
|
||||
## WP6 — The rest of the nodes, and the predecessor's remains
|
||||
|
||||
*The live mesh and mesh-catalog. One day.*
|
||||
|
||||
**Order.** Assign the module on the second workstation and on the servers whose account is stated;
|
||||
remove the predecessor's files on the second workstation. Log in on a workstation under a licence's
|
||||
account and watch the offer be adopted — and under the wrong account, and watch it refused and
|
||||
notified. Retire `anthropic-manager` and `anthropic-consumer` from the catalogue. Set designs 36 and
|
||||
39 to `implemented` for what runs, with the as-is written
|
||||
([playbook 02](../../00-META/process/02-graduation.md)).
|
||||
|
||||
**Proof.** Every node with an account runs the module and `licence_status` answers on each; the
|
||||
refused login's notification arrived; the catalogue has no module built on the old placement.
|
||||
|
||||
## What is deliberately not here
|
||||
|
||||
- **The package repository seat** for a distribution that does not carry the agent's package
|
||||
(design 36 §7). A fresh node refuses the module in the package manager's words until it exists.
|
||||
- **An automated switch on exhaustion.** The readings are kept from WP4; the policy is a later record.
|
||||
- **Workers and the mesh's own sessions as consumers.** The manager's bindings and fallbacks know them
|
||||
from WP2; the consumers themselves do not exist yet ([to-be 15](15-the-agent-session.md),
|
||||
[ADR 0003](../../02-DECISIONS/0003-agents-are-persistent-employees.md)).
|
||||
- **Whether a refresh token is single-use.** WP4 may measure it on a licence deliberately refreshed
|
||||
twice; the design holds either way.
|
||||
|
||||
## How this list is kept true
|
||||
|
||||
Each package's proof is run when the package is finished and its line here gains the date and the
|
||||
commit, as design 38 does. A package whose proof fails is not reworded; the failure is recorded under
|
||||
it and the package stays open. When WP5 is proven, designs 36 and 39 move to `in-progress` with their
|
||||
owning repository, and when WP6 is proven to `implemented`, with the as-is written.
|
||||
@@ -1,9 +1,9 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
opened: 2026-09-26
|
||||
located-in: [mesh-controller internal/catalogue/declaration.go, mesh-sdk src/provisioner, mesh-catalog modules/minio]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
located-in: [mesh-controller internal/catalogue, mesh-sdk src/provisioner, mesh-catalog modules/minio]
|
||||
fixed-by: 02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md
|
||||
amended-design: 03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md
|
||||
---
|
||||
|
||||
# 124 — A consumer cannot be told a value its provider derived for it, so it transcribes one
|
||||
@@ -63,3 +63,27 @@ compares it to what the provider will actually create. The one wrong instance wa
|
||||
- What would have caught the wrong instance? A test that resolves a consumer's grant and compares the
|
||||
bucket in its own configuration against the one the provider would create is a check that could
|
||||
exist today, for any interface, without the mechanism above.
|
||||
|
||||
## Answered, 2026-10-02 — [ADR 0188](../../02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md)
|
||||
|
||||
The channel is the provider's own `serves` block, which may now name the consumer the mesh is
|
||||
serving: `${consumer:as}` and `${consumer:as:dns}`. The mesh fills it once, where it knows who the
|
||||
consumer is, and delivers the one filled value to both ends — the consumer's binding and its
|
||||
`${bound:…}` substitutions, and the provider's contributions entry, so a provisioner is told the
|
||||
name rather than deriving it. Each open question above, answered:
|
||||
|
||||
- **Should a provider return values from provisioning?** No. It would make a grant carry data the
|
||||
provider wrote, make a consumer's declaration wait on its provider's reconcile loop, and put the
|
||||
rule where nothing can refuse it. The reasoning is in the record.
|
||||
- **Or should `serves` say a value is derived?** Yes, and the mesh performs the derivation — but it
|
||||
learns no protocol doing it. The only fact is the identity the mesh itself minted, in one of two
|
||||
alphabets it already knows.
|
||||
- **Should a consumer that names the resource be refused?** Yes. A consumer's file that already
|
||||
contains the value the mesh is about to derive for it is refused at resolution, naming the
|
||||
placeholder to write instead. That is the check this report asked for, and it is exact rather than
|
||||
heuristic: a derived value carries the identity minted for this consumer on this machine, which
|
||||
nothing else would spell out.
|
||||
|
||||
minio's `bucketFor` is gone; its manifest serves `"bucket": "${consumer:as:dns}"`. The three
|
||||
consumers' hand-written bucket names are gone with it — each of them also named the machine the
|
||||
module happens to run on, which is the second thing wrong with a transcription.
|
||||
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-02
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by: 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 201 — A push recreated the controller at a digest older than the seat row its successor had written
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the
|
||||
controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan
|
||||
built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The
|
||||
host's plan then pushed the control node with the controller digest it had recorded when it was made —
|
||||
the previous build — and recreated the controller container on it. The older binary read the row, found
|
||||
a verb it could not run, and refused to start:
|
||||
|
||||
```
|
||||
mesh-controller: the mesh-controller seat's row declares "command", which this control plane
|
||||
cannot run: "command" is not a verb the mesh-controller seat serves
|
||||
```
|
||||
|
||||
A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since
|
||||
the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the
|
||||
newer image, outside the service, to push the control node again; the push sent the newer digest and
|
||||
the controller came up.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md));
|
||||
a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of
|
||||
2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that
|
||||
wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md));
|
||||
for every other module an older digest is a brief regression a later push corrects. For the controller
|
||||
it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct
|
||||
it is the thing that is down. Two plans that overlap will happen again whenever two people merge within
|
||||
a minute.
|
||||
|
||||
## What a fix would have to do
|
||||
|
||||
Either of two, and the first is the smaller:
|
||||
|
||||
- A push never sends a controller digest older than the one the running controller is — the controller
|
||||
knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words.
|
||||
- Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves
|
||||
what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check
|
||||
exists to refuse.
|
||||
|
||||
Until one is built: do not merge a controller change while another plan is rolling, and after merging
|
||||
one, wait for `node show` on the control node to report the new controller before merging anything else.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
||||
- mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`
|
||||
|
||||
## Half of it is closed, 2026-10-02
|
||||
|
||||
[ADR 0185](../../02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) takes
|
||||
the outage out of it: a control plane behind its seat's row now serves every verb it can run, says
|
||||
which it cannot, and answers the reason when one of those is called. The same race today would cost
|
||||
the verbs the newer build added, for as long as the older binary is in place, and the ordinary
|
||||
"this machine is behind" machinery would put the newer one back without a hand.
|
||||
|
||||
**The race itself is still open**, and this report stays open for it. What was established while
|
||||
closing the other half, so the next reader does not redo it:
|
||||
|
||||
- Composing and sending are serialised per machine by a session advisory lock in the store, so two
|
||||
control planes cannot compose one machine's declaration at the same time. The stale content did
|
||||
not come from two concurrent composes.
|
||||
- A container's image is resolved into the module's manifest when it is *built*, and a push composes
|
||||
from the catalogue as it is at that moment, under the hold. So a compose that ran after the build
|
||||
was taken in could not have named the older image.
|
||||
- The declaration's sequence orders arrival and nothing else (the numbering of
|
||||
[issue 107](../107-a-declaration-carries-no-order/00-report.md)); it cannot tell a later send
|
||||
carrying earlier content from a later send carrying later content. The host refuses a declaration
|
||||
numbered below the last it applied, and both of these were above it.
|
||||
- The machine's own journal shows the two applies ten seconds apart and which replaced what; it does
|
||||
not record which image each declaration named, which is the one fact that would settle it. A host
|
||||
that recorded the digest it was told, per apply, would have answered this in a minute.
|
||||
|
||||
So the trigger is not yet pinned, and guessing at the push path is the most expensive place in the
|
||||
mesh to guess. The fix the report first suggested — a push never sending a control plane a digest
|
||||
older than the one that machine reports running — closes the class without needing the trigger, and
|
||||
is now a correctness nicety rather than the difference between a working mesh and a dead one.
|
||||
Reference in New Issue
Block a user