Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d57289e049 | ||
|
|
d4a2f99ab5 | ||
|
|
a9f91fdd0c | ||
|
|
131a5e4714 | ||
|
|
329a24fdae | ||
|
|
7f72f3b79a | ||
|
|
114a71f36f | ||
|
|
0f407417f3 | ||
|
|
d0d5799884 | ||
|
|
9ba4de5557 |
+132
@@ -0,0 +1,132 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
---
|
||||
|
||||
# 179. The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail
|
||||
|
||||
## Context
|
||||
|
||||
Read on the control node on 2026-10-02, the day the machines were confirmed filtered by the mesh
|
||||
alone ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)): the intrusion
|
||||
prevention watched one door. Its two jails read the ssh daemon's journal and its own log, banned
|
||||
five failures in ten minutes for ten minutes, and in a day had seen twelve thousand failed logins
|
||||
from three hundred addresses and banned none of the busiest, which paced themselves at one try every
|
||||
ten minutes. The mail submission port took a hundred and sixty password guesses in the same day from
|
||||
thirty-eight addresses with no jail reading it at all; the forge and the public proxy had no jail
|
||||
either, and the proxy logged nothing a jail could read. Nobody could see the jails without a shell:
|
||||
the module's three tools existed in code and were served by nothing, and the seat it holds declared
|
||||
no verbs.
|
||||
|
||||
Three things were missing and they are three shapes the mesh already has. The packet filter's seat
|
||||
serves verbs every holder owes ([ADR 0170](0170-the-firewall-seat-serves-its-verbs.md)); the
|
||||
intrusion seat serves none. A module's `listens` compose into the machine's filter, and [to-be 31](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
|
||||
says a module's `jails` compose into the machine's intrusion prevention the same way — the controller
|
||||
composes them, and no module declares one. And a jail reads a log; a container's output goes to a
|
||||
file of the runtime's own under a path that changes when the container is recreated, which is why
|
||||
no jail could read the mail front end, the forge or the proxy, however they logged.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. The `node-intrusion-prevention` seat serves four verbs**, and a module that claims it serves
|
||||
all four or is refused the claim, as with every seat:
|
||||
|
||||
- `status` — every jail with what it watches, how many addresses it is counting failures against and
|
||||
holding now, and the totals since it started; one jail's detail when named. Read-only.
|
||||
- `banned` — every address banned now, with the jail holding it, when it was banned and when the ban
|
||||
ends. Read-only.
|
||||
- `ban` — ban one address in one jail now, for that jail's ban time. An operator's act on the live
|
||||
ban list, which the mesh composes the rules for and never writes itself.
|
||||
- `unban` — let one address go, from one jail or from every jail.
|
||||
|
||||
A holder may serve its own tools beside these; the fail2ban module reads one jail's effective
|
||||
settings as its own.
|
||||
|
||||
**2. A container may log to the journal.** `logging: journald` on a container has the host run it
|
||||
with the journal as its log driver; the journal keeps the container's name on every line, and
|
||||
`docker logs` keeps working. Where a container logs is part of its spec, so moving it recreates the
|
||||
container, and the only place besides the runtime's own file is the journal: a machine's intrusion
|
||||
prevention reads the journal already, for the ssh daemon, and a container that logs there is read
|
||||
the same way, by the container's name, whatever the container is called by the runtime this time.
|
||||
|
||||
**3. A module with a door declares its jail, and the holder composes them.** What to-be 31 designed
|
||||
is now the rule: a module whose service authenticates from outside — the mail front end, the forge,
|
||||
the public proxy — declares in its manifest what a failed attempt looks like in its log and how to
|
||||
ban on it, naming no node and no path; the module that holds the intrusion seat declares where the
|
||||
composed jails and filters land, and the mesh writes them on every machine that runs both. A machine
|
||||
not running the module has no such jail. The holder restarts its daemon on the composed file.
|
||||
|
||||
**4. The base is strict, and the mesh's own range is never banned.** Three failures in a day ban for
|
||||
a day, on every jail unless the jail says otherwise; banned twice in two weeks, by any jail, is
|
||||
banned for four. The attackers this mesh sees pace themselves under any ten-minute window; a day's
|
||||
window counts them. A person who mistypes three times from one address is out for a day from that
|
||||
address, and never from a machine of the mesh, whose range stays in the never-banned list the module
|
||||
has carried since [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md). The operator
|
||||
chose this knowing it.
|
||||
|
||||
**5. The proxy says a refused name in its log.** A request for a name this mesh does not serve, from
|
||||
outside, is what a scanner does; the proxy already logged a certificate refused for such a name, and
|
||||
now logs the plain request too, with the asking address last, as its own jail's filter expects it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The seat's row gains four verbs; a mesh that already runs widens its row at the next controller
|
||||
start. The fail2ban module claims them and gains a runtime — a tool server whose image carries the
|
||||
fail2ban client, with the daemon's socket shared in from the machine, and nothing else of the
|
||||
machine. The daemon stays the machine's; what runs in the container is only the client.
|
||||
- **That runtime is the shape the catalogue has today, and it is on its way out.**
|
||||
[ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md),
|
||||
accepted the same day as this record, replaces a tool container per module with one tool runtime
|
||||
per node on the host side, taking each module's tools as a bundle. Nothing here depends on the
|
||||
container: the verbs, the client that speaks to the daemon over its socket, and the jails are the
|
||||
same code under either. This module converts with the packet filter's, whose runtime that record
|
||||
names, and the socket it needs becomes the node runtime's to reach rather than a mount of its own.
|
||||
- The host's container vocabulary grows by `logging`; an older host refuses a declaration that carries
|
||||
it, so the host rolls before the modules. Three containers are recreated once, when their modules
|
||||
are pushed with the field: the mail front end, the forge and the proxy — each a moment's outage.
|
||||
- The fail2ban module declares where jails compose (`jailing`) and the directory the filters go in;
|
||||
the mail, forge and proxy modules each declare one jail reading the journal by their container's
|
||||
name. The composed jail file is the one resource the daemon restarts on when a module arrives or
|
||||
leaves a machine.
|
||||
- The two base jails and the composed ones take the day's window; the ssh jail's ten minutes are
|
||||
gone. An address banned on the first day of this record stays banned for the day.
|
||||
- The module's three old tools, served by nothing, are replaced by the seat's four verbs and one
|
||||
own tool; `fail2ban_status` as a name is gone.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The seat declares the four verbs; a claim that serves fewer is refused by name | the catalogue's seat tests |
|
||||
| `status`, `banned`, `ban` and `unban` read and steer the daemon through its client, with the shapes fail2ban 1.1.0 printed live; a non-address and a non-name are refused before anything runs | the module's tests over a fake command runner |
|
||||
| A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests |
|
||||
| A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) |
|
||||
| The proxy logs a refused name with the address last | the proxy's tests |
|
||||
| A jail's pattern names `<HOST>` once per shape, since two is a duplicate capture group and costs the machine every ban | the catalogue's manifest tests |
|
||||
| Live | done 2026-10-02: `status` and `banned` answered on both servers through the console; the proxy's jail counted seven refusals on the home server; a documentation address banned in the ssh jail came back with its end time and was released |
|
||||
|
||||
## Built and proven live, 2026-10-02
|
||||
|
||||
All five rules are in the mesh. The host carries `logging`; the controller's seat row carries the four
|
||||
verbs and the proxy says a refused name in its log; the fail2ban module holds the seat from a runtime
|
||||
with the daemon's socket shared in, composes the jails, and the mail front end, the forge and the
|
||||
proxy each declare one. Through the console on the control node: `status` listed five jails with what
|
||||
each watches, `banned` listed the nine the long jail holds, and a documentation address banned in the
|
||||
ssh jail came back with its ban's end time and was released again. On the home server the proxy's jail
|
||||
had counted seven refusals within minutes of starting.
|
||||
|
||||
**One fault, found by the machine and not by a test.** The proxy's pattern matched two shapes of
|
||||
refusal in one expression and so named `<HOST>` twice. fail2ban expands that placeholder into a named
|
||||
capture group; two of them is a duplicate group name, and the daemon refuses *its whole configuration*
|
||||
and exits — both servers kept no bans at all for about ten minutes, every jail and not the one at
|
||||
fault. The pattern is now one per shape. A manifest check refuses the mistake at merge time, naming
|
||||
what it would cost, which is the only reason this record can claim the rule rather than the instance.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0170](0170-the-firewall-seat-serves-its-verbs.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
- [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md), [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md), [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md)
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 184. A service the mesh asked to run is still running a moment later
|
||||
|
||||
## Context
|
||||
|
||||
The host already refuses to take a service manager's word for it. Three places in one function read
|
||||
a unit back after acting on it, each with a comment saying why: *a service manager accepting a
|
||||
command says the transaction was accepted, not that the unit is running — one that starts and
|
||||
immediately dies satisfies it.* The intent was right and the implementation did not reach it.
|
||||
|
||||
On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the
|
||||
files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at
|
||||
that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public
|
||||
machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said
|
||||
so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing.
|
||||
|
||||
The read-back races the failure. A service manager returns when it has started the process; a daemon
|
||||
that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One
|
||||
look sees `activating` or `active` whatever the process is about to do, and *the host reports success
|
||||
for a machine that is already wrong* — the one shape of failure this host exists to refuse
|
||||
([ADR 0005](0005-the-node-host.md)).
|
||||
|
||||
A command the module declares — *test the configuration before restarting* — was considered and
|
||||
rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification
|
||||
command is a command: a declaration that carried one would be remote execution over the bus,
|
||||
arriving as root on every machine, which is a far larger door than the fault it closes. The host
|
||||
does not need one. It already knows what it asked for.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A unit the host has just asked to run is read twice**, with a pause between the reads long
|
||||
enough for a daemon that refuses its configuration to have exited. Not running at the second look is
|
||||
a failure of that resource, named with the unit and the state it is in — the same failure the single
|
||||
read was always meant to catch.
|
||||
|
||||
**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both
|
||||
looks and is accepted, exactly as before. What the second look catches is a unit that *was* running
|
||||
and is not any more. A service asked to be stopped is not waited on at all.
|
||||
|
||||
**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking
|
||||
the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and
|
||||
no command reaches a machine that did not already come from a built artifact.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is
|
||||
bounded by the number of services that changed in that apply, which is usually none.
|
||||
- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the
|
||||
resolver — now fails its apply when the composition is bad, instead of reporting success onto a
|
||||
dead daemon. `status` names the machine, which is how the operator finds out.
|
||||
- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s
|
||||
manifest check is what refuses the one that caused this, at merge time; this record is what makes
|
||||
the *next* one visible within a minute rather than invisible until something asks the daemon a
|
||||
question.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does |
|
||||
| A unit still starting is accepted at both looks | a host test |
|
||||
| A service asked to be stopped is not waited on | a host test |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
|
||||
---
|
||||
|
||||
# 185. A control plane behind its seat's row serves what it can
|
||||
|
||||
## Context
|
||||
|
||||
The mesh's own verbs are the controller seat's tools, and the seat's row is the store's
|
||||
([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). A control plane reads the
|
||||
row at start and installs a handler per verb; a verb the row carries that the binary cannot run was
|
||||
refused at start rather than at the first call, so that a disagreement between the row and the
|
||||
binary was said early. The refusal aborted the start.
|
||||
|
||||
On 2026-10-02 a merge added one verb. The new control plane started, widened the row, and ran. A
|
||||
push a few seconds later recreated its container at the previous image — a stale declaration from
|
||||
an overlapping wave, [issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) —
|
||||
and the older binary read a row naming a word it had never heard. It refused to start, and kept
|
||||
refusing. The mesh had no voice for ten minutes: no verb answered, no node could be pushed, no build
|
||||
was dispatched, and `status` said nothing because `status` is one of the verbs that had stopped
|
||||
being served. The way back was a person running the binary by hand outside its service, because the
|
||||
push that would have replaced it is itself a verb of the control plane that was down.
|
||||
|
||||
The check was right about the fact and wrong about the cost. A row ahead of a binary is the ordinary
|
||||
state of a roll-out: the row is widened by whichever control plane starts first, and a mesh with one
|
||||
control plane sees that gap on every merge that adds a verb. Making it fatal turned a transient into
|
||||
an outage with no path out that did not need a human.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A control plane serves the verbs it can run and does not refuse to start for the ones it
|
||||
cannot.** The row remains the authority on what the seat serves; this is only about what this binary
|
||||
does when it is behind the row.
|
||||
|
||||
**2. A verb it cannot run answers the reason.** Not silence and not a missing subject: a caller gets
|
||||
a sentence naming the verb, saying this control plane cannot run it and that it is a verb of a newer
|
||||
build. A verb that is simply absent from the row is still not served at all — that is the row
|
||||
deciding, which is unchanged.
|
||||
|
||||
**3. It says so once at start**, naming every verb of the row it cannot run, so the gap is visible
|
||||
in the log of the thing that has it rather than only at the moment somebody calls one.
|
||||
|
||||
**4. A mesh with no controller seat at all is still a refusal.** That is not a version gap, it is a
|
||||
mesh that has not been seeded, and nothing this control plane does would be meaningful.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An overlapping roll-out costs the verbs the newer build added, for as long as the older binary is
|
||||
in place. Everything else — every push, every build, every read — keeps working, and the ordinary
|
||||
machinery that notices a machine is behind is what puts the newer binary back.
|
||||
- The log gains one line on a control plane that is behind, and nothing on one that is not.
|
||||
- Issue 201's other half remains: the push that sent a stale declaration is a race worth closing on
|
||||
its own terms. This record makes that race survivable rather than fatal, which is the difference
|
||||
between a transient and an outage, and is deliberately the cheaper half.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A row carrying a verb this build cannot run still serves every verb it can, and names the one it cannot | a controller test over a widened row |
|
||||
| The unknown verb answers a sentence naming itself and saying this build is behind | the same test |
|
||||
| A mesh with no controller seat is refused | the existing start-up path |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
||||
- [Issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)
|
||||
- [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
---
|
||||
|
||||
# 186. A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) gave the
|
||||
public proxy a jail. Within the hour the home server's ban list held `192.168.1.1` — the house's own
|
||||
router. The router reflects local traffic, so every client in the building reaches that machine as
|
||||
the gateway's address; one local request for a name the mesh does not serve, three times in a day,
|
||||
and the whole house is refused by the machine it was asking. The jails inherited an `ignoreip` of
|
||||
the loopback and the mesh's own range, which was right when the only jail read the ssh daemon and
|
||||
the only clients were the mesh's; a jail on a public front door sees the neighbours too.
|
||||
|
||||
The same jail broke the other half of [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md).
|
||||
The home server began reading *NOT the mesh alone: 1 rule set the mesh did not write refuses traffic
|
||||
here*, and the rule set named was the mesh's own ban chain, written by the mesh's own intrusion
|
||||
prevention minutes earlier. The host's reader of the legacy filter required every path into a chain
|
||||
of refusals to come from a built-in chain whose policy accepts, before it would call that chain a
|
||||
ban. On that machine the chain hangs off the container runtime's user chain as well as the input
|
||||
chain, and the runtime had set the forward policy to DROP — so the mesh reported its own work as a
|
||||
foreigner's, on the one machine where the group's exit condition was supposed to hold.
|
||||
|
||||
Both faults are one mistake in two places: a rule written about the public internet, applied to
|
||||
everything that arrives.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A ban list never holds a neighbour.** The jails the mesh composes never ban a source on a
|
||||
private range — the mesh's own range, which was already named rather than written
|
||||
([ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md)), and every address space
|
||||
reserved for private use beside it, in both families. A machine behind a router that reflects local
|
||||
traffic sees its whole building as one address; a ban there is a self-inflicted outage, and the
|
||||
sources worth banning are not on those ranges in the first place.
|
||||
|
||||
**2. The mesh's own bans are its own wherever they hang.** A chain of refusals is a ban list when
|
||||
every refusal names the sources it refuses and the chain accepts nothing — the rule the host already
|
||||
applied to the packet filter's own tables, now applied to the legacy filter too, and nothing more.
|
||||
The policy of the chains that jump into it says nothing about what it is: that policy is already
|
||||
classified where it belongs, as the container runtime's, and requiring it here counted it twice.
|
||||
|
||||
**3. A chain that accepts anything is still not a ban.** That is what keeps a predecessor's
|
||||
allow-these-and-drop-the-rest chain classified as something an operator must look at, which is the
|
||||
distinction [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md) exists to draw.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The composed jails gain the private ranges in their never-ban list. An address already banned
|
||||
stays banned until it is released; the house's router was released by hand the moment it was found.
|
||||
- The home server reads *the mesh alone* again, which is group 7's exit condition and was false for
|
||||
about an hour.
|
||||
- A machine whose apply fails for an unrelated reason does not revisit its found firewall's record
|
||||
at all — the step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)).
|
||||
The home server's record therefore still reads *retired by the mesh* although the front end is
|
||||
uninstalled, and will correct itself once that machine's own stuck module is fixed. It is a stale
|
||||
record, not a wrong machine.
|
||||
- The record number the front end's removal was given moved under it: another session took 0175
|
||||
while that record was in review, and it is now
|
||||
[ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md). The citations
|
||||
the host and the control plane print were pointing at an unrelated record and are corrected here.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A private source is never banned | the module's jail configuration, read back by `fail2ban.fail2ban_settings` on a machine |
|
||||
| The mesh's own ban chain reads as a ban behind a dropping forward policy | a host test over the home server's own captured rule set |
|
||||
| A chain that accepts anything is not a ban | a host test |
|
||||
| Live | the home server reads *the mesh alone*; no ban held on either machine is a private address |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
|
||||
- [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md), [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
|
||||
@@ -182,7 +182,11 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)
|
||||
- **0170** — [The firewall seat serves its verbs, and a foreign rule set is removed through one of them](0170-the-firewall-seat-serves-its-verbs.md)
|
||||
- **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md)
|
||||
- **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
- **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
|
||||
- **0184** — [A service the mesh asked to run is still running a moment later](0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md)
|
||||
- **0185** — [A control plane behind its seat's row serves what it can](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)
|
||||
- **0186** — [A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang](0186-a-ban-list-never-holds-a-neighbour.md)
|
||||
|
||||
### Its tiers, from the bottom up
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ status: in-progress
|
||||
code: [mesh-host]
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md
|
||||
- 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md
|
||||
- 02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md
|
||||
- 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
|
||||
@@ -498,3 +499,17 @@ run and reported; the exit follows an in-flight apply rather than interrupting i
|
||||
not start is rolled back once and the second failure halts; a completed reconcile retires what is older
|
||||
than the predecessor and never the predecessor; and the newest of two delivered versions is the one
|
||||
that runs.
|
||||
|
||||
## A service is still running a moment later, 2026-10-02
|
||||
|
||||
[ADR 0184](../../02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md).
|
||||
The host has always read a unit back after acting on it, because a service manager accepting a
|
||||
command says the transaction was accepted and nothing about the process. The read raced the failure:
|
||||
a daemon that refuses the configuration the mesh just wrote exits a fraction of a second after the
|
||||
manager returns, and one look sees it alive. So the host looks twice, with a pause between, and a
|
||||
unit that was running and is not any more fails its resource by name. A unit still coming up reads
|
||||
as running at both looks and is accepted; a service asked to stop is not waited on.
|
||||
|
||||
No command for this reaches a machine. A module declaring *how to test my configuration* was weighed
|
||||
and refused: the link carries no actions, and a verification command is one. The host is checking
|
||||
the state it was told to establish, which is what it is for. *How it is checked:* ADR 0184's table.
|
||||
|
||||
@@ -1,10 +1,15 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code: []
|
||||
updated: 2026-09-27
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller: internal/catalogue/jails_into.go, internal/catalogue/manifest.go (Jail, Jailing)
|
||||
- mesh-catalog: modules/fail2ban (jailing, the base and the seat's verbs), modules/mailu, modules/route-proxy, modules/gitea (jails)
|
||||
- mesh-host: internal/declaration/declaration.go (a container's logging)
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
|
||||
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
- 02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md
|
||||
---
|
||||
|
||||
# 31 — A module declares its fail2ban jail, and the mesh composes them per node
|
||||
@@ -63,3 +68,38 @@ jail, composed from the postgres module's manifest, without anyone editing a nod
|
||||
beside)
|
||||
- mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules
|
||||
(`postgres`, `mssql`, `mailu`) that will declare jails
|
||||
|
||||
## Decided and built, 2026-10-02
|
||||
|
||||
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
made this the rule and built it. A module declares `jails` — each a name, the `failregex` of a
|
||||
failed attempt in its log, and the stanza's own keys — and the fail2ban module declares `jailing`:
|
||||
the one file the stanzas compose into and the directory each filter lands in. The controller gathers
|
||||
every assigned module's jails per node into those; the holder's daemon restarts on the composed file.
|
||||
|
||||
What made it workable was the log. A container's output went to a file of the runtime's own, under
|
||||
a path that changes when the container is recreated, so no jail could read a container's service
|
||||
however it logged. A container now declares `logging: journald`, the host runs it with the journal as
|
||||
its driver, and a jail reads it with `backend = systemd` and a `journalmatch` on the container's
|
||||
name — the same way the base's ssh jail has always read the ssh daemon. The first three doors: the
|
||||
mail front end (every login failure on its proxying ports), the forge (a failed authentication
|
||||
attempt) and the public proxy (a certificate or request for a name the mesh does not serve, which
|
||||
the proxy now says in its log). The base is strict — three in a day for a day; twice banned in two
|
||||
weeks for four — and the mesh's own range stays never banned.
|
||||
|
||||
The seat the module holds serves `status`, `banned`, `ban` and `unban`, from a runtime that carries
|
||||
only the fail2ban client with the daemon's socket shared in; the jails are composed, the ban list is
|
||||
the daemon's, and both are read through the console.
|
||||
|
||||
*How it is checked:* ADR 0179's table.
|
||||
|
||||
## What the first jails taught, 2026-10-02
|
||||
|
||||
[ADR 0186](../../02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md). Within an hour of the
|
||||
first public jail the home server had banned the house's own router: the router reflects local
|
||||
traffic, so every client in the building arrives as the gateway's address. The never-ban list now
|
||||
holds every private range as well as the mesh's own. And the mesh read its own ban chain as a
|
||||
foreign rule set on that machine, because the chain hangs off the container runtime's user chain and
|
||||
that machine's forward policy is the runtime's DROP — the reader now calls a chain of source-named
|
||||
refusals a ban wherever it hangs, as it already did for the packet filter's own tables.
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-tools]
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
|
||||
- 02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md
|
||||
- 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
|
||||
@@ -184,6 +185,18 @@ container to declare a capability. Removing a predecessor's rule set is an opera
|
||||
through the seat, recorded on the bus, instead of a shell on the machine. *How it is checked:* ADR
|
||||
0169's table.
|
||||
|
||||
## The intrusion seat's verbs, 2026-10-02
|
||||
|
||||
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md).
|
||||
The second node-scoped seat to carry verbs: `node-intrusion-prevention` serves `status` (every jail
|
||||
with what it watches and holds), `banned` (every address held now, with its jail and when the ban
|
||||
ends), `ban` and `unban` (an operator's act on the live ban list). The fail2ban module serves them
|
||||
from a runtime that carries only the daemon's client, the socket shared in from the machine — no
|
||||
capability, no machine network, since the daemon on the machine does the banning. That runtime is the
|
||||
per-module container [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
retires; the verbs and the client are the same code once the node's own runtime loads them as a bundle.
|
||||
The module's own tool beside them reads one jail's effective settings. *How it is checked:* ADR 0179's table.
|
||||
|
||||
## What this does not settle
|
||||
|
||||
- Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a
|
||||
|
||||
@@ -1,226 +0,0 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0183-the-anthropic-licence-manager-is-a-module-and-hands-tokens-to-the-agent-over-the-bus.md
|
||||
- 02-DECISIONS/0181-the-operator-account-is-a-node-fact-and-a-home-is-a-placement-root.md
|
||||
- 02-DECISIONS/0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md
|
||||
- 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
|
||||
- 02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md
|
||||
- 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# 40. Building the operator's agent and its licence manager
|
||||
|
||||
**The work of [design 36](36-the-operators-agent-on-a-machine.md) and [design 39](39-the-anthropic-licence-manager.md),
|
||||
broken into packages that each end at something a person can see run, in the order their
|
||||
dependencies allow.** The two designs are the authority on *what* is built; this document holds the
|
||||
packages, their order, their sizes and their proofs, and is wrong the moment it disagrees with them.
|
||||
It is the shape [design 38](38-building-the-operators-machine.md) gave the operator's machine, applied
|
||||
to the two modules that make its agent work.
|
||||
|
||||
## How this is built, and where it is run
|
||||
|
||||
**On the live mesh, by the operator's decision** ([ADR 0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md),
|
||||
and design 38's words on the same day). Every package is written with unit tests, committed on one
|
||||
branch per repository ([playbook 07](../../00-META/process/07-feature-branches.md)), and proven on the
|
||||
machines: the control node first for the manager, one workstation first for the agent, then the rest.
|
||||
The cost accepted: a broken agent module leaves a workstation's agent without the mesh's instructions
|
||||
or with a stale token until the next push; the person's own files under the home are never in reach of
|
||||
the failure, by [ADR 0182](../../02-DECISIONS/0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md).
|
||||
|
||||
Each package names what proves it. A package that cannot name its proof is divided until it can.
|
||||
|
||||
## What exists already, measured
|
||||
|
||||
Counted 2026-10-02 in the repositories and on the machines. Nothing here is new ground; every package
|
||||
reshapes something standing.
|
||||
|
||||
| Piece | Today | Becomes |
|
||||
|---|---|---|
|
||||
| the vendor's refresh, the sealed box, the grant file | `anthropic-manager` in the catalogue: a token-endpoint client, a sealed-box primitive, adoption of a grant sealed to a node's key; assigned to nothing | the manager's refresh and adoption, with the lease, the floor and the cadence the predecessor's manager had |
|
||||
| the credentials write, the refresh-token strip, the identity read | `anthropic-consumer` in the catalogue: tested; assigned to nothing | the agent module's write, unchanged in shape |
|
||||
| the predecessor's manager and consumer | two modules in the retired system: the lease per licence, the expiry floor, the lineage comparison, the identity guard, three touchpoints with fallbacks, cooldowns on alarms | ported as logic with its tests; nothing of its registry or its bus |
|
||||
| the host's `process` and `archive` shapes | fetch a bundle by digest and run it supervised; fetch and unpack an artifact | **unchanged** — the manager's daemon is one process; both modules' tools are bundles |
|
||||
| the manifest's `uses`, `claims`, `invokes seat:<seat>.<verb>`, node-scoped `provides` with `serves: {port}` | all four exist and are used by other modules | **unchanged** — the agent uses the seat and invokes its verbs; the console provides its endpoint |
|
||||
| the console | a container per node, MCP on loopback, no provision | gains one provision; becomes the node-tools runtime's serving mode under design 38's WP3 |
|
||||
| the agent's package | present on both workstations from a build the predecessor's helper made; the distribution's repositories do not carry it | declared; satisfied where present, refused where not, until a package repository seat exists |
|
||||
| the operator account | a column on every node record, **empty on all four** | stated by the operator, before anything home-scoped lands |
|
||||
|
||||
**One dependency decides the order.** Both modules serve tools and the agent module's tools write
|
||||
under `/etc` and, as the operator, under the home. Under [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
tools run in the node's tool runtime, host-side, which design 38 builds in its WP1–WP3. Writing a
|
||||
per-module tool container for these two modules would be building the pattern that record retires, so
|
||||
**the live proofs of WP3 to WP5 below wait for design 38's WP3.** Everything before a live proof —
|
||||
manifests, code, tests — does not, and is written now.
|
||||
|
||||
## The order the work allows
|
||||
|
||||
```
|
||||
WP0 the operator states the facts (the live mesh) ── accounts, roles, licences to adopt
|
||||
WP1 the console provides its endpoint (mesh-catalog) ── small, independent
|
||||
WP2 the licence manager, built and tested (mesh-catalog) ──┐ independent of each other;
|
||||
WP3 the agent module, built and tested (mesh-catalog) ──┘ both wait on design 38 WP3 to run
|
||||
│
|
||||
WP4 the manager live on the control node (the live mesh) ── three licences adopted, a refresh seen
|
||||
WP5 the agent live on one workstation (the live mesh) ── the hand-over, the switch, the instructions
|
||||
WP6 the rest of the nodes, and the predecessor's remains ── adoption from a login, the retirements
|
||||
```
|
||||
|
||||
WP1, WP2 and WP3 touch different directories of one repository and meet only at the seat's name and
|
||||
the provision's name; they are built in parallel. WP4 is the first time anything on a machine changes.
|
||||
WP5 is the proof of the whole.
|
||||
|
||||
## WP0 — The operator states the facts
|
||||
|
||||
*The live mesh. An hour, and it is the operator's.*
|
||||
|
||||
The account on each node record, through the controller's node command — none is stated today, and
|
||||
[ADR 0181](../../02-DECISIONS/0181-the-operator-account-is-a-node-fact-and-a-home-is-a-placement-root.md)
|
||||
refuses a home-scoped module without one. The role of each node, as the agent module's setting on the
|
||||
node layer, once the module exists. Which three licences exist and what each is called.
|
||||
|
||||
**Proof.** The controller's node command lists an account for every node.
|
||||
|
||||
## WP1 — The console provides its endpoint
|
||||
|
||||
*mesh-catalog. Half a day.*
|
||||
|
||||
**What changes.** The console's manifest gains a node-scoped provision — working name
|
||||
`console-endpoint`, fixed when the manifest is written — serving the port the machine gave it, as the
|
||||
local model server already does for its API. Co-location resolves it
|
||||
([ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md),
|
||||
[to-be 34](34-the-console.md) as amended). When design 38's WP3 moves the console into the node-tools
|
||||
module, the provision moves with it; it is a line in a manifest either way.
|
||||
|
||||
**Proof.** The controller's plan for a workstation shows a consumer of the provision bound to the
|
||||
console's port; the same consumer on a machine without the console is refused naming the provision;
|
||||
the catalogue's tests pass.
|
||||
|
||||
## WP2 — The licence manager, built and tested
|
||||
|
||||
*mesh-catalog. Two to three days; the largest package.*
|
||||
|
||||
**What is written**, as design 39 says:
|
||||
|
||||
1. **The manifest.** Claims the mesh-scoped seat `anthropic-licence-manager` with its verbs; requires a
|
||||
database, a `secret` for the key its grants are encrypted with, and the bus; a `bundle` of tools; a
|
||||
`process` for the daemon that refreshes, collects usage and notifies, on a schedule; declared
|
||||
settings for the cadence, the usage threshold and the cooldown, each with a default; `invokes` the
|
||||
agent module's `apply`.
|
||||
2. **The store.** Migrations for licences, bindings, usage and audit, with the lease and the
|
||||
notification slot as columns, numbered and idempotent.
|
||||
3. **The refresh.** The token-endpoint client and the sealed box from `anthropic-manager`; the plan
|
||||
(floor, cadence, forced, cannot) and the lease from the predecessor, as pure functions with their
|
||||
tests; the vendor's reason logged on failure; counted failures, one notification per cooldown.
|
||||
4. **Adoption.** From a file on the manager's node for the API key; from a sealed grant a node offers;
|
||||
the identity guard that refuses a mismatch and notifies.
|
||||
5. **Usage.** The vendor's reading per licence on a schedule, stored raw and normalised
|
||||
([ADR 0054](../../02-DECISIONS/0054-model-usage-is-recorded-at-two-grains.md)), one notification
|
||||
per threshold crossing.
|
||||
6. **The verbs**: `licences`, `bindings`, `bind`, `switch`, `release`, `refresh`, `usage`, `adopt`,
|
||||
`register`, `current` — the last answering a consumer's token sealed to the key that consumer
|
||||
registered.
|
||||
7. **The hand-over**: on rotation or switch, one call to `claude-code.apply@<node>` per bound node,
|
||||
the token sealed to that node's key, the answer recorded.
|
||||
|
||||
**Proof, before anything runs live.** Unit tests: two refresh runs started together rotate one grant
|
||||
once; a grant with a mismatching identity is refused; a worker bound to a dead licence is refused and
|
||||
never lent another; every event the daemon emits is free of a token; the hand-over payload opens only
|
||||
with the registered key. The catalogue's checks: no installation named, no secret in a declared file.
|
||||
|
||||
## WP3 — The agent module, built and tested
|
||||
|
||||
*mesh-catalog. Two days.*
|
||||
|
||||
**What is written**, as design 36 says:
|
||||
|
||||
1. **The manifest.** The agent's package; the state directory; the facts file carrying the node's
|
||||
name, the operator account and its home, the console's bound port, the role and the extra tool
|
||||
servers from settings; requires the console's endpoint and the bus; `uses` the seat and `invokes`
|
||||
its `register`, `current` and `adopt`; a `bundle` of tools. **No file resource under a home or
|
||||
under `/etc`.**
|
||||
2. **The renderer.** From the facts file and the current binding, the managed settings file (the tool
|
||||
servers under the entry `mesh`, the attribution trailers, and the key-helper for an API-key binding)
|
||||
and the managed instruction file (§3 of design 36), written under the agent's managed directory
|
||||
with the escalation the tool performs for itself; idempotent; re-run when the facts file changes.
|
||||
3. **The keypair**, made once in the state directory, the public half registered with the seat at
|
||||
start and at every start.
|
||||
4. **The consumer side**: `apply` (a rotation applied only if newer within one lineage, a switch applied
|
||||
regardless, the answer naming the outcome and never a token); the pull at start and near expiry;
|
||||
the credentials write as the operator, access-token-only, from `anthropic-consumer` with its tests;
|
||||
the key-helper program for the API key; the offer of a login to the seat, sealed, after reading the
|
||||
account's identity.
|
||||
5. **The tools**: `apply`, `licence_status`, `mcp_configure` (validates a server, sets the module's
|
||||
setting through the controller's settings verb), `render` (re-render now, for a person).
|
||||
6. **The documentation**: the six predecessor files and the hand-made console entry a person removes
|
||||
on a workstation that carried the predecessor.
|
||||
|
||||
**Proof, before anything runs live.** Unit tests: the renderer writes only the mesh's keys and leaves
|
||||
every other key of a seeded settings file; the credentials write strips a refresh token and is atomic;
|
||||
the lineage comparison from the predecessor, with its cases; a login offer carries the identity it read.
|
||||
The catalogue's checks pass.
|
||||
|
||||
## WP4 — The manager live on the control node
|
||||
|
||||
*The live mesh. Half a day, after design 38's WP3.*
|
||||
|
||||
**Order.** Assign the manager on the control node; push. Adopt the API key from a file there. Adopt
|
||||
the two subscription grants: a login in a throwaway home on the control node, offered to the seat the
|
||||
way a node's agent module will. Bind each node's agent to a licence.
|
||||
|
||||
**Proof.** Through the console: `anthropic-licence-manager.licences` lists three licences with
|
||||
identity and expiry; within the cadence, the audit shows a rotation and `licences` shows a later
|
||||
expiry; a forced `refresh` on one licence is logged with the vendor's answer; the two retired catalogue
|
||||
modules are still assigned to nothing.
|
||||
|
||||
## WP5 — The agent live on one workstation
|
||||
|
||||
*The live mesh. Half a day. The proof of the whole.*
|
||||
|
||||
**Order.** Set the workstation's role in the module's settings. Record the checksums of everything
|
||||
under the person's agent directory. Assign the module; push. Remove the six predecessor files and the
|
||||
hand-made console entry. Start a new session.
|
||||
|
||||
**Proof.** The managed directory holds the settings and instruction files, owned by root. Everything
|
||||
under the person's agent directory is byte-identical to before except the credentials file, which is
|
||||
owned by the operator, readable by nobody else, and names no refresh token. The new session lists the
|
||||
mesh's tools under `mesh` once, answers *which node am I* from the instruction file, and makes a model
|
||||
request. `anthropic-licence-manager.switch` to the second subscription licence changes the token on
|
||||
the workstation within a minute, and neither the verb's answer nor either module's log holds a token.
|
||||
Switched to the API-key licence, the credentials file is left as it was and the agent authenticates
|
||||
through the key-helper. Switched back.
|
||||
|
||||
## WP6 — The rest of the nodes, and the predecessor's remains
|
||||
|
||||
*The live mesh and mesh-catalog. One day.*
|
||||
|
||||
**Order.** Assign the module on the second workstation and on the servers whose account is stated;
|
||||
remove the predecessor's files on the second workstation. Log in on a workstation under a licence's
|
||||
account and watch the offer be adopted — and under the wrong account, and watch it refused and
|
||||
notified. Retire `anthropic-manager` and `anthropic-consumer` from the catalogue. Set designs 36 and
|
||||
39 to `implemented` for what runs, with the as-is written
|
||||
([playbook 02](../../00-META/process/02-graduation.md)).
|
||||
|
||||
**Proof.** Every node with an account runs the module and `licence_status` answers on each; the
|
||||
refused login's notification arrived; the catalogue has no module built on the old placement.
|
||||
|
||||
## What is deliberately not here
|
||||
|
||||
- **The package repository seat** for a distribution that does not carry the agent's package
|
||||
(design 36 §7). A fresh node refuses the module in the package manager's words until it exists.
|
||||
- **An automated switch on exhaustion.** The readings are kept from WP4; the policy is a later record.
|
||||
- **Workers and the mesh's own sessions as consumers.** The manager's bindings and fallbacks know them
|
||||
from WP2; the consumers themselves do not exist yet ([to-be 15](15-the-agent-session.md),
|
||||
[ADR 0003](../../02-DECISIONS/0003-agents-are-persistent-employees.md)).
|
||||
- **Whether a refresh token is single-use.** WP4 may measure it on a licence deliberately refreshed
|
||||
twice; the design holds either way.
|
||||
|
||||
## How this list is kept true
|
||||
|
||||
Each package's proof is run when the package is finished and its line here gains the date and the
|
||||
commit, as design 38 does. A package whose proof fails is not reworded; the failure is recorded under
|
||||
it and the package stays open. When WP5 is proven, designs 36 and 39 move to `in-progress` with their
|
||||
owning repository, and when WP6 is proven to `implemented`, with the as-is written.
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-02
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by: 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 201 — A push recreated the controller at a digest older than the seat row its successor had written
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the
|
||||
controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan
|
||||
built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The
|
||||
host's plan then pushed the control node with the controller digest it had recorded when it was made —
|
||||
the previous build — and recreated the controller container on it. The older binary read the row, found
|
||||
a verb it could not run, and refused to start:
|
||||
|
||||
```
|
||||
mesh-controller: the mesh-controller seat's row declares "command", which this control plane
|
||||
cannot run: "command" is not a verb the mesh-controller seat serves
|
||||
```
|
||||
|
||||
A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since
|
||||
the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the
|
||||
newer image, outside the service, to push the control node again; the push sent the newer digest and
|
||||
the controller came up.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md));
|
||||
a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of
|
||||
2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that
|
||||
wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md));
|
||||
for every other module an older digest is a brief regression a later push corrects. For the controller
|
||||
it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct
|
||||
it is the thing that is down. Two plans that overlap will happen again whenever two people merge within
|
||||
a minute.
|
||||
|
||||
## What a fix would have to do
|
||||
|
||||
Either of two, and the first is the smaller:
|
||||
|
||||
- A push never sends a controller digest older than the one the running controller is — the controller
|
||||
knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words.
|
||||
- Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves
|
||||
what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check
|
||||
exists to refuse.
|
||||
|
||||
Until one is built: do not merge a controller change while another plan is rolling, and after merging
|
||||
one, wait for `node show` on the control node to report the new controller before merging anything else.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
||||
- mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`
|
||||
|
||||
## Half of it is closed, 2026-10-02
|
||||
|
||||
[ADR 0185](../../02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) takes
|
||||
the outage out of it: a control plane behind its seat's row now serves every verb it can run, says
|
||||
which it cannot, and answers the reason when one of those is called. The same race today would cost
|
||||
the verbs the newer build added, for as long as the older binary is in place, and the ordinary
|
||||
"this machine is behind" machinery would put the newer one back without a hand.
|
||||
|
||||
**The race itself is still open**, and this report stays open for it. What was established while
|
||||
closing the other half, so the next reader does not redo it:
|
||||
|
||||
- Composing and sending are serialised per machine by a session advisory lock in the store, so two
|
||||
control planes cannot compose one machine's declaration at the same time. The stale content did
|
||||
not come from two concurrent composes.
|
||||
- A container's image is resolved into the module's manifest when it is *built*, and a push composes
|
||||
from the catalogue as it is at that moment, under the hold. So a compose that ran after the build
|
||||
was taken in could not have named the older image.
|
||||
- The declaration's sequence orders arrival and nothing else (the numbering of
|
||||
[issue 107](../107-a-declaration-carries-no-order/00-report.md)); it cannot tell a later send
|
||||
carrying earlier content from a later send carrying later content. The host refuses a declaration
|
||||
numbered below the last it applied, and both of these were above it.
|
||||
- The machine's own journal shows the two applies ten seconds apart and which replaced what; it does
|
||||
not record which image each declaration named, which is the one fact that would settle it. A host
|
||||
that recorded the digest it was told, per apply, would have answered this in a minute.
|
||||
|
||||
So the trigger is not yet pinned, and guessing at the push path is the most expensive place in the
|
||||
mesh to guess. The fix the report first suggested — a push never sending a control plane a digest
|
||||
older than the one that machine reports running — closes the class without needing the trigger, and
|
||||
is now a correctness nicety rather than the difference between a working mesh and a dead one.
|
||||
Reference in New Issue
Block a user