2026-10-02 15:28:48 +00:00
5 changed files with 236 additions and 3 deletions
@@ -0,0 +1,132 @@
 ---
 topic: the mesh
 status: accepted
 date: 2026-10-02
 deciders: jochen
 reconstructed: false
 extends: 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
 ---
 # 179. The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail
 ## Context
 Read on the control node on 2026-10-02, the day the machines were confirmed filtered by the mesh
 alone ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)): the intrusion
 prevention watched one door. Its two jails read the ssh daemon's journal and its own log, banned
 five failures in ten minutes for ten minutes, and in a day had seen twelve thousand failed logins
 from three hundred addresses and banned none of the busiest, which paced themselves at one try every
 ten minutes. The mail submission port took a hundred and sixty password guesses in the same day from
 thirty-eight addresses with no jail reading it at all; the forge and the public proxy had no jail
 either, and the proxy logged nothing a jail could read. Nobody could see the jails without a shell:
 the module's three tools existed in code and were served by nothing, and the seat it holds declared
 no verbs.
 Three things were missing and they are three shapes the mesh already has. The packet filter's seat
 serves verbs every holder owes ([ADR 0170](0170-the-firewall-seat-serves-its-verbs.md)); the
 intrusion seat serves none. A module's `listens` compose into the machine's filter, and [to-be 31](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
 says a module's `jails` compose into the machine's intrusion prevention the same way — the controller
 composes them, and no module declares one. And a jail reads a log; a container's output goes to a
 file of the runtime's own under a path that changes when the container is recreated, which is why
 no jail could read the mail front end, the forge or the proxy, however they logged.
 ## Decision
 **1. The `node-intrusion-prevention` seat serves four verbs**, and a module that claims it serves
 all four or is refused the claim, as with every seat:
 - `status` — every jail with what it watches, how many addresses it is counting failures against and
  holding now, and the totals since it started; one jail's detail when named. Read-only.
 - `banned` — every address banned now, with the jail holding it, when it was banned and when the ban
  ends. Read-only.
 - `ban` — ban one address in one jail now, for that jail's ban time. An operator's act on the live
  ban list, which the mesh composes the rules for and never writes itself.
 - `unban` — let one address go, from one jail or from every jail.
 A holder may serve its own tools beside these; the fail2ban module reads one jail's effective
 settings as its own.
 **2. A container may log to the journal.** `logging: journald` on a container has the host run it
 with the journal as its log driver; the journal keeps the container's name on every line, and
 `docker logs` keeps working. Where a container logs is part of its spec, so moving it recreates the
 container, and the only place besides the runtime's own file is the journal: a machine's intrusion
 prevention reads the journal already, for the ssh daemon, and a container that logs there is read
 the same way, by the container's name, whatever the container is called by the runtime this time.
 **3. A module with a door declares its jail, and the holder composes them.** What to-be 31 designed
 is now the rule: a module whose service authenticates from outside — the mail front end, the forge,
 the public proxy — declares in its manifest what a failed attempt looks like in its log and how to
 ban on it, naming no node and no path; the module that holds the intrusion seat declares where the
 composed jails and filters land, and the mesh writes them on every machine that runs both. A machine
 not running the module has no such jail. The holder restarts its daemon on the composed file.
 **4. The base is strict, and the mesh's own range is never banned.** Three failures in a day ban for
 a day, on every jail unless the jail says otherwise; banned twice in two weeks, by any jail, is
 banned for four. The attackers this mesh sees pace themselves under any ten-minute window; a day's
 window counts them. A person who mistypes three times from one address is out for a day from that
 address, and never from a machine of the mesh, whose range stays in the never-banned list the module
 has carried since [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md). The operator
 chose this knowing it.
 **5. The proxy says a refused name in its log.** A request for a name this mesh does not serve, from
 outside, is what a scanner does; the proxy already logged a certificate refused for such a name, and
 now logs the plain request too, with the asking address last, as its own jail's filter expects it.
 ## Consequences
 - The seat's row gains four verbs; a mesh that already runs widens its row at the next controller
  start. The fail2ban module claims them and gains a runtime — a tool server whose image carries the
  fail2ban client, with the daemon's socket shared in from the machine, and nothing else of the
  machine. The daemon stays the machine's; what runs in the container is only the client.
 - **That runtime is the shape the catalogue has today, and it is on its way out.**
  [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md),
  accepted the same day as this record, replaces a tool container per module with one tool runtime
  per node on the host side, taking each module's tools as a bundle. Nothing here depends on the
  container: the verbs, the client that speaks to the daemon over its socket, and the jails are the
  same code under either. This module converts with the packet filter's, whose runtime that record
  names, and the socket it needs becomes the node runtime's to reach rather than a mount of its own.
 - The host's container vocabulary grows by `logging`; an older host refuses a declaration that carries
  it, so the host rolls before the modules. Three containers are recreated once, when their modules
  are pushed with the field: the mail front end, the forge and the proxy — each a moment's outage.
 - The fail2ban module declares where jails compose (`jailing`) and the directory the filters go in;
  the mail, forge and proxy modules each declare one jail reading the journal by their container's
  name. The composed jail file is the one resource the daemon restarts on when a module arrives or
  leaves a machine.
 - The two base jails and the composed ones take the day's window; the ssh jail's ten minutes are
  gone. An address banned on the first day of this record stays banned for the day.
 - The module's three old tools, served by nothing, are replaced by the seat's four verbs and one
  own tool; `fail2ban_status` as a name is gone.
 ## How this is checked
 | Rule | Checked by |
 |---|---|
 | The seat declares the four verbs; a claim that serves fewer is refused by name | the catalogue's seat tests |
 | `status`, `banned`, `ban` and `unban` read and steer the daemon through its client, with the shapes fail2ban 1.1.0 printed live; a non-address and a non-name are refused before anything runs | the module's tests over a fake command runner |
 | A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests |
 | A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) |
 | The proxy logs a refused name with the address last | the proxy's tests |
 | A jail's pattern names `<HOST>` once per shape, since two is a duplicate capture group and costs the machine every ban | the catalogue's manifest tests |
 | Live | done 2026-10-02: `status` and `banned` answered on both servers through the console; the proxy's jail counted seven refusals on the home server; a documentation address banned in the ssh jail came back with its end time and was released |
 ## Built and proven live, 2026-10-02
 All five rules are in the mesh. The host carries `logging`; the controller's seat row carries the four
 verbs and the proxy says a refused name in its log; the fail2ban module holds the seat from a runtime
 with the daemon's socket shared in, composes the jails, and the mail front end, the forge and the
 proxy each declare one. Through the console on the control node: `status` listed five jails with what
 each watches, `banned` listed the nine the long jail holds, and a documentation address banned in the
 ssh jail came back with its ban's end time and was released again. On the home server the proxy's jail
 had counted seven refusals within minutes of starting.
 **One fault, found by the machine and not by a test.** The proxy's pattern matched two shapes of
 refusal in one expression and so named `<HOST>` twice. fail2ban expands that placeholder into a named
 capture group; two of them is a duplicate group name, and the daemon refuses *its whole configuration*
 and exits — both servers kept no bans at all for about ten minutes, every jail and not the one at
 fault. The pattern is now one per shape. A manifest check refuses the mistake at merge time, naming
 what it would cost, which is the only reason this record can claim the rule rather than the instance.
 ## References
 - [ADR 0170](0170-the-firewall-seat-serves-its-verbs.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
 - [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md), [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md), [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md)
@@ -182,6 +182,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)
 - **0170** — [The firewall seat serves its verbs, and a foreign rule set is removed through one of them](0170-the-firewall-seat-serves-its-verbs.md)
 - **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md)
 - **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
 - **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
 ### Its tiers, from the bottom up
@@ -1,10 +1,14 @@
 ---
 layer: to-be
-status: proposed
+status: in-progress
-code: []
+code:
-updated: 2026-09-27
+  - mesh-controller: internal/catalogue/jails_into.go, internal/catalogue/manifest.go (Jail, Jailing)
  - mesh-catalog: modules/fail2ban (jailing, the base and the seat's verbs), modules/mailu, modules/route-proxy, modules/gitea (jails)
  - mesh-host: internal/declaration/declaration.go (a container's logging)
 updated: 2026-10-02
 decisions:
  - 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
  - 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
 ---
 # 31 — A module declares its fail2ban jail, and the mesh composes them per node
@@ -63,3 +67,28 @@ jail, composed from the postgres module's manifest, without anyone editing a nod
  beside)
 - mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules
  (`postgres`, `mssql`, `mailu`) that will declare jails
 ## Decided and built, 2026-10-02
 [ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
 made this the rule and built it. A module declares `jails` — each a name, the `failregex` of a
 failed attempt in its log, and the stanza's own keys — and the fail2ban module declares `jailing`:
 the one file the stanzas compose into and the directory each filter lands in. The controller gathers
 every assigned module's jails per node into those; the holder's daemon restarts on the composed file.
 What made it workable was the log. A container's output went to a file of the runtime's own, under
 a path that changes when the container is recreated, so no jail could read a container's service
 however it logged. A container now declares `logging: journald`, the host runs it with the journal as
 its driver, and a jail reads it with `backend = systemd` and a `journalmatch` on the container's
 name — the same way the base's ssh jail has always read the ssh daemon. The first three doors: the
 mail front end (every login failure on its proxying ports), the forge (a failed authentication
 attempt) and the public proxy (a certificate or request for a name the mesh does not serve, which
 the proxy now says in its log). The base is strict — three in a day for a day; twice banned in two
 weeks for four — and the mesh's own range stays never banned.
 The seat the module holds serves `status`, `banned`, `ban` and `unban`, from a runtime that carries
 only the fail2ban client with the daemon's socket shared in; the jails are composed, the ban list is
 the daemon's, and both are read through the console.
 *How it is checked:* ADR 0179's table.
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-tools]
 updated: 2026-10-02
 decisions:
  - 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
  - 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
  - 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
  - 02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md
  - 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
@@ -184,6 +185,18 @@ container to declare a capability. Removing a predecessor's rule set is an opera
 through the seat, recorded on the bus, instead of a shell on the machine. *How it is checked:* ADR
 0169's table.
 ## The intrusion seat's verbs, 2026-10-02
 [ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md).
 The second node-scoped seat to carry verbs: `node-intrusion-prevention` serves `status` (every jail
 with what it watches and holds), `banned` (every address held now, with its jail and when the ban
 ends), `ban` and `unban` (an operator's act on the live ban list). The fail2ban module serves them
 from a runtime that carries only the daemon's client, the socket shared in from the machine — no
 capability, no machine network, since the daemon on the machine does the banning. That runtime is the
 per-module container [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
 retires; the verbs and the client are the same code once the node's own runtime loads them as a bundle.
 The module's own tool beside them reads one jail's effective settings. *How it is checked:* ADR 0179's table.
 ## What this does not settle
 - Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a
@@ -0,0 +1,58 @@
 ---
 status: open
 opened: 2026-10-02
 located-in:
  - mesh-controller
 fixed-by:
 amended-design:
 ---
 # 201 — A push recreated the controller at a digest older than the seat row its successor had written
 ## What was observed
 2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the
 controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan
 built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The
 host's plan then pushed the control node with the controller digest it had recorded when it was made —
 the previous build — and recreated the controller container on it. The older binary read the row, found
 a verb it could not run, and refused to start:
 ```
 mesh-controller: the mesh-controller seat's row declares "command", which this control plane
 cannot run: "command" is not a verb the mesh-controller seat serves
 ```
 A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since
 the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the
 newer image, outside the service, to push the control node again; the push sent the newer digest and
 the controller came up.
 ## Why it matters beyond this instance
 The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md));
 a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of
 2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that
 wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md));
 for every other module an older digest is a brief regression a later push corrects. For the controller
 it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct
 it is the thing that is down. Two plans that overlap will happen again whenever two people merge within
 a minute.
 ## What a fix would have to do
 Either of two, and the first is the smaller:
 - A push never sends a controller digest older than the one the running controller is — the controller
  knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words.
 - Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves
  what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check
  exists to refuse.
 Until one is built: do not merge a controller change while another plan is rolling, and after merging
 one, wait for `node show` on the control node to report the new controller before merging anything else.
 ## References
 - [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
 - mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`