From 9ba4de5557113fc7061b17b11bb980a2785ebcb0 Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 2 Oct 2026 17:02:49 +0200 Subject: [PATCH 1/3] ADR 0179: the intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail; designs 31 and 33 --- ...-verbs-and-every-door-declares-its-jail.md | 107 ++++++++++++++++++ 02-DECISIONS/README.md | 1 + .../31-a-module-declares-its-fail2ban-jail.md | 35 +++++- .../01-to-be/33-the-tools-the-mesh-answers.md | 11 ++ 4 files changed, 151 insertions(+), 3 deletions(-) create mode 100644 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md diff --git a/02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md b/02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md new file mode 100644 index 0000000..e671b62 --- /dev/null +++ b/02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md @@ -0,0 +1,107 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-02 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md +--- + +# 179. The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail + +## Context + +Read on the control node on 2026-10-02, the day the machines were confirmed filtered by the mesh +alone ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)): the intrusion +prevention watched one door. Its two jails read the ssh daemon's journal and its own log, banned +five failures in ten minutes for ten minutes, and in a day had seen twelve thousand failed logins +from three hundred addresses and banned none of the busiest, which paced themselves at one try every +ten minutes. The mail submission port took a hundred and sixty password guesses in the same day from +thirty-eight addresses with no jail reading it at all; the forge and the public proxy had no jail +either, and the proxy logged nothing a jail could read. Nobody could see the jails without a shell: +the module's three tools existed in code and were served by nothing, and the seat it holds declared +no verbs. + +Three things were missing and they are three shapes the mesh already has. The packet filter's seat +serves verbs every holder owes ([ADR 0170](0170-the-firewall-seat-serves-its-verbs.md)); the +intrusion seat serves none. A module's `listens` compose into the machine's filter, and [to-be 31](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md) +says a module's `jails` compose into the machine's intrusion prevention the same way — the controller +composes them, and no module declares one. And a jail reads a log; a container's output goes to a +file of the runtime's own under a path that changes when the container is recreated, which is why +no jail could read the mail front end, the forge or the proxy, however they logged. + +## Decision + +**1. The `node-intrusion-prevention` seat serves four verbs**, and a module that claims it serves +all four or is refused the claim, as with every seat: + +- `status` — every jail with what it watches, how many addresses it is counting failures against and + holding now, and the totals since it started; one jail's detail when named. Read-only. +- `banned` — every address banned now, with the jail holding it, when it was banned and when the ban + ends. Read-only. +- `ban` — ban one address in one jail now, for that jail's ban time. An operator's act on the live + ban list, which the mesh composes the rules for and never writes itself. +- `unban` — let one address go, from one jail or from every jail. + +A holder may serve its own tools beside these; the fail2ban module reads one jail's effective +settings as its own. + +**2. A container may log to the journal.** `logging: journald` on a container has the host run it +with the journal as its log driver; the journal keeps the container's name on every line, and +`docker logs` keeps working. Where a container logs is part of its spec, so moving it recreates the +container, and the only place besides the runtime's own file is the journal: a machine's intrusion +prevention reads the journal already, for the ssh daemon, and a container that logs there is read +the same way, by the container's name, whatever the container is called by the runtime this time. + +**3. A module with a door declares its jail, and the holder composes them.** What to-be 31 designed +is now the rule: a module whose service authenticates from outside — the mail front end, the forge, +the public proxy — declares in its manifest what a failed attempt looks like in its log and how to +ban on it, naming no node and no path; the module that holds the intrusion seat declares where the +composed jails and filters land, and the mesh writes them on every machine that runs both. A machine +not running the module has no such jail. The holder restarts its daemon on the composed file. + +**4. The base is strict, and the mesh's own range is never banned.** Three failures in a day ban for +a day, on every jail unless the jail says otherwise; banned twice in two weeks, by any jail, is +banned for four. The attackers this mesh sees pace themselves under any ten-minute window; a day's +window counts them. A person who mistypes three times from one address is out for a day from that +address, and never from a machine of the mesh, whose range stays in the never-banned list the module +has carried since [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md). The operator +chose this knowing it. + +**5. The proxy says a refused name in its log.** A request for a name this mesh does not serve, from +outside, is what a scanner does; the proxy already logged a certificate refused for such a name, and +now logs the plain request too, with the asking address last, as its own jail's filter expects it. + +## Consequences + +- The seat's row gains four verbs; a mesh that already runs widens its row at the next controller + start. The fail2ban module claims them and gains a runtime — a tool server whose image carries the + fail2ban client, with the daemon's socket shared in from the machine, and nothing else of the + machine. The daemon stays the machine's; what runs in the container is only the client. +- The host's container vocabulary grows by `logging`; an older host refuses a declaration that carries + it, so the host rolls before the modules. Three containers are recreated once, when their modules + are pushed with the field: the mail front end, the forge and the proxy — each a moment's outage. +- The fail2ban module declares where jails compose (`jailing`) and the directory the filters go in; + the mail, forge and proxy modules each declare one jail reading the journal by their container's + name. The composed jail file is the one resource the daemon restarts on when a module arrives or + leaves a machine. +- The two base jails and the composed ones take the day's window; the ssh jail's ten minutes are + gone. An address banned on the first day of this record stays banned for the day. +- The module's three old tools, served by nothing, are replaced by the seat's four verbs and one + own tool; `fail2ban_status` as a name is gone. + +## How this is checked + +| Rule | Checked by | +|---|---| +| The seat declares the four verbs; a claim that serves fewer is refused by name | the catalogue's seat tests | +| `status`, `banned`, `ban` and `unban` read and steer the daemon through its client, with the shapes fail2ban 1.1.0 printed live; a non-address and a non-name are refused before anything runs | the module's tests over a fake command runner | +| A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests | +| A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) | +| The proxy logs a refused name with the address last | the proxy's tests | +| Live | `node-intrusion-prevention.status@` and `banned` answer on every machine through the console; the mail front end's jail reads the journal and counts a guess within the day; `ban` of a documentation address lands in the ban list and `unban` releases it; `status` is well | + +## References + +- [ADR 0170](0170-the-firewall-seat-serves-its-verbs.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md) +- [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md), [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md), [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 4d2ecd8..95112e4 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -183,6 +183,7 @@ python3 00-META/checks/index.py fail if stale - **0170** — [The firewall seat serves its verbs, and a foreign rule set is removed through one of them](0170-the-firewall-seat-serves-its-verbs.md) - **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md) - **0175** — [The found front end is uninstalled once a machine is converged](0175-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md) +- **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md b/03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md index e477693..05bdeae 100644 --- a/03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md +++ b/03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md @@ -1,10 +1,14 @@ --- layer: to-be -status: proposed -code: [] -updated: 2026-09-27 +status: in-progress +code: + - mesh-controller: internal/catalogue/jails_into.go, internal/catalogue/manifest.go (Jail, Jailing) + - mesh-catalog: modules/fail2ban (jailing, the base and the seat's verbs), modules/mailu, modules/route-proxy, modules/gitea (jails) + - mesh-host: internal/declaration/declaration.go (a container's logging) +updated: 2026-10-02 decisions: - 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md + - 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md --- # 31 — A module declares its fail2ban jail, and the mesh composes them per node @@ -63,3 +67,28 @@ jail, composed from the postgres module's manifest, without anyone editing a nod beside) - mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules (`postgres`, `mssql`, `mailu`) that will declare jails + +## Decided and built, 2026-10-02 + +[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) +made this the rule and built it. A module declares `jails` — each a name, the `failregex` of a +failed attempt in its log, and the stanza's own keys — and the fail2ban module declares `jailing`: +the one file the stanzas compose into and the directory each filter lands in. The controller gathers +every assigned module's jails per node into those; the holder's daemon restarts on the composed file. + +What made it workable was the log. A container's output went to a file of the runtime's own, under +a path that changes when the container is recreated, so no jail could read a container's service +however it logged. A container now declares `logging: journald`, the host runs it with the journal as +its driver, and a jail reads it with `backend = systemd` and a `journalmatch` on the container's +name — the same way the base's ssh jail has always read the ssh daemon. The first three doors: the +mail front end (every login failure on its proxying ports), the forge (a failed authentication +attempt) and the public proxy (a certificate or request for a name the mesh does not serve, which +the proxy now says in its log). The base is strict — three in a day for a day; twice banned in two +weeks for four — and the mesh's own range stays never banned. + +The seat the module holds serves `status`, `banned`, `ban` and `unban`, from a runtime that carries +only the fail2ban client with the daemon's socket shared in; the jails are composed, the ban list is +the daemon's, and both are read through the console. + +*How it is checked:* ADR 0179's table. + diff --git a/03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md b/03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md index c2926b6..02f32dd 100644 --- a/03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md +++ b/03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md @@ -5,6 +5,7 @@ code: [mesh-controller, mesh-tools] updated: 2026-10-02 decisions: - 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md + - 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md - 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md - 02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md - 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md @@ -182,6 +183,16 @@ container to declare a capability. Removing a predecessor's rule set is an opera through the seat, recorded on the bus, instead of a shell on the machine. *How it is checked:* ADR 0169's table. +## The intrusion seat's verbs, 2026-10-02 + +[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md). +The second node-scoped seat to carry verbs: `node-intrusion-prevention` serves `status` (every jail +with what it watches and holds), `banned` (every address held now, with its jail and when the ban +ends), `ban` and `unban` (an operator's act on the live ban list). The fail2ban module serves them +from a runtime that carries only the daemon's client, the socket shared in from the machine — no +capability, no machine network, since the daemon on the machine does the banning. The module's own +tool beside them reads one jail's effective settings. *How it is checked:* ADR 0179's table. + ## What this does not settle - Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a -- 2.54.0 From d0d57998846acc594f0b8c0ccd9d78f78dbdb39e Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 2 Oct 2026 17:15:22 +0200 Subject: [PATCH 2/3] Issue 201: a push recreated the controller at a digest older than the seat row its successor wrote --- .../00-report.md | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md diff --git a/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md b/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md new file mode 100644 index 0000000..0daa672 --- /dev/null +++ b/04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md @@ -0,0 +1,58 @@ +--- +status: open +opened: 2026-10-02 +located-in: + - mesh-controller +fixed-by: +amended-design: +--- + +# 201 — A push recreated the controller at a digest older than the seat row its successor had written + +## What was observed + +2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the +controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan +built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The +host's plan then pushed the control node with the controller digest it had recorded when it was made — +the previous build — and recreated the controller container on it. The older binary read the row, found +a verb it could not run, and refused to start: + +``` +mesh-controller: the mesh-controller seat's row declares "command", which this control plane +cannot run: "command" is not a verb the mesh-controller seat serves +``` + +A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since +the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the +newer image, outside the service, to push the control node again; the push sent the newer digest and +the controller came up. + +## Why it matters beyond this instance + +The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)); +a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of +2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that +wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)); +for every other module an older digest is a brief regression a later push corrects. For the controller +it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct +it is the thing that is down. Two plans that overlap will happen again whenever two people merge within +a minute. + +## What a fix would have to do + +Either of two, and the first is the smaller: + +- A push never sends a controller digest older than the one the running controller is — the controller + knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words. +- Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves + what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check + exists to refuse. + +Until one is built: do not merge a controller change while another plan is rolling, and after merging +one, wait for `node show` on the control node to report the new controller before merging anything else. + +## References + +- [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) +- mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go` -- 2.54.0 From 114a71f36f41b930bbbef69be0cd35b59d309993 Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 2 Oct 2026 17:28:38 +0200 Subject: [PATCH 3/3] ADR 0179: built and proven live; the one fault the machine found, and the check that refuses it --- ...-verbs-and-every-door-declares-its-jail.md | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md b/02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md index c1a4ebf..c132c86 100644 --- a/02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md +++ b/02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md @@ -106,7 +106,25 @@ now logs the plain request too, with the asking address last, as its own jail's | A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests | | A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) | | The proxy logs a refused name with the address last | the proxy's tests | -| Live | `node-intrusion-prevention.status@` and `banned` answer on every machine through the console; the mail front end's jail reads the journal and counts a guess within the day; `ban` of a documentation address lands in the ban list and `unban` releases it; `status` is well | +| A jail's pattern names `` once per shape, since two is a duplicate capture group and costs the machine every ban | the catalogue's manifest tests | +| Live | done 2026-10-02: `status` and `banned` answered on both servers through the console; the proxy's jail counted seven refusals on the home server; a documentation address banned in the ssh jail came back with its end time and was released | + +## Built and proven live, 2026-10-02 + +All five rules are in the mesh. The host carries `logging`; the controller's seat row carries the four +verbs and the proxy says a refused name in its log; the fail2ban module holds the seat from a runtime +with the daemon's socket shared in, composes the jails, and the mail front end, the forge and the +proxy each declare one. Through the console on the control node: `status` listed five jails with what +each watches, `banned` listed the nine the long jail holds, and a documentation address banned in the +ssh jail came back with its ban's end time and was released again. On the home server the proxy's jail +had counted seven refusals within minutes of starting. + +**One fault, found by the machine and not by a test.** The proxy's pattern matched two shapes of +refusal in one expression and so named `` twice. fail2ban expands that placeholder into a named +capture group; two of them is a duplicate group name, and the daemon refuses *its whole configuration* +and exits — both servers kept no bans at all for about ten minutes, every jail and not the one at +fault. The pattern is now one per shape. A manifest check refuses the mistake at merge time, naming +what it would cost, which is the only reason this record can claim the rule rather than the instance. ## References -- 2.54.0