ADR 0179: the intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail #295
+132
@@ -0,0 +1,132 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
---
|
||||
|
||||
# 179. The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail
|
||||
|
||||
## Context
|
||||
|
||||
Read on the control node on 2026-10-02, the day the machines were confirmed filtered by the mesh
|
||||
alone ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)): the intrusion
|
||||
prevention watched one door. Its two jails read the ssh daemon's journal and its own log, banned
|
||||
five failures in ten minutes for ten minutes, and in a day had seen twelve thousand failed logins
|
||||
from three hundred addresses and banned none of the busiest, which paced themselves at one try every
|
||||
ten minutes. The mail submission port took a hundred and sixty password guesses in the same day from
|
||||
thirty-eight addresses with no jail reading it at all; the forge and the public proxy had no jail
|
||||
either, and the proxy logged nothing a jail could read. Nobody could see the jails without a shell:
|
||||
the module's three tools existed in code and were served by nothing, and the seat it holds declared
|
||||
no verbs.
|
||||
|
||||
Three things were missing and they are three shapes the mesh already has. The packet filter's seat
|
||||
serves verbs every holder owes ([ADR 0170](0170-the-firewall-seat-serves-its-verbs.md)); the
|
||||
intrusion seat serves none. A module's `listens` compose into the machine's filter, and [to-be 31](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
|
||||
says a module's `jails` compose into the machine's intrusion prevention the same way — the controller
|
||||
composes them, and no module declares one. And a jail reads a log; a container's output goes to a
|
||||
file of the runtime's own under a path that changes when the container is recreated, which is why
|
||||
no jail could read the mail front end, the forge or the proxy, however they logged.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. The `node-intrusion-prevention` seat serves four verbs**, and a module that claims it serves
|
||||
all four or is refused the claim, as with every seat:
|
||||
|
||||
- `status` — every jail with what it watches, how many addresses it is counting failures against and
|
||||
holding now, and the totals since it started; one jail's detail when named. Read-only.
|
||||
- `banned` — every address banned now, with the jail holding it, when it was banned and when the ban
|
||||
ends. Read-only.
|
||||
- `ban` — ban one address in one jail now, for that jail's ban time. An operator's act on the live
|
||||
ban list, which the mesh composes the rules for and never writes itself.
|
||||
- `unban` — let one address go, from one jail or from every jail.
|
||||
|
||||
A holder may serve its own tools beside these; the fail2ban module reads one jail's effective
|
||||
settings as its own.
|
||||
|
||||
**2. A container may log to the journal.** `logging: journald` on a container has the host run it
|
||||
with the journal as its log driver; the journal keeps the container's name on every line, and
|
||||
`docker logs` keeps working. Where a container logs is part of its spec, so moving it recreates the
|
||||
container, and the only place besides the runtime's own file is the journal: a machine's intrusion
|
||||
prevention reads the journal already, for the ssh daemon, and a container that logs there is read
|
||||
the same way, by the container's name, whatever the container is called by the runtime this time.
|
||||
|
||||
**3. A module with a door declares its jail, and the holder composes them.** What to-be 31 designed
|
||||
is now the rule: a module whose service authenticates from outside — the mail front end, the forge,
|
||||
the public proxy — declares in its manifest what a failed attempt looks like in its log and how to
|
||||
ban on it, naming no node and no path; the module that holds the intrusion seat declares where the
|
||||
composed jails and filters land, and the mesh writes them on every machine that runs both. A machine
|
||||
not running the module has no such jail. The holder restarts its daemon on the composed file.
|
||||
|
||||
**4. The base is strict, and the mesh's own range is never banned.** Three failures in a day ban for
|
||||
a day, on every jail unless the jail says otherwise; banned twice in two weeks, by any jail, is
|
||||
banned for four. The attackers this mesh sees pace themselves under any ten-minute window; a day's
|
||||
window counts them. A person who mistypes three times from one address is out for a day from that
|
||||
address, and never from a machine of the mesh, whose range stays in the never-banned list the module
|
||||
has carried since [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md). The operator
|
||||
chose this knowing it.
|
||||
|
||||
**5. The proxy says a refused name in its log.** A request for a name this mesh does not serve, from
|
||||
outside, is what a scanner does; the proxy already logged a certificate refused for such a name, and
|
||||
now logs the plain request too, with the asking address last, as its own jail's filter expects it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The seat's row gains four verbs; a mesh that already runs widens its row at the next controller
|
||||
start. The fail2ban module claims them and gains a runtime — a tool server whose image carries the
|
||||
fail2ban client, with the daemon's socket shared in from the machine, and nothing else of the
|
||||
machine. The daemon stays the machine's; what runs in the container is only the client.
|
||||
- **That runtime is the shape the catalogue has today, and it is on its way out.**
|
||||
[ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md),
|
||||
accepted the same day as this record, replaces a tool container per module with one tool runtime
|
||||
per node on the host side, taking each module's tools as a bundle. Nothing here depends on the
|
||||
container: the verbs, the client that speaks to the daemon over its socket, and the jails are the
|
||||
same code under either. This module converts with the packet filter's, whose runtime that record
|
||||
names, and the socket it needs becomes the node runtime's to reach rather than a mount of its own.
|
||||
- The host's container vocabulary grows by `logging`; an older host refuses a declaration that carries
|
||||
it, so the host rolls before the modules. Three containers are recreated once, when their modules
|
||||
are pushed with the field: the mail front end, the forge and the proxy — each a moment's outage.
|
||||
- The fail2ban module declares where jails compose (`jailing`) and the directory the filters go in;
|
||||
the mail, forge and proxy modules each declare one jail reading the journal by their container's
|
||||
name. The composed jail file is the one resource the daemon restarts on when a module arrives or
|
||||
leaves a machine.
|
||||
- The two base jails and the composed ones take the day's window; the ssh jail's ten minutes are
|
||||
gone. An address banned on the first day of this record stays banned for the day.
|
||||
- The module's three old tools, served by nothing, are replaced by the seat's four verbs and one
|
||||
own tool; `fail2ban_status` as a name is gone.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The seat declares the four verbs; a claim that serves fewer is refused by name | the catalogue's seat tests |
|
||||
| `status`, `banned`, `ban` and `unban` read and steer the daemon through its client, with the shapes fail2ban 1.1.0 printed live; a non-address and a non-name are refused before anything runs | the module's tests over a fake command runner |
|
||||
| A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests |
|
||||
| A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) |
|
||||
| The proxy logs a refused name with the address last | the proxy's tests |
|
||||
| A jail's pattern names `<HOST>` once per shape, since two is a duplicate capture group and costs the machine every ban | the catalogue's manifest tests |
|
||||
| Live | done 2026-10-02: `status` and `banned` answered on both servers through the console; the proxy's jail counted seven refusals on the home server; a documentation address banned in the ssh jail came back with its end time and was released |
|
||||
|
||||
## Built and proven live, 2026-10-02
|
||||
|
||||
All five rules are in the mesh. The host carries `logging`; the controller's seat row carries the four
|
||||
verbs and the proxy says a refused name in its log; the fail2ban module holds the seat from a runtime
|
||||
with the daemon's socket shared in, composes the jails, and the mail front end, the forge and the
|
||||
proxy each declare one. Through the console on the control node: `status` listed five jails with what
|
||||
each watches, `banned` listed the nine the long jail holds, and a documentation address banned in the
|
||||
ssh jail came back with its ban's end time and was released again. On the home server the proxy's jail
|
||||
had counted seven refusals within minutes of starting.
|
||||
|
||||
**One fault, found by the machine and not by a test.** The proxy's pattern matched two shapes of
|
||||
refusal in one expression and so named `<HOST>` twice. fail2ban expands that placeholder into a named
|
||||
capture group; two of them is a duplicate group name, and the daemon refuses *its whole configuration*
|
||||
and exits — both servers kept no bans at all for about ten minutes, every jail and not the one at
|
||||
fault. The pattern is now one per shape. A manifest check refuses the mistake at merge time, naming
|
||||
what it would cost, which is the only reason this record can claim the rule rather than the instance.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0170](0170-the-firewall-seat-serves-its-verbs.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
- [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md), [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md), [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md)
|
||||
@@ -182,6 +182,7 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)
|
||||
- **0170** — [The firewall seat serves its verbs, and a foreign rule set is removed through one of them](0170-the-firewall-seat-serves-its-verbs.md)
|
||||
- **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md)
|
||||
- **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
- **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
|
||||
|
||||
### Its tiers, from the bottom up
|
||||
|
||||
@@ -1,10 +1,14 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code: []
|
||||
updated: 2026-09-27
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller: internal/catalogue/jails_into.go, internal/catalogue/manifest.go (Jail, Jailing)
|
||||
- mesh-catalog: modules/fail2ban (jailing, the base and the seat's verbs), modules/mailu, modules/route-proxy, modules/gitea (jails)
|
||||
- mesh-host: internal/declaration/declaration.go (a container's logging)
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
|
||||
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
---
|
||||
|
||||
# 31 — A module declares its fail2ban jail, and the mesh composes them per node
|
||||
@@ -63,3 +67,28 @@ jail, composed from the postgres module's manifest, without anyone editing a nod
|
||||
beside)
|
||||
- mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules
|
||||
(`postgres`, `mssql`, `mailu`) that will declare jails
|
||||
|
||||
## Decided and built, 2026-10-02
|
||||
|
||||
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
made this the rule and built it. A module declares `jails` — each a name, the `failregex` of a
|
||||
failed attempt in its log, and the stanza's own keys — and the fail2ban module declares `jailing`:
|
||||
the one file the stanzas compose into and the directory each filter lands in. The controller gathers
|
||||
every assigned module's jails per node into those; the holder's daemon restarts on the composed file.
|
||||
|
||||
What made it workable was the log. A container's output went to a file of the runtime's own, under
|
||||
a path that changes when the container is recreated, so no jail could read a container's service
|
||||
however it logged. A container now declares `logging: journald`, the host runs it with the journal as
|
||||
its driver, and a jail reads it with `backend = systemd` and a `journalmatch` on the container's
|
||||
name — the same way the base's ssh jail has always read the ssh daemon. The first three doors: the
|
||||
mail front end (every login failure on its proxying ports), the forge (a failed authentication
|
||||
attempt) and the public proxy (a certificate or request for a name the mesh does not serve, which
|
||||
the proxy now says in its log). The base is strict — three in a day for a day; twice banned in two
|
||||
weeks for four — and the mesh's own range stays never banned.
|
||||
|
||||
The seat the module holds serves `status`, `banned`, `ban` and `unban`, from a runtime that carries
|
||||
only the fail2ban client with the daemon's socket shared in; the jails are composed, the ban list is
|
||||
the daemon's, and both are read through the console.
|
||||
|
||||
*How it is checked:* ADR 0179's table.
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-tools]
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
|
||||
- 02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md
|
||||
- 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
|
||||
@@ -184,6 +185,18 @@ container to declare a capability. Removing a predecessor's rule set is an opera
|
||||
through the seat, recorded on the bus, instead of a shell on the machine. *How it is checked:* ADR
|
||||
0169's table.
|
||||
|
||||
## The intrusion seat's verbs, 2026-10-02
|
||||
|
||||
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md).
|
||||
The second node-scoped seat to carry verbs: `node-intrusion-prevention` serves `status` (every jail
|
||||
with what it watches and holds), `banned` (every address held now, with its jail and when the ban
|
||||
ends), `ban` and `unban` (an operator's act on the live ban list). The fail2ban module serves them
|
||||
from a runtime that carries only the daemon's client, the socket shared in from the machine — no
|
||||
capability, no machine network, since the daemon on the machine does the banning. That runtime is the
|
||||
per-module container [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
retires; the verbs and the client are the same code once the node's own runtime loads them as a bundle.
|
||||
The module's own tool beside them reads one jail's effective settings. *How it is checked:* ADR 0179's table.
|
||||
|
||||
## What this does not settle
|
||||
|
||||
- Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a
|
||||
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-02
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 201 — A push recreated the controller at a digest older than the seat row its successor had written
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the
|
||||
controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan
|
||||
built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The
|
||||
host's plan then pushed the control node with the controller digest it had recorded when it was made —
|
||||
the previous build — and recreated the controller container on it. The older binary read the row, found
|
||||
a verb it could not run, and refused to start:
|
||||
|
||||
```
|
||||
mesh-controller: the mesh-controller seat's row declares "command", which this control plane
|
||||
cannot run: "command" is not a verb the mesh-controller seat serves
|
||||
```
|
||||
|
||||
A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since
|
||||
the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the
|
||||
newer image, outside the service, to push the control node again; the push sent the newer digest and
|
||||
the controller came up.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md));
|
||||
a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of
|
||||
2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that
|
||||
wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md));
|
||||
for every other module an older digest is a brief regression a later push corrects. For the controller
|
||||
it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct
|
||||
it is the thing that is down. Two plans that overlap will happen again whenever two people merge within
|
||||
a minute.
|
||||
|
||||
## What a fix would have to do
|
||||
|
||||
Either of two, and the first is the smaller:
|
||||
|
||||
- A push never sends a controller digest older than the one the running controller is — the controller
|
||||
knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words.
|
||||
- Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves
|
||||
what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check
|
||||
exists to refuse.
|
||||
|
||||
Until one is built: do not merge a controller change while another plan is rolling, and after merging
|
||||
one, wait for `node show` on the control node to report the new controller before merging anything else.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
||||
- mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`
|
||||
Reference in New Issue
Block a user