Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b69ae663bc | ||
|
|
0b9fc90885 | ||
|
|
983fd412c6 | ||
|
|
9ac2493e2c | ||
|
|
560f25c2c7 | ||
|
|
9e0288128b | ||
|
|
d57289e049 | ||
|
|
d4a2f99ab5 | ||
|
|
a9f91fdd0c | ||
|
|
131a5e4714 | ||
|
|
329a24fdae | ||
|
|
7f72f3b79a | ||
|
|
114a71f36f | ||
|
|
0f407417f3 | ||
|
|
d0d5799884 | ||
|
|
9ba4de5557 |
+132
@@ -0,0 +1,132 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
---
|
||||
|
||||
# 179. The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail
|
||||
|
||||
## Context
|
||||
|
||||
Read on the control node on 2026-10-02, the day the machines were confirmed filtered by the mesh
|
||||
alone ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)): the intrusion
|
||||
prevention watched one door. Its two jails read the ssh daemon's journal and its own log, banned
|
||||
five failures in ten minutes for ten minutes, and in a day had seen twelve thousand failed logins
|
||||
from three hundred addresses and banned none of the busiest, which paced themselves at one try every
|
||||
ten minutes. The mail submission port took a hundred and sixty password guesses in the same day from
|
||||
thirty-eight addresses with no jail reading it at all; the forge and the public proxy had no jail
|
||||
either, and the proxy logged nothing a jail could read. Nobody could see the jails without a shell:
|
||||
the module's three tools existed in code and were served by nothing, and the seat it holds declared
|
||||
no verbs.
|
||||
|
||||
Three things were missing and they are three shapes the mesh already has. The packet filter's seat
|
||||
serves verbs every holder owes ([ADR 0170](0170-the-firewall-seat-serves-its-verbs.md)); the
|
||||
intrusion seat serves none. A module's `listens` compose into the machine's filter, and [to-be 31](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
|
||||
says a module's `jails` compose into the machine's intrusion prevention the same way — the controller
|
||||
composes them, and no module declares one. And a jail reads a log; a container's output goes to a
|
||||
file of the runtime's own under a path that changes when the container is recreated, which is why
|
||||
no jail could read the mail front end, the forge or the proxy, however they logged.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. The `node-intrusion-prevention` seat serves four verbs**, and a module that claims it serves
|
||||
all four or is refused the claim, as with every seat:
|
||||
|
||||
- `status` — every jail with what it watches, how many addresses it is counting failures against and
|
||||
holding now, and the totals since it started; one jail's detail when named. Read-only.
|
||||
- `banned` — every address banned now, with the jail holding it, when it was banned and when the ban
|
||||
ends. Read-only.
|
||||
- `ban` — ban one address in one jail now, for that jail's ban time. An operator's act on the live
|
||||
ban list, which the mesh composes the rules for and never writes itself.
|
||||
- `unban` — let one address go, from one jail or from every jail.
|
||||
|
||||
A holder may serve its own tools beside these; the fail2ban module reads one jail's effective
|
||||
settings as its own.
|
||||
|
||||
**2. A container may log to the journal.** `logging: journald` on a container has the host run it
|
||||
with the journal as its log driver; the journal keeps the container's name on every line, and
|
||||
`docker logs` keeps working. Where a container logs is part of its spec, so moving it recreates the
|
||||
container, and the only place besides the runtime's own file is the journal: a machine's intrusion
|
||||
prevention reads the journal already, for the ssh daemon, and a container that logs there is read
|
||||
the same way, by the container's name, whatever the container is called by the runtime this time.
|
||||
|
||||
**3. A module with a door declares its jail, and the holder composes them.** What to-be 31 designed
|
||||
is now the rule: a module whose service authenticates from outside — the mail front end, the forge,
|
||||
the public proxy — declares in its manifest what a failed attempt looks like in its log and how to
|
||||
ban on it, naming no node and no path; the module that holds the intrusion seat declares where the
|
||||
composed jails and filters land, and the mesh writes them on every machine that runs both. A machine
|
||||
not running the module has no such jail. The holder restarts its daemon on the composed file.
|
||||
|
||||
**4. The base is strict, and the mesh's own range is never banned.** Three failures in a day ban for
|
||||
a day, on every jail unless the jail says otherwise; banned twice in two weeks, by any jail, is
|
||||
banned for four. The attackers this mesh sees pace themselves under any ten-minute window; a day's
|
||||
window counts them. A person who mistypes three times from one address is out for a day from that
|
||||
address, and never from a machine of the mesh, whose range stays in the never-banned list the module
|
||||
has carried since [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md). The operator
|
||||
chose this knowing it.
|
||||
|
||||
**5. The proxy says a refused name in its log.** A request for a name this mesh does not serve, from
|
||||
outside, is what a scanner does; the proxy already logged a certificate refused for such a name, and
|
||||
now logs the plain request too, with the asking address last, as its own jail's filter expects it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The seat's row gains four verbs; a mesh that already runs widens its row at the next controller
|
||||
start. The fail2ban module claims them and gains a runtime — a tool server whose image carries the
|
||||
fail2ban client, with the daemon's socket shared in from the machine, and nothing else of the
|
||||
machine. The daemon stays the machine's; what runs in the container is only the client.
|
||||
- **That runtime is the shape the catalogue has today, and it is on its way out.**
|
||||
[ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md),
|
||||
accepted the same day as this record, replaces a tool container per module with one tool runtime
|
||||
per node on the host side, taking each module's tools as a bundle. Nothing here depends on the
|
||||
container: the verbs, the client that speaks to the daemon over its socket, and the jails are the
|
||||
same code under either. This module converts with the packet filter's, whose runtime that record
|
||||
names, and the socket it needs becomes the node runtime's to reach rather than a mount of its own.
|
||||
- The host's container vocabulary grows by `logging`; an older host refuses a declaration that carries
|
||||
it, so the host rolls before the modules. Three containers are recreated once, when their modules
|
||||
are pushed with the field: the mail front end, the forge and the proxy — each a moment's outage.
|
||||
- The fail2ban module declares where jails compose (`jailing`) and the directory the filters go in;
|
||||
the mail, forge and proxy modules each declare one jail reading the journal by their container's
|
||||
name. The composed jail file is the one resource the daemon restarts on when a module arrives or
|
||||
leaves a machine.
|
||||
- The two base jails and the composed ones take the day's window; the ssh jail's ten minutes are
|
||||
gone. An address banned on the first day of this record stays banned for the day.
|
||||
- The module's three old tools, served by nothing, are replaced by the seat's four verbs and one
|
||||
own tool; `fail2ban_status` as a name is gone.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The seat declares the four verbs; a claim that serves fewer is refused by name | the catalogue's seat tests |
|
||||
| `status`, `banned`, `ban` and `unban` read and steer the daemon through its client, with the shapes fail2ban 1.1.0 printed live; a non-address and a non-name are refused before anything runs | the module's tests over a fake command runner |
|
||||
| A container's `logging` reaches the runtime's arguments and its spec; a place other than the journal is refused | host tests |
|
||||
| A module's jails compose into the holder's file and a filter per jail, and the file is written empty when none is declared | the controller's composition tests (to-be 31) |
|
||||
| The proxy logs a refused name with the address last | the proxy's tests |
|
||||
| A jail's pattern names `<HOST>` once per shape, since two is a duplicate capture group and costs the machine every ban | the catalogue's manifest tests |
|
||||
| Live | done 2026-10-02: `status` and `banned` answered on both servers through the console; the proxy's jail counted seven refusals on the home server; a documentation address banned in the ssh jail came back with its end time and was released |
|
||||
|
||||
## Built and proven live, 2026-10-02
|
||||
|
||||
All five rules are in the mesh. The host carries `logging`; the controller's seat row carries the four
|
||||
verbs and the proxy says a refused name in its log; the fail2ban module holds the seat from a runtime
|
||||
with the daemon's socket shared in, composes the jails, and the mail front end, the forge and the
|
||||
proxy each declare one. Through the console on the control node: `status` listed five jails with what
|
||||
each watches, `banned` listed the nine the long jail holds, and a documentation address banned in the
|
||||
ssh jail came back with its ban's end time and was released again. On the home server the proxy's jail
|
||||
had counted seven refusals within minutes of starting.
|
||||
|
||||
**One fault, found by the machine and not by a test.** The proxy's pattern matched two shapes of
|
||||
refusal in one expression and so named `<HOST>` twice. fail2ban expands that placeholder into a named
|
||||
capture group; two of them is a duplicate group name, and the daemon refuses *its whole configuration*
|
||||
and exits — both servers kept no bans at all for about ten minutes, every jail and not the one at
|
||||
fault. The pattern is now one per shape. A manifest check refuses the mistake at merge time, naming
|
||||
what it would cost, which is the only reason this record can claim the rule rather than the instance.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0170](0170-the-firewall-seat-serves-its-verbs.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
- [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md), [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md), [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md)
|
||||
@@ -60,7 +60,7 @@ configuration file; the rollback path it described is given up on purpose.
|
||||
|---|---|
|
||||
| An absent package is removed when present, left when not, and read back | host tests over a fake package manager |
|
||||
| An uninstalled front end is recorded as removed and nothing is asked of it | a host test with ufw missing on a converged apply |
|
||||
| Live | the two machines report ufw gone: `pacman -Q ufw` has no answer, `node show` says removed, `status` is well |
|
||||
| Live | done 2026-10-02: both machines report ufw gone — `pacman -Q ufw` has no answer, `node show` says *removed*, `status` is well. The home server said *retired* for six hours after the package went, because this record's step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)) and one dead tracker was failing its applies ([ADR 0187](0187-a-dead-tracker-is-not-the-machines-failure.md)) |
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 184. A service the mesh asked to run is still running a moment later
|
||||
|
||||
## Context
|
||||
|
||||
The host already refuses to take a service manager's word for it. Three places in one function read
|
||||
a unit back after acting on it, each with a comment saying why: *a service manager accepting a
|
||||
command says the transaction was accepted, not that the unit is running — one that starts and
|
||||
immediately dies satisfies it.* The intent was right and the implementation did not reach it.
|
||||
|
||||
On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the
|
||||
files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at
|
||||
that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public
|
||||
machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said
|
||||
so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing.
|
||||
|
||||
The read-back races the failure. A service manager returns when it has started the process; a daemon
|
||||
that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One
|
||||
look sees `activating` or `active` whatever the process is about to do, and *the host reports success
|
||||
for a machine that is already wrong* — the one shape of failure this host exists to refuse
|
||||
([ADR 0005](0005-the-node-host.md)).
|
||||
|
||||
A command the module declares — *test the configuration before restarting* — was considered and
|
||||
rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification
|
||||
command is a command: a declaration that carried one would be remote execution over the bus,
|
||||
arriving as root on every machine, which is a far larger door than the fault it closes. The host
|
||||
does not need one. It already knows what it asked for.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A unit the host has just asked to run is read twice**, with a pause between the reads long
|
||||
enough for a daemon that refuses its configuration to have exited. Not running at the second look is
|
||||
a failure of that resource, named with the unit and the state it is in — the same failure the single
|
||||
read was always meant to catch.
|
||||
|
||||
**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both
|
||||
looks and is accepted, exactly as before. What the second look catches is a unit that *was* running
|
||||
and is not any more. A service asked to be stopped is not waited on at all.
|
||||
|
||||
**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking
|
||||
the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and
|
||||
no command reaches a machine that did not already come from a built artifact.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is
|
||||
bounded by the number of services that changed in that apply, which is usually none.
|
||||
- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the
|
||||
resolver — now fails its apply when the composition is bad, instead of reporting success onto a
|
||||
dead daemon. `status` names the machine, which is how the operator finds out.
|
||||
- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s
|
||||
manifest check is what refuses the one that caused this, at merge time; this record is what makes
|
||||
the *next* one visible within a minute rather than invisible until something asks the daemon a
|
||||
question.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does |
|
||||
| A unit still starting is accepted at both looks | a host test |
|
||||
| A service asked to be stopped is not waited on | a host test |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
|
||||
---
|
||||
|
||||
# 185. A control plane behind its seat's row serves what it can
|
||||
|
||||
## Context
|
||||
|
||||
The mesh's own verbs are the controller seat's tools, and the seat's row is the store's
|
||||
([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). A control plane reads the
|
||||
row at start and installs a handler per verb; a verb the row carries that the binary cannot run was
|
||||
refused at start rather than at the first call, so that a disagreement between the row and the
|
||||
binary was said early. The refusal aborted the start.
|
||||
|
||||
On 2026-10-02 a merge added one verb. The new control plane started, widened the row, and ran. A
|
||||
push a few seconds later recreated its container at the previous image — a stale declaration from
|
||||
an overlapping wave, [issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md) —
|
||||
and the older binary read a row naming a word it had never heard. It refused to start, and kept
|
||||
refusing. The mesh had no voice for ten minutes: no verb answered, no node could be pushed, no build
|
||||
was dispatched, and `status` said nothing because `status` is one of the verbs that had stopped
|
||||
being served. The way back was a person running the binary by hand outside its service, because the
|
||||
push that would have replaced it is itself a verb of the control plane that was down.
|
||||
|
||||
The check was right about the fact and wrong about the cost. A row ahead of a binary is the ordinary
|
||||
state of a roll-out: the row is widened by whichever control plane starts first, and a mesh with one
|
||||
control plane sees that gap on every merge that adds a verb. Making it fatal turned a transient into
|
||||
an outage with no path out that did not need a human.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A control plane serves the verbs it can run and does not refuse to start for the ones it
|
||||
cannot.** The row remains the authority on what the seat serves; this is only about what this binary
|
||||
does when it is behind the row.
|
||||
|
||||
**2. A verb it cannot run answers the reason.** Not silence and not a missing subject: a caller gets
|
||||
a sentence naming the verb, saying this control plane cannot run it and that it is a verb of a newer
|
||||
build. A verb that is simply absent from the row is still not served at all — that is the row
|
||||
deciding, which is unchanged.
|
||||
|
||||
**3. It says so once at start**, naming every verb of the row it cannot run, so the gap is visible
|
||||
in the log of the thing that has it rather than only at the moment somebody calls one.
|
||||
|
||||
**4. A mesh with no controller seat at all is still a refusal.** That is not a version gap, it is a
|
||||
mesh that has not been seeded, and nothing this control plane does would be meaningful.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An overlapping roll-out costs the verbs the newer build added, for as long as the older binary is
|
||||
in place. Everything else — every push, every build, every read — keeps working, and the ordinary
|
||||
machinery that notices a machine is behind is what puts the newer binary back.
|
||||
- The log gains one line on a control plane that is behind, and nothing on one that is not.
|
||||
- Issue 201's other half remains: the push that sent a stale declaration is a race worth closing on
|
||||
its own terms. This record makes that race survivable rather than fatal, which is the difference
|
||||
between a transient and an outage, and is deliberately the cheaper half.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A row carrying a verb this build cannot run still serves every verb it can, and names the one it cannot | a controller test over a widened row |
|
||||
| The unknown verb answers a sentence naming itself and saying this build is behind | the same test |
|
||||
| A mesh with no controller seat is refused | the existing start-up path |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
||||
- [Issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)
|
||||
- [Design 33 — The tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
---
|
||||
|
||||
# 186. A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md) gave the
|
||||
public proxy a jail. Within the hour the home server's ban list held `192.168.1.1` — the house's own
|
||||
router. The router reflects local traffic, so every client in the building reaches that machine as
|
||||
the gateway's address; one local request for a name the mesh does not serve, three times in a day,
|
||||
and the whole house is refused by the machine it was asking. The jails inherited an `ignoreip` of
|
||||
the loopback and the mesh's own range, which was right when the only jail read the ssh daemon and
|
||||
the only clients were the mesh's; a jail on a public front door sees the neighbours too.
|
||||
|
||||
The same jail broke the other half of [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md).
|
||||
The home server began reading *NOT the mesh alone: 1 rule set the mesh did not write refuses traffic
|
||||
here*, and the rule set named was the mesh's own ban chain, written by the mesh's own intrusion
|
||||
prevention minutes earlier. The host's reader of the legacy filter required every path into a chain
|
||||
of refusals to come from a built-in chain whose policy accepts, before it would call that chain a
|
||||
ban. On that machine the chain hangs off the container runtime's user chain as well as the input
|
||||
chain, and the runtime had set the forward policy to DROP — so the mesh reported its own work as a
|
||||
foreigner's, on the one machine where the group's exit condition was supposed to hold.
|
||||
|
||||
Both faults are one mistake in two places: a rule written about the public internet, applied to
|
||||
everything that arrives.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A ban list never holds a neighbour.** The jails the mesh composes never ban a source on a
|
||||
private range — the mesh's own range, which was already named rather than written
|
||||
([ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md)), and every address space
|
||||
reserved for private use beside it, in both families. A machine behind a router that reflects local
|
||||
traffic sees its whole building as one address; a ban there is a self-inflicted outage, and the
|
||||
sources worth banning are not on those ranges in the first place.
|
||||
|
||||
**2. The mesh's own bans are its own wherever they hang.** A chain of refusals is a ban list when
|
||||
every refusal names the sources it refuses and the chain accepts nothing — the rule the host already
|
||||
applied to the packet filter's own tables, now applied to the legacy filter too, and nothing more.
|
||||
The policy of the chains that jump into it says nothing about what it is: that policy is already
|
||||
classified where it belongs, as the container runtime's, and requiring it here counted it twice.
|
||||
|
||||
**3. A chain that accepts anything is still not a ban.** That is what keeps a predecessor's
|
||||
allow-these-and-drop-the-rest chain classified as something an operator must look at, which is the
|
||||
distinction [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md) exists to draw.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The composed jails gain the private ranges in their never-ban list. An address already banned
|
||||
stays banned until it is released; the house's router was released by hand the moment it was found.
|
||||
- The home server reads *the mesh alone* again, which is group 7's exit condition and was false for
|
||||
about an hour.
|
||||
- A machine whose apply fails for an unrelated reason does not revisit its found firewall's record
|
||||
at all — the step runs only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)).
|
||||
The home server's record therefore still reads *retired by the mesh* although the front end is
|
||||
uninstalled, and will correct itself once that machine's own stuck module is fixed. It is a stale
|
||||
record, not a wrong machine.
|
||||
- The record number the front end's removal was given moved under it: another session took 0175
|
||||
while that record was in review, and it is now
|
||||
[ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md). The citations
|
||||
the host and the control plane print were pointing at an unrelated record and are corrected here.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A private source is never banned | the module's jail configuration, read back by `fail2ban.fail2ban_settings` on a machine |
|
||||
| The mesh's own ban chain reads as a ban behind a dropping forward policy | a host test over the home server's own captured rule set |
|
||||
| A chain that accepts anything is not a ban | a host test |
|
||||
| Live | done 2026-10-02: all four machines read *the mesh alone*, the home server counting its own ban chain as a ban; no ban held anywhere is a private address |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
|
||||
- [Design 08 — Connectivity](../03-DESIGN/01-to-be/08-connectivity.md), [Design 31 — A module declares its fail2ban jail](../03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
|
||||
---
|
||||
|
||||
# 187. A dead tracker is not the machine's failure
|
||||
|
||||
## Context
|
||||
|
||||
The home server had not applied a declaration cleanly since midday. One run-once step — the one
|
||||
that writes a media app's download clients and indexers through the app's own API — exited
|
||||
non-zero, forty-nine times over six hours, for one public tracker that had stopped answering. The
|
||||
step's own words: the entry was *written*, and the app's test of it then failed with a 400 from the
|
||||
indexer proxy. The machine reported *not doing what it was told* for the rest of the day.
|
||||
|
||||
What that gated matters more than the step. A converged machine retires the firewall it was found
|
||||
with only after a clean apply ([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)),
|
||||
so that machine went on recording its found front end as merely *retired* long after the package
|
||||
had been uninstalled ([ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)).
|
||||
A dead public tracker was holding a firewall record hostage, which is not a connection anybody
|
||||
would design.
|
||||
|
||||
The step already knew this was not its business. It had a rule for exactly this: an entry the mesh
|
||||
only *found and re-pointed*, rather than one it was told to make, whose feed is gone, is said and
|
||||
left as found — *failing the node's apply on every heartbeat for it reports the mesh as wrong about
|
||||
a tracker*. The rule was there and matched one shape of the fault. An app can refuse to save such an
|
||||
entry, and it can save it and then fail its own test; saving validates settings, and the test runs a
|
||||
live search. The rule caught the first and let the second through.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. An entry the mesh only found is never the machine's failure.** Whatever shape the app's
|
||||
refusal takes — it would not save it, or it saved it and its own test fails — an indexer the mesh
|
||||
found and re-pointed is reported as a notice and left as found. What decides is whose entry it is,
|
||||
not which sentence the app returned.
|
||||
|
||||
**2. What the mesh is answerable for is the plumbing.** That the entry exists, points at this
|
||||
mesh's indexer proxy, and carries the credential the mesh delivered — which was checked against the
|
||||
proxy before anything was written. Whether a public tracker answers today is not the mesh's to
|
||||
promise, and a machine that reports itself broken because one did is lying about itself.
|
||||
|
||||
**3. An entry the operator listed is theirs to insist on.** An indexer named in the step's settings
|
||||
is one the mesh was told to make, and it still fails the step when it cannot be made to work. The
|
||||
notice says so, and says that listing the indexer is how to turn it back into a failure.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The home server applies cleanly again, and everything a clean apply gates — its found firewall's
|
||||
record among it — follows.
|
||||
- A tracker that dies is a line in a report rather than a machine that reads as broken. An operator
|
||||
who wants it gone removes the entry or repairs the feed; the mesh says which, every time it runs.
|
||||
- The four Servarr modules carry one byte-identical copy of this step each
|
||||
([ADR 0069](0069-a-module-is-a-repository-and-a-path.md)), so the change lands in four places and
|
||||
a test refuses any drift between them.
|
||||
- It does not widen to a download client: one the mesh was told to write and cannot is still a
|
||||
failure, because the mesh chose it and nothing else will fix it.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A found feed whose tracker answers an error after the entry was written is a notice | the step's tests, with the home server's own message and the app's two validations modelled apart |
|
||||
| An indexer the settings list is still a failure | the same test |
|
||||
| The four copies of the step do not drift | the step's own sameness test |
|
||||
| Live | done 2026-10-02: the home server applies cleanly after six hours of failing, `status` holds no machine wrong or behind, and its found firewall reads *removed* |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0136](0136-a-step-gates-its-module-not-the-machine.md), [ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md), [ADR 0180](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md), [ADR 0069](0069-a-module-is-a-repository-and-a-path.md)
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
|
||||
---
|
||||
|
||||
# 188. A provider declares what it derives for each consumer, and the mesh tells both ends
|
||||
|
||||
## Context
|
||||
|
||||
An arrangement between a consumer and a provider is delivered entirely by the mesh. Where the
|
||||
provider is, which port it answers on, what name the consumer must present, where its password
|
||||
is — each arrives as a fact the consumer reads from its binding, or as `${bound:…}` filled into a
|
||||
file before the declaration leaves the control plane. The provider invents none of it and hands
|
||||
none of it back ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)).
|
||||
|
||||
One kind of value escapes that. Where the **provider names the resource** — a bucket, a database,
|
||||
a vhost — the name is derived from the consumer, per consumer, and the mesh has no way to carry
|
||||
it. `serves` is a literal block in the provider's definition: the same values for every consumer.
|
||||
A provisioner's contract takes a provision and returns nothing. So a value the mesh's own rule
|
||||
produced reaches neither end as a statement; it is recomputed at one end and transcribed at the
|
||||
other.
|
||||
|
||||
The object store is the instance ([issue 124](../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)).
|
||||
Its provisioner normalises the login the mesh minted into a bucket name and creates, checks and
|
||||
removes exactly that; the rule lives in twenty lines of the module's own TypeScript. Its three
|
||||
consumers each write the answer into their own definition by hand. Two transcribed it correctly;
|
||||
one named a predecessor's bucket, and would have authenticated successfully and been refused on
|
||||
every object, which reads like a credential fault and is not one.
|
||||
|
||||
Even corrected, the transcriptions are wrong in a second way. Each is `mesh-<node>-<slug>`, so
|
||||
each **names the machine the module happens to run on today** — a definition stating a fact about
|
||||
one installation, which [ADR 0155](0155-a-definition-names-no-installation-and-how-that-is-checked.md)
|
||||
forbids and whose check does not catch because the name is not a domain. Move any of the three to
|
||||
another machine and its configuration points at a bucket its key cannot open.
|
||||
|
||||
The shape is not the object store's. A database provisioner that prefixed names, a queue provider
|
||||
that scoped vhosts, any provider that derives a resource from who is asking: each forces the
|
||||
consumer to reproduce somebody else's rule and keep it in agreement by hand.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. A served value may name the consumer the mesh is serving.** A `serves` block, which is
|
||||
literal today, may interpolate the mesh's own statement of who the consumer is:
|
||||
|
||||
- `${consumer:as}` — the identity the mesh minted for this consumer, exactly as the login it is
|
||||
told to present ([ADR 0049](0049-a-consumers-identity-fits-the-tightest-backend.md));
|
||||
- `${consumer:as:dns}` — the same identity written as a DNS label.
|
||||
|
||||
Nothing else. **The mesh learns no protocol here; it spells its own name in an alphabet it already
|
||||
knows.** The identity is the mesh's, minted by the mesh, already capped at twenty characters
|
||||
because of what an S3 access key accepts; `dns` is that same name with its separator written `-`
|
||||
instead of `_`, which is the whole of the difference between the mesh's identifier alphabet and
|
||||
the one buckets, vhosts and hostnames use. A provider that needs a prefix or a suffix writes it
|
||||
around the placeholder, because a served value is a string.
|
||||
|
||||
The rejected alternative is **the provider returning values from provisioning** — the natural
|
||||
channel, since the provider is what derived them. It is rejected for three reasons, in order of
|
||||
weight. It inverts the delivery the mesh is built on: a grant would carry data the provider wrote
|
||||
rather than only data the mesh minted, and [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)
|
||||
removed exactly that second path once already. It makes a consumer's declaration incomplete until
|
||||
its provider's reconcile loop has run, so a consumer could not be composed before a provider
|
||||
answered — a bootstrap order the mesh does not have and does not want. And it puts the rule where
|
||||
nothing can check it: a value that arrives from a running process cannot be refused at resolution,
|
||||
only discovered wrong later, which is the failure this record exists to end.
|
||||
|
||||
**2. The mesh resolves it once, per consumer, and tells both ends from the one resolution.** At the
|
||||
moment a consumer's declaration is composed, the mesh knows exactly who the consumer is. There, and
|
||||
only there, the placeholders are filled. The result reaches:
|
||||
|
||||
- the **consumer**, as the served facts in its binding file and as `${bound:<provision>:<key>}` in
|
||||
any file it writes — unchanged mechanisms, carrying one more key;
|
||||
- the **provider**, as `serves` on that consumer's entry in its contributions file, so the
|
||||
provisioner is *told* the name rather than recomputing it.
|
||||
|
||||
**The provider stops deriving in code and starts declaring.** One statement, filled once, delivered
|
||||
to both ends: the two cannot disagree, because there is no second computation to disagree with.
|
||||
|
||||
**3. A served value stays settled before it is per-consumer.** Settings still compose into `serves`
|
||||
([ADR 0174](0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)), and the consumer
|
||||
placeholders are filled after that, so an operator may set a prefix and the mesh still derives the
|
||||
rest. A `${consumer:…}` naming a fact or an alphabet the mesh does not have is refused when the
|
||||
definition is parsed, with what it may say.
|
||||
|
||||
**4. A consumer may no longer name the resource its provider derives.** With the value delivered,
|
||||
a literal in a consumer's definition is not merely redundant — it is the one thing that can
|
||||
disagree with what the provider will actually create. The three object-store consumers lose their
|
||||
hand-written bucket names in this change.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One more thing a definition may say, and one less thing a module may be wrong about. The
|
||||
vocabulary grows by a placeholder; the catalogue loses three literals that named this
|
||||
installation's control node.
|
||||
- A provider's naming rule becomes readable in its definition instead of in its source. `minio`'s
|
||||
`bucketFor` goes; the manifest says `"bucket": "${consumer:as:dns}"` and the provisioner uses
|
||||
what it is given.
|
||||
- A provider that already serves consumers keeps serving them: the derived value equals what the
|
||||
code derived, so no bucket, database or login changes name. This is a change of **who says it**,
|
||||
not of **what is said**.
|
||||
- The mesh now holds a rule in another system's alphabet — one rule, `dns`, stated once. A second
|
||||
alphabet is a decision, not an addition: the cost of each is that the mesh must be right about
|
||||
somebody else's naming, and that cost is only worth paying where the mesh already mints the name.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- A served value naming an unknown fact or alphabet is refused at parse, with the list of what it
|
||||
may say — tested on both halves of the message.
|
||||
- Resolving a consumer whose provider derives a value puts that value in the consumer's binding
|
||||
file, in its `${bound:…}` substitutions, and in the provider's contributions entry for that
|
||||
consumer — one test asserting the three agree, because agreeing is the whole point.
|
||||
- Two consumers of one provider on one machine get two different derived values, and neither gets
|
||||
the other's.
|
||||
- A catalogue-wide test refuses a consumer definition that writes a literal where its provider
|
||||
derives: the provider's `serves` names the key, so the catalogue can say which definitions
|
||||
transcribe one.
|
||||
- `dns` is checked against the identity the mesh actually mints, not against an invented string:
|
||||
the test derives an identity with `ConsumerIdentity` and asserts the label it becomes.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 124 — a consumer cannot be told a value its provider derived for it](../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)
|
||||
- [ADR 0048 — a provider creates the credential the mesh minted, and seals nothing](0048-a-provider-creates-the-credential-the-mesh-minted.md)
|
||||
- [ADR 0049 — a consumer's identity fits the tightest backend](0049-a-consumers-identity-fits-the-tightest-backend.md)
|
||||
- [ADR 0174 — a node varies a module through settings and kept regions, never through an edit](0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)
|
||||
- [ADR 0155 — a definition names no installation, and how that is checked](0155-a-definition-names-no-installation-and-how-that-is-checked.md)
|
||||
- [design 27 — a module requires, the mesh resolves](../03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md)
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-02
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
|
||||
---
|
||||
|
||||
# 189. The store keeps what the records name, and a maintenance step holds its writers still
|
||||
|
||||
## Context
|
||||
|
||||
The mesh's artifact store has never collected anything
|
||||
([issue 108](../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md)).
|
||||
Every build pushes another layer set; nothing has ever removed one. The predecessor ran a routine
|
||||
on a timer — stop the registry, collect, start it — and the conversion carried the settings that
|
||||
routine depends on without the routine, because the routine was a script beside the module and not
|
||||
a resource in it. The store now holds fifty-three repositories on the machine that serves
|
||||
everything else, and the only outcome of leaving it is a full disk reported as somebody else's
|
||||
failure.
|
||||
|
||||
Three things stood in the way, and the issue names all three.
|
||||
|
||||
**Nothing in the mesh's vocabulary expresses a maintenance window.** The collector requires every
|
||||
writer stopped while it runs. A `run-once` step runs *beside* containers, not instead of them, and
|
||||
a scheduled step is the same container on a cadence. There is no way for a module to say *hold this
|
||||
container of mine still while this runs*.
|
||||
|
||||
**Deletion is not enabled, and the door it would be enabled on has no accounts.** The store is
|
||||
internal, reached by name over the overlay, trusted because being on that network is the permission
|
||||
([ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)). The predecessor
|
||||
kept deletion behind an authenticated door, which it could, having one.
|
||||
|
||||
**Nothing says what may be removed.** The registry's own answer — collect everything no tag names —
|
||||
is wrong here. The mesh pushes each artifact under one moving tag and pins machines by digest, so
|
||||
every build but the newest is untagged and some machine may still be running it.
|
||||
|
||||
## Decision
|
||||
|
||||
**1. Deletion is enabled on the store's one door, and the overlay stays the permission.** The
|
||||
objection dissolves on inspection: that door **already accepts a push**, and a writer who can push
|
||||
can replace any tag in the store with anything it likes. Delete takes nothing a push did not
|
||||
already have, and the machines that can reach the door are the ones the mesh's own filter admits
|
||||
([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)). Putting an authenticated
|
||||
door in front of deletion while leaving push open would be a lock on the window beside an open
|
||||
door, and it would cost the thing ADR 0082 bought: a store every machine can reach without a
|
||||
credential to distribute first.
|
||||
|
||||
**2. The mesh deletes what it made and no longer keeps; the store reclaims the bytes.** Two halves,
|
||||
each doing what only it can.
|
||||
|
||||
The **mesh** decides. It does not need to enumerate the store to do it — it has never put anything
|
||||
there it did not record, so **every digest it could remove is already in its own build records**.
|
||||
It deletes those manifests through the store's door, by digest, and remembers that it did.
|
||||
|
||||
The **store** reclaims. A deleted manifest frees no bytes until the registry's own collector walks
|
||||
the storage with nothing writing to it, so the module declares that collector as a scheduled step
|
||||
with the server held still for its duration. Plain collection, not `--delete-untagged`: what the
|
||||
mesh keeps is still a manifest in the store, so it is still referenced, so its blobs stay — the
|
||||
dangerous flag is not needed at all once the mesh is the one deciding.
|
||||
|
||||
**3. What the mesh keeps, stated as three reasons rather than a number.** A digest is kept because:
|
||||
|
||||
- **a definition names it** — every artifact reference in any module's current recorded manifest,
|
||||
which is what the mesh would hand a machine now. No age limit: this is the floor;
|
||||
- **the mesh can still go back to it** — every artifact of the **five most recent successful
|
||||
builds** of each module, so a release that turns out wrong has somewhere to return to;
|
||||
- **nothing else.** An artifact older than that, which no definition names, is what the store is
|
||||
carrying for no stated reason.
|
||||
|
||||
A digest the mesh did not record making is never touched. That is not a safety margin, it is the
|
||||
whole rule restated: the mesh removes what it put there and can account for, and the images genesis
|
||||
pushed before any record existed are exactly what this must not reach
|
||||
([04-ISSUES/102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md), F4).
|
||||
|
||||
**4. A scheduled step may hold its module's own containers still while it runs** —
|
||||
`while-stopped`, naming resource ids in the same module. The host stops each, runs the step, and
|
||||
starts them again **whatever the step did**, including when it failed or the host was interrupted.
|
||||
Three boundaries:
|
||||
|
||||
- **Its own module's containers only.** A module that could quiesce a neighbour could stop the
|
||||
mesh; a maintenance window is a statement about one service's own insides.
|
||||
- **Scheduled steps only, not `run-once`.** At apply time the host already has a window: the
|
||||
declaration is applied in order and a step gates what follows, so a one-time offline migration
|
||||
says *before* rather than *instead of*. A recurring window is the case order cannot express.
|
||||
- **Restoring is not conditional.** A step that fails must leave the service running; the whole
|
||||
risk of this field is a window that never closes.
|
||||
|
||||
**5. The sweep runs where the records change — after a build the mesh recorded.** That is the
|
||||
moment new bytes landed and the moment the keep set moved, and it needs no new timer. The
|
||||
store's collection runs nightly, because reclaiming is slow and the thing it reclaims is already
|
||||
unreferenced.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Disk stops growing without bound on the machine that serves the mesh. That is the whole point
|
||||
and it has no other way to be true.
|
||||
- A machine behind by more than five builds of a module, which recreates a container, cannot pull
|
||||
what it was running. It is already a machine the mesh reports as behind, and the answer is the
|
||||
one the mesh already gives it: the current declaration. Stated here rather than discovered.
|
||||
- The store is a little less of a museum. A digest in an old build record may no longer be
|
||||
fetchable, and the record still says what that build made — the record is history, not an
|
||||
index of what is on disk. The collected mark is kept beside it so the two can be told apart.
|
||||
- `while-stopped` is a second thing the host does to a container it did not start this pass. It is
|
||||
deliberately the narrowest form: the module's own, by id, restored unconditionally.
|
||||
- The store is briefly unavailable each night, for as long as collection takes. Everything that
|
||||
pulls from it retries; nothing in the mesh treats a momentary store as a failure
|
||||
([ADR 0185](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)).
|
||||
|
||||
## How this is checked
|
||||
|
||||
- The host: a scheduled step with `while-stopped` stops the named containers before the run and
|
||||
starts them after; it starts them again **when the step fails**; it refuses an id that is not a
|
||||
container of the same module, its own id, and `while-stopped` on a `run-once` step. Each refusal
|
||||
is tested for what it says, not only that it says something.
|
||||
- The controller: given build records and current manifests, the keep set holds every reference a
|
||||
manifest names and every reference of the five most recent builds per module, and nothing else;
|
||||
a reference the mesh never recorded is never in the delete set; a delete that answers 404 is
|
||||
recorded as collected rather than retried forever.
|
||||
- The sweep is tested against a fake store that records what it was asked to delete, so what is
|
||||
asserted is the decision and not the registry's behaviour.
|
||||
- Live: the store's size before and after the first nightly collection, read from the machine.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 108 — the registry has no garbage collection](../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md)
|
||||
- [ADR 0082 — the registry is reached by name and trusted by the overlay](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)
|
||||
- [ADR 0053 — a step that runs on a schedule](0053-a-step-that-runs-on-a-schedule.md)
|
||||
- [ADR 0156 — an artifact is what a build produces, and the store is named for its scope](0156-an-artifact-is-what-a-build-produces-and-the-store-is-named-for-its-scope.md)
|
||||
- [design 32 — what a module declares](../03-DESIGN/01-to-be/32-what-a-module-declares.md)
|
||||
@@ -182,7 +182,14 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0169** — [A machine joins through the tunnel, and the bus is never public](0169-a-machine-joins-through-the-tunnel-and-the-bus-is-never-public.md)
|
||||
- **0170** — [The firewall seat serves its verbs, and a foreign rule set is removed through one of them](0170-the-firewall-seat-serves-its-verbs.md)
|
||||
- **0172** — [The lab is a module, and runs a bed when the mesh asks](0172-the-lab-is-a-module-and-runs-a-bed-when-the-mesh-asks.md)
|
||||
- **0179** — [The intrusion seat serves its verbs, a container may log to the journal, and every door declares its jail](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
- **0180** — [The found front end is uninstalled once a machine is converged](0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md)
|
||||
- **0184** — [A service the mesh asked to run is still running a moment later](0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md)
|
||||
- **0185** — [A control plane behind its seat's row serves what it can](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)
|
||||
- **0186** — [A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang](0186-a-ban-list-never-holds-a-neighbour.md)
|
||||
- **0187** — [A dead tracker is not the machine's failure](0187-a-dead-tracker-is-not-the-machines-failure.md)
|
||||
- **0188** — [A provider declares what it derives for each consumer, and the mesh tells both ends](0188-a-provider-declares-what-it-derives-for-each-consumer.md)
|
||||
- **0189** — [The store keeps what the records name, and a maintenance step holds its writers still](0189-the-store-keeps-what-the-records-name.md)
|
||||
|
||||
### Its tiers, from the bottom up
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ status: in-progress
|
||||
code: [mesh-host]
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md
|
||||
- 02-DECISIONS/0168-a-converged-machine-is-filtered-by-the-mesh-alone.md
|
||||
- 02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md
|
||||
- 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
|
||||
@@ -498,3 +499,17 @@ run and reported; the exit follows an in-flight apply rather than interrupting i
|
||||
not start is rolled back once and the second failure halts; a completed reconcile retires what is older
|
||||
than the predecessor and never the predecessor; and the newest of two delivered versions is the one
|
||||
that runs.
|
||||
|
||||
## A service is still running a moment later, 2026-10-02
|
||||
|
||||
[ADR 0184](../../02-DECISIONS/0184-a-service-the-mesh-asked-to-run-is-still-running-a-moment-later.md).
|
||||
The host has always read a unit back after acting on it, because a service manager accepting a
|
||||
command says the transaction was accepted and nothing about the process. The read raced the failure:
|
||||
a daemon that refuses the configuration the mesh just wrote exits a fraction of a second after the
|
||||
manager returns, and one look sees it alive. So the host looks twice, with a pause between, and a
|
||||
unit that was running and is not any more fails its resource by name. A unit still coming up reads
|
||||
as running at both looks and is accepted; a service asked to stop is not waited on.
|
||||
|
||||
No command for this reaches a machine. A module declaring *how to test my configuration* was weighed
|
||||
and refused: the link carries no actions, and a verification command is one. The host is checking
|
||||
the state it was told to establish, which is what it is for. *How it is checked:* ADR 0184's table.
|
||||
|
||||
@@ -5,9 +5,10 @@ code:
|
||||
- mesh-controller cmd/mesh-builder
|
||||
- mesh-controller internal/builder
|
||||
- mesh-catalog modules/builder
|
||||
updated: 2026-10-01
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
|
||||
- 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md
|
||||
- 02-DECISIONS/0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md
|
||||
- 02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md
|
||||
- 02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md
|
||||
@@ -293,6 +294,44 @@ build's lines reach a reader of its subject in order and the stream holds them a
|
||||
against a real server); the seat verb with an id reads the log (controller test); and, live, a build
|
||||
after the roll-out read line by line through the console.
|
||||
|
||||
## The store keeps what the records name
|
||||
|
||||
*2026-10-02 — [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md),
|
||||
[issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
|
||||
|
||||
Every build pushes another layer set and, until this, nothing ever removed one. The registry's own
|
||||
answer — collect what no tag names — is wrong for this mesh: each artifact is pushed under one
|
||||
moving tag and machines are pinned by digest, so every build but the newest is untagged and some
|
||||
machine may still be running it.
|
||||
|
||||
**The mesh decides and the store reclaims.** Deletion is enabled on the store's one door — that
|
||||
door already accepts a push, and a writer who can push can replace any tag, so delete takes
|
||||
nothing a push did not already have, and ADR 0082's bargain (a store every machine reaches with no
|
||||
credential to distribute first) is kept. The mesh then removes what it put there and no longer
|
||||
keeps, **naming it from its own build records** rather than enumerating the store: it has never
|
||||
put anything there it did not record, so a digest it did not record making is never named, which
|
||||
is what keeps the sweep away from the images genesis pushed before any record existed.
|
||||
|
||||
An artifact stays for one of two reasons and otherwise goes: a definition the mesh holds names it
|
||||
(no age limit — this is the floor), or it belongs to one of the five most recent successful builds
|
||||
of its module (somewhere for a wrong release to return to). The sweep runs after a build the mesh
|
||||
recorded, which is the moment new bytes landed and the moment the keep set moved; it needs no
|
||||
timer. Deleting a manifest frees no bytes, so the store's own collector runs nightly as a
|
||||
scheduled step with the server held still — which is what `while-stopped` exists for
|
||||
([design 20](20-writing-a-module.md)). Plain collection, not `--delete-untagged`: what the mesh
|
||||
keeps is still a manifest and so still referenced, and the dangerous flag is not needed once the
|
||||
mesh is the one deciding.
|
||||
|
||||
A machine behind by more than five builds of a module, recreating a container, cannot pull what it
|
||||
was running. It is already a machine the mesh reports as behind, and the answer is the current
|
||||
declaration.
|
||||
|
||||
*How it is checked:* the keep set, against records, holds what a manifest names and the five most
|
||||
recent builds and nothing else; a reference the mesh never recorded is never in the delete set; an
|
||||
image and an archive are asked for at their own endpoints; a store with deletion off names the
|
||||
remedy rather than the status code; a store that does not have it is recorded collected rather
|
||||
than retried for ever. Live: the store's size before and after the first nightly collection.
|
||||
|
||||
## The builder compiles the languages the mesh is written in
|
||||
|
||||
*2026-09-29 —
|
||||
|
||||
@@ -5,11 +5,12 @@ code:
|
||||
- mesh-catalog modules/showcase
|
||||
- mesh-controller internal/builder
|
||||
- mesh-sdk src
|
||||
updated: 2026-09-30
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md
|
||||
- 02-DECISIONS/0099-a-step-that-runs-once-names-what-it-reads.md
|
||||
- 02-DECISIONS/0053-a-step-that-runs-on-a-schedule.md
|
||||
- 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md
|
||||
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
|
||||
- 02-DECISIONS/0040-what-a-module-is.md
|
||||
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
|
||||
@@ -208,3 +209,27 @@ is recreated with the new fact
|
||||
([ADR 0099](../../02-DECISIONS/0099-a-step-that-runs-once-names-what-it-reads.md)). *How it is
|
||||
checked:* the host's unit tests run a step again when its named file changed and not otherwise,
|
||||
and recreate a container naming a step after the step ran.
|
||||
|
||||
## A recurring step may hold its own module's containers still
|
||||
|
||||
*Written 2026-10-02, from [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md)
|
||||
and [issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
|
||||
|
||||
Some work cannot be done underneath a running service: an artifact store's collector walks the
|
||||
storage and requires every writer stopped. A `run-once` step runs *beside* containers and a
|
||||
scheduled one is the same container again, so until this a module had no way to say it — and the
|
||||
mesh inherited a store that has never collected anything, because the predecessor said it with a
|
||||
shell script and a script beside a module is not a resource in it.
|
||||
|
||||
A scheduled step may name `while-stopped`: resource ids of **its own module's** containers, which
|
||||
the host stops before the run and starts again after it, in the reverse order, **whatever the step
|
||||
did**. Three boundaries, each refused where it can be seen earliest — its own module's containers
|
||||
only, because a module that could quiesce a neighbour could stop the mesh; scheduled steps only,
|
||||
because at apply the declaration is applied in order and a step already gates what follows, so a
|
||||
one-time offline job says *before* rather than *instead of*; and restoring that is not conditional
|
||||
on anything, because the only real risk of the field is a window that never closes.
|
||||
|
||||
*How it is checked:* the host's unit tests assert stop–run–start in that order, the restart after a
|
||||
step that **failed**, the reverse order for several containers, and a service left down said
|
||||
loudly. The controller refuses, from the definition alone, a window with no schedule, one on a
|
||||
run-once step, one naming a container the module does not declare, and one naming itself.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-controller internal/catalogue]
|
||||
updated: 2026-09-30
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
|
||||
- 02-DECISIONS/0155-a-definition-names-no-installation-and-how-that-is-checked.md
|
||||
@@ -14,6 +14,7 @@ decisions:
|
||||
- 02-DECISIONS/0084-which-provider-serves-a-consumer.md
|
||||
- 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md
|
||||
- 02-DECISIONS/0038-the-mesh-assigns-the-port.md
|
||||
- 02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md
|
||||
---
|
||||
|
||||
# 27 — A module requires, the mesh resolves
|
||||
@@ -206,6 +207,22 @@ name when nothing sets it. That is the contract half of this design's operator p
|
||||
the placeholder allows: the definition says which values reach which requirement, and nothing else
|
||||
does. *How it is checked:* the unit tests named in issue 173, and the plan comparison that closed it.
|
||||
|
||||
*A provider says once what it derives for each consumer (2026-10-02,
|
||||
[ADR 0188](../../02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md),
|
||||
[issue 124](../../04-ISSUES/124-a-consumer-cannot-be-told-what-its-provider-derived/00-report.md)):*
|
||||
where a provider **names the resource** it gives each consumer — a bucket, a database, a vhost — the
|
||||
name is derived per consumer, and a literal `serves` block could not carry it. A served value may
|
||||
now name the consumer the mesh is serving: `${consumer:as}`, the identity the mesh minted, and
|
||||
`${consumer:as:dns}`, that same identity written as a DNS label. Nothing else — **the mesh learns no
|
||||
protocol here; it spells its own name in an alphabet it already knows.** Settings are laid on first,
|
||||
so an operator may still set a prefix and the mesh derives the rest. The mesh fills it at the one
|
||||
moment it knows who the consumer is, and the one filled value reaches both ends: the consumer, as
|
||||
its binding's served facts and as `${bound:<provision>:<key>}` in any file it writes; the provider,
|
||||
as `derived` on that consumer's entry in its contributions file, so its provisioner is told the name
|
||||
rather than recomputing it. A consumer that writes the derived value into its own definition instead
|
||||
of asking for it is refused, naming the placeholder to use. *How it is checked:* the unit tests in
|
||||
ADR 0188's "how this is checked", each run against the unchanged controller first.
|
||||
|
||||
## How a definition reads what was resolved
|
||||
|
||||
**One form, naming a requirement and a field of its contract.** A definition that needs the database's
|
||||
@@ -214,7 +231,9 @@ name in a configuration file writes the same thing: the requirement's name and t
|
||||
controller fills it at resolution.
|
||||
|
||||
This one form replaces the placeholders that exist today, one per mechanism: bound values, secrets,
|
||||
ports and machine facts.
|
||||
ports and machine facts. It subsumes the consumer placeholder too — a value a provider derives is
|
||||
read by the consumer exactly as any other field of the contract is, and `${consumer:…}` is only
|
||||
how the *provider* states the rule.
|
||||
|
||||
**The seat placeholder stays, for the controller alone.** The controller composes its own
|
||||
declaration and reaches the store and broker it made before any module existed, so it cannot be
|
||||
|
||||
@@ -1,10 +1,15 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code: []
|
||||
updated: 2026-09-27
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller: internal/catalogue/jails_into.go, internal/catalogue/manifest.go (Jail, Jailing)
|
||||
- mesh-catalog: modules/fail2ban (jailing, the base and the seat's verbs), modules/mailu, modules/route-proxy, modules/gitea (jails)
|
||||
- mesh-host: internal/declaration/declaration.go (a container's logging)
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
|
||||
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
- 02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md
|
||||
---
|
||||
|
||||
# 31 — A module declares its fail2ban jail, and the mesh composes them per node
|
||||
@@ -63,3 +68,38 @@ jail, composed from the postgres module's manifest, without anyone editing a nod
|
||||
beside)
|
||||
- mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules
|
||||
(`postgres`, `mssql`, `mailu`) that will declare jails
|
||||
|
||||
## Decided and built, 2026-10-02
|
||||
|
||||
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
||||
made this the rule and built it. A module declares `jails` — each a name, the `failregex` of a
|
||||
failed attempt in its log, and the stanza's own keys — and the fail2ban module declares `jailing`:
|
||||
the one file the stanzas compose into and the directory each filter lands in. The controller gathers
|
||||
every assigned module's jails per node into those; the holder's daemon restarts on the composed file.
|
||||
|
||||
What made it workable was the log. A container's output went to a file of the runtime's own, under
|
||||
a path that changes when the container is recreated, so no jail could read a container's service
|
||||
however it logged. A container now declares `logging: journald`, the host runs it with the journal as
|
||||
its driver, and a jail reads it with `backend = systemd` and a `journalmatch` on the container's
|
||||
name — the same way the base's ssh jail has always read the ssh daemon. The first three doors: the
|
||||
mail front end (every login failure on its proxying ports), the forge (a failed authentication
|
||||
attempt) and the public proxy (a certificate or request for a name the mesh does not serve, which
|
||||
the proxy now says in its log). The base is strict — three in a day for a day; twice banned in two
|
||||
weeks for four — and the mesh's own range stays never banned.
|
||||
|
||||
The seat the module holds serves `status`, `banned`, `ban` and `unban`, from a runtime that carries
|
||||
only the fail2ban client with the daemon's socket shared in; the jails are composed, the ban list is
|
||||
the daemon's, and both are read through the console.
|
||||
|
||||
*How it is checked:* ADR 0179's table.
|
||||
|
||||
## What the first jails taught, 2026-10-02
|
||||
|
||||
[ADR 0186](../../02-DECISIONS/0186-a-ban-list-never-holds-a-neighbour.md). Within an hour of the
|
||||
first public jail the home server had banned the house's own router: the router reflects local
|
||||
traffic, so every client in the building arrives as the gateway's address. The never-ban list now
|
||||
holds every private range as well as the mesh's own. And the mesh read its own ban chain as a
|
||||
foreign rule set on that machine, because the chain hangs off the container runtime's user chain and
|
||||
that machine's forward policy is the runtime's DROP — the reader now calls a chain of source-named
|
||||
refusals a ban wherever it hangs, as it already did for the packet filter's own tables.
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ code:
|
||||
- mesh-host internal/apply/apply.go
|
||||
- mesh-tools src/main.ts
|
||||
- mesh-catalog modules/mesh-catalog
|
||||
updated: 2026-09-28
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
|
||||
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
|
||||
@@ -24,6 +24,7 @@ decisions:
|
||||
- 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
|
||||
- 02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md
|
||||
- 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
|
||||
- 02-DECISIONS/0187-a-dead-tracker-is-not-the-machines-failure.md
|
||||
- 02-DECISIONS/0134-the-mesh-says-what-it-applied.md
|
||||
---
|
||||
|
||||
@@ -470,6 +471,21 @@ moment the mesh can mint for itself: **the bus's own accounts** (§the bootstrap
|
||||
needs an account before it can run) and **the vault's own credential**. Any third exception is a
|
||||
design failure, and naming these two is what makes a third one visible.
|
||||
|
||||
## What a step is answerable for, 2026-10-02
|
||||
|
||||
[ADR 0187](../../02-DECISIONS/0187-a-dead-tracker-is-not-the-machines-failure.md). A step gates its
|
||||
module and not the machine ([ADR 0136](../../02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md)),
|
||||
but a step that exits non-zero still leaves the machine reporting that it is not doing what it was
|
||||
told — and a clean apply gates other things entirely, the found firewall's retirement among them. So
|
||||
what a step calls a failure matters beyond the step.
|
||||
|
||||
The rule the media step now follows, and the one to copy: a step fails for what the mesh chose and
|
||||
can fix, and reports what it merely found and cannot. An indexer entry the mesh re-pointed at this
|
||||
mesh's proxy is plumbing the mesh is answerable for; whether the public tracker behind it answers
|
||||
today is not. An entry the operator listed is the operator's to insist on, and still fails. Six
|
||||
hours of a machine reading as broken, for one tracker that had died, is what the distinction costs
|
||||
when it is missing.
|
||||
|
||||
## 11. Open
|
||||
|
||||
**Semantic change has no mechanical defence** (§8). Recorded as open rather than solved, because
|
||||
|
||||
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-tools]
|
||||
updated: 2026-10-02
|
||||
decisions:
|
||||
- 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
- 02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md
|
||||
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
|
||||
- 02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md
|
||||
- 02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md
|
||||
@@ -184,6 +185,18 @@ container to declare a capability. Removing a predecessor's rule set is an opera
|
||||
through the seat, recorded on the bus, instead of a shell on the machine. *How it is checked:* ADR
|
||||
0169's table.
|
||||
|
||||
## The intrusion seat's verbs, 2026-10-02
|
||||
|
||||
[ADR 0179](../../02-DECISIONS/0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md).
|
||||
The second node-scoped seat to carry verbs: `node-intrusion-prevention` serves `status` (every jail
|
||||
with what it watches and holds), `banned` (every address held now, with its jail and when the ban
|
||||
ends), `ban` and `unban` (an operator's act on the live ban list). The fail2ban module serves them
|
||||
from a runtime that carries only the daemon's client, the socket shared in from the machine — no
|
||||
capability, no machine network, since the daemon on the machine does the banning. That runtime is the
|
||||
per-module container [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md)
|
||||
retires; the verbs and the client are the same code once the node's own runtime loads them as a bundle.
|
||||
The module's own tool beside them reads one jail's effective settings. *How it is checked:* ADR 0179's table.
|
||||
|
||||
## What this does not settle
|
||||
|
||||
- Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a
|
||||
|
||||
+33
-4
@@ -1,9 +1,9 @@
|
||||
---
|
||||
status: open
|
||||
status: resolved
|
||||
opened: 2026-09-23
|
||||
located-in: []
|
||||
fixed-by:
|
||||
amended-design:
|
||||
located-in: [mesh-controller internal/inventory, mesh-controller internal/artifacts, mesh-host internal/apply, mesh-catalog modules/distribution]
|
||||
fixed-by: 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md
|
||||
amended-design: 03-DESIGN/01-to-be/18-building-a-module.md
|
||||
---
|
||||
|
||||
# 108 — The registry has no garbage collection, and two doors make it harder to add
|
||||
@@ -75,3 +75,32 @@ real thing services need, and the mesh cannot express one.
|
||||
images by digest and moves by version — is retention "the digests no recorded build names"?
|
||||
- Who owns the routine when the store and its public door are two modules — the store, since the
|
||||
volume is its?
|
||||
|
||||
## Answered, 2026-10-02 — [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md)
|
||||
|
||||
The three open questions, answered:
|
||||
|
||||
- **A maintenance step, or a backend that does not need its writers stopped?** The step. A
|
||||
scheduled container may name `while-stopped` — resource ids of **its own module's** containers,
|
||||
which the host stops before the run and starts again after it whatever the step did. A storage
|
||||
backend the mesh does not run would be a bigger thing to own than the mechanism it avoids, and
|
||||
the mechanism is wanted anyway: a service that cannot have work done underneath it is a real
|
||||
shape and the mesh could not express it at all.
|
||||
- **Is retention "the digests no recorded build names"?** Nearly. An artifact stays because a
|
||||
definition the mesh holds names it (no age limit), or because it belongs to one of the five most
|
||||
recent successful builds of its module. Last-N-tags was the predecessor's rule for a registry
|
||||
that knew nothing else; this mesh knows what each digest is for.
|
||||
- **Who owns the routine now the second door is gone?** Both halves, each where it can be. The
|
||||
**mesh** decides what may go — only it holds the records — and asks the store to drop it. The
|
||||
**store** reclaims the bytes, because only it can stop its own server. Neither half can be done
|
||||
by the other.
|
||||
|
||||
And the sharpened point — enabling deletion on a door with no accounts — dissolved on inspection:
|
||||
**that door already accepts a push**, so a writer who can reach it can already replace any tag.
|
||||
Delete takes nothing a push did not have. What it does not do is undo ADR 0082's bargain, which
|
||||
putting an authenticated door in front of deletion would have.
|
||||
|
||||
The second registry process is not built, as the 2026-09-26 note says, so the shared blob cache
|
||||
and the deletion-cached-by-the-other-door problem never arise. Plain `garbage-collect` is enough:
|
||||
what the mesh keeps is still a manifest in the store, so `--delete-untagged` — the flag that would
|
||||
delete images machines are running — is not needed at all.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
opened: 2026-09-26
|
||||
located-in: [mesh-controller internal/catalogue/declaration.go, mesh-sdk src/provisioner, mesh-catalog modules/minio]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
located-in: [mesh-controller internal/catalogue, mesh-sdk src/provisioner, mesh-catalog modules/minio]
|
||||
fixed-by: 02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md
|
||||
amended-design: 03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md
|
||||
---
|
||||
|
||||
# 124 — A consumer cannot be told a value its provider derived for it, so it transcribes one
|
||||
@@ -63,3 +63,27 @@ compares it to what the provider will actually create. The one wrong instance wa
|
||||
- What would have caught the wrong instance? A test that resolves a consumer's grant and compares the
|
||||
bucket in its own configuration against the one the provider would create is a check that could
|
||||
exist today, for any interface, without the mechanism above.
|
||||
|
||||
## Answered, 2026-10-02 — [ADR 0188](../../02-DECISIONS/0188-a-provider-declares-what-it-derives-for-each-consumer.md)
|
||||
|
||||
The channel is the provider's own `serves` block, which may now name the consumer the mesh is
|
||||
serving: `${consumer:as}` and `${consumer:as:dns}`. The mesh fills it once, where it knows who the
|
||||
consumer is, and delivers the one filled value to both ends — the consumer's binding and its
|
||||
`${bound:…}` substitutions, and the provider's contributions entry, so a provisioner is told the
|
||||
name rather than deriving it. Each open question above, answered:
|
||||
|
||||
- **Should a provider return values from provisioning?** No. It would make a grant carry data the
|
||||
provider wrote, make a consumer's declaration wait on its provider's reconcile loop, and put the
|
||||
rule where nothing can refuse it. The reasoning is in the record.
|
||||
- **Or should `serves` say a value is derived?** Yes, and the mesh performs the derivation — but it
|
||||
learns no protocol doing it. The only fact is the identity the mesh itself minted, in one of two
|
||||
alphabets it already knows.
|
||||
- **Should a consumer that names the resource be refused?** Yes. A consumer's file that already
|
||||
contains the value the mesh is about to derive for it is refused at resolution, naming the
|
||||
placeholder to write instead. That is the check this report asked for, and it is exact rather than
|
||||
heuristic: a derived value carries the identity minted for this consumer on this machine, which
|
||||
nothing else would spell out.
|
||||
|
||||
minio's `bucketFor` is gone; its manifest serves `"bucket": "${consumer:as:dns}"`. The three
|
||||
consumers' hand-written bucket names are gone with it — each of them also named the machine the
|
||||
module happens to run on, which is the second thing wrong with a transcription.
|
||||
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-02
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by: 02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 201 — A push recreated the controller at a digest older than the seat row its successor had written
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the
|
||||
controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan
|
||||
built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The
|
||||
host's plan then pushed the control node with the controller digest it had recorded when it was made —
|
||||
the previous build — and recreated the controller container on it. The older binary read the row, found
|
||||
a verb it could not run, and refused to start:
|
||||
|
||||
```
|
||||
mesh-controller: the mesh-controller seat's row declares "command", which this control plane
|
||||
cannot run: "command" is not a verb the mesh-controller seat serves
|
||||
```
|
||||
|
||||
A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since
|
||||
the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the
|
||||
newer image, outside the service, to push the control node again; the push sent the newer digest and
|
||||
the controller came up.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md));
|
||||
a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of
|
||||
2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that
|
||||
wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md));
|
||||
for every other module an older digest is a brief regression a later push corrects. For the controller
|
||||
it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct
|
||||
it is the thing that is down. Two plans that overlap will happen again whenever two people merge within
|
||||
a minute.
|
||||
|
||||
## What a fix would have to do
|
||||
|
||||
Either of two, and the first is the smaller:
|
||||
|
||||
- A push never sends a controller digest older than the one the running controller is — the controller
|
||||
knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words.
|
||||
- Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves
|
||||
what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check
|
||||
exists to refuse.
|
||||
|
||||
Until one is built: do not merge a controller change while another plan is rolling, and after merging
|
||||
one, wait for `node show` on the control node to report the new controller before merging anything else.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
||||
- mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`
|
||||
|
||||
## Half of it is closed, 2026-10-02
|
||||
|
||||
[ADR 0185](../../02-DECISIONS/0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md) takes
|
||||
the outage out of it: a control plane behind its seat's row now serves every verb it can run, says
|
||||
which it cannot, and answers the reason when one of those is called. The same race today would cost
|
||||
the verbs the newer build added, for as long as the older binary is in place, and the ordinary
|
||||
"this machine is behind" machinery would put the newer one back without a hand.
|
||||
|
||||
**The race itself is still open**, and this report stays open for it. What was established while
|
||||
closing the other half, so the next reader does not redo it:
|
||||
|
||||
- Composing and sending are serialised per machine by a session advisory lock in the store, so two
|
||||
control planes cannot compose one machine's declaration at the same time. The stale content did
|
||||
not come from two concurrent composes.
|
||||
- A container's image is resolved into the module's manifest when it is *built*, and a push composes
|
||||
from the catalogue as it is at that moment, under the hold. So a compose that ran after the build
|
||||
was taken in could not have named the older image.
|
||||
- The declaration's sequence orders arrival and nothing else (the numbering of
|
||||
[issue 107](../107-a-declaration-carries-no-order/00-report.md)); it cannot tell a later send
|
||||
carrying earlier content from a later send carrying later content. The host refuses a declaration
|
||||
numbered below the last it applied, and both of these were above it.
|
||||
- The machine's own journal shows the two applies ten seconds apart and which replaced what; it does
|
||||
not record which image each declaration named, which is the one fact that would settle it. A host
|
||||
that recorded the digest it was told, per apply, would have answered this in a minute.
|
||||
|
||||
So the trigger is not yet pinned, and guessing at the push path is the most expensive place in the
|
||||
mesh to guess. The fix the report first suggested — a push never sending a control plane a digest
|
||||
older than the one that machine reports running — closes the class without needing the trigger, and
|
||||
is now a correctness nicety rather than the difference between a working mesh and a dead one.
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-02
|
||||
located-in: [mesh-catalog modules/dnsmasq, mesh-controller cmd/mesh-controller]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 202 — A module whose required setting nobody set is left out of the machine, and the resolver is the module it happened to
|
||||
|
||||
## What was observed
|
||||
|
||||
Running the controller's own test suite against the catalogue beside it, 2026-10-02.
|
||||
`TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves` fails with *"the resolver
|
||||
was not handed the machines"*. Composing the same machine by hand and listing what it receives
|
||||
shows why: **dnsmasq contributes nothing at all.** Four resources are composed for that node, all
|
||||
of them the overlay's. The resolver's package, its configuration, its service and the fact that
|
||||
carries every machine's name are simply not there.
|
||||
|
||||
The cause is one line added to `dnsmasq`'s configuration earlier the same day: the addresses it
|
||||
listens on beside the machine's own became an operator setting,
|
||||
`listen-address=${setting:listen-addresses}`, with no default. A `${setting:…}` nothing sets is
|
||||
refused, a module that cannot be composed is **left out** rather than failing the whole machine
|
||||
([ADR 0163](../../02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md)), and so a node
|
||||
assigned the resolver is handed a declaration with no resolver in it.
|
||||
|
||||
The failing test is the symptom that surfaced it. The test is not what is wrong.
|
||||
|
||||
**Proven rather than inferred.** Composing the same machine a second time with
|
||||
`listen-addresses` set to `127.0.0.1` and nothing else changed, every one of dnsmasq's eight
|
||||
resources appears — `needs-broker`, `mesh-state`, `package`, `config`, `runtime-dns`, `runtime`,
|
||||
`service` and `fact-node-zones`. The only difference between a machine with a resolver and a
|
||||
machine without one is whether somebody set a value that did not exist yesterday.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**Leaving a module out is right, and being quiet about it is not.** The rule exists so one
|
||||
module's broken setting cannot stop a machine converging — a good rule. But the outcome here is a
|
||||
machine that applies cleanly, reports current, and is missing its DNS resolver. Every name on that
|
||||
machine then resolves through whatever was there before, or not at all, and nothing in the mesh
|
||||
says the resolver was dropped. That is the shape
|
||||
[issue 152](../152-a-nodes-plan-failure-silently-drops-its-routed-names/00-report.md) records for
|
||||
routed names, here for a whole module.
|
||||
|
||||
**And a setting with no default is a definition that cannot be assigned.** Every other
|
||||
`${setting:…}` in the catalogue names something that is genuinely particular to one installation —
|
||||
a public domain, an issuer. "Which addresses besides my own do I answer on" has an obvious correct
|
||||
default for every machine that is not a LAN gateway: none beside loopback. A definition that
|
||||
refuses to compose until somebody sets a value most machines do not need is a definition that
|
||||
breaks the next node to be assigned it, and genesis with it.
|
||||
|
||||
## What this does not claim
|
||||
|
||||
Whether the live machines are affected was not checked — those four have had the setting set, or
|
||||
their resolvers would already be gone. The claim is about a machine assigned the resolver *from
|
||||
now on*, and about the silence.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should the declaration say which modules it left out, where a person or the console can see it?
|
||||
`left_out` already travels to the host ([ADR 0163](../../02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md));
|
||||
what is missing is anything that reads it back and says so.
|
||||
- Should a `${setting:…}` be allowed a default in the definition — making "unset" mean "the
|
||||
default" rather than "refuse" — or is a setting with a default no longer the operator's value?
|
||||
- Is leaving a module out ever right for a module a node is **assigned**, as opposed to one it
|
||||
merely pulls in? An assignment is somebody saying *this machine runs this*; silently not running
|
||||
it is the one answer nobody asked for.
|
||||
Reference in New Issue
Block a user