Compare commits

...
Author SHA1 Message Date
jschoubben bff32e3370 Issue 136: a module may name a program the machine does not have, and everything reports success 2026-09-28 20:50:35 +02:00
mesh-admin d6b62387f2 Merge pull request 'Issue 135: a container's mesh names are not compared' (#165) from issue/135-a-containers-mesh-names into main 2026-09-28 15:19:37 +00:00
jschoubben 4789624857 Issue 135: a container's mesh names are not compared, so a moved address is never noticed
One container restarted 2286 times over five days while the mesh reported the machine as doing what it
was told. Its overlay address was five days out of date: the host compares a container by a digest of
its spec, and the mesh's names were not in it, so a container whose image and files never changed was
left alone holding a name that no longer resolved. Forty-eight others were current only because
something else had recreated them.

The same fault as issue 045, in the field that was left out. Resolved by putting the names in the
digest.
2026-09-28 17:19:35 +02:00
mesh-admin e00862e317 Merge pull request 'ADR 0112 is accepted, and issue 134 records what it is not yet' (#164) from decision/0112-accepted-and-issue-134 into main 2026-09-28 15:12:15 +00:00
jschoubben 3a92e4b80c ADR 0112 is accepted, and issue 134 records what it is not yet
0120 was already accepted; what failed the check was that it rests on 0112, still marked proposed —
and so do four designs. The decision stands: a definition names no node, no mesh and no host path, and
everything a module needs is a requirement the mesh resolves.

Accepting it makes the gap visible rather than hiding it, so issue 134 states it. 0112 says how it is
checked — 'a catalogue test finds no domain name in any definition value' — and there is no such test.
Asked by hand: seven modules name this installation in a value the mesh acts on, and eight mention a
public name in prose nothing reads. The two are not the same fault and the fixes differ, which is why
the issue separates them rather than counting to fifteen.
2026-09-28 17:12:12 +02:00
mesh-admin bfddf78bf3 Merge pull request 'Design 32: what shipped today, and the records it rests on' (#163) from design/28-and-32-what-shipped into main 2026-09-28 14:50:44 +00:00
jschoubben 0b291d89f3 Design 32: what shipped today, and the records it rests on
Two of its statements are built — a version preparing its state, and the mesh saying what it applied —
so the document is in-progress rather than proposed, and names the code that owns them. Resting on a
live record rather than a superseded one: 0127 was replaced by 0131.

What this exposes is pre-existing: it also rests on ADR 0112, which is still proposed, and a document
that is not itself proposed may not. ADR 0120 has rested on it the same way for a while. Accepting or
superseding 0112 is a decision, not a cleanup, so it stays visible in the check rather than papered
over.
2026-09-28 16:50:41 +02:00
mesh-admin a924efcc28 Merge pull request 'ADR 0136: a step gates its module, not the machine' (#162) from decision/0136-a-step-gates-its-module into main 2026-09-28 13:38:18 +00:00
6 changed files with 245 additions and 8 deletions
@@ -1,6 +1,6 @@
---
topic: what runs on it
status: proposed
status: accepted
date: 2026-09-25
deciders: jochen
reconstructed: false
+1 -1
View File
@@ -205,7 +205,7 @@ python3 00-META/checks/index.py fail if stale
- **0091** — [A mount is declared, and there are three things it can be](0091-a-mount-is-declared-three-ways.md)
- **0099** — [A step that runs once names what it reads, and runs again when it changed](0099-a-step-that-runs-once-names-what-it-reads.md)
- **0110** — [A seat is held by one assignment, from a closed set, and it may deliver a provision](0110-a-seat-is-a-module-assignment-from-a-closed-set.md)
- **0112** — [A module definition names no node, no mesh and no path: everything it needs is a requirement the mesh resolves](0112-a-module-definition-names-no-node-mesh-or-path.md) *(proposed)*
- **0112** — [A module definition names no node, no mesh and no path: everything it needs is a requirement the mesh resolves](0112-a-module-definition-names-no-node-mesh-or-path.md)
- **0113** — [The vault makes every shared secret, a provider makes resources and data, and the mesh carries both](0113-the-vault-makes-every-secret.md) *(proposed)*
- **0114** — [A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached](0114-a-shared-credential-rotates-over-two-credentials.md) *(proposed)*
- **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md) *(proposed)*
@@ -1,11 +1,20 @@
---
layer: to-be
status: proposed
code: []
status: in-progress
code:
- mesh-controller internal/catalogue/declaration.go
- mesh-controller internal/catalogue/manifest.go
- mesh-controller internal/link/serve.go
- mesh-controller internal/link/bus.go
- mesh-controller internal/broker/nats.go
- mesh-controller internal/inventory/nodes.go
- mesh-host internal/apply/apply.go
- mesh-tools src/main.ts
- mesh-catalog modules/mesh-catalog
updated: 2026-09-28
decisions:
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
- 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
- 02-DECISIONS/0128-the-mesh-bus-is-required-not-ambient.md
- 02-DECISIONS/0106-the-bus-is-nats.md
- 02-DECISIONS/0041-events-are-a-relationship.md
@@ -261,8 +270,8 @@ queue.
and publishes it last-per-subject. A node that was away gets exactly the current one, never a
queue of superseded ones, and a replayed older one is refused by sequence.
**A version prepares its state before it runs.** A module version may declare an entrypoint that brings
its state to the shape that version needs — the same vocabulary as the entrypoints it declares for its
**A version prepares its state before it runs.** *Built 2026-09-28.* A module version may declare an
entrypoint that brings its state to the shape that version needs — the same vocabulary as the entrypoints it declares for its
tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it
runs the module's own code, to completion, in the module's own context, and a version whose preparation
did not succeed does not run: the step gates that module and nothing else on the machine
@@ -277,7 +286,7 @@ mesh provisions is per consumer and preparation is too. No level to choose, and
not to an address it was given at genesis. Held and retried while the store restarts
([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)).
**And the mesh says what it applied.** A report is control traffic only the control plane reads, so the
**And the mesh says what it applied.** *Built 2026-09-28.* A report is control traffic only the control plane reads, so the
chain above went dark at the moment it touched a machine: nothing said which version a machine now runs,
or that it refused to. The control plane states those as facts under its own seat's namespace, when what
a machine runs changes rather than on every convergence pass, and anything that cares subscribes the way
@@ -0,0 +1,66 @@
---
status: open
opened: 2026-09-28
located-in: [mesh-catalog, mesh-controller internal/catalogue]
fixed-by:
amended-design:
---
# 134 — A definition may still name the mesh, and the check that would say so does not exist
## What was observed
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) says a module
definition names no node, no mesh and no host path, and states how that is checked:
> A catalogue test finds no domain name in any definition value.
There is no such test. Run by hand on 2026-09-28, across the 72 manifests in the catalogue, the
question it asks has 15 answers. They are not all the same kind of thing, and the difference matters
more than the count:
**Values the mesh acts on** — seven:
| module | where | what it names |
|---|---|---|
| keycloak | `env.KC_HOSTNAME` | this installation's public name for itself |
| minio | `env.MINIO_BROWSER_REDIRECT_URL` | the same, for its console |
| invoicing | a resource's `image` | a named registry rather than the mesh's artifact store |
| builder | `build.artifacts[].context.repository` | the forge, by URL |
| route-proxy | `build.artifacts[].context.repository` | the forge, by URL |
| route-adapter | a resource's `content` | a proxy's dynamic configuration |
| novox.be | `module` | the module is named after the domain it serves |
**Prose** — eight, in `listens[].why`: de-spiegel, mailu, n8n, only-office, photos, photos-eef,
photos-filip, portainer. Each explains what a port is for and mentions the public name it is reached
by. Nothing reads these; a check written as a string search would report them, and reporting them as
violations of the same rule would be wrong.
## Why it matters beyond this instance
**An unenforced rule is indistinguishable from a wrong one, and costs more, because people believe
it.** The record says the mesh is name-agnostic, four design documents rest on that, and a reader
checking whether it holds finds that it does not — in the places that matter most. The two forge URLs
are what a build reaches into for its source; the two hostnames are what a service tells a browser
about itself.
**It is the difference between a mesh and this mesh.** A definition carrying `novox.be` is a
definition that can only be installed here. The whole point of the rule is that the same catalogue
raises a different mesh with a different name, and today seven modules would need editing to do it.
**And the shape of the fix is not the same for each.** A public name is an operator's choice about an
assignment, which ADR 0112 already provides for; a forge URL should be a path on the git seat
([ADR 0111](../../02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md)); an image from a named
registry is a question about the artifact store, not about naming. Counting them together would hide
that.
## Open questions
- Does a domain in a `why` string break the rule? It is documentation the mesh never reads, and a
check that cannot tell the two apart will either pass things it should catch or fail things nobody
should change.
- Where does a service's public name live, concretely — a setting on the assignment, or a fact the
mesh composes from the node's domain? ADR 0112 says a requirement the mesh resolves; the two
hostnames above are the first real cases.
- Should a build context name a repository on the git seat rather than by URL, and if so, what does
that mean for a context in *another* mesh's forge?
@@ -0,0 +1,70 @@
---
status: resolved
opened: 2026-09-28
located-in: [mesh-host internal/apply]
fixed-by: mesh-host — a container's mesh names are part of the spec digest the host compares, sorted so the digest does not move for a reordering. A container whose names moved is now recreated like a container whose image moved, and the test fails against the previous behaviour.
amended-design:
---
# 135 — A container's mesh names are not compared, so a moved address is never noticed
## What was observed
One container on this mesh had been restarting every thirty seconds for five days — 2286 times — and
the mesh reported the machine as doing what it was told.
Its logs said its database connected and then a query timed out. The database was reachable: the same
query from the same network, with the same credential, answered in milliseconds. What differed was the
name. Inside that container, `novox.internal` resolved to `10.42.0.1`; in every other container on the
machine it resolved to `10.10.0.1`. The mesh's overlay range had moved, and this container still held
the old one:
```
umami created 2026-09-23 novox.internal:10.42.0.1
mesh-catalog created today novox.internal:10.10.0.1
```
A container resolves other machines and public names through the entries the mesh gives it when it is
created, and nothing re-reads them afterwards. The host compares a container against what was declared
by a digest of its spec — image, name, environment, ports, volumes, arguments, resolver, address, and
what it reads — and **the mesh's names were not in it**. So this container matched what was declared,
was left alone, and kept an address that had not existed for five days.
Forty-eight other containers had current names. Not because anything corrected them: each had been
recreated for some other reason — a new image, a changed file — and picked up the current roster on the
way. This one's image is an upstream release that had not moved, and nothing else about it changed, so
nothing ever recreated it.
## Why it matters beyond this instance
**It is the exact fault [issue 045](../045-a-container-keeps-the-values-it-started-with/00-report.md)
named, in the one field that was left out.** That issue is why the digest carries what a container
reads: "a container whose configuration had since been rewritten compared equal and was left alone —
running values the machine no longer holds, while every check reported success." The same sentence
describes this, with *names* in place of *files*.
**The failure is invisible in exactly the way that matters.** The container runs, so the machine
reports it applied. It restarts, but a restarting container is a normal sight during an upgrade. The
only account of the fault is inside the container's own log, in the words of the application rather
than of the mesh — and what it says is that a query timed out, which points at the database.
**And it is most likely to bite what changes least.** Every container that is rebuilt often repairs
itself by accident. The victim is the module whose image is stable — which is to say, the module that
was working fine.
## What was done
The mesh's names are part of the digest, sorted so the digest does not move for a reordering nobody
made. A container whose names moved is now recreated exactly as one whose image moved.
The first apply after this recreates every container that carries mesh names — one restart each,
already the price the mesh pays for any image update — because their recorded digests predate the
field.
## What is still true
The mesh gives a container its names at creation and has no way to change them in place. That is the
container runtime's shape, not a choice; the answer is to recreate, which is what this does. A module
that would rather re-read a roster from a file can already ask for one as a fact
([ADR 0120](../../02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md)) and restart on
it.
@@ -0,0 +1,92 @@
---
status: resolved
opened: 2026-09-28
located-in: [mesh-catalog modules/fail2ban]
fixed-by: mesh-catalog — the intrusion-prevention module bans through an action it ships itself, already in use on every machine, instead of naming a firewall front-end two of them do not have. The instance is closed; the class in "What is still true" is not.
amended-design:
---
# 136 — A module may name a program the machine does not have, and everything reports success
## What was observed
Two machines were given the intrusion-prevention module on 2026-09-28. Both refused to start it:
```
ERROR Failed during configuration: Have not found any log file for 'recidive' jail.
ERROR Async configuration of server failed
fail2ban.service: Main process exited, code=exited, status=255/EXCEPTION
```
The jail that bans whoever keeps coming back reads the service's *own* log, and the service checks
every jail's log file while it configures itself — before it has created that log. The module
declared the jail and shipped the rotation for that log, and never declared the log. On the two
machines where it had run for years the file was simply there, so nothing had ever noticed.
That failure was loud. Fixing it uncovered a second one in the same module that is not.
The module's defaults named `ufw` as the way to ban an address. Two of these four machines have no
`ufw` — they filter with nftables — and nothing checks that until an address is banned. Asked to ban
a documentation address on such a machine, the service accepted the instruction, counted it, ran the
command, and wrote this to a log nobody reads:
```
ERROR ... -- stderr: '/bin/sh: line 5: ufw: command not found'
ERROR ... -- returned 127
ERROR Failed to execute ban jail 'sshd' action 'ufw' ... Error banning 192.0.2.99
```
No rule existed afterwards. Throughout, the unit was `active`, the module was applied, and the
machine's report said so. **A machine had been added to the mesh's intrusion prevention, reported as
protected, and was banning nobody.**
## Why it matters beyond this instance
**The two faults are the same mistake with opposite symptoms.** Both are the module assuming
something about the machine — a file that happens to exist, a program that happens to be installed.
One stopped the service, which anybody notices. The other left it running and empty, which nobody
does. A mesh that only catches the loud one is a mesh whose coverage is unknown.
**"The unit is running" was taken for "the module is doing its job".** That is the only health a
service resource has. It is the right answer for most modules and it is silent for any module whose
work happens later, on an event — a ban, a renewal, a backup, a notification. The report cannot
distinguish "protecting this machine" from "installed and inert".
**And it is exactly the naming rule, one level down.**
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) says a
definition names no node, no mesh and no host path, because the same definition has to raise a
different mesh. `ufw` is not a node name, but it is the same class of assumption: a value the module
cannot know, true on some machines and false on others, written as though it were a constant. The
module already knew how to do better a few lines away — the mesh's own address range is named there
as something the machine fills in.
## What was done
The module declares the log its own jail reads, created once and never touched again, since what
grows in it is the service's and the rotation the module already ships is what keeps it small. And
it bans through the action it ships itself, which every machine here can run, which was already in
use by the other jail on all four, and which covers a container's published port as well as the
host's own.
All four machines now run it, with both jails, and a ban lands on each — verified by banning and
unbanning a documentation address on every one.
## What is still true
**Nothing would have caught either fault before it shipped.** The control plane reads a manifest, not
a machine; `ufw` and `/var/log/…` are strings in a file it has no way to evaluate. The host could in
principle be asked whether a declared program exists, but no resource says "this file names a command
that must be there", so there is nothing to check.
**Two machines' bans from before this are stale rules in the old front-end**, which the service no
longer knows about and will never lift. They reject two addresses for ever. Harmless, and a reminder
that changing how a module enforces something leaves what it already enforced behind.
## Open questions
- What does a service resource's health mean for a module whose work is event-driven? A unit being
active is the weakest claim available, and four of this mesh's modules are of that kind.
- Should a declaration be able to say that a resource depends on a program, so the machine can refuse
what it cannot carry out rather than reporting success?
- Where should the packet filter a module bans through come from — the module's own choice, as now,
or the seat that owns the machine's filtering?