Compare commits

..
Author SHA1 Message Date
mesh-admin d6b62387f2 Merge pull request 'Issue 135: a container's mesh names are not compared' (#165) from issue/135-a-containers-mesh-names into main 2026-09-28 15:19:37 +00:00
jschoubben 4789624857 Issue 135: a container's mesh names are not compared, so a moved address is never noticed
One container restarted 2286 times over five days while the mesh reported the machine as doing what it
was told. Its overlay address was five days out of date: the host compares a container by a digest of
its spec, and the mesh's names were not in it, so a container whose image and files never changed was
left alone holding a name that no longer resolved. Forty-eight others were current only because
something else had recreated them.

The same fault as issue 045, in the field that was left out. Resolved by putting the names in the
digest.
2026-09-28 17:19:35 +02:00
mesh-admin e00862e317 Merge pull request 'ADR 0112 is accepted, and issue 134 records what it is not yet' (#164) from decision/0112-accepted-and-issue-134 into main 2026-09-28 15:12:15 +00:00
jschoubben 3a92e4b80c ADR 0112 is accepted, and issue 134 records what it is not yet
0120 was already accepted; what failed the check was that it rests on 0112, still marked proposed —
and so do four designs. The decision stands: a definition names no node, no mesh and no host path, and
everything a module needs is a requirement the mesh resolves.

Accepting it makes the gap visible rather than hiding it, so issue 134 states it. 0112 says how it is
checked — 'a catalogue test finds no domain name in any definition value' — and there is no such test.
Asked by hand: seven modules name this installation in a value the mesh acts on, and eight mention a
public name in prose nothing reads. The two are not the same fault and the fixes differ, which is why
the issue separates them rather than counting to fifteen.
2026-09-28 17:12:12 +02:00
mesh-admin bfddf78bf3 Merge pull request 'Design 32: what shipped today, and the records it rests on' (#163) from design/28-and-32-what-shipped into main 2026-09-28 14:50:44 +00:00
jschoubben 0b291d89f3 Design 32: what shipped today, and the records it rests on
Two of its statements are built — a version preparing its state, and the mesh saying what it applied —
so the document is in-progress rather than proposed, and names the code that owns them. Resting on a
live record rather than a superseded one: 0127 was replaced by 0131.

What this exposes is pre-existing: it also rests on ADR 0112, which is still proposed, and a document
that is not itself proposed may not. ADR 0120 has rested on it the same way for a while. Accepting or
superseding 0112 is a decision, not a cleanup, so it stays visible in the check rather than papered
over.
2026-09-28 16:50:41 +02:00
mesh-admin a924efcc28 Merge pull request 'ADR 0136: a step gates its module, not the machine' (#162) from decision/0136-a-step-gates-its-module into main 2026-09-28 13:38:18 +00:00
jschoubben b421c73a5a ADR 0136: a step gates its module, not the machine
ADR 0135 made a step something the mesh derives for any module that prepares its state, which turned
ADR 0052's reach into a fault: a module whose database is briefly unreachable would stop every module
declared after it on that machine — the fault issue 011 already removed for every other shape, and
the reason the catalogue migrates itself at start rather than in a step.

A step now stops the rest of its own module and nothing else; an action still gates the machine,
because genesis is a row of them and they belong to no module. What was not attempted is reported as
skipped rather than left to be inferred from silence.
2026-09-28 15:38:16 +02:00
mesh-admin 017b1401e4 Merge pull request 'ADR 0135: a module version prepares its state before it runs' (#161) from decision/0133-0134-migrations-and-deploy-facts into main 2026-09-28 09:57:47 +00:00
mesh-admin 74ba3ff1d4 Merge pull request 'ADR 0133 and 0134: who runs migrations, and the mesh saying what it applied' (#160) from decision/0133-0134-migrations-and-deploy-facts into main 2026-09-28 09:45:49 +00:00
6 changed files with 264 additions and 9 deletions
@@ -1,6 +1,6 @@
---
topic: what runs on it
status: proposed
status: accepted
date: 2026-09-25
deciders: jochen
reconstructed: false
@@ -0,0 +1,106 @@
---
topic: what runs on it
status: accepted
date: 2026-09-28
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md
---
# 136. A step gates its module, not the machine
## Context
[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) made a run-once container a step the host
runs to completion, and gave it the same reach a failed action has: it stops everything the declaration
places after it. When the only steps on the mesh were a broker's seed and a forge's admin account, that
reach was invisible — the thing after the step was the container the step existed for, in the same
module.
[ADR 0135](0135-a-module-version-prepares-its-state-before-it-runs.md) made a step something the mesh
derives for **any** module that prepares its state, and that turns the reach into a fault. A module
whose database is briefly unreachable now stops every module declared after it on that machine, for as
long as it is unreachable.
**The host already rejected this for every other shape, and says why in its own loop.** From
[issue 011](../04-ISSUES/011-one-broken-module-blocks-every-other/00-report.md):
> It used to stop at the first one, and that made one broken resource hold the whole machine hostage: a
> module declaring a package that does not exist meant every module ordered after it was never applied,
> for ever, and the mesh reported "failed" without saying that the rest had not been tried. A machine
> with one bad module and nine good ones ran none of the nine.
Everything is attempted and every failure reported — except an action and a run-once step, kept as the
deliberate exceptions. So the mesh has two rules about the same question and the wider one is now
reachable by any module that declares a schema.
**And it deadlocks a case the catalogue already named.** The catalogue migrates its own schema when it
starts rather than in a step, and says why in its code: *a schema step that had to reach the provider
over the overlay would block the very apply that brings the overlay up*. With a machine-wide gate that
is exactly right — the step fails, the apply stops, the overlay module after it is never applied, and
the next reconcile is blocked the same way. The module that most obviously wants a step could not have
one.
## Decision
**A step gates its own module.** A run-once container that does not complete stops the rest of *that
module's* resources and nothing else. Every other module on the machine is attempted, as every other
shape already is.
**An action still gates the machine.** Genesis is a row of actions, each making the next possible, and
they belong to no module — there is nothing narrower for their reach to be.
**What was not attempted is reported, not inferred from silence.** A skipped resource appears in the
machine's account of the apply as skipped, with the reason, because "not attempted" and "nothing to do"
are different answers and only one of them is somebody's to fix.
**A module is the part of a resource's identity before the first dot**, which is how the mesh composes
them. What the mesh declares in its own right — a guard, an opening, the adoption's own resources —
belongs to no module, and its gate is therefore the machine's.
## Options considered
1. **Leave the reach as it is.** Rejected: it reintroduces, through a mechanism now derived for every
module, exactly the fault issue 011 removed. A mesh where one module's unreachable database stops a
machine converging is worse than one where that module alone is behind.
2. **Make preparation not a gate at all** — run it and carry on. Rejected: then a version serves against
a state nobody shaped, which is the whole of what ADR 0135 exists to prevent.
3. **Order every module's step before everything else on the machine**, so a gate stops nothing that
matters. Rejected: it inverts the order a module needs — its files and directories are declared before
its step because the step reads them — and it would still stop later modules.
4. **Let a module declare how far its step reaches.** Rejected: the answer is the same for every module,
and a field would let one be wrong about it.
## Consequences
**The catalogue can move to a step.** The reason it migrates at start — that a step blocks the apply
that would make its provider reachable — stops being true: the step fails, that module waits, the
overlay comes up, and the next reconcile prepares it. One shape for the whole mesh, which is what
ADR 0135 asked for and could not have had.
**A module can sit behind while the machine is otherwise current.** That is the honest state and it is
what the report now says. It also means a preparation that never succeeds is a module that never
upgrades, quietly, until somebody reads the report — which is an argument for
[ADR 0134](0134-the-mesh-says-what-it-applied.md) rather than against this.
**A module's resources must be ordered within the module for the gate to mean anything.** They already
are: the mesh composes a module's resources in the order its manifest declares them, and its own
workload comes after the files it reads.
## How this is checked
- **A failed step stops its module and nothing else.** A test with two modules: the one whose step
failed does not start its workload, the other starts, and the error still says the failure gated
something. It fails against the previous behaviour, which is how it was written.
- **An action still stops the machine.** The existing test for a failed action is unchanged, and a step
with no module in its identity — which is what genesis carries — takes the same path.
- **The report names what was skipped.** Asserted in the same test, because a gate nobody can see is
indistinguishable from a module that had nothing to do.
## References
- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the step this narrows
- [ADR 0135](0135-a-module-version-prepares-its-state-before-it-runs.md) — what made the reach reachable
- [issue 011](../04-ISSUES/011-one-broken-module-blocks-every-other/00-report.md) — the same fault, removed once already
- [ADR 0134](0134-the-mesh-says-what-it-applied.md) — how a module left behind becomes visible
- mesh-host `internal/apply` — the loop whose own comment argued this case for every other shape
+2 -1
View File
@@ -205,7 +205,7 @@ python3 00-META/checks/index.py fail if stale
- **0091** — [A mount is declared, and there are three things it can be](0091-a-mount-is-declared-three-ways.md)
- **0099** — [A step that runs once names what it reads, and runs again when it changed](0099-a-step-that-runs-once-names-what-it-reads.md)
- **0110** — [A seat is held by one assignment, from a closed set, and it may deliver a provision](0110-a-seat-is-a-module-assignment-from-a-closed-set.md)
- **0112** — [A module definition names no node, no mesh and no path: everything it needs is a requirement the mesh resolves](0112-a-module-definition-names-no-node-mesh-or-path.md) *(proposed)*
- **0112** — [A module definition names no node, no mesh and no path: everything it needs is a requirement the mesh resolves](0112-a-module-definition-names-no-node-mesh-or-path.md)
- **0113** — [The vault makes every shared secret, a provider makes resources and data, and the mesh carries both](0113-the-vault-makes-every-secret.md) *(proposed)*
- **0114** — [A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached](0114-a-shared-credential-rotates-over-two-credentials.md) *(proposed)*
- **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md) *(proposed)*
@@ -216,6 +216,7 @@ python3 00-META/checks/index.py fail if stale
- **0122** — [A seat is data the controller owns, and a rename is a database update](0122-a-seat-is-data-a-rename-is-a-database-update.md)
- **0133** — [A module owns its migrations, and the mesh owns when they run](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) *(superseded)*
- **0135** — [A module version prepares its state before it runs](0135-a-module-version-prepares-its-state-before-it-runs.md)
- **0136** — [A step gates its module, not the machine](0136-a-step-gates-its-module-not-the-machine.md)
### How it is built
@@ -1,11 +1,20 @@
---
layer: to-be
status: proposed
code: []
status: in-progress
code:
- mesh-controller internal/catalogue/declaration.go
- mesh-controller internal/catalogue/manifest.go
- mesh-controller internal/link/serve.go
- mesh-controller internal/link/bus.go
- mesh-controller internal/broker/nats.go
- mesh-controller internal/inventory/nodes.go
- mesh-host internal/apply/apply.go
- mesh-tools src/main.ts
- mesh-catalog modules/mesh-catalog
updated: 2026-09-28
decisions:
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
- 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
- 02-DECISIONS/0128-the-mesh-bus-is-required-not-ambient.md
- 02-DECISIONS/0106-the-bus-is-nats.md
- 02-DECISIONS/0041-events-are-a-relationship.md
@@ -13,6 +22,7 @@ decisions:
- 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md
- 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
- 02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md
- 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
- 02-DECISIONS/0134-the-mesh-says-what-it-applied.md
---
@@ -260,11 +270,13 @@ queue.
and publishes it last-per-subject. A node that was away gets exactly the current one, never a
queue of superseded ones, and a replayed older one is refused by sequence.
**A version prepares its state before it runs.** A module version may declare an entrypoint that brings
its state to the shape that version needs — the same vocabulary as the entrypoints it declares for its
**A version prepares its state before it runs.** *Built 2026-09-28.* A module version may declare an
entrypoint that brings its state to the shape that version needs — the same vocabulary as the entrypoints it declares for its
tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it
runs the module's own code, to completion, in the module's own context, and a version whose preparation
did not succeed does not run: the rollout stops at the first machine that did not take it
did not succeed does not run: the step gates that module and nothing else on the machine
([ADR 0136](../../02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md)), and the rollout stops
at the first machine that did not take it
([ADR 0135](../../02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md), superseding
[ADR 0133](../../02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)).
Once per state, and the mesh derives what a state is: a consumer is a module on a machine, so what the
@@ -274,7 +286,7 @@ mesh provisions is per consumer and preparation is too. No level to choose, and
not to an address it was given at genesis. Held and retried while the store restarts
([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)).
**And the mesh says what it applied.** A report is control traffic only the control plane reads, so the
**And the mesh says what it applied.** *Built 2026-09-28.* A report is control traffic only the control plane reads, so the
chain above went dark at the moment it touched a machine: nothing said which version a machine now runs,
or that it refused to. The control plane states those as facts under its own seat's namespace, when what
a machine runs changes rather than on every convergence pass, and anything that cares subscribes the way
@@ -0,0 +1,66 @@
---
status: open
opened: 2026-09-28
located-in: [mesh-catalog, mesh-controller internal/catalogue]
fixed-by:
amended-design:
---
# 134 — A definition may still name the mesh, and the check that would say so does not exist
## What was observed
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) says a module
definition names no node, no mesh and no host path, and states how that is checked:
> A catalogue test finds no domain name in any definition value.
There is no such test. Run by hand on 2026-09-28, across the 72 manifests in the catalogue, the
question it asks has 15 answers. They are not all the same kind of thing, and the difference matters
more than the count:
**Values the mesh acts on** — seven:
| module | where | what it names |
|---|---|---|
| keycloak | `env.KC_HOSTNAME` | this installation's public name for itself |
| minio | `env.MINIO_BROWSER_REDIRECT_URL` | the same, for its console |
| invoicing | a resource's `image` | a named registry rather than the mesh's artifact store |
| builder | `build.artifacts[].context.repository` | the forge, by URL |
| route-proxy | `build.artifacts[].context.repository` | the forge, by URL |
| route-adapter | a resource's `content` | a proxy's dynamic configuration |
| novox.be | `module` | the module is named after the domain it serves |
**Prose** — eight, in `listens[].why`: de-spiegel, mailu, n8n, only-office, photos, photos-eef,
photos-filip, portainer. Each explains what a port is for and mentions the public name it is reached
by. Nothing reads these; a check written as a string search would report them, and reporting them as
violations of the same rule would be wrong.
## Why it matters beyond this instance
**An unenforced rule is indistinguishable from a wrong one, and costs more, because people believe
it.** The record says the mesh is name-agnostic, four design documents rest on that, and a reader
checking whether it holds finds that it does not — in the places that matter most. The two forge URLs
are what a build reaches into for its source; the two hostnames are what a service tells a browser
about itself.
**It is the difference between a mesh and this mesh.** A definition carrying `novox.be` is a
definition that can only be installed here. The whole point of the rule is that the same catalogue
raises a different mesh with a different name, and today seven modules would need editing to do it.
**And the shape of the fix is not the same for each.** A public name is an operator's choice about an
assignment, which ADR 0112 already provides for; a forge URL should be a path on the git seat
([ADR 0111](../../02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md)); an image from a named
registry is a question about the artifact store, not about naming. Counting them together would hide
that.
## Open questions
- Does a domain in a `why` string break the rule? It is documentation the mesh never reads, and a
check that cannot tell the two apart will either pass things it should catch or fail things nobody
should change.
- Where does a service's public name live, concretely — a setting on the assignment, or a fact the
mesh composes from the node's domain? ADR 0112 says a requirement the mesh resolves; the two
hostnames above are the first real cases.
- Should a build context name a repository on the git seat rather than by URL, and if so, what does
that mean for a context in *another* mesh's forge?
@@ -0,0 +1,70 @@
---
status: resolved
opened: 2026-09-28
located-in: [mesh-host internal/apply]
fixed-by: mesh-host — a container's mesh names are part of the spec digest the host compares, sorted so the digest does not move for a reordering. A container whose names moved is now recreated like a container whose image moved, and the test fails against the previous behaviour.
amended-design:
---
# 135 — A container's mesh names are not compared, so a moved address is never noticed
## What was observed
One container on this mesh had been restarting every thirty seconds for five days — 2286 times — and
the mesh reported the machine as doing what it was told.
Its logs said its database connected and then a query timed out. The database was reachable: the same
query from the same network, with the same credential, answered in milliseconds. What differed was the
name. Inside that container, `novox.internal` resolved to `10.42.0.1`; in every other container on the
machine it resolved to `10.10.0.1`. The mesh's overlay range had moved, and this container still held
the old one:
```
umami created 2026-09-23 novox.internal:10.42.0.1
mesh-catalog created today novox.internal:10.10.0.1
```
A container resolves other machines and public names through the entries the mesh gives it when it is
created, and nothing re-reads them afterwards. The host compares a container against what was declared
by a digest of its spec — image, name, environment, ports, volumes, arguments, resolver, address, and
what it reads — and **the mesh's names were not in it**. So this container matched what was declared,
was left alone, and kept an address that had not existed for five days.
Forty-eight other containers had current names. Not because anything corrected them: each had been
recreated for some other reason — a new image, a changed file — and picked up the current roster on the
way. This one's image is an upstream release that had not moved, and nothing else about it changed, so
nothing ever recreated it.
## Why it matters beyond this instance
**It is the exact fault [issue 045](../045-a-container-keeps-the-values-it-started-with/00-report.md)
named, in the one field that was left out.** That issue is why the digest carries what a container
reads: "a container whose configuration had since been rewritten compared equal and was left alone —
running values the machine no longer holds, while every check reported success." The same sentence
describes this, with *names* in place of *files*.
**The failure is invisible in exactly the way that matters.** The container runs, so the machine
reports it applied. It restarts, but a restarting container is a normal sight during an upgrade. The
only account of the fault is inside the container's own log, in the words of the application rather
than of the mesh — and what it says is that a query timed out, which points at the database.
**And it is most likely to bite what changes least.** Every container that is rebuilt often repairs
itself by accident. The victim is the module whose image is stable — which is to say, the module that
was working fine.
## What was done
The mesh's names are part of the digest, sorted so the digest does not move for a reordering nobody
made. A container whose names moved is now recreated exactly as one whose image moved.
The first apply after this recreates every container that carries mesh names — one restart each,
already the price the mesh pays for any image update — because their recorded digests predate the
field.
## What is still true
The mesh gives a container its names at creation and has no way to change them in place. That is the
container runtime's shape, not a choice; the answer is to recreate, which is what this does. A module
that would rather re-read a roster from a file can already ask for one as a fact
([ADR 0120](../../02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md)) and restart on
it.