Merge pull request 'ADR 0133 and 0134: who runs migrations, and the mesh saying what it applied' (#160) from decision/0133-0134-migrations-and-deploy-facts into main

02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md

@@ -0,0 +1,158 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-09-28
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md
+---
+
+# 133. A module owns its migrations, and the mesh owns when they run
+
+## Context
+
+On 2026-09-28 the mesh replaced its own control plane, through its own upgrade path, with a build
+carrying a migration. Nothing applied it. For the next three quarters of an hour every build the mesh
+made was refused by the store with one line — *column "built_contexts" does not exist* — which reached
+only whoever happened to be waiting on that build's reply. The images were built and published, so the
+registry filled with artifacts the mesh has no record of, and the overview went on reporting that every
+module was current ([issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)).
+
+The schema had been created once, at genesis, by an action in the foundation bundle. Nothing ran it
+again, through many updates of the control plane since.
+
+**The mechanism to do this right already existed and one module used it wrong.**
+[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) makes a run-once container a step the host
+runs to completion before whatever the declaration places after it, and names migrating a schema as the
+case it exists for. Three facts about how it is used today:
+
+- The control plane's manifest had no step at all. The immediate fix was to write one by hand, and that
+  hand-written step repeats three environment variables and three volume mounts from the server
+  resource it precedes — six chances to drift from the thing it prepares.
+- Two other modules hand-write the same shape for the same reason: gitea's admin bootstrap and
+  mosquitto's dynsec seed, each repeating its sibling's image, environment and mounts. One of them
+  ends in `|| true`, which is a lock implemented as a shrug.
+- The catalogue module takes the other road: it migrates its own schema in its own code when it starts.
+  That failure mode is a crash loop rather than a stop — the catalogue restarted 338 times this
+  morning on an unrelated start-time failure, and nothing anywhere said the mesh's graph had a gap.
+
+**What the mesh already has, and what HAL needed stages for.** Ordering a provider before its consumer
+is `providersFirst`, which topologically orders a node's modules. Ordering within a module is
+declaration order, and a run-once container gates everything after it. Remembering that a step has
+already run is the digest of its declaration, recorded only after it exits 0
+([ADR 0018](0018-a-picture-is-read-from-what-runs.md)) — and the image is part of that digest, so a new
+build re-runs it. Three of the four things a stage system provides are therefore already here. The
+fourth — that a module has a schema at all — is the only thing missing.
+
+**Nothing in the catalogue ships a migrations directory.** Of 72 modules, none has one; the modules that
+migrate do it in their own code. So this is not a decision about where SQL files live. It is a decision
+about who runs them and when.
+
+**Two facts bound what is safely expressible.** A node converges toward its own declaration without
+waiting on any other node. And of the five modules that run on more than one machine today — dnsmasq,
+fail2ban, networking, networkmanager, sshd — not one wants a store; every module with a database is on
+exactly one machine.
+
+## Decision
+
+**A container may declare steps to run before it.** The same container, run to completion, with
+different arguments, in order, before it starts. The mesh derives the run-once resources from that
+declaration, so the image, the environment, the volumes, the network and the credentials come from the
+one place they are already described and cannot drift from it.
+
+**A module's migrations are the first user of this, and the module owns them entirely.** The SQL, the
+order, the idempotence, the lock, and which dialect it speaks. The mesh never learns that postgres and
+mssql differ, because it runs the module's own image with the module's own arguments against the
+module's own binding and requires exit 0. A module needing both stores runs one step that does both.
+
+**The mesh owns the moment, and the gate is the guarantee.** Whether a version may serve when its
+schema is not there yet is a deployment question, and the mesh is the only thing that can answer it,
+because the mesh is what starts the container. A step that fails stops the container it precedes, so
+a failed migration is a version that does not serve rather than a version serving against a store it
+does not match.
+
+**Per node, and there is no level.** The step runs wherever the module runs. A step that ran "once,
+somewhere" would leave every other machine with no gate at all, and additive migrations protect old
+code against a new schema, never new code against an old one. The cost is an obligation a migration
+runner already carries: a version table and a lock.
+
+**"Once, mesh-wide" is what holding a seat means.** A step that is not idempotent — seeding an
+account, sending a notice, taking a backup — belongs to a module that holds a seat, where the mesh
+already guarantees one holder, on record, handed over deliberately. That is the answer to the level
+question rather than a field that has to invent an election and keep it somewhere.
+
+**Migrations are forward-only and additive.** The step runs before the *new* container starts, so the
+old one is still running against the new schema for the length of the apply.
+
+**Declared, never inferred.** The control plane cannot see inside an image, so a module that ships
+migrations and declares no step is not refusable at registration; it breaks on its first upgrade. This
+record says so rather than implying a check that cannot exist.
+
+## Options considered
+
+1. **Each module migrates itself when it starts** — what the catalogue does today. Rejected: it turns a
+   schema failure into a crash loop instead of a stop, it is invisible in the declaration so nothing can
+   say the module even has a schema, and two machines running the module both migrate at start with
+   nothing sequencing them.
+2. **The mesh applies migrations itself**, with a driver and a version table per store — HAL's shape.
+   Rejected: the mesh would have to know one store type from another, hold another module's store
+   credentials, and reach a machine to use them, which [ADR 0005](0005-the-node-host.md) forbids. It is
+   also the reason that shape needs levels: something central has to decide where the once happens.
+3. **A hook lifecycle** — pre-build, post-build, pre-deploy, post-deploy. Rejected: there is no deploy
+   event here to hook. A declaration is a desired state applied in order and reconciled forever, so
+   "pre-deploy" is exactly "a step before this container", pre- and post-build are what a Dockerfile and
+   the artifact list already are, and "post-deploy" has no moment to name.
+4. **A hook level** — once per module, or once per module-node assignment. Rejected as a field, kept as
+   a property: see the decision. A once-per-module step needs cross-node ordering underneath it to be
+   safe, and a node converging without waiting on its neighbours is worth losing on purpose rather than
+   by accident.
+5. **Every module hand-writes its own run-once step** — the immediate fix for the control plane.
+   Rejected as the general answer: it duplicates the resource it precedes, in three places already, and
+   a hand-written step is one the next module forgets. Forgetting it is the fault this record exists
+   for.
+6. **Record a schema level per module in the store.** Rejected: gating makes the invariant true by
+   construction, so a level is a second account of the same fact and the first one to go stale.
+
+## Consequences
+
+**Three hand-written steps collapse into one line each**, and the control plane's own migrate step stops
+repeating its server's environment and mounts.
+
+**The catalogue's self-migration becomes the exception to remove.** One shape, and the mesh's own
+control plane is not an exception to it either.
+
+**A module on two machines with one shared store must lock.** Today none is, so this is an obligation
+stated before it is needed rather than discovered by two concurrent migrations.
+
+**There is still no readiness-gated step.** Only an action carries `verify`; a container has no health
+notion, so "run this once the service answers" remains unexpressible and seeding through a running
+service's API has no home. That is its own decision about a container's readiness, and this record does
+not make it.
+
+**Genesis keeps its own action.** At birth there is no control plane to derive anything from, which is
+what [ADR 0067](0067-genesis-is-a-pivot.md) already says about that moment.
+
+## How this is checked
+
+- **The composition carries the step.** A test on a node's composed declaration: every container that
+  declares steps before it is preceded by them, and the derived step's image, environment, volumes and
+  network equal the container's — so the two cannot drift, which is the failure the hand-written kind
+  has.
+- **A failed step stops what follows.** The host already refuses to go on past a run-once step that did
+  not exit 0; the test for that is extended to a derived one, so the gate is checked rather than
+  assumed.
+- **The mesh's own schema is covered by the same mechanism as everything else.** The control plane
+  declares its step in its own manifest, so the case that failed on 2026-09-28 is the case the test
+  covers.
+- **A module claiming a seat for a once-only step is checked where seats are checked** — the conditions
+  of holding, not a new mechanism.
+
+## References
+
+- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the step this extends
+- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — a digest is the record that something happened
+- [ADR 0005](0005-the-node-host.md) — the control plane decides and never touches a machine
+- [ADR 0067](0067-genesis-is-a-pivot.md) — why genesis does it differently, once
+- [issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md) — the failure that produced this record
+- [`03-DESIGN/01-to-be/32-what-a-module-declares.md`](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §6 — the lifecycle this sits in
+- Measured 2026-09-28: three hand-written run-once steps repeating their sibling's resource; 0 of 72 modules with a migrations directory; 5 modules on more than one machine, none of them wanting a store

02-DECISIONS/0134-the-mesh-says-what-it-applied.md

@@ -0,0 +1,126 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-09-28
+deciders: jochen
+reconstructed: false
+---
+
+# 134. The mesh says what it applied
+
+## Context
+
+The pipeline is observable on the bus from a merge to an artifact, and modules already plug into it:
+the forge emits `pull.merged`, the build machine's seat emits `built`, the catalogue emits `registered`,
+`upgraded` and `rebuild-needed`, providers emit `postgres.database.provisioned` and its siblings. Things
+consume them today — the catalogue consumes `built`, each provider consumes its own provisioning events,
+model-usage consumes `*.usage.*`, the audit logger consumes `**`. Nothing had to be invented for any of
+that; subscribing *is* plugging in.
+
+**It goes dark at the moment it touches a machine.** A host applies a declaration and reports to the
+control plane on the control branch, which only the control plane may read — correctly, because a report
+carries what a machine is and enrolment travels the same way. So nothing on the mesh says *this machine
+now runs version Y of module Z*, or that it refused to, or why.
+
+What that cost on 2026-09-28, in one morning:
+
+- A build result the store refused was visible only to whoever was waiting on that build's reply. For
+  three quarters of an hour the mesh built things and recorded none of them, while the overview said
+  every module was current ([issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)).
+- A module crash-looping at start — 338 restarts — was found by reading a container's logs by hand.
+  Nothing on the bus said the mesh's graph had stopped learning.
+- A run-once step that fails now stops an upgrade by design ([ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)),
+  and the same silence would cover it: the version simply would not appear.
+
+**And the one thing the control plane does emit is refused by its own account.** Answering a catalogue
+that asks to catch up, it publishes each recorded build under `mesh.mod.control-plane.event.…` — a
+module namespace for a module that does not exist. Its own permissions refuse it, so a catalogue that
+restarts gets nothing and keeps its gap. The control plane has facts to state and nowhere to state
+them.
+
+## Decision
+
+**The mesh emits the deploy half of the pipeline as facts on the bus.** What a machine now runs, and
+what it refused to run and why. Both are facts about the mesh doing its work, in the same form as every
+other fact on the bus, so anything that wants them subscribes the way the catalogue subscribes to
+`built`.
+
+**The control plane states them, as the holder of the `mesh-controller` seat.** Its facts live under the
+seat's own namespace, which is where a role's events belong
+([ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md),
+[ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md)) and which survives the control plane being
+replaced. That is also what gives the catch-up replay a subject it may publish instead of an invented
+module namespace.
+
+**Emitted when what a machine runs changes, not on every convergence pass.** A host reconciles
+continuously and reports each time; a fact per pass would be a fact per minute per machine that says
+nothing. The report carries the declaration it applied and what changed, so the control plane has what
+it needs to speak only when there is something to say.
+
+**A refusal is a fact with a subject in it** — which machine, which resource, and the reason as the host
+gave it. A refusal that names only the machine is the silence this record is about, one level up.
+
+**Reports stay where they are.** A node's report remains control traffic that only the control plane
+reads. The deploy facts are derived from it, which makes them second-hand on purpose: one emitter, one
+ordering, and no widening of the narrowest account in the mesh.
+
+## Options considered
+
+1. **Leave it as it is, and let whatever cares ask the control plane.** Rejected: asking for a fact that
+   already arrives is the shape the mesh removed everywhere else, and nothing can react at the moment a
+   machine changes — which is exactly when a graph, an audit or an operator wants to know.
+2. **Each node emits its own facts.** Rejected: it widens every host's account to an event namespace, and
+   a host's authority is deliberately the narrowest in the mesh. Its report already reaches the one thing
+   that can speak for it.
+3. **Widen who may read the control branch.** Rejected: that branch carries what machines say *to* the
+   control plane, enrolment included. Widening its readers widens that too, for an unrelated reason.
+4. **A registry of deploy hooks** — something registers interest and is called. Rejected: an event is
+   already the mechanism; there is nothing to register, and a callback is an address the mesh spent
+   [issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
+   learning not to keep.
+5. **Put the facts on the node's own declaration stream.** Rejected: that stream is last-per-subject by
+   design, so a machine away for an hour gets exactly the current declaration and nothing older. A
+   history of what happened cannot live in a stream built to forget.
+
+## Consequences
+
+**The audit logger gets the deploy half for nothing**, because it consumes everything.
+
+**A failure becomes visible where the mesh is watched** rather than where someone happened to be
+looking. That answers the open question [issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)
+left about a record the store refused.
+
+**The catch-up replay stops being a burst of events.** With the control plane able to state its own
+facts, replaying history as if it were happening now is a choice rather than the only option — and the
+better shape is the question the catalogue is actually asking, answered once
+([design 33](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)).
+
+**The facts are second-hand.** The control plane says what a machine reported, so a machine that cannot
+reach the bus produces no fact at all. Absence is not health, and what a machine was last heard from
+stays the place that says so.
+
+**The events stream carries more.** Bounded by emitting on change rather than on every pass, and each
+fact is small; the stream's own limits remain what keeps it finite.
+
+## How this is checked
+
+- **The control plane's grant names exactly the subjects it emits**, derived from its seat like every
+  other principal's, and the composed user list is compared against a golden file — so a fact it cannot
+  publish fails a test rather than a catalogue's replay.
+- **A convergence that changed nothing emits nothing.** A test with two identical reports and one
+  expected fact, because the failure this guards against is a fact per minute per machine.
+- **A refusal names its resource.** A test where a host reports a failed resource and the emitted fact
+  carries which one and why, not merely that something went wrong.
+- **What the mesh emits is what something consumes.** The subject a module declares it consumes derives
+  to the subject the control plane publishes — the same agreement test that already keeps the
+  controller's own subscriptions honest.
+
+## References
+
+- [ADR 0041](0041-events-are-a-relationship.md) — an event is a relationship, not a call
+- [ADR 0126](0126-a-module-declares-its-own-seats.md) — an event is addressed to its emitter, because the emitter's identity is the meaning
+- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md), [ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md) — a role's events belong to the role
+- [ADR 0083](0083-one-push-leaves-the-mesh-consistent.md) — a report is held for the store rather than lost
+- [ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) — the gate whose failure this makes visible
+- [`03-DESIGN/01-to-be/32-what-a-module-declares.md`](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §6 — the lifecycle, which ends today at a report nobody else may read
+- Measured 2026-09-28: 45 minutes of builds recorded nowhere with the overview reporting health; a module at 338 restarts found by hand; the control plane's only emitted event refused by its own permissions

02-DECISIONS/README.md

@@ -141,6 +141,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0130** — [The predecessor is ending, and its broker goes with it](0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)
 - **0131** — [Everything on the mesh speaks to the broker seat, and AMQP is not a provision](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)
 - **0132** — [A seat carries the tools its holder must serve](0132-a-seat-carries-the-tools-its-holder-must-serve.md)
+- **0134** — [The mesh says what it applied](0134-the-mesh-says-what-it-applied.md)
 
 ### Its tiers, from the bottom up
 
@@ -213,6 +214,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0120** — [A roster fact carries its format as a template: the mesh owns the data, the module owns the format](0120-a-roster-fact-carries-its-format-as-a-template.md)
 - **0121** — [A system seat is named for its scope, and a module may define its own](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md)
 - **0122** — [A seat is data the controller owns, and a rename is a database update](0122-a-seat-is-data-a-rename-is-a-database-update.md)
+- **0133** — [A module owns its migrations, and the mesh owns when they run](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)
 
 ### How it is built
 

03-DESIGN/01-to-be/32-what-a-module-declares.md

@@ -2,7 +2,7 @@
 layer: to-be
 status: proposed
 code: []
-updated: 2026-09-27
+updated: 2026-09-28
 decisions:
   - 02-DECISIONS/0126-a-module-declares-its-own-seats.md
   - 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md
@@ -12,6 +12,8 @@ decisions:
   - 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
   - 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md
   - 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
+  - 02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md
+  - 02-DECISIONS/0134-the-mesh-says-what-it-applied.md
 ---
 
 # 32. What a module declares, and what the bus makes of it
@@ -258,10 +260,26 @@ queue.
 and publishes it last-per-subject. A node that was away gets exactly the current one, never a
 queue of superseded ones, and a replayed older one is refused by sequence.
 
+**A version that needs its store prepared prepares it first.** A container may declare steps to run
+before it — the same container, to completion, with different arguments — and the mesh derives them as
+run-once resources placed ahead of it, so a schema change and the code that needs it arrive together or
+not at all. The module owns what the step does; the mesh owns when it runs, and refuses to start the
+container if it failed ([ADR 0133](../../02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)).
+Per node, because a node converges without waiting on its neighbours; a step that must happen once
+mesh-wide belongs to a module holding a seat, which is what one holder on record already means.
+
 **Applying is reported to a role.** The host applies and reports to the `mesh-controller` seat —
 not to an address it was given at genesis. Held and retried while the store restarts
 ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)).
 
+**And the mesh says what it applied.** A report is control traffic only the control plane reads, so the
+chain above went dark at the moment it touched a machine: nothing said which version a machine now runs,
+or that it refused to. The control plane states those as facts under its own seat's namespace, when what
+a machine runs changes rather than on every convergence pass, and anything that cares subscribes the way
+the catalogue subscribes to `built` ([ADR 0134](../../02-DECISIONS/0134-the-mesh-says-what-it-applied.md)).
+The facts are second-hand by design — one emitter, one ordering — and a machine that cannot reach the bus
+produces none, so absence is not health.
+
 What disappears across that chain is every address. No webhook URL, no registered callback, no
 "which node is the builder on", no controller endpoint baked into a joining node. That is the
 class of bug
@@ -439,6 +457,10 @@ it is the residue of a question the rest of §8 answers and the part a fingerpri
 **Whether a module may declare a seat it does not itself claim** — the contract as one thing, the
 implementation as another, which is how two competing implementations would ever exist.
 
+**Whether a container should have a readiness notion.** Only an action carries `verify`, so a step that
+must run once a service *answers* — seeding through its own API — cannot be declared at all. Named here
+because the steps above make the gap obvious, not because they caused it.
+
 **Whether `consumes` naming another module couples too tightly.** It is kept here deliberately —
 an event's provenance is its meaning — but a consumer of `billing.order.placed` does depend on
 billing existing under that name.
@@ -447,6 +469,11 @@ billing existing under that name.
 
 - **A manifest holds no subject.** A catalogue test: no manifest contains a string matching the
   subject grammar. The rule is worthless if it is followed by convention.
+- **A derived step is the container it precedes.** A composition test: the step's image, environment,
+  volumes and network equal that container's, so the two cannot drift — which is the failure the
+  hand-written kind has, three times over in the catalogue today.
+- **A convergence that changed nothing says nothing.** Two identical reports, one emitted fact: what is
+  guarded against is a fact per minute per machine, which is a stream nobody reads.
 - **Permissions are exactly the three namespaces.** A composition test per module: the derived
   permission set equals what its declaration implies, and a hand-written addition to it fails.
 - **A sender cannot read the queue it writes to.** A bed: a module declaring `uses` is refused

04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md

@@ -3,7 +3,7 @@ status: resolved
 opened: 2026-09-28
 located-in: [mesh-controller module.json]
 fixed-by: mesh-controller — the control plane's module declares a run-once `migrate` step before its server, which is the shape ADR 0052 prescribes for exactly this. A step's record of having run is the digest of its declaration and the image is part of that digest, so a new build of the control plane re-runs it; and because a run-once step gates what the declaration places after it, a migration that fails stops the new server from starting at all rather than letting it run against a schema it does not have.
-amended-design:
+amended-design: 03-DESIGN/01-to-be/32-what-a-module-declares.md
 ---
 
 # 133 — The control plane's schema is migrated at birth and never again
@@ -54,6 +54,14 @@ such step; it went straight from a state directory to the server.
 server starts, which means the old binary briefly runs against the new schema. A migration that
 removes or renames something would break the running control plane in the window between the two.
 
+**A hand-written step is one the next module forgets**, which is why this fix is not where the matter
+ends: [ADR 0133](../../02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)
+makes the step derived — a container declares what must run before it, and the mesh composes the gated
+resource — so the control plane stops being the only module that had to remember. The same record
+settles the level question HAL answered with stages, and
+[ADR 0134](../../02-DECISIONS/0134-the-mesh-says-what-it-applied.md) answers the second open question
+below: what a machine applied, and what it refused, become facts on the bus rather than a line in a log.
+
 **The mesh now has two shapes for one problem.** The catalogue module migrates its own schema in its
 own code when it starts; the control plane migrates in a step the host gates on. Both work and the
 reasons differ — a module that owns its store entirely can do it at start, while a step is visible in