diff --git a/02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md b/02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md new file mode 100644 index 0000000..a73092e --- /dev/null +++ b/02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md @@ -0,0 +1,158 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-28 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md +--- + +# 133. A module owns its migrations, and the mesh owns when they run + +## Context + +On 2026-09-28 the mesh replaced its own control plane, through its own upgrade path, with a build +carrying a migration. Nothing applied it. For the next three quarters of an hour every build the mesh +made was refused by the store with one line — *column "built_contexts" does not exist* — which reached +only whoever happened to be waiting on that build's reply. The images were built and published, so the +registry filled with artifacts the mesh has no record of, and the overview went on reporting that every +module was current ([issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)). + +The schema had been created once, at genesis, by an action in the foundation bundle. Nothing ran it +again, through many updates of the control plane since. + +**The mechanism to do this right already existed and one module used it wrong.** +[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) makes a run-once container a step the host +runs to completion before whatever the declaration places after it, and names migrating a schema as the +case it exists for. Three facts about how it is used today: + +- The control plane's manifest had no step at all. The immediate fix was to write one by hand, and that + hand-written step repeats three environment variables and three volume mounts from the server + resource it precedes — six chances to drift from the thing it prepares. +- Two other modules hand-write the same shape for the same reason: gitea's admin bootstrap and + mosquitto's dynsec seed, each repeating its sibling's image, environment and mounts. One of them + ends in `|| true`, which is a lock implemented as a shrug. +- The catalogue module takes the other road: it migrates its own schema in its own code when it starts. + That failure mode is a crash loop rather than a stop — the catalogue restarted 338 times this + morning on an unrelated start-time failure, and nothing anywhere said the mesh's graph had a gap. + +**What the mesh already has, and what HAL needed stages for.** Ordering a provider before its consumer +is `providersFirst`, which topologically orders a node's modules. Ordering within a module is +declaration order, and a run-once container gates everything after it. Remembering that a step has +already run is the digest of its declaration, recorded only after it exits 0 +([ADR 0018](0018-a-picture-is-read-from-what-runs.md)) — and the image is part of that digest, so a new +build re-runs it. Three of the four things a stage system provides are therefore already here. The +fourth — that a module has a schema at all — is the only thing missing. + +**Nothing in the catalogue ships a migrations directory.** Of 72 modules, none has one; the modules that +migrate do it in their own code. So this is not a decision about where SQL files live. It is a decision +about who runs them and when. + +**Two facts bound what is safely expressible.** A node converges toward its own declaration without +waiting on any other node. And of the five modules that run on more than one machine today — dnsmasq, +fail2ban, networking, networkmanager, sshd — not one wants a store; every module with a database is on +exactly one machine. + +## Decision + +**A container may declare steps to run before it.** The same container, run to completion, with +different arguments, in order, before it starts. The mesh derives the run-once resources from that +declaration, so the image, the environment, the volumes, the network and the credentials come from the +one place they are already described and cannot drift from it. + +**A module's migrations are the first user of this, and the module owns them entirely.** The SQL, the +order, the idempotence, the lock, and which dialect it speaks. The mesh never learns that postgres and +mssql differ, because it runs the module's own image with the module's own arguments against the +module's own binding and requires exit 0. A module needing both stores runs one step that does both. + +**The mesh owns the moment, and the gate is the guarantee.** Whether a version may serve when its +schema is not there yet is a deployment question, and the mesh is the only thing that can answer it, +because the mesh is what starts the container. A step that fails stops the container it precedes, so +a failed migration is a version that does not serve rather than a version serving against a store it +does not match. + +**Per node, and there is no level.** The step runs wherever the module runs. A step that ran "once, +somewhere" would leave every other machine with no gate at all, and additive migrations protect old +code against a new schema, never new code against an old one. The cost is an obligation a migration +runner already carries: a version table and a lock. + +**"Once, mesh-wide" is what holding a seat means.** A step that is not idempotent — seeding an +account, sending a notice, taking a backup — belongs to a module that holds a seat, where the mesh +already guarantees one holder, on record, handed over deliberately. That is the answer to the level +question rather than a field that has to invent an election and keep it somewhere. + +**Migrations are forward-only and additive.** The step runs before the *new* container starts, so the +old one is still running against the new schema for the length of the apply. + +**Declared, never inferred.** The control plane cannot see inside an image, so a module that ships +migrations and declares no step is not refusable at registration; it breaks on its first upgrade. This +record says so rather than implying a check that cannot exist. + +## Options considered + +1. **Each module migrates itself when it starts** — what the catalogue does today. Rejected: it turns a + schema failure into a crash loop instead of a stop, it is invisible in the declaration so nothing can + say the module even has a schema, and two machines running the module both migrate at start with + nothing sequencing them. +2. **The mesh applies migrations itself**, with a driver and a version table per store — HAL's shape. + Rejected: the mesh would have to know one store type from another, hold another module's store + credentials, and reach a machine to use them, which [ADR 0005](0005-the-node-host.md) forbids. It is + also the reason that shape needs levels: something central has to decide where the once happens. +3. **A hook lifecycle** — pre-build, post-build, pre-deploy, post-deploy. Rejected: there is no deploy + event here to hook. A declaration is a desired state applied in order and reconciled forever, so + "pre-deploy" is exactly "a step before this container", pre- and post-build are what a Dockerfile and + the artifact list already are, and "post-deploy" has no moment to name. +4. **A hook level** — once per module, or once per module-node assignment. Rejected as a field, kept as + a property: see the decision. A once-per-module step needs cross-node ordering underneath it to be + safe, and a node converging without waiting on its neighbours is worth losing on purpose rather than + by accident. +5. **Every module hand-writes its own run-once step** — the immediate fix for the control plane. + Rejected as the general answer: it duplicates the resource it precedes, in three places already, and + a hand-written step is one the next module forgets. Forgetting it is the fault this record exists + for. +6. **Record a schema level per module in the store.** Rejected: gating makes the invariant true by + construction, so a level is a second account of the same fact and the first one to go stale. + +## Consequences + +**Three hand-written steps collapse into one line each**, and the control plane's own migrate step stops +repeating its server's environment and mounts. + +**The catalogue's self-migration becomes the exception to remove.** One shape, and the mesh's own +control plane is not an exception to it either. + +**A module on two machines with one shared store must lock.** Today none is, so this is an obligation +stated before it is needed rather than discovered by two concurrent migrations. + +**There is still no readiness-gated step.** Only an action carries `verify`; a container has no health +notion, so "run this once the service answers" remains unexpressible and seeding through a running +service's API has no home. That is its own decision about a container's readiness, and this record does +not make it. + +**Genesis keeps its own action.** At birth there is no control plane to derive anything from, which is +what [ADR 0067](0067-genesis-is-a-pivot.md) already says about that moment. + +## How this is checked + +- **The composition carries the step.** A test on a node's composed declaration: every container that + declares steps before it is preceded by them, and the derived step's image, environment, volumes and + network equal the container's — so the two cannot drift, which is the failure the hand-written kind + has. +- **A failed step stops what follows.** The host already refuses to go on past a run-once step that did + not exit 0; the test for that is extended to a derived one, so the gate is checked rather than + assumed. +- **The mesh's own schema is covered by the same mechanism as everything else.** The control plane + declares its step in its own manifest, so the case that failed on 2026-09-28 is the case the test + covers. +- **A module claiming a seat for a once-only step is checked where seats are checked** — the conditions + of holding, not a new mechanism. + +## References + +- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the step this extends +- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — a digest is the record that something happened +- [ADR 0005](0005-the-node-host.md) — the control plane decides and never touches a machine +- [ADR 0067](0067-genesis-is-a-pivot.md) — why genesis does it differently, once +- [issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md) — the failure that produced this record +- [`03-DESIGN/01-to-be/32-what-a-module-declares.md`](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §6 — the lifecycle this sits in +- Measured 2026-09-28: three hand-written run-once steps repeating their sibling's resource; 0 of 72 modules with a migrations directory; 5 modules on more than one machine, none of them wanting a store diff --git a/02-DECISIONS/0134-the-mesh-says-what-it-applied.md b/02-DECISIONS/0134-the-mesh-says-what-it-applied.md new file mode 100644 index 0000000..6e325b4 --- /dev/null +++ b/02-DECISIONS/0134-the-mesh-says-what-it-applied.md @@ -0,0 +1,126 @@ +--- +topic: the mesh +status: accepted +date: 2026-09-28 +deciders: jochen +reconstructed: false +--- + +# 134. The mesh says what it applied + +## Context + +The pipeline is observable on the bus from a merge to an artifact, and modules already plug into it: +the forge emits `pull.merged`, the build machine's seat emits `built`, the catalogue emits `registered`, +`upgraded` and `rebuild-needed`, providers emit `postgres.database.provisioned` and its siblings. Things +consume them today — the catalogue consumes `built`, each provider consumes its own provisioning events, +model-usage consumes `*.usage.*`, the audit logger consumes `**`. Nothing had to be invented for any of +that; subscribing *is* plugging in. + +**It goes dark at the moment it touches a machine.** A host applies a declaration and reports to the +control plane on the control branch, which only the control plane may read — correctly, because a report +carries what a machine is and enrolment travels the same way. So nothing on the mesh says *this machine +now runs version Y of module Z*, or that it refused to, or why. + +What that cost on 2026-09-28, in one morning: + +- A build result the store refused was visible only to whoever was waiting on that build's reply. For + three quarters of an hour the mesh built things and recorded none of them, while the overview said + every module was current ([issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)). +- A module crash-looping at start — 338 restarts — was found by reading a container's logs by hand. + Nothing on the bus said the mesh's graph had stopped learning. +- A run-once step that fails now stops an upgrade by design ([ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)), + and the same silence would cover it: the version simply would not appear. + +**And the one thing the control plane does emit is refused by its own account.** Answering a catalogue +that asks to catch up, it publishes each recorded build under `mesh.mod.control-plane.event.…` — a +module namespace for a module that does not exist. Its own permissions refuse it, so a catalogue that +restarts gets nothing and keeps its gap. The control plane has facts to state and nowhere to state +them. + +## Decision + +**The mesh emits the deploy half of the pipeline as facts on the bus.** What a machine now runs, and +what it refused to run and why. Both are facts about the mesh doing its work, in the same form as every +other fact on the bus, so anything that wants them subscribes the way the catalogue subscribes to +`built`. + +**The control plane states them, as the holder of the `mesh-controller` seat.** Its facts live under the +seat's own namespace, which is where a role's events belong +([ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md), +[ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md)) and which survives the control plane being +replaced. That is also what gives the catch-up replay a subject it may publish instead of an invented +module namespace. + +**Emitted when what a machine runs changes, not on every convergence pass.** A host reconciles +continuously and reports each time; a fact per pass would be a fact per minute per machine that says +nothing. The report carries the declaration it applied and what changed, so the control plane has what +it needs to speak only when there is something to say. + +**A refusal is a fact with a subject in it** — which machine, which resource, and the reason as the host +gave it. A refusal that names only the machine is the silence this record is about, one level up. + +**Reports stay where they are.** A node's report remains control traffic that only the control plane +reads. The deploy facts are derived from it, which makes them second-hand on purpose: one emitter, one +ordering, and no widening of the narrowest account in the mesh. + +## Options considered + +1. **Leave it as it is, and let whatever cares ask the control plane.** Rejected: asking for a fact that + already arrives is the shape the mesh removed everywhere else, and nothing can react at the moment a + machine changes — which is exactly when a graph, an audit or an operator wants to know. +2. **Each node emits its own facts.** Rejected: it widens every host's account to an event namespace, and + a host's authority is deliberately the narrowest in the mesh. Its report already reaches the one thing + that can speak for it. +3. **Widen who may read the control branch.** Rejected: that branch carries what machines say *to* the + control plane, enrolment included. Widening its readers widens that too, for an unrelated reason. +4. **A registry of deploy hooks** — something registers interest and is called. Rejected: an event is + already the mechanism; there is nothing to register, and a callback is an address the mesh spent + [issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md) + learning not to keep. +5. **Put the facts on the node's own declaration stream.** Rejected: that stream is last-per-subject by + design, so a machine away for an hour gets exactly the current declaration and nothing older. A + history of what happened cannot live in a stream built to forget. + +## Consequences + +**The audit logger gets the deploy half for nothing**, because it consumes everything. + +**A failure becomes visible where the mesh is watched** rather than where someone happened to be +looking. That answers the open question [issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md) +left about a record the store refused. + +**The catch-up replay stops being a burst of events.** With the control plane able to state its own +facts, replaying history as if it were happening now is a choice rather than the only option — and the +better shape is the question the catalogue is actually asking, answered once +([design 33](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)). + +**The facts are second-hand.** The control plane says what a machine reported, so a machine that cannot +reach the bus produces no fact at all. Absence is not health, and what a machine was last heard from +stays the place that says so. + +**The events stream carries more.** Bounded by emitting on change rather than on every pass, and each +fact is small; the stream's own limits remain what keeps it finite. + +## How this is checked + +- **The control plane's grant names exactly the subjects it emits**, derived from its seat like every + other principal's, and the composed user list is compared against a golden file — so a fact it cannot + publish fails a test rather than a catalogue's replay. +- **A convergence that changed nothing emits nothing.** A test with two identical reports and one + expected fact, because the failure this guards against is a fact per minute per machine. +- **A refusal names its resource.** A test where a host reports a failed resource and the emitted fact + carries which one and why, not merely that something went wrong. +- **What the mesh emits is what something consumes.** The subject a module declares it consumes derives + to the subject the control plane publishes — the same agreement test that already keeps the + controller's own subscriptions honest. + +## References + +- [ADR 0041](0041-events-are-a-relationship.md) — an event is a relationship, not a call +- [ADR 0126](0126-a-module-declares-its-own-seats.md) — an event is addressed to its emitter, because the emitter's identity is the meaning +- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md), [ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md) — a role's events belong to the role +- [ADR 0083](0083-one-push-leaves-the-mesh-consistent.md) — a report is held for the store rather than lost +- [ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) — the gate whose failure this makes visible +- [`03-DESIGN/01-to-be/32-what-a-module-declares.md`](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §6 — the lifecycle, which ends today at a report nobody else may read +- Measured 2026-09-28: 45 minutes of builds recorded nowhere with the overview reporting health; a module at 338 restarts found by hand; the control plane's only emitted event refused by its own permissions diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index d9ea7b9..8e06a63 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -141,6 +141,7 @@ python3 00-META/checks/index.py fail if stale - **0130** — [The predecessor is ending, and its broker goes with it](0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md) - **0131** — [Everything on the mesh speaks to the broker seat, and AMQP is not a provision](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) - **0132** — [A seat carries the tools its holder must serve](0132-a-seat-carries-the-tools-its-holder-must-serve.md) +- **0134** — [The mesh says what it applied](0134-the-mesh-says-what-it-applied.md) ### Its tiers, from the bottom up @@ -213,6 +214,7 @@ python3 00-META/checks/index.py fail if stale - **0120** — [A roster fact carries its format as a template: the mesh owns the data, the module owns the format](0120-a-roster-fact-carries-its-format-as-a-template.md) - **0121** — [A system seat is named for its scope, and a module may define its own](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) - **0122** — [A seat is data the controller owns, and a rename is a database update](0122-a-seat-is-data-a-rename-is-a-database-update.md) +- **0133** — [A module owns its migrations, and the mesh owns when they run](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) ### How it is built diff --git a/03-DESIGN/01-to-be/32-what-a-module-declares.md b/03-DESIGN/01-to-be/32-what-a-module-declares.md index 0d2dcdf..1987f5e 100644 --- a/03-DESIGN/01-to-be/32-what-a-module-declares.md +++ b/03-DESIGN/01-to-be/32-what-a-module-declares.md @@ -2,7 +2,7 @@ layer: to-be status: proposed code: [] -updated: 2026-09-27 +updated: 2026-09-28 decisions: - 02-DECISIONS/0126-a-module-declares-its-own-seats.md - 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md @@ -12,6 +12,8 @@ decisions: - 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md - 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md - 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md + - 02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md + - 02-DECISIONS/0134-the-mesh-says-what-it-applied.md --- # 32. What a module declares, and what the bus makes of it @@ -258,10 +260,26 @@ queue. and publishes it last-per-subject. A node that was away gets exactly the current one, never a queue of superseded ones, and a replayed older one is refused by sequence. +**A version that needs its store prepared prepares it first.** A container may declare steps to run +before it — the same container, to completion, with different arguments — and the mesh derives them as +run-once resources placed ahead of it, so a schema change and the code that needs it arrive together or +not at all. The module owns what the step does; the mesh owns when it runs, and refuses to start the +container if it failed ([ADR 0133](../../02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)). +Per node, because a node converges without waiting on its neighbours; a step that must happen once +mesh-wide belongs to a module holding a seat, which is what one holder on record already means. + **Applying is reported to a role.** The host applies and reports to the `mesh-controller` seat — not to an address it was given at genesis. Held and retried while the store restarts ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)). +**And the mesh says what it applied.** A report is control traffic only the control plane reads, so the +chain above went dark at the moment it touched a machine: nothing said which version a machine now runs, +or that it refused to. The control plane states those as facts under its own seat's namespace, when what +a machine runs changes rather than on every convergence pass, and anything that cares subscribes the way +the catalogue subscribes to `built` ([ADR 0134](../../02-DECISIONS/0134-the-mesh-says-what-it-applied.md)). +The facts are second-hand by design — one emitter, one ordering — and a machine that cannot reach the bus +produces none, so absence is not health. + What disappears across that chain is every address. No webhook URL, no registered callback, no "which node is the builder on", no controller endpoint baked into a joining node. That is the class of bug @@ -439,6 +457,10 @@ it is the residue of a question the rest of §8 answers and the part a fingerpri **Whether a module may declare a seat it does not itself claim** — the contract as one thing, the implementation as another, which is how two competing implementations would ever exist. +**Whether a container should have a readiness notion.** Only an action carries `verify`, so a step that +must run once a service *answers* — seeding through its own API — cannot be declared at all. Named here +because the steps above make the gap obvious, not because they caused it. + **Whether `consumes` naming another module couples too tightly.** It is kept here deliberately — an event's provenance is its meaning — but a consumer of `billing.order.placed` does depend on billing existing under that name. @@ -447,6 +469,11 @@ billing existing under that name. - **A manifest holds no subject.** A catalogue test: no manifest contains a string matching the subject grammar. The rule is worthless if it is followed by convention. +- **A derived step is the container it precedes.** A composition test: the step's image, environment, + volumes and network equal that container's, so the two cannot drift — which is the failure the + hand-written kind has, three times over in the catalogue today. +- **A convergence that changed nothing says nothing.** Two identical reports, one emitted fact: what is + guarded against is a fact per minute per machine, which is a stream nobody reads. - **Permissions are exactly the three namespaces.** A composition test per module: the derived permission set equals what its declaration implies, and a hand-written addition to it fails. - **A sender cannot read the queue it writes to.** A bed: a module declaring `uses` is refused diff --git a/04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md b/04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md index a71d68f..98bc035 100644 --- a/04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md +++ b/04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md @@ -3,7 +3,7 @@ status: resolved opened: 2026-09-28 located-in: [mesh-controller module.json] fixed-by: mesh-controller — the control plane's module declares a run-once `migrate` step before its server, which is the shape ADR 0052 prescribes for exactly this. A step's record of having run is the digest of its declaration and the image is part of that digest, so a new build of the control plane re-runs it; and because a run-once step gates what the declaration places after it, a migration that fails stops the new server from starting at all rather than letting it run against a schema it does not have. -amended-design: +amended-design: 03-DESIGN/01-to-be/32-what-a-module-declares.md --- # 133 — The control plane's schema is migrated at birth and never again @@ -54,6 +54,14 @@ such step; it went straight from a state directory to the server. server starts, which means the old binary briefly runs against the new schema. A migration that removes or renames something would break the running control plane in the window between the two. +**A hand-written step is one the next module forgets**, which is why this fix is not where the matter +ends: [ADR 0133](../../02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) +makes the step derived — a container declares what must run before it, and the mesh composes the gated +resource — so the control plane stops being the only module that had to remember. The same record +settles the level question HAL answered with stages, and +[ADR 0134](../../02-DECISIONS/0134-the-mesh-says-what-it-applied.md) answers the second open question +below: what a machine applied, and what it refused, become facts on the bus rather than a line in a log. + **The mesh now has two shapes for one problem.** The catalogue module migrates its own schema in its own code when it starts; the control plane migrates in a step the host gates on. Both work and the reasons differ — a module that owns its store entirely can do it at start, while a step is visible in