hq/02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md

---
topic: what runs on it
status: superseded
date: 2026-09-28
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md
superseded-by: 02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md
---

# 133. A module owns its migrations, and the mesh owns when they run

## Context

On 2026-09-28 the mesh replaced its own control plane, through its own upgrade path, with a build
carrying a migration. Nothing applied it. For the next three quarters of an hour every build the mesh
made was refused by the store with one line — *column "built_contexts" does not exist* — which reached
only whoever happened to be waiting on that build's reply. The images were built and published, so the
registry filled with artifacts the mesh has no record of, and the overview went on reporting that every
module was current ([issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)).

The schema had been created once, at genesis, by an action in the foundation bundle. Nothing ran it
again, through many updates of the control plane since.

**The mechanism to do this right already existed and one module used it wrong.**
[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) makes a run-once container a step the host
runs to completion before whatever the declaration places after it, and names migrating a schema as the
case it exists for. Three facts about how it is used today:

- The control plane's manifest had no step at all. The immediate fix was to write one by hand, and that
  hand-written step repeats three environment variables and three volume mounts from the server
  resource it precedes — six chances to drift from the thing it prepares.
- Two other modules hand-write the same shape for the same reason: gitea's admin bootstrap and
  mosquitto's dynsec seed, each repeating its sibling's image, environment and mounts. One of them
  ends in `|| true`, which is a lock implemented as a shrug.
- The catalogue module takes the other road: it migrates its own schema in its own code when it starts.
  That failure mode is a crash loop rather than a stop — the catalogue restarted 338 times this
  morning on an unrelated start-time failure, and nothing anywhere said the mesh's graph had a gap.

**What the mesh already has, and what HAL needed stages for.** Ordering a provider before its consumer
is `providersFirst`, which topologically orders a node's modules. Ordering within a module is
declaration order, and a run-once container gates everything after it. Remembering that a step has
already run is the digest of its declaration, recorded only after it exits 0
([ADR 0018](0018-a-picture-is-read-from-what-runs.md)) — and the image is part of that digest, so a new
build re-runs it. Three of the four things a stage system provides are therefore already here. The
fourth — that a module has a schema at all — is the only thing missing.

**Nothing in the catalogue ships a migrations directory.** Of 72 modules, none has one; the modules that
migrate do it in their own code. So this is not a decision about where SQL files live. It is a decision
about who runs them and when.

**Two facts bound what is safely expressible.** A node converges toward its own declaration without
waiting on any other node. And of the five modules that run on more than one machine today — dnsmasq,
fail2ban, networking, networkmanager, sshd — not one wants a store; every module with a database is on
exactly one machine.

## Decision

**A container may declare steps to run before it.** The same container, run to completion, with
different arguments, in order, before it starts. The mesh derives the run-once resources from that
declaration, so the image, the environment, the volumes, the network and the credentials come from the
one place they are already described and cannot drift from it.

**A module's migrations are the first user of this, and the module owns them entirely.** The SQL, the
order, the idempotence, the lock, and which dialect it speaks. The mesh never learns that postgres and
mssql differ, because it runs the module's own image with the module's own arguments against the
module's own binding and requires exit 0. A module needing both stores runs one step that does both.

**The mesh owns the moment, and the gate is the guarantee.** Whether a version may serve when its
schema is not there yet is a deployment question, and the mesh is the only thing that can answer it,
because the mesh is what starts the container. A step that fails stops the container it precedes, so
a failed migration is a version that does not serve rather than a version serving against a store it
does not match.

**Per node, and there is no level.** The step runs wherever the module runs. A step that ran "once,
somewhere" would leave every other machine with no gate at all, and additive migrations protect old
code against a new schema, never new code against an old one. The cost is an obligation a migration
runner already carries: a version table and a lock.

**"Once, mesh-wide" is what holding a seat means.** A step that is not idempotent — seeding an
account, sending a notice, taking a backup — belongs to a module that holds a seat, where the mesh
already guarantees one holder, on record, handed over deliberately. That is the answer to the level
question rather than a field that has to invent an election and keep it somewhere.

**Migrations are forward-only and additive.** The step runs before the *new* container starts, so the
old one is still running against the new schema for the length of the apply.

**Declared, never inferred.** The control plane cannot see inside an image, so a module that ships
migrations and declares no step is not refusable at registration; it breaks on its first upgrade. This
record says so rather than implying a check that cannot exist.

## Options considered

1. **Each module migrates itself when it starts** — what the catalogue does today. Rejected: it turns a
   schema failure into a crash loop instead of a stop, it is invisible in the declaration so nothing can
   say the module even has a schema, and two machines running the module both migrate at start with
   nothing sequencing them.
2. **The mesh applies migrations itself**, with a driver and a version table per store — HAL's shape.
   Rejected: the mesh would have to know one store type from another, hold another module's store
   credentials, and reach a machine to use them, which [ADR 0005](0005-the-node-host.md) forbids. It is
   also the reason that shape needs levels: something central has to decide where the once happens.
3. **A hook lifecycle** — pre-build, post-build, pre-deploy, post-deploy. Rejected: there is no deploy
   event here to hook. A declaration is a desired state applied in order and reconciled forever, so
   "pre-deploy" is exactly "a step before this container", pre- and post-build are what a Dockerfile and
   the artifact list already are, and "post-deploy" has no moment to name.
4. **A hook level** — once per module, or once per module-node assignment. Rejected as a field, kept as
   a property: see the decision. A once-per-module step needs cross-node ordering underneath it to be
   safe, and a node converging without waiting on its neighbours is worth losing on purpose rather than
   by accident.
5. **Every module hand-writes its own run-once step** — the immediate fix for the control plane.
   Rejected as the general answer: it duplicates the resource it precedes, in three places already, and
   a hand-written step is one the next module forgets. Forgetting it is the fault this record exists
   for.
6. **Record a schema level per module in the store.** Rejected: gating makes the invariant true by
   construction, so a level is a second account of the same fact and the first one to go stale.

## Consequences

**Three hand-written steps collapse into one line each**, and the control plane's own migrate step stops
repeating its server's environment and mounts.

**The catalogue's self-migration becomes the exception to remove.** One shape, and the mesh's own
control plane is not an exception to it either.

**A module on two machines with one shared store must lock.** Today none is, so this is an obligation
stated before it is needed rather than discovered by two concurrent migrations.

**There is still no readiness-gated step.** Only an action carries `verify`; a container has no health
notion, so "run this once the service answers" remains unexpressible and seeding through a running
service's API has no home. That is its own decision about a container's readiness, and this record does
not make it.

**Genesis keeps its own action.** At birth there is no control plane to derive anything from, which is
what [ADR 0067](0067-genesis-is-a-pivot.md) already says about that moment.

## How this is checked

- **The composition carries the step.** A test on a node's composed declaration: every container that
  declares steps before it is preceded by them, and the derived step's image, environment, volumes and
  network equal the container's — so the two cannot drift, which is the failure the hand-written kind
  has.
- **A failed step stops what follows.** The host already refuses to go on past a run-once step that did
  not exit 0; the test for that is extended to a derived one, so the gate is checked rather than
  assumed.
- **The mesh's own schema is covered by the same mechanism as everything else.** The control plane
  declares its step in its own manifest, so the case that failed on 2026-09-28 is the case the test
  covers.
- **A module claiming a seat for a once-only step is checked where seats are checked** — the conditions
  of holding, not a new mechanism.

## References

- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the step this extends
- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — a digest is the record that something happened
- [ADR 0005](0005-the-node-host.md) — the control plane decides and never touches a machine
- [ADR 0067](0067-genesis-is-a-pivot.md) — why genesis does it differently, once
- [issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md) — the failure that produced this record
- [`03-DESIGN/01-to-be/32-what-a-module-declares.md`](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §6 — the lifecycle this sits in
- Measured 2026-09-28: three hand-written run-once steps repeating their sibling's resource; 0 of 72 modules with a migrations directory; 5 modules on more than one machine, none of them wanting a store