hq/02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md

topic, status, date, deciders, reconstructed, extends

topic	status	date	deciders	reconstructed	extends
what runs on it	accepted	2026-09-05	jochen	false	0005-the-node-host.md

52. An init step is a container run once to completion, gating what follows

Context

A module can declare things that exist; it cannot declare a step that runs. The host owns a finite vocabulary of shapes — file, directory, service, package, container, action — and every one but action describes state: a thing that should be present, with content or a mode or an image, which the host reconciles toward (ADR 0005). That is right for what it covers. But a real class of modules needs, once, to run their own code at a point in their own lifecycle — and the vocabulary has no word for it (04-ISSUES/037).

mosquitto is the sharp case, and it fails silently without this. Its Dynamic Security plugin loads at broker start and refuses to come up unless dynamic-security.json already holds an admin client. Seeding that file is a step that must happen after the data directory exists and before the broker container starts. The manifest can declare the directory, the config file and the broker container; it cannot declare "seed this, once, before that container starts." Written as it is today, the broker starts against an unseeded store and the plugin aborts — and the next reconcile does not fix it, because nothing in the declaration ever seeds the file.

It is not one module's defect. The database providers need the same to run a first-boot migration, an extension enable, or a health gate before they are announced ready; today that works only where the image happens to seed itself from an environment variable, and anything the mesh must run once against the server has no home. This is the timing face of the same gap 04-ISSUES/035 records from the content side: the manifest needed the file to exist before first start, and had only the file has this content, forever.

The obvious answer is the one that already went wrong. An earlier mesh had exactly this as a feature — event-driven hooks that ran custom code at phases of build, publish and deploy. It was powerful and it was complex to set up and flaky, and that fragility, not the need, is the content of the issue. Whatever this becomes must not rebuild that engine.

The ground has shifted since that engine, in a way that makes a much smaller answer possible. A module with tools or events now runs a process of its own — a container carrying the module's compiled code, holding the single broker account the mesh scoped to it, isolated from every other module (ADR 0047). The code that must run at first boot is already in that image, under that account. So the mesh does not need a way to run a module's code — it has one. It needs a way to say run this container to completion, and start the one that depends on it only after it has.

Considered Options

1. A host run shape — a command the host executes on the machine. The direct reading of "run code at a lifecycle phase." Rejected. The host's vocabulary is finite and every added shape is a security decision, because it widens what a compromised control plane can express (ADR 0005). A general "run this command as the host" is the largest such widening there is: the blast radius is the whole machine, as root. The mesh already drew this exact line for action — it runs a command, and so it is permitted from the bundle and refused from the link, because the bundle arrives with the binary and the link is a separate party with an unbounded reach. A new host-command shape usable by an ordinary module would be an action from the link by another name, which is precisely what is refused.

2. Per-phase lifecycle hooks on a module — pre-start, post-start, pre-remove, and their build/publish cousins, each naming code the mesh runs at that phase. The general answer, and the old feature. Rejected for now. It is the flaky engine the issue warns against, and most of its phases have no present need. Deciding the full set of phases, where each one's code runs, and how each is made idempotent is a large design taken to buy capability nothing yet asks for. The three blocked modules all need one phase — before a container starts — and a mechanism narrow enough to be obviously correct beats a general one that is not.

3. A distinct one-shot resource type — a new shape, sibling to container, that names an image and runs it once. Rejected. It grows the host vocabulary by a whole shape (a Type, a struct, an applier, a place in every host's shape list) to express something a container almost already is. A one-shot is a container — a pinned image, an account, volumes, an environment — that happens to exit. Spending a new shape on the difference is the cost of option 1 in smaller type, for a capability the existing shape can carry with one modifier.

4. A modifier on the existing container shape: this container runs once, to completion, and the host gates the apply on it. Adopted. It reuses the shape the host already has, adds no new host action, and leans on two guarantees the host already gives — apply in declared order, never sorted, and a failed step fails the apply — to turn "before that container starts" into an emergent property of ordering rather than a dependency graph the host must resolve.

Decision

A run-once step is an ordinary container, marked to run to completion. The manifest sets run-once: true on a container resource. Everything else about it is a container as before — a digest-pinned image (ADR 0006), volumes, an environment, and for a module's own code the same scoped account its runtime already holds (ADR 0047). The mesh adds no way to run code; it marks a container the host must run to completion and require to exit 0, rather than start and leave running.

The gate is declaration order, not a named dependency. The host applies a declaration in the order it is given, does not sort, and does not resolve dependencies — ordering is a decision, and it is the control plane's (ADR 0005, 04-ISSUES/013). A run-once step is placed before the container that depends on it, and a failed run-once step halts the apply, exactly as a failed action does — so everything the declaration places after it, the broker included, is never reached until the step has completed. "Before the broker starts" is therefore expressed by list position plus completion, and the host cross-references nothing.

Completion is recorded, and a re-apply does not re-run it. The host records what it applied only after the fact, as the digest of the declaration that produced it (ADR 0018) — a run-once step no differently. Because the step leaves nothing running to inspect, that persisted digest, not a live container, is the marker that it happened. On a later apply the host finds the digest already recorded for this exact declaration and does nothing; it re-runs only when the declaration's digest has changed, and a step that exited non-zero recorded nothing and so is retried next apply. This is the reconcilable, idempotent discipline the state shapes get for free, made explicit for a step.

This is a decision and not a patch because it settles what a module may say about running its own code, which the whole catalogue of providers — a seed before start, a first-boot migration, a health gate — now and later depends on, and because it draws the line the issue asked for: the narrowest sound mechanism that unblocks the three modules without rebuilding the hook engine whose fragility is the warning.

The security bound, stated plainly

The host gains no new action and no new shape. run-once is a boolean modifier on the container shape that already exists. A run-once container is strictly less powerful than an action: it cannot run an arbitrary host command, only a digest-pinned image under an account the mesh scoped — which is exactly the capability container already grants from the link. A control plane that is compromised can express nothing through run-once it could not already express by declaring an ordinary container. The dangerous expansion of option 1 — a command the host runs on the machine — is not made.

How each claim is checked

• A run-once step runs to completion and its exit 0 is required. A host unit test applies a run-once container whose image exits 0, asserts the host ran it to completion (not detached, not left running) and reported it done; a sibling test applies one that exits non-zero and asserts the apply fails, naming the step.
• A failed run-once step gates what follows. A host unit test places a run-once container that exits non-zero before another container and asserts the second is never started and the apply is reported gated — the mirror of the existing test that a failed action stops what follows.
• It is not re-run once it has completed. A host unit test applies a run-once step, then applies the identical declaration again with the first run's record present, and asserts the second apply runs nothing and reports the step unchanged.
• A changed declaration re-runs it. A host unit test applies a run-once step, then applies one whose image or environment differs, and asserts it runs again — the digest moved, so the marker no longer matches.
• The vocabulary carries the field end to end. A control-plane unit test resolves a module whose manifest marks a container run-once and asserts the rendered host declaration carries the field, in author order before the container it gates; the manifest parser refuses a run-once that is not boolean.
• mosquitto seeds before the broker. mosquitto's manifest declares a run-once init container, before the server (broker) container, that writes the admin client into dynamic-security.json and exits — checked by the control-plane resolver, which now understands the field, and by the ordering of the rendered declaration. The end-to-end proof that it runs exactly once, at the right phase, and converges on re-apply is owed to a lab scenario ([04-ISSUES/037] open question), which this record does not close.

Consequences

• The three blocked modules gain a home for their step. mosquitto seeds its dynsec admin before the broker; a provider that must migrate or health-gate at first boot declares a run-once step in its own runtime image, under its own account, before the container that depends on it.
• The general lifecycle hook is deferred, deliberately. Only before a container starts is bought here. post-start, pre-remove and the build/publish phases remain unbuilt, and the day one is genuinely needed it is decided then, against a need, not speculatively — the same restraint that kept this from being the old engine.
• The seed-then-mutate file is safe if the step is written to be. A run-once seed writes dynamic-security.json only when it is absent and never reconciles it, so what the running plugin grows in that file afterward is never wiped (04-ISSUES/035). The host's marker guarantees the step is not re-run; the step's own code guarantees it does not clobber on the pass it does run.
• The host vocabulary did not grow, and that is the point. The cost of a run-once step is one boolean and a completion path in the container applier, not a new shape and not a new action. The mesh expresses ordering and completion; the module runs its own code, where it already runs it.
• A run-once step that never converges is a stuck apply, loudly. A step that exits non-zero every time halts the apply every time, and the container it gates never starts — which is the correct failure, reported, rather than a broker that half-starts against an unseeded store and a reconcile that reports success. It is failed forward, not failed silent.

References

• 04-ISSUES/037 — a module cannot run its own code at a lifecycle phase; the gap this resolves, and the warning about the old hook engine
• 04-ISSUES/035 — the content face of the same gap: a file needed before first start, that the running program then mutates
• 04-ISSUES/013 — the order of a declaration is the control plane's, and the host applies it as given; the gate rests on this
• ADR 0005 — the host's finite vocabulary, the ordered declaration it does not sort, and action as the shape a command already is and why it is refused from the link
• ADR 0047 — a module runs its code as its own process under its own account; a run-once step is that process, run to completion
• ADR 0018 — what was applied is recorded after it works; the completion marker is that record
• ADR 0006 — a container is pinned by digest; a run-once container no differently
• mesh-control feat/lifecycle-run-once, mesh-host feat/apply-run-once, mesh-catalog feat/mosquitto-bootstrap — the vocabulary, the apply support, and mosquitto's seeded broker