From ab6db9369b0d69b3f2da87971e87f7396d99a672 Mon Sep 17 00:00:00 2001 From: jochen Date: Mon, 28 Sep 2026 10:27:30 +0200 Subject: [PATCH] Issue 133: the control plane's schema is migrated at birth and never again MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mesh replaced its own control plane with a build carrying a migration, applied none of it, and then recorded no build for three quarters of an hour while saying everything was fine. ADR 0052 already prescribes the shape — a run-once step that gates the server — and the control plane was the one module that did not use it. --- .../00-report.md | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md diff --git a/04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md b/04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md new file mode 100644 index 0000000..a71d68f --- /dev/null +++ b/04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md @@ -0,0 +1,69 @@ +--- +status: resolved +opened: 2026-09-28 +located-in: [mesh-controller module.json] +fixed-by: mesh-controller — the control plane's module declares a run-once `migrate` step before its server, which is the shape ADR 0052 prescribes for exactly this. A step's record of having run is the digest of its declaration and the image is part of that digest, so a new build of the control plane re-runs it; and because a run-once step gates what the declaration places after it, a migration that fails stops the new server from starting at all rather than letting it run against a schema it does not have. +amended-design: +--- + +# 133 — The control plane's schema is migrated at birth and never again + +## What was observed + +On 2026-09-28 at 08:17 the control plane was replaced, by the mesh's own upgrade path, with a build +whose code writes a column that a migration **in that same build** creates. Nothing ran the migration. + +For the next three quarters of an hour the mesh built things and recorded none of them. Every build +answered: + +> ERROR: column "built_contexts" of relation "build" does not exist (SQLSTATE 42703) + +and that sentence went only to whoever happened to be waiting on a build's reply. The overview kept +saying the mesh was fine. The builds themselves worked — images were built and published — so the +registry filled up with artifacts the mesh has no record of, and the graph stopped learning without +anything saying so. + +The schema was created once, at genesis, by an action in the foundation bundle that runs the same +binary's `migrate`. Nothing runs it again. The mesh has updated its own control plane many times since +that bundle, and every one of those updates carried whatever migrations the new build brought and +applied none of them. This is the first time a build needed one. + +## Why it matters beyond this instance + +**The schema and the code that needs it ship as one artifact and are applied by two mechanisms, only +one of which is automatic.** A module's version is atomic everywhere else in the mesh — the manifest, +the image and what the machine runs move together. Its schema did not, so "the mesh updates itself on +a push" was true of the code and false of what the code needs. + +**The failure is quiet exactly where quiet is worst.** A build that cannot be recorded is a build that +happened and left no trace, which is the fault [issue 050](../050-the-catalogue-knows-nothing-built-before-it/00-report.md) +and [issue 131](../131-nothing-tells-the-mesh-a-source-moved/00-report.md) are both about. The mesh +has three mechanisms for noticing a module is behind its source and none for noticing that what it +recorded was refused. + +**The shape was already decided, and the control plane was the one module that did not use it.** +[ADR 0052](../../02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md) says a run-once container is a +step the host runs to completion before whatever the declaration places after it, and names migrating +a schema as the case it exists for. The genesis code's own comment says a manifest may name its image +in more than one resource — "a migrate step beside the server". The control plane's manifest had no +such step; it went straight from a state directory to the server. + +## What is still true + +**Additive migrations are load-bearing, not a style preference.** The step runs before the *new* +server starts, which means the old binary briefly runs against the new schema. A migration that +removes or renames something would break the running control plane in the window between the two. + +**The mesh now has two shapes for one problem.** The catalogue module migrates its own schema in its +own code when it starts; the control plane migrates in a step the host gates on. Both work and the +reasons differ — a module that owns its store entirely can do it at start, while a step is visible in +the declaration and refuses to let a broken upgrade serve. Which one the mesh should standardise on is +a decision, not a fix, and it is not made here. + +## Open questions + +- Should a module be refusable at registration when it ships migrations and declares no step and no + other way to apply them? The mesh can see both halves. +- Should a record the store refuses reach the overview? Today the only reader of that failure is + whoever asked for the thing that failed, and for an event arriving on the bus there is no such + person.