to-be 30: the mesh updates itself on a push

Records the manual update process (module moved -> build -> reconcile; and the breaking-change freeze/re-register recovery), and the two things that make self-update more than a webhook: the build-on-push trigger is currently HAL's (hal-gitea-tools on :9877), a retirement gap the mesh must replace with its own forge-webhook trigger wired to every repo including mesh-controller; and the builder validates manifests too, so a breaking change couples controller + builder + manifests + hosts, and renaming the builder's own seat deadlocks its rebuild. Names the transition discipline (accept old+new for one release) that self-update needs so a push does not auto-freeze.
2026-09-27 14:49:02 +02:00
parent bf7297b7ed
commit b8cdfce16d
1 changed files with 137 additions and 0 deletions
@@ -0,0 +1,137 @@
 ---
 layer: to-be
 status: proposed
 code: []
 updated: 2026-09-27
 decisions:
  - 02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md
  - 02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md
 ---
 # 30 — The mesh updates itself on a push
 **Today the mesh does not update itself; a person drives the pipeline by hand, and one class of
 change freezes it.** A code change lands in `mesh-controller` or `mesh-catalog`, and getting it onto
 the machines is a sequence somebody types. The predecessor's pipelines rebuilt and redeployed on a
 push without anyone watching; the successor should too. This records the process as it is done by
 hand now — so it can be read, and then coded — and the two things that make it more than "add a
 webhook".
 ## The process, as done by hand
 **An ordinary (non-breaking) change** — new module code, a bug fix, a manifest tweak that changes no
 seat or schema:
 1. `module moved <module> <commit>` — tell the mesh its source advanced (the controller repo has no
   trigger, so this is manual; the catalogue's webhook does it automatically — see below).
 2. `build --behind` (or `build <repo> [--ref] [--path <subdir>]`) — the build machine rebuilds and
   records the new image.
 3. The mesh **reconciles on its own**: the module's declaration now names the new image, the next
   push/heartbeat sends it, and the host swaps the container. For the control plane this is a
   self-upgrade — the running controller composes its own new image and the host replaces it. No
   restart is typed.
 **A breaking change** — a manifest schema the controller parses differently (a fact's shape, a
 seat's name), where the new control plane cannot read the manifests the old one stored:
 4. Land the code (controller + catalogue together — they are one change).
 5. Rebuild + deploy the new controller (steps 1–3). **The moment it is live it refuses the
   still-old-shape stored manifests, and composition freezes for every node that runs an affected
   module.** Running services are untouched; only new declarations stop.
 6. **Re-register each affected manifest under the new shape**, which the *new* controller accepts —
   `module add <file> -source <repo> -ref <ref> -commit <commit>`. This writes the manifest to the
   store without a build, so it is the fast way to lift the freeze. (The controller container is
   distroless: `docker cp` the file to the container root `/x.json`; `/tmp` does not exist; the
   root filesystem is writable. The file is lost when the container is recreated on the next image
   swap, so copy it *after* the swap.)
 7. `push --behind`, then verify `status` is clean and `seats` (or the relevant surface) shows the
   new shape held by the right holders.
 The freeze in a breaking change has been paid three times in one session (a fact-shape change, the
 `/etc/hosts` region, a seat rename); each time it lasted seconds and no service dropped. It is
 recoverable, but it is not something a push should trigger unwatched — which is the crux of what
 automating this must solve.
 ## Why it is more than "add a webhook"
 ### 1. The trigger today is HAL's, not the mesh's
 Build-on-push works for the catalogue because its repository has a Gitea webhook pointing at
 `http://host.docker.internal:9877/webhook/gitea` — and **that receiver is `hal-gitea-tools.service`**
 (`~/.hal/modules/hal/gitea/tools/server.js`), a *predecessor* component. The nox builder consumes
 build work; it does not receive Git events. So the mesh's own build pipeline currently rides on a
 HAL service, and:
 - the `mesh-controller` repository was never wired to it, which is why the control plane is the one
  thing that does **not** self-update — every controller deploy this session was `module moved` +
  `build` by hand;
 - when HAL is retired, build-on-push stops for the whole mesh.
 **The mesh needs its own forge-webhook→build trigger**, a nox component (a module, and likely a
 seat — `mesh-forge-trigger` or folded into the git seat's holder) that receives Git events and turns
 them into build work over the broker, for **every** repository including `mesh-controller`. Replacing
 `hal-gitea-tools` is the concrete first build. Its logic already exists to copy: match the pushed
 repository (and changed paths, for a monorepo like the catalogue) against the build-context
 repository of every registered module, and rebuild the matches.
 ### 2. The builder validates too — and a breaking change deadlocks it
 The build machine embeds the same catalogue package the controller does, so **it validates a
 manifest against its own compiled-in seat/schema set**. A breaking change therefore couples *four*
 things, not two: the controller, the **builder**, every affected manifest, and every node's host.
 This session's seat rename rebuilt the controller but not the builder, and the stale builder then
 refused every manifest claiming a renamed seat.
 Worse, one rename **deadlocked** the builder: the build machine's own seat was renamed
 (`the-build-machine` → `mesh-build-machine`). To refresh the builder you must build it; to build it
 the *running* (old) builder must accept the new builder's manifest — which claims the new name it
 does not know. The old builder cannot build the new builder. Escapes:
 - **Never rename a seat whose holder validates manifests** in an ordinary pass — the build machine's
  seat belongs with the deferred delivering seats (ADR 0121). Reverting `mesh-build-machine` to
  `the-build-machine` (deferred) lets the old builder build the new builder, which then knows the
  new names.
 - Or bootstrap a new builder image **out of band** (build locally, publish to the registry, register
  the module at that digest), the way genesis loads the first builder — bypassing the old builder's
  validation once.
 Either way, self-update for breaking changes needs a **transition discipline** so a push does not
 auto-freeze: the new control plane (and builder) should accept the *old and new* shape together for
 one release — deprecated aliases in the seat set, a schema that reads both — then a later release
 drops the old. With that, a breaking change rolls out on a push like any other: everything reads
 both, the manifests migrate, the compatibility is removed. Without it, self-update would simply
 automate the freeze.
 ## What to build
 - **A nox forge-webhook trigger** (replaces `hal-gitea-tools`): receives Git events for every mesh
  repository, dispatches build work to the builder over the broker, and records `module moved`
  automatically. Wire `mesh-controller` to it so the control plane self-updates like everything else.
 - **A transition discipline for breaking changes**: the control plane and builder accept old+new for
  one release; the tooling that lands a schema/seat change emits the compatibility shim and the
  follow-up that removes it. This is what makes step 4–7 above safe to trigger unwatched.
 - **Config/package modules need no builder** — `module add` registers their manifest directly
  (this is how the uplink managers and the re-registrations above were done). Only image-bearing
  modules need the build machine, which narrows what the deadlock above can block.
 ## Why now, and why not yet
 **Why it matters:** self-update is the difference between a mesh a person maintains by typing
 pipeline steps and one that maintains itself, and it is a stated goal (parity with the predecessor's
 pipelines). The HAL trigger dependency also makes it a retirement blocker: build-on-push dies with
 HAL.
 **Why not reflexively:** the trigger is a new component with the broker and forge in its blast
 radius, and the transition discipline changes how every breaking change is written. Both should be
 designed, not bolted on beside a freeze. The manual process above is the interim, and it works.
 ## References
 - [ADR 0121](../../02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md)
  — the seat rename whose migration and builder deadlock this record is drawn from
 - [ADR 0120](../../02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md) — the
  fact-shape change that first showed the breaking-change freeze
 - `hal-gitea-tools.service` (`~/.hal/modules/hal/gitea/tools/server.js`) — the predecessor webhook
  receiver on `:9877` the mesh currently rides on
 - mesh-controller `cmd/mesh-builder` (the build machine), `internal/catalogue` (the seat/schema
  validation the builder shares with the controller)