Merge pull request 'Issue 206: a seat's worker changing type strands its holder, and the build that would fix it' (#309) from issues/206-a-worker-changing-type-under-its-holder into main

2026-10-03 01:46:12 +00:00
parent d8a58dadcf 561f26fb34
commit 733cff4c3c
1 changed files with 57 additions and 0 deletions
@@ -0,0 +1,57 @@
+---
+status: open
+opened: 2026-10-03
+located-in:
+  - mesh-controller
+fixed-by:
+amended-design:
+---
+
+# 206 — A seat's worker changing type strands its holder, and the build that would fix it
+
+## What was observed
+
+2026-10-03, the switch to shared build work ([ADR 0190](../../02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md)).
+The controller change made a seat's worker a pull consumer and the build machine pull from it. The
+merge's plan put the **build machine** in tier 0 and the controller in tier 1, so the new build machine
+rolled first, onto a bus where the running controller had defined the worker as push:
+
+```
+mesh-builder: this machine cannot take work from mesh-build-machine: nats: cannot pull subscribe to
+push based consumer. The mesh creates that queue and this machine's worker on it, and a build
+machine may not create one
+```
+
+It restarted every few seconds. The old controller kept asking for tier 1 — the new controller image —
+on that worker, and nothing took it. The only thing that would redefine the worker is the controller
+that could not be built; there is no verb to run a module's previous build. The way out was a
+person running the previous build machine image by hand on the control node until the new controller
+had rolled, then removing it.
+
+Two smaller faults surfaced on the way and were each a step of the same handover: the build machine's
+credential, issued on 2026-09-28, carried no `claims`, so the new binary's "serve the seat your
+credential claims" fell back to the new seat it had no grant for (re-issuing the credential fixed it);
+and the build machine's container restarts on its environment file, not on its credential, so the
+re-issued credential reached it only because it was already restarting.
+
+## Why it matters beyond this instance
+
+A consumer's type is part of the contract between the controller that defines a worker and the holder
+that binds it, and the two are built and rolled by different plans in an order the dependency graph
+decides, not the contract. Any future change to a worker's shape — ack wait, filters, type — can strand
+every holder the same way, and when the holder is the build machine, the mesh cannot build its way out.
+[ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) orders tiers by
+artifacts; it says nothing about what must be *running* before what, and
+[issue 201](../201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)
+is the same gap seen from the controller's side.
+
+## Questions HQ must answer
+
+- Does the controller own the worker's shape fully — redefining an existing consumer to the shape it
+  derives on every start — or does a holder bind whatever shape it finds? Either answer must hold
+  across a handover where the two are at different versions.
+- Should a plan that changes the build machine always roll the controller first, or should the mesh
+  keep a way to run a module's previous build without a person on the machine?
+- A credential issued before claims existed names none: should the mesh re-issue credentials whose
+  shape is older than what the binary reads, or should every holder treat an unclaimed credential as
+  the seat its manifest claims?