2026-10-03 09:06:17 +00:00
1 changed files with 49 additions and 0 deletions
@@ -0,0 +1,49 @@
+---
+status: open
+opened: 2026-10-03
+located-in:
+  - mesh-controller
+fixed-by:
+amended-design:
+---
+
+# 207 — A re-made worker replayed every ask the stream kept, and the mesh re-registered its past
+
+## What was observed
+
+2026-10-03, recovering from [issue 206](../206-a-seats-worker-changing-type-strands-the-holder-and-the-build-that-would-fix-it/00-report.md).
+The build seat's worker, a push consumer the new controller could not change to pull, was deleted by hand
+and the controller restarted. On start it recreated the worker in the shape it derives — pull — and with
+the delivery policy a new consumer gets when nothing says otherwise: *every message the stream holds*.
+The seat's stream keeps its history. The build machine then took, in order, every build ask since
+1 October:
+
+```
+a build request arrived for …/mesh-catalog.git (build-1790856308080864567)
+  [built] baserow from 6afc1160
+```
+
+Each outcome was heard and taken in as any build's is. In the minute before the build machine was taken
+off the control node to stop it, nine modules were re-registered from the 1 October commit — baserow,
+cloudflare-dns, gitlab, grafana, icecast, influxdb, jira, keycloak, letta — the catalogue's recorded
+head moved back to that commit, so status listed almost every module as "behind", and the upgrade
+policy re-sent the machine running five of them, which replaced their tools containers with the old
+images. Recovery: the nine rebuilt from main by hand, the worker re-made by hand as pull delivering only
+new asks, the build machine assigned again.
+
+## Why it matters beyond this instance
+
+A work queue that keeps its history is a replay waiting for a consumer that starts from the beginning,
+and a new consumer starts there unless told not to. Nothing in the controller's consumer derivation says
+where a seat's worker starts, so any re-creation — by hand, or by the reconciliation issue 206's fix
+adds — can replay the mesh's whole build history into the catalogue and onto machines. And the taking-in
+of an outcome trusts the outcome's commit absolutely: an outcome for a commit older than what the
+catalogue holds is registered as if it were news, and rolls out.
+
+## Questions HQ must answer
+
+- Does a seat's work queue keep acknowledged asks at all? If it is a work queue, acknowledged work
+  should leave it ([design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 calls it one); if it
+  keeps history for the record, its worker must be derived to start at new messages, always.
+- Should a build outcome for a commit the catalogue has already moved past be recorded and **not**
+  registered — a build of the past is a fact, not a change — and never sent to machines?