Merge pull request 'Issue 207: a re-made worker replayed every ask the stream kept, and the mesh re-registered its past' (#311) from issues/207-a-re-made-worker-replayed-history into main
This commit was merged in pull request #311.
This commit is contained in:
@@ -0,0 +1,49 @@
|
|||||||
|
---
|
||||||
|
status: open
|
||||||
|
opened: 2026-10-03
|
||||||
|
located-in:
|
||||||
|
- mesh-controller
|
||||||
|
fixed-by:
|
||||||
|
amended-design:
|
||||||
|
---
|
||||||
|
|
||||||
|
# 207 — A re-made worker replayed every ask the stream kept, and the mesh re-registered its past
|
||||||
|
|
||||||
|
## What was observed
|
||||||
|
|
||||||
|
2026-10-03, recovering from [issue 206](../206-a-seats-worker-changing-type-strands-the-holder-and-the-build-that-would-fix-it/00-report.md).
|
||||||
|
The build seat's worker, a push consumer the new controller could not change to pull, was deleted by hand
|
||||||
|
and the controller restarted. On start it recreated the worker in the shape it derives — pull — and with
|
||||||
|
the delivery policy a new consumer gets when nothing says otherwise: *every message the stream holds*.
|
||||||
|
The seat's stream keeps its history. The build machine then took, in order, every build ask since
|
||||||
|
1 October:
|
||||||
|
|
||||||
|
```
|
||||||
|
a build request arrived for …/mesh-catalog.git (build-1790856308080864567)
|
||||||
|
[built] baserow from 6afc1160
|
||||||
|
```
|
||||||
|
|
||||||
|
Each outcome was heard and taken in as any build's is. In the minute before the build machine was taken
|
||||||
|
off the control node to stop it, nine modules were re-registered from the 1 October commit — baserow,
|
||||||
|
cloudflare-dns, gitlab, grafana, icecast, influxdb, jira, keycloak, letta — the catalogue's recorded
|
||||||
|
head moved back to that commit, so status listed almost every module as "behind", and the upgrade
|
||||||
|
policy re-sent the machine running five of them, which replaced their tools containers with the old
|
||||||
|
images. Recovery: the nine rebuilt from main by hand, the worker re-made by hand as pull delivering only
|
||||||
|
new asks, the build machine assigned again.
|
||||||
|
|
||||||
|
## Why it matters beyond this instance
|
||||||
|
|
||||||
|
A work queue that keeps its history is a replay waiting for a consumer that starts from the beginning,
|
||||||
|
and a new consumer starts there unless told not to. Nothing in the controller's consumer derivation says
|
||||||
|
where a seat's worker starts, so any re-creation — by hand, or by the reconciliation issue 206's fix
|
||||||
|
adds — can replay the mesh's whole build history into the catalogue and onto machines. And the taking-in
|
||||||
|
of an outcome trusts the outcome's commit absolutely: an outcome for a commit older than what the
|
||||||
|
catalogue holds is registered as if it were news, and rolls out.
|
||||||
|
|
||||||
|
## Questions HQ must answer
|
||||||
|
|
||||||
|
- Does a seat's work queue keep acknowledged asks at all? If it is a work queue, acknowledged work
|
||||||
|
should leave it ([design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 calls it one); if it
|
||||||
|
keeps history for the record, its worker must be derived to start at new messages, always.
|
||||||
|
- Should a build outcome for a commit the catalogue has already moved past be recorded and **not**
|
||||||
|
registered — a build of the past is a fact, not a change — and never sent to machines?
|
||||||
Reference in New Issue
Block a user