Merge pull request 'ADR 0219: the build queue is controlled through the controller and the build seat' (#105) from decision/0219-the-build-queue-is-controlled into main

2026-10-05 18:22:01 +00:00
parent ab1fbe3202 77429488b0
commit 035d4a27e3
3 changed files with 145 additions and 0 deletions
@@ -0,0 +1,120 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-05
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md
+---
+
+# 219. The build queue is controlled through the controller and the build seat
+
+## Context
+
+The operator asked for tools to control the mesh's builds: see what is queued and running, cancel,
+clear, stop a build immediately, pause and continue, restart and replay. On the day of the request none
+existed. The controller could ask for a build and list finished ones. Nothing could see an ask waiting on
+the build seat's work queue, or one being built. Nothing could take an ask back, and a running build
+could only be stopped by restarting the machine's build agent, which hands the ask to another holder.
+
+How builds run today, read from the code and the bus
+([ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md),
+[ADR 0157](0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)):
+
+- an ask is a message on the seat's work-queue stream;
+- every holder pulls one at a time from one shared worker, and acknowledges after it has announced the
+  outcome;
+- a build says it started, logs its steps, and announces what it built, all on the bus;
+- an ask delivered five times without an answer stays in the stream for a week, with nothing saying so.
+
+Three facts constrain any control:
+
+- **Only the controller may act on the bus's streams and consumers** (design 25 §3, enforced in the bus's
+  user list). Holders may only take from their worker and acknowledge.
+- **A plan waits for a build's outcome and has no timeout.** An ask that disappears without one leaves
+  its plan waiting, said only as late after half an hour.
+- **The bus server in use cannot pause a consumer**; that arrived in a later version.
+
+## Considered Options
+
+1. **Pause and cancel by remaking the worker consumer.** Rejected: a remade worker delivers from now on,
+   so every queued ask would be skipped silently. Issues 206 and 207 are this mistake both ways round.
+2. **Upgrade the bus first and use its consumer pause.** Not now: it gives pause and continue, and
+   nothing else on the list, and upgrading the bus is a change of its own.
+3. **Queue actions are the controller's verbs, process actions are the build seat's verbs, and every
+   action that drops work leaves a failed outcome.** Chosen.
+
+## Decision
+
+**1. The queue is the controller's.** Its verbs act on the work-queue stream, the one thing only it may
+touch:
+
+| verb | what it does |
+|---|---|
+| `queue` | lists every ask: waiting, in flight (with the machine building it, from its started event), and dead (delivered as often as allowed, still in the stream) |
+| `cancel <id>` | takes back a waiting or dead ask; an ask in flight is refused, and `kill` is named instead |
+| `clear` | cancels every waiting ask, and with `--dead` every dead one |
+| `rebuild <module or build>` | asks again for a module's current source, or a past build's source and ref, under a new id |
+| `replay <build>` | asks again for a past build at its commit, as a **dry run** unless told to register |
+| `kill <id>`, `pause [node]`, `resume [node]` | pass the request on to the build seat on the right machine, or on every machine |
+
+**2. The process is the build seat's.** Each holder serves verbs on its own machine:
+
+- `current`: the build running here, its step and how long, and whether this holder is paused;
+- `kill`: stops a running build at once. The build's whole process group is ended, along with every
+  container it started. Its outcome is announced as failed, and the ask is acknowledged, so it is not
+  delivered again;
+- `pause` and `resume`: a paused holder takes nothing new, and a running build finishes. The flag
+  survives the holder's restart.
+
+A holder restarted mid-build keeps today's behaviour: the ask is not acknowledged, and another holder
+takes it.
+
+**3. Nothing dropped is silent.** Cancel, clear and kill each leave a failed outcome for the ask's id,
+taken in like any other. A plan waiting on that build fails, and says why, instead of waiting. A plan keeps
+the id it asked for, so it can match its outcome exactly. A holder checks a cancelled ask before it builds
+it, so an ask taken in the instant it was cancelled is not built.
+
+**4. A plan says what its builds are waiting on, and a failed plan can go on.**
+
+- A plan whose build waits on a paused seat says the seat is paused, and on which machines, and is not
+  counted late while it waits.
+- `plans retry <id>` asks again for the modules a failed plan could not build, under new ids. The plan
+  resumes at that tier and goes on through its later ones. A plan another has superseded, or one
+  already done, is refused.
+- `rebuild` of a module that an open or failed plan has not yet built joins that plan, so the plan and
+  the build are one thing.
+
+**5. Replay does not move the mesh backwards unasked.** A replayed build is a dry run: built, its log
+kept, nothing registered. With `--register` it is registered. If a newer build of the module is already
+registered, that is refused unless `--older` is said as well: registering an older commit makes it the
+current one, and the rollout policy sends it to the machines ([issue 207](../04-ISSUES/207-a-re-made-worker-replayed-every-ask-the-stream-kept/00-report.md)).
+
+## Consequences
+
+- Every build in the mesh can be seen, taken back, stopped or asked again from the console. None of it
+  needs a shell on a machine.
+- A cancelled or killed build shows in the build records as failed, with who stopped it. Its plan fails
+  saying the same.
+- **What got harder:** a holder now serves verbs as well as taking work, and keeps one small flag on
+  disk. Pause is per holder, so "pause the mesh" is the controller asking every holder in turn. A holder
+  away at the time misses it, and the answer names that holder.
+
+## How it is checked
+
+| Rule | Checked by |
+|---|---|
+| the queue is read and classified right | the controller's test: waiting, in flight and dead asks told apart from the stream and the worker; a live test on a throwaway bus |
+| nothing dropped is silent | the controller's test: cancel and clear delete the ask and record a failed outcome; a plan asked for that id fails |
+| a cancelled ask is not built | the holder's test: an ask taken after its cancel is answered failed without building |
+| kill stops everything it started | the holder's test: the build's process group and its labelled containers are ended; the outcome is failed and the ask acknowledged |
+| pause survives a restart | the holder's test: the flag is read back at start, and nothing is taken while it is set |
+| plans follow the queue | the controller's test: a plan waiting on a paused seat says so and is not late; `plans retry` resumes a failed plan and its later tiers are asked; `rebuild` joins the plan that holds the module |
+| replay is safe | the controller's test: a dry run by default, and `--register` over a newer build refused without `--older` |
+
+## References
+
+- [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md) — holders share the seat's work, one at a time
+- [ADR 0157](0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md) — a build's own events are its record
+- [issue 207](../04-ISSUES/207-a-re-made-worker-replayed-every-ask-the-stream-kept/00-report.md) — a remade worker replayed every ask
+- [to-be 18](../03-DESIGN/01-to-be/18-building-a-module.md) — the design this amends
@@ -195,6 +195,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0210** — [A tool's configuration is its seat holder's, and every other module extends it through the seat](0210-a-tools-configuration-is-its-seat-holders-and-every-other-module-extends-it-through-the-seat.md)
 - **0212** — [A seat says what it receives, and the machine's hotkeys are a seat](0212-a-seat-says-what-it-receives-and-the-machines-hotkeys-are-a-seat.md)
 - **0218** — [A plan sends grants before code, rolls a module out one machine first, and a newer merge takes over an older plan](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)
+- **0219** — [The build queue is controlled through the controller and the build seat](0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md)

 ### Its tiers, from the bottom up

@@ -7,6 +7,7 @@ code:
  - mesh-catalog modules/build-agent
 updated: 2026-10-05
 decisions:
+  - 02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md
  - 02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md
  - 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
  - 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md
@@ -394,3 +395,26 @@ not of the recipe: one artifact declared per target, one build each.
 A component's version stops being stamped in at link time. It is unpacked into a directory named for
 its version, so it reads its version from its own path, and a build no longer has to know what it will
 be called.
+
+## The build queue is controlled
+
+*2026-10-05 — [ADR 0219](../../02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md).*
+
+What is asked of the build seat can be seen and controlled from the console. The controller holds the
+queue, and each machine's build agent holds its own process.
+
+- **The controller's verbs:**
+  - `queue` lists every ask, waiting, in flight or dead;
+  - `cancel` and `clear` take asks back;
+  - `rebuild` asks again under a new id;
+  - `replay` asks again for a past build at its commit, as a dry run unless told to register, and never
+    over a newer build without saying so;
+  - `kill`, `pause` and `resume` are passed on to the seat on the right machine.
+- **The build seat's verbs, on each machine:**
+  - `current` says what is building here and whether this holder is paused;
+  - `kill` ends the build's whole process group and every container it started;
+  - `pause` and `resume` set a flag the holder reads before it takes the next ask, which survives its
+    restart.
+
+Every action that drops work leaves a failed outcome, so a plan waiting on that build fails and says why
+rather than waiting. A plan keeps the id of every build it asked for. A plan waiting on a paused seat says so and is not counted late; a failed plan can be resumed with `plans retry`, and a `rebuild` of a module a plan holds joins that plan.