From 3944e4f914dce2c69fbfe92c49ce0c7c04c8335d Mon Sep 17 00:00:00 2001 From: jochen Date: Mon, 5 Oct 2026 18:54:44 +0200 Subject: [PATCH 1/2] ADR 0219: the build queue is controlled through the controller and the build seat The operator asked for tools to see, cancel, clear, stop, pause, continue, restart and replay builds; none existed. Queue actions are the controller's, process actions the build seat's per machine, and every action that drops work leaves a failed outcome so no plan waits on it. To-be 18 amended. --- ...rough-the-controller-and-the-build-seat.md | 109 ++++++++++++++++++ 02-DECISIONS/README.md | 1 + 03-DESIGN/01-to-be/18-building-a-module.md | 24 ++++ 3 files changed, 134 insertions(+) create mode 100644 02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md diff --git a/02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md b/02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md new file mode 100644 index 0000000..c378aaa --- /dev/null +++ b/02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md @@ -0,0 +1,109 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-05 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md +--- + +# 219. The build queue is controlled through the controller and the build seat + +## Context + +The operator asked for tools to control the mesh's builds: see what is queued and running, cancel, +clear, stop a build immediately, pause and continue, restart and replay. On the day of the request none +existed. The controller could ask for a build and list finished ones. Nothing could see an ask waiting on +the build seat's work queue, or one being built. Nothing could take an ask back, and a running build +could only be stopped by restarting the machine's build agent, which hands the ask to another holder. + +How builds run today, read from the code and the bus +([ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md), +[ADR 0157](0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)): + +- an ask is a message on the seat's work-queue stream; +- every holder pulls one at a time from one shared worker, and acknowledges after it has announced the + outcome; +- a build says it started, logs its steps, and announces what it built, all on the bus; +- an ask delivered five times without an answer stays in the stream for a week, with nothing saying so. + +Three facts constrain any control: + +- **Only the controller may act on the bus's streams and consumers** (design 25 §3, enforced in the bus's + user list). Holders may only take from their worker and acknowledge. +- **A plan waits for a build's outcome and has no timeout.** An ask that disappears without one leaves + its plan waiting, said only as late after half an hour. +- **The bus server in use cannot pause a consumer**; that arrived in a later version. + +## Considered Options + +1. **Pause and cancel by remaking the worker consumer.** Rejected: a remade worker delivers from now on, + so every queued ask would be skipped silently. Issues 206 and 207 are this mistake both ways round. +2. **Upgrade the bus first and use its consumer pause.** Not now: it gives pause and continue, and + nothing else on the list, and upgrading the bus is a change of its own. +3. **Queue actions are the controller's verbs, process actions are the build seat's verbs, and every + action that drops work leaves a failed outcome.** Chosen. + +## Decision + +**1. The queue is the controller's.** Its verbs act on the work-queue stream, the one thing only it may +touch: + +| verb | what it does | +|---|---| +| `queue` | lists every ask: waiting, in flight (with the machine building it, from its started event), and dead (delivered as often as allowed, still in the stream) | +| `cancel ` | takes back a waiting or dead ask; an ask in flight is refused, and `kill` is named instead | +| `clear` | cancels every waiting ask, and with `--dead` every dead one | +| `rebuild ` | asks again for a module's current source, or a past build's source and ref, under a new id | +| `replay ` | asks again for a past build at its commit, as a **dry run** unless told to register | +| `kill `, `pause [node]`, `resume [node]` | pass the request on to the build seat on the right machine, or on every machine | + +**2. The process is the build seat's.** Each holder serves verbs on its own machine: + +- `current`: the build running here, its step and how long, and whether this holder is paused; +- `kill`: stops a running build at once. The build's whole process group is ended, along with every + container it started. Its outcome is announced as failed, and the ask is acknowledged, so it is not + delivered again; +- `pause` and `resume`: a paused holder takes nothing new, and a running build finishes. The flag + survives the holder's restart. + +A holder restarted mid-build keeps today's behaviour: the ask is not acknowledged, and another holder +takes it. + +**3. Nothing dropped is silent.** Cancel, clear and kill each leave a failed outcome for the ask's id, +taken in like any other. A plan waiting on that build fails, and says why, instead of waiting. A plan keeps +the id it asked for, so it can match its outcome exactly. A holder checks a cancelled ask before it builds +it, so an ask taken in the instant it was cancelled is not built. + +**4. Replay does not move the mesh backwards unasked.** A replayed build is a dry run: built, its log +kept, nothing registered. With `--register` it is registered. If a newer build of the module is already +registered, that is refused unless `--older` is said as well: registering an older commit makes it the +current one, and the rollout policy sends it to the machines ([issue 207](../04-ISSUES/207-a-re-made-worker-replayed-every-ask-the-stream-kept/00-report.md)). + +## Consequences + +- Every build in the mesh can be seen, taken back, stopped or asked again from the console. None of it + needs a shell on a machine. +- A cancelled or killed build shows in the build records as failed, with who stopped it. Its plan fails + saying the same. +- **What got harder:** a holder now serves verbs as well as taking work, and keeps one small flag on + disk. Pause is per holder, so "pause the mesh" is the controller asking every holder in turn. A holder + away at the time misses it, and the answer names that holder. + +## How it is checked + +| Rule | Checked by | +|---|---| +| the queue is read and classified right | the controller's test: waiting, in flight and dead asks told apart from the stream and the worker; a live test on a throwaway bus | +| nothing dropped is silent | the controller's test: cancel and clear delete the ask and record a failed outcome; a plan asked for that id fails | +| a cancelled ask is not built | the holder's test: an ask taken after its cancel is answered failed without building | +| kill stops everything it started | the holder's test: the build's process group and its labelled containers are ended; the outcome is failed and the ask acknowledged | +| pause survives a restart | the holder's test: the flag is read back at start, and nothing is taken while it is set | +| replay is safe | the controller's test: a dry run by default, and `--register` over a newer build refused without `--older` | + +## References + +- [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md) — holders share the seat's work, one at a time +- [ADR 0157](0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md) — a build's own events are its record +- [issue 207](../04-ISSUES/207-a-re-made-worker-replayed-every-ask-the-stream-kept/00-report.md) — a remade worker replayed every ask +- [to-be 18](../03-DESIGN/01-to-be/18-building-a-module.md) — the design this amends diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index d4eceb1..6adb839 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -195,6 +195,7 @@ python3 00-META/checks/index.py fail if stale - **0210** — [A tool's configuration is its seat holder's, and every other module extends it through the seat](0210-a-tools-configuration-is-its-seat-holders-and-every-other-module-extends-it-through-the-seat.md) - **0212** — [A seat says what it receives, and the machine's hotkeys are a seat](0212-a-seat-says-what-it-receives-and-the-machines-hotkeys-are-a-seat.md) - **0218** — [A plan sends grants before code, rolls a module out one machine first, and a newer merge takes over an older plan](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md) +- **0219** — [The build queue is controlled through the controller and the build seat](0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/18-building-a-module.md b/03-DESIGN/01-to-be/18-building-a-module.md index 707f73d..9408236 100644 --- a/03-DESIGN/01-to-be/18-building-a-module.md +++ b/03-DESIGN/01-to-be/18-building-a-module.md @@ -7,6 +7,7 @@ code: - mesh-catalog modules/build-agent updated: 2026-10-05 decisions: + - 02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md - 02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md - 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md - 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md @@ -394,3 +395,26 @@ not of the recipe: one artifact declared per target, one build each. A component's version stops being stamped in at link time. It is unpacked into a directory named for its version, so it reads its version from its own path, and a build no longer has to know what it will be called. + +## The build queue is controlled + +*2026-10-05 — [ADR 0219](../../02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md).* + +What is asked of the build seat can be seen and controlled from the console. The controller holds the +queue, and each machine's build agent holds its own process. + +- **The controller's verbs:** + - `queue` lists every ask, waiting, in flight or dead; + - `cancel` and `clear` take asks back; + - `rebuild` asks again under a new id; + - `replay` asks again for a past build at its commit, as a dry run unless told to register, and never + over a newer build without saying so; + - `kill`, `pause` and `resume` are passed on to the seat on the right machine. +- **The build seat's verbs, on each machine:** + - `current` says what is building here and whether this holder is paused; + - `kill` ends the build's whole process group and every container it started; + - `pause` and `resume` set a flag the holder reads before it takes the next ask, which survives its + restart. + +Every action that drops work leaves a failed outcome, so a plan waiting on that build fails and says why +rather than waiting. A plan keeps the id of every build it asked for. From 77429488b01dc56279647ef463d0702e89b0f0e3 Mon Sep 17 00:00:00 2001 From: jochen Date: Mon, 5 Oct 2026 18:56:18 +0200 Subject: [PATCH 2/2] ADR 0219: plans say what their builds wait on, and a failed plan can go on --- ...led-through-the-controller-and-the-build-seat.md | 13 ++++++++++++- 03-DESIGN/01-to-be/18-building-a-module.md | 2 +- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md b/02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md index c378aaa..d68c861 100644 --- a/02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md +++ b/02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md @@ -75,7 +75,17 @@ taken in like any other. A plan waiting on that build fails, and says why, inste the id it asked for, so it can match its outcome exactly. A holder checks a cancelled ask before it builds it, so an ask taken in the instant it was cancelled is not built. -**4. Replay does not move the mesh backwards unasked.** A replayed build is a dry run: built, its log +**4. A plan says what its builds are waiting on, and a failed plan can go on.** + +- A plan whose build waits on a paused seat says the seat is paused, and on which machines, and is not + counted late while it waits. +- `plans retry ` asks again for the modules a failed plan could not build, under new ids. The plan + resumes at that tier and goes on through its later ones. A plan another has superseded, or one + already done, is refused. +- `rebuild` of a module that an open or failed plan has not yet built joins that plan, so the plan and + the build are one thing. + +**5. Replay does not move the mesh backwards unasked.** A replayed build is a dry run: built, its log kept, nothing registered. With `--register` it is registered. If a newer build of the module is already registered, that is refused unless `--older` is said as well: registering an older commit makes it the current one, and the rollout policy sends it to the machines ([issue 207](../04-ISSUES/207-a-re-made-worker-replayed-every-ask-the-stream-kept/00-report.md)). @@ -99,6 +109,7 @@ current one, and the rollout policy sends it to the machines ([issue 207](../04- | a cancelled ask is not built | the holder's test: an ask taken after its cancel is answered failed without building | | kill stops everything it started | the holder's test: the build's process group and its labelled containers are ended; the outcome is failed and the ask acknowledged | | pause survives a restart | the holder's test: the flag is read back at start, and nothing is taken while it is set | +| plans follow the queue | the controller's test: a plan waiting on a paused seat says so and is not late; `plans retry` resumes a failed plan and its later tiers are asked; `rebuild` joins the plan that holds the module | | replay is safe | the controller's test: a dry run by default, and `--register` over a newer build refused without `--older` | ## References diff --git a/03-DESIGN/01-to-be/18-building-a-module.md b/03-DESIGN/01-to-be/18-building-a-module.md index 9408236..456ed81 100644 --- a/03-DESIGN/01-to-be/18-building-a-module.md +++ b/03-DESIGN/01-to-be/18-building-a-module.md @@ -417,4 +417,4 @@ queue, and each machine's build agent holds its own process. restart. Every action that drops work leaves a failed outcome, so a plan waiting on that build fails and says why -rather than waiting. A plan keeps the id of every build it asked for. +rather than waiting. A plan keeps the id of every build it asked for. A plan waiting on a paused seat says so and is not counted late; a failed plan can be resumed with `plans retry`, and a `rebuild` of a module a plan holds joins that plan.