7.4 KiB
topic, status, date, deciders, reconstructed, extends
| topic | status | date | deciders | reconstructed | extends |
|---|---|---|---|---|---|
| the mesh | accepted | 2026-10-05 | jochen | false | 02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md |
219. The build queue is controlled through the controller and the build seat
Context
The operator asked for tools to control the mesh's builds: see what is queued and running, cancel, clear, stop a build immediately, pause and continue, restart and replay. On the day of the request none existed. The controller could ask for a build and list finished ones. Nothing could see an ask waiting on the build seat's work queue, or one being built. Nothing could take an ask back, and a running build could only be stopped by restarting the machine's build agent, which hands the ask to another holder.
How builds run today, read from the code and the bus (ADR 0190, ADR 0157):
- an ask is a message on the seat's work-queue stream;
- every holder pulls one at a time from one shared worker, and acknowledges after it has announced the outcome;
- a build says it started, logs its steps, and announces what it built, all on the bus;
- an ask delivered five times without an answer stays in the stream for a week, with nothing saying so.
Three facts constrain any control:
- Only the controller may act on the bus's streams and consumers (design 25 §3, enforced in the bus's user list). Holders may only take from their worker and acknowledge.
- A plan waits for a build's outcome and has no timeout. An ask that disappears without one leaves its plan waiting, said only as late after half an hour.
- The bus server in use cannot pause a consumer; that arrived in a later version.
Considered Options
- Pause and cancel by remaking the worker consumer. Rejected: a remade worker delivers from now on, so every queued ask would be skipped silently. Issues 206 and 207 are this mistake both ways round.
- Upgrade the bus first and use its consumer pause. Not now: it gives pause and continue, and nothing else on the list, and upgrading the bus is a change of its own.
- Queue actions are the controller's verbs, process actions are the build seat's verbs, and every action that drops work leaves a failed outcome. Chosen.
Decision
1. The queue is the controller's. Its verbs act on the work-queue stream, the one thing only it may touch:
| verb | what it does |
|---|---|
queue |
lists every ask: waiting, in flight (with the machine building it, from its started event), and dead (delivered as often as allowed, still in the stream) |
cancel <id> |
takes back a waiting or dead ask; an ask in flight is refused, and kill is named instead |
clear |
cancels every waiting ask, and with --dead every dead one |
rebuild <module or build> |
asks again for a module's current source, or a past build's source and ref, under a new id |
replay <build> |
asks again for a past build at its commit, as a dry run unless told to register |
kill <id>, pause [node], resume [node] |
pass the request on to the build seat on the right machine, or on every machine |
2. The process is the build seat's. Each holder serves verbs on its own machine:
current: the build running here, its step and how long, and whether this holder is paused;kill: stops a running build at once. The build's whole process group is ended, along with every container it started. Its outcome is announced as failed, and the ask is acknowledged, so it is not delivered again;pauseandresume: a paused holder takes nothing new, and a running build finishes. The flag survives the holder's restart.
A holder restarted mid-build keeps today's behaviour: the ask is not acknowledged, and another holder takes it.
3. Nothing dropped is silent. Cancel, clear and kill each leave a failed outcome for the ask's id, taken in like any other. A plan waiting on that build fails, and says why, instead of waiting. A plan keeps the id it asked for, so it can match its outcome exactly. A holder checks a cancelled ask before it builds it, so an ask taken in the instant it was cancelled is not built.
4. A plan says what its builds are waiting on, and a failed plan can go on.
- A plan whose build waits on a paused seat says the seat is paused, and on which machines, and is not counted late while it waits.
plans retry <id>asks again for the modules a failed plan could not build, under new ids. The plan resumes at that tier and goes on through its later ones. A plan another has superseded, or one already done, is refused.rebuildof a module that an open or failed plan has not yet built joins that plan, so the plan and the build are one thing.
5. Replay does not move the mesh backwards unasked. A replayed build is a dry run: built, its log
kept, nothing registered. With --register it is registered. If a newer build of the module is already
registered, that is refused unless --older is said as well: registering an older commit makes it the
current one, and the rollout policy sends it to the machines (issue 207).
Consequences
- Every build in the mesh can be seen, taken back, stopped or asked again from the console. None of it needs a shell on a machine.
- A cancelled or killed build shows in the build records as failed, with who stopped it. Its plan fails saying the same.
- What got harder: a holder now serves verbs as well as taking work, and keeps one small flag on disk. Pause is per holder, so "pause the mesh" is the controller asking every holder in turn. A holder away at the time misses it, and the answer names that holder.
How it is checked
| Rule | Checked by |
|---|---|
| the queue is read and classified right | the controller's test: waiting, in flight and dead asks told apart from the stream and the worker; a live test on a throwaway bus |
| nothing dropped is silent | the controller's test: cancel and clear delete the ask and record a failed outcome; a plan asked for that id fails |
| a cancelled ask is not built | the holder's test: an ask taken after its cancel is answered failed without building |
| kill stops everything it started | the holder's test: the build's process group and its labelled containers are ended; the outcome is failed and the ask acknowledged |
| pause survives a restart | the holder's test: the flag is read back at start, and nothing is taken while it is set |
| plans follow the queue | the controller's test: a plan waiting on a paused seat says so and is not late; plans retry resumes a failed plan and its later tiers are asked; rebuild joins the plan that holds the module |
| replay is safe | the controller's test: a dry run by default, and --register over a newer build refused without --older |