118 lines
8.1 KiB
Markdown
118 lines
8.1 KiB
Markdown
---
|
|
topic: the mesh
|
|
status: accepted
|
|
date: 2026-10-02
|
|
deciders: jochen
|
|
reconstructed: false
|
|
extends: 02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md
|
|
---
|
|
|
|
# 190. A seat's work is shared by its holders, and building is the first such role
|
|
|
|
## Context
|
|
|
|
Work addressed to a role goes to the seat's `accept` subjects, on a per-seat work queue
|
|
([design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1, [ADR 0041](0041-events-are-a-relationship.md)).
|
|
The holder's worker on that queue is already a queue group — *"even though the seat guarantees one
|
|
holder … the day somebody allows two holders for throughput, every message is processed twice with
|
|
nothing reporting it"* — and design 25 already says what a build queue shared by several machines is:
|
|
*a seat's `accept` subjects, on a work queue with a queue group of holders*. The mechanism was drawn.
|
|
Two things stopped it being used.
|
|
|
|
First, the build role is a **mesh-scoped** seat, `mesh-build-machine`, so there is one holder in the
|
|
whole mesh ([ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md):
|
|
*the mesh's single build machine*). Second, the worker is a push consumer with **one delivery in
|
|
flight** — set so after 2026-10-01, when a push consumer handing out many at once left twenty-six of
|
|
forty-three asks undelivered ([issue 175](../04-ISSUES/175-an-announcement-behind-a-long-build-comes-back/00-report.md)) —
|
|
and one in flight on a shared consumer is one build at a time across every holder there could be.
|
|
|
|
Measured on 2026-10-02: a change to code comments in the tool runtime rebuilt its thirty-five
|
|
dependent images, one after another, on one machine, for about half an hour, while three other
|
|
machines with a container runtime sat idle; the work the mesh wanted next waited behind it. The
|
|
operator's words: *this is our first occurrence of a mesh advantage* — and: *make sure the setup is
|
|
done generically, so if another module also requires mesh functionality it can re-use the pattern.*
|
|
|
|
## Considered Options
|
|
|
|
1. **A second build machine by configuration** — a concurrency setting on the one holder, or a second
|
|
holder admitted by hand. Rejected: a setting on one machine shares nothing, and a second holder
|
|
of a mesh-scoped seat contradicts what a mesh seat means.
|
|
2. **A build-specific dispatcher** — the controller choosing a machine per build and asking it by
|
|
name. Rejected: it reinvents the queue the bus already is, it makes the controller a scheduler,
|
|
and it is specific to building; the next role needing the same would build its own.
|
|
3. **A seat's work is shared by its holders, and the build role becomes node-scoped.** Chosen. It is
|
|
what the bus was drawn to do, it is one rule for every role rather than one for building, and
|
|
"the machines that are online and hold the seat" is exactly the set a queue group's members is.
|
|
|
|
## Decision
|
|
|
|
**1. Work asked of a seat is taken by whichever of its holders is idle.** Every holder of a seat with
|
|
`accepts` reads the seat's one work queue; a node-scoped seat held on several machines has several
|
|
holders, and an ask goes to one of them. The asker addresses the role — `mesh.seat.<seat>.accept.<verb>`
|
|
— and never a machine. The outcome, the role's own event, says which machine did the work (`on`), as a
|
|
build's already does ([ADR 0157](0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)).
|
|
|
|
**2. A holder takes one ask at a time, when it is idle, by pulling.** The worker is a pull consumer:
|
|
a holder fetches one ask, works it, acknowledges, fetches the next. The server never hands an ask
|
|
to a busy holder, so a slow machine never holds work an idle one could take — the fault issue 175
|
|
found in push delivery is removed by the shape rather than by a limit, and the one-in-flight limit
|
|
that made the shared queue serial goes with it. A holder that dies mid-work leaves its ask to be
|
|
redelivered to another, as today.
|
|
|
|
**3. Work that must run on one particular machine is not a work queue.** That is a node seat's verb
|
|
asked of that machine ([design 33](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md) §4), and
|
|
nothing here changes it. A role's work queue is for work whose result is the same whichever holder
|
|
does it: a build is, because what comes out is published by digest to the mesh's store.
|
|
|
|
**4. This is one pattern, not one role's.** Any module that declares a node-scoped seat with
|
|
`accepts` gets decisions 1 and 2 with no further mechanism: the controller derives the queue and the
|
|
worker, the holders pull, the module's manifest says what every holding machine must have. The
|
|
build agent is the first; a module needing work done *somewhere on the mesh* — a scan, a
|
|
conversion, a fetch — declares a seat of its own the same way
|
|
([ADR 0126](0126-a-module-declares-its-own-seats.md)).
|
|
|
|
**5. Building is the first such role.** The build role is `node-build-agent`, scope node, with the
|
|
same `build` ask and the same `started`, `built` and `log.<id>` events as before. Its holder is the
|
|
`build-agent` module: the builder as it is — a container runtime, the artifact store and the package
|
|
registry resolved as provisions, a workspace, the bus credential — assignable to every machine that
|
|
has a container runtime. `mesh-build-machine` and the `builder` module are retired when the new
|
|
holder is assigned where the old one was. The tiered plan ([ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md))
|
|
is unchanged: a tier's asks go out together and are now worked together.
|
|
|
|
## Consequences
|
|
|
|
- A tier of thirty-five images is built by as many machines as hold the seat and are online. A
|
|
machine that is off builds nothing and blocks nothing.
|
|
- Every holding machine fetches base images from the store and pushes what it builds; the store is
|
|
reached as a provision, so this is what the provision was for. A machine with a slow link builds
|
|
slowly, and takes fewer asks for it, which is the point of pulling.
|
|
- A build's outcome carries which machine built it, so a build that fails on one machine and not
|
|
another is a fact the record shows, not a mystery.
|
|
- What got harder: a build's cache is per machine, so a cold machine pays the first pull of every
|
|
base it has never seen; the artifact store is now asked by several machines at once, and the
|
|
package registry likewise. Both are provisions and both are made for that.
|
|
- ADR 0121's *"the mesh's single build machine"* and ADR 0162's *"built by whichever build machine
|
|
is running — the only one there could be"* were true and are no longer; both records carry a note.
|
|
|
|
## How it is checked
|
|
|
|
| Rule | Checked by |
|
|
|---|---|
|
|
| Two holders of one seat each take one of two asks, and a third ask waits for the first to be idle | the controller's test over the work queue against a real bus: two machines bound to one worker, three asks |
|
|
| An ask is never delivered to a busy holder | the same test: the busy holder's ask count stays at one until it acknowledges |
|
|
| A holder that dies mid-work leaves its ask for another | the same test, one holder closed mid-ask |
|
|
| The asker names no machine | the controller's seat table: `node-build-agent` accepts `build` and the asking side publishes to the seat's accept subject, as the existing tests of `build` already require |
|
|
| Live | `builds` shows a tier's builds `on` more than one machine within one plan; `seats` shows `node-build-agent` held on every machine with a container runtime |
|
|
|
|
## References
|
|
|
|
- [Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1 and §5 — the work queue and the queue
|
|
group of holders this uses as drawn
|
|
- [Design 18](../03-DESIGN/01-to-be/18-building-a-module.md) — building a module, amended for
|
|
where a build runs
|
|
- [ADR 0041](0041-events-are-a-relationship.md), [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md),
|
|
[ADR 0126](0126-a-module-declares-its-own-seats.md), [ADR 0157](0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md),
|
|
[ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
|
- [Issue 175](../04-ISSUES/175-an-announcement-behind-a-long-build-comes-back/00-report.md) — why the
|
|
worker had one in flight, and why pulling removes the cause rather than the symptom
|