Raise a check's store without durability and remove what earlier holders left (hq issue 306)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered

On the control node the store-bound packages of the controller's suite ran
five to seven times slower than on any other holder, and every controller
check that landed there ran past the suite's thirty minutes: a throwaway
store flushing to a disk the mesh's own store, bus and forge keep busy.
A store that lives for one check needs no crash safety.

A holder recreated mid-check left the check's store and bus running, and
the redelivery went to another machine, so nothing removed them: eleven
pairs across four machines. A starting holder has taken nothing, so every
container labelled with an ask of the seat is an earlier holder's.
This commit is contained in:
jochen
2026-10-08 10:20:28 +02:00
parent c714d07739
commit 85b2a1855b
7 changed files with 178 additions and 1 deletions
+50
View File
@@ -76,3 +76,53 @@ func RemoveContainersOf(ctx context.Context, run Runner, id string) (int, error)
}
return len(ids), nil
}
// A holder that stops mid-build leaves its containers running (novox/hq issue 306).
//
// A build's containers are removed by the build itself, by a kill, or by the next delivery of the same
// ask — but only on the machine that delivery reaches. A holder recreated by a rollout while a check
// ran left the check's throwaway store and bus running, the ask went to another machine, and nothing
// on the first one ever looked at them again: eleven pairs across four machines in two days, each a
// postgres that had written its share of the disk. A holder starting has taken nothing yet, so every
// container labelled with an ask of the seat is one an earlier holder here left, and is removed then.
// RemoveLeftBehind removes every container on this machine labelled with an ask of the build seat —
// an id of the seat's shape, `build-<n>` — and says how many. Called by a holder before it takes
// anything, so none of them can be a build it runs. A container labelled with any other id — a check
// a person runs by hand (`check-here-…`), a test's — is not the seat's, and is left.
func RemoveLeftBehind(ctx context.Context, run Runner) (int, error) {
out, err := run(ctx, "", "docker", "ps", "-a", "--filter", "label="+BuildLabel,
"--format", `{{.ID}} {{.Label "`+BuildLabel+`"}}`)
if err != nil {
return 0, err
}
var ids []string
for _, line := range strings.Split(out, "\n") {
fields := strings.Fields(line)
if len(fields) == 2 && anAskOfTheSeat(fields[1]) {
ids = append(ids, fields[0])
}
}
if len(ids) == 0 {
return 0, nil
}
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f"}, ids...)...); err != nil {
return 0, err
}
return len(ids), nil
}
// anAskOfTheSeat is whether an id is of the shape the controller gives the seat's asks: `build-` and the
// moment it was asked, in nanoseconds.
func anAskOfTheSeat(id string) bool {
n, ok := strings.CutPrefix(id, "build-")
if !ok || n == "" {
return false
}
for _, c := range n {
if c < '0' || c > '9' {
return false
}
}
return true
}