Raise a check's store without durability and remove what earlier holders left (hq issue 306)
On the control node the store-bound packages of the controller's suite ran five to seven times slower than on any other holder, and every controller check that landed there ran past the suite's thirty minutes: a throwaway store flushing to a disk the mesh's own store, bus and forge keep busy. A store that lives for one check needs no crash safety. A holder recreated mid-check left the check's store and bus running, and the redelivery went to another machine, so nothing removed them: eleven pairs across four machines. A starting holder has taken nothing, so every container labelled with an ask of the seat is an earlier holder's.
This commit is contained in:
@@ -76,3 +76,53 @@ func RemoveContainersOf(ctx context.Context, run Runner, id string) (int, error)
|
||||
}
|
||||
return len(ids), nil
|
||||
}
|
||||
|
||||
// A holder that stops mid-build leaves its containers running (novox/hq issue 306).
|
||||
//
|
||||
// A build's containers are removed by the build itself, by a kill, or by the next delivery of the same
|
||||
// ask — but only on the machine that delivery reaches. A holder recreated by a rollout while a check
|
||||
// ran left the check's throwaway store and bus running, the ask went to another machine, and nothing
|
||||
// on the first one ever looked at them again: eleven pairs across four machines in two days, each a
|
||||
// postgres that had written its share of the disk. A holder starting has taken nothing yet, so every
|
||||
// container labelled with an ask of the seat is one an earlier holder here left, and is removed then.
|
||||
|
||||
// RemoveLeftBehind removes every container on this machine labelled with an ask of the build seat —
|
||||
// an id of the seat's shape, `build-<n>` — and says how many. Called by a holder before it takes
|
||||
// anything, so none of them can be a build it runs. A container labelled with any other id — a check
|
||||
// a person runs by hand (`check-here-…`), a test's — is not the seat's, and is left.
|
||||
func RemoveLeftBehind(ctx context.Context, run Runner) (int, error) {
|
||||
out, err := run(ctx, "", "docker", "ps", "-a", "--filter", "label="+BuildLabel,
|
||||
"--format", `{{.ID}} {{.Label "`+BuildLabel+`"}}`)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var ids []string
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) == 2 && anAskOfTheSeat(fields[1]) {
|
||||
ids = append(ids, fields[0])
|
||||
}
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f"}, ids...)...); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return len(ids), nil
|
||||
}
|
||||
|
||||
// anAskOfTheSeat is whether an id is of the shape the controller gives the seat's asks: `build-` and the
|
||||
// moment it was asked, in nanoseconds.
|
||||
func anAskOfTheSeat(id string) bool {
|
||||
n, ok := strings.CutPrefix(id, "build-")
|
||||
if !ok || n == "" {
|
||||
return false
|
||||
}
|
||||
for _, c := range n {
|
||||
if c < '0' || c > '9' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user