The operator's desktop slows while it builds (shanks at 92%, the Go build
runtime at 1650% of 24 processors). The build modules declare cpu-cap
(default shanks=50,g14=50,ace=75), which the composer turns into the runtime's
quota at idle priority; each step's go, node and buildah commands get
GOMAXPROCS and UV_THREADPOOL_SIZE for their part of it, and the status tools
show the limits and what the kernel holds the runtime to.
mesh/check failed: ace: the check of novox/mesh-catalog#492 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
mesh/check checking 15a5fe09b62e on c51a46b802d3 | novox/mesh-catalog#488 in group fix/1921-every-failure-r
Pins mesh-sdk fix/1921-every-failure-r, whose check ask no longer stops a level at its first failing test step. A captured fixture with a failing test in each of two packages proves the build module's test step names both.
mesh/check failed: integration: meshbuildnode/cmd/mesh-build-node.TestAFixtureBundleBuiltTwiceHasOneDigest: no toolchain of this lock is installed und…
The review found a cut read took an arbitrary 1000 keys, so the newest steps could be missing. A step id is
base-36 milliseconds plus random digits, so its string order is its start order; the newest max(limit*4, 100)
keys are read now, 1000 at most. The toolchains read logs when it drops keys past its cap.
On a steps state of 1164 keys / 20 MB (mesh-build-node: 10034 keys) the read tools listed every key and
then read each value one round trip at a time, filtering by node only after the Get, so they timed out
at 30 s. Now the keys are filtered by the node's prefix before any Get, the values are read 16 at a
time, at most 1000 a call (the answer says it was cut), and one log line gives rows read and time.
build_*_toolchains reads its values the same parallel way. The kind: a read tool that reads a whole
state key by key; every read tool of the three modules is covered.
mesh/delivery-group group fix/steps-bucket-ages held: a member is held: novox/mesh-sdk@518dae5ec9c8 (merged without a passing check: only a person decides what…
mesh/check pass 4e82a824be7a on cdbe2945cf6e | unit 1994 run, 2322 kept, 5 skipped | integration 2016 run, 2341 kept, 9 skipped | over bound: unit, in…
Without it in Kept, stdio.KeptState refuses the state before the bus (mesh-sdk go/keeps.go
refusedUnannounced), so the pause could be neither read nor written.
mesh-build-go's steps bucket filled to the bus's 64 MiB cap (27,370 keys, none ever removed) and
refused the end of a passed check, so the check read failed; mesh-build-node's (17.8 MB) and
mesh-build-image's grow the same way. A step's key is per step: its reader reads its end within
the step's bound, and a check's verdict is kept by the checker, so two days is ample. The node's
pause key lived in the same bucket and would have ended by itself; it moves to a pauses state
without an age (mesh-sdk fix/steps-bucket-ages, Puller.PausePut). No node is paused today, so
nothing is carried over.
g14's Go builder sat idle at capacity 2 because nothing woke its parked
puller. mesh-build-go, mesh-build-node and mesh-build-image now pin mesh-sdk
b9911dc1, whose puller watches the config value capacity as mesh-config
resolved it and wakes on each change, and hand it that watch.
mesh/delivery-group group fix/1901-package-cause held: a member is held: novox/mesh-catalog@155124b59af4 (merged without a passing check: only a person decides…
mesh/check pass 155124b59af4 on f27f4c8cd58c | unit 2015 run, 2883 kept, 5 skipped | integration 2037 run, 2978 kept, 9 skipped | over bound: unit, in…
An indented log line of another running test that held 'panic:' won over the
real timeout panic, since the lines were trimmed before matching. Go prints
the runtime's lines at column 0; a test's log is indented.
mesh-controller's try was worded 'PASS' again because the race detector
reported after the tests passed and the program exited 66. The cause is now
the race: its access, the frame that made it and where its goroutine was
created, instead of PASS or testing's generic 'race detected during execution
of test'. Tests from captured go test -race -json (1901/AC4).
A try of mesh-controller kept 'PASS' as its only failure: the package's test
program stopped after printing PASS and the package-level failure took its
first line. Now a failure that is not one test's assertion names the tests that
were running and carries the panic line (a timeout's with its running tests),
a signal or fatal error line, a build's first error, or words saying the
program exited during which tests or after they passed; go test's summary lines
are never kept. Tests from captured go test -json of eight real failures
(1901/AC4).
mesh/check failed: shanks: novox/mesh-catalog#473: modules/mesh-build-go/go.mod names mesh-sdk 19c1d35939bc, which is neither on mesh-sdk's trunk nor …
mesh/check failed: mesh-build-go: the check of novox/mesh-catalog#470: the ask mvlnph6fd84a7b's step on shanks did not end within 30m0s of its start |…
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792875397682979170 (novox/mesh-catalog at 6fb0ceee into main,…
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-catalog@753e8fc202c3 (merged as 4b1bed4b into main, walk plan-17927949853…
mesh/check failed: shanks: the check of novox/mesh-catalog#465 failed: the unit level: packages is a list of package patterns | 70ef99595aa9 on e52024…
mesh/check failed: integration: mesh-delivery-gate/cmd/mesh-d… | c32474ee40f4 on 3bec357968da | judged by the candidate's mesh-build-go at 2e667ac15ce7
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/chec… | d6b9cef9d005 on 3bec357968da | judged by the candidate's mesh-build-go at 769f9d906be2
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/chec… | b60077850dfb on 3bec357968da | judged by the candidate's mesh-build-go at 47bcc1b5810e
mesh/delivery-group group feat/fastcheck rejected: composed together: the checker's verdict at the members' heads (a5ea6d8de38e): integration: nats-tools/cmd/m…
The checker asks runner workspace when a merge changes modules/mesh-build-go; the step's
shut-off process now takes a step spec (digest, unit, integration) and every answer says
judged_by. ADR 0535 §1, §3, §5.
mesh/check failed: novox: the check ran past its bound of 30m0s, stuck at: test the unit level of novox/mesh-catalog#458 on novox | 3ca48ff23005 on 89…
mesh-build-go, mesh-build-node and mesh-build-image refuse every fetch, test
and build, and say unhealthy, while their work or cache filesystem has less
than 20 GiB free (build-agent's floor), so the builder chooses another node
instead of filling this one. Each serves build_<x>_release, removing a
workspace once its caller's last step answered; a failed fetch removes its
own; a workspace never released goes two hours after its last use (ADR 0509)
in place of a day, never while a step holds it. Pins mesh-sdk's
fix/1839-build-floor (buildstep.Floor, buildstep.Workspaces).
mesh-build-go binds /var/lib/mesh-build-node-cache/toolchain read-only into the
integration level's one shut-off process when it is there; the unit level never.
mesh-build-node opens its installed toolchains to other accounts (folders 0755,
files 0644, writable by its owner alone; its cache 0711, so the npm cache beside
them stays its own), and realMachine copies the toolchain of its own lock from
there instead of npm ci from the registry, which the level cannot reach.
jschoubben