novox/mesh-catalog
18
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

The build modules refuse every step and report unhealthy under a 20 GiB free-space floor, release workspaces, and sweep after 2 h (issues 1839 and 1894, ADR 0532) #458

Merged
jschoubben merged 2 commits from fix/1839-build-floor into main 2026-10-23 11:29:04 +00:00
Conversation 3 Commits 2 Files Changed 34
+724 -73
mesh-tools commented 2026-10-23 06:40:12 +00:00
Member
Copy Link
Copy Source

Issue 1839 (blocking, 2026-10-23): the new build modules filled a build node's root filesystem twice this morning (no floor, 24 h workspaces); stopped and unassigned from that node by hand until this is delivered. Group with mesh-sdk on the same branch (pins its head 6147a204).

mesh-build-go, mesh-build-node, mesh-build-image: room.go refuses fetch/test/build while a filesystem the step writes (go: work + Go caches; node: work + toolchain cache; image: work + image store) is under the floor, and the module's health is unhealthy with the same text until it has room; build__release {workspace}; a failed fetch removes its own workspace; the sweep is 2 h and never removes a held workspace; mesh-build-node sweeps every 10 minutes. build_image_copy is not checked (it writes nothing). Decision: ADR 0532 (accepted); ADR 0509 for release and the 2 h bound.

Tests seen failing first (1839/AC1-AC4); go test and vet (with and without the integration tag) green in all three modules; the catalogue's words, particulars, toolchains and paths checks pass. The first try failed only at its pin check (mesh-sdk had no pull request yet).

Also: issue 1894 (the real-buildah tests skip where the module's guard refuses).

Reviewed-by: review-1894-888@shanks at 539a31103b

Issue 1839 (blocking, 2026-10-23): the new build modules filled a build node's root filesystem twice this morning (no floor, 24 h workspaces); stopped and unassigned from that node by hand until this is delivered. Group with mesh-sdk on the same branch (pins its head 6147a204). mesh-build-go, mesh-build-node, mesh-build-image: room.go refuses fetch/test/build while a filesystem the step writes (go: work + Go caches; node: work + toolchain cache; image: work + image store) is under the floor, and the module's health is unhealthy with the same text until it has room; build_<x>_release {workspace}; a failed fetch removes its own workspace; the sweep is 2 h and never removes a held workspace; mesh-build-node sweeps every 10 minutes. build_image_copy is not checked (it writes nothing). Decision: ADR 0532 (accepted); ADR 0509 for release and the 2 h bound. Tests seen failing first (1839/AC1-AC4); go test and vet (with and without the integration tag) green in all three modules; the catalogue's words, particulars, toolchains and paths checks pass. The first try failed only at its pin check (mesh-sdk had no pull request yet). Also: issue 1894 (the real-buildah tests skip where the module's guard refuses). Reviewed-by: review-1894-888@shanks at 539a31103b534561ee04153836742d53ba6910a3
mesh-tools commented 2026-10-23 06:40:18 +00:00
Author
Member
Copy Link
Copy Source

Delivery novox/mesh-catalog@539a31103b53 — delivered since 2026-10-23T11:54:37Z

Delivery plan — builds mesh-build-go, mesh-build-image, mesh-build-node → g14, novox, shanks; no bus step

  • tier 0: mesh-build-go, mesh-build-image, mesh-build-node
  • g14: receives mesh-build-go, mesh-build-node
  • novox: receives mesh-build-go, mesh-build-node
  • shanks: receives mesh-build-go, mesh-build-image, mesh-build-node
  • grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before mesh-build-go, mesh-build-image, mesh-build-node is judged on its first machine

Group fix/1839-build-floor: novox/mesh-catalog@539a31103b, novox/mesh-sdk@6147a20454

  • composed together: fail — the checker's verdict at the members' heads (e77fb3c3b256): integration: meshbuildnode/cmd/mesh-build-node.TestAFixtureBundleBuiltTwiceHasOneDigest

Machines

  • mesh-build-go on every machine running it: sent
  • mesh-build-image on every machine running it: sent
  • mesh-build-node on every machine running it: sent
  • mesh-sdk on every machine running it: sent
  • mesh-tools on every machine running it: sent
  • audit-logger on every machine running it: sent
  • baserow on every machine running it: sent
  • bazarr on every machine running it: sent
  • bookshelf on every machine running it: sent
  • confluence on every machine running it: sent
  • gitea on every machine running it: sent
  • gitlab on every machine running it: sent
  • grafana on every machine running it: sent
  • home-assistant on every machine running it: sent
  • icecast on every machine running it: sent
  • influxdb on every machine running it: sent
  • jackett on every machine running it: sent
  • jira on every machine running it: sent
  • kometa on every machine running it: sent
  • lab on every machine running it: sent
  • letta on every machine running it: sent
  • lidarr on every machine running it: sent
  • mesh-catalog on every machine running it: sent
  • model-usage on every machine running it: sent
  • mosquitto on every machine running it: sent
  • netcheck on every machine running it: sent
  • nextcloud on every machine running it: sent
  • nodered on every machine running it: sent
  • ombi on every machine running it: sent
  • openai-consumer on every machine running it: sent
  • radarr on every machine running it: sent
  • redis on every machine running it: sent
  • route-adapter on every machine running it: sent
  • searxng on every machine running it: sent
  • showcase on every machine running it: sent
  • sonarr on every machine running it: sent
  • tautulli on every machine running it: sent
  • umami on every machine running it: sent
  • unifi on every machine running it: sent
  • zsh on every machine running it: sent

Transitions

  • 2026-10-23 10:38 (new) → proposed (announced): novox/mesh-catalog#458's head announced
  • 2026-10-23 11:22 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
  • 2026-10-23 11:22 checked → rejected (refused): the gate failed or could not run, or the repository's own check failed
  • 2026-10-23 11:29 rejected → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on
  • 2026-10-23 11:30 held → delivering (release): the operator merged mesh-catalog #458 at the forge by hand at 13:3x (chat with lead-47@shanks): the only failure is TestAFixtureBundleBuiltTwiceHasOneDigest, environmental (the installed mesh-build-node predates its 0711 cache, issue 1713's aftermath); 6158 unit and 6236 integration tests otherwise passed; this delivery brings the release that opens the cache (issues 1839, 1894, ADR 0532)
  • 2026-10-23 11:54 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-catalog@539a31103b53` — **delivered** since 2026-10-23T11:54:37Z **Delivery plan** — builds mesh-build-go, mesh-build-image, mesh-build-node → g14, novox, shanks; no bus step - tier 0: mesh-build-go, mesh-build-image, mesh-build-node - g14: receives mesh-build-go, mesh-build-node - novox: receives mesh-build-go, mesh-build-node - shanks: receives mesh-build-go, mesh-build-image, mesh-build-node - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before mesh-build-go, mesh-build-image, mesh-build-node is judged on its first machine **Group** `fix/1839-build-floor`: novox/mesh-catalog@539a31103b53, novox/mesh-sdk@6147a2045467 - composed together: fail — the checker's verdict at the members' heads (e77fb3c3b256): integration: meshbuildnode/cmd/mesh-build-node.TestAFixtureBundleBuiltTwiceHasOneDigest **Machines** - mesh-build-go on every machine running it: sent - mesh-build-image on every machine running it: sent - mesh-build-node on every machine running it: sent - mesh-sdk on every machine running it: sent - mesh-tools on every machine running it: sent - audit-logger on every machine running it: sent - baserow on every machine running it: sent - bazarr on every machine running it: sent - bookshelf on every machine running it: sent - confluence on every machine running it: sent - gitea on every machine running it: sent - gitlab on every machine running it: sent - grafana on every machine running it: sent - home-assistant on every machine running it: sent - icecast on every machine running it: sent - influxdb on every machine running it: sent - jackett on every machine running it: sent - jira on every machine running it: sent - kometa on every machine running it: sent - lab on every machine running it: sent - letta on every machine running it: sent - lidarr on every machine running it: sent - mesh-catalog on every machine running it: sent - model-usage on every machine running it: sent - mosquitto on every machine running it: sent - netcheck on every machine running it: sent - nextcloud on every machine running it: sent - nodered on every machine running it: sent - ombi on every machine running it: sent - openai-consumer on every machine running it: sent - radarr on every machine running it: sent - redis on every machine running it: sent - route-adapter on every machine running it: sent - searxng on every machine running it: sent - showcase on every machine running it: sent - sonarr on every machine running it: sent - tautulli on every machine running it: sent - umami on every machine running it: sent - unifi on every machine running it: sent - zsh on every machine running it: sent **Transitions** - 2026-10-23 10:38 (new) → proposed (announced): novox/mesh-catalog#458's head announced - 2026-10-23 11:22 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-23 11:22 checked → rejected (refused): the gate failed or could not run, or the repository's own check failed - 2026-10-23 11:29 rejected → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on - 2026-10-23 11:30 held → delivering (release): the operator merged mesh-catalog #458 at the forge by hand at 13:3x (chat with lead-47@shanks): the only failure is TestAFixtureBundleBuiltTwiceHasOneDigest, environmental (the installed mesh-build-node predates its 0711 cache, issue 1713's aftermath); 6158 unit and 6236 integration tests otherwise passed; this delivery brings the release that opens the cache (issues 1839, 1894, ADR 0532) - 2026-10-23 11:54 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
mesh-tools commented 2026-10-23 06:52:44 +00:00
Author
Member
Copy Link
Copy Source

Merge check at 60010aa1
Judged: head 60010aa1fa12 merged into trunk 89e15c7bcda6; read beside it: mesh-controller 219fdcd7f00c, mesh-controller-main 219fdcd7f00c, mesh-host 48c3e3834c69, mesh-tools e0fadc43d9dd

  • mesh/merge-gate: PASS — the change touches no module of the mesh's graph
  • mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds mesh-build-go, mesh-build-image, mesh-build-node → g14, novox, shanks; no bus step

  • tier 0: mesh-build-go, mesh-build-image, mesh-build-node
  • g14: receives mesh-build-go, mesh-build-node
  • novox: receives mesh-build-go, mesh-build-node
  • shanks: receives mesh-build-go, mesh-build-image, mesh-build-node
  • grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before mesh-build-go, mesh-build-image, mesh-build-node is judged on its first machine
  • read by no module's build: modules/mesh-build-go/README.md, modules/mesh-build-go/cmd/mesh-build-go/fetch_test.go, modules/mesh-build-go/cmd/mesh-build-go/module_test.go, modules/mesh-build-go/cmd/mesh-build-go/room_test.go, modules/mesh-build-image/README.md, modules/mesh-build-image/cmd/mesh-build-image/room_test.go, modules/mesh-build-node/README.md, modules/mesh-build-node/cmd/mesh-build-node/module_test.go, modules/mesh-build-node/cmd/mesh-build-node/room_test.go

Run by the build seat on shanks as build-1792737625729421749 (builds --log build-1792737625729421749).

--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.003s
words: 12 retired words and 70 daemons' lines, none in the 62856 strings of 977 files
ok  	git.novox.be/novox/mesh-catalog/checks/particulars	0.007s
particulars: none of 4 node names in the 2 covered modules
ok  	git.novox.be/novox/mesh-catalog/checks/toolchains	0.012s
toolchains: go, nodejs, npm and buildah are declared only by their owners
ok  	git.novox.be/novox/mesh-catalog/checks/imagebuild	0.011s
imagebuild: rootless-image-build is provided only by buildah
ok  	git.novox.be/novox/mesh-catalog/checks/imageplace	0.011s
imageplace: every image builds from a Dockerfile in its own module's directory
ok  	git.novox.be/novox/mesh-catalog/checks/locks	0.004s
locks: every TypeScript module that names packages beyond the SDK commits its lock
ok  	git.novox.be/novox/mesh-catalog/checks/agenthome	0.014s
agent-home: no archive and no file written into under the agent's home
ok  	git.novox.be/novox/mesh-catalog/checks/mailstore	0.010s
mail-store: claimed only by mesh-mail
ok  	git.novox.be/novox/mesh-catalog/checks/sshdir	0.022s
~/.ssh: named only by ssh-client
ok  	git.novox.be/novox/mesh-catalog/checks/owned	0.028s
owned paths: every directory and archive says how it treats unknown files; no module declares inside another's directory, or in a system package's
ok  	git.novox.be/novox/mesh-catalog/checks/secrets	0.016s
secrets: no setting can replace a vault secret in a mergeable file (1 known, nested)
ok  	git.novox.be/novox/mesh-catalog/checks/predefined	0.273s
predefined.json: 3 key(s), none removed that a module names
ok  	git.novox.be/novox/mesh-catalog/checks/config	0.017s
config: no moved module says settings, takes-settings or ${setting:<key>}, or takes a whole layer (0 not yet moved)
ok  	git.novox.be/novox/mesh-catalog/checks/templates	0.013s
templates: no module keeps a file's text inline but the 0 listed
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.006s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 58, as kept
ok  	git.novox.be/novox/mesh-catalog/checks/deps	0.235s
testing modules/mesh-build-go
ok  	meshbuildgo/cmd/mesh-build-go	4.375s
testing modules/mesh-build-image
ok  	meshbuildimage/cmd/mesh-build-image	1.603s
testing modules/mesh-build-node
ok  	meshbuildnode/cmd/mesh-build-node	1.854s
--- its merge-check-typescript.sh (typescript toolchain)
**Merge check** at `60010aa1` Judged: head `60010aa1fa12` merged into trunk `89e15c7bcda6`; read beside it: mesh-controller `219fdcd7f00c`, mesh-controller-main `219fdcd7f00c`, mesh-host `48c3e3834c69`, mesh-tools `e0fadc43d9dd` - `mesh/merge-gate`: **PASS** — the change touches no module of the mesh's graph - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds mesh-build-go, mesh-build-image, mesh-build-node → g14, novox, shanks; no bus step - tier 0: mesh-build-go, mesh-build-image, mesh-build-node - g14: receives mesh-build-go, mesh-build-node - novox: receives mesh-build-go, mesh-build-node - shanks: receives mesh-build-go, mesh-build-image, mesh-build-node - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before mesh-build-go, mesh-build-image, mesh-build-node is judged on its first machine - read by no module's build: modules/mesh-build-go/README.md, modules/mesh-build-go/cmd/mesh-build-go/fetch_test.go, modules/mesh-build-go/cmd/mesh-build-go/module_test.go, modules/mesh-build-go/cmd/mesh-build-go/room_test.go, modules/mesh-build-image/README.md, modules/mesh-build-image/cmd/mesh-build-image/room_test.go, modules/mesh-build-node/README.md, modules/mesh-build-node/cmd/mesh-build-node/module_test.go, modules/mesh-build-node/cmd/mesh-build-node/room_test.go Run by the build seat on shanks as `build-1792737625729421749` (`builds --log build-1792737625729421749`). ``` --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.003s words: 12 retired words and 70 daemons' lines, none in the 62856 strings of 977 files ok git.novox.be/novox/mesh-catalog/checks/particulars 0.007s particulars: none of 4 node names in the 2 covered modules ok git.novox.be/novox/mesh-catalog/checks/toolchains 0.012s toolchains: go, nodejs, npm and buildah are declared only by their owners ok git.novox.be/novox/mesh-catalog/checks/imagebuild 0.011s imagebuild: rootless-image-build is provided only by buildah ok git.novox.be/novox/mesh-catalog/checks/imageplace 0.011s imageplace: every image builds from a Dockerfile in its own module's directory ok git.novox.be/novox/mesh-catalog/checks/locks 0.004s locks: every TypeScript module that names packages beyond the SDK commits its lock ok git.novox.be/novox/mesh-catalog/checks/agenthome 0.014s agent-home: no archive and no file written into under the agent's home ok git.novox.be/novox/mesh-catalog/checks/mailstore 0.010s mail-store: claimed only by mesh-mail ok git.novox.be/novox/mesh-catalog/checks/sshdir 0.022s ~/.ssh: named only by ssh-client ok git.novox.be/novox/mesh-catalog/checks/owned 0.028s owned paths: every directory and archive says how it treats unknown files; no module declares inside another's directory, or in a system package's ok git.novox.be/novox/mesh-catalog/checks/secrets 0.016s secrets: no setting can replace a vault secret in a mergeable file (1 known, nested) ok git.novox.be/novox/mesh-catalog/checks/predefined 0.273s predefined.json: 3 key(s), none removed that a module names ok git.novox.be/novox/mesh-catalog/checks/config 0.017s config: no moved module says settings, takes-settings or ${setting:<key>}, or takes a whole layer (0 not yet moved) ok git.novox.be/novox/mesh-catalog/checks/templates 0.013s templates: no module keeps a file's text inline but the 0 listed ok git.novox.be/novox/mesh-catalog/checks/paths 0.006s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 58, as kept ok git.novox.be/novox/mesh-catalog/checks/deps 0.235s testing modules/mesh-build-go ok meshbuildgo/cmd/mesh-build-go 4.375s testing modules/mesh-build-image ok meshbuildimage/cmd/mesh-build-image 1.603s testing modules/mesh-build-node ok meshbuildnode/cmd/mesh-build-node 1.854s --- its merge-check-typescript.sh (typescript toolchain) ```
mesh-tools commented 2026-10-23 10:03:41 +00:00
Author
Member
Copy Link
Copy Source

mesh-delivery: not merged. This pull request's checks passed at 60010aa1fa, and now its check failed: failed: unit: logrotate/cmd/logrotate-tools.TestTheBaseParses: manifest_test.go:52: logrotate -d: exit status 1 | 60010aa1fa on 89e15c7bc…. A new head, or a recheck that passes, asks the operator again.

mesh-delivery: **not merged.** This pull request's checks passed at 60010aa1fa12e4e8ca6b966623dff9c72813a340, and now its check failed: failed: unit: logrotate/cmd/logrotate-tools.TestTheBaseParses: manifest_test.go:52: logrotate -d: exit status 1 | 60010aa1fa12 on 89e15c7bc…. A new head, or a recheck that passes, asks the operator again.
jschoubben added 1 commit 2026-10-23 10:18:11 +00:00
Refuse build steps under a free-space floor and release workspaces, so the build modules stop filling a build node's disk (issue 1839)
mesh/delivery-group group fix/1839-build-floor checking: 1 of 2 member(s) ready
Details
mesh/delivery superseded: a newer head of the same pull request
Details
mesh/check failed: novox: the check ran past its bound of 30m0s, stuck at: test the unit level of novox/mesh-catalog#458 on novox | 3ca48ff23005 on 89…
3ca48ff230 
mesh-build-go, mesh-build-node and mesh-build-image refuse every fetch, test
and build, and say unhealthy, while their work or cache filesystem has less
than 20 GiB free (build-agent's floor), so the builder chooses another node
instead of filling this one. Each serves build_<x>_release, removing a
workspace once its caller's last step answered; a failed fetch removes its
own; a workspace never released goes two hours after its last use (ADR 0509)
in place of a day, never while a step holds it. Pins mesh-sdk's
fix/1839-build-floor (buildstep.Floor, buildstep.Workspaces).
jschoubben force-pushed fix/1839-build-floor from 60010aa1fa to 3ca48ff230 2026-10-23 10:18:11 +00:00 Compare
jschoubben added 1 commit 2026-10-23 10:38:11 +00:00
mesh-build-image: let the module's own guard decide whether the real-buildah tests run, so a check sandbox where rootless buildah cannot unshare skips them loudly instead of failing every catalogue check (issue 1894)
mesh/delivery delivered
Details
mesh/delivery-group group fix/1839-build-floor delivered: every member is delivered
Details
mesh/check failed: g14: the check ran past its bound of 30m0s, stuck at: test the integration level of novox/mesh-catalog#458 on g14 | 539a31103b53 on…
539a31103b
mesh-tools changed title from The build modules refuse every step and report unhealthy under a 20 GiB free-space floor, release workspaces, and sweep after 2 h (issue 1839, ADR 0532) to The build modules refuse every step and report unhealthy under a 20 GiB free-space floor, release workspaces, and sweep after 2 h (issues 1839 and 1894, ADR 0532) 2026-10-23 10:40:08 +00:00
jschoubben scheduled this pull request to auto merge when all checks succeed 2026-10-23 11:13:37 +00:00
jschoubben merged commit 15378cfba7 into main 2026-10-23 11:29:04 +00:00
jschoubben deleted branch fix/1839-build-floor 2026-10-23 11:29:05 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
mesh/ready
Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-catalog#458