docker: the module's state is in the mesh's directory, never /var/lib/docker (hq issue 452) #217

Merged
mesh-admin merged 1 commits from fix/452-docker-state-is-its-own into main 2026-10-11 16:07:36 +00:00
Contributor

Fixes the failed rollout of #205 (novox/hq issue 452).

What failed

#205 (merged as e2d25b22) declared the state directory with "place": ".". The controller resolves that to <root>/<module> (mesh-controller internal/catalogue/dir_into.go, dirsFor), which for module docker on the default root is /var/lib/docker: Docker's own directory. The node-engine refuses to place anything in or below the container runtimes' data (mesh-host internal/apply/placement_guard.go, novox/hq issue 339). ace's apply at 12:42:08 failed docker.state and the walk put the module back; g14, novox and shanks never got it.

The change

  • state now says "place": "mesh", which the controller resolves to <root>/mesh/<module>: /var/lib/mesh/docker on a node with the default data root. Gitea, nextcloud, mailu, build-agent and others already place a directory this way. log-rotation (${dir:state}/log-rotation.conf) and the step's DOCKER_LOG_ROTATION follow it by reference, unchanged in text.
  • Nothing else in the module names a path under /var/lib/docker, /var/lib/containers or /var/lib/containerd.
  • statedir_test.go: resolves every directory as dirsFor does on the default root and fails when a directory, a resource path or an env value lands in a runtime's data; and pins state at /var/lib/mesh/docker. Both tests failed on main (state resolved to /var/lib/docker; log-rotation and the step named /var/lib/docker/log-rotation.conf).
  • README: says where the directory is and why never /var/lib/docker.

Checks run

go vet ./..., go test -race ./... -count=1 in modules/docker/cmd/docker-tools: pass. checks/paths: pass (3 known). checks/words: pass. No database, no containers.

Why the gate did not catch it, and whether checks/paths can

The merge gate composes and validates the declaration; the controller's own guard (systemPath in placement.go) is applied to the places and accesses settings only, not to a directory the default layout resolves. The refusal lives only in the node-engine's apply. checks/paths cannot catch the class: it reads only stated path values (a pathless place has none) and judges one module naming another's path, not a path landing in a machine's own tree. The class for every module is best caught in the controller, by running systemPath over every dirsFor result at compose time, so the gate refuses it; that is a mesh-control change, not part of this PR.

How a reviewer verifies the path

Before merge: read dirsFor (placeMesh → dataRoot + "/mesh/" + m.Module) and run the test. After the branch is registered or merged: mesh-controller.plan {"node":"ace","diff":true} shows docker.state at /var/lib/mesh/docker and docker.log-rotation at /var/lib/mesh/docker/log-rotation.conf, unless ace's assignment sets a data root, in which case <that root>/mesh/docker. /var/lib/mesh/docker is below /var/lib/mesh, which the engine protects only as a root (never the directory itself or one holding it), as for gitea's mesh-state.

Fixes the failed rollout of #205 (novox/hq issue 452). ## What failed #205 (merged as e2d25b22) declared the `state` directory with `"place": "."`. The controller resolves that to `<root>/<module>` (mesh-controller `internal/catalogue/dir_into.go`, `dirsFor`), which for module `docker` on the default root is `/var/lib/docker`: Docker's own directory. The node-engine refuses to place anything in or below the container runtimes' data (mesh-host `internal/apply/placement_guard.go`, novox/hq issue 339). ace's apply at 12:42:08 failed `docker.state` and the walk put the module back; g14, novox and shanks never got it. ## The change - `state` now says `"place": "mesh"`, which the controller resolves to `<root>/mesh/<module>`: `/var/lib/mesh/docker` on a node with the default data root. Gitea, nextcloud, mailu, build-agent and others already place a directory this way. `log-rotation` (`${dir:state}/log-rotation.conf`) and the step's `DOCKER_LOG_ROTATION` follow it by reference, unchanged in text. - Nothing else in the module names a path under `/var/lib/docker`, `/var/lib/containers` or `/var/lib/containerd`. - `statedir_test.go`: resolves every directory as `dirsFor` does on the default root and fails when a directory, a resource path or an env value lands in a runtime's data; and pins `state` at `/var/lib/mesh/docker`. Both tests failed on main (state resolved to /var/lib/docker; log-rotation and the step named /var/lib/docker/log-rotation.conf). - README: says where the directory is and why never /var/lib/docker. ## Checks run `go vet ./...`, `go test -race ./... -count=1` in modules/docker/cmd/docker-tools: pass. `checks/paths`: pass (3 known). `checks/words`: pass. No database, no containers. ## Why the gate did not catch it, and whether checks/paths can The merge gate composes and validates the declaration; the controller's own guard (`systemPath` in `placement.go`) is applied to the `places` and `accesses` settings only, not to a directory the default layout resolves. The refusal lives only in the node-engine's apply. `checks/paths` cannot catch the class: it reads only stated `path` values (a pathless `place` has none) and judges one module naming another's path, not a path landing in a machine's own tree. The class for every module is best caught in the controller, by running `systemPath` over every `dirsFor` result at compose time, so the gate refuses it; that is a mesh-control change, not part of this PR. ## How a reviewer verifies the path Before merge: read `dirsFor` (placeMesh → `dataRoot + "/mesh/" + m.Module`) and run the test. After the branch is registered or merged: `mesh-controller.plan` `{"node":"ace","diff":true}` shows `docker.state` at `/var/lib/mesh/docker` and `docker.log-rotation` at `/var/lib/mesh/docker/log-rotation.conf`, unless ace's assignment sets a data root, in which case `<that root>/mesh/docker`. `/var/lib/mesh/docker` is below `/var/lib/mesh`, which the engine protects only as a root (never the directory itself or one holding it), as for gitea's `mesh-state`.
mesh-admin added 1 commit 2026-10-11 16:01:42 +00:00
docker: keep the module's state in the mesh's directory, never Docker's own
mesh/merge-gate pass: builds docker → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
7e6889a646
#205's "place": "." resolved to /var/lib/docker, where the node-engine places
nothing (issue 339), so ace refused the apply and put the module back. The state
directory now says "place": "mesh", <root>/mesh/docker, and a test holds every
directory and path of the module out of the container runtimes' data (novox/hq
issue 452).
Author
Contributor

Delivery novox/mesh-catalog@7e6889a64689 — delivered since 2026-10-11T16:13:01Z

Delivery plan — builds docker → ace, g14, novox, shanks; no bus step

  • tier 0: docker
  • ace: receives docker
  • g14: receives docker
  • novox: receives docker
  • shanks: receives docker

Machines

  • docker on ace: passed — healthy 3 times over 2m30s
  • docker on the rest running it: sent

Transitions

  • 2026-10-11 16:01 (new) → proposed (announced): novox/mesh-catalog#217's head announced
  • 2026-10-11 16:04 proposed → checked (checked): the verdict names this commit
  • 2026-10-11 16:04 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-11 16:09 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-11 16:09 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
  • 2026-10-11 16:13 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-catalog@7e6889a64689` — **delivered** since 2026-10-11T16:13:01Z **Delivery plan** — builds docker → ace, g14, novox, shanks; no bus step - tier 0: docker - ace: receives docker - g14: receives docker - novox: receives docker - shanks: receives docker **Machines** - docker on ace: passed — healthy 3 times over 2m30s - docker on the rest running it: sent **Transitions** - 2026-10-11 16:01 (new) → proposed (announced): novox/mesh-catalog#217's head announced - 2026-10-11 16:04 proposed → checked (checked): the verdict names this commit - 2026-10-11 16:04 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-11 16:09 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-11 16:09 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit - 2026-10-11 16:13 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
Author
Contributor

Merge check at 7e6889a6

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: docker)
  • mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds docker → ace, g14, novox, shanks; no bus step

  • tier 0: docker
  • ace: receives docker
  • g14: receives docker
  • novox: receives docker
  • shanks: receives docker
  • read by no module's build: modules/docker/README.md, modules/docker/cmd/docker-tools/statedir_test.go

Run by the build seat on novox as build-1791734506573285106 (builds --log build-1791734506573285106).

--- the gate: docker
docker: ok, 22 tool(s), invokes seat:mesh-controller.images; WARNING: socket, runtime, prune stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 3
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-11T11:22:35Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-11T11:22:35Z

modules the change moves:
  - docker (changed)

machines:
  - a machine (f35): composes; the change adds 3 resource(s) and removes 0 (+directory:docker.state, +file:docker.log-rotation, +process:docker.log-rotation-check)
  - holds mesh-dns-resolver (idt): composes; the change adds 3 resource(s) and removes 0 (+directory:docker.state, +file:docker.log-rotation, +process:docker.log-rotation-check)
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes; the change adds 3 resource(s) and removes 0 (+directory:docker.state, +file:docker.log-rotation, +process:docker.log-rotation-check)
  - a machine (yvbpsm): composes; the change adds 3 resource(s) and removes 0 (+directory:docker.state, +file:docker.log-rotation, +process:docker.log-rotation-check)

a merge would rebuild 1 module(s) in 1 tier(s): docker

notes:
  - modules/docker/README.md and modules/docker/cmd/docker-tools/statedir_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.005s
words: 12 retired words and 70 daemons' lines, none in the 41738 strings of 676 files
ok  	git.novox.be/novox/mesh-catalog/checks/agenthome	0.015s
agent-home: no archive and no file written into under the agent's home
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.007s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 69, as kept
testing modules/docker
ok  	docker/cmd/docker-tools	1.062s
--- its merge-check-typescript.sh (typescript toolchain)
**Merge check** at `7e6889a6` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: docker) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds docker → ace, g14, novox, shanks; no bus step - tier 0: docker - ace: receives docker - g14: receives docker - novox: receives docker - shanks: receives docker - read by no module's build: modules/docker/README.md, modules/docker/cmd/docker-tools/statedir_test.go Run by the build seat on novox as `build-1791734506573285106` (`builds --log build-1791734506573285106`). ``` --- the gate: docker docker: ok, 22 tool(s), invokes seat:mesh-controller.images; WARNING: socket, runtime, prune stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 3 files asking for a setting without saying whether it is trusted: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-11T11:22:35Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-11T11:22:35Z modules the change moves: - docker (changed) machines: - a machine (f35): composes; the change adds 3 resource(s) and removes 0 (+directory:docker.state, +file:docker.log-rotation, +process:docker.log-rotation-check) - holds mesh-dns-resolver (idt): composes; the change adds 3 resource(s) and removes 0 (+directory:docker.state, +file:docker.log-rotation, +process:docker.log-rotation-check) - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes; the change adds 3 resource(s) and removes 0 (+directory:docker.state, +file:docker.log-rotation, +process:docker.log-rotation-check) - a machine (yvbpsm): composes; the change adds 3 resource(s) and removes 0 (+directory:docker.state, +file:docker.log-rotation, +process:docker.log-rotation-check) a merge would rebuild 1 module(s) in 1 tier(s): docker notes: - modules/docker/README.md and modules/docker/cmd/docker-tools/statedir_test.go read by no module's build, and rebuild nothing (issue 280) --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.005s words: 12 retired words and 70 daemons' lines, none in the 41738 strings of 676 files ok git.novox.be/novox/mesh-catalog/checks/agenthome 0.015s agent-home: no archive and no file written into under the agent's home ok git.novox.be/novox/mesh-catalog/checks/paths 0.007s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 69, as kept testing modules/docker ok docker/cmd/docker-tools 1.062s --- its merge-check-typescript.sh (typescript toolchain) ```
mesh-admin merged commit 0a55831992 into main 2026-10-11 16:07:36 +00:00
mesh-admin deleted branch fix/452-docker-state-is-its-own 2026-10-11 16:07:36 +00:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-catalog#217