novox/mesh-host
9
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Keep a replaced build reachable, and delete it only once no process runs from it (hq issue 289) #48

Merged
mesh-admin merged 1 commits from fix/a-verb-survives-the-handover into main 2026-10-07 00:55:24 +00:00
Conversation 3 Commits 1 Files Changed 4
+348 -14
mesh-admin commented 2026-10-07 00:28:29 +00:00
Contributor
Copy Link
Copy Source

novox/hq issue 289. While a controller update was rolling out, the witness moved the running controller's build into .witness/mesh-controller/previous/. That directory is 0700 and owned by root, and the old controller was still serving and running its own binary as each verb. Every verb failed with permission denied. On proof the witness also deleted previous/ without checking whether anything still ran from it.

The order stays the same: move the running build aside, unpack the new one, restart (systemd stops the old process, then starts the new), prove, retire. What changes is that each step is now safe for a process still running from the old build (internal/witness/draining.go):

  • The directories the witness keeps for a process are 0711. The process's own user can enter them; only root can list them. Existing 0700 directories are chmod'ed on the next placement or save.
  • A build that is no longer needed is retired, not deleted. That covers the previous build once proved, a build on trial replaced by a newer one, and the failed build after a restore. It is renamed under .witness/<name>/retired/ and deleted only when no process runs from it. RunningFrom reads every /proc/<pid>/exe, including ones that show (deleted). That finds the PID the old build ran as and every command it started.
  • Watcher.Look sweeps retired builds on every look. The "healthy" line says whether the old build was deleted or is waiting for named PIDs.

Also included: internal/bootstrap/publish_test.go is restructured so that the toolchain's gofmt (Go 1.26) and a newer gofmt lay it out the same way. The build seat's check failed on that file as it was on main.

Tests (real processes, Linux):

  • TestABuildIsKeptReachableAndUndeletedWhileAProcessRunsFromIt: the directories are 0711, the running PID is found in previous/, and after proof the build is still kept until the process stops, then removed at the next sweep.
  • TestABuildOnTrialReplacedIsRetiredNotDeletedUnderItsProcess
  • TestAProcessRunsFromABuildDeletedUnderIt

mesh-controller check-here: merge-gate PASS (4 of 4 compose), repo-check PASS.

novox/hq issue 289. While a controller update was rolling out, the witness moved the running controller's build into `.witness/mesh-controller/previous/`. That directory is 0700 and owned by root, and the old controller was still serving and running its own binary as each verb. Every verb failed with permission denied. On proof the witness also deleted `previous/` without checking whether anything still ran from it. The order stays the same: move the running build aside, unpack the new one, restart (systemd stops the old process, then starts the new), prove, retire. What changes is that each step is now safe for a process still running from the old build (`internal/witness/draining.go`): - The directories the witness keeps for a process are **0711**. The process's own user can enter them; only root can list them. Existing 0700 directories are chmod'ed on the next placement or save. - A build that is no longer needed is **retired, not deleted**. That covers the previous build once proved, a build on trial replaced by a newer one, and the failed build after a restore. It is renamed under `.witness/<name>/retired/` and deleted only when **no process runs from it**. `RunningFrom` reads every `/proc/<pid>/exe`, including ones that show `(deleted)`. That finds the PID the old build ran as and every command it started. - `Watcher.Look` sweeps retired builds on every look. The "healthy" line says whether the old build was deleted or is waiting for named PIDs. Also included: `internal/bootstrap/publish_test.go` is restructured so that the toolchain's gofmt (Go 1.26) and a newer gofmt lay it out the same way. The build seat's check failed on that file as it was on main. Tests (real processes, Linux): - `TestABuildIsKeptReachableAndUndeletedWhileAProcessRunsFromIt`: the directories are 0711, the running PID is found in `previous/`, and after proof the build is still kept until the process stops, then removed at the next sweep. - `TestABuildOnTrialReplacedIsRetiredNotDeletedUnderItsProcess` - `TestAProcessRunsFromABuildDeletedUnderIt` `mesh-controller check-here`: merge-gate PASS (4 of 4 compose), repo-check PASS.
mesh-admin commented 2026-10-07 00:29:23 +00:00
Author
Contributor
Copy Link
Copy Source

Delivery novox/mesh-host@3a117c2d2bdd — delivered since 2026-10-07T01:05:07Z

Delivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-host
  • ace: receives mesh-host
  • g14: receives mesh-host
  • novox: receives mesh-host
  • shanks: receives mesh-host

Machines

  • mesh-host on ace: passed — healthy 3 times over 8m59s
  • mesh-host on the rest running it: sent

Transitions

  • 2026-10-07 00:45 (new) → proposed (announced): novox/mesh-host#48's head announced
  • 2026-10-07 00:48 proposed → checked (checked): the verdict names this commit
  • 2026-10-07 00:48 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-07 00:55 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-07 00:55 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
  • 2026-10-07 01:05 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-host@3a117c2d2bdd` — **delivered** since 2026-10-07T01:05:07Z **Delivery plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host **Machines** - mesh-host on ace: passed — healthy 3 times over 8m59s - mesh-host on the rest running it: sent **Transitions** - 2026-10-07 00:45 (new) → proposed (announced): novox/mesh-host#48's head announced - 2026-10-07 00:48 proposed → checked (checked): the verdict names this commit - 2026-10-07 00:48 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-07 00:55 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-07 00:55 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-07 01:05 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
mesh-admin commented 2026-10-07 00:35:48 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 4b23bed3

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
  • mesh/repo-check: PASS — its merge-check.sh passed

Change plan — builds mesh-host → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-host
  • ace: receives mesh-host
  • g14: receives mesh-host
  • novox: receives mesh-host
  • shanks: receives mesh-host

Run by the build seat on g14 as build-1791332936840909516 (builds --log build-1791332936840909516).

--- the gate: mesh-host
mesh-host: ok
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T00:30:23Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-07T00:30:23Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
--- its merge-check.sh (go toolchain)
go: downloading golang.org/x/crypto v0.57.0
go: downloading github.com/nats-io/nats.go v1.54.0
go: downloading github.com/nats-io/nuid v1.0.1
go: downloading github.com/klauspost/compress v1.20.0
go: downloading github.com/nats-io/nkeys v0.4.16
go: downloading golang.org/x/sys v0.48.0
ok  	github.com/novox/mesh-host/cmd/mesh-bootstrap	1.009s
ok  	github.com/novox/mesh-host/cmd/mesh-host	1.242s
ok  	github.com/novox/mesh-host/examples	1.013s
ok  	github.com/novox/mesh-host/internal/apply	1.691s
ok  	github.com/novox/mesh-host/internal/bootstrap	1.328s
ok  	github.com/novox/mesh-host/internal/bundle	1.012s
ok  	github.com/novox/mesh-host/internal/declaration	1.047s
ok  	github.com/novox/mesh-host/internal/firewall	1.734s
ok  	github.com/novox/mesh-host/internal/identity	1.048s
ok  	github.com/novox/mesh-host/internal/image	1.011s
ok  	github.com/novox/mesh-host/internal/inventory	1.026s
ok  	github.com/novox/mesh-host/internal/link	10.366s
ok  	github.com/novox/mesh-host/internal/outward	1.029s
ok  	github.com/novox/mesh-host/internal/profile	1.498s
ok  	github.com/novox/mesh-host/internal/reachable	1.029s
ok  	github.com/novox/mesh-host/internal/store	1.099s
ok  	github.com/novox/mesh-host/internal/system	1.030s
ok  	github.com/novox/mesh-host/internal/tunnel	1.032s
ok  	github.com/novox/mesh-host/internal/upgrade	1.050s
ok  	github.com/novox/mesh-host/internal/witness	2.071s
ok  	github.com/novox/mesh-host/packaging	1.008s
ok  	github.com/novox/mesh-host/validate	1.007s
**Merge check** at `4b23bed3` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Change plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host Run by the build seat on g14 as `build-1791332936840909516` (`builds --log build-1791332936840909516`). ``` --- the gate: mesh-host mesh-host: ok 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T00:30:23Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-07T00:30:23Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): mesh-host --- its merge-check.sh (go toolchain) go: downloading golang.org/x/crypto v0.57.0 go: downloading github.com/nats-io/nats.go v1.54.0 go: downloading github.com/nats-io/nuid v1.0.1 go: downloading github.com/klauspost/compress v1.20.0 go: downloading github.com/nats-io/nkeys v0.4.16 go: downloading golang.org/x/sys v0.48.0 ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.009s ok github.com/novox/mesh-host/cmd/mesh-host 1.242s ok github.com/novox/mesh-host/examples 1.013s ok github.com/novox/mesh-host/internal/apply 1.691s ok github.com/novox/mesh-host/internal/bootstrap 1.328s ok github.com/novox/mesh-host/internal/bundle 1.012s ok github.com/novox/mesh-host/internal/declaration 1.047s ok github.com/novox/mesh-host/internal/firewall 1.734s ok github.com/novox/mesh-host/internal/identity 1.048s ok github.com/novox/mesh-host/internal/image 1.011s ok github.com/novox/mesh-host/internal/inventory 1.026s ok github.com/novox/mesh-host/internal/link 10.366s ok github.com/novox/mesh-host/internal/outward 1.029s ok github.com/novox/mesh-host/internal/profile 1.498s ok github.com/novox/mesh-host/internal/reachable 1.029s ok github.com/novox/mesh-host/internal/store 1.099s ok github.com/novox/mesh-host/internal/system 1.030s ok github.com/novox/mesh-host/internal/tunnel 1.032s ok github.com/novox/mesh-host/internal/upgrade 1.050s ok github.com/novox/mesh-host/internal/witness 2.071s ok github.com/novox/mesh-host/packaging 1.008s ok github.com/novox/mesh-host/validate 1.007s ```
jschoubben added 1 commit 2026-10-07 00:45:39 +00:00
Keep a replaced build reachable, and delete it only once no process runs from it (hq issue 289)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
Details
mesh/repo-check pass: its merge-check.sh passed
Details
mesh/delivery delivered
Details
3a117c2d2b 
The witness moved a running controller's build into a 0700 directory and deleted it
on proof, while the old process could still be serving. Its directories are now
0711, and a build without a reader is retired and swept once /proc shows nothing
runs from it.
jschoubben force-pushed fix/a-verb-survives-the-handover from 4b23bed325 to 3a117c2d2b 2026-10-07 00:45:39 +00:00 Compare
mesh-admin commented 2026-10-07 00:48:01 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 3a117c2d

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
  • mesh/repo-check: PASS — its merge-check.sh passed

Change plan — builds mesh-host → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-host
  • ace: receives mesh-host
  • g14: receives mesh-host
  • novox: receives mesh-host
  • shanks: receives mesh-host

Run by the build seat on ace as build-1791333959468090814 (builds --log build-1791333959468090814).

--- the gate: mesh-host
mesh-host: ok
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T00:39:23Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-07T00:39:23Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-host/cmd/mesh-bootstrap	1.014s
ok  	github.com/novox/mesh-host/cmd/mesh-host	1.302s
ok  	github.com/novox/mesh-host/examples	1.018s
ok  	github.com/novox/mesh-host/internal/apply	3.981s
ok  	github.com/novox/mesh-host/internal/bootstrap	1.515s
ok  	github.com/novox/mesh-host/internal/bundle	1.033s
ok  	github.com/novox/mesh-host/internal/declaration	1.060s
ok  	github.com/novox/mesh-host/internal/firewall	1.637s
ok  	github.com/novox/mesh-host/internal/identity	1.105s
ok  	github.com/novox/mesh-host/internal/image	1.027s
ok  	github.com/novox/mesh-host/internal/inventory	1.033s
ok  	github.com/novox/mesh-host/internal/link	10.331s
ok  	github.com/novox/mesh-host/internal/liveness	1.057s
ok  	github.com/novox/mesh-host/internal/outward	1.038s
ok  	github.com/novox/mesh-host/internal/profile	1.498s
ok  	github.com/novox/mesh-host/internal/reachable	1.035s
ok  	github.com/novox/mesh-host/internal/store	1.222s
ok  	github.com/novox/mesh-host/internal/system	1.039s
ok  	github.com/novox/mesh-host/internal/tunnel	1.047s
ok  	github.com/novox/mesh-host/internal/upgrade	1.148s
ok  	github.com/novox/mesh-host/internal/witness	4.556s
ok  	github.com/novox/mesh-host/packaging	1.032s
ok  	github.com/novox/mesh-host/validate	1.032s
**Merge check** at `3a117c2d` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Change plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host Run by the build seat on ace as `build-1791333959468090814` (`builds --log build-1791333959468090814`). ``` --- the gate: mesh-host mesh-host: ok 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T00:39:23Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-07T00:39:23Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): mesh-host --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.014s ok github.com/novox/mesh-host/cmd/mesh-host 1.302s ok github.com/novox/mesh-host/examples 1.018s ok github.com/novox/mesh-host/internal/apply 3.981s ok github.com/novox/mesh-host/internal/bootstrap 1.515s ok github.com/novox/mesh-host/internal/bundle 1.033s ok github.com/novox/mesh-host/internal/declaration 1.060s ok github.com/novox/mesh-host/internal/firewall 1.637s ok github.com/novox/mesh-host/internal/identity 1.105s ok github.com/novox/mesh-host/internal/image 1.027s ok github.com/novox/mesh-host/internal/inventory 1.033s ok github.com/novox/mesh-host/internal/link 10.331s ok github.com/novox/mesh-host/internal/liveness 1.057s ok github.com/novox/mesh-host/internal/outward 1.038s ok github.com/novox/mesh-host/internal/profile 1.498s ok github.com/novox/mesh-host/internal/reachable 1.035s ok github.com/novox/mesh-host/internal/store 1.222s ok github.com/novox/mesh-host/internal/system 1.039s ok github.com/novox/mesh-host/internal/tunnel 1.047s ok github.com/novox/mesh-host/internal/upgrade 1.148s ok github.com/novox/mesh-host/internal/witness 4.556s ok github.com/novox/mesh-host/packaging 1.032s ok github.com/novox/mesh-host/validate 1.032s ```
mesh-admin merged commit 038eff5ca0 into main 2026-10-07 00:55:24 +00:00
mesh-admin deleted branch fix/a-verb-survives-the-handover 2026-10-07 00:55:24 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
mesh/ready
Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-host#48