novox/mesh-host
9
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Judge whether what a module runs stays up, and say it (hq ADR 0240, to-be 48 Phase A) #47

Merged
mesh-admin merged 1 commits from feat/a-module-says-how-it-is-healthy into main 2026-10-07 00:39:40 +00:00
Conversation 3 Commits 1 Files Changed 11
+1346 -4
mesh-admin commented 2026-10-07 00:19:36 +00:00
Contributor
Copy Link
Copy Source

The gate passed a container that crash-looped after it applied. Nothing looked at what a module runs. The agent server restarted about a hundred times while the mesh read it applied (research 032).

The node-engine now judges liveness (internal/liveness):

  • What it judges: every long-running resource of a module, with no declaration. That is a container that stays up, a process that stays up, and a service stated running.
  • The states: healthy, unhealthy (down or restarting), starting (inside the 60 s grace), held (a maintenance window holds it; ADR 0189), and unknown (the runtime does not answer, which is never read as down).
  • Restarts: two counted within the 10-minute settle window after grace make it unhealthy. A restart inside grace is not counted.
  • The count is kept: in liveness.json beside the node's state, across recreates and across the engine's own restarts. The runtime's count is read only to see it move.
  • What is a new start: a recreate (new identity, count not moved) and a restart somebody made (start time moved, count not moved) are new starts from a fresh grace. The counted total is kept.
  • Cost: one look every 15 s. That is one container inspect of every container and one systemctl show per manager. It never runs anything inside a container.
  • It reads only (rule 6). Nothing is restarted, recreated or stopped for being unhealthy, and a test holds every command it runs to a read.

How it is said:

  • In every report: health: {contract, at, resources: [{module, resource, kind, target, state, reason, since, streak, restarts}]}. It is judged right after the apply, so whatever the apply started says starting.
  • As an event on mesh.control.<node>.health (core NATS, inside the existing mesh.control.<node>.> grant, so no grant changed). Sent on a change, again every minute while anything is not healthy, and every 5 minutes anyway. The queue's SayHealth uses the open link and does not wait behind an apply.

Compatibility: the new report field is additive. The controller decodes reports leniently, and an older controller ignores the field.

Tests (ADR 0240 rules 1 and 6): TestARestartInsideGraceIsNotCountedAndTwoAfterItAreUnhealthy, TestARecreatedContainerKeepsItsCountedRestartsAndStartsAgain, TestTheEngineRestartedKeepsWhatItCounted, TestAContainerHeldByAWindowIsHeldAndJudgedAfreshAfter, TestDownAfterGraceAndUnknownWhenNothingCouldBeRead, TestAUnitRestartedByItsManagerIsCountedAndOneStartedAgainIsNot, TestWhatIsLongRunning, TestAnUnhealthyContainerIsNeitherRestartedRecreatedNorStopped, TestOneLookIsOneReadOfEveryContainer, TestAHealthStatementIsSaidOnTheLinkOpenNow, plus the wire names. TestReplayCrashLoopIsSaidUnhealthy is the engine's half of mesh-lab's R-crashloop: a real container that exits at start is said unhealthy with 2 restarts counted in about 7 s at a 3 s grace. It is skipped unless the lab names a container or MESH_REPLAY_RUNTIME=1.

Checks:

  • merge-check.sh passes locally under -race.
  • check-here: mesh/merge-gate PASS (4 of 4 compose). mesh/repo-check fails only on internal/bootstrap/publish_test.go, which is the existing gofmt issue #46 fixes. With #46 merged in locally, repo-check PASSES. Merge #46 first.

Rolls out before the mesh-controller change. Feature a-module-says-how-it-is-healthy (same branch in mesh-controller, mesh-lab, hq).

**The gate passed a container that crash-looped after it applied.** Nothing looked at what a module runs. The agent server restarted about a hundred times while the mesh read it applied (research 032). The node-engine now judges liveness (`internal/liveness`): - **What it judges:** every long-running resource of a module, with no declaration. That is a container that stays up, a process that stays up, and a service stated `running`. - **The states:** `healthy`, `unhealthy` (`down` or `restarting`), `starting` (inside the 60 s grace), `held` (a maintenance window holds it; ADR 0189), and `unknown` (the runtime does not answer, which is never read as down). - **Restarts:** two counted within the 10-minute settle window after grace make it unhealthy. A restart inside grace is not counted. - **The count is kept:** in `liveness.json` beside the node's state, across recreates and across the engine's own restarts. The runtime's count is read only to see it move. - **What is a new start:** a recreate (new identity, count not moved) and a restart somebody made (start time moved, count not moved) are new starts from a fresh grace. The counted total is kept. - **Cost:** one look every 15 s. That is one `container inspect` of every container and one `systemctl show` per manager. It never runs anything inside a container. - **It reads only (rule 6).** Nothing is restarted, recreated or stopped for being unhealthy, and a test holds every command it runs to a read. How it is said: - **In every report:** `health: {contract, at, resources: [{module, resource, kind, target, state, reason, since, streak, restarts}]}`. It is judged right after the apply, so whatever the apply started says `starting`. - **As an event** on `mesh.control.<node>.health` (core NATS, inside the existing `mesh.control.<node>.>` grant, so no grant changed). Sent on a change, again every minute while anything is not healthy, and every 5 minutes anyway. The queue's `SayHealth` uses the open link and does not wait behind an apply. **Compatibility:** the new report field is additive. The controller decodes reports leniently, and an older controller ignores the field. **Tests** (ADR 0240 rules 1 and 6): `TestARestartInsideGraceIsNotCountedAndTwoAfterItAreUnhealthy`, `TestARecreatedContainerKeepsItsCountedRestartsAndStartsAgain`, `TestTheEngineRestartedKeepsWhatItCounted`, `TestAContainerHeldByAWindowIsHeldAndJudgedAfreshAfter`, `TestDownAfterGraceAndUnknownWhenNothingCouldBeRead`, `TestAUnitRestartedByItsManagerIsCountedAndOneStartedAgainIsNot`, `TestWhatIsLongRunning`, `TestAnUnhealthyContainerIsNeitherRestartedRecreatedNorStopped`, `TestOneLookIsOneReadOfEveryContainer`, `TestAHealthStatementIsSaidOnTheLinkOpenNow`, plus the wire names. `TestReplayCrashLoopIsSaidUnhealthy` is the engine's half of mesh-lab's R-crashloop: a real container that exits at start is said unhealthy with 2 restarts counted in about 7 s at a 3 s grace. It is skipped unless the lab names a container or `MESH_REPLAY_RUNTIME=1`. **Checks:** - `merge-check.sh` passes locally under `-race`. - `check-here`: `mesh/merge-gate` PASS (4 of 4 compose). `mesh/repo-check` fails only on `internal/bootstrap/publish_test.go`, which is the existing gofmt issue #46 fixes. With #46 merged in locally, repo-check PASSES. **Merge #46 first.** Rolls out before the mesh-controller change. Feature `a-module-says-how-it-is-healthy` (same branch in mesh-controller, mesh-lab, hq).
mesh-admin commented 2026-10-07 00:23:05 +00:00
Author
Contributor
Copy Link
Copy Source

Delivery novox/mesh-host@1fc1e74cc3c9 — delivered since 2026-10-07T00:44:37Z

Delivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-host
  • ace: receives mesh-host
  • g14: receives mesh-host
  • novox: receives mesh-host
  • shanks: receives mesh-host

Group feat/a-module-says-how-it-is-healthy, in order: novox/hq@dea72dc4fc → novox/mesh-lab@5e3ae7b835

Machines

  • mesh-host on ace: passed — healthy 3 times over 2m59s
  • mesh-host on the rest running it: sent

Transitions

  • 2026-10-07 00:28 (new) → proposed (announced): novox/mesh-host#47's head announced
  • 2026-10-07 00:35 proposed → checked (checked): the verdict names this commit
  • 2026-10-07 00:35 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-07 00:40 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-07 00:40 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
  • 2026-10-07 00:44 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-host@1fc1e74cc3c9` — **delivered** since 2026-10-07T00:44:37Z **Delivery plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host **Group** `feat/a-module-says-how-it-is-healthy`, in order: novox/hq@dea72dc4fc8c → novox/mesh-lab@5e3ae7b835b2 - novox/mesh-controller@725fcd977e75 before novox/mesh-host@1fc1e74cc3c9: built by - novox/mesh-host@1fc1e74cc3c9 before novox/mesh-controller@725fcd977e75: engine before controller **Machines** - mesh-host on ace: passed — healthy 3 times over 2m59s - mesh-host on the rest running it: sent **Transitions** - 2026-10-07 00:28 (new) → proposed (announced): novox/mesh-host#47's head announced - 2026-10-07 00:35 proposed → checked (checked): the verdict names this commit - 2026-10-07 00:35 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-07 00:40 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-07 00:40 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-07 00:44 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
mesh-admin commented 2026-10-07 00:24:33 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at fad17501

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
  • mesh/repo-check: FAIL — its merge-check.sh failed: not gofmt'd by the toolchain's gofmt: internal/bootstrap/publish_test.go

Change plan — builds mesh-host → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-host
  • ace: receives mesh-host
  • g14: receives mesh-host
  • novox: receives mesh-host
  • shanks: receives mesh-host

Run by the build seat on ace as build-1791332571082496755 (builds --log build-1791332571082496755).

--- the gate: mesh-host
mesh-host: ok
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T00:21:01Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-07T00:21:01Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
--- its merge-check.sh (go toolchain)
not gofmt'd:
internal/bootstrap/publish_test.go
**Merge check** at `fad17501` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host) - `mesh/repo-check`: **FAIL** — its merge-check.sh failed: not gofmt'd by the toolchain's gofmt: internal/bootstrap/publish_test.go **Change plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host Run by the build seat on ace as `build-1791332571082496755` (`builds --log build-1791332571082496755`). ``` --- the gate: mesh-host mesh-host: ok 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T00:21:01Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-07T00:21:01Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): mesh-host --- its merge-check.sh (go toolchain) not gofmt'd: internal/bootstrap/publish_test.go ```
jschoubben added 1 commit 2026-10-07 00:28:53 +00:00
Judge whether what a module runs stays up, and say it (hq ADR 0240, to-be 48 Phase A)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
Details
mesh/repo-check pass: its merge-check.sh passed
Details
mesh/delivery delivered
Details
mesh/delivery-group group feat/a-module-says-how-it-is-healthy merged: every member is merged
Details
1fc1e74cc3 
A container that crash-looped after its compose applied passed every check the
gate had: nothing looked at what a module runs. The node-engine now judges every
long-running resource on every look — one read of the runtime, one per service
manager — keeps the restarts it counts across recreates and its own restarts,
and says the state in every report and as an event on change, again every minute
while not healthy. It reads only; nothing is restarted for being unhealthy.
jschoubben force-pushed feat/a-module-says-how-it-is-healthy from fad17501f4 to 1fc1e74cc3 2026-10-07 00:28:53 +00:00 Compare
mesh-admin commented 2026-10-07 00:35:53 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 1fc1e74c

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
  • mesh/repo-check: PASS — its merge-check.sh passed

Change plan — builds mesh-host → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-host
  • ace: receives mesh-host
  • g14: receives mesh-host
  • novox: receives mesh-host
  • shanks: receives mesh-host

Run by the build seat on novox as build-1791332936221958733 (builds --log build-1791332936221958733).

--- the gate: mesh-host
mesh-host: ok
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T00:30:23Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-07T00:30:23Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-host/cmd/mesh-bootstrap	1.016s
ok  	github.com/novox/mesh-host/cmd/mesh-host	2.142s
ok  	github.com/novox/mesh-host/examples	1.021s
ok  	github.com/novox/mesh-host/internal/apply	40.854s
ok  	github.com/novox/mesh-host/internal/bootstrap	3.616s
ok  	github.com/novox/mesh-host/internal/bundle	1.016s
ok  	github.com/novox/mesh-host/internal/declaration	1.058s
ok  	github.com/novox/mesh-host/internal/firewall	1.854s
ok  	github.com/novox/mesh-host/internal/identity	1.669s
ok  	github.com/novox/mesh-host/internal/image	1.020s
ok  	github.com/novox/mesh-host/internal/inventory	1.021s
ok  	github.com/novox/mesh-host/internal/link	10.429s
ok  	github.com/novox/mesh-host/internal/liveness	1.482s
ok  	github.com/novox/mesh-host/internal/outward	1.028s
ok  	github.com/novox/mesh-host/internal/profile	1.484s
ok  	github.com/novox/mesh-host/internal/reachable	1.016s
ok  	github.com/novox/mesh-host/internal/store	2.514s
ok  	github.com/novox/mesh-host/internal/system	1.018s
ok  	github.com/novox/mesh-host/internal/tunnel	1.019s
ok  	github.com/novox/mesh-host/internal/upgrade	1.971s
ok  	github.com/novox/mesh-host/internal/witness	49.275s
ok  	github.com/novox/mesh-host/packaging	1.013s
ok  	github.com/novox/mesh-host/validate	1.013s
**Merge check** at `1fc1e74c` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Change plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host Run by the build seat on novox as `build-1791332936221958733` (`builds --log build-1791332936221958733`). ``` --- the gate: mesh-host mesh-host: ok 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T00:30:23Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-07T00:30:23Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): mesh-host --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.016s ok github.com/novox/mesh-host/cmd/mesh-host 2.142s ok github.com/novox/mesh-host/examples 1.021s ok github.com/novox/mesh-host/internal/apply 40.854s ok github.com/novox/mesh-host/internal/bootstrap 3.616s ok github.com/novox/mesh-host/internal/bundle 1.016s ok github.com/novox/mesh-host/internal/declaration 1.058s ok github.com/novox/mesh-host/internal/firewall 1.854s ok github.com/novox/mesh-host/internal/identity 1.669s ok github.com/novox/mesh-host/internal/image 1.020s ok github.com/novox/mesh-host/internal/inventory 1.021s ok github.com/novox/mesh-host/internal/link 10.429s ok github.com/novox/mesh-host/internal/liveness 1.482s ok github.com/novox/mesh-host/internal/outward 1.028s ok github.com/novox/mesh-host/internal/profile 1.484s ok github.com/novox/mesh-host/internal/reachable 1.016s ok github.com/novox/mesh-host/internal/store 2.514s ok github.com/novox/mesh-host/internal/system 1.018s ok github.com/novox/mesh-host/internal/tunnel 1.019s ok github.com/novox/mesh-host/internal/upgrade 1.971s ok github.com/novox/mesh-host/internal/witness 49.275s ok github.com/novox/mesh-host/packaging 1.013s ok github.com/novox/mesh-host/validate 1.013s ```
mesh-admin merged commit 03031a46dd into main 2026-10-07 00:39:40 +00:00
mesh-admin deleted branch feat/a-module-says-how-it-is-healthy 2026-10-07 00:39:40 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
mesh/ready
Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-host#47