novox/mesh-controller
Archived
9
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Raise a failed unit, the module's and the machine's (hq issue 315) #137

Merged
mesh-admin merged 1 commits from fix/315-a-failed-unit-is-a-condition into main 2026-10-08 09:40:24 +00:00
Conversation 2 Commits 1 Files Changed 8
+382 -8
mesh-admin commented 2026-10-08 09:29:23 +00:00
Contributor
Copy Link
Copy Source

hq issue 315: a failed systemd unit belonging to a mesh module was never raised, and a degraded service manager hid silently.

What changes

  • link.Health.Units (state, failed[], unread[]) and link.KindUnit, the node-engine's (mesh-host same-named branch).
  • node_health.units jsonb (migration 0080-a-failed-unit-is-never-silent.sql), kept and read with the machine's newest statement.
  • A module's failed unit arrives among its resources (kind unit) and raises module.<m>.<machine>.unhealthy on the second statement as any unhealthy resource; its summary now names the unit (its unit openrazer-daemon.service failed in the account's own service manager (exit-code)), the placing resource is evidence.
  • The machine's own failed units — no module places them — raise one warning machine.<m>.units, listing each unit (summary) and how/since (evidence); cleared on the first statement that lists none. A statement with no units (older engine) raises and clears nothing.
  • The gate never holds a send on machine-units: nothing a send moved is among them (a module's failed unit is its own condition, which the gate already holds).
  • node show lists the service managers' state and each failed unit with its owner.

Tests: cmd/mesh-controller/machine_units_test.go through the store and keeper (the workstation's four failures as found 2026-10-08), and the gate. Full suite run locally against a throwaway Postgres (go test -race ./...): green; the operator-channel content rule (TestMain) holds for the new summaries.

Pairs with novox/mesh-host fix/315-a-failed-unit-is-a-condition; either order is safe.

hq issue 315: a failed systemd unit belonging to a mesh module was never raised, and a degraded service manager hid silently. **What changes** - `link.Health.Units` (`state`, `failed[]`, `unread[]`) and `link.KindUnit`, the node-engine's (mesh-host same-named branch). - `node_health.units` jsonb (migration `0080-a-failed-unit-is-never-silent.sql`), kept and read with the machine's newest statement. - A module's failed unit arrives among its resources (kind `unit`) and raises `module.<m>.<machine>.unhealthy` on the second statement as any unhealthy resource; its summary now names the unit (`its unit openrazer-daemon.service failed in the account's own service manager (exit-code)`), the placing resource is evidence. - The machine's own failed units — no module places them — raise one warning `machine.<m>.units`, listing each unit (summary) and how/since (evidence); cleared on the first statement that lists none. A statement with no `units` (older engine) raises and clears nothing. - The gate never holds a send on `machine-units`: nothing a send moved is among them (a module's failed unit is its own condition, which the gate already holds). - `node show` lists the service managers' state and each failed unit with its owner. **Tests**: `cmd/mesh-controller/machine_units_test.go` through the store and keeper (the workstation's four failures as found 2026-10-08), and the gate. Full suite run locally against a throwaway Postgres (`go test -race ./...`): green; the operator-channel content rule (TestMain) holds for the new summaries. Pairs with novox/mesh-host fix/315-a-failed-unit-is-a-condition; either order is safe.
mesh-admin added 1 commit 2026-10-08 09:29:23 +00:00
Raise a failed unit, the module's and the machine's (hq issue 315)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
Details
mesh/repo-check pass: its merge-check.sh passed
Details
mesh/delivery delivered
Details
mesh/delivery-group group fix/315-a-failed-unit-is-a-condition merged: every member is merged
Details
6c98e7ea51 
A module's failed unit now arrives among its resources and raises the
module's own condition, named by the unit. The machine's own failed
units, which no module places, are one warning for the machine listing
them, cleared when none is listed. Nothing a send moved is among them,
so the gate never holds a send on that finding. The statement's units
are kept with the machine's health (migration 0080) and node show says
them.
mesh-admin commented 2026-10-08 09:29:53 +00:00
Author
Contributor
Copy Link
Copy Source

Delivery novox/mesh-controller@6c98e7ea5103 — delivered since 2026-10-08T09:57:54Z

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Group fix/315-a-failed-unit-is-a-condition, in order: novox/mesh-controller@6c98e7ea51 → novox/mesh-host@14e5d91c9a

Machines

  • mesh-controller on novox: passed — healthy 3 times over 3m19s
  • mesh-controller on the rest running it: sent
  • build-agent on ace: passed — healthy 3 times over 3m30s
  • build-agent on the rest running it: sent
  • route-proxy on ace: passed — healthy 3 times over 3m22s
  • route-proxy on the rest running it: sent

Transitions

  • 2026-10-08 09:29 (new) → proposed (announced): novox/mesh-controller#137's head announced
  • 2026-10-08 09:34 proposed → checked (checked): the verdict names this commit
  • 2026-10-08 09:34 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-08 09:40 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-08 09:40 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
  • 2026-10-08 09:57 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-controller@6c98e7ea5103` — **delivered** since 2026-10-08T09:57:54Z **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves **Group** `fix/315-a-failed-unit-is-a-condition`, in order: novox/mesh-controller@6c98e7ea5103 → novox/mesh-host@14e5d91c9a2e - novox/mesh-controller@6c98e7ea5103 before novox/mesh-host@14e5d91c9a2e: built by, over engine before controller - composed together: pass — every machine composes with the change as it did without (4 of 4 compose) **Machines** - mesh-controller on novox: passed — healthy 3 times over 3m19s - mesh-controller on the rest running it: sent - build-agent on ace: passed — healthy 3 times over 3m30s - build-agent on the rest running it: sent - route-proxy on ace: passed — healthy 3 times over 3m22s - route-proxy on the rest running it: sent **Transitions** - 2026-10-08 09:29 (new) → proposed (announced): novox/mesh-controller#137's head announced - 2026-10-08 09:34 proposed → checked (checked): the verdict names this commit - 2026-10-08 09:34 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-08 09:40 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-08 09:40 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-08 09:57 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
mesh-admin commented 2026-10-08 09:34:24 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 6c98e7ea

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
  • mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on g14 as build-1791451784228554858 (builds --log build-1791451784228554858).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 47 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T09:22:44Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-08T09:22:44Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.037s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	153.282s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.016s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.022s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.011s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.199s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.046s
ok  	github.com/novox/mesh-controller/internal/broker	2.948s
ok  	github.com/novox/mesh-controller/internal/builder	1.755s
ok  	github.com/novox/mesh-controller/internal/catalogue	4.514s
ok  	github.com/novox/mesh-controller/internal/conditions	1.967s
ok  	github.com/novox/mesh-controller/internal/envfile	1.033s
ok  	github.com/novox/mesh-controller/internal/facts	1.038s
ok  	github.com/novox/mesh-controller/internal/identity	4.620s
ok  	github.com/novox/mesh-controller/internal/inventory	69.468s
ok  	github.com/novox/mesh-controller/internal/lease	17.698s
ok  	github.com/novox/mesh-controller/internal/licences	4.209s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.019s
ok  	github.com/novox/mesh-controller/internal/link	43.194s
ok  	github.com/novox/mesh-controller/internal/lint	2.240s
ok  	github.com/novox/mesh-controller/internal/outward	1.069s
ok  	github.com/novox/mesh-controller/internal/overlay	1.061s
ok  	github.com/novox/mesh-controller/internal/secrets	1.727s
ok  	github.com/novox/mesh-controller/internal/store	3.490s
ok  	github.com/novox/mesh-controller/internal/testbus	1.170s
ok  	github.com/novox/mesh-controller/internal/token	1.017s
**Merge check** at `6c98e7ea` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on g14 as `build-1791451784228554858` (`builds --log build-1791451784228554858`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 47 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T09:22:44Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-08T09:22:44Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.037s ok github.com/novox/mesh-controller/cmd/mesh-controller 153.282s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.016s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.022s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.011s ok github.com/novox/mesh-controller/examples/route-proxy 1.199s ok github.com/novox/mesh-controller/internal/artifacts 1.046s ok github.com/novox/mesh-controller/internal/broker 2.948s ok github.com/novox/mesh-controller/internal/builder 1.755s ok github.com/novox/mesh-controller/internal/catalogue 4.514s ok github.com/novox/mesh-controller/internal/conditions 1.967s ok github.com/novox/mesh-controller/internal/envfile 1.033s ok github.com/novox/mesh-controller/internal/facts 1.038s ok github.com/novox/mesh-controller/internal/identity 4.620s ok github.com/novox/mesh-controller/internal/inventory 69.468s ok github.com/novox/mesh-controller/internal/lease 17.698s ok github.com/novox/mesh-controller/internal/licences 4.209s ok github.com/novox/mesh-controller/internal/licences/adapters 1.019s ok github.com/novox/mesh-controller/internal/link 43.194s ok github.com/novox/mesh-controller/internal/lint 2.240s ok github.com/novox/mesh-controller/internal/outward 1.069s ok github.com/novox/mesh-controller/internal/overlay 1.061s ok github.com/novox/mesh-controller/internal/secrets 1.727s ok github.com/novox/mesh-controller/internal/store 3.490s ok github.com/novox/mesh-controller/internal/testbus 1.170s ok github.com/novox/mesh-controller/internal/token 1.017s ```
mesh-admin merged commit 4d337ff897 into main 2026-10-08 09:40:24 +00:00
mesh-admin deleted branch fix/315-a-failed-unit-is-a-condition 2026-10-08 09:40:24 +00:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No Reviewers
Labels
Clear labels
mesh/ready
Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-controller#137