mounts: never leave a failed mount unit when a source is away, or the failure of a unit it takes over (hq ADR 0263 rules 4 and 6) #162

Merged
mesh-admin merged 3 commits from fix/mounts-leaves-no-failed-unit into main 2026-10-09 12:55:38 +00:00
Contributor

Found while planning the workstation's library and gaming-device mounts (2026-10-09).

Symptom it prevents. The node-engine reports a failed unit as a module's only when the node-engine itself placed the unit (a declared unit, file or package). The mounts module's apply process writes its unit pairs itself, so a failed .mount of the module's is said as a failed unit of the machine's own (condition machine.<node>.units), with a notification every time it re-fails. Two cases hit this:

  1. Takeover. The fstab generator's unit for the same mount point (e.g. storage-media.mount) has already failed. The module's unit of the same name inherits that failed state; judge then says the share failed and, five minutes later, unhealthy, although the module's pair never mounted at all.
  2. A source that is away. Opening the mount point of an occasional source that is off (the gaming device), or of a share whose server is away, makes the .mount fail after its timeout and stay failed. ADR 0263 rule 6 promises an occasional source "never leaves a failed unit".

Change. apply (the acting process only):

  • clears (systemctl reset-failed) a mount unit it has just written anew over one that had failed;
  • every 5 s between its 30 s passes, clears a failed mount of its own pairs whose source does not answer (TCP 2049 / 445, or the disk's device). 5 s is under the node-engine's two looks of 15 s.
  • A mount that failed while its source answers is left failed: that is a fault, and the share stays failed.

The module's own states are unchanged: absent for an occasional source, unreachable for a share (and its health after five minutes, rule 8).

Tests: failed_test.go (takeover cleared; away source cleared and still said absent/unreachable; answering source left failed; the judging process changes nothing). MESH_CHECK_MODULES=mounts sh merge-check.sh passes with -race locally.

Follow-up not in this PR: the node-engine could attribute a unit file carrying a module's marker to that module instead (a design question for hq).

Not merged: waits for a person's approval.

Found while planning the workstation's library and gaming-device mounts (2026-10-09). **Symptom it prevents.** The node-engine reports a failed unit as a module's only when the node-engine itself placed the unit (a declared unit, file or package). The mounts module's `apply` process writes its unit pairs itself, so a failed `.mount` of the module's is said as a failed unit *of the machine's own* (condition `machine.<node>.units`), with a notification every time it re-fails. Two cases hit this: 1. **Takeover.** The fstab generator's unit for the same mount point (e.g. `storage-media.mount`) has already failed. The module's unit of the same name inherits that failed state; `judge` then says the share `failed` and, five minutes later, unhealthy, although the module's pair never mounted at all. 2. **A source that is away.** Opening the mount point of an occasional source that is off (the gaming device), or of a share whose server is away, makes the `.mount` fail after its timeout and stay failed. ADR 0263 rule 6 promises an occasional source "never leaves a failed unit". **Change.** `apply` (the acting process only): - clears (`systemctl reset-failed`) a mount unit it has just written anew over one that had failed; - every 5 s between its 30 s passes, clears a failed mount of its own pairs whose source does not answer (TCP 2049 / 445, or the disk's device). 5 s is under the node-engine's two looks of 15 s. - A mount that failed while its source answers is left failed: that is a fault, and the share stays `failed`. The module's own states are unchanged: `absent` for an occasional source, `unreachable` for a share (and its health after five minutes, rule 8). Tests: `failed_test.go` (takeover cleared; away source cleared and still said absent/unreachable; answering source left failed; the judging process changes nothing). `MESH_CHECK_MODULES=mounts sh merge-check.sh` passes with `-race` locally. Follow-up not in this PR: the node-engine could attribute a unit file carrying a module's marker to that module instead (a design question for hq). Not merged: waits for a person's approval.
mesh-admin added 1 commit 2026-10-09 12:26:32 +00:00
mounts: clear a failed mount of its own whose source was away, and the failure of a unit it takes over, so neither is said as the machine's own failed unit (hq ADR 0263 rules 4 and 6)
mesh/merge-gate pass: builds mounts, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
0f330fc57d
Author
Contributor

Delivery novox/mesh-catalog@6d3cb73e3b28 — delivered since 2026-10-09T12:56:46Z

Delivery plan — builds mounts, sent nowhere; no bus step

  • tier 0: mounts

Machines

  • mounts on every machine running it: sent

Transitions

  • 2026-10-09 12:43 (new) → proposed (announced): novox/mesh-catalog#162's head announced
  • 2026-10-09 12:45 proposed → checked (checked): the verdict names this commit
  • 2026-10-09 12:45 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-09 12:56 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-09 12:56 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
  • 2026-10-09 12:56 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-catalog@6d3cb73e3b28` — **delivered** since 2026-10-09T12:56:46Z **Delivery plan** — builds mounts, sent nowhere; no bus step - tier 0: mounts **Machines** - mounts on every machine running it: sent **Transitions** - 2026-10-09 12:43 (new) → proposed (announced): novox/mesh-catalog#162's head announced - 2026-10-09 12:45 proposed → checked (checked): the verdict names this commit - 2026-10-09 12:45 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-09 12:56 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-09 12:56 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-09 12:56 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
Author
Contributor

Merge check at 0f330fc5

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mounts)
  • mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds mounts, sent nowhere; no bus step

  • tier 0: mounts

Run by the build seat on novox as build-1791548818713439328 (builds --log build-1791548818713439328).

--- the gate: mounts
mounts: ok, 3 tool(s), invokes seat:mesh-controller.data, keeps state shares, reads nfs-server.exports, ready: apply by unit every 30s; watch by its tool mounts_health every 1m0s; WARNING: config ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339)
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:24:01Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T12:24:01Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): mounts
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.002s
words: 12 retired words, none in the 35162 strings of 614 files
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.007s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 70, as kept
testing modules/mounts
ok  	mounts/cmd/mounts	1.154s
--- its merge-check-typescript.sh (typescript toolchain)
**Merge check** at `0f330fc5` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mounts) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds mounts, sent nowhere; no bus step - tier 0: mounts Run by the build seat on novox as `build-1791548818713439328` (`builds --log build-1791548818713439328`). ``` --- the gate: mounts mounts: ok, 3 tool(s), invokes seat:mesh-controller.data, keeps state shares, reads nfs-server.exports, ready: apply by unit every 30s; watch by its tool mounts_health every 1m0s; WARNING: config ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339) long-running resources without health: 0 files asking for a setting without saying whether it is trusted: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:24:01Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-09T12:24:01Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): mounts --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.002s words: 12 retired words, none in the 35162 strings of 614 files ok git.novox.be/novox/mesh-catalog/checks/paths 0.007s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 70, as kept testing modules/mounts ok mounts/cmd/mounts 1.154s --- its merge-check-typescript.sh (typescript toolchain) ```
jschoubben added 1 commit 2026-10-09 12:43:18 +00:00
jschoubben added 1 commit 2026-10-09 12:43:27 +00:00
mounts: gofmt the clearing tests
mesh/merge-gate pass: builds mounts, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
6d3cb73e3b
Author
Contributor

Merge check at 6d3cb73e

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mounts)
  • mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds mounts, sent nowhere; no bus step

  • tier 0: mounts

Run by the build seat on shanks as build-1791549820931617139 (builds --log build-1791549820931617139).

--- the gate: mounts
mounts: ok, 3 tool(s), invokes seat:mesh-controller.data, keeps state shares, reads nfs-server.exports, ready: apply by unit every 30s; watch by its tool mounts_health every 1m0s; WARNING: config ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339)
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:43:20Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T12:43:20Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): mounts
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.002s
words: 12 retired words, none in the 35176 strings of 614 files
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.006s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 70, as kept
testing modules/mounts
ok  	mounts/cmd/mounts	1.124s
--- its merge-check-typescript.sh (typescript toolchain)
**Merge check** at `6d3cb73e` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mounts) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds mounts, sent nowhere; no bus step - tier 0: mounts Run by the build seat on shanks as `build-1791549820931617139` (`builds --log build-1791549820931617139`). ``` --- the gate: mounts mounts: ok, 3 tool(s), invokes seat:mesh-controller.data, keeps state shares, reads nfs-server.exports, ready: apply by unit every 30s; watch by its tool mounts_health every 1m0s; WARNING: config ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339) long-running resources without health: 0 files asking for a setting without saying whether it is trusted: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:43:20Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-09T12:43:20Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): mounts --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.002s words: 12 retired words, none in the 35176 strings of 614 files ok git.novox.be/novox/mesh-catalog/checks/paths 0.006s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 70, as kept testing modules/mounts ok mounts/cmd/mounts 1.124s --- its merge-check-typescript.sh (typescript toolchain) ```
mesh-admin merged commit 26d6af3589 into main 2026-10-09 12:55:38 +00:00
mesh-admin deleted branch fix/mounts-leaves-no-failed-unit 2026-10-09 12:55:38 +00:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-catalog#162