A version that fails on one machine while another runs it healthy is that machine's failure, and goes no further meanwhile (issue 1496, ADR 0557 §1-§3) #81

2026-10-26T21:37:16Z

mesh-tools commented

2026-10-26 21:37:16 +00:00

Issue 1496, ADR 0557 §1 to §3 (accepted, the operator's Approve), in mesh-delivery-assigner:

A failed health of F holds F only when another node's have names F failed too, or no other online node (by presence) runs F healthy; otherwise no hold and no set-back, the failing node keeps its want, and the condition mesh-delivery-assigner.failed-alone.<node>.<module> (warning) names the node, the module, F, the have's words and the healthy nodes, raised once and cleared once (also when the node leaves, and for every module and node on the first pass after a start).
While a node fails F alone, a node that neither wants nor runs F gets no new want for it; wants already written and nodes already running F are untouched (the issue-1945 recovery path).
A second failure, or no healthy online witness, holds and sets back as ADR 0429 §6 has it.
Member of the delivery group fix/1496-start-health.

Reviewed in four rounds (online witness, clear after leave and restart, the second-failure hold, the no-further-node rule and its exemption for a node already running F).

Tests: rows in TestAFailedHealthHoldsAndSetsBack and TestThePassGathersTheModulesOnNeitherList, no new test function.

Reviewed-by: review-start@shanks at f007ca4227

Issue 1496, ADR 0557 §1 to §3 (accepted, the operator's Approve), in mesh-delivery-assigner: - A failed health of F holds F only when another node's have names F failed too, or no other online node (by presence) runs F healthy; otherwise no hold and no set-back, the failing node keeps its want, and the condition `mesh-delivery-assigner.failed-alone.<node>.<module>` (warning) names the node, the module, F, the have's words and the healthy nodes, raised once and cleared once (also when the node leaves, and for every module and node on the first pass after a start). - While a node fails F alone, a node that neither wants nor runs F gets no new want for it; wants already written and nodes already running F are untouched (the issue-1945 recovery path). - A second failure, or no healthy online witness, holds and sets back as ADR 0429 §6 has it. - Member of the delivery group fix/1496-start-health. Reviewed in four rounds (online witness, clear after leave and restart, the second-failure hold, the no-further-node rule and its exemption for a node already running F). Tests: rows in TestAFailedHealthHoldsAndSetsBack and TestThePassGathersTheModulesOnNeitherList, no new test function. Reviewed-by: review-start@shanks at f007ca42279a77058d59ca3ef106a160a2cb1cc8

mesh-tools added 5 commits 2026-10-26 21:37:17 +00:00

A version healthy on another node is not held for one node's failed health check (ADR 0557, issue 1496) 2ee7589474

On ace, 2026-10-26 21:19, a newly assigned mesh-build-node failed its first
health check at 8a98386f (later 31be2f24) while shanks, g14 and novox had run it
healthy for an hour. Being first by name, ace was the canary, and the assigner
held the version and moved the three healthy nodes back.

A failed health check of the newest fingerprint now holds it only when no other
node's have names it healthy (ADR 0557 §2 keeps ADR 0429 §6's hold and set-back
for that case). Otherwise the failure is that node's (ADR 0557 §1): nothing is
held or set back, the node keeps its want, and the assigner raises the node's
condition once (failed-alone.<node>.<module>), naming the node, the module, the
fingerprint and the have's words, and clears it once when it ends; the first
pass after a start clears what a previous process may have raised.

Only an online healthy witness, and only a first failure, keep a version from being held; failed-alone clears when a node leaves (ADR 0557, issue 1496) 16e4e61a0f

Review of 2ee7589: a node that sleeps after running F healthy kept its stale
have as a witness, so F stayed unheld while it failed on the online nodes; and a
failed-alone condition was never cleared once its node left the module.

A witness must now be online by the watchman's presence. F failing on a second
node is no longer one node's failure: the hold and the set-back stand (ADR 0557
version 6, §2). A remembered failed-alone key whose node is no longer one of the
module's nodes is cleared once; the first pass after a start clears it for every
node the presence bucket knows, for a module with a failed have.

The first pass after a start clears failed-alone for every module and known node (ADR 0557, issue 1496) 92aedb9365

Review round 2 of 16e4e61: gating the first-pass clear on a failed have left a
condition an earlier process raised standing for good once its node recovered
while no assigner watched. The first pass now clears it for every module and
every node the presence bucket knows, a quiet clear at the router for one that
is not open.

While a node fails a version alone, no further node gets a want for it (ADR 0557 §2, issue 1496) 50c4674dfa

ADR 0557 version 9: a failure that is one node's holds nothing and sets nothing
back, but the version reaches no node that does not want it yet until it is
over. A node whose want does not name F gets no new want naming it (rule 2, in
words naming the failing nodes); wants already written stay.

A node that runs a version alone-failed elsewhere still gets its want at it (ADR 0557 §2, issue 1496)

mesh/delivery merged: source.main moved to novox/mesh-delivery@bd107d20154c

Details

mesh/delivery-group group fix/1496-start-health merged: every member is merged

Details

mesh/check checking f007ca42279a on 4b36cc29a6ae | novox/mesh-delivery#81 in group fix/1496-start-health

f007ca4227

Review of 50c4674: the bar on new wants naming F while a node fails it alone
also caught a node that runs F with no live want, the issue-1945 recovery path:
it went on neither list and its node-engine gave the module up. The bar now
holds only a node that neither wants nor runs F.

mesh-tools commented

2026-10-26 21:37:25 +00:00

Delivery novox/mesh-delivery@f007ca42279a — merged since 2026-10-26T21:38:00Z

Source — source.main moved to novox/mesh-delivery@bd107d2015. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Group fix/1496-start-health: novox/mesh-delivery@f007ca4227, novox/mesh-host@b225c7bb4c, novox/mesh-tools@8cdb7c3032

Transitions

2026-10-26 21:37 (new) → proposed (announced): novox/mesh-delivery#81's head announced
2026-10-26 21:37 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-26 21:38 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-delivery@f007ca42279a` — **merged** since 2026-10-26T21:38:00Z **Source** — source.main moved to novox/mesh-delivery@bd107d20154c. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Group** `fix/1496-start-health`: novox/mesh-delivery@f007ca42279a, novox/mesh-host@b225c7bb4cf5, novox/mesh-tools@8cdb7c303210 **Transitions** - 2026-10-26 21:37 (new) → proposed (announced): novox/mesh-delivery#81's head announced - 2026-10-26 21:37 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-26 21:38 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit bd107d2015 into main

2026-10-26 21:37:59 +00:00

mesh-tools deleted branch fix/1496-start-health

2026-10-26 21:37:59 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-delivery#81