Reconcile a module from the machine's facts at start and on a down edge (issue 1957, ADR 0568) #166

2026-10-28T17:13:49Z

mesh-tools commented

2026-10-28 17:13:49 +00:00

After a reboot a declared-running unit that failed to start stayed down for 90 minutes: the engine believed its running have and nothing raised an alarm.
At engine start the baseline now also reads the live unit and container state and applies the module once when a declared-running resource is not active.
The judge stays read-only; its look calls Agent.Reconcile once per down edge of a resource, never on a timer. A held module and a module with no want are left alone, and a held have keeps its hold at start.

Fixes issue 1957

Decision: ADR 0568 clauses 1, 2, 4 and the node side of 3 (a failed re-apply writes the have failed with the apply's own words; already so, now asserted).

Quick fix: count a down unit as missing in newlyMissing (about 30 minutes). Thorough fix: the engine start reads the machine's facts, and the judge's down edge calls the agent's reconcile in process. The thorough one was chosen because the kind recurs at every reboot of every node, hides a failure, and relates to issues 1802 and 1372.

Tests: they fail without the fix (checked by disabling it). They are folded into existing tests, so the counts are unchanged: 1262 unit and 6 integration. The integration test boots the agent on a real bus with a running have and a stopped unit (a fake systemctl) and requires the unit started once.

Clause 3's watchman and condition-keeper part (S21 row, needs on apply-failing) is milestone L4, issue 1952, and is not in this pull request.

Reviewed-by: rev-1957@shanks at bf667a602d
Pushed-by: fix-1957@shanks

After a reboot a declared-running unit that failed to start stayed down for 90 minutes: the engine believed its `running` have and nothing raised an alarm. At engine start the baseline now also reads the live unit and container state and applies the module once when a declared-running resource is not active. The judge stays read-only; its look calls `Agent.Reconcile` once per down edge of a resource, never on a timer. A held module and a module with no want are left alone, and a held have keeps its hold at start. Fixes issue 1957 Decision: ADR 0568 clauses 1, 2, 4 and the node side of 3 (a failed re-apply writes the have `failed` with the apply's own words; already so, now asserted). Quick fix: count a down unit as missing in `newlyMissing` (about 30 minutes). Thorough fix: the engine start reads the machine's facts, and the judge's down edge calls the agent's reconcile in process. The thorough one was chosen because the kind recurs at every reboot of every node, hides a failure, and relates to issues 1802 and 1372. Tests: they fail without the fix (checked by disabling it). They are folded into existing tests, so the counts are unchanged: 1262 unit and 6 integration. The integration test boots the agent on a real bus with a `running` have and a stopped unit (a fake systemctl) and requires the unit started once. Clause 3's watchman and condition-keeper part (S21 row, `needs` on apply-failing) is milestone L4, issue 1952, and is not in this pull request. Reviewed-by: rev-1957@shanks at bf667a602d211a202ad38da1ccb8f4e3d3266c2f Pushed-by: fix-1957@shanks

mesh-tools added 1 commit 2026-10-28 17:13:49 +00:00

Reconcile a module from the machine's facts at engine start and at each edge of a declared-running resource going down (issue 1957, ADR 0568)

mesh/delivery merged: source.main moved to novox/mesh-host@58b4f8b6bc07

Details

mesh/check being carried to bf667a602d21 on 58b4f8b6bc07 | from 9a8b3870bad0

bf667a602d

A reboot left the have running, so a unit that failed to start at boot was never started again and nothing said so. Baseline now asks the machine; the judge's look calls Agent.Reconcile once per down edge, leaving held and unwanted modules alone.

mesh-tools commented

2026-10-28 17:13:52 +00:00

Delivery novox/mesh-host@bf667a602d21 — merged since 2026-10-28T17:41:57Z

Source — source.main moved to novox/mesh-host@58b4f8b6bc. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-28 17:13 (new) → proposed (announced): novox/mesh-host#166's head announced
2026-10-28 17:13 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-28 17:41 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@bf667a602d21` — **merged** since 2026-10-28T17:41:57Z **Source** — source.main moved to novox/mesh-host@58b4f8b6bc07. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-28 17:13 (new) → proposed (announced): novox/mesh-host#166's head announced - 2026-10-28 17:13 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-28 17:41 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 58b4f8b6bc into main

2026-10-28 17:41:54 +00:00

mesh-tools deleted branch fix/1957-boot-reconcile

2026-10-28 17:41:56 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#166