The controller gives its lease back at the stop signal and says a takeover once, so a controller delivery raises no urgent alarm (issue 906, ADR 0558) #478

2026-10-26T23:57:10Z

mesh-tools commented

2026-10-26 23:57:10 +00:00

Issue 906 (notes 8-11). At every controller replacement on novox (five times in a day) the urgent condition core.controller.lease.lost was raised and stayed an hour. Cause, read in the journal of 10-26 22:02: the old controller did give its lease back and the successor took it the same second, but serveUnderTheLease's close hook recorded the epoch released only if l.Epoch() passed the gate, which had closed 3 s earlier; the successor's TookEpoch ended it expired and S12 raised it for advisoryQuiet.

Decision: ADR 0558 (accepted; amends ADR 0229 §5, §7; home to-be 45).

On SIGTERM/SIGINT the lease is given back first: the gate closes, a revision-guarded delete within 5 s; a second release gives nothing back twice.
The release writes its own delete marker naming the epoch it gives back (Mesh-Lease-Released); the record is made from the delete, not the gate.
A candidate reads the key's last message before taking it; TookEpoch ends as released only the epoch the marker names (a command's lease or a failed take names none), every other open epoch expired.
A lost or expired takeover is raised and cleared at once by the new holder; a lease given back says nothing; an earlier end that cannot be read is raised as urgent.

Research 072 (the controller, NATS-native) piece moved: the lease in 03-domains (the mesh-controller row) and 04-findings §2 (a watchdog reading a signal's age becomes an event: the takeover is said on the take, not observed for an hour).

Quick fix weighed: drop the gate check in the close hook (a few lines); not chosen: the release would still come last in a 10 s shutdown, the record would be judged from the store alone, and a real takeover would still alarm for an hour after the mesh recovered.

Tests: no new test function (mesh-controller is over its budget); lease_test handover/expiry compare the epoch given back; signals_test covers raise-then-clear and nothing said for a release; TestTwoControllersOneActs stops A and checks B holds within 1 s, A recorded released, no condition. Local: gofmt, go vet ./..., go vet -tags integration ./..., the untagged suite, internal/lease integration on the embedded bus (9 s) pass. TestTwoControllersOneActs needs MESH_TEST_POSTGRES and first runs in this check.

Live case: after delivery, the next controller replacement on novox raises no core.controller.lease.lost.

Reviewed-by: review-906r@shanks at 8f84731896

Issue 906 (notes 8-11). At every controller replacement on novox (five times in a day) the urgent condition core.controller.lease.lost was raised and stayed an hour. Cause, read in the journal of 10-26 22:02: the old controller did give its lease back and the successor took it the same second, but serveUnderTheLease's close hook recorded the epoch released only if l.Epoch() passed the gate, which had closed 3 s earlier; the successor's TookEpoch ended it expired and S12 raised it for advisoryQuiet. Decision: ADR 0558 (accepted; amends ADR 0229 §5, §7; home to-be 45). - On SIGTERM/SIGINT the lease is given back first: the gate closes, a revision-guarded delete within 5 s; a second release gives nothing back twice. - The release writes its own delete marker naming the epoch it gives back (Mesh-Lease-Released); the record is made from the delete, not the gate. - A candidate reads the key's last message before taking it; TookEpoch ends as released only the epoch the marker names (a command's lease or a failed take names none), every other open epoch expired. - A lost or expired takeover is raised and cleared at once by the new holder; a lease given back says nothing; an earlier end that cannot be read is raised as urgent. Research 072 (the controller, NATS-native) piece moved: the lease in 03-domains (the mesh-controller row) and 04-findings §2 (a watchdog reading a signal's age becomes an event: the takeover is said on the take, not observed for an hour). Quick fix weighed: drop the gate check in the close hook (a few lines); not chosen: the release would still come last in a 10 s shutdown, the record would be judged from the store alone, and a real takeover would still alarm for an hour after the mesh recovered. Tests: no new test function (mesh-controller is over its budget); lease_test handover/expiry compare the epoch given back; signals_test covers raise-then-clear and nothing said for a release; TestTwoControllersOneActs stops A and checks B holds within 1 s, A recorded released, no condition. Local: gofmt, go vet ./..., go vet -tags integration ./..., the untagged suite, internal/lease integration on the embedded bus (9 s) pass. TestTwoControllersOneActs needs MESH_TEST_POSTGRES and first runs in this check. Live case: after delivery, the next controller replacement on novox raises no core.controller.lease.lost. Reviewed-by: review-906r@shanks at 8f847318963c9ab56a5bd0117fb9629615d10f26

mesh-tools added 2 commits 2026-10-26 23:57:10 +00:00

Give the controller lease back at the stop signal and say a takeover once, so a controller delivery raises no urgent alarm (issue 906, ADR 0558) fbaa463a38

The 2026-10-26 replacements showed the lease was given back, but last: the
shutdown took 10 s, the gate had closed at half the key's age, and the release
recorded the epoch released only through the gate, so the next holder found
it expired and S12 said it, urgent, for an hour.

Now the stop signal gives the key back first and records the epoch released
whenever the delete succeeded; a candidate reads the key's delete marker on
the bus to tell a handover from an expiry; and a real takeover (lost or
expired) is raised and cleared once by the controller that took it, instead
of an hour of S12 observation.

Research 072 piece moved: the lease's takeover read from its own key on the
bus and said as an event (03-domains, mesh-controller row; 04-findings 2,
the watchdogs' age reading).
Quick fix weighed: drop the gate check from the release record only; it
leaves the release last in the shutdown and the hour-long alarm for a real
takeover.

Name the epoch given back in the lease's delete marker, so only a holder's own release counts as a handover (issue 906, ADR 0558 §3, review round 1)

mesh/delivery merged: source.main moved to novox/mesh-controller@0fbb5ef35991

Details

mesh/check being carried to 8f847318963c on 0fbb5ef35991 | from 2dafbe60e939

8f84731896

A plain delete marker on the lease key is also written by a command's own
lease and by a take given back over the floor; counted as a handover, it
would record a crashed holder's epoch released and say nothing. Release now
writes the marker itself, compare-and-set on the key's revision, with the
header Mesh-Lease-Released naming its epoch, and the next holder ends only
that epoch as released. A takeover whose predecessor cannot be read is now
raised as an expiry, not only printed.

mesh-tools commented

2026-10-26 23:57:15 +00:00

Delivery novox/mesh-controller@8f847318963c — merged since 2026-10-27T00:07:48Z

Source — source.main moved to novox/mesh-controller@0fbb5ef359. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-26 23:57 (new) → proposed (announced): novox/mesh-controller#478's head announced
2026-10-26 23:57 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-27 00:07 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@8f847318963c` — **merged** since 2026-10-27T00:07:48Z **Source** — source.main moved to novox/mesh-controller@0fbb5ef35991. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-26 23:57 (new) → proposed (announced): novox/mesh-controller#478's head announced - 2026-10-26 23:57 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-27 00:07 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 0fbb5ef359 into main

2026-10-27 00:07:47 +00:00

mesh-tools deleted branch fix/906-lease-release

2026-10-27 00:07:47 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#478