Compose the packages mirror only on a change, end every serve goroutine before serve returns, and raise no alarm for a cancellation (issue 1945) #464

2026-10-25T16:11:44Z

mesh-tools commented

2026-10-25 16:11:44 +00:00

Issue 1945 (R6 recovery, fix 12).

Mirror on change (a4cd5ba3): the declared-packages mirror recomposed every node every 10 s (about 150 % CPU on novox). It now composes when what it composes from changes: composed and assignment bucket writes, this controller taking the lease, and a 10-minute safety pass.
Serving group (c75d7262): every goroutine serve starts is counted; serve cancels and waits for them before it undoes anything, the watchdogs have their own group, the condition store opens before its readers and closes after them, askerFrom is atomic. This fixes a DATA RACE on conditionsFrom (watchdogs.go watchTheMesh vs pending.go settlingPending) that the race detector reported after the tests' PASS: the cause of the try that failed at 15:32 with its words hidden (issue 1901). fingerprint-not-composed has plain words.
Cancellation is no fault (c7e31640): a take-in, layout or put cut by a stopping serve or by its own step's cancellation raises and clears nothing. This was the cause of about 30 false urgent fingerprint-not-composed conditions today.
#463's take-ins waited for (cc5c9ae8): inBackground counts in the serving group.
Clear on every success (8e02c472): a successful take-in clears the module's condition whoever raised it, so alarms from before a restart clear. The error path calls stopRunning before the undoing defers.

Tests, each failing first: TestATakeInCutByACancellationRaisesNothing, TestAPutCutByACancellationRaisesNothing, TestASucceedingTakeInClearsAConditionAnotherRaised, plus a leaked-global test fixed. Checked: gofmt, go vet with and without -tags integration, the unit suite with -race, and -race -tags integration -count=2 on TestServed|DoneKey|TakeIn|Put|Plain|Words. No try: the operator decided no checks run during the recovery (issue 1945 note 27).

NATS-native research piece moved: none. This recovery fix was written and reviewed before the instruction fix-the-controller-at-its-root was approved (2026-10-25). The mirror's NATS-native shape (compose on the bus's change events, state in the bus) is part of the research "The controller, NATS-native" being written now.

After delivery, the controller's start pass takes in every done key, which clears the stale fingerprint-not-composed alarms.

Reviewed-by: reviewer-r6m3@shanks at 8e02c4726e

Issue 1945 (R6 recovery, fix 12). - **Mirror on change** (a4cd5ba3): the declared-packages mirror recomposed every node every 10 s (about 150 % CPU on novox). It now composes when what it composes from changes: composed and assignment bucket writes, this controller taking the lease, and a 10-minute safety pass. - **Serving group** (c75d7262): every goroutine serve starts is counted; serve cancels and waits for them before it undoes anything, the watchdogs have their own group, the condition store opens before its readers and closes after them, askerFrom is atomic. This fixes a DATA RACE on conditionsFrom (watchdogs.go watchTheMesh vs pending.go settlingPending) that the race detector reported after the tests' PASS: the cause of the try that failed at 15:32 with its words hidden (issue 1901). fingerprint-not-composed has plain words. - **Cancellation is no fault** (c7e31640): a take-in, layout or put cut by a stopping serve or by its own step's cancellation raises and clears nothing. This was the cause of about 30 false urgent fingerprint-not-composed conditions today. - **#463's take-ins waited for** (cc5c9ae8): inBackground counts in the serving group. - **Clear on every success** (8e02c472): a successful take-in clears the module's condition whoever raised it, so alarms from before a restart clear. The error path calls stopRunning before the undoing defers. Tests, each failing first: TestATakeInCutByACancellationRaisesNothing, TestAPutCutByACancellationRaisesNothing, TestASucceedingTakeInClearsAConditionAnotherRaised, plus a leaked-global test fixed. Checked: gofmt, go vet with and without -tags integration, the unit suite with -race, and -race -tags integration -count=2 on TestServed|DoneKey|TakeIn|Put|Plain|Words. No try: the operator decided no checks run during the recovery (issue 1945 note 27). NATS-native research piece moved: none. This recovery fix was written and reviewed before the instruction fix-the-controller-at-its-root was approved (2026-10-25). The mirror's NATS-native shape (compose on the bus's change events, state in the bus) is part of the research "The controller, NATS-native" being written now. After delivery, the controller's start pass takes in every done key, which clears the stale fingerprint-not-composed alarms. Reviewed-by: reviewer-r6m3@shanks at 8e02c4726eb4a8cf7de9cc3d2ce0560f00b35157

mesh-tools added 5 commits 2026-10-25 16:11:45 +00:00

Compose the declared-packages mirror only when what it composes from changed (issue 1945) a4cd5ba375

The mirror composed every node in full on the record mirrors' 10 s tick; a
pass takes about 45 s on the control node, so the controller never rested
(130-175 % CPU, issue 1945 note 22). It now passes on a watch of the
compositions' digests and of the assignments, one pass at a time with the
wakes of a pass coalesced into one after it (at least 30 s apart), skips the
composition when the stores' Input digest is unchanged, and keeps a safety
pass every 10 minutes that composes nothing when nothing changed.

End every goroutine serve starts before serve returns, so no loop outlives its controller (issue 1945) c75d726205

The try of the on-change mirror failed with every test passed: goroutines a
served test started outlived its serve and met the next test's globals and its
closed stores. Under -race the served tests show it as a race on conditionsFrom
(watchTheMesh writing it while settlingPending, already running, read it), and
a take-in cut by the stop raised fingerprint-not-composed, a kind with no plain
words, which fails the package after its PASS line.

- serve counts every goroutine it and its parts start (serving_group.go) and
  ends and waits for them before anything else is undone: the lease's watch,
  the pending settler, the status summary, the facts, the mirrors and the new
  declared-packages keeper with its two bucket watches, the declaration sweep,
  the read-call counter, the watches' closers, the have watches, the vault move
  on a report, and the watchdogs, doctor, healers and asker under their own
  group.
- The condition store opens before anything that reads conditionsFrom starts;
  askerFrom is an atomic pointer, read from the keeper's goroutines.
- A done key's take-in cut by the stop raises nothing, and fingerprint-not-composed
  has plain words.

Raise no condition for a take-in, a layout or a put cut by a cancellation (issue 1945) c7e31640cf

Thirty urgent fingerprint-not-composed conditions were live on 2026-10-25, each a
done key's take-in a stop or a restart cut with "context canceled". A
cancellation is no fault of the build, the chain or the bucket: a take-in or a
layout whose controller stops returns and the next one does it; one whose own
step was cancelled is tried again without a condition; a send cut by a
cancellation counts as no failure to put its files.

The put test now starts and leaves the node's failure count at nothing, so a
second run of it (-count=2) finds no count left by the first.

Count the take-ins and every other background work in the serving group, so serve's stop waits for them too (issue 1945) cc5c9ae888

inBackground ran the take-ins of done keys (now in #463's slots), the start's
look, the given values' replacement and the announced reads beside serve without
serve waiting for them: a take-in could outlive its controller as the loops did.
It now takes the context and is counted in its serving group as well, and a
serve that is stopping starts none.

Clear a module's fingerprint-not-composed on every successful take-in, and stop serve's goroutines before an error return undoes anything (issue 1945)

mesh/check pass 8e02c4726eb4 on c3db00ceac25 | unit 981 run, 171 kept, 2 skipped | integration 1641 run, 232 kept, 7 skipped | over bound: integration…

mesh/delivery held for a person: merged without a passing check: only a person decides what follows

Details

8e02c4726e

A success cleared the condition only when the same follower had raised it, so
one raised before a restart or by another follower stayed open after the take-in
worked, and the operator read that a build needed a look. Clearing a condition
that is not open costs one read. On an error return, serve now ends its
goroutines before the keeper closes and before handActConn and the bus are put
back (the review of fix 12).

mesh-tools commented

2026-10-25 16:11:49 +00:00

Delivery novox/mesh-controller@8e02c4726eb4 — held since 2026-10-25T16:20:58Z

Held for a person: merged without a passing check: only a person decides what follows — the merge reached the trunk without a passing check.

Source — source.main moved to novox/mesh-controller@c3db00ceac. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-25 16:11 (new) → proposed (announced): novox/mesh-controller#464's head announced
2026-10-25 16:20 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides what follows

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@8e02c4726eb4` — **held** since 2026-10-25T16:20:58Z Held for a person: merged without a passing check: only a person decides what follows — the merge reached the trunk without a passing check. **Source** — source.main moved to novox/mesh-controller@c3db00ceac25. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 16:11 (new) → proposed (announced): novox/mesh-controller#464's head announced - 2026-10-25 16:20 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides what follows The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

jschoubben merged commit c3db00ceac into main

2026-10-25 16:20:55 +00:00

jschoubben deleted branch fix/r6-mirror-on-change-2

2026-10-25 16:20:56 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#464