Compose each machine once per pass, so take-ins of done builds stop colliding and every new build gets its composition and credential (issue 1945) #460

2026-10-25T11:52:44Z

mesh-tools commented

2026-10-25 11:52:44 +00:00

Since the switch to release 6, every take-in of a done build recomposed every machine of its module itself. Dozens ran at once and collided on the compositions' revisions ("wrong last sequence ... key revision mismatch; tried again in 30s"), so new builds (mesh-host 55e07c82 for g14 and shanks) got no composition and no credential, and wants froze on a digest the bucket no longer held (node-tools on shanks and g14).

What changes:

8a3c0cd0: a waiting take-in says why once and is looked at again every 30 s, one follower per module.
0470869f: the digest kept beside a composition follows the composition the bucket holds, read back after writing.
baa8f584 + 8114cee2: one composer per set of stores. A take-in registers its module and asks the composer for its nodes, then the bus holder, then the grants; the composer runs one pass at a time, composing each node asked since the last pass began once, then the grants once. Each caller hears only its own nodes' errors; grants are written whenever any caller asked for them. Every composition of a node holds a per-node lock. The start pass looks at 16 done keys side by side.
f35aa355 + c47a5904 + 3b4cb69e: no request is dropped by a "busy, return at once" guard. Every such guard in cmd/mesh-controller (takeDoneKey, takeDoneKeyAtStart, replaceGiven) leaves a mark when it finds the work running, and the holder reads the mark after letting go and goes round. The other locks block or refuse aloud.

Tests (each failed first): the burst test, the caller hears its pass's failure, TestAWakeDuringATakeInIsTakenAfterIt, TestANodeThatFailsFailsOnlyItsCallers, the digest race test, TestAWakeJustBeforeTheFollowerLetsGoIsTaken, TestAWakeDuringTheStartsTakeInIsTaken, TestAStartHeardDuringAReplacementIsTaken.

Tries: try-1792929772594650167 passed on 3b4cb69e (962 unit, 1618 integration).
Reviewed-by: reviewer-r6m@shanks at 3b4cb69e84
Pushed-by: lead-47@shanks

Since the switch to release 6, every take-in of a done build recomposed every machine of its module itself. Dozens ran at once and collided on the compositions' revisions ("wrong last sequence ... key revision mismatch; tried again in 30s"), so new builds (mesh-host 55e07c82 for g14 and shanks) got no composition and no credential, and wants froze on a digest the bucket no longer held (node-tools on shanks and g14). What changes: 1. 8a3c0cd0: a waiting take-in says why once and is looked at again every 30 s, one follower per module. 2. 0470869f: the digest kept beside a composition follows the composition the bucket holds, read back after writing. 3. baa8f584 + 8114cee2: one composer per set of stores. A take-in registers its module and asks the composer for its nodes, then the bus holder, then the grants; the composer runs one pass at a time, composing each node asked since the last pass began once, then the grants once. Each caller hears only its own nodes' errors; grants are written whenever any caller asked for them. Every composition of a node holds a per-node lock. The start pass looks at 16 done keys side by side. 4. f35aa355 + c47a5904 + 3b4cb69e: no request is dropped by a "busy, return at once" guard. Every such guard in cmd/mesh-controller (takeDoneKey, takeDoneKeyAtStart, replaceGiven) leaves a mark when it finds the work running, and the holder reads the mark after letting go and goes round. The other locks block or refuse aloud. Tests (each failed first): the burst test, the caller hears its pass's failure, TestAWakeDuringATakeInIsTakenAfterIt, TestANodeThatFailsFailsOnlyItsCallers, the digest race test, TestAWakeJustBeforeTheFollowerLetsGoIsTaken, TestAWakeDuringTheStartsTakeInIsTaken, TestAStartHeardDuringAReplacementIsTaken. Tries: try-1792929772594650167 passed on 3b4cb69e (962 unit, 1618 integration). Reviewed-by: reviewer-r6m@shanks at 3b4cb69e84ddd59759f07342d56fbdd77d6a8ff2 Pushed-by: lead-47@shanks

mesh-tools added 4 commits 2026-10-25 11:52:44 +00:00

Say each wait of a done key's take-in once with its reason and look again every 30 s, one follower per module (issue 1945) 8a3c0cd022

Keep the digest of the composition the bucket holds, following another writer's composition written in between (issue 1945) 0470869fbb

Compose each node once per pass in one composer for the done keys' take-ins, one composition of a node at a time, and the start's look side by side (issue 1945) baa8f58473

Give each caller of a pass its own nodes' errors and write the grants whenever asked; a wake during a take-in makes it look once more (issue 1945)

mesh/check pass 8114cee2af47 on 0fff3efd4e0b | unit 959 run, 172 kept, 2 skipped | integration 1615 run, 236 kept, 7 skipped | over bound: integration…

mesh/delivery closed: a newer head of the same pull request

Details

8114cee2af

mesh-tools commented

2026-10-25 11:52:50 +00:00

Delivery novox/mesh-controller@3b4cb69e84dd — merged since 2026-10-25T12:30:34Z

Source — source.main moved to novox/mesh-controller@af2b0844a1. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-25 12:12 (new) → proposed (announced): novox/mesh-controller#460's head announced
2026-10-25 12:19 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail
2026-10-25 12:30 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@3b4cb69e84dd` — **merged** since 2026-10-25T12:30:34Z **Source** — source.main moved to novox/mesh-controller@af2b0844a1fc. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 12:12 (new) → proposed (announced): novox/mesh-controller#460's head announced - 2026-10-25 12:19 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail - 2026-10-25 12:30 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

jschoubben added 3 commits 2026-10-25 12:12:22 +00:00

Look at a module's mark again after letting go of it, so a wake just before is taken (issue 1945) f35aa35541

The start pass reads a module's look-again mark after letting go, and hands it on to takeDoneKey (issue 1945) c47a590463

Take again a replacement of given values asked while one ran on the node (issue 1945)

mesh/check pass 3b4cb69e84dd on af2b0844a1fc | unit 963 run, 171 kept, 2 skipped | integration 1622 run, 232 kept, 7 skipped | over bound: integration…

mesh/delivery merged: source.main moved to novox/mesh-controller@af2b0844a1fc

Details

3b4cb69e84

mesh-tools commented

2026-10-25 12:12:27 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 8114cee2af is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 8114cee2af47 is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools merged commit af2b0844a1 into main

2026-10-25 12:30:32 +00:00

mesh-tools deleted branch fix/r6-compose-once

2026-10-25 12:30:32 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#460