Build modules' step and toolchain lists read only the newest keys they answer, in parallel (issue 1827) #485

2026-10-25T23:49:45Z

mesh-tools commented

2026-10-25 23:49:45 +00:00

Issue 1827, notes 3-6. On shanks, 2026-10-26 00:28-00:33, mesh-build-go.build_go_steps did not answer within 30 s. It read every step of the node one Get at a time to show a few (1164 keys, 20 MB in mesh-build-go_steps; mesh-build-node_steps holds 10034).

The change, in mesh-build-go, mesh-build-node and mesh-build-image (read.go):

RecentSteps sorts the node's keys newest first. Step ids start with base-36 milliseconds, so string order is start order.
It reads only the newest min(max(limit*4, 100), 1000) of them, 16 at a time.
The answer says cut with a note when older steps were not read, and each call logs one line.
ReadToolchains reads in parallel and logs a cut line past 1000 keys.

Tests: one table test per module with a slow counting fake bucket. It covers reads, overlap and the cut, and checks that the newest step comes first when cut.

Run locally (the check does not gate):

unit with -race: pass in all three modules
go vet -tags integration: clean
integration: pass in go (77 s) and node (26 s)

In image, TestARealBuildIsRootlessWithNoSocketAndNoWayOut and TestARealBuildFindsNoCredentialsOfTheAccount fail on shanks. They fail the same way on main (environmental: real buildah builds).

Review: round 1 found that the cut kept an arbitrary 1000 keys; fixed in round 2.

Note, not blocking: an ask step's id carries the ask time, so a long-queued ask can fall outside the read window. The answer says cut, and build_*_step reads it alone.

Acceptance after delivery: shanks/mesh-build-go.build_go_steps and shanks/mesh-build-node.build_node_steps answer within a few seconds, newest first.

Reviewed-by: review-reads-cat@shanks at e901769cb2

Issue 1827, notes 3-6. On shanks, 2026-10-26 00:28-00:33, mesh-build-go.build_go_steps did not answer within 30 s. It read every step of the node one Get at a time to show a few (1164 keys, 20 MB in mesh-build-go_steps; mesh-build-node_steps holds 10034). The change, in mesh-build-go, mesh-build-node and mesh-build-image (read.go): - RecentSteps sorts the node's keys newest first. Step ids start with base-36 milliseconds, so string order is start order. - It reads only the newest min(max(limit*4, 100), 1000) of them, 16 at a time. - The answer says `cut` with a note when older steps were not read, and each call logs one line. - ReadToolchains reads in parallel and logs a cut line past 1000 keys. Tests: one table test per module with a slow counting fake bucket. It covers reads, overlap and the cut, and checks that the newest step comes first when cut. Run locally (the check does not gate): - unit with -race: pass in all three modules - go vet -tags integration: clean - integration: pass in go (77 s) and node (26 s) In image, TestARealBuildIsRootlessWithNoSocketAndNoWayOut and TestARealBuildFindsNoCredentialsOfTheAccount fail on shanks. They fail the same way on main (environmental: real buildah builds). Review: round 1 found that the cut kept an arbitrary 1000 keys; fixed in round 2. Note, not blocking: an ask step's id carries the ask time, so a long-queued ask can fall outside the read window. The answer says cut, and build_*_step reads it alone. Acceptance after delivery: shanks/mesh-build-go.build_go_steps and shanks/mesh-build-node.build_node_steps answer within a few seconds, newest first. Reviewed-by: review-reads-cat@shanks at e901769cb2c2bf88cc32e044fcf2b9a9c99c8776

mesh-tools added 2 commits 2026-10-25 23:49:46 +00:00

Read only the node's own steps, in parallel and bounded, so build_*_steps answers within the call's 30 s (issue 1827) 8f9c78bcba

On a steps state of 1164 keys / 20 MB (mesh-build-node: 10034 keys) the read tools listed every key and
then read each value one round trip at a time, filtering by node only after the Get, so they timed out
at 30 s. Now the keys are filtered by the node's prefix before any Get, the values are read 16 at a
time, at most 1000 a call (the answer says it was cut), and one log line gives rows read and time.
build_*_toolchains reads its values the same parallel way. The kind: a read tool that reads a whole
state key by key; every read tool of the three modules is covered.

Read the newest steps first: sort the node's keys by id (start order) before the cut, and say older steps were not read

mesh/delivery merged: source.main moved to novox/mesh-catalog@567736f7795a

Details

mesh/check failed: integration: meshbuildnode/cmd/mesh-build-node.TestAFixtureBundleBuiltTwiceHasOneDigest: no toolchain of this lock is installed und…

e901769cb2

The review found a cut read took an arbitrary 1000 keys, so the newest steps could be missing. A step id is
base-36 milliseconds plus random digits, so its string order is its start order; the newest max(limit*4, 100)
keys are read now, 1000 at most. The toolchains read logs when it drops keys past its cap.

mesh-tools commented

2026-10-25 23:49:49 +00:00

Delivery novox/mesh-catalog@e901769cb2c2 — merged since 2026-10-25T23:56:13Z

Source — source.main moved to novox/mesh-catalog@567736f779. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-25 23:49 (new) → proposed (announced): novox/mesh-catalog#485's head announced
2026-10-25 23:49 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-25 23:56 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@e901769cb2c2` — **merged** since 2026-10-25T23:56:13Z **Source** — source.main moved to novox/mesh-catalog@567736f7795a. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 23:49 (new) → proposed (announced): novox/mesh-catalog#485's head announced - 2026-10-25 23:49 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-25 23:56 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 567736f779 into main

2026-10-25 23:56:13 +00:00

mesh-tools deleted branch fix/1827-bounded-reads

2026-10-25 23:56:13 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#485