Indexer: write a module's value only when its fingerprint changes, so a catalogue merge no longer floods the take-in (issue 1945) #48

2026-10-25T20:22:37Z

mesh-tools commented

2026-10-25 20:22:37 +00:00

Issue 1945 (the take-in flood of 2026-10-25: a catalogue merge rewrote ~186 module values and made a backlog of 605). Design 47, ADR 0429 (parts watch and write keys on the bus).

eaf27f3 The indexer writes a module's value only when the module changed (fingerprint, manifest digest, path, inputs, base module, toolchain). The value keeps the commit where its fingerprint first appeared; Src is stamped once. A catalogue merge now writes one to a few keys instead of every module of the repository.
9afef80 The builder merges a candidate's preview onto the trunk head it was previewed onto, not onto the module value's commit (which is now older).
8d38c6b The indexer fills the existing mesh-sdk field Candidate.Base with the source revision whose trunk heads it previewed against, so the builder finds that trunk head. No mesh-sdk change. The builder's test starts from the candidate the indexer's test writes (testdata/candidate-written.json).
ac82187 Test only: TestIndexerCountsOnlyTheAssignedBuildNodesToolchains fails on main 4efe6cf, because #47's last commit (keep a computed fingerprint while builders disagree, issue 1940) never updated #47's own test. The test now expects the fingerprint kept at n2's assignment and the disagreement at the next merge; it still proves n2's report counts once assigned and stops counting once unassigned.

Checked: go test -race and go vet -tags integration pass in the indexer, builder, view and checker; the integration level of the indexer, builder and checker passes locally at ac82187 (the first merge check failed on the test above, which fails on main too). No try (recovery mode: the operator merges by hand). Test counts unchanged (573 unit, 76 integration in the repository, already over the 500/50 budget; this change adds none).

Review notes kept for later (not blocking): a candidate whose fingerprints stay the same keeps an old Base, and after 64+ merges unbuilt the builder logs 'not heard' until the PR or trunk moves; a missing source bucket at builder start means no candidates until restart; candidates.go:79 logs a misleading 'is not built' for an already-built fingerprint; the builder test reads the indexer's testdata across modules; Src stays 0 if the indexer misses its revision; the fixed 500 ms wait before the 'kept' read in the corrected test could miss a later blanking (the disagree and unassign checks stay valid).

Reviewed-by: review-indexer@shanks at ac82187b85

Issue 1945 (the take-in flood of 2026-10-25: a catalogue merge rewrote ~186 module values and made a backlog of 605). Design 47, ADR 0429 (parts watch and write keys on the bus). - **eaf27f3** The indexer writes a module's value only when the module changed (fingerprint, manifest digest, path, inputs, base module, toolchain). The value keeps the commit where its fingerprint first appeared; Src is stamped once. A catalogue merge now writes one to a few keys instead of every module of the repository. - **9afef80** The builder merges a candidate's preview onto the trunk head it was previewed onto, not onto the module value's commit (which is now older). - **8d38c6b** The indexer fills the existing mesh-sdk field Candidate.Base with the source revision whose trunk heads it previewed against, so the builder finds that trunk head. No mesh-sdk change. The builder's test starts from the candidate the indexer's test writes (testdata/candidate-written.json). - **ac82187** Test only: TestIndexerCountsOnlyTheAssignedBuildNodesToolchains fails on main 4efe6cf, because #47's last commit (keep a computed fingerprint while builders disagree, issue 1940) never updated #47's own test. The test now expects the fingerprint kept at n2's assignment and the disagreement at the next merge; it still proves n2's report counts once assigned and stops counting once unassigned. Checked: go test -race and go vet -tags integration pass in the indexer, builder, view and checker; the integration level of the indexer, builder and checker passes locally at ac82187 (the first merge check failed on the test above, which fails on main too). No try (recovery mode: the operator merges by hand). Test counts unchanged (573 unit, 76 integration in the repository, already over the 500/50 budget; this change adds none). Review notes kept for later (not blocking): a candidate whose fingerprints stay the same keeps an old Base, and after 64+ merges unbuilt the builder logs 'not heard' until the PR or trunk moves; a missing source bucket at builder start means no candidates until restart; candidates.go:79 logs a misleading 'is not built' for an already-built fingerprint; the builder test reads the indexer's testdata across modules; Src stays 0 if the indexer misses its revision; the fixed 500 ms wait before the 'kept' read in the corrected test could miss a later blanking (the disagree and unassign checks stay valid). Reviewed-by: review-indexer@shanks at ac82187b852ce1e2624a8d2c9d0b26da9dfeda82

mesh-tools added 3 commits 2026-10-25 20:22:37 +00:00

Write a module's value only when the module changed, so a merge no longer rewrites every module of its repository eaf27f36af

The operator, 2026-10-25: only if a module has a new version in commit x is a new value written naming commit x. A merge
wrote every module of the repository at the new commit, fingerprint unchanged (about 186 keys per mesh-catalog merge), and
the controller's take-in took each in again: twice tonight it flooded. Now a merge that gives a module the value it holds
but for the commit writes nothing, and the value keeps the commit its fingerprint first appeared at. The kind: a write
that differs only in where it was computed. The other instance, a source revision re-stamping Src on every module at its
repository's head after any merge anywhere, now stamps Src once. The trunk head stays the gate's source.main.

Merge a candidate's preview onto the trunk head it was previewed onto, not the module value's commit 9afef800c0

The builder took a candidate build's merge base from the module's value commit. Now that a value keeps the commit its
fingerprint first appeared at, that commit can be many merges behind the trunk, so a preview would be built onto a stale
trunk (review round 1). The base is now the head of the pull request's repository in the gate's source revision the
candidate names (Candidate.Base): the builder reads mesh-delivery-gate.source, watched with its history (a direct get of
one revision is not what a read grants), and a candidate heard before its revision is built once the revision is heard.

Write the source revision a candidate was previewed onto, so the builder merges its preview onto that trunk head

mesh/delivery closed: a newer head of the same pull request

Details

mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…

8d38c6bac9

Candidate.Base was never written non-zero (review round 2): the builder's base lookup would have found no trunk for any
live candidate and built no preview at all. The indexer now writes, whenever it writes a candidate's fingerprints, the gate's
source revision whose trunk heads it read the previews onto; that head of the pull request's repository is the preview's
trunk. No mesh-sdk change: the field and its meaning were there. The builder's test now takes the candidate the indexer's
own test writes (testdata/candidate-written.json, kept equal to it), not a hand-made one.

mesh-tools commented

2026-10-25 20:22:40 +00:00

Delivery novox/mesh-delivery@ac82187b852c — merged since 2026-10-25T20:35:26Z

Source — source.main moved to novox/mesh-delivery@82180bffd2. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-25 20:28 (new) → proposed (announced): novox/mesh-delivery#48's head announced
2026-10-25 20:30 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail
2026-10-25 20:35 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-delivery@ac82187b852c` — **merged** since 2026-10-25T20:35:26Z **Source** — source.main moved to novox/mesh-delivery@82180bffd2c6. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 20:28 (new) → proposed (announced): novox/mesh-delivery#48's head announced - 2026-10-25 20:30 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail - 2026-10-25 20:35 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

jschoubben added 1 commit 2026-10-25 20:28:05 +00:00

Indexer integration test: a disagreement keeps a computed fingerprint, as issue 1940's fix decided

mesh/delivery merged: source.main moved to novox/mesh-delivery@82180bffd2c6

Details

mesh/check pass ac82187b852c on 82180bffd2c6 | unit 167 run, 432 kept | integration 177 run, 498 kept | composition passed on 2 machines

ac82187b85

The test predates the fix that keeps a held fingerprint while builders disagree (#47, 3e7f852) and still asserted the
blanking that fix removed, so it fails on main 4efe6cf as well. It now asserts the fingerprint is kept after n2's
assignment, and that a merge moving avahi during the disagreement gets none until the builders agree.