The sweep keeps a fresh build's files, and a take-in holds every file before it records a build (issue 1933) #451

2026-10-24T00:49:34Z

mesh-tools commented

2026-10-24 00:49:34 +00:00

Issue 1933 (critical, met three times tonight: the indexer 00:37, route-proxy 01:52 and 02:43). The controller's own collection sweep let go of a fresh build's archives seconds after the take-in, so the definition it sent named blobs the artifact store no longer held, and novox and ace failed every apply.

Cause (note 8): Inventory.Artifacts read KeepSet before the records; take-ins run concurrently and each ends in a sweep, so a RecordBuild landing between the two reads was 'recorded, not kept' and let go. A new module's build was also kept by nothing between RecordBuild and its registration.

Fix:

Artifacts reads the records first; KeepSet keeps what a build recorded within KeptLately (1 h, 'recorded-lately') and what an open plan's builds name ('open-plan').
The after-build sweep names every reference it lets go of.
buildOfKey holds every artifact of the key before anything is recorded: a Gone blob fails the take-in 'not in the artifact store' and records no build; an unreachable store makes the take-in wait.
RecordBuild clears artifact_collected for what the build made.
No schema change.

Tests: four unit tests fail first on main (the race, the fresh/open-plan keep, the take-in hold, the sweep's log) and pass here; two integration tests (a new module's build kept during its take-in; a collected reference recorded again is held) are compiled, not run by hand. Unit suite 13-24 s (over the 2 s guideline: said, not cut, issue 1921); go test -race on the touched packages passes; go vet -tags integration ./... clean.

What-if: builds mesh-controller only, sent to novox. After its delivery, route-proxy needs one more rebuild, taken in by this controller.

Later (noted on 1933): a sweep that judged an identical digest eligible before a new build records it again can still let it go (needs a lock or a second KeepSet read); KeepSet's cost as builds grow; a key whose blob is gone is refused on every retry until the builder asks the store before reusing a done key.

Temporary build -> publish -> deliver mode (issue 1921 note 13): the operator merges by hand.

Reviewed-by: review-1933c@shanks at 0741880d31
Pushed-by: lead-47@shanks

Issue 1933 (critical, met three times tonight: the indexer 00:37, route-proxy 01:52 and 02:43). The controller's own collection sweep let go of a fresh build's archives seconds after the take-in, so the definition it sent named blobs the artifact store no longer held, and novox and ace failed every apply. Cause (note 8): Inventory.Artifacts read KeepSet before the records; take-ins run concurrently and each ends in a sweep, so a RecordBuild landing between the two reads was 'recorded, not kept' and let go. A new module's build was also kept by nothing between RecordBuild and its registration. Fix: - Artifacts reads the records first; KeepSet keeps what a build recorded within KeptLately (1 h, 'recorded-lately') and what an open plan's builds name ('open-plan'). - The after-build sweep names every reference it lets go of. - buildOfKey holds every artifact of the key before anything is recorded: a Gone blob fails the take-in 'not in the artifact store' and records no build; an unreachable store makes the take-in wait. - RecordBuild clears artifact_collected for what the build made. No schema change. Tests: four unit tests fail first on main (the race, the fresh/open-plan keep, the take-in hold, the sweep's log) and pass here; two integration tests (a new module's build kept during its take-in; a collected reference recorded again is held) are compiled, not run by hand. Unit suite 13-24 s (over the 2 s guideline: said, not cut, issue 1921); `go test -race` on the touched packages passes; `go vet -tags integration ./...` clean. What-if: builds mesh-controller only, sent to novox. After its delivery, route-proxy needs one more rebuild, taken in by this controller. Later (noted on 1933): a sweep that judged an identical digest eligible before a new build records it again can still let it go (needs a lock or a second KeepSet read); KeepSet's cost as builds grow; a key whose blob is gone is refused on every retry until the builder asks the store before reusing a done key. Temporary build -> publish -> deliver mode (issue 1921 note 13): the operator merges by hand. Reviewed-by: review-1933c@shanks at 0741880d3137223747a15063ad038668e6f69efd Pushed-by: lead-47@shanks

mesh-tools added 1 commit 2026-10-24 00:49:35 +00:00

Keep a fresh build's archives from the controller's own sweep (issue 1933)

mesh/delivery delivered

Details

0741880d31

A sweep read what it keeps before what was ever recorded, so a take-in's
build recorded between the two reads, by a concurrent take-in off the
plans' hold, was eligible and let go of seconds after the take-in read
it; a new module's build is unkept until it is registered. Now the sweep
reads the records first and keeps what a build recorded within the hour
or an open plan names, and names every reference it lets go of; a take-in
holds every artifact in the store before recording, failing on one the
store lacks; and RecordBuild clears artifact_collected as RecordMirrored
does.

mesh-tools commented

2026-10-24 00:49:38 +00:00

Delivery novox/mesh-controller@0741880d3137 — delivered since 2026-10-24T00:59:40Z

Delivery plan — builds mesh-controller → novox; no bus step

tier 0: mesh-controller
novox: receives mesh-controller

Machines

mesh-controller on every machine running it: sent

Transitions

2026-10-24 00:49 (new) → proposed (announced): novox/mesh-controller#451's head announced
2026-10-24 00:54 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on
2026-10-24 00:55 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's plans go
2026-10-24 00:59 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@0741880d3137` — **delivered** since 2026-10-24T00:59:40Z **Delivery plan** — builds mesh-controller → novox; no bus step - tier 0: mesh-controller - novox: receives mesh-controller **Machines** - mesh-controller on every machine running it: sent **Transitions** - 2026-10-24 00:49 (new) → proposed (announced): novox/mesh-controller#451's head announced - 2026-10-24 00:54 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on - 2026-10-24 00:55 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's `plans go` - 2026-10-24 00:59 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

jschoubben merged commit 760351b4ae into main

2026-10-24 00:54:55 +00:00

jschoubben deleted branch fix/1933-sweep-keeps-fresh-builds

2026-10-24 00:54:55 +00:00

mesh-tools referenced this pull request

2026-10-24 00:57:15 +00:00

Give route-proxy a build the fixed controller takes in, dropping the incident note #450 added (issue 1933) #452

mesh-tools referenced this pull request

2026-10-24 02:39:02 +00:00

Answer held and say artifacts-gone, so the builder rebuilds a done key whose blob is gone (issue 1937) #453

Sign in to join this conversation.