Answer held and say artifacts-gone; write the grants again; integration tests match R4g (issues 1937, 1938, 1931) #453

Merged

jschoubben merged 30 commits from fix/1937-done-key-blobs into main 2026-10-24 10:35:15 +00:00

Conversation 1 Commits 30 Files Changed 30

+574 -65

mesh-tools commented 2026-10-24 02:39:02 +00:00

Member

Copy Link

Copy Source

Issue 1937 (critical). The controller serves held (read-only, using #451's store check) and says artifacts-gone on mesh.seat.mesh-controller.event.artifacts-gone when a take-in is refused because blobs are gone, naming every gone blob. The grant lets it publish the event.

Also carries, merged from fix/1931-reconnects-verb:

• Issue 1938: chainBuildByID (internal/inventory/chain_builds.go) returns the build's SourceFingerprint again, so the controller writes the grants the assigner reads; it has written none since 2026-10-18 (the live grants bucket holds 0 keys).
• Issue 1931: about 20 integration test files made to match R4g (modules taken from the builder's done key, no build asked), so mesh-controller's integration level passes: 2566 tests, nothing kept, in 5m38s (try-1792817721004558884); and the rollout test's race between anchor's report and the silence bound removed (anchor reports on its send).

Group fix/1937-done-key-blobs: mesh-sdk, mesh-controller, mesh-delivery. The builder's recovery at its start holds when this controller is delivered before the builder (the delivery plan's tiers will be read). Nothing the operator sees changes.

Reviewed-by: review-1937@shanks at 3b51d3f86e
Reviewed-by: review-453b@shanks at 16ea033124
Reviewed-by: review-453b@shanks at c06f75d083
Pushed-by: lead-47@shanks

Issue 1937 (critical). The controller serves `held` (read-only, using #451's store check) and says `artifacts-gone` on `mesh.seat.mesh-controller.event.artifacts-gone` when a take-in is refused because blobs are gone, naming every gone blob. The grant lets it publish the event. Also carries, merged from fix/1931-reconnects-verb: - Issue 1938: chainBuildByID (internal/inventory/chain_builds.go) returns the build's SourceFingerprint again, so the controller writes the grants the assigner reads; it has written none since 2026-10-18 (the live grants bucket holds 0 keys). - Issue 1931: about 20 integration test files made to match R4g (modules taken from the builder's done key, no build asked), so mesh-controller's integration level passes: 2566 tests, nothing kept, in 5m38s (try-1792817721004558884); and the rollout test's race between anchor's report and the silence bound removed (anchor reports on its send). Group `fix/1937-done-key-blobs`: mesh-sdk, mesh-controller, mesh-delivery. The builder's recovery at its start holds when this controller is delivered before the builder (the delivery plan's tiers will be read). Nothing the operator sees changes. Reviewed-by: review-1937@shanks at 3b51d3f86ea65c17af9753430550ee271a8d8703 Reviewed-by: review-453b@shanks at 16ea033124e1a6b9b2aa7ac74dd0778046915708 Reviewed-by: review-453b@shanks at c06f75d083a990e43cd25f7e69c4f54056edf327 Pushed-by: lead-47@shanks

mesh-tools added 1 commit 2026-10-24 02:39:02 +00:00

Answer held and say artifacts-gone, so the builder rebuilds a done key whose blob the artifact store no longer holds (issue 1937) ... 3b51d3f86e

A done key the controller never took in holds no manifest, and the store's nightly collector deleted its bare blob:
node-tools', mesh-host's and the forge bridge's keys name blobs that answer 404. Since issue 1933 the take-in refuses such a
key, and nothing built it again. held (in the serving process, HEADs only) lets the builder ask before it reuses a key; a
refused take-in now names every gone archive and says artifacts-gone with the module, key, fingerprint and references.

mesh-tools commented 2026-10-24 02:39:05 +00:00

Author

Member

Copy Link

Copy Source

Delivery novox/mesh-controller@c06f75d083a9 — stopped since 2026-10-24T10:53:06Z

Covered: covered after its stop: a later walk that contains it was delivered: plan-1792856223268385098 (novox/mesh-controller at 34b105ea into main, done, contains its merge 1b0fd8b4)

Delivery plan — builds mesh-controller → novox; no bus step

• tier 0: mesh-controller
• novox: receives mesh-controller

Group fix/1937-done-key-blobs: novox/mesh-controller@c06f75d083, novox/mesh-delivery@28ee22ef27, novox/mesh-sdk@34f210e5eb

• composed together: pass — the checker's verdict at the members' heads (5bd558b99155)

Transitions

• 2026-10-24 05:16 (new) → proposed (announced): novox/mesh-controller#453's head announced
• 2026-10-24 10:35 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on
• 2026-10-24 10:35 held → delivering (release): the operator merged the reviewed group fix/1937-done-key-blobs by hand at 12:35 (temporary build-publish-deliver mode since 00:24; their merge is their word to release); the group's composed check passed at 07:26 but never reached the forge (issue 1726)
• 2026-10-24 10:53 delivering → stopped (stop): its walk plan-1792838111606250980 was stopped by a person (mesh-controller): the operator's word 12:53: close it. Its tier 0 waits for ever on mesh-sdk ('builders disagree': a stale ace.typescript key in mesh-build-node_toolchains, issue 1940), and the 1940 fix can become a plan only after this one ends (ADR 0276 §4). Retried after the 1940 fix is delivered.

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-controller@c06f75d083a9` — **stopped** since 2026-10-24T10:53:06Z **Covered**: covered after its stop: a later walk that contains it was delivered: plan-1792856223268385098 (novox/mesh-controller at 34b105ea into main, done, contains its merge 1b0fd8b4) **Delivery plan** — builds mesh-controller → novox; no bus step - tier 0: mesh-controller - novox: receives mesh-controller **Group** `fix/1937-done-key-blobs`: novox/mesh-controller@c06f75d083a9, novox/mesh-delivery@28ee22ef27e2, novox/mesh-sdk@34f210e5eba1 - composed together: pass — the checker's verdict at the members' heads (5bd558b99155) **Transitions** - 2026-10-24 05:16 (new) → proposed (announced): novox/mesh-controller#453's head announced - 2026-10-24 10:35 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on - 2026-10-24 10:35 held → delivering (release): the operator merged the reviewed group fix/1937-done-key-blobs by hand at 12:35 (temporary build-publish-deliver mode since 00:24; their merge is their word to release); the group's composed check passed at 07:26 but never reached the forge (issue 1726) - 2026-10-24 10:53 delivering → stopped (stop): its walk plan-1792838111606250980 was stopped by a person (mesh-controller): the operator's word 12:53: close it. Its tier 0 waits for ever on mesh-sdk ('builders disagree': a stale ace.typescript key in mesh-build-node_toolchains, issue 1940), and the 1940 fix can become a plan only after this one ends (ADR 0276 §4). Retried after the 1940 fix is delivered. The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

jschoubben added 27 commits 2026-10-24 05:02:55 +00:00

Call a verb that still exists in the reconnects test, since queue was removed (issue 1931) ab9a7771f4

Assert a Retry waits for the builder's key, since the controller asks no build now (issue 1931) d951aa954e

End the reconnects test's verb in the serving process: a command verb re-runs the test binary (issue 1931) a9de578cb9

Hold tier 1's key in the away test and count one shared key in the backups test, since a walk takes keys at once now (issue 1931) a77aa8520e

Assert the cut takes its first tier from the key at the carried commit, since the cut asks no build (issue 1931) 64fbef7713

Assert the three-repository walk takes each first-tier module from its key at its commit (issue 1931) 443ba6b265

Assert the merge walked alone is taken from its key at its own commit, since a cut records no ask (issue 1931) 5ee08cb5e1

Keep the started walk started in the let-go test by holding its key, since a walk takes keys at once now (issue 1931) 2ba9eef58d

Format the batches test imports (issue 1931) 893aa85349

Read the started walk again before ending it in the let-go test: the steps wrote it (issue 1931) a1969403c6

Hold the controller walk's key in the own-path test and read its walk again before ending it (issue 1931) fecfb40ed4

Hold the keys in the walk-beside test so its walks stay open (issue 1931) 9131ef0dbc

Derive the late holder's worker from a build seat, since the build agent's seat is gone (issue 1931) bb2073b6bb

Seed the seats before the late holder claims a build seat (issue 1931) e57224c25a

Give the late build-seat holder a broker account so its worker is derived (issue 1931) 3c9a4d56c0

Format the busobjects test (issue 1931) f99e499288

Step the walk once before counting its key takes in the delivery-word test (issue 1931) 3cfcf5e3e4

Assert the key takes where a walk asks no build, and hold keys where a walk must stand still (issue 1931) 9db1459cb0

Read a build's fingerprint back by its id, so the controller writes the grants the assigner reads (issue 1938) ... b2bf14ab01

chainBuildByID answered a trimmed build without SourceFingerprint since builds moved to the builds bucket (issue 1090), so
currentFingerprints read every fingerprint empty and grantsFrom skipped every module: mesh-controller_grants has never held
a key, and TestTheGrantOfAnIssuedCredentialIsOnTheBus failed on main.

Hold the expiry of a pending assignment to the words it has said since R4g: its last build, when, and that it failed (issue 1938) ... 491cbe2548

The test was kept from older runs and ran again only once the inventory changed beneath it; e419d018 changed the words.

Hold the replay of issue 614 to the plan since R4g: a new SDK plans itself and what stands on its package, no toolchain edge to the bundles (issue 1938) a5b874a2a0

Hold the rollout test's tier 1 to a key taken at b's own trunk value: the walk carries only the catalogue's commit, so KeyAt stays empty (issue 1938, after issue 1931) 5b03b2e26b

Have anchor report on tier 1's send in the rollout test only once the plan keeps when it was sent (issue 1938, after issue 1931) f2da7d0540

Name tier 1's take in the rollout test by b's own repository, novox/b (issue 1938, after issue 1931) e8b66a2eb3

Give the rollout test a silence window anchor's report fits in while the whole integration level runs at once (issue 1938) d0f5ae26de

Widen the rollout test's silence window to 10 s: under the whole integration level anchor's report came 5.7 s after its send (issue 1938) 9088db9a04

Merge remote-tracking branch 'origin/fix/1931-reconnects-verb' into fix/1937-done-key-blobs 16ea033124

mesh-tools changed title from Answer held and say artifacts-gone, so the builder rebuilds a done key whose blob is gone (issue 1937) to Answer held and say artifacts-gone; write the grants again; integration tests match R4g (issues 1937, 1938, 1931) 2026-10-24 05:07:11 +00:00

jschoubben added 2 commits 2026-10-24 05:16:28 +00:00

Have anchor report on tier 1 inside its send in the rollout test, so its report never races the silence bound (issue 1938, met twice) ... 5f2da0af69

Under a loaded run the report came 5.7 s and then more than 10 s after the send, and anchor was named silent. Now the
fake send makes anchor's report as soon as tier 1 is recorded, and the test waits for it; laptop never reports and is
named silent.

Merge remote-tracking branch 'origin/fix/1931-reconnects-verb' into fix/1937-done-key-blobs c06f75d083

jschoubben merged commit 1b0fd8b442 into main 2026-10-24 10:35:15 +00:00

jschoubben deleted branch fix/1937-done-key-blobs 2026-10-24 10:35:16 +00:00

jschoubben referenced this issue from a commit 2026-10-24 11:30:44 +00:00

Merge main into feat/r4g-finish (#453)

mesh-tools referenced this pull request 2026-10-24 11:49:34 +00:00

R4g finish: reads-outside markers (issue 1935) #454

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

mesh/ready

Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#453