Builder: announce a done build again, so the controller composes a module whose composition and grant were lost (recovery of novox and ace, 2026-10-29) #111

2026-10-29T17:54:57Z

mesh-tools commented

2026-10-29 17:54:57 +00:00

Why

Recovery, 2026-10-29: photos and link2pay on novox, snake, txt-game and f1-circuits on ace have been down since 10:01 CET. Their builds are done; the controller's composition and grant for them are gone, so the assigner's rule 6 writes no want. The operator re-registered all five at the controller's terminal at 18:45 CET (now registered at their src2 fingerprints), but nothing composed them since: the controller's take-in composes only when a write of a build key reaches its consumer, and no tool writes a done key again (builder_build_again refuses done keys; indexer_take_in refuses a module it holds).

What changes

builder_announce_again (module, fp, why) rewrites a done key with its own bytes as a new revision of the same key (content, artifacts, manifest identical), by compare-and-set at the revision read; refuses a key that is not done (failed: builder_build_again; building/queued: builder_ask_again), absent, or with a bad fingerprint; logs who and why; goes through the reasks state like builder_build_again.

Quick fix or thorough fix

Quick (this change): a hand tool to make the controller's take-in run again for one build.
Thorough (not here): ADR 0547 and design 47, the node agent composes its own and the assigner stops waiting on controller compositions, so a lost composition cannot block a want. Chosen quick because five live modules are down now.

How it is checked

One table-driven unit test (5 cases), internal/build/announce_test.go; the manifest test lists the tool. gofmt, go vet (both tags) and go test pass; 95 unit tests, 0 integration (budget 500/50).
The reviewer read mesh-controller's take-in (composed_keys.go, take_in_consumer.go at e04846e9): the consumer hears any put on the builder bucket; for a module registered at its fingerprint, the batch asks taken(), which is false, and composes and writes the grant.
Live after delivery: builder_announce_again for the five; the controller's journal says each composed with its grant; the assigner writes their wants; the five containers run.

Reviewed-by: rev-announce@shanks at ba4bfea0a4

## Why Recovery, 2026-10-29: photos and link2pay on novox, snake, txt-game and f1-circuits on ace have been down since 10:01 CET. Their builds are done; the controller's composition and grant for them are gone, so the assigner's rule 6 writes no want. The operator re-registered all five at the controller's terminal at 18:45 CET (now registered at their src2 fingerprints), but nothing composed them since: the controller's take-in composes only when a write of a build key reaches its consumer, and no tool writes a done key again (builder_build_again refuses done keys; indexer_take_in refuses a module it holds). ## What changes `builder_announce_again` (module, fp, why) rewrites a done key with its own bytes as a new revision of the same key (content, artifacts, manifest identical), by compare-and-set at the revision read; refuses a key that is not done (failed: builder_build_again; building/queued: builder_ask_again), absent, or with a bad fingerprint; logs who and why; goes through the reasks state like builder_build_again. ## Quick fix or thorough fix - Quick (this change): a hand tool to make the controller's take-in run again for one build. - Thorough (not here): ADR 0547 and design 47, the node agent composes its own and the assigner stops waiting on controller compositions, so a lost composition cannot block a want. Chosen quick because five live modules are down now. ## How it is checked - One table-driven unit test (5 cases), internal/build/announce_test.go; the manifest test lists the tool. gofmt, go vet (both tags) and go test pass; 95 unit tests, 0 integration (budget 500/50). - The reviewer read mesh-controller's take-in (composed_keys.go, take_in_consumer.go at e04846e9): the consumer hears any put on the builder bucket; for a module registered at its fingerprint, the batch asks taken(), which is false, and composes and writes the grant. - Live after delivery: builder_announce_again for the five; the controller's journal says each composed with its grant; the assigner writes their wants; the five containers run. Reviewed-by: rev-announce@shanks at ba4bfea0a407db41c5a80146a7130f127ed63f2c

mesh-tools added 2 commits 2026-10-29 17:54:57 +00:00

Add builder_announce_again so a done build can be taken in and composed again fb7ad5cd73

Recovery of photos and link2pay on novox, 2026-10-29: their composition and grant were lost and no tool wrote a done key again.
Quick fix: rewrite the done key unchanged as a new revision. Thorough fix: ADR 0547, the node agent composes its own and the
assigner stops waiting on controller compositions (design 47).

Remove the PR description draft from the branch; it is kept outside the repository

mesh/check checking ba4bfea0a407 on 883d976b5f00

mesh/delivery merged: source.main moved to novox/mesh-delivery@883d976b5f00

Details

ba4bfea0a4

mesh-tools commented

2026-10-29 17:55:01 +00:00

Delivery novox/mesh-delivery@ba4bfea0a407 — merged since 2026-10-29T19:47:59Z

Source — source.main moved to novox/mesh-delivery@883d976b5f. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-29 17:55 (new) → proposed (announced): novox/mesh-delivery#111's head announced
2026-10-29 17:55 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-29 19:47 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-delivery@ba4bfea0a407` — **merged** since 2026-10-29T19:47:59Z **Source** — source.main moved to novox/mesh-delivery@883d976b5f00. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-29 17:55 (new) → proposed (announced): novox/mesh-delivery#111's head announced - 2026-10-29 17:55 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-29 19:47 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 883d976b5f into main

2026-10-29 19:47:58 +00:00

mesh-tools deleted branch fix/announce-done-again

2026-10-29 19:47:58 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-delivery#111