The node-engine fetches every artifact before it replaces anything, and a failed apply no longer claims the old composition runs (issue 1966, recovery) #215

2026-11-09T00:16:19Z

mesh-tools commented

2026-11-09 00:16:19 +00:00

Recovery fix (recovery-fixes-merge-and-tell; the operator: "you can deliver stuff yourself").

Symptom. 2026-11-08 23:25 UTC: applying distribution 83cd1a3d replaced the artifact store's container (replaced) and then failed to fetch distribution.program from the store it had just cut off. The store was unreachable for 33 minutes, and every build's store step failed. A put back to the old build changed nothing: the engine still said the old composition ran, so it judged the put unchanged. Recovered by the operator's hand act (issue 1966 note 109).

Fix.

22a37b75: fetchFirst fetches and checks every archive and process of an apply right after pullFirst, before any resource is touched. A fetch that fails stops the module whole: nothing is replaced and its orphans are kept. applyArchive and applyProcess use the bytes already fetched.
9f6801a6: every failed apply marks what runs of the module as failed (ranFailed), so a want of the composition that ran is applied again. The composition tool shows runs.phase failed.

Tests. "an artifact that cannot be fetched replaces nothing of its module" (replace_image_test.go) and "1966: after a failed apply, the composition that ran is applied again when wanted" (agent_test.go). Both failed first. go vet (plain and integration), go test ./..., gofmt and the packaging scripts pass.

Costs, said.

A data module put back after a failed apply is backed up again before it applies.
A module's fetched artifacts are held in memory until their turn (up to 512 MiB each).

Noted, not fixed. A failed image pull ends a converged node's whole apply at the first resource, because pullFirst failures feed the guard check.

Delivery. The node-engine on all four machines, ace first. Then distribution's put on novox is released, so the store's move to 83cd1a3d fetches its program before replacing the container.

Undo. Revert this merge.

https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

Recovery fix (recovery-fixes-merge-and-tell; the operator: "you can deliver stuff yourself"). **Symptom.** 2026-11-08 23:25 UTC: applying distribution 83cd1a3d replaced the artifact store's container (`replaced`) and then failed to fetch distribution.program from the store it had just cut off. The store was unreachable for 33 minutes, and every build's store step failed. A put back to the old build changed nothing: the engine still said the old composition ran, so it judged the put `unchanged`. Recovered by the operator's hand act (issue 1966 note 109). **Fix.** - 22a37b75: `fetchFirst` fetches and checks every archive and process of an apply right after `pullFirst`, before any resource is touched. A fetch that fails stops the module whole: nothing is replaced and its orphans are kept. `applyArchive` and `applyProcess` use the bytes already fetched. - 9f6801a6: every failed apply marks what runs of the module as failed (`ranFailed`), so a want of the composition that ran is applied again. The composition tool shows runs.phase failed. **Tests.** "an artifact that cannot be fetched replaces nothing of its module" (replace_image_test.go) and "1966: after a failed apply, the composition that ran is applied again when wanted" (agent_test.go). Both failed first. `go vet` (plain and integration), `go test ./...`, gofmt and the packaging scripts pass. **Costs, said.** - A data module put back after a failed apply is backed up again before it applies. - A module's fetched artifacts are held in memory until their turn (up to 512 MiB each). **Noted, not fixed.** A failed image pull ends a converged node's whole apply at the first resource, because `pullFirst` failures feed the guard check. **Delivery.** The node-engine on all four machines, ace first. Then distribution's put on novox is released, so the store's move to 83cd1a3d fetches its program before replacing the container. **Undo.** Revert this merge. https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

mesh-tools added 2 commits 2026-11-09 00:16:20 +00:00

Fetch every archive and process of an apply before anything is replaced (issue 1966, recovery) 22a37b758c

On 2026-11-08 23:25 UTC the node-engine replaced distribution's store container
and only then fetched distribution's program from the store it had just cut off:
the module was left half applied and the artifact store unreachable from every
other machine. An apply now fetches and checks every archive and process first,
as it already pulls a module's images first; one that cannot be had gates its
module, so none of its resources is replaced and none of its orphans removed.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

A module whose apply failed is no longer said to run what it ran (issue 1966, recovery)

mesh/delivery merged: source.main moved to novox/mesh-host@9b55bcb12743

Details

9f6801a6c5

On 2026-11-08 a failed apply of distribution's new build had already replaced
its store container, yet the node-engine still held the old build as running,
so the put back to it at 23:47 UTC was judged unchanged and applied nothing.
A failed apply now marks what runs of the module failed, and a want of the
composition that ran before is applied again.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5

mesh-tools commented

2026-11-09 00:16:22 +00:00

Delivery novox/mesh-host@9f6801a6c563 — merged since 2026-11-09T00:16:28Z

Source — source.main moved to novox/mesh-host@9b55bcb127. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-11-09 00:16 (new) → proposed (announced): novox/mesh-host#215's head announced
2026-11-09 00:16 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-11-09 00:16 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@9f6801a6c563` — **merged** since 2026-11-09T00:16:28Z **Source** — source.main moved to novox/mesh-host@9b55bcb12743. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-11-09 00:16 (new) → proposed (announced): novox/mesh-host#215's head announced - 2026-11-09 00:16 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-11-09 00:16 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 9b55bcb127 into main

2026-11-09 00:16:26 +00:00

mesh-tools deleted branch fix/1966-fetch-before-replace

2026-11-09 00:16:26 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#215