Node-engine: a stop never cuts a save off, an archive's unpack is recorded at once, and an answer the want carries is acted on at start (issue 1585) #207

2026-11-04T09:55:04Z

mesh-tools commented

2026-11-04 09:55:04 +00:00

Issue 1585 (notes 45-48). On novox the node-engine lost the records of nats's two archives (about 260 leftover temporary state saves), so a new nats build was refused, and after a restart the operator's "Back up again" answer was never acted on (nats held since 10:04).

Cause (read in the code): a stop ended the process while an apply ran detached (applyContext WithoutCancel), so its save was cut between write and rename.

84a14b0: what an archive unpacked is recorded as soon as it is unpacked.
ee0f46b: one state writer (write, fsync, rename, directory fsync); the stop waits up to 60 s for the apply in flight and no save starts after it; leftover temporary saves are removed at start and said; a save that drops a record nothing forgot says so loudly.
768f637: at start, a module held for a failed backup whose want already carries the answer (again: back up then apply; skip: apply) is acted on.
5dc3a8e (review round 1): the unpack record lives in the engine's own state directory (0600, directory 0700), never beside the archive; a record naming a non-local path, a wrong parent or another target is refused; it is removed with the archive. The trial launcher's grace is 75 s (was 20 s); the unit gets TimeoutStopSec=120 (reaches a node only with a package reinstall).

Quick fix: remove the leftover files and restore the records by hand (done once by the operator's decision on novox), answer the hold again. Thorough fix, chosen: the kind (the engine loses what it knew across a stop) is met twice today and can hit any module on any node.

Tests: rows and subtests in existing tests only (repository at 1772 unit test functions, unchanged); gofmt, go vet, go vet -tags integration, go test, packaging/launch_test.sh pass.

Live proof after delivery: nats on novox is backed up and applied at the new engine's first start (have phase running, no hold); no .state-*.json left in the node-engine's state directory on any node.

Reviewed-by: review-1585-state@shanks at 5dc3a8e17f

https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn

Issue 1585 (notes 45-48). On novox the node-engine lost the records of nats's two archives (about 260 leftover temporary state saves), so a new nats build was refused, and after a restart the operator's "Back up again" answer was never acted on (nats held since 10:04). Cause (read in the code): a stop ended the process while an apply ran detached (`applyContext` WithoutCancel), so its save was cut between write and rename. - 84a14b0: what an archive unpacked is recorded as soon as it is unpacked. - ee0f46b: one state writer (write, fsync, rename, directory fsync); the stop waits up to 60 s for the apply in flight and no save starts after it; leftover temporary saves are removed at start and said; a save that drops a record nothing forgot says so loudly. - 768f637: at start, a module held for a failed backup whose want already carries the answer (again: back up then apply; skip: apply) is acted on. - 5dc3a8e (review round 1): the unpack record lives in the engine's own state directory (0600, directory 0700), never beside the archive; a record naming a non-local path, a wrong parent or another target is refused; it is removed with the archive. The trial launcher's grace is 75 s (was 20 s); the unit gets TimeoutStopSec=120 (reaches a node only with a package reinstall). Quick fix: remove the leftover files and restore the records by hand (done once by the operator's decision on novox), answer the hold again. Thorough fix, chosen: the kind (the engine loses what it knew across a stop) is met twice today and can hit any module on any node. Tests: rows and subtests in existing tests only (repository at 1772 unit test functions, unchanged); gofmt, go vet, go vet -tags integration, go test, packaging/launch_test.sh pass. Live proof after delivery: nats on novox is backed up and applied at the new engine's first start (have phase running, no hold); no `.state-*.json` left in the node-engine's state directory on any node. Reviewed-by: review-1585-state@shanks at 5dc3a8e17f97094d5d31dee07af886883998a2ab https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn

mesh-tools added 4 commits 2026-11-04 09:55:04 +00:00

Write down what an archive unpacked as soon as it is unpacked, so a lost record never makes the host's own files foreign (issue 1585) 84a14b08d5

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn

Stop between applies and save the node's state as one writer, so a stop never cuts a save off and loses its records (issue 1585) ee0f46b78f

On novox the node-engine's state had lost the records of nats's two
archives, which only a temporary save file of 2026-10-24 still held, and
about 260 such files (~80 MB) lay beside it. The cause: an apply runs
detached from the engine's stop (applyContext, WithoutCancel), and
runLink returned, and the process ended, without waiting for it; a save
cut between its write and its rename left its temporary file and the
state before it, and whatever the apply did was never recorded.

Quick fix: remove the leftover files by hand and restore the records.
Thorough fix, chosen: the engine ends between applies whichever way it
ends (a stop, standing aside, a move to another bus): it waits up to 60 s
for the apply in flight, then lets no save start (store.Quiesce). Save is
one writer per process, flushes the directory after its rename, and says
loudly each record the state saved before held that this save drops and
nothing forgot (Forget, Release mark what is dropped on purpose, and no
longer write into an array an earlier copy reads). The engine's start
removes leftover temporary saves under the apply lock and says how many.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn

Act at the engine's start on a backup answer the want already carries, so an answer written before a restart is never swallowed (issue 1585) 768f637bd0

nats on novox, 2026-11-04: the operator answered "Back up again", the
engine restarted, and baseline() took the hold with AnsweredRev = the
want's current revision, so the answer in that very revision was never
acted on and compose refused "held for a failed backup".

Quick fix: answer once more (a new want revision). Thorough fix, chosen:
at baseline, a held module whose want carries again or skip is applied
(again backs it up first, skip goes on without one); a hold with no
answer stays held. The cost said in the code: an answer already acted on
before a stop, whose backup failed again, is tried once more per start.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn

Keep what an archive unpacked in the engine's own state directory, never beside the archive, and stop no slower than the host waits (issue 1585, review round 1)

mesh/delivery merged: source.main moved to novox/mesh-host@998d8f737008

Details

mesh/check being carried to 5dc3a8e17f97 on 998d8f737008 | from fabfdf04f78a

5dc3a8e17f

The journal of an unpack sat at <target>.unpacked.json, in a directory
the module's account may write: a forged one would make the root engine
remove or adopt any path it named. It now lives under
<statedir>/unpacked/<sha256 of the target>.json, mode 0600, names its
target, and is refused whole when an entry is not filepath.IsLocal or a
parent is not above the target. It goes when the archive is taken off,
whether its state record still said what it unpacked or not.

The launcher's stop grace for a host on trial rises from 20 s to 75 s and
the unit's TimeoutStopSec is 120 s (openrc the same), past the host's own
60 s wait for the apply in flight, so neither cuts a save off again.

Tests folded as rows into existing ones (the repository is over its unit
budget): the journal's rows, a forged entry and parent refused, the
journal gone with a record that lost its Unpacked; the test that passed
on main dropped; the stop test waits on its own line, not a sleep, and no
longer consumes the save it waits for.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn

mesh-tools commented

2026-11-04 09:55:07 +00:00

Delivery novox/mesh-host@5dc3a8e17f97 — merged since 2026-11-04T10:04:28Z

Source — source.main moved to novox/mesh-host@998d8f7370. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-11-04 09:55 (new) → proposed (announced): novox/mesh-host#207's head announced
2026-11-04 09:55 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-11-04 10:04 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@5dc3a8e17f97` — **merged** since 2026-11-04T10:04:28Z **Source** — source.main moved to novox/mesh-host@998d8f737008. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-11-04 09:55 (new) → proposed (announced): novox/mesh-host#207's head announced - 2026-11-04 09:55 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-11-04 10:04 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 998d8f7370 into main

2026-11-04 10:04:26 +00:00

mesh-tools deleted branch fix/archive-record-at-unpack

2026-11-04 10:04:26 +00:00

jschoubben referenced this issue from a commit

2026-11-04 10:42:35 +00:00

Act on a backup answer whatever runs, say why a module stays held, and let the engine's read tool show its want and its hold (issue 1585)

mesh-tools referenced this pull request

2026-11-04 10:42:41 +00:00

Node-engine: act on a backup answer whatever runs, say why a module stays held, and show the want and the hold in the composition verb (issue 1585) #208

Sign in to join this conversation.