Resume a backup the node agent's own stop interrupted, and keep what a held module runs

On novox (2026-10-31) postgres's repeat backup was interrupted by the node-engine's
restart: its held have named no build, so the node's own composition took postgres in
and was refused for its secret reader, and nothing ran the backup again. Issue 1289
(R4c, its delivery held).

- A held have names what runs: read from the module's have before backing-up when
  the agent has no running entry, as the not-composed mark does.
- A have in any phase but running that names nothing is not run inside the node's own.
- A backup interrupted by the agent's stop is no failure: the next start backs the
  module up again and applies while the want asks a backup. A real failure stays
  held for the operator's answer.

Quick fix: a new backup ask by hand. Thorough, chosen: the fault fixed where it lives
in the node agent, so every data module's interrupted backup resumes by itself.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
This commit is contained in:
2026-10-31 16:33:49 +01:00
parent a4713e2d28
commit 39365c62c9
6 changed files with 94 additions and 5 deletions
+37
View File
@@ -590,6 +590,38 @@ func TestADataModuleIsBackedUpFirstAndHeldWhenItFails(t *testing.T) {
h := b.haves.last("db")
return h.Phase == string(delivery.PhaseRunning) && h.FP == fp2 && h.Composed == dig2
})
// At a start: a backup the node agent's own stop interrupted (postgres on novox, 2026-10-31) is no failure, so it is
// backed up again and applied while the want asks a backup; a backup that failed stays held for the operator.
for _, row := range []struct {
detail, backup string
backups, applied int
phase delivery.Phase
}{
{BackupInterrupted, delivery.BackupAgain, 1, 1, delivery.PhaseRunning},
{BackupInterrupted, "", 1, 1, delivery.PhaseRunning},
{"restic: the repository is locked", delivery.BackupAgain, 0, 0, delivery.PhaseHeld},
} {
r := newBed(t)
r.comps.before(ck1, craw1)
r.comps.before(ck2, craw2)
wk, wraw := wantFor(t, "db", fp2, dig2, row.backup)
r.wants.before(wk, wraw)
r.haves.stored = map[string]Have{"db": {Module: "db", FP: fp1, Composed: dig1, Phase: string(delivery.PhaseHeld), Detail: row.detail}}
stopR := r.run()
if row.applied > 0 {
eventually(t, row.detail+": not applied after its backup", func() bool { return len(r.appliedList()) == row.applied })
}
time.Sleep(50 * time.Millisecond)
r.mu.Lock()
n := len(r.backups)
r.mu.Unlock()
h := r.haves.last("db")
if n != row.backups || len(r.appliedList()) != row.applied || h.Phase != string(row.phase) {
t.Errorf("%s, want backup %q: %d backups, applied %v, have %+v", row.detail, row.backup, n, r.appliedList(), h)
}
stopR()
}
}
// A data module with a good backup is applied after it; one that never ran has nothing to back up.
@@ -1640,6 +1672,11 @@ func testReconcileAppliesARunningModuleAgainAndLeavesTheHeldAndTheUnwantedAlone(
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied at the start: %v", got)
}
// Held, it keeps naming what runs, which this agent knows only from the have (postgres on novox, 2026-10-31): not
// the build it was to move to, and never nothing.
if h := b.haves.last("db"); h.FP != fpOf("0") || h.Composed != "sha256:"+strings.Repeat("0", 64) {
t.Fatalf("the held have does not name what runs: %+v", h)
}
b.ag.Reconcile("db", "ghost")
time.Sleep(50 * time.Millisecond)