Act at the engine's start on a backup answer the want already carries, so an answer written before a restart is never swallowed (issue 1585)
nats on novox, 2026-11-04: the operator answered "Back up again", the engine restarted, and baseline() took the hold with AnsweredRev = the want's current revision, so the answer in that very revision was never acted on and compose refused "held for a failed backup". Quick fix: answer once more (a new want revision). Thorough fix, chosen: at baseline, a held module whose want carries again or skip is applied (again backs it up first, skip goes on without one); a hold with no answer stays held. The cost said in the code: an answer already acted on before a stop, whose backup failed again, is tried once more per start. Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
This commit is contained in:
@@ -1375,6 +1375,19 @@ func (a *Agent) baseline(ctx context.Context, module string, w wantNow, comp com
|
|||||||
a.mu.Unlock()
|
a.mu.Unlock()
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
if h.Phase == string(delivery.PhaseHeld) && (w.Want.Backup == delivery.BackupAgain || w.Want.Backup == delivery.BackupSkip) {
|
||||||
|
// Held for a failed backup, and the want already carries the operator's answer, written before this engine
|
||||||
|
// started (nats on novox, 2026-11-04: "Back up again" answered, then the engine restarted and took the
|
||||||
|
// answer for the hold's own revision, so it was never acted on; issue 1585). The answer is acted on now:
|
||||||
|
// again backs it up and applies (a.prior makes the apply back it up first), skip applies without one.
|
||||||
|
// An answer the engine already acted on before its stop, and whose backup failed once more, is so acted
|
||||||
|
// on once more at this start: one backup more, never one skipped.
|
||||||
|
a.mu.Lock()
|
||||||
|
a.sayOnce("answered:"+module, fmt.Sprintf("%s is held for a failed backup, and its want carries the operator's "+
|
||||||
|
"answer %q: acted on at this start", module, w.Want.Backup))
|
||||||
|
a.mu.Unlock()
|
||||||
|
return false
|
||||||
|
}
|
||||||
if h.Phase == string(delivery.PhaseHeld) {
|
if h.Phase == string(delivery.PhaseHeld) {
|
||||||
// Held for a failed backup, and the operator's answer is still to come: the hold is kept, nothing is
|
// Held for a failed backup, and the operator's answer is still to come: the hold is kept, nothing is
|
||||||
// backed up again or applied, and the next revision of the want is the answer (ADR 0568 §4).
|
// backed up again or applied, and the next revision of the want is the answer (ADR 0568 §4).
|
||||||
|
|||||||
@@ -595,7 +595,8 @@ func TestADataModuleIsBackedUpFirstAndHeldWhenItFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// At a start: a backup the node agent's own stop interrupted (postgres on novox, 2026-10-31) is no failure, so it is
|
// At a start: a backup the node agent's own stop interrupted (postgres on novox, 2026-10-31) is no failure, so it is
|
||||||
// backed up again and applied while the want asks a backup; a backup that failed stays held for the operator.
|
// backed up again and applied while the want asks a backup; a backup that failed stays held for the operator's
|
||||||
|
// answer, and an answer the want already carries is acted on.
|
||||||
for _, row := range []struct {
|
for _, row := range []struct {
|
||||||
detail, backup string
|
detail, backup string
|
||||||
backups, applied int
|
backups, applied int
|
||||||
@@ -605,7 +606,11 @@ func TestADataModuleIsBackedUpFirstAndHeldWhenItFails(t *testing.T) {
|
|||||||
{BackupInterrupted, delivery.BackupAgain, 1, 1, delivery.PhaseRunning, false},
|
{BackupInterrupted, delivery.BackupAgain, 1, 1, delivery.PhaseRunning, false},
|
||||||
// novox's postgres: its held have, written before mesh-host #189, names nothing it runs.
|
// novox's postgres: its held have, written before mesh-host #189, names nothing it runs.
|
||||||
{BackupInterrupted, "", 1, 1, delivery.PhaseRunning, true},
|
{BackupInterrupted, "", 1, 1, delivery.PhaseRunning, true},
|
||||||
{"restic: the repository is locked", delivery.BackupAgain, 0, 0, delivery.PhaseHeld, false},
|
// A backup that failed, and the want carries the operator's answer, written before this start (nats on novox,
|
||||||
|
// 2026-11-04, issue 1585): again backs it up and applies, skip applies without one; no answer keeps the hold.
|
||||||
|
{"restic: the repository is locked", delivery.BackupAgain, 1, 1, delivery.PhaseRunning, false},
|
||||||
|
{"restic: the repository is locked", delivery.BackupSkip, 0, 1, delivery.PhaseRunning, false},
|
||||||
|
{"restic: the repository is locked", "", 0, 0, delivery.PhaseHeld, false},
|
||||||
} {
|
} {
|
||||||
r := newBed(t)
|
r := newBed(t)
|
||||||
r.comps.before(ck1, craw1)
|
r.comps.before(ck1, craw1)
|
||||||
|
|||||||
Reference in New Issue
Block a user