Resume a backup the node agent's own stop interrupted, and keep what a held module runs
On novox (2026-10-31) postgres's repeat backup was interrupted by the node-engine's restart: its held have named no build, so the node's own composition took postgres in and was refused for its secret reader, and nothing ran the backup again. Issue 1289 (R4c, its delivery held). - A held have names what runs: read from the module's have before backing-up when the agent has no running entry, as the not-composed mark does. - A have in any phase but running that names nothing is not run inside the node's own. - A backup interrupted by the agent's stop is no failure: the next start backs the module up again and applies while the want asks a backup. A real failure stays held for the operator's answer. Quick fix: a new backup ask by hand. Thorough, chosen: the fault fixed where it lives in the node agent, so every data module's interrupted backup resumes by itself. Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
This commit is contained in:
+35
-3
@@ -84,9 +84,15 @@ type Result struct {
|
||||
}
|
||||
|
||||
// Backup is the answer of the backup asked before a module moves.
|
||||
// BackupInterrupted is the words of a backup the node agent itself stopped waiting for (its context ended): no failure, so
|
||||
// the next start backs the module up again rather than holding it for the operator.
|
||||
const BackupInterrupted = "the node agent was stopped while the backup ran"
|
||||
|
||||
type Backup struct {
|
||||
Done bool
|
||||
Words string
|
||||
// Interrupted says the node agent stopped waiting for it (its context ended): not a failure of the backup.
|
||||
Interrupted bool
|
||||
}
|
||||
|
||||
// Env is everything the agent needs of the node.
|
||||
@@ -1325,6 +1331,15 @@ func (a *Agent) baseline(ctx context.Context, module string, w wantNow, comp com
|
||||
a.mu.Lock()
|
||||
a.prior[module] = true
|
||||
a.mu.Unlock()
|
||||
if h.Phase == string(delivery.PhaseHeld) && h.Detail == BackupInterrupted && w.Want.Backup != delivery.BackupSkip {
|
||||
// Held only because this node agent stopped while the backup ran (postgres on novox, 2026-10-31): no
|
||||
// failure and nothing for the operator to answer. It is backed up again and applied (a.prior makes the
|
||||
// apply back it up first); the want still asking a backup is all the answer needed.
|
||||
a.mu.Lock()
|
||||
a.sayOnce("interrupted:"+module, fmt.Sprintf("%s's backup was interrupted by this node agent's stop: it is backed up again", module))
|
||||
a.mu.Unlock()
|
||||
return false
|
||||
}
|
||||
if h.Phase == string(delivery.PhaseHeld) {
|
||||
// Held for a failed backup, and the operator's answer is still to come: the hold is kept, nothing is
|
||||
// backed up again or applied, and the next revision of the want is the answer (ADR 0568 §4).
|
||||
@@ -1414,16 +1429,33 @@ func (a *Agent) apply(ctx context.Context, module string, w wantNow, comp compNo
|
||||
// A data module is backed up before a new composition of it moves (ADR 0429 §14), unless the want says to go on
|
||||
// without one (ADR 0539 §16).
|
||||
if comp.C.Data && ranBefore && a.env.Backup != nil && w.Want.Backup != delivery.BackupSkip {
|
||||
// What runs, read before the have says backing-up: a module known only from its have (a.prior) has no running
|
||||
// entry, and its held have kept naming nothing (postgres on novox, 2026-10-31), as the not-composed mark keeps it.
|
||||
a.mu.Lock()
|
||||
prev, runs := a.running[module]
|
||||
a.mu.Unlock()
|
||||
if !runs {
|
||||
if h, found, _ := a.readHave(ctx, module); found {
|
||||
prev = delivery.Have{FP: h.FP, Composed: h.Composed}
|
||||
}
|
||||
}
|
||||
a.writeHave(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseBackingUp)})
|
||||
if got := a.env.Backup(ctx, module); !got.Done {
|
||||
a.env.Say(fmt.Sprintf("%s is held: its backup before %s failed: %s", module, fp, got.Words))
|
||||
words := got.Words
|
||||
if got.Interrupted || ctx.Err() != nil {
|
||||
// The node agent stopped while it waited: no failure of the backup, so the next start backs it up again
|
||||
// (baseline) rather than holding it for the operator.
|
||||
words = BackupInterrupted
|
||||
a.env.Say(fmt.Sprintf("%s waits: %s; it is backed up again at the next start", module, words))
|
||||
} else {
|
||||
a.env.Say(fmt.Sprintf("%s is held: its backup before %s failed: %s", module, fp, words))
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev, Held: true, AnsweredRev: w.Rev}
|
||||
prev := a.running[module]
|
||||
a.mu.Unlock()
|
||||
// It keeps running what it has: the have says what it holds and why it waits.
|
||||
a.writeHave(ctx, Have{Module: module, FP: prev.FP, Composed: prev.Composed, Phase: string(delivery.PhaseHeld),
|
||||
Detail: got.Words})
|
||||
Detail: words})
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user