The launcher supervises the host instead of exec'ing it
Jochen: "I thought we did not want to run the host under a systemd/openrc/init loop, but instead had our own host-init program?" -- and that was right. I had moved the give-up logic out of unit files and left RESTART in them, with the launcher exec'ing the host and disappearing. So init still decided when the host came back, which is the arrangement 0061 exists to remove. The launcher now stays and supervises: starts the host as a child, waits, decides. Init is asked for one thing, run this at boot. There is an OpenRC script beside the systemd unit now, four lines each, which is the point -- a second init is transcription rather than a port. The cost of not exec'ing is signals. A supervisor that exits while its child runs leaves the host to be killed rather than to stop, and an apply interrupted that way is the half-configured machine this project is about. So SIGTERM is trapped, passed down, and waited on. Two bugs, both found by the tests rather than by review: A clean exit was counted as a failure. The host exits cleanly to stand aside for a new binary after an upgrade (0057), so a host that upgraded itself three times rolled itself back having worked perfectly every time. The counter now counts CONSECUTIVE FAILURES, incremented after the wait rather than before the start. And when rolling back I reset the counter file but not the variable, so the next failure counted from the old value -- the rolled-back version got one attempt instead of three. Also: the host now clears the counter when it completes a reconcile, at the same moment it records known-good and for the same reason. Without it the count only climbs, and a node up for months rolls itself back on its third ordinary restart -- a healthy machine undone by its own recovery. One test expectation was tightened rather than fixed: "resets the counter after rolling back" asserted exactly 0, which was true only under the old count-before-start semantics. It now asserts the property -- below the limit -- since 1 is correct after a rollback plus one failure. 32 launcher tests, all confirmed to bite.
This commit is contained in:
@@ -19,9 +19,12 @@ import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
// KnownGoodName is the file a rollback script reads. Next to the store, because it is node
|
||||
// Files the launcher reads and this binary writes. Next to the store, because they are node
|
||||
// state of exactly the same kind.
|
||||
const KnownGoodName = "known-good"
|
||||
const (
|
||||
KnownGoodName = "known-good"
|
||||
AttemptsName = "start-attempts"
|
||||
)
|
||||
|
||||
// Self is the executable this process started from, remembered.
|
||||
//
|
||||
@@ -80,6 +83,24 @@ func KnownGoodPath(statePath string) string {
|
||||
return filepath.Join(filepath.Dir(statePath), KnownGoodName)
|
||||
}
|
||||
|
||||
// AttemptsPath is where the launcher counts starts that have not yet worked.
|
||||
func AttemptsPath(statePath string) string {
|
||||
return filepath.Join(filepath.Dir(statePath), AttemptsName)
|
||||
}
|
||||
|
||||
// ClearAttempts tells the launcher this start worked.
|
||||
//
|
||||
// Written at the same moment as known-good and for the same reason: a completed reconcile is
|
||||
// the evidence, and it is the only evidence either of them has. Without this the counter only
|
||||
// ever climbs, so a node that has been up for months rolls itself back on its third ordinary
|
||||
// restart — a healthy machine undone by its own recovery.
|
||||
func ClearAttempts(path string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(path, []byte("0\n"), 0o644)
|
||||
}
|
||||
|
||||
// RecordKnownGood marks a version as one that started and completed a reconcile.
|
||||
//
|
||||
// Written atomically and as one bare line. The reader is a shell script running on a machine
|
||||
|
||||
Reference in New Issue
Block a user