Files
mesh-host/internal/upgrade/upgrade.go
T
jschoubben 5b7b280e3a The launcher supervises the host instead of exec'ing it
Jochen: "I thought we did not want to run the host under a systemd/openrc/init
loop, but instead had our own host-init program?" -- and that was right. I had
moved the give-up logic out of unit files and left RESTART in them, with the
launcher exec'ing the host and disappearing. So init still decided when the
host came back, which is the arrangement 0061 exists to remove.

The launcher now stays and supervises: starts the host as a child, waits,
decides. Init is asked for one thing, run this at boot. There is an OpenRC
script beside the systemd unit now, four lines each, which is the point --
a second init is transcription rather than a port.

The cost of not exec'ing is signals. A supervisor that exits while its child
runs leaves the host to be killed rather than to stop, and an apply interrupted
that way is the half-configured machine this project is about. So SIGTERM is
trapped, passed down, and waited on.

Two bugs, both found by the tests rather than by review:

A clean exit was counted as a failure. The host exits cleanly to stand aside
for a new binary after an upgrade (0057), so a host that upgraded itself three
times rolled itself back having worked perfectly every time. The counter now
counts CONSECUTIVE FAILURES, incremented after the wait rather than before the
start.

And when rolling back I reset the counter file but not the variable, so the
next failure counted from the old value -- the rolled-back version got one
attempt instead of three.

Also: the host now clears the counter when it completes a reconcile, at the
same moment it records known-good and for the same reason. Without it the count
only climbs, and a node up for months rolls itself back on its third ordinary
restart -- a healthy machine undone by its own recovery.

One test expectation was tightened rather than fixed: "resets the counter after
rolling back" asserted exactly 0, which was true only under the old
count-before-start semantics. It now asserts the property -- below the limit --
since 1 is correct after a rollback plus one failure.

32 launcher tests, all confirmed to bite.
2026-08-28 00:37:52 +02:00

160 lines
5.6 KiB
Go

// Package upgrade is how the host survives replacing itself.
//
// novox/hq ADR 0057 and ADR 0059. Two facts, and neither is the host judging its own health:
//
// - whether the executable this process started from has been replaced on disk, which is how
// it knows to stand aside for a new one;
// - which version last got as far as a completed reconcile, which is what a rollback outside
// this binary reads when this binary will not start.
//
// The second is written for a reader that is not the host. A binary that cannot start cannot be
// its own recovery, so what it leaves behind has to be plain enough for a shell script.
package upgrade
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
)
// Files the launcher reads and this binary writes. Next to the store, because they are node
// state of exactly the same kind.
const (
KnownGoodName = "known-good"
AttemptsName = "start-attempts"
)
// Self is the executable this process started from, remembered.
//
// Identity is taken once, at start, and compared later. The obvious alternative — asking
// /proc/self/exe whether it is marked deleted — was tried and is worse in two ways: it is Linux
// procfs behaviour rather than a fact about files, and it catches only *unlink*, so a binary
// swapped by rename onto the same path reads as untouched. Remembering what we started from
// needs no special filesystem and misses neither case.
type Self struct {
path string
info os.FileInfo
}
// Current captures the running executable's identity.
//
// path is what os.Executable() returned; a test passes one it can manipulate, because the
// boundary being tested is the filesystem and a fake would assert that the fake behaves as
// expected (novox/hq ADR 0034).
func Current(path string) (Self, error) {
info, err := os.Stat(path)
if err != nil {
return Self{}, fmt.Errorf(
"cannot stat %s, so this host cannot tell whether it is later replaced: %w", path, err)
}
return Self{path: path, info: info}, nil
}
// Path is where the executable was when this process started.
func (s Self) Path() string { return s.path }
// Replaced reports whether a different file is at that path now, or none.
//
// Never a silent false: a host that cannot read its own image says so rather than assuming it is
// current, which is the shape of every fault this repository catalogues.
func (s Self) Replaced() (bool, error) {
if s.info == nil {
return false, errors.New("this host never captured its own identity, so it cannot tell " +
"whether it has been replaced")
}
now, err := os.Stat(s.path)
if errors.Is(err, os.ErrNotExist) {
// Removed rather than upgraded. Still not what is running, and saying "unchanged"
// would leave the host claiming a version that is no longer installed.
return true, nil
}
if err != nil {
return false, err
}
return !os.SameFile(s.info, now), nil
}
// KnownGoodPath is where the marker lives, given where the store lives.
func KnownGoodPath(statePath string) string {
return filepath.Join(filepath.Dir(statePath), KnownGoodName)
}
// AttemptsPath is where the launcher counts starts that have not yet worked.
func AttemptsPath(statePath string) string {
return filepath.Join(filepath.Dir(statePath), AttemptsName)
}
// ClearAttempts tells the launcher this start worked.
//
// Written at the same moment as known-good and for the same reason: a completed reconcile is
// the evidence, and it is the only evidence either of them has. Without this the counter only
// ever climbs, so a node that has been up for months rolls itself back on its third ordinary
// restart — a healthy machine undone by its own recovery.
func ClearAttempts(path string) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
return os.WriteFile(path, []byte("0\n"), 0o644)
}
// RecordKnownGood marks a version as one that started and completed a reconcile.
//
// Written atomically and as one bare line. The reader is a shell script running on a machine
// where the host is failing to start, so the format is the least it can be: no JSON, no
// escaping, nothing that needs a parser to be present and working.
func RecordKnownGood(path, version string) error {
if strings.TrimSpace(version) == "" {
return errors.New("refusing to record an empty version as known-good: a rollback " +
"reading it would install nothing and report success")
}
if strings.ContainsAny(version, "\n\r") {
return fmt.Errorf("refusing to record %q as known-good: it must be one line", version)
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(path), ".known-good-*")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
if _, err := fmt.Fprintln(tmp, version); err != nil {
tmp.Close()
return err
}
if err := tmp.Sync(); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
if err := os.Chmod(tmp.Name(), 0o644); err != nil {
return err
}
return os.Rename(tmp.Name(), path)
}
// ReadKnownGood returns the recorded version, or "" if there has never been one.
//
// Absence is not an error. A machine whose host has never completed a reconcile has no version
// to go back to, and that is a real state rather than a fault: the node was never working, so
// the failure belongs to the installation and not to an upgrade. A rollback that guessed here
// would become a second fault.
func ReadKnownGood(path string) (string, error) {
raw, err := os.ReadFile(path)
if errors.Is(err, os.ErrNotExist) {
return "", nil
}
if err != nil {
return "", err
}
return strings.TrimSpace(string(raw)), nil
}