jschoubben
057f34f924
ADR 0061. Recovery was the most systemd-specific part of the host, and it is the part that must work on a machine where nothing else does -- which made unit-file syntax a poor place for it, because syntax cannot be tested and the one time it runs is the one time nobody can afford it wrong. So StartLimitBurst and OnFailure move into a launcher script that init starts instead of the host. The unit drops to start-at-boot and restart-on-exit, which OpenRC, runit, s6 and an Android init.rc can all express. Everything 0059 decided is kept: two watchdogs, roll back once, recovery is local, the rollback shares no code with the host. The counter is the whole mechanism, so it is what the tests are mostly about. Three real problems came out of writing them: A counter file holding "1 2" became "12" -- `tr -d [:space:]` concatenates rather than rejecting -- which is past the limit, so a HEALTHY node rolled itself back. Now it reads the first field and insists on a plain integer. The corrupt-counter test used "not-a-number", which shell arithmetic happens to evaluate to 0, so it passed with the guard removed and proved nothing. Replaced with values that discriminate: "5x" errors under set -e and kills the launcher, and "0x10" is read as HEX 16 -- past the limit, so again a healthy node rolls back. And the test harness itself was wrong. With `set -e` and a bare launcher call, removing a guard killed the script at the first corrupt case and silently skipped everything after -- reporting a full pass over tests that never ran. Every launcher call now records its failure instead of aborting. Same class as the placebo assertion found last time, and the reason to keep injecting faults rather than trusting green. Both scripts run in `make check`. 27 launcher tests, 9 rollback tests, all confirmed to bite.
66 lines
2.6 KiB
Bash
Executable File
66 lines
2.6 KiB
Bash
Executable File
#!/bin/sh
|
|
# Put the host back on the last version that worked.
|
|
#
|
|
# novox/hq ADR 0059. This runs when nox-mesh-host will not start, so it shares no code with it
|
|
# and calls none of it: a binary that cannot start cannot be its own recovery. POSIX sh, no
|
|
# bashisms, nothing that has to be installed.
|
|
#
|
|
# It is deliberately dull. Everything it does is one of: read a file, run the package manager,
|
|
# ask the service manager to try again.
|
|
set -eu
|
|
|
|
STATE_DIR="${MESH_HOST_STATE_DIR:-/var/lib/mesh-host}"
|
|
PKG_CACHE="${MESH_HOST_PKG_CACHE:-/var/cache/pacman/pkg}"
|
|
PACKAGE="${MESH_HOST_PACKAGE:-nox-mesh-host}"
|
|
|
|
KNOWN_GOOD="$STATE_DIR/known-good"
|
|
ATTEMPTED="$STATE_DIR/rollback-attempted"
|
|
|
|
say() { echo "nox-mesh-host-rollback: $*" >&2; }
|
|
|
|
# Roll back once. A second failure is a different diagnosis: the previously working binary also
|
|
# does not run, so the binary is not the problem — the machine is. Rolling back again would flap
|
|
# between two versions forever and bury the actual cause under a loop.
|
|
if [ -e "$ATTEMPTED" ]; then
|
|
say "already rolled back once, to $(cat "$ATTEMPTED" 2>/dev/null || echo unknown)."
|
|
say "the previous version also failed to start, so this is the machine and not the binary."
|
|
say "not rolling back again. this node needs a person."
|
|
exit 0
|
|
fi
|
|
|
|
# A machine whose host never completed a reconcile has no version to go back to. That is a real
|
|
# state rather than a fault: the node was never working, so the failure belongs to the
|
|
# installation. Guessing a version here is how a recovery becomes a second fault.
|
|
if [ ! -s "$KNOWN_GOOD" ]; then
|
|
say "no known-good version recorded — this host has never completed a reconcile."
|
|
say "there is nothing to roll back to. this is an installation failure, not an upgrade one."
|
|
exit 0
|
|
fi
|
|
|
|
VERSION="$(tr -d '[:space:]' < "$KNOWN_GOOD")"
|
|
if [ -z "$VERSION" ]; then
|
|
say "known-good is empty. refusing to guess."
|
|
exit 0
|
|
fi
|
|
|
|
PKG="$(ls "$PKG_CACHE"/"$PACKAGE"-"$VERSION"-*.pkg.tar.* 2>/dev/null | head -n 1 || true)"
|
|
if [ -z "$PKG" ]; then
|
|
say "known-good is $VERSION and no package for it is in $PKG_CACHE."
|
|
say "the cache was cleaned, or that version was never installed from here."
|
|
say "cannot roll back. this node needs a person."
|
|
exit 1
|
|
fi
|
|
|
|
say "rolling back to $VERSION ($PKG)"
|
|
printf '%s\n' "$VERSION" > "$ATTEMPTED"
|
|
|
|
if ! pacman -U --noconfirm "$PKG"; then
|
|
say "the package manager refused to install $PKG."
|
|
exit 1
|
|
fi
|
|
|
|
# Deliberately does NOT start anything. The launcher called this and will exec the host next,
|
|
# so starting it here would run two. novox/hq ADR 0061 moved that responsibility; this script
|
|
# installs a version and says so, and nothing else.
|
|
say "rolled back to $VERSION. the launcher will start it."
|