The host delivers its own successor, and versions live side by side

The supervision was already right: a clean exit means the host stood aside, and
the launcher's next turn runs what is on disk. Two things made it dead code —
nothing told the running host a successor was waiting, and the rollback resolved
its known-good version through pacman, which no machine here uses and which two
of three operating systems do not have.

Keeping a version rather than a path was the clue. Versions now live in
directories named for them:

- the launcher picks the newest delivered one every time round the loop, or the
  one a rollback pinned, or the host placed by hand when nothing is delivered;
- the running host stands aside between reconciles, never inside one, by exiting
  cleanly — and returns nil so the launcher does not count it as a crash;
- a completed reconcile retires what is older than the predecessor, keeping the
  predecessor because that is what a rollback starts, and never the running one;
- rollback pins the predecessor instead of reinstalling a package: no package
  manager, no cache anyone may clean, same script on every operating system;
- the report says which host version produced it, so 'behind' is answerable.

Newest is when it arrived, never how the name sorts: '1.10' orders before '1.9',
and ordering by name would start an older host and call it an upgrade.

novox/hq ADR 0141. The delivery half — a module carrying the next host — follows;
until then nothing delivers a version and every machine takes the fallback, which
is what it does today.

packaging/nox-mesh-host-rollback

@@ -1,20 +1,29 @@
 #!/bin/sh
 # Put the host back on the last version that worked.
 #
-# novox/hq ADR 0005. This runs when nox-mesh-host will not start, so it shares no code with it
-# and calls none of it: a binary that cannot start cannot be its own recovery. POSIX sh, no
+# novox/hq ADR 0005 and ADR 0141. This runs when nox-mesh-host will not start, so it shares no code
+# with it and calls none of it: a binary that cannot start cannot be its own recovery. POSIX sh, no
 # bashisms, nothing that has to be installed.
 #
-# It is deliberately dull. Everything it does is one of: read a file, run the package manager,
-# ask the service manager to try again.
+# It is deliberately dull. Everything it does is one of: read a file, look at a directory, write a
+# file.
+#
+# **It used to reinstall a package.** It read the known-good version and asked one operating system's
+# package manager for it, out of that package manager's cache. Two things were wrong with that. No
+# machine in this mesh had the host installed as a package, so the recovery could not run on any of
+# them; and the host is built per operating system (ADR 0005), so a recovery written in one package
+# manager's terms could not run on two of the three. Versions now live side by side in directories
+# named for them, so going back is choosing a directory — which is the same on every machine.
 set -eu
 
 STATE_DIR="${MESH_HOST_STATE_DIR:-/var/lib/mesh-host}"
-PKG_CACHE="${MESH_HOST_PKG_CACHE:-/var/cache/pacman/pkg}"
-PACKAGE="${MESH_HOST_PACKAGE:-nox-mesh-host}"
+LIBEXEC="${MESH_HOST_LIBEXEC:-/usr/lib/nox-mesh-host}"
+VERSIONS="$LIBEXEC/versions"
+BINARY="nox-mesh-host"
 
 KNOWN_GOOD="$STATE_DIR/known-good"
 ATTEMPTED="$STATE_DIR/rollback-attempted"
+PINNED="$STATE_DIR/rollback-pinned"
 
 say() { echo "nox-mesh-host-rollback: $*" >&2; }
 
@@ -43,23 +52,24 @@ if [ -z "$VERSION" ]; then
 	exit 0
 fi
 
-PKG="$(ls "$PKG_CACHE"/"$PACKAGE"-"$VERSION"-*.pkg.tar.* 2>/dev/null | head -n 1 || true)"
-if [ -z "$PKG" ]; then
-	say "known-good is $VERSION and no package for it is in $PKG_CACHE."
-	say "the cache was cleaned, or that version was never installed from here."
+# The version that last worked may be the one that was placed by hand, which is not delivered and has
+# no directory. Nothing to choose, and saying so is better than pinning a version that is not there —
+# the launcher would ignore the pin and start the newest again, which is the binary that is failing.
+if [ ! -x "$VERSIONS/$VERSION/$BINARY" ]; then
+	say "known-good is $VERSION and no such version is delivered under $VERSIONS."
+	say "it was retired, or that host was placed by hand and never delivered."
 	say "cannot roll back. this node needs a person."
 	exit 1
 fi
 
-say "rolling back to $VERSION ($PKG)"
+say "rolling back to $VERSION ($VERSIONS/$VERSION/$BINARY)"
 printf '%s\n' "$VERSION" > "$ATTEMPTED"
 
-if ! pacman -U --noconfirm "$PKG"; then
-	say "the package manager refused to install $PKG."
-	exit 1
-fi
+# The pin is what stops the launcher starting the newest again. Written last, so a failure above
+# leaves the machine choosing for itself rather than pinned to something this script did not verify.
+printf '%s\n' "$VERSION" > "$PINNED"
 
-# Deliberately does NOT start anything. The launcher called this and will exec the host next,
-# so starting it here would run two. novox/hq ADR 0005 moved that responsibility; this script
-# installs a version and says so, and nothing else.
-say "rolled back to $VERSION. the launcher will start it."
+# Deliberately does NOT start anything. The launcher called this and will run the host next, so
+# starting it here would run two. novox/hq ADR 0005 moved that responsibility; this script chooses a
+# version and says so, and nothing else.
+say "pinned $VERSION. the launcher will start it."