The launcher supervises the host instead of exec'ing it

Jochen: "I thought we did not want to run the host under a systemd/openrc/init
loop, but instead had our own host-init program?" -- and that was right. I had
moved the give-up logic out of unit files and left RESTART in them, with the
launcher exec'ing the host and disappearing. So init still decided when the
host came back, which is the arrangement 0061 exists to remove.

The launcher now stays and supervises: starts the host as a child, waits,
decides. Init is asked for one thing, run this at boot. There is an OpenRC
script beside the systemd unit now, four lines each, which is the point --
a second init is transcription rather than a port.

The cost of not exec'ing is signals. A supervisor that exits while its child
runs leaves the host to be killed rather than to stop, and an apply interrupted
that way is the half-configured machine this project is about. So SIGTERM is
trapped, passed down, and waited on.

Two bugs, both found by the tests rather than by review:

A clean exit was counted as a failure. The host exits cleanly to stand aside
for a new binary after an upgrade (0057), so a host that upgraded itself three
times rolled itself back having worked perfectly every time. The counter now
counts CONSECUTIVE FAILURES, incremented after the wait rather than before the
start.

And when rolling back I reset the counter file but not the variable, so the
next failure counted from the old value -- the rolled-back version got one
attempt instead of three.

Also: the host now clears the counter when it completes a reconcile, at the
same moment it records known-good and for the same reason. Without it the count
only climbs, and a node up for months rolls itself back on its third ordinary
restart -- a healthy machine undone by its own recovery.

One test expectation was tightened rather than fixed: "resets the counter after
rolling back" asserted exactly 0, which was true only under the old
count-before-start semantics. It now asserts the property -- below the limit --
since 1 is correct after a rollback plus one failure.

32 launcher tests, all confirmed to bite.
This commit is contained in:
2026-08-28 00:37:52 +02:00
parent f04294c3c1
commit 5b7b280e3a
6 changed files with 207 additions and 59 deletions
+58 -2
View File
@@ -15,6 +15,8 @@ setup() {
export MESH_HOST_LIBEXEC="$WORK/libexec"
export MESH_HOST_BIN="$WORK/bin/nox-mesh-host"
export MESH_HOST_START_LIMIT=3
export MESH_HOST_BACKOFF=0
export MESH_HOST_RUN_ONCE=1
mkdir -p "$MESH_HOST_STATE_DIR" "$MESH_HOST_LIBEXEC" "$WORK/bin"
# A host that records being started. It exits immediately, which is what the launcher's
@@ -23,7 +25,7 @@ setup() {
cat > "$MESH_HOST_BIN" <<'STUB'
#!/bin/sh
echo "$@" >> "$MESH_HOST_STATE_DIR/host.starts"
exit 0
exit "${STUB_HOST_EXIT:-1}"
STUB
cat > "$MESH_HOST_LIBEXEC/rollback" <<'STUB'
#!/bin/sh
@@ -66,8 +68,11 @@ echo "1.4.2" > "$MESH_HOST_STATE_DIR/known-good"
i=1; while [ $i -le 4 ]; do "$LAUNCH" >/dev/null 2>&1 || true; i=$((i+1)); done
check "the fourth start rolls back" "three failures is a binary that does not work" "$(rolled)" "yes"
check "and still starts the host" "the rolled-back version has to be run" "$(started)" "yes"
# Below the limit, not exactly zero. The rollback resets it and the rolled-back version then
# fails once here, so 1 is right — the property is that it did NOT inherit a count already at
# the limit, which would halt the new version on its first attempt.
check "resets the counter after rolling back" "the new version deserves its own attempts, or it halts at once" \
"$(count)" "0"
"$([ "$(count)" -lt 3 ] && echo below-limit || echo "at-limit($(count))")" "below-limit"
# --- the host clears the counter on success ----------------------------------------------------
setup
@@ -131,5 +136,56 @@ for corrupt in "5x" "0x10" "1 2" ""; do
"$(count | grep -cE '^[0-9]+$')" "1"
done
# --- the loop, and shutting down ------------------------------------------------------------
#
# These need the launcher to actually run as a supervisor rather than one iteration, so they do
# not set MESH_HOST_RUN_ONCE.
# A host that exits 0 has upgraded itself and stood aside (novox/hq ADR 0057). The launcher must
# start it again — and must NOT count it, because it did not fail.
setup
unset MESH_HOST_RUN_ONCE
cat > "$MESH_HOST_BIN" <<'STUB'
#!/bin/sh
echo start >> "$MESH_HOST_STATE_DIR/host.starts"
# Exit 0 three times, then hang so the launcher stops looping and can be killed.
if [ "$(wc -l < "$MESH_HOST_STATE_DIR/host.starts")" -lt 3 ]; then exit 0; fi
sleep 30
STUB
chmod +x "$MESH_HOST_BIN"
"$LAUNCH" >/dev/null 2>&1 &
LP=$!
sleep 1
check "a clean exit restarts the host" "that is how it stands aside for a new binary" \
"$([ "$(wc -l < "$MESH_HOST_STATE_DIR/host.starts" 2>/dev/null || echo 0)" -ge 3 ] && echo looped || echo stopped)" "looped"
# No counter file at all: nothing has failed, so nothing has been counted.
check "a clean exit is not counted as a failure" "it finished, it did not fail" "$(count)" "MISSING"
# Shutting down: the signal must reach the host, and the launcher must wait for it rather than
# exiting and leaving the host to be killed mid-apply.
kill -TERM "$LP" 2>/dev/null
sleep 1
check "SIGTERM stops the launcher" "a supervisor that ignores shutdown hangs the machine" \
"$(kill -0 "$LP" 2>/dev/null && echo running || echo stopped)" "stopped"
check "and does not leave the host running" "the child must go down with it" \
"$(pgrep -f "$MESH_HOST_BIN" >/dev/null 2>&1 && echo orphaned || echo reaped)" "reaped"
# A crash IS counted, and the launcher keeps going.
setup
unset MESH_HOST_RUN_ONCE
export MESH_HOST_BACKOFF=0
cat > "$MESH_HOST_BIN" <<'STUB'
#!/bin/sh
echo start >> "$MESH_HOST_STATE_DIR/host.starts"
if [ "$(wc -l < "$MESH_HOST_STATE_DIR/host.starts")" -lt 2 ]; then exit 3; fi
sleep 30
STUB
chmod +x "$MESH_HOST_BIN"
"$LAUNCH" >/dev/null 2>&1 &
LP=$!
sleep 1
check "a crash is counted" "unlike a clean exit, which is not" "$(count)" "1"
kill -TERM "$LP" 2>/dev/null; sleep 1; pkill -f "$MESH_HOST_BIN" 2>/dev/null || true
printf '\nlaunch: %d passed, %d failed\n' "$PASS" "$FAIL"
[ "$FAIL" -eq 0 ]