Replay the crash loop: a container that exits at start fails its gate on the first machine (hq ADR 0240) #54

2026-10-07T00:20:01Z

mesh-admin commented

2026-10-07 00:20:01 +00:00

Adds R-crashloop to the replay register (hq ADR 0240 rule 1, to-be 48 Phase A done-when). It replays the agent server's crash loop (issue 268, research 032 §6). This is the register's first replay of a module rather than a core incident, so ids may now be R-<name>.

It is a new kind, liveness, and it runs two repositories, each at its own commit: Repository = mesh-controller, and With = mesh-host. replays/crashloop_test.go:

Raises alpine:3.20 running sh -c 'exit 3' the way the node-engine raises a module's container: unless-stopped, labelled mesh-host.id=app.server. It goes through the runtime API; Run gained Name, Labels and Restart.
Runs the node-engine's TestReplayCrashLoopIsSaidUnhealthy at its commit against that container. That writes what the engine states. An engine older than the judging states nothing, and its reports carry no health.
Runs the controller's TestReplayCrashLoopFailsItsGateOnTheFirstMachine at its commit from that statement. The replay passes only if the gate fails the build on the first machine and puts it back.

The container is removed afterwards. The replay needs Docker, Go and MESH_TEST_POSTGRES, and skips otherwise.

Proved:

go run ./cmd/prove R-crashloop
R-crashloop (issue 268): PROVED — before its fix fails, on its fix passes

Before the fix, the engine at origin/main judges nothing, and the controller at origin/main passes the gate and sends the build to the second machine. On the fix, the engine says unhealthy (restarting), 2 restarts counted about 7 s after it starts judging (grace shortened to 3 s), and the gate fails at the bound and is put back. Nothing is left behind (containers, worktrees).

Fix/Before name the feature branch and origin/main until mesh-controller#105 and mesh-host#47 merge. Then they should be set to the merge commits, as for R236.

Checks: go vet and gofmt are clean in replays/ (also with Go 1.26's gofmt). check-here: mesh/repo-check PASS. Two TypeScript tests (listing instances/networks fails rather than reporting none) fail only on a workstation where incus answers. They fail the same way on main and are not touched here.

Feature a-module-says-how-it-is-healthy.

**Adds `R-crashloop` to the replay register** (hq ADR 0240 rule 1, to-be 48 Phase A done-when). It replays the agent server's crash loop (issue 268, research 032 §6). This is the register's first replay of a module rather than a core incident, so ids may now be `R-<name>`. It is a new kind, `liveness`, and it runs two repositories, each at its own commit: Repository = mesh-controller, and `With` = mesh-host. `replays/crashloop_test.go`: 1. Raises `alpine:3.20` running `sh -c 'exit 3'` the way the node-engine raises a module's container: `unless-stopped`, labelled `mesh-host.id=app.server`. It goes through the runtime API; `Run` gained `Name`, `Labels` and `Restart`. 2. Runs the node-engine's `TestReplayCrashLoopIsSaidUnhealthy` at its commit against that container. That writes what the engine states. An engine older than the judging states nothing, and its reports carry no health. 3. Runs the controller's `TestReplayCrashLoopFailsItsGateOnTheFirstMachine` at its commit from that statement. The replay passes only if the gate fails the build on the first machine and puts it back. The container is removed afterwards. The replay needs Docker, Go and `MESH_TEST_POSTGRES`, and skips otherwise. **Proved:** ``` go run ./cmd/prove R-crashloop R-crashloop (issue 268): PROVED — before its fix fails, on its fix passes ``` Before the fix, the engine at origin/main judges nothing, and the controller at origin/main passes the gate and sends the build to the second machine. On the fix, the engine says `unhealthy (restarting), 2 restarts counted` about 7 s after it starts judging (grace shortened to 3 s), and the gate fails at the bound and is put back. Nothing is left behind (containers, worktrees). `Fix`/`Before` name the feature branch and `origin/main` until mesh-controller#105 and mesh-host#47 merge. Then they should be set to the merge commits, as for R236. **Checks:** `go vet` and `gofmt` are clean in `replays/` (also with Go 1.26's gofmt). `check-here`: `mesh/repo-check` PASS. Two TypeScript tests (`listing instances/networks fails rather than reporting none`) fail only on a workstation where incus answers. They fail the same way on main and are not touched here. Feature `a-module-says-how-it-is-healthy`.

mesh-admin commented

2026-10-07 00:20:20 +00:00

Correction to the description: the controller's pull request is mesh-controller#107, not #105. Confirmed: the two TypeScript failures (incus answering on this workstation) show the same on origin/main, 158 pass and 2 fail.

jschoubben added 1 commit 2026-10-07 00:28:56 +00:00

Replay the crash loop: a container that exits at start fails its gate on the first machine (hq ADR 0240)

mesh/merge-gate pass: the change touches no module of the mesh's graph

Details

mesh/repo-check pass: its merge-check.sh passed

Details

mesh/delivery delivered

Details

mesh/delivery-group group feat/a-module-says-how-it-is-healthy delivered: every member is delivered

Details

5e3ae7b835

The agent server crash-looped about a hundred times behind every passing check.
R-crashloop raises a container whose program exits at start, has the node-engine
at its commit judge it through the runtime, and the controller at its commit
judge the gate from what the engine said — two repositories, each at its own
commit before the fix and on it.

jschoubben force-pushed feat/a-module-says-how-it-is-healthy from 4db1616bbd to 5e3ae7b835

2026-10-07 00:28:56 +00:00

Compare

mesh-admin commented

2026-10-07 00:30:01 +00:00

Delivery novox/mesh-lab@5e3ae7b835b2 — delivered since 2026-10-07T00:40:12Z

Delivery plan — builds nothing: the change touches no module of the mesh's graph

Group feat/a-module-says-how-it-is-healthy, in order: novox/hq@dea72dc4fc → novox/mesh-lab@5e3ae7b835

novox/mesh-controller@725fcd977e before novox/mesh-host@1fc1e74cc3: built by
novox/mesh-host@1fc1e74cc3 before novox/mesh-controller@725fcd977e: engine before controller

Transitions

2026-10-07 00:29 (new) → proposed (announced): novox/mesh-lab#54's head announced
2026-10-07 00:36 proposed → checked (checked): the verdict names this commit
2026-10-07 00:36 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-07 00:40 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-07 00:40 published → delivered (done): nothing for a walk to move

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-lab@5e3ae7b835b2` — **delivered** since 2026-10-07T00:40:12Z **Delivery plan** — builds nothing: the change touches no module of the mesh's graph **Group** `feat/a-module-says-how-it-is-healthy`, in order: novox/hq@dea72dc4fc8c → novox/mesh-lab@5e3ae7b835b2 - novox/mesh-controller@725fcd977e75 before novox/mesh-host@1fc1e74cc3c9: built by - novox/mesh-host@1fc1e74cc3c9 before novox/mesh-controller@725fcd977e75: engine before controller **Transitions** - 2026-10-07 00:29 (new) → proposed (announced): novox/mesh-lab#54's head announced - 2026-10-07 00:36 proposed → checked (checked): the verdict names this commit - 2026-10-07 00:36 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-07 00:40 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-07 00:40 published → delivered (done): nothing for a walk to move The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin merged commit 69e1215424 into main

2026-10-07 00:39:41 +00:00

mesh-admin deleted branch feat/a-module-says-how-it-is-healthy

2026-10-07 00:39:41 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-lab#54