Core upgrades that roll back: the host side (hq to-be 45 Phase 4, ADR 0227 rule 8) #40

2026-10-06T16:24:24Z

mesh-admin commented

2026-10-06 16:24:24 +00:00

Host side of Phase 4. The controller side (the probes, the gate, the conditions, the grants) is being built in parallel in mesh-controller.

Before this change

The launcher's rollback could not run anywhere. Only a by-hand reconcile wrote known-good and cleared start-attempts; the daemon did neither. (g14 has 53 delivered host versions and no known-good.)
rollback-attempted was global and never cleared, and nothing ever lifted the pin.
The controller and node-tools bundles were replaced in place (RemoveAll), so there was no previous build to go back to.
Retire could delete a newer version that had just been delivered. The one-shot path also retired around the link stamp, which could remove the running version.

Changes

The node-engine and its launcher

A delivered host that is not known-good runs on trial for 10 minutes (MESH_HOST_TRIAL_BOUND).
It is rolled back if it:
- crashes 3 times,
- exits cleanly when there is nothing newer to stand aside for, or
- does not report before the bound.
A rollback goes to known-good once per version and appends to rolled-back (TSV: from, to, epoch, outcome, why). A rolled-back version is never started again. A newer delivery drops the pin and runs on trial.
If the version it went back to fails too, the launcher halts and records that.
The host proves itself on the first report the mesh takes about a declaration under its own build. Proof writes known-good, clears the counter, retires older versions and ends earlier verdicts.
While unproved, the host sends its startup report even if nothing changed.
The host never stands aside for a rolled-back version.
A replaced launcher runs itself at the next clean exit. If the launcher running the host is an older one that was replaced on disk, the host asks systemd to restart the service once.
The rollback logic now lives in the launcher itself. nox-mesh-host-rollback is unchanged and stays for older launchers.

The controller and node-tools witness (internal/witness)

The previous build is kept at /var/lib/mesh/daemons/.witness/<name>/previous. The new build is judged:
- controller: the mesh-controller_lease/holder key (read-only direct get) must be held on this machine, taken after the new build started, and renewed within 15 s, all within 60 s;
- node-tools: $SRV.PING.node-tools.<node> must be answered within 5 s, all within 60 s.
Only time when the host could ask counts toward the bound. If it cannot ask (no grant, no bucket, no link), that time does not count; after 30 minutes the verdict is unwitnessed.
Not healthy: the previous build is restored once and the verdict is said. A rolled-back digest is refused while the mesh still declares it.
Proved: the previous build is deleted.
A build declared not-reversible is never rolled back. It is left running and the verdict is not-reversible.

Reports

rollbacks: every standing verdict, {component, from, to, outcome, why, at}.
witness: 1: the witness contract version.

Process declaration

New optional fields: witness (lease / ping / none; the default comes from the name: mesh-controller is lease, node-tools is ping) and not-reversible (why).

Grant the controller must compose (node principal)

publish $SRV.PING.node-tools.<node> (every node)
publish $JS.API.DIRECT.GET.KV_mesh-controller_lease.$KV.mesh-controller_lease.holder (at least the control node)

Replies use the node's existing _INBOX.node.<node>.>. The genesis lock carries no node user, so it is unchanged.

Tests

go test -race ./..., vet and gofmt are clean. make packaging-test passes 60 launcher checks, also run under busybox ash.
Each rollback path is induced: a new engine that crashes, one that exits 0, one that never reports; a controller that never takes the lease; a runtime that never answers PING. Each ends on the previous build, reported once.
A healthy update deletes nothing until it is proved, then cleans the previous build.
A not-reversible build is never rolled back.
A host that cannot ask never rolls back.
TestNatsWitnessAsksWhatItIsGranted passes against nats 2.11 with exactly the two grants (testdata/witness-grants.conf, MESH_TEST_NATS_GRANTS). Without the grants the host can't ask, and that is not counted as a failure.

Do not merge before the review.

Host side of Phase 4. The controller side (the probes, the gate, the conditions, the grants) is being built in parallel in mesh-controller. ## Before this change - The launcher's rollback could not run anywhere. Only a by-hand `reconcile` wrote `known-good` and cleared `start-attempts`; the daemon did neither. (g14 has 53 delivered host versions and no known-good.) - `rollback-attempted` was global and never cleared, and nothing ever lifted the pin. - The controller and node-tools bundles were replaced in place (`RemoveAll`), so there was no previous build to go back to. - `Retire` could delete a newer version that had just been delivered. The one-shot path also retired around the link stamp, which could remove the running version. ## Changes **The node-engine and its launcher** - A delivered host that is not known-good runs **on trial** for 10 minutes (`MESH_HOST_TRIAL_BOUND`). - It is rolled back if it: - crashes 3 times, - exits cleanly when there is nothing newer to stand aside for, or - does not report before the bound. - A rollback goes to known-good **once per version** and appends to `rolled-back` (TSV: from, to, epoch, outcome, why). A rolled-back version is never started again. A newer delivery drops the pin and runs on trial. - If the version it went back to fails too, the launcher halts and records that. - The host proves itself on the first report the mesh takes about a declaration under its own build. Proof writes known-good, clears the counter, retires older versions and ends earlier verdicts. - While unproved, the host sends its startup report even if nothing changed. - The host never stands aside for a rolled-back version. - A replaced launcher runs itself at the next clean exit. If the launcher running the host is an older one that was replaced on disk, the host asks systemd to restart the service once. - The rollback logic now lives in the launcher itself. `nox-mesh-host-rollback` is unchanged and stays for older launchers. **The controller and node-tools witness** (`internal/witness`) - The previous build is kept at `/var/lib/mesh/daemons/.witness/<name>/previous`. The new build is judged: - controller: the `mesh-controller_lease/holder` key (read-only direct get) must be held on this machine, taken after the new build started, and renewed within 15 s, all within 60 s; - node-tools: `$SRV.PING.node-tools.<node>` must be answered within 5 s, all within 60 s. - Only time when the host could ask counts toward the bound. If it cannot ask (no grant, no bucket, no link), that time does not count; after 30 minutes the verdict is `unwitnessed`. - Not healthy: the previous build is restored once and the verdict is said. A rolled-back digest is refused while the mesh still declares it. - Proved: the previous build is deleted. - A build declared `not-reversible` is never rolled back. It is left running and the verdict is `not-reversible`. **Reports** - `rollbacks`: every standing verdict, `{component, from, to, outcome, why, at}`. - `witness: 1`: the witness contract version. **Process declaration** - New optional fields: `witness` (`lease` / `ping` / `none`; the default comes from the name: `mesh-controller` is lease, `node-tools` is ping) and `not-reversible` (why). ## Grant the controller must compose (node principal) - publish `$SRV.PING.node-tools.<node>` (every node) - publish `$JS.API.DIRECT.GET.KV_mesh-controller_lease.$KV.mesh-controller_lease.holder` (at least the control node) Replies use the node's existing `_INBOX.node.<node>.>`. The genesis lock carries no node user, so it is unchanged. ## Tests - `go test -race ./...`, vet and gofmt are clean. `make packaging-test` passes 60 launcher checks, also run under busybox ash. - Each rollback path is induced: a new engine that crashes, one that exits 0, one that never reports; a controller that never takes the lease; a runtime that never answers PING. Each ends on the previous build, reported once. - A healthy update deletes nothing until it is proved, then cleans the previous build. - A `not-reversible` build is never rolled back. - A host that cannot ask never rolls back. - `TestNatsWitnessAsksWhatItIsGranted` passes against nats 2.11 with exactly the two grants (`testdata/witness-grants.conf`, `MESH_TEST_NATS_GRANTS`). Without the grants the host can't ask, and that is not counted as a failure. Do not merge before the review.

mesh-admin added 1 commit 2026-10-06 16:24:24 +00:00

Roll a core build back by a witness that is not the new build (hq to-be 45 Phase 4) 0c405b70cc

The launcher trusted a counter only a by-hand reconcile ever cleared and a
known-good nothing in the daemon wrote, so no machine could roll its host back;
the controller and the node tools were replaced in place with nothing kept.

- The launcher runs a delivered host that is not known-good on trial: one that
  crashes, stops for nothing, or does not report within ten minutes goes back
  to known-good, once per version, recorded in rolled-back. The host proves
  itself when the mesh takes a report under its own build, says every standing
  verdict on its reports, never stands aside for a rolled-back version, and
  restarts its service once when its launcher was replaced on disk.
- The engine keeps the controller's and the node tools' previous build beside
  the new one and judges the new one: the lease taken by the controller it
  started (read-only direct get of mesh-controller_lease/holder), or this
  machine's runtime answering $SRV.PING.node-tools.<node>, within sixty seconds
  of time it could ask. Not healthy: the previous restored, once, said. Proved:
  the previous deleted. A build declared not-reversible is never rolled back.
- Retire never removes a version newer than the running one.

mesh-admin referenced this pull request

2026-10-06 16:39:56 +00:00

Genesis grants for the gate's event and the bus step's snapshot (hq ADR 0236) #41

mesh-admin merged commit 7a6f9218a0 into main

2026-10-06 18:00:08 +00:00

mesh-admin deleted branch feat/core-upgrades-roll-back

2026-10-06 18:00:08 +00:00

mesh-admin commented

2026-10-06 23:10:41 +00:00

Delivery novox/mesh-host@7a6f9218a05f — delivered since 2026-10-06T23:05:28Z

Machines

mesh-host on ace: passed — healthy 3 times over 2m32s
mesh-host on the rest running it: sent

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-host#40 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on
2026-10-06 23:05 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's plans go
2026-10-06 23:05 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@7a6f9218a05f` — **delivered** since 2026-10-06T23:05:28Z **Machines** - mesh-host on ace: passed — healthy 3 times over 2m32s - mesh-host on the rest running it: sent **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-host#40 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on - 2026-10-06 23:05 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's `plans go` - 2026-10-06 23:05 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#40