Phase 4: gate a plan's first machine, roll a failed build back there, roll out by default, the bus as a planned step (hq ADR 0236) #92

2026-10-06T16:59:14Z

mesh-admin commented

2026-10-06 16:59:14 +00:00

hq ADR 0236 / to-be 45 Phase 4, controller side.

Health definitions as probes: H-controller (lease held, fresh, controller says ready: doctor ran, D9 passed), H-engine, H-tools (node tools answer discovery), H-bus (streams + durable consumers + a round trip); DG (gate verdicts + witnesses' rollbacks as conditions), DB (bus step).

The gate on every release plan's first machine: applied, no witness verdict since the send, no new condition about the machine / the module there, the component's health (core) or tools served (module); 3 healthy judgings ≥ 40 s apart, ≥ 2 min after the send, within 10 min. Recorded in PlanModule.Gate (plans <id> shows it) and in build_gate.

Rollback once per build: verdict written first (never overwritten), registered build restored to what the first machine ran (PreviousBuild, within the five kept), that machine sent it; build.<m>.<node>.rolled-back / core.<component>.<node>.rolled-back / rollback-failed, event rolled-back. A gate-failed build is refused registration; plans retry refuses it. A first machine that fails/refuses is rolled back too.

Policy: migration 0073 makes module.upgrade a person's choice (null = derived); default roll; manifest upgrade {policy, why}; irreplaceable data → record; the bus → record always. upgrade lists every module's policy and source; upgrade <m> roll-out|record --why|default.

Witness contract (internal/lease/witness.go, matching mesh-host PR #40): node principal granted $SRV.PING.node-tools.<node> everywhere and the lease's direct get where the controller runs; Report.rollbacks / witness; core.<component>.<node>.<outcome> conditions. The lease value carries health (ready) for the gate; host ignores it.

Bus: bus / bus upgrade --why --reversible|--irreversible [--snapshot-taken]; snapshot via the bus machine's node-backup now (ADR 0235's dump) + backed-up; bus-maintenance until H-bus passes, bus-upgrade-failed after 15 min. No send reaches the bus's machine while a new bus build waits (the restart seen live today); a rebuild with the same artifacts+manifest is no move. Controller grant gains node-backup.now and event rolled-back (genesis in mesh-host PR #41).

After a controller passes its gate it carries its user list to the bus's machine when changed and nothing held goes with it.

Deleted modules: SourceMoved.removed — a module whose manifest the merge deleted is forgotten (or said) and never built; a build finding no manifest marks it deleted and the plan goes on.

Tests (postgres:17 + nats:2.11, -race): all pass except TestTheCataloguesBusIsBackedUpBySnapshot, which reads the sibling mesh-catalog and fails on main too until mesh-catalog feat/bus-snapshot merges (passes with it). New: gate fail/pass/bound, health definitions, bus never rolled, deleted-module merge, witness verdicts, grants, policy derivation.

Merge order: mesh-host #41 (genesis grants) → this → mesh-catalog #95 (manifests use upgrade, which older builders refuse). Migration 0073.

hq ADR 0236 / to-be 45 Phase 4, controller side. **Health definitions as probes**: H-controller (lease held, fresh, controller says ready: doctor ran, D9 passed), H-engine, H-tools (node tools answer discovery), H-bus (streams + durable consumers + a round trip); DG (gate verdicts + witnesses' rollbacks as conditions), DB (bus step). **The gate** on every release plan's first machine: applied, no witness verdict since the send, no new condition about the machine / the module there, the component's health (core) or tools served (module); 3 healthy judgings ≥ 40 s apart, ≥ 2 min after the send, within 10 min. Recorded in `PlanModule.Gate` (`plans <id>` shows it) and in `build_gate`. **Rollback** once per build: verdict written first (never overwritten), registered build restored to what the first machine ran (`PreviousBuild`, within the five kept), that machine sent it; `build.<m>.<node>.rolled-back` / `core.<component>.<node>.rolled-back` / `rollback-failed`, event `rolled-back`. A gate-failed build is refused registration; `plans retry` refuses it. A first machine that fails/refuses is rolled back too. **Policy**: migration 0073 makes `module.upgrade` a person's choice (null = derived); default `roll`; manifest `upgrade {policy, why}`; irreplaceable data → record; the bus → record always. `upgrade` lists every module's policy and source; `upgrade <m> roll-out|record --why|default`. **Witness contract** (`internal/lease/witness.go`, matching mesh-host PR #40): node principal granted `$SRV.PING.node-tools.<node>` everywhere and the lease's direct get where the controller runs; `Report.rollbacks` / `witness`; `core.<component>.<node>.<outcome>` conditions. The lease value carries `health` (ready) for the gate; host ignores it. **Bus**: `bus` / `bus upgrade --why --reversible|--irreversible [--snapshot-taken]`; snapshot via the bus machine's `node-backup now` (ADR 0235's dump) + `backed-up`; `bus-maintenance` until H-bus passes, `bus-upgrade-failed` after 15 min. **No send reaches the bus's machine while a new bus build waits** (the restart seen live today); a rebuild with the same artifacts+manifest is no move. Controller grant gains `node-backup.now` and event `rolled-back` (genesis in mesh-host PR #41). **After a controller passes its gate** it carries its user list to the bus's machine when changed and nothing held goes with it. **Deleted modules**: `SourceMoved.removed` — a module whose manifest the merge deleted is forgotten (or said) and never built; a build finding no manifest marks it `deleted` and the plan goes on. Tests (postgres:17 + nats:2.11, `-race`): all pass except `TestTheCataloguesBusIsBackedUpBySnapshot`, which reads the sibling mesh-catalog and fails on main too until mesh-catalog `feat/bus-snapshot` merges (passes with it). New: gate fail/pass/bound, health definitions, bus never rolled, deleted-module merge, witness verdicts, grants, policy derivation. **Merge order**: mesh-host #41 (genesis grants) → this → mesh-catalog #95 (manifests use `upgrade`, which older builders refuse). Migration 0073.

mesh-admin added 5 commits 2026-10-06 16:59:14 +00:00

Gate a release plan's first machine and roll a failed build back there (hq ADR 0236) d9289ef6d4

A build that reported applied was sent everywhere; one that then did nothing, served
no tools or broke its machine's word reached every machine. Now the first machine is
judged by the component's health (the core's definitions, as doctor probes H-*, or a
module's own) three times over two minutes within ten; a failing gate puts the previous
build back there once, marks the build, and says it as a condition and an event.
Upgrades roll out by default; the bus is a planned step; a module deleted at its
source is not built (the public-acme plan failure).

Take the bus's snapshot through its machine's backup holder before the planned step (hq ADR 0235, 0236) c6f3d8cdfa

Name the decision this builds: hq ADR 0236 (0235 is the bus's snapshot) d7bf1bae83

Refuse every send that would replace the bus outside its planned step (hq ADR 0236) 37229b4db5

A plan's send to the bus's machine for another module carried the bus's new build and
restarted it under every machine with nobody asking (2026-10-06). The guard is in the one
send everything uses; only the bus step passes it. A rebuild that made the same artifacts
is no move.

Tell a rebuild that changes nothing by its artifacts and its manifest (hq ADR 0236) 41f7b2c152

jschoubben added 1 commit 2026-10-06 17:14:31 +00:00

No build reaches a machine without a gate; a release plan walks what waits (hq ADR 0236) 2bfa6ae4a0

A send carries the machine's whole declaration, so at the switch to roll the next send of
anything would have carried the old default's backlog, unjudged, to every machine. A gated
send now carries and judges everything waiting on its machine; every other send is refused
or leaves the machine; a release plan walks what waits one machine at a time, the control
node last, and one that fails holds the next until a person releases it.

jschoubben added 1 commit 2026-10-06 17:14:38 +00:00

gofmt dcec6a4db3

mesh-admin merged commit 4635341a9d into main

2026-10-06 17:15:08 +00:00

mesh-admin deleted branch feat/core-upgrades-that-roll-back

2026-10-06 17:15:08 +00:00

mesh-admin commented

2026-10-06 23:09:43 +00:00

Delivery novox/mesh-controller@4635341a9da4 — delivered since 2026-10-06T23:05:28Z

Machines

build-agent on ace: passed — healthy 3 times over 2m47s
build-agent on the rest running it: sent
mesh-controller on novox: passed — healthy 3 times over 8m53s
mesh-controller on the rest running it: sent
route-proxy on ace: passed — healthy 3 times over 2m33s
route-proxy on the rest running it: sent

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-controller#92 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on
2026-10-06 23:05 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's plans go
2026-10-06 23:05 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@4635341a9da4` — **delivered** since 2026-10-06T23:05:28Z **Machines** - build-agent on ace: passed — healthy 3 times over 2m47s - build-agent on the rest running it: sent - mesh-controller on novox: passed — healthy 3 times over 8m53s - mesh-controller on the rest running it: sent - route-proxy on ace: passed — healthy 3 times over 2m33s - route-proxy on the rest running it: sent **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-controller#92 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on - 2026-10-06 23:05 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's `plans go` - 2026-10-06 23:05 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin referenced this pull request

2026-10-13 00:30:20 +00:00

Replay issue 166 with only what the catalogue had before its fix (for mesh-lab R166) #287

mesh-admin referenced this pull request

2026-10-13 00:37:54 +00:00

Replay issue 166 with only what the catalogue had before its fix (for mesh-lab R166) #287

This repo is archived. You cannot comment on pull requests.

2 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#92