Send a plan's tier to each machine once, and blame no module for its machine (hq issue 281) #100

2026-10-06T20:24:18Z

mesh-admin commented

2026-10-06 20:24:18 +00:00

hq issue 281 (novox/hq PR to follow, 04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine).

What happened (2026-10-06, plan-1791315388197806253)

advanceOnce sent each module of a ~70-module tier to its first machine in a send of its own (gatedSend per module, return, next step). A send carries the machine's whole declaration (ADR 0221), so the laptop got a new declaration every few seconds; its node-engine set aside one for the next, its reports never caught up with what it was last sent, and D10 raised machine.<laptop>.core-behind ("runs the node tools it was last sent, not yet reported applied"). judgeHealth read any machine-scope condition raised since the send as the judged module's, so blueman's gate failed at the 10 min bound for it.

The rollback then said "no build of blueman from b792bf4b is still kept": Previous was read from the machine's sent builds after an earlier send of the same tier had already carried blueman's b792bf4b build there, so the plan recorded b792bf4b → b792bf4b and looked for an earlier build of the commit it had just failed. The build blueman really ran before (7f99fb4a) was kept all along, and its bundle digest is identical to b792bf4b's: nothing of blueman ever changed on the machine.

Fix

One send per machine per tier. Every module of the tier whose first machine is the same goes there in one gatedSend (now taking several own moves); Previous is read once, before that send. One gate judges the send, kept on the first module; the others carry GatedBy and take its verdict. The rest of the tier's machines are sent once, together, when the gates have passed. Rollbacks of a failed send go back in one send per machine (batchingRollbacks/sendRollbacks), also for release plans. plans retry of stopped rollouts sends each machine once.
No module blamed for its machine. judgeMoves sorts machine-scope conditions per machine (aboutTheMachine): one naming a moved module is that module's; core-behind is the node-engine's / node tools' when the send moved them (their settle window is the gate's bound) and no evidence otherwise; anything else is the machine's as a whole - every module of the send waits on it and fails together, once, with " as a whole: ...". The module a gate is kept on is put back only when it is among the failing.
A send that changed nothing of a module is no verdict on its build (unchangedBy: same commit, same source fingerprint, or same artifacts): left as it was, not marked failed, no rollback condition. The first-ever-build and not-kept messages now say what they mean (incl. the kept count).
plans retry accepts a plan stopped at such a gate (Rollback: unchanged, or a recorded From == To as in the incident): the module is asked again under a new id and every module of the tier sent first and never passed is sent again, the tier whole.

Rebased on #99 (issue 280, unchanged source is no move): its no-move check runs first; unchangedBy uses the source fingerprints too.

Tests (`cmd/mesh-controller/tier_send_test.go`)

a tier of 12 modules on two machines: exactly [[anchor] [laptop]] sent, one gate, every build's pass kept;
core-behind while node-tools (sent in the same tier) settles: only node-tools fails and is put back; app modules stay at the new build, not marked failed, plan note does not blame the gate's module;
a machine-level condition about anything else: the whole send fails together with "anchor as a whole", put back in one send;
aboutTheMachine sorting, pure;
the incident's From == To case: left as it was, not marked, no rollback condition, and plans retry asks it again.

go vet ./..., gofmt, and the full suite go test -race ./... (vendor, GOPROXY=off, per-test buses, throwaway postgres) pass.

Not merged; needs a human approval.

hq issue 281 (novox/hq PR to follow, `04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine`). ## What happened (2026-10-06, plan-1791315388197806253) `advanceOnce` sent each module of a ~70-module tier to its first machine **in a send of its own** (`gatedSend` per module, return, next step). A send carries the machine's whole declaration (ADR 0221), so the laptop got a new declaration every few seconds; its node-engine set aside one for the next, its reports never caught up with what it was last sent, and D10 raised `machine.<laptop>.core-behind` ("runs the node tools it was last sent, not yet reported applied"). `judgeHealth` read **any** machine-scope condition raised since the send as the judged module's, so blueman's gate failed at the 10 min bound for it. The rollback then said "no build of blueman from b792bf4b is still kept": `Previous` was read from the machine's sent builds *after* an earlier send of the same tier had already carried blueman's b792bf4b build there, so the plan recorded b792bf4b → b792bf4b and looked for an earlier build **of the commit it had just failed**. The build blueman really ran before (7f99fb4a) was kept all along, and its bundle digest is identical to b792bf4b's: nothing of blueman ever changed on the machine. ## Fix 1. **One send per machine per tier.** Every module of the tier whose first machine is the same goes there in one `gatedSend` (now taking several own moves); `Previous` is read once, before that send. One gate judges the send, kept on the first module; the others carry `GatedBy` and take its verdict. The rest of the tier's machines are sent once, together, when the gates have passed. Rollbacks of a failed send go back in one send per machine (`batchingRollbacks`/`sendRollbacks`), also for release plans. `plans retry` of stopped rollouts sends each machine once. 2. **No module blamed for its machine.** `judgeMoves` sorts machine-scope conditions per machine (`aboutTheMachine`): one naming a moved module is that module's; `core-behind` is the node-engine's / node tools' when the send moved them (their settle window is the gate's bound) and no evidence otherwise; anything else is the machine's *as a whole* - every module of the send waits on it and fails together, once, with "<machine> as a whole: ...". The module a gate is kept on is put back only when it is among the failing. 3. **A send that changed nothing of a module is no verdict on its build** (`unchangedBy`: same commit, same source fingerprint, or same artifacts): left as it was, not marked failed, no rollback condition. The first-ever-build and not-kept messages now say what they mean (incl. the kept count). 4. **`plans retry`** accepts a plan stopped at such a gate (`Rollback: unchanged`, or a recorded From == To as in the incident): the module is asked again under a new id and every module of the tier sent first and never passed is sent again, the tier whole. Rebased on #99 (issue 280, unchanged source is no move): its no-move check runs first; `unchangedBy` uses the source fingerprints too. ## Tests (`cmd/mesh-controller/tier_send_test.go`) - a tier of 12 modules on two machines: exactly `[[anchor] [laptop]]` sent, one gate, every build's pass kept; - core-behind while node-tools (sent in the same tier) settles: only node-tools fails and is put back; app modules stay at the new build, not marked failed, plan note does not blame the gate's module; - a machine-level condition about anything else: the whole send fails together with "anchor as a whole", put back in one send; - `aboutTheMachine` sorting, pure; - the incident's From == To case: left as it was, not marked, no rollback condition, and `plans retry` asks it again. `go vet ./...`, gofmt, and the full suite `go test -race ./...` (vendor, GOPROXY=off, per-test buses, throwaway postgres) pass. Not merged; needs a human approval.

mesh-admin added 1 commit 2026-10-06 20:24:19 +00:00

Send a plan's tier to each machine once, and blame no module for its machine (hq issue 281)

mesh/merge-gate error: the check could not run: a throwaway postgres:17-alpine could not be raised: docker run --label mesh.build=build-1791318263948250337…

mesh/delivery delivered

Details

d6e0a8250a

mesh-admin merged commit d0580a17e5 into main

2026-10-06 20:26:04 +00:00

mesh-admin deleted branch fix/one-send-per-machine-per-tier

2026-10-06 20:26:04 +00:00

mesh-admin commented

2026-10-06 20:36:34 +00:00

Merge check: ERROR at d6e0a825 — the check could not run: a throwaway postgres:17-alpine could not be raised: docker run --label mesh.build=build-1791318263948250337 -d --rm --name build-1791318263948250337-store -p 127.0.0.1::5432 -e POSTGRES_PASSWORD=check postgres:17-alpine: exit status 125

Run by the build seat on ace as build-1791318263948250337 (builds --log build-1791318263948250337).

**Merge check: ERROR** at `d6e0a825` — the check could not run: a throwaway postgres:17-alpine could not be raised: docker run --label mesh.build=build-1791318263948250337 -d --rm --name build-1791318263948250337-store -p 127.0.0.1::5432 -e POSTGRES_PASSWORD=check postgres:17-alpine: exit status 125 Run by the build seat on ace as `build-1791318263948250337` (`builds --log build-1791318263948250337`).

mesh-admin commented

2026-10-06 23:10:22 +00:00

Delivery novox/mesh-controller@d6e0a8250afb — delivered since 2026-10-06T23:05:28Z

Machines

build-agent on ace: passed — healthy 3 times over 2m30s
build-agent on the rest running it: sent
mesh-controller on novox: passed — healthy 3 times over 3m47s
mesh-controller on the rest running it: sent
route-proxy on ace: passed — healthy 3 times over 2m35s
route-proxy on the rest running it: sent

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-controller#100's head announced
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on
2026-10-06 23:05 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's plans go
2026-10-06 23:05 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@d6e0a8250afb` — **delivered** since 2026-10-06T23:05:28Z **Machines** - build-agent on ace: passed — healthy 3 times over 2m30s - build-agent on the rest running it: sent - mesh-controller on novox: passed — healthy 3 times over 3m47s - mesh-controller on the rest running it: sent - route-proxy on ace: passed — healthy 3 times over 2m35s - route-proxy on the rest running it: sent **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-controller#100's head announced - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on - 2026-10-06 23:05 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's `plans go` - 2026-10-06 23:05 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

This repo is archived. You cannot comment on pull requests.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#100

Send a plan's tier to each machine once, and blame no module for its machine (hq issue 281) #100

What happened (2026-10-06, plan-1791315388197806253)

Fix

Tests (cmd/mesh-controller/tier_send_test.go)

Tests (`cmd/mesh-controller/tier_send_test.go`)