A gate judges its own send and the build it sent, and never puts the controller back behind its store (hq issue 352) #185

Merged
mesh-admin merged 1 commits from fix/a-gate-outlives-the-controller-and-judges-the-build-it-sent into main 2026-10-09 16:10:13 +00:00
Contributor

hq issue 352 (hq PR #252). On 2026-10-09, 16:19-16:36 local on the control node: a release's gate read the machine's report against a newer send another plan had just made there, failed three builds the machine had reported healthy ("has not reported on what it was sent"), put them back on every machine to a controller older than the store's schema, and that controller then passed the newer plan's gate from its own health.

D1 — a gate reads the report against its own send. PlanGate.Sent keeps, per machine, the declaration the send carried (digest and sequence, read from the node's record right after the send; kept with the plan, so a restarted controller reads the same). A report is on it when it names that declaration, one sequenced after it, or the declaration sent last. judgeHealth and the merge plan's first-machine wait (nextRollout) read this; a gate kept without it reads the last send, as before.

D2 — a gate judges only the build the machine was last sent. Each gate judges sent_builds[machine][module] against the commit it judges; another build there is a new verdict, superseded: decided at once, no verdict on the build, nothing marked, nothing put back. A superseded release ends and what still waits is released again; a superseded merge plan ends, said. A module the gate itself put back at once (Returned) is not read as superseded.

D3 — no put-back to a controller older than the store's schema. The controller's declaration tells it which build it is (MESH_CONTROLLER_VERSION: ${version} in module.json; ${version} is now composed inside a process's env too). When it serves it records the highest migration it carries by that version (controller_schema, migration 0087). A failed controller gate reads the record of the build it would put back: one reaching less than the store has applied, or that never recorded, is not put back — the current build is kept and rollback-failed (urgent) says why. The lease holder's Build is that version.

Serialised walks. walkedBy reads a release's open gate as its walk, and a plan's own first send is held by another walk of the same module on that machine with another build (the same build there is already there). The merge plan's send at 16:31:37 would have waited for the release's gate.

Tests: TestReplay352 fails on the commit before the fix with "has not reported on what it was sent"; TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere, TestAPlansJudgingIsSupersededByASendThatMovesItsModule, TestAReportIsHeldAgainstTheGatesOwnSend, TestAGateJudgesOnlyTheBuildTheMachineWasLastSent, TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine, TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema, TestTheHolderNamesTheBuildTheDeclarationToldIt; inventory TestASendAndAReportAreKnownByTheirDeclaration, TestASchemaReachIsKeptByBuild; catalogue TestAProcessEnvCanNameTheBuildsOwnVersion. 32 mutations, each failing a test. Two fixtures were completed (a machine that "has not reported" now reports on no send at all).

Local merge-check.sh: every package passes but internal/broker's installer test, which fails on main as well when the mesh-host checkout beside it is behind — see the thread. Not merged; no rollout is triggered by this PR.

hq issue 352 (hq PR #252). On 2026-10-09, 16:19-16:36 local on the control node: a release's gate read the machine's report against a newer send another plan had just made there, failed three builds the machine had reported healthy ("has not reported on what it was sent"), put them back on every machine to a controller older than the store's schema, and that controller then passed the newer plan's gate from its own health. **D1 — a gate reads the report against its own send.** `PlanGate.Sent` keeps, per machine, the declaration the send carried (digest and sequence, read from the node's record right after the send; kept with the plan, so a restarted controller reads the same). A report is on it when it names that declaration, one sequenced after it, or the declaration sent last. `judgeHealth` and the merge plan's first-machine wait (`nextRollout`) read this; a gate kept without it reads the last send, as before. **D2 — a gate judges only the build the machine was last sent.** Each gate judges `sent_builds[machine][module]` against the commit it judges; another build there is a new verdict, `superseded`: decided at once, no verdict on the build, nothing marked, nothing put back. A superseded release ends and what still waits is released again; a superseded merge plan ends, said. A module the gate itself put back at once (`Returned`) is not read as superseded. **D3 — no put-back to a controller older than the store's schema.** The controller's declaration tells it which build it is (`MESH_CONTROLLER_VERSION: ${version}` in module.json; `${version}` is now composed inside a process's env too). When it serves it records the highest migration it carries by that version (`controller_schema`, migration 0087). A failed controller gate reads the record of the build it would put back: one reaching less than the store has applied, or that never recorded, is not put back — the current build is kept and `rollback-failed` (urgent) says why. The lease holder's `Build` is that version. **Serialised walks.** `walkedBy` reads a release's open gate as its walk, and a plan's own first send is held by another walk of the same module on that machine with another build (the same build there is already there). The merge plan's send at 16:31:37 would have waited for the release's gate. Tests: `TestReplay352` fails on the commit before the fix with "has not reported on what it was sent"; `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule`, `TestAReportIsHeldAgainstTheGatesOwnSend`, `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent`, `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, `TestTheHolderNamesTheBuildTheDeclarationToldIt`; inventory `TestASendAndAReportAreKnownByTheirDeclaration`, `TestASchemaReachIsKeptByBuild`; catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`. 32 mutations, each failing a test. Two fixtures were completed (a machine that "has not reported" now reports on no send at all). Local `merge-check.sh`: every package passes but `internal/broker`'s installer test, which fails on main as well when the mesh-host checkout beside it is behind — see the thread. Not merged; no rollout is triggered by this PR.
jschoubben added 1 commit 2026-10-09 15:15:43 +00:00
A gate judges its own send and the build it sent, and never puts the controller back behind its store (hq issue 352)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
e6e1e3bc89
On 2026-10-09 a release's gate on the control node read the machine's
report against a newer send another plan had just made there, failed
three builds the machine had reported healthy, and put them back on
every machine to a controller older than the store's schema; that
controller then passed the newer plan's gate from its own health.

- A gate keeps what its send carried (digest, sequence) and reads the
  report against it; a report on the last send is on it too.
- A gate judges only the build the machine was last sent: another build
  there supersedes the judging — no verdict, nothing put back.
- A controller is told its build (MESH_CONTROLLER_VERSION, ${version}
  in a process's env) and records how far it reads the store's schema;
  a put-back to a build that reaches less, or never said, is refused
  and the current build kept, said as urgent.
- A release's open gate holds other sends of its modules there, and a
  plan's own first send waits on it.
jschoubben force-pushed fix/a-gate-outlives-the-controller-and-judges-the-build-it-sent from 1de509e536 to e6e1e3bc89 2026-10-09 15:15:43 +00:00 Compare
Author
Contributor

Delivery novox/mesh-controller@e6e1e3bc8958 — delivered since 2026-10-09T16:23:35Z

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Machines

  • mesh-controller on novox: judging — carried by plan-1791562212728626197
  • build-agent on ace: passed — healthy 3 times over 3m39s
  • build-agent on the rest running it: sent
  • route-proxy on ace: passed — healthy 3 times over 3m45s
  • route-proxy on the rest running it: sent

Transitions

  • 2026-10-09 15:15 (new) → proposed (announced): novox/mesh-controller#185's head announced
  • 2026-10-09 15:21 proposed → checked (checked): the verdict names this commit
  • 2026-10-09 15:21 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-09 16:11 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-09 16:11 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
  • 2026-10-09 16:23 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-controller@e6e1e3bc8958` — **delivered** since 2026-10-09T16:23:35Z **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves **Machines** - mesh-controller on novox: judging — carried by plan-1791562212728626197 - build-agent on ace: passed — healthy 3 times over 3m39s - build-agent on the rest running it: sent - route-proxy on ace: passed — healthy 3 times over 3m45s - route-proxy on the rest running it: sent **Transitions** - 2026-10-09 15:15 (new) → proposed (announced): novox/mesh-controller#185's head announced - 2026-10-09 15:21 proposed → checked (checked): the verdict names this commit - 2026-10-09 15:21 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-09 16:11 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-09 16:11 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-09 16:23 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
Author
Contributor

Merge check at e6e1e3bc

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
  • mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on ace as build-1791558948668151721 (builds --log build-1791558948668151721).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 54 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T15:13:05Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T15:13:05Z

modules the change moves:
  - mesh-controller (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.035s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	199.912s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.018s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.028s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.018s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.133s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.073s
ok  	github.com/novox/mesh-controller/internal/broker	2.737s
ok  	github.com/novox/mesh-controller/internal/builder	1.997s
ok  	github.com/novox/mesh-controller/internal/catalogue	4.781s
ok  	github.com/novox/mesh-controller/internal/conditions	2.527s
ok  	github.com/novox/mesh-controller/internal/envfile	1.051s
ok  	github.com/novox/mesh-controller/internal/facts	1.073s
ok  	github.com/novox/mesh-controller/internal/identity	4.547s
ok  	github.com/novox/mesh-controller/internal/inventory	77.728s
ok  	github.com/novox/mesh-controller/internal/lease	17.706s
ok  	github.com/novox/mesh-controller/internal/licences	4.166s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.049s
ok  	github.com/novox/mesh-controller/internal/link	51.891s
ok  	github.com/novox/mesh-controller/internal/lint	2.154s
ok  	github.com/novox/mesh-controller/internal/outward	1.052s
ok  	github.com/novox/mesh-controller/internal/overlay	1.060s
ok  	github.com/novox/mesh-controller/internal/secrets	1.478s
ok  	github.com/novox/mesh-controller/internal/store	3.418s
ok  	github.com/novox/mesh-controller/internal/testbus	1.262s
ok  	github.com/novox/mesh-controller/internal/token	1.017s
**Merge check** at `e6e1e3bc` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on ace as `build-1791558948668151721` (`builds --log build-1791558948668151721`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 54 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 files asking for a setting without saying whether it is trusted: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T15:13:05Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-09T15:13:05Z modules the change moves: - mesh-controller (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.035s ok github.com/novox/mesh-controller/cmd/mesh-controller 199.912s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.018s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.028s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.018s ok github.com/novox/mesh-controller/examples/route-proxy 1.133s ok github.com/novox/mesh-controller/internal/artifacts 1.073s ok github.com/novox/mesh-controller/internal/broker 2.737s ok github.com/novox/mesh-controller/internal/builder 1.997s ok github.com/novox/mesh-controller/internal/catalogue 4.781s ok github.com/novox/mesh-controller/internal/conditions 2.527s ok github.com/novox/mesh-controller/internal/envfile 1.051s ok github.com/novox/mesh-controller/internal/facts 1.073s ok github.com/novox/mesh-controller/internal/identity 4.547s ok github.com/novox/mesh-controller/internal/inventory 77.728s ok github.com/novox/mesh-controller/internal/lease 17.706s ok github.com/novox/mesh-controller/internal/licences 4.166s ok github.com/novox/mesh-controller/internal/licences/adapters 1.049s ok github.com/novox/mesh-controller/internal/link 51.891s ok github.com/novox/mesh-controller/internal/lint 2.154s ok github.com/novox/mesh-controller/internal/outward 1.052s ok github.com/novox/mesh-controller/internal/overlay 1.060s ok github.com/novox/mesh-controller/internal/secrets 1.478s ok github.com/novox/mesh-controller/internal/store 3.418s ok github.com/novox/mesh-controller/internal/testbus 1.262s ok github.com/novox/mesh-controller/internal/token 1.017s ```
mesh-admin merged commit 1c7c385839 into main 2026-10-09 16:10:13 +00:00
mesh-admin deleted branch fix/a-gate-outlives-the-controller-and-judges-the-build-it-sent 2026-10-09 16:10:13 +00:00
This repo is archived. You cannot comment on pull requests.
No Reviewers
2 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-controller#185