hq issue 352 (hq PR #252). On 2026-10-09, 16:19-16:36 local on the control node: a release's gate read the machine's report against a newer send another plan had just made there, failed three builds the machine had reported healthy ("has not reported on what it was sent"), put them back on every machine to a controller older than the store's schema, and that controller then passed the newer plan's gate from its own health.
D1 — a gate reads the report against its own send.PlanGate.Sent keeps, per machine, the declaration the send carried (digest and sequence, read from the node's record right after the send; kept with the plan, so a restarted controller reads the same). A report is on it when it names that declaration, one sequenced after it, or the declaration sent last. judgeHealth and the merge plan's first-machine wait (nextRollout) read this; a gate kept without it reads the last send, as before.
D2 — a gate judges only the build the machine was last sent. Each gate judges sent_builds[machine][module] against the commit it judges; another build there is a new verdict, superseded: decided at once, no verdict on the build, nothing marked, nothing put back. A superseded release ends and what still waits is released again; a superseded merge plan ends, said. A module the gate itself put back at once (Returned) is not read as superseded.
D3 — no put-back to a controller older than the store's schema. The controller's declaration tells it which build it is (MESH_CONTROLLER_VERSION: ${version} in module.json; ${version} is now composed inside a process's env too). When it serves it records the highest migration it carries by that version (controller_schema, migration 0087). A failed controller gate reads the record of the build it would put back: one reaching less than the store has applied, or that never recorded, is not put back — the current build is kept and rollback-failed (urgent) says why. The lease holder's Build is that version.
Serialised walks.walkedBy reads a release's open gate as its walk, and a plan's own first send is held by another walk of the same module on that machine with another build (the same build there is already there). The merge plan's send at 16:31:37 would have waited for the release's gate.
Tests: TestReplay352 fails on the commit before the fix with "has not reported on what it was sent"; TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere, TestAPlansJudgingIsSupersededByASendThatMovesItsModule, TestAReportIsHeldAgainstTheGatesOwnSend, TestAGateJudgesOnlyTheBuildTheMachineWasLastSent, TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine, TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema, TestTheHolderNamesTheBuildTheDeclarationToldIt; inventory TestASendAndAReportAreKnownByTheirDeclaration, TestASchemaReachIsKeptByBuild; catalogue TestAProcessEnvCanNameTheBuildsOwnVersion. 32 mutations, each failing a test. Two fixtures were completed (a machine that "has not reported" now reports on no send at all).
Local merge-check.sh: every package passes but internal/broker's installer test, which fails on main as well when the mesh-host checkout beside it is behind — see the thread. Not merged; no rollout is triggered by this PR.
hq issue 352 (hq PR #252). On 2026-10-09, 16:19-16:36 local on the control node: a release's gate read the machine's report against a newer send another plan had just made there, failed three builds the machine had reported healthy ("has not reported on what it was sent"), put them back on every machine to a controller older than the store's schema, and that controller then passed the newer plan's gate from its own health.
**D1 — a gate reads the report against its own send.** `PlanGate.Sent` keeps, per machine, the declaration the send carried (digest and sequence, read from the node's record right after the send; kept with the plan, so a restarted controller reads the same). A report is on it when it names that declaration, one sequenced after it, or the declaration sent last. `judgeHealth` and the merge plan's first-machine wait (`nextRollout`) read this; a gate kept without it reads the last send, as before.
**D2 — a gate judges only the build the machine was last sent.** Each gate judges `sent_builds[machine][module]` against the commit it judges; another build there is a new verdict, `superseded`: decided at once, no verdict on the build, nothing marked, nothing put back. A superseded release ends and what still waits is released again; a superseded merge plan ends, said. A module the gate itself put back at once (`Returned`) is not read as superseded.
**D3 — no put-back to a controller older than the store's schema.** The controller's declaration tells it which build it is (`MESH_CONTROLLER_VERSION: ${version}` in module.json; `${version}` is now composed inside a process's env too). When it serves it records the highest migration it carries by that version (`controller_schema`, migration 0087). A failed controller gate reads the record of the build it would put back: one reaching less than the store has applied, or that never recorded, is not put back — the current build is kept and `rollback-failed` (urgent) says why. The lease holder's `Build` is that version.
**Serialised walks.** `walkedBy` reads a release's open gate as its walk, and a plan's own first send is held by another walk of the same module on that machine with another build (the same build there is already there). The merge plan's send at 16:31:37 would have waited for the release's gate.
Tests: `TestReplay352` fails on the commit before the fix with "has not reported on what it was sent"; `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule`, `TestAReportIsHeldAgainstTheGatesOwnSend`, `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent`, `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, `TestTheHolderNamesTheBuildTheDeclarationToldIt`; inventory `TestASendAndAReportAreKnownByTheirDeclaration`, `TestASchemaReachIsKeptByBuild`; catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`. 32 mutations, each failing a test. Two fixtures were completed (a machine that "has not reported" now reports on no send at all).
Local `merge-check.sh`: every package passes but `internal/broker`'s installer test, which fails on main as well when the mesh-host checkout beside it is behind — see the thread. Not merged; no rollout is triggered by this PR.
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
On 2026-10-09 a release's gate on the control node read the machine's
report against a newer send another plan had just made there, failed
three builds the machine had reported healthy, and put them back on
every machine to a controller older than the store's schema; that
controller then passed the newer plan's gate from its own health.
- A gate keeps what its send carried (digest, sequence) and reads the
report against it; a report on the last send is on it too.
- A gate judges only the build the machine was last sent: another build
there supersedes the judging — no verdict, nothing put back.
- A controller is told its build (MESH_CONTROLLER_VERSION, ${version}
in a process's env) and records how far it reads the store's schema;
a put-back to a build that reaches less, or never said, is refused
and the current build kept, said as urgent.
- A release's open gate holds other sends of its modules there, and a
plan's own first send waits on it.
jschoubben
force-pushed fix/a-gate-outlives-the-controller-and-judges-the-build-it-sent from 1de509e536 to e6e1e3bc892026-10-09 15:15:43 +00:00Compare
build-agent keeps data (ADR 0233): what it holds is backed up before it moves
route-proxy provides route: its consumers are sent again after it
route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
Machines
mesh-controller on novox: judging — carried by plan-1791562212728626197
build-agent on ace: passed — healthy 3 times over 3m39s
build-agent on the rest running it: sent
route-proxy on ace: passed — healthy 3 times over 3m45s
route-proxy on the rest running it: sent
Transitions
2026-10-09 15:15 (new) → proposed (announced): novox/mesh-controller#185's head announced
2026-10-09 15:21 proposed → checked (checked): the verdict names this commit
2026-10-09 15:21 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-09 16:11 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-09 16:11 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
2026-10-09 16:23 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says
The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.
<!-- mesh-delivery:view -->
**Delivery** `novox/mesh-controller@e6e1e3bc8958` — **delivered** since 2026-10-09T16:23:35Z
**Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
- tier 0: mesh-controller
- tier 1: build-agent
- tier 2: route-proxy
- ace: receives build-agent, route-proxy
- g14: receives build-agent
- novox: receives mesh-controller, build-agent, route-proxy
- shanks: receives build-agent
- build-agent keeps data (ADR 0233): what it holds is backed up before it moves
- route-proxy provides route: its consumers are sent again after it
- route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
**Machines**
- mesh-controller on novox: judging — carried by plan-1791562212728626197
- build-agent on ace: passed — healthy 3 times over 3m39s
- build-agent on the rest running it: sent
- route-proxy on ace: passed — healthy 3 times over 3m45s
- route-proxy on the rest running it: sent
**Transitions**
- 2026-10-09 15:15 (new) → proposed (announced): novox/mesh-controller#185's head announced
- 2026-10-09 15:21 proposed → checked (checked): the verdict names this commit
- 2026-10-09 15:21 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
- 2026-10-09 16:11 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
- 2026-10-09 16:11 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
- 2026-10-09 16:23 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says
The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
build-agent keeps data (ADR 0233): what it holds is backed up before it moves
route-proxy provides route: its consumers are sent again after it
route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
Run by the build seat on ace as build-1791558948668151721 (builds --log build-1791558948668151721).
--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 54 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T15:13:05Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T15:13:05Z
modules the change moves:
- mesh-controller (changed)
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-controller/cmd/mesh-builder 1.035s
ok github.com/novox/mesh-controller/cmd/mesh-controller 199.912s
ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.018s
ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.028s
ok github.com/novox/mesh-controller/examples/redis-provisioner 1.018s
ok github.com/novox/mesh-controller/examples/route-proxy 1.133s
ok github.com/novox/mesh-controller/internal/artifacts 1.073s
ok github.com/novox/mesh-controller/internal/broker 2.737s
ok github.com/novox/mesh-controller/internal/builder 1.997s
ok github.com/novox/mesh-controller/internal/catalogue 4.781s
ok github.com/novox/mesh-controller/internal/conditions 2.527s
ok github.com/novox/mesh-controller/internal/envfile 1.051s
ok github.com/novox/mesh-controller/internal/facts 1.073s
ok github.com/novox/mesh-controller/internal/identity 4.547s
ok github.com/novox/mesh-controller/internal/inventory 77.728s
ok github.com/novox/mesh-controller/internal/lease 17.706s
ok github.com/novox/mesh-controller/internal/licences 4.166s
ok github.com/novox/mesh-controller/internal/licences/adapters 1.049s
ok github.com/novox/mesh-controller/internal/link 51.891s
ok github.com/novox/mesh-controller/internal/lint 2.154s
ok github.com/novox/mesh-controller/internal/outward 1.052s
ok github.com/novox/mesh-controller/internal/overlay 1.060s
ok github.com/novox/mesh-controller/internal/secrets 1.478s
ok github.com/novox/mesh-controller/internal/store 3.418s
ok github.com/novox/mesh-controller/internal/testbus 1.262s
ok github.com/novox/mesh-controller/internal/token 1.017s
**Merge check** at `e6e1e3bc`
- `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
- `mesh/repo-check`: **PASS** — its merge-check.sh passed
**Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
- tier 0: mesh-controller
- tier 1: build-agent
- tier 2: route-proxy
- ace: receives build-agent, route-proxy
- g14: receives build-agent
- novox: receives mesh-controller, build-agent, route-proxy
- shanks: receives build-agent
- build-agent keeps data (ADR 0233): what it holds is backed up before it moves
- route-proxy provides route: its consumers are sent again after it
- route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
Run by the build seat on ace as `build-1791558948668151721` (`builds --log build-1791558948668151721`).
```
--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 54 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T15:13:05Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T15:13:05Z
modules the change moves:
- mesh-controller (changed)
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-controller/cmd/mesh-builder 1.035s
ok github.com/novox/mesh-controller/cmd/mesh-controller 199.912s
ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.018s
ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.028s
ok github.com/novox/mesh-controller/examples/redis-provisioner 1.018s
ok github.com/novox/mesh-controller/examples/route-proxy 1.133s
ok github.com/novox/mesh-controller/internal/artifacts 1.073s
ok github.com/novox/mesh-controller/internal/broker 2.737s
ok github.com/novox/mesh-controller/internal/builder 1.997s
ok github.com/novox/mesh-controller/internal/catalogue 4.781s
ok github.com/novox/mesh-controller/internal/conditions 2.527s
ok github.com/novox/mesh-controller/internal/envfile 1.051s
ok github.com/novox/mesh-controller/internal/facts 1.073s
ok github.com/novox/mesh-controller/internal/identity 4.547s
ok github.com/novox/mesh-controller/internal/inventory 77.728s
ok github.com/novox/mesh-controller/internal/lease 17.706s
ok github.com/novox/mesh-controller/internal/licences 4.166s
ok github.com/novox/mesh-controller/internal/licences/adapters 1.049s
ok github.com/novox/mesh-controller/internal/link 51.891s
ok github.com/novox/mesh-controller/internal/lint 2.154s
ok github.com/novox/mesh-controller/internal/outward 1.052s
ok github.com/novox/mesh-controller/internal/overlay 1.060s
ok github.com/novox/mesh-controller/internal/secrets 1.478s
ok github.com/novox/mesh-controller/internal/store 3.418s
ok github.com/novox/mesh-controller/internal/testbus 1.262s
ok github.com/novox/mesh-controller/internal/token 1.017s
```
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
hq issue 352 (hq PR #252). On 2026-10-09, 16:19-16:36 local on the control node: a release's gate read the machine's report against a newer send another plan had just made there, failed three builds the machine had reported healthy ("has not reported on what it was sent"), put them back on every machine to a controller older than the store's schema, and that controller then passed the newer plan's gate from its own health.
D1 — a gate reads the report against its own send.
PlanGate.Sentkeeps, per machine, the declaration the send carried (digest and sequence, read from the node's record right after the send; kept with the plan, so a restarted controller reads the same). A report is on it when it names that declaration, one sequenced after it, or the declaration sent last.judgeHealthand the merge plan's first-machine wait (nextRollout) read this; a gate kept without it reads the last send, as before.D2 — a gate judges only the build the machine was last sent. Each gate judges
sent_builds[machine][module]against the commit it judges; another build there is a new verdict,superseded: decided at once, no verdict on the build, nothing marked, nothing put back. A superseded release ends and what still waits is released again; a superseded merge plan ends, said. A module the gate itself put back at once (Returned) is not read as superseded.D3 — no put-back to a controller older than the store's schema. The controller's declaration tells it which build it is (
MESH_CONTROLLER_VERSION: ${version}in module.json;${version}is now composed inside a process's env too). When it serves it records the highest migration it carries by that version (controller_schema, migration 0087). A failed controller gate reads the record of the build it would put back: one reaching less than the store has applied, or that never recorded, is not put back — the current build is kept androllback-failed(urgent) says why. The lease holder'sBuildis that version.Serialised walks.
walkedByreads a release's open gate as its walk, and a plan's own first send is held by another walk of the same module on that machine with another build (the same build there is already there). The merge plan's send at 16:31:37 would have waited for the release's gate.Tests:
TestReplay352fails on the commit before the fix with "has not reported on what it was sent";TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere,TestAPlansJudgingIsSupersededByASendThatMovesItsModule,TestAReportIsHeldAgainstTheGatesOwnSend,TestAGateJudgesOnlyTheBuildTheMachineWasLastSent,TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine,TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema,TestTheHolderNamesTheBuildTheDeclarationToldIt; inventoryTestASendAndAReportAreKnownByTheirDeclaration,TestASchemaReachIsKeptByBuild; catalogueTestAProcessEnvCanNameTheBuildsOwnVersion. 32 mutations, each failing a test. Two fixtures were completed (a machine that "has not reported" now reports on no send at all).Local
merge-check.sh: every package passes butinternal/broker's installer test, which fails on main as well when the mesh-host checkout beside it is behind — see the thread. Not merged; no rollout is triggered by this PR.On 2026-10-09 a release's gate on the control node read the machine's report against a newer send another plan had just made there, failed three builds the machine had reported healthy, and put them back on every machine to a controller older than the store's schema; that controller then passed the newer plan's gate from its own health. - A gate keeps what its send carried (digest, sequence) and reads the report against it; a report on the last send is on it too. - A gate judges only the build the machine was last sent: another build there supersedes the judging — no verdict, nothing put back. - A controller is told its build (MESH_CONTROLLER_VERSION, ${version} in a process's env) and records how far it reads the store's schema; a put-back to a build that reaches less, or never said, is refused and the current build kept, said as urgent. - A release's open gate holds other sends of its modules there, and a plan's own first send waits on it.1de509e536toe6e1e3bc89Delivery
novox/mesh-controller@e6e1e3bc8958— delivered since 2026-10-09T16:23:35ZDelivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
Machines
Transitions
The commit's note under
refs/notes/mesh-plankeeps every transition:git log --notes=mesh-plan.Merge check at
e6e1e3bcmesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
Run by the build seat on ace as
build-1791558948668151721(builds --log build-1791558948668151721).