novox/mesh-controller
Archived
9
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Switch the memory store's failure under its lock, so a test cannot race the keeper #148

Merged
mesh-admin merged 1 commits from fix/conditions-store-race into main 2026-10-08 23:35:59 +00:00
Conversation 4 Commits 1 Files Changed 4
+75 -27
mesh-admin commented 2026-10-08 13:18:36 +00:00
Contributor
Copy Link
Copy Source

Why

go test -race ./internal/conditions failed once in TestAnUnreadableStoreClearsNothing (store_test.go:202 against memory.go:93).

Cause. The keeper keeps every transition from its own goroutine (telling), which calls InMemory.Append and reads InMemory.Fail under the store's lock. The test assigned the exported Fail field bare, with no lock. When the telling goroutine appended the transition from the test's first Observe in the window between that assignment and the test's next store call, the two accesses were unordered and the race detector failed the test. The window is narrow, so the failure was rare (0 of 100 isolated runs here).

Test-only. InMemory and Told are test doubles; the controller runs the bus store (OnTheBus), and no production code constructs InMemory. The keeper itself is sound.

What changes

  • InMemory.Fail and Told.Fail become unexported and are set only through SetFail, under the lock, so a test cannot reintroduce the race.
  • The planted unreadable entry in TestAnUnreadableStoreClearsNothing is written under the store's lock.
  • TestATransitionIsOfferedAgainWhileTheBusIsAway and cmd/mesh-controller's TestUnreadableConditionsAreNotAWellMesh use SetFail.
  • New TestTheStoreIsTakenAwayWhileTheKeeperIsTelling: 100 transitions still being kept while the failure is switched 100 times.

Evidence

  • The new test's body with the old bare field writes failed under -race in 30 of 30 separate processes; with SetFail it passed 30 of 30.
  • go test -race -count=50 ./internal/conditions: ok.
  • sh merge-check.sh: exit 0.
## Why `go test -race ./internal/conditions` failed once in `TestAnUnreadableStoreClearsNothing` (store_test.go:202 against memory.go:93). **Cause.** The keeper keeps every transition from its own goroutine (`telling`), which calls `InMemory.Append` and reads `InMemory.Fail` under the store's lock. The test assigned the exported `Fail` field bare, with no lock. When the telling goroutine appended the transition from the test's first `Observe` in the window between that assignment and the test's next store call, the two accesses were unordered and the race detector failed the test. The window is narrow, so the failure was rare (0 of 100 isolated runs here). **Test-only.** `InMemory` and `Told` are test doubles; the controller runs the bus store (`OnTheBus`), and no production code constructs `InMemory`. The keeper itself is sound. ## What changes - `InMemory.Fail` and `Told.Fail` become unexported and are set only through `SetFail`, under the lock, so a test cannot reintroduce the race. - The planted unreadable entry in `TestAnUnreadableStoreClearsNothing` is written under the store's lock. - `TestATransitionIsOfferedAgainWhileTheBusIsAway` and `cmd/mesh-controller`'s `TestUnreadableConditionsAreNotAWellMesh` use `SetFail`. - New `TestTheStoreIsTakenAwayWhileTheKeeperIsTelling`: 100 transitions still being kept while the failure is switched 100 times. ## Evidence - The new test's body with the old bare field writes failed under `-race` in 30 of 30 separate processes; with `SetFail` it passed 30 of 30. - `go test -race -count=50 ./internal/conditions`: ok. - `sh merge-check.sh`: exit 0.
mesh-admin commented 2026-10-08 13:18:49 +00:00
Author
Contributor
Copy Link
Copy Source

Delivery novox/mesh-controller@fe857ba0817d — delivered since 2026-10-08T23:51:25Z

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Machines

  • mesh-controller on novox: passed — healthy 3 times over 3m38s
  • mesh-controller on the rest running it: sent
  • build-agent on ace: passed — healthy 3 times over 3m43s
  • build-agent on the rest running it: sent
  • route-proxy on ace: passed — healthy 3 times over 3m16s
  • route-proxy on the rest running it: sent

Transitions

  • 2026-10-08 23:26 (new) → proposed (announced): novox/mesh-controller#148's head announced
  • 2026-10-08 23:31 proposed → checked (checked): the verdict names this commit
  • 2026-10-08 23:31 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-08 23:36 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-08 23:36 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
  • 2026-10-08 23:51 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-controller@fe857ba0817d` — **delivered** since 2026-10-08T23:51:25Z **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves **Machines** - mesh-controller on novox: passed — healthy 3 times over 3m38s - mesh-controller on the rest running it: sent - build-agent on ace: passed — healthy 3 times over 3m43s - build-agent on the rest running it: sent - route-proxy on ace: passed — healthy 3 times over 3m16s - route-proxy on the rest running it: sent **Transitions** - 2026-10-08 23:26 (new) → proposed (announced): novox/mesh-controller#148's head announced - 2026-10-08 23:31 proposed → checked (checked): the verdict names this commit - 2026-10-08 23:31 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-08 23:36 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-08 23:36 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-08 23:51 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
mesh-admin commented 2026-10-08 13:22:38 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 60e636c1

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
  • mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on ace as build-1791465519048050703 (builds --log build-1791465519048050703).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 50 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T13:13:50Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-08T13:13:50Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.038s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	144.101s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.012s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.015s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.012s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.105s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.030s
ok  	github.com/novox/mesh-controller/internal/broker	2.460s
ok  	github.com/novox/mesh-controller/internal/builder	1.712s
ok  	github.com/novox/mesh-controller/internal/catalogue	3.976s
ok  	github.com/novox/mesh-controller/internal/conditions	2.413s
ok  	github.com/novox/mesh-controller/internal/envfile	1.042s
ok  	github.com/novox/mesh-controller/internal/facts	1.043s
ok  	github.com/novox/mesh-controller/internal/identity	4.061s
ok  	github.com/novox/mesh-controller/internal/inventory	67.254s
ok  	github.com/novox/mesh-controller/internal/lease	17.698s
ok  	github.com/novox/mesh-controller/internal/licences	3.703s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.068s
ok  	github.com/novox/mesh-controller/internal/link	43.739s
ok  	github.com/novox/mesh-controller/internal/lint	1.891s
ok  	github.com/novox/mesh-controller/internal/outward	1.099s
ok  	github.com/novox/mesh-controller/internal/overlay	1.050s
ok  	github.com/novox/mesh-controller/internal/secrets	1.413s
ok  	github.com/novox/mesh-controller/internal/store	3.260s
ok  	github.com/novox/mesh-controller/internal/testbus	1.205s
ok  	github.com/novox/mesh-controller/internal/token	1.018s
**Merge check** at `60e636c1` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on ace as `build-1791465519048050703` (`builds --log build-1791465519048050703`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 50 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T13:13:50Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-08T13:13:50Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.038s ok github.com/novox/mesh-controller/cmd/mesh-controller 144.101s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.012s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.015s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.012s ok github.com/novox/mesh-controller/examples/route-proxy 1.105s ok github.com/novox/mesh-controller/internal/artifacts 1.030s ok github.com/novox/mesh-controller/internal/broker 2.460s ok github.com/novox/mesh-controller/internal/builder 1.712s ok github.com/novox/mesh-controller/internal/catalogue 3.976s ok github.com/novox/mesh-controller/internal/conditions 2.413s ok github.com/novox/mesh-controller/internal/envfile 1.042s ok github.com/novox/mesh-controller/internal/facts 1.043s ok github.com/novox/mesh-controller/internal/identity 4.061s ok github.com/novox/mesh-controller/internal/inventory 67.254s ok github.com/novox/mesh-controller/internal/lease 17.698s ok github.com/novox/mesh-controller/internal/licences 3.703s ok github.com/novox/mesh-controller/internal/licences/adapters 1.068s ok github.com/novox/mesh-controller/internal/link 43.739s ok github.com/novox/mesh-controller/internal/lint 1.891s ok github.com/novox/mesh-controller/internal/outward 1.099s ok github.com/novox/mesh-controller/internal/overlay 1.050s ok github.com/novox/mesh-controller/internal/secrets 1.413s ok github.com/novox/mesh-controller/internal/store 3.260s ok github.com/novox/mesh-controller/internal/testbus 1.205s ok github.com/novox/mesh-controller/internal/token 1.018s ```
mesh-admin commented 2026-10-08 23:15:18 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 60e636c1

  • mesh/merge-gate: ERROR — the merge gate could not judge the change: mesh-controller: the snapshot's dunst is not a manifest: json: unknown field "settings" (modules: build-agent, mesh-controller, route-proxy)
  • mesh/repo-check: FAIL — its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.79s)

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on shanks as build-1791501110809710260 (builds --log build-1791501110809710260).

release-1791501284472541991: sent laptop 2 build(s) that waited for a gate; judging them there
release-1791501284980175672: 2 build(s) wait for a gate on anchor, laptop; one machine at a time, each judged
release-1791501284980175672: sent anchor 1 build(s) that waited for a gate; judging them there
app's build c2 failed its gate on anchor and was put back to c1: anchor failed what it was sent
release-1791501284980175672: failed its gate on anchor: anchor failed what it was sent — app failed its gate on anchor in tier 0: anchor failed what it was sent; put back app to c1 on anchor, in one send
this act by hand could NOT be recorded in the hand-act log, and is done anyway: this control plane has no MESH_BUS_NATS, so it cannot reach the mesh's bus
release-1791501285057336624: 1 build(s) wait for a gate on laptop; one machine at a time, each judged
release-1791501285057336624 releases it; `plans release-1791501285057336624` says where it is
release-1791501285057336624: sent laptop 1 build(s) that waited for a gate; judging them there
release-1791501285537433182: 3 build(s) wait for a gate on laptop, workstation; one machine at a time, each judged
release-1791501285537433182: sent laptop 3 build(s) that waited for a gate; judging them there
release-1791501285537433182: sent workstation 3 build(s) that waited for a gate; judging them there
plan-crashloop: tier 0 built; sent 1 module(s) to anchor first in one send (app), the rest once its gate passes
app's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: its container app.server on anchor keeps restarting (2 restart(s) counted)
plan-crashloop: app failed its gate on anchor in tier 0: not healthy within -1s of its apply: its container app.server on anchor keeps restarting (2 restart(s) counted); put back app to c1 on anchor, in one send; none reported left as it was
novox/mesh-catalog merged into main (3ad22356); plan plan-1791501288774160900, 1 module(s) in 1 tier(s)
  0: app
  tier 0:   modules/systemd-resolved is a new module in novox/mesh-catalog: build build-modules/systemd-resolved asked at main; registered when it lands, assigned nowhere
novox/mesh-catalog merged into main (3da80a4b); it changed no module the mesh holds, and adds modules/systemd-resolved: built, registered when the build lands, and sent nowhere
build-1791497690126725438: the build seat did not say whether c1111111 is on its repository's trunk (it predates ADR 0238); registered as before
build-1791497750126725438: the build seat did not say whether c2222222 is on its repository's trunk (it predates ADR 0238); registered as before
recorded as act-1791501290236321040-1 in the hand-act log: a push of recorded builds (resolver from c1111111 → c2222222) by an unnamed account at a shell on Shanks, because "operator's yes: the resolver's new build on anchor" — the person's word their upgrade policy asks for, which no healer is wanted for
recorded as act-1791501290287008324-2 in the hand-act log: a push of recorded builds (resolver from c1111111 → c2222222) by an unnamed account at a shell on Shanks, because "operator's yes: the resolver's new build on laptop" — the person's word their upgrade policy asks for, which no healer is wanted for
laptop is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
laptop is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
build-1791497693268395679: the build seat did not say whether c1aaaaaa is on its repository's trunk (it predates ADR 0238); registered as before
build-1791497753268395679: the build seat did not say whether c2bbbbbb is on its repository's trunk (it predates ADR 0238); registered as before
app at c2bbbbbb was made from the source build-1791497693268395679 was: registered with its artifacts, no move
build-1791497813268395679: the build seat did not say whether c3cccccc is on its repository's trunk (it predates ADR 0238); registered as before
build-1791497873268395679: the build seat did not say whether c4dddddd is on its repository's trunk (it predates ADR 0238); registered as before
build-1791497693709772460: the build seat did not say whether n1111111 is on its repository's trunk (it predates ADR 0238); registered as before
build-1791497753709772460: the build seat did not say whether n2222222 is on its repository's trunk (it predates ADR 0238); registered as before
nats at n2222222 was made from the source build-1791497693709772460 was: registered with its artifacts, no move
every machine running nats runs the build the mesh holds (n2222222): nothing to upgrade
build-1791497813709772460: the build seat did not say whether n3333333 is on its repository's trunk (it predates ADR 0238); registered as before
plan-gate: app built from the source every machine running it runs: no move, nothing sent
plan-gate: done — novox/mesh-catalog at c2, 1 tier(s)
plan-gate: tier 0 built; sent 1 module(s) to anchor first in one send (app), the rest once its gate passes
status could not be composed (after 0s): the store did not answer — `status` answers the last summary, with its age
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
anchor is not counted as on the network: it does not resolve: these assignments cannot be applied:
  - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node
this act by hand could NOT be recorded in the hand-act log, and is done anyway: this control plane has no MESH_BUS_NATS, so it cannot reach the mesh's bus
plan-97b1b2b closed at tier 1 of 2; what was asked still builds and registers, nothing further is asked
plan-tier: tier 0 built; sent 12 module(s) to anchor first in one send (m01, m02, m03, m04, m05, m06, m07, m08, m09, m10, m11, m12), the rest once its gate passes
plan-tier: m01 passed its gate on anchor (healthy 3 times over 0s); the rest are sent
plan-tier: tier 0 built; sent m01, m02, m03, m04, m05, m06, m07, m08, m09, m10, m11, m12 to laptop, one send each
plan-tier: done — novox/mesh-catalog at c2, 1 tier(s)
plan-tier: tier 0 built; sent 3 module(s) to anchor first in one send (app1, app2, node-tools), the rest once its gate passes
node-tools's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: raised since it was sent: machine.anchor.core-behind — anchor runs the node tools it was last sent, not yet reported applied, and no plan is rolling them out (its settle window runs to the gate's bound)
plan-tier: the send to anchor in tier 0 failed its gate: not healthy within -1s of its apply: raised since it was sent: machine.anchor.core-behind — anchor runs the node tools it was last sent, not yet reported applied, and no plan is rolling them out (its settle window runs to the gate's bound); app1 was not found wanting and is left as it is; passed on their own and kept: app2; node-tools failed its gate on anchor in tier 0: not healthy within -1s of its apply: raised since it was sent: machine.anchor.core-behind — anchor runs the node tools it was last sent, not yet reported applied, and no plan is rolling them out (its settle window runs to the gate's bound); put back node-tools to c1 on anchor, in one send; none reported left as it was
plan-tier: tier 0 built; sent 3 module(s) to anchor first in one send (app1, app2, app3), the rest once its gate passes
app1's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full
app2's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full
app3's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full
plan-tier: app1 failed its gate on anchor in tier 0: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full; app2 failed its gate on anchor in tier 0: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full; app3 failed its gate on anchor in tier 0: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full; put back app1 to c1, app2 to c1, app3 to c1 on anchor, in one send; none reported left as it was
plan-gate: tier 0 built; sent 1 module(s) to anchor first in one send (app), the rest once its gate passes
plan-gate: the send to anchor in tier 0 failed its gate: not healthy within -1s of its apply: anchor has not reported on what it was sent; app was left as it was — anchor already ran app c2, or a build identical to it, before the send, so nothing of it moved and nothing is put back
plan-gate: the send to anchor in tier 0 failed its gate: not healthy within -1s of its apply: anchor has not reported on what it was sent; app was left as it was — anchor already ran app c2, or a build identical to it, before the send, so nothing of it moved and nothing is put back; none reported left as it was
  tier 0: FAIL
FAIL	github.com/novox/mesh-controller/cmd/mesh-controller	167.436s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.009s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.022s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.014s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.185s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.064s
--- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.79s)
    genesis_template_test.go:46: the installer's user list lets the controller publish [$JS.ACK.DEAD_LETTER_NOTICES.controller.> mesh.again.> mesh.events.dead.>], which it does not derive
FAIL
FAIL	github.com/novox/mesh-controller/internal/broker	1.680s
ok  	github.com/novox/mesh-controller/internal/builder	1.871s
--- FAIL: TestEveryCatalogueManifestParses (0.28s)
    catalogue_check_test.go:44: dunst: this is not a module manifest: json: unknown field "settings"
    catalogue_check_test.go:44: i3status-rust: this is not a module manifest: json: unknown field "renders"
    catalogue_check_test.go:44: mounts: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
    catalogue_check_test.go:44: nfs-server: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
    catalogue_check_test.go:44: power: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
    catalogue_check_test.go:57: 120 manifests, 70 named endpoints, 30 routed endpoints resolved
--- FAIL: TestNoCatalogueManifestNamesAnInstallation (0.80s)
    catalogue_check_test.go:80: ../../../mesh-catalog/modules/dunst/module.json: this is not a module manifest: json: unknown field "settings"
    catalogue_check_test.go:80: ../../../mesh-catalog/modules/i3status-rust/module.json: this is not a module manifest: json: unknown field "renders"
    catalogue_check_test.go:80: ../../../mesh-catalog/modules/mounts/module.json: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
    catalogue_check_test.go:80: ../../../mesh-catalog/modules/nfs-server/module.json: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
    catalogue_check_test.go:80: ../../../mesh-catalog/modules/power/module.json: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
--- FAIL: TestEveryCatalogueIdentityFitsWhatItRequires (0.04s)
    catalogue_check_test.go:142: ../../../mesh-catalog/modules/dunst/module.json: this is not a module manifest: json: unknown field "settings"
--- FAIL: TestEveryCatalogueManifestDeclaresWhatItMounts (0.25s)
    mounts_test.go:87: dunst: this is not a module manifest: json: unknown field "settings"
    mounts_test.go:87: i3status-rust: this is not a module manifest: json: unknown field "renders"
    mounts_test.go:87: mounts: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
    mounts_test.go:87: nfs-server: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
    mounts_test.go:87: power: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine"
--- FAIL: TestEveryManifestsEventNamesAreLocal (0.02s)
    no_subjects_test.go:82: dunst: json: unknown field "settings"
--- FAIL: TestTheCataloguesResolverHoldersWriteTheFileAndProvideSplitDNS (0.03s)
    node_resolver_test.go:156: dunst: json: unknown field "settings"
--- FAIL: TestTheCataloguesUplinksWriteTheResolverFileAndNothingElseDoes (0.02s)
    resolver_file_is_the_uplinks_test.go:26: dunst: json: unknown field "settings"
--- FAIL: TestEveryManifestInUseClaimsASeatTheMeshDefines (0.04s)
    seats_test.go:215: ../../../mesh-catalog/modules/dunst/module.json: json: unknown field "settings"
--- FAIL: TestEveryManifestsStateIsLocalAndEveryReadIsKept (0.03s)
    state_test.go:73: dunst: json: unknown field "settings"
FAIL
FAIL	github.com/novox/mesh-controller/internal/catalogue	2.886s
ok  	github.com/novox/mesh-controller/internal/conditions	2.535s
ok  	github.com/novox/mesh-controller/internal/envfile	1.043s
ok  	github.com/novox/mesh-controller/internal/facts	1.046s
ok  	github.com/novox/mesh-controller/internal/identity	4.815s
ok  	github.com/novox/mesh-controller/internal/inventory	82.388s
ok  	github.com/novox/mesh-controller/internal/lease	17.700s
ok  	github.com/novox/mesh-controller/internal/licences	4.407s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.045s
ok  	github.com/novox/mesh-controller/internal/link	46.755s
ok  	github.com/novox/mesh-controller/internal/lint	2.117s
ok  	github.com/novox/mesh-controller/internal/outward	1.074s
ok  	github.com/novox/mesh-controller/internal/overlay	1.056s
ok  	github.com/novox/mesh-controller/internal/secrets	1.514s
ok  	github.com/novox/mesh-controller/internal/store	3.587s
ok  	github.com/novox/mesh-controller/internal/testbus	1.237s
ok  	github.com/novox/mesh-controller/internal/token	1.017s
FAIL
**Merge check** at `60e636c1` - `mesh/merge-gate`: **ERROR** — the merge gate could not judge the change: mesh-controller: the snapshot's dunst is not a manifest: json: unknown field "settings" (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **FAIL** — its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.79s) **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on shanks as `build-1791501110809710260` (`builds --log build-1791501110809710260`). ``` release-1791501284472541991: sent laptop 2 build(s) that waited for a gate; judging them there release-1791501284980175672: 2 build(s) wait for a gate on anchor, laptop; one machine at a time, each judged release-1791501284980175672: sent anchor 1 build(s) that waited for a gate; judging them there app's build c2 failed its gate on anchor and was put back to c1: anchor failed what it was sent release-1791501284980175672: failed its gate on anchor: anchor failed what it was sent — app failed its gate on anchor in tier 0: anchor failed what it was sent; put back app to c1 on anchor, in one send this act by hand could NOT be recorded in the hand-act log, and is done anyway: this control plane has no MESH_BUS_NATS, so it cannot reach the mesh's bus release-1791501285057336624: 1 build(s) wait for a gate on laptop; one machine at a time, each judged release-1791501285057336624 releases it; `plans release-1791501285057336624` says where it is release-1791501285057336624: sent laptop 1 build(s) that waited for a gate; judging them there release-1791501285537433182: 3 build(s) wait for a gate on laptop, workstation; one machine at a time, each judged release-1791501285537433182: sent laptop 3 build(s) that waited for a gate; judging them there release-1791501285537433182: sent workstation 3 build(s) that waited for a gate; judging them there plan-crashloop: tier 0 built; sent 1 module(s) to anchor first in one send (app), the rest once its gate passes app's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: its container app.server on anchor keeps restarting (2 restart(s) counted) plan-crashloop: app failed its gate on anchor in tier 0: not healthy within -1s of its apply: its container app.server on anchor keeps restarting (2 restart(s) counted); put back app to c1 on anchor, in one send; none reported left as it was novox/mesh-catalog merged into main (3ad22356); plan plan-1791501288774160900, 1 module(s) in 1 tier(s) 0: app tier 0: modules/systemd-resolved is a new module in novox/mesh-catalog: build build-modules/systemd-resolved asked at main; registered when it lands, assigned nowhere novox/mesh-catalog merged into main (3da80a4b); it changed no module the mesh holds, and adds modules/systemd-resolved: built, registered when the build lands, and sent nowhere build-1791497690126725438: the build seat did not say whether c1111111 is on its repository's trunk (it predates ADR 0238); registered as before build-1791497750126725438: the build seat did not say whether c2222222 is on its repository's trunk (it predates ADR 0238); registered as before recorded as act-1791501290236321040-1 in the hand-act log: a push of recorded builds (resolver from c1111111 → c2222222) by an unnamed account at a shell on Shanks, because "operator's yes: the resolver's new build on anchor" — the person's word their upgrade policy asks for, which no healer is wanted for recorded as act-1791501290287008324-2 in the hand-act log: a push of recorded builds (resolver from c1111111 → c2222222) by an unnamed account at a shell on Shanks, because "operator's yes: the resolver's new build on laptop" — the person's word their upgrade policy asks for, which no healer is wanted for laptop is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node laptop is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node build-1791497693268395679: the build seat did not say whether c1aaaaaa is on its repository's trunk (it predates ADR 0238); registered as before build-1791497753268395679: the build seat did not say whether c2bbbbbb is on its repository's trunk (it predates ADR 0238); registered as before app at c2bbbbbb was made from the source build-1791497693268395679 was: registered with its artifacts, no move build-1791497813268395679: the build seat did not say whether c3cccccc is on its repository's trunk (it predates ADR 0238); registered as before build-1791497873268395679: the build seat did not say whether c4dddddd is on its repository's trunk (it predates ADR 0238); registered as before build-1791497693709772460: the build seat did not say whether n1111111 is on its repository's trunk (it predates ADR 0238); registered as before build-1791497753709772460: the build seat did not say whether n2222222 is on its repository's trunk (it predates ADR 0238); registered as before nats at n2222222 was made from the source build-1791497693709772460 was: registered with its artifacts, no move every machine running nats runs the build the mesh holds (n2222222): nothing to upgrade build-1791497813709772460: the build seat did not say whether n3333333 is on its repository's trunk (it predates ADR 0238); registered as before plan-gate: app built from the source every machine running it runs: no move, nothing sent plan-gate: done — novox/mesh-catalog at c2, 1 tier(s) plan-gate: tier 0 built; sent 1 module(s) to anchor first in one send (app), the rest once its gate passes status could not be composed (after 0s): the store did not answer — `status` answers the last summary, with its age anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is left out of the rest of the mesh: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node anchor is not counted as on the network: it does not resolve: these assignments cannot be applied: - rival-one and rival-two both claim "the-seat", and only one thing may hold it per node this act by hand could NOT be recorded in the hand-act log, and is done anyway: this control plane has no MESH_BUS_NATS, so it cannot reach the mesh's bus plan-97b1b2b closed at tier 1 of 2; what was asked still builds and registers, nothing further is asked plan-tier: tier 0 built; sent 12 module(s) to anchor first in one send (m01, m02, m03, m04, m05, m06, m07, m08, m09, m10, m11, m12), the rest once its gate passes plan-tier: m01 passed its gate on anchor (healthy 3 times over 0s); the rest are sent plan-tier: tier 0 built; sent m01, m02, m03, m04, m05, m06, m07, m08, m09, m10, m11, m12 to laptop, one send each plan-tier: done — novox/mesh-catalog at c2, 1 tier(s) plan-tier: tier 0 built; sent 3 module(s) to anchor first in one send (app1, app2, node-tools), the rest once its gate passes node-tools's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: raised since it was sent: machine.anchor.core-behind — anchor runs the node tools it was last sent, not yet reported applied, and no plan is rolling them out (its settle window runs to the gate's bound) plan-tier: the send to anchor in tier 0 failed its gate: not healthy within -1s of its apply: raised since it was sent: machine.anchor.core-behind — anchor runs the node tools it was last sent, not yet reported applied, and no plan is rolling them out (its settle window runs to the gate's bound); app1 was not found wanting and is left as it is; passed on their own and kept: app2; node-tools failed its gate on anchor in tier 0: not healthy within -1s of its apply: raised since it was sent: machine.anchor.core-behind — anchor runs the node tools it was last sent, not yet reported applied, and no plan is rolling them out (its settle window runs to the gate's bound); put back node-tools to c1 on anchor, in one send; none reported left as it was plan-tier: tier 0 built; sent 3 module(s) to anchor first in one send (app1, app2, app3), the rest once its gate passes app1's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full app2's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full app3's build c2 failed its gate on anchor and was put back to c1: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full plan-tier: app1 failed its gate on anchor in tier 0: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full; app2 failed its gate on anchor in tier 0: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full; app3 failed its gate on anchor in tier 0: not healthy within -1s of its apply: anchor as a whole: raised since it was sent: machine.anchor.disk-full — anchor's root file system is full; put back app1 to c1, app2 to c1, app3 to c1 on anchor, in one send; none reported left as it was plan-gate: tier 0 built; sent 1 module(s) to anchor first in one send (app), the rest once its gate passes plan-gate: the send to anchor in tier 0 failed its gate: not healthy within -1s of its apply: anchor has not reported on what it was sent; app was left as it was — anchor already ran app c2, or a build identical to it, before the send, so nothing of it moved and nothing is put back plan-gate: the send to anchor in tier 0 failed its gate: not healthy within -1s of its apply: anchor has not reported on what it was sent; app was left as it was — anchor already ran app c2, or a build identical to it, before the send, so nothing of it moved and nothing is put back; none reported left as it was tier 0: FAIL FAIL github.com/novox/mesh-controller/cmd/mesh-controller 167.436s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.009s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.022s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.014s ok github.com/novox/mesh-controller/examples/route-proxy 1.185s ok github.com/novox/mesh-controller/internal/artifacts 1.064s --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.79s) genesis_template_test.go:46: the installer's user list lets the controller publish [$JS.ACK.DEAD_LETTER_NOTICES.controller.> mesh.again.> mesh.events.dead.>], which it does not derive FAIL FAIL github.com/novox/mesh-controller/internal/broker 1.680s ok github.com/novox/mesh-controller/internal/builder 1.871s --- FAIL: TestEveryCatalogueManifestParses (0.28s) catalogue_check_test.go:44: dunst: this is not a module manifest: json: unknown field "settings" catalogue_check_test.go:44: i3status-rust: this is not a module manifest: json: unknown field "renders" catalogue_check_test.go:44: mounts: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" catalogue_check_test.go:44: nfs-server: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" catalogue_check_test.go:44: power: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" catalogue_check_test.go:57: 120 manifests, 70 named endpoints, 30 routed endpoints resolved --- FAIL: TestNoCatalogueManifestNamesAnInstallation (0.80s) catalogue_check_test.go:80: ../../../mesh-catalog/modules/dunst/module.json: this is not a module manifest: json: unknown field "settings" catalogue_check_test.go:80: ../../../mesh-catalog/modules/i3status-rust/module.json: this is not a module manifest: json: unknown field "renders" catalogue_check_test.go:80: ../../../mesh-catalog/modules/mounts/module.json: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" catalogue_check_test.go:80: ../../../mesh-catalog/modules/nfs-server/module.json: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" catalogue_check_test.go:80: ../../../mesh-catalog/modules/power/module.json: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" --- FAIL: TestEveryCatalogueIdentityFitsWhatItRequires (0.04s) catalogue_check_test.go:142: ../../../mesh-catalog/modules/dunst/module.json: this is not a module manifest: json: unknown field "settings" --- FAIL: TestEveryCatalogueManifestDeclaresWhatItMounts (0.25s) mounts_test.go:87: dunst: this is not a module manifest: json: unknown field "settings" mounts_test.go:87: i3status-rust: this is not a module manifest: json: unknown field "renders" mounts_test.go:87: mounts: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" mounts_test.go:87: nfs-server: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" mounts_test.go:87: power: this is not a module manifest: a state is either a name or {name, history, ttl-seconds}: json: unknown field "per-machine" --- FAIL: TestEveryManifestsEventNamesAreLocal (0.02s) no_subjects_test.go:82: dunst: json: unknown field "settings" --- FAIL: TestTheCataloguesResolverHoldersWriteTheFileAndProvideSplitDNS (0.03s) node_resolver_test.go:156: dunst: json: unknown field "settings" --- FAIL: TestTheCataloguesUplinksWriteTheResolverFileAndNothingElseDoes (0.02s) resolver_file_is_the_uplinks_test.go:26: dunst: json: unknown field "settings" --- FAIL: TestEveryManifestInUseClaimsASeatTheMeshDefines (0.04s) seats_test.go:215: ../../../mesh-catalog/modules/dunst/module.json: json: unknown field "settings" --- FAIL: TestEveryManifestsStateIsLocalAndEveryReadIsKept (0.03s) state_test.go:73: dunst: json: unknown field "settings" FAIL FAIL github.com/novox/mesh-controller/internal/catalogue 2.886s ok github.com/novox/mesh-controller/internal/conditions 2.535s ok github.com/novox/mesh-controller/internal/envfile 1.043s ok github.com/novox/mesh-controller/internal/facts 1.046s ok github.com/novox/mesh-controller/internal/identity 4.815s ok github.com/novox/mesh-controller/internal/inventory 82.388s ok github.com/novox/mesh-controller/internal/lease 17.700s ok github.com/novox/mesh-controller/internal/licences 4.407s ok github.com/novox/mesh-controller/internal/licences/adapters 1.045s ok github.com/novox/mesh-controller/internal/link 46.755s ok github.com/novox/mesh-controller/internal/lint 2.117s ok github.com/novox/mesh-controller/internal/outward 1.074s ok github.com/novox/mesh-controller/internal/overlay 1.056s ok github.com/novox/mesh-controller/internal/secrets 1.514s ok github.com/novox/mesh-controller/internal/store 3.587s ok github.com/novox/mesh-controller/internal/testbus 1.237s ok github.com/novox/mesh-controller/internal/token 1.017s FAIL ```
jschoubben added 1 commit 2026-10-08 23:26:28 +00:00
Switch the memory store's failure under its lock, so a test cannot race the keeper
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
Details
mesh/repo-check pass: its merge-check.sh passed
Details
mesh/delivery delivered
Details
fe857ba081 
The keeper keeps each transition from its own goroutine, which reads the memory
store's Fail field under the store's lock; tests assigned the exported field
bare, so TestAnUnreadableStoreClearsNothing failed under -race whenever the
goroutine appended in that window. The field is now set only through SetFail
(and Told's likewise), and a test makes the race certain rather than rare.
Test-only: the controller runs the bus store, never InMemory.
jschoubben force-pushed fix/conditions-store-race from 60e636c1e3 to fe857ba081 2026-10-08 23:26:28 +00:00 Compare
mesh-admin commented 2026-10-08 23:31:13 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at fe857ba0

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
  • mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on ace as build-1791502003338266414 (builds --log build-1791502003338266414).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T23:23:48Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-08T23:23:48Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.033s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	167.084s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.013s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.018s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.015s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.163s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.036s
ok  	github.com/novox/mesh-controller/internal/broker	2.603s
ok  	github.com/novox/mesh-controller/internal/builder	1.886s
ok  	github.com/novox/mesh-controller/internal/catalogue	4.285s
ok  	github.com/novox/mesh-controller/internal/conditions	2.431s
ok  	github.com/novox/mesh-controller/internal/envfile	1.067s
ok  	github.com/novox/mesh-controller/internal/facts	1.101s
ok  	github.com/novox/mesh-controller/internal/identity	4.175s
ok  	github.com/novox/mesh-controller/internal/inventory	71.774s
ok  	github.com/novox/mesh-controller/internal/lease	17.733s
ok  	github.com/novox/mesh-controller/internal/licences	3.811s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.094s
ok  	github.com/novox/mesh-controller/internal/link	50.770s
ok  	github.com/novox/mesh-controller/internal/lint	2.026s
ok  	github.com/novox/mesh-controller/internal/outward	1.088s
ok  	github.com/novox/mesh-controller/internal/overlay	1.106s
ok  	github.com/novox/mesh-controller/internal/secrets	1.497s
ok  	github.com/novox/mesh-controller/internal/store	3.333s
ok  	github.com/novox/mesh-controller/internal/testbus	1.271s
ok  	github.com/novox/mesh-controller/internal/token	1.016s
**Merge check** at `fe857ba0` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on ace as `build-1791502003338266414` (`builds --log build-1791502003338266414`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T23:23:48Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-08T23:23:48Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.033s ok github.com/novox/mesh-controller/cmd/mesh-controller 167.084s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.013s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.018s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.015s ok github.com/novox/mesh-controller/examples/route-proxy 1.163s ok github.com/novox/mesh-controller/internal/artifacts 1.036s ok github.com/novox/mesh-controller/internal/broker 2.603s ok github.com/novox/mesh-controller/internal/builder 1.886s ok github.com/novox/mesh-controller/internal/catalogue 4.285s ok github.com/novox/mesh-controller/internal/conditions 2.431s ok github.com/novox/mesh-controller/internal/envfile 1.067s ok github.com/novox/mesh-controller/internal/facts 1.101s ok github.com/novox/mesh-controller/internal/identity 4.175s ok github.com/novox/mesh-controller/internal/inventory 71.774s ok github.com/novox/mesh-controller/internal/lease 17.733s ok github.com/novox/mesh-controller/internal/licences 3.811s ok github.com/novox/mesh-controller/internal/licences/adapters 1.094s ok github.com/novox/mesh-controller/internal/link 50.770s ok github.com/novox/mesh-controller/internal/lint 2.026s ok github.com/novox/mesh-controller/internal/outward 1.088s ok github.com/novox/mesh-controller/internal/overlay 1.106s ok github.com/novox/mesh-controller/internal/secrets 1.497s ok github.com/novox/mesh-controller/internal/store 3.333s ok github.com/novox/mesh-controller/internal/testbus 1.271s ok github.com/novox/mesh-controller/internal/token 1.016s ```
mesh-admin merged commit 2a9697cf70 into main 2026-10-08 23:35:59 +00:00
mesh-admin deleted branch fix/conditions-store-race 2026-10-08 23:35:59 +00:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No Reviewers
Labels
Clear labels
mesh/ready
Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent
No Assignees
2 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-controller#148