Read how a module says each resource is ready, and send it to engines that read it (hq ADR 0240, to-be 48 Phase B) #110

Merged

mesh-admin merged 1 commits from feat/health-the-field into main 2026-10-07 16:32:26 +00:00

Conversation 4 Commits 1 Files Changed 22

+1205 -10

mesh-admin commented 2026-10-07 12:52:28 +00:00

Contributor

Copy Link

Copy Source

Phase B of to-be 48, the controller's half.

• health on every long-running manifest resource (internal/catalogue/health.go), refused in ParseManifest (so module check and registration) for: no/unknown kind, a port/address/host/url/ip, an unknown key, an endpoint not declared under listens or not tcp, interval < 10 s, timeout ≥ interval, looks < 2, grace + looks×interval > 5 min, a bad duration/number/path/status/scheme, http-only fields on another kind, exec without command, runtime/exec on a non-container, unit on a container, a tool the module does not serve, a tool check with no other kind on the module, needs the module does not require, health on a step / schedule / non-running service. A test per refusal.
• Composition (healthInto): the endpoint becomes the port this machine published it on, defaults written out — only for an engine whose statement says contract ≥ 2 (Rendering.ReadsHealth, from node_health); for an older, strict engine the field is stripped. A test that every composed kind passes the node-engine's own validator.
• Bus: the machine principal gets mesh.mod.<module>.tool.<tool>.<node> for each tool its modules' health asks — its own instance only (no genesis-lock change: the controller's own grants are unchanged).
• module check: per module ready: … and a WARNING for each long-running resource without health; the line long-running resources without health: N (the catalogue counter reads it, Phase E); refuses them from 2026-11-18 (six weeks after Phase A went live).
• Statements carry check/needs (kept in the JSON column, no migration). A check's findings (paths, addresses) stay out of the condition's summary (ADR 0234 §6) — found by the content rule test.
• Replay TestReplaySilentWebAppIsRaisedWithinTwoLooks (R145's controller half).
• vendor/: mesh-host pinned at its Phase B branch commit bdd4415 so the validator knows health. Move the pin to the merge commit once mesh-host's PR merges.

Order: merge + roll out mesh-host first; this controller is safe against older engines (it strips the field). The catalogue may only add health after this controller is live.

Full suite with a test store passes. check-here: merge-gate PASS (4 of 4 compose), repo-check PASS.

Phase B of to-be 48, the controller's half. - `health` on every long-running manifest resource (`internal/catalogue/health.go`), refused in `ParseManifest` (so `module check` and registration) for: no/unknown kind, a `port`/`address`/`host`/`url`/`ip`, an unknown key, an endpoint not declared under `listens` or not tcp, interval < 10 s, timeout ≥ interval, looks < 2, grace + looks×interval > 5 min, a bad duration/number/path/status/scheme, http-only fields on another kind, exec without command, runtime/exec on a non-container, unit on a container, a tool the module does not serve, a tool check with no other kind on the module, `needs` the module does not require, health on a step / schedule / non-running service. A test per refusal. - Composition (`healthInto`): the endpoint becomes the port this machine published it on, defaults written out — **only for an engine whose statement says contract ≥ 2** (`Rendering.ReadsHealth`, from `node_health`); for an older, strict engine the field is stripped. A test that every composed kind passes the node-engine's own validator. - Bus: the machine principal gets `mesh.mod.<module>.tool.<tool>.<node>` for each tool its modules' health asks — its own instance only (no genesis-lock change: the controller's own grants are unchanged). - `module check`: per module `ready: …` and a WARNING for each long-running resource without `health`; the line `long-running resources without health: N` (the catalogue counter reads it, Phase E); refuses them from 2026-11-18 (six weeks after Phase A went live). - Statements carry `check`/`needs` (kept in the JSON column, no migration). A check's findings (paths, addresses) stay out of the condition's summary (ADR 0234 §6) — found by the content rule test. - Replay `TestReplaySilentWebAppIsRaisedWithinTwoLooks` (R145's controller half). - **vendor/**: mesh-host pinned at its Phase B branch commit `bdd4415` so the validator knows `health`. Move the pin to the merge commit once mesh-host's PR merges. Order: merge + roll out mesh-host first; this controller is safe against older engines (it strips the field). The catalogue may only add `health` after this controller is live. Full suite with a test store passes. check-here: merge-gate PASS (4 of 4 compose), repo-check PASS.

mesh-admin commented 2026-10-07 12:59:58 +00:00

Author

Contributor

Copy Link

Copy Source

Delivery novox/mesh-controller@b98fd0f396c8 — covered since 2026-10-09T07:31:59Z

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Group feat/health-the-field, in order: novox/mesh-lab@d93f1e4eab

• novox/mesh-controller@b98fd0f396 before novox/mesh-host@96bf9415aa: built by
• novox/mesh-host@96bf9415aa before novox/mesh-controller@b98fd0f396: engine before controller

Transitions

• 2026-10-07 14:18 (new) → proposed (announced): novox/mesh-controller#110's head announced
• 2026-10-07 14:27 proposed → checked (checked): the verdict names this commit
• 2026-10-07 14:27 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
• 2026-10-07 16:42 ready → held (hold): something holds it for a person: its group's composed check did not pass for the heads that merged, or it merged and no walk was opened for it
• 2026-10-09 07:31 held → covered (covered): covered: a later merge that contains it was delivered: novox/mesh-controller@4291fee68e (merged as a5a132ac into main, walk plan-1791390776050013480): build-agent, mesh-controller, route-proxy

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-controller@b98fd0f396c8` — **covered** since 2026-10-09T07:31:59Z **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves **Group** `feat/health-the-field`, in order: novox/mesh-lab@d93f1e4eab73 - novox/mesh-controller@b98fd0f396c8 before novox/mesh-host@96bf9415aae4: built by - novox/mesh-host@96bf9415aae4 before novox/mesh-controller@b98fd0f396c8: engine before controller **Transitions** - 2026-10-07 14:18 (new) → proposed (announced): novox/mesh-controller#110's head announced - 2026-10-07 14:27 proposed → checked (checked): the verdict names this commit - 2026-10-07 14:27 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-07 16:42 ready → held (hold): something holds it for a person: its group's composed check did not pass for the heads that merged, or it merged and no walk was opened for it - 2026-10-09 07:31 held → covered (covered): covered: a later merge that contains it was delivered: novox/mesh-controller@4291fee68e95 (merged as a5a132ac into main, walk plan-1791390776050013480): build-agent, mesh-controller, route-proxy The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin referenced this pull request 2026-10-07 13:07:29 +00:00

Hold a consumer's findings under its unhealthy provider, and say them once there (hq ADR 0240, to-be 48 Phase C) #111

mesh-admin commented 2026-10-07 13:24:31 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at 468d5094

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
• mesh/repo-check: PASS — its merge-check.sh passed

Change plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on shanks as build-1791378896745632406 (builds --log build-1791378896745632406).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 46 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T13:06:37Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-07T13:06:37Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.035s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	446.858s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.022s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.042s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.033s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.224s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.070s
ok  	github.com/novox/mesh-controller/internal/broker	2.529s
ok  	github.com/novox/mesh-controller/internal/builder	1.489s
ok  	github.com/novox/mesh-controller/internal/catalogue	4.222s
ok  	github.com/novox/mesh-controller/internal/conditions	1.997s
ok  	github.com/novox/mesh-controller/internal/envfile	1.062s
ok  	github.com/novox/mesh-controller/internal/facts	1.058s
ok  	github.com/novox/mesh-controller/internal/identity	44.261s
ok  	github.com/novox/mesh-controller/internal/inventory	282.644s
ok  	github.com/novox/mesh-controller/internal/lease	17.691s
ok  	github.com/novox/mesh-controller/internal/licences	40.380s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.046s
ok  	github.com/novox/mesh-controller/internal/link	87.042s
ok  	github.com/novox/mesh-controller/internal/lint	1.905s
ok  	github.com/novox/mesh-controller/internal/outward	1.084s
ok  	github.com/novox/mesh-controller/internal/overlay	1.068s
ok  	github.com/novox/mesh-controller/internal/secrets	1.501s
ok  	github.com/novox/mesh-controller/internal/store	10.915s
ok  	github.com/novox/mesh-controller/internal/testbus	1.214s
ok  	github.com/novox/mesh-controller/internal/token	1.012s

**Merge check** at `468d5094` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Change plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on shanks as `build-1791378896745632406` (`builds --log build-1791378896745632406`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 46 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T13:06:37Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-07T13:06:37Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.035s ok github.com/novox/mesh-controller/cmd/mesh-controller 446.858s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.022s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.042s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.033s ok github.com/novox/mesh-controller/examples/route-proxy 1.224s ok github.com/novox/mesh-controller/internal/artifacts 1.070s ok github.com/novox/mesh-controller/internal/broker 2.529s ok github.com/novox/mesh-controller/internal/builder 1.489s ok github.com/novox/mesh-controller/internal/catalogue 4.222s ok github.com/novox/mesh-controller/internal/conditions 1.997s ok github.com/novox/mesh-controller/internal/envfile 1.062s ok github.com/novox/mesh-controller/internal/facts 1.058s ok github.com/novox/mesh-controller/internal/identity 44.261s ok github.com/novox/mesh-controller/internal/inventory 282.644s ok github.com/novox/mesh-controller/internal/lease 17.691s ok github.com/novox/mesh-controller/internal/licences 40.380s ok github.com/novox/mesh-controller/internal/licences/adapters 1.046s ok github.com/novox/mesh-controller/internal/link 87.042s ok github.com/novox/mesh-controller/internal/lint 1.905s ok github.com/novox/mesh-controller/internal/outward 1.084s ok github.com/novox/mesh-controller/internal/overlay 1.068s ok github.com/novox/mesh-controller/internal/secrets 1.501s ok github.com/novox/mesh-controller/internal/store 10.915s ok github.com/novox/mesh-controller/internal/testbus 1.214s ok github.com/novox/mesh-controller/internal/token 1.012s ```

mesh-admin commented 2026-10-07 13:33:08 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at 468d5094

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
• mesh/repo-check: FAIL — its merge-check.sh failed: FAIL github.com/novox/mesh-controller/cmd/mesh-controller 1800.059s

Change plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on novox as build-1791377986374127161 (builds --log build-1791377986374127161).

github.com/nats-io/nats-server/v2/server.(*fileStore).flushStreamStateLoop(0xc0005cd408, 0xc0001e6ee0, 0xc0001e6f50)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/filestore.go:10009 +0x1cc
created by github.com/nats-io/nats-server/v2/server.newFileStoreWithCreated in goroutine 6467
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/filestore.go:603 +0x2c8b

goroutine 6455 [IO wait]:
internal/poll.runtime_pollWait(0x7f9f383c9600, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0005f4020, 0x72, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005f4000, {0xc0003ef200, 0x200, 0x200})
	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x499
net.(*netFD).Read(0xc0005f4000, {0xc0003ef200, 0x200, 0x200})
	/usr/local/go/src/net/fd_posix.go:68 +0x4b
net.(*conn).Read(0xc000650000, {0xc0003ef200, 0x200, 0x200})
	/usr/local/go/src/net/net.go:196 +0xad
github.com/nats-io/nats-server/v2/server.(*client).readLoop(0xc000846a08, {0x0, 0x0, 0x0})
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/client.go:1420 +0x8ee
github.com/nats-io/nats-server/v2/server.(*Server).createClientEx.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3417 +0x55
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6411
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6305 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).startRateLimitLogExpiration.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:4552 +0x253
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6453 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).internalReceiveLoop(0xc000918008, 0xc000352b00)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/events.go:474 +0x12e
created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:1824 +0xa6d

goroutine 6472 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6470 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6469 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6473 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6454 [IO wait]:
internal/poll.runtime_pollWait(0x7f9ef017a400, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000352c20, 0x72, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000352c00)
	/usr/local/go/src/internal/poll/fd_unix.go:613 +0x4ea
net.(*netFD).accept(0xc000352c00)
	/usr/local/go/src/net/fd_unix.go:150 +0x45
net.(*TCPListener).accept(0xc0003f6000)
	/usr/local/go/src/net/tcpsock_posix.go:159 +0x4e
net.(*TCPListener).Accept(0xc0003f6000)
	/usr/local/go/src/net/tcpsock.go:387 +0x65
github.com/nats-io/nats-server/v2/server.(*Server).acceptConnections(0xc000918008, {0x203af20, 0xc0003f6000}, {0x1f49308, 0x6}, 0xc000516030, 0xc000516040)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:2819 +0x71
created by github.com/nats-io/nats-server/v2/server.(*Server).AcceptLoop in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:2768 +0xbdd

goroutine 6466 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6452 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).internalReceiveLoop(0xc000918008, 0xc000352a80)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/events.go:474 +0x12e
created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:1822 +0x9ad

goroutine 6467 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6474 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6450 [select]:
github.com/nats-io/nats-server/v2/server.(*sendq).internalLoop(0xc0005987a0)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/sendq.go:65 +0x439
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6468 [select]:
github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac
github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a
created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8

goroutine 6174 [select]:
github.com/nats-io/nats-server/v2/server.(*consumerFileStore).flushLoop(0xc0000f68f0, 0xc0005fe460, 0xc0005fe540)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/filestore.go:11007 +0x16e
created by github.com/nats-io/nats-server/v2/server.(*fileStore).ConsumerStore in goroutine 6467
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/filestore.go:10878 +0x17cb

goroutine 6436 [select]:
github.com/jackc/pgx/v5/pgxpool.(*Pool).backgroundHealthCheck(0xc0001600e0)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:493 +0x1cc
github.com/jackc/pgx/v5/pgxpool.NewWithConfig.func5()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:336 +0x8f
created by github.com/jackc/pgx/v5/pgxpool.NewWithConfig in goroutine 6303
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:333 +0xc1d

goroutine 6532 [select]:
github.com/jackc/pgx/v5/pgxpool.(*Pool).backgroundHealthCheck(0xc00091c2a0)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:493 +0x1cc
github.com/jackc/pgx/v5/pgxpool.NewWithConfig.func5()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:336 +0x8f
created by github.com/jackc/pgx/v5/pgxpool.NewWithConfig in goroutine 6303
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:333 +0xc1d

goroutine 6566 [select]:
github.com/nats-io/nats-server/v2/server.(*consumer).processInboundNextMsgReqs(0xc00032b088, 0xc000003650)
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/consumer.go:4550 +0x1c9
github.com/nats-io/nats-server/v2/server.(*consumer).setLeader.func4()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/consumer.go:1544 +0x74
created by github.com/nats-io/nats-server/v2/server.(*consumer).setLeader in goroutine 6472
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/consumer.go:1542 +0x1cf3

goroutine 6567 [select]:
github.com/nats-io/nats%2ego.(*Subscription).Fetch.func3()
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats.go/js.go:3304 +0xca
created by github.com/nats-io/nats%2ego.(*Subscription).Fetch in goroutine 6303
	/var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats.go/js.go:3303 +0x160b
FAIL	github.com/novox/mesh-controller/cmd/mesh-controller	1800.059s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.025s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.030s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.021s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.131s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.052s
ok  	github.com/novox/mesh-controller/internal/broker	2.711s
ok  	github.com/novox/mesh-controller/internal/builder	1.679s
ok  	github.com/novox/mesh-controller/internal/catalogue	4.335s
ok  	github.com/novox/mesh-controller/internal/conditions	2.036s
ok  	github.com/novox/mesh-controller/internal/envfile	1.029s
ok  	github.com/novox/mesh-controller/internal/facts	1.038s
ok  	github.com/novox/mesh-controller/internal/identity	249.739s
ok  	github.com/novox/mesh-controller/internal/inventory	1461.481s
ok  	github.com/novox/mesh-controller/internal/lease	17.654s
ok  	github.com/novox/mesh-controller/internal/licences	228.029s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.010s
ok  	github.com/novox/mesh-controller/internal/link	320.761s
ok  	github.com/novox/mesh-controller/internal/lint	1.865s
ok  	github.com/novox/mesh-controller/internal/outward	1.030s
ok  	github.com/novox/mesh-controller/internal/overlay	1.019s
ok  	github.com/novox/mesh-controller/internal/secrets	1.404s
ok  	github.com/novox/mesh-controller/internal/store	80.437s
ok  	github.com/novox/mesh-controller/internal/testbus	1.167s
ok  	github.com/novox/mesh-controller/internal/token	1.015s
FAIL

**Merge check** at `468d5094` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **FAIL** — its merge-check.sh failed: FAIL github.com/novox/mesh-controller/cmd/mesh-controller 1800.059s **Change plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on novox as `build-1791377986374127161` (`builds --log build-1791377986374127161`). ``` github.com/nats-io/nats-server/v2/server.(*fileStore).flushStreamStateLoop(0xc0005cd408, 0xc0001e6ee0, 0xc0001e6f50) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/filestore.go:10009 +0x1cc created by github.com/nats-io/nats-server/v2/server.newFileStoreWithCreated in goroutine 6467 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/filestore.go:603 +0x2c8b goroutine 6455 [IO wait]: internal/poll.runtime_pollWait(0x7f9f383c9600, 0x72) /usr/local/go/src/runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc0005f4020, 0x72, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xb1 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0005f4000, {0xc0003ef200, 0x200, 0x200}) /usr/local/go/src/internal/poll/fd_unix.go:165 +0x499 net.(*netFD).Read(0xc0005f4000, {0xc0003ef200, 0x200, 0x200}) /usr/local/go/src/net/fd_posix.go:68 +0x4b net.(*conn).Read(0xc000650000, {0xc0003ef200, 0x200, 0x200}) /usr/local/go/src/net/net.go:196 +0xad github.com/nats-io/nats-server/v2/server.(*client).readLoop(0xc000846a08, {0x0, 0x0, 0x0}) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/client.go:1420 +0x8ee github.com/nats-io/nats-server/v2/server.(*Server).createClientEx.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3417 +0x55 github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6411 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6305 [select]: github.com/nats-io/nats-server/v2/server.(*Server).startRateLimitLogExpiration.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:4552 +0x253 github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6453 [select]: github.com/nats-io/nats-server/v2/server.(*Server).internalReceiveLoop(0xc000918008, 0xc000352b00) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/events.go:474 +0x12e created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:1824 +0xa6d goroutine 6472 [select]: github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6470 [select]: github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6469 [select]: github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6473 [select]: github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6454 [IO wait]: internal/poll.runtime_pollWait(0x7f9ef017a400, 0x72) /usr/local/go/src/runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc000352c20, 0x72, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xb1 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc000352c00) /usr/local/go/src/internal/poll/fd_unix.go:613 +0x4ea net.(*netFD).accept(0xc000352c00) /usr/local/go/src/net/fd_unix.go:150 +0x45 net.(*TCPListener).accept(0xc0003f6000) /usr/local/go/src/net/tcpsock_posix.go:159 +0x4e net.(*TCPListener).Accept(0xc0003f6000) /usr/local/go/src/net/tcpsock.go:387 +0x65 github.com/nats-io/nats-server/v2/server.(*Server).acceptConnections(0xc000918008, {0x203af20, 0xc0003f6000}, {0x1f49308, 0x6}, 0xc000516030, 0xc000516040) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:2819 +0x71 created by github.com/nats-io/nats-server/v2/server.(*Server).AcceptLoop in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:2768 +0xbdd goroutine 6466 [select]: github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6452 [select]: github.com/nats-io/nats-server/v2/server.(*Server).internalReceiveLoop(0xc000918008, 0xc000352a80) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/events.go:474 +0x12e created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:1822 +0x9ad goroutine 6467 [select]: github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6474 [select]: github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6450 [select]: github.com/nats-io/nats-server/v2/server.(*sendq).internalLoop(0xc0005987a0) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/sendq.go:65 +0x439 github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6468 [select]: github.com/nats-io/nats-server/v2/server.(*Server).processJSAPIRoutedRequests(0xc000918008) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/jetstream_api.go:914 +0x2ac github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3917 +0x5a created by github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine in goroutine 6304 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/server.go:3915 +0x1a8 goroutine 6174 [select]: github.com/nats-io/nats-server/v2/server.(*consumerFileStore).flushLoop(0xc0000f68f0, 0xc0005fe460, 0xc0005fe540) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/filestore.go:11007 +0x16e created by github.com/nats-io/nats-server/v2/server.(*fileStore).ConsumerStore in goroutine 6467 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/filestore.go:10878 +0x17cb goroutine 6436 [select]: github.com/jackc/pgx/v5/pgxpool.(*Pool).backgroundHealthCheck(0xc0001600e0) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:493 +0x1cc github.com/jackc/pgx/v5/pgxpool.NewWithConfig.func5() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:336 +0x8f created by github.com/jackc/pgx/v5/pgxpool.NewWithConfig in goroutine 6303 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:333 +0xc1d goroutine 6532 [select]: github.com/jackc/pgx/v5/pgxpool.(*Pool).backgroundHealthCheck(0xc00091c2a0) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:493 +0x1cc github.com/jackc/pgx/v5/pgxpool.NewWithConfig.func5() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:336 +0x8f created by github.com/jackc/pgx/v5/pgxpool.NewWithConfig in goroutine 6303 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/jackc/pgx/v5/pgxpool/pool.go:333 +0xc1d goroutine 6566 [select]: github.com/nats-io/nats-server/v2/server.(*consumer).processInboundNextMsgReqs(0xc00032b088, 0xc000003650) /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/consumer.go:4550 +0x1c9 github.com/nats-io/nats-server/v2/server.(*consumer).setLeader.func4() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/consumer.go:1544 +0x74 created by github.com/nats-io/nats-server/v2/server.(*consumer).setLeader in goroutine 6472 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats-server/v2/server/consumer.go:1542 +0x1cf3 goroutine 6567 [select]: github.com/nats-io/nats%2ego.(*Subscription).Fetch.func3() /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats.go/js.go:3304 +0xca created by github.com/nats-io/nats%2ego.(*Subscription).Fetch in goroutine 6303 /var/lib/build-agent/workspace/check/mesh-controller/vendor/github.com/nats-io/nats.go/js.go:3303 +0x160b FAIL github.com/novox/mesh-controller/cmd/mesh-controller 1800.059s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.025s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.030s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.021s ok github.com/novox/mesh-controller/examples/route-proxy 1.131s ok github.com/novox/mesh-controller/internal/artifacts 1.052s ok github.com/novox/mesh-controller/internal/broker 2.711s ok github.com/novox/mesh-controller/internal/builder 1.679s ok github.com/novox/mesh-controller/internal/catalogue 4.335s ok github.com/novox/mesh-controller/internal/conditions 2.036s ok github.com/novox/mesh-controller/internal/envfile 1.029s ok github.com/novox/mesh-controller/internal/facts 1.038s ok github.com/novox/mesh-controller/internal/identity 249.739s ok github.com/novox/mesh-controller/internal/inventory 1461.481s ok github.com/novox/mesh-controller/internal/lease 17.654s ok github.com/novox/mesh-controller/internal/licences 228.029s ok github.com/novox/mesh-controller/internal/licences/adapters 1.010s ok github.com/novox/mesh-controller/internal/link 320.761s ok github.com/novox/mesh-controller/internal/lint 1.865s ok github.com/novox/mesh-controller/internal/outward 1.030s ok github.com/novox/mesh-controller/internal/overlay 1.019s ok github.com/novox/mesh-controller/internal/secrets 1.404s ok github.com/novox/mesh-controller/internal/store 80.437s ok github.com/novox/mesh-controller/internal/testbus 1.167s ok github.com/novox/mesh-controller/internal/token 1.015s FAIL ```

jschoubben added 1 commit 2026-10-07 14:17:53 +00:00

Read how a module says each resource is ready, and send it to engines that read it (hq ADR 0240, to-be 48 Phase B) ... b98fd0f396

A module could say nothing about what ready means for what it runs, so a web
application with its port open and its requests hanging passed everything for
eleven hours (issue 145). A long-running resource now carries `health` — the
image's own check adopted by name, http, tcp, exec, unit or a module's own tool,
with its timing — refused near its author when it names a port or an address,
an endpoint the module does not declare, a tool it does not serve, a tool check
alone, or a timing outside the record's bounds. It is composed with the endpoint
as the port this machine published it on, and sent only to a node-engine whose
statement says it reads it: an older one would refuse the whole declaration.
The engine is granted its own machine's instance of each health tool. `module
check` warns of every long-running resource without `health`, counts them for
the catalogue, and refuses them from 2026-11-18. A check's findings stay out of
a condition's summary. The node-engine's validator is vendored at its Phase B
commit, so what is composed is judged by the words the engine takes.

jschoubben force-pushed feat/health-the-field from 468d509462 to b98fd0f396 2026-10-07 14:17:53 +00:00 Compare

mesh-admin commented 2026-10-07 14:27:58 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at b98fd0f3

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
• mesh/repo-check: PASS — its merge-check.sh passed

Change plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on ace as build-1791382696710557555 (builds --log build-1791382696710557555).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 46 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T13:56:37Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-07T13:56:37Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.030s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	395.004s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.013s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.067s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.013s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.130s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.029s
ok  	github.com/novox/mesh-controller/internal/broker	2.407s
ok  	github.com/novox/mesh-controller/internal/builder	1.415s
ok  	github.com/novox/mesh-controller/internal/catalogue	3.544s
ok  	github.com/novox/mesh-controller/internal/conditions	2.024s
ok  	github.com/novox/mesh-controller/internal/envfile	1.063s
ok  	github.com/novox/mesh-controller/internal/facts	1.085s
ok  	github.com/novox/mesh-controller/internal/identity	39.793s
ok  	github.com/novox/mesh-controller/internal/inventory	241.833s
ok  	github.com/novox/mesh-controller/internal/lease	17.715s
ok  	github.com/novox/mesh-controller/internal/licences	32.533s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.080s
ok  	github.com/novox/mesh-controller/internal/link	79.190s
ok  	github.com/novox/mesh-controller/internal/lint	1.763s
ok  	github.com/novox/mesh-controller/internal/outward	1.090s
ok  	github.com/novox/mesh-controller/internal/overlay	1.076s
ok  	github.com/novox/mesh-controller/internal/secrets	1.427s
ok  	github.com/novox/mesh-controller/internal/store	18.863s
ok  	github.com/novox/mesh-controller/internal/testbus	1.221s
ok  	github.com/novox/mesh-controller/internal/token	1.017s

**Merge check** at `b98fd0f3` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Change plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on ace as `build-1791382696710557555` (`builds --log build-1791382696710557555`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 46 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T13:56:37Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-07T13:56:37Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.030s ok github.com/novox/mesh-controller/cmd/mesh-controller 395.004s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.013s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.067s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.013s ok github.com/novox/mesh-controller/examples/route-proxy 1.130s ok github.com/novox/mesh-controller/internal/artifacts 1.029s ok github.com/novox/mesh-controller/internal/broker 2.407s ok github.com/novox/mesh-controller/internal/builder 1.415s ok github.com/novox/mesh-controller/internal/catalogue 3.544s ok github.com/novox/mesh-controller/internal/conditions 2.024s ok github.com/novox/mesh-controller/internal/envfile 1.063s ok github.com/novox/mesh-controller/internal/facts 1.085s ok github.com/novox/mesh-controller/internal/identity 39.793s ok github.com/novox/mesh-controller/internal/inventory 241.833s ok github.com/novox/mesh-controller/internal/lease 17.715s ok github.com/novox/mesh-controller/internal/licences 32.533s ok github.com/novox/mesh-controller/internal/licences/adapters 1.080s ok github.com/novox/mesh-controller/internal/link 79.190s ok github.com/novox/mesh-controller/internal/lint 1.763s ok github.com/novox/mesh-controller/internal/outward 1.090s ok github.com/novox/mesh-controller/internal/overlay 1.076s ok github.com/novox/mesh-controller/internal/secrets 1.427s ok github.com/novox/mesh-controller/internal/store 18.863s ok github.com/novox/mesh-controller/internal/testbus 1.221s ok github.com/novox/mesh-controller/internal/token 1.017s ```

mesh-admin merged commit 7f25666cee into main 2026-10-07 16:32:26 +00:00

mesh-admin deleted branch feat/health-the-field 2026-10-07 16:32:26 +00:00

mesh-admin referenced this pull request 2026-10-07 16:50:26 +00:00

Vendor the node-engine's validator at its merged Phase B, not the branch commit #112

This repo is archived. You cannot comment on pull requests.

Reviewers

No Reviewers

Labels

Clear labels

mesh/ready

Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent

No Assignees

2 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#110