novox/mesh-catalog
9
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

nats: say who opens and closes connections to the bus, and why each closed (hq issue 327, ADR 0265) #139

Merged
mesh-admin merged 2 commits from fix/327-a-verb-reads-on-the-serving-connection into main 2026-10-08 18:21:27 +00:00
Conversation 3 Commits 2 Files Changed 5
+244 -13
mesh-admin commented 2026-10-08 15:56:06 +00:00
Contributor
Copy Link
Copy Source

hq issue 327: the bus's connection total rose by hundreds an hour while the open count stayed flat, and nothing could say which clients those were. The server keeps a record of closed connections (/connz?state=closed), which no tool read.

  • New tool nats_closed_connections (args hours, default 1, and user): per user and connection name, how many closed in the window, how many were dropped (any reason but Client Closed), the reasons, the median time each lived, the rate per hour, and whether the server's record (its newest 10 000) reaches back the whole window.
  • The controller's self-check probe D15 asks it every run (mesh-controller, same group) and raises bus-reconnects for a user dropped more than 12 times an hour.
  • Tests: the counts against a captured record, and against a real server's monitoring endpoint (two connections a client closed, by user and name).

merge-check.sh with a controller built from the group's head as MESH_GATE: passed locally (testing modules/nats ok).

Delivery group: fix/327-a-verb-reads-on-the-serving-connection with mesh-controller, mesh-lab and hq.

hq issue 327: the bus's connection total rose by hundreds an hour while the open count stayed flat, and nothing could say which clients those were. The server keeps a record of closed connections (`/connz?state=closed`), which no tool read. - New tool **`nats_closed_connections`** (args `hours`, default 1, and `user`): per user and connection name, how many closed in the window, how many were *dropped* (any reason but `Client Closed`), the reasons, the median time each lived, the rate per hour, and whether the server's record (its newest 10 000) reaches back the whole window. - The controller's self-check probe D15 asks it every run (mesh-controller, same group) and raises `bus-reconnects` for a user dropped more than 12 times an hour. - Tests: the counts against a captured record, and against a real server's monitoring endpoint (two connections a client closed, by user and name). merge-check.sh with a controller built from the group's head as MESH_GATE: passed locally (`testing modules/nats` ok). Delivery group: `fix/327-a-verb-reads-on-the-serving-connection` with mesh-controller, mesh-lab and hq.
mesh-admin added 1 commit 2026-10-08 15:56:06 +00:00
Say who opens and closes connections to the bus, and why each closed (hq issue 327)
mesh/merge-gate pass: builds nats, sent nowhere; a bus step; 1 wait(s) for a person; every machine composes with the change as it did without (4 of 4 compo…
Details
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheImageIsTheServerTestedHere (0.00s)
Details
mesh/delivery-group group fix/327-a-verb-reads-on-the-serving-connection rejected: a member's own check failed
Details
mesh/delivery superseded: a newer head of the same pull request
Details
0063cc2ad1 
The connection total rose by hundreds an hour while the open count stayed
flat, and nothing could say which clients those were.
mesh-admin commented 2026-10-08 15:56:39 +00:00
Author
Contributor
Copy Link
Copy Source

Delivery novox/mesh-catalog@ea69e795ab9d — delivered since 2026-10-08T18:22:32Z

Delivery plan — builds nats, sent nowhere; a bus step; 1 wait(s) for a person

  • tier 0: nats
  • novox: waits for a person: nats
  • a planned bus step: the bus is upgraded by bus upgrade, never by an ordinary send
  • nats provides mesh-bus: its consumers are sent again after it
  • nats keeps data (ADR 0233): what it holds is backed up before it moves

Group fix/327-a-verb-reads-on-the-serving-connection, in order: novox/hq@0a38d9b21d → novox/mesh-controller@a1a7cf451a → novox/mesh-catalog@ea69e795ab → novox/mesh-lab@75854f5182

Transitions

  • 2026-10-08 16:07 (new) → proposed (announced): novox/mesh-catalog#139's head announced
  • 2026-10-08 16:09 proposed → checked (checked): the verdict names this commit
  • 2026-10-08 16:09 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-08 18:21 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-08 18:21 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
  • 2026-10-08 18:22 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-catalog@ea69e795ab9d` — **delivered** since 2026-10-08T18:22:32Z **Delivery plan** — builds nats, sent nowhere; a bus step; 1 wait(s) for a person - tier 0: nats - novox: waits for a person: nats - a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send - nats provides mesh-bus: its consumers are sent again after it - nats keeps data (ADR 0233): what it holds is backed up before it moves **Group** `fix/327-a-verb-reads-on-the-serving-connection`, in order: novox/hq@0a38d9b21d64 → novox/mesh-controller@a1a7cf451abd → novox/mesh-catalog@ea69e795ab9d → novox/mesh-lab@75854f5182fb - novox/mesh-controller@a1a7cf451abd before novox/mesh-catalog@ea69e795ab9d: built by - composed together: pass — every machine composes with the change as it did without (4 of 4 compose) **Transitions** - 2026-10-08 16:07 (new) → proposed (announced): novox/mesh-catalog#139's head announced - 2026-10-08 16:09 proposed → checked (checked): the verdict names this commit - 2026-10-08 16:09 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-08 18:21 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-08 18:21 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-08 18:22 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
mesh-admin commented 2026-10-08 15:59:44 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 0063cc2a

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: nats)
  • mesh/repo-check: FAIL — its merge-check.sh failed: --- FAIL: TestTheImageIsTheServerTestedHere (0.00s)

Delivery plan — builds nats, sent nowhere; a bus step; 1 wait(s) for a person

  • tier 0: nats
  • novox: waits for a person: nats
  • a planned bus step: the bus is upgraded by bus upgrade, never by an ordinary send
  • nats provides mesh-bus: its consumers are sent again after it
  • nats keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on novox as build-1791474993024009552 (builds --log build-1791474993024009552).

--- the gate: nats
nats: ok, 9 tool(s), keeps jetstream (valuable), snapshots (rebuildable); WARNING: server stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T15:52:40Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-08T15:52:40Z

modules the change moves:
  - nats (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): nats
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.002s
words: 12 retired words, none in the 30672 strings of 561 files
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.006s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 70, as kept
testing modules/nats
--- FAIL: TestTheImageIsTheServerTestedHere (0.00s)
    server_test.go:43: these tests run no nats-server
FAIL
FAIL	nats-tools/cmd/nats-tools	5.279s
FAIL
**Merge check** at `0063cc2a` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: nats) - `mesh/repo-check`: **FAIL** — its merge-check.sh failed: --- FAIL: TestTheImageIsTheServerTestedHere (0.00s) **Delivery plan** — builds nats, sent nowhere; a bus step; 1 wait(s) for a person - tier 0: nats - novox: waits for a person: nats - a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send - nats provides mesh-bus: its consumers are sent again after it - nats keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on novox as `build-1791474993024009552` (`builds --log build-1791474993024009552`). ``` --- the gate: nats nats: ok, 9 tool(s), keeps jetstream (valuable), snapshots (rebuildable); WARNING: server stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T15:52:40Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-08T15:52:40Z modules the change moves: - nats (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): nats --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.002s words: 12 retired words, none in the 30672 strings of 561 files ok git.novox.be/novox/mesh-catalog/checks/paths 0.006s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 70, as kept testing modules/nats --- FAIL: TestTheImageIsTheServerTestedHere (0.00s) server_test.go:43: these tests run no nats-server FAIL FAIL nats-tools/cmd/nats-tools 5.279s FAIL ```
jschoubben added 1 commit 2026-10-08 16:07:30 +00:00
Read the tested nats release from the server itself
mesh/merge-gate pass: builds nats, sent nowhere; a bus step; 1 wait(s) for a person; every machine composes with the change as it did without (4 of 4 compo…
Details
mesh/repo-check pass: its merge-check.sh passed
Details
mesh/delivery delivered
Details
mesh/delivery-group group fix/327-a-verb-reads-on-the-serving-connection merged: every member is merged
Details
ea69e795ab 
The build seat's test binary listed no dependencies, so the check that the
image and the tests run one release failed on a release that matched.
mesh-admin commented 2026-10-08 16:09:56 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at ea69e795

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: nats)
  • mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds nats, sent nowhere; a bus step; 1 wait(s) for a person

  • tier 0: nats
  • novox: waits for a person: nats
  • a planned bus step: the bus is upgraded by bus upgrade, never by an ordinary send
  • nats provides mesh-bus: its consumers are sent again after it
  • nats keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on novox as build-1791475659413208805 (builds --log build-1791475659413208805).

--- the gate: nats
nats: ok, 9 tool(s), keeps jetstream (valuable), snapshots (rebuildable); WARNING: server stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T16:06:42Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-08T16:06:42Z

modules the change moves:
  - nats (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): nats
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.002s
words: 12 retired words, none in the 30672 strings of 561 files
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.007s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 70, as kept
testing modules/nats
ok  	nats-tools/cmd/nats-tools	6.282s
**Merge check** at `ea69e795` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: nats) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds nats, sent nowhere; a bus step; 1 wait(s) for a person - tier 0: nats - novox: waits for a person: nats - a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send - nats provides mesh-bus: its consumers are sent again after it - nats keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on novox as `build-1791475659413208805` (`builds --log build-1791475659413208805`). ``` --- the gate: nats nats: ok, 9 tool(s), keeps jetstream (valuable), snapshots (rebuildable); WARNING: server stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T16:06:42Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-08T16:06:42Z modules the change moves: - nats (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): nats --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.002s words: 12 retired words, none in the 30672 strings of 561 files ok git.novox.be/novox/mesh-catalog/checks/paths 0.007s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 70, as kept testing modules/nats ok nats-tools/cmd/nats-tools 6.282s ```
mesh-admin merged commit 32307bd18b into main 2026-10-08 18:21:27 +00:00
mesh-admin deleted branch fix/327-a-verb-reads-on-the-serving-connection 2026-10-08 18:21:27 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
mesh/ready
Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-catalog#139