ADR 0265 and issue 327: answer read verbs on the serving connection and name every connection
Hundreds of short connections an hour, all named mesh-controller, hid any client reconnecting in a loop.
This commit is contained in:
jochen
+115
@@ -0,0 +1,115 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-08
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 265. A verb that reads is answered on the serving controller's own connection, and every connection says what it is
|
||||
|
||||
## Context
|
||||
|
||||
The controller serves its seat's verbs by running the command each names, in a fresh process of its own
|
||||
binary ([ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md)). The logic lives once,
|
||||
in the command, and each command opens its own stores and its own bus connection, as it does from a shell.
|
||||
|
||||
[Issue 327](../04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md)
|
||||
measured what that costs on the bus. The server's connection total rose by 5.2 a minute while the same 16
|
||||
connections stayed open, and three calls of `conditions` in one second added three. The operator's channel
|
||||
reads `conditions` at least once a minute. The serving controller dialled too: each watchdog tick while a
|
||||
walk waited for a build, to read whether the build seat paused, and each build it asked for. Every one of
|
||||
those connections was named `mesh-controller`, as were the build agents' long-lived ones.
|
||||
|
||||
[Research 036](../01-RESEARCH/036-everything-over-nats/00-overview.md) asks the bus to watch itself (G7),
|
||||
and a client reconnecting in a loop is one of the things it should say. The connection total is the
|
||||
number that would show it, and hundreds of short connections an hour that are normal hide it. The server
|
||||
also keeps a record of the connections it closed, with why (`/connz?state=closed`), which nothing read.
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Where a read verb is answered
|
||||
|
||||
1. **Keep every verb a fresh process, and only name its connection.** The rate stays; a loop is still
|
||||
found only by reading names. Rejected as the whole answer; kept for the verbs that act.
|
||||
2. **Answer every verb in the serving process.** A command prints to the process's output, and commands
|
||||
that act take holds and leases a shell command takes; moving all of them at once risks the
|
||||
answers being mixed up or an act behaving differently. Rejected.
|
||||
3. **Answer the verbs that only read the bus, in the serving process, by running the same command
|
||||
function writing into the answer.** `conditions` (its list, one condition, its history), `hand-acts`
|
||||
and `queue`. The logic still lives once, in the command; only where its words are written changes.
|
||||
Chosen.
|
||||
|
||||
### What the serving controller does for a moment
|
||||
|
||||
1. **Dial and close, as before.** Rejected: a connection per watchdog tick.
|
||||
2. **Lend the serving connection**: whatever the serving controller does for a moment is done on its own
|
||||
connection, and the code that closes what it was handed closes nothing. Chosen.
|
||||
|
||||
### How a reconnect loop is said
|
||||
|
||||
1. **Count the server's connection total.** It counts clients of short connections too. Rejected.
|
||||
2. **Read the server's record of closed connections**, per user and name, and say a user whose
|
||||
connections were dropped (closed by anything but its own client) too often. Chosen. The bus's module
|
||||
reads it, since only it reaches the server's monitoring; the controller's self-check asks it.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **`conditions` (list, show, history), `hand-acts` and `queue` are answered by the serving controller**,
|
||||
on its own connection and its own condition keeper, by the same command function the shell runs,
|
||||
writing into the answer. A call of any of them opens no connection. `conditions silence` and every verb
|
||||
that acts still run as their own process.
|
||||
2. **The serving controller lends its connection** to what it does for a moment: the hand-act log, the
|
||||
build queue, a build's log, whether the build seat paused, whether the bus answers, a build it asks for.
|
||||
A lent connection's close closes nothing. Its long-lived connections are two: the one it serves on and
|
||||
the one that holds the lease.
|
||||
3. **Every connection says what it is**, in the server's list:
|
||||
- `mesh-controller serving`, and `mesh-controller serving lease`;
|
||||
- `mesh-controller verb <verb>` for a verb run as its own process;
|
||||
- `mesh-controller command <command>` for a command run at a shell;
|
||||
- `build agent on <machine>` for a build agent, which no longer uses the controller's name.
|
||||
4. **The bus's module has a tool, `nats_closed_connections`**: per user and connection name, how many
|
||||
connections closed in the last hours, how many were dropped rather than closed by their client, why,
|
||||
how long each lived, and the rate per hour. It says whether the server's record (its newest ten
|
||||
thousand) reaches back the whole window.
|
||||
5. **Self-check probe D15** asks that tool every run, for the last hour, and raises `bus-reconnects` (a
|
||||
warning) for each user whose connections were dropped more than twelve times in the hour, a provisional
|
||||
bound. Its words say who keeps losing the bus and that what it says and is asked waits while it
|
||||
reconnects. It needs nothing of the operator. A bus module older than the tool is passed over.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The connection total of a quiet hour is the long-lived clients' reconnects, the verbs that act, and the
|
||||
commands people run.
|
||||
- A verb answered in the serving controller is answered by the build that serves, not by the binary on
|
||||
disk, which is what a fresh process ran. Since issue 289 the fresh process runs the serving image too, so
|
||||
nothing changes in which build answers.
|
||||
- A verb that runs as its own process still opens a connection: the delivery owner's `delivery-check`,
|
||||
`deliver` and `delivery-stop`, a silence, every act. Each now says which verb it is.
|
||||
- D15 counts a user, not a connection name. A module whose runtime and own process share a user are counted
|
||||
together. That is acceptable at a bound of twelve an hour.
|
||||
- The serving controller no longer opens a connection per build it asks for, nor per watchdog tick.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- mesh-controller `cmd/mesh-controller/replay327_test.go`, **R327** in mesh-lab's replays register: three
|
||||
`conditions` calls served by the serving controller open no connection to a bus that counts them. It
|
||||
fails on the commit before the fix (three connections) and passes on it.
|
||||
- mesh-controller `cmd/mesh-controller/reconnects_test.go`: the serving controller lends its connection
|
||||
and `hand-acts` and `queue` open none; a verb run as its own process names its connection
|
||||
`mesh-controller verb conditions`; each process's name; D15 says a user dropped forty times an hour in
|
||||
plain words and not one whose client closes its own short connections; the bus module is asked on its
|
||||
machine over the bus.
|
||||
- mesh-controller `internal/broker/jetstream_test.go`: a connection carries the process's name unless the
|
||||
caller names it; a lent connection's close leaves its owner's open.
|
||||
- mesh-catalog `modules/nats`: the tool's counts per user and name, dropped and why, against a captured
|
||||
record and against a real server's monitoring endpoint.
|
||||
- **Live**, once delivered: `nats.nats_closed_connections` lists no `mesh-controller verb conditions`;
|
||||
`nats.nats_connections` names the serving controller, its lease and the build agents apart.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 327](../04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md)
|
||||
- [Research 036](../01-RESEARCH/036-everything-over-nats/00-overview.md) (G7)
|
||||
- [ADR 0154](0154-the-meshs-own-verbs-are-the-controller-seats-tools.md);
|
||||
[to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §4
|
||||
@@ -4,6 +4,7 @@ status: in-progress
|
||||
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
|
||||
updated: 2026-10-08
|
||||
decisions:
|
||||
- 02-DECISIONS/0265-a-verb-that-reads-is-answered-on-the-serving-controllers-own-connection-and-every-connection-says-what-it-is.md
|
||||
- 02-DECISIONS/0261-an-assignment-made-while-its-modules-build-runs-is-kept-pending-and-the-controller-makes-it.md
|
||||
- 02-DECISIONS/0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md
|
||||
- 02-DECISIONS/0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md
|
||||
@@ -278,6 +279,7 @@ producer that wrote it.
|
||||
| D11 | no provider holds a consumer retired more than thirty days: each provider assigned is asked `provisioner_retirement` on its machine (ADR 0230); one that does not serve it yet is named, not failed | ADR 0230 |
|
||||
| D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (`binding-kept`, `binding-moving`, `binding-moved`) | ADR 0232 |
|
||||
| D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's `node-backup` holder is asked `backed-up`; every provider of kept consumer data `provisioner_retirement` (with each held consumer's size) — see §2 for the kinds | ADR 0233, issue 273 |
|
||||
| D15 | no user of the bus had its connection dropped more than twelve times in the last hour (*provisional*): the bus module's `nats_closed_connections`, from the server's record of closed connections; `bus-reconnects` (ADR 0265) | 327 |
|
||||
| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
|
||||
| H-* | the health probes of §8, run for every core component on every machine: H-controller, H-engine, H-tools, H-bus (as built, [ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)) | rule 8 |
|
||||
| DG | every build that failed its gate, and every core build a witness put back, is said — `build.<module>.<machine>.rolled-back`, `core.<component>.<machine>.<outcome>`, `rollback-failed` — until a newer build passes or the reports stop saying it (ADR 0236) | rule 8 |
|
||||
|
||||
+24
-2
@@ -2,8 +2,9 @@
|
||||
status: located
|
||||
opened: 2026-10-08
|
||||
located-in: [mesh-controller cmd/mesh-controller (seatverbs.go, a verb runs as a fresh process; handacts.go onTheBus, acting.go forACommand, queue.go dialTheBus, localDoctor, each opening its own connection; plan_retry.go buildSeatPause, a connection per watchdog tick), mesh-controller internal/broker (jetstream.go Dial, every connection named "mesh-controller")]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
fixed-by: mesh-controller#161, mesh-catalog#139, mesh-lab#71
|
||||
replay: R327
|
||||
amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
|
||||
---
|
||||
|
||||
# 327. Every controller verb opens its own bus connection, so the connection count hides a reconnect loop
|
||||
@@ -69,3 +70,24 @@ why none is possible.
|
||||
- Where a fresh process stays, its connection is named for what it is (`mesh-controller verb
|
||||
<verb>`), and the build agent names its own as the build agent.
|
||||
- The `nats` module gains the closed-connection tool above.
|
||||
|
||||
## Resolution
|
||||
|
||||
Decided in [ADR 0265](../../02-DECISIONS/0265-a-verb-that-reads-is-answered-on-the-serving-controllers-own-connection-and-every-connection-says-what-it-is.md),
|
||||
and [to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §4 names its probe:
|
||||
|
||||
- **`conditions` (its list, one condition, its history), `hand-acts` and `queue` are answered by the serving
|
||||
controller** on its own connection and keeper, by the same command writing into the answer. Three
|
||||
`conditions` calls open no connection.
|
||||
- **The serving controller lends its connection** to what it does for a moment: the build seat's pause on a
|
||||
watchdog tick, the bus's readiness, a build it asks for, the queue, the hand-act log.
|
||||
- **Every connection says what it is**: `mesh-controller serving`, `mesh-controller serving lease`,
|
||||
`mesh-controller verb <verb>`, `mesh-controller command <command>`, and `build agent on <machine>`.
|
||||
- **The bus's module answers `nats_closed_connections`**: per user and name, the connections closed in the
|
||||
last hours, which were dropped, why, and how long each lived.
|
||||
- **Probe D15** raises `bus-reconnects` for a user whose connections were dropped more than twelve times in
|
||||
an hour.
|
||||
|
||||
**Replay R327** (mesh-controller `cmd/mesh-controller/replay327_test.go`, registered in mesh-lab) fails on
|
||||
the commit before the fix, with three connections, and passes on it.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user