Console: wait for every runtime that answered, name the ones it missed, add mesh_runtimes #13

2026-10-05T13:36:32Z

mesh-admin commented

2026-10-05 13:36:32 +00:00

Symptom (2026-10-05, g14)

mesh_machine {node: g14} answered modules: null, mesh_call g14/slack.slack_check answered "nothing in the mesh is called slack", and g14/nextcloud-client.nextcloud_check once answered "does not run on g14; it runs on shanks" and worked a minute later. g14's runtime was serving 275 tools for 48 modules and logged nothing about it.

Root cause, measured

A read-only probe on the runtime's own credential timed $SRV.INFO against the live broker (25 rounds, quiet link):

g14's answer: 163,930 bytes, 341 endpoints. Well under the 1 MiB max payload, and never shortened.
It arrives last every round: median ~365 ms (322 to 405 ms), and 813 ms in one of 25. That is past the console's fixed 750 ms Window. novox, ace and shanks answer in 180 to 280 ms, and the controller in 47 ms.
Inside the console the same gather also runs alongside the two controller calls and every other runtime's ~400 kB coming down the same link. A runtime that is re-serving (48 memberships were re-issued at 15:18:14) is slower again.

When g14's answer missed the window, the index had no g14 modules. resolve then said the module did not exist, or ran only where it was heard. Nothing logged it, because the answer was sent; it just came too late.

Hypotheses (b) to (e): (b) not the cause: 164 kB is far below 1 MiB, and STATS carries no descriptions. (c) not observed: the runtime had restarted after the broker swap, and nats.go resubscribes Raw subscriptions. (d) the console's own runtime answer is counted, because it comes back through the broker; it is just the slowest. (e) the merge keeps every answer and accumulates machines; no drop found.

Fix

announce.Gather asks $SRV.PING and $SRV.INFO together through a new bus.Collect. It waits at least Window (750 ms), then until every instance that answered PING has sent its INFO, up to Patience (5 s). PING answers are ~120 bytes and arrive in under 200 ms. If nothing is late, it still returns at 750 ms.
The result is a Discovery: each answer with its time, size and shortening, plus the Silent instances (answered PING, no INFO in time).
An INFO still too large after first-line descriptions drops descriptions entirely. The service metadata says which (shortened), and the service's own metadata map is no longer mutated.
bus.Gather is removed (it has no callers now).

Discovery's failures are visible

The index carries Unheard: runtimes that answered PING but not INFO, and runtimes heard within the last 15 minutes but not this round.
While a runtime that could serve it is unheard, resolve never answers "nothing in the mesh is called X" or "X does not run on N". It names the runtime and why, and uses the controller's records ("X is assigned to N in the mesh's records, but ...").
mesh_machine returns [] instead of null, plus an incomplete note. mesh_search and mesh_overview (not answering) name the unheard runtimes.

New tool: `mesh_runtimes`

Read-only, no arguments, always asks discovery anew. For each runtime it reports: machine, instance, took, bytes, modules, tools, seat verbs, shortened (and how), and last heard (and how long ago). It also lists not heard: silent or lost runtimes, and machines the controller knows with no runtime heard since the console started. It is announced in discovery() and covered in the MCP instructions and the README, which also replaces the stale discovery paragraph.

Tests

announce/gather_test.go, against a real bus:
- A runtime whose INFO comes 400 ms after the window is heard. With Patience = Window, i.e. the old behaviour, the test fails with "the slow runtime was not heard".
- A PING-only runtime is named in Silent, and Gather returns at Patience.
- With nobody slow, Gather returns at the window.
- The two-step shortening.
console/runtimes_test.go:
- The symptom end to end: a late runtime's module is found and called. A silent runtime's addresses get the "did not answer discovery" messages, never "nothing in the mesh" or "does not run on". mesh_runtimes reports took ≥ window, bytes, 2 modules and 2 tools, and lists the silent machine.
- A unit test for runtimes heard before but not now, and that the plain messages still stand when everyone answered.

go vet ./... is clean. go test ./... passes with and without MESH_TEST_NATS (nats-server 2.15.0, -p 1), except TestTheSDKsStateReachesTheRuntime. That test fails identically on origin/main here, because the local SDK copy (0.1.6) has no state support. It is unrelated to this change.

Not built, assigned or rolled out.

## Symptom (2026-10-05, g14) `mesh_machine {node: g14}` answered `modules: null`, `mesh_call g14/slack.slack_check` answered "nothing in the mesh is called slack", and `g14/nextcloud-client.nextcloud_check` once answered "does not run on g14; it runs on shanks" and worked a minute later. g14's runtime was serving 275 tools for 48 modules and logged nothing about it. ## Root cause, measured A read-only probe on the runtime's own credential timed `$SRV.INFO` against the live broker (25 rounds, quiet link): - g14's answer: **163,930 bytes, 341 endpoints**. Well under the 1 MiB max payload, and never shortened. - It **arrives last every round**: median ~365 ms (322 to 405 ms), and **813 ms in one of 25**. That is past the console's fixed 750 ms `Window`. novox, ace and shanks answer in 180 to 280 ms, and the controller in 47 ms. - Inside the console the same gather also runs alongside the two controller calls and every other runtime's ~400 kB coming down the same link. A runtime that is re-serving (48 memberships were re-issued at 15:18:14) is slower again. When g14's answer missed the window, the index had no g14 modules. `resolve` then said the module did not exist, or ran only where it was heard. Nothing logged it, because the answer was sent; it just came too late. Hypotheses (b) to (e): (b) not the cause: 164 kB is far below 1 MiB, and STATS carries no descriptions. (c) not observed: the runtime had restarted after the broker swap, and nats.go resubscribes `Raw` subscriptions. (d) the console's own runtime answer is counted, because it comes back through the broker; it is just the slowest. (e) the merge keeps every answer and accumulates machines; no drop found. ## Fix - `announce.Gather` asks `$SRV.PING` and `$SRV.INFO` together through a new `bus.Collect`. It waits at least `Window` (750 ms), then until every instance that answered PING has sent its INFO, up to `Patience` (5 s). PING answers are ~120 bytes and arrive in under 200 ms. If nothing is late, it still returns at 750 ms. - The result is a `Discovery`: each answer with its time, size and shortening, plus the `Silent` instances (answered PING, no INFO in time). - An INFO still too large after first-line descriptions drops descriptions entirely. The service metadata says which (`shortened`), and the service's own metadata map is no longer mutated. - `bus.Gather` is removed (it has no callers now). ## Discovery's failures are visible - The index carries `Unheard`: runtimes that answered PING but not INFO, and runtimes heard within the last 15 minutes but not this round. - While a runtime that could serve it is unheard, `resolve` never answers "nothing in the mesh is called X" or "X does not run on N". It names the runtime and why, and uses the controller's records ("X is assigned to N in the mesh's records, but ..."). - `mesh_machine` returns `[]` instead of `null`, plus an `incomplete` note. `mesh_search` and `mesh_overview` (`not answering`) name the unheard runtimes. ## New tool: `mesh_runtimes` Read-only, no arguments, always asks discovery anew. For each runtime it reports: machine, instance, `took`, `bytes`, `modules`, `tools`, `seat verbs`, `shortened` (and how), and `last heard` (and how long ago). It also lists `not heard`: silent or lost runtimes, and machines the controller knows with no runtime heard since the console started. It is announced in `discovery()` and covered in the MCP instructions and the README, which also replaces the stale discovery paragraph. ## Tests - `announce/gather_test.go`, against a real bus: - A runtime whose INFO comes 400 ms after the window is heard. With `Patience = Window`, i.e. the old behaviour, the test fails with "the slow runtime was not heard". - A PING-only runtime is named in `Silent`, and Gather returns at `Patience`. - With nobody slow, Gather returns at the window. - The two-step shortening. - `console/runtimes_test.go`: - The symptom end to end: a late runtime's module is found and called. A silent runtime's addresses get the "did not answer discovery" messages, never "nothing in the mesh" or "does not run on". `mesh_runtimes` reports took ≥ window, bytes, 2 modules and 2 tools, and lists the silent machine. - A unit test for runtimes heard before but not now, and that the plain messages still stand when everyone answered. `go vet ./...` is clean. `go test ./...` passes with and without `MESH_TEST_NATS` (nats-server 2.15.0, `-p 1`), except `TestTheSDKsStateReachesTheRuntime`. That test fails identically on origin/main here, because the local SDK copy (0.1.6) has no state support. It is unrelated to this change. Not built, assigned or rolled out.

mesh-admin added 1 commit 2026-10-05 13:36:32 +00:00

Wait for every runtime discovery hears from, and name the ones it misses 379d19d907

The console gathered $SRV.INFO answers for a fixed 750 ms. The laptop's
runtime (48 modules, 341 endpoints, 164 kB) answers last every time: its
answer crosses to the broker on another machine and back, a median of
365 ms on a quiet link and 813 ms in one of 25 rounds measured. When it
missed the window the console said its modules ran nowhere ("nothing in
the mesh is called slack") or only on another machine.

Discovery now asks $SRV.PING alongside $SRV.INFO and waits, past the
window and up to 5 s, for every instance that answered PING. A runtime
that never sends what it serves, or answered before and not now, is
named in every answer that might concern it instead of the module being
called missing. An answer still too large after first-line descriptions
drops them, and says so in its metadata.

mesh_runtimes reports, per runtime, its machine, how long its answer
took, its size, its modules and tools, whether it was shortened and when
it was last heard, and the runtimes and machines not heard.

jschoubben added 1 commit 2026-10-05 13:38:33 +00:00

Forget a runtime's old instance once it answers under a new one, so a restart is not reported as a missed answer 52a0cc4fcb

mesh-admin merged commit 93c1ad8c4a into main

2026-10-05 13:38:49 +00:00

mesh-admin deleted branch fix/console-sees-every-runtime

2026-10-05 13:38:49 +00:00

mesh-admin commented

2026-10-06 23:11:27 +00:00

Delivery novox/mesh-tools@93c1ad8c4a1c — held since 2026-10-06T23:05:28Z

Held for a person: merged without a passing check: only a person decides that it goes on — mesh-delivery.release with why lets it go on.

Delivery plan — builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step

tier 0: mesh-tools, node-tools
ace: receives node-tools
g14: receives node-tools
novox: receives node-tools
shanks: receives node-tools
node-tools provides mcp-endpoint: its consumers are sent again after it

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-tools#13 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-tools@93c1ad8c4a1c` — **held** since 2026-10-06T23:05:28Z Held for a person: merged without a passing check: only a person decides that it goes on — `mesh-delivery.release` with why lets it go on. **Delivery plan** — builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step - tier 0: mesh-tools, node-tools - ace: receives node-tools - g14: receives node-tools - novox: receives node-tools - shanks: receives node-tools - node-tools provides mcp-endpoint: its consumers are sent again after it **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-tools#13 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-tools#13