Issue 246: the console says a module runs nowhere when a runtime answers late

2026-10-05 15:40:14 +02:00
parent 8bd56a2820
commit 9306192f93
1 changed files with 65 additions and 0 deletions
@@ -0,0 +1,65 @@
+---
+status: resolved
+opened: 2026-10-05
+located-in: [mesh-tools]
+fixed-by: mesh-tools pull request 13 — discovery waits for every runtime that answered PING, names the ones it missed, and mesh_runtimes says who answered
+amended-design:
+---
+
+# 246 — The console says a module runs nowhere when a runtime answers late
+
+## What was observed
+
+2026-10-05. On the laptop, the console answered `mesh_machine` for the laptop with no modules and no
+seats, and a call to one of the laptop's modules with "nothing in the mesh is called slack". The
+laptop's own runtime said at the same moment that it served 275 tools for 48 modules. A minute
+earlier, a call to another laptop module had been answered with "it does not run on the laptop; it
+runs on the workstation", and a retry of the same call worked. Nothing was logged anywhere.
+
+An agent worked around it by calling the runtime's local MCP port directly, which gives the right
+answer and bypasses everything the console stands for: one way in, one account, one record of what
+was called. The operator asked for a tool instead.
+
+## What was measured
+
+A read-only probe on the laptop's own runtime credential timed the discovery answers over 25 rounds
+against the live bus, whose round trip from the laptop was about 40 ms:
+
+- The laptop's runtime answer was the largest on the mesh, at about 164 kB with 341 endpoints. That
+  is far below the bus's message limit, and it was never shortened.
+- It arrived last in every round: a median of about 365 ms, and once 813 ms. The other runtimes
+  answered within 180 to 275 ms, and the controller within 50 ms.
+- The console gathered discovery answers for a fixed 750 ms. Inside the console, the gather runs
+  beside two controller calls and about 570 kB of answers on the same link, and it is slower still
+  while a runtime re-serves after a restart.
+
+## Root cause
+
+Discovery decided who was there by who answered in a fixed window. A late answer was not a failure
+to anyone, so nobody said it. The index simply lacked that runtime, and every answer built on the
+index then stated as fact that the runtime's modules did not exist, or ran only elsewhere.
+
+Ruled out by measurement or by reading the code: an answer too large for the bus, subscriptions lost
+when the bus reconnects, the console not counting its own machine's answer, and the merge of two
+answers dropping a machine.
+
+## Resolution
+
+- Discovery asks who is there (PING, a hundred bytes, answered at once) beside what each serves
+  (INFO). It waits at least the old window, and then up to five seconds for every instance that said
+  it is there, so a large answer is waited for and a quiet mesh costs nothing extra.
+- A runtime that said it is there and did not say what it serves in time, or that answered recently
+  and not now, is named. While one is unheard, the console never says an address is missing or runs
+  elsewhere: it says which runtime was not heard, and where the controller's records place the module.
+- A new console tool, `mesh_runtimes`, says for every runtime how long its answer took, how large it
+  was, how many modules and tools it announced, whether it was shortened, and when it was last heard,
+  and which runtimes or machines were not heard.
+- An announcement still too large after its descriptions are cut to their first line now leaves the
+  descriptions out, and says so.
+
+## How it is checked
+
+The fix ships with tests against a real bus: a runtime that answers after the old window is found and
+called (the same test fails with the fixed window), a runtime that answers PING and never INFO is
+named, and a restarted runtime, which answers under a new instance, is not reported as missed. Live,
+`mesh_runtimes` shows every machine's answer and its time.