diff --git a/04-ISSUES/246-the-console-says-a-module-runs-nowhere-when-a-runtime-answers-late/00-report.md b/04-ISSUES/246-the-console-says-a-module-runs-nowhere-when-a-runtime-answers-late/00-report.md new file mode 100644 index 0000000..dd3dc13 --- /dev/null +++ b/04-ISSUES/246-the-console-says-a-module-runs-nowhere-when-a-runtime-answers-late/00-report.md @@ -0,0 +1,65 @@ +--- +status: resolved +opened: 2026-10-05 +located-in: [mesh-tools] +fixed-by: mesh-tools pull request 13 — discovery waits for every runtime that answered PING, names the ones it missed, and mesh_runtimes says who answered +amended-design: +--- + +# 246 — The console says a module runs nowhere when a runtime answers late + +## What was observed + +2026-10-05. On the laptop, the console answered `mesh_machine` for the laptop with no modules and no +seats, and a call to one of the laptop's modules with "nothing in the mesh is called slack". The +laptop's own runtime said at the same moment that it served 275 tools for 48 modules. A minute +earlier, a call to another laptop module had been answered with "it does not run on the laptop; it +runs on the workstation", and a retry of the same call worked. Nothing was logged anywhere. + +An agent worked around it by calling the runtime's local MCP port directly, which gives the right +answer and bypasses everything the console stands for: one way in, one account, one record of what +was called. The operator asked for a tool instead. + +## What was measured + +A read-only probe on the laptop's own runtime credential timed the discovery answers over 25 rounds +against the live bus, whose round trip from the laptop was about 40 ms: + +- The laptop's runtime answer was the largest on the mesh, at about 164 kB with 341 endpoints. That + is far below the bus's message limit, and it was never shortened. +- It arrived last in every round: a median of about 365 ms, and once 813 ms. The other runtimes + answered within 180 to 275 ms, and the controller within 50 ms. +- The console gathered discovery answers for a fixed 750 ms. Inside the console, the gather runs + beside two controller calls and about 570 kB of answers on the same link, and it is slower still + while a runtime re-serves after a restart. + +## Root cause + +Discovery decided who was there by who answered in a fixed window. A late answer was not a failure +to anyone, so nobody said it. The index simply lacked that runtime, and every answer built on the +index then stated as fact that the runtime's modules did not exist, or ran only elsewhere. + +Ruled out by measurement or by reading the code: an answer too large for the bus, subscriptions lost +when the bus reconnects, the console not counting its own machine's answer, and the merge of two +answers dropping a machine. + +## Resolution + +- Discovery asks who is there (PING, a hundred bytes, answered at once) beside what each serves + (INFO). It waits at least the old window, and then up to five seconds for every instance that said + it is there, so a large answer is waited for and a quiet mesh costs nothing extra. +- A runtime that said it is there and did not say what it serves in time, or that answered recently + and not now, is named. While one is unheard, the console never says an address is missing or runs + elsewhere: it says which runtime was not heard, and where the controller's records place the module. +- A new console tool, `mesh_runtimes`, says for every runtime how long its answer took, how large it + was, how many modules and tools it announced, whether it was shortened, and when it was last heard, + and which runtimes or machines were not heard. +- An announcement still too large after its descriptions are cut to their first line now leaves the + descriptions out, and says so. + +## How it is checked + +The fix ships with tests against a real bus: a runtime that answers after the old window is found and +called (the same test fails with the fixed window), a runtime that answers PING and never INFO is +named, and a restarted runtime, which answers under a new instance, is not reported as missed. Live, +`mesh_runtimes` shows every machine's answer and its time.