Issue 331: record why no declared tool check was ever asked on a live machine

2026-10-08 16:45:21 +02:00
parent 1782f1ad74
commit 08e065fbb6
2 changed files with 101 additions and 0 deletions
@@ -0,0 +1,63 @@
+---
+status: located
+opened: 2026-10-08
+located-in: [mesh-host internal/link (hearing_nats.go, the NATS link; queue.go, AskTool)]
+fixed-by:
+amended-design:
+---
+
+# 331. A declared tool check is never asked on the live link
+
+## Symptom
+
+On 2026-10-08 the controller held the condition `module.sensors.<machine>.unhealthy` for the laptop from
+13:26 UTC on. Its evidence read:
+
+> sensors.watch (process sensors-watch.service): its tool sensors_health: asking it: no link to the bus
+> is open, 296 look(s) in a row, 0 restart(s) counted, since 2026-10-08 13:25:26 UTC
+
+At the same time the node-engine on the laptop was linked to the bus: it reported, said its health and
+received its declarations. Calling the same module's `sensors_summary` tool through the mesh MCP server
+answered at once. So the tool was served, the machine was on the bus, and only the node-engine's own
+question was refused before it was asked.
+
+## Why it is an issue
+
+[ADR 0240](../../02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) lets a module whose function
+no endpoint shows declare a **tool check**: the node-engine asks one of the module's own tools on its own
+machine and reads `healthy` from the answer ([to-be 48](../../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md)
+§3). In practice no tool check was ever asked on a live machine. Every one fails with the words that mean
+"this machine has no link", so the condition blames the network rather than the code, and a module that
+declares a tool check is reported unhealthy for ever. The suite passed because its links were fakes that
+could ask.
+
+The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
+
+## Impact
+
+- Every module with a tool check is reported not working on every machine it runs on. On 2026-10-08 that
+  was one module (sensors, on the laptop); the other open module conditions were unit failures.
+- The words point at the bus, so whoever reads the condition looks at the network first.
+- The design's promise that the mesh can judge a module no endpoint shows is not kept.
+
+## How it is checked
+
+- **In the node-engine's tests:** a queue attached to the NATS link the engine dials must reach that
+  link's `Ask`; it fails on the commit before the fix with "the NATS link was not asked". A second test,
+  run against a real bus (`MESH_TEST_NATS`), asks a served tool through the queue and reads it healthy.
+- **At build time:** the NATS link is asserted to satisfy every optional interface the queue asks of it
+  (health, tool, key reading, being asked to report), so a link that drops one does not build.
+- **Live:** after the fix reaches the laptop, `mesh-controller.conditions` no longer holds the sensors
+  condition with "no link to the bus is open".
+
+## Core or not
+
+**Core.** It is located in the node-engine, so it resolves with a replay in mesh-lab's replays register,
+proved to fail on the commit before the fix and to pass on it (`replay:`), or with `replay-none:` saying
+why none is possible.
+
+## Suggested fix
+
+Give the NATS link an `Ask` (pull request 57 on `novox/mesh-host`), assert its interfaces at build time, and
+say "the link open now cannot ask a tool" apart from "no link open", so the same words cannot hide the
+same fault again.
@@ -0,0 +1,38 @@
+# 331 — Diagnosis
+
+## 2026-10-08
+
+**The records first.** `records_search` for "no link to the bus is open" found nothing.
+
+**Where the words come from.** In the node-engine they are said in two places: the network judgement
+(ADR 0241), which says them when the queue holds no link, and `Queue.AskTool` (internal/link/queue.go),
+which runs a tool check. The network judgement on the laptop passed at the same time, so the queue held
+a link: the words came from `AskTool`.
+
+**Why `AskTool` said them with a link open.** It takes the link the queue holds and asserts it to
+`ToolBus`, the interface with `Ask`. It said "no link to the bus is open" both when the link was nil and
+when the assertion failed. Only `OverNATS` had an `Ask`. The link the node-engine attaches when it dials
+the bus is a different type, the NATS link (`natsLink`, internal/link/hearing_nats.go), which forwards
+`Report`, `Alive` and `Health` to `OverNATS` and has its own `ReadKey` and `Ping`. It had no `Ask`. So the
+assertion failed on every live link, on every machine, and every tool check read as "no link".
+
+**Why the tests passed.** The tests of the tool check attach fake links that implement `Ask`. No test
+attached the NATS link and asked a tool through the queue.
+
+**Ruled out.**
+
+- *The bus grant.* The controller composes a machine's grant for each declared check as
+  `mesh.mod.<module>.tool.<tool>.<machine>` (internal/broker, CheckSubjects), and the NATS link's
+  inbox is granted for its other questions (the witness's `Ping`). The question never reached the bus,
+  so the grant was never tried.
+- *The tool runner.* The same module's tools answered through the mesh MCP server.
+- *A link opened late and never retried.* The queue re-reads the link on every look, and the condition
+  counted 296 failed looks in a row over five hours with the link up.
+
+**Other modules.** `mesh-controller.conditions` at 14:36 UTC held three module conditions: sensors on the
+laptop (this one) and openrazer on the laptop and the desktop, which are unit failures, not tool checks.
+Any module that declares a tool check is affected in the same way.
+
+**Fix.** Pull request 57 on `novox/mesh-host`: `natsLink.Ask`, through `OverNATS.Ask`, naming "nothing on
+this machine answers" for no responders and a refused grant the way `Ping` does; build-time assertions of
+the NATS link's interfaces; and distinct words for a link that cannot ask.