Merge pull request 'Issue 331: a declared tool check is never asked on the live link' (#218) from issue/331-a-tool-check-is-never-asked into main

2026-10-08 15:06:55 +00:00
parent 31664fcd80 33b0ee3d53
commit 72e0a851cf
2 changed files with 115 additions and 0 deletions
@@ -0,0 +1,71 @@
+---
+status: located
+opened: 2026-10-08
+located-in: [mesh-host internal/link (hearing_nats.go, the NATS link; queue.go, AskTool)]
+fixed-by:
+amended-design:
+---
+
+# 331. A declared tool check is never asked on the live link
+
+## Symptom
+
+On 2026-10-08 the controller held the condition `module.sensors.<machine>.unhealthy` for the laptop from
+13:26 UTC on. Its evidence read:
+
+> sensors.watch (process sensors-watch.service): its tool sensors_health: asking it: no link to the bus
+> is open, 296 look(s) in a row, 0 restart(s) counted, since 2026-10-08 13:25:26 UTC
+
+At the same time the node-engine on the laptop was linked to the bus: it reported, said its health and
+received its declarations. Calling the same module's `sensors_summary` tool through the mesh MCP server
+answered at once. So the tool was served, the machine was on the bus, and only the node-engine's own
+question was refused before it was asked.
+
+## Why it is an issue
+
+[ADR 0240](../../02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) lets a module whose function
+no endpoint shows declare a **tool check**: the node-engine asks one of the module's own tools on its own
+machine and reads `healthy` from the answer ([to-be 48](../../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md)
+§3). In practice no tool check was ever asked on a live machine. Every one fails with the words that mean
+"this machine has no link", so the condition blames the network rather than the code, and a module that
+declares a tool check is reported unhealthy for ever. The suite passed because its links were fakes that
+could ask.
+
+The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
+
+## Impact
+
+- Every module with a tool check is reported not working on every machine it runs on. On 2026-10-08 that
+  was one module (sensors, on the laptop); the other open module conditions were unit failures.
+- The words point at the bus, so whoever reads the condition looks at the network first.
+- The design's promise that the mesh can judge a module no endpoint shows is not kept.
+
+## How it is checked
+
+- **In the node-engine's tests, with no bus:** a queue attached to the NATS link the engine dials must
+  reach that link's `Ask`, and a link that cannot ask is said apart from no link. On the commit before
+  the fix it fails at its second step, with "on a link that cannot ask: no link to the bus is open":
+  the queue said the same words for both.
+- **In the node-engine's tests, on a real bus** (`MESH_TEST_NATS`): a served tool is asked through the
+  queue and read healthy, and a tool nothing serves is said as such. **The pull request's repository
+  check skips this test today**: the build seat's Go toolchain holds no nats-server, so nothing sets
+  `MESH_TEST_NATS` there. The node-engine's `merge-check.sh` now starts a throwaway nats-server and runs
+  these tests whenever the toolchain holds one, and prints "NOT RUN AGAINST A BUS" when it does not. It
+  passed so locally. **Open:** the Go toolchain image does not carry a nats-server yet, so in the build
+  seat the real-bus tests remain skipped and say so.
+- **At build time:** the NATS link is asserted to satisfy every optional interface the queue asks of it
+  (health, tool, key reading, being asked to report), so a link that drops one does not build.
+- **Live:** after the fix reaches the laptop, `mesh-controller.conditions` no longer holds the sensors
+  condition with "no link to the bus is open".
+
+## Core or not
+
+**Core.** It is located in the node-engine, so it resolves with a replay in mesh-lab's replays register,
+proved to fail on the commit before the fix and to pass on it (`replay:`), or with `replay-none:` saying
+why none is possible.
+
+## Suggested fix
+
+Give the NATS link an `Ask` (pull request 57 on `novox/mesh-host`), assert its interfaces at build time, and
+say "the link open now cannot ask a tool" apart from "no link open", so the same words cannot hide the
+same fault again.
@@ -0,0 +1,44 @@
+# 331 — Diagnosis
+
+## 2026-10-08
+
+**The records first.** `records_search` for "no link to the bus is open" found nothing.
+
+**Where the words come from.** In the node-engine they are said in two places: the network judgement
+(ADR 0241), which says them when the queue holds no link, and `Queue.AskTool` (internal/link/queue.go),
+which runs a tool check. The network judgement on the laptop passed at the same time, so the queue held
+a link: the words came from `AskTool`.
+
+**Why `AskTool` said them with a link open.** It takes the link the queue holds and asserts it to
+`ToolBus`, the interface with `Ask`. It said "no link to the bus is open" both when the link was nil and
+when the assertion failed. Only `OverNATS` had an `Ask`. The link the node-engine attaches when it dials
+the bus is a different type, the NATS link (`natsLink`, internal/link/hearing_nats.go), which forwards
+`Report`, `Alive` and `Health` to `OverNATS` and has its own `ReadKey` and `Ping`. It had no `Ask`. So the
+assertion failed on every live link, on every machine, and every tool check read as "no link".
+
+**Why the tests passed.** The tests of the tool check attach fake links that implement `Ask`. No test
+attached the NATS link and asked a tool through the queue.
+
+**Ruled out.**
+
+- *The bus grant.* The controller composes a machine's grant for each declared check as
+  `mesh.mod.<module>.tool.<tool>.<machine>` (internal/broker, CheckSubjects). **Verified live** on
+  2026-10-08 with `nats.nats_users`: the laptop's node-engine user may publish
+  `mesh.mod.sensors.tool.sensors_health.<machine>` and subscribe to its own inbox, and the laptop's tool
+  runner subscribes to `mesh.mod.sensors.tool.>` and may respond. Calling `sensors_health` on the laptop
+  through the mesh MCP server answered healthy. The question never reached the bus, so the grant was
+  never tried.
+- *The tool runner.* The same module's tools answered through the mesh MCP server.
+- *A link opened late and never retried.* The queue re-reads the link on every look, and the condition
+  counted 296 failed looks in a row over five hours with the link up.
+
+**Other modules.** `mesh-controller.conditions` at 14:36 UTC held three module conditions: sensors on the
+laptop (this one) and openrazer on the laptop and the desktop, which are unit failures, not tool checks.
+Any module that declares a tool check is affected in the same way.
+
+**Fix.** Pull request 57 on `novox/mesh-host`: `natsLink.Ask`, through `OverNATS.Ask`, naming "nothing on
+this machine answers" for no responders and a refused grant the way `Ping` does; build-time assertions of
+the NATS link's interfaces; and distinct words for a link that cannot ask. The same pull request says a
+check's error once (the evidence read "asking it: asking …"), says a deadline as "no answer within" the
+time the check gave it, and names a refusal only when the bus refused this question: the client keeps its
+last error for the life of the connection, so a refusal under a grant since widened was said again.