diff --git a/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/00-report.md b/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/00-report.md new file mode 100644 index 00000000..86b8aca6 --- /dev/null +++ b/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/00-report.md @@ -0,0 +1,71 @@ +--- +status: located +opened: 2026-10-08 +located-in: [mesh-host internal/link (hearing_nats.go, the NATS link; queue.go, AskTool)] +fixed-by: +amended-design: +--- + +# 331. A declared tool check is never asked on the live link + +## Symptom + +On 2026-10-08 the controller held the condition `module.sensors..unhealthy` for the laptop from +13:26 UTC on. Its evidence read: + +> sensors.watch (process sensors-watch.service): its tool sensors_health: asking it: no link to the bus +> is open, 296 look(s) in a row, 0 restart(s) counted, since 2026-10-08 13:25:26 UTC + +At the same time the node-engine on the laptop was linked to the bus: it reported, said its health and +received its declarations. Calling the same module's `sensors_summary` tool through the mesh MCP server +answered at once. So the tool was served, the machine was on the bus, and only the node-engine's own +question was refused before it was asked. + +## Why it is an issue + +[ADR 0240](../../02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) lets a module whose function +no endpoint shows declare a **tool check**: the node-engine asks one of the module's own tools on its own +machine and reads `healthy` from the answer ([to-be 48](../../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md) +§3). In practice no tool check was ever asked on a live machine. Every one fails with the words that mean +"this machine has no link", so the condition blames the network rather than the code, and a module that +declares a tool check is reported unhealthy for ever. The suite passed because its links were fakes that +could ask. + +The diagnosis is in [01-diagnosis.md](01-diagnosis.md). + +## Impact + +- Every module with a tool check is reported not working on every machine it runs on. On 2026-10-08 that + was one module (sensors, on the laptop); the other open module conditions were unit failures. +- The words point at the bus, so whoever reads the condition looks at the network first. +- The design's promise that the mesh can judge a module no endpoint shows is not kept. + +## How it is checked + +- **In the node-engine's tests, with no bus:** a queue attached to the NATS link the engine dials must + reach that link's `Ask`, and a link that cannot ask is said apart from no link. On the commit before + the fix it fails at its second step, with "on a link that cannot ask: no link to the bus is open": + the queue said the same words for both. +- **In the node-engine's tests, on a real bus** (`MESH_TEST_NATS`): a served tool is asked through the + queue and read healthy, and a tool nothing serves is said as such. **The pull request's repository + check skips this test today**: the build seat's Go toolchain holds no nats-server, so nothing sets + `MESH_TEST_NATS` there. The node-engine's `merge-check.sh` now starts a throwaway nats-server and runs + these tests whenever the toolchain holds one, and prints "NOT RUN AGAINST A BUS" when it does not. It + passed so locally. **Open:** the Go toolchain image does not carry a nats-server yet, so in the build + seat the real-bus tests remain skipped and say so. +- **At build time:** the NATS link is asserted to satisfy every optional interface the queue asks of it + (health, tool, key reading, being asked to report), so a link that drops one does not build. +- **Live:** after the fix reaches the laptop, `mesh-controller.conditions` no longer holds the sensors + condition with "no link to the bus is open". + +## Core or not + +**Core.** It is located in the node-engine, so it resolves with a replay in mesh-lab's replays register, +proved to fail on the commit before the fix and to pass on it (`replay:`), or with `replay-none:` saying +why none is possible. + +## Suggested fix + +Give the NATS link an `Ask` (pull request 57 on `novox/mesh-host`), assert its interfaces at build time, and +say "the link open now cannot ask a tool" apart from "no link open", so the same words cannot hide the +same fault again. diff --git a/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/01-diagnosis.md b/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/01-diagnosis.md new file mode 100644 index 00000000..9980322e --- /dev/null +++ b/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/01-diagnosis.md @@ -0,0 +1,44 @@ +# 331 — Diagnosis + +## 2026-10-08 + +**The records first.** `records_search` for "no link to the bus is open" found nothing. + +**Where the words come from.** In the node-engine they are said in two places: the network judgement +(ADR 0241), which says them when the queue holds no link, and `Queue.AskTool` (internal/link/queue.go), +which runs a tool check. The network judgement on the laptop passed at the same time, so the queue held +a link: the words came from `AskTool`. + +**Why `AskTool` said them with a link open.** It takes the link the queue holds and asserts it to +`ToolBus`, the interface with `Ask`. It said "no link to the bus is open" both when the link was nil and +when the assertion failed. Only `OverNATS` had an `Ask`. The link the node-engine attaches when it dials +the bus is a different type, the NATS link (`natsLink`, internal/link/hearing_nats.go), which forwards +`Report`, `Alive` and `Health` to `OverNATS` and has its own `ReadKey` and `Ping`. It had no `Ask`. So the +assertion failed on every live link, on every machine, and every tool check read as "no link". + +**Why the tests passed.** The tests of the tool check attach fake links that implement `Ask`. No test +attached the NATS link and asked a tool through the queue. + +**Ruled out.** + +- *The bus grant.* The controller composes a machine's grant for each declared check as + `mesh.mod..tool..` (internal/broker, CheckSubjects). **Verified live** on + 2026-10-08 with `nats.nats_users`: the laptop's node-engine user may publish + `mesh.mod.sensors.tool.sensors_health.` and subscribe to its own inbox, and the laptop's tool + runner subscribes to `mesh.mod.sensors.tool.>` and may respond. Calling `sensors_health` on the laptop + through the mesh MCP server answered healthy. The question never reached the bus, so the grant was + never tried. +- *The tool runner.* The same module's tools answered through the mesh MCP server. +- *A link opened late and never retried.* The queue re-reads the link on every look, and the condition + counted 296 failed looks in a row over five hours with the link up. + +**Other modules.** `mesh-controller.conditions` at 14:36 UTC held three module conditions: sensors on the +laptop (this one) and openrazer on the laptop and the desktop, which are unit failures, not tool checks. +Any module that declares a tool check is affected in the same way. + +**Fix.** Pull request 57 on `novox/mesh-host`: `natsLink.Ask`, through `OverNATS.Ask`, naming "nothing on +this machine answers" for no responders and a refused grant the way `Ping` does; build-time assertions of +the NATS link's interfaces; and distinct words for a link that cannot ask. The same pull request says a +check's error once (the evidence read "asking it: asking …"), says a deadline as "no answer within" the +time the check gave it, and names a refusal only when the bus refused this question: the client keeps its +last error for the life of the connection, so a refusal under a grant since widened was said again.