From 08e065fbb6a3c2762f9ee81c569d720a87a1618e Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 16:45:21 +0200 Subject: [PATCH] Issue 331: record why no declared tool check was ever asked on a live machine --- .../00-report.md | 63 +++++++++++++++++++ .../01-diagnosis.md | 38 +++++++++++ 2 files changed, 101 insertions(+) create mode 100644 04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/00-report.md create mode 100644 04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/01-diagnosis.md diff --git a/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/00-report.md b/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/00-report.md new file mode 100644 index 00000000..bf35af53 --- /dev/null +++ b/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/00-report.md @@ -0,0 +1,63 @@ +--- +status: located +opened: 2026-10-08 +located-in: [mesh-host internal/link (hearing_nats.go, the NATS link; queue.go, AskTool)] +fixed-by: +amended-design: +--- + +# 331. A declared tool check is never asked on the live link + +## Symptom + +On 2026-10-08 the controller held the condition `module.sensors..unhealthy` for the laptop from +13:26 UTC on. Its evidence read: + +> sensors.watch (process sensors-watch.service): its tool sensors_health: asking it: no link to the bus +> is open, 296 look(s) in a row, 0 restart(s) counted, since 2026-10-08 13:25:26 UTC + +At the same time the node-engine on the laptop was linked to the bus: it reported, said its health and +received its declarations. Calling the same module's `sensors_summary` tool through the mesh MCP server +answered at once. So the tool was served, the machine was on the bus, and only the node-engine's own +question was refused before it was asked. + +## Why it is an issue + +[ADR 0240](../../02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) lets a module whose function +no endpoint shows declare a **tool check**: the node-engine asks one of the module's own tools on its own +machine and reads `healthy` from the answer ([to-be 48](../../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md) +§3). In practice no tool check was ever asked on a live machine. Every one fails with the words that mean +"this machine has no link", so the condition blames the network rather than the code, and a module that +declares a tool check is reported unhealthy for ever. The suite passed because its links were fakes that +could ask. + +The diagnosis is in [01-diagnosis.md](01-diagnosis.md). + +## Impact + +- Every module with a tool check is reported not working on every machine it runs on. On 2026-10-08 that + was one module (sensors, on the laptop); the other open module conditions were unit failures. +- The words point at the bus, so whoever reads the condition looks at the network first. +- The design's promise that the mesh can judge a module no endpoint shows is not kept. + +## How it is checked + +- **In the node-engine's tests:** a queue attached to the NATS link the engine dials must reach that + link's `Ask`; it fails on the commit before the fix with "the NATS link was not asked". A second test, + run against a real bus (`MESH_TEST_NATS`), asks a served tool through the queue and reads it healthy. +- **At build time:** the NATS link is asserted to satisfy every optional interface the queue asks of it + (health, tool, key reading, being asked to report), so a link that drops one does not build. +- **Live:** after the fix reaches the laptop, `mesh-controller.conditions` no longer holds the sensors + condition with "no link to the bus is open". + +## Core or not + +**Core.** It is located in the node-engine, so it resolves with a replay in mesh-lab's replays register, +proved to fail on the commit before the fix and to pass on it (`replay:`), or with `replay-none:` saying +why none is possible. + +## Suggested fix + +Give the NATS link an `Ask` (pull request 57 on `novox/mesh-host`), assert its interfaces at build time, and +say "the link open now cannot ask a tool" apart from "no link open", so the same words cannot hide the +same fault again. diff --git a/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/01-diagnosis.md b/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/01-diagnosis.md new file mode 100644 index 00000000..ea20e21f --- /dev/null +++ b/04-ISSUES/331-a-declared-tool-check-is-never-asked-on-the-live-link/01-diagnosis.md @@ -0,0 +1,38 @@ +# 331 — Diagnosis + +## 2026-10-08 + +**The records first.** `records_search` for "no link to the bus is open" found nothing. + +**Where the words come from.** In the node-engine they are said in two places: the network judgement +(ADR 0241), which says them when the queue holds no link, and `Queue.AskTool` (internal/link/queue.go), +which runs a tool check. The network judgement on the laptop passed at the same time, so the queue held +a link: the words came from `AskTool`. + +**Why `AskTool` said them with a link open.** It takes the link the queue holds and asserts it to +`ToolBus`, the interface with `Ask`. It said "no link to the bus is open" both when the link was nil and +when the assertion failed. Only `OverNATS` had an `Ask`. The link the node-engine attaches when it dials +the bus is a different type, the NATS link (`natsLink`, internal/link/hearing_nats.go), which forwards +`Report`, `Alive` and `Health` to `OverNATS` and has its own `ReadKey` and `Ping`. It had no `Ask`. So the +assertion failed on every live link, on every machine, and every tool check read as "no link". + +**Why the tests passed.** The tests of the tool check attach fake links that implement `Ask`. No test +attached the NATS link and asked a tool through the queue. + +**Ruled out.** + +- *The bus grant.* The controller composes a machine's grant for each declared check as + `mesh.mod..tool..` (internal/broker, CheckSubjects), and the NATS link's + inbox is granted for its other questions (the witness's `Ping`). The question never reached the bus, + so the grant was never tried. +- *The tool runner.* The same module's tools answered through the mesh MCP server. +- *A link opened late and never retried.* The queue re-reads the link on every look, and the condition + counted 296 failed looks in a row over five hours with the link up. + +**Other modules.** `mesh-controller.conditions` at 14:36 UTC held three module conditions: sensors on the +laptop (this one) and openrazer on the laptop and the desktop, which are unit failures, not tool checks. +Any module that declares a tool check is affected in the same way. + +**Fix.** Pull request 57 on `novox/mesh-host`: `natsLink.Ask`, through `OverNATS.Ask`, naming "nothing on +this machine answers" for no responders and a refused grant the way `Ping` does; build-time assertions of +the NATS link's interfaces; and distinct words for a link that cannot ask.