sensors: raise only what the hardware says is failing, and say a hung watch

The review of #129 found false alarms and a silent failure: the NVMe temperature bit, old-age
attributes, a fan's minimum or latched alarm, an unrelated hot sensor and a board chip's limits could
raise a condition, while a hung smartctl stopped the fans' looks and an old readings file passed as
healthy. Read spinning drives hourly (smartd too) so a read never keeps one from spinning down, and skip
a drive that cannot say whether it sleeps.

modules/sensors/README.md

@@ -28,13 +28,29 @@ seat's verbs. The full reasoning is in ADR 0256.
 |---|---|---|
 | `smartmontools` | the package: `smartctl` and `smartd` | |
 | `/etc/smartd.conf` | every drive (`DEVICESCAN`), every check, never waking a drive in standby, no mail | written whole |
-| `smartd.service` drop-in | runs on a virtual machine too, and stays up with no drive to watch (`-q never`) | written whole |
+| `smartd.service` drop-in | runs on a virtual machine too, stays up with no drive to watch (`-q never`), and checks each drive hourly (`-i 3600`) | written whole |
 | `smartd` | watches every drive between looks, and logs what it finds to the journal | root, the package's unit |
-| `watch` (`sensors watch`) | every 10 min each drive's health with `smartctl --json -a -n standby`; every 30 s which fans turn; one file, `/run/mesh-sensors/readings.json` (0644) | root, the node-engine's process |
-| the tools | read `/sys`, `/proc` and the watch's file | the operator account, in the tool runner |
+| `watch` (`sensors watch`) | each drive's health with `/usr/bin/smartctl --json -a -n standby,2,3`: a spinning drive hourly, any other every 10 min; every 30 s which fans turn; one file, `/run/mesh-sensors/readings.json` (0644, in a 0755 directory) | root, the node-engine's process |
+| the tools | read `/sys`, `/proc` and the watch's file, once per call | the operator account, in the tool runner |
 
-The tools need no root, because the watch has already read what only root may read. A drive in standby
-is never woken. The watch keeps its last reading of that drive and says that it was asleep.
+The tools need no root, because the watch has already read what only root may read.
+
+**The drives never hold up the fans.** The watch reads the drives beside its looks at the fans, one pass
+at a time. Each smartctl is bounded by 20 s, plus 2 s for a child that keeps its output open. The file is
+written every 30 s whatever a drive does, so a file older than five minutes means the watch is hung or
+gone, and the health check says so.
+
+**A drive is never woken, and is read seldom enough to sleep.**
+- smartctl's `-n standby,2,3` stops with status 2 and the drive's power mode when the drive is in standby
+  or asleep. The watch then keeps the reading before and says the drive was asleep. A drive is asleep only
+  on status 2 with a parsed power mode (or smartctl's own "STANDBY (OS)").
+- It stops with status 3 when the drive cannot say (a USB bridge that does not pass CHECK POWER MODE on).
+  Such a drive is not read, so that reading it never wakes it, and it is said as not read.
+- Reading SMART from an awake drive can restart its standby timer, so a drive read more often than its
+  timer is set for never spins down. No mesh tool reads a drive's standby timer, so whether the servers'
+  drives spin down is not known. A spinning drive is therefore read hourly, by the watch and by smartd.
+  A drive whose timer is set longer than an hour may still be kept awake.
+- smartd has no per-drive equivalent of status 3. A drive behind such a bridge may be woken by smartd.
 
 ## Tools
 
@@ -55,20 +71,26 @@ All six only read.
 |---|---|
 | temperature HIGH | at or above its own maximum, or a zone's `hot` trip (without one, its lowest `passive` trip) |
 | temperature CRITICAL | at or above its own critical or emergency value, or a zone's `critical` trip |
-| drive FAILING | SMART's overall verdict failed, an attribute is failing now, or an NVMe critical warning bit is set |
-| drive HIGH | reallocated, pending or unreadable sectors; an attribute that failed in the past; NVMe media errors or 90 % of rated endurance used; errors in the error log; a last self-test that did not pass |
-| fan STOPPED | its chip's alarm, or under its own minimum, once it has been seen turning; or it ran since boot and reads 0 while a temperature is at or above its own maximum |
+| drive FAILING | SMART's overall verdict failed (SATA), a pre-failure attribute is failing now, or an NVMe critical warning bit other than temperature is set |
+| drive HIGH | an old-age attribute failing now; an attribute that failed in the past; the NVMe temperature bit alone; reallocated, pending or unreadable sectors; NVMe media errors or 90 % of rated endurance used; errors in the error log; a last self-test that did not pass |
+| fan STOPPED | it ran since boot, reads 0 now, and the CPU, the board or a sensor on its own chip is at or above its own maximum |
+| fan note | under its chip's minimum, or its chip's alarm bit set: said, never flagged |
 | memory HIGH / CRITICAL | corrected / uncorrectable errors since boot |
 | PCIe HIGH | fatal or non-fatal errors since boot |
 
-**What is no threshold.** A threshold given as the format's largest value is no threshold. The laptop's
-NVMe drive gives 65261.85 °C as the maximum of two of its sensors. A threshold of 0 is no threshold
-either, and neither is anything outside −40 to 200 °C. The reading is then said without a flag. A CPU
-sensor that gives no threshold, such as AMD's `Tctl`, is never flagged.
+**What is no threshold, and no reading.**
+- A threshold given as the format's largest value is no threshold. The laptop's NVMe drive gives
+  65261.85 °C as the maximum of two of its sensors.
+- A threshold of 0 is no threshold either, and neither is anything outside −40 to 200 °C. The reading
+  is then said without a flag.
+- A CPU sensor that gives no threshold, such as AMD's `Tctl`, is never flagged.
+- 0 °C is a reading. 127 and 127.5 °C are not: they are what a board chip's input with nothing on it
+  answers (a phantom AUXTIN). Such a sensor is said as not read.
 
-**Why a fan at 0 is not enough.** Laptop fans and GPU fans in zero-fan mode stop on purpose when the
-machine is cool. A board header with nothing on it reads 0, and may carry an alarm or a minimum that the
-firmware set. So a fan is judged only once it has been seen turning.
+**Why a fan at 0, a minimum or an alarm is not enough.** Laptop fans and GPU fans in zero-fan mode stop on
+purpose when the machine is cool, under the minimum their chip gives. A board chip latches an alarm bit,
+and a header with nothing on it reads 0. Only a fan that ran and stopped while what it cools is hot is a
+fault. A hot drive or network adapter says nothing of a fan.
 
 ## Health
 
@@ -80,15 +102,18 @@ The module declares its health under ADR 0240:
 
 `sensors_health` is unhealthy when:
 
-- a temperature is at or above its own critical value;
-- a drive says it is failing;
-- a fan that ran has stopped while the machine is hot;
-- memory reports an uncorrectable error.
+- a CPU, GPU, drive or zone temperature is at or above its own critical value. A board chip's (Super I/O)
+  critical value is shown CRITICAL by the tools and never makes the module unhealthy: its limits are
+  whatever the firmware left;
+- a drive says it is failing (the FAILING row above);
+- a fan is STOPPED (the row above);
+- memory reports an uncorrectable error;
+- the watch has written no readings since the machine started, or none for five minutes.
 
 Its `why` is one plain sentence per finding, for the operator. For example: "The drive sdb (WDC
-WUH722222ALE6L4) says it is failing: its overall SMART health check failed." HIGH readings raise nothing.
-A drive whose health was not read is said in `why` and does not make the check unhealthy. The watch not
-running is caught by the watch's own liveness.
+WUH722222ALE6L4) says it is failing: its overall SMART health check failed." HIGH readings and fan notes
+raise nothing. A drive whose health was not read (asleep since boot, or behind a bridge) is said in `why`
+and does not make the check unhealthy.
 
 The condition this raises is `module.sensors.<machine>.unhealthy`. ADR 0256 records what this costs:
 
@@ -100,7 +125,7 @@ The machine judging its own hardware, as it judges its network, is the follow-up
 ## Tests
 
 `go test ./...`. The tests read a tree standing in for `/sys`, `/proc` and `/run`, and smartctl answers
-recorded as smartmontools 7 writes them. There are three machines:
+as smartmontools 7.5 writes them (its standby answer as its source sets it). There are three machines:
 
 - a laptop, as the laptop showed itself on 2026-10-08;
 - a server with ECC memory, three SATA drives read by drivetemp and a board chip with an empty fan header;
@@ -110,14 +135,20 @@ They cover:
 
 - thresholds taken from the hardware only, and a zone's hwmon copy left out;
 - HIGH at the maximum and CRITICAL at the critical value; a zone's passive trip as its high;
-- a sensor that gives no reading said as not read;
+- 0 °C as a reading, a phantom 127 °C as not read, and a board chip's critical value raising nothing;
 - every class not present on the virtual machine, which is healthy;
-- a failing SATA drive, an NVMe critical warning, and early signs that are HIGH and raise nothing;
-- a missing, stale or earlier-boot reading said as such;
-- a fan at rest, a fan that stopped while hot, an empty header, and a fan under its minimum;
+- a failing SATA drive; an NVMe critical warning; the NVMe temperature bit alone as HIGH; NVMe without
+  `smart_status`; an old-age attribute failing now and a past failure as HIGH;
+- a missing, stale or earlier-boot reading, and the watch's staleness making the module unhealthy;
+- a fan at rest, a fan that stopped while the board is hot, a hot drive saying nothing of a fan, a hot
+  sensor on the fan's own chip, an empty header, a zero-fan GPU under its minimum, and a latched alarm;
 - memory and PCIe errors, and CPU throttling;
-- the watch reading as root without waking a drive, keeping a reading for a drive asleep, reading drives
-  only every 10 min, and keeping what it saw across a restart;
+- the watch reading as root without waking a drive: smartctl's real standby answer, an open failure
+  that is not sleep, and a bridge without a power mode;
+- a spinning drive read hourly and a solid-state one every 10 min;
+- a smartctl that hangs with a child holding its output, bounded, and the fans looked at and written
+  while a drive hangs;
+- a restart keeping what the last watch read; the readings directory given its mode;
 - a watch that is not root asking through `sudo -n`, and saying what stops it;
 - the health answer in the shape the node-engine reads;
 - the manifest: the tools listed are the tools served, all read-only, no seat claimed, no machine named,