sensors: say a stuck drive read, and be the drives' only reader

The second review of #129 found a read stuck in the kernel kept the file fresh and the module healthy,
smartd as a second reader could wake or keep awake the drives the watch spares, and a board chip's
maximum could stop a fan. Read each drive in its own goroutine from a root reader of the module's own,
mark each read in flight, and say a read past its bound or a drive not looked at for twice its cadence;
drop smartd; tell smartctl each drive's kind; take a SAS drive's standby; say a drive asleep since when.

modules/sensors/README.md

@@ -26,31 +26,40 @@ seat's verbs. The full reasoning is in ADR 0256.
 
 | resource | what | as |
 |---|---|---|
-| `smartmontools` | the package: `smartctl` and `smartd` | |
-| `/etc/smartd.conf` | every drive (`DEVICESCAN`), every check, never waking a drive in standby, no mail | written whole |
-| `smartd.service` drop-in | runs on a virtual machine too, stays up with no drive to watch (`-q never`), and checks each drive hourly (`-i 3600`) | written whole |
-| `smartd` | watches every drive between looks, and logs what it finds to the journal | root, the package's unit |
-| `watch` (`sensors watch`) | each drive's health with `/usr/bin/smartctl --json -a -n standby,2,3`: a spinning drive hourly, any other every 10 min; every 30 s which fans turn; one file, `/run/mesh-sensors/readings.json` (0644, in a 0755 directory) | root, the node-engine's process |
-| the tools | read `/sys`, `/proc` and the watch's file, once per call | the operator account, in the tool runner |
+| `smartmontools` | the package, for `smartctl` | |
+| `drives` (`sensors drives`) | each drive's health with `/usr/bin/smartctl --json -a -d <kind> -n standby,2,3`: a spinning drive hourly, any other every 10 min, each drive in a goroutine of its own; writes `/run/mesh-sensors/drives.json` every 30 s | root, the node-engine's process; health `unit` |
+| `watch` (`sensors watch`) | every 30 s which fans turn; writes `/run/mesh-sensors/fans.json` | the node-engine's process; health `tool` (`sensors_health`) |
+| the tools | read `/sys`, `/proc` and the two files, each once per call | the operator account, in the tool runner |
 
-The tools need no root, because the watch has already read what only root may read.
+Both files are 0644 in a 0755 directory. The tools need no root, because the drive reader has already read
+what only root may read.
 
-**The drives never hold up the fans.** The watch reads the drives beside its looks at the fans, one pass
-at a time. Each smartctl is bounded by 20 s, plus 2 s for a child that keeps its output open. The file is
-written every 30 s whatever a drive does, so a file older than five minutes means the watch is hung or
-gone, and the health check says so.
+**The module is the only reader of the drives; it does not run smartd.** smartd reads every drive when it
+starts or restarts, which can spin a sleeping drive up. Its `-n standby` is for ATA drives only, so it
+polls a SAS drive in standby anyway. And as a second reader on its own schedule, it would read each drive
+at another phase of the hour, shortening the idle stretch a drive needs to spin down.
 
-**A drive is never woken, and is read seldom enough to sleep.**
-- smartctl's `-n standby,2,3` stops with status 2 and the drive's power mode when the drive is in standby
-  or asleep. The watch then keeps the reading before and says the drive was asleep. A drive is asleep only
-  on status 2 with a parsed power mode (or smartctl's own "STANDBY (OS)").
+**No drive holds up another, and a stuck one is said.** Each smartctl is bounded by 20 s, plus 2 s for a
+child that keeps its output open. A smartctl stuck in the kernel cannot be ended at all. Its read stays in
+flight, marked since when, and the health check says so once it is older than any read can take (20 s +
+2 s + 30 s). Each drive is read in its own goroutine, so the other drives go on being read. The fans are
+another process, and go on whatever the drives do.
+
+**A drive is not woken, as far as it can say so, and is read seldom enough to sleep.**
+- `-n standby,2,3` makes smartctl stop with status 2 when the drive is in standby or asleep. The drive
+  reader then keeps the reading before, and since when the drive has been asleep. A drive counts as asleep
+  on status 2 with either an ATA power mode of STANDBY or SLEEP, or smartctl's own words "Device is in
+  <mode> mode, exit(2)" (a SAS drive, or one the operating system powered down). An open failure is
+  neither, whatever its words.
 - It stops with status 3 when the drive cannot say (a USB bridge that does not pass CHECK POWER MODE on).
-  Such a drive is not read, so that reading it never wakes it, and it is said as not read.
+  Such a drive is not read, and is said as not read.
+- smartctl is told the drive's kind (`-d nvme`, `-d sat` for an ATA drive or one behind USB, `-d scsi`)
+  so it does not probe. A drive of a kind the module cannot tell (virtio, MMC) is read with smartctl's own
+  detection.
 - Reading SMART from an awake drive can restart its standby timer, so a drive read more often than its
   timer is set for never spins down. No mesh tool reads a drive's standby timer, so whether the servers'
-  drives spin down is not known. A spinning drive is therefore read hourly, by the watch and by smartd.
-  A drive whose timer is set longer than an hour may still be kept awake.
-- smartd has no per-drive equivalent of status 3. A drive behind such a bridge may be woken by smartd.
+  drives spin down is not known. A spinning drive is therefore read once an hour, timed from its last
+  look. A drive whose timer is set longer than an hour may still be kept awake.
 
 ## Tools
 
@@ -60,7 +69,7 @@ All six only read.
 |---|---|---|
 | `sensors_summary` | one line per finding, flagged CRITICAL, HIGH, FAILING or STOPPED; then one line per class: temperatures with the hottest, fans, drives, memory and PCIe errors | `sensors`, `smartctl -H` |
 | `sensors_temperatures` | every thermal zone with its trip points, and every hwmon temperature with its label, its kind (cpu, gpu, drive, board, network), the drive it sits in, and its own maximum and critical values | `sensors` |
-| `sensors_fans` | every fan's RPM, its chip's minimum and alarm, and whether the watch saw it turn since boot | `sensors` |
+| `sensors_fans` | every fan's RPM, its chip's minimum and alarm, and whether the fan watch saw it turn since boot | `sensors` |
 | `sensors_drives` | each drive: model, serial, size, whether it spins, the SMART verdict; for NVMe the critical warning bits, spare, endurance used and media errors; for SATA the reallocated, pending and unreadable sectors, the error log and the last self-test | `smartctl -a`, `nvme smart-log` |
 | `sensors_faults` | EDAC memory errors per controller and module, PCIe AER errors per device, and how often an Intel CPU was slowed for heat | `edac-util` |
 | `sensors_health` | the module's health check: healthy or not, with `why` | |
@@ -73,8 +82,9 @@ All six only read.
 | temperature CRITICAL | at or above its own critical or emergency value, or a zone's `critical` trip |
 | drive FAILING | SMART's overall verdict failed (SATA), a pre-failure attribute is failing now, or an NVMe critical warning bit other than temperature is set |
 | drive HIGH | an old-age attribute failing now; an attribute that failed in the past; the NVMe temperature bit alone; reallocated, pending or unreadable sectors; NVMe media errors or 90 % of rated endurance used; errors in the error log; a last self-test that did not pass |
-| fan STOPPED | it ran since boot, reads 0 now, and the CPU, the board or a sensor on its own chip is at or above its own maximum |
+| fan STOPPED | it ran since boot, reads 0 now, and a CPU sensor, a thermal zone, or a sensor on its own chip (not a board chip) is at or above its own maximum |
 | fan note | under its chip's minimum, or its chip's alarm bit set: said, never flagged |
+| drive asleep HIGH | asleep at every look for a day: its health has not been read since, and a drive that stopped answering can look asleep |
 | memory HIGH / CRITICAL | corrected / uncorrectable errors since boot |
 | PCIe HIGH | fatal or non-fatal errors since boot |
 
@@ -90,30 +100,31 @@ All six only read.
 **Why a fan at 0, a minimum or an alarm is not enough.** Laptop fans and GPU fans in zero-fan mode stop on
 purpose when the machine is cool, under the minimum their chip gives. A board chip latches an alarm bit,
 and a header with nothing on it reads 0. Only a fan that ran and stopped while what it cools is hot is a
-fault. A hot drive or network adapter says nothing of a fan.
+fault. A hot drive or network adapter says nothing of a fan, and neither does a board chip's own sensor:
+its maximum is whatever the firmware left, as its critical value is.
 
 ## Health
 
 The module declares its health under ADR 0240:
 
-- `smartd` has the `unit` kind: the unit's own readiness, which the module does not judge itself.
-- The watch has the `tool` kind: `sensors_health`, every 60 s, unhealthy after 2 looks in a row, with a
-  60 s grace.
+- the drive reader has the `unit` kind: the unit's own readiness, which the module does not judge itself;
+- the fan watch has the `tool` kind: `sensors_health`, every 60 s, unhealthy after 2 looks in a row, with
+  a 60 s grace.
 
 `sensors_health` is unhealthy when:
 
 - a CPU, GPU, drive or zone temperature is at or above its own critical value. A board chip's (Super I/O)
-  critical value is shown CRITICAL by the tools and never makes the module unhealthy: its limits are
-  whatever the firmware left;
+  critical value is shown CRITICAL by the tools and never makes the module unhealthy;
 - a drive says it is failing (the FAILING row above);
 - a fan is STOPPED (the row above);
 - memory reports an uncorrectable error;
-- the watch has written no readings since the machine started, or none for five minutes.
+- the module's own processes cannot be trusted: either file missing, from an earlier boot or five minutes
+  old; a drive's read in flight past its bound; a drive not looked at for twice its cadence.
 
 Its `why` is one plain sentence per finding, for the operator. For example: "The drive sdb (WDC
-WUH722222ALE6L4) says it is failing: its overall SMART health check failed." HIGH readings and fan notes
-raise nothing. A drive whose health was not read (asleep since boot, or behind a bridge) is said in `why`
-and does not make the check unhealthy.
+WUH722222ALE6L4) says it is failing: its overall SMART health check failed." HIGH readings, fan notes and
+a drive asleep raise nothing. **A drive asleep never makes the module unhealthy, at any length** (ADR
+0256): a drive kept for backups may sleep for weeks. It is HIGH after a day, said with since when.
 
 The condition this raises is `module.sensors.<machine>.unhealthy`. ADR 0256 records what this costs:
 
@@ -139,20 +150,23 @@ They cover:
 - every class not present on the virtual machine, which is healthy;
 - a failing SATA drive; an NVMe critical warning; the NVMe temperature bit alone as HIGH; NVMe without
   `smart_status`; an old-age attribute failing now and a past failure as HIGH;
-- a missing, stale or earlier-boot reading, and the watch's staleness making the module unhealthy;
-- a fan at rest, a fan that stopped while the board is hot, a hot drive saying nothing of a fan, a hot
-  sensor on the fan's own chip, an empty header, a zero-fan GPU under its minimum, and a latched alarm;
+- a missing, stale or earlier-boot reading, and either process's staleness making the module unhealthy;
+- a fan at rest, a fan that stopped while the board's zone is hot, a hot drive saying nothing of a fan,
+  a hot sensor on the fan's own chip, a board chip's own sensor saying nothing of its fan while a CPU
+  sensor does, an empty header, a zero-fan GPU under its minimum, and a latched alarm;
 - memory and PCIe errors, and CPU throttling;
-- the watch reading as root without waking a drive: smartctl's real standby answer, an open failure
-  that is not sleep, and a bridge without a power mode;
+- the drive reader reading as root without waking a drive: smartctl's real standby answer, a SAS drive's,
+  an open failure that is not sleep, and a bridge without a power mode; asleep since when kept across
+  looks, and HIGH after a day without making the module unhealthy;
+- smartctl told each drive's kind;
 - a spinning drive read hourly and a solid-state one every 10 min;
-- a smartctl that hangs with a child holding its output, bounded, and the fans looked at and written
-  while a drive hangs;
-- a restart keeping what the last watch read; the readings directory given its mode;
-- a watch that is not root asking through `sudo -n`, and saying what stops it;
+- a smartctl that hangs with a child holding its output, bounded; a read stuck past its bound said as
+  unhealthy while the other drives are read; a drive not looked at for twice its cadence;
+- a restart keeping what the last reader read; the readings directory given its mode;
+- a drive reader that is not root asking through `sudo -n`, and saying what stops it;
 - the health answer in the shape the node-engine reads;
 - the manifest: the tools listed are the tools served, all read-only, no seat claimed, no machine named,
-  and the two health checks.
+  the two health checks, and no smartd.
 
 ## Follow-ups
 

modules/sensors/cmd/sensors/drives.go

@@ -6,21 +6,30 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"regexp"
 	"strconv"
 	"strings"
 	"time"
 )
 
 // Drives and their health. Which drives the machine has is read from /sys by anyone; what each says of its
-// own health (the NVMe health log, SATA SMART) is read by the module's watch, as root, with smartctl, and
-// left in ReadingsPath for the tools (novox/hq ADR 0256).
+// own health (the NVMe health log, SATA and SAS SMART) is read by the module's drive reader, as root, with
+// smartctl, and left in DrivesPath for the tools (novox/hq ADR 0256).
 
-// ReadingsPath is the watch's one file: what it last read of each drive, and which fans it saw turning.
-const ReadingsPath = "/run/mesh-sensors/readings.json"
+// The two files the module's processes write: the drive reader's (DrivesPath) and the fan watch's
+// (FansPath). Each process writes only its own.
+const (
+	DrivesPath = "/run/mesh-sensors/drives.json"
+	FansPath   = "/run/mesh-sensors/fans.json"
+)
 
-// StaleAfter is how old the watch's file may be before the module is unhealthy: the watch writes it every
-// FanEvery, so ten looks missed is a watch that is hung or gone, never a slow drive (drives are read
-// beside the fans, not before them).
+// AsleepHighAfter is how long a drive may be found asleep at every look before the tools say it as HIGH:
+// its health has not been read for that long, and a drive that stops answering can look asleep. It never
+// makes the module unhealthy on its own (ADR 0256): a drive kept for backups may sleep for weeks.
+const AsleepHighAfter = 24 * time.Hour
+
+// StaleAfter is how old either process's file may be before the module is unhealthy: each writes its own
+// every FanEvery, whatever a drive does, so ten writes missed is a process that is hung or gone.
 const StaleAfter = 5 * time.Minute
 
 // Smartctl is the program, by its path: the watch runs as root and runs nothing found on a PATH.
@@ -29,7 +38,7 @@ const Smartctl = "/usr/bin/smartctl"
 // FlagFailing is said beside a drive that says it is failing.
 const FlagFailing = "FAILING"
 
-// Readings is the watch's file.
+// Readings is one process's file: the drive reader fills Drives, the fan watch FansRan.
 type Readings struct {
 	BootID    string                  `json:"boot_id"`
 	WrittenAt time.Time               `json:"written_at"`
@@ -49,6 +58,11 @@ type DriveReading struct {
 	NotRead string `json:"not_read,omitempty"`
 	// LookedAt is the newest look at the drive, asleep or not: what the next look is timed from.
 	LookedAt time.Time `json:"looked_at"`
+	// AsleepSince is the first of the looks in a row that found the drive asleep.
+	AsleepSince *time.Time `json:"asleep_since,omitempty"`
+	// InFlightSince is when the read under way began: a read that never ends (a drive or a controller that
+	// stopped answering, a smartctl stuck in the kernel) is seen by how old this is.
+	InFlightSince *time.Time `json:"in_flight_since,omitempty"`
 }
 
 // Drive is one drive, as the tools say it.
@@ -79,6 +93,7 @@ type Drive struct {
 	LastSelfTest     string   `json:"last_self_test,omitempty"`
 	ReadAt           string   `json:"read_at,omitempty"`
 	Asleep           bool     `json:"asleep_at_last_look,omitempty"`
+	AsleepSince      string   `json:"asleep_since,omitempty"`
 	NotRead          string   `json:"not_read,omitempty"`
 	failingSentences []string // why it is failing, each a clause for the operator
 }
@@ -109,6 +124,9 @@ func (m *Machine) BlockDrives() []Drive {
 		}
 		dir := "/sys/block/" + b
 		d := Drive{Device: b, Model: m.read(dir + "/device/model"), Transport: transport(b, m.read(dir+"/device/vendor"))}
+		if where, err := filepath.EvalSymlinks(m.path(dir)); err == nil && strings.Contains(where, "/usb") {
+			d.Transport = "usb"
+		}
 		if s, ok := m.readInt(dir + "/size"); ok {
 			d.SizeGiB = round1(float64(s) * 512 / (1 << 30))
 		}
@@ -134,18 +152,26 @@ func transport(name, vendor string) string {
 	return "scsi"
 }
 
-// ReadReadings is the watch's file, or why there is none to use.
-func (m *Machine) ReadReadings() (*Readings, string) {
-	raw, err := os.ReadFile(m.path(ReadingsPath))
+// who names the process that writes a file, as a person reads it.
+func who(path string) string {
+	if path == DrivesPath {
+		return "the module's drive reader"
+	}
+	return "the module's fan watch"
+}
+
+// ReadReadings is one process's file, or why there is none to use.
+func (m *Machine) ReadReadings(path string) (*Readings, string) {
+	raw, err := os.ReadFile(m.path(path))
 	if err != nil {
-		return nil, "the module's watch, which reads each drive's health as root, has left no reading (is it running?)"
+		return nil, who(path) + " has left no readings (is it running?)"
 	}
 	var r Readings
 	if err := json.Unmarshal(raw, &r); err != nil {
-		return nil, "the module's watch left a reading this tool cannot read: " + err.Error()
+		return nil, who(path) + " left readings this tool cannot read: " + err.Error()
 	}
 	if id := m.bootID(); id != "" && r.BootID != "" && r.BootID != id {
-		return nil, "the module's watch has not read anything since the machine started"
+		return nil, who(path) + " has not written anything since the machine started"
 	}
 	return &r, ""
 }
@@ -156,7 +182,7 @@ func (m *Machine) Drives(r *Readings, why string) ([]Drive, string) {
 	drives := m.BlockDrives()
 	note := why
 	if r != nil && m.Now().Sub(r.WrittenAt) > StaleAfter {
-		note = fmt.Sprintf("the module's watch last wrote %s ago; it may not be running", ago(m.Now().Sub(r.WrittenAt)))
+		note = fmt.Sprintf("the module's drive reader last wrote %s ago; it may not be running", ago(m.Now().Sub(r.WrittenAt)))
 	}
 	for i := range drives {
 		d := &drives[i]
@@ -169,7 +195,7 @@ func (m *Machine) Drives(r *Readings, why string) ([]Drive, string) {
 			d.Health, d.NotRead = "not read", "the module's watch has not read this drive yet"
 			continue
 		}
-		judgeDrive(d, reading)
+		judgeDrive(d, reading, m.Now())
 	}
 	return drives, note
 }
@@ -271,11 +297,27 @@ func num(v int64) *int64 { return &v }
 
 // judgeDrive reads one drive's smartctl answer into the drive, with its findings: FAILING when the drive
 // says so itself, HIGH for the early signs a drive gives before it fails.
-func judgeDrive(d *Drive, reading DriveReading) {
+func judgeDrive(d *Drive, reading DriveReading, now time.Time) {
 	if !reading.ReadAt.IsZero() {
 		d.ReadAt = reading.ReadAt.UTC().Format(time.RFC3339)
 	}
 	d.Asleep = reading.Asleep
+	var asleepLong string
+	if reading.Asleep && reading.AsleepSince != nil {
+		d.AsleepSince = reading.AsleepSince.UTC().Format(time.RFC3339)
+		if for_ := now.Sub(*reading.AsleepSince); for_ >= AsleepHighAfter {
+			asleepLong = fmt.Sprintf("it has been asleep at every look for %s, so its health has not been read since; "+
+				"a drive that stopped answering can look asleep", ago(for_))
+		}
+	}
+	defer func() {
+		if asleepLong != "" {
+			d.Findings = append(d.Findings, asleepLong)
+			if d.Flag == "" {
+				d.Flag = FlagHigh
+			}
+		}
+	}()
 	var s smartJSON
 	if len(reading.Smartctl) == 0 || json.Unmarshal(reading.Smartctl, &s) != nil {
 		d.Health, d.NotRead = "not read", reading.NotRead
@@ -404,13 +446,32 @@ func firstError(s smartJSON, fallback string) string {
 	return "smartctl gave no health for it"
 }
 
+// deviceType is smartctl's -d for a drive whose kind is known, so smartctl does not probe for it: probing
+// a drive behind a USB bridge can spin it up. Empty for a drive smartctl must find out about itself.
+func deviceType(d Drive) string {
+	switch d.Transport {
+	case "nvme":
+		return "nvme"
+	case "ata", "usb":
+		return "sat"
+	case "scsi":
+		return "scsi"
+	}
+	return ""
+}
+
 // ReadDrive asks smartctl for one drive's health, as root (directly when this process is root, else
 // through sudo -n), without waking a drive in standby. `-n standby,2,3` makes smartctl stop with status 2
-// and the drive's power mode when it is asleep, and with status 3 when the drive cannot say (a USB bridge
-// that does not pass the question on): such a drive is not read, because reading it could wake it. A
-// drive asleep keeps the reading before.
-func (m *Machine) ReadDrive(device string, before *DriveReading) DriveReading {
-	args := []string{"--json=c", "-a", "-n", "standby,2,3", "/dev/" + device}
+// when the drive is in standby or asleep (an ATA drive with its power mode, a SAS drive or one the
+// operating system powered down with its own words), and with status 3 when the drive cannot say (a USB
+// bridge that does not pass the question on): such a drive is not read, because reading it could wake it.
+// A drive asleep keeps the reading before, and since when it has been asleep.
+func (m *Machine) ReadDrive(d Drive, before *DriveReading) DriveReading {
+	args := []string{"--json=c", "-a"}
+	if t := deviceType(d); t != "" {
+		args = append(args, "-d", t)
+	}
+	args = append(args, "-n", "standby,2,3", "/dev/"+d.Device)
 	program := Smartctl
 	if m.UID != 0 {
 		program, args = "sudo", append([]string{"-n", Smartctl}, args...)
@@ -423,7 +484,7 @@ func (m *Machine) ReadDrive(device string, before *DriveReading) DriveReading {
 		out.NotRead = "smartctl is not installed; the module declares smartmontools, so the package has not been applied yet"
 		return out
 	case r.Err == "ENOENT":
-		out.NotRead = "the watch is not root and sudo is not installed to read the drive as root"
+		out.NotRead = "the drive reader is not root and sudo is not installed to read the drive as root"
 		return out
 	case r.Err != "":
 		out.NotRead = "smartctl did not answer: " + r.Err
@@ -432,20 +493,23 @@ func (m *Machine) ReadDrive(device string, before *DriveReading) DriveReading {
 		out.NotRead = "smartctl is not installed; the module declares smartmontools, so the package has not been applied yet"
 		return out
 	case program == "sudo" && strings.HasPrefix(strings.TrimSpace(r.Stderr), "sudo:"):
-		out.NotRead = "the watch is not root and may not run smartctl through sudo without a prompt"
+		out.NotRead = "the drive reader is not root and may not run smartctl through sudo without a prompt"
 		return out
 	}
 	var s smartJSON
 	parsed := json.Unmarshal([]byte(r.Stdout), &s) == nil
 	switch {
 	case parsed && r.Status == 2 && asleep(s):
+		since := now
+		kept := DriveReading{NotRead: "the drive was asleep at every look, and is not woken to be read"}
 		if before != nil {
-			kept := *before
-			kept.Asleep, kept.LookedAt = true, now
-			return kept
+			kept = *before
+			if before.Asleep && before.AsleepSince != nil {
+				since = *before.AsleepSince
+			}
 		}
-		out.Asleep, out.NotRead = true, "the drive was asleep at every look, and is not woken to be read"
-		return out
+		kept.Asleep, kept.LookedAt, kept.AsleepSince, kept.InFlightSince = true, now, &since, nil
+		return kept
 	case parsed && r.Status == 3 && saysNoPowerMode(s):
 		out.NotRead = "the drive cannot say whether it is asleep (a USB bridge, usually), so it is not read, never to wake it"
 		return out
@@ -457,8 +521,10 @@ func (m *Machine) ReadDrive(device string, before *DriveReading) DriveReading {
 	return out
 }
 
-// asleep says smartctl stopped because the drive is in standby or asleep: by the power mode it parsed,
-// or by its own word that the operating system has the drive powered down.
+// asleep says smartctl stopped because the drive is in standby or asleep: by the ATA power mode it
+// parsed, or by its own words "Device is in <mode> mode, exit(2)", which it says for a SAS drive and for
+// a drive the operating system powered down (ataprint.cpp, scsiprint.cpp, smartctl.cpp of smartmontools
+// 7.5). A failure to open the drive says neither, whatever else its words hold.
 func asleep(s smartJSON) bool {
 	if s.PowerMode != nil {
 		switch s.PowerMode.Name {
@@ -467,13 +533,15 @@ func asleep(s smartJSON) bool {
 		}
 	}
 	for _, msg := range s.Smartctl.Messages {
-		if strings.HasPrefix(msg.String, "Device is in STANDBY (OS) mode") {
+		if sleepWords.MatchString(strings.TrimSpace(msg.String)) {
 			return true
 		}
 	}
 	return false
 }
 
+var sleepWords = regexp.MustCompile(`^Device is in [A-Z_]+( \(OS\))? mode, exit\(2\)$`)
+
 func saysNoPowerMode(s smartJSON) bool {
 	for _, msg := range s.Smartctl.Messages {
 		if strings.HasPrefix(msg.String, "CHECK POWER MODE not implemented") || strings.HasPrefix(msg.String, "CHECK POWER MODE returned unknown value") {
@@ -483,10 +551,10 @@ func saysNoPowerMode(s smartJSON) bool {
 	return false
 }
 
-// WriteReadings writes the watch's file whole, through a rename, readable by every account: it holds no
-// secret, and the tools read it as the operator account.
-func (m *Machine) WriteReadings(r Readings) error {
-	path := m.path(ReadingsPath)
+// WriteReadings writes one process's file whole, through a rename, readable by every account: it holds
+// no secret, and the tools read it as the operator account.
+func (m *Machine) WriteReadings(file string, r Readings) error {
+	path := m.path(file)
 	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
 		return err
 	}

modules/sensors/cmd/sensors/fans.go

@@ -10,8 +10,8 @@ import (
 // in its zero-fan mode), so 0 RPM alone is not a fault; nor is a chip's minimum or alarm bit, which a
 // zero-fan GPU is under by design and a Super I/O chip latches. Those are said in a note and raise nothing.
 // A fan is STOPPED only when it ran since the machine started, reads 0 now, and a temperature that fan
-// cools is at or above the hardware's own maximum: the CPU's, the board's, or one on the fan's own chip
-// (novox/hq ADR 0256 rule 4).
+// cools is at or above the hardware's own maximum: a CPU sensor's, a thermal zone's, or one on the fan's
+// own chip unless that is a board chip (novox/hq ADR 0256 rule 4).
 
 // FlagStopped is said beside a fan that should be turning and is not.
 const FlagStopped = "STOPPED"
@@ -107,8 +107,10 @@ func judgeFans(fans []Fan, ran map[string]bool, temps []Temperature) []Fan {
 }
 
 // hottestFor is the temperature furthest at or above its own high value among those this fan may be
-// cooling: the CPU's, the board's, or one on the fan's own chip. A drive or a network adapter running hot
-// says nothing of a fan.
+// cooling: a CPU sensor, a thermal zone (whose trips the firmware sets for the machine as a whole), or a
+// sensor on the fan's own chip when that chip is not a board chip. A board chip's (a Super I/O's) limits
+// are whatever the firmware left, so its maximum is no more trusted here than its critical value is for
+// health. A drive or a network adapter running hot says nothing of a fan.
 func hottestFor(f Fan, ts []Temperature) *Temperature {
 	chip := path.Dir(f.Where)
 	var best *Temperature
@@ -117,7 +119,8 @@ func hottestFor(f Fan, ts []Temperature) *Temperature {
 		if t.C == nil || t.High == nil || *t.C < *t.High {
 			continue
 		}
-		related := t.Kind == "cpu" || t.Kind == "board" || (t.Source == "hwmon" && path.Dir(t.Where) == chip)
+		related := t.Kind == "cpu" || t.Source == "thermal zone" ||
+			(t.Source == "hwmon" && path.Dir(t.Where) == chip && kindOf(f.Chip) != "board")
 		if !related {
 			continue
 		}

modules/sensors/cmd/sensors/findings.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"fmt"
+	"sort"
 	"strings"
 )
 
@@ -15,28 +16,54 @@ type Look struct {
 	Drives       []Drive
 	DrivesNote   string
 	Faults       Faults
-	// Watch is why the watch's readings are not to be trusted now: none, from an earlier boot, or older
-	// than StaleAfter. Empty when they are fresh.
-	Watch string
+	// Stale is why the module's own processes are not to be trusted now: a file missing, from an earlier
+	// boot or older than StaleAfter, a drive's read stuck, a drive not looked at for twice its cadence.
+	Stale []string
 }
 
-// LookNow reads every class, the watch's file once, and judges the fans against what the watch saw
-// turning since the machine started.
+// LookNow reads every class and each process's file once, and judges the fans against what the fan watch
+// saw turning since the machine started, and the drive reader by how old its file and its reads are.
 func (m *Machine) LookNow() Look {
 	l := Look{Temperatures: m.Temperatures(), Fans: m.Fans(), Faults: m.Faults()}
-	r, why := m.ReadReadings()
-	l.Drives, l.DrivesNote = m.Drives(r, why)
-	var ran map[string]bool
-	switch {
-	case r == nil:
-		l.Watch = "The module's watch has left no readings since the machine started, so drive health and fan history are not being read."
-	case m.Now().Sub(r.WrittenAt) > StaleAfter:
-		l.Watch = fmt.Sprintf("The module's watch has not written its readings for %s, so drive health and fan history are not being read.",
-			ago(m.Now().Sub(r.WrittenAt)))
+	now := m.Now()
+	dr, dwhy := m.ReadReadings(DrivesPath)
+	l.Drives, l.DrivesNote = m.Drives(dr, dwhy)
+	fr, fwhy := m.ReadReadings(FansPath)
+	for _, f := range []struct {
+		r    *Readings
+		why  string
+		path string
+	}{{dr, dwhy, DrivesPath}, {fr, fwhy, FansPath}} {
+		switch {
+		case f.r == nil:
+			l.Stale = append(l.Stale, capital(f.why)+".")
+		case now.Sub(f.r.WrittenAt) > StaleAfter:
+			l.Stale = append(l.Stale, fmt.Sprintf("%s has not written its readings for %s, so it is hung or gone.",
+				capital(who(f.path)), ago(now.Sub(f.r.WrittenAt))))
+		}
 	}
-	if r != nil {
+	if dr != nil {
+		byName := map[string]Drive{}
+		for _, d := range l.Drives {
+			byName[d.Device] = d
+		}
+		for _, name := range sortedKeys(dr.Drives) {
+			rd := dr.Drives[name]
+			d, present := byName[name]
+			switch {
+			case rd.InFlightSince != nil && now.Sub(*rd.InFlightSince) > ReadBound():
+				l.Stale = append(l.Stale, fmt.Sprintf("The read of the drive %s has not finished for %s; the drive or its "+
+					"controller may have stopped answering.", name, ago(now.Sub(*rd.InFlightSince))))
+			case present && rd.InFlightSince == nil && !rd.LookedAt.IsZero() && now.Sub(rd.LookedAt) > 2*cadence(d):
+				l.Stale = append(l.Stale, fmt.Sprintf("The drive %s has not been looked at for %s, more than twice as long "+
+					"as it should be.", name, ago(now.Sub(rd.LookedAt))))
+			}
+		}
+	}
+	var ran map[string]bool
+	if fr != nil {
 		ran = map[string]bool{}
-		for k := range r.FansRan {
+		for k := range fr.FansRan {
 			ran[k] = true
 		}
 	}
@@ -44,6 +71,15 @@ func (m *Machine) LookNow() Look {
 	return l
 }
 
+func sortedKeys(m map[string]DriveReading) []string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	sort.Strings(out)
+	return out
+}
+
 // Problems is what makes the module unhealthy: each a sentence for the operator.
 func (l Look) Problems() []string {
 	var out []string
@@ -68,10 +104,7 @@ func (l Look) Problems() []string {
 				c.Uncorrectable, c.Controller))
 		}
 	}
-	if l.Watch != "" {
-		out = append(out, l.Watch)
-	}
-	return out
+	return append(out, l.Stale...)
 }
 
 // Health is the answer the node-engine reads from a tool check: healthy or not, with why.
@@ -142,8 +175,8 @@ func (l Look) Said() []string {
 			out = append(out, fmt.Sprintf("HIGH: the PCIe device %s (%s) counted %d fatal and %d non-fatal errors since the machine started", e.Device, orUnknown(e.Driver), e.Fatal, e.NonFatal))
 		}
 	}
-	if l.Watch != "" {
-		out = append(out, "STALE: "+l.Watch)
+	for _, st := range l.Stale {
+		out = append(out, "STALE: "+st)
 	}
 	return append(out, l.overview()...)
 }

modules/sensors/cmd/sensors/helpers_test.go

@@ -151,7 +151,8 @@ func itoa(n int64) string {
 	return string(b)
 }
 
-// readings writes the watch's file as if it had read each drive with the smartctl JSON given.
+// readings writes both processes' files as if the drive reader had read each drive with the smartctl JSON
+// given, and the fan watch had seen the fans given turning.
 func (f *fake) readings(written time.Time, boot string, drives map[string]string, fansRan ...string) {
 	r := Readings{BootID: boot, WrittenAt: written, Drives: map[string]DriveReading{}, FansRan: map[string]time.Time{}}
 	for d, j := range drives {
@@ -160,11 +161,23 @@ func (f *fake) readings(written time.Time, boot string, drives map[string]string
 	for _, k := range fansRan {
 		r.FansRan[k] = written
 	}
+	for d := range r.Drives {
+		x := r.Drives[d]
+		x.LookedAt = written
+		r.Drives[d] = x
+	}
+	f.write(DrivesPath, Readings{BootID: boot, WrittenAt: written, Drives: r.Drives})
+	f.write(FansPath, Readings{BootID: boot, WrittenAt: written, FansRan: r.FansRan})
+}
+
+// write writes one process's file.
+func (f *fake) write(path string, r Readings) {
+	f.t.Helper()
 	raw, err := json.Marshal(r)
 	if err != nil {
 		f.t.Fatal(err)
 	}
-	f.file(ReadingsPath, string(raw))
+	f.file(path, string(raw))
 }
 
 // smartctl answers, as smartmontools 7 writes them (trimmed to what the module reads).
@@ -190,9 +203,14 @@ const (
 
 type ctxT = context.Context
 
-// smartctlLine is the command the watch runs for one drive, as root.
-func smartctlLine(device string) string {
-	return "/usr/bin/smartctl --json=c -a -n standby,2,3 /dev/" + device
+// smartctlLine is the command the drive reader runs for one drive, as root, with smartctl's -d when the
+// drive's kind is known.
+func smartctlLine(device, typ string) string {
+	d := ""
+	if typ != "" {
+		d = "-d " + typ + " "
+	}
+	return "/usr/bin/smartctl --json=c -a " + d + "-n standby,2,3 /dev/" + device
 }
 
 func chmod(path string, mode os.FileMode) error { return os.Chmod(path, mode) }

modules/sensors/cmd/sensors/main.go

@@ -3,9 +3,9 @@
 // errors — read from /sys and /proc, and changing nothing.
 //
 // One binary, two ways to run. With no argument it is the bundle the node's tool runner launches and
-// speaks MCP to over stdio (ADR 0188, ADR 0193), as the operator account. With `watch` it is the module's
-// long-running process, run by the node-engine as root, which reads the drives' health and notes which
-// fans turn (watch.go).
+// speaks MCP to over stdio (ADR 0188, ADR 0193), as the operator account. With `watch` and `drives` it is
+// one of the module's two long-running processes, run by the node-engine as root: the fan watch, and the
+// drive reader (watch.go).
 package main
 
 import (
@@ -19,16 +19,24 @@ import (
 )
 
 func main() {
-	if len(os.Args) > 1 && os.Args[1] == "watch" {
+	if len(os.Args) > 1 {
 		ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
 		defer stop()
-		NewWatch(Here()).Run(ctx)
+		switch os.Args[1] {
+		case "watch":
+			w := NewFanWatch(Here())
+			every(ctx, "watch", w.Look)
+		case "drives":
+			r := NewDriveReader(Here())
+			r.Async = true
+			every(ctx, "drives", r.Look)
+		default:
+			fmt.Fprintf(os.Stderr, "sensors: %q is not a way to run; with no argument it serves its tools, with watch it "+
+				"watches the fans, with drives it reads the drives\n", os.Args[1])
+			os.Exit(2)
+		}
 		return
 	}
-	if len(os.Args) > 1 {
-		fmt.Fprintf(os.Stderr, "sensors: %q is not a way to run; with no argument it serves its tools, with watch it watches\n", os.Args[1])
-		os.Exit(2)
-	}
 	// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): sensors.
 	if err := stdio.Serve("", Tools(Here())); err != nil {
 		fmt.Fprintln(os.Stderr, err)

modules/sensors/cmd/sensors/manifest_test.go

@@ -51,43 +51,33 @@ func TestTheManifestNamesExactlyTheToolsTheBundleServesAllReadOnlyAndNoMachine(t
 	}
 }
 
-// ADR 0240 rule 2 and ADR 0256 rule 4: the tool check on the watch, beside the unit's own readiness on
-// smartd, which the module does not judge itself.
-func TestTheHealthIsTheDriveDaemonsUnitAndTheWatchsToolCheck(t *testing.T) {
-	m, _ := readManifest(t)
+// ADR 0240 rule 2 and ADR 0256 rule 4: the tool check on the fan watch, beside the drive reader's unit's
+// readiness, which the module does not judge itself; and no smartd, so the module is the only reader.
+func TestTheHealthIsTheDriveReadersUnitAndTheFanWatchsToolCheckAndNoSmartd(t *testing.T) {
+	m, raw := readManifest(t)
 	byID := map[string]map[string]any{}
 	for _, r := range m.Resources {
 		byID[r["id"].(string)] = r
 	}
-	smartd, watch := byID["smartd"], byID["watch"]
-	if smartd == nil || watch == nil {
+	drives, watch := byID["drives"], byID["watch"]
+	if drives == nil || watch == nil || len(byID) != 3 {
 		t.Fatalf("resources: %v", byID)
 	}
-	if h, _ := smartd["health"].(map[string]any); h["kind"] != "unit" || smartd["state"] != "running" {
-		t.Errorf("smartd is judged by its unit: %v", smartd)
+	if h, _ := drives["health"].(map[string]any); h["kind"] != "unit" || drives["type"] != "process" || drives["user"] != nil {
+		t.Errorf("the drive reader runs as root and is judged by its unit: %v", drives)
 	}
 	h, _ := watch["health"].(map[string]any)
-	if h["kind"] != "tool" || h["tool"] != "sensors_health" || watch["type"] != "process" || watch["user"] != nil {
-		t.Errorf("the watch runs as root and is judged by sensors_health: %v", watch)
+	if h["kind"] != "tool" || h["tool"] != "sensors_health" || watch["type"] != "process" {
+		t.Errorf("the fan watch is judged by sensors_health: %v", watch)
 	}
-	run, _ := watch["run"].([]any)
-	if len(run) != 2 || run[1] != "watch" {
-		t.Errorf("the watch runs the bundle's watch: %v", run)
-	}
-	conf, _ := byID["smartd-conf"]["content"].(string)
-	if !strings.Contains(conf, "\nDEVICESCAN -a -n standby,q\n") || strings.Contains(conf, " -m ") {
-		t.Errorf("smartd never wakes a drive and sends no mail: %q", conf)
-	}
-	dropIn, _ := byID["smartd-drop-in"]["content"].(string)
-	for _, want := range []string{"ConditionVirtualization=\n", "ExecStart=\nExecStart=/usr/bin/smartd -n -q never -i 3600\n"} {
-		if !strings.Contains(dropIn, want) {
-			t.Errorf("the drop-in misses %q", want)
+	for id, mode := range map[string]string{"drives": "drives", "watch": "watch"} {
+		run, _ := byID[id]["run"].([]any)
+		if len(run) != 2 || run[1] != mode {
+			t.Errorf("%s runs the bundle's %s: %v", id, mode, run)
 		}
 	}
-	for _, f := range []string{"smartd-conf", "smartd-drop-in"} {
-		if !strings.Contains(strings.Join(anyStrings(smartd["restart-on"]), ","), f) {
-			t.Errorf("smartd is restarted when %s changes", f)
-		}
+	if strings.Contains(raw, "smartd") {
+		t.Errorf("the module runs no second reader of the drives")
 	}
 }
 

modules/sensors/cmd/sensors/review_test.go

@@ -183,11 +183,12 @@ func TestZeroDegreesIsAReading(t *testing.T) {
 func TestAMissingOrStaleWatchMakesTheModuleUnhealthy(t *testing.T) {
 	f := newFake(t)
 	f.laptop()
-	if h := f.machine().LookNow().Health(); h.Healthy || !strings.Contains(h.Why, "watch") {
+	if h := f.machine().LookNow().Health(); h.Healthy || !strings.Contains(h.Why, "drive reader has left no readings") ||
+		!strings.Contains(h.Why, "fan watch has left no readings") {
 		t.Errorf("no readings at all: %+v", h)
 	}
 	f.readings(f.now.Add(-StaleAfter-time.Minute), bootID, map[string]string{"nvme0n1": nvmePassed})
-	if h := f.machine().LookNow().Health(); h.Healthy || !strings.Contains(h.Why, "has not written its readings for") {
+	if h := f.machine().LookNow().Health(); h.Healthy || !strings.Contains(h.Why, "has not written its readings for 6 min, so it is hung or gone") {
 		t.Errorf("stale readings: %+v", h)
 	}
 	f.readings(f.now.Add(-time.Minute), bootID, map[string]string{"nvme0n1": nvmePassed})
@@ -199,25 +200,26 @@ func TestAMissingOrStaleWatchMakesTheModuleUnhealthy(t *testing.T) {
 func TestSmartctlsStandbyAnswerIsAsleepOnlyWithItsPowerMode(t *testing.T) {
 	f := newFake(t)
 	f.drive("sda", "X", "ATA     ", 100, true)
-	f.answers[smartctlLine("sda")] = Ran{Stdout: standbyJSON, Status: 2}
+	d := f.machine().BlockDrives()[0]
+	f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: standbyJSON, Status: 2}
 	before := DriveReading{ReadAt: f.now.Add(-time.Hour), Smartctl: []byte(ataPassed)}
-	got := f.machine().ReadDrive("sda", &before)
+	got := f.machine().ReadDrive(d, &before)
 	if !got.Asleep || string(got.Smartctl) != ataPassed {
 		t.Errorf("standby keeps the reading before: %+v", got)
 	}
-	f.answers[smartctlLine("sda")] = Ran{Stdout: openFailedJSON, Status: 2}
-	got = f.machine().ReadDrive("sda", &before)
+	f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: openFailedJSON, Status: 2}
+	got = f.machine().ReadDrive(d, &before)
 	if got.Asleep {
 		t.Errorf("an open failure is not sleep, whatever its words: %+v", got)
 	}
-	f.answers[smartctlLine("sda")] = Ran{Stdout: noPowerModeJSON, Status: 3}
-	got = f.machine().ReadDrive("sda", nil)
+	f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: noPowerModeJSON, Status: 3}
+	got = f.machine().ReadDrive(d, nil)
 	if got.Asleep || !strings.Contains(got.NotRead, "cannot say whether it is asleep") {
 		t.Errorf("a bridge without a power mode is skipped and said: %+v", got)
 	}
 }
 
-func TestASmartctlThatHangsCostsOneLineAndTheFansGoOn(t *testing.T) {
+func TestASmartctlThatHangsCostsOneLineAndTheReaderGoesOn(t *testing.T) {
 	old := CommandTimeout
 	CommandTimeout = 300 * time.Millisecond
 	defer func() { CommandTimeout = old }()
@@ -236,7 +238,7 @@ func TestASmartctlThatHangsCostsOneLineAndTheFansGoOn(t *testing.T) {
 	release := make(chan struct{})
 	m := f.machine()
 	m.Run = func(_ ctxT, name string, args ...string) Ran { <-release; return Ran{Stdout: ataPassed} }
-	w := NewWatch(m)
+	w := NewDriveReader(m)
 	w.Async = true
 	if err := w.Look(); err != nil {
 		t.Fatal(err)
@@ -245,12 +247,12 @@ func TestASmartctlThatHangsCostsOneLineAndTheFansGoOn(t *testing.T) {
 	if err := w.Look(); err != nil {
 		t.Fatal(err)
 	}
-	r2, why := m.ReadReadings()
-	if why != "" || !r2.WrittenAt.Equal(f.now) || len(r2.FansRan) != 1 {
-		t.Errorf("the fans are looked at and written while a drive hangs: %+v %q", r2, why)
+	r2, why := m.ReadReadings(DrivesPath)
+	if why != "" || !r2.WrittenAt.Equal(f.now) || r2.Drives["sda"].InFlightSince == nil {
+		t.Errorf("the reader writes its file, each read marked in flight, while the drives hang: %+v %q", r2, why)
 	}
 	close(release)
-	w.WaitDrives()
+	w.WaitReads()
 }
 
 func TestASpinningDriveIsReadAtMostHourlyAndASolidStateOneEveryTenMinutes(t *testing.T) {
@@ -258,10 +260,10 @@ func TestASpinningDriveIsReadAtMostHourlyAndASolidStateOneEveryTenMinutes(t *tes
 	f.server()
 	f.drive("nvme0n1", "N", "", 100, false)
 	for _, d := range []string{"sda", "sdb", "sdc"} {
-		f.answers[smartctlLine(d)] = Ran{Stdout: ataPassed}
+		f.answers[smartctlLine(d, "sat")] = Ran{Stdout: ataPassed}
 	}
-	f.answers[smartctlLine("nvme0n1")] = Ran{Stdout: nvmePassed}
-	w := NewWatch(f.machine())
+	f.answers[smartctlLine("nvme0n1", "nvme")] = Ran{Stdout: nvmePassed}
+	w := NewDriveReader(f.machine())
 	if err := w.Look(); err != nil || len(f.calls) != 4 {
 		t.Fatalf("first look reads every drive: %q %v", f.calls, err)
 	}
@@ -285,7 +287,7 @@ func TestAnExistingReadingsDirectoryGetsItsMode(t *testing.T) {
 	if err := chmod(f.root+"/run/mesh-sensors", 0o700); err != nil {
 		t.Fatal(err)
 	}
-	if err := f.machine().WriteReadings(Readings{}); err != nil {
+	if err := f.machine().WriteReadings(DrivesPath, Readings{}); err != nil {
 		t.Fatal(err)
 	}
 	if mode := modeOf(t, f.root+"/run/mesh-sensors"); mode != 0o755 {

modules/sensors/cmd/sensors/second_review_test.go

@@ -0,0 +1,158 @@
+package main
+
+// The reproductions of the second review of 2026-10-08 (mesh-catalog #129, findings A to E).
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+)
+
+// A: a read stuck in the kernel is never ended, so it is seen by its age, and one stuck drive does not
+// stop the others being read.
+func TestAStuckReadIsUnhealthyAndDoesNotStopTheOtherDrives(t *testing.T) {
+	f := newFake(t)
+	f.server()
+	stuck := make(chan struct{})
+	m := f.machine()
+	m.Run = func(_ context.Context, name string, args ...string) Ran {
+		if strings.HasSuffix(args[len(args)-1], "/dev/sda") {
+			<-stuck
+		}
+		return Ran{Stdout: ataPassed}
+	}
+	w := NewDriveReader(m)
+	w.Async = true
+	looked := make(chan error, 1)
+	go func() { looked <- w.Look() }()
+	select {
+	case err := <-looked:
+		if err != nil {
+			t.Fatal(err)
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("a look waited on a stuck drive")
+	}
+	deadline := time.Now().Add(5 * time.Second)
+	for {
+		_ = w.write()
+		r, _ := m.ReadReadings(DrivesPath)
+		if len(r.Drives["sdb"].Smartctl) > 0 && len(r.Drives["sdc"].Smartctl) > 0 {
+			if r.Drives["sda"].InFlightSince == nil {
+				t.Fatalf("sda is in flight: %+v", r.Drives["sda"])
+			}
+			break
+		}
+		if time.Now().After(deadline) {
+			t.Fatalf("one stuck drive held up the others: %+v", r.Drives)
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	f.fresh2(w)
+	if h := m.LookNow().Health(); !h.Healthy {
+		t.Fatalf("a read under way is not yet stuck: %+v", h)
+	}
+	f.now = f.now.Add(ReadBound() + time.Minute)
+	_ = w.Look()
+	f.write(FansPath, Readings{BootID: bootID, WrittenAt: f.now})
+	h := m.LookNow().Health()
+	if h.Healthy || !strings.Contains(h.Why, "The read of the drive sda has not finished for 1 min; the drive or its controller may have stopped answering.") {
+		t.Errorf("health = %+v", h)
+	}
+	close(stuck)
+	w.WaitReads()
+}
+
+// fresh2 writes the fan watch's file now, beside a drive reader that has just written its own.
+func (f *fake) fresh2(w *DriveReader) {
+	_ = w.write()
+	f.write(FansPath, Readings{BootID: bootID, WrittenAt: f.now})
+}
+
+// A: a drive the reader has not looked at for twice its cadence is said, though the file is fresh.
+func TestADriveNotLookedAtForTwiceItsCadenceIsUnhealthy(t *testing.T) {
+	f := newFake(t)
+	f.server()
+	f.fresh()
+	r, _ := f.machine().ReadReadings(DrivesPath)
+	x := r.Drives["sdb"]
+	x.LookedAt = f.now.Add(-2*SpinningEvery - time.Minute)
+	r.Drives["sdb"] = x
+	f.write(DrivesPath, *r)
+	h := f.machine().LookNow().Health()
+	if h.Healthy || !strings.Contains(h.Why, "The drive sdb has not been looked at for 2.0 h, more than twice as long as it should be.") {
+		t.Errorf("health = %+v", h)
+	}
+}
+
+// B: a board chip's own sensor at its maximum says nothing of the board chip's fan; a CPU sensor does.
+func TestABoardChipsOwnSensorDoesNotStopItsFanAndACPUSensorDoes(t *testing.T) {
+	f := newFake(t)
+	f.server()
+	f.files("/sys/class/hwmon/hwmon1", map[string]string{"fan1_input": "0", "temp1_input": "75000", "temp1_max": "70000", "temp1_label": "SYSTIN"})
+	f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataPassed, "sdc": ataPassed}, "nct6775/fan1")
+	l := f.machine().LookNow()
+	if l.Fans[0].Flag != "" || !l.Health().Healthy {
+		t.Fatalf("a Super I/O's maximum is not trusted: %+v %+v", l.Fans[0], l.Health())
+	}
+	f.file("/sys/class/hwmon/hwmon0/temp1_input", "81000")
+	l = f.machine().LookNow()
+	if l.Fans[0].Flag != FlagStopped || !strings.Contains(l.Fans[0].Why, "the CPU (coretemp Package id 0) is at 81.0 °C") {
+		t.Errorf("the CPU at its maximum with a fan that ran reading 0: %+v", l.Fans[0])
+	}
+}
+
+// D: a SAS drive in standby says so in words and no ATA power mode; status 2 with those words is asleep.
+func TestASASDriveInStandbyIsAsleep(t *testing.T) {
+	f := newFake(t)
+	f.drive("sdm", "ST4000NM0023", "SEAGATE ", 100, true)
+	d := f.machine().BlockDrives()[0]
+	sas := `{"smartctl":{"messages":[{"string":"Device is in STANDBY mode, exit(2)","severity":"information"}],"exit_status":2},` +
+		`"device":{"name":"/dev/sdm","type":"scsi","protocol":"SCSI"}}`
+	f.answers[smartctlLine("sdm", "scsi")] = Ran{Stdout: sas, Status: 2}
+	before := DriveReading{ReadAt: f.now.Add(-time.Hour), Smartctl: []byte(ataPassed)}
+	got := f.machine().ReadDrive(d, &before)
+	if !got.Asleep || string(got.Smartctl) != ataPassed || got.AsleepSince == nil {
+		t.Errorf("a SAS drive in standby keeps its reading: %+v", got)
+	}
+}
+
+// E: a drive asleep at every look for a day is HIGH, said with since when, and the module stays healthy.
+func TestADriveAsleepForADayIsHighAndNotUnhealthy(t *testing.T) {
+	f := newFake(t)
+	f.server()
+	f.fresh()
+	r, _ := f.machine().ReadReadings(DrivesPath)
+	since := f.now.Add(-25 * time.Hour)
+	x := r.Drives["sdc"]
+	x.Asleep, x.AsleepSince = true, &since
+	r.Drives["sdc"] = x
+	f.write(DrivesPath, *r)
+	l := f.machine().LookNow()
+	d := l.Drives[2]
+	if d.Flag != FlagHigh || d.AsleepSince != "2026-10-07T11:00:00Z" || !strings.Contains(strings.Join(d.Findings, ";"), "asleep at every look for 25.0 h") {
+		t.Errorf("drive = %+v", d)
+	}
+	if !l.Health().Healthy {
+		t.Errorf("a drive asleep is never unhealthy on its own: %+v", l.Health())
+	}
+	if !strings.Contains(strings.Join(l.Said(), "\n"), "HIGH: the drive sdc") {
+		t.Errorf("the summary says it: %q", l.Said())
+	}
+}
+
+// C: smartctl is told the drive's kind where it is known, so it does not probe a drive to find out.
+func TestSmartctlIsToldTheDrivesKind(t *testing.T) {
+	f := newFake(t)
+	f.drive("nvme0n1", "N", "", 100, false)
+	f.drive("sda", "A", "ATA     ", 100, true)
+	f.drive("sdm", "S", "SEAGATE ", 100, true)
+	f.drive("vda", "", "", 100, true)
+	w := NewDriveReader(f.machine())
+	_ = w.Look()
+	want := []string{smartctlLine("nvme0n1", "nvme"), smartctlLine("sda", "sat"), smartctlLine("sdm", "scsi"), smartctlLine("vda", "")}
+	if strings.Join(f.calls, "\n") != strings.Join(want, "\n") {
+		t.Errorf("calls:\n%s\nwant:\n%s", strings.Join(f.calls, "\n"), strings.Join(want, "\n"))
+	}
+}

modules/sensors/cmd/sensors/sensors_test.go

@@ -122,7 +122,7 @@ func TestADriveThatSaysItIsFailingMakesTheModuleUnhealthy(t *testing.T) {
 	f := newFake(t)
 	f.server()
 	f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataFailing, "sdc": ataPending})
-	ds, note := f.machine().Drives(f.machine().ReadReadings())
+	ds, note := f.machine().Drives(f.machine().ReadReadings(DrivesPath))
 	if note != "" || len(ds) != 3 {
 		t.Fatalf("three drives, the RAID volume left out, no note: %d, %q", len(ds), note)
 	}
@@ -147,7 +147,7 @@ func TestAnNVMeCriticalWarningBitIsFailing(t *testing.T) {
 	f := newFake(t)
 	f.laptop()
 	f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmeSpareLow})
-	ds, _ := f.machine().Drives(f.machine().ReadReadings())
+	ds, _ := f.machine().Drives(f.machine().ReadReadings(DrivesPath))
 	d := ds[0]
 	if d.Flag != FlagFailing || *d.CriticalWarning != 1 || *d.PercentageUsed != 97 || d.Model != "Samsung SSD 990 PRO 2TB" {
 		t.Fatalf("spare below threshold is failing: %+v", d)
@@ -184,11 +184,11 @@ func TestAHealthyLaptopReadsHealthyWithItsDriveRead(t *testing.T) {
 func TestAMissingStaleOrEarlierBootReadingIsSaid(t *testing.T) {
 	f := newFake(t)
 	f.laptop()
-	if ds, note := f.machine().Drives(f.machine().ReadReadings()); ds[0].Health != "not read" || !strings.Contains(note, "has left no reading") {
+	if ds, note := f.machine().Drives(f.machine().ReadReadings(DrivesPath)); ds[0].Health != "not read" || !strings.Contains(note, "has left no readings") {
 		t.Errorf("no file: %+v %q", ds[0], note)
 	}
 	f.readings(f.now.Add(-2*time.Hour), bootID, map[string]string{"nvme0n1": nvmePassed})
-	if ds, note := f.machine().Drives(f.machine().ReadReadings()); ds[0].Health != "passed" || note != "the module's watch last wrote 2.0 h ago; it may not be running" {
+	if ds, note := f.machine().Drives(f.machine().ReadReadings(DrivesPath)); ds[0].Health != "passed" || note != "the module's drive reader last wrote 2.0 h ago; it may not be running" {
 		t.Errorf("stale: %+v %q", ds[0], note)
 	}
 	f.readings(f.now, "an-earlier-boot", map[string]string{"nvme0n1": nvmeSpareLow}, "asus/fan2")
@@ -277,64 +277,76 @@ func TestPCIeErrorsAndThrottlingAreCountedWhereTheMachineHasThem(t *testing.T) {
 	}
 }
 
-func TestTheWatchReadsEachDriveAsRootWithoutWakingItAndNotesTheFans(t *testing.T) {
+func TestTheDriveReaderReadsEachDriveAsRootWithoutWakingItAndTheFanWatchNotesTheFans(t *testing.T) {
 	f := newFake(t)
 	f.server()
-	f.answers[smartctlLine("sda")] = Ran{Stdout: ataPassed}
-	f.answers[smartctlLine("sdb")] = Ran{Stdout: ataFailing, Status: 24}
-	f.answers[smartctlLine("sdc")] = Ran{Stdout: strings.Replace(standbyJSON, "/dev/sda", "/dev/sdc", 1), Status: 2}
-	w := NewWatch(f.machine())
+	f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: ataPassed}
+	f.answers[smartctlLine("sdb", "sat")] = Ran{Stdout: ataFailing, Status: 24}
+	f.answers[smartctlLine("sdc", "sat")] = Ran{Stdout: strings.Replace(standbyJSON, "/dev/sda", "/dev/sdc", 1), Status: 2}
+	w := NewDriveReader(f.machine())
 	if err := w.Look(); err != nil {
 		t.Fatal(err)
 	}
 	if len(f.calls) != 3 {
-		t.Fatalf("one smartctl per drive, directly as root: %q", f.calls)
+		t.Fatalf("one smartctl per drive, directly as root, told it is SATA: %q", f.calls)
 	}
-	info, err := os.Stat(filepath.Join(f.root, ReadingsPath))
-	if err != nil || info.Mode().Perm() != 0o644 {
-		t.Fatalf("the readings are readable by the operator account: %v %v", info, err)
+	fw := NewFanWatch(f.machine())
+	if err := fw.Look(); err != nil {
+		t.Fatal(err)
 	}
-	r, why := f.machine().ReadReadings()
-	if why != "" || r.Drives["sdc"].NotRead != "the drive was asleep at every look, and is not woken to be read" || !r.Drives["sdc"].Asleep {
-		t.Fatalf("sdc asleep: %+v %q", r.Drives["sdc"], why)
+	for _, p := range []string{DrivesPath, FansPath} {
+		info, err := os.Stat(filepath.Join(f.root, p))
+		if err != nil || info.Mode().Perm() != 0o644 {
+			t.Fatalf("%s is readable by the operator account: %v %v", p, info, err)
+		}
 	}
-	if _, ok := r.FansRan["nct6775/fan1"]; !ok || len(r.FansRan) != 1 {
-		t.Errorf("the turning fan is noted, the empty header not: %v", r.FansRan)
+	r, why := f.machine().ReadReadings(DrivesPath)
+	if why != "" || r.Drives["sdc"].NotRead != "the drive was asleep at every look, and is not woken to be read" || !r.Drives["sdc"].Asleep ||
+		r.Drives["sdc"].AsleepSince == nil || !r.Drives["sdc"].AsleepSince.Equal(f.now) {
+		t.Fatalf("sdc asleep since now: %+v %q", r.Drives["sdc"], why)
 	}
-	// An hour has not passed: the fans are looked at, the spinning drives are not read again.
+	fr, _ := f.machine().ReadReadings(FansPath)
+	if _, ok := fr.FansRan["nct6775/fan1"]; !ok || len(fr.FansRan) != 1 {
+		t.Errorf("the turning fan is noted, the empty header not: %v", fr.FansRan)
+	}
+	// An hour has not passed: the spinning drives are not read again.
 	f.now = f.now.Add(FanEvery)
 	f.calls = nil
 	if err := w.Look(); err != nil || len(f.calls) != 0 {
 		t.Fatalf("drives read again within the hour: %q %v", f.calls, err)
 	}
-	// A new watch (a restart) keeps what the last one read, and a drive asleep keeps its reading before.
+	// A new reader (a restart) keeps what the last one read, and a drive asleep keeps its reading before.
 	f.now = f.now.Add(SpinningEvery)
-	f.answers[smartctlLine("sda")] = Ran{Stdout: strings.Replace(standbyJSON, `"STANDBY"}`, `"SLEEP"}`, 1), Status: 2}
-	w2 := NewWatch(f.machine())
+	f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: strings.Replace(standbyJSON, `"STANDBY"}`, `"SLEEP"}`, 1), Status: 2}
+	w2 := NewDriveReader(f.machine())
 	if err := w2.Look(); err != nil {
 		t.Fatal(err)
 	}
-	r, _ = f.machine().ReadReadings()
+	r, _ = f.machine().ReadReadings(DrivesPath)
 	if !r.Drives["sda"].Asleep || len(r.Drives["sda"].Smartctl) == 0 {
 		t.Errorf("sda asleep keeps its reading: %+v", r.Drives["sda"])
 	}
-	ds, _ := f.machine().Drives(f.machine().ReadReadings())
+	if since := r.Drives["sdc"].AsleepSince; since == nil || !since.Equal(f.now.Add(-SpinningEvery-FanEvery)) {
+		t.Errorf("sdc asleep again keeps since when: %+v", r.Drives["sdc"])
+	}
+	ds, _ := f.machine().Drives(f.machine().ReadReadings(DrivesPath))
 	if ds[0].Health != "passed" || !ds[0].Asleep || ds[1].Health != "failing" {
 		t.Errorf("drives = %+v", ds)
 	}
 }
 
-func TestTheWatchNotRootAsksThroughSudoAndSaysWhatStopsIt(t *testing.T) {
+func TestTheDriveReaderNotRootAsksThroughSudoAndSaysWhatStopsIt(t *testing.T) {
 	f := newFake(t)
 	f.uid = 1000
 	f.drive("sda", "X", "ATA     ", 100, true)
-	f.answers["sudo -n "+smartctlLine("sda")] = Ran{Stderr: "sudo: a password is required\n", Status: 1}
-	got := f.machine().ReadDrive("sda", nil)
-	if got.NotRead != "the watch is not root and may not run smartctl through sudo without a prompt" {
+	d := f.machine().BlockDrives()[0]
+	f.answers["sudo -n "+smartctlLine("sda", "sat")] = Ran{Stderr: "sudo: a password is required\n", Status: 1}
+	got := f.machine().ReadDrive(d, nil)
+	if got.NotRead != "the drive reader is not root and may not run smartctl through sudo without a prompt" {
 		t.Errorf("not read = %q", got.NotRead)
 	}
 	f.uid = 0
-	got = f.machine().ReadDrive("sda", nil)
+	got = f.machine().ReadDrive(d, nil)
 	if !strings.Contains(got.NotRead, "smartctl is not installed") {
 		t.Errorf("not read = %q", got.NotRead)
 	}

modules/sensors/cmd/sensors/tools.go

@@ -5,7 +5,7 @@ import (
 )
 
 // Tools is the module's tools over one machine. Every one only reads, and none needs root: what only root
-// may read, the watch has read (drives.go).
+// may read, the drive reader has read (drives.go, watch.go).
 func Tools(m *Machine) []stdio.Tool {
 	return []stdio.Tool{
 		{
@@ -35,8 +35,8 @@ func Tools(m *Machine) []stdio.Tool {
 			Name: "sensors_fans",
 			Description: "Every fan the machine reports: RPM, the chip's own minimum and alarm where it has them (said in a note, never " +
 				"flagged: a zero-fan GPU is under its minimum by design and a board chip latches its alarm), and whether the " +
-				"module's watch saw it turning since the machine started. STOPPED only when it ran and reads 0 while the CPU, the " +
-				"board or a sensor on its own chip is at or above its own maximum. " +
+				"module's fan watch saw it turning since the machine started. STOPPED only when it ran and reads 0 while a CPU " +
+				"sensor, a thermal zone or a sensor on its own non-board chip is at or above its own maximum. " +
 				"Replaces sensors. (r)",
 			Run: func(map[string]any) (any, error) {
 				l := m.LookNow()
@@ -45,13 +45,13 @@ func Tools(m *Machine) []stdio.Tool {
 		},
 		{
 			Name: "sensors_drives",
-			Description: "Each drive's health, as the module's watch last read it as root with smartctl (every 10 minutes, never " +
-				"waking a drive in standby): model, serial, size, whether it spins, the overall SMART verdict; for NVMe the critical " +
+			Description: "Each drive's health, as the module's drive reader last read it as root with smartctl (a spinning drive " +
+				"hourly, any other every 10 minutes; a drive in standby is not read, and said as asleep since when): model, serial, size, whether it spins, the overall SMART verdict; for NVMe the critical " +
 				"warning bits, spare, endurance used and media errors; for SATA the reallocated, pending and unreadable sectors, the " +
 				"error log and the last self-test. FAILING when the drive says so, HIGH for the early signs. Replaces smartctl -a " +
 				"and nvme smart-log. (r)",
 			Run: func(map[string]any) (any, error) {
-				ds, note := m.Drives(m.ReadReadings())
+				ds, note := m.Drives(m.ReadReadings(DrivesPath))
 				out := map[string]any{"drives": ds, "count": len(ds)}
 				if note != "" {
 					out["note"] = note
@@ -73,8 +73,9 @@ func Tools(m *Machine) []stdio.Tool {
 			Name: "sensors_health",
 			Description: "Whether the hardware is failing, as the module's health check (novox/hq ADR 0256): unhealthy when a " +
 				"CPU, GPU, drive or zone temperature is at or above its own critical value (a board chip's is shown, never judged), " +
-				"a drive says it is failing, a fan that ran reads 0 while the CPU, the board or its own chip is at its own maximum, " +
-				"memory reports an uncorrectable error, or the module's watch has not written its readings for five minutes; with " +
+				"a drive says it is failing, a fan that ran reads 0 while a CPU sensor, a thermal zone or its own (non-board) chip is at its own maximum, " +
+				"memory reports an uncorrectable error, or the module's own processes are stale (a file five minutes old, a drive's " +
+				"read stuck, a drive not looked at for twice its cadence); with " +
 				"why, one sentence per finding. HIGH readings, a fan's alarm bit and a fan under its minimum do not make it " +
 				"unhealthy. (r)",
 			Run: func(map[string]any) (any, error) {

modules/sensors/cmd/sensors/watch.go

@@ -8,21 +8,25 @@ import (
 	"time"
 )
 
-// The watch: the module's long-running process, run by the node-engine as root (novox/hq ADR 0256). It
-// reads each drive's health, so the tools, which run as the operator account, need no root; it never wakes
-// a drive in standby; and it notes every FanEvery which fans are turning, so a fan that ran and stopped can
-// be told from one that never runs. It writes ReadingsPath and nothing else.
+// The module's two long-running processes, both of this binary, run by the node-engine (novox/hq ADR
+// 0256). Each writes its own file every FanEvery and nothing else:
 //
-// **The drives never hold up the fans.** In Run, the drives are read beside the looks at the fans, one
-// pass at a time, each smartctl bounded by CommandTimeout; the file is written every FanEvery whatever a
-// drive does, so a watch whose file grows old is a watch that is hung or gone, and the health check says
-// so (StaleAfter).
+//   - **the fan watch** (`sensors watch`) notes which fans are turning, so a fan that ran and stopped can
+//     be told from one that never runs. It is judged by the module's health tool, sensors_health;
+//   - **the drive reader** (`sensors drives`), as root, reads each drive's health, so the tools, which run
+//     as the operator account, need no root. It is judged by its unit.
 //
-// **A spinning drive is read at most hourly.** Reading SMART from an awake drive can restart its standby
-// timer, so a drive read more often than its timer is set for never spins down. Whether the machines'
-// drives are set to spin down at all is not known to the mesh (no tool reads a drive's standby timer), so
-// the watch reads a rotational drive every SpinningEvery and smartd checks every drive at the same
-// interval (its -i, in the module's drop-in). A drive in standby is never woken: smartctl is asked with -n.
+// **One drive never holds up another, nor the fans.** Each drive is read in its own goroutine, one read at
+// a time per drive, bounded by CommandTimeout and WaitDelay. A smartctl stuck in the kernel (a drive or a
+// controller that stopped answering) cannot be ended at all: its read stays in flight, marked since when,
+// and the health check says so once it is older than any read can take (ReadBound). The fans are another
+// process, so they are looked at whatever the drives do.
+//
+// **The module is the only reader, and a spinning drive is read hourly.** Reading SMART from an awake
+// drive can restart its standby timer, so a drive read more often than its timer is set for never spins
+// down. Whether the machines' drives are set to spin down is not known to the mesh, so a rotational drive
+// is read every SpinningEvery. No other reader runs: the module does not run smartd, whose own schedule
+// would read the same drives at another phase of the hour, and whose start reads every drive.
 
 const (
 	FanEvery = 30 * time.Second
@@ -32,129 +36,143 @@ const (
 	SpinningEvery = time.Hour
 )
 
-// Watch keeps the readings between looks.
-type Watch struct {
+// ReadBound is the longest one drive's read can take when the drive answers or smartctl can be ended: a
+// read in flight for longer is stuck, and said.
+func ReadBound() time.Duration { return CommandTimeout + WaitDelay + FanEvery }
+
+// cadence is how often a drive is read.
+func cadence(d Drive) time.Duration {
+	if d.Rotational {
+		return SpinningEvery
+	}
+	return SolidStateEvery
+}
+
+// FanWatch keeps which fans ran since the machine started.
+type FanWatch struct {
+	M        *Machine
+	readings Readings
+}
+
+// NewFanWatch starts from what an earlier fan watch of this boot left.
+func NewFanWatch(m *Machine) *FanWatch {
+	w := &FanWatch{M: m, readings: Readings{FansRan: map[string]time.Time{}}}
+	if r, why := m.ReadReadings(FansPath); why == "" && r != nil && r.FansRan != nil {
+		w.readings.FansRan = r.FansRan
+	}
+	w.readings.BootID = m.bootID()
+	return w
+}
+
+// Look notes the fans turning now, and writes the file.
+func (w *FanWatch) Look() error {
+	now := w.M.Now()
+	for _, f := range w.M.Fans() {
+		if f.RPM != nil && *f.RPM > 0 {
+			w.readings.FansRan[f.Key] = now
+		}
+	}
+	w.readings.WrittenAt = now
+	return w.M.WriteReadings(FansPath, w.readings)
+}
+
+// DriveReader keeps each drive's newest reading.
+type DriveReader struct {
 	M *Machine
-	// Async reads the drives beside the fans instead of before them; Run sets it.
+	// Async reads each due drive in a goroutine of its own instead of before writing; Run sets it.
 	Async bool
 
 	mu       sync.Mutex
 	readings Readings
-	reading  bool
 	done     sync.WaitGroup
 }
 
-// NewWatch starts from what an earlier watch of this boot left, so a restart neither forgets the fans
-// that ran nor reads every drive again at once.
-func NewWatch(m *Machine) *Watch {
-	w := &Watch{M: m, readings: Readings{Drives: map[string]DriveReading{}, FansRan: map[string]time.Time{}}}
-	if r, why := m.ReadReadings(); why == "" && r != nil {
-		w.readings = *r
-		if w.readings.Drives == nil {
-			w.readings.Drives = map[string]DriveReading{}
-		}
-		if w.readings.FansRan == nil {
-			w.readings.FansRan = map[string]time.Time{}
+// NewDriveReader starts from what an earlier reader of this boot left, so a restart does not read every
+// drive again at once. A read that was in flight when it stopped is not.
+func NewDriveReader(m *Machine) *DriveReader {
+	w := &DriveReader{M: m, readings: Readings{Drives: map[string]DriveReading{}}}
+	if r, why := m.ReadReadings(DrivesPath); why == "" && r != nil && r.Drives != nil {
+		w.readings.Drives = r.Drives
+		for name, d := range w.readings.Drives {
+			d.InFlightSince = nil
+			w.readings.Drives[name] = d
 		}
 	}
 	w.readings.BootID = m.bootID()
 	return w
 }
 
-// due lists the drives whose next look has come: never looked at, or looked at longer ago than their
-// cadence.
-func (w *Watch) due(now time.Time) (due []Drive, present map[string]bool) {
-	present = map[string]bool{}
+// Look starts a read of every drive that is due and not already being read, forgets drives that are gone,
+// and writes the file.
+func (w *DriveReader) Look() error {
+	now := w.M.Now()
+	var due []Drive
+	present := map[string]bool{}
+	w.mu.Lock()
 	for _, d := range w.M.BlockDrives() {
 		present[d.Device] = true
-		every := SolidStateEvery
-		if d.Rotational {
-			every = SpinningEvery
-		}
 		last, ok := w.readings.Drives[d.Device]
-		looked := last.LookedAt
-		if looked.IsZero() {
-			looked = last.ReadAt
+		if ok && last.InFlightSince != nil {
+			continue
 		}
-		if !ok || now.Sub(looked) >= every {
+		if !ok || now.Sub(last.LookedAt) >= cadence(d) {
+			last.InFlightSince = &now
+			w.readings.Drives[d.Device] = last
 			due = append(due, d)
 		}
 	}
-	return due, present
-}
-
-// Look is one look: the fans always, the drives that are due. It writes the file.
-func (w *Watch) Look() error {
-	now := w.M.Now()
-	fans := w.M.Fans()
-	w.mu.Lock()
-	for _, f := range fans {
-		if f.RPM != nil && *f.RPM > 0 {
-			w.readings.FansRan[f.Key] = now
-		}
-	}
-	due, present := w.due(now)
-	for name := range w.readings.Drives {
-		if !present[name] {
+	for name, d := range w.readings.Drives {
+		if !present[name] && d.InFlightSince == nil {
 			delete(w.readings.Drives, name)
 		}
 	}
-	start := len(due) > 0 && !w.reading
-	if start {
-		w.reading = true
-	}
 	w.mu.Unlock()
-	if start {
+	for _, d := range due {
 		w.done.Add(1)
 		if w.Async {
-			go w.readDrives(due)
+			go w.read(d)
 		} else {
-			w.readDrives(due)
+			w.read(d)
 		}
 	}
 	return w.write()
 }
 
-// readDrives reads each due drive, one after another, and keeps each reading as it comes.
-func (w *Watch) readDrives(due []Drive) {
+// read reads one drive and keeps its reading.
+func (w *DriveReader) read(d Drive) {
 	defer w.done.Done()
-	for _, d := range due {
-		w.mu.Lock()
-		var before *DriveReading
-		if b, ok := w.readings.Drives[d.Device]; ok && len(b.Smartctl) > 0 {
-			before = &b
-		}
-		w.mu.Unlock()
-		got := w.M.ReadDrive(d.Device, before)
-		w.mu.Lock()
-		w.readings.Drives[d.Device] = got
-		w.mu.Unlock()
-	}
 	w.mu.Lock()
-	w.reading = false
+	var before *DriveReading
+	if b, ok := w.readings.Drives[d.Device]; ok && (len(b.Smartctl) > 0 || b.Asleep) {
+		b.InFlightSince = nil
+		before = &b
+	}
+	w.mu.Unlock()
+	got := w.M.ReadDrive(d, before)
+	got.InFlightSince = nil
+	w.mu.Lock()
+	w.readings.Drives[d.Device] = got
 	w.mu.Unlock()
 }
 
-// WaitDrives waits for a pass over the drives that is under way.
-func (w *Watch) WaitDrives() { w.done.Wait() }
+// WaitReads waits for the reads under way.
+func (w *DriveReader) WaitReads() { w.done.Wait() }
 
-func (w *Watch) write() error {
+func (w *DriveReader) write() error {
 	w.mu.Lock()
 	defer w.mu.Unlock()
 	w.readings.WrittenAt = w.M.Now()
-	return w.M.WriteReadings(w.readings)
+	return w.M.WriteReadings(DrivesPath, w.readings)
 }
 
-// Run looks at once and every FanEvery until ctx ends, the drives beside the fans. A look that cannot
-// write is said on standard error, which the node-engine keeps in the unit's journal, and the watch goes
-// on.
-func (w *Watch) Run(ctx context.Context) {
-	w.Async = true
+// every runs look at once and every FanEvery until ctx ends. A look that cannot write is said on standard
+// error, which the node-engine keeps in the unit's journal, and the process goes on.
+func every(ctx context.Context, name string, look func() error) {
 	t := time.NewTicker(FanEvery)
 	defer t.Stop()
 	for {
-		if err := w.Look(); err != nil {
-			fmt.Fprintln(os.Stderr, "sensors watch: the readings were not written:", err)
+		if err := look(); err != nil {
+			fmt.Fprintf(os.Stderr, "sensors %s: the readings were not written: %v\n", name, err)
 		}
 		select {
 		case <-ctx.Done():

modules/sensors/module.json

@@ -20,34 +20,13 @@
       "package": "smartmontools"
     },
     {
-      "id": "smartd-conf",
-      "type": "file",
-      "path": "/etc/smartd.conf",
-      "mode": "0644",
-      "content": "# The mesh's (module sensors, novox/hq ADR 0256): what smartd watches. Written whole at every push;\n# an edit here is overwritten.\n#\n# Every drive smartd can find, with every check (-a). A drive in standby is skipped, never woken to be\n# checked, and the skip is not logged (-n standby,q): a server's spinning drives may sleep. A drive\n# behind a USB bridge that cannot say whether it is asleep is checked anyway by smartd, and may be woken\n# (novox/hq ADR 0256); the watch does not read such a drive. No mail is\n# sent (no -m): what smartd finds goes to the journal, and the module's watch reads each drive's health\n# for the mesh's tools and its health check.\nDEVICESCAN -a -n standby,q\n"
-    },
-    {
-      "id": "smartd-drop-ins",
-      "type": "directory",
-      "path": "/etc/systemd/system/smartd.service.d",
-      "mode": "0755"
-    },
-    {
-      "id": "smartd-drop-in",
-      "type": "file",
-      "path": "/etc/systemd/system/smartd.service.d/10-mesh.conf",
-      "mode": "0644",
-      "content": "# The mesh's (module sensors, novox/hq ADR 0256). Written whole at every push; edit nothing here.\n#\n# smartd runs on every machine the module is assigned to, a virtual machine included, where the package's\n# unit would not start; and it stays up on a machine with no drive it can watch (-q never), where it\n# would otherwise exit, because the mesh judges this service as one that stays up. It checks each drive\n# hourly (-i 3600), not every half hour: a drive read more often than its standby timer never spins down,\n# and the module's watch reads a spinning drive hourly too.\n[Unit]\nConditionVirtualization=\n\n[Service]\nExecStart=\nExecStart=/usr/bin/smartd -n -q never -i 3600\n"
-    },
-    {
-      "id": "smartd",
-      "type": "service",
-      "unit": "smartd.service",
-      "state": "running",
-      "boot": "enabled",
-      "restart-on": [
-        "smartd-conf",
-        "smartd-drop-in"
+      "id": "drives",
+      "type": "process",
+      "name": "sensors-drives",
+      "artifact": "tools",
+      "run": [
+        "./sensors",
+        "drives"
       ],
       "health": {
         "kind": "unit"