ADR 0256: one reader of the drives, and a stuck read is said
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request

After the second review: smartd is dropped as a second reader that could wake or keep awake the drives,
a stuck read or a drive not looked at is unhealthy, a drive asleep for a day is HIGH and never unhealthy,
and a board chip's maximum says nothing of a fan. The never-woken and hourly claims now say exactly what
holds.
This commit is contained in:
jochen
2026-10-08 14:59:12 +02:00
parent f63828ce1c
commit 0406373545
2 changed files with 68 additions and 47 deletions
@@ -85,13 +85,19 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
2. *An event, as `memory-pressure` emits `pressure.high`.* Rejected: an event is a sample, not a state
(ADR 0240's own reason), and no event raises a condition.
3. **ADR 0240's health, as it stands.** Chosen, because it raises a condition today with no core change.
The module gets the two long-running resources that rule 2 needs, and each has a real job:
- **the drive daemon (`smartd`)**, judged by the unit's own readiness, which the module does not run
itself. It watches every drive between looks and writes what it finds to the journal when it happens;
- **the module's watch**, a process of its own bundle run as root, judged by the module's
`sensors_health` tool. The watch reads each drive's health (a spinning drive hourly, any other every
ten minutes), so the tool runner needs no root. It skips a drive that is asleep, so a spinning drive
is never woken to be read. It also records which fans have run since the machine started.
Rule 2 needs a check of another kind beside the tool check, so the module runs two long-running
processes of its own bundle, each with a job of its own and its own file:
- **the drive reader**, run as root and judged by the unit's own readiness, which the module does not
run itself. It reads each drive's health, so the tool runner needs no root. A drive read that never
ends (a drive or a controller that stopped answering) cannot stop the fans being looked at;
- **the fan watch**, judged by the module's `sensors_health` tool. It records which fans have run
since the machine started.
*Running `smartd` beside it* was the first choice and is rejected. `smartd` reads every drive when it
starts or restarts, which can spin a sleeping drive up. Its `-n standby` applies to ATA drives only, so
it polls a SAS drive in standby anyway. And as a second reader on its own schedule, it reads each drive
at another phase of the hour than the module does, which shortens the idle stretch a drive needs to
spin down. One reader, the module's, avoids all three.
## Decision
@@ -112,8 +118,8 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
nothing on it answers.
3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR
0246 when a caller has to reach it by role.
4. **It declares its health under ADR 0240.** `smartd` is judged by its unit. The watch is judged by
`sensors_health`, which is unhealthy when, and only when:
4. **It declares its health under ADR 0240.** The drive reader is judged by its unit. The fan watch is
judged by `sensors_health`, which is unhealthy when, and only when:
- a CPU, GPU, drive or thermal-zone temperature is at or above its own critical value. A board chip's
(a Super I/O's) critical value is shown as CRITICAL by the tools and never makes the module
unhealthy, because its inputs and limits are whatever the firmware left;
@@ -123,14 +129,23 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
smartctl derives from them, and by its health log when smartctl gives no verdict at all. The
temperature bit alone, an old-age attribute failing now (a temperature, a count of hours) and an
attribute that failed in the past are HIGH;
- a fan that ran since the machine started reads 0 RPM now while the CPU, the board, or a sensor on
the fan's own chip is at or above its own maximum. A hot drive or network adapter says nothing of a
fan. A fan under its chip's minimum, or with its chip's alarm bit set, is said in a note and raises
nothing: a zero-fan GPU is under its minimum by design, and a board chip latches its alarm;
- a fan that ran since the machine started reads 0 RPM now while a CPU sensor, a thermal zone, or a
sensor on the fan's own chip is at or above its own maximum, the last only when that chip is not a
board chip. A board chip's maximum is no more trusted than its critical value. A hot drive or network
adapter says nothing of a fan. A fan under its chip's minimum, or with its chip's alarm bit set, is
said in a note and raises nothing: a zero-fan GPU is under its minimum by design, and a board chip
latches its alarm;
- memory reports an uncorrectable error;
- the watch's readings are not to be trusted: none since the machine started, or none written for five
minutes. The watch writes them every 30 seconds and reads the drives beside that, each `smartctl`
bounded in time, so old readings mean the watch is hung or gone, never that a drive is slow.
- the module's own processes cannot be trusted now: either process's file is missing, from an earlier
boot, or not written for five minutes (each writes every 30 seconds); a drive's read has been in
flight longer than any read can take (smartctl's 20 s bound, 2 s for its output, and one write of
slack), which is a read stuck in the kernel, since such a read cannot be ended; or a drive has not
been looked at for twice its cadence.
A drive found asleep at every look is said as asleep since when, and HIGH once that is a day, because
its health has not been read for that long and a drive that stopped answering can look asleep. It
never makes the module unhealthy on its own, at any length: a drive kept for backups may sleep for
weeks, and its health being unread is said, not raised.
What the tool answers as `why` is one plain sentence per finding, for the operator, such as "The drive
nvme0n1 (Samsung SSD 990 PRO 2TB) says it is failing: its spare capacity is below its threshold."
@@ -150,23 +165,26 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
with a failing drive is not healthy within the gate's bound, and is put back there (ADR 0236 §2–3). If
that machine is the build's first machine, the build reaches no other machine. Option 1 of the third
question removes this. Until it is built, the laptop is the right first machine for this module.
- The module installs a package (`smartmontools`) and runs a root daemon on every machine it is assigned
to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail,
checked hourly. It also keeps running on a machine with no drive it can watch, such as a virtual
machine.
- **Reading a drive may keep it from spinning down.** Neither the watch nor `smartd` wakes a drive in
standby. But reading SMART from an awake drive can restart its standby timer, so a drive read more often
than its timer is set for never reaches standby. Whether the servers' drives are set to spin down is not
known: no mesh tool reads a drive's standby timer or power mode, and no module sets one. On 2026-10-08,
`disk-load` showed the anchor's two drives written to continuously (they hold its root, so they never
idle) and the home server's twelve spinning drives without I/O over a ten-second sample, which says
nothing of their timers. So a spinning drive is read hourly, by the watch and by `smartd` (`-i 3600`). A
drive whose timer is set longer than an hour can still be kept awake by these reads. Reading the power
mode and the timer is a follow-up for this module's tools.
- **A drive behind a USB bridge** that does not pass the power-mode question on is not read by the watch
(smartctl is asked to stop, not to read on), and is said as not read. `smartd` has no such per-drive
stop, and may wake it.
- The watch runs as root. It writes one file of readings under the runtime directory, and nothing else.
- The module installs a package (`smartmontools`) for `smartctl`, and runs two processes of its own as
root, each writing one file under the runtime directory and nothing else. It does not run `smartd`.
- **A drive is not woken by the module, as far as the drive can say so.** Each read is asked with
`-n standby`, and smartctl stops without reading when the drive says it is in standby or asleep (an ATA
drive by its power mode, a SAS drive or one the operating system powered down by smartctl's own words).
smartctl is told the drive's kind (`-d nvme`, `sat` or `scsi`) where it is known, so it does not probe a
drive to find out. A drive that cannot say whether it is asleep (a USB bridge that does not pass the
question on) is not read at all, and is said as not read. What remains: a drive of a kind the module
cannot tell is read with smartctl's own detection.
- **Reading a drive may keep it from spinning down.** Reading SMART from an awake drive can restart its
standby timer, so a drive read more often than its timer is set for never reaches standby. Whether the
servers' drives are set to spin down is not known: no mesh tool reads a drive's standby timer or power
mode, and no module sets one. On 2026-10-08, `disk-load` showed the anchor's two drives written to
continuously (they hold its root, so they never idle) and the home server's twelve spinning drives
without I/O over a ten-second sample, which says nothing of their timers. So the module, the only
reader, reads a spinning drive once an hour, timed from its last look. A drive whose timer is set
longer than an hour can still be kept awake. Reading the power mode and the timer is a follow-up for
this module's tools.
- **A drive that stops answering may look asleep**, and is then HIGH after a day, never unhealthy (rule
4). A drive whose read hangs instead is unhealthy within a minute of its bound.
## Follow-ups
@@ -196,5 +214,5 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
| 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only |
| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged; 0 °C is a reading and 127 °C is not |
| 3 No seat | the manifest test that `sensors` claims nothing |
| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing reading is unhealthy, that a hung `smartctl` is bounded and the fans are still written, and that its `why` is one sentence per finding; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has |
| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing file is unhealthy, that a hung `smartctl` is bounded, that a read stuck past its bound is unhealthy and does not stop the other drives, that a drive not looked at for twice its cadence is unhealthy, that a drive asleep for a day is HIGH and healthy, that a SAS drive in standby is asleep, that smartctl is told each drive's kind, and that its `why` is one sentence per finding; the manifest test that no `smartd` runs; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has |
| 5 Vendor extras only | review, when the vendor module is next changed |
@@ -284,24 +284,27 @@ health of this design, not yet through the machine's own judgement as §10 does
```
sensors on one machine node-engine controller
┌──────────────────────────────────────────────┐
│ smartd (unit) watches every drive, │ unit ──► liveness + readiness
│ never wakes one asleep │
│ watch (root) each drive's health │
│ (spinning: hourly); fans │ tool ──► asks sensors_health ──► module.sensors.<m>.unhealthy
│ that ran since boot │ every 60 s, 2 looks (the why is its evidence)
│ tools (operator) /sys and the watch's │
│ readings; read-only │
│ drive reader (root) each drive's health, │ unit ──► liveness + readiness
│ never one asleep; each │
│ read bounded, or said │
│ fan watch fans that ran since boot │ tool ──► asks sensors_health ──► module.sensors.<m>.unhealthy
│ │ every 60 s, 2 looks (the why is its evidence)
│ tools (operator) /sys and the two files; │
│ read-only │
└──────────────────────────────────────────────┘
```
- **Unhealthy means the hardware is failing, measured against its own thresholds:** a CPU, GPU, drive or
zone temperature at or above its own critical value (a board chip's is shown, never judged); a drive that
says it is failing (a SATA drive's verdict, a pre-failure attribute failing now, an NVMe critical warning
bit other than temperature); a fan that ran since boot and reads 0 while the CPU, the board or a sensor on
its own chip is at or above its own maximum; memory that reports an uncorrectable error; or the watch's
readings older than five minutes. HIGH readings and fan notes are said by the tools and raise nothing.
- **A drive is never woken, and is read seldom enough to sleep:** a spinning drive hourly, by the watch and
by smartd, because a read can restart a standby timer the mesh cannot yet see.
bit other than temperature); a fan that ran since boot and reads 0 while a CPU sensor, a thermal zone or a
sensor on its own non-board chip is at or above its own maximum; memory that reports an uncorrectable
error. **Or that the module's own processes are stale:** a file five minutes old, a drive's read stuck
past its bound, a drive not looked at for twice its cadence. HIGH readings, fan notes and a drive asleep
for a day are said by the tools and raise nothing.
- **One reader, never waking a drive that can say it sleeps:** the module's drive reader alone reads the
drives (no `smartd`), a spinning drive hourly, because a read can restart a standby timer the mesh cannot
yet see.
- **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no
SMART drive is healthy.
- **The cost of using module health for the machine's fault** is that the condition names the module, and a
@@ -320,7 +323,7 @@ health of this design, not yet through the machine's own judgement as §10 does
| D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine |
| E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses |
| F — the machine's network | `mesh-host` (node-engine), `mesh-controller`, `mesh-catalog` | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile |
| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, `smartd` and the watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy |
| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, the drive reader and the fan watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy |
Phases A and B may be built together; A is live first, because it judges without a single declaration.