ADR 0256: one reader of the drives, and a stuck read is said
After the second review: smartd is dropped as a second reader that could wake or keep awake the drives, a stuck read or a drive not looked at is unhealthy, a drive asleep for a day is HIGH and never unhealthy, and a board chip's maximum says nothing of a fan. The never-woken and hourly claims now say exactly what holds.
This commit is contained in:
+52
-34
@@ -85,13 +85,19 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
|
||||
2. *An event, as `memory-pressure` emits `pressure.high`.* Rejected: an event is a sample, not a state
|
||||
(ADR 0240's own reason), and no event raises a condition.
|
||||
3. **ADR 0240's health, as it stands.** Chosen, because it raises a condition today with no core change.
|
||||
The module gets the two long-running resources that rule 2 needs, and each has a real job:
|
||||
- **the drive daemon (`smartd`)**, judged by the unit's own readiness, which the module does not run
|
||||
itself. It watches every drive between looks and writes what it finds to the journal when it happens;
|
||||
- **the module's watch**, a process of its own bundle run as root, judged by the module's
|
||||
`sensors_health` tool. The watch reads each drive's health (a spinning drive hourly, any other every
|
||||
ten minutes), so the tool runner needs no root. It skips a drive that is asleep, so a spinning drive
|
||||
is never woken to be read. It also records which fans have run since the machine started.
|
||||
Rule 2 needs a check of another kind beside the tool check, so the module runs two long-running
|
||||
processes of its own bundle, each with a job of its own and its own file:
|
||||
- **the drive reader**, run as root and judged by the unit's own readiness, which the module does not
|
||||
run itself. It reads each drive's health, so the tool runner needs no root. A drive read that never
|
||||
ends (a drive or a controller that stopped answering) cannot stop the fans being looked at;
|
||||
- **the fan watch**, judged by the module's `sensors_health` tool. It records which fans have run
|
||||
since the machine started.
|
||||
|
||||
*Running `smartd` beside it* was the first choice and is rejected. `smartd` reads every drive when it
|
||||
starts or restarts, which can spin a sleeping drive up. Its `-n standby` applies to ATA drives only, so
|
||||
it polls a SAS drive in standby anyway. And as a second reader on its own schedule, it reads each drive
|
||||
at another phase of the hour than the module does, which shortens the idle stretch a drive needs to
|
||||
spin down. One reader, the module's, avoids all three.
|
||||
|
||||
## Decision
|
||||
|
||||
@@ -112,8 +118,8 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
|
||||
nothing on it answers.
|
||||
3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR
|
||||
0246 when a caller has to reach it by role.
|
||||
4. **It declares its health under ADR 0240.** `smartd` is judged by its unit. The watch is judged by
|
||||
`sensors_health`, which is unhealthy when, and only when:
|
||||
4. **It declares its health under ADR 0240.** The drive reader is judged by its unit. The fan watch is
|
||||
judged by `sensors_health`, which is unhealthy when, and only when:
|
||||
- a CPU, GPU, drive or thermal-zone temperature is at or above its own critical value. A board chip's
|
||||
(a Super I/O's) critical value is shown as CRITICAL by the tools and never makes the module
|
||||
unhealthy, because its inputs and limits are whatever the firmware left;
|
||||
@@ -123,14 +129,23 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
|
||||
smartctl derives from them, and by its health log when smartctl gives no verdict at all. The
|
||||
temperature bit alone, an old-age attribute failing now (a temperature, a count of hours) and an
|
||||
attribute that failed in the past are HIGH;
|
||||
- a fan that ran since the machine started reads 0 RPM now while the CPU, the board, or a sensor on
|
||||
the fan's own chip is at or above its own maximum. A hot drive or network adapter says nothing of a
|
||||
fan. A fan under its chip's minimum, or with its chip's alarm bit set, is said in a note and raises
|
||||
nothing: a zero-fan GPU is under its minimum by design, and a board chip latches its alarm;
|
||||
- a fan that ran since the machine started reads 0 RPM now while a CPU sensor, a thermal zone, or a
|
||||
sensor on the fan's own chip is at or above its own maximum, the last only when that chip is not a
|
||||
board chip. A board chip's maximum is no more trusted than its critical value. A hot drive or network
|
||||
adapter says nothing of a fan. A fan under its chip's minimum, or with its chip's alarm bit set, is
|
||||
said in a note and raises nothing: a zero-fan GPU is under its minimum by design, and a board chip
|
||||
latches its alarm;
|
||||
- memory reports an uncorrectable error;
|
||||
- the watch's readings are not to be trusted: none since the machine started, or none written for five
|
||||
minutes. The watch writes them every 30 seconds and reads the drives beside that, each `smartctl`
|
||||
bounded in time, so old readings mean the watch is hung or gone, never that a drive is slow.
|
||||
- the module's own processes cannot be trusted now: either process's file is missing, from an earlier
|
||||
boot, or not written for five minutes (each writes every 30 seconds); a drive's read has been in
|
||||
flight longer than any read can take (smartctl's 20 s bound, 2 s for its output, and one write of
|
||||
slack), which is a read stuck in the kernel, since such a read cannot be ended; or a drive has not
|
||||
been looked at for twice its cadence.
|
||||
|
||||
A drive found asleep at every look is said as asleep since when, and HIGH once that is a day, because
|
||||
its health has not been read for that long and a drive that stopped answering can look asleep. It
|
||||
never makes the module unhealthy on its own, at any length: a drive kept for backups may sleep for
|
||||
weeks, and its health being unread is said, not raised.
|
||||
|
||||
What the tool answers as `why` is one plain sentence per finding, for the operator, such as "The drive
|
||||
nvme0n1 (Samsung SSD 990 PRO 2TB) says it is failing: its spare capacity is below its threshold."
|
||||
@@ -150,23 +165,26 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
|
||||
with a failing drive is not healthy within the gate's bound, and is put back there (ADR 0236 §2–3). If
|
||||
that machine is the build's first machine, the build reaches no other machine. Option 1 of the third
|
||||
question removes this. Until it is built, the laptop is the right first machine for this module.
|
||||
- The module installs a package (`smartmontools`) and runs a root daemon on every machine it is assigned
|
||||
to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail,
|
||||
checked hourly. It also keeps running on a machine with no drive it can watch, such as a virtual
|
||||
machine.
|
||||
- **Reading a drive may keep it from spinning down.** Neither the watch nor `smartd` wakes a drive in
|
||||
standby. But reading SMART from an awake drive can restart its standby timer, so a drive read more often
|
||||
than its timer is set for never reaches standby. Whether the servers' drives are set to spin down is not
|
||||
known: no mesh tool reads a drive's standby timer or power mode, and no module sets one. On 2026-10-08,
|
||||
`disk-load` showed the anchor's two drives written to continuously (they hold its root, so they never
|
||||
idle) and the home server's twelve spinning drives without I/O over a ten-second sample, which says
|
||||
nothing of their timers. So a spinning drive is read hourly, by the watch and by `smartd` (`-i 3600`). A
|
||||
drive whose timer is set longer than an hour can still be kept awake by these reads. Reading the power
|
||||
mode and the timer is a follow-up for this module's tools.
|
||||
- **A drive behind a USB bridge** that does not pass the power-mode question on is not read by the watch
|
||||
(smartctl is asked to stop, not to read on), and is said as not read. `smartd` has no such per-drive
|
||||
stop, and may wake it.
|
||||
- The watch runs as root. It writes one file of readings under the runtime directory, and nothing else.
|
||||
- The module installs a package (`smartmontools`) for `smartctl`, and runs two processes of its own as
|
||||
root, each writing one file under the runtime directory and nothing else. It does not run `smartd`.
|
||||
- **A drive is not woken by the module, as far as the drive can say so.** Each read is asked with
|
||||
`-n standby`, and smartctl stops without reading when the drive says it is in standby or asleep (an ATA
|
||||
drive by its power mode, a SAS drive or one the operating system powered down by smartctl's own words).
|
||||
smartctl is told the drive's kind (`-d nvme`, `sat` or `scsi`) where it is known, so it does not probe a
|
||||
drive to find out. A drive that cannot say whether it is asleep (a USB bridge that does not pass the
|
||||
question on) is not read at all, and is said as not read. What remains: a drive of a kind the module
|
||||
cannot tell is read with smartctl's own detection.
|
||||
- **Reading a drive may keep it from spinning down.** Reading SMART from an awake drive can restart its
|
||||
standby timer, so a drive read more often than its timer is set for never reaches standby. Whether the
|
||||
servers' drives are set to spin down is not known: no mesh tool reads a drive's standby timer or power
|
||||
mode, and no module sets one. On 2026-10-08, `disk-load` showed the anchor's two drives written to
|
||||
continuously (they hold its root, so they never idle) and the home server's twelve spinning drives
|
||||
without I/O over a ten-second sample, which says nothing of their timers. So the module, the only
|
||||
reader, reads a spinning drive once an hour, timed from its last look. A drive whose timer is set
|
||||
longer than an hour can still be kept awake. Reading the power mode and the timer is a follow-up for
|
||||
this module's tools.
|
||||
- **A drive that stops answering may look asleep**, and is then HIGH after a day, never unhealthy (rule
|
||||
4). A drive whose read hangs instead is unhealthy within a minute of its bound.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
@@ -196,5 +214,5 @@ as not read, never as cool. Nothing here conflicts with GENESIS.
|
||||
| 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only |
|
||||
| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged; 0 °C is a reading and 127 °C is not |
|
||||
| 3 No seat | the manifest test that `sensors` claims nothing |
|
||||
| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing reading is unhealthy, that a hung `smartctl` is bounded and the fans are still written, and that its `why` is one sentence per finding; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has |
|
||||
| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing file is unhealthy, that a hung `smartctl` is bounded, that a read stuck past its bound is unhealthy and does not stop the other drives, that a drive not looked at for twice its cadence is unhealthy, that a drive asleep for a day is HIGH and healthy, that a SAS drive in standby is asleep, that smartctl is told each drive's kind, and that its `why` is one sentence per finding; the manifest test that no `smartd` runs; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has |
|
||||
| 5 Vendor extras only | review, when the vendor module is next changed |
|
||||
|
||||
@@ -284,24 +284,27 @@ health of this design, not yet through the machine's own judgement as §10 does
|
||||
```
|
||||
sensors on one machine node-engine controller
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ smartd (unit) watches every drive, │ unit ──► liveness + readiness
|
||||
│ never wakes one asleep │
|
||||
│ watch (root) each drive's health │
|
||||
│ (spinning: hourly); fans │ tool ──► asks sensors_health ──► module.sensors.<m>.unhealthy
|
||||
│ that ran since boot │ every 60 s, 2 looks (the why is its evidence)
|
||||
│ tools (operator) /sys and the watch's │
|
||||
│ readings; read-only │
|
||||
│ drive reader (root) each drive's health, │ unit ──► liveness + readiness
|
||||
│ never one asleep; each │
|
||||
│ read bounded, or said │
|
||||
│ fan watch fans that ran since boot │ tool ──► asks sensors_health ──► module.sensors.<m>.unhealthy
|
||||
│ │ every 60 s, 2 looks (the why is its evidence)
|
||||
│ tools (operator) /sys and the two files; │
|
||||
│ read-only │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- **Unhealthy means the hardware is failing, measured against its own thresholds:** a CPU, GPU, drive or
|
||||
zone temperature at or above its own critical value (a board chip's is shown, never judged); a drive that
|
||||
says it is failing (a SATA drive's verdict, a pre-failure attribute failing now, an NVMe critical warning
|
||||
bit other than temperature); a fan that ran since boot and reads 0 while the CPU, the board or a sensor on
|
||||
its own chip is at or above its own maximum; memory that reports an uncorrectable error; or the watch's
|
||||
readings older than five minutes. HIGH readings and fan notes are said by the tools and raise nothing.
|
||||
- **A drive is never woken, and is read seldom enough to sleep:** a spinning drive hourly, by the watch and
|
||||
by smartd, because a read can restart a standby timer the mesh cannot yet see.
|
||||
bit other than temperature); a fan that ran since boot and reads 0 while a CPU sensor, a thermal zone or a
|
||||
sensor on its own non-board chip is at or above its own maximum; memory that reports an uncorrectable
|
||||
error. **Or that the module's own processes are stale:** a file five minutes old, a drive's read stuck
|
||||
past its bound, a drive not looked at for twice its cadence. HIGH readings, fan notes and a drive asleep
|
||||
for a day are said by the tools and raise nothing.
|
||||
- **One reader, never waking a drive that can say it sleeps:** the module's drive reader alone reads the
|
||||
drives (no `smartd`), a spinning drive hourly, because a read can restart a standby timer the mesh cannot
|
||||
yet see.
|
||||
- **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no
|
||||
SMART drive is healthy.
|
||||
- **The cost of using module health for the machine's fault** is that the condition names the module, and a
|
||||
@@ -320,7 +323,7 @@ health of this design, not yet through the machine's own judgement as §10 does
|
||||
| D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine |
|
||||
| E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses |
|
||||
| F — the machine's network | `mesh-host` (node-engine), `mesh-controller`, `mesh-catalog` | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile |
|
||||
| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, `smartd` and the watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy |
|
||||
| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, the drive reader and the fan watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy |
|
||||
|
||||
Phases A and B may be built together; A is live first, because it judges without a single declaration.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user