From f63828ce1cc442db60a1ddbe4f41f3578cecddb3 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 14:35:21 +0200 Subject: [PATCH] ADR 0256: say exactly what raises a condition, and why a drive is read only hourly Rule 4 now matches the module after review: the NVMe temperature bit, old-age attributes, a fan's minimum or alarm and a board chip's limits raise nothing; a stale watch does. The standby caveat is recorded: no mesh tool sees a drive's spin-down timer, so spinning drives are read hourly. --- ...-and-a-fault-is-said-through-its-health.md | 60 ++++++++++++++----- .../48-a-module-says-how-it-is-healthy.md | 21 ++++--- 2 files changed, 56 insertions(+), 25 deletions(-) diff --git a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md index ece2e37c..fc7e7ef5 100644 --- a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md +++ b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md @@ -89,9 +89,9 @@ as not read, never as cool. Nothing here conflicts with GENESIS. - **the drive daemon (`smartd`)**, judged by the unit's own readiness, which the module does not run itself. It watches every drive between looks and writes what it finds to the journal when it happens; - **the module's watch**, a process of its own bundle run as root, judged by the module's - `sensors_health` tool. The watch reads each drive's health every ten minutes, so the tool runner needs - no root. It skips a drive that is asleep, so a spinning drive is never woken to be read. It also - records which fans have run since the machine started. + `sensors_health` tool. The watch reads each drive's health (a spinning drive hourly, any other every + ten minutes), so the tool runner needs no root. It skips a drive that is asleep, so a spinning drive + is never woken to be read. It also records which fans have run since the machine started. ## Decision @@ -108,19 +108,32 @@ as not read, never as cool. Nothing here conflicts with GENESIS. 2. **Every flag is measured against the hardware's own threshold.** A temperature is HIGH at its own maximum (or the zone's `hot` or `passive` trip). It is CRITICAL at its own critical value (or the zone's `critical` trip). The mesh adds no threshold of its own. A sensor that reports none is never - flagged. + flagged. 0 °C is a reading; 127 and 127.5 °C are not, because they are what a board chip's input with + nothing on it answers. 3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR 0246 when a caller has to reach it by role. 4. **It declares its health under ADR 0240.** `smartd` is judged by its unit. The watch is judged by - `sensors_health`, which is unhealthy when: - - a temperature is at or above its own critical threshold; - - a drive says it is failing: SMART's overall verdict, an attribute failing now, or an NVMe critical - warning bit; - - a fan that ran since the machine started reads 0 while a temperature is at or above its own maximum; - - memory reports an uncorrectable error. + `sensors_health`, which is unhealthy when, and only when: + - a CPU, GPU, drive or thermal-zone temperature is at or above its own critical value. A board chip's + (a Super I/O's) critical value is shown as CRITICAL by the tools and never makes the module + unhealthy, because its inputs and limits are whatever the firmware left; + - a drive says it is failing: a SATA drive's overall SMART verdict, a pre-failure attribute failing + now, or an NVMe critical warning bit other than temperature (spare below threshold, reliability + degraded, read-only, backup failed). An NVMe drive is judged by its bits, not by the overall verdict + smartctl derives from them, and by its health log when smartctl gives no verdict at all. The + temperature bit alone, an old-age attribute failing now (a temperature, a count of hours) and an + attribute that failed in the past are HIGH; + - a fan that ran since the machine started reads 0 RPM now while the CPU, the board, or a sensor on + the fan's own chip is at or above its own maximum. A hot drive or network adapter says nothing of a + fan. A fan under its chip's minimum, or with its chip's alarm bit set, is said in a note and raises + nothing: a zero-fan GPU is under its minimum by design, and a board chip latches its alarm; + - memory reports an uncorrectable error; + - the watch's readings are not to be trusted: none since the machine started, or none written for five + minutes. The watch writes them every 30 seconds and reads the drives beside that, each `smartctl` + bounded in time, so old readings mean the watch is hung or gone, never that a drive is slow. - What the tool answers as `why` is one plain sentence for the operator, such as "The drive nvme0 - (Samsung 990 PRO) says its spare capacity is below its threshold." + What the tool answers as `why` is one plain sentence per finding, for the operator, such as "The drive + nvme0n1 (Samsung SSD 990 PRO 2TB) says it is failing: its spare capacity is below its threshold." 5. **A vendor module keeps only what is its vendor's.** For example, the laptop's fan curves and performance profiles. Its reading of the generic sensors moves to this module when that module is next changed. @@ -138,8 +151,21 @@ as not read, never as cool. Nothing here conflicts with GENESIS. that machine is the build's first machine, the build reaches no other machine. Option 1 of the third question removes this. Until it is built, the laptop is the right first machine for this module. - The module installs a package (`smartmontools`) and runs a root daemon on every machine it is assigned - to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail. - It also keeps running on a machine with no drive it can watch, such as a virtual machine. + to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail, + checked hourly. It also keeps running on a machine with no drive it can watch, such as a virtual + machine. +- **Reading a drive may keep it from spinning down.** Neither the watch nor `smartd` wakes a drive in + standby. But reading SMART from an awake drive can restart its standby timer, so a drive read more often + than its timer is set for never reaches standby. Whether the servers' drives are set to spin down is not + known: no mesh tool reads a drive's standby timer or power mode, and no module sets one. On 2026-10-08, + `disk-load` showed the anchor's two drives written to continuously (they hold its root, so they never + idle) and the home server's twelve spinning drives without I/O over a ten-second sample, which says + nothing of their timers. So a spinning drive is read hourly, by the watch and by `smartd` (`-i 3600`). A + drive whose timer is set longer than an hour can still be kept awake by these reads. Reading the power + mode and the timer is a follow-up for this module's tools. +- **A drive behind a USB bridge** that does not pass the power-mode question on is not read by the watch + (smartctl is asked to stop, not to read on), and is said as not read. `smartd` has no such per-drive + stop, and may wake it. - The watch runs as root. It writes one file of readings under the runtime directory, and nothing else. ## Follow-ups @@ -148,6 +174,8 @@ as not read, never as cool. Nothing here conflicts with GENESIS. condition. Then `sensors` keeps its tools and drops the tool check from its health. - **The module-health wording takes a tool check's `why`** as its explanation (ADR 0253 rule 2). - **Temperature blocks for the bar**, once the generic contract for a module's bar blocks exists. +- **A drive's power mode and standby timer in `sensors_drives`**, so the cadence above can be set from + what each drive is set to, rather than from a guess that it may be an hour. - **The vendor module's `thermals` tool** points at this module. ## References @@ -166,7 +194,7 @@ as not read, never as cool. Nothing here conflicts with GENESIS. | Rule | Checked by | |---|---| | 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only | -| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged | +| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged; 0 °C is a reading and 127 °C is not | | 3 No seat | the manifest test that `sensors` claims nothing | -| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case, and that its `why` is one sentence; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | +| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing reading is unhealthy, that a hung `smartctl` is bounded and the fans are still written, and that its `why` is one sentence per finding; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | | 5 Vendor extras only | review, when the vendor module is next changed | diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 3998ae0f..44cddad6 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -283,22 +283,25 @@ health of this design, not yet through the machine's own judgement as §10 does ``` sensors on one machine node-engine controller - ┌─────────────────────────────────────────────┐ + ┌──────────────────────────────────────────────┐ │ smartd (unit) watches every drive, │ unit ──► liveness + readiness │ never wakes one asleep │ - │ watch (root) every 10 min each │ - │ drive's health; fans │ tool ──► asks sensors_health ──► module.sensors..unhealthy + │ watch (root) each drive's health │ + │ (spinning: hourly); fans │ tool ──► asks sensors_health ──► module.sensors..unhealthy │ that ran since boot │ every 60 s, 2 looks (the why is its evidence) │ tools (operator) /sys and the watch's │ │ readings; read-only │ - └─────────────────────────────────────────────┘ + └──────────────────────────────────────────────┘ ``` -- **Unhealthy means the hardware is failing, measured against its own thresholds:** a temperature at or - above its own critical value, a drive that says it is failing (SMART's verdict, an attribute failing now, - an NVMe critical warning bit), a fan that ran since boot and reads 0 while a temperature is at or above its - own maximum, or memory that reports an uncorrectable error. HIGH readings are said by the tools and raise - nothing. +- **Unhealthy means the hardware is failing, measured against its own thresholds:** a CPU, GPU, drive or + zone temperature at or above its own critical value (a board chip's is shown, never judged); a drive that + says it is failing (a SATA drive's verdict, a pre-failure attribute failing now, an NVMe critical warning + bit other than temperature); a fan that ran since boot and reads 0 while the CPU, the board or a sensor on + its own chip is at or above its own maximum; memory that reports an uncorrectable error; or the watch's + readings older than five minutes. HIGH readings and fan notes are said by the tools and raise nothing. +- **A drive is never woken, and is read seldom enough to sleep:** a spinning drive hourly, by the watch and + by smartd, because a read can restart a standby timer the mesh cannot yet see. - **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no SMART drive is healthy. - **The cost of using module health for the machine's fault** is that the condition names the module, and a