From 0406373545bbe4513d4a85cf4f30b3fd6209f3dc Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 14:59:12 +0200 Subject: [PATCH] ADR 0256: one reader of the drives, and a stuck read is said After the second review: smartd is dropped as a second reader that could wake or keep awake the drives, a stuck read or a drive not looked at is unhealthy, a drive asleep for a day is HIGH and never unhealthy, and a board chip's maximum says nothing of a fan. The never-woken and hourly claims now say exactly what holds. --- ...-and-a-fault-is-said-through-its-health.md | 86 +++++++++++-------- .../48-a-module-says-how-it-is-healthy.md | 29 ++++--- 2 files changed, 68 insertions(+), 47 deletions(-) diff --git a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md index fc7e7ef5..c386ba6e 100644 --- a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md +++ b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md @@ -85,13 +85,19 @@ as not read, never as cool. Nothing here conflicts with GENESIS. 2. *An event, as `memory-pressure` emits `pressure.high`.* Rejected: an event is a sample, not a state (ADR 0240's own reason), and no event raises a condition. 3. **ADR 0240's health, as it stands.** Chosen, because it raises a condition today with no core change. - The module gets the two long-running resources that rule 2 needs, and each has a real job: - - **the drive daemon (`smartd`)**, judged by the unit's own readiness, which the module does not run - itself. It watches every drive between looks and writes what it finds to the journal when it happens; - - **the module's watch**, a process of its own bundle run as root, judged by the module's - `sensors_health` tool. The watch reads each drive's health (a spinning drive hourly, any other every - ten minutes), so the tool runner needs no root. It skips a drive that is asleep, so a spinning drive - is never woken to be read. It also records which fans have run since the machine started. + Rule 2 needs a check of another kind beside the tool check, so the module runs two long-running + processes of its own bundle, each with a job of its own and its own file: + - **the drive reader**, run as root and judged by the unit's own readiness, which the module does not + run itself. It reads each drive's health, so the tool runner needs no root. A drive read that never + ends (a drive or a controller that stopped answering) cannot stop the fans being looked at; + - **the fan watch**, judged by the module's `sensors_health` tool. It records which fans have run + since the machine started. + + *Running `smartd` beside it* was the first choice and is rejected. `smartd` reads every drive when it + starts or restarts, which can spin a sleeping drive up. Its `-n standby` applies to ATA drives only, so + it polls a SAS drive in standby anyway. And as a second reader on its own schedule, it reads each drive + at another phase of the hour than the module does, which shortens the idle stretch a drive needs to + spin down. One reader, the module's, avoids all three. ## Decision @@ -112,8 +118,8 @@ as not read, never as cool. Nothing here conflicts with GENESIS. nothing on it answers. 3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR 0246 when a caller has to reach it by role. -4. **It declares its health under ADR 0240.** `smartd` is judged by its unit. The watch is judged by - `sensors_health`, which is unhealthy when, and only when: +4. **It declares its health under ADR 0240.** The drive reader is judged by its unit. The fan watch is + judged by `sensors_health`, which is unhealthy when, and only when: - a CPU, GPU, drive or thermal-zone temperature is at or above its own critical value. A board chip's (a Super I/O's) critical value is shown as CRITICAL by the tools and never makes the module unhealthy, because its inputs and limits are whatever the firmware left; @@ -123,14 +129,23 @@ as not read, never as cool. Nothing here conflicts with GENESIS. smartctl derives from them, and by its health log when smartctl gives no verdict at all. The temperature bit alone, an old-age attribute failing now (a temperature, a count of hours) and an attribute that failed in the past are HIGH; - - a fan that ran since the machine started reads 0 RPM now while the CPU, the board, or a sensor on - the fan's own chip is at or above its own maximum. A hot drive or network adapter says nothing of a - fan. A fan under its chip's minimum, or with its chip's alarm bit set, is said in a note and raises - nothing: a zero-fan GPU is under its minimum by design, and a board chip latches its alarm; + - a fan that ran since the machine started reads 0 RPM now while a CPU sensor, a thermal zone, or a + sensor on the fan's own chip is at or above its own maximum, the last only when that chip is not a + board chip. A board chip's maximum is no more trusted than its critical value. A hot drive or network + adapter says nothing of a fan. A fan under its chip's minimum, or with its chip's alarm bit set, is + said in a note and raises nothing: a zero-fan GPU is under its minimum by design, and a board chip + latches its alarm; - memory reports an uncorrectable error; - - the watch's readings are not to be trusted: none since the machine started, or none written for five - minutes. The watch writes them every 30 seconds and reads the drives beside that, each `smartctl` - bounded in time, so old readings mean the watch is hung or gone, never that a drive is slow. + - the module's own processes cannot be trusted now: either process's file is missing, from an earlier + boot, or not written for five minutes (each writes every 30 seconds); a drive's read has been in + flight longer than any read can take (smartctl's 20 s bound, 2 s for its output, and one write of + slack), which is a read stuck in the kernel, since such a read cannot be ended; or a drive has not + been looked at for twice its cadence. + + A drive found asleep at every look is said as asleep since when, and HIGH once that is a day, because + its health has not been read for that long and a drive that stopped answering can look asleep. It + never makes the module unhealthy on its own, at any length: a drive kept for backups may sleep for + weeks, and its health being unread is said, not raised. What the tool answers as `why` is one plain sentence per finding, for the operator, such as "The drive nvme0n1 (Samsung SSD 990 PRO 2TB) says it is failing: its spare capacity is below its threshold." @@ -150,23 +165,26 @@ as not read, never as cool. Nothing here conflicts with GENESIS. with a failing drive is not healthy within the gate's bound, and is put back there (ADR 0236 §2–3). If that machine is the build's first machine, the build reaches no other machine. Option 1 of the third question removes this. Until it is built, the laptop is the right first machine for this module. -- The module installs a package (`smartmontools`) and runs a root daemon on every machine it is assigned - to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail, - checked hourly. It also keeps running on a machine with no drive it can watch, such as a virtual - machine. -- **Reading a drive may keep it from spinning down.** Neither the watch nor `smartd` wakes a drive in - standby. But reading SMART from an awake drive can restart its standby timer, so a drive read more often - than its timer is set for never reaches standby. Whether the servers' drives are set to spin down is not - known: no mesh tool reads a drive's standby timer or power mode, and no module sets one. On 2026-10-08, - `disk-load` showed the anchor's two drives written to continuously (they hold its root, so they never - idle) and the home server's twelve spinning drives without I/O over a ten-second sample, which says - nothing of their timers. So a spinning drive is read hourly, by the watch and by `smartd` (`-i 3600`). A - drive whose timer is set longer than an hour can still be kept awake by these reads. Reading the power - mode and the timer is a follow-up for this module's tools. -- **A drive behind a USB bridge** that does not pass the power-mode question on is not read by the watch - (smartctl is asked to stop, not to read on), and is said as not read. `smartd` has no such per-drive - stop, and may wake it. -- The watch runs as root. It writes one file of readings under the runtime directory, and nothing else. +- The module installs a package (`smartmontools`) for `smartctl`, and runs two processes of its own as + root, each writing one file under the runtime directory and nothing else. It does not run `smartd`. +- **A drive is not woken by the module, as far as the drive can say so.** Each read is asked with + `-n standby`, and smartctl stops without reading when the drive says it is in standby or asleep (an ATA + drive by its power mode, a SAS drive or one the operating system powered down by smartctl's own words). + smartctl is told the drive's kind (`-d nvme`, `sat` or `scsi`) where it is known, so it does not probe a + drive to find out. A drive that cannot say whether it is asleep (a USB bridge that does not pass the + question on) is not read at all, and is said as not read. What remains: a drive of a kind the module + cannot tell is read with smartctl's own detection. +- **Reading a drive may keep it from spinning down.** Reading SMART from an awake drive can restart its + standby timer, so a drive read more often than its timer is set for never reaches standby. Whether the + servers' drives are set to spin down is not known: no mesh tool reads a drive's standby timer or power + mode, and no module sets one. On 2026-10-08, `disk-load` showed the anchor's two drives written to + continuously (they hold its root, so they never idle) and the home server's twelve spinning drives + without I/O over a ten-second sample, which says nothing of their timers. So the module, the only + reader, reads a spinning drive once an hour, timed from its last look. A drive whose timer is set + longer than an hour can still be kept awake. Reading the power mode and the timer is a follow-up for + this module's tools. +- **A drive that stops answering may look asleep**, and is then HIGH after a day, never unhealthy (rule + 4). A drive whose read hangs instead is unhealthy within a minute of its bound. ## Follow-ups @@ -196,5 +214,5 @@ as not read, never as cool. Nothing here conflicts with GENESIS. | 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only | | 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged; 0 °C is a reading and 127 °C is not | | 3 No seat | the manifest test that `sensors` claims nothing | -| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing reading is unhealthy, that a hung `smartctl` is bounded and the fans are still written, and that its `why` is one sentence per finding; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | +| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing file is unhealthy, that a hung `smartctl` is bounded, that a read stuck past its bound is unhealthy and does not stop the other drives, that a drive not looked at for twice its cadence is unhealthy, that a drive asleep for a day is HIGH and healthy, that a SAS drive in standby is asleep, that smartctl is told each drive's kind, and that its `why` is one sentence per finding; the manifest test that no `smartd` runs; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | | 5 Vendor extras only | review, when the vendor module is next changed | diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 44cddad6..df7167bf 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -284,24 +284,27 @@ health of this design, not yet through the machine's own judgement as §10 does ``` sensors on one machine node-engine controller ┌──────────────────────────────────────────────┐ - │ smartd (unit) watches every drive, │ unit ──► liveness + readiness - │ never wakes one asleep │ - │ watch (root) each drive's health │ - │ (spinning: hourly); fans │ tool ──► asks sensors_health ──► module.sensors..unhealthy - │ that ran since boot │ every 60 s, 2 looks (the why is its evidence) - │ tools (operator) /sys and the watch's │ - │ readings; read-only │ + │ drive reader (root) each drive's health, │ unit ──► liveness + readiness + │ never one asleep; each │ + │ read bounded, or said │ + │ fan watch fans that ran since boot │ tool ──► asks sensors_health ──► module.sensors..unhealthy + │ │ every 60 s, 2 looks (the why is its evidence) + │ tools (operator) /sys and the two files; │ + │ read-only │ └──────────────────────────────────────────────┘ ``` - **Unhealthy means the hardware is failing, measured against its own thresholds:** a CPU, GPU, drive or zone temperature at or above its own critical value (a board chip's is shown, never judged); a drive that says it is failing (a SATA drive's verdict, a pre-failure attribute failing now, an NVMe critical warning - bit other than temperature); a fan that ran since boot and reads 0 while the CPU, the board or a sensor on - its own chip is at or above its own maximum; memory that reports an uncorrectable error; or the watch's - readings older than five minutes. HIGH readings and fan notes are said by the tools and raise nothing. -- **A drive is never woken, and is read seldom enough to sleep:** a spinning drive hourly, by the watch and - by smartd, because a read can restart a standby timer the mesh cannot yet see. + bit other than temperature); a fan that ran since boot and reads 0 while a CPU sensor, a thermal zone or a + sensor on its own non-board chip is at or above its own maximum; memory that reports an uncorrectable + error. **Or that the module's own processes are stale:** a file five minutes old, a drive's read stuck + past its bound, a drive not looked at for twice its cadence. HIGH readings, fan notes and a drive asleep + for a day are said by the tools and raise nothing. +- **One reader, never waking a drive that can say it sleeps:** the module's drive reader alone reads the + drives (no `smartd`), a spinning drive hourly, because a read can restart a standby timer the mesh cannot + yet see. - **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no SMART drive is healthy. - **The cost of using module health for the machine's fault** is that the condition names the module, and a @@ -320,7 +323,7 @@ health of this design, not yet through the machine's own judgement as §10 does | D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine | | E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses | | F — the machine's network | `mesh-host` (node-engine), `mesh-controller`, `mesh-catalog` | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile | -| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, `smartd` and the watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy | +| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, the drive reader and the fan watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy | Phases A and B may be built together; A is live first, because it judges without a single declaration.