From c8b6a4eea850cc1eb8b6e99a7119aeed07edfe80 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 13:48:52 +0200 Subject: [PATCH 1/4] ADR 0256: one module reads the machine's hardware, so a failing drive or fan is raised, not found dead MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A machine's temperatures, fans and drive health were read by the laptop's vendor module only; the machines' 20 drives were watched by nothing. The sensors module reads them on every machine, holds no seat until a caller must reach it by role, and says a hardware fault through ADR 0240's health until the machine judges its own hardware. To-be 48 gains §11 and phase G. --- ...-and-a-fault-is-said-through-its-health.md | 172 ++++++++++++++++++ .../48-a-module-says-how-it-is-healthy.md | 35 +++- 2 files changed, 206 insertions(+), 1 deletion(-) create mode 100644 02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md diff --git a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md new file mode 100644 index 00000000..ece2e37c --- /dev/null +++ b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md @@ -0,0 +1,172 @@ +--- +topic: what runs on it +status: accepted +date: 2026-10-08 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md +--- + +# 256. A machine's hardware condition is read by one module on every machine, and a fault is said through its health + +## Context + +The operator set the boundary on 2026-10-08, in three sentences: + +- **What hardware a machine has is a node-engine fact.** +- **A module holding a generic role reads and provides it.** +- **A vendor module adds only vendor-specific extras.** + +The `power` module already holds the battery and the power draw (`node-power`). This record is about +everything else a Linux machine reports about the condition of its own hardware: temperatures, fans, drive +health and the kernel's count of hardware errors. + +What the mesh had on 2026-10-08, measured on the catalogue's main branch and on the live mesh: + +- **One of 116 catalogue modules reads a temperature, and it is the laptop's vendor module.** It reads + every hwmon chip for its `thermals` tool. The workstation and the servers have no vendor module, so the + mesh cannot say how hot they are. +- **No module reads a drive's health.** No manifest declares `smartmontools`, and no code calls `smartctl`. + `disk-load` reads how busy a disk is, not whether it is failing. The machines hold 20 drives between + them: the home server has twelve spinning drives and two NVMe drives, the anchor has two spinning drives + in a mirror, the workstation has one spinning drive and two NVMe drives, and the laptop has one NVMe drive + (`disk-load.disk_devices` on each machine). The media on the home server is data the operator counts as + irreplaceable. +- **The hardware gives its own thresholds, and not always sane ones.** On the laptop, nine hwmon chips and + one thermal zone answer. The NVMe drive's composite sensor reports its own maximum and critical + temperatures (83.85 °C and 87.85 °C). Its other two sensors report a maximum of 65261.85 °C, which is + the format's largest value and means "none". The CPU's sensor reports no threshold at all. The thermal + zone gives trip points (`passive`, `hot`, `critical`). +- **Drive health needs root.** The NVMe controller's device is readable by root only, and so are SATA + drives' SMART commands. The tool runner runs as the operator account (ADR 0175 §4). +- **A module's health is the only way a module raises a condition.** ADR 0240 judges a module's + long-running resources. A `tool` check is allowed only beside a check of another kind on the same + module, because a module never judges itself alone (rule 2). A module that runs nothing long-lived + declares no health. A failing check raises `module...unhealthy`. Its summary names the + check, and what the check found is kept as evidence (to-be 48 §4). + +**Checked against GENESIS.** *The mesh notices when something is wrong before you do* +([effect](../00-META/effect.md)): a drive that is failing or a fan that has stopped is the kind of fault +nobody notices until it costs data or hardware. *Evidence over assertion*: each threshold is the +hardware's own, not a number the mesh chose. *Failure must be loud*: a sensor that cannot be read is said +as not read, never as cool. Nothing here conflicts with GENESIS. + +## Considered Options + +**Where the reading lives.** + +1. *In each vendor module.* Rejected: it breaks the boundary, and the workstation and the servers have no + vendor module. +2. *In `disk-load` and `power`.* Rejected: `disk-load` reads load, needs no root and declares no + resources, so it can be assigned anywhere. Drive health would give it root, a package and a daemon. + `power` holds its own seat, and this is not about power. +3. **One module, `sensors`, that reads every generic sensor class and can be assigned to every machine.** + Chosen. A class the machine does not have (no fan, no SMART, no memory-error counters) is said as not + present, never as an error. + +**Whether it claims a seat.** + +1. *A node seat, such as `node-sensors`, whose verbs are these tools.* Rejected for now. A seat is a role + that one module holds on each machine, and callers reach it by the role rather than by the module. That + matters when two modules could hold the role, or when another part of the mesh must ask for it without + knowing who answers. Neither is true yet. There is one reader, and it only reads. A new seat is defined + in the controller, and the verbs of a new seat are required from the start (ADR 0246), so the module + would wait on a controller change for nothing a caller needs today. +2. **A module with tools, like `disk-load`.** Chosen. The seat is added when a caller has to ask by role: + the bar's temperature blocks, or a vendor module's fan policy reading the machine's temperatures. Then + these tools become its verbs, as `disk-load`'s README already says for the storage tools. + +**How a fault becomes a condition.** + +1. *The node-engine judges the machine's hardware itself, beside its network (ADR 0241), and raises + `machine..hardware`.* This is the right long-term home, because the hardware is the machine's, as its + network is, and a machine's fault must not be pinned on a module (issue 281). It needs a change to the + node-engine and to the controller. It is not chosen now and is left as the follow-up below. +2. *An event, as `memory-pressure` emits `pressure.high`.* Rejected: an event is a sample, not a state + (ADR 0240's own reason), and no event raises a condition. +3. **ADR 0240's health, as it stands.** Chosen, because it raises a condition today with no core change. + The module gets the two long-running resources that rule 2 needs, and each has a real job: + - **the drive daemon (`smartd`)**, judged by the unit's own readiness, which the module does not run + itself. It watches every drive between looks and writes what it finds to the journal when it happens; + - **the module's watch**, a process of its own bundle run as root, judged by the module's + `sensors_health` tool. The watch reads each drive's health every ten minutes, so the tool runner needs + no root. It skips a drive that is asleep, so a spinning drive is never woken to be read. It also + records which fans have run since the machine started. + +## Decision + +1. **The `sensors` module reads the condition of a machine's hardware.** It only reads. It has read-only + tools for: + - temperatures (hwmon and thermal zones, each with its label and its own thresholds); + - fans; + - drives (NVMe health log, SATA SMART); + - the kernel's hardware-error counters (memory errors from EDAC, PCIe errors); + - a summary that says one line per finding. + + A sensor class the machine lacks is said as not present. A threshold the hardware gives as its format's + largest value, or as nothing, is no threshold, and the reading is said without a flag. +2. **Every flag is measured against the hardware's own threshold.** A temperature is HIGH at its own + maximum (or the zone's `hot` or `passive` trip). It is CRITICAL at its own critical value (or the + zone's `critical` trip). The mesh adds no threshold of its own. A sensor that reports none is never + flagged. +3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR + 0246 when a caller has to reach it by role. +4. **It declares its health under ADR 0240.** `smartd` is judged by its unit. The watch is judged by + `sensors_health`, which is unhealthy when: + - a temperature is at or above its own critical threshold; + - a drive says it is failing: SMART's overall verdict, an attribute failing now, or an NVMe critical + warning bit; + - a fan that ran since the machine started reads 0 while a temperature is at or above its own maximum; + - memory reports an uncorrectable error. + + What the tool answers as `why` is one plain sentence for the operator, such as "The drive nvme0 + (Samsung 990 PRO) says its spare capacity is below its threshold." +5. **A vendor module keeps only what is its vendor's.** For example, the laptop's fan curves and + performance profiles. Its reading of the generic sensors moves to this module when that module is + next changed. + +## Consequences + +- Every machine can say how hot it is and whether its drives and fans are well. A failing drive on the + home server is raised as a condition instead of being found when it dies. +- **The condition names the module, not the machine.** A hardware fault on the home server is raised as + `module.sensors..unhealthy`, and its summary names the failing check. The sentence that says + what is wrong is in the condition's evidence. The plain words of ADR 0253 can carry it to the operator + once the module-health wording uses a tool check's `why`. +- **A standing hardware fault holds the module's own builds.** A new build of `sensors` sent to a machine + with a failing drive is not healthy within the gate's bound, and is put back there (ADR 0236 §2–3). If + that machine is the build's first machine, the build reaches no other machine. Option 1 of the third + question removes this. Until it is built, the laptop is the right first machine for this module. +- The module installs a package (`smartmontools`) and runs a root daemon on every machine it is assigned + to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail. + It also keeps running on a machine with no drive it can watch, such as a virtual machine. +- The watch runs as root. It writes one file of readings under the runtime directory, and nothing else. + +## Follow-ups + +- **The machine's hardware judged by the node-engine** (third question, option 1), raising a machine + condition. Then `sensors` keeps its tools and drops the tool check from its health. +- **The module-health wording takes a tool check's `why`** as its explanation (ADR 0253 rule 2). +- **Temperature blocks for the bar**, once the generic contract for a module's bar blocks exists. +- **The vendor module's `thermals` tool** points at this module. + +## References + +- [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) and + [to-be 48](../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md) §11, which this record adds. +- [ADR 0241](0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md) + (the machine's network, the shape of the follow-up), + [ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md) (a new seat's verbs), + [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md) + (the gate), [issue 281](../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md). +- mesh-catalog `modules/sensors`, beside `modules/disk-load` (its closest relative). + +## How it is checked + +| Rule | Checked by | +|---|---| +| 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only | +| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged | +| 3 No seat | the manifest test that `sensors` claims nothing | +| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case, and that its `why` is one sentence; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | +| 5 Vendor extras only | review, when the vendor module is next changed | diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 7458ce7f..3998ae0f 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -2,13 +2,14 @@ layer: to-be status: in-progress code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog] -updated: 2026-10-07 +updated: 2026-10-08 decisions: - 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md - 02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md - 02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md - 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md + - 02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md --- # 48 — A module says how it is healthy @@ -273,6 +274,37 @@ the machine itself, beside its liveness looks, and says it in the same statement [ADR 0246](../../02-DECISIONS/0246-a-seats-new-verb-is-promised-before-it-is-required.md), from [issue 298](../../04-ISSUES/298-a-new-seat-verb-deadlocks-across-two-repositories/00-report.md)). +## 11. The machine under the modules: its hardware + +([ADR 0256](../../02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md).) +Every module on a machine runs on its hardware, and none of them owns it. The `sensors` module reads its +condition on every machine it is assigned to. It holds no seat, and states a fault through the module +health of this design, not yet through the machine's own judgement as §10 does for the network. + +``` + sensors on one machine node-engine controller + ┌─────────────────────────────────────────────┐ + │ smartd (unit) watches every drive, │ unit ──► liveness + readiness + │ never wakes one asleep │ + │ watch (root) every 10 min each │ + │ drive's health; fans │ tool ──► asks sensors_health ──► module.sensors..unhealthy + │ that ran since boot │ every 60 s, 2 looks (the why is its evidence) + │ tools (operator) /sys and the watch's │ + │ readings; read-only │ + └─────────────────────────────────────────────┘ +``` + +- **Unhealthy means the hardware is failing, measured against its own thresholds:** a temperature at or + above its own critical value, a drive that says it is failing (SMART's verdict, an attribute failing now, + an NVMe critical warning bit), a fan that ran since boot and reads 0 while a temperature is at or above its + own maximum, or memory that reports an uncorrectable error. HIGH readings are said by the tools and raise + nothing. +- **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no + SMART drive is healthy. +- **The cost of using module health for the machine's fault** is that the condition names the module, and a + standing fault holds the module's own builds at their gate. The machine judging its own hardware, as it + judges its network in §10, is the follow-up that removes both. + ## Phases | Phase | Repository | Delivers | Done when | @@ -285,6 +317,7 @@ the machine itself, beside its liveness looks, and says it in the same statement | D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine | | E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses | | F — the machine's network | `mesh-host` (node-engine), `mesh-controller`, `mesh-catalog` | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile | +| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, `smartd` and the watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy | Phases A and B may be built together; A is live first, because it judges without a single declaration. From f63828ce1cc442db60a1ddbe4f41f3578cecddb3 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 14:35:21 +0200 Subject: [PATCH 2/4] ADR 0256: say exactly what raises a condition, and why a drive is read only hourly Rule 4 now matches the module after review: the NVMe temperature bit, old-age attributes, a fan's minimum or alarm and a board chip's limits raise nothing; a stale watch does. The standby caveat is recorded: no mesh tool sees a drive's spin-down timer, so spinning drives are read hourly. --- ...-and-a-fault-is-said-through-its-health.md | 60 ++++++++++++++----- .../48-a-module-says-how-it-is-healthy.md | 21 ++++--- 2 files changed, 56 insertions(+), 25 deletions(-) diff --git a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md index ece2e37c..fc7e7ef5 100644 --- a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md +++ b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md @@ -89,9 +89,9 @@ as not read, never as cool. Nothing here conflicts with GENESIS. - **the drive daemon (`smartd`)**, judged by the unit's own readiness, which the module does not run itself. It watches every drive between looks and writes what it finds to the journal when it happens; - **the module's watch**, a process of its own bundle run as root, judged by the module's - `sensors_health` tool. The watch reads each drive's health every ten minutes, so the tool runner needs - no root. It skips a drive that is asleep, so a spinning drive is never woken to be read. It also - records which fans have run since the machine started. + `sensors_health` tool. The watch reads each drive's health (a spinning drive hourly, any other every + ten minutes), so the tool runner needs no root. It skips a drive that is asleep, so a spinning drive + is never woken to be read. It also records which fans have run since the machine started. ## Decision @@ -108,19 +108,32 @@ as not read, never as cool. Nothing here conflicts with GENESIS. 2. **Every flag is measured against the hardware's own threshold.** A temperature is HIGH at its own maximum (or the zone's `hot` or `passive` trip). It is CRITICAL at its own critical value (or the zone's `critical` trip). The mesh adds no threshold of its own. A sensor that reports none is never - flagged. + flagged. 0 °C is a reading; 127 and 127.5 °C are not, because they are what a board chip's input with + nothing on it answers. 3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR 0246 when a caller has to reach it by role. 4. **It declares its health under ADR 0240.** `smartd` is judged by its unit. The watch is judged by - `sensors_health`, which is unhealthy when: - - a temperature is at or above its own critical threshold; - - a drive says it is failing: SMART's overall verdict, an attribute failing now, or an NVMe critical - warning bit; - - a fan that ran since the machine started reads 0 while a temperature is at or above its own maximum; - - memory reports an uncorrectable error. + `sensors_health`, which is unhealthy when, and only when: + - a CPU, GPU, drive or thermal-zone temperature is at or above its own critical value. A board chip's + (a Super I/O's) critical value is shown as CRITICAL by the tools and never makes the module + unhealthy, because its inputs and limits are whatever the firmware left; + - a drive says it is failing: a SATA drive's overall SMART verdict, a pre-failure attribute failing + now, or an NVMe critical warning bit other than temperature (spare below threshold, reliability + degraded, read-only, backup failed). An NVMe drive is judged by its bits, not by the overall verdict + smartctl derives from them, and by its health log when smartctl gives no verdict at all. The + temperature bit alone, an old-age attribute failing now (a temperature, a count of hours) and an + attribute that failed in the past are HIGH; + - a fan that ran since the machine started reads 0 RPM now while the CPU, the board, or a sensor on + the fan's own chip is at or above its own maximum. A hot drive or network adapter says nothing of a + fan. A fan under its chip's minimum, or with its chip's alarm bit set, is said in a note and raises + nothing: a zero-fan GPU is under its minimum by design, and a board chip latches its alarm; + - memory reports an uncorrectable error; + - the watch's readings are not to be trusted: none since the machine started, or none written for five + minutes. The watch writes them every 30 seconds and reads the drives beside that, each `smartctl` + bounded in time, so old readings mean the watch is hung or gone, never that a drive is slow. - What the tool answers as `why` is one plain sentence for the operator, such as "The drive nvme0 - (Samsung 990 PRO) says its spare capacity is below its threshold." + What the tool answers as `why` is one plain sentence per finding, for the operator, such as "The drive + nvme0n1 (Samsung SSD 990 PRO 2TB) says it is failing: its spare capacity is below its threshold." 5. **A vendor module keeps only what is its vendor's.** For example, the laptop's fan curves and performance profiles. Its reading of the generic sensors moves to this module when that module is next changed. @@ -138,8 +151,21 @@ as not read, never as cool. Nothing here conflicts with GENESIS. that machine is the build's first machine, the build reaches no other machine. Option 1 of the third question removes this. Until it is built, the laptop is the right first machine for this module. - The module installs a package (`smartmontools`) and runs a root daemon on every machine it is assigned - to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail. - It also keeps running on a machine with no drive it can watch, such as a virtual machine. + to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail, + checked hourly. It also keeps running on a machine with no drive it can watch, such as a virtual + machine. +- **Reading a drive may keep it from spinning down.** Neither the watch nor `smartd` wakes a drive in + standby. But reading SMART from an awake drive can restart its standby timer, so a drive read more often + than its timer is set for never reaches standby. Whether the servers' drives are set to spin down is not + known: no mesh tool reads a drive's standby timer or power mode, and no module sets one. On 2026-10-08, + `disk-load` showed the anchor's two drives written to continuously (they hold its root, so they never + idle) and the home server's twelve spinning drives without I/O over a ten-second sample, which says + nothing of their timers. So a spinning drive is read hourly, by the watch and by `smartd` (`-i 3600`). A + drive whose timer is set longer than an hour can still be kept awake by these reads. Reading the power + mode and the timer is a follow-up for this module's tools. +- **A drive behind a USB bridge** that does not pass the power-mode question on is not read by the watch + (smartctl is asked to stop, not to read on), and is said as not read. `smartd` has no such per-drive + stop, and may wake it. - The watch runs as root. It writes one file of readings under the runtime directory, and nothing else. ## Follow-ups @@ -148,6 +174,8 @@ as not read, never as cool. Nothing here conflicts with GENESIS. condition. Then `sensors` keeps its tools and drops the tool check from its health. - **The module-health wording takes a tool check's `why`** as its explanation (ADR 0253 rule 2). - **Temperature blocks for the bar**, once the generic contract for a module's bar blocks exists. +- **A drive's power mode and standby timer in `sensors_drives`**, so the cadence above can be set from + what each drive is set to, rather than from a guess that it may be an hour. - **The vendor module's `thermals` tool** points at this module. ## References @@ -166,7 +194,7 @@ as not read, never as cool. Nothing here conflicts with GENESIS. | Rule | Checked by | |---|---| | 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only | -| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged | +| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged; 0 °C is a reading and 127 °C is not | | 3 No seat | the manifest test that `sensors` claims nothing | -| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case, and that its `why` is one sentence; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | +| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing reading is unhealthy, that a hung `smartctl` is bounded and the fans are still written, and that its `why` is one sentence per finding; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | | 5 Vendor extras only | review, when the vendor module is next changed | diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 3998ae0f..44cddad6 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -283,22 +283,25 @@ health of this design, not yet through the machine's own judgement as §10 does ``` sensors on one machine node-engine controller - ┌─────────────────────────────────────────────┐ + ┌──────────────────────────────────────────────┐ │ smartd (unit) watches every drive, │ unit ──► liveness + readiness │ never wakes one asleep │ - │ watch (root) every 10 min each │ - │ drive's health; fans │ tool ──► asks sensors_health ──► module.sensors..unhealthy + │ watch (root) each drive's health │ + │ (spinning: hourly); fans │ tool ──► asks sensors_health ──► module.sensors..unhealthy │ that ran since boot │ every 60 s, 2 looks (the why is its evidence) │ tools (operator) /sys and the watch's │ │ readings; read-only │ - └─────────────────────────────────────────────┘ + └──────────────────────────────────────────────┘ ``` -- **Unhealthy means the hardware is failing, measured against its own thresholds:** a temperature at or - above its own critical value, a drive that says it is failing (SMART's verdict, an attribute failing now, - an NVMe critical warning bit), a fan that ran since boot and reads 0 while a temperature is at or above its - own maximum, or memory that reports an uncorrectable error. HIGH readings are said by the tools and raise - nothing. +- **Unhealthy means the hardware is failing, measured against its own thresholds:** a CPU, GPU, drive or + zone temperature at or above its own critical value (a board chip's is shown, never judged); a drive that + says it is failing (a SATA drive's verdict, a pre-failure attribute failing now, an NVMe critical warning + bit other than temperature); a fan that ran since boot and reads 0 while the CPU, the board or a sensor on + its own chip is at or above its own maximum; memory that reports an uncorrectable error; or the watch's + readings older than five minutes. HIGH readings and fan notes are said by the tools and raise nothing. +- **A drive is never woken, and is read seldom enough to sleep:** a spinning drive hourly, by the watch and + by smartd, because a read can restart a standby timer the mesh cannot yet see. - **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no SMART drive is healthy. - **The cost of using module health for the machine's fault** is that the condition names the module, and a From 0406373545bbe4513d4a85cf4f30b3fd6209f3dc Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 14:59:12 +0200 Subject: [PATCH 3/4] ADR 0256: one reader of the drives, and a stuck read is said After the second review: smartd is dropped as a second reader that could wake or keep awake the drives, a stuck read or a drive not looked at is unhealthy, a drive asleep for a day is HIGH and never unhealthy, and a board chip's maximum says nothing of a fan. The never-woken and hourly claims now say exactly what holds. --- ...-and-a-fault-is-said-through-its-health.md | 86 +++++++++++-------- .../48-a-module-says-how-it-is-healthy.md | 29 ++++--- 2 files changed, 68 insertions(+), 47 deletions(-) diff --git a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md index fc7e7ef5..c386ba6e 100644 --- a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md +++ b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md @@ -85,13 +85,19 @@ as not read, never as cool. Nothing here conflicts with GENESIS. 2. *An event, as `memory-pressure` emits `pressure.high`.* Rejected: an event is a sample, not a state (ADR 0240's own reason), and no event raises a condition. 3. **ADR 0240's health, as it stands.** Chosen, because it raises a condition today with no core change. - The module gets the two long-running resources that rule 2 needs, and each has a real job: - - **the drive daemon (`smartd`)**, judged by the unit's own readiness, which the module does not run - itself. It watches every drive between looks and writes what it finds to the journal when it happens; - - **the module's watch**, a process of its own bundle run as root, judged by the module's - `sensors_health` tool. The watch reads each drive's health (a spinning drive hourly, any other every - ten minutes), so the tool runner needs no root. It skips a drive that is asleep, so a spinning drive - is never woken to be read. It also records which fans have run since the machine started. + Rule 2 needs a check of another kind beside the tool check, so the module runs two long-running + processes of its own bundle, each with a job of its own and its own file: + - **the drive reader**, run as root and judged by the unit's own readiness, which the module does not + run itself. It reads each drive's health, so the tool runner needs no root. A drive read that never + ends (a drive or a controller that stopped answering) cannot stop the fans being looked at; + - **the fan watch**, judged by the module's `sensors_health` tool. It records which fans have run + since the machine started. + + *Running `smartd` beside it* was the first choice and is rejected. `smartd` reads every drive when it + starts or restarts, which can spin a sleeping drive up. Its `-n standby` applies to ATA drives only, so + it polls a SAS drive in standby anyway. And as a second reader on its own schedule, it reads each drive + at another phase of the hour than the module does, which shortens the idle stretch a drive needs to + spin down. One reader, the module's, avoids all three. ## Decision @@ -112,8 +118,8 @@ as not read, never as cool. Nothing here conflicts with GENESIS. nothing on it answers. 3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR 0246 when a caller has to reach it by role. -4. **It declares its health under ADR 0240.** `smartd` is judged by its unit. The watch is judged by - `sensors_health`, which is unhealthy when, and only when: +4. **It declares its health under ADR 0240.** The drive reader is judged by its unit. The fan watch is + judged by `sensors_health`, which is unhealthy when, and only when: - a CPU, GPU, drive or thermal-zone temperature is at or above its own critical value. A board chip's (a Super I/O's) critical value is shown as CRITICAL by the tools and never makes the module unhealthy, because its inputs and limits are whatever the firmware left; @@ -123,14 +129,23 @@ as not read, never as cool. Nothing here conflicts with GENESIS. smartctl derives from them, and by its health log when smartctl gives no verdict at all. The temperature bit alone, an old-age attribute failing now (a temperature, a count of hours) and an attribute that failed in the past are HIGH; - - a fan that ran since the machine started reads 0 RPM now while the CPU, the board, or a sensor on - the fan's own chip is at or above its own maximum. A hot drive or network adapter says nothing of a - fan. A fan under its chip's minimum, or with its chip's alarm bit set, is said in a note and raises - nothing: a zero-fan GPU is under its minimum by design, and a board chip latches its alarm; + - a fan that ran since the machine started reads 0 RPM now while a CPU sensor, a thermal zone, or a + sensor on the fan's own chip is at or above its own maximum, the last only when that chip is not a + board chip. A board chip's maximum is no more trusted than its critical value. A hot drive or network + adapter says nothing of a fan. A fan under its chip's minimum, or with its chip's alarm bit set, is + said in a note and raises nothing: a zero-fan GPU is under its minimum by design, and a board chip + latches its alarm; - memory reports an uncorrectable error; - - the watch's readings are not to be trusted: none since the machine started, or none written for five - minutes. The watch writes them every 30 seconds and reads the drives beside that, each `smartctl` - bounded in time, so old readings mean the watch is hung or gone, never that a drive is slow. + - the module's own processes cannot be trusted now: either process's file is missing, from an earlier + boot, or not written for five minutes (each writes every 30 seconds); a drive's read has been in + flight longer than any read can take (smartctl's 20 s bound, 2 s for its output, and one write of + slack), which is a read stuck in the kernel, since such a read cannot be ended; or a drive has not + been looked at for twice its cadence. + + A drive found asleep at every look is said as asleep since when, and HIGH once that is a day, because + its health has not been read for that long and a drive that stopped answering can look asleep. It + never makes the module unhealthy on its own, at any length: a drive kept for backups may sleep for + weeks, and its health being unread is said, not raised. What the tool answers as `why` is one plain sentence per finding, for the operator, such as "The drive nvme0n1 (Samsung SSD 990 PRO 2TB) says it is failing: its spare capacity is below its threshold." @@ -150,23 +165,26 @@ as not read, never as cool. Nothing here conflicts with GENESIS. with a failing drive is not healthy within the gate's bound, and is put back there (ADR 0236 §2–3). If that machine is the build's first machine, the build reaches no other machine. Option 1 of the third question removes this. Until it is built, the laptop is the right first machine for this module. -- The module installs a package (`smartmontools`) and runs a root daemon on every machine it is assigned - to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail, - checked hourly. It also keeps running on a machine with no drive it can watch, such as a virtual - machine. -- **Reading a drive may keep it from spinning down.** Neither the watch nor `smartd` wakes a drive in - standby. But reading SMART from an awake drive can restart its standby timer, so a drive read more often - than its timer is set for never reaches standby. Whether the servers' drives are set to spin down is not - known: no mesh tool reads a drive's standby timer or power mode, and no module sets one. On 2026-10-08, - `disk-load` showed the anchor's two drives written to continuously (they hold its root, so they never - idle) and the home server's twelve spinning drives without I/O over a ten-second sample, which says - nothing of their timers. So a spinning drive is read hourly, by the watch and by `smartd` (`-i 3600`). A - drive whose timer is set longer than an hour can still be kept awake by these reads. Reading the power - mode and the timer is a follow-up for this module's tools. -- **A drive behind a USB bridge** that does not pass the power-mode question on is not read by the watch - (smartctl is asked to stop, not to read on), and is said as not read. `smartd` has no such per-drive - stop, and may wake it. -- The watch runs as root. It writes one file of readings under the runtime directory, and nothing else. +- The module installs a package (`smartmontools`) for `smartctl`, and runs two processes of its own as + root, each writing one file under the runtime directory and nothing else. It does not run `smartd`. +- **A drive is not woken by the module, as far as the drive can say so.** Each read is asked with + `-n standby`, and smartctl stops without reading when the drive says it is in standby or asleep (an ATA + drive by its power mode, a SAS drive or one the operating system powered down by smartctl's own words). + smartctl is told the drive's kind (`-d nvme`, `sat` or `scsi`) where it is known, so it does not probe a + drive to find out. A drive that cannot say whether it is asleep (a USB bridge that does not pass the + question on) is not read at all, and is said as not read. What remains: a drive of a kind the module + cannot tell is read with smartctl's own detection. +- **Reading a drive may keep it from spinning down.** Reading SMART from an awake drive can restart its + standby timer, so a drive read more often than its timer is set for never reaches standby. Whether the + servers' drives are set to spin down is not known: no mesh tool reads a drive's standby timer or power + mode, and no module sets one. On 2026-10-08, `disk-load` showed the anchor's two drives written to + continuously (they hold its root, so they never idle) and the home server's twelve spinning drives + without I/O over a ten-second sample, which says nothing of their timers. So the module, the only + reader, reads a spinning drive once an hour, timed from its last look. A drive whose timer is set + longer than an hour can still be kept awake. Reading the power mode and the timer is a follow-up for + this module's tools. +- **A drive that stops answering may look asleep**, and is then HIGH after a day, never unhealthy (rule + 4). A drive whose read hangs instead is unhealthy within a minute of its bound. ## Follow-ups @@ -196,5 +214,5 @@ as not read, never as cool. Nothing here conflicts with GENESIS. | 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only | | 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged; 0 °C is a reading and 127 °C is not | | 3 No seat | the manifest test that `sensors` claims nothing | -| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing reading is unhealthy, that a hung `smartctl` is bounded and the fans are still written, and that its `why` is one sentence per finding; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | +| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing file is unhealthy, that a hung `smartctl` is bounded, that a read stuck past its bound is unhealthy and does not stop the other drives, that a drive not looked at for twice its cadence is unhealthy, that a drive asleep for a day is HIGH and healthy, that a SAS drive in standby is asleep, that smartctl is told each drive's kind, and that its `why` is one sentence per finding; the manifest test that no `smartd` runs; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | | 5 Vendor extras only | review, when the vendor module is next changed | diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 44cddad6..df7167bf 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -284,24 +284,27 @@ health of this design, not yet through the machine's own judgement as §10 does ``` sensors on one machine node-engine controller ┌──────────────────────────────────────────────┐ - │ smartd (unit) watches every drive, │ unit ──► liveness + readiness - │ never wakes one asleep │ - │ watch (root) each drive's health │ - │ (spinning: hourly); fans │ tool ──► asks sensors_health ──► module.sensors..unhealthy - │ that ran since boot │ every 60 s, 2 looks (the why is its evidence) - │ tools (operator) /sys and the watch's │ - │ readings; read-only │ + │ drive reader (root) each drive's health, │ unit ──► liveness + readiness + │ never one asleep; each │ + │ read bounded, or said │ + │ fan watch fans that ran since boot │ tool ──► asks sensors_health ──► module.sensors..unhealthy + │ │ every 60 s, 2 looks (the why is its evidence) + │ tools (operator) /sys and the two files; │ + │ read-only │ └──────────────────────────────────────────────┘ ``` - **Unhealthy means the hardware is failing, measured against its own thresholds:** a CPU, GPU, drive or zone temperature at or above its own critical value (a board chip's is shown, never judged); a drive that says it is failing (a SATA drive's verdict, a pre-failure attribute failing now, an NVMe critical warning - bit other than temperature); a fan that ran since boot and reads 0 while the CPU, the board or a sensor on - its own chip is at or above its own maximum; memory that reports an uncorrectable error; or the watch's - readings older than five minutes. HIGH readings and fan notes are said by the tools and raise nothing. -- **A drive is never woken, and is read seldom enough to sleep:** a spinning drive hourly, by the watch and - by smartd, because a read can restart a standby timer the mesh cannot yet see. + bit other than temperature); a fan that ran since boot and reads 0 while a CPU sensor, a thermal zone or a + sensor on its own non-board chip is at or above its own maximum; memory that reports an uncorrectable + error. **Or that the module's own processes are stale:** a file five minutes old, a drive's read stuck + past its bound, a drive not looked at for twice its cadence. HIGH readings, fan notes and a drive asleep + for a day are said by the tools and raise nothing. +- **One reader, never waking a drive that can say it sleeps:** the module's drive reader alone reads the + drives (no `smartd`), a spinning drive hourly, because a read can restart a standby timer the mesh cannot + yet see. - **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no SMART drive is healthy. - **The cost of using module health for the machine's fault** is that the condition names the module, and a @@ -320,7 +323,7 @@ health of this design, not yet through the machine's own judgement as §10 does | D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine | | E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses | | F — the machine's network | `mesh-host` (node-engine), `mesh-controller`, `mesh-catalog` | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile | -| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, `smartd` and the watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy | +| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, the drive reader and the fan watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy | Phases A and B may be built together; A is live first, because it judges without a single declaration. From df41dfff38b4aa690799e1d54bf0220022a033bb Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 15:04:03 +0200 Subject: [PATCH 4/4] ADR 0256: say exactly how a SAS drive, a USB drive and a stuck read are treated The SAS sentence now holds for modes with spaces, a hung read is unhealthy in two to three minutes, not one, every USB drive is told sat, and the limits of a stuck read are recorded. --- ...-and-a-fault-is-said-through-its-health.md | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md index c386ba6e..26e99be0 100644 --- a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md +++ b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md @@ -168,12 +168,16 @@ as not read, never as cool. Nothing here conflicts with GENESIS. - The module installs a package (`smartmontools`) for `smartctl`, and runs two processes of its own as root, each writing one file under the runtime directory and nothing else. It does not run `smartd`. - **A drive is not woken by the module, as far as the drive can say so.** Each read is asked with - `-n standby`, and smartctl stops without reading when the drive says it is in standby or asleep (an ATA - drive by its power mode, a SAS drive or one the operating system powered down by smartctl's own words). - smartctl is told the drive's kind (`-d nvme`, `sat` or `scsi`) where it is known, so it does not probe a - drive to find out. A drive that cannot say whether it is asleep (a USB bridge that does not pass the - question on) is not read at all, and is said as not read. What remains: a drive of a kind the module - cannot tell is read with smartctl's own detection. + `-n standby`, and smartctl stops without reading when the drive says it is in standby or asleep. An ATA + drive says it by its power mode (STANDBY, STANDBY_Y, SLEEP). A SAS drive, or one the operating system + powered down, says it in smartctl's own words, "Device is in mode, exit(2)", where a SAS mode + holds spaces (STANDBY BY TIMER, STANDBY_Y BY COMMAND, LOW POWER); the module takes every such mode, and + an idle mode never stops smartctl. smartctl is told the drive's kind (`-d nvme`, `sat` or `scsi`) where + it is known, so it does not probe a drive to find out. A drive behind USB is always told `sat`: one + behind a USB-to-NVMe enclosure or a vendor's own bridge does not answer to it, and is said as not read. + A drive that cannot say whether it is asleep (a USB bridge that does not pass the question on) is not + read at all, and is said as not read. What remains: a drive of a kind the module cannot tell (a virtual + disk, a memory card) is read with smartctl's own detection. - **Reading a drive may keep it from spinning down.** Reading SMART from an awake drive can restart its standby timer, so a drive read more often than its timer is set for never reaches standby. Whether the servers' drives are set to spin down is not known: no mesh tool reads a drive's standby timer or power @@ -184,7 +188,12 @@ as not read, never as cool. Nothing here conflicts with GENESIS. longer than an hour can still be kept awake. Reading the power mode and the timer is a follow-up for this module's tools. - **A drive that stops answering may look asleep**, and is then HIGH after a day, never unhealthy (rule - 4). A drive whose read hangs instead is unhealthy within a minute of its bound. + 4). A drive whose read hangs instead is unhealthy in about two to three minutes: its read is said once + it is past its 52 s bound, and the health check needs two looks 60 s apart to be unhealthy. +- **Known limits of a stuck read.** A smartctl stuck in the kernel cannot be ended, and outlives the + drive reader: a restarted reader starts a new read of that drive, which may stick beside the first. The + reads of all drives run at once, with no cap per controller or bridge; a server with many drives on one + controller asks it all at once, once an hour. ## Follow-ups