diff --git a/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md new file mode 100644 index 00000000..26e99be0 --- /dev/null +++ b/02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md @@ -0,0 +1,227 @@ +--- +topic: what runs on it +status: accepted +date: 2026-10-08 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md +--- + +# 256. A machine's hardware condition is read by one module on every machine, and a fault is said through its health + +## Context + +The operator set the boundary on 2026-10-08, in three sentences: + +- **What hardware a machine has is a node-engine fact.** +- **A module holding a generic role reads and provides it.** +- **A vendor module adds only vendor-specific extras.** + +The `power` module already holds the battery and the power draw (`node-power`). This record is about +everything else a Linux machine reports about the condition of its own hardware: temperatures, fans, drive +health and the kernel's count of hardware errors. + +What the mesh had on 2026-10-08, measured on the catalogue's main branch and on the live mesh: + +- **One of 116 catalogue modules reads a temperature, and it is the laptop's vendor module.** It reads + every hwmon chip for its `thermals` tool. The workstation and the servers have no vendor module, so the + mesh cannot say how hot they are. +- **No module reads a drive's health.** No manifest declares `smartmontools`, and no code calls `smartctl`. + `disk-load` reads how busy a disk is, not whether it is failing. The machines hold 20 drives between + them: the home server has twelve spinning drives and two NVMe drives, the anchor has two spinning drives + in a mirror, the workstation has one spinning drive and two NVMe drives, and the laptop has one NVMe drive + (`disk-load.disk_devices` on each machine). The media on the home server is data the operator counts as + irreplaceable. +- **The hardware gives its own thresholds, and not always sane ones.** On the laptop, nine hwmon chips and + one thermal zone answer. The NVMe drive's composite sensor reports its own maximum and critical + temperatures (83.85 °C and 87.85 °C). Its other two sensors report a maximum of 65261.85 °C, which is + the format's largest value and means "none". The CPU's sensor reports no threshold at all. The thermal + zone gives trip points (`passive`, `hot`, `critical`). +- **Drive health needs root.** The NVMe controller's device is readable by root only, and so are SATA + drives' SMART commands. The tool runner runs as the operator account (ADR 0175 §4). +- **A module's health is the only way a module raises a condition.** ADR 0240 judges a module's + long-running resources. A `tool` check is allowed only beside a check of another kind on the same + module, because a module never judges itself alone (rule 2). A module that runs nothing long-lived + declares no health. A failing check raises `module...unhealthy`. Its summary names the + check, and what the check found is kept as evidence (to-be 48 §4). + +**Checked against GENESIS.** *The mesh notices when something is wrong before you do* +([effect](../00-META/effect.md)): a drive that is failing or a fan that has stopped is the kind of fault +nobody notices until it costs data or hardware. *Evidence over assertion*: each threshold is the +hardware's own, not a number the mesh chose. *Failure must be loud*: a sensor that cannot be read is said +as not read, never as cool. Nothing here conflicts with GENESIS. + +## Considered Options + +**Where the reading lives.** + +1. *In each vendor module.* Rejected: it breaks the boundary, and the workstation and the servers have no + vendor module. +2. *In `disk-load` and `power`.* Rejected: `disk-load` reads load, needs no root and declares no + resources, so it can be assigned anywhere. Drive health would give it root, a package and a daemon. + `power` holds its own seat, and this is not about power. +3. **One module, `sensors`, that reads every generic sensor class and can be assigned to every machine.** + Chosen. A class the machine does not have (no fan, no SMART, no memory-error counters) is said as not + present, never as an error. + +**Whether it claims a seat.** + +1. *A node seat, such as `node-sensors`, whose verbs are these tools.* Rejected for now. A seat is a role + that one module holds on each machine, and callers reach it by the role rather than by the module. That + matters when two modules could hold the role, or when another part of the mesh must ask for it without + knowing who answers. Neither is true yet. There is one reader, and it only reads. A new seat is defined + in the controller, and the verbs of a new seat are required from the start (ADR 0246), so the module + would wait on a controller change for nothing a caller needs today. +2. **A module with tools, like `disk-load`.** Chosen. The seat is added when a caller has to ask by role: + the bar's temperature blocks, or a vendor module's fan policy reading the machine's temperatures. Then + these tools become its verbs, as `disk-load`'s README already says for the storage tools. + +**How a fault becomes a condition.** + +1. *The node-engine judges the machine's hardware itself, beside its network (ADR 0241), and raises + `machine..hardware`.* This is the right long-term home, because the hardware is the machine's, as its + network is, and a machine's fault must not be pinned on a module (issue 281). It needs a change to the + node-engine and to the controller. It is not chosen now and is left as the follow-up below. +2. *An event, as `memory-pressure` emits `pressure.high`.* Rejected: an event is a sample, not a state + (ADR 0240's own reason), and no event raises a condition. +3. **ADR 0240's health, as it stands.** Chosen, because it raises a condition today with no core change. + Rule 2 needs a check of another kind beside the tool check, so the module runs two long-running + processes of its own bundle, each with a job of its own and its own file: + - **the drive reader**, run as root and judged by the unit's own readiness, which the module does not + run itself. It reads each drive's health, so the tool runner needs no root. A drive read that never + ends (a drive or a controller that stopped answering) cannot stop the fans being looked at; + - **the fan watch**, judged by the module's `sensors_health` tool. It records which fans have run + since the machine started. + + *Running `smartd` beside it* was the first choice and is rejected. `smartd` reads every drive when it + starts or restarts, which can spin a sleeping drive up. Its `-n standby` applies to ATA drives only, so + it polls a SAS drive in standby anyway. And as a second reader on its own schedule, it reads each drive + at another phase of the hour than the module does, which shortens the idle stretch a drive needs to + spin down. One reader, the module's, avoids all three. + +## Decision + +1. **The `sensors` module reads the condition of a machine's hardware.** It only reads. It has read-only + tools for: + - temperatures (hwmon and thermal zones, each with its label and its own thresholds); + - fans; + - drives (NVMe health log, SATA SMART); + - the kernel's hardware-error counters (memory errors from EDAC, PCIe errors); + - a summary that says one line per finding. + + A sensor class the machine lacks is said as not present. A threshold the hardware gives as its format's + largest value, or as nothing, is no threshold, and the reading is said without a flag. +2. **Every flag is measured against the hardware's own threshold.** A temperature is HIGH at its own + maximum (or the zone's `hot` or `passive` trip). It is CRITICAL at its own critical value (or the + zone's `critical` trip). The mesh adds no threshold of its own. A sensor that reports none is never + flagged. 0 °C is a reading; 127 and 127.5 °C are not, because they are what a board chip's input with + nothing on it answers. +3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR + 0246 when a caller has to reach it by role. +4. **It declares its health under ADR 0240.** The drive reader is judged by its unit. The fan watch is + judged by `sensors_health`, which is unhealthy when, and only when: + - a CPU, GPU, drive or thermal-zone temperature is at or above its own critical value. A board chip's + (a Super I/O's) critical value is shown as CRITICAL by the tools and never makes the module + unhealthy, because its inputs and limits are whatever the firmware left; + - a drive says it is failing: a SATA drive's overall SMART verdict, a pre-failure attribute failing + now, or an NVMe critical warning bit other than temperature (spare below threshold, reliability + degraded, read-only, backup failed). An NVMe drive is judged by its bits, not by the overall verdict + smartctl derives from them, and by its health log when smartctl gives no verdict at all. The + temperature bit alone, an old-age attribute failing now (a temperature, a count of hours) and an + attribute that failed in the past are HIGH; + - a fan that ran since the machine started reads 0 RPM now while a CPU sensor, a thermal zone, or a + sensor on the fan's own chip is at or above its own maximum, the last only when that chip is not a + board chip. A board chip's maximum is no more trusted than its critical value. A hot drive or network + adapter says nothing of a fan. A fan under its chip's minimum, or with its chip's alarm bit set, is + said in a note and raises nothing: a zero-fan GPU is under its minimum by design, and a board chip + latches its alarm; + - memory reports an uncorrectable error; + - the module's own processes cannot be trusted now: either process's file is missing, from an earlier + boot, or not written for five minutes (each writes every 30 seconds); a drive's read has been in + flight longer than any read can take (smartctl's 20 s bound, 2 s for its output, and one write of + slack), which is a read stuck in the kernel, since such a read cannot be ended; or a drive has not + been looked at for twice its cadence. + + A drive found asleep at every look is said as asleep since when, and HIGH once that is a day, because + its health has not been read for that long and a drive that stopped answering can look asleep. It + never makes the module unhealthy on its own, at any length: a drive kept for backups may sleep for + weeks, and its health being unread is said, not raised. + + What the tool answers as `why` is one plain sentence per finding, for the operator, such as "The drive + nvme0n1 (Samsung SSD 990 PRO 2TB) says it is failing: its spare capacity is below its threshold." +5. **A vendor module keeps only what is its vendor's.** For example, the laptop's fan curves and + performance profiles. Its reading of the generic sensors moves to this module when that module is + next changed. + +## Consequences + +- Every machine can say how hot it is and whether its drives and fans are well. A failing drive on the + home server is raised as a condition instead of being found when it dies. +- **The condition names the module, not the machine.** A hardware fault on the home server is raised as + `module.sensors..unhealthy`, and its summary names the failing check. The sentence that says + what is wrong is in the condition's evidence. The plain words of ADR 0253 can carry it to the operator + once the module-health wording uses a tool check's `why`. +- **A standing hardware fault holds the module's own builds.** A new build of `sensors` sent to a machine + with a failing drive is not healthy within the gate's bound, and is put back there (ADR 0236 §2–3). If + that machine is the build's first machine, the build reaches no other machine. Option 1 of the third + question removes this. Until it is built, the laptop is the right first machine for this module. +- The module installs a package (`smartmontools`) for `smartctl`, and runs two processes of its own as + root, each writing one file under the runtime directory and nothing else. It does not run `smartd`. +- **A drive is not woken by the module, as far as the drive can say so.** Each read is asked with + `-n standby`, and smartctl stops without reading when the drive says it is in standby or asleep. An ATA + drive says it by its power mode (STANDBY, STANDBY_Y, SLEEP). A SAS drive, or one the operating system + powered down, says it in smartctl's own words, "Device is in mode, exit(2)", where a SAS mode + holds spaces (STANDBY BY TIMER, STANDBY_Y BY COMMAND, LOW POWER); the module takes every such mode, and + an idle mode never stops smartctl. smartctl is told the drive's kind (`-d nvme`, `sat` or `scsi`) where + it is known, so it does not probe a drive to find out. A drive behind USB is always told `sat`: one + behind a USB-to-NVMe enclosure or a vendor's own bridge does not answer to it, and is said as not read. + A drive that cannot say whether it is asleep (a USB bridge that does not pass the question on) is not + read at all, and is said as not read. What remains: a drive of a kind the module cannot tell (a virtual + disk, a memory card) is read with smartctl's own detection. +- **Reading a drive may keep it from spinning down.** Reading SMART from an awake drive can restart its + standby timer, so a drive read more often than its timer is set for never reaches standby. Whether the + servers' drives are set to spin down is not known: no mesh tool reads a drive's standby timer or power + mode, and no module sets one. On 2026-10-08, `disk-load` showed the anchor's two drives written to + continuously (they hold its root, so they never idle) and the home server's twelve spinning drives + without I/O over a ten-second sample, which says nothing of their timers. So the module, the only + reader, reads a spinning drive once an hour, timed from its last look. A drive whose timer is set + longer than an hour can still be kept awake. Reading the power mode and the timer is a follow-up for + this module's tools. +- **A drive that stops answering may look asleep**, and is then HIGH after a day, never unhealthy (rule + 4). A drive whose read hangs instead is unhealthy in about two to three minutes: its read is said once + it is past its 52 s bound, and the health check needs two looks 60 s apart to be unhealthy. +- **Known limits of a stuck read.** A smartctl stuck in the kernel cannot be ended, and outlives the + drive reader: a restarted reader starts a new read of that drive, which may stick beside the first. The + reads of all drives run at once, with no cap per controller or bridge; a server with many drives on one + controller asks it all at once, once an hour. + +## Follow-ups + +- **The machine's hardware judged by the node-engine** (third question, option 1), raising a machine + condition. Then `sensors` keeps its tools and drops the tool check from its health. +- **The module-health wording takes a tool check's `why`** as its explanation (ADR 0253 rule 2). +- **Temperature blocks for the bar**, once the generic contract for a module's bar blocks exists. +- **A drive's power mode and standby timer in `sensors_drives`**, so the cadence above can be set from + what each drive is set to, rather than from a guess that it may be an hour. +- **The vendor module's `thermals` tool** points at this module. + +## References + +- [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) and + [to-be 48](../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md) §11, which this record adds. +- [ADR 0241](0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md) + (the machine's network, the shape of the follow-up), + [ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md) (a new seat's verbs), + [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md) + (the gate), [issue 281](../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md). +- mesh-catalog `modules/sensors`, beside `modules/disk-load` (its closest relative). + +## How it is checked + +| Rule | Checked by | +|---|---| +| 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only | +| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged; 0 °C is a reading and 127 °C is not | +| 3 No seat | the manifest test that `sensors` claims nothing | +| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing file is unhealthy, that a hung `smartctl` is bounded, that a read stuck past its bound is unhealthy and does not stop the other drives, that a drive not looked at for twice its cadence is unhealthy, that a drive asleep for a day is HIGH and healthy, that a SAS drive in standby is asleep, that smartctl is told each drive's kind, and that its `why` is one sentence per finding; the manifest test that no `smartd` runs; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has | +| 5 Vendor extras only | review, when the vendor module is next changed | diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 441b16e5..10a620e0 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -10,6 +10,7 @@ decisions: - 02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md - 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md + - 02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md --- # 48 — A module says how it is healthy @@ -292,6 +293,43 @@ the machine itself, beside its liveness looks, and says it in the same statement [ADR 0246](../../02-DECISIONS/0246-a-seats-new-verb-is-promised-before-it-is-required.md), from [issue 298](../../04-ISSUES/298-a-new-seat-verb-deadlocks-across-two-repositories/00-report.md)). +## 11. The machine under the modules: its hardware + +([ADR 0256](../../02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md).) +Every module on a machine runs on its hardware, and none of them owns it. The `sensors` module reads its +condition on every machine it is assigned to. It holds no seat, and states a fault through the module +health of this design, not yet through the machine's own judgement as §10 does for the network. + +``` + sensors on one machine node-engine controller + ┌──────────────────────────────────────────────┐ + │ drive reader (root) each drive's health, │ unit ──► liveness + readiness + │ never one asleep; each │ + │ read bounded, or said │ + │ fan watch fans that ran since boot │ tool ──► asks sensors_health ──► module.sensors..unhealthy + │ │ every 60 s, 2 looks (the why is its evidence) + │ tools (operator) /sys and the two files; │ + │ read-only │ + └──────────────────────────────────────────────┘ +``` + +- **Unhealthy means the hardware is failing, measured against its own thresholds:** a CPU, GPU, drive or + zone temperature at or above its own critical value (a board chip's is shown, never judged); a drive that + says it is failing (a SATA drive's verdict, a pre-failure attribute failing now, an NVMe critical warning + bit other than temperature); a fan that ran since boot and reads 0 while a CPU sensor, a thermal zone or a + sensor on its own non-board chip is at or above its own maximum; memory that reports an uncorrectable + error. **Or that the module's own processes are stale:** a file five minutes old, a drive's read stuck + past its bound, a drive not looked at for twice its cadence. HIGH readings, fan notes and a drive asleep + for a day are said by the tools and raise nothing. +- **One reader, never waking a drive that can say it sleeps:** the module's drive reader alone reads the + drives (no `smartd`), a spinning drive hourly, because a read can restart a standby timer the mesh cannot + yet see. +- **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no + SMART drive is healthy. +- **The cost of using module health for the machine's fault** is that the condition names the module, and a + standing fault holds the module's own builds at their gate. The machine judging its own hardware, as it + judges its network in §10, is the follow-up that removes both. + ## Phases | Phase | Repository | Delivers | Done when | @@ -304,6 +342,7 @@ the machine itself, beside its liveness looks, and says it in the same statement | D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine | | E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses | | F — the machine's network | `mesh-host` (node-engine), `mesh-controller`, `mesh-catalog` | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile | +| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, the drive reader and the fan watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy | Phases A and B may be built together; A is live first, because it judges without a single declaration.