novox/hq
6
0
Fork 0
Code Issues Pull Requests 5 Releases Wiki Activity

Merge pull request 'ADR 0256: one module reads a machine's hardware condition, and a fault is said through its health' (#206) from decision/hardware-sensors into main

Browse Source
This commit was merged in pull request #206.
This commit is contained in:
mesh-admin
2026-10-08 13:20:58 +00:00
parent f08342ad50 df41dfff38
commit 4196b711f8
2 changed files with 266 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md
+227
View File
@@ -0,0 +1,227 @@
---
topic: what runs on it
status: accepted
date: 2026-10-08
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md
---
# 256. A machine's hardware condition is read by one module on every machine, and a fault is said through its health
## Context
The operator set the boundary on 2026-10-08, in three sentences:
- **What hardware a machine has is a node-engine fact.**
- **A module holding a generic role reads and provides it.**
- **A vendor module adds only vendor-specific extras.**
The `power` module already holds the battery and the power draw (`node-power`). This record is about
everything else a Linux machine reports about the condition of its own hardware: temperatures, fans, drive
health and the kernel's count of hardware errors.
What the mesh had on 2026-10-08, measured on the catalogue's main branch and on the live mesh:
- **One of 116 catalogue modules reads a temperature, and it is the laptop's vendor module.** It reads
every hwmon chip for its `thermals` tool. The workstation and the servers have no vendor module, so the
mesh cannot say how hot they are.
- **No module reads a drive's health.** No manifest declares `smartmontools`, and no code calls `smartctl`.
`disk-load` reads how busy a disk is, not whether it is failing. The machines hold 20 drives between
them: the home server has twelve spinning drives and two NVMe drives, the anchor has two spinning drives
in a mirror, the workstation has one spinning drive and two NVMe drives, and the laptop has one NVMe drive
(`disk-load.disk_devices` on each machine). The media on the home server is data the operator counts as
irreplaceable.
- **The hardware gives its own thresholds, and not always sane ones.** On the laptop, nine hwmon chips and
one thermal zone answer. The NVMe drive's composite sensor reports its own maximum and critical
temperatures (83.85 °C and 87.85 °C). Its other two sensors report a maximum of 65261.85 °C, which is
the format's largest value and means "none". The CPU's sensor reports no threshold at all. The thermal
zone gives trip points (`passive`, `hot`, `critical`).
- **Drive health needs root.** The NVMe controller's device is readable by root only, and so are SATA
drives' SMART commands. The tool runner runs as the operator account (ADR 0175 §4).
- **A module's health is the only way a module raises a condition.** ADR 0240 judges a module's
long-running resources. A `tool` check is allowed only beside a check of another kind on the same
module, because a module never judges itself alone (rule 2). A module that runs nothing long-lived
declares no health. A failing check raises `module.<module>.<machine>.unhealthy`. Its summary names the
check, and what the check found is kept as evidence (to-be 48 §4).
**Checked against GENESIS.** *The mesh notices when something is wrong before you do*
([effect](../00-META/effect.md)): a drive that is failing or a fan that has stopped is the kind of fault
nobody notices until it costs data or hardware. *Evidence over assertion*: each threshold is the
hardware's own, not a number the mesh chose. *Failure must be loud*: a sensor that cannot be read is said
as not read, never as cool. Nothing here conflicts with GENESIS.
## Considered Options
**Where the reading lives.**
1. *In each vendor module.* Rejected: it breaks the boundary, and the workstation and the servers have no
vendor module.
2. *In `disk-load` and `power`.* Rejected: `disk-load` reads load, needs no root and declares no
resources, so it can be assigned anywhere. Drive health would give it root, a package and a daemon.
`power` holds its own seat, and this is not about power.
3. **One module, `sensors`, that reads every generic sensor class and can be assigned to every machine.**
Chosen. A class the machine does not have (no fan, no SMART, no memory-error counters) is said as not
present, never as an error.
**Whether it claims a seat.**
1. *A node seat, such as `node-sensors`, whose verbs are these tools.* Rejected for now. A seat is a role
that one module holds on each machine, and callers reach it by the role rather than by the module. That
matters when two modules could hold the role, or when another part of the mesh must ask for it without
knowing who answers. Neither is true yet. There is one reader, and it only reads. A new seat is defined
in the controller, and the verbs of a new seat are required from the start (ADR 0246), so the module
would wait on a controller change for nothing a caller needs today.
2. **A module with tools, like `disk-load`.** Chosen. The seat is added when a caller has to ask by role:
the bar's temperature blocks, or a vendor module's fan policy reading the machine's temperatures. Then
these tools become its verbs, as `disk-load`'s README already says for the storage tools.
**How a fault becomes a condition.**
1. *The node-engine judges the machine's hardware itself, beside its network (ADR 0241), and raises
`machine.<m>.hardware`.* This is the right long-term home, because the hardware is the machine's, as its
network is, and a machine's fault must not be pinned on a module (issue 281). It needs a change to the
node-engine and to the controller. It is not chosen now and is left as the follow-up below.
2. *An event, as `memory-pressure` emits `pressure.high`.* Rejected: an event is a sample, not a state
(ADR 0240's own reason), and no event raises a condition.
3. **ADR 0240's health, as it stands.** Chosen, because it raises a condition today with no core change.
Rule 2 needs a check of another kind beside the tool check, so the module runs two long-running
processes of its own bundle, each with a job of its own and its own file:
- **the drive reader**, run as root and judged by the unit's own readiness, which the module does not
run itself. It reads each drive's health, so the tool runner needs no root. A drive read that never
ends (a drive or a controller that stopped answering) cannot stop the fans being looked at;
- **the fan watch**, judged by the module's `sensors_health` tool. It records which fans have run
since the machine started.
*Running `smartd` beside it* was the first choice and is rejected. `smartd` reads every drive when it
starts or restarts, which can spin a sleeping drive up. Its `-n standby` applies to ATA drives only, so
it polls a SAS drive in standby anyway. And as a second reader on its own schedule, it reads each drive
at another phase of the hour than the module does, which shortens the idle stretch a drive needs to
spin down. One reader, the module's, avoids all three.
## Decision
1. **The `sensors` module reads the condition of a machine's hardware.** It only reads. It has read-only
tools for:
- temperatures (hwmon and thermal zones, each with its label and its own thresholds);
- fans;
- drives (NVMe health log, SATA SMART);
- the kernel's hardware-error counters (memory errors from EDAC, PCIe errors);
- a summary that says one line per finding.
A sensor class the machine lacks is said as not present. A threshold the hardware gives as its format's
largest value, or as nothing, is no threshold, and the reading is said without a flag.
2. **Every flag is measured against the hardware's own threshold.** A temperature is HIGH at its own
maximum (or the zone's `hot` or `passive` trip). It is CRITICAL at its own critical value (or the
zone's `critical` trip). The mesh adds no threshold of its own. A sensor that reports none is never
flagged. 0 °C is a reading; 127 and 127.5 °C are not, because they are what a board chip's input with
nothing on it answers.
3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR
0246 when a caller has to reach it by role.
4. **It declares its health under ADR 0240.** The drive reader is judged by its unit. The fan watch is
judged by `sensors_health`, which is unhealthy when, and only when:
- a CPU, GPU, drive or thermal-zone temperature is at or above its own critical value. A board chip's
(a Super I/O's) critical value is shown as CRITICAL by the tools and never makes the module
unhealthy, because its inputs and limits are whatever the firmware left;
- a drive says it is failing: a SATA drive's overall SMART verdict, a pre-failure attribute failing
now, or an NVMe critical warning bit other than temperature (spare below threshold, reliability
degraded, read-only, backup failed). An NVMe drive is judged by its bits, not by the overall verdict
smartctl derives from them, and by its health log when smartctl gives no verdict at all. The
temperature bit alone, an old-age attribute failing now (a temperature, a count of hours) and an
attribute that failed in the past are HIGH;
- a fan that ran since the machine started reads 0 RPM now while a CPU sensor, a thermal zone, or a
sensor on the fan's own chip is at or above its own maximum, the last only when that chip is not a
board chip. A board chip's maximum is no more trusted than its critical value. A hot drive or network
adapter says nothing of a fan. A fan under its chip's minimum, or with its chip's alarm bit set, is
said in a note and raises nothing: a zero-fan GPU is under its minimum by design, and a board chip
latches its alarm;
- memory reports an uncorrectable error;
- the module's own processes cannot be trusted now: either process's file is missing, from an earlier
boot, or not written for five minutes (each writes every 30 seconds); a drive's read has been in
flight longer than any read can take (smartctl's 20 s bound, 2 s for its output, and one write of
slack), which is a read stuck in the kernel, since such a read cannot be ended; or a drive has not
been looked at for twice its cadence.
A drive found asleep at every look is said as asleep since when, and HIGH once that is a day, because
its health has not been read for that long and a drive that stopped answering can look asleep. It
never makes the module unhealthy on its own, at any length: a drive kept for backups may sleep for
weeks, and its health being unread is said, not raised.
What the tool answers as `why` is one plain sentence per finding, for the operator, such as "The drive
nvme0n1 (Samsung SSD 990 PRO 2TB) says it is failing: its spare capacity is below its threshold."
5. **A vendor module keeps only what is its vendor's.** For example, the laptop's fan curves and
performance profiles. Its reading of the generic sensors moves to this module when that module is
next changed.
## Consequences
- Every machine can say how hot it is and whether its drives and fans are well. A failing drive on the
home server is raised as a condition instead of being found when it dies.
- **The condition names the module, not the machine.** A hardware fault on the home server is raised as
`module.sensors.<machine>.unhealthy`, and its summary names the failing check. The sentence that says
what is wrong is in the condition's evidence. The plain words of ADR 0253 can carry it to the operator
once the module-health wording uses a tool check's `why`.
- **A standing hardware fault holds the module's own builds.** A new build of `sensors` sent to a machine
with a failing drive is not healthy within the gate's bound, and is put back there (ADR 0236 §2–3). If
that machine is the build's first machine, the build reaches no other machine. Option 1 of the third
question removes this. Until it is built, the laptop is the right first machine for this module.
- The module installs a package (`smartmontools`) for `smartctl`, and runs two processes of its own as
root, each writing one file under the runtime directory and nothing else. It does not run `smartd`.
- **A drive is not woken by the module, as far as the drive can say so.** Each read is asked with
`-n standby`, and smartctl stops without reading when the drive says it is in standby or asleep. An ATA
drive says it by its power mode (STANDBY, STANDBY_Y, SLEEP). A SAS drive, or one the operating system
powered down, says it in smartctl's own words, "Device is in <mode> mode, exit(2)", where a SAS mode
holds spaces (STANDBY BY TIMER, STANDBY_Y BY COMMAND, LOW POWER); the module takes every such mode, and
an idle mode never stops smartctl. smartctl is told the drive's kind (`-d nvme`, `sat` or `scsi`) where
it is known, so it does not probe a drive to find out. A drive behind USB is always told `sat`: one
behind a USB-to-NVMe enclosure or a vendor's own bridge does not answer to it, and is said as not read.
A drive that cannot say whether it is asleep (a USB bridge that does not pass the question on) is not
read at all, and is said as not read. What remains: a drive of a kind the module cannot tell (a virtual
disk, a memory card) is read with smartctl's own detection.
- **Reading a drive may keep it from spinning down.** Reading SMART from an awake drive can restart its
standby timer, so a drive read more often than its timer is set for never reaches standby. Whether the
servers' drives are set to spin down is not known: no mesh tool reads a drive's standby timer or power
mode, and no module sets one. On 2026-10-08, `disk-load` showed the anchor's two drives written to
continuously (they hold its root, so they never idle) and the home server's twelve spinning drives
without I/O over a ten-second sample, which says nothing of their timers. So the module, the only
reader, reads a spinning drive once an hour, timed from its last look. A drive whose timer is set
longer than an hour can still be kept awake. Reading the power mode and the timer is a follow-up for
this module's tools.
- **A drive that stops answering may look asleep**, and is then HIGH after a day, never unhealthy (rule
4). A drive whose read hangs instead is unhealthy in about two to three minutes: its read is said once
it is past its 52 s bound, and the health check needs two looks 60 s apart to be unhealthy.
- **Known limits of a stuck read.** A smartctl stuck in the kernel cannot be ended, and outlives the
drive reader: a restarted reader starts a new read of that drive, which may stick beside the first. The
reads of all drives run at once, with no cap per controller or bridge; a server with many drives on one
controller asks it all at once, once an hour.
## Follow-ups
- **The machine's hardware judged by the node-engine** (third question, option 1), raising a machine
condition. Then `sensors` keeps its tools and drops the tool check from its health.
- **The module-health wording takes a tool check's `why`** as its explanation (ADR 0253 rule 2).
- **Temperature blocks for the bar**, once the generic contract for a module's bar blocks exists.
- **A drive's power mode and standby timer in `sensors_drives`**, so the cadence above can be set from
what each drive is set to, rather than from a guess that it may be an hour.
- **The vendor module's `thermals` tool** points at this module.
## References
- [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) and
[to-be 48](../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md) §11, which this record adds.
- [ADR 0241](0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md)
(the machine's network, the shape of the follow-up),
[ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md) (a new seat's verbs),
[ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)
(the gate), [issue 281](../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md).
- mesh-catalog `modules/sensors`, beside `modules/disk-load` (its closest relative).
## How it is checked
| Rule | Checked by |
|---|---|
| 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only |
| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged; 0 °C is a reading and 127 °C is not |
| 3 No seat | the manifest test that `sensors` claims nothing |
| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case and for each case rule 4 says raises nothing (the NVMe temperature bit, an old-age attribute, a past failure, a zero-fan GPU under its minimum, a latched alarm, a hot drive beside a stopped fan, a board chip's critical value); tests that a stale or missing file is unhealthy, that a hung `smartctl` is bounded, that a read stuck past its bound is unhealthy and does not stop the other drives, that a drive not looked at for twice its cadence is unhealthy, that a drive asleep for a day is HIGH and healthy, that a SAS drive in standby is asleep, that smartctl is told each drive's kind, and that its `why` is one sentence per finding; the manifest test that no `smartd` runs; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has |
| 5 Vendor extras only | review, when the vendor module is next changed |
03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md
+39
View File
@@ -10,6 +10,7 @@ decisions:
- 02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md
- 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md
- 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
- 02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md
---
# 48 — A module says how it is healthy
@@ -292,6 +293,43 @@ the machine itself, beside its liveness looks, and says it in the same statement
[ADR 0246](../../02-DECISIONS/0246-a-seats-new-verb-is-promised-before-it-is-required.md), from
[issue 298](../../04-ISSUES/298-a-new-seat-verb-deadlocks-across-two-repositories/00-report.md)).
## 11. The machine under the modules: its hardware
([ADR 0256](../../02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md).)
Every module on a machine runs on its hardware, and none of them owns it. The `sensors` module reads its
condition on every machine it is assigned to. It holds no seat, and states a fault through the module
health of this design, not yet through the machine's own judgement as §10 does for the network.
```
sensors on one machine node-engine controller
┌──────────────────────────────────────────────┐
│ drive reader (root) each drive's health, │ unit ──► liveness + readiness
│ never one asleep; each │
│ read bounded, or said │
│ fan watch fans that ran since boot │ tool ──► asks sensors_health ──► module.sensors.<m>.unhealthy
│ │ every 60 s, 2 looks (the why is its evidence)
│ tools (operator) /sys and the two files; │
│ read-only │
└──────────────────────────────────────────────┘
```
- **Unhealthy means the hardware is failing, measured against its own thresholds:** a CPU, GPU, drive or
zone temperature at or above its own critical value (a board chip's is shown, never judged); a drive that
says it is failing (a SATA drive's verdict, a pre-failure attribute failing now, an NVMe critical warning
bit other than temperature); a fan that ran since boot and reads 0 while a CPU sensor, a thermal zone or a
sensor on its own non-board chip is at or above its own maximum; memory that reports an uncorrectable
error. **Or that the module's own processes are stale:** a file five minutes old, a drive's read stuck
past its bound, a drive not looked at for twice its cadence. HIGH readings, fan notes and a drive asleep
for a day are said by the tools and raise nothing.
- **One reader, never waking a drive that can say it sleeps:** the module's drive reader alone reads the
drives (no `smartd`), a spinning drive hourly, because a read can restart a standby timer the mesh cannot
yet see.
- **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no
SMART drive is healthy.
- **The cost of using module health for the machine's fault** is that the condition names the module, and a
standing fault holds the module's own builds at their gate. The machine judging its own hardware, as it
judges its network in §10, is the follow-up that removes both.
## Phases
| Phase | Repository | Delivers | Done when |
@@ -304,6 +342,7 @@ the machine itself, beside its liveness looks, and says it in the same statement
| D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine |
| E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses |
| F — the machine's network | `mesh-host` (node-engine), `mesh-controller`, `mesh-catalog` | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile |
| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, the drive reader and the fan watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy |
Phases A and B may be built together; A is live first, because it judges without a single declaration.