ADR 0256: one module reads the machine's hardware, so a failing drive or fan is raised, not found dead
A machine's temperatures, fans and drive health were read by the laptop's vendor module only; the machines' 20 drives were watched by nothing. The sensors module reads them on every machine, holds no seat until a caller must reach it by role, and says a hardware fault through ADR 0240's health until the machine judges its own hardware. To-be 48 gains §11 and phase G.
This commit is contained in:
jochen
+172
@@ -0,0 +1,172 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-10-08
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md
|
||||
---
|
||||
|
||||
# 256. A machine's hardware condition is read by one module on every machine, and a fault is said through its health
|
||||
|
||||
## Context
|
||||
|
||||
The operator set the boundary on 2026-10-08, in three sentences:
|
||||
|
||||
- **What hardware a machine has is a node-engine fact.**
|
||||
- **A module holding a generic role reads and provides it.**
|
||||
- **A vendor module adds only vendor-specific extras.**
|
||||
|
||||
The `power` module already holds the battery and the power draw (`node-power`). This record is about
|
||||
everything else a Linux machine reports about the condition of its own hardware: temperatures, fans, drive
|
||||
health and the kernel's count of hardware errors.
|
||||
|
||||
What the mesh had on 2026-10-08, measured on the catalogue's main branch and on the live mesh:
|
||||
|
||||
- **One of 116 catalogue modules reads a temperature, and it is the laptop's vendor module.** It reads
|
||||
every hwmon chip for its `thermals` tool. The workstation and the servers have no vendor module, so the
|
||||
mesh cannot say how hot they are.
|
||||
- **No module reads a drive's health.** No manifest declares `smartmontools`, and no code calls `smartctl`.
|
||||
`disk-load` reads how busy a disk is, not whether it is failing. The machines hold 20 drives between
|
||||
them: the home server has twelve spinning drives and two NVMe drives, the anchor has two spinning drives
|
||||
in a mirror, the workstation has one spinning drive and two NVMe drives, and the laptop has one NVMe drive
|
||||
(`disk-load.disk_devices` on each machine). The media on the home server is data the operator counts as
|
||||
irreplaceable.
|
||||
- **The hardware gives its own thresholds, and not always sane ones.** On the laptop, nine hwmon chips and
|
||||
one thermal zone answer. The NVMe drive's composite sensor reports its own maximum and critical
|
||||
temperatures (83.85 °C and 87.85 °C). Its other two sensors report a maximum of 65261.85 °C, which is
|
||||
the format's largest value and means "none". The CPU's sensor reports no threshold at all. The thermal
|
||||
zone gives trip points (`passive`, `hot`, `critical`).
|
||||
- **Drive health needs root.** The NVMe controller's device is readable by root only, and so are SATA
|
||||
drives' SMART commands. The tool runner runs as the operator account (ADR 0175 §4).
|
||||
- **A module's health is the only way a module raises a condition.** ADR 0240 judges a module's
|
||||
long-running resources. A `tool` check is allowed only beside a check of another kind on the same
|
||||
module, because a module never judges itself alone (rule 2). A module that runs nothing long-lived
|
||||
declares no health. A failing check raises `module.<module>.<machine>.unhealthy`. Its summary names the
|
||||
check, and what the check found is kept as evidence (to-be 48 §4).
|
||||
|
||||
**Checked against GENESIS.** *The mesh notices when something is wrong before you do*
|
||||
([effect](../00-META/effect.md)): a drive that is failing or a fan that has stopped is the kind of fault
|
||||
nobody notices until it costs data or hardware. *Evidence over assertion*: each threshold is the
|
||||
hardware's own, not a number the mesh chose. *Failure must be loud*: a sensor that cannot be read is said
|
||||
as not read, never as cool. Nothing here conflicts with GENESIS.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**Where the reading lives.**
|
||||
|
||||
1. *In each vendor module.* Rejected: it breaks the boundary, and the workstation and the servers have no
|
||||
vendor module.
|
||||
2. *In `disk-load` and `power`.* Rejected: `disk-load` reads load, needs no root and declares no
|
||||
resources, so it can be assigned anywhere. Drive health would give it root, a package and a daemon.
|
||||
`power` holds its own seat, and this is not about power.
|
||||
3. **One module, `sensors`, that reads every generic sensor class and can be assigned to every machine.**
|
||||
Chosen. A class the machine does not have (no fan, no SMART, no memory-error counters) is said as not
|
||||
present, never as an error.
|
||||
|
||||
**Whether it claims a seat.**
|
||||
|
||||
1. *A node seat, such as `node-sensors`, whose verbs are these tools.* Rejected for now. A seat is a role
|
||||
that one module holds on each machine, and callers reach it by the role rather than by the module. That
|
||||
matters when two modules could hold the role, or when another part of the mesh must ask for it without
|
||||
knowing who answers. Neither is true yet. There is one reader, and it only reads. A new seat is defined
|
||||
in the controller, and the verbs of a new seat are required from the start (ADR 0246), so the module
|
||||
would wait on a controller change for nothing a caller needs today.
|
||||
2. **A module with tools, like `disk-load`.** Chosen. The seat is added when a caller has to ask by role:
|
||||
the bar's temperature blocks, or a vendor module's fan policy reading the machine's temperatures. Then
|
||||
these tools become its verbs, as `disk-load`'s README already says for the storage tools.
|
||||
|
||||
**How a fault becomes a condition.**
|
||||
|
||||
1. *The node-engine judges the machine's hardware itself, beside its network (ADR 0241), and raises
|
||||
`machine.<m>.hardware`.* This is the right long-term home, because the hardware is the machine's, as its
|
||||
network is, and a machine's fault must not be pinned on a module (issue 281). It needs a change to the
|
||||
node-engine and to the controller. It is not chosen now and is left as the follow-up below.
|
||||
2. *An event, as `memory-pressure` emits `pressure.high`.* Rejected: an event is a sample, not a state
|
||||
(ADR 0240's own reason), and no event raises a condition.
|
||||
3. **ADR 0240's health, as it stands.** Chosen, because it raises a condition today with no core change.
|
||||
The module gets the two long-running resources that rule 2 needs, and each has a real job:
|
||||
- **the drive daemon (`smartd`)**, judged by the unit's own readiness, which the module does not run
|
||||
itself. It watches every drive between looks and writes what it finds to the journal when it happens;
|
||||
- **the module's watch**, a process of its own bundle run as root, judged by the module's
|
||||
`sensors_health` tool. The watch reads each drive's health every ten minutes, so the tool runner needs
|
||||
no root. It skips a drive that is asleep, so a spinning drive is never woken to be read. It also
|
||||
records which fans have run since the machine started.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **The `sensors` module reads the condition of a machine's hardware.** It only reads. It has read-only
|
||||
tools for:
|
||||
- temperatures (hwmon and thermal zones, each with its label and its own thresholds);
|
||||
- fans;
|
||||
- drives (NVMe health log, SATA SMART);
|
||||
- the kernel's hardware-error counters (memory errors from EDAC, PCIe errors);
|
||||
- a summary that says one line per finding.
|
||||
|
||||
A sensor class the machine lacks is said as not present. A threshold the hardware gives as its format's
|
||||
largest value, or as nothing, is no threshold, and the reading is said without a flag.
|
||||
2. **Every flag is measured against the hardware's own threshold.** A temperature is HIGH at its own
|
||||
maximum (or the zone's `hot` or `passive` trip). It is CRITICAL at its own critical value (or the
|
||||
zone's `critical` trip). The mesh adds no threshold of its own. A sensor that reports none is never
|
||||
flagged.
|
||||
3. **It holds no seat.** It is a module with tools, assignable to every machine. A seat is added under ADR
|
||||
0246 when a caller has to reach it by role.
|
||||
4. **It declares its health under ADR 0240.** `smartd` is judged by its unit. The watch is judged by
|
||||
`sensors_health`, which is unhealthy when:
|
||||
- a temperature is at or above its own critical threshold;
|
||||
- a drive says it is failing: SMART's overall verdict, an attribute failing now, or an NVMe critical
|
||||
warning bit;
|
||||
- a fan that ran since the machine started reads 0 while a temperature is at or above its own maximum;
|
||||
- memory reports an uncorrectable error.
|
||||
|
||||
What the tool answers as `why` is one plain sentence for the operator, such as "The drive nvme0
|
||||
(Samsung 990 PRO) says its spare capacity is below its threshold."
|
||||
5. **A vendor module keeps only what is its vendor's.** For example, the laptop's fan curves and
|
||||
performance profiles. Its reading of the generic sensors moves to this module when that module is
|
||||
next changed.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every machine can say how hot it is and whether its drives and fans are well. A failing drive on the
|
||||
home server is raised as a condition instead of being found when it dies.
|
||||
- **The condition names the module, not the machine.** A hardware fault on the home server is raised as
|
||||
`module.sensors.<machine>.unhealthy`, and its summary names the failing check. The sentence that says
|
||||
what is wrong is in the condition's evidence. The plain words of ADR 0253 can carry it to the operator
|
||||
once the module-health wording uses a tool check's `why`.
|
||||
- **A standing hardware fault holds the module's own builds.** A new build of `sensors` sent to a machine
|
||||
with a failing drive is not healthy within the gate's bound, and is put back there (ADR 0236 §2–3). If
|
||||
that machine is the build's first machine, the build reaches no other machine. Option 1 of the third
|
||||
question removes this. Until it is built, the laptop is the right first machine for this module.
|
||||
- The module installs a package (`smartmontools`) and runs a root daemon on every machine it is assigned
|
||||
to. `smartd`'s configuration is written by the module: every drive, never woken from standby, no mail.
|
||||
It also keeps running on a machine with no drive it can watch, such as a virtual machine.
|
||||
- The watch runs as root. It writes one file of readings under the runtime directory, and nothing else.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- **The machine's hardware judged by the node-engine** (third question, option 1), raising a machine
|
||||
condition. Then `sensors` keeps its tools and drops the tool check from its health.
|
||||
- **The module-health wording takes a tool check's `why`** as its explanation (ADR 0253 rule 2).
|
||||
- **Temperature blocks for the bar**, once the generic contract for a module's bar blocks exists.
|
||||
- **The vendor module's `thermals` tool** points at this module.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) and
|
||||
[to-be 48](../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md) §11, which this record adds.
|
||||
- [ADR 0241](0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md)
|
||||
(the machine's network, the shape of the follow-up),
|
||||
[ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md) (a new seat's verbs),
|
||||
[ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)
|
||||
(the gate), [issue 281](../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md).
|
||||
- mesh-catalog `modules/sensors`, beside `modules/disk-load` (its closest relative).
|
||||
|
||||
## How it is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| 1 Reads only, a missing class is not present | the module's tests over fixture trees: a laptop, a server with SATA drives and EDAC, and an empty machine with no sensor at all, each answered without an error; the manifest test that every tool is read-only |
|
||||
| 2 The hardware's own thresholds | tests: a sensor at its maximum is HIGH and at its critical value is CRITICAL; the NVMe's 65261.85 °C maximum and a sensor with no threshold are never flagged |
|
||||
| 3 No seat | the manifest test that `sensors` claims nothing |
|
||||
| 4 Health | `module check` in the catalogue's merge check accepts the two checks; tests of `sensors_health` for each unhealthy case, and that its `why` is one sentence; live, the first machine states both checks healthy, and `sensors_summary` there answers every sensor class it has |
|
||||
| 5 Vendor extras only | review, when the vendor module is next changed |
|
||||
@@ -2,13 +2,14 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog]
|
||||
updated: 2026-10-07
|
||||
updated: 2026-10-08
|
||||
decisions:
|
||||
- 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md
|
||||
- 02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md
|
||||
- 02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md
|
||||
- 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md
|
||||
- 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
|
||||
- 02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md
|
||||
---
|
||||
|
||||
# 48 — A module says how it is healthy
|
||||
@@ -273,6 +274,37 @@ the machine itself, beside its liveness looks, and says it in the same statement
|
||||
[ADR 0246](../../02-DECISIONS/0246-a-seats-new-verb-is-promised-before-it-is-required.md), from
|
||||
[issue 298](../../04-ISSUES/298-a-new-seat-verb-deadlocks-across-two-repositories/00-report.md)).
|
||||
|
||||
## 11. The machine under the modules: its hardware
|
||||
|
||||
([ADR 0256](../../02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md).)
|
||||
Every module on a machine runs on its hardware, and none of them owns it. The `sensors` module reads its
|
||||
condition on every machine it is assigned to. It holds no seat, and states a fault through the module
|
||||
health of this design, not yet through the machine's own judgement as §10 does for the network.
|
||||
|
||||
```
|
||||
sensors on one machine node-engine controller
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ smartd (unit) watches every drive, │ unit ──► liveness + readiness
|
||||
│ never wakes one asleep │
|
||||
│ watch (root) every 10 min each │
|
||||
│ drive's health; fans │ tool ──► asks sensors_health ──► module.sensors.<m>.unhealthy
|
||||
│ that ran since boot │ every 60 s, 2 looks (the why is its evidence)
|
||||
│ tools (operator) /sys and the watch's │
|
||||
│ readings; read-only │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- **Unhealthy means the hardware is failing, measured against its own thresholds:** a temperature at or
|
||||
above its own critical value, a drive that says it is failing (SMART's verdict, an attribute failing now,
|
||||
an NVMe critical warning bit), a fan that ran since boot and reads 0 while a temperature is at or above its
|
||||
own maximum, or memory that reports an uncorrectable error. HIGH readings are said by the tools and raise
|
||||
nothing.
|
||||
- **A class the machine lacks is not present**, never a failure: a virtual machine with no sensor and no
|
||||
SMART drive is healthy.
|
||||
- **The cost of using module health for the machine's fault** is that the condition names the module, and a
|
||||
standing fault holds the module's own builds at their gate. The machine judging its own hardware, as it
|
||||
judges its network in §10, is the follow-up that removes both.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Repository | Delivers | Done when |
|
||||
@@ -285,6 +317,7 @@ the machine itself, beside its liveness looks, and says it in the same statement
|
||||
| D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine |
|
||||
| E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses |
|
||||
| F — the machine's network | `mesh-host` (node-engine), `mesh-controller`, `mesh-catalog` | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile |
|
||||
| G — the machine's hardware | `mesh-catalog` | §11: the `sensors` module, its tools, `smartd` and the watch, both checks declared | its tests pass; `module check` accepts its health; on the first machine both checks are stated healthy |
|
||||
|
||||
Phases A and B may be built together; A is live first, because it judges without a single declaration.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user