Merge pull request 'sensors: read the machine's hardware condition on every machine (hq ADR 0256)' (#129) from feat/sensors into main
This commit was merged in pull request #129.
This commit is contained in:
@@ -0,0 +1,180 @@
|
||||
# sensors
|
||||
|
||||
The condition of a machine's own hardware: temperatures with the thresholds the hardware gives for
|
||||
itself, fans, each drive's health, and the kernel's counts of memory and PCIe errors (novox/hq ADR
|
||||
0256). It reads, and changes nothing on the hardware. Every flag is measured against the hardware's own
|
||||
threshold, never one the mesh chose. A class the machine does not have is said as not present, never
|
||||
as an error, so the module can be assigned to every machine: a laptop, a desktop, a server, a virtual
|
||||
machine.
|
||||
|
||||
## Why it exists, and why here
|
||||
|
||||
The operator set the boundary on 2026-10-08. What hardware a machine has is a node-engine fact. A module
|
||||
holding a generic role reads and provides it. A vendor module adds only vendor-specific extras.
|
||||
|
||||
Before this module, only the laptop's vendor module read a temperature, and no module read a drive's
|
||||
health. The machines' 20 drives, 15 of them spinning, were watched by nothing. `power` keeps the battery
|
||||
and the power draw. `disk-load` keeps how busy the disks are. This module keeps everything else the
|
||||
kernel says about the hardware's condition.
|
||||
|
||||
It holds no seat. A seat is a role that callers reach without knowing the module, and nothing asks for
|
||||
this role yet. A new seat would also be a controller change first (ADR 0246). When a caller needs to ask
|
||||
by role, such as the bar's temperature blocks or a vendor module's fan policy, these tools become the
|
||||
seat's verbs. The full reasoning is in ADR 0256.
|
||||
|
||||
## What runs
|
||||
|
||||
| resource | what | as |
|
||||
|---|---|---|
|
||||
| `smartmontools` | the package, for `smartctl` | |
|
||||
| `drives` (`sensors drives`) | each drive's health with `/usr/bin/smartctl --json -a -d <kind> -n standby,2,3`: a spinning drive hourly, any other every 10 min, each drive in a goroutine of its own; writes `/run/mesh-sensors/drives.json` every 30 s | root, the node-engine's process; health `unit` |
|
||||
| `watch` (`sensors watch`) | every 30 s which fans turn; writes `/run/mesh-sensors/fans.json` | the node-engine's process; health `tool` (`sensors_health`) |
|
||||
| the tools | read `/sys`, `/proc` and the two files, each once per call | the operator account, in the tool runner |
|
||||
|
||||
Both files are 0644 in a 0755 directory. The tools need no root, because the drive reader has already read
|
||||
what only root may read.
|
||||
|
||||
**The module is the only reader of the drives; it does not run smartd.** smartd reads every drive when it
|
||||
starts or restarts, which can spin a sleeping drive up. Its `-n standby` is for ATA drives only, so it
|
||||
polls a SAS drive in standby anyway. And as a second reader on its own schedule, it would read each drive
|
||||
at another phase of the hour, shortening the idle stretch a drive needs to spin down.
|
||||
|
||||
**No drive holds up another, and a stuck one is said.** Each smartctl is bounded by 20 s, plus 2 s for a
|
||||
child that keeps its output open. A smartctl stuck in the kernel cannot be ended at all. Its read stays in
|
||||
flight, marked since when, and the health check says so once it is older than any read can take (20 s +
|
||||
2 s + 30 s). Each drive is read in its own goroutine, so the other drives go on being read. The fans are
|
||||
another process, and go on whatever the drives do.
|
||||
|
||||
**A drive is not woken, as far as it can say so, and is read seldom enough to sleep.**
|
||||
- `-n standby,2,3` makes smartctl stop with status 2 when the drive is in standby or asleep. The drive
|
||||
reader then keeps the reading before, and since when the drive has been asleep. A drive counts as asleep
|
||||
on status 2 with either an ATA power mode of STANDBY or SLEEP, or smartctl's own words "Device is in
|
||||
<mode> mode, exit(2)" (a SAS drive, whose modes hold spaces such as STANDBY BY TIMER or LOW POWER, or
|
||||
one the operating system powered down). An open failure is neither, whatever its words.
|
||||
- It stops with status 3 when the drive cannot say (a USB bridge that does not pass CHECK POWER MODE on).
|
||||
Such a drive is not read, and is said as not read.
|
||||
- smartctl is told the drive's kind (`-d nvme`, `-d sat` for an ATA drive or any drive behind USB,
|
||||
`-d scsi`) so it does not probe. A drive behind a USB-to-NVMe enclosure or a vendor's own bridge does not
|
||||
answer to `sat`, and is said as not read. A drive of a kind the module cannot tell (virtio, MMC) is read
|
||||
with smartctl's own detection.
|
||||
- Known limits: a smartctl stuck in the kernel outlives the drive reader, and a restarted reader starts a
|
||||
new read beside it; all drives are read at once, with no cap per controller or bridge.
|
||||
- Reading SMART from an awake drive can restart its standby timer, so a drive read more often than its
|
||||
timer is set for never spins down. No mesh tool reads a drive's standby timer, so whether the servers'
|
||||
drives spin down is not known. A spinning drive is therefore read once an hour, timed from its last
|
||||
look. A drive whose timer is set longer than an hour may still be kept awake.
|
||||
|
||||
## Tools
|
||||
|
||||
All six only read.
|
||||
|
||||
| tool | what | replaces |
|
||||
|---|---|---|
|
||||
| `sensors_summary` | one line per finding, flagged CRITICAL, HIGH, FAILING or STOPPED; then one line per class: temperatures with the hottest, fans, drives, memory and PCIe errors | `sensors`, `smartctl -H` |
|
||||
| `sensors_temperatures` | every thermal zone with its trip points, and every hwmon temperature with its label, its kind (cpu, gpu, drive, board, network), the drive it sits in, and its own maximum and critical values | `sensors` |
|
||||
| `sensors_fans` | every fan's RPM, its chip's minimum and alarm, and whether the fan watch saw it turn since boot | `sensors` |
|
||||
| `sensors_drives` | each drive: model, serial, size, whether it spins, the SMART verdict; for NVMe the critical warning bits, spare, endurance used and media errors; for SATA the reallocated, pending and unreadable sectors, the error log and the last self-test | `smartctl -a`, `nvme smart-log` |
|
||||
| `sensors_faults` | EDAC memory errors per controller and module, PCIe AER errors per device, and how often an Intel CPU was slowed for heat | `edac-util` |
|
||||
| `sensors_health` | the module's health check: healthy or not, with `why` | |
|
||||
|
||||
### The flags
|
||||
|
||||
| flag | when |
|
||||
|---|---|
|
||||
| temperature HIGH | at or above its own maximum, or a zone's `hot` trip (without one, its lowest `passive` trip) |
|
||||
| temperature CRITICAL | at or above its own critical or emergency value, or a zone's `critical` trip |
|
||||
| drive FAILING | SMART's overall verdict failed (SATA), a pre-failure attribute is failing now, or an NVMe critical warning bit other than temperature is set |
|
||||
| drive HIGH | an old-age attribute failing now; an attribute that failed in the past; the NVMe temperature bit alone; reallocated, pending or unreadable sectors; NVMe media errors or 90 % of rated endurance used; errors in the error log; a last self-test that did not pass |
|
||||
| fan STOPPED | it ran since boot, reads 0 now, and a CPU sensor, a thermal zone, or a sensor on its own chip (not a board chip) is at or above its own maximum |
|
||||
| fan note | under its chip's minimum, or its chip's alarm bit set: said, never flagged |
|
||||
| drive asleep HIGH | asleep at every look for a day: its health has not been read since, and a drive that stopped answering can look asleep |
|
||||
| memory HIGH / CRITICAL | corrected / uncorrectable errors since boot |
|
||||
| PCIe HIGH | fatal or non-fatal errors since boot |
|
||||
|
||||
**What is no threshold, and no reading.**
|
||||
- A threshold given as the format's largest value is no threshold. The laptop's NVMe drive gives
|
||||
65261.85 °C as the maximum of two of its sensors.
|
||||
- A threshold of 0 is no threshold either, and neither is anything outside −40 to 200 °C. The reading
|
||||
is then said without a flag.
|
||||
- A CPU sensor that gives no threshold, such as AMD's `Tctl`, is never flagged.
|
||||
- 0 °C is a reading. 127 and 127.5 °C are not: they are what a board chip's input with nothing on it
|
||||
answers (a phantom AUXTIN). Such a sensor is said as not read.
|
||||
|
||||
**Why a fan at 0, a minimum or an alarm is not enough.** Laptop fans and GPU fans in zero-fan mode stop on
|
||||
purpose when the machine is cool, under the minimum their chip gives. A board chip latches an alarm bit,
|
||||
and a header with nothing on it reads 0. Only a fan that ran and stopped while what it cools is hot is a
|
||||
fault. A hot drive or network adapter says nothing of a fan, and neither does a board chip's own sensor:
|
||||
its maximum is whatever the firmware left, as its critical value is.
|
||||
|
||||
## Health
|
||||
|
||||
The module declares its health under ADR 0240:
|
||||
|
||||
- the drive reader has the `unit` kind: the unit's own readiness, which the module does not judge itself;
|
||||
- the fan watch has the `tool` kind: `sensors_health`, every 60 s, unhealthy after 2 looks in a row, with
|
||||
a 60 s grace.
|
||||
|
||||
`sensors_health` is unhealthy when:
|
||||
|
||||
- a CPU, GPU, drive or zone temperature is at or above its own critical value. A board chip's (Super I/O)
|
||||
critical value is shown CRITICAL by the tools and never makes the module unhealthy;
|
||||
- a drive says it is failing (the FAILING row above);
|
||||
- a fan is STOPPED (the row above);
|
||||
- memory reports an uncorrectable error;
|
||||
- the module's own processes cannot be trusted: either file missing, from an earlier boot or five minutes
|
||||
old; a drive's read in flight past its bound; a drive not looked at for twice its cadence.
|
||||
|
||||
Its `why` is one plain sentence per finding, for the operator. For example: "The drive sdb (WDC
|
||||
WUH722222ALE6L4) says it is failing: its overall SMART health check failed." HIGH readings, fan notes and
|
||||
a drive asleep raise nothing. **A drive asleep never makes the module unhealthy, at any length** (ADR
|
||||
0256): a drive kept for backups may sleep for weeks. It is HIGH after a day, said with since when.
|
||||
|
||||
The condition this raises is `module.sensors.<machine>.unhealthy`. ADR 0256 records what this costs:
|
||||
|
||||
- the condition names the module, not the machine;
|
||||
- a standing hardware fault holds this module's own new builds at their gate on that machine.
|
||||
|
||||
The machine judging its own hardware, as it judges its network, is the follow-up.
|
||||
|
||||
## Tests
|
||||
|
||||
`go test ./...`. The tests read a tree standing in for `/sys`, `/proc` and `/run`, and smartctl answers
|
||||
as smartmontools 7.5 writes them (its standby answer as its source sets it). There are three machines:
|
||||
|
||||
- a laptop, as the laptop showed itself on 2026-10-08;
|
||||
- a server with ECC memory, three SATA drives read by drivetemp and a board chip with an empty fan header;
|
||||
- a virtual machine with no sensor at all.
|
||||
|
||||
They cover:
|
||||
|
||||
- thresholds taken from the hardware only, and a zone's hwmon copy left out;
|
||||
- HIGH at the maximum and CRITICAL at the critical value; a zone's passive trip as its high;
|
||||
- 0 °C as a reading, a phantom 127 °C as not read, and a board chip's critical value raising nothing;
|
||||
- every class not present on the virtual machine, which is healthy;
|
||||
- a failing SATA drive; an NVMe critical warning; the NVMe temperature bit alone as HIGH; NVMe without
|
||||
`smart_status`; an old-age attribute failing now and a past failure as HIGH;
|
||||
- a missing, stale or earlier-boot reading, and either process's staleness making the module unhealthy;
|
||||
- a fan at rest, a fan that stopped while the board's zone is hot, a hot drive saying nothing of a fan,
|
||||
a hot sensor on the fan's own chip, a board chip's own sensor saying nothing of its fan while a CPU
|
||||
sensor does, an empty header, a zero-fan GPU under its minimum, and a latched alarm;
|
||||
- memory and PCIe errors, and CPU throttling;
|
||||
- the drive reader reading as root without waking a drive: smartctl's real standby answer, a SAS drive's,
|
||||
an open failure that is not sleep, and a bridge without a power mode; asleep since when kept across
|
||||
looks, and HIGH after a day without making the module unhealthy;
|
||||
- smartctl told each drive's kind;
|
||||
- a spinning drive read hourly and a solid-state one every 10 min;
|
||||
- a smartctl that hangs with a child holding its output, bounded; a read stuck past its bound said as
|
||||
unhealthy while the other drives are read; a drive not looked at for twice its cadence;
|
||||
- a restart keeping what the last reader read; the readings directory given its mode;
|
||||
- a drive reader that is not root asking through `sudo -n`, and saying what stops it;
|
||||
- the health answer in the shape the node-engine reads;
|
||||
- the manifest: the tools listed are the tools served, all read-only, no seat claimed, no machine named,
|
||||
the two health checks, and no smartd.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- The machine's hardware judged by the node-engine, raising a machine condition, instead of this
|
||||
module's health (ADR 0256).
|
||||
- The module-health condition carrying the tool check's `why` as its plain words (ADR 0253).
|
||||
- Temperature blocks for the bar, once there is a generic way for a module to provide bar blocks.
|
||||
- The laptop vendor module's `zephyrus_thermals` keeps only what is the vendor's.
|
||||
@@ -0,0 +1,580 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Drives and their health. Which drives the machine has is read from /sys by anyone; what each says of its
|
||||
// own health (the NVMe health log, SATA and SAS SMART) is read by the module's drive reader, as root, with
|
||||
// smartctl, and left in DrivesPath for the tools (novox/hq ADR 0256).
|
||||
|
||||
// The two files the module's processes write: the drive reader's (DrivesPath) and the fan watch's
|
||||
// (FansPath). Each process writes only its own.
|
||||
const (
|
||||
DrivesPath = "/run/mesh-sensors/drives.json"
|
||||
FansPath = "/run/mesh-sensors/fans.json"
|
||||
)
|
||||
|
||||
// AsleepHighAfter is how long a drive may be found asleep at every look before the tools say it as HIGH:
|
||||
// its health has not been read for that long, and a drive that stops answering can look asleep. It never
|
||||
// makes the module unhealthy on its own (ADR 0256): a drive kept for backups may sleep for weeks.
|
||||
const AsleepHighAfter = 24 * time.Hour
|
||||
|
||||
// StaleAfter is how old either process's file may be before the module is unhealthy: each writes its own
|
||||
// every FanEvery, whatever a drive does, so ten writes missed is a process that is hung or gone.
|
||||
const StaleAfter = 5 * time.Minute
|
||||
|
||||
// Smartctl is the program, by its path: the watch runs as root and runs nothing found on a PATH.
|
||||
const Smartctl = "/usr/bin/smartctl"
|
||||
|
||||
// FlagFailing is said beside a drive that says it is failing.
|
||||
const FlagFailing = "FAILING"
|
||||
|
||||
// Readings is one process's file: the drive reader fills Drives, the fan watch FansRan.
|
||||
type Readings struct {
|
||||
BootID string `json:"boot_id"`
|
||||
WrittenAt time.Time `json:"written_at"`
|
||||
Drives map[string]DriveReading `json:"drives"`
|
||||
// FansRan is when the watch last saw each fan turning, by its key (asus/fan1), since the machine started.
|
||||
FansRan map[string]time.Time `json:"fans_ran"`
|
||||
}
|
||||
|
||||
// DriveReading is one drive as smartctl last answered for it.
|
||||
type DriveReading struct {
|
||||
ReadAt time.Time `json:"read_at"`
|
||||
Smartctl json.RawMessage `json:"smartctl,omitempty"`
|
||||
Status int `json:"status"`
|
||||
// Asleep says the drive was in standby at the newest look, so it was not woken and the reading is the
|
||||
// one before.
|
||||
Asleep bool `json:"asleep,omitempty"`
|
||||
NotRead string `json:"not_read,omitempty"`
|
||||
// LookedAt is the newest look at the drive, asleep or not: what the next look is timed from.
|
||||
LookedAt time.Time `json:"looked_at"`
|
||||
// AsleepSince is the first of the looks in a row that found the drive asleep.
|
||||
AsleepSince *time.Time `json:"asleep_since,omitempty"`
|
||||
// InFlightSince is when the read under way began: a read that never ends (a drive or a controller that
|
||||
// stopped answering, a smartctl stuck in the kernel) is seen by how old this is.
|
||||
InFlightSince *time.Time `json:"in_flight_since,omitempty"`
|
||||
}
|
||||
|
||||
// Drive is one drive, as the tools say it.
|
||||
type Drive struct {
|
||||
Device string `json:"device"`
|
||||
Model string `json:"model,omitempty"`
|
||||
Serial string `json:"serial,omitempty"`
|
||||
Firmware string `json:"firmware,omitempty"`
|
||||
SizeGiB float64 `json:"size_gib"`
|
||||
Rotational bool `json:"rotational"`
|
||||
Transport string `json:"transport"`
|
||||
// Health is "passed", "failing" or "not read".
|
||||
Health string `json:"health"`
|
||||
Flag string `json:"flag,omitempty"`
|
||||
Findings []string `json:"findings,omitempty"`
|
||||
|
||||
TemperatureC *int64 `json:"temperature_c,omitempty"`
|
||||
PowerOnHours *int64 `json:"power_on_hours,omitempty"`
|
||||
PercentageUsed *int64 `json:"percentage_used,omitempty"`
|
||||
AvailableSpare *int64 `json:"available_spare_pct,omitempty"`
|
||||
SpareThreshold *int64 `json:"available_spare_threshold_pct,omitempty"`
|
||||
MediaErrors *int64 `json:"media_errors,omitempty"`
|
||||
CriticalWarning *int64 `json:"critical_warning,omitempty"`
|
||||
Reallocated *int64 `json:"reallocated_sectors,omitempty"`
|
||||
Pending *int64 `json:"pending_sectors,omitempty"`
|
||||
Uncorrectable *int64 `json:"offline_uncorrectable,omitempty"`
|
||||
ErrorLogCount *int64 `json:"error_log_count,omitempty"`
|
||||
LastSelfTest string `json:"last_self_test,omitempty"`
|
||||
ReadAt string `json:"read_at,omitempty"`
|
||||
Asleep bool `json:"asleep_at_last_look,omitempty"`
|
||||
AsleepSince string `json:"asleep_since,omitempty"`
|
||||
NotRead string `json:"not_read,omitempty"`
|
||||
failingSentences []string // why it is failing, each a clause for the operator
|
||||
}
|
||||
|
||||
// Name is how a person reads which drive this is.
|
||||
func (d Drive) Name() string {
|
||||
if d.Model != "" {
|
||||
return fmt.Sprintf("the drive %s (%s)", d.Device, d.Model)
|
||||
}
|
||||
return "the drive " + d.Device
|
||||
}
|
||||
|
||||
// notDrives are block devices that are not a drive of their own: loop files, memory, device-mapper and
|
||||
// software RAID volumes (built on drives that are listed themselves), optical and floppy drives, network
|
||||
// block devices.
|
||||
var notDrives = []string{"loop", "ram", "zram", "dm-", "md", "sr", "fd", "nbd", "rbd"}
|
||||
|
||||
// BlockDrives lists the machine's drives from /sys/block: whole devices backed by hardware.
|
||||
func (m *Machine) BlockDrives() []Drive {
|
||||
var out []Drive
|
||||
for _, b := range m.names("/sys/block") {
|
||||
skip := strings.HasSuffix(b, "boot0") || strings.HasSuffix(b, "boot1") || strings.HasSuffix(b, "rpmb")
|
||||
for _, p := range notDrives {
|
||||
skip = skip || strings.HasPrefix(b, p)
|
||||
}
|
||||
if skip || !m.exists("/sys/block/"+b+"/device") {
|
||||
continue
|
||||
}
|
||||
dir := "/sys/block/" + b
|
||||
d := Drive{Device: b, Model: m.read(dir + "/device/model"), Transport: transport(b, m.read(dir+"/device/vendor"))}
|
||||
if where, err := filepath.EvalSymlinks(m.path(dir)); err == nil && strings.Contains(where, "/usb") {
|
||||
d.Transport = "usb"
|
||||
}
|
||||
if s, ok := m.readInt(dir + "/size"); ok {
|
||||
d.SizeGiB = round1(float64(s) * 512 / (1 << 30))
|
||||
}
|
||||
if r, ok := m.readInt(dir + "/queue/rotational"); ok {
|
||||
d.Rotational = r == 1
|
||||
}
|
||||
out = append(out, d)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func transport(name, vendor string) string {
|
||||
switch {
|
||||
case strings.HasPrefix(name, "nvme"):
|
||||
return "nvme"
|
||||
case strings.HasPrefix(name, "vd"), strings.HasPrefix(name, "xvd"):
|
||||
return "virtual"
|
||||
case strings.HasPrefix(name, "mmcblk"):
|
||||
return "mmc"
|
||||
case strings.TrimSpace(vendor) == "ATA":
|
||||
return "ata"
|
||||
}
|
||||
return "scsi"
|
||||
}
|
||||
|
||||
// who names the process that writes a file, as a person reads it.
|
||||
func who(path string) string {
|
||||
if path == DrivesPath {
|
||||
return "the module's drive reader"
|
||||
}
|
||||
return "the module's fan watch"
|
||||
}
|
||||
|
||||
// ReadReadings is one process's file, or why there is none to use.
|
||||
func (m *Machine) ReadReadings(path string) (*Readings, string) {
|
||||
raw, err := os.ReadFile(m.path(path))
|
||||
if err != nil {
|
||||
return nil, who(path) + " has left no readings (is it running?)"
|
||||
}
|
||||
var r Readings
|
||||
if err := json.Unmarshal(raw, &r); err != nil {
|
||||
return nil, who(path) + " left readings this tool cannot read: " + err.Error()
|
||||
}
|
||||
if id := m.bootID(); id != "" && r.BootID != "" && r.BootID != id {
|
||||
return nil, who(path) + " has not written anything since the machine started"
|
||||
}
|
||||
return &r, ""
|
||||
}
|
||||
|
||||
// Drives is every drive with its health as the watch's readings say it, judged. r is nil, with why, when
|
||||
// there are none to use.
|
||||
func (m *Machine) Drives(r *Readings, why string) ([]Drive, string) {
|
||||
drives := m.BlockDrives()
|
||||
note := why
|
||||
if r != nil && m.Now().Sub(r.WrittenAt) > StaleAfter {
|
||||
note = fmt.Sprintf("the module's drive reader last wrote %s ago; it may not be running", ago(m.Now().Sub(r.WrittenAt)))
|
||||
}
|
||||
for i := range drives {
|
||||
d := &drives[i]
|
||||
if r == nil {
|
||||
d.Health, d.NotRead = "not read", why
|
||||
continue
|
||||
}
|
||||
reading, ok := r.Drives[d.Device]
|
||||
if !ok {
|
||||
d.Health, d.NotRead = "not read", "the module's watch has not read this drive yet"
|
||||
continue
|
||||
}
|
||||
judgeDrive(d, reading, m.Now())
|
||||
}
|
||||
return drives, note
|
||||
}
|
||||
|
||||
func ago(d time.Duration) string {
|
||||
switch {
|
||||
case d < time.Minute:
|
||||
return fmt.Sprintf("%d s", int(d.Seconds()))
|
||||
case d < 2*time.Hour:
|
||||
return fmt.Sprintf("%d min", int(d.Minutes()))
|
||||
}
|
||||
return fmt.Sprintf("%.1f h", d.Hours())
|
||||
}
|
||||
|
||||
// smartctl's JSON, the parts the module reads (smartmontools 7).
|
||||
type smartJSON struct {
|
||||
Smartctl struct {
|
||||
Messages []struct {
|
||||
String string `json:"string"`
|
||||
Severity string `json:"severity"`
|
||||
} `json:"messages"`
|
||||
} `json:"smartctl"`
|
||||
ModelName string `json:"model_name"`
|
||||
SerialNumber string `json:"serial_number"`
|
||||
FirmwareVersion string `json:"firmware_version"`
|
||||
SmartSupport *struct {
|
||||
Available bool `json:"available"`
|
||||
Enabled bool `json:"enabled"`
|
||||
} `json:"smart_support"`
|
||||
SmartStatus *struct {
|
||||
Passed bool `json:"passed"`
|
||||
} `json:"smart_status"`
|
||||
PowerMode *struct {
|
||||
Name string `json:"name"`
|
||||
} `json:"power_mode"`
|
||||
NVMe *struct {
|
||||
CriticalWarning int64 `json:"critical_warning"`
|
||||
AvailableSpare int64 `json:"available_spare"`
|
||||
AvailableSpareThreshold int64 `json:"available_spare_threshold"`
|
||||
PercentageUsed int64 `json:"percentage_used"`
|
||||
MediaErrors int64 `json:"media_errors"`
|
||||
PowerOnHours int64 `json:"power_on_hours"`
|
||||
} `json:"nvme_smart_health_information_log"`
|
||||
ATA *struct {
|
||||
Table []struct {
|
||||
ID int `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Value int64 `json:"value"`
|
||||
Thresh int64 `json:"thresh"`
|
||||
WhenFailed string `json:"when_failed"`
|
||||
Flags struct {
|
||||
Prefailure bool `json:"prefailure"`
|
||||
} `json:"flags"`
|
||||
Raw struct {
|
||||
Value int64 `json:"value"`
|
||||
} `json:"raw"`
|
||||
} `json:"table"`
|
||||
} `json:"ata_smart_attributes"`
|
||||
ErrorLog *struct {
|
||||
Summary struct {
|
||||
Count int64 `json:"count"`
|
||||
} `json:"summary"`
|
||||
} `json:"ata_smart_error_log"`
|
||||
SelfTests *struct {
|
||||
Standard struct {
|
||||
Table []struct {
|
||||
Type struct {
|
||||
String string `json:"string"`
|
||||
} `json:"type"`
|
||||
Status struct {
|
||||
String string `json:"string"`
|
||||
Passed *bool `json:"passed"`
|
||||
} `json:"status"`
|
||||
} `json:"table"`
|
||||
} `json:"standard"`
|
||||
} `json:"ata_smart_self_test_log"`
|
||||
Temperature *struct {
|
||||
Current int64 `json:"current"`
|
||||
} `json:"temperature"`
|
||||
PowerOnTime *struct {
|
||||
Hours int64 `json:"hours"`
|
||||
} `json:"power_on_time"`
|
||||
}
|
||||
|
||||
// nvmeWarnings are the NVMe health log's critical warning bits, as a person reads each.
|
||||
var nvmeWarnings = []struct {
|
||||
bit int64
|
||||
said string
|
||||
}{
|
||||
{0x01, "its spare capacity is below its threshold"},
|
||||
{0x02, "its temperature is past one of its own thresholds"},
|
||||
{0x04, "its reliability is degraded by media or internal errors"},
|
||||
{0x08, "it has put itself in read-only mode"},
|
||||
{0x10, "its volatile memory backup has failed"},
|
||||
{0x20, "its persistent memory region has become read-only"},
|
||||
}
|
||||
|
||||
func num(v int64) *int64 { return &v }
|
||||
|
||||
// judgeDrive reads one drive's smartctl answer into the drive, with its findings: FAILING when the drive
|
||||
// says so itself, HIGH for the early signs a drive gives before it fails.
|
||||
func judgeDrive(d *Drive, reading DriveReading, now time.Time) {
|
||||
if !reading.ReadAt.IsZero() {
|
||||
d.ReadAt = reading.ReadAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
d.Asleep = reading.Asleep
|
||||
var asleepLong string
|
||||
if reading.Asleep && reading.AsleepSince != nil {
|
||||
d.AsleepSince = reading.AsleepSince.UTC().Format(time.RFC3339)
|
||||
if for_ := now.Sub(*reading.AsleepSince); for_ >= AsleepHighAfter {
|
||||
asleepLong = fmt.Sprintf("it has been asleep at every look for %s, so its health has not been read since; "+
|
||||
"a drive that stopped answering can look asleep", ago(for_))
|
||||
}
|
||||
}
|
||||
defer func() {
|
||||
if asleepLong != "" {
|
||||
d.Findings = append(d.Findings, asleepLong)
|
||||
if d.Flag == "" {
|
||||
d.Flag = FlagHigh
|
||||
}
|
||||
}
|
||||
}()
|
||||
var s smartJSON
|
||||
if len(reading.Smartctl) == 0 || json.Unmarshal(reading.Smartctl, &s) != nil {
|
||||
d.Health, d.NotRead = "not read", reading.NotRead
|
||||
if d.NotRead == "" {
|
||||
d.NotRead = "smartctl gave no answer for it"
|
||||
}
|
||||
return
|
||||
}
|
||||
if s.ModelName != "" {
|
||||
d.Model = s.ModelName
|
||||
}
|
||||
d.Serial, d.Firmware = s.SerialNumber, s.FirmwareVersion
|
||||
if s.SmartStatus == nil && s.NVMe == nil {
|
||||
d.Health = "not read"
|
||||
switch {
|
||||
case s.SmartSupport != nil && !s.SmartSupport.Available:
|
||||
d.NotRead = "the drive does not support SMART"
|
||||
case s.SmartSupport != nil && !s.SmartSupport.Enabled:
|
||||
d.NotRead = "SMART is turned off on the drive"
|
||||
default:
|
||||
d.NotRead = firstError(s, reading.NotRead)
|
||||
}
|
||||
return
|
||||
}
|
||||
d.Health = "passed"
|
||||
var failing, high []string
|
||||
// An NVMe drive's overall verdict is its critical warning byte, so it is judged from the bits below;
|
||||
// the temperature bit alone is a drive running hot, not one failing.
|
||||
if s.SmartStatus != nil && !s.SmartStatus.Passed && s.NVMe == nil {
|
||||
failing = append(failing, "its overall SMART health check failed")
|
||||
}
|
||||
if s.Temperature != nil && s.Temperature.Current != 0 {
|
||||
d.TemperatureC = num(s.Temperature.Current)
|
||||
}
|
||||
if s.PowerOnTime != nil {
|
||||
d.PowerOnHours = num(s.PowerOnTime.Hours)
|
||||
}
|
||||
if n := s.NVMe; n != nil {
|
||||
d.CriticalWarning, d.PercentageUsed = num(n.CriticalWarning), num(n.PercentageUsed)
|
||||
d.AvailableSpare, d.SpareThreshold, d.MediaErrors = num(n.AvailableSpare), num(n.AvailableSpareThreshold), num(n.MediaErrors)
|
||||
if d.PowerOnHours == nil {
|
||||
d.PowerOnHours = num(n.PowerOnHours)
|
||||
}
|
||||
for _, w := range nvmeWarnings {
|
||||
switch {
|
||||
case n.CriticalWarning&w.bit == 0:
|
||||
case w.bit == 0x02:
|
||||
high = append(high, w.said)
|
||||
default:
|
||||
failing = append(failing, w.said)
|
||||
}
|
||||
}
|
||||
if n.MediaErrors > 0 {
|
||||
high = append(high, fmt.Sprintf("it has had %d media errors", n.MediaErrors))
|
||||
}
|
||||
if n.PercentageUsed >= 90 {
|
||||
high = append(high, fmt.Sprintf("it has used %d %% of its rated endurance", n.PercentageUsed))
|
||||
}
|
||||
}
|
||||
if a := s.ATA; a != nil {
|
||||
for _, at := range a.Table {
|
||||
switch at.ID {
|
||||
case 5:
|
||||
d.Reallocated = num(at.Raw.Value)
|
||||
case 197:
|
||||
d.Pending = num(at.Raw.Value)
|
||||
case 198:
|
||||
d.Uncorrectable = num(at.Raw.Value)
|
||||
}
|
||||
switch at.WhenFailed {
|
||||
case "now":
|
||||
// A pre-failure attribute at its threshold is the drive saying it is about to fail; an
|
||||
// old-age one (a temperature, a count of hours) is wear or a hot day.
|
||||
said := fmt.Sprintf("its attribute %s is at %d, at or below its own threshold of %d", at.Name, at.Value, at.Thresh)
|
||||
if at.Flags.Prefailure {
|
||||
failing = append(failing, said)
|
||||
} else {
|
||||
high = append(high, said+" (an old-age attribute)")
|
||||
}
|
||||
case "past":
|
||||
high = append(high, fmt.Sprintf("its attribute %s was at or below its own threshold in the past", at.Name))
|
||||
}
|
||||
}
|
||||
for _, c := range []struct {
|
||||
v *int64
|
||||
said string
|
||||
}{{d.Reallocated, "sectors reallocated"}, {d.Pending, "sectors waiting to be reallocated"}, {d.Uncorrectable, "sectors it could not read"}} {
|
||||
if c.v != nil && *c.v > 0 {
|
||||
high = append(high, fmt.Sprintf("it has %d %s", *c.v, c.said))
|
||||
}
|
||||
}
|
||||
}
|
||||
if s.ErrorLog != nil {
|
||||
d.ErrorLogCount = num(s.ErrorLog.Summary.Count)
|
||||
if s.ErrorLog.Summary.Count > 0 {
|
||||
high = append(high, fmt.Sprintf("its error log holds %d errors", s.ErrorLog.Summary.Count))
|
||||
}
|
||||
}
|
||||
if s.SelfTests != nil && len(s.SelfTests.Standard.Table) > 0 {
|
||||
last := s.SelfTests.Standard.Table[0]
|
||||
d.LastSelfTest = strings.TrimSpace(last.Type.String + ": " + last.Status.String)
|
||||
if last.Status.Passed != nil && !*last.Status.Passed {
|
||||
high = append(high, "its last self-test did not pass ("+last.Status.String+")")
|
||||
}
|
||||
}
|
||||
d.failingSentences = failing
|
||||
switch {
|
||||
case len(failing) > 0:
|
||||
d.Health, d.Flag = "failing", FlagFailing
|
||||
d.Findings = append(d.Findings, failing...)
|
||||
case len(high) > 0:
|
||||
d.Flag = FlagHigh
|
||||
}
|
||||
d.Findings = append(d.Findings, high...)
|
||||
}
|
||||
|
||||
func firstError(s smartJSON, fallback string) string {
|
||||
for _, msg := range s.Smartctl.Messages {
|
||||
if msg.Severity == "error" && strings.TrimSpace(msg.String) != "" {
|
||||
return "smartctl: " + strings.TrimSpace(msg.String)
|
||||
}
|
||||
}
|
||||
if fallback != "" {
|
||||
return fallback
|
||||
}
|
||||
return "smartctl gave no health for it"
|
||||
}
|
||||
|
||||
// deviceType is smartctl's -d for a drive whose kind is known, so smartctl does not probe for it: probing
|
||||
// a drive behind a USB bridge can spin it up. Empty for a drive smartctl must find out about itself.
|
||||
func deviceType(d Drive) string {
|
||||
switch d.Transport {
|
||||
case "nvme":
|
||||
return "nvme"
|
||||
case "ata", "usb":
|
||||
return "sat"
|
||||
case "scsi":
|
||||
return "scsi"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ReadDrive asks smartctl for one drive's health, as root (directly when this process is root, else
|
||||
// through sudo -n), without waking a drive in standby. `-n standby,2,3` makes smartctl stop with status 2
|
||||
// when the drive is in standby or asleep (an ATA drive with its power mode, a SAS drive or one the
|
||||
// operating system powered down with its own words), and with status 3 when the drive cannot say (a USB
|
||||
// bridge that does not pass the question on): such a drive is not read, because reading it could wake it.
|
||||
// A drive asleep keeps the reading before, and since when it has been asleep.
|
||||
func (m *Machine) ReadDrive(d Drive, before *DriveReading) DriveReading {
|
||||
args := []string{"--json=c", "-a"}
|
||||
if t := deviceType(d); t != "" {
|
||||
args = append(args, "-d", t)
|
||||
}
|
||||
args = append(args, "-n", "standby,2,3", "/dev/"+d.Device)
|
||||
program := Smartctl
|
||||
if m.UID != 0 {
|
||||
program, args = "sudo", append([]string{"-n", Smartctl}, args...)
|
||||
}
|
||||
r := m.Run(context.Background(), program, args...)
|
||||
now := m.Now()
|
||||
out := DriveReading{ReadAt: now, LookedAt: now, Status: r.Status}
|
||||
switch {
|
||||
case r.Err == "ENOENT" && program == Smartctl:
|
||||
out.NotRead = "smartctl is not installed; the module declares smartmontools, so the package has not been applied yet"
|
||||
return out
|
||||
case r.Err == "ENOENT":
|
||||
out.NotRead = "the drive reader is not root and sudo is not installed to read the drive as root"
|
||||
return out
|
||||
case r.Err != "":
|
||||
out.NotRead = "smartctl did not answer: " + r.Err
|
||||
return out
|
||||
case program == "sudo" && strings.HasPrefix(strings.TrimSpace(r.Stderr), "sudo:") && strings.Contains(r.Stderr, "command not found"):
|
||||
out.NotRead = "smartctl is not installed; the module declares smartmontools, so the package has not been applied yet"
|
||||
return out
|
||||
case program == "sudo" && strings.HasPrefix(strings.TrimSpace(r.Stderr), "sudo:"):
|
||||
out.NotRead = "the drive reader is not root and may not run smartctl through sudo without a prompt"
|
||||
return out
|
||||
}
|
||||
var s smartJSON
|
||||
parsed := json.Unmarshal([]byte(r.Stdout), &s) == nil
|
||||
switch {
|
||||
case parsed && r.Status == 2 && asleep(s):
|
||||
since := now
|
||||
kept := DriveReading{NotRead: "the drive was asleep at every look, and is not woken to be read"}
|
||||
if before != nil {
|
||||
kept = *before
|
||||
if before.Asleep && before.AsleepSince != nil {
|
||||
since = *before.AsleepSince
|
||||
}
|
||||
}
|
||||
kept.Asleep, kept.LookedAt, kept.AsleepSince, kept.InFlightSince = true, now, &since, nil
|
||||
return kept
|
||||
case parsed && r.Status == 3 && saysNoPowerMode(s):
|
||||
out.NotRead = "the drive cannot say whether it is asleep (a USB bridge, usually), so it is not read, never to wake it"
|
||||
return out
|
||||
case !parsed:
|
||||
out.NotRead = "smartctl answered something other than JSON (status " + strconv.Itoa(r.Status) + ")"
|
||||
return out
|
||||
}
|
||||
out.Smartctl = json.RawMessage(strings.TrimSpace(r.Stdout))
|
||||
return out
|
||||
}
|
||||
|
||||
// asleep says smartctl stopped because the drive is in standby or asleep: by the ATA power mode it
|
||||
// parsed, or by its own words "Device is in <mode> mode, exit(2)", which it says for a SAS drive and for
|
||||
// a drive the operating system powered down (ataprint.cpp, scsiprint.cpp, smartctl.cpp of smartmontools
|
||||
// 7.5). A failure to open the drive says neither, whatever else its words hold.
|
||||
func asleep(s smartJSON) bool {
|
||||
if s.PowerMode != nil {
|
||||
switch s.PowerMode.Name {
|
||||
case "STANDBY", "STANDBY_Y", "SLEEP":
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, msg := range s.Smartctl.Messages {
|
||||
if sleepWords.MatchString(strings.TrimSpace(msg.String)) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// sleepWords are smartctl's words when -n standby stopped it: an ATA mode (STANDBY, SLEEP), a SAS mode,
|
||||
// which holds spaces (STANDBY BY TIMER, STANDBY_Y BY COMMAND, LOW POWER), or STANDBY (OS). An idle mode
|
||||
// never stops smartctl under -n standby, so it never appears here.
|
||||
var sleepWords = regexp.MustCompile(`^Device is in [A-Z_]+( [A-Z_]+)*( \(OS\))? mode, exit\(2\)$`)
|
||||
|
||||
func saysNoPowerMode(s smartJSON) bool {
|
||||
for _, msg := range s.Smartctl.Messages {
|
||||
if strings.HasPrefix(msg.String, "CHECK POWER MODE not implemented") || strings.HasPrefix(msg.String, "CHECK POWER MODE returned unknown value") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// WriteReadings writes one process's file whole, through a rename, readable by every account: it holds
|
||||
// no secret, and the tools read it as the operator account.
|
||||
func (m *Machine) WriteReadings(file string, r Readings) error {
|
||||
path := m.path(file)
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
// MkdirAll leaves a directory that was already there as it was: the tools must be able to enter it.
|
||||
if err := os.Chmod(filepath.Dir(path), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
raw, err := json.Marshal(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := path + ".new"
|
||||
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Chmod(tmp, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Fans, from the hwmon chips. Many fans stop on purpose when the machine is cool (a laptop at rest, a GPU
|
||||
// in its zero-fan mode), so 0 RPM alone is not a fault; nor is a chip's minimum or alarm bit, which a
|
||||
// zero-fan GPU is under by design and a Super I/O chip latches. Those are said in a note and raise nothing.
|
||||
// A fan is STOPPED only when it ran since the machine started, reads 0 now, and a temperature that fan
|
||||
// cools is at or above the hardware's own maximum: a CPU sensor's, a thermal zone's, or one on the fan's
|
||||
// own chip unless that is a board chip (novox/hq ADR 0256 rule 4).
|
||||
|
||||
// FlagStopped is said beside a fan that should be turning and is not.
|
||||
const FlagStopped = "STOPPED"
|
||||
|
||||
// Fan is one fan's reading.
|
||||
type Fan struct {
|
||||
Chip string `json:"chip"`
|
||||
Label string `json:"label"`
|
||||
// Key names the fan across readings: its chip and its input (asus/fan1).
|
||||
Key string `json:"key"`
|
||||
RPM *int64 `json:"rpm,omitempty"`
|
||||
// Min is the chip's own lowest speed for it, where it gives one.
|
||||
Min *int64 `json:"min_rpm,omitempty"`
|
||||
Alarm bool `json:"alarm,omitempty"`
|
||||
// RanSinceBoot says the module's watch saw it turning since the machine started; absent when the
|
||||
// watch has not said.
|
||||
RanSinceBoot *bool `json:"ran_since_boot,omitempty"`
|
||||
Flag string `json:"flag,omitempty"`
|
||||
Why string `json:"why,omitempty"`
|
||||
// Note is what the chip says that raises nothing: under its own minimum, its alarm bit.
|
||||
Note string `json:"note,omitempty"`
|
||||
NotRead string `json:"not_read,omitempty"`
|
||||
Where string `json:"where"`
|
||||
}
|
||||
|
||||
// Name is how a person reads which fan this is.
|
||||
func (f Fan) Name() string {
|
||||
if f.Label != "" && !strings.HasPrefix(f.Label, "fan") {
|
||||
return fmt.Sprintf("the fan %s (%s)", f.Label, f.Chip)
|
||||
}
|
||||
return fmt.Sprintf("the fan %s of %s", f.Label, f.Chip)
|
||||
}
|
||||
|
||||
// Fans is every fan the machine reports, read now.
|
||||
func (m *Machine) Fans() []Fan {
|
||||
var out []Fan
|
||||
for _, h := range m.numbered("/sys/class/hwmon", "hwmon") {
|
||||
dir := "/sys/class/hwmon/" + h
|
||||
chip := m.read(dir + "/name")
|
||||
for _, stem := range m.inputs(dir, "fan") {
|
||||
f := Fan{Chip: chip, Label: m.read(dir + "/" + stem + "_label"), Key: chip + "/" + stem, Where: dir + "/" + stem}
|
||||
if f.Label == "" {
|
||||
f.Label = stem
|
||||
}
|
||||
v, ok := m.readInt(dir + "/" + stem + "_input")
|
||||
if !ok || v < 0 {
|
||||
f.NotRead = "the fan gave no speed"
|
||||
out = append(out, f)
|
||||
continue
|
||||
}
|
||||
f.RPM = &v
|
||||
if mn, ok := m.readInt(dir + "/" + stem + "_min"); ok && mn > 0 {
|
||||
f.Min = &mn
|
||||
}
|
||||
if a, ok := m.readInt(dir + "/" + stem + "_alarm"); ok && a != 0 {
|
||||
f.Alarm = true
|
||||
}
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// judgeFans flags each fan, given which fans the watch saw turning since the machine started (nil when it
|
||||
// has not said) and the temperatures read now.
|
||||
func judgeFans(fans []Fan, ran map[string]bool, temps []Temperature) []Fan {
|
||||
for i := range fans {
|
||||
f := &fans[i]
|
||||
if ran != nil {
|
||||
r := ran[f.Key]
|
||||
f.RanSinceBoot = &r
|
||||
}
|
||||
if f.RPM == nil {
|
||||
continue
|
||||
}
|
||||
var notes []string
|
||||
if f.Alarm {
|
||||
notes = append(notes, fmt.Sprintf("its chip's alarm bit is set at %d RPM", *f.RPM))
|
||||
}
|
||||
if f.Min != nil && *f.RPM < *f.Min {
|
||||
notes = append(notes, fmt.Sprintf("it turns at %d RPM, under its chip's minimum of %d RPM", *f.RPM, *f.Min))
|
||||
}
|
||||
f.Note = strings.Join(notes, "; ")
|
||||
if *f.RPM != 0 || f.RanSinceBoot == nil || !*f.RanSinceBoot {
|
||||
continue
|
||||
}
|
||||
if hot := hottestFor(*f, temps); hot != nil {
|
||||
f.Flag, f.Why = FlagStopped, fmt.Sprintf("it ran earlier and reads 0 RPM while %s is at %.1f °C, at or above %s of %.1f °C",
|
||||
hot.Name(), *hot.C, hot.HighIs, *hot.High)
|
||||
}
|
||||
}
|
||||
return fans
|
||||
}
|
||||
|
||||
// hottestFor is the temperature furthest at or above its own high value among those this fan may be
|
||||
// cooling: a CPU sensor, a thermal zone (whose trips the firmware sets for the machine as a whole), or a
|
||||
// sensor on the fan's own chip when that chip is not a board chip. A board chip's (a Super I/O's) limits
|
||||
// are whatever the firmware left, so its maximum is no more trusted here than its critical value is for
|
||||
// health. A drive or a network adapter running hot says nothing of a fan.
|
||||
func hottestFor(f Fan, ts []Temperature) *Temperature {
|
||||
chip := path.Dir(f.Where)
|
||||
var best *Temperature
|
||||
for i := range ts {
|
||||
t := &ts[i]
|
||||
if t.C == nil || t.High == nil || *t.C < *t.High {
|
||||
continue
|
||||
}
|
||||
related := t.Kind == "cpu" || t.Source == "thermal zone" ||
|
||||
(t.Source == "hwmon" && path.Dir(t.Where) == chip && kindOf(f.Chip) != "board")
|
||||
if !related {
|
||||
continue
|
||||
}
|
||||
if best == nil || *t.C-*t.High > *best.C-*best.High {
|
||||
best = t
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
// The kernel's own counts of hardware errors since the machine started, where the hardware reports them:
|
||||
// memory errors from EDAC (a server with ECC memory has them, a laptop has none), PCIe errors from AER,
|
||||
// and how often an Intel CPU was slowed for heat. Each is a few small files; a machine without them has
|
||||
// the class not present.
|
||||
|
||||
// MemoryController is one EDAC memory controller's error counts.
|
||||
type MemoryController struct {
|
||||
Controller string `json:"controller"`
|
||||
Name string `json:"name,omitempty"`
|
||||
Corrected int64 `json:"corrected"`
|
||||
Uncorrectable int64 `json:"uncorrectable"`
|
||||
// Modules names each memory module that counted an error, by its label, with its counts.
|
||||
Modules []MemoryModule `json:"modules,omitempty"`
|
||||
Flag string `json:"flag,omitempty"`
|
||||
}
|
||||
|
||||
// MemoryModule is one DIMM's counts, where the controller says them per module.
|
||||
type MemoryModule struct {
|
||||
Label string `json:"label"`
|
||||
Corrected int64 `json:"corrected"`
|
||||
Uncorrectable int64 `json:"uncorrectable"`
|
||||
}
|
||||
|
||||
// PCIError is one PCIe device that counted an error.
|
||||
type PCIError struct {
|
||||
Device string `json:"device"`
|
||||
Driver string `json:"driver,omitempty"`
|
||||
Correctable int64 `json:"correctable"`
|
||||
NonFatal int64 `json:"non_fatal"`
|
||||
Fatal int64 `json:"fatal"`
|
||||
Flag string `json:"flag,omitempty"`
|
||||
}
|
||||
|
||||
// Faults is everything the kernel counted.
|
||||
type Faults struct {
|
||||
Memory []MemoryController `json:"memory"`
|
||||
// MemoryNotPresent says why no memory controller is listed.
|
||||
MemoryNotPresent string `json:"memory_not_present,omitempty"`
|
||||
// PCIDevicesCounting is how many PCIe devices report their errors; only those that counted one are
|
||||
// listed.
|
||||
PCIDevicesCounting int `json:"pci_devices_counting"`
|
||||
PCI []PCIError `json:"pci"`
|
||||
// Throttled is how often the CPU was slowed for heat since the machine started, summed over its
|
||||
// cores, where the CPU counts it.
|
||||
ThrottledCore *int64 `json:"cpu_core_throttled,omitempty"`
|
||||
ThrottledPackage *int64 `json:"cpu_package_throttled,omitempty"`
|
||||
}
|
||||
|
||||
// Faults reads the kernel's counts of hardware errors.
|
||||
func (m *Machine) Faults() Faults {
|
||||
f := Faults{Memory: []MemoryController{}, PCI: []PCIError{}}
|
||||
base := "/sys/devices/system/edac/mc"
|
||||
for _, mc := range m.numbered(base, "mc") {
|
||||
dir := base + "/" + mc
|
||||
c := MemoryController{Controller: mc, Name: m.read(dir + "/mc_name")}
|
||||
c.Corrected, _ = m.readInt(dir + "/ce_count")
|
||||
c.Uncorrectable, _ = m.readInt(dir + "/ue_count")
|
||||
for _, n := range m.names(dir) {
|
||||
if !strings.HasPrefix(n, "dimm") && !strings.HasPrefix(n, "rank") {
|
||||
continue
|
||||
}
|
||||
ce, _ := m.readInt(dir + "/" + n + "/dimm_ce_count")
|
||||
ue, _ := m.readInt(dir + "/" + n + "/dimm_ue_count")
|
||||
if ce+ue == 0 {
|
||||
continue
|
||||
}
|
||||
label := m.read(dir + "/" + n + "/dimm_label")
|
||||
if label == "" {
|
||||
label = n
|
||||
}
|
||||
c.Modules = append(c.Modules, MemoryModule{Label: label, Corrected: ce, Uncorrectable: ue})
|
||||
}
|
||||
switch {
|
||||
case c.Uncorrectable > 0:
|
||||
c.Flag = FlagCritical
|
||||
case c.Corrected > 0:
|
||||
c.Flag = FlagHigh
|
||||
}
|
||||
f.Memory = append(f.Memory, c)
|
||||
}
|
||||
if len(f.Memory) == 0 {
|
||||
f.MemoryNotPresent = "no memory controller reports its errors here (no EDAC driver, or memory without ECC)"
|
||||
}
|
||||
for _, dev := range m.names("/sys/bus/pci/devices") {
|
||||
dir := "/sys/bus/pci/devices/" + dev
|
||||
if !m.exists(dir + "/aer_dev_correctable") {
|
||||
continue
|
||||
}
|
||||
f.PCIDevicesCounting++
|
||||
e := PCIError{Device: dev, Driver: uevent(m.read(dir+"/uevent"), "DRIVER"),
|
||||
Correctable: aerTotal(m.read(dir + "/aer_dev_correctable")),
|
||||
NonFatal: aerTotal(m.read(dir + "/aer_dev_nonfatal")),
|
||||
Fatal: aerTotal(m.read(dir + "/aer_dev_fatal"))}
|
||||
if e.Correctable+e.NonFatal+e.Fatal == 0 {
|
||||
continue
|
||||
}
|
||||
if e.Fatal+e.NonFatal > 0 {
|
||||
e.Flag = FlagHigh
|
||||
}
|
||||
f.PCI = append(f.PCI, e)
|
||||
}
|
||||
var core, pkg int64
|
||||
counted := false
|
||||
for _, cpu := range m.numbered("/sys/devices/system/cpu", "cpu") {
|
||||
dir := "/sys/devices/system/cpu/" + cpu + "/thermal_throttle"
|
||||
if c, ok := m.readInt(dir + "/core_throttle_count"); ok {
|
||||
core += c
|
||||
counted = true
|
||||
}
|
||||
if p, ok := m.readInt(dir + "/package_throttle_count"); ok && p > pkg {
|
||||
pkg = p
|
||||
}
|
||||
}
|
||||
if counted {
|
||||
f.ThrottledCore, f.ThrottledPackage = &core, &pkg
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// aerTotal is the TOTAL_ERR_ line of one AER counter file.
|
||||
func aerTotal(content string) int64 {
|
||||
for _, l := range strings.Split(content, "\n") {
|
||||
fields := strings.Fields(l)
|
||||
if len(fields) == 2 && strings.HasPrefix(fields[0], "TOTAL_ERR_") {
|
||||
var n int64
|
||||
for _, r := range fields[1] {
|
||||
if r < '0' || r > '9' {
|
||||
return 0
|
||||
}
|
||||
n = n*10 + int64(r-'0')
|
||||
}
|
||||
return n
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func uevent(content, key string) string {
|
||||
for _, l := range strings.Split(content, "\n") {
|
||||
if v, ok := strings.CutPrefix(l, key+"="); ok {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// What the module finds, said two ways: a line per finding for whoever asks (sensors_summary), and the
|
||||
// module's health for the node-engine (sensors_health, novox/hq ADR 0240 and ADR 0256 rule 4).
|
||||
|
||||
// Look is one reading of every class.
|
||||
type Look struct {
|
||||
Temperatures []Temperature
|
||||
Fans []Fan
|
||||
Drives []Drive
|
||||
DrivesNote string
|
||||
Faults Faults
|
||||
// Stale is why the module's own processes are not to be trusted now: a file missing, from an earlier
|
||||
// boot or older than StaleAfter, a drive's read stuck, a drive not looked at for twice its cadence.
|
||||
Stale []string
|
||||
}
|
||||
|
||||
// LookNow reads every class and each process's file once, and judges the fans against what the fan watch
|
||||
// saw turning since the machine started, and the drive reader by how old its file and its reads are.
|
||||
func (m *Machine) LookNow() Look {
|
||||
l := Look{Temperatures: m.Temperatures(), Fans: m.Fans(), Faults: m.Faults()}
|
||||
now := m.Now()
|
||||
dr, dwhy := m.ReadReadings(DrivesPath)
|
||||
l.Drives, l.DrivesNote = m.Drives(dr, dwhy)
|
||||
fr, fwhy := m.ReadReadings(FansPath)
|
||||
for _, f := range []struct {
|
||||
r *Readings
|
||||
why string
|
||||
path string
|
||||
}{{dr, dwhy, DrivesPath}, {fr, fwhy, FansPath}} {
|
||||
switch {
|
||||
case f.r == nil:
|
||||
l.Stale = append(l.Stale, capital(f.why)+".")
|
||||
case now.Sub(f.r.WrittenAt) > StaleAfter:
|
||||
l.Stale = append(l.Stale, fmt.Sprintf("%s has not written its readings for %s, so it is hung or gone.",
|
||||
capital(who(f.path)), ago(now.Sub(f.r.WrittenAt))))
|
||||
}
|
||||
}
|
||||
if dr != nil {
|
||||
byName := map[string]Drive{}
|
||||
for _, d := range l.Drives {
|
||||
byName[d.Device] = d
|
||||
}
|
||||
for _, name := range sortedKeys(dr.Drives) {
|
||||
rd := dr.Drives[name]
|
||||
d, present := byName[name]
|
||||
switch {
|
||||
case rd.InFlightSince != nil && now.Sub(*rd.InFlightSince) > ReadBound():
|
||||
l.Stale = append(l.Stale, fmt.Sprintf("The read of the drive %s has not finished for %s; the drive or its "+
|
||||
"controller may have stopped answering.", name, ago(now.Sub(*rd.InFlightSince))))
|
||||
case present && rd.InFlightSince == nil && !rd.LookedAt.IsZero() && now.Sub(rd.LookedAt) > 2*cadence(d):
|
||||
l.Stale = append(l.Stale, fmt.Sprintf("The drive %s has not been looked at for %s, more than twice as long "+
|
||||
"as it should be.", name, ago(now.Sub(rd.LookedAt))))
|
||||
}
|
||||
}
|
||||
}
|
||||
var ran map[string]bool
|
||||
if fr != nil {
|
||||
ran = map[string]bool{}
|
||||
for k := range fr.FansRan {
|
||||
ran[k] = true
|
||||
}
|
||||
}
|
||||
l.Fans = judgeFans(l.Fans, ran, l.Temperatures)
|
||||
return l
|
||||
}
|
||||
|
||||
func sortedKeys(m map[string]DriveReading) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// Problems is what makes the module unhealthy: each a sentence for the operator.
|
||||
func (l Look) Problems() []string {
|
||||
var out []string
|
||||
for _, t := range l.Temperatures {
|
||||
if t.Flag == FlagCritical && t.DrivesHealth() {
|
||||
out = append(out, fmt.Sprintf("%s is at %.1f °C, at or above %s of %.1f °C.", capital(t.Name()), *t.C, t.CriticalIs, *t.Critical))
|
||||
}
|
||||
}
|
||||
for _, d := range l.Drives {
|
||||
if d.Flag == FlagFailing {
|
||||
out = append(out, fmt.Sprintf("%s says it is failing: %s.", capital(d.Name()), joinAnd(d.failingSentences)))
|
||||
}
|
||||
}
|
||||
for _, f := range l.Fans {
|
||||
if f.Flag == FlagStopped {
|
||||
out = append(out, fmt.Sprintf("%s has stopped: %s.", capital(f.Name()), f.Why))
|
||||
}
|
||||
}
|
||||
for _, c := range l.Faults.Memory {
|
||||
if c.Uncorrectable > 0 {
|
||||
out = append(out, fmt.Sprintf("The memory reports %d uncorrectable errors since the machine started (memory controller %s).",
|
||||
c.Uncorrectable, c.Controller))
|
||||
}
|
||||
}
|
||||
return append(out, l.Stale...)
|
||||
}
|
||||
|
||||
// Health is the answer the node-engine reads from a tool check: healthy or not, with why.
|
||||
type Health struct {
|
||||
Healthy bool `json:"healthy"`
|
||||
Why string `json:"why"`
|
||||
Found []string `json:"found,omitempty"`
|
||||
}
|
||||
|
||||
// Health judges the look. Unhealthy only on what the hardware says is failing; HIGH readings and what
|
||||
// could not be read are said by the other tools and raise nothing.
|
||||
func (l Look) Health() Health {
|
||||
p := l.Problems()
|
||||
if len(p) == 0 {
|
||||
why := "No temperature is at its critical value, no drive says it is failing, no fan has stopped and no memory error is uncorrectable."
|
||||
unread := 0
|
||||
for _, d := range l.Drives {
|
||||
if d.Health == "not read" {
|
||||
unread++
|
||||
}
|
||||
}
|
||||
// Said, never passed as read: a drive asleep since boot, or one the watch cannot read.
|
||||
if unread > 0 {
|
||||
why += fmt.Sprintf(" The health of %d of %d drives has not been read.", unread, len(l.Drives))
|
||||
}
|
||||
return Health{Healthy: true, Why: why}
|
||||
}
|
||||
return Health{Healthy: false, Why: strings.Join(p, " "), Found: p}
|
||||
}
|
||||
|
||||
// Said is one line per finding, flagged, then one line per class saying what was read.
|
||||
func (l Look) Said() []string {
|
||||
var out []string
|
||||
for _, t := range l.Temperatures {
|
||||
switch t.Flag {
|
||||
case FlagCritical:
|
||||
out = append(out, fmt.Sprintf("CRITICAL: %s at %.1f °C, at or above %s (%.1f °C)", t.Name(), *t.C, t.CriticalIs, *t.Critical))
|
||||
case FlagHigh:
|
||||
out = append(out, fmt.Sprintf("HIGH: %s at %.1f °C, at or above %s (%.1f °C)", t.Name(), *t.C, t.HighIs, *t.High))
|
||||
}
|
||||
if t.Alarm && t.Flag == "" {
|
||||
out = append(out, fmt.Sprintf("HIGH: %s raises its chip's alarm at %.1f °C", t.Name(), *t.C))
|
||||
}
|
||||
}
|
||||
for _, d := range l.Drives {
|
||||
switch d.Flag {
|
||||
case FlagFailing:
|
||||
out = append(out, fmt.Sprintf("FAILING: %s: %s", d.Name(), strings.Join(d.Findings, "; ")))
|
||||
case FlagHigh:
|
||||
out = append(out, fmt.Sprintf("HIGH: %s: %s", d.Name(), strings.Join(d.Findings, "; ")))
|
||||
}
|
||||
}
|
||||
for _, f := range l.Fans {
|
||||
if f.Flag == FlagStopped {
|
||||
out = append(out, fmt.Sprintf("STOPPED: %s: %s", f.Name(), f.Why))
|
||||
}
|
||||
}
|
||||
for _, c := range l.Faults.Memory {
|
||||
switch c.Flag {
|
||||
case FlagCritical:
|
||||
out = append(out, fmt.Sprintf("CRITICAL: memory controller %s counted %d uncorrectable and %d corrected errors since the machine started", c.Controller, c.Uncorrectable, c.Corrected))
|
||||
case FlagHigh:
|
||||
out = append(out, fmt.Sprintf("HIGH: memory controller %s corrected %d errors since the machine started", c.Controller, c.Corrected))
|
||||
}
|
||||
}
|
||||
for _, e := range l.Faults.PCI {
|
||||
if e.Flag != "" {
|
||||
out = append(out, fmt.Sprintf("HIGH: the PCIe device %s (%s) counted %d fatal and %d non-fatal errors since the machine started", e.Device, orUnknown(e.Driver), e.Fatal, e.NonFatal))
|
||||
}
|
||||
}
|
||||
for _, st := range l.Stale {
|
||||
out = append(out, "STALE: "+st)
|
||||
}
|
||||
return append(out, l.overview()...)
|
||||
}
|
||||
|
||||
// overview is a line per class: how many were read, the hottest, and what is not present.
|
||||
func (l Look) overview() []string {
|
||||
var out []string
|
||||
read, notRead := 0, 0
|
||||
var hot *Temperature
|
||||
for i, t := range l.Temperatures {
|
||||
if t.C == nil {
|
||||
notRead++
|
||||
continue
|
||||
}
|
||||
read++
|
||||
if hot == nil || *t.C > *hot.C {
|
||||
hot = &l.Temperatures[i]
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case read == 0 && notRead == 0:
|
||||
out = append(out, "temperatures: not present (no hwmon chip and no thermal zone reports one)")
|
||||
case read == 0:
|
||||
out = append(out, fmt.Sprintf("temperatures: %d sensors, none gave a reading", notRead))
|
||||
default:
|
||||
line := fmt.Sprintf("temperatures: %d read; the hottest is %s at %.1f °C", read, hot.Name(), *hot.C)
|
||||
switch {
|
||||
case hot.Critical != nil:
|
||||
line += fmt.Sprintf(" (critical at %.1f °C)", *hot.Critical)
|
||||
case hot.High != nil:
|
||||
line += fmt.Sprintf(" (high at %.1f °C)", *hot.High)
|
||||
default:
|
||||
line += ", which reports no threshold of its own"
|
||||
}
|
||||
if notRead > 0 {
|
||||
line += fmt.Sprintf("; %d gave no reading", notRead)
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
if len(l.Fans) == 0 {
|
||||
out = append(out, "fans: not present (no hwmon chip reports a fan)")
|
||||
} else {
|
||||
var parts []string
|
||||
for _, f := range l.Fans {
|
||||
if f.RPM == nil {
|
||||
parts = append(parts, f.Label+" not read")
|
||||
continue
|
||||
}
|
||||
parts = append(parts, fmt.Sprintf("%s %d RPM", f.Label, *f.RPM))
|
||||
}
|
||||
out = append(out, "fans: "+strings.Join(parts, ", "))
|
||||
}
|
||||
if len(l.Drives) == 0 {
|
||||
out = append(out, "drives: not present (no drive backed by hardware)")
|
||||
} else {
|
||||
passed, failing, unread := 0, 0, 0
|
||||
for _, d := range l.Drives {
|
||||
switch d.Health {
|
||||
case "passed":
|
||||
passed++
|
||||
case "failing":
|
||||
failing++
|
||||
default:
|
||||
unread++
|
||||
}
|
||||
}
|
||||
line := fmt.Sprintf("drives: %d, %d passed, %d failing", len(l.Drives), passed, failing)
|
||||
if unread > 0 {
|
||||
line += fmt.Sprintf(", %d not read", unread)
|
||||
for _, d := range l.Drives {
|
||||
if d.NotRead != "" {
|
||||
line += " (" + d.Device + ": " + d.NotRead + ")"
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if l.DrivesNote != "" && unread < len(l.Drives) {
|
||||
line += "; " + l.DrivesNote
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
errs := "hardware errors: "
|
||||
if len(l.Faults.Memory) == 0 {
|
||||
errs += "memory not present"
|
||||
} else {
|
||||
var ce, ue int64
|
||||
for _, c := range l.Faults.Memory {
|
||||
ce, ue = ce+c.Corrected, ue+c.Uncorrectable
|
||||
}
|
||||
errs += fmt.Sprintf("memory %d corrected, %d uncorrectable", ce, ue)
|
||||
}
|
||||
if l.Faults.PCIDevicesCounting == 0 {
|
||||
errs += "; PCIe not present"
|
||||
} else {
|
||||
errs += fmt.Sprintf("; PCIe %d of %d devices counted an error", len(l.Faults.PCI), l.Faults.PCIDevicesCounting)
|
||||
}
|
||||
if l.Faults.ThrottledCore != nil {
|
||||
errs += fmt.Sprintf("; the CPU was slowed for heat %d times", *l.Faults.ThrottledCore)
|
||||
}
|
||||
return append(out, errs+", since the machine started")
|
||||
}
|
||||
|
||||
func capital(s string) string {
|
||||
if s == "" {
|
||||
return s
|
||||
}
|
||||
return strings.ToUpper(s[:1]) + s[1:]
|
||||
}
|
||||
|
||||
func joinAnd(parts []string) string {
|
||||
switch len(parts) {
|
||||
case 0:
|
||||
return ""
|
||||
case 1:
|
||||
return parts[0]
|
||||
}
|
||||
return strings.Join(parts[:len(parts)-1], ", ") + " and " + parts[len(parts)-1]
|
||||
}
|
||||
|
||||
func orUnknown(s string) string {
|
||||
if s == "" {
|
||||
return "no driver"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// fake is a machine for a test: a tree standing in for /sys, /proc and /run, a runner answering smartctl
|
||||
// from a table, and a clock.
|
||||
type fake struct {
|
||||
t *testing.T
|
||||
root string
|
||||
now time.Time
|
||||
uid int
|
||||
answers map[string]Ran
|
||||
calls []string
|
||||
}
|
||||
|
||||
const bootID = "11111111-2222-3333-4444-555555555555"
|
||||
|
||||
func newFake(t *testing.T) *fake {
|
||||
t.Helper()
|
||||
f := &fake{t: t, root: t.TempDir(), now: time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC), answers: map[string]Ran{}}
|
||||
f.file("/proc/sys/kernel/random/boot_id", bootID)
|
||||
return f
|
||||
}
|
||||
|
||||
func (f *fake) machine() *Machine {
|
||||
return &Machine{
|
||||
Root: f.root,
|
||||
UID: f.uid,
|
||||
Now: func() time.Time { return f.now },
|
||||
Run: func(_ context.Context, name string, args ...string) Ran {
|
||||
line := strings.TrimSpace(name + " " + strings.Join(args, " "))
|
||||
f.calls = append(f.calls, line)
|
||||
if r, ok := f.answers[line]; ok {
|
||||
return r
|
||||
}
|
||||
return Ran{Status: 127, Err: "ENOENT"}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// file writes a file under the fake root.
|
||||
func (f *fake) file(path, content string) {
|
||||
f.t.Helper()
|
||||
full := filepath.Join(f.root, path)
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(full, []byte(content), 0o644); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// dir makes a directory: sysfs's entries that are links on a real machine are directories here, because
|
||||
// the module only lists their names and reads files through them.
|
||||
func (f *fake) dir(path string) {
|
||||
f.t.Helper()
|
||||
if err := os.MkdirAll(filepath.Join(f.root, path), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// files writes several files under one directory.
|
||||
func (f *fake) files(dir string, kv map[string]string) {
|
||||
for k, v := range kv {
|
||||
f.file(dir+"/"+k, v)
|
||||
}
|
||||
}
|
||||
|
||||
// laptop is this kind of machine as the laptop of 2026-10-08 showed it: a CPU sensor with no threshold, an
|
||||
// integrated GPU, an NVMe drive whose composite sensor has its own maximum and critical values and whose
|
||||
// two others give the format's largest value, an ACPI zone that a hwmon chip copies, a network adapter and
|
||||
// four fans, two of them the vendor's.
|
||||
func (f *fake) laptop() {
|
||||
f.files("/sys/class/hwmon/hwmon0", map[string]string{"name": "ACAD"})
|
||||
f.files("/sys/class/hwmon/hwmon1", map[string]string{"name": "acpi_fan", "fan1_input": "4015"})
|
||||
f.files("/sys/class/hwmon/hwmon2", map[string]string{"name": "acpitz", "temp1_input": "60000"})
|
||||
f.files("/sys/class/hwmon/hwmon4", map[string]string{"name": "asus",
|
||||
"fan1_input": "4000", "fan1_label": "cpu_fan", "fan2_input": "0", "fan2_label": "gpu_fan", "fan3_input": "4900", "fan3_label": "mid_fan"})
|
||||
f.files("/sys/class/hwmon/hwmon6", map[string]string{"name": "amdgpu", "temp1_input": "57000", "temp1_label": "edge"})
|
||||
f.files("/sys/class/hwmon/hwmon7", map[string]string{"name": "nvme",
|
||||
"temp1_input": "38850", "temp1_label": "Composite", "temp1_max": "83850", "temp1_crit": "87850", "temp1_min": "-40150", "temp1_alarm": "0",
|
||||
"temp2_input": "58850", "temp2_label": "Sensor 1", "temp2_max": "65261850", "temp2_min": "-273150",
|
||||
"temp3_input": "38850", "temp3_label": "Sensor 2", "temp3_max": "65261850", "temp3_min": "-273150"})
|
||||
f.files("/sys/class/hwmon/hwmon8", map[string]string{"name": "k10temp", "temp1_input": "60875", "temp1_label": "Tctl"})
|
||||
f.files("/sys/class/hwmon/hwmon11", map[string]string{"name": "mt7921_phy0", "temp1_input": "56000"})
|
||||
f.files("/sys/class/thermal/thermal_zone0", map[string]string{"type": "acpitz", "temp": "60000", "mode": "enabled",
|
||||
"trip_point_0_type": "critical", "trip_point_0_temp": "124000",
|
||||
"trip_point_1_type": "hot", "trip_point_1_temp": "124000",
|
||||
"trip_point_2_type": "passive", "trip_point_2_temp": "110000",
|
||||
"trip_point_3_type": "active", "trip_point_3_temp": "60000"})
|
||||
f.dir("/sys/class/thermal/thermal_zone0/hwmon2")
|
||||
f.files("/sys/class/nvme/nvme0", map[string]string{"model": "WD PC SN560 SDDPNQE-1T00-1102"})
|
||||
f.dir("/sys/class/nvme/nvme0/hwmon7")
|
||||
f.dir("/sys/class/nvme/nvme0/nvme0n1")
|
||||
f.dir("/sys/class/nvme/nvme0/ng0n1")
|
||||
f.drive("nvme0n1", "WD PC SN560 SDDPNQE-1T00-1102", "", 2000409264, false)
|
||||
f.drive("loop0", "", "", 100, false)
|
||||
f.drive("zram0", "", "", 100, false)
|
||||
f.files("/sys/bus/pci/devices/0000:04:00.0", map[string]string{"uevent": "DRIVER=nvme\nPCI_CLASS=10802\n",
|
||||
"aer_dev_correctable": "RxErr 0\nTOTAL_ERR_COR 0\n", "aer_dev_nonfatal": "Undefined 0\nTOTAL_ERR_NONFATAL 0\n", "aer_dev_fatal": "TOTAL_ERR_FATAL 0\n"})
|
||||
f.dir("/sys/devices/system/edac/mc")
|
||||
}
|
||||
|
||||
// server is a machine with ECC memory, a CPU sensor with its own maximum and critical values, a board
|
||||
// chip with a header that has nothing on it, and three SATA drives read by drivetemp.
|
||||
func (f *fake) server() {
|
||||
f.files("/sys/class/hwmon/hwmon0", map[string]string{"name": "coretemp",
|
||||
"temp1_input": "45000", "temp1_label": "Package id 0", "temp1_max": "80000", "temp1_crit": "100000", "temp1_crit_alarm": "0",
|
||||
"temp2_input": "43000", "temp2_label": "Core 0", "temp2_max": "80000", "temp2_crit": "100000"})
|
||||
f.files("/sys/class/hwmon/hwmon1", map[string]string{"name": "nct6775",
|
||||
"fan1_input": "1200", "fan1_min": "300", "fan1_alarm": "0",
|
||||
"fan2_input": "0", "fan2_min": "300", "fan2_alarm": "1"})
|
||||
for i, d := range []string{"sda", "sdb", "sdc"} {
|
||||
f.drive(d, "WDC WUH722222AL", "ATA ", 42970644480, true)
|
||||
h := "hwmon" + string(rune('2'+i))
|
||||
f.files("/sys/class/hwmon/"+h, map[string]string{"name": "drivetemp", "temp1_input": "36000", "temp1_max": "60000", "temp1_crit": "70000"})
|
||||
f.dir("/sys/block/" + d + "/device/hwmon/" + h)
|
||||
}
|
||||
f.drive("md0", "", "", 100, true)
|
||||
f.files("/sys/devices/system/edac/mc/mc0", map[string]string{"mc_name": "Skylake Socket#0 IMC#0", "ce_count": "0", "ue_count": "0"})
|
||||
f.files("/sys/devices/system/edac/mc/mc0/dimm0", map[string]string{"dimm_label": "CPU_SrcID#0_MC#0_Chan#0_DIMM#0", "dimm_ce_count": "0", "dimm_ue_count": "0"})
|
||||
}
|
||||
|
||||
// drive adds a block device.
|
||||
func (f *fake) drive(name, model, vendor string, sectors int64, spins bool) {
|
||||
f.dir("/sys/block/" + name + "/device")
|
||||
if model != "" {
|
||||
f.file("/sys/block/"+name+"/device/model", model)
|
||||
}
|
||||
if vendor != "" {
|
||||
f.file("/sys/block/"+name+"/device/vendor", vendor)
|
||||
}
|
||||
f.file("/sys/block/"+name+"/size", itoa(sectors))
|
||||
rot := "0"
|
||||
if spins {
|
||||
rot = "1"
|
||||
}
|
||||
f.file("/sys/block/"+name+"/queue/rotational", rot)
|
||||
}
|
||||
|
||||
func itoa(n int64) string {
|
||||
b, _ := json.Marshal(n)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// readings writes both processes' files as if the drive reader had read each drive with the smartctl JSON
|
||||
// given, and the fan watch had seen the fans given turning.
|
||||
func (f *fake) readings(written time.Time, boot string, drives map[string]string, fansRan ...string) {
|
||||
r := Readings{BootID: boot, WrittenAt: written, Drives: map[string]DriveReading{}, FansRan: map[string]time.Time{}}
|
||||
for d, j := range drives {
|
||||
r.Drives[d] = DriveReading{ReadAt: written, Smartctl: json.RawMessage(j)}
|
||||
}
|
||||
for _, k := range fansRan {
|
||||
r.FansRan[k] = written
|
||||
}
|
||||
for d := range r.Drives {
|
||||
x := r.Drives[d]
|
||||
x.LookedAt = written
|
||||
r.Drives[d] = x
|
||||
}
|
||||
f.write(DrivesPath, Readings{BootID: boot, WrittenAt: written, Drives: r.Drives})
|
||||
f.write(FansPath, Readings{BootID: boot, WrittenAt: written, FansRan: r.FansRan})
|
||||
}
|
||||
|
||||
// write writes one process's file.
|
||||
func (f *fake) write(path string, r Readings) {
|
||||
f.t.Helper()
|
||||
raw, err := json.Marshal(r)
|
||||
if err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
f.file(path, string(raw))
|
||||
}
|
||||
|
||||
// smartctl answers, as smartmontools 7 writes them (trimmed to what the module reads).
|
||||
const (
|
||||
nvmePassed = `{"smartctl":{"exit_status":0},"device":{"protocol":"NVMe"},"model_name":"WD PC SN560 SDDPNQE-1T00-1102","serial_number":"S1","firmware_version":"74115000",` +
|
||||
`"smart_support":{"available":true,"enabled":true},"smart_status":{"passed":true,"nvme":{"value":0}},` +
|
||||
`"nvme_smart_health_information_log":{"critical_warning":0,"temperature":39,"available_spare":100,"available_spare_threshold":10,"percentage_used":3,"power_on_hours":2210,"media_errors":0},` +
|
||||
`"temperature":{"current":39},"power_on_time":{"hours":2210}}`
|
||||
nvmeSpareLow = `{"smartctl":{"exit_status":8},"model_name":"Samsung SSD 990 PRO 2TB","serial_number":"S2","smart_status":{"passed":false,"nvme":{"value":1}},` +
|
||||
`"nvme_smart_health_information_log":{"critical_warning":1,"available_spare":5,"available_spare_threshold":10,"percentage_used":97,"power_on_hours":40000,"media_errors":3}}`
|
||||
ataPassed = `{"smartctl":{"exit_status":0},"model_name":"WDC WUH722222ALE6L4","serial_number":"A1","smart_support":{"available":true,"enabled":true},"smart_status":{"passed":true},` +
|
||||
`"ata_smart_attributes":{"table":[{"id":5,"name":"Reallocated_Sector_Ct","value":100,"thresh":1,"when_failed":"","raw":{"value":0}},` +
|
||||
`{"id":197,"name":"Current_Pending_Sector","value":100,"thresh":0,"when_failed":"","raw":{"value":0}},{"id":198,"name":"Offline_Uncorrectable","value":100,"thresh":0,"when_failed":"","raw":{"value":0}}]},` +
|
||||
`"ata_smart_error_log":{"summary":{"count":0}},"ata_smart_self_test_log":{"standard":{"table":[{"type":{"string":"Short offline"},"status":{"string":"Completed without error","passed":true}}]}},` +
|
||||
`"temperature":{"current":36},"power_on_time":{"hours":17000}}`
|
||||
ataFailing = `{"smartctl":{"exit_status":24},"model_name":"WDC WUH722222ALE6L4","serial_number":"A2","smart_support":{"available":true,"enabled":true},"smart_status":{"passed":false},` +
|
||||
`"ata_smart_attributes":{"table":[{"id":5,"name":"Reallocated_Sector_Ct","value":1,"thresh":10,"when_failed":"now","flags":{"prefailure":true},"raw":{"value":4012}},` +
|
||||
`{"id":197,"name":"Current_Pending_Sector","value":100,"thresh":0,"when_failed":"","raw":{"value":16}}]},"ata_smart_error_log":{"summary":{"count":12}}}`
|
||||
ataPending = `{"smartctl":{"exit_status":64},"model_name":"WDC WUH722222ALE6L4","serial_number":"A3","smart_support":{"available":true,"enabled":true},"smart_status":{"passed":true},` +
|
||||
`"ata_smart_attributes":{"table":[{"id":197,"name":"Current_Pending_Sector","value":100,"thresh":0,"when_failed":"","raw":{"value":8}}]},"ata_smart_error_log":{"summary":{"count":0}}}`
|
||||
noSMART = `{"smartctl":{"exit_status":4,"messages":[{"string":"Read Device Identity failed: Unknown","severity":"error"}]},"smart_support":{"available":false}}`
|
||||
)
|
||||
|
||||
type ctxT = context.Context
|
||||
|
||||
// smartctlLine is the command the drive reader runs for one drive, as root, with smartctl's -d when the
|
||||
// drive's kind is known.
|
||||
func smartctlLine(device, typ string) string {
|
||||
d := ""
|
||||
if typ != "" {
|
||||
d = "-d " + typ + " "
|
||||
}
|
||||
return "/usr/bin/smartctl --json=c -a " + d + "-n standby,2,3 /dev/" + device
|
||||
}
|
||||
|
||||
func chmod(path string, mode os.FileMode) error { return os.Chmod(path, mode) }
|
||||
|
||||
func modeOf(t *testing.T, path string) os.FileMode {
|
||||
t.Helper()
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return info.Mode().Perm()
|
||||
}
|
||||
|
||||
// fresh writes readings the watch has just written, with every drive of the server passed.
|
||||
func (f *fake) fresh() {
|
||||
f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataPassed, "sdc": ataPassed})
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// What the module reads, and as whom.
|
||||
//
|
||||
// The tools run as the operator account in the node's tool runner (novox/hq ADR 0175 §4) and read only
|
||||
// /sys and /proc, which every account may. A drive's health needs root: the NVMe controller's device and a
|
||||
// SATA drive's SMART commands are root's. So the module's watch, a process of this same binary that the
|
||||
// node-engine runs as root, reads it with smartctl and leaves its readings in one file the tools read
|
||||
// (novox/hq ADR 0256).
|
||||
|
||||
// CommandTimeout bounds one smartctl: a drive that does not answer costs one line, never the watch. A
|
||||
// variable only so a test can shorten it.
|
||||
var CommandTimeout = 20 * time.Second
|
||||
|
||||
// WaitDelay is how long a command ended for its timeout may keep its output open (a child it left behind)
|
||||
// before the pipes are closed under it: without it, the wait for the output outlives the timeout.
|
||||
const WaitDelay = 2 * time.Second
|
||||
|
||||
// Ran is what one command did: its output and exit status, and why it never ran to an answer.
|
||||
type Ran struct {
|
||||
Stdout string
|
||||
Stderr string
|
||||
Status int
|
||||
// Err is "ENOENT" when the program is not there, or that it was ended for taking too long.
|
||||
Err string
|
||||
}
|
||||
|
||||
// Runner runs one command; injected, so the watch is tested over recorded answers.
|
||||
type Runner func(ctx context.Context, name string, args ...string) Ran
|
||||
|
||||
// ExecRunner runs a command on this machine, in the C locale, bounded by CommandTimeout.
|
||||
func ExecRunner(ctx context.Context, name string, args ...string) Ran {
|
||||
ctx, cancel := context.WithTimeout(ctx, CommandTimeout)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
cmd.Env = append(os.Environ(), "LC_ALL=C")
|
||||
cmd.WaitDelay = WaitDelay
|
||||
var out, errb bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &errb
|
||||
err := cmd.Run()
|
||||
r := Ran{Stdout: out.String(), Stderr: errb.String()}
|
||||
if ctx.Err() == context.DeadlineExceeded || errors.Is(err, exec.ErrWaitDelay) {
|
||||
r.Status, r.Err = 124, fmt.Sprintf("no answer within %s", CommandTimeout)
|
||||
return r
|
||||
}
|
||||
var exit *exec.ExitError
|
||||
switch {
|
||||
case err == nil:
|
||||
case errors.As(err, &exit):
|
||||
r.Status = exit.ExitCode()
|
||||
case errors.Is(err, exec.ErrNotFound) || errors.Is(err, fs.ErrNotExist):
|
||||
r.Status, r.Err = 127, "ENOENT"
|
||||
default:
|
||||
r.Status, r.Err = 126, err.Error()
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// Machine is what the module reads: a filesystem root (the real one, or a test's tree standing in for
|
||||
// /sys, /proc and /run), a way to run a command, and the time.
|
||||
type Machine struct {
|
||||
Root string
|
||||
Run Runner
|
||||
Now func() time.Time
|
||||
// UID is who this process is: smartctl is run directly by root and through sudo -n by anyone else.
|
||||
UID int
|
||||
}
|
||||
|
||||
// Here is the machine this process runs on.
|
||||
func Here() *Machine {
|
||||
return &Machine{Root: "/", Run: ExecRunner, Now: time.Now, UID: os.Getuid()}
|
||||
}
|
||||
|
||||
func (m *Machine) path(p string) string { return filepath.Join(m.Root, p) }
|
||||
|
||||
// read is a file's content, trimmed; "" when it cannot be read.
|
||||
func (m *Machine) read(p string) string {
|
||||
b, err := os.ReadFile(m.path(p))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
// readInt is a file holding one whole number; false when it is absent, unreadable or not a number — a
|
||||
// sensor whose driver answers an error (ENODATA, EIO) is said as not read, never as 0.
|
||||
func (m *Machine) readInt(p string) (int64, bool) {
|
||||
s := m.read(p)
|
||||
if s == "" {
|
||||
return 0, false
|
||||
}
|
||||
n, err := strconv.ParseInt(s, 10, 64)
|
||||
return n, err == nil
|
||||
}
|
||||
|
||||
func (m *Machine) exists(p string) bool {
|
||||
_, err := os.Stat(m.path(p))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// names lists the entries of a directory, sorted; none when it cannot be read.
|
||||
func (m *Machine) names(dir string) []string {
|
||||
entries, err := os.ReadDir(m.path(dir))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// numbered lists a directory's entries named prefix<number>, in numeric order (hwmon2 before hwmon10).
|
||||
func (m *Machine) numbered(dir, prefix string) []string {
|
||||
var out []string
|
||||
for _, n := range m.names(dir) {
|
||||
if rest, ok := strings.CutPrefix(n, prefix); ok {
|
||||
if _, err := strconv.Atoi(rest); err == nil {
|
||||
out = append(out, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
sortNumbered(out, prefix)
|
||||
return out
|
||||
}
|
||||
|
||||
func sortNumbered(names []string, prefix string) {
|
||||
num := func(s string) int {
|
||||
n, _ := strconv.Atoi(strings.TrimPrefix(s, prefix))
|
||||
return n
|
||||
}
|
||||
for i := 1; i < len(names); i++ {
|
||||
for j := i; j > 0 && num(names[j]) < num(names[j-1]); j-- {
|
||||
names[j], names[j-1] = names[j-1], names[j]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// bootID names this boot, so a reading from before the machine last started is not taken for this one.
|
||||
func (m *Machine) bootID() string { return m.read("/proc/sys/kernel/random/boot_id") }
|
||||
|
||||
// round1 rounds to one decimal, for what a person reads.
|
||||
func round1(f float64) float64 {
|
||||
if f < 0 {
|
||||
return -float64(int64(-f*10+0.5)) / 10
|
||||
}
|
||||
return float64(int64(f*10+0.5)) / 10
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
// The sensors module's Go bundle (novox/hq ADR 0256): the condition of a machine's own hardware —
|
||||
// temperatures with their own thresholds, fans, each drive's health, the kernel's counts of hardware
|
||||
// errors — read from /sys and /proc, and changing nothing.
|
||||
//
|
||||
// One binary, two ways to run. With no argument it is the bundle the node's tool runner launches and
|
||||
// speaks MCP to over stdio (ADR 0188, ADR 0193), as the operator account. With `watch` and `drives` it is
|
||||
// one of the module's two long-running processes, run by the node-engine as root: the fan watch, and the
|
||||
// drive reader (watch.go).
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) > 1 {
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
|
||||
defer stop()
|
||||
switch os.Args[1] {
|
||||
case "watch":
|
||||
w := NewFanWatch(Here())
|
||||
every(ctx, "watch", w.Look)
|
||||
case "drives":
|
||||
r := NewDriveReader(Here())
|
||||
r.Async = true
|
||||
every(ctx, "drives", r.Look)
|
||||
default:
|
||||
fmt.Fprintf(os.Stderr, "sensors: %q is not a way to run; with no argument it serves its tools, with watch it "+
|
||||
"watches the fans, with drives it reads the drives\n", os.Args[1])
|
||||
os.Exit(2)
|
||||
}
|
||||
return
|
||||
}
|
||||
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): sensors.
|
||||
if err := stdio.Serve("", Tools(Here())); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type manifest struct {
|
||||
Tools []string `json:"tools"`
|
||||
Claims []any `json:"claims"`
|
||||
Resources []map[string]any `json:"resources"`
|
||||
}
|
||||
|
||||
func readManifest(t *testing.T) (manifest, string) {
|
||||
t.Helper()
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m manifest
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return m, string(raw)
|
||||
}
|
||||
|
||||
func TestTheManifestNamesExactlyTheToolsTheBundleServesAllReadOnlyAndNoMachine(t *testing.T) {
|
||||
m, raw := readManifest(t)
|
||||
var served []string
|
||||
for _, tl := range Tools(newFake(t).machine()) {
|
||||
served = append(served, tl.Name)
|
||||
if !strings.HasSuffix(tl.Description, "(r)") {
|
||||
t.Errorf("%s only reads and does not say so", tl.Name)
|
||||
}
|
||||
}
|
||||
sort.Strings(served)
|
||||
sort.Strings(m.Tools)
|
||||
if strings.Join(served, ",") != strings.Join(m.Tools, ",") {
|
||||
t.Fatalf("served %v, listed %v", served, m.Tools)
|
||||
}
|
||||
if len(m.Claims) != 0 {
|
||||
t.Errorf("the module holds no seat (ADR 0256 rule 3), and claims %v", m.Claims)
|
||||
}
|
||||
// A machine's or a person's name, as a word: "novox/hq" names the record, not the machine.
|
||||
if hit := regexp.MustCompile(`\b(g14|novox|ace|shanks|jochen)\b|/home/`).FindString(strings.ReplaceAll(raw, "novox/hq", "")); hit != "" {
|
||||
t.Errorf("the manifest says %q", hit)
|
||||
}
|
||||
}
|
||||
|
||||
// ADR 0240 rule 2 and ADR 0256 rule 4: the tool check on the fan watch, beside the drive reader's unit's
|
||||
// readiness, which the module does not judge itself; and no smartd, so the module is the only reader.
|
||||
func TestTheHealthIsTheDriveReadersUnitAndTheFanWatchsToolCheckAndNoSmartd(t *testing.T) {
|
||||
m, raw := readManifest(t)
|
||||
byID := map[string]map[string]any{}
|
||||
for _, r := range m.Resources {
|
||||
byID[r["id"].(string)] = r
|
||||
}
|
||||
drives, watch := byID["drives"], byID["watch"]
|
||||
if drives == nil || watch == nil || len(byID) != 3 {
|
||||
t.Fatalf("resources: %v", byID)
|
||||
}
|
||||
if h, _ := drives["health"].(map[string]any); h["kind"] != "unit" || drives["type"] != "process" || drives["user"] != nil {
|
||||
t.Errorf("the drive reader runs as root and is judged by its unit: %v", drives)
|
||||
}
|
||||
h, _ := watch["health"].(map[string]any)
|
||||
if h["kind"] != "tool" || h["tool"] != "sensors_health" || watch["type"] != "process" {
|
||||
t.Errorf("the fan watch is judged by sensors_health: %v", watch)
|
||||
}
|
||||
for id, mode := range map[string]string{"drives": "drives", "watch": "watch"} {
|
||||
run, _ := byID[id]["run"].([]any)
|
||||
if len(run) != 2 || run[1] != mode {
|
||||
t.Errorf("%s runs the bundle's %s: %v", id, mode, run)
|
||||
}
|
||||
}
|
||||
if strings.Contains(raw, "smartd") {
|
||||
t.Errorf("the module runs no second reader of the drives")
|
||||
}
|
||||
}
|
||||
|
||||
func anyStrings(v any) []string {
|
||||
var out []string
|
||||
l, _ := v.([]any)
|
||||
for _, x := range l {
|
||||
s, _ := x.(string)
|
||||
out = append(out, s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,296 @@
|
||||
package main
|
||||
|
||||
// The reproductions of the review of 2026-10-08 (mesh-catalog #129): each fails on the first build of the
|
||||
// module and passes on the fix.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// smartmontools 7.5's own answer for a drive in standby with -n standby,2,3: the power mode is set
|
||||
// before the information message, and the exit status is the one -n asked for (ataprint.cpp).
|
||||
standbyJSON = `{"json_format_version":[1,0],"smartctl":{"version":[7,5],"messages":[{"string":"Device is in STANDBY mode, exit(2)",` +
|
||||
`"severity":"information"}],"exit_status":2},"device":{"name":"/dev/sda","info_name":"/dev/sda [SAT]","type":"sat","protocol":"ATA"},` +
|
||||
`"power_mode":{"ata_value":0,"name":"STANDBY"}}`
|
||||
// A bridge that cannot answer CHECK POWER MODE, with -n standby,2,3: exit 3 and no power mode.
|
||||
noPowerModeJSON = `{"smartctl":{"messages":[{"string":"CHECK POWER MODE not implemented, exit(3)","severity":"information"}],` +
|
||||
`"exit_status":3},"device":{"name":"/dev/sdd","type":"sat","protocol":"ATA"}}`
|
||||
// A drive that could not be opened, whose model happens to hold the word SLEEP: exit 2, no power mode.
|
||||
openFailedJSON = `{"smartctl":{"messages":[{"string":"Smartctl open device: /dev/sda failed: No such device","severity":"error"}],` +
|
||||
`"exit_status":2},"model_name":"SLEEPY DRIVE"}`
|
||||
nvmeHot = `{"smartctl":{"exit_status":8},"model_name":"Samsung SSD 990 PRO 2TB","smart_status":{"passed":false,"nvme":{"value":2}},` +
|
||||
`"nvme_smart_health_information_log":{"critical_warning":2,"available_spare":100,"available_spare_threshold":10,"percentage_used":1,"media_errors":0}}`
|
||||
nvmeNoStatusDegraded = `{"smartctl":{"exit_status":0},"model_name":"Old NVMe","nvme_smart_health_information_log":{"critical_warning":4,"available_spare":100,` +
|
||||
`"available_spare_threshold":10,"percentage_used":20,"media_errors":0}}`
|
||||
nvmeNoStatusWell = `{"smartctl":{"exit_status":0},"model_name":"Old NVMe","nvme_smart_health_information_log":{"critical_warning":0,"available_spare":100,` +
|
||||
`"available_spare_threshold":10,"percentage_used":20,"media_errors":0}}`
|
||||
// An old-age attribute failing now, and another that failed in the past; the overall verdict passed.
|
||||
ataOldAge = `{"smartctl":{"exit_status":32},"model_name":"ST16000NM001G","smart_support":{"available":true,"enabled":true},"smart_status":{"passed":true},` +
|
||||
`"ata_smart_attributes":{"table":[{"id":5,"name":"Reallocated_Sector_Ct","value":100,"thresh":10,"when_failed":"","flags":{"prefailure":true},"raw":{"value":0}},` +
|
||||
`{"id":194,"name":"Temperature_Celsius","value":40,"thresh":45,"when_failed":"now","flags":{"prefailure":false},"raw":{"value":60}},` +
|
||||
`{"id":190,"name":"Airflow_Temperature_Cel","value":40,"thresh":45,"when_failed":"past","flags":{"prefailure":false},"raw":{"value":60}}]}}`
|
||||
)
|
||||
|
||||
func TestTheNVMeTemperatureBitAloneIsHighNotFailing(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmeHot})
|
||||
l := f.machine().LookNow()
|
||||
d := l.Drives[0]
|
||||
if d.Flag != FlagHigh || d.Health != "passed" {
|
||||
t.Errorf("the temperature bit alone is HIGH, and the verdict it implies is not a failure: %+v", d)
|
||||
}
|
||||
if !l.Health().Healthy {
|
||||
t.Errorf("health = %+v", l.Health())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNVMeWithoutSmartStatusIsJudgedFromItsHealthLog(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmeNoStatusDegraded})
|
||||
if d := f.machine().LookNow().Drives[0]; d.Flag != FlagFailing {
|
||||
t.Errorf("a degraded log is failing without smart_status: %+v", d)
|
||||
}
|
||||
f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmeNoStatusWell})
|
||||
if d := f.machine().LookNow().Drives[0]; d.Health != "passed" || d.Flag != "" {
|
||||
t.Errorf("a clean log has passed: %+v", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOldAgeAttributeFailingNowAndAPastFailureAreHigh(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.readings(f.now, bootID, map[string]string{"sda": ataOldAge})
|
||||
l := f.machine().LookNow()
|
||||
d := l.Drives[0]
|
||||
if d.Flag != FlagHigh || d.Health != "passed" {
|
||||
t.Fatalf("an old-age attribute and a past failure are HIGH: %+v", d)
|
||||
}
|
||||
joined := strings.Join(d.Findings, "; ")
|
||||
for _, want := range []string{"Temperature_Celsius", "Airflow_Temperature_Cel was at or below its own threshold in the past"} {
|
||||
if !strings.Contains(joined, want) {
|
||||
t.Errorf("findings miss %q: %s", want, joined)
|
||||
}
|
||||
}
|
||||
if !l.Health().Healthy {
|
||||
t.Errorf("health = %+v", l.Health())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAZeroFanGPUUnderItsMinimumIsNotStopped(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
f.files("/sys/class/hwmon/hwmon12", map[string]string{"name": "amdgpu", "fan1_input": "0", "fan1_min": "500", "temp1_input": "45000", "temp1_label": "edge"})
|
||||
f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmePassed}, "amdgpu/fan1")
|
||||
l := f.machine().LookNow()
|
||||
var gpu Fan
|
||||
for _, x := range l.Fans {
|
||||
if x.Key == "amdgpu/fan1" {
|
||||
gpu = x
|
||||
}
|
||||
}
|
||||
if gpu.Flag != "" || gpu.Note == "" {
|
||||
t.Errorf("a zero-fan GPU at rest is said, not flagged: %+v", gpu)
|
||||
}
|
||||
if !l.Health().Healthy {
|
||||
t.Errorf("health = %+v", l.Health())
|
||||
}
|
||||
}
|
||||
|
||||
func TestALatchedAlarmOnATurningFanRaisesNothing(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.file("/sys/class/hwmon/hwmon1/fan1_alarm", "1")
|
||||
f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataPassed, "sdc": ataPassed}, "nct6775/fan1")
|
||||
l := f.machine().LookNow()
|
||||
if l.Fans[0].Flag != "" || !strings.Contains(l.Fans[0].Note, "alarm") {
|
||||
t.Errorf("a latched alarm while turning is said, not flagged: %+v", l.Fans[0])
|
||||
}
|
||||
if !l.Health().Healthy {
|
||||
t.Errorf("health = %+v", l.Health())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnrelatedHotSensorDoesNotMakeAFanStopped(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
f.file("/sys/class/hwmon/hwmon7/temp1_input", "84000") // the NVMe drive past its own maximum
|
||||
f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmePassed}, "asus/fan1", "asus/fan2")
|
||||
l := f.machine().LookNow()
|
||||
if l.Fans[2].Flag != "" {
|
||||
t.Fatalf("a hot drive says nothing of the GPU fan: %+v", l.Fans[2])
|
||||
}
|
||||
// The board's zone at its hot trip does.
|
||||
f.file("/sys/class/thermal/thermal_zone0/temp", "124000")
|
||||
l = f.machine().LookNow()
|
||||
if l.Fans[2].Flag != FlagStopped {
|
||||
t.Fatalf("the board at its hot trip with a fan that ran reading 0: %+v", l.Fans[2])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFanIsStoppedByAHotSensorOnItsOwnChip(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.files("/sys/class/hwmon/hwmon9", map[string]string{"name": "pwmchip", "fan1_input": "0", "temp1_input": "71000", "temp1_max": "70000"})
|
||||
f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataPassed, "sdc": ataPassed}, "pwmchip/fan1")
|
||||
h := f.machine().LookNow().Health()
|
||||
if h.Healthy || !strings.Contains(h.Why, "at 71.0 °C, at or above its own maximum of 70.0 °C") {
|
||||
t.Errorf("health = %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPhantom127SensorIsNotReadAndABoardChipsCriticalRaisesNothing(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.files("/sys/class/hwmon/hwmon1", map[string]string{"temp1_input": "105000", "temp1_label": "SYSTIN", "temp1_crit": "100000",
|
||||
"temp2_input": "127000", "temp2_label": "AUXTIN0", "temp2_crit": "100000", "temp3_input": "127500", "temp3_label": "AUXTIN1"})
|
||||
f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataPassed, "sdc": ataPassed})
|
||||
l := f.machine().LookNow()
|
||||
for _, x := range l.Temperatures {
|
||||
if x.Chip != "nct6775" {
|
||||
continue
|
||||
}
|
||||
switch x.Label {
|
||||
case "SYSTIN":
|
||||
if x.Flag != FlagCritical {
|
||||
t.Errorf("SYSTIN is shown CRITICAL: %+v", x)
|
||||
}
|
||||
default:
|
||||
if x.C != nil || x.NotRead == "" {
|
||||
t.Errorf("%s at 127 °C is a phantom: %+v", x.Label, x)
|
||||
}
|
||||
}
|
||||
}
|
||||
if !l.Health().Healthy {
|
||||
t.Errorf("a board chip's critical value drives no health: %+v", l.Health())
|
||||
}
|
||||
}
|
||||
|
||||
func TestZeroDegreesIsAReading(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.files("/sys/class/hwmon/hwmon0", map[string]string{"name": "outdoor", "temp1_input": "0"})
|
||||
ts := f.machine().Temperatures()
|
||||
if len(ts) != 1 || ts[0].C == nil || *ts[0].C != 0 {
|
||||
t.Errorf("0 °C is a reading: %+v", ts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMissingOrStaleWatchMakesTheModuleUnhealthy(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
if h := f.machine().LookNow().Health(); h.Healthy || !strings.Contains(h.Why, "drive reader has left no readings") ||
|
||||
!strings.Contains(h.Why, "fan watch has left no readings") {
|
||||
t.Errorf("no readings at all: %+v", h)
|
||||
}
|
||||
f.readings(f.now.Add(-StaleAfter-time.Minute), bootID, map[string]string{"nvme0n1": nvmePassed})
|
||||
if h := f.machine().LookNow().Health(); h.Healthy || !strings.Contains(h.Why, "has not written its readings for 6 min, so it is hung or gone") {
|
||||
t.Errorf("stale readings: %+v", h)
|
||||
}
|
||||
f.readings(f.now.Add(-time.Minute), bootID, map[string]string{"nvme0n1": nvmePassed})
|
||||
if h := f.machine().LookNow().Health(); !h.Healthy {
|
||||
t.Errorf("fresh readings: %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSmartctlsStandbyAnswerIsAsleepOnlyWithItsPowerMode(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.drive("sda", "X", "ATA ", 100, true)
|
||||
d := f.machine().BlockDrives()[0]
|
||||
f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: standbyJSON, Status: 2}
|
||||
before := DriveReading{ReadAt: f.now.Add(-time.Hour), Smartctl: []byte(ataPassed)}
|
||||
got := f.machine().ReadDrive(d, &before)
|
||||
if !got.Asleep || string(got.Smartctl) != ataPassed {
|
||||
t.Errorf("standby keeps the reading before: %+v", got)
|
||||
}
|
||||
f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: openFailedJSON, Status: 2}
|
||||
got = f.machine().ReadDrive(d, &before)
|
||||
if got.Asleep {
|
||||
t.Errorf("an open failure is not sleep, whatever its words: %+v", got)
|
||||
}
|
||||
f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: noPowerModeJSON, Status: 3}
|
||||
got = f.machine().ReadDrive(d, nil)
|
||||
if got.Asleep || !strings.Contains(got.NotRead, "cannot say whether it is asleep") {
|
||||
t.Errorf("a bridge without a power mode is skipped and said: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASmartctlThatHangsCostsOneLineAndTheReaderGoesOn(t *testing.T) {
|
||||
old := CommandTimeout
|
||||
CommandTimeout = 300 * time.Millisecond
|
||||
defer func() { CommandTimeout = old }()
|
||||
start := time.Now()
|
||||
// A child that keeps the output open after its parent is ended.
|
||||
r := ExecRunner(context.Background(), "sh", "-c", "sleep 30 & sleep 30")
|
||||
if took := time.Since(start); took > 5*time.Second {
|
||||
t.Fatalf("a hung command held the watch for %s", took)
|
||||
}
|
||||
if r.Err == "" {
|
||||
t.Errorf("the hang is said: %+v", r)
|
||||
}
|
||||
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
release := make(chan struct{})
|
||||
m := f.machine()
|
||||
m.Run = func(_ ctxT, name string, args ...string) Ran { <-release; return Ran{Stdout: ataPassed} }
|
||||
w := NewDriveReader(m)
|
||||
w.Async = true
|
||||
if err := w.Look(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.now = f.now.Add(FanEvery)
|
||||
if err := w.Look(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r2, why := m.ReadReadings(DrivesPath)
|
||||
if why != "" || !r2.WrittenAt.Equal(f.now) || r2.Drives["sda"].InFlightSince == nil {
|
||||
t.Errorf("the reader writes its file, each read marked in flight, while the drives hang: %+v %q", r2, why)
|
||||
}
|
||||
close(release)
|
||||
w.WaitReads()
|
||||
}
|
||||
|
||||
func TestASpinningDriveIsReadAtMostHourlyAndASolidStateOneEveryTenMinutes(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.drive("nvme0n1", "N", "", 100, false)
|
||||
for _, d := range []string{"sda", "sdb", "sdc"} {
|
||||
f.answers[smartctlLine(d, "sat")] = Ran{Stdout: ataPassed}
|
||||
}
|
||||
f.answers[smartctlLine("nvme0n1", "nvme")] = Ran{Stdout: nvmePassed}
|
||||
w := NewDriveReader(f.machine())
|
||||
if err := w.Look(); err != nil || len(f.calls) != 4 {
|
||||
t.Fatalf("first look reads every drive: %q %v", f.calls, err)
|
||||
}
|
||||
f.calls = nil
|
||||
f.now = f.now.Add(SolidStateEvery)
|
||||
_ = w.Look()
|
||||
if len(f.calls) != 1 || !strings.HasSuffix(f.calls[0], "/dev/nvme0n1") {
|
||||
t.Fatalf("after ten minutes only the solid-state drive: %q", f.calls)
|
||||
}
|
||||
f.calls = nil
|
||||
f.now = f.now.Add(SpinningEvery)
|
||||
_ = w.Look()
|
||||
if len(f.calls) != 4 {
|
||||
t.Fatalf("after an hour every drive: %q", f.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnExistingReadingsDirectoryGetsItsMode(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.dir("/run/mesh-sensors")
|
||||
if err := chmod(f.root+"/run/mesh-sensors", 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.machine().WriteReadings(DrivesPath, Readings{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if mode := modeOf(t, f.root+"/run/mesh-sensors"); mode != 0o755 {
|
||||
t.Errorf("mode %o", mode)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
package main
|
||||
|
||||
// The reproductions of the second review of 2026-10-08 (mesh-catalog #129, findings A to E).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A: a read stuck in the kernel is never ended, so it is seen by its age, and one stuck drive does not
|
||||
// stop the others being read.
|
||||
func TestAStuckReadIsUnhealthyAndDoesNotStopTheOtherDrives(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
stuck := make(chan struct{})
|
||||
m := f.machine()
|
||||
m.Run = func(_ context.Context, name string, args ...string) Ran {
|
||||
if strings.HasSuffix(args[len(args)-1], "/dev/sda") {
|
||||
<-stuck
|
||||
}
|
||||
return Ran{Stdout: ataPassed}
|
||||
}
|
||||
w := NewDriveReader(m)
|
||||
w.Async = true
|
||||
looked := make(chan error, 1)
|
||||
go func() { looked <- w.Look() }()
|
||||
select {
|
||||
case err := <-looked:
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("a look waited on a stuck drive")
|
||||
}
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
_ = w.write()
|
||||
r, _ := m.ReadReadings(DrivesPath)
|
||||
if len(r.Drives["sdb"].Smartctl) > 0 && len(r.Drives["sdc"].Smartctl) > 0 {
|
||||
if r.Drives["sda"].InFlightSince == nil {
|
||||
t.Fatalf("sda is in flight: %+v", r.Drives["sda"])
|
||||
}
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("one stuck drive held up the others: %+v", r.Drives)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
f.fresh2(w)
|
||||
if h := m.LookNow().Health(); !h.Healthy {
|
||||
t.Fatalf("a read under way is not yet stuck: %+v", h)
|
||||
}
|
||||
f.now = f.now.Add(ReadBound() + time.Minute)
|
||||
_ = w.Look()
|
||||
f.write(FansPath, Readings{BootID: bootID, WrittenAt: f.now})
|
||||
h := m.LookNow().Health()
|
||||
if h.Healthy || !strings.Contains(h.Why, "The read of the drive sda has not finished for 1 min; the drive or its controller may have stopped answering.") {
|
||||
t.Errorf("health = %+v", h)
|
||||
}
|
||||
close(stuck)
|
||||
w.WaitReads()
|
||||
}
|
||||
|
||||
// fresh2 writes the fan watch's file now, beside a drive reader that has just written its own.
|
||||
func (f *fake) fresh2(w *DriveReader) {
|
||||
_ = w.write()
|
||||
f.write(FansPath, Readings{BootID: bootID, WrittenAt: f.now})
|
||||
}
|
||||
|
||||
// A: a drive the reader has not looked at for twice its cadence is said, though the file is fresh.
|
||||
func TestADriveNotLookedAtForTwiceItsCadenceIsUnhealthy(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.fresh()
|
||||
r, _ := f.machine().ReadReadings(DrivesPath)
|
||||
x := r.Drives["sdb"]
|
||||
x.LookedAt = f.now.Add(-2*SpinningEvery - time.Minute)
|
||||
r.Drives["sdb"] = x
|
||||
f.write(DrivesPath, *r)
|
||||
h := f.machine().LookNow().Health()
|
||||
if h.Healthy || !strings.Contains(h.Why, "The drive sdb has not been looked at for 2.0 h, more than twice as long as it should be.") {
|
||||
t.Errorf("health = %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
// B: a board chip's own sensor at its maximum says nothing of the board chip's fan; a CPU sensor does.
|
||||
func TestABoardChipsOwnSensorDoesNotStopItsFanAndACPUSensorDoes(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.files("/sys/class/hwmon/hwmon1", map[string]string{"fan1_input": "0", "temp1_input": "75000", "temp1_max": "70000", "temp1_label": "SYSTIN"})
|
||||
f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataPassed, "sdc": ataPassed}, "nct6775/fan1")
|
||||
l := f.machine().LookNow()
|
||||
if l.Fans[0].Flag != "" || !l.Health().Healthy {
|
||||
t.Fatalf("a Super I/O's maximum is not trusted: %+v %+v", l.Fans[0], l.Health())
|
||||
}
|
||||
f.file("/sys/class/hwmon/hwmon0/temp1_input", "81000")
|
||||
l = f.machine().LookNow()
|
||||
if l.Fans[0].Flag != FlagStopped || !strings.Contains(l.Fans[0].Why, "the CPU (coretemp Package id 0) is at 81.0 °C") {
|
||||
t.Errorf("the CPU at its maximum with a fan that ran reading 0: %+v", l.Fans[0])
|
||||
}
|
||||
}
|
||||
|
||||
// D: a SAS drive in standby says so in words and no ATA power mode, and its modes hold spaces
|
||||
// (scsiprint.cpp of smartmontools 7.5); status 2 with those words is asleep. An idle mode never stops
|
||||
// smartctl under -n standby, so it never says so.
|
||||
func TestASASDriveInStandbyIsAsleep(t *testing.T) {
|
||||
for _, mode := range []string{"STANDBY BY TIMER", "STANDBY BY COMMAND", "STANDBY_Y BY TIMER", "STANDBY_Y BY COMMAND", "LOW POWER", "SLEEP"} {
|
||||
f := newFake(t)
|
||||
f.drive("sdm", "ST4000NM0023", "SEAGATE ", 100, true)
|
||||
d := f.machine().BlockDrives()[0]
|
||||
sas := `{"smartctl":{"messages":[{"string":"Device is in ` + mode + ` mode, exit(2)","severity":"information"}],"exit_status":2},` +
|
||||
`"device":{"name":"/dev/sdm","type":"scsi","protocol":"SCSI"}}`
|
||||
f.answers[smartctlLine("sdm", "scsi")] = Ran{Stdout: sas, Status: 2}
|
||||
before := DriveReading{ReadAt: f.now.Add(-time.Hour), Smartctl: []byte(ataPassed)}
|
||||
got := f.machine().ReadDrive(d, &before)
|
||||
if !got.Asleep || string(got.Smartctl) != ataPassed || got.AsleepSince == nil {
|
||||
t.Errorf("a SAS drive in %s keeps its reading: %+v", mode, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// E: a drive asleep at every look for a day is HIGH, said with since when, and the module stays healthy.
|
||||
func TestADriveAsleepForADayIsHighAndNotUnhealthy(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.fresh()
|
||||
r, _ := f.machine().ReadReadings(DrivesPath)
|
||||
since := f.now.Add(-25 * time.Hour)
|
||||
x := r.Drives["sdc"]
|
||||
x.Asleep, x.AsleepSince = true, &since
|
||||
r.Drives["sdc"] = x
|
||||
f.write(DrivesPath, *r)
|
||||
l := f.machine().LookNow()
|
||||
d := l.Drives[2]
|
||||
if d.Flag != FlagHigh || d.AsleepSince != "2026-10-07T11:00:00Z" || !strings.Contains(strings.Join(d.Findings, ";"), "asleep at every look for 25.0 h") {
|
||||
t.Errorf("drive = %+v", d)
|
||||
}
|
||||
if !l.Health().Healthy {
|
||||
t.Errorf("a drive asleep is never unhealthy on its own: %+v", l.Health())
|
||||
}
|
||||
if !strings.Contains(strings.Join(l.Said(), "\n"), "HIGH: the drive sdc") {
|
||||
t.Errorf("the summary says it: %q", l.Said())
|
||||
}
|
||||
}
|
||||
|
||||
// C: smartctl is told the drive's kind where it is known, so it does not probe a drive to find out.
|
||||
func TestSmartctlIsToldTheDrivesKind(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.drive("nvme0n1", "N", "", 100, false)
|
||||
f.drive("sda", "A", "ATA ", 100, true)
|
||||
f.drive("sdm", "S", "SEAGATE ", 100, true)
|
||||
f.drive("vda", "", "", 100, true)
|
||||
w := NewDriveReader(f.machine())
|
||||
_ = w.Look()
|
||||
want := []string{smartctlLine("nvme0n1", "nvme"), smartctlLine("sda", "sat"), smartctlLine("sdm", "scsi"), smartctlLine("vda", "")}
|
||||
if strings.Join(f.calls, "\n") != strings.Join(want, "\n") {
|
||||
t.Errorf("calls:\n%s\nwant:\n%s", strings.Join(f.calls, "\n"), strings.Join(want, "\n"))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,382 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func find(ts []Temperature, chip, label string) Temperature {
|
||||
for _, t := range ts {
|
||||
if t.Chip == chip && t.Label == label {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return Temperature{}
|
||||
}
|
||||
|
||||
func TestALaptopsTemperaturesCarryOnlyTheHardwaresOwnThresholds(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
ts := f.machine().Temperatures()
|
||||
if len(ts) != 7 {
|
||||
t.Fatalf("read %d temperatures, want 7 (the zone's hwmon copy left out): %+v", len(ts), ts)
|
||||
}
|
||||
if ts[0].Chip != "k10temp" || ts[0].Kind != "cpu" {
|
||||
t.Errorf("the CPU is said first, got %+v", ts[0])
|
||||
}
|
||||
comp := find(ts, "nvme", "Composite")
|
||||
if comp.High == nil || *comp.High != 83.9 || comp.Critical == nil || *comp.Critical != 87.9 || comp.Drive != "nvme0n1 (WD PC SN560 SDDPNQE-1T00-1102)" {
|
||||
t.Errorf("the NVMe composite sensor keeps its own thresholds and names its drive: %+v", comp)
|
||||
}
|
||||
if s1 := find(ts, "nvme", "Sensor 1"); s1.High != nil || s1.Flag != "" {
|
||||
t.Errorf("a maximum of 65261.85 °C is no threshold: %+v", s1)
|
||||
}
|
||||
if cpu := find(ts, "k10temp", "Tctl"); cpu.High != nil || cpu.Critical != nil || cpu.Flag != "" || *cpu.C != 60.9 {
|
||||
t.Errorf("a sensor with no threshold is read and never flagged: %+v", cpu)
|
||||
}
|
||||
zone := find(ts, "acpitz", "thermal_zone0")
|
||||
if zone.Source != "thermal zone" || *zone.High != 124 || zone.HighIs != "the zone's hot trip" || *zone.Critical != 124 {
|
||||
t.Errorf("the zone takes its hot and critical trips, not its active one: %+v", zone)
|
||||
}
|
||||
for _, x := range ts {
|
||||
if x.Flag != "" {
|
||||
t.Errorf("nothing on a cool laptop is flagged: %+v", x)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestATemperatureIsHighAtItsMaximumAndCriticalAtItsCriticalValue(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.file("/sys/class/hwmon/hwmon0/temp2_input", "80000")
|
||||
f.file("/sys/class/hwmon/hwmon0/temp1_input", "100000")
|
||||
f.fresh()
|
||||
ts := f.machine().Temperatures()
|
||||
if core := find(ts, "coretemp", "Core 0"); core.Flag != FlagHigh {
|
||||
t.Errorf("at its own maximum it is HIGH: %+v", core)
|
||||
}
|
||||
pkg := find(ts, "coretemp", "Package id 0")
|
||||
if pkg.Flag != FlagCritical {
|
||||
t.Fatalf("at its own critical value it is CRITICAL: %+v", pkg)
|
||||
}
|
||||
h := f.machine().LookNow().Health()
|
||||
want := "The CPU (coretemp Package id 0) is at 100.0 °C, at or above its own critical value of 100.0 °C."
|
||||
if h.Healthy || h.Why != want {
|
||||
t.Errorf("health = %+v, want unhealthy with %q", h, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAZoneWithoutAHotTripIsHighAtItsPassiveTrip(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.files("/sys/class/thermal/thermal_zone3", map[string]string{"type": "x86_pkg_temp", "temp": "96000",
|
||||
"trip_point_0_type": "passive", "trip_point_0_temp": "95000", "trip_point_1_type": "passive", "trip_point_1_temp": "98000"})
|
||||
ts := f.machine().Temperatures()
|
||||
if len(ts) != 1 || ts[0].Kind != "cpu" || *ts[0].High != 95 || ts[0].Flag != FlagHigh || ts[0].Critical != nil {
|
||||
t.Fatalf("the lowest passive trip is its high, and it has no critical: %+v", ts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASensorThatGivesNoReadingIsSaidAsNotRead(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.files("/sys/class/hwmon/hwmon0", map[string]string{"name": "iwlwifi_1", "temp1_label": "x"})
|
||||
f.dir("/sys/class/hwmon/hwmon0/temp1_input") // a read that fails, as ENODATA does
|
||||
f.files("/sys/class/thermal/thermal_zone0", map[string]string{"type": "acpitz", "temp": "-274000"})
|
||||
ts := f.machine().Temperatures()
|
||||
if len(ts) != 2 {
|
||||
t.Fatalf("both are listed: %+v", ts)
|
||||
}
|
||||
for _, x := range ts {
|
||||
if x.C != nil || x.NotRead == "" || x.Flag != "" {
|
||||
t.Errorf("not read, never cool: %+v", x)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(ts[0].NotRead, "-274.0 °C, which is no temperature") {
|
||||
t.Errorf("the zone says what it answered: %q", ts[0].NotRead)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMachineWithNoSensorAtAllHasEveryClassNotPresentAndIsHealthy(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.drive("vda", "", "", 41943040, true)
|
||||
f.readings(f.now, bootID, map[string]string{"vda": noSMART})
|
||||
l := f.machine().LookNow()
|
||||
said := strings.Join(l.Said(), "\n")
|
||||
for _, want := range []string{
|
||||
"temperatures: not present", "fans: not present", "drives: 1, 0 passed, 0 failing, 1 not read (vda: the drive does not support SMART)",
|
||||
"memory not present", "PCIe not present",
|
||||
} {
|
||||
if !strings.Contains(said, want) {
|
||||
t.Errorf("summary misses %q:\n%s", want, said)
|
||||
}
|
||||
}
|
||||
if h := l.Health(); !h.Healthy || !strings.Contains(h.Why, "The health of 1 of 1 drives has not been read.") {
|
||||
t.Errorf("a machine without sensors is healthy, and says what it could not read: %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
func TestADriveThatSaysItIsFailingMakesTheModuleUnhealthy(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataFailing, "sdc": ataPending})
|
||||
ds, note := f.machine().Drives(f.machine().ReadReadings(DrivesPath))
|
||||
if note != "" || len(ds) != 3 {
|
||||
t.Fatalf("three drives, the RAID volume left out, no note: %d, %q", len(ds), note)
|
||||
}
|
||||
if ds[0].Health != "passed" || ds[0].Flag != "" || *ds[0].Reallocated != 0 || ds[0].LastSelfTest != "Short offline: Completed without error" || ds[0].Transport != "ata" {
|
||||
t.Errorf("sda passed: %+v", ds[0])
|
||||
}
|
||||
if ds[1].Health != "failing" || ds[1].Flag != FlagFailing {
|
||||
t.Errorf("sdb is failing: %+v", ds[1])
|
||||
}
|
||||
if ds[2].Health != "passed" || ds[2].Flag != FlagHigh || ds[2].Findings[0] != "it has 8 sectors waiting to be reallocated" {
|
||||
t.Errorf("sdc's pending sectors are an early sign, HIGH: %+v", ds[2])
|
||||
}
|
||||
h := f.machine().LookNow().Health()
|
||||
want := "The drive sdb (WDC WUH722222ALE6L4) says it is failing: its overall SMART health check failed and " +
|
||||
"its attribute Reallocated_Sector_Ct is at 1, at or below its own threshold of 10."
|
||||
if h.Healthy || h.Why != want {
|
||||
t.Errorf("health = %+v\nwant why %q", h, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnNVMeCriticalWarningBitIsFailing(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmeSpareLow})
|
||||
ds, _ := f.machine().Drives(f.machine().ReadReadings(DrivesPath))
|
||||
d := ds[0]
|
||||
if d.Flag != FlagFailing || *d.CriticalWarning != 1 || *d.PercentageUsed != 97 || d.Model != "Samsung SSD 990 PRO 2TB" {
|
||||
t.Fatalf("spare below threshold is failing: %+v", d)
|
||||
}
|
||||
joined := strings.Join(d.Findings, "; ")
|
||||
for _, want := range []string{"its spare capacity is below its threshold", "it has had 3 media errors", "it has used 97 % of its rated endurance"} {
|
||||
if !strings.Contains(joined, want) {
|
||||
t.Errorf("findings miss %q: %s", want, joined)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAHealthyLaptopReadsHealthyWithItsDriveRead(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
f.readings(f.now.Add(-5*time.Minute), bootID, map[string]string{"nvme0n1": nvmePassed}, "asus/fan2")
|
||||
l := f.machine().LookNow()
|
||||
if h := l.Health(); !h.Healthy || strings.Contains(h.Why, "not been read") {
|
||||
t.Errorf("health = %+v", h)
|
||||
}
|
||||
d := l.Drives[0]
|
||||
if d.Health != "passed" || *d.PowerOnHours != 2210 || *d.AvailableSpare != 100 || d.Serial != "S1" {
|
||||
t.Errorf("drive = %+v", d)
|
||||
}
|
||||
said := l.Said()
|
||||
if len(said) != 4 || !strings.HasPrefix(said[0], "temperatures: 7 read; the hottest is the CPU (k10temp Tctl) at 60.9 °C, which reports no threshold of its own") {
|
||||
t.Errorf("one line per class and no finding: %q", said)
|
||||
}
|
||||
if said[2] != "drives: 1, 1 passed, 0 failing" {
|
||||
t.Errorf("drives line = %q", said[2])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMissingStaleOrEarlierBootReadingIsSaid(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
if ds, note := f.machine().Drives(f.machine().ReadReadings(DrivesPath)); ds[0].Health != "not read" || !strings.Contains(note, "has left no readings") {
|
||||
t.Errorf("no file: %+v %q", ds[0], note)
|
||||
}
|
||||
f.readings(f.now.Add(-2*time.Hour), bootID, map[string]string{"nvme0n1": nvmePassed})
|
||||
if ds, note := f.machine().Drives(f.machine().ReadReadings(DrivesPath)); ds[0].Health != "passed" || note != "the module's drive reader last wrote 2.0 h ago; it may not be running" {
|
||||
t.Errorf("stale: %+v %q", ds[0], note)
|
||||
}
|
||||
f.readings(f.now, "an-earlier-boot", map[string]string{"nvme0n1": nvmeSpareLow}, "asus/fan2")
|
||||
l := f.machine().LookNow()
|
||||
if l.Drives[0].Health != "not read" || l.Fans[2].RanSinceBoot != nil {
|
||||
t.Errorf("a reading from before the machine started is not this boot's: %+v %+v", l.Drives[0], l.Fans[2])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFanAtRestOnACoolMachineIsNotStopped(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmePassed}, "asus/fan2")
|
||||
l := f.machine().LookNow()
|
||||
gpu := l.Fans[2]
|
||||
if gpu.Label != "gpu_fan" || *gpu.RPM != 0 || gpu.RanSinceBoot == nil || !*gpu.RanSinceBoot || gpu.Flag != "" {
|
||||
t.Errorf("a fan that ran and rests while nothing is hot is not stopped: %+v", gpu)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFanThatRanAndReadsZeroWhileTheBoardIsHotHasStopped(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
f.file("/sys/class/thermal/thermal_zone0/temp", "124500")
|
||||
f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmePassed}, "asus/fan1", "asus/fan2")
|
||||
l := f.machine().LookNow()
|
||||
if l.Fans[2].Flag != FlagStopped {
|
||||
t.Fatalf("gpu_fan ran and reads 0 while the board is past its hot trip: %+v", l.Fans[2])
|
||||
}
|
||||
h := l.Health()
|
||||
want := "The board (acpitz thermal_zone0) is at 124.5 °C, at or above the zone's critical trip of 124.0 °C. " +
|
||||
"The fan gpu_fan (asus) has stopped: it ran earlier and reads 0 RPM while the board (acpitz thermal_zone0) " +
|
||||
"is at 124.5 °C, at or above the zone's hot trip of 124.0 °C."
|
||||
if h.Healthy || h.Why != want {
|
||||
t.Errorf("health = %+v\nwant %q", h, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEmptyHeaderAndAFanUnderItsMinimumAreSaidAndNotFlagged(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
l := f.machine().LookNow()
|
||||
if l.Fans[1].Flag != "" {
|
||||
t.Errorf("a header never seen turning, with an alarm the firmware set, is no fan: %+v", l.Fans[1])
|
||||
}
|
||||
f.file("/sys/class/hwmon/hwmon1/fan1_input", "200")
|
||||
l = f.machine().LookNow()
|
||||
if l.Fans[0].Flag != "" || l.Fans[0].Note != "it turns at 200 RPM, under its chip's minimum of 300 RPM" {
|
||||
t.Errorf("under its own minimum is a note: %+v", l.Fans[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestMemoryErrorsAreCountedAndAnUncorrectableOneIsUnhealthy(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.fresh()
|
||||
f.file("/sys/devices/system/edac/mc/mc0/ce_count", "3")
|
||||
f.file("/sys/devices/system/edac/mc/mc0/dimm0/dimm_ce_count", "3")
|
||||
l := f.machine().LookNow()
|
||||
if c := l.Faults.Memory[0]; c.Flag != FlagHigh || len(c.Modules) != 1 || c.Modules[0].Label != "CPU_SrcID#0_MC#0_Chan#0_DIMM#0" {
|
||||
t.Errorf("corrected errors are HIGH, by module: %+v", c)
|
||||
}
|
||||
if !l.Health().Healthy {
|
||||
t.Errorf("corrected errors raise nothing")
|
||||
}
|
||||
f.file("/sys/devices/system/edac/mc/mc0/ue_count", "1")
|
||||
h := f.machine().LookNow().Health()
|
||||
if h.Healthy || h.Why != "The memory reports 1 uncorrectable errors since the machine started (memory controller mc0)." {
|
||||
t.Errorf("health = %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPCIeErrorsAndThrottlingAreCountedWhereTheMachineHasThem(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.laptop()
|
||||
fs := f.machine().Faults()
|
||||
if fs.PCIDevicesCounting != 1 || len(fs.PCI) != 0 || fs.ThrottledCore != nil || fs.MemoryNotPresent == "" {
|
||||
t.Errorf("faults = %+v", fs)
|
||||
}
|
||||
f.file("/sys/bus/pci/devices/0000:04:00.0/aer_dev_fatal", "TOTAL_ERR_FATAL 2\n")
|
||||
f.file("/sys/devices/system/cpu/cpu0/thermal_throttle/core_throttle_count", "5")
|
||||
f.file("/sys/devices/system/cpu/cpu1/thermal_throttle/core_throttle_count", "7")
|
||||
fs = f.machine().Faults()
|
||||
if len(fs.PCI) != 1 || fs.PCI[0].Fatal != 2 || fs.PCI[0].Driver != "nvme" || fs.PCI[0].Flag != FlagHigh || *fs.ThrottledCore != 12 {
|
||||
t.Errorf("faults = %+v", fs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheDriveReaderReadsEachDriveAsRootWithoutWakingItAndTheFanWatchNotesTheFans(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: ataPassed}
|
||||
f.answers[smartctlLine("sdb", "sat")] = Ran{Stdout: ataFailing, Status: 24}
|
||||
f.answers[smartctlLine("sdc", "sat")] = Ran{Stdout: strings.Replace(standbyJSON, "/dev/sda", "/dev/sdc", 1), Status: 2}
|
||||
w := NewDriveReader(f.machine())
|
||||
if err := w.Look(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(f.calls) != 3 {
|
||||
t.Fatalf("one smartctl per drive, directly as root, told it is SATA: %q", f.calls)
|
||||
}
|
||||
fw := NewFanWatch(f.machine())
|
||||
if err := fw.Look(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, p := range []string{DrivesPath, FansPath} {
|
||||
info, err := os.Stat(filepath.Join(f.root, p))
|
||||
if err != nil || info.Mode().Perm() != 0o644 {
|
||||
t.Fatalf("%s is readable by the operator account: %v %v", p, info, err)
|
||||
}
|
||||
}
|
||||
r, why := f.machine().ReadReadings(DrivesPath)
|
||||
if why != "" || r.Drives["sdc"].NotRead != "the drive was asleep at every look, and is not woken to be read" || !r.Drives["sdc"].Asleep ||
|
||||
r.Drives["sdc"].AsleepSince == nil || !r.Drives["sdc"].AsleepSince.Equal(f.now) {
|
||||
t.Fatalf("sdc asleep since now: %+v %q", r.Drives["sdc"], why)
|
||||
}
|
||||
fr, _ := f.machine().ReadReadings(FansPath)
|
||||
if _, ok := fr.FansRan["nct6775/fan1"]; !ok || len(fr.FansRan) != 1 {
|
||||
t.Errorf("the turning fan is noted, the empty header not: %v", fr.FansRan)
|
||||
}
|
||||
// An hour has not passed: the spinning drives are not read again.
|
||||
f.now = f.now.Add(FanEvery)
|
||||
f.calls = nil
|
||||
if err := w.Look(); err != nil || len(f.calls) != 0 {
|
||||
t.Fatalf("drives read again within the hour: %q %v", f.calls, err)
|
||||
}
|
||||
// A new reader (a restart) keeps what the last one read, and a drive asleep keeps its reading before.
|
||||
f.now = f.now.Add(SpinningEvery)
|
||||
f.answers[smartctlLine("sda", "sat")] = Ran{Stdout: strings.Replace(standbyJSON, `"STANDBY"}`, `"SLEEP"}`, 1), Status: 2}
|
||||
w2 := NewDriveReader(f.machine())
|
||||
if err := w2.Look(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, _ = f.machine().ReadReadings(DrivesPath)
|
||||
if !r.Drives["sda"].Asleep || len(r.Drives["sda"].Smartctl) == 0 {
|
||||
t.Errorf("sda asleep keeps its reading: %+v", r.Drives["sda"])
|
||||
}
|
||||
if since := r.Drives["sdc"].AsleepSince; since == nil || !since.Equal(f.now.Add(-SpinningEvery-FanEvery)) {
|
||||
t.Errorf("sdc asleep again keeps since when: %+v", r.Drives["sdc"])
|
||||
}
|
||||
ds, _ := f.machine().Drives(f.machine().ReadReadings(DrivesPath))
|
||||
if ds[0].Health != "passed" || !ds[0].Asleep || ds[1].Health != "failing" {
|
||||
t.Errorf("drives = %+v", ds)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheDriveReaderNotRootAsksThroughSudoAndSaysWhatStopsIt(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.uid = 1000
|
||||
f.drive("sda", "X", "ATA ", 100, true)
|
||||
d := f.machine().BlockDrives()[0]
|
||||
f.answers["sudo -n "+smartctlLine("sda", "sat")] = Ran{Stderr: "sudo: a password is required\n", Status: 1}
|
||||
got := f.machine().ReadDrive(d, nil)
|
||||
if got.NotRead != "the drive reader is not root and may not run smartctl through sudo without a prompt" {
|
||||
t.Errorf("not read = %q", got.NotRead)
|
||||
}
|
||||
f.uid = 0
|
||||
got = f.machine().ReadDrive(d, nil)
|
||||
if !strings.Contains(got.NotRead, "smartctl is not installed") {
|
||||
t.Errorf("not read = %q", got.NotRead)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheHealthAnswerIsWhatTheNodeEngineReads(t *testing.T) {
|
||||
f := newFake(t)
|
||||
f.server()
|
||||
f.file("/sys/class/hwmon/hwmon0/temp1_input", "101000")
|
||||
f.file("/sys/devices/system/edac/mc/mc0/ue_count", "2")
|
||||
f.fresh()
|
||||
var tool func(map[string]any) (any, error)
|
||||
for _, tl := range Tools(f.machine()) {
|
||||
if tl.Name == "sensors_health" {
|
||||
tool = tl.Run
|
||||
}
|
||||
}
|
||||
out, err := tool(map[string]any{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := json.Marshal(out)
|
||||
var a struct {
|
||||
Healthy bool `json:"healthy"`
|
||||
Why string `json:"why"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &a); err != nil || a.Healthy {
|
||||
t.Fatalf("answer %s", raw)
|
||||
}
|
||||
if strings.Count(a.Why, ". ") != 1 || !strings.HasSuffix(a.Why, ".") || strings.ContainsAny(a.Why, "\n`") {
|
||||
t.Errorf("one sentence per finding, plain: %q", a.Why)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Temperatures, from the kernel's hwmon chips and its thermal zones, each with the thresholds the hardware
|
||||
// gives for itself (novox/hq ADR 0256 rule 2). The mesh adds no threshold of its own: a sensor that gives
|
||||
// none is said without a flag.
|
||||
|
||||
// Flags, said beside a reading that has reached one of the hardware's own thresholds.
|
||||
const (
|
||||
FlagHigh = "HIGH"
|
||||
FlagCritical = "CRITICAL"
|
||||
)
|
||||
|
||||
// Temperature is one sensor's reading, in °C.
|
||||
type Temperature struct {
|
||||
// Source is "hwmon" or "thermal zone".
|
||||
Source string `json:"source"`
|
||||
// Chip is the hwmon chip's name (k10temp, nvme, amdgpu) or the zone's type (acpitz, x86_pkg_temp).
|
||||
Chip string `json:"chip"`
|
||||
// Kind is what it measures: cpu, gpu, drive, board, network or other.
|
||||
Kind string `json:"kind"`
|
||||
// Drive is the drive a drive sensor sits in, by its block device, with its model.
|
||||
Drive string `json:"drive,omitempty"`
|
||||
Label string `json:"label"`
|
||||
C *float64 `json:"c,omitempty"`
|
||||
// High is where the hardware says it runs hot: its own maximum, or a zone's hot or passive trip.
|
||||
High *float64 `json:"high_c,omitempty"`
|
||||
HighIs string `json:"high_is,omitempty"`
|
||||
// Critical is where the hardware says it must not go: its own critical or emergency value, or a zone's
|
||||
// critical trip.
|
||||
Critical *float64 `json:"critical_c,omitempty"`
|
||||
CriticalIs string `json:"critical_is,omitempty"`
|
||||
// Alarm is the chip's own alarm bit for this sensor, where it has one and it is set.
|
||||
Alarm bool `json:"alarm,omitempty"`
|
||||
Flag string `json:"flag,omitempty"`
|
||||
NotRead string `json:"not_read,omitempty"`
|
||||
// Where is the directory it was read from, for whoever looks closer.
|
||||
Where string `json:"where"`
|
||||
}
|
||||
|
||||
// DrivesHealth says whether this sensor's critical value makes the module unhealthy. A board chip's
|
||||
// (a Super I/O's) is not to be trusted: its inputs and their limits are whatever the firmware left, and a
|
||||
// wrongly wired input reads far past them on a healthy machine. It is shown, and raises nothing.
|
||||
func (t Temperature) DrivesHealth() bool { return !(t.Source == "hwmon" && t.Kind == "board") }
|
||||
|
||||
// Name is how a person reads which sensor this is.
|
||||
func (t Temperature) Name() string {
|
||||
what := t.Chip
|
||||
if t.Label != "" && t.Label != t.Chip {
|
||||
what += " " + t.Label
|
||||
}
|
||||
switch {
|
||||
case t.Drive != "":
|
||||
return fmt.Sprintf("the drive %s, sensor %s", t.Drive, t.Label)
|
||||
case t.Kind == "cpu":
|
||||
return fmt.Sprintf("the CPU (%s)", what)
|
||||
case t.Kind == "gpu":
|
||||
return fmt.Sprintf("the GPU (%s)", what)
|
||||
case t.Kind == "board":
|
||||
return fmt.Sprintf("the board (%s)", what)
|
||||
case t.Kind == "network":
|
||||
return fmt.Sprintf("the network adapter (%s)", what)
|
||||
}
|
||||
return "the sensor " + what
|
||||
}
|
||||
|
||||
// sane says whether a value in °C is a temperature or a threshold the hardware means as one. A
|
||||
// threshold given as the format's largest value (an NVMe sensor's 65261.85 °C) or as 0 means none, and
|
||||
// so does anything a machine cannot be at and still answer.
|
||||
func sane(c float64) bool { return reading(c) && c != 0 }
|
||||
|
||||
// reading says whether a value in °C is a temperature a machine can be at. 0 °C is one. 127 and 127.5 °C
|
||||
// are not: they are what a board chip's input with nothing on it answers (a phantom AUXTIN, the largest
|
||||
// value of its eight-bit register), never a real sensor's reading.
|
||||
func reading(c float64) bool { return c > -40 && c < 200 && c != 127 && c != 127.5 }
|
||||
|
||||
func milli(v int64) *float64 {
|
||||
c := round1(float64(v) / 1000)
|
||||
return &c
|
||||
}
|
||||
|
||||
// chipKinds is what a hwmon chip or a thermal zone's type measures, by the driver's name.
|
||||
var chipKinds = []struct {
|
||||
prefix, kind string
|
||||
}{
|
||||
{"k10temp", "cpu"}, {"coretemp", "cpu"}, {"zenpower", "cpu"}, {"cpu_thermal", "cpu"}, {"cpu-thermal", "cpu"},
|
||||
{"x86_pkg_temp", "cpu"}, {"via_cputemp", "cpu"}, {"soc_thermal", "cpu"}, {"TCPU", "cpu"},
|
||||
{"amdgpu", "gpu"}, {"radeon", "gpu"}, {"nouveau", "gpu"}, {"i915", "gpu"}, {"xe", "gpu"}, {"nvidia", "gpu"},
|
||||
{"gpu", "gpu"},
|
||||
{"nvme", "drive"}, {"drivetemp", "drive"},
|
||||
{"iwlwifi", "network"}, {"mt79", "network"}, {"mt76", "network"}, {"ath", "network"}, {"r8169", "network"},
|
||||
{"igb", "network"}, {"ixgbe", "network"}, {"mlx", "network"}, {"bnxt", "network"},
|
||||
{"acpitz", "board"}, {"pch_", "board"}, {"nct", "board"}, {"it87", "board"}, {"it86", "board"},
|
||||
{"w83", "board"}, {"f71", "board"}, {"asus", "board"}, {"dell_smm", "board"}, {"thinkpad", "board"},
|
||||
{"applesmc", "board"}, {"INT3400", "board"}, {"SEN", "board"}, {"TSKN", "board"},
|
||||
}
|
||||
|
||||
func kindOf(chip string) string {
|
||||
for _, k := range chipKinds {
|
||||
if strings.HasPrefix(chip, k.prefix) {
|
||||
return k.kind
|
||||
}
|
||||
}
|
||||
return "other"
|
||||
}
|
||||
|
||||
var kindOrder = map[string]int{"cpu": 0, "gpu": 1, "drive": 2, "board": 3, "network": 4, "other": 5}
|
||||
|
||||
// Temperatures is every temperature the machine reports: each thermal zone, then each hwmon sensor that is
|
||||
// not a zone's copy (the zone has the same reading and its trip points).
|
||||
func (m *Machine) Temperatures() []Temperature {
|
||||
var out []Temperature
|
||||
inZone := map[string]bool{}
|
||||
for _, zone := range m.numbered("/sys/class/thermal", "thermal_zone") {
|
||||
dir := "/sys/class/thermal/" + zone
|
||||
for _, n := range m.names(dir) {
|
||||
if strings.HasPrefix(n, "hwmon") {
|
||||
inZone[n] = true
|
||||
}
|
||||
}
|
||||
out = append(out, m.zone(dir))
|
||||
}
|
||||
drives := m.driveOfHwmon()
|
||||
for _, h := range m.numbered("/sys/class/hwmon", "hwmon") {
|
||||
if inZone[h] {
|
||||
continue
|
||||
}
|
||||
dir := "/sys/class/hwmon/" + h
|
||||
chip := m.read(dir + "/name")
|
||||
for _, input := range m.inputs(dir, "temp") {
|
||||
out = append(out, m.hwmonTemperature(dir, chip, input, drives[h]))
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
if kindOrder[out[i].Kind] != kindOrder[out[j].Kind] {
|
||||
return kindOrder[out[i].Kind] < kindOrder[out[j].Kind]
|
||||
}
|
||||
return false
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// inputs lists a hwmon directory's <kind>N_input files by their stem (temp1, temp2), in numeric order.
|
||||
func (m *Machine) inputs(dir, kind string) []string {
|
||||
var stems []string
|
||||
for _, n := range m.names(dir) {
|
||||
if strings.HasPrefix(n, kind) && strings.HasSuffix(n, "_input") {
|
||||
stems = append(stems, strings.TrimSuffix(n, "_input"))
|
||||
}
|
||||
}
|
||||
sortNumbered(stems, kind)
|
||||
return stems
|
||||
}
|
||||
|
||||
func (m *Machine) hwmonTemperature(dir, chip, stem, drive string) Temperature {
|
||||
t := Temperature{Source: "hwmon", Chip: chip, Kind: kindOf(chip), Drive: drive, Label: m.read(dir + "/" + stem + "_label"),
|
||||
Where: dir + "/" + stem}
|
||||
if t.Label == "" {
|
||||
t.Label = stem
|
||||
}
|
||||
if drive != "" {
|
||||
t.Kind = "drive"
|
||||
}
|
||||
v, ok := m.readInt(dir + "/" + stem + "_input")
|
||||
if !ok || !reading(float64(v)/1000) {
|
||||
t.NotRead = "the sensor gave no temperature"
|
||||
if ok {
|
||||
t.NotRead = fmt.Sprintf("the sensor answered %.1f °C, which is no temperature", float64(v)/1000)
|
||||
}
|
||||
return t
|
||||
}
|
||||
t.C = milli(v)
|
||||
if h, ok := m.readInt(dir + "/" + stem + "_max"); ok && sane(float64(h)/1000) {
|
||||
t.High, t.HighIs = milli(h), "its own maximum"
|
||||
}
|
||||
if c, ok := m.readInt(dir + "/" + stem + "_crit"); ok && sane(float64(c)/1000) {
|
||||
t.Critical, t.CriticalIs = milli(c), "its own critical value"
|
||||
} else if e, ok := m.readInt(dir + "/" + stem + "_emergency"); ok && sane(float64(e)/1000) {
|
||||
t.Critical, t.CriticalIs = milli(e), "its own emergency value"
|
||||
}
|
||||
for _, alarm := range []string{"_alarm", "_crit_alarm", "_max_alarm", "_emergency_alarm"} {
|
||||
if a, ok := m.readInt(dir + "/" + stem + alarm); ok && a != 0 {
|
||||
t.Alarm = true
|
||||
}
|
||||
}
|
||||
t.Flag = flagOf(t)
|
||||
return t
|
||||
}
|
||||
|
||||
// zone reads one thermal zone and its trip points: critical is the critical trip, high the hot trip or,
|
||||
// without one, the lowest passive trip (where the kernel starts to slow the machine down). Active trips
|
||||
// are where a fan steps up, and are nothing to report.
|
||||
func (m *Machine) zone(dir string) Temperature {
|
||||
typ := m.read(dir + "/type")
|
||||
t := Temperature{Source: "thermal zone", Chip: typ, Kind: kindOf(typ), Label: strings.TrimPrefix(dir, "/sys/class/thermal/"),
|
||||
Where: dir}
|
||||
v, ok := m.readInt(dir + "/temp")
|
||||
if !ok || !reading(float64(v)/1000) {
|
||||
t.NotRead = "the zone gave no temperature"
|
||||
if ok {
|
||||
t.NotRead = fmt.Sprintf("the zone answered %.1f °C, which is no temperature", float64(v)/1000)
|
||||
}
|
||||
return t
|
||||
}
|
||||
t.C = milli(v)
|
||||
var hot, passive *float64
|
||||
for _, n := range m.names(dir) {
|
||||
stem, ok := strings.CutSuffix(n, "_type")
|
||||
if !ok || !strings.HasPrefix(stem, "trip_point_") {
|
||||
continue
|
||||
}
|
||||
tv, ok := m.readInt(dir + "/" + stem + "_temp")
|
||||
if !ok || !sane(float64(tv)/1000) {
|
||||
continue
|
||||
}
|
||||
c := milli(tv)
|
||||
switch m.read(dir + "/" + n) {
|
||||
case "critical":
|
||||
if t.Critical == nil || *c < *t.Critical {
|
||||
t.Critical, t.CriticalIs = c, "the zone's critical trip"
|
||||
}
|
||||
case "hot":
|
||||
if hot == nil || *c < *hot {
|
||||
hot = c
|
||||
}
|
||||
case "passive":
|
||||
if passive == nil || *c < *passive {
|
||||
passive = c
|
||||
}
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case hot != nil:
|
||||
t.High, t.HighIs = hot, "the zone's hot trip"
|
||||
case passive != nil:
|
||||
t.High, t.HighIs = passive, "the zone's passive trip, where the kernel slows the machine down"
|
||||
}
|
||||
t.Flag = flagOf(t)
|
||||
return t
|
||||
}
|
||||
|
||||
func flagOf(t Temperature) string {
|
||||
if t.C == nil {
|
||||
return ""
|
||||
}
|
||||
switch {
|
||||
case t.Critical != nil && *t.C >= *t.Critical:
|
||||
return FlagCritical
|
||||
case t.High != nil && *t.C >= *t.High:
|
||||
return FlagHigh
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// driveOfHwmon names the drive each drive sensor's hwmon chip sits in: an NVMe controller lists its hwmon
|
||||
// chip and its namespaces, and a SATA drive read by drivetemp lists its chip under its device.
|
||||
func (m *Machine) driveOfHwmon() map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, ctrl := range m.numbered("/sys/class/nvme", "nvme") {
|
||||
dir := "/sys/class/nvme/" + ctrl
|
||||
name := ctrl
|
||||
for _, n := range m.names(dir) {
|
||||
if strings.HasPrefix(n, ctrl+"n") && !strings.Contains(n, "p") {
|
||||
name = n
|
||||
break
|
||||
}
|
||||
}
|
||||
if model := m.read(dir + "/model"); model != "" {
|
||||
name += " (" + model + ")"
|
||||
}
|
||||
for _, n := range m.names(dir) {
|
||||
if strings.HasPrefix(n, "hwmon") {
|
||||
out[n] = name
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, b := range m.names("/sys/block") {
|
||||
for _, n := range m.names("/sys/block/" + b + "/device/hwmon") {
|
||||
name := b
|
||||
if model := m.read("/sys/block/" + b + "/device/model"); model != "" {
|
||||
name += " (" + model + ")"
|
||||
}
|
||||
out[n] = name
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// Tools is the module's tools over one machine. Every one only reads, and none needs root: what only root
|
||||
// may read, the drive reader has read (drives.go, watch.go).
|
||||
func Tools(m *Machine) []stdio.Tool {
|
||||
return []stdio.Tool{
|
||||
{
|
||||
Name: "sensors_summary",
|
||||
Description: "The condition of this machine's hardware in a few lines: one per finding, flagged CRITICAL, HIGH, FAILING or " +
|
||||
"STOPPED against the hardware's own thresholds (never the mesh's), then one per class — temperatures with the hottest, " +
|
||||
"fans, drives' health, memory and PCIe errors — saying what was read and what the machine does not have. Replaces " +
|
||||
"sensors and smartctl -H. (r)",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
l := m.LookNow()
|
||||
h := l.Health()
|
||||
return map[string]any{"said": l.Said(), "healthy": h.Healthy}, nil
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "sensors_temperatures",
|
||||
Description: "Every temperature the machine reports, in °C: each thermal zone with its trip points, and each hwmon sensor " +
|
||||
"(CPU, GPU, drives, board, network adapters) with its label and its own maximum and critical values. HIGH at its own " +
|
||||
"maximum (or a zone's hot or passive trip), CRITICAL at its own critical value; a sensor that gives no threshold is " +
|
||||
"never flagged, and one that gives no reading is said as not read. Replaces sensors. (r)",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
ts := m.Temperatures()
|
||||
return map[string]any{"temperatures": ts, "count": len(ts)}, nil
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "sensors_fans",
|
||||
Description: "Every fan the machine reports: RPM, the chip's own minimum and alarm where it has them (said in a note, never " +
|
||||
"flagged: a zero-fan GPU is under its minimum by design and a board chip latches its alarm), and whether the " +
|
||||
"module's fan watch saw it turning since the machine started. STOPPED only when it ran and reads 0 while a CPU " +
|
||||
"sensor, a thermal zone or a sensor on its own non-board chip is at or above its own maximum. " +
|
||||
"Replaces sensors. (r)",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
l := m.LookNow()
|
||||
return map[string]any{"fans": l.Fans, "count": len(l.Fans)}, nil
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "sensors_drives",
|
||||
Description: "Each drive's health, as the module's drive reader last read it as root with smartctl (a spinning drive " +
|
||||
"hourly, any other every 10 minutes; a drive in standby is not read, and said as asleep since when): model, serial, size, whether it spins, the overall SMART verdict; for NVMe the critical " +
|
||||
"warning bits, spare, endurance used and media errors; for SATA the reallocated, pending and unreadable sectors, the " +
|
||||
"error log and the last self-test. FAILING when the drive says so, HIGH for the early signs. Replaces smartctl -a " +
|
||||
"and nvme smart-log. (r)",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
ds, note := m.Drives(m.ReadReadings(DrivesPath))
|
||||
out := map[string]any{"drives": ds, "count": len(ds)}
|
||||
if note != "" {
|
||||
out["note"] = note
|
||||
}
|
||||
return out, nil
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "sensors_faults",
|
||||
Description: "The kernel's counts of hardware errors since the machine started: corrected and uncorrectable memory " +
|
||||
"errors per memory controller and module (EDAC, where the memory has ECC), PCIe errors per device that counted one " +
|
||||
"(AER), and how often an Intel CPU was slowed for heat. A machine without a class has it as not present. Replaces " +
|
||||
"edac-util and reading /sys by hand. (r)",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
return m.Faults(), nil
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "sensors_health",
|
||||
Description: "Whether the hardware is failing, as the module's health check (novox/hq ADR 0256): unhealthy when a " +
|
||||
"CPU, GPU, drive or zone temperature is at or above its own critical value (a board chip's is shown, never judged), " +
|
||||
"a drive says it is failing, a fan that ran reads 0 while a CPU sensor, a thermal zone or its own (non-board) chip is at its own maximum, " +
|
||||
"memory reports an uncorrectable error, or the module's own processes are stale (a file five minutes old, a drive's " +
|
||||
"read stuck, a drive not looked at for twice its cadence); with " +
|
||||
"why, one sentence per finding. HIGH readings, a fan's alarm bit and a fan under its minimum do not make it " +
|
||||
"unhealthy. (r)",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
return m.LookNow().Health(), nil
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The module's two long-running processes, both of this binary, run by the node-engine (novox/hq ADR
|
||||
// 0256). Each writes its own file every FanEvery and nothing else:
|
||||
//
|
||||
// - **the fan watch** (`sensors watch`) notes which fans are turning, so a fan that ran and stopped can
|
||||
// be told from one that never runs. It is judged by the module's health tool, sensors_health;
|
||||
// - **the drive reader** (`sensors drives`), as root, reads each drive's health, so the tools, which run
|
||||
// as the operator account, need no root. It is judged by its unit.
|
||||
//
|
||||
// **One drive never holds up another, nor the fans.** Each drive is read in its own goroutine, one read at
|
||||
// a time per drive, bounded by CommandTimeout and WaitDelay. A smartctl stuck in the kernel (a drive or a
|
||||
// controller that stopped answering) cannot be ended at all: its read stays in flight, marked since when,
|
||||
// and the health check says so once it is older than any read can take (ReadBound). The fans are another
|
||||
// process, so they are looked at whatever the drives do.
|
||||
//
|
||||
// **The module is the only reader, and a spinning drive is read hourly.** Reading SMART from an awake
|
||||
// drive can restart its standby timer, so a drive read more often than its timer is set for never spins
|
||||
// down. Whether the machines' drives are set to spin down is not known to the mesh, so a rotational drive
|
||||
// is read every SpinningEvery. No other reader runs: the module does not run smartd, whose own schedule
|
||||
// would read the same drives at another phase of the hour, and whose start reads every drive.
|
||||
|
||||
const (
|
||||
FanEvery = 30 * time.Second
|
||||
// SolidStateEvery is how often a drive that does not spin is read.
|
||||
SolidStateEvery = 10 * time.Minute
|
||||
// SpinningEvery is how often a rotational drive is read: longer than a spin-down timer is commonly set.
|
||||
SpinningEvery = time.Hour
|
||||
)
|
||||
|
||||
// ReadBound is the longest one drive's read can take when the drive answers or smartctl can be ended: a
|
||||
// read in flight for longer is stuck, and said.
|
||||
func ReadBound() time.Duration { return CommandTimeout + WaitDelay + FanEvery }
|
||||
|
||||
// cadence is how often a drive is read.
|
||||
func cadence(d Drive) time.Duration {
|
||||
if d.Rotational {
|
||||
return SpinningEvery
|
||||
}
|
||||
return SolidStateEvery
|
||||
}
|
||||
|
||||
// FanWatch keeps which fans ran since the machine started.
|
||||
type FanWatch struct {
|
||||
M *Machine
|
||||
readings Readings
|
||||
}
|
||||
|
||||
// NewFanWatch starts from what an earlier fan watch of this boot left.
|
||||
func NewFanWatch(m *Machine) *FanWatch {
|
||||
w := &FanWatch{M: m, readings: Readings{FansRan: map[string]time.Time{}}}
|
||||
if r, why := m.ReadReadings(FansPath); why == "" && r != nil && r.FansRan != nil {
|
||||
w.readings.FansRan = r.FansRan
|
||||
}
|
||||
w.readings.BootID = m.bootID()
|
||||
return w
|
||||
}
|
||||
|
||||
// Look notes the fans turning now, and writes the file.
|
||||
func (w *FanWatch) Look() error {
|
||||
now := w.M.Now()
|
||||
for _, f := range w.M.Fans() {
|
||||
if f.RPM != nil && *f.RPM > 0 {
|
||||
w.readings.FansRan[f.Key] = now
|
||||
}
|
||||
}
|
||||
w.readings.WrittenAt = now
|
||||
return w.M.WriteReadings(FansPath, w.readings)
|
||||
}
|
||||
|
||||
// DriveReader keeps each drive's newest reading.
|
||||
type DriveReader struct {
|
||||
M *Machine
|
||||
// Async reads each due drive in a goroutine of its own instead of before writing; Run sets it.
|
||||
Async bool
|
||||
|
||||
mu sync.Mutex
|
||||
readings Readings
|
||||
done sync.WaitGroup
|
||||
}
|
||||
|
||||
// NewDriveReader starts from what an earlier reader of this boot left, so a restart does not read every
|
||||
// drive again at once. A read that was in flight when it stopped is not.
|
||||
func NewDriveReader(m *Machine) *DriveReader {
|
||||
w := &DriveReader{M: m, readings: Readings{Drives: map[string]DriveReading{}}}
|
||||
if r, why := m.ReadReadings(DrivesPath); why == "" && r != nil && r.Drives != nil {
|
||||
w.readings.Drives = r.Drives
|
||||
for name, d := range w.readings.Drives {
|
||||
d.InFlightSince = nil
|
||||
w.readings.Drives[name] = d
|
||||
}
|
||||
}
|
||||
w.readings.BootID = m.bootID()
|
||||
return w
|
||||
}
|
||||
|
||||
// Look starts a read of every drive that is due and not already being read, forgets drives that are gone,
|
||||
// and writes the file.
|
||||
func (w *DriveReader) Look() error {
|
||||
now := w.M.Now()
|
||||
var due []Drive
|
||||
present := map[string]bool{}
|
||||
w.mu.Lock()
|
||||
for _, d := range w.M.BlockDrives() {
|
||||
present[d.Device] = true
|
||||
last, ok := w.readings.Drives[d.Device]
|
||||
if ok && last.InFlightSince != nil {
|
||||
continue
|
||||
}
|
||||
if !ok || now.Sub(last.LookedAt) >= cadence(d) {
|
||||
last.InFlightSince = &now
|
||||
w.readings.Drives[d.Device] = last
|
||||
due = append(due, d)
|
||||
}
|
||||
}
|
||||
for name, d := range w.readings.Drives {
|
||||
if !present[name] && d.InFlightSince == nil {
|
||||
delete(w.readings.Drives, name)
|
||||
}
|
||||
}
|
||||
w.mu.Unlock()
|
||||
for _, d := range due {
|
||||
w.done.Add(1)
|
||||
if w.Async {
|
||||
go w.read(d)
|
||||
} else {
|
||||
w.read(d)
|
||||
}
|
||||
}
|
||||
return w.write()
|
||||
}
|
||||
|
||||
// read reads one drive and keeps its reading.
|
||||
func (w *DriveReader) read(d Drive) {
|
||||
defer w.done.Done()
|
||||
w.mu.Lock()
|
||||
var before *DriveReading
|
||||
if b, ok := w.readings.Drives[d.Device]; ok && (len(b.Smartctl) > 0 || b.Asleep) {
|
||||
b.InFlightSince = nil
|
||||
before = &b
|
||||
}
|
||||
w.mu.Unlock()
|
||||
got := w.M.ReadDrive(d, before)
|
||||
got.InFlightSince = nil
|
||||
w.mu.Lock()
|
||||
w.readings.Drives[d.Device] = got
|
||||
w.mu.Unlock()
|
||||
}
|
||||
|
||||
// WaitReads waits for the reads under way.
|
||||
func (w *DriveReader) WaitReads() { w.done.Wait() }
|
||||
|
||||
func (w *DriveReader) write() error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
w.readings.WrittenAt = w.M.Now()
|
||||
return w.M.WriteReadings(DrivesPath, w.readings)
|
||||
}
|
||||
|
||||
// every runs look at once and every FanEvery until ctx ends. A look that cannot write is said on standard
|
||||
// error, which the node-engine keeps in the unit's journal, and the process goes on.
|
||||
func every(ctx context.Context, name string, look func() error) {
|
||||
t := time.NewTicker(FanEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
if err := look(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "sensors %s: the readings were not written: %v\n", name, err)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
module sensors
|
||||
|
||||
go 1.22
|
||||
|
||||
require git.novox.be/novox/mesh-sdk/go v0.1.6
|
||||
@@ -0,0 +1,2 @@
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.6 h1:9qzdYONYbJdWcu6sxQcq9v1LI0JxcfkiKYkMUzJSkVQ=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.6/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"module": "sensors",
|
||||
"version": "1",
|
||||
"capabilities": [
|
||||
"package-manager",
|
||||
"service-manager"
|
||||
],
|
||||
"tools": [
|
||||
"sensors_summary",
|
||||
"sensors_temperatures",
|
||||
"sensors_fans",
|
||||
"sensors_drives",
|
||||
"sensors_faults",
|
||||
"sensors_health"
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"id": "smartmontools",
|
||||
"type": "package",
|
||||
"package": "smartmontools"
|
||||
},
|
||||
{
|
||||
"id": "drives",
|
||||
"type": "process",
|
||||
"name": "sensors-drives",
|
||||
"artifact": "tools",
|
||||
"run": [
|
||||
"./sensors",
|
||||
"drives"
|
||||
],
|
||||
"health": {
|
||||
"kind": "unit"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "watch",
|
||||
"type": "process",
|
||||
"name": "sensors-watch",
|
||||
"artifact": "tools",
|
||||
"run": [
|
||||
"./sensors",
|
||||
"watch"
|
||||
],
|
||||
"health": {
|
||||
"kind": "tool",
|
||||
"tool": "sensors_health",
|
||||
"interval": "60s",
|
||||
"timeout": "10s",
|
||||
"looks": 2,
|
||||
"grace": "60s"
|
||||
}
|
||||
}
|
||||
],
|
||||
"build": {
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "tools",
|
||||
"kind": "bundle",
|
||||
"language": "go",
|
||||
"system": "arch",
|
||||
"from": "cmd/sensors",
|
||||
"binary": "sensors",
|
||||
"loads": [
|
||||
"sensors"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user