From 73610f8e3cd917e0457fae50bd0b377586f30e51 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 13:48:53 +0200 Subject: [PATCH] sensors: read the machine's hardware condition on every machine (hq ADR 0256) Temperatures against the hardware's own thresholds, fans, each drive's health and the kernel's error counts, read-only. A root watch reads SMART so the tools need no root and never wake a sleeping drive; smartd and the watch carry the health that raises a condition when the hardware says it is failing. --- modules/sensors/README.md | 132 ++++++ modules/sensors/cmd/sensors/drives.go | 447 +++++++++++++++++++ modules/sensors/cmd/sensors/fans.go | 116 +++++ modules/sensors/cmd/sensors/faults.go | 151 +++++++ modules/sensors/cmd/sensors/findings.go | 254 +++++++++++ modules/sensors/cmd/sensors/helpers_test.go | 189 ++++++++ modules/sensors/cmd/sensors/machine.go | 157 +++++++ modules/sensors/cmd/sensors/main.go | 37 ++ modules/sensors/cmd/sensors/manifest_test.go | 102 +++++ modules/sensors/cmd/sensors/sensors_test.go | 366 +++++++++++++++ modules/sensors/cmd/sensors/temperatures.go | 282 ++++++++++++ modules/sensors/cmd/sensors/tools.go | 82 ++++ modules/sensors/cmd/sensors/watch.go | 96 ++++ modules/sensors/go.mod | 5 + modules/sensors/go.sum | 2 + modules/sensors/module.json | 90 ++++ 16 files changed, 2508 insertions(+) create mode 100644 modules/sensors/README.md create mode 100644 modules/sensors/cmd/sensors/drives.go create mode 100644 modules/sensors/cmd/sensors/fans.go create mode 100644 modules/sensors/cmd/sensors/faults.go create mode 100644 modules/sensors/cmd/sensors/findings.go create mode 100644 modules/sensors/cmd/sensors/helpers_test.go create mode 100644 modules/sensors/cmd/sensors/machine.go create mode 100644 modules/sensors/cmd/sensors/main.go create mode 100644 modules/sensors/cmd/sensors/manifest_test.go create mode 100644 modules/sensors/cmd/sensors/sensors_test.go create mode 100644 modules/sensors/cmd/sensors/temperatures.go create mode 100644 modules/sensors/cmd/sensors/tools.go create mode 100644 modules/sensors/cmd/sensors/watch.go create mode 100644 modules/sensors/go.mod create mode 100644 modules/sensors/go.sum create mode 100644 modules/sensors/module.json diff --git a/modules/sensors/README.md b/modules/sensors/README.md new file mode 100644 index 00000000..132a91ec --- /dev/null +++ b/modules/sensors/README.md @@ -0,0 +1,132 @@ +# sensors + +The condition of a machine's own hardware: temperatures with the thresholds the hardware gives for +itself, fans, each drive's health, and the kernel's counts of memory and PCIe errors (novox/hq ADR +0256). It reads, and changes nothing on the hardware. Every flag is measured against the hardware's own +threshold, never one the mesh chose. A class the machine does not have is said as not present, never +as an error, so the module can be assigned to every machine: a laptop, a desktop, a server, a virtual +machine. + +## Why it exists, and why here + +The operator set the boundary on 2026-10-08. What hardware a machine has is a node-engine fact. A module +holding a generic role reads and provides it. A vendor module adds only vendor-specific extras. + +Before this module, only the laptop's vendor module read a temperature, and no module read a drive's +health. The machines' 20 drives, 15 of them spinning, were watched by nothing. `power` keeps the battery +and the power draw. `disk-load` keeps how busy the disks are. This module keeps everything else the +kernel says about the hardware's condition. + +It holds no seat. A seat is a role that callers reach without knowing the module, and nothing asks for +this role yet. A new seat would also be a controller change first (ADR 0246). When a caller needs to ask +by role, such as the bar's temperature blocks or a vendor module's fan policy, these tools become the +seat's verbs. The full reasoning is in ADR 0256. + +## What runs + +| resource | what | as | +|---|---|---| +| `smartmontools` | the package: `smartctl` and `smartd` | | +| `/etc/smartd.conf` | every drive (`DEVICESCAN`), every check, never waking a drive in standby, no mail | written whole | +| `smartd.service` drop-in | runs on a virtual machine too, and stays up with no drive to watch (`-q never`) | written whole | +| `smartd` | watches every drive between looks, and logs what it finds to the journal | root, the package's unit | +| `watch` (`sensors watch`) | every 10 min each drive's health with `smartctl --json -a -n standby`; every 30 s which fans turn; one file, `/run/mesh-sensors/readings.json` (0644) | root, the node-engine's process | +| the tools | read `/sys`, `/proc` and the watch's file | the operator account, in the tool runner | + +The tools need no root, because the watch has already read what only root may read. A drive in standby +is never woken. The watch keeps its last reading of that drive and says that it was asleep. + +## Tools + +All six only read. + +| tool | what | replaces | +|---|---|---| +| `sensors_summary` | one line per finding, flagged CRITICAL, HIGH, FAILING or STOPPED; then one line per class: temperatures with the hottest, fans, drives, memory and PCIe errors | `sensors`, `smartctl -H` | +| `sensors_temperatures` | every thermal zone with its trip points, and every hwmon temperature with its label, its kind (cpu, gpu, drive, board, network), the drive it sits in, and its own maximum and critical values | `sensors` | +| `sensors_fans` | every fan's RPM, its chip's minimum and alarm, and whether the watch saw it turn since boot | `sensors` | +| `sensors_drives` | each drive: model, serial, size, whether it spins, the SMART verdict; for NVMe the critical warning bits, spare, endurance used and media errors; for SATA the reallocated, pending and unreadable sectors, the error log and the last self-test | `smartctl -a`, `nvme smart-log` | +| `sensors_faults` | EDAC memory errors per controller and module, PCIe AER errors per device, and how often an Intel CPU was slowed for heat | `edac-util` | +| `sensors_health` | the module's health check: healthy or not, with `why` | | + +### The flags + +| flag | when | +|---|---| +| temperature HIGH | at or above its own maximum, or a zone's `hot` trip (without one, its lowest `passive` trip) | +| temperature CRITICAL | at or above its own critical or emergency value, or a zone's `critical` trip | +| drive FAILING | SMART's overall verdict failed, an attribute is failing now, or an NVMe critical warning bit is set | +| drive HIGH | reallocated, pending or unreadable sectors; an attribute that failed in the past; NVMe media errors or 90 % of rated endurance used; errors in the error log; a last self-test that did not pass | +| fan STOPPED | its chip's alarm, or under its own minimum, once it has been seen turning; or it ran since boot and reads 0 while a temperature is at or above its own maximum | +| memory HIGH / CRITICAL | corrected / uncorrectable errors since boot | +| PCIe HIGH | fatal or non-fatal errors since boot | + +**What is no threshold.** A threshold given as the format's largest value is no threshold. The laptop's +NVMe drive gives 65261.85 °C as the maximum of two of its sensors. A threshold of 0 is no threshold +either, and neither is anything outside −40 to 200 °C. The reading is then said without a flag. A CPU +sensor that gives no threshold, such as AMD's `Tctl`, is never flagged. + +**Why a fan at 0 is not enough.** Laptop fans and GPU fans in zero-fan mode stop on purpose when the +machine is cool. A board header with nothing on it reads 0, and may carry an alarm or a minimum that the +firmware set. So a fan is judged only once it has been seen turning. + +## Health + +The module declares its health under ADR 0240: + +- `smartd` has the `unit` kind: the unit's own readiness, which the module does not judge itself. +- The watch has the `tool` kind: `sensors_health`, every 60 s, unhealthy after 2 looks in a row, with a + 60 s grace. + +`sensors_health` is unhealthy when: + +- a temperature is at or above its own critical value; +- a drive says it is failing; +- a fan that ran has stopped while the machine is hot; +- memory reports an uncorrectable error. + +Its `why` is one plain sentence per finding, for the operator. For example: "The drive sdb (WDC +WUH722222ALE6L4) says it is failing: its overall SMART health check failed." HIGH readings raise nothing. +A drive whose health was not read is said in `why` and does not make the check unhealthy. The watch not +running is caught by the watch's own liveness. + +The condition this raises is `module.sensors..unhealthy`. ADR 0256 records what this costs: + +- the condition names the module, not the machine; +- a standing hardware fault holds this module's own new builds at their gate on that machine. + +The machine judging its own hardware, as it judges its network, is the follow-up. + +## Tests + +`go test ./...`. The tests read a tree standing in for `/sys`, `/proc` and `/run`, and smartctl answers +recorded as smartmontools 7 writes them. There are three machines: + +- a laptop, as the laptop showed itself on 2026-10-08; +- a server with ECC memory, three SATA drives read by drivetemp and a board chip with an empty fan header; +- a virtual machine with no sensor at all. + +They cover: + +- thresholds taken from the hardware only, and a zone's hwmon copy left out; +- HIGH at the maximum and CRITICAL at the critical value; a zone's passive trip as its high; +- a sensor that gives no reading said as not read; +- every class not present on the virtual machine, which is healthy; +- a failing SATA drive, an NVMe critical warning, and early signs that are HIGH and raise nothing; +- a missing, stale or earlier-boot reading said as such; +- a fan at rest, a fan that stopped while hot, an empty header, and a fan under its minimum; +- memory and PCIe errors, and CPU throttling; +- the watch reading as root without waking a drive, keeping a reading for a drive asleep, reading drives + only every 10 min, and keeping what it saw across a restart; +- a watch that is not root asking through `sudo -n`, and saying what stops it; +- the health answer in the shape the node-engine reads; +- the manifest: the tools listed are the tools served, all read-only, no seat claimed, no machine named, + and the two health checks. + +## Follow-ups + +- The machine's hardware judged by the node-engine, raising a machine condition, instead of this + module's health (ADR 0256). +- The module-health condition carrying the tool check's `why` as its plain words (ADR 0253). +- Temperature blocks for the bar, once there is a generic way for a module to provide bar blocks. +- The laptop vendor module's `zephyrus_thermals` keeps only what is the vendor's. diff --git a/modules/sensors/cmd/sensors/drives.go b/modules/sensors/cmd/sensors/drives.go new file mode 100644 index 00000000..4fc01a85 --- /dev/null +++ b/modules/sensors/cmd/sensors/drives.go @@ -0,0 +1,447 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strconv" + "strings" + "time" +) + +// Drives and their health. Which drives the machine has is read from /sys by anyone; what each says of its +// own health (the NVMe health log, SATA SMART) is read by the module's watch, as root, with smartctl, and +// left in ReadingsPath for the tools (novox/hq ADR 0256). + +// ReadingsPath is the watch's one file: what it last read of each drive, and which fans it saw turning. +const ReadingsPath = "/run/mesh-sensors/readings.json" + +// StaleAfter is how old the watch's file may be before the tools say the watch may not be running: three +// of its drive readings missed. +const StaleAfter = 30 * time.Minute + +// FlagFailing is said beside a drive that says it is failing. +const FlagFailing = "FAILING" + +// Readings is the watch's file. +type Readings struct { + BootID string `json:"boot_id"` + WrittenAt time.Time `json:"written_at"` + Drives map[string]DriveReading `json:"drives"` + // FansRan is when the watch last saw each fan turning, by its key (asus/fan1), since the machine started. + FansRan map[string]time.Time `json:"fans_ran"` +} + +// DriveReading is one drive as smartctl last answered for it. +type DriveReading struct { + ReadAt time.Time `json:"read_at"` + Smartctl json.RawMessage `json:"smartctl,omitempty"` + Status int `json:"status"` + // Asleep says the drive was in standby at the newest look, so it was not woken and the reading is the + // one before. + Asleep bool `json:"asleep,omitempty"` + NotRead string `json:"not_read,omitempty"` +} + +// Drive is one drive, as the tools say it. +type Drive struct { + Device string `json:"device"` + Model string `json:"model,omitempty"` + Serial string `json:"serial,omitempty"` + Firmware string `json:"firmware,omitempty"` + SizeGiB float64 `json:"size_gib"` + Rotational bool `json:"rotational"` + Transport string `json:"transport"` + // Health is "passed", "failing" or "not read". + Health string `json:"health"` + Flag string `json:"flag,omitempty"` + Findings []string `json:"findings,omitempty"` + + TemperatureC *int64 `json:"temperature_c,omitempty"` + PowerOnHours *int64 `json:"power_on_hours,omitempty"` + PercentageUsed *int64 `json:"percentage_used,omitempty"` + AvailableSpare *int64 `json:"available_spare_pct,omitempty"` + SpareThreshold *int64 `json:"available_spare_threshold_pct,omitempty"` + MediaErrors *int64 `json:"media_errors,omitempty"` + CriticalWarning *int64 `json:"critical_warning,omitempty"` + Reallocated *int64 `json:"reallocated_sectors,omitempty"` + Pending *int64 `json:"pending_sectors,omitempty"` + Uncorrectable *int64 `json:"offline_uncorrectable,omitempty"` + ErrorLogCount *int64 `json:"error_log_count,omitempty"` + LastSelfTest string `json:"last_self_test,omitempty"` + ReadAt string `json:"read_at,omitempty"` + Asleep bool `json:"asleep_at_last_look,omitempty"` + NotRead string `json:"not_read,omitempty"` + failingSentences []string // why it is failing, each a clause for the operator +} + +// Name is how a person reads which drive this is. +func (d Drive) Name() string { + if d.Model != "" { + return fmt.Sprintf("the drive %s (%s)", d.Device, d.Model) + } + return "the drive " + d.Device +} + +// notDrives are block devices that are not a drive of their own: loop files, memory, device-mapper and +// software RAID volumes (built on drives that are listed themselves), optical and floppy drives, network +// block devices. +var notDrives = []string{"loop", "ram", "zram", "dm-", "md", "sr", "fd", "nbd", "rbd"} + +// BlockDrives lists the machine's drives from /sys/block: whole devices backed by hardware. +func (m *Machine) BlockDrives() []Drive { + var out []Drive + for _, b := range m.names("/sys/block") { + skip := strings.HasSuffix(b, "boot0") || strings.HasSuffix(b, "boot1") || strings.HasSuffix(b, "rpmb") + for _, p := range notDrives { + skip = skip || strings.HasPrefix(b, p) + } + if skip || !m.exists("/sys/block/"+b+"/device") { + continue + } + dir := "/sys/block/" + b + d := Drive{Device: b, Model: m.read(dir + "/device/model"), Transport: transport(b, m.read(dir+"/device/vendor"))} + if s, ok := m.readInt(dir + "/size"); ok { + d.SizeGiB = round1(float64(s) * 512 / (1 << 30)) + } + if r, ok := m.readInt(dir + "/queue/rotational"); ok { + d.Rotational = r == 1 + } + out = append(out, d) + } + return out +} + +func transport(name, vendor string) string { + switch { + case strings.HasPrefix(name, "nvme"): + return "nvme" + case strings.HasPrefix(name, "vd"), strings.HasPrefix(name, "xvd"): + return "virtual" + case strings.HasPrefix(name, "mmcblk"): + return "mmc" + case strings.TrimSpace(vendor) == "ATA": + return "ata" + } + return "scsi" +} + +// ReadReadings is the watch's file, or why there is none to use. +func (m *Machine) ReadReadings() (*Readings, string) { + raw, err := os.ReadFile(m.path(ReadingsPath)) + if err != nil { + return nil, "the module's watch, which reads each drive's health as root, has left no reading (is it running?)" + } + var r Readings + if err := json.Unmarshal(raw, &r); err != nil { + return nil, "the module's watch left a reading this tool cannot read: " + err.Error() + } + if id := m.bootID(); id != "" && r.BootID != "" && r.BootID != id { + return nil, "the module's watch has not read anything since the machine started" + } + return &r, "" +} + +// Drives is every drive with its health as the watch last read it, judged. +func (m *Machine) Drives() ([]Drive, string) { + drives := m.BlockDrives() + r, why := m.ReadReadings() + note := why + if r != nil && m.Now().Sub(r.WrittenAt) > StaleAfter { + note = fmt.Sprintf("the module's watch last wrote %s ago; it may not be running", ago(m.Now().Sub(r.WrittenAt))) + } + for i := range drives { + d := &drives[i] + if r == nil { + d.Health, d.NotRead = "not read", why + continue + } + reading, ok := r.Drives[d.Device] + if !ok { + d.Health, d.NotRead = "not read", "the module's watch has not read this drive yet" + continue + } + judgeDrive(d, reading, m.Now()) + } + return drives, note +} + +func ago(d time.Duration) string { + switch { + case d < time.Minute: + return fmt.Sprintf("%d s", int(d.Seconds())) + case d < 2*time.Hour: + return fmt.Sprintf("%d min", int(d.Minutes())) + } + return fmt.Sprintf("%.1f h", d.Hours()) +} + +// smartctl's JSON, the parts the module reads (smartmontools 7). +type smartJSON struct { + Smartctl struct { + Messages []struct { + String string `json:"string"` + Severity string `json:"severity"` + } `json:"messages"` + } `json:"smartctl"` + ModelName string `json:"model_name"` + SerialNumber string `json:"serial_number"` + FirmwareVersion string `json:"firmware_version"` + SmartSupport *struct { + Available bool `json:"available"` + Enabled bool `json:"enabled"` + } `json:"smart_support"` + SmartStatus *struct { + Passed bool `json:"passed"` + } `json:"smart_status"` + NVMe *struct { + CriticalWarning int64 `json:"critical_warning"` + AvailableSpare int64 `json:"available_spare"` + AvailableSpareThreshold int64 `json:"available_spare_threshold"` + PercentageUsed int64 `json:"percentage_used"` + MediaErrors int64 `json:"media_errors"` + PowerOnHours int64 `json:"power_on_hours"` + } `json:"nvme_smart_health_information_log"` + ATA *struct { + Table []struct { + ID int `json:"id"` + Name string `json:"name"` + Value int64 `json:"value"` + Thresh int64 `json:"thresh"` + WhenFailed string `json:"when_failed"` + Raw struct { + Value int64 `json:"value"` + } `json:"raw"` + } `json:"table"` + } `json:"ata_smart_attributes"` + ErrorLog *struct { + Summary struct { + Count int64 `json:"count"` + } `json:"summary"` + } `json:"ata_smart_error_log"` + SelfTests *struct { + Standard struct { + Table []struct { + Type struct { + String string `json:"string"` + } `json:"type"` + Status struct { + String string `json:"string"` + Passed *bool `json:"passed"` + } `json:"status"` + } `json:"table"` + } `json:"standard"` + } `json:"ata_smart_self_test_log"` + Temperature *struct { + Current int64 `json:"current"` + } `json:"temperature"` + PowerOnTime *struct { + Hours int64 `json:"hours"` + } `json:"power_on_time"` +} + +// nvmeWarnings are the NVMe health log's critical warning bits, as a person reads each. +var nvmeWarnings = []struct { + bit int64 + said string +}{ + {0x01, "its spare capacity is below its threshold"}, + {0x02, "its temperature is past one of its own thresholds"}, + {0x04, "its reliability is degraded by media or internal errors"}, + {0x08, "it has put itself in read-only mode"}, + {0x10, "its volatile memory backup has failed"}, + {0x20, "its persistent memory region has become read-only"}, +} + +func num(v int64) *int64 { return &v } + +// judgeDrive reads one drive's smartctl answer into the drive, with its findings: FAILING when the drive +// says so itself, HIGH for the early signs a drive gives before it fails. +func judgeDrive(d *Drive, reading DriveReading, now time.Time) { + if !reading.ReadAt.IsZero() { + d.ReadAt = reading.ReadAt.UTC().Format(time.RFC3339) + } + d.Asleep = reading.Asleep + var s smartJSON + if len(reading.Smartctl) == 0 || json.Unmarshal(reading.Smartctl, &s) != nil { + d.Health, d.NotRead = "not read", reading.NotRead + if d.NotRead == "" { + d.NotRead = "smartctl gave no answer for it" + } + return + } + if s.ModelName != "" { + d.Model = s.ModelName + } + d.Serial, d.Firmware = s.SerialNumber, s.FirmwareVersion + if s.SmartStatus == nil { + d.Health = "not read" + switch { + case s.SmartSupport != nil && !s.SmartSupport.Available: + d.NotRead = "the drive does not support SMART" + case s.SmartSupport != nil && !s.SmartSupport.Enabled: + d.NotRead = "SMART is turned off on the drive" + default: + d.NotRead = firstError(s, reading.NotRead) + } + return + } + d.Health = "passed" + var failing, high []string + if !s.SmartStatus.Passed { + failing = append(failing, "its overall SMART health check failed") + } + if s.Temperature != nil && s.Temperature.Current != 0 { + d.TemperatureC = num(s.Temperature.Current) + } + if s.PowerOnTime != nil { + d.PowerOnHours = num(s.PowerOnTime.Hours) + } + if n := s.NVMe; n != nil { + d.CriticalWarning, d.PercentageUsed = num(n.CriticalWarning), num(n.PercentageUsed) + d.AvailableSpare, d.SpareThreshold, d.MediaErrors = num(n.AvailableSpare), num(n.AvailableSpareThreshold), num(n.MediaErrors) + if d.PowerOnHours == nil { + d.PowerOnHours = num(n.PowerOnHours) + } + for _, w := range nvmeWarnings { + if n.CriticalWarning&w.bit != 0 { + failing = append(failing, w.said) + } + } + if n.MediaErrors > 0 { + high = append(high, fmt.Sprintf("it has had %d media errors", n.MediaErrors)) + } + if n.PercentageUsed >= 90 { + high = append(high, fmt.Sprintf("it has used %d %% of its rated endurance", n.PercentageUsed)) + } + } + if a := s.ATA; a != nil { + for _, at := range a.Table { + switch at.ID { + case 5: + d.Reallocated = num(at.Raw.Value) + case 197: + d.Pending = num(at.Raw.Value) + case 198: + d.Uncorrectable = num(at.Raw.Value) + } + switch at.WhenFailed { + case "now": + failing = append(failing, fmt.Sprintf("its attribute %s is at %d, at or below its own threshold of %d", at.Name, at.Value, at.Thresh)) + case "past": + high = append(high, fmt.Sprintf("its attribute %s was at or below its own threshold in the past", at.Name)) + } + } + for _, c := range []struct { + v *int64 + said string + }{{d.Reallocated, "sectors reallocated"}, {d.Pending, "sectors waiting to be reallocated"}, {d.Uncorrectable, "sectors it could not read"}} { + if c.v != nil && *c.v > 0 { + high = append(high, fmt.Sprintf("it has %d %s", *c.v, c.said)) + } + } + } + if s.ErrorLog != nil { + d.ErrorLogCount = num(s.ErrorLog.Summary.Count) + if s.ErrorLog.Summary.Count > 0 { + high = append(high, fmt.Sprintf("its error log holds %d errors", s.ErrorLog.Summary.Count)) + } + } + if s.SelfTests != nil && len(s.SelfTests.Standard.Table) > 0 { + last := s.SelfTests.Standard.Table[0] + d.LastSelfTest = strings.TrimSpace(last.Type.String + ": " + last.Status.String) + if last.Status.Passed != nil && !*last.Status.Passed { + high = append(high, "its last self-test did not pass ("+last.Status.String+")") + } + } + d.failingSentences = failing + switch { + case len(failing) > 0: + d.Health, d.Flag = "failing", FlagFailing + d.Findings = append(d.Findings, failing...) + case len(high) > 0: + d.Flag = FlagHigh + } + d.Findings = append(d.Findings, high...) +} + +func firstError(s smartJSON, fallback string) string { + for _, msg := range s.Smartctl.Messages { + if msg.Severity == "error" && strings.TrimSpace(msg.String) != "" { + return "smartctl: " + strings.TrimSpace(msg.String) + } + } + if fallback != "" { + return fallback + } + return "smartctl gave no health for it" +} + +// ReadDrive asks smartctl for one drive's health, as root (directly when this process is root, else +// through sudo -n), without waking a drive in standby. A drive asleep keeps the reading before. +func (m *Machine) ReadDrive(device string, before *DriveReading) DriveReading { + args := []string{"--json=c", "-a", "-n", "standby", "/dev/" + device} + program := "smartctl" + if m.UID != 0 { + program, args = "sudo", append([]string{"-n", "smartctl"}, args...) + } + r := m.Run(context.Background(), program, args...) + now := m.Now() + out := DriveReading{ReadAt: now, Status: r.Status} + switch { + case r.Err == "ENOENT" && program == "smartctl": + out.NotRead = "smartctl is not installed; the module declares smartmontools, so the package has not been applied yet" + return out + case r.Err == "ENOENT": + out.NotRead = "the watch is not root and sudo is not installed to read the drive as root" + return out + case r.Err != "": + out.NotRead = "smartctl did not answer: " + r.Err + return out + case program == "sudo" && strings.HasPrefix(strings.TrimSpace(r.Stderr), "sudo:") && strings.Contains(r.Stderr, "command not found"): + out.NotRead = "smartctl is not installed; the module declares smartmontools, so the package has not been applied yet" + return out + case program == "sudo" && strings.HasPrefix(strings.TrimSpace(r.Stderr), "sudo:"): + out.NotRead = "the watch is not root and may not run smartctl through sudo without a prompt" + return out + } + said := strings.ToUpper(r.Stdout + r.Stderr) + if r.Status&2 != 0 && (strings.Contains(said, "STANDBY") || strings.Contains(said, "SLEEP")) { + if before != nil { + kept := *before + kept.Asleep = true + return kept + } + out.Asleep, out.NotRead = true, "the drive was asleep at every look, and is not woken to be read" + return out + } + if !json.Valid([]byte(r.Stdout)) { + out.NotRead = "smartctl answered something other than JSON (status " + strconv.Itoa(r.Status) + ")" + return out + } + out.Smartctl = json.RawMessage(strings.TrimSpace(r.Stdout)) + return out +} + +// WriteReadings writes the watch's file whole, through a rename, readable by every account: it holds no +// secret, and the tools read it as the operator account. +func (m *Machine) WriteReadings(r Readings) error { + path := m.path(ReadingsPath) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + raw, err := json.Marshal(r) + if err != nil { + return err + } + tmp := path + ".new" + if err := os.WriteFile(tmp, raw, 0o644); err != nil { + return err + } + if err := os.Chmod(tmp, 0o644); err != nil { + return err + } + return os.Rename(tmp, path) +} diff --git a/modules/sensors/cmd/sensors/fans.go b/modules/sensors/cmd/sensors/fans.go new file mode 100644 index 00000000..d65344f5 --- /dev/null +++ b/modules/sensors/cmd/sensors/fans.go @@ -0,0 +1,116 @@ +package main + +import ( + "fmt" + "strings" +) + +// Fans, from the hwmon chips. Many fans stop on purpose when the machine is cool (a laptop at rest, a GPU +// in its zero-fan mode), so 0 RPM alone is not a fault. A fan is said STOPPED only when the hardware says +// so itself (its alarm bit, or a reading under its own minimum) or when it ran since the machine started +// and reads 0 while a temperature is at or above the hardware's own maximum (novox/hq ADR 0256 rule 4). + +// FlagStopped is said beside a fan that should be turning and is not. +const FlagStopped = "STOPPED" + +// Fan is one fan's reading. +type Fan struct { + Chip string `json:"chip"` + Label string `json:"label"` + // Key names the fan across readings: its chip and its input (asus/fan1). + Key string `json:"key"` + RPM *int64 `json:"rpm,omitempty"` + // Min is the chip's own lowest speed for it, where it gives one. + Min *int64 `json:"min_rpm,omitempty"` + Alarm bool `json:"alarm,omitempty"` + // RanSinceBoot says the module's watch saw it turning since the machine started; absent when the + // watch has not said. + RanSinceBoot *bool `json:"ran_since_boot,omitempty"` + Flag string `json:"flag,omitempty"` + Why string `json:"why,omitempty"` + NotRead string `json:"not_read,omitempty"` + Where string `json:"where"` +} + +// Name is how a person reads which fan this is. +func (f Fan) Name() string { + if f.Label != "" && !strings.HasPrefix(f.Label, "fan") { + return fmt.Sprintf("the fan %s (%s)", f.Label, f.Chip) + } + return fmt.Sprintf("the fan %s of %s", f.Label, f.Chip) +} + +// Fans is every fan the machine reports, read now. +func (m *Machine) Fans() []Fan { + var out []Fan + for _, h := range m.numbered("/sys/class/hwmon", "hwmon") { + dir := "/sys/class/hwmon/" + h + chip := m.read(dir + "/name") + for _, stem := range m.inputs(dir, "fan") { + f := Fan{Chip: chip, Label: m.read(dir + "/" + stem + "_label"), Key: chip + "/" + stem, Where: dir + "/" + stem} + if f.Label == "" { + f.Label = stem + } + v, ok := m.readInt(dir + "/" + stem + "_input") + if !ok || v < 0 { + f.NotRead = "the fan gave no speed" + out = append(out, f) + continue + } + f.RPM = &v + if mn, ok := m.readInt(dir + "/" + stem + "_min"); ok && mn > 0 { + f.Min = &mn + } + if a, ok := m.readInt(dir + "/" + stem + "_alarm"); ok && a != 0 { + f.Alarm = true + } + out = append(out, f) + } + } + return out +} + +// judgeFans flags each fan by what the hardware says of it and, given which fans ran since the machine +// started (nil when the watch has not said) and the hottest temperature at or above its own maximum, a fan +// that stopped while the machine is hot. +func judgeFans(fans []Fan, ran map[string]bool, hot *Temperature) []Fan { + for i := range fans { + f := &fans[i] + if ran != nil { + r := ran[f.Key] + f.RanSinceBoot = &r + } + if f.RPM == nil { + continue + } + // A header with nothing on it reads 0 and may carry an alarm or a minimum the firmware set: a fan + // is judged only once it has been seen turning, now or since the machine started. + ranBefore := f.RanSinceBoot != nil && *f.RanSinceBoot + seen := *f.RPM > 0 || ranBefore + switch { + case f.Alarm && seen: + f.Flag, f.Why = FlagStopped, fmt.Sprintf("its chip raises its alarm at %d RPM", *f.RPM) + case f.Min != nil && *f.RPM < *f.Min && seen: + f.Flag, f.Why = FlagStopped, fmt.Sprintf("it turns at %d RPM, under its own minimum of %d RPM", *f.RPM, *f.Min) + case *f.RPM == 0 && ranBefore && hot != nil: + f.Flag, f.Why = FlagStopped, fmt.Sprintf("it ran earlier and reads 0 RPM while %s is at %.0f °C, at or above its own maximum of %.0f °C", + hot.Name(), *hot.C, *hot.High) + } + } + return fans +} + +// hottest is the temperature furthest at or above its own maximum, or nil when none is. +func hottest(ts []Temperature) *Temperature { + var best *Temperature + for i := range ts { + t := &ts[i] + if t.C == nil || t.High == nil || *t.C < *t.High { + continue + } + if best == nil || *t.C-*t.High > *best.C-*best.High { + best = t + } + } + return best +} diff --git a/modules/sensors/cmd/sensors/faults.go b/modules/sensors/cmd/sensors/faults.go new file mode 100644 index 00000000..e3814604 --- /dev/null +++ b/modules/sensors/cmd/sensors/faults.go @@ -0,0 +1,151 @@ +package main + +import ( + "strings" +) + +// The kernel's own counts of hardware errors since the machine started, where the hardware reports them: +// memory errors from EDAC (a server with ECC memory has them, a laptop has none), PCIe errors from AER, +// and how often an Intel CPU was slowed for heat. Each is a few small files; a machine without them has +// the class not present. + +// MemoryController is one EDAC memory controller's error counts. +type MemoryController struct { + Controller string `json:"controller"` + Name string `json:"name,omitempty"` + Corrected int64 `json:"corrected"` + Uncorrectable int64 `json:"uncorrectable"` + // Modules names each memory module that counted an error, by its label, with its counts. + Modules []MemoryModule `json:"modules,omitempty"` + Flag string `json:"flag,omitempty"` +} + +// MemoryModule is one DIMM's counts, where the controller says them per module. +type MemoryModule struct { + Label string `json:"label"` + Corrected int64 `json:"corrected"` + Uncorrectable int64 `json:"uncorrectable"` +} + +// PCIError is one PCIe device that counted an error. +type PCIError struct { + Device string `json:"device"` + Driver string `json:"driver,omitempty"` + Correctable int64 `json:"correctable"` + NonFatal int64 `json:"non_fatal"` + Fatal int64 `json:"fatal"` + Flag string `json:"flag,omitempty"` +} + +// Faults is everything the kernel counted. +type Faults struct { + Memory []MemoryController `json:"memory"` + // MemoryNotPresent says why no memory controller is listed. + MemoryNotPresent string `json:"memory_not_present,omitempty"` + // PCIDevicesCounting is how many PCIe devices report their errors; only those that counted one are + // listed. + PCIDevicesCounting int `json:"pci_devices_counting"` + PCI []PCIError `json:"pci"` + // Throttled is how often the CPU was slowed for heat since the machine started, summed over its + // cores, where the CPU counts it. + ThrottledCore *int64 `json:"cpu_core_throttled,omitempty"` + ThrottledPackage *int64 `json:"cpu_package_throttled,omitempty"` +} + +// Faults reads the kernel's counts of hardware errors. +func (m *Machine) Faults() Faults { + f := Faults{Memory: []MemoryController{}, PCI: []PCIError{}} + base := "/sys/devices/system/edac/mc" + for _, mc := range m.numbered(base, "mc") { + dir := base + "/" + mc + c := MemoryController{Controller: mc, Name: m.read(dir + "/mc_name")} + c.Corrected, _ = m.readInt(dir + "/ce_count") + c.Uncorrectable, _ = m.readInt(dir + "/ue_count") + for _, n := range m.names(dir) { + if !strings.HasPrefix(n, "dimm") && !strings.HasPrefix(n, "rank") { + continue + } + ce, _ := m.readInt(dir + "/" + n + "/dimm_ce_count") + ue, _ := m.readInt(dir + "/" + n + "/dimm_ue_count") + if ce+ue == 0 { + continue + } + label := m.read(dir + "/" + n + "/dimm_label") + if label == "" { + label = n + } + c.Modules = append(c.Modules, MemoryModule{Label: label, Corrected: ce, Uncorrectable: ue}) + } + switch { + case c.Uncorrectable > 0: + c.Flag = FlagCritical + case c.Corrected > 0: + c.Flag = FlagHigh + } + f.Memory = append(f.Memory, c) + } + if len(f.Memory) == 0 { + f.MemoryNotPresent = "no memory controller reports its errors here (no EDAC driver, or memory without ECC)" + } + for _, dev := range m.names("/sys/bus/pci/devices") { + dir := "/sys/bus/pci/devices/" + dev + if !m.exists(dir + "/aer_dev_correctable") { + continue + } + f.PCIDevicesCounting++ + e := PCIError{Device: dev, Driver: uevent(m.read(dir+"/uevent"), "DRIVER"), + Correctable: aerTotal(m.read(dir + "/aer_dev_correctable")), + NonFatal: aerTotal(m.read(dir + "/aer_dev_nonfatal")), + Fatal: aerTotal(m.read(dir + "/aer_dev_fatal"))} + if e.Correctable+e.NonFatal+e.Fatal == 0 { + continue + } + if e.Fatal+e.NonFatal > 0 { + e.Flag = FlagHigh + } + f.PCI = append(f.PCI, e) + } + var core, pkg int64 + counted := false + for _, cpu := range m.numbered("/sys/devices/system/cpu", "cpu") { + dir := "/sys/devices/system/cpu/" + cpu + "/thermal_throttle" + if c, ok := m.readInt(dir + "/core_throttle_count"); ok { + core += c + counted = true + } + if p, ok := m.readInt(dir + "/package_throttle_count"); ok && p > pkg { + pkg = p + } + } + if counted { + f.ThrottledCore, f.ThrottledPackage = &core, &pkg + } + return f +} + +// aerTotal is the TOTAL_ERR_ line of one AER counter file. +func aerTotal(content string) int64 { + for _, l := range strings.Split(content, "\n") { + fields := strings.Fields(l) + if len(fields) == 2 && strings.HasPrefix(fields[0], "TOTAL_ERR_") { + var n int64 + for _, r := range fields[1] { + if r < '0' || r > '9' { + return 0 + } + n = n*10 + int64(r-'0') + } + return n + } + } + return 0 +} + +func uevent(content, key string) string { + for _, l := range strings.Split(content, "\n") { + if v, ok := strings.CutPrefix(l, key+"="); ok { + return v + } + } + return "" +} diff --git a/modules/sensors/cmd/sensors/findings.go b/modules/sensors/cmd/sensors/findings.go new file mode 100644 index 00000000..37d646b3 --- /dev/null +++ b/modules/sensors/cmd/sensors/findings.go @@ -0,0 +1,254 @@ +package main + +import ( + "fmt" + "strings" +) + +// What the module finds, said two ways: a line per finding for whoever asks (sensors_summary), and the +// module's health for the node-engine (sensors_health, novox/hq ADR 0240 and ADR 0256 rule 4). + +// Look is one reading of every class. +type Look struct { + Temperatures []Temperature + Fans []Fan + Drives []Drive + DrivesNote string + Faults Faults +} + +// LookNow reads every class, and judges the fans against what the watch saw turning since the machine +// started. +func (m *Machine) LookNow() Look { + l := Look{Temperatures: m.Temperatures(), Fans: m.Fans(), Faults: m.Faults()} + l.Drives, l.DrivesNote = m.Drives() + var ran map[string]bool + if r, why := m.ReadReadings(); why == "" && r != nil { + ran = map[string]bool{} + for k := range r.FansRan { + ran[k] = true + } + } + l.Fans = judgeFans(l.Fans, ran, hottest(l.Temperatures)) + return l +} + +// Problems is what makes the module unhealthy: each a sentence for the operator. +func (l Look) Problems() []string { + var out []string + for _, t := range l.Temperatures { + if t.Flag == FlagCritical { + out = append(out, fmt.Sprintf("%s is at %.0f °C, at or above %s of %.0f °C.", capital(t.Name()), *t.C, t.CriticalIs, *t.Critical)) + } + } + for _, d := range l.Drives { + if d.Flag == FlagFailing { + out = append(out, fmt.Sprintf("%s says it is failing: %s.", capital(d.Name()), joinAnd(d.failingSentences))) + } + } + for _, f := range l.Fans { + if f.Flag == FlagStopped { + out = append(out, fmt.Sprintf("%s has stopped: %s.", capital(f.Name()), f.Why)) + } + } + for _, c := range l.Faults.Memory { + if c.Uncorrectable > 0 { + out = append(out, fmt.Sprintf("The memory reports %d uncorrectable errors since the machine started (memory controller %s).", + c.Uncorrectable, c.Controller)) + } + } + return out +} + +// Health is the answer the node-engine reads from a tool check: healthy or not, with why. +type Health struct { + Healthy bool `json:"healthy"` + Why string `json:"why"` + Found []string `json:"found,omitempty"` +} + +// Health judges the look. Unhealthy only on what the hardware says is failing; HIGH readings and what +// could not be read are said by the other tools and raise nothing. +func (l Look) Health() Health { + p := l.Problems() + if len(p) == 0 { + why := "No temperature is at its critical value, no drive says it is failing, no fan has stopped and no memory error is uncorrectable." + unread := 0 + for _, d := range l.Drives { + if d.Health == "not read" { + unread++ + } + } + // Said, never passed as read. A watch that is not running is the watch's liveness to judge. + if unread > 0 { + why += fmt.Sprintf(" The health of %d of %d drives has not been read.", unread, len(l.Drives)) + } + return Health{Healthy: true, Why: why} + } + return Health{Healthy: false, Why: strings.Join(p, " "), Found: p} +} + +// Said is one line per finding, flagged, then one line per class saying what was read. +func (l Look) Said() []string { + var out []string + for _, t := range l.Temperatures { + switch t.Flag { + case FlagCritical: + out = append(out, fmt.Sprintf("CRITICAL: %s at %.1f °C, at or above %s (%.1f °C)", t.Name(), *t.C, t.CriticalIs, *t.Critical)) + case FlagHigh: + out = append(out, fmt.Sprintf("HIGH: %s at %.1f °C, at or above %s (%.1f °C)", t.Name(), *t.C, t.HighIs, *t.High)) + } + if t.Alarm && t.Flag == "" { + out = append(out, fmt.Sprintf("HIGH: %s raises its chip's alarm at %.1f °C", t.Name(), *t.C)) + } + } + for _, d := range l.Drives { + switch d.Flag { + case FlagFailing: + out = append(out, fmt.Sprintf("FAILING: %s: %s", d.Name(), strings.Join(d.Findings, "; "))) + case FlagHigh: + out = append(out, fmt.Sprintf("HIGH: %s: %s", d.Name(), strings.Join(d.Findings, "; "))) + } + } + for _, f := range l.Fans { + if f.Flag == FlagStopped { + out = append(out, fmt.Sprintf("STOPPED: %s: %s", f.Name(), f.Why)) + } + } + for _, c := range l.Faults.Memory { + switch c.Flag { + case FlagCritical: + out = append(out, fmt.Sprintf("CRITICAL: memory controller %s counted %d uncorrectable and %d corrected errors since the machine started", c.Controller, c.Uncorrectable, c.Corrected)) + case FlagHigh: + out = append(out, fmt.Sprintf("HIGH: memory controller %s corrected %d errors since the machine started", c.Controller, c.Corrected)) + } + } + for _, e := range l.Faults.PCI { + if e.Flag != "" { + out = append(out, fmt.Sprintf("HIGH: the PCIe device %s (%s) counted %d fatal and %d non-fatal errors since the machine started", e.Device, orUnknown(e.Driver), e.Fatal, e.NonFatal)) + } + } + return append(out, l.overview()...) +} + +// overview is a line per class: how many were read, the hottest, and what is not present. +func (l Look) overview() []string { + var out []string + read, notRead := 0, 0 + var hot *Temperature + for i, t := range l.Temperatures { + if t.C == nil { + notRead++ + continue + } + read++ + if hot == nil || *t.C > *hot.C { + hot = &l.Temperatures[i] + } + } + switch { + case read == 0 && notRead == 0: + out = append(out, "temperatures: not present (no hwmon chip and no thermal zone reports one)") + case read == 0: + out = append(out, fmt.Sprintf("temperatures: %d sensors, none gave a reading", notRead)) + default: + line := fmt.Sprintf("temperatures: %d read; the hottest is %s at %.1f °C", read, hot.Name(), *hot.C) + switch { + case hot.Critical != nil: + line += fmt.Sprintf(" (critical at %.1f °C)", *hot.Critical) + case hot.High != nil: + line += fmt.Sprintf(" (high at %.1f °C)", *hot.High) + default: + line += ", which reports no threshold of its own" + } + if notRead > 0 { + line += fmt.Sprintf("; %d gave no reading", notRead) + } + out = append(out, line) + } + if len(l.Fans) == 0 { + out = append(out, "fans: not present (no hwmon chip reports a fan)") + } else { + var parts []string + for _, f := range l.Fans { + if f.RPM == nil { + parts = append(parts, f.Label+" not read") + continue + } + parts = append(parts, fmt.Sprintf("%s %d RPM", f.Label, *f.RPM)) + } + out = append(out, "fans: "+strings.Join(parts, ", ")) + } + if len(l.Drives) == 0 { + out = append(out, "drives: not present (no drive backed by hardware)") + } else { + passed, failing, unread := 0, 0, 0 + for _, d := range l.Drives { + switch d.Health { + case "passed": + passed++ + case "failing": + failing++ + default: + unread++ + } + } + line := fmt.Sprintf("drives: %d, %d passed, %d failing", len(l.Drives), passed, failing) + if unread > 0 { + line += fmt.Sprintf(", %d not read", unread) + for _, d := range l.Drives { + if d.NotRead != "" { + line += " (" + d.Device + ": " + d.NotRead + ")" + break + } + } + } + if l.DrivesNote != "" && unread < len(l.Drives) { + line += "; " + l.DrivesNote + } + out = append(out, line) + } + errs := "hardware errors: " + if len(l.Faults.Memory) == 0 { + errs += "memory not present" + } else { + var ce, ue int64 + for _, c := range l.Faults.Memory { + ce, ue = ce+c.Corrected, ue+c.Uncorrectable + } + errs += fmt.Sprintf("memory %d corrected, %d uncorrectable", ce, ue) + } + if l.Faults.PCIDevicesCounting == 0 { + errs += "; PCIe not present" + } else { + errs += fmt.Sprintf("; PCIe %d of %d devices counted an error", len(l.Faults.PCI), l.Faults.PCIDevicesCounting) + } + if l.Faults.ThrottledCore != nil { + errs += fmt.Sprintf("; the CPU was slowed for heat %d times", *l.Faults.ThrottledCore) + } + return append(out, errs+", since the machine started") +} + +func capital(s string) string { + if s == "" { + return s + } + return strings.ToUpper(s[:1]) + s[1:] +} + +func joinAnd(parts []string) string { + switch len(parts) { + case 0: + return "" + case 1: + return parts[0] + } + return strings.Join(parts[:len(parts)-1], ", ") + " and " + parts[len(parts)-1] +} + +func orUnknown(s string) string { + if s == "" { + return "no driver" + } + return s +} diff --git a/modules/sensors/cmd/sensors/helpers_test.go b/modules/sensors/cmd/sensors/helpers_test.go new file mode 100644 index 00000000..c43945d9 --- /dev/null +++ b/modules/sensors/cmd/sensors/helpers_test.go @@ -0,0 +1,189 @@ +package main + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// fake is a machine for a test: a tree standing in for /sys, /proc and /run, a runner answering smartctl +// from a table, and a clock. +type fake struct { + t *testing.T + root string + now time.Time + uid int + answers map[string]Ran + calls []string +} + +const bootID = "11111111-2222-3333-4444-555555555555" + +func newFake(t *testing.T) *fake { + t.Helper() + f := &fake{t: t, root: t.TempDir(), now: time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC), answers: map[string]Ran{}} + f.file("/proc/sys/kernel/random/boot_id", bootID) + return f +} + +func (f *fake) machine() *Machine { + return &Machine{ + Root: f.root, + UID: f.uid, + Now: func() time.Time { return f.now }, + Run: func(_ context.Context, name string, args ...string) Ran { + line := strings.TrimSpace(name + " " + strings.Join(args, " ")) + f.calls = append(f.calls, line) + if r, ok := f.answers[line]; ok { + return r + } + return Ran{Status: 127, Err: "ENOENT"} + }, + } +} + +// file writes a file under the fake root. +func (f *fake) file(path, content string) { + f.t.Helper() + full := filepath.Join(f.root, path) + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { + f.t.Fatal(err) + } + if err := os.WriteFile(full, []byte(content), 0o644); err != nil { + f.t.Fatal(err) + } +} + +// dir makes a directory: sysfs's entries that are links on a real machine are directories here, because +// the module only lists their names and reads files through them. +func (f *fake) dir(path string) { + f.t.Helper() + if err := os.MkdirAll(filepath.Join(f.root, path), 0o755); err != nil { + f.t.Fatal(err) + } +} + +// files writes several files under one directory. +func (f *fake) files(dir string, kv map[string]string) { + for k, v := range kv { + f.file(dir+"/"+k, v) + } +} + +// laptop is this kind of machine as the laptop of 2026-10-08 showed it: a CPU sensor with no threshold, an +// integrated GPU, an NVMe drive whose composite sensor has its own maximum and critical values and whose +// two others give the format's largest value, an ACPI zone that a hwmon chip copies, a network adapter and +// four fans, two of them the vendor's. +func (f *fake) laptop() { + f.files("/sys/class/hwmon/hwmon0", map[string]string{"name": "ACAD"}) + f.files("/sys/class/hwmon/hwmon1", map[string]string{"name": "acpi_fan", "fan1_input": "4015"}) + f.files("/sys/class/hwmon/hwmon2", map[string]string{"name": "acpitz", "temp1_input": "60000"}) + f.files("/sys/class/hwmon/hwmon4", map[string]string{"name": "asus", + "fan1_input": "4000", "fan1_label": "cpu_fan", "fan2_input": "0", "fan2_label": "gpu_fan", "fan3_input": "4900", "fan3_label": "mid_fan"}) + f.files("/sys/class/hwmon/hwmon6", map[string]string{"name": "amdgpu", "temp1_input": "57000", "temp1_label": "edge"}) + f.files("/sys/class/hwmon/hwmon7", map[string]string{"name": "nvme", + "temp1_input": "38850", "temp1_label": "Composite", "temp1_max": "83850", "temp1_crit": "87850", "temp1_min": "-40150", "temp1_alarm": "0", + "temp2_input": "58850", "temp2_label": "Sensor 1", "temp2_max": "65261850", "temp2_min": "-273150", + "temp3_input": "38850", "temp3_label": "Sensor 2", "temp3_max": "65261850", "temp3_min": "-273150"}) + f.files("/sys/class/hwmon/hwmon8", map[string]string{"name": "k10temp", "temp1_input": "60875", "temp1_label": "Tctl"}) + f.files("/sys/class/hwmon/hwmon11", map[string]string{"name": "mt7921_phy0", "temp1_input": "56000"}) + f.files("/sys/class/thermal/thermal_zone0", map[string]string{"type": "acpitz", "temp": "60000", "mode": "enabled", + "trip_point_0_type": "critical", "trip_point_0_temp": "124000", + "trip_point_1_type": "hot", "trip_point_1_temp": "124000", + "trip_point_2_type": "passive", "trip_point_2_temp": "110000", + "trip_point_3_type": "active", "trip_point_3_temp": "60000"}) + f.dir("/sys/class/thermal/thermal_zone0/hwmon2") + f.files("/sys/class/nvme/nvme0", map[string]string{"model": "WD PC SN560 SDDPNQE-1T00-1102"}) + f.dir("/sys/class/nvme/nvme0/hwmon7") + f.dir("/sys/class/nvme/nvme0/nvme0n1") + f.dir("/sys/class/nvme/nvme0/ng0n1") + f.drive("nvme0n1", "WD PC SN560 SDDPNQE-1T00-1102", "", 2000409264, false) + f.drive("loop0", "", "", 100, false) + f.drive("zram0", "", "", 100, false) + f.files("/sys/bus/pci/devices/0000:04:00.0", map[string]string{"uevent": "DRIVER=nvme\nPCI_CLASS=10802\n", + "aer_dev_correctable": "RxErr 0\nTOTAL_ERR_COR 0\n", "aer_dev_nonfatal": "Undefined 0\nTOTAL_ERR_NONFATAL 0\n", "aer_dev_fatal": "TOTAL_ERR_FATAL 0\n"}) + f.dir("/sys/devices/system/edac/mc") +} + +// server is a machine with ECC memory, a CPU sensor with its own maximum and critical values, a board +// chip with a header that has nothing on it, and three SATA drives read by drivetemp. +func (f *fake) server() { + f.files("/sys/class/hwmon/hwmon0", map[string]string{"name": "coretemp", + "temp1_input": "45000", "temp1_label": "Package id 0", "temp1_max": "80000", "temp1_crit": "100000", "temp1_crit_alarm": "0", + "temp2_input": "43000", "temp2_label": "Core 0", "temp2_max": "80000", "temp2_crit": "100000"}) + f.files("/sys/class/hwmon/hwmon1", map[string]string{"name": "nct6775", + "fan1_input": "1200", "fan1_min": "300", "fan1_alarm": "0", + "fan2_input": "0", "fan2_min": "300", "fan2_alarm": "1"}) + for i, d := range []string{"sda", "sdb", "sdc"} { + f.drive(d, "WDC WUH722222AL", "ATA ", 42970644480, true) + h := "hwmon" + string(rune('2'+i)) + f.files("/sys/class/hwmon/"+h, map[string]string{"name": "drivetemp", "temp1_input": "36000", "temp1_max": "60000", "temp1_crit": "70000"}) + f.dir("/sys/block/" + d + "/device/hwmon/" + h) + } + f.drive("md0", "", "", 100, true) + f.files("/sys/devices/system/edac/mc/mc0", map[string]string{"mc_name": "Skylake Socket#0 IMC#0", "ce_count": "0", "ue_count": "0"}) + f.files("/sys/devices/system/edac/mc/mc0/dimm0", map[string]string{"dimm_label": "CPU_SrcID#0_MC#0_Chan#0_DIMM#0", "dimm_ce_count": "0", "dimm_ue_count": "0"}) +} + +// drive adds a block device. +func (f *fake) drive(name, model, vendor string, sectors int64, spins bool) { + f.dir("/sys/block/" + name + "/device") + if model != "" { + f.file("/sys/block/"+name+"/device/model", model) + } + if vendor != "" { + f.file("/sys/block/"+name+"/device/vendor", vendor) + } + f.file("/sys/block/"+name+"/size", itoa(sectors)) + rot := "0" + if spins { + rot = "1" + } + f.file("/sys/block/"+name+"/queue/rotational", rot) +} + +func itoa(n int64) string { + b, _ := json.Marshal(n) + return string(b) +} + +// readings writes the watch's file as if it had read each drive with the smartctl JSON given. +func (f *fake) readings(written time.Time, boot string, drives map[string]string, fansRan ...string) { + r := Readings{BootID: boot, WrittenAt: written, Drives: map[string]DriveReading{}, FansRan: map[string]time.Time{}} + for d, j := range drives { + r.Drives[d] = DriveReading{ReadAt: written, Smartctl: json.RawMessage(j)} + } + for _, k := range fansRan { + r.FansRan[k] = written + } + raw, err := json.Marshal(r) + if err != nil { + f.t.Fatal(err) + } + f.file(ReadingsPath, string(raw)) +} + +// smartctl answers, as smartmontools 7 writes them (trimmed to what the module reads). +const ( + nvmePassed = `{"smartctl":{"exit_status":0},"device":{"protocol":"NVMe"},"model_name":"WD PC SN560 SDDPNQE-1T00-1102","serial_number":"S1","firmware_version":"74115000",` + + `"smart_support":{"available":true,"enabled":true},"smart_status":{"passed":true,"nvme":{"value":0}},` + + `"nvme_smart_health_information_log":{"critical_warning":0,"temperature":39,"available_spare":100,"available_spare_threshold":10,"percentage_used":3,"power_on_hours":2210,"media_errors":0},` + + `"temperature":{"current":39},"power_on_time":{"hours":2210}}` + nvmeSpareLow = `{"smartctl":{"exit_status":8},"model_name":"Samsung SSD 990 PRO 2TB","serial_number":"S2","smart_status":{"passed":false,"nvme":{"value":1}},` + + `"nvme_smart_health_information_log":{"critical_warning":1,"available_spare":5,"available_spare_threshold":10,"percentage_used":97,"power_on_hours":40000,"media_errors":3}}` + ataPassed = `{"smartctl":{"exit_status":0},"model_name":"WDC WUH722222ALE6L4","serial_number":"A1","smart_support":{"available":true,"enabled":true},"smart_status":{"passed":true},` + + `"ata_smart_attributes":{"table":[{"id":5,"name":"Reallocated_Sector_Ct","value":100,"thresh":1,"when_failed":"","raw":{"value":0}},` + + `{"id":197,"name":"Current_Pending_Sector","value":100,"thresh":0,"when_failed":"","raw":{"value":0}},{"id":198,"name":"Offline_Uncorrectable","value":100,"thresh":0,"when_failed":"","raw":{"value":0}}]},` + + `"ata_smart_error_log":{"summary":{"count":0}},"ata_smart_self_test_log":{"standard":{"table":[{"type":{"string":"Short offline"},"status":{"string":"Completed without error","passed":true}}]}},` + + `"temperature":{"current":36},"power_on_time":{"hours":17000}}` + ataFailing = `{"smartctl":{"exit_status":24},"model_name":"WDC WUH722222ALE6L4","serial_number":"A2","smart_support":{"available":true,"enabled":true},"smart_status":{"passed":false},` + + `"ata_smart_attributes":{"table":[{"id":5,"name":"Reallocated_Sector_Ct","value":1,"thresh":10,"when_failed":"now","raw":{"value":4012}},` + + `{"id":197,"name":"Current_Pending_Sector","value":100,"thresh":0,"when_failed":"","raw":{"value":16}}]},"ata_smart_error_log":{"summary":{"count":12}}}` + ataPending = `{"smartctl":{"exit_status":64},"model_name":"WDC WUH722222ALE6L4","serial_number":"A3","smart_support":{"available":true,"enabled":true},"smart_status":{"passed":true},` + + `"ata_smart_attributes":{"table":[{"id":197,"name":"Current_Pending_Sector","value":100,"thresh":0,"when_failed":"","raw":{"value":8}}]},"ata_smart_error_log":{"summary":{"count":0}}}` + noSMART = `{"smartctl":{"exit_status":4,"messages":[{"string":"Read Device Identity failed: Unknown","severity":"error"}]},"smart_support":{"available":false}}` +) diff --git a/modules/sensors/cmd/sensors/machine.go b/modules/sensors/cmd/sensors/machine.go new file mode 100644 index 00000000..925b3669 --- /dev/null +++ b/modules/sensors/cmd/sensors/machine.go @@ -0,0 +1,157 @@ +package main + +import ( + "bytes" + "context" + "errors" + "fmt" + "io/fs" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "time" +) + +// What the module reads, and as whom. +// +// The tools run as the operator account in the node's tool runner (novox/hq ADR 0175 §4) and read only +// /sys and /proc, which every account may. A drive's health needs root: the NVMe controller's device and a +// SATA drive's SMART commands are root's. So the module's watch, a process of this same binary that the +// node-engine runs as root, reads it with smartctl and leaves its readings in one file the tools read +// (novox/hq ADR 0256). + +// CommandTimeout bounds one smartctl: a drive that does not answer costs one line, never the watch. +const CommandTimeout = 20 * time.Second + +// Ran is what one command did: its output and exit status, and why it never ran to an answer. +type Ran struct { + Stdout string + Stderr string + Status int + // Err is "ENOENT" when the program is not there, or that it was ended for taking too long. + Err string +} + +// Runner runs one command; injected, so the watch is tested over recorded answers. +type Runner func(ctx context.Context, name string, args ...string) Ran + +// ExecRunner runs a command on this machine, in the C locale, bounded by CommandTimeout. +func ExecRunner(ctx context.Context, name string, args ...string) Ran { + ctx, cancel := context.WithTimeout(ctx, CommandTimeout) + defer cancel() + cmd := exec.CommandContext(ctx, name, args...) + cmd.Env = append(os.Environ(), "LC_ALL=C") + var out, errb bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errb + err := cmd.Run() + r := Ran{Stdout: out.String(), Stderr: errb.String()} + if ctx.Err() == context.DeadlineExceeded { + r.Status, r.Err = 124, fmt.Sprintf("no answer within %d s", int(CommandTimeout.Seconds())) + return r + } + var exit *exec.ExitError + switch { + case err == nil: + case errors.As(err, &exit): + r.Status = exit.ExitCode() + case errors.Is(err, exec.ErrNotFound) || errors.Is(err, fs.ErrNotExist): + r.Status, r.Err = 127, "ENOENT" + default: + r.Status, r.Err = 126, err.Error() + } + return r +} + +// Machine is what the module reads: a filesystem root (the real one, or a test's tree standing in for +// /sys, /proc and /run), a way to run a command, and the time. +type Machine struct { + Root string + Run Runner + Now func() time.Time + // UID is who this process is: smartctl is run directly by root and through sudo -n by anyone else. + UID int +} + +// Here is the machine this process runs on. +func Here() *Machine { + return &Machine{Root: "/", Run: ExecRunner, Now: time.Now, UID: os.Getuid()} +} + +func (m *Machine) path(p string) string { return filepath.Join(m.Root, p) } + +// read is a file's content, trimmed; "" when it cannot be read. +func (m *Machine) read(p string) string { + b, err := os.ReadFile(m.path(p)) + if err != nil { + return "" + } + return strings.TrimSpace(string(b)) +} + +// readInt is a file holding one whole number; false when it is absent, unreadable or not a number — a +// sensor whose driver answers an error (ENODATA, EIO) is said as not read, never as 0. +func (m *Machine) readInt(p string) (int64, bool) { + s := m.read(p) + if s == "" { + return 0, false + } + n, err := strconv.ParseInt(s, 10, 64) + return n, err == nil +} + +func (m *Machine) exists(p string) bool { + _, err := os.Stat(m.path(p)) + return err == nil +} + +// names lists the entries of a directory, sorted; none when it cannot be read. +func (m *Machine) names(dir string) []string { + entries, err := os.ReadDir(m.path(dir)) + if err != nil { + return nil + } + out := make([]string, 0, len(entries)) + for _, e := range entries { + out = append(out, e.Name()) + } + return out +} + +// numbered lists a directory's entries named prefix, in numeric order (hwmon2 before hwmon10). +func (m *Machine) numbered(dir, prefix string) []string { + var out []string + for _, n := range m.names(dir) { + if rest, ok := strings.CutPrefix(n, prefix); ok { + if _, err := strconv.Atoi(rest); err == nil { + out = append(out, n) + } + } + } + sortNumbered(out, prefix) + return out +} + +func sortNumbered(names []string, prefix string) { + num := func(s string) int { + n, _ := strconv.Atoi(strings.TrimPrefix(s, prefix)) + return n + } + for i := 1; i < len(names); i++ { + for j := i; j > 0 && num(names[j]) < num(names[j-1]); j-- { + names[j], names[j-1] = names[j-1], names[j] + } + } +} + +// bootID names this boot, so a reading from before the machine last started is not taken for this one. +func (m *Machine) bootID() string { return m.read("/proc/sys/kernel/random/boot_id") } + +// round1 rounds to one decimal, for what a person reads. +func round1(f float64) float64 { + if f < 0 { + return -float64(int64(-f*10+0.5)) / 10 + } + return float64(int64(f*10+0.5)) / 10 +} diff --git a/modules/sensors/cmd/sensors/main.go b/modules/sensors/cmd/sensors/main.go new file mode 100644 index 00000000..e77f4301 --- /dev/null +++ b/modules/sensors/cmd/sensors/main.go @@ -0,0 +1,37 @@ +// The sensors module's Go bundle (novox/hq ADR 0256): the condition of a machine's own hardware — +// temperatures with their own thresholds, fans, each drive's health, the kernel's counts of hardware +// errors — read from /sys and /proc, and changing nothing. +// +// One binary, two ways to run. With no argument it is the bundle the node's tool runner launches and +// speaks MCP to over stdio (ADR 0188, ADR 0193), as the operator account. With `watch` it is the module's +// long-running process, run by the node-engine as root, which reads the drives' health and notes which +// fans turn (watch.go). +package main + +import ( + "context" + "fmt" + "os" + "os/signal" + "syscall" + + stdio "git.novox.be/novox/mesh-sdk/go" +) + +func main() { + if len(os.Args) > 1 && os.Args[1] == "watch" { + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt) + defer stop() + NewWatch(Here()).Run(ctx) + return + } + if len(os.Args) > 1 { + fmt.Fprintf(os.Stderr, "sensors: %q is not a way to run; with no argument it serves its tools, with watch it watches\n", os.Args[1]) + os.Exit(2) + } + // An empty name serves as the module the runtime names (MESH_SERVED_MODULE): sensors. + if err := stdio.Serve("", Tools(Here())); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} diff --git a/modules/sensors/cmd/sensors/manifest_test.go b/modules/sensors/cmd/sensors/manifest_test.go new file mode 100644 index 00000000..9fcf9829 --- /dev/null +++ b/modules/sensors/cmd/sensors/manifest_test.go @@ -0,0 +1,102 @@ +package main + +import ( + "encoding/json" + "os" + "regexp" + "sort" + "strings" + "testing" +) + +type manifest struct { + Tools []string `json:"tools"` + Claims []any `json:"claims"` + Resources []map[string]any `json:"resources"` +} + +func readManifest(t *testing.T) (manifest, string) { + t.Helper() + raw, err := os.ReadFile("../../module.json") + if err != nil { + t.Fatal(err) + } + var m manifest + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + return m, string(raw) +} + +func TestTheManifestNamesExactlyTheToolsTheBundleServesAllReadOnlyAndNoMachine(t *testing.T) { + m, raw := readManifest(t) + var served []string + for _, tl := range Tools(newFake(t).machine()) { + served = append(served, tl.Name) + if !strings.HasSuffix(tl.Description, "(r)") { + t.Errorf("%s only reads and does not say so", tl.Name) + } + } + sort.Strings(served) + sort.Strings(m.Tools) + if strings.Join(served, ",") != strings.Join(m.Tools, ",") { + t.Fatalf("served %v, listed %v", served, m.Tools) + } + if len(m.Claims) != 0 { + t.Errorf("the module holds no seat (ADR 0256 rule 3), and claims %v", m.Claims) + } + // A machine's or a person's name, as a word: "novox/hq" names the record, not the machine. + if hit := regexp.MustCompile(`\b(g14|novox|ace|shanks|jochen)\b|/home/`).FindString(strings.ReplaceAll(raw, "novox/hq", "")); hit != "" { + t.Errorf("the manifest says %q", hit) + } +} + +// ADR 0240 rule 2 and ADR 0256 rule 4: the tool check on the watch, beside the unit's own readiness on +// smartd, which the module does not judge itself. +func TestTheHealthIsTheDriveDaemonsUnitAndTheWatchsToolCheck(t *testing.T) { + m, _ := readManifest(t) + byID := map[string]map[string]any{} + for _, r := range m.Resources { + byID[r["id"].(string)] = r + } + smartd, watch := byID["smartd"], byID["watch"] + if smartd == nil || watch == nil { + t.Fatalf("resources: %v", byID) + } + if h, _ := smartd["health"].(map[string]any); h["kind"] != "unit" || smartd["state"] != "running" { + t.Errorf("smartd is judged by its unit: %v", smartd) + } + h, _ := watch["health"].(map[string]any) + if h["kind"] != "tool" || h["tool"] != "sensors_health" || watch["type"] != "process" || watch["user"] != nil { + t.Errorf("the watch runs as root and is judged by sensors_health: %v", watch) + } + run, _ := watch["run"].([]any) + if len(run) != 2 || run[1] != "watch" { + t.Errorf("the watch runs the bundle's watch: %v", run) + } + conf, _ := byID["smartd-conf"]["content"].(string) + if !strings.Contains(conf, "\nDEVICESCAN -a -n standby,q\n") || strings.Contains(conf, " -m ") { + t.Errorf("smartd never wakes a drive and sends no mail: %q", conf) + } + dropIn, _ := byID["smartd-drop-in"]["content"].(string) + for _, want := range []string{"ConditionVirtualization=\n", "ExecStart=\nExecStart=/usr/bin/smartd -n -q never\n"} { + if !strings.Contains(dropIn, want) { + t.Errorf("the drop-in misses %q", want) + } + } + for _, f := range []string{"smartd-conf", "smartd-drop-in"} { + if !strings.Contains(strings.Join(anyStrings(smartd["restart-on"]), ","), f) { + t.Errorf("smartd is restarted when %s changes", f) + } + } +} + +func anyStrings(v any) []string { + var out []string + l, _ := v.([]any) + for _, x := range l { + s, _ := x.(string) + out = append(out, s) + } + return out +} diff --git a/modules/sensors/cmd/sensors/sensors_test.go b/modules/sensors/cmd/sensors/sensors_test.go new file mode 100644 index 00000000..931a4b41 --- /dev/null +++ b/modules/sensors/cmd/sensors/sensors_test.go @@ -0,0 +1,366 @@ +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func find(ts []Temperature, chip, label string) Temperature { + for _, t := range ts { + if t.Chip == chip && t.Label == label { + return t + } + } + return Temperature{} +} + +func TestALaptopsTemperaturesCarryOnlyTheHardwaresOwnThresholds(t *testing.T) { + f := newFake(t) + f.laptop() + ts := f.machine().Temperatures() + if len(ts) != 7 { + t.Fatalf("read %d temperatures, want 7 (the zone's hwmon copy left out): %+v", len(ts), ts) + } + if ts[0].Chip != "k10temp" || ts[0].Kind != "cpu" { + t.Errorf("the CPU is said first, got %+v", ts[0]) + } + comp := find(ts, "nvme", "Composite") + if comp.High == nil || *comp.High != 83.9 || comp.Critical == nil || *comp.Critical != 87.9 || comp.Drive != "nvme0n1 (WD PC SN560 SDDPNQE-1T00-1102)" { + t.Errorf("the NVMe composite sensor keeps its own thresholds and names its drive: %+v", comp) + } + if s1 := find(ts, "nvme", "Sensor 1"); s1.High != nil || s1.Flag != "" { + t.Errorf("a maximum of 65261.85 °C is no threshold: %+v", s1) + } + if cpu := find(ts, "k10temp", "Tctl"); cpu.High != nil || cpu.Critical != nil || cpu.Flag != "" || *cpu.C != 60.9 { + t.Errorf("a sensor with no threshold is read and never flagged: %+v", cpu) + } + zone := find(ts, "acpitz", "thermal_zone0") + if zone.Source != "thermal zone" || *zone.High != 124 || zone.HighIs != "the zone's hot trip" || *zone.Critical != 124 { + t.Errorf("the zone takes its hot and critical trips, not its active one: %+v", zone) + } + for _, x := range ts { + if x.Flag != "" { + t.Errorf("nothing on a cool laptop is flagged: %+v", x) + } + } +} + +func TestATemperatureIsHighAtItsMaximumAndCriticalAtItsCriticalValue(t *testing.T) { + f := newFake(t) + f.server() + f.file("/sys/class/hwmon/hwmon0/temp2_input", "80000") + f.file("/sys/class/hwmon/hwmon0/temp1_input", "100000") + ts := f.machine().Temperatures() + if core := find(ts, "coretemp", "Core 0"); core.Flag != FlagHigh { + t.Errorf("at its own maximum it is HIGH: %+v", core) + } + pkg := find(ts, "coretemp", "Package id 0") + if pkg.Flag != FlagCritical { + t.Fatalf("at its own critical value it is CRITICAL: %+v", pkg) + } + h := f.machine().LookNow().Health() + want := "The CPU (coretemp Package id 0) is at 100 °C, at or above its own critical value of 100 °C." + if h.Healthy || h.Why != want { + t.Errorf("health = %+v, want unhealthy with %q", h, want) + } +} + +func TestAZoneWithoutAHotTripIsHighAtItsPassiveTrip(t *testing.T) { + f := newFake(t) + f.files("/sys/class/thermal/thermal_zone3", map[string]string{"type": "x86_pkg_temp", "temp": "96000", + "trip_point_0_type": "passive", "trip_point_0_temp": "95000", "trip_point_1_type": "passive", "trip_point_1_temp": "98000"}) + ts := f.machine().Temperatures() + if len(ts) != 1 || ts[0].Kind != "cpu" || *ts[0].High != 95 || ts[0].Flag != FlagHigh || ts[0].Critical != nil { + t.Fatalf("the lowest passive trip is its high, and it has no critical: %+v", ts) + } +} + +func TestASensorThatGivesNoReadingIsSaidAsNotRead(t *testing.T) { + f := newFake(t) + f.files("/sys/class/hwmon/hwmon0", map[string]string{"name": "iwlwifi_1", "temp1_label": "x"}) + f.dir("/sys/class/hwmon/hwmon0/temp1_input") // a read that fails, as ENODATA does + f.files("/sys/class/thermal/thermal_zone0", map[string]string{"type": "acpitz", "temp": "-274000"}) + ts := f.machine().Temperatures() + if len(ts) != 2 { + t.Fatalf("both are listed: %+v", ts) + } + for _, x := range ts { + if x.C != nil || x.NotRead == "" || x.Flag != "" { + t.Errorf("not read, never cool: %+v", x) + } + } + if !strings.Contains(ts[0].NotRead, "-274.0 °C, which is no temperature") { + t.Errorf("the zone says what it answered: %q", ts[0].NotRead) + } +} + +func TestAMachineWithNoSensorAtAllHasEveryClassNotPresentAndIsHealthy(t *testing.T) { + f := newFake(t) + f.drive("vda", "", "", 41943040, true) + f.readings(f.now, bootID, map[string]string{"vda": noSMART}) + l := f.machine().LookNow() + said := strings.Join(l.Said(), "\n") + for _, want := range []string{ + "temperatures: not present", "fans: not present", "drives: 1, 0 passed, 0 failing, 1 not read (vda: the drive does not support SMART)", + "memory not present", "PCIe not present", + } { + if !strings.Contains(said, want) { + t.Errorf("summary misses %q:\n%s", want, said) + } + } + if h := l.Health(); !h.Healthy || !strings.Contains(h.Why, "The health of 1 of 1 drives has not been read.") { + t.Errorf("a machine without sensors is healthy, and says what it could not read: %+v", h) + } +} + +func TestADriveThatSaysItIsFailingMakesTheModuleUnhealthy(t *testing.T) { + f := newFake(t) + f.server() + f.readings(f.now, bootID, map[string]string{"sda": ataPassed, "sdb": ataFailing, "sdc": ataPending}) + ds, note := f.machine().Drives() + if note != "" || len(ds) != 3 { + t.Fatalf("three drives, the RAID volume left out, no note: %d, %q", len(ds), note) + } + if ds[0].Health != "passed" || ds[0].Flag != "" || *ds[0].Reallocated != 0 || ds[0].LastSelfTest != "Short offline: Completed without error" || ds[0].Transport != "ata" { + t.Errorf("sda passed: %+v", ds[0]) + } + if ds[1].Health != "failing" || ds[1].Flag != FlagFailing { + t.Errorf("sdb is failing: %+v", ds[1]) + } + if ds[2].Health != "passed" || ds[2].Flag != FlagHigh || ds[2].Findings[0] != "it has 8 sectors waiting to be reallocated" { + t.Errorf("sdc's pending sectors are an early sign, HIGH: %+v", ds[2]) + } + h := f.machine().LookNow().Health() + want := "The drive sdb (WDC WUH722222ALE6L4) says it is failing: its overall SMART health check failed and " + + "its attribute Reallocated_Sector_Ct is at 1, at or below its own threshold of 10." + if h.Healthy || h.Why != want { + t.Errorf("health = %+v\nwant why %q", h, want) + } +} + +func TestAnNVMeCriticalWarningBitIsFailing(t *testing.T) { + f := newFake(t) + f.laptop() + f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmeSpareLow}) + ds, _ := f.machine().Drives() + d := ds[0] + if d.Flag != FlagFailing || *d.CriticalWarning != 1 || *d.PercentageUsed != 97 || d.Model != "Samsung SSD 990 PRO 2TB" { + t.Fatalf("spare below threshold is failing: %+v", d) + } + joined := strings.Join(d.Findings, "; ") + for _, want := range []string{"its spare capacity is below its threshold", "it has had 3 media errors", "it has used 97 % of its rated endurance"} { + if !strings.Contains(joined, want) { + t.Errorf("findings miss %q: %s", want, joined) + } + } +} + +func TestAHealthyLaptopReadsHealthyWithItsDriveRead(t *testing.T) { + f := newFake(t) + f.laptop() + f.readings(f.now.Add(-5*time.Minute), bootID, map[string]string{"nvme0n1": nvmePassed}, "asus/fan2") + l := f.machine().LookNow() + if h := l.Health(); !h.Healthy || strings.Contains(h.Why, "not been read") { + t.Errorf("health = %+v", h) + } + d := l.Drives[0] + if d.Health != "passed" || *d.PowerOnHours != 2210 || *d.AvailableSpare != 100 || d.Serial != "S1" { + t.Errorf("drive = %+v", d) + } + said := l.Said() + if len(said) != 4 || !strings.HasPrefix(said[0], "temperatures: 7 read; the hottest is the CPU (k10temp Tctl) at 60.9 °C, which reports no threshold of its own") { + t.Errorf("one line per class and no finding: %q", said) + } + if said[2] != "drives: 1, 1 passed, 0 failing" { + t.Errorf("drives line = %q", said[2]) + } +} + +func TestAMissingStaleOrEarlierBootReadingIsSaid(t *testing.T) { + f := newFake(t) + f.laptop() + if ds, note := f.machine().Drives(); ds[0].Health != "not read" || !strings.Contains(note, "has left no reading") { + t.Errorf("no file: %+v %q", ds[0], note) + } + f.readings(f.now.Add(-2*time.Hour), bootID, map[string]string{"nvme0n1": nvmePassed}) + if ds, note := f.machine().Drives(); ds[0].Health != "passed" || note != "the module's watch last wrote 2.0 h ago; it may not be running" { + t.Errorf("stale: %+v %q", ds[0], note) + } + f.readings(f.now, "an-earlier-boot", map[string]string{"nvme0n1": nvmeSpareLow}, "asus/fan2") + l := f.machine().LookNow() + if l.Drives[0].Health != "not read" || l.Fans[2].RanSinceBoot != nil { + t.Errorf("a reading from before the machine started is not this boot's: %+v %+v", l.Drives[0], l.Fans[2]) + } +} + +func TestAFanAtRestOnACoolMachineIsNotStopped(t *testing.T) { + f := newFake(t) + f.laptop() + f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmePassed}, "asus/fan2") + l := f.machine().LookNow() + gpu := l.Fans[2] + if gpu.Label != "gpu_fan" || *gpu.RPM != 0 || gpu.RanSinceBoot == nil || !*gpu.RanSinceBoot || gpu.Flag != "" { + t.Errorf("a fan that ran and rests while nothing is hot is not stopped: %+v", gpu) + } +} + +func TestAFanThatRanAndReadsZeroWhileTheMachineIsHotHasStopped(t *testing.T) { + f := newFake(t) + f.laptop() + f.file("/sys/class/hwmon/hwmon7/temp1_input", "84000") + f.readings(f.now, bootID, map[string]string{"nvme0n1": nvmePassed}, "asus/fan1", "asus/fan2") + l := f.machine().LookNow() + if l.Fans[2].Flag != FlagStopped { + t.Fatalf("gpu_fan ran and reads 0 while the drive is past its maximum: %+v", l.Fans[2]) + } + h := l.Health() + want := "The fan gpu_fan (asus) has stopped: it ran earlier and reads 0 RPM while the drive nvme0n1 (WD PC SN560 SDDPNQE-1T00-1102), " + + "sensor Composite is at 84 °C, at or above its own maximum of 84 °C." + if h.Healthy || h.Why != want { + t.Errorf("health = %+v\nwant %q", h, want) + } +} + +func TestAnEmptyHeaderIsNotAStoppedFanAndAFanUnderItsMinimumIs(t *testing.T) { + f := newFake(t) + f.server() + l := f.machine().LookNow() + if l.Fans[1].Flag != "" { + t.Errorf("a header never seen turning, with an alarm the firmware set, is no fan: %+v", l.Fans[1]) + } + f.file("/sys/class/hwmon/hwmon1/fan1_input", "200") + l = f.machine().LookNow() + if l.Fans[0].Flag != FlagStopped || l.Fans[0].Why != "it turns at 200 RPM, under its own minimum of 300 RPM" { + t.Errorf("under its own minimum: %+v", l.Fans[0]) + } +} + +func TestMemoryErrorsAreCountedAndAnUncorrectableOneIsUnhealthy(t *testing.T) { + f := newFake(t) + f.server() + f.file("/sys/devices/system/edac/mc/mc0/ce_count", "3") + f.file("/sys/devices/system/edac/mc/mc0/dimm0/dimm_ce_count", "3") + l := f.machine().LookNow() + if c := l.Faults.Memory[0]; c.Flag != FlagHigh || len(c.Modules) != 1 || c.Modules[0].Label != "CPU_SrcID#0_MC#0_Chan#0_DIMM#0" { + t.Errorf("corrected errors are HIGH, by module: %+v", c) + } + if !l.Health().Healthy { + t.Errorf("corrected errors raise nothing") + } + f.file("/sys/devices/system/edac/mc/mc0/ue_count", "1") + h := f.machine().LookNow().Health() + if h.Healthy || h.Why != "The memory reports 1 uncorrectable errors since the machine started (memory controller mc0)." { + t.Errorf("health = %+v", h) + } +} + +func TestPCIeErrorsAndThrottlingAreCountedWhereTheMachineHasThem(t *testing.T) { + f := newFake(t) + f.laptop() + fs := f.machine().Faults() + if fs.PCIDevicesCounting != 1 || len(fs.PCI) != 0 || fs.ThrottledCore != nil || fs.MemoryNotPresent == "" { + t.Errorf("faults = %+v", fs) + } + f.file("/sys/bus/pci/devices/0000:04:00.0/aer_dev_fatal", "TOTAL_ERR_FATAL 2\n") + f.file("/sys/devices/system/cpu/cpu0/thermal_throttle/core_throttle_count", "5") + f.file("/sys/devices/system/cpu/cpu1/thermal_throttle/core_throttle_count", "7") + fs = f.machine().Faults() + if len(fs.PCI) != 1 || fs.PCI[0].Fatal != 2 || fs.PCI[0].Driver != "nvme" || fs.PCI[0].Flag != FlagHigh || *fs.ThrottledCore != 12 { + t.Errorf("faults = %+v", fs) + } +} + +func TestTheWatchReadsEachDriveAsRootWithoutWakingItAndNotesTheFans(t *testing.T) { + f := newFake(t) + f.server() + f.answers["smartctl --json=c -a -n standby /dev/sda"] = Ran{Stdout: ataPassed} + f.answers["smartctl --json=c -a -n standby /dev/sdb"] = Ran{Stdout: ataFailing, Status: 24} + f.answers["smartctl --json=c -a -n standby /dev/sdc"] = Ran{Stdout: "Device is in STANDBY mode, exit(2)\n", Status: 2} + w := NewWatch(f.machine()) + if err := w.Look(); err != nil { + t.Fatal(err) + } + if len(f.calls) != 3 { + t.Fatalf("one smartctl per drive, directly as root: %q", f.calls) + } + info, err := os.Stat(filepath.Join(f.root, ReadingsPath)) + if err != nil || info.Mode().Perm() != 0o644 { + t.Fatalf("the readings are readable by the operator account: %v %v", info, err) + } + r, why := f.machine().ReadReadings() + if why != "" || r.Drives["sdc"].NotRead != "the drive was asleep at every look, and is not woken to be read" || !r.Drives["sdc"].Asleep { + t.Fatalf("sdc asleep: %+v %q", r.Drives["sdc"], why) + } + if _, ok := r.FansRan["nct6775/fan1"]; !ok || len(r.FansRan) != 1 { + t.Errorf("the turning fan is noted, the empty header not: %v", r.FansRan) + } + // Ten minutes have not passed: the fans are looked at, the drives are not read again. + f.now = f.now.Add(FanEvery) + f.calls = nil + if err := w.Look(); err != nil || len(f.calls) != 0 { + t.Fatalf("drives read again within ten minutes: %q %v", f.calls, err) + } + // A new watch (a restart) keeps what the last one read, and a drive asleep keeps its reading before. + f.now = f.now.Add(DriveEvery) + f.answers["smartctl --json=c -a -n standby /dev/sda"] = Ran{Stdout: "Device is in SLEEP mode, exit(2)\n", Status: 2} + w2 := NewWatch(f.machine()) + if err := w2.Look(); err != nil { + t.Fatal(err) + } + r, _ = f.machine().ReadReadings() + if !r.Drives["sda"].Asleep || len(r.Drives["sda"].Smartctl) == 0 { + t.Errorf("sda asleep keeps its reading: %+v", r.Drives["sda"]) + } + ds, _ := f.machine().Drives() + if ds[0].Health != "passed" || !ds[0].Asleep || ds[1].Health != "failing" { + t.Errorf("drives = %+v", ds) + } +} + +func TestTheWatchNotRootAsksThroughSudoAndSaysWhatStopsIt(t *testing.T) { + f := newFake(t) + f.uid = 1000 + f.drive("sda", "X", "ATA ", 100, true) + f.answers["sudo -n smartctl --json=c -a -n standby /dev/sda"] = Ran{Stderr: "sudo: a password is required\n", Status: 1} + got := f.machine().ReadDrive("sda", nil) + if got.NotRead != "the watch is not root and may not run smartctl through sudo without a prompt" { + t.Errorf("not read = %q", got.NotRead) + } + f.uid = 0 + got = f.machine().ReadDrive("sda", nil) + if !strings.Contains(got.NotRead, "smartctl is not installed") { + t.Errorf("not read = %q", got.NotRead) + } +} + +func TestTheHealthAnswerIsWhatTheNodeEngineReads(t *testing.T) { + f := newFake(t) + f.server() + f.file("/sys/class/hwmon/hwmon0/temp1_input", "101000") + f.file("/sys/devices/system/edac/mc/mc0/ue_count", "2") + var tool func(map[string]any) (any, error) + for _, tl := range Tools(f.machine()) { + if tl.Name == "sensors_health" { + tool = tl.Run + } + } + out, err := tool(map[string]any{}) + if err != nil { + t.Fatal(err) + } + raw, _ := json.Marshal(out) + var a struct { + Healthy bool `json:"healthy"` + Why string `json:"why"` + } + if err := json.Unmarshal(raw, &a); err != nil || a.Healthy { + t.Fatalf("answer %s", raw) + } + if strings.Count(a.Why, ". ") != 1 || !strings.HasSuffix(a.Why, ".") || strings.ContainsAny(a.Why, "\n`") { + t.Errorf("one sentence per finding, plain: %q", a.Why) + } +} diff --git a/modules/sensors/cmd/sensors/temperatures.go b/modules/sensors/cmd/sensors/temperatures.go new file mode 100644 index 00000000..a89ceca0 --- /dev/null +++ b/modules/sensors/cmd/sensors/temperatures.go @@ -0,0 +1,282 @@ +package main + +import ( + "fmt" + "sort" + "strings" +) + +// Temperatures, from the kernel's hwmon chips and its thermal zones, each with the thresholds the hardware +// gives for itself (novox/hq ADR 0256 rule 2). The mesh adds no threshold of its own: a sensor that gives +// none is said without a flag. + +// Flags, said beside a reading that has reached one of the hardware's own thresholds. +const ( + FlagHigh = "HIGH" + FlagCritical = "CRITICAL" +) + +// Temperature is one sensor's reading, in °C. +type Temperature struct { + // Source is "hwmon" or "thermal zone". + Source string `json:"source"` + // Chip is the hwmon chip's name (k10temp, nvme, amdgpu) or the zone's type (acpitz, x86_pkg_temp). + Chip string `json:"chip"` + // Kind is what it measures: cpu, gpu, drive, board, network or other. + Kind string `json:"kind"` + // Drive is the drive a drive sensor sits in, by its block device, with its model. + Drive string `json:"drive,omitempty"` + Label string `json:"label"` + C *float64 `json:"c,omitempty"` + // High is where the hardware says it runs hot: its own maximum, or a zone's hot or passive trip. + High *float64 `json:"high_c,omitempty"` + HighIs string `json:"high_is,omitempty"` + // Critical is where the hardware says it must not go: its own critical or emergency value, or a zone's + // critical trip. + Critical *float64 `json:"critical_c,omitempty"` + CriticalIs string `json:"critical_is,omitempty"` + // Alarm is the chip's own alarm bit for this sensor, where it has one and it is set. + Alarm bool `json:"alarm,omitempty"` + Flag string `json:"flag,omitempty"` + NotRead string `json:"not_read,omitempty"` + // Where is the directory it was read from, for whoever looks closer. + Where string `json:"where"` +} + +// Name is how a person reads which sensor this is. +func (t Temperature) Name() string { + what := t.Chip + if t.Label != "" && t.Label != t.Chip { + what += " " + t.Label + } + switch { + case t.Drive != "": + return fmt.Sprintf("the drive %s, sensor %s", t.Drive, t.Label) + case t.Kind == "cpu": + return fmt.Sprintf("the CPU (%s)", what) + case t.Kind == "gpu": + return fmt.Sprintf("the GPU (%s)", what) + case t.Kind == "board": + return fmt.Sprintf("the board (%s)", what) + case t.Kind == "network": + return fmt.Sprintf("the network adapter (%s)", what) + } + return "the sensor " + what +} + +// sane says whether a value in °C is a temperature or a threshold the hardware means as one. A +// threshold given as the format's largest value (an NVMe sensor's 65261.85 °C) or as 0 means none, and +// so does anything a machine cannot be at and still answer. +func sane(c float64) bool { return c > -40 && c < 200 && c != 0 } + +func milli(v int64) *float64 { + c := round1(float64(v) / 1000) + return &c +} + +// chipKinds is what a hwmon chip or a thermal zone's type measures, by the driver's name. +var chipKinds = []struct { + prefix, kind string +}{ + {"k10temp", "cpu"}, {"coretemp", "cpu"}, {"zenpower", "cpu"}, {"cpu_thermal", "cpu"}, {"cpu-thermal", "cpu"}, + {"x86_pkg_temp", "cpu"}, {"via_cputemp", "cpu"}, {"soc_thermal", "cpu"}, {"TCPU", "cpu"}, + {"amdgpu", "gpu"}, {"radeon", "gpu"}, {"nouveau", "gpu"}, {"i915", "gpu"}, {"xe", "gpu"}, {"nvidia", "gpu"}, + {"gpu", "gpu"}, + {"nvme", "drive"}, {"drivetemp", "drive"}, + {"iwlwifi", "network"}, {"mt79", "network"}, {"mt76", "network"}, {"ath", "network"}, {"r8169", "network"}, + {"igb", "network"}, {"ixgbe", "network"}, {"mlx", "network"}, {"bnxt", "network"}, + {"acpitz", "board"}, {"pch_", "board"}, {"nct", "board"}, {"it87", "board"}, {"it86", "board"}, + {"w83", "board"}, {"f71", "board"}, {"asus", "board"}, {"dell_smm", "board"}, {"thinkpad", "board"}, + {"applesmc", "board"}, {"INT3400", "board"}, {"SEN", "board"}, {"TSKN", "board"}, +} + +func kindOf(chip string) string { + for _, k := range chipKinds { + if strings.HasPrefix(chip, k.prefix) { + return k.kind + } + } + return "other" +} + +var kindOrder = map[string]int{"cpu": 0, "gpu": 1, "drive": 2, "board": 3, "network": 4, "other": 5} + +// Temperatures is every temperature the machine reports: each thermal zone, then each hwmon sensor that is +// not a zone's copy (the zone has the same reading and its trip points). +func (m *Machine) Temperatures() []Temperature { + var out []Temperature + inZone := map[string]bool{} + for _, zone := range m.numbered("/sys/class/thermal", "thermal_zone") { + dir := "/sys/class/thermal/" + zone + for _, n := range m.names(dir) { + if strings.HasPrefix(n, "hwmon") { + inZone[n] = true + } + } + out = append(out, m.zone(dir)) + } + drives := m.driveOfHwmon() + for _, h := range m.numbered("/sys/class/hwmon", "hwmon") { + if inZone[h] { + continue + } + dir := "/sys/class/hwmon/" + h + chip := m.read(dir + "/name") + for _, input := range m.inputs(dir, "temp") { + out = append(out, m.hwmonTemperature(dir, chip, input, drives[h])) + } + } + sort.SliceStable(out, func(i, j int) bool { + if kindOrder[out[i].Kind] != kindOrder[out[j].Kind] { + return kindOrder[out[i].Kind] < kindOrder[out[j].Kind] + } + return false + }) + return out +} + +// inputs lists a hwmon directory's N_input files by their stem (temp1, temp2), in numeric order. +func (m *Machine) inputs(dir, kind string) []string { + var stems []string + for _, n := range m.names(dir) { + if strings.HasPrefix(n, kind) && strings.HasSuffix(n, "_input") { + stems = append(stems, strings.TrimSuffix(n, "_input")) + } + } + sortNumbered(stems, kind) + return stems +} + +func (m *Machine) hwmonTemperature(dir, chip, stem, drive string) Temperature { + t := Temperature{Source: "hwmon", Chip: chip, Kind: kindOf(chip), Drive: drive, Label: m.read(dir + "/" + stem + "_label"), + Where: dir + "/" + stem} + if t.Label == "" { + t.Label = stem + } + if drive != "" { + t.Kind = "drive" + } + v, ok := m.readInt(dir + "/" + stem + "_input") + if !ok || !sane(float64(v)/1000) { + t.NotRead = "the sensor gave no temperature" + if ok { + t.NotRead = fmt.Sprintf("the sensor answered %.1f °C, which is no temperature", float64(v)/1000) + } + return t + } + t.C = milli(v) + if h, ok := m.readInt(dir + "/" + stem + "_max"); ok && sane(float64(h)/1000) { + t.High, t.HighIs = milli(h), "its own maximum" + } + if c, ok := m.readInt(dir + "/" + stem + "_crit"); ok && sane(float64(c)/1000) { + t.Critical, t.CriticalIs = milli(c), "its own critical value" + } else if e, ok := m.readInt(dir + "/" + stem + "_emergency"); ok && sane(float64(e)/1000) { + t.Critical, t.CriticalIs = milli(e), "its own emergency value" + } + for _, alarm := range []string{"_alarm", "_crit_alarm", "_max_alarm", "_emergency_alarm"} { + if a, ok := m.readInt(dir + "/" + stem + alarm); ok && a != 0 { + t.Alarm = true + } + } + t.Flag = flagOf(t) + return t +} + +// zone reads one thermal zone and its trip points: critical is the critical trip, high the hot trip or, +// without one, the lowest passive trip (where the kernel starts to slow the machine down). Active trips +// are where a fan steps up, and are nothing to report. +func (m *Machine) zone(dir string) Temperature { + typ := m.read(dir + "/type") + t := Temperature{Source: "thermal zone", Chip: typ, Kind: kindOf(typ), Label: strings.TrimPrefix(dir, "/sys/class/thermal/"), + Where: dir} + v, ok := m.readInt(dir + "/temp") + if !ok || !sane(float64(v)/1000) { + t.NotRead = "the zone gave no temperature" + if ok { + t.NotRead = fmt.Sprintf("the zone answered %.1f °C, which is no temperature", float64(v)/1000) + } + return t + } + t.C = milli(v) + var hot, passive *float64 + for _, n := range m.names(dir) { + stem, ok := strings.CutSuffix(n, "_type") + if !ok || !strings.HasPrefix(stem, "trip_point_") { + continue + } + tv, ok := m.readInt(dir + "/" + stem + "_temp") + if !ok || !sane(float64(tv)/1000) { + continue + } + c := milli(tv) + switch m.read(dir + "/" + n) { + case "critical": + if t.Critical == nil || *c < *t.Critical { + t.Critical, t.CriticalIs = c, "the zone's critical trip" + } + case "hot": + if hot == nil || *c < *hot { + hot = c + } + case "passive": + if passive == nil || *c < *passive { + passive = c + } + } + } + switch { + case hot != nil: + t.High, t.HighIs = hot, "the zone's hot trip" + case passive != nil: + t.High, t.HighIs = passive, "the zone's passive trip, where the kernel slows the machine down" + } + t.Flag = flagOf(t) + return t +} + +func flagOf(t Temperature) string { + if t.C == nil { + return "" + } + switch { + case t.Critical != nil && *t.C >= *t.Critical: + return FlagCritical + case t.High != nil && *t.C >= *t.High: + return FlagHigh + } + return "" +} + +// driveOfHwmon names the drive each drive sensor's hwmon chip sits in: an NVMe controller lists its hwmon +// chip and its namespaces, and a SATA drive read by drivetemp lists its chip under its device. +func (m *Machine) driveOfHwmon() map[string]string { + out := map[string]string{} + for _, ctrl := range m.numbered("/sys/class/nvme", "nvme") { + dir := "/sys/class/nvme/" + ctrl + name := ctrl + for _, n := range m.names(dir) { + if strings.HasPrefix(n, ctrl+"n") && !strings.Contains(n, "p") { + name = n + break + } + } + if model := m.read(dir + "/model"); model != "" { + name += " (" + model + ")" + } + for _, n := range m.names(dir) { + if strings.HasPrefix(n, "hwmon") { + out[n] = name + } + } + } + for _, b := range m.names("/sys/block") { + for _, n := range m.names("/sys/block/" + b + "/device/hwmon") { + name := b + if model := m.read("/sys/block/" + b + "/device/model"); model != "" { + name += " (" + model + ")" + } + out[n] = name + } + } + return out +} diff --git a/modules/sensors/cmd/sensors/tools.go b/modules/sensors/cmd/sensors/tools.go new file mode 100644 index 00000000..1bf8e6d0 --- /dev/null +++ b/modules/sensors/cmd/sensors/tools.go @@ -0,0 +1,82 @@ +package main + +import ( + stdio "git.novox.be/novox/mesh-sdk/go" +) + +// Tools is the module's tools over one machine. Every one only reads, and none needs root: what only root +// may read, the watch has read (drives.go). +func Tools(m *Machine) []stdio.Tool { + return []stdio.Tool{ + { + Name: "sensors_summary", + Description: "The condition of this machine's hardware in a few lines: one per finding, flagged CRITICAL, HIGH, FAILING or " + + "STOPPED against the hardware's own thresholds (never the mesh's), then one per class — temperatures with the hottest, " + + "fans, drives' health, memory and PCIe errors — saying what was read and what the machine does not have. Replaces " + + "sensors and smartctl -H. (r)", + Run: func(map[string]any) (any, error) { + l := m.LookNow() + h := l.Health() + return map[string]any{"said": l.Said(), "healthy": h.Healthy}, nil + }, + }, + { + Name: "sensors_temperatures", + Description: "Every temperature the machine reports, in °C: each thermal zone with its trip points, and each hwmon sensor " + + "(CPU, GPU, drives, board, network adapters) with its label and its own maximum and critical values. HIGH at its own " + + "maximum (or a zone's hot or passive trip), CRITICAL at its own critical value; a sensor that gives no threshold is " + + "never flagged, and one that gives no reading is said as not read. Replaces sensors. (r)", + Run: func(map[string]any) (any, error) { + ts := m.Temperatures() + return map[string]any{"temperatures": ts, "count": len(ts)}, nil + }, + }, + { + Name: "sensors_fans", + Description: "Every fan the machine reports: RPM, the chip's own minimum and alarm where it has them, and whether the " + + "module's watch saw it turning since the machine started. STOPPED when its chip says so, or when it ran and reads 0 " + + "while a temperature is at or above its own maximum; a fan resting at 0 on a cool machine is not stopped. " + + "Replaces sensors. (r)", + Run: func(map[string]any) (any, error) { + l := m.LookNow() + return map[string]any{"fans": l.Fans, "count": len(l.Fans)}, nil + }, + }, + { + Name: "sensors_drives", + Description: "Each drive's health, as the module's watch last read it as root with smartctl (every 10 minutes, never " + + "waking a drive in standby): model, serial, size, whether it spins, the overall SMART verdict; for NVMe the critical " + + "warning bits, spare, endurance used and media errors; for SATA the reallocated, pending and unreadable sectors, the " + + "error log and the last self-test. FAILING when the drive says so, HIGH for the early signs. Replaces smartctl -a " + + "and nvme smart-log. (r)", + Run: func(map[string]any) (any, error) { + ds, note := m.Drives() + out := map[string]any{"drives": ds, "count": len(ds)} + if note != "" { + out["note"] = note + } + return out, nil + }, + }, + { + Name: "sensors_faults", + Description: "The kernel's counts of hardware errors since the machine started: corrected and uncorrectable memory " + + "errors per memory controller and module (EDAC, where the memory has ECC), PCIe errors per device that counted one " + + "(AER), and how often an Intel CPU was slowed for heat. A machine without a class has it as not present. Replaces " + + "edac-util and reading /sys by hand. (r)", + Run: func(map[string]any) (any, error) { + return m.Faults(), nil + }, + }, + { + Name: "sensors_health", + Description: "Whether the hardware is failing, as the module's health check (novox/hq ADR 0256): unhealthy when a " + + "temperature is at or above its own critical value, a drive says it is failing, a fan that ran has stopped while " + + "the machine is hot, or memory reports an uncorrectable error; with why, one sentence per finding. HIGH readings " + + "do not make it unhealthy. (r)", + Run: func(map[string]any) (any, error) { + return m.LookNow().Health(), nil + }, + }, + } +} diff --git a/modules/sensors/cmd/sensors/watch.go b/modules/sensors/cmd/sensors/watch.go new file mode 100644 index 00000000..0d32a1fa --- /dev/null +++ b/modules/sensors/cmd/sensors/watch.go @@ -0,0 +1,96 @@ +package main + +import ( + "context" + "fmt" + "os" + "time" +) + +// The watch: the module's long-running process, run by the node-engine as root (novox/hq ADR 0256). It +// reads each drive's health every DriveEvery, so the tools, which run as the operator account, need no +// root; it never wakes a drive in standby; and it notes every FanEvery which fans are turning, so a fan +// that ran and stopped can be told from one that never runs. It writes ReadingsPath and nothing else. +// +// The module's health is judged on it: by its unit (alive), and by asking sensors_health what the +// hardware says. + +const ( + FanEvery = 30 * time.Second + DriveEvery = 10 * time.Minute +) + +// Watch keeps the readings between looks. +type Watch struct { + M *Machine + readings Readings + lastRead time.Time +} + +// NewWatch starts from what an earlier watch of this boot left, so a restart neither forgets the fans +// that ran nor reads every drive again at once. +func NewWatch(m *Machine) *Watch { + w := &Watch{M: m, readings: Readings{Drives: map[string]DriveReading{}, FansRan: map[string]time.Time{}}} + if r, why := m.ReadReadings(); why == "" && r != nil { + w.readings = *r + if w.readings.Drives == nil { + w.readings.Drives = map[string]DriveReading{} + } + if w.readings.FansRan == nil { + w.readings.FansRan = map[string]time.Time{} + } + for _, d := range r.Drives { + if d.ReadAt.After(w.lastRead) { + w.lastRead = d.ReadAt + } + } + } + w.readings.BootID = m.bootID() + return w +} + +// Look is one look: the fans always, the drives when DriveEvery has passed. It writes the file. +func (w *Watch) Look() error { + now := w.M.Now() + for _, f := range w.M.Fans() { + if f.RPM != nil && *f.RPM > 0 { + w.readings.FansRan[f.Key] = now + } + } + if w.lastRead.IsZero() || now.Sub(w.lastRead) >= DriveEvery { + present := map[string]bool{} + for _, d := range w.M.BlockDrives() { + present[d.Device] = true + var before *DriveReading + if b, ok := w.readings.Drives[d.Device]; ok && len(b.Smartctl) > 0 { + before = &b + } + w.readings.Drives[d.Device] = w.M.ReadDrive(d.Device, before) + } + for name := range w.readings.Drives { + if !present[name] { + delete(w.readings.Drives, name) + } + } + w.lastRead = now + } + w.readings.WrittenAt = w.M.Now() + return w.M.WriteReadings(w.readings) +} + +// Run looks at once and every FanEvery until ctx ends. A look that cannot write is said on standard +// error, which the node-engine keeps in the unit's journal, and the watch goes on. +func (w *Watch) Run(ctx context.Context) { + t := time.NewTicker(FanEvery) + defer t.Stop() + for { + if err := w.Look(); err != nil { + fmt.Fprintln(os.Stderr, "sensors watch: the readings were not written:", err) + } + select { + case <-ctx.Done(): + return + case <-t.C: + } + } +} diff --git a/modules/sensors/go.mod b/modules/sensors/go.mod new file mode 100644 index 00000000..90087d61 --- /dev/null +++ b/modules/sensors/go.mod @@ -0,0 +1,5 @@ +module sensors + +go 1.22 + +require git.novox.be/novox/mesh-sdk/go v0.1.6 diff --git a/modules/sensors/go.sum b/modules/sensors/go.sum new file mode 100644 index 00000000..0dd60617 --- /dev/null +++ b/modules/sensors/go.sum @@ -0,0 +1,2 @@ +git.novox.be/novox/mesh-sdk/go v0.1.6 h1:9qzdYONYbJdWcu6sxQcq9v1LI0JxcfkiKYkMUzJSkVQ= +git.novox.be/novox/mesh-sdk/go v0.1.6/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY= diff --git a/modules/sensors/module.json b/modules/sensors/module.json new file mode 100644 index 00000000..49f141dc --- /dev/null +++ b/modules/sensors/module.json @@ -0,0 +1,90 @@ +{ + "module": "sensors", + "version": "1", + "capabilities": [ + "package-manager", + "service-manager" + ], + "tools": [ + "sensors_summary", + "sensors_temperatures", + "sensors_fans", + "sensors_drives", + "sensors_faults", + "sensors_health" + ], + "resources": [ + { + "id": "smartmontools", + "type": "package", + "package": "smartmontools" + }, + { + "id": "smartd-conf", + "type": "file", + "path": "/etc/smartd.conf", + "mode": "0644", + "content": "# The mesh's (module sensors, novox/hq ADR 0256): what smartd watches. Written whole at every push;\n# an edit here is overwritten.\n#\n# Every drive smartd can find, with every check (-a). A drive in standby is skipped, never woken to be\n# checked, and the skip is not logged (-n standby,q): a server's spinning drives may sleep. No mail is\n# sent (no -m): what smartd finds goes to the journal, and the module's watch reads each drive's health\n# for the mesh's tools and its health check.\nDEVICESCAN -a -n standby,q\n" + }, + { + "id": "smartd-drop-ins", + "type": "directory", + "path": "/etc/systemd/system/smartd.service.d", + "mode": "0755" + }, + { + "id": "smartd-drop-in", + "type": "file", + "path": "/etc/systemd/system/smartd.service.d/10-mesh.conf", + "mode": "0644", + "content": "# The mesh's (module sensors, novox/hq ADR 0256). Written whole at every push; edit nothing here.\n#\n# smartd runs on every machine the module is assigned to, a virtual machine included, where the package's\n# unit would not start; and it stays up on a machine with no drive it can watch (-q never), where it\n# would otherwise exit, because the mesh judges this service as one that stays up.\n[Unit]\nConditionVirtualization=\n\n[Service]\nExecStart=\nExecStart=/usr/bin/smartd -n -q never\n" + }, + { + "id": "smartd", + "type": "service", + "unit": "smartd.service", + "state": "running", + "boot": "enabled", + "restart-on": [ + "smartd-conf", + "smartd-drop-in" + ], + "health": { + "kind": "unit" + } + }, + { + "id": "watch", + "type": "process", + "name": "sensors-watch", + "artifact": "tools", + "run": [ + "./sensors", + "watch" + ], + "health": { + "kind": "tool", + "tool": "sensors_health", + "interval": "60s", + "timeout": "10s", + "looks": 2, + "grace": "60s" + } + } + ], + "build": { + "artifacts": [ + { + "name": "tools", + "kind": "bundle", + "language": "go", + "system": "arch", + "from": "cmd/sensors", + "binary": "sensors", + "loads": [ + "sensors" + ] + } + ] + } +}