mesh-catalog/modules/disk-load/README.md

# disk-load

How loaded a machine's storage is, read from `/proc` and `/sys` and nothing else: the share of CPU time
spent waiting on I/O, I/O pressure stall, each disk's busy share, throughput, IOPS, wait and queue
depth, the filesystems' space, and the units, containers and processes doing the I/O (novox/hq issue
312). It changes nothing and declares no resources, so it can be assigned to every machine.

## Why it exists, and why here

Investigating why the controller's checks ran five to seven times slower on the control node (hq issue
306) needed disk numbers, and no tool in the mesh reported any: not iowait, not a disk's utilisation,
not throughput, latency or which process wrote. The operator's rule is that a missing tool is built in
the module that owns the area, never worked round with a login to the machine.

No module owned the area. `memory-pressure` reads the same kind of thing for memory, but it owns
compressed swap and systemd-oomd: assigning it to a machine to read the disks would change how that
machine handles memory. No node seat holds the machine's storage either. Every seat is a role one
module holds on each machine, with verbs that act through it, and a new seat costs a controller change
first (hq ADR 0246). A report that reads kernel counters needs neither a role nor exclusivity. So this
is a small module with tools only, like `netcheck` beside the network. If the mesh later gives a
machine's resources a seat, these tools become its verbs.

## Tools

All four only read.

| tool | what | replaces |
|---|---|---|
| `disk_load` | Samples a window (`seconds`, default 2, at most 10). It answers the CPU's iowait, user, system and idle %; I/O pressure (some and full, over the window and as avg10/60/300); per disk busy %, read and write MiB/s, IOPS, wait per request in ms, queue depth and requests in flight, with the filesystems each disk holds; the filesystems' space; and the busiest units and processes (`processes`, default 5, 0 for none). `said` puts each finding in one line, flagged past a line (below) | `iostat -x`, `vmstat`, `iotop`, `df` |
| `disk_top` | Over a window. `by=unit` (the default) ranks systemd units and containers by bytes moved, from their cgroups. `by=container` ranks containers by name, and `by=process` ranks processes | `iotop` |
| `disk_filesystems` | Each filesystem that stores data, said once however many places it is mounted: size, used and free GiB, used % as `df` counts it, inodes used % | `df -h`, `df -i` |
| `disk_devices` | Each disk without sampling: model, size, whether it spins, scheduler, queue limit, what it is built on, and since boot the GiB read and written, the busy share and the wait per read and per write | `lsblk`, `cat /proc/diskstats` |

### How each number is read

- **iowait** is from `/proc/stat`'s `cpu` line: the share of every CPU's time in the window spent idle
  with I/O outstanding. Guest time is left out, because the kernel already counts it in user.
- **A disk** is read from `/proc/diskstats` at both ends of the window, with iostat's arithmetic.
  Busy is the share of the window with any request in flight. Wait is the time spent on the
  requests completed, divided by their number. Queue depth is the weighted time divided by the window.
  Sectors there are always 512 bytes. A counter that went backwards (a device removed and added)
  reads as 0, never as a huge number. Loop devices, RAM disks and zram are left out unless `all`;
  zram is memory, and `memory-pressure` reports it.
- **Pressure** is `/proc/pressure/io`. The share of the window comes from the `total` counter, in
  microseconds. A kernel without PSI is said as not available.
- **Units and containers** come from each cgroup's `io.stat`, which every account can read. Only the
  innermost `.service` or `.scope` counts, because `io.stat` includes descendants. Only whole disks
  built on nothing count: a write through an encrypted or logical volume is charged to the volume and
  again to the disk under it, and a loop device's writes are charged again to the filesystem its file
  sits on. Container names come from `docker ps`; if the runtime cannot be asked, a container is
  given by its short id.
- **Processes** come from `/proc/<pid>/io` (`read_bytes`, `write_bytes`, which is storage, not the
  page cache). Another account's process can be read only when the tool runner is root. The unreadable
  are counted and said, and the units above miss none of them. A process that started inside the
  window is left out.
- **Filesystems** come from the process's own mount table, keeping only types that store data
  (ext4, xfs, btrfs, vfat, NFS and the like). Each one is checked with statfs, given 2 s per mount so
  a dead network mount costs one line.

### The lines

| finding | past |
|---|---|
| iowait HIGH | 20 % of CPU time |
| I/O pressure HIGH | some task stalled 10 % of the window, or avg60 ≥ 10 % |
| disk SATURATED | busy 90 % of the window |
| filesystem NEARLY FULL | 90 % of its space or of its inodes |

Busy % is how iostat counts it, so it overstates an NVMe disk, which serves many requests at once. Read
it beside the wait and the queue depth.

## Tests

`go test ./...`. The tests read a tree standing in for `/proc` and `/sys`. Its fixture is a server
writing 100 MiB in 2 s to an encrypted root on an NVMe disk, with a spinning disk idle beside it. They
cover:

- iowait, each disk's numbers, and the encrypted root named, with what it sits on and what it holds;
- loop, zram and partitions left out unless asked for;
- pressure, and a kernel without it;
- a counter reset, and busy held at 100;
- units counted once, by innermost cgroup and bottom disk;
- containers by name, or by short id if the runtime cannot be asked;
- processes ranked, one that started inside the window left out, unreadable ones counted;
- filesystems said once, mount escapes undone, a statfs that fails said;
- the window's bounds;
- the manifest: tools listed equal tools served, all read-only, no resources, no machine named.