Add disk-load: a machine's iowait, disk load and who does the I/O (hq issue 312) #120

2026-10-08T08:47:58Z

mesh-admin commented

2026-10-08 08:47:58 +00:00

No tool in the mesh said a machine's CPU iowait, a disk's utilisation, throughput, latency or queue depth, or which unit, container or process did the I/O. Investigating hq issue 306 (checks 5-7x slower on the control node) needed exactly those numbers. This adds them as a small read-only module, disk-load, registered as hq issue 312.

Why a module and not a seat verb. No node seat owns the machine's storage. memory-pressure is the nearest relative, but it owns zram and systemd-oomd, so assigning it to read disks would change how a machine handles memory. A seat is a role that one module holds per machine, and a new one needs ADR 0246's three steps. A report built from kernel counters needs neither. It sits beside netcheck, which reads the network the same way. It declares no resources and can be assigned to every machine.

Tools (all read-only, (r)):

disk_load {seconds=2 (1-10), processes=5 (0-50), all=false}: CPU iowait/user/system/idle %; I/O PSI (some/full over the window and avg10/60/300); per disk busy %, read and write MiB/s, IOPS, wait in ms, queue depth and in-flight requests, with the filesystems each holds; filesystems' space; busiest units (from cgroup io.stat) and processes (from /proc//io). said gives one readable line per finding, flagged HIGH / SATURATED / NEARLY FULL past fixed lines.
disk_top {seconds, limit, by=unit|container|process}: ranks by bytes moved. Units and containers come from cgroup io.stat, which any account can read, so this is complete even when the tool runner is not root. Only the innermost unit and only the bottom disk are counted, so no byte is counted twice.
disk_filesystems: df -h and df -i, one line per filesystem however many mounts it has.
disk_devices: model, size, rotational, scheduler, and GiB read and written plus wait since boot.

Tests: fixture /proc and /sys trees: an encrypted root on NVMe under a 100 MiB/2 s write, an idle spinning disk, loop and zram, nested user cgroups and an unreadable process. Also checks that the manifest's tools equal the tools served. Run once on the laptop by hand: it named the build agent's container writing 83 MiB/s, which /proc//io could not see as non-root.

Checks run locally: merge-check.sh with MESH_GATE set to a controller built from main: words clean, module check passes, health-undeclared count 70 as kept, vet and race tests pass.

After merge: assign disk-load to the machines through the controller (mesh-controller.assign); the control node first, for issue 306.

No tool in the mesh said a machine's CPU iowait, a disk's utilisation, throughput, latency or queue depth, or which unit, container or process did the I/O. Investigating hq issue 306 (checks 5-7x slower on the control node) needed exactly those numbers. This adds them as a small read-only module, `disk-load`, registered as hq issue 312. **Why a module and not a seat verb.** No node seat owns the machine's storage. `memory-pressure` is the nearest relative, but it owns zram and systemd-oomd, so assigning it to read disks would change how a machine handles memory. A seat is a role that one module holds per machine, and a new one needs ADR 0246's three steps. A report built from kernel counters needs neither. It sits beside `netcheck`, which reads the network the same way. It declares no resources and can be assigned to every machine. **Tools** (all read-only, `(r)`): - `disk_load {seconds=2 (1-10), processes=5 (0-50), all=false}`: CPU iowait/user/system/idle %; I/O PSI (some/full over the window and avg10/60/300); per disk busy %, read and write MiB/s, IOPS, wait in ms, queue depth and in-flight requests, with the filesystems each holds; filesystems' space; busiest units (from cgroup io.stat) and processes (from /proc/<pid>/io). `said` gives one readable line per finding, flagged HIGH / SATURATED / NEARLY FULL past fixed lines. - `disk_top {seconds, limit, by=unit|container|process}`: ranks by bytes moved. Units and containers come from cgroup io.stat, which any account can read, so this is complete even when the tool runner is not root. Only the innermost unit and only the bottom disk are counted, so no byte is counted twice. - `disk_filesystems`: df -h and df -i, one line per filesystem however many mounts it has. - `disk_devices`: model, size, rotational, scheduler, and GiB read and written plus wait since boot. **Tests:** fixture /proc and /sys trees: an encrypted root on NVMe under a 100 MiB/2 s write, an idle spinning disk, loop and zram, nested user cgroups and an unreadable process. Also checks that the manifest's tools equal the tools served. Run once on the laptop by hand: it named the build agent's container writing 83 MiB/s, which /proc/<pid>/io could not see as non-root. **Checks run locally:** `merge-check.sh` with `MESH_GATE` set to a controller built from main: words clean, `module check` passes, health-undeclared count 70 as kept, vet and race tests pass. **After merge:** assign `disk-load` to the machines through the controller (`mesh-controller.assign`); the control node first, for issue 306.

mesh-admin added 1 commit 2026-10-08 08:47:59 +00:00

Add disk-load: no tool said a machine's iowait, disk load or who does the I/O (hq issue 312)

mesh/merge-gate pass: builds new: modules/disk-load, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)

Details

mesh/repo-check pass: its merge-check.sh passed

Details

mesh/delivery delivered

Details

6fdf5293eb

mesh-admin commented

2026-10-08 08:48:44 +00:00

Delivery novox/mesh-catalog@6fdf5293eb4e — delivered since 2026-10-09T07:31:48Z

Delivery plan — builds new: modules/disk-load, sent nowhere; no bus step

Machines

route-proxy on ace: passed — carried by plan-1791449197631987621

Transitions

2026-10-08 08:48 (new) → proposed (announced): novox/mesh-catalog#120's head announced
2026-10-08 08:49 proposed → checked (checked): the verdict names this commit
2026-10-08 08:49 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-08 09:06 ready → held (hold): something holds it for a person: its group's composed check did not pass for the heads that merged, or it merged and no walk was opened for it
2026-10-09 07:31 held → delivered (done): merged, and nothing for a walk to move: builds new: modules/disk-load, sent nowhere; no bus step

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@6fdf5293eb4e` — **delivered** since 2026-10-09T07:31:48Z **Delivery plan** — builds new: modules/disk-load, sent nowhere; no bus step **Machines** - route-proxy on ace: passed — carried by plan-1791449197631987621 **Transitions** - 2026-10-08 08:48 (new) → proposed (announced): novox/mesh-catalog#120's head announced - 2026-10-08 08:49 proposed → checked (checked): the verdict names this commit - 2026-10-08 08:49 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-08 09:06 ready → held (hold): something holds it for a person: its group's composed check did not pass for the heads that merged, or it merged and no walk was opened for it - 2026-10-09 07:31 held → delivered (done): merged, and nothing for a walk to move: builds new: modules/disk-load, sent nowhere; no bus step The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin commented

2026-10-08 08:49:56 +00:00

Merge check at 6fdf5293

mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: new:modules/disk-load)
mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds new: modules/disk-load, sent nowhere; no bus step

Run by the build seat on ace as build-1791449289559819580 (builds --log build-1791449289559819580).

--- the gate: new:modules/disk-load
disk-load: ok, 4 tool(s)
long-running resources without health: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T08:44:32Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-08T08:44:32Z

modules the change moves:
  - disk-load (added)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 0 module(s) in 0 tier(s)
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.002s
words: 12 retired words, none in the 28633 strings of 533 files
long-running resources without health: 70, as kept
testing modules/disk-load
ok  	diskload/cmd/disk-load	1.096s

**Merge check** at `6fdf5293` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: new:modules/disk-load) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds new: modules/disk-load, sent nowhere; no bus step Run by the build seat on ace as `build-1791449289559819580` (`builds --log build-1791449289559819580`). ``` --- the gate: new:modules/disk-load disk-load: ok, 4 tool(s) long-running resources without health: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-08T08:44:32Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-08T08:44:32Z modules the change moves: - disk-load (added) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 0 module(s) in 0 tier(s) --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.002s words: 12 retired words, none in the 28633 strings of 533 files long-running resources without health: 70, as kept testing modules/disk-load ok diskload/cmd/disk-load 1.096s ```

mesh-admin merged commit 8134627e18 into main

2026-10-08 08:55:39 +00:00

mesh-admin deleted branch feat/disk-load

2026-10-08 08:55:39 +00:00

mesh-admin referenced this pull request

2026-10-08 23:33:08 +00:00

systemd-resolved: a machine's own resolver, routing a VPN's domains by link (hq ADR 0247) #113

jschoubben referenced this issue from a commit

2026-10-28 20:56:16 +00:00

Pin mesh-sdk 8eafc0c, the L3 group's head on mesh-sdk main with #120 and #121 (issue 774)

Sign in to join this conversation.