jschoubben
c816cafc00
4.4 KiB
kernel
The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). Today it holds one: transparent huge pages, set per machine. The rest of the row (the kernel package, microcode, boot entries) is not built yet.
Why (novox/hq issue 746)
With transparent huge pages always, the page cache of a file mapped by a program is collapsed into
2 MiB folios (khugepaged), and a small write into such a folio dirties all of it: writeback then writes
2 MiB. journald writes its journal files through mmap, and every journal reader (fail2ban's journal
jails among them) maps them too. Measured on the control node on 2026-10-14: journald's cgroup had
written 2.5 TB in 12 days while the journal grew by about 25-30 KB/s, fail2ban's cgroup 63 GB in 4 h
while its own files did not grow, both holding hundreds of MiB of file pages in huge folios; the bursts
saturated the mirrored spinning root disk that holds the bus store and the store, and the controller
lost its lease four times in 30 minutes. With madvise, only memory a program asks to have in huge pages
(madvise(MADV_HUGEPAGE)) gets them; a database or a JVM that wants them still asks.
What it owns
| what | notes | |
|---|---|---|
| setting | transparent-huge-pages |
always, madvise or never. Default madvise, the value the mesh chose for every node (ADR 0381), not what the distribution's kernel boots with (always); a machine that needs another value is given it as its own setting, with its reason |
| file | ${dir:state}/huge-pages |
the script, written from the setting: apply writes the mode to /sys/kernel/mm/transparent_hugepage/enabled and reads it back (a value that does not take fails the unit); restore gives back the mode found before the module first set it, kept in ${dir:state}/found |
| file | /etc/systemd/system/mesh-kernel-huge-pages.service |
oneshot, early in boot (sysinit.target, before the services that fill the page cache); reloading it runs apply, stopping it runs restore, which also removes found. Sandboxed: the file system read-only but for the module's state (/sys stays writable), no home, no network, no new privileges; the journal's rotation goes over its local socket, which a private network leaves reachable |
| service | mesh-kernel-huge-pages.service |
running and enabled; reloaded when the script (so the setting) changes, so a new value is set directly and never passes back through the found one; restarted only when the unit changes; its health is the unit's |
The per-size controls (hugepages-<n>kB/enabled) are left alone: on the machines read, 2048 kB says
inherit (it follows the mode) and every other size never. kernel_huge_pages shows each one.
A folio already in the page cache keeps its size. A new mode applies to pages faulted or collapsed
from then on. The active journal files would keep their 2 MiB folios until archived, so apply rotates
the journal (journalctl --rotate) whenever it moves the mode away from always: new files are mapped
under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. No
rotation is made while the machine is still starting (systemctl is-system-running says initializing
or starting): nothing is in huge pages yet, and the persistent journal opens after this unit.
Tools
| tool | answers | |
|---|---|---|
kernel_huge_pages |
r | the mode, defrag and shared-memory mode; each per-size control and what inherit resolves to; huge pages by kind from /proc/meminfo (MiB); the units holding the most file pages in huge folios (memory.stat file_thp) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (io.stat, the most any one device saw: a mirror and its members each count the same write) |
Tests
go test ./...: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the
manifest's shape (the default, the order, the unit and its sandbox, reload, restart and health); and the script as the mesh
writes it, run with sh against a stand-in directory: it sets and reads back, keeps what it found once,
rotates the journal once and not for always nor while the machine starts, goes straight to a new value on a
reload without rewriting what it found, gives back what it found and then keeps nothing, refuses a value the kernel
does not take before writing anything, fails when the write does not take, and says when the kernel has
no transparent huge pages.