jschoubben
3a84044ece
With THP 'always' journald's mmapped journal files sit in 2 MiB folios, so each small journal write dirties and writes back 2 MiB; on the control node that saturated the root mirror holding the bus store and the store, and the controller lost its lease. The module sets the mode from a per-machine setting (default always, the kernel's own, so assigning it changes nothing until a machine sets madvise), verifies it took, rotates the journal once so new files are mapped under the new mode, gives back what it found when stopped, and serves kernel_huge_pages to measure file_thp per unit.
51 lines
3.7 KiB
Markdown
51 lines
3.7 KiB
Markdown
# kernel
|
|
|
|
The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). Today it holds one:
|
|
**transparent huge pages**, set per machine. The rest of the row (the kernel package, microcode, boot
|
|
entries) is not built yet.
|
|
|
|
## Why (novox/hq issue 746)
|
|
|
|
With transparent huge pages `always`, the page cache of a file mapped by a program is collapsed into
|
|
2 MiB folios (khugepaged), and a small write into such a folio dirties all of it: writeback then writes
|
|
2 MiB. journald writes its journal files through mmap, and every journal reader (fail2ban's journal
|
|
jails among them) maps them too. Measured on the control node on 2026-10-14: journald's cgroup had
|
|
written 2.5 TB in 12 days while the journal grew by about 25-30 KB/s, fail2ban's cgroup 63 GB in 4 h
|
|
while its own files did not grow, both holding hundreds of MiB of file pages in huge folios; the bursts
|
|
saturated the mirrored spinning root disk that holds the bus store and the store, and the controller
|
|
lost its lease four times in 30 minutes. With `madvise`, only memory a program asks to have in huge pages
|
|
(`madvise(MADV_HUGEPAGE)`) gets them; a database or a JVM that wants them still asks.
|
|
|
|
## What it owns
|
|
|
|
| | what | notes |
|
|
|---|---|---|
|
|
| setting | `transparent-huge-pages` | `always`, `madvise` or `never`. Default `always`, what the distribution's kernel boots with, so assigning the module changes nothing until a machine sets another value |
|
|
| file | `${dir:state}/huge-pages` | the script, written from the setting: `apply` writes the mode to `/sys/kernel/mm/transparent_hugepage/enabled` and reads it back (a value that does not take fails the unit); `restore` gives back the mode found before the module first set it, kept in `${dir:state}/found` |
|
|
| file | `/etc/systemd/system/mesh-kernel-huge-pages.service` | oneshot, early in boot (`sysinit.target`, before the services that fill the page cache); stopping it runs `restore` |
|
|
| service | `mesh-kernel-huge-pages.service` | running and enabled, restarted when the script (so the setting) or the unit changes; its health is the unit's |
|
|
|
|
The per-size controls (`hugepages-<n>kB/enabled`) are left alone: on the machines read, 2048 kB says
|
|
`inherit` (it follows the mode) and every other size `never`. `kernel_huge_pages` shows each one.
|
|
|
|
**A folio already in the page cache keeps its size.** A new mode applies to pages faulted or collapsed
|
|
from then on. The active journal files would keep their 2 MiB folios until archived, so `apply` rotates
|
|
the journal (`journalctl --rotate`) whenever it moves the mode away from `always`: new files are mapped
|
|
under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. At
|
|
boot the kernel starts at `always`, so a machine set to `madvise` rotates its journal once per boot.
|
|
|
|
## Tools
|
|
|
|
| tool | | answers |
|
|
|---|---|---|
|
|
| `kernel_huge_pages` | r | the mode, defrag and shared-memory mode; each per-size control and what `inherit` resolves to; huge pages by kind from `/proc/meminfo` (MiB); the units holding the most file pages in huge folios (`memory.stat` `file_thp`) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (`io.stat`, the most any one device saw: a mirror and its members each count the same write) |
|
|
|
|
## Tests
|
|
|
|
`go test ./...`: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the
|
|
manifest's shape (the default, the order, the unit, restart and health); and the script as the mesh
|
|
writes it, run with `sh` against a stand-in directory: it sets and reads back, keeps what it found once,
|
|
rotates the journal once and not for `always`, gives back what it found, refuses a value the kernel
|
|
does not take before writing anything, fails when the write does not take, and says when the kernel has
|
|
no transparent huge pages.
|