jschoubben
bc77afa6bb
ADR 0508 §9: the operator sees whether every boot menu lists the fallback kernel, and the room left on /boot.
7.4 KiB
kernel
The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). It holds two things:
transparent huge pages, set per machine, and the fallback kernel (linux-lts with its headers) and
its place on the boot menu (novox/hq ADR 0508 §9 and §10). The default kernel's package, and microcode, are
not built yet.
Why (novox/hq issue 746)
With transparent huge pages always, the page cache of a file mapped by a program is collapsed into
2 MiB folios (khugepaged), and a small write into such a folio dirties all of it: writeback then writes
2 MiB. journald writes its journal files through mmap, and every journal reader (fail2ban's journal
jails among them) maps them too. Measured on the control node on 2026-10-14: journald's cgroup had
written 2.5 TB in 12 days while the journal grew by about 25-30 KB/s, fail2ban's cgroup 63 GB in 4 h
while its own files did not grow, both holding hundreds of MiB of file pages in huge folios; the bursts
saturated the mirrored spinning root disk that holds the bus store and the store, and the controller
lost its lease four times in 30 minutes. With madvise, only memory a program asks to have in huge pages
(madvise(MADV_HUGEPAGE)) gets them; a database or a JVM that wants them still asks.
What it owns
| what | notes | |
|---|---|---|
| setting | transparent-huge-pages |
always, madvise or never. Default madvise, the value the mesh chose for every node (ADR 0381), not what the distribution's kernel boots with (always); a machine that needs another value is given it as its own setting, with its reason |
| file | ${dir:state}/huge-pages |
the script, written from the setting: apply writes the mode to /sys/kernel/mm/transparent_hugepage/enabled and reads it back (a value that does not take fails the unit); restore gives back the mode found before the module first set it, kept in ${dir:state}/found |
| file | /etc/systemd/system/mesh-kernel-huge-pages.service |
oneshot, early in boot (sysinit.target, before the services that fill the page cache); reloading it runs apply, stopping it runs restore, which also removes found. Sandboxed: the file system read-only but for the module's state (/sys stays writable), no home, no network, no new privileges; the journal's rotation goes over its local socket, which a private network leaves reachable |
| service | mesh-kernel-huge-pages.service |
running and enabled; reloaded when the script (so the setting) changes, so a new value is set directly and never passes back through the found one; restarted only when the unit changes; its health is the unit's |
The per-size controls (hugepages-<n>kB/enabled) are left alone: on the machines read, 2048 kB says
inherit (it follows the mode) and every other size never. kernel_huge_pages shows each one.
A folio already in the page cache keeps its size. A new mode applies to pages faulted or collapsed
from then on. The active journal files would keep their 2 MiB folios until archived, so apply rotates
the journal (journalctl --rotate) whenever it moves the mode away from always: new files are mapped
under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. No
rotation is made while the machine is still starting (systemctl is-system-running says initializing
or starting): nothing is in huge pages yet, and the persistent journal opens after this unit.
The fallback kernel (novox/hq ADR 0508, issue 1841)
Every machine keeps a second, older kernel to start from, so a kernel upgrade that does not boot is not the
end of the machine. The module declares the packages linux-lts and linux-lts-headers (so every DKMS module is
built for the fallback kernel too) after its boot-entries service, so the entry is in place before the kernel is.
| what | notes | |
|---|---|---|
| file | ${dir:state}/boot-entries |
the script. It reads from the machine which loader booted it: the EFI variable LoaderEntrySelected present means systemd-boot, a /boot/grub/grub.cfg without it means GRUB, neither fails the unit loudly |
| systemd-boot | writes /boot/loader/entries/mesh-linux-lts.conf from /proc/cmdline without initrd= and BOOT_IMAGE=, with the microcode images present in /boot and the fallback initramfs as initrd lines; loader.conf and the default entry are never touched |
|
| GRUB | runs grub-mkconfig -o /boot/grub/grub.cfg only when the set of /boot/vmlinuz-* differs from the kernels the menu names; the default is left as it is |
|
| room | while /boot/vmlinuz-linux-lts is not there yet, less than 96 MiB free on /boot fails the unit and says so (the fallback kernel writes about 72 MB, issue 1836) |
|
| a kernel-copying hook | a hook in /etc/pacman.d/hooks that the mesh did not write, triggered by kernel images and copying to one fixed file name in /boot, would put the fallback kernel in the default kernel's place: it is moved into ${dir:state}/taken-over/ and named in the unit's journal. Each kernel is then placed under its own package's name by the distribution's hooks |
|
| file | /etc/systemd/system/mesh-kernel-boot-entries.service |
oneshot, sandboxed: the file system read-only but for /boot, /etc/pacman.d/hooks and the module's state; no home, no network |
| file | /etc/pacman.d/hooks/90-mesh-kernel-boot-entries.hook |
runs the script after a kernel image is installed, upgraded or removed |
| packages | linux-lts, linux-lts-headers |
declared after the service and the hook |
Tools
| tool | answers | |
|---|---|---|
kernel_boot |
r | the loader (systemd-boot, grub or none), the menu's entries, the kernels in /boot, the entry booted (systemd-boot) or the kernel image booted (GRUB), whether the fallback kernel is on the menu with its kernel in /boot, and the room free on /boot in KiB |
kernel_huge_pages |
r | the mode, defrag and shared-memory mode; each per-size control and what inherit resolves to; huge pages by kind from /proc/meminfo (MiB); the units holding the most file pages in huge folios (memory.stat file_thp) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (io.stat, the most any one device saw: a mirror and its members each count the same write) |
Tests
go test ./...: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the
manifest's shape (the default, the order, the unit and its sandbox, reload, restart and health); the boot-entries script run with sh over stand-in roots (a systemd-boot entries directory, a GRUB menu, a hook that copies kernels to one fixed name): the entry from the booted command line, a menu generated again only when the kernels changed, no loader, no room, the taken-over hook; kernel_boot over the same roots; and the huge-pages script as the mesh
writes it, run with sh against a stand-in directory: it sets and reads back, keeps what it found once,
rotates the journal once and not for always nor while the machine starts, goes straight to a new value on a
reload without rewriting what it found, gives back what it found and then keeps nothing, refuses a value the kernel
does not take before writing anything, fails when the write does not take, and says when the kernel has
no transparent huge pages.
Files it does not know
Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).
${dir:state}: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).