mesh-catalog/modules/kernel/README.md

kernel

The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). It holds three things: transparent huge pages, set per machine, the fallback kernel (linux-lts with its headers) and its place on the boot menu (novox/hq ADR 0508 §9 and §10), and microcode in every initramfs (novox/hq issue 1977). The default kernel's package is not built yet.

Why (novox/hq issue 746)

With transparent huge pages always, the page cache of a file mapped by a program is collapsed into 2 MiB folios (khugepaged), and a small write into such a folio dirties all of it: writeback then writes 2 MiB. journald writes its journal files through mmap, and every journal reader (fail2ban's journal jails among them) maps them too. Measured on the control node on 2026-10-14: journald's cgroup had written 2.5 TB in 12 days while the journal grew by about 25-30 KB/s, fail2ban's cgroup 63 GB in 4 h while its own files did not grow, both holding hundreds of MiB of file pages in huge folios; the bursts saturated the mirrored spinning root disk that holds the bus store and the store, and the controller lost its lease four times in 30 minutes. With madvise, only memory a program asks to have in huge pages (madvise(MADV_HUGEPAGE)) gets them; a database or a JVM that wants them still asks.

What it owns

	what	notes
setting	transparent-huge-pages	always, madvise or never. Default madvise, the value the mesh chose for every node (ADR 0381), not what the distribution's kernel boots with (always); a machine that needs another value is given it as its own setting, with its reason
file	${dir:state}/huge-pages	the script, written from the setting: apply writes the mode to /sys/kernel/mm/transparent_hugepage/enabled and reads it back (a value that does not take fails the unit); restore gives back the mode found before the module first set it, kept in ${dir:state}/found
file	/etc/systemd/system/mesh-kernel-huge-pages.service	oneshot, early in boot (sysinit.target, before the services that fill the page cache); reloading it runs apply, stopping it runs restore, which also removes found. Sandboxed: the file system read-only but for the module's state (/sys stays writable), no home, no network, no new privileges; the journal's rotation goes over its local socket, which a private network leaves reachable
service	mesh-kernel-huge-pages.service	running and enabled; reloaded when the script (so the setting) changes, so a new value is set directly and never passes back through the found one; restarted only when the unit changes; its health is the unit's

The per-size controls (hugepages-<n>kB/enabled) are left alone: on the machines read, 2048 kB says inherit (it follows the mode) and every other size never. kernel_huge_pages shows each one.

A folio already in the page cache keeps its size. A new mode applies to pages faulted or collapsed from then on. The active journal files would keep their 2 MiB folios until archived, so apply rotates the journal (journalctl --rotate) whenever it moves the mode away from always: new files are mapped under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. No rotation is made while the machine is still starting (systemctl is-system-running says initializing or starting): nothing is in huge pages yet, and the persistent journal opens after this unit.

The fallback kernel (novox/hq ADR 0508, issue 1841)

Every machine keeps a second, older kernel to start from, so a kernel upgrade that does not boot is not the end of the machine. The module declares the packages linux-lts and linux-lts-headers (so every DKMS module is built for the fallback kernel too) after its boot-entries service, so the entry is in place before the kernel is.

	what	notes
file	${dir:state}/boot-entries	the script. It reads from the machine which loader booted it: the EFI variable LoaderEntrySelected present means systemd-boot, a /boot/grub/grub.cfg without it means GRUB, neither fails the unit loudly
	systemd-boot	writes /boot/loader/entries/mesh-linux-lts.conf from /proc/cmdline without initrd= and BOOT_IMAGE=, with the microcode images present in /boot and the fallback initramfs as initrd lines; loader.conf and the default entry are never touched
	GRUB	runs grub-mkconfig -o /boot/grub/grub.cfg only when the set of /boot/vmlinuz-* differs from the kernels the menu names, with GRUB_TOP_LEVEL naming the kernel the machine booted (BOOT_IMAGE; else the non-fallback kernel), since GRUB lists kernels in reverse version order and would otherwise make the fallback kernel the default; /etc/default/grub still overrides it when it sets one
	room	while /boot/vmlinuz-linux-lts is not there yet, less than 96 MiB free on /boot fails the unit and says so (the fallback kernel writes about 72 MB, issue 1836)
	a kernel-copying hook	a hook in /etc/pacman.d/hooks that the mesh did not write, triggered by kernel images and copying to one fixed file name in /boot, would put the fallback kernel in the default kernel's place: it is moved into ${dir:state}/taken-over/ and named in the unit's journal. Each kernel is then placed under its own package's name by the distribution's hooks
file	/etc/systemd/system/mesh-kernel-boot-entries.service	oneshot, sandboxed: the file system read-only but for /boot, /etc/pacman.d/hooks and the module's state; no home, no network
file	/etc/pacman.d/hooks/90-mesh-kernel-boot-entries.hook	runs the script after a kernel image is installed, upgraded or removed
packages	linux-lts, linux-lts-headers	declared after the service and the hook

Microcode in every initramfs (novox/hq issue 1977)

Before this, amd-ucode was installed by hand on two machines and no initramfs carried it, so a machine booted from its default entry ran the CPU's older microcode (ace: revision 0x0a201016 while the package held 0x0a20102e). The module declares the packages and makes mkinitcpio's own microcode hook part of every image, so every boot entry loads it, a default entry written by hand included, and no entry is edited. The fallback entry's own microcode lines stay (a second early archive is harmless).

	what	notes
packages	amd-ucode, intel-ucode	both on every machine; the autodetect hook keeps only the running CPU's microcode in the default image
file	/etc/mkinitcpio.conf.d/mesh-microcode.conf	read by mkinitcpio after mkinitcpio.conf: appends microcode to HOOKS unless it names it
file	${dir:state}/initramfs	the script, apply: when the drop-in differs from the one the images were last built with (${dir:state}/initramfs-built) it runs mkinitcpio -P and then records the drop-in; otherwise it says "as it was". A failed mkinitcpio fails the unit, says so and records nothing
file	/etc/systemd/system/mesh-kernel-initramfs.service	oneshot, sandboxed: the file system read-only but for /boot and the module's state, a private /tmp, no home, no network
service	mesh-kernel-initramfs.service	running and enabled; restarted when the unit, the script or the drop-in changes. Later kernel upgrades build their images through the distribution's own hook, with the drop-in in place

The module does not reboot: a machine loads the new microcode at its next boot. kernel_boot says the revision the CPU runs now and whether the images were built with the drop-in.

Kernel modules at boot, and the running kernel's modules kept

A machine that runs for weeks meets a kernel upgrade that removes /usr/lib/modules/<the running kernel>: from then on no module that was not loaded yet can be loaded until the next boot. On 2026-10-28 a build node, up 24 days on linux-lts 6.18.54 with only 6.18.58's modules on disk, had /dev/net/tun and no tun driver, and every image build there failed in buildah's network setup ("Failed to open() /dev/net/tun: No such device").

	what	notes
file	/etc/modules-load.d/mesh-kernel.conf	the kernel modules loaded at every boot by systemd-modules-load: tun (pasta, which buildah uses for a RUN line's network, and every user-space tunnel open /dev/net/tun). Read at boot only: nothing is loaded by an apply
package	kernel-modules-hook	keeps the running kernel's modules across an upgrade (saved before the transaction, put back after), so a module first needed later still loads
service	linux-modules-cleanup.service	the package's oneshot, enabled at boot and stopped otherwise: it moves the modules of a kernel that is neither running nor owned by a package to /usr/lib/modules/.old, which the package's tmpfiles rule empties after four weeks. An apply never starts it

A machine whose running kernel's modules are already gone loads nothing new until it is started again; the image build module says so in its health and takes no image build there meanwhile.

Tools

tool		answers
kernel_boot	r	the loader (systemd-boot, grub or none), the menu's entries, the kernels in /boot, the entry booted (systemd-boot) or the kernel image booted (GRUB), whether the fallback kernel is on the menu with its kernel in /boot, the room free on /boot in KiB, the microcode revision the CPU runs (/proc/cpuinfo) and whether the initramfs images were built with the microcode drop-in (read from /var/lib/mesh-kernel/initramfs-built)
kernel_huge_pages	r	the mode, defrag and shared-memory mode; each per-size control and what inherit resolves to; huge pages by kind from /proc/meminfo (MiB); the units holding the most file pages in huge folios (memory.stat file_thp) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (io.stat, the most any one device saw: a mirror and its members each count the same write)

Tests

go test ./...: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the manifest's shape (the default, the order, the unit and its sandbox, reload, restart and health); the kernel modules loaded at boot (tun) and the package and boot service that keep the running kernel's modules; the boot-entries script run with sh over stand-in roots (a systemd-boot entries directory, a GRUB menu, a hook that copies kernels to one fixed name): the entry from the booted command line, a menu generated again only when the kernels changed, no loader, no room, the taken-over hook; kernel_boot over the same roots; and the microcode drop-in sourced by bash, the initramfs script run with sh over a stand-in mkinitcpio (once, again on a changed drop-in, a failure recording nothing) and kernel_boot's microcode answer; and the huge-pages script as the mesh writes it, run with sh against a stand-in directory: it sets and reads back, keeps what it found once, rotates the journal once and not for always nor while the machine starts, goes straight to a new value on a reload without rewriting what it found, gives back what it found and then keeps nothing, refuses a value the kernel does not take before writing anything, fails when the write does not take, and says when the kernel has no transparent huge pages.

Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

• ${dir:state}: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).