mesh-catalog/modules/kernel/README.md

# kernel

The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). It holds two things:
**transparent huge pages**, set per machine, and **the fallback kernel** (`linux-lts` with its headers) and
its place on the boot menu (novox/hq ADR 0508 §9 and §10). The default kernel's package, and microcode, are
not built yet.

## Why (novox/hq issue 746)

With transparent huge pages `always`, the page cache of a file mapped by a program is collapsed into
2 MiB folios (khugepaged), and a small write into such a folio dirties all of it: writeback then writes
2 MiB. journald writes its journal files through mmap, and every journal reader (fail2ban's journal
jails among them) maps them too. Measured on the control node on 2026-10-14: journald's cgroup had
written 2.5 TB in 12 days while the journal grew by about 25-30 KB/s, fail2ban's cgroup 63 GB in 4 h
while its own files did not grow, both holding hundreds of MiB of file pages in huge folios; the bursts
saturated the mirrored spinning root disk that holds the bus store and the store, and the controller
lost its lease four times in 30 minutes. With `madvise`, only memory a program asks to have in huge pages
(`madvise(MADV_HUGEPAGE)`) gets them; a database or a JVM that wants them still asks.

## What it owns

| | what | notes |
|---|---|---|
| setting | `transparent-huge-pages` | `always`, `madvise` or `never`. Default `madvise`, the value the mesh chose for every node (ADR 0381), not what the distribution's kernel boots with (`always`); a machine that needs another value is given it as its own setting, with its reason |
| file | `${dir:state}/huge-pages` | the script, written from the setting: `apply` writes the mode to `/sys/kernel/mm/transparent_hugepage/enabled` and reads it back (a value that does not take fails the unit); `restore` gives back the mode found before the module first set it, kept in `${dir:state}/found` |
| file | `/etc/systemd/system/mesh-kernel-huge-pages.service` | oneshot, early in boot (`sysinit.target`, before the services that fill the page cache); reloading it runs `apply`, stopping it runs `restore`, which also removes `found`. Sandboxed: the file system read-only but for the module's state (`/sys` stays writable), no home, no network, no new privileges; the journal's rotation goes over its local socket, which a private network leaves reachable |
| service | `mesh-kernel-huge-pages.service` | running and enabled; **reloaded** when the script (so the setting) changes, so a new value is set directly and never passes back through the found one; restarted only when the unit changes; its health is the unit's |

The per-size controls (`hugepages-<n>kB/enabled`) are left alone: on the machines read, 2048 kB says
`inherit` (it follows the mode) and every other size `never`. `kernel_huge_pages` shows each one.

**A folio already in the page cache keeps its size.** A new mode applies to pages faulted or collapsed
from then on. The active journal files would keep their 2 MiB folios until archived, so `apply` rotates
the journal (`journalctl --rotate`) whenever it moves the mode away from `always`: new files are mapped
under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. No
rotation is made while the machine is still starting (`systemctl is-system-running` says `initializing`
or `starting`): nothing is in huge pages yet, and the persistent journal opens after this unit.

## The fallback kernel (novox/hq ADR 0508, issue 1841)

Every machine keeps a second, older kernel to start from, so a kernel upgrade that does not boot is not the
end of the machine. The module declares the packages `linux-lts` and `linux-lts-headers` (so every DKMS module is
built for the fallback kernel too) after its boot-entries service, so the entry is in place before the kernel is.

| | what | notes |
|---|---|---|
| file | `${dir:state}/boot-entries` | the script. It reads from the machine which loader booted it: the EFI variable `LoaderEntrySelected` present means systemd-boot, a `/boot/grub/grub.cfg` without it means GRUB, neither fails the unit loudly |
| | systemd-boot | writes `/boot/loader/entries/mesh-linux-lts.conf` from `/proc/cmdline` without `initrd=` and `BOOT_IMAGE=`, with the microcode images present in `/boot` and the fallback initramfs as initrd lines; `loader.conf` and the default entry are never touched |
| | GRUB | runs `grub-mkconfig -o /boot/grub/grub.cfg` only when the set of `/boot/vmlinuz-*` differs from the kernels the menu names, with `GRUB_TOP_LEVEL` naming the kernel the machine booted (`BOOT_IMAGE`; else the non-fallback kernel), since GRUB lists kernels in reverse version order and would otherwise make the fallback kernel the default; `/etc/default/grub` still overrides it when it sets one |
| | room | while `/boot/vmlinuz-linux-lts` is not there yet, less than 96 MiB free on `/boot` fails the unit and says so (the fallback kernel writes about 72 MB, issue 1836) |
| | a kernel-copying hook | a hook in `/etc/pacman.d/hooks` that the mesh did not write, triggered by kernel images and copying to one fixed file name in `/boot`, would put the fallback kernel in the default kernel's place: it is moved into `${dir:state}/taken-over/` and named in the unit's journal. Each kernel is then placed under its own package's name by the distribution's hooks |
| file | `/etc/systemd/system/mesh-kernel-boot-entries.service` | oneshot, sandboxed: the file system read-only but for `/boot`, `/etc/pacman.d/hooks` and the module's state; no home, no network |
| file | `/etc/pacman.d/hooks/90-mesh-kernel-boot-entries.hook` | runs the script after a kernel image is installed, upgraded or removed |
| packages | `linux-lts`, `linux-lts-headers` | declared after the service and the hook |

## Kernel modules at boot, and the running kernel's modules kept

A machine that runs for weeks meets a kernel upgrade that removes `/usr/lib/modules/<the running kernel>`: from then on
no module that was not loaded yet can be loaded until the next boot. On 2026-10-28 a build node, up 24 days on
`linux-lts` 6.18.54 with only 6.18.58's modules on disk, had `/dev/net/tun` and no tun driver, and every image build
there failed in buildah's network setup ("Failed to open() /dev/net/tun: No such device").

| | what | notes |
|---|---|---|
| file | `/etc/modules-load.d/mesh-kernel.conf` | the kernel modules loaded at every boot by systemd-modules-load: `tun` (pasta, which buildah uses for a RUN line's network, and every user-space tunnel open `/dev/net/tun`). Read at boot only: nothing is loaded by an apply |
| package | `kernel-modules-hook` | keeps the running kernel's modules across an upgrade (saved before the transaction, put back after), so a module first needed later still loads |
| service | `linux-modules-cleanup.service` | the package's oneshot, enabled at boot and stopped otherwise: it moves the modules of a kernel that is neither running nor owned by a package to `/usr/lib/modules/.old`, which the package's tmpfiles rule empties after four weeks. An apply never starts it |

A machine whose running kernel's modules are already gone loads nothing new until it is started again; the image build
module says so in its health and takes no image build there meanwhile.

## Tools

| tool | | answers |
|---|---|---|
| `kernel_boot` | r | the loader (`systemd-boot`, `grub` or `none`), the menu's entries, the kernels in `/boot`, the entry booted (systemd-boot) or the kernel image booted (GRUB), whether the fallback kernel is on the menu with its kernel in `/boot`, and the room free on `/boot` in KiB |
| `kernel_huge_pages` | r | the mode, defrag and shared-memory mode; each per-size control and what `inherit` resolves to; huge pages by kind from `/proc/meminfo` (MiB); the units holding the most file pages in huge folios (`memory.stat` `file_thp`) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (`io.stat`, the most any one device saw: a mirror and its members each count the same write) |

## Tests

`go test ./...`: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the
manifest's shape (the default, the order, the unit and its sandbox, reload, restart and health); the kernel modules loaded at boot (`tun`) and the package and boot service that keep the running kernel's modules; the boot-entries script run with `sh` over stand-in roots (a systemd-boot entries directory, a GRUB menu, a hook that copies kernels to one fixed name): the entry from the booted command line, a menu generated again only when the kernels changed, no loader, no room, the taken-over hook; `kernel_boot` over the same roots; and the huge-pages script as the mesh
writes it, run with `sh` against a stand-in directory: it sets and reads back, keeps what it found once,
rotates the journal once and not for `always` nor while the machine starts, goes straight to a new value on a
reload without rewriting what it found, gives back what it found and then keeps nothing, refuses a value the kernel
does not take before writing anything, fails when the write does not take, and says when the kernel has
no transparent huge pages.

## Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

- `${dir:state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).