diff --git a/modules/kernel/README.md b/modules/kernel/README.md index 032fd9059..2da9207db 100644 --- a/modules/kernel/README.md +++ b/modules/kernel/README.md @@ -1,9 +1,9 @@ # kernel -The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). It holds two things: -**transparent huge pages**, set per machine, and **the fallback kernel** (`linux-lts` with its headers) and -its place on the boot menu (novox/hq ADR 0508 §9 and §10). The default kernel's package, and microcode, are -not built yet. +The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). It holds three things: +**transparent huge pages**, set per machine, **the fallback kernel** (`linux-lts` with its headers) and +its place on the boot menu (novox/hq ADR 0508 §9 and §10), and **microcode** in every initramfs (novox/hq issue 1977). +The default kernel's package is not built yet. ## Why (novox/hq issue 746) @@ -53,6 +53,25 @@ built for the fallback kernel too) after its boot-entries service, so the entry | file | `/etc/pacman.d/hooks/90-mesh-kernel-boot-entries.hook` | runs the script after a kernel image is installed, upgraded or removed | | packages | `linux-lts`, `linux-lts-headers` | declared after the service and the hook | +## Microcode in every initramfs (novox/hq issue 1977) + +Before this, `amd-ucode` was installed by hand on two machines and no initramfs carried it, so a machine booted from +its default entry ran the CPU's older microcode (ace: revision 0x0a201016 while the package held 0x0a20102e). The module +declares the packages and makes mkinitcpio's own `microcode` hook part of every image, so every boot entry loads it, +a default entry written by hand included, and no entry is edited. The fallback entry's own microcode lines stay (a second +early archive is harmless). + +| | what | notes | +|---|---|---| +| packages | `amd-ucode`, `intel-ucode` | both on every machine; the `autodetect` hook keeps only the running CPU's microcode in the default image | +| file | `/etc/mkinitcpio.conf.d/mesh-microcode.conf` | read by mkinitcpio after `mkinitcpio.conf`: appends `microcode` to `HOOKS` unless it names it | +| file | `${dir:state}/initramfs` | the script, `apply`: when the drop-in differs from the one the images were last built with (`${dir:state}/initramfs-built`) it runs `mkinitcpio -P` and then records the drop-in; otherwise it says "as it was". A failed mkinitcpio fails the unit, says so and records nothing | +| file | `/etc/systemd/system/mesh-kernel-initramfs.service` | oneshot, sandboxed: the file system read-only but for `/boot` and the module's state, a private `/tmp`, no home, no network | +| service | `mesh-kernel-initramfs.service` | running and enabled; restarted when the unit, the script or the drop-in changes. Later kernel upgrades build their images through the distribution's own hook, with the drop-in in place | + +The module does not reboot: a machine loads the new microcode at its next boot. `kernel_boot` says the revision the CPU +runs now and whether the images were built with the drop-in. + ## Kernel modules at boot, and the running kernel's modules kept A machine that runs for weeks meets a kernel upgrade that removes `/usr/lib/modules/`: from then on @@ -73,13 +92,13 @@ module says so in its health and takes no image build there meanwhile. | tool | | answers | |---|---|---| -| `kernel_boot` | r | the loader (`systemd-boot`, `grub` or `none`), the menu's entries, the kernels in `/boot`, the entry booted (systemd-boot) or the kernel image booted (GRUB), whether the fallback kernel is on the menu with its kernel in `/boot`, and the room free on `/boot` in KiB | +| `kernel_boot` | r | the loader (`systemd-boot`, `grub` or `none`), the menu's entries, the kernels in `/boot`, the entry booted (systemd-boot) or the kernel image booted (GRUB), whether the fallback kernel is on the menu with its kernel in `/boot`, the room free on `/boot` in KiB, the microcode revision the CPU runs (`/proc/cpuinfo`) and whether the initramfs images were built with the microcode drop-in (read from `/var/lib/mesh-kernel/initramfs-built`) | | `kernel_huge_pages` | r | the mode, defrag and shared-memory mode; each per-size control and what `inherit` resolves to; huge pages by kind from `/proc/meminfo` (MiB); the units holding the most file pages in huge folios (`memory.stat` `file_thp`) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (`io.stat`, the most any one device saw: a mirror and its members each count the same write) | ## Tests `go test ./...`: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the -manifest's shape (the default, the order, the unit and its sandbox, reload, restart and health); the kernel modules loaded at boot (`tun`) and the package and boot service that keep the running kernel's modules; the boot-entries script run with `sh` over stand-in roots (a systemd-boot entries directory, a GRUB menu, a hook that copies kernels to one fixed name): the entry from the booted command line, a menu generated again only when the kernels changed, no loader, no room, the taken-over hook; `kernel_boot` over the same roots; and the huge-pages script as the mesh +manifest's shape (the default, the order, the unit and its sandbox, reload, restart and health); the kernel modules loaded at boot (`tun`) and the package and boot service that keep the running kernel's modules; the boot-entries script run with `sh` over stand-in roots (a systemd-boot entries directory, a GRUB menu, a hook that copies kernels to one fixed name): the entry from the booted command line, a menu generated again only when the kernels changed, no loader, no room, the taken-over hook; `kernel_boot` over the same roots; and the microcode drop-in sourced by bash, the initramfs script run with `sh` over a stand-in mkinitcpio (once, again on a changed drop-in, a failure recording nothing) and `kernel_boot`'s microcode answer; and the huge-pages script as the mesh writes it, run with `sh` against a stand-in directory: it sets and reads back, keeps what it found once, rotates the journal once and not for `always` nor while the machine starts, goes straight to a new value on a reload without rewriting what it found, gives back what it found and then keeps nothing, refuses a value the kernel diff --git a/modules/kernel/cmd/kernel-tools/boot.go b/modules/kernel/cmd/kernel-tools/boot.go index 10a895a17..8ebfd49e0 100644 --- a/modules/kernel/cmd/kernel-tools/boot.go +++ b/modules/kernel/cmd/kernel-tools/boot.go @@ -23,6 +23,30 @@ type BootState struct { Booted string `json:"booted"` FallbackPresent bool `json:"fallback_present"` BootFreeKiB int64 `json:"boot_free_kib"` + // MicrocodeRevision is the revision the CPU runs, the first "microcode" line of /proc/cpuinfo ("" when it has none). + MicrocodeRevision string `json:"microcode_revision"` + // ImagesBuiltWithMicrocode is true when the initramfs images were last built with the microcode drop-in as it is now + // (novox/hq issue 1977). + ImagesBuiltWithMicrocode bool `json:"images_built_with_microcode"` +} + +// stateDir is the module's own directory on the machine, where the initramfs script records the drop-in it built with. +const stateDir = "var/lib/mesh-kernel" + +const microcodeDropIn = "etc/mkinitcpio.conf.d/mesh-microcode.conf" + +var cpuMicrocode = regexp.MustCompile(`(?m)^microcode\s*:\s*(\S+)`) + +// microcode reads the revision the CPU runs and whether the images carry the drop-in as it is now. +func (m *Machine) microcode(b *BootState) { + if raw, err := m.read("proc/cpuinfo"); err == nil { + if r := cpuMicrocode.FindStringSubmatch(raw); r != nil { + b.MicrocodeRevision = r[1] + } + } + built, err := m.read(stateDir + "/initramfs-built") + now, err2 := m.read(microcodeDropIn) + b.ImagesBuiltWithMicrocode = err == nil && err2 == nil && built == now } var grubKernel = regexp.MustCompile(`(?m)^\s*linux\s+(?:/boot)?/(vmlinuz-\S+)`) @@ -33,6 +57,7 @@ var bootImage = regexp.MustCompile(`BOOT_IMAGE=(?:/boot)?/?(\S+)`) // systemd-boot, a grub.cfg without it GRUB. func (m *Machine) Boot() (*BootState, error) { b := &BootState{Loader: "none", Entries: []string{}, Kernels: []string{}} + m.microcode(b) files, _ := filepath.Glob(m.path("boot/vmlinuz-*")) for _, f := range files { b.Kernels = append(b.Kernels, filepath.Base(f)) diff --git a/modules/kernel/cmd/kernel-tools/main.go b/modules/kernel/cmd/kernel-tools/main.go index 93cdece80..e93c4efb5 100644 --- a/modules/kernel/cmd/kernel-tools/main.go +++ b/modules/kernel/cmd/kernel-tools/main.go @@ -51,7 +51,8 @@ func tools(m *Machine) []stdio.Tool { Description: "How this machine boots and whether the fallback kernel is on its menu (novox/hq ADR 0508 §9): the loader that booted it " + "(systemd-boot, read from the EFI variable it sets, grub, or none), the menu's entries, the kernels in /boot, the entry " + "the machine booted (systemd-boot) or the kernel image it booted (grub), whether the fallback kernel linux-lts is on " + - "the menu with its kernel in /boot, and the room free on /boot in KiB (the fallback kernel needs about 72 MB). " + + "the menu with its kernel in /boot, and the room free on /boot in KiB (the fallback kernel needs about 72 MB), the microcode revision the CPU runs " + + "(/proc/cpuinfo) and whether the initramfs images were built with the microcode drop-in (novox/hq issue 1977). " + "Replaces ls /boot, bootctl list, reading grub.cfg and df /boot. (r)", Input: schema(map[string]any{}), Run: func(args map[string]any) (any, error) { return m.Boot() }, diff --git a/modules/kernel/cmd/kernel-tools/manifest_test.go b/modules/kernel/cmd/kernel-tools/manifest_test.go index 21195061f..e7c548f2b 100644 --- a/modules/kernel/cmd/kernel-tools/manifest_test.go +++ b/modules/kernel/cmd/kernel-tools/manifest_test.go @@ -24,7 +24,7 @@ func TestTheDefaultIsMadviseAndTheUnitFollowsTheSetting(t *testing.T) { for _, r := range m.Resources { ids = append(ids, r["id"].(string)) } - if strings.Join(ids, " ") != "state huge-pages unit apply boot-entries boot-unit boot-hook boot-apply linux-lts linux-lts-headers modules-load modules-keep modules-cleanup" { + if strings.Join(ids, " ") != "state huge-pages unit apply boot-entries boot-unit boot-hook boot-apply linux-lts linux-lts-headers modules-load modules-keep modules-cleanup amd-ucode intel-ucode microcode-hook initramfs initramfs-unit initramfs-apply" { t.Fatalf("order: %v", ids) } script := m.resource(t, "huge-pages") diff --git a/modules/kernel/cmd/kernel-tools/microcode_test.go b/modules/kernel/cmd/kernel-tools/microcode_test.go new file mode 100644 index 000000000..144e1d31c --- /dev/null +++ b/modules/kernel/cmd/kernel-tools/microcode_test.go @@ -0,0 +1,137 @@ +package main + +// Microcode in every initramfs (novox/hq issue 1977, lead answers L1-L4): the manifest, the drop-in sourced by bash, +// the initramfs script run with sh over stand-in files and a stand-in mkinitcpio, and kernel_boot's answer. + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// 1977/AC1 +func TestAC1TheManifestDeclaresTheMicrocodePackagesDropInScriptAndService(t *testing.T) { + m := manifest(t) + var ids []string + for _, r := range m.Resources { + ids = append(ids, r["id"].(string)) + } + if !strings.HasSuffix(strings.Join(ids, " "), "modules-cleanup amd-ucode intel-ucode microcode-hook initramfs initramfs-unit initramfs-apply") { + t.Fatalf("order: %v", ids) + } + for id, pkg := range map[string]string{"amd-ucode": "amd-ucode", "intel-ucode": "intel-ucode"} { + if r := m.resource(t, id); r["type"] != "package" || r["package"] != pkg { + t.Fatalf("%s: %v", id, r) + } + } + if r := m.resource(t, "microcode-hook"); r["path"] != "/etc/mkinitcpio.conf.d/mesh-microcode.conf" { + t.Fatalf("drop-in: %v", r) + } + if r := m.resource(t, "initramfs"); r["path"] != "${dir:state}/initramfs" || r["mode"] != "0755" || r["trusted"] != true { + t.Fatalf("script: %v", r) + } + if r := m.resource(t, "initramfs-unit"); r["path"] != "/etc/systemd/system/mesh-kernel-initramfs.service" { + t.Fatalf("unit: %v", r) + } + a := m.resource(t, "initramfs-apply") + on := a["restart-on"].([]any) + if a["unit"] != "mesh-kernel-initramfs.service" || a["state"] != "running" || a["boot"] != "enabled" || + len(on) != 3 || on[0] != "initramfs-unit" || on[1] != "initramfs" || on[2] != "microcode-hook" { + t.Fatalf("service: %v", a) + } +} + +// 1977/AC2 +func TestAC2TheDropInAddsMicrocodeToHooksOnce(t *testing.T) { + drop := filepath.Join(t.TempDir(), "mesh-microcode.conf") + write(t, drop, manifest(t).resource(t, "microcode-hook").text(t), 0o644) + for _, c := range []struct{ hooks, want string }{ + {"base udev autodetect modconf block filesystems keyboard fsck", "base udev autodetect modconf block filesystems keyboard fsck microcode"}, + {"base udev autodetect microcode block", "base udev autodetect microcode block"}, + } { + out, err := exec.Command("bash", "-c", "HOOKS=("+c.hooks+"); . "+drop+"; echo \"${HOOKS[*]}\"").CombinedOutput() + if err != nil || strings.TrimSpace(string(out)) != c.want { + t.Errorf("%q: %v %q", c.hooks, err, out) + } + } +} + +type initBed struct{ script, dropin, state, runs, bin string } + +func newInitBed(t *testing.T, mkinitcpioExit string) initBed { + t.Helper() + dir := t.TempDir() + b := initBed{script: filepath.Join(dir, "initramfs"), dropin: filepath.Join(dir, "mesh-microcode.conf"), state: filepath.Join(dir, "state"), + runs: filepath.Join(dir, "runs"), bin: filepath.Join(dir, "bin")} + for _, d := range []string{b.state, b.bin} { + if err := os.MkdirAll(d, 0o755); err != nil { + t.Fatal(err) + } + } + content := strings.ReplaceAll(manifest(t).resource(t, "initramfs").text(t), "${dir:state}", b.state) + if strings.Contains(content, "${dir") { + t.Fatal("a substitution is left in the script") + } + write(t, b.script, content, 0o755) + write(t, b.dropin, "HOOKS+=(microcode)\n", 0o644) + write(t, filepath.Join(b.bin, "mkinitcpio"), "#!/bin/sh\necho \"$@\" >>"+b.runs+"\nexit "+mkinitcpioExit+"\n", 0o755) + return b +} + +func (b initBed) run(t *testing.T) (string, error) { + cmd := exec.Command("sh", b.script, "apply", b.dropin, b.state) + cmd.Env = append(os.Environ(), "PATH="+b.bin+string(os.PathListSeparator)+os.Getenv("PATH")) + out, err := cmd.CombinedOutput() + return string(out), err +} + +func (b initBed) builds(t *testing.T) string { + raw, _ := os.ReadFile(b.runs) + return string(raw) +} + +// 1977/AC3 +func TestAC3TheImagesAreBuiltOnceAndAgainWhenTheDropInChanges(t *testing.T) { + b := newInitBed(t, "0") + if out, err := b.run(t); err != nil || b.builds(t) != "-P\n" { + t.Fatalf("first: %v %s %q", err, out, b.builds(t)) + } + if raw, _ := os.ReadFile(filepath.Join(b.state, "initramfs-built")); string(raw) != "HOOKS+=(microcode)\n" { + t.Fatalf("recorded: %q", raw) + } + if out, err := b.run(t); err != nil || !strings.Contains(out, "as it was") || b.builds(t) != "-P\n" { + t.Fatalf("again: %v %s %q", err, out, b.builds(t)) + } + write(t, b.dropin, "HOOKS+=(microcode)\n# new\n", 0o644) + if _, err := b.run(t); err != nil || b.builds(t) != "-P\n-P\n" { + t.Fatalf("changed: %v %q", err, b.builds(t)) + } +} + +// 1977/AC4 +func TestAC4AFailedMkinitcpioFailsTheUnitAndRecordsNothing(t *testing.T) { + b := newInitBed(t, "1") + out, err := b.run(t) + if err == nil || !strings.Contains(out, "mkinitcpio failed") { + t.Fatalf("%v %s", err, out) + } + if _, err := os.Stat(filepath.Join(b.state, "initramfs-built")); !os.IsNotExist(err) { + t.Fatalf("recorded after a failure: %v", err) + } +} + +// 1977/AC5 +func TestAC5KernelBootNamesTheRunningMicrocodeAndWhetherTheImagesCarryTheDropIn(t *testing.T) { + root := t.TempDir() + put(t, root, "proc/cpuinfo", "processor\t: 0\nmicrocode\t: 0x0a201016\nmodel name\t: x\n") + put(t, root, "etc/mkinitcpio.conf.d/mesh-microcode.conf", "HOOKS+=(microcode)\n") + if got := bootOf(t, root); got.MicrocodeRevision != "0x0a201016" || got.ImagesBuiltWithMicrocode { + t.Fatalf("no record: %+v", got) + } + put(t, root, "var/lib/mesh-kernel/initramfs-built", "HOOKS+=(microcode)\n") + if got := bootOf(t, root); got.MicrocodeRevision != "0x0a201016" || !got.ImagesBuiltWithMicrocode { + t.Fatalf("recorded: %+v", got) + } +} diff --git a/modules/kernel/files/initramfs.sh b/modules/kernel/files/initramfs.sh new file mode 100644 index 000000000..92982f467 --- /dev/null +++ b/modules/kernel/files/initramfs.sh @@ -0,0 +1,36 @@ +#!/bin/sh +# The initramfs images as the mesh builds them (module kernel, novox/hq issue 1977). Run by +# mesh-kernel-initramfs.service: +# +# initramfs apply [ [ []]] +# +# `apply` builds every image again (mkinitcpio -P) when the drop-in differs from the one the images were last built +# with, kept in the state as `initramfs-built`, and records the drop-in only after mkinitcpio succeeded. Later +# kernel upgrades build the images through the distribution's own hook, with the drop-in in place. The arguments +# are only for the module's tests. +set -eu + +dropin=/etc/mkinitcpio.conf.d/mesh-microcode.conf +state='${dir:state}' +[ $# -ge 2 ] && dropin=$2 +[ $# -ge 3 ] && state=$3 + +[ $# -ge 1 ] && [ "$1" = apply ] || { + echo "usage: initramfs apply [ []]" >&2 + exit 2 +} +[ -f "$dropin" ] || { + echo "initramfs: $dropin is not there, so the images cannot be built with the microcode hook" >&2 + exit 1 +} +if [ -f "$state/initramfs-built" ] && cmp -s "$dropin" "$state/initramfs-built"; then + echo "initramfs: the images were built with this drop-in, as it was" + exit 0 +fi +if ! mkinitcpio -P; then + echo "initramfs: mkinitcpio failed; nothing recorded, the next start tries again" >&2 + exit 1 +fi +cp "$dropin" "$state/initramfs-built.new" +mv "$state/initramfs-built.new" "$state/initramfs-built" +echo "initramfs: the images are built with the microcode drop-in" diff --git a/modules/kernel/files/mesh-kernel-initramfs.service b/modules/kernel/files/mesh-kernel-initramfs.service new file mode 100644 index 000000000..95df9ba3f --- /dev/null +++ b/modules/kernel/files/mesh-kernel-initramfs.service @@ -0,0 +1,22 @@ +[Unit] +Description=The initramfs images with the CPU's current microcode +# Module kernel, novox/hq issue 1977. Starting this unit builds the images again when the microcode drop-in +# changed since they were last built; a failed mkinitcpio fails the unit. The microcode packages are declared +# before this unit. +After=local-fs.target +Wants=local-fs.target + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=${dir:state}/initramfs apply +# It writes the boot partition and its own state; mkinitcpio keeps its work in a private /tmp. +ProtectSystem=strict +ReadWritePaths=/boot ${dir:state} +PrivateTmp=yes +ProtectHome=yes +PrivateNetwork=yes +NoNewPrivileges=yes + +[Install] +WantedBy=multi-user.target diff --git a/modules/kernel/files/mesh-microcode.conf b/modules/kernel/files/mesh-microcode.conf new file mode 100644 index 000000000..6025eac81 --- /dev/null +++ b/modules/kernel/files/mesh-microcode.conf @@ -0,0 +1,8 @@ +# Module kernel, novox/hq issue 1977. mkinitcpio reads this after /etc/mkinitcpio.conf: it adds the microcode +# hook, which puts the CPU's current microcode (amd-ucode, intel-ucode) at the front of every initramfs, so +# every boot entry loads it, the default one too. HOOKS that name it already are left as they are. +# Written by the mesh; change the module. +case "$(declare -p HOOKS)" in +*'"microcode"'*) ;; +*) HOOKS+=(microcode) ;; +esac diff --git a/modules/kernel/module.json b/modules/kernel/module.json index 8fc90d09c..69f064d2d 100644 --- a/modules/kernel/module.json +++ b/modules/kernel/module.json @@ -119,6 +119,53 @@ "unit": "linux-modules-cleanup.service", "state": "stopped", "boot": "enabled" + }, + { + "id": "amd-ucode", + "type": "package", + "package": "amd-ucode" + }, + { + "id": "intel-ucode", + "type": "package", + "package": "intel-ucode" + }, + { + "id": "microcode-hook", + "type": "file", + "path": "/etc/mkinitcpio.conf.d/mesh-microcode.conf", + "mode": "0644", + "template": "files/mesh-microcode.conf" + }, + { + "id": "initramfs", + "type": "file", + "path": "${dir:state}/initramfs", + "mode": "0755", + "trusted": true, + "template": "files/initramfs.sh" + }, + { + "id": "initramfs-unit", + "type": "file", + "path": "/etc/systemd/system/mesh-kernel-initramfs.service", + "mode": "0644", + "template": "files/mesh-kernel-initramfs.service" + }, + { + "id": "initramfs-apply", + "type": "service", + "unit": "mesh-kernel-initramfs.service", + "state": "running", + "boot": "enabled", + "restart-on": [ + "initramfs-unit", + "initramfs", + "microcode-hook" + ], + "health": { + "kind": "unit" + } } ], "build": {