From 3a84044eceba6f28fb42743df89ad0e1cae239b9 Mon Sep 17 00:00:00 2001 From: jochen Date: Wed, 14 Oct 2026 23:31:13 +0200 Subject: [PATCH 1/2] Add the kernel module: transparent huge pages set per machine (hq issue 746) With THP 'always' journald's mmapped journal files sit in 2 MiB folios, so each small journal write dirties and writes back 2 MiB; on the control node that saturated the root mirror holding the bus store and the store, and the controller lost its lease. The module sets the mode from a per-machine setting (default always, the kernel's own, so assigning it changes nothing until a machine sets madvise), verifies it took, rotates the journal once so new files are mapped under the new mode, gives back what it found when stopped, and serves kernel_huge_pages to measure file_thp per unit. --- modules/kernel/README.md | 50 ++++ modules/kernel/cmd/kernel-tools/hugepages.go | 241 ++++++++++++++++++ .../kernel/cmd/kernel-tools/hugepages_test.go | 113 ++++++++ modules/kernel/cmd/kernel-tools/machine.go | 65 +++++ modules/kernel/cmd/kernel-tools/main.go | 50 ++++ .../kernel/cmd/kernel-tools/manifest_test.go | 181 +++++++++++++ modules/kernel/cmd/kernel-tools/shape_test.go | 79 ++++++ modules/kernel/go.mod | 5 + modules/kernel/go.sum | 2 + modules/kernel/module.json | 69 +++++ 10 files changed, 855 insertions(+) create mode 100644 modules/kernel/README.md create mode 100644 modules/kernel/cmd/kernel-tools/hugepages.go create mode 100644 modules/kernel/cmd/kernel-tools/hugepages_test.go create mode 100644 modules/kernel/cmd/kernel-tools/machine.go create mode 100644 modules/kernel/cmd/kernel-tools/main.go create mode 100644 modules/kernel/cmd/kernel-tools/manifest_test.go create mode 100644 modules/kernel/cmd/kernel-tools/shape_test.go create mode 100644 modules/kernel/go.mod create mode 100644 modules/kernel/go.sum create mode 100644 modules/kernel/module.json diff --git a/modules/kernel/README.md b/modules/kernel/README.md new file mode 100644 index 00000000..faf4cc94 --- /dev/null +++ b/modules/kernel/README.md @@ -0,0 +1,50 @@ +# kernel + +The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). Today it holds one: +**transparent huge pages**, set per machine. The rest of the row (the kernel package, microcode, boot +entries) is not built yet. + +## Why (novox/hq issue 746) + +With transparent huge pages `always`, the page cache of a file mapped by a program is collapsed into +2 MiB folios (khugepaged), and a small write into such a folio dirties all of it: writeback then writes +2 MiB. journald writes its journal files through mmap, and every journal reader (fail2ban's journal +jails among them) maps them too. Measured on the control node on 2026-10-14: journald's cgroup had +written 2.5 TB in 12 days while the journal grew by about 25-30 KB/s, fail2ban's cgroup 63 GB in 4 h +while its own files did not grow, both holding hundreds of MiB of file pages in huge folios; the bursts +saturated the mirrored spinning root disk that holds the bus store and the store, and the controller +lost its lease four times in 30 minutes. With `madvise`, only memory a program asks to have in huge pages +(`madvise(MADV_HUGEPAGE)`) gets them; a database or a JVM that wants them still asks. + +## What it owns + +| | what | notes | +|---|---|---| +| setting | `transparent-huge-pages` | `always`, `madvise` or `never`. Default `always`, what the distribution's kernel boots with, so assigning the module changes nothing until a machine sets another value | +| file | `${dir:state}/huge-pages` | the script, written from the setting: `apply` writes the mode to `/sys/kernel/mm/transparent_hugepage/enabled` and reads it back (a value that does not take fails the unit); `restore` gives back the mode found before the module first set it, kept in `${dir:state}/found` | +| file | `/etc/systemd/system/mesh-kernel-huge-pages.service` | oneshot, early in boot (`sysinit.target`, before the services that fill the page cache); stopping it runs `restore` | +| service | `mesh-kernel-huge-pages.service` | running and enabled, restarted when the script (so the setting) or the unit changes; its health is the unit's | + +The per-size controls (`hugepages-kB/enabled`) are left alone: on the machines read, 2048 kB says +`inherit` (it follows the mode) and every other size `never`. `kernel_huge_pages` shows each one. + +**A folio already in the page cache keeps its size.** A new mode applies to pages faulted or collapsed +from then on. The active journal files would keep their 2 MiB folios until archived, so `apply` rotates +the journal (`journalctl --rotate`) whenever it moves the mode away from `always`: new files are mapped +under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. At +boot the kernel starts at `always`, so a machine set to `madvise` rotates its journal once per boot. + +## Tools + +| tool | | answers | +|---|---|---| +| `kernel_huge_pages` | r | the mode, defrag and shared-memory mode; each per-size control and what `inherit` resolves to; huge pages by kind from `/proc/meminfo` (MiB); the units holding the most file pages in huge folios (`memory.stat` `file_thp`) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (`io.stat`, the most any one device saw: a mirror and its members each count the same write) | + +## Tests + +`go test ./...`: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the +manifest's shape (the default, the order, the unit, restart and health); and the script as the mesh +writes it, run with `sh` against a stand-in directory: it sets and reads back, keeps what it found once, +rotates the journal once and not for `always`, gives back what it found, refuses a value the kernel +does not take before writing anything, fails when the write does not take, and says when the kernel has +no transparent huge pages. diff --git a/modules/kernel/cmd/kernel-tools/hugepages.go b/modules/kernel/cmd/kernel-tools/hugepages.go new file mode 100644 index 00000000..697f4fd8 --- /dev/null +++ b/modules/kernel/cmd/kernel-tools/hugepages.go @@ -0,0 +1,241 @@ +package main + +import ( + "fmt" + "os" + "path" + "sort" + "strconv" + "strings" +) + +// THPDir is where the kernel says and takes its transparent huge pages. +const THPDir = "sys/kernel/mm/transparent_hugepage" + +// CgroupRoot is the unified cgroup hierarchy. +const CgroupRoot = "sys/fs/cgroup" + +// HugePagesAnswer is what kernel_huge_pages answers. +type HugePagesAnswer struct { + // Mode is the word the kernel brackets in enabled: always, madvise or never. + Mode string `json:"mode"` + Defrag string `json:"defrag,omitempty"` + Shmem string `json:"shmem_enabled,omitempty"` + PerSize []Size `json:"per_size,omitempty"` + // Meminfo is the machine's huge pages by kind, in MiB. + Meminfo map[string]float64 `json:"meminfo_mib,omitempty"` + Units []Unit `json:"units"` + Said []string `json:"said"` +} + +// Size is one per-size control (hugepages-kB/enabled). +type Size struct { + Size string `json:"size"` + Enabled string `json:"enabled"` + // Means is what inherit resolves to: the top-level mode. + Means string `json:"means"` +} + +// Unit is one cgroup holding file pages in huge folios. +type Unit struct { + Cgroup string `json:"cgroup"` + FileTHP int64 `json:"file_thp_bytes"` + FileDirty int64 `json:"file_dirty_bytes"` + FileWriteback int64 `json:"file_writeback_bytes"` + FileMapped int64 `json:"file_mapped_bytes"` + CollapsedByKhug int64 `json:"thp_collapse_alloc"` + Written int64 `json:"written_bytes"` +} + +// bracketed is the word a sysfs choice file brackets: "always [madvise] never" is madvise. +func bracketed(text string) string { + open := strings.IndexByte(text, '[') + end := strings.IndexByte(text, ']') + if open < 0 || end < open { + return "" + } + return text[open+1 : end] +} + +// HugePages reads the mode, the per-size controls, meminfo and the top units by file_thp. +func (m *Machine) HugePages(units int) (*HugePagesAnswer, error) { + enabled, err := m.read(THPDir + "/enabled") + if err != nil { + if os.IsNotExist(err) { + return nil, fmt.Errorf("this kernel has no /%s/enabled: it was built without transparent huge pages", THPDir) + } + return nil, fmt.Errorf("reading /%s/enabled: %w", THPDir, err) + } + a := &HugePagesAnswer{Mode: bracketed(enabled), Units: []Unit{}} + if s, err := m.read(THPDir + "/defrag"); err == nil { + a.Defrag = bracketed(s) + } + if s, err := m.read(THPDir + "/shmem_enabled"); err == nil { + a.Shmem = bracketed(s) + } + a.Said = append(a.Said, fmt.Sprintf("transparent huge pages: %s, defrag %s", a.Mode, orUnknown(a.Defrag))) + + sizes, _ := m.dirs(THPDir) + for _, d := range sizes { + if !strings.HasPrefix(d, "hugepages-") { + continue + } + s, err := m.read(THPDir + "/" + d + "/enabled") + if err != nil { + continue + } + sz := Size{Size: strings.TrimPrefix(d, "hugepages-"), Enabled: bracketed(s)} + sz.Means = sz.Enabled + if sz.Enabled == "inherit" { + sz.Means = a.Mode + } + a.PerSize = append(a.PerSize, sz) + } + sort.Slice(a.PerSize, func(i, j int) bool { return kib(a.PerSize[i].Size) < kib(a.PerSize[j].Size) }) + var on []string + for _, s := range a.PerSize { + if s.Means != "never" { + on = append(on, s.Size+" "+s.Means) + } + } + if len(a.PerSize) > 0 { + a.Said = append(a.Said, "sizes not never: "+orNone(on)) + } + + if mi, err := m.read("proc/meminfo"); err == nil { + a.Meminfo = meminfo(mi) + a.Said = append(a.Said, fmt.Sprintf("huge pages held: anonymous %.0f MiB, shared memory %.0f MiB, file %.0f MiB (mapped at 2 MiB %.0f MiB)", + a.Meminfo["AnonHugePages"], a.Meminfo["ShmemHugePages"], a.Meminfo["FileHugePages"], a.Meminfo["FilePmdMapped"])) + } + + if units > 0 { + all := m.cgroups() + sort.Slice(all, func(i, j int) bool { + if all[i].FileTHP != all[j].FileTHP { + return all[i].FileTHP > all[j].FileTHP + } + return all[i].Cgroup < all[j].Cgroup + }) + if len(all) > units { + all = all[:units] + } + a.Units = all + for _, u := range all { + a.Said = append(a.Said, fmt.Sprintf("%s: file in huge folios %s, dirty %s, writeback %s, written %s since it started", + u.Cgroup, mib(u.FileTHP), mib(u.FileDirty), mib(u.FileWriteback), mib(u.Written))) + } + if len(all) == 0 { + a.Said = append(a.Said, "no unit holds file pages in huge folios") + } + } + return a, nil +} + +// cgroups walks the leaf services and scopes (to three levels below the root) that hold file_thp. +func (m *Machine) cgroups() []Unit { + var out []Unit + var walk func(rel string, depth int) + walk = func(rel string, depth int) { + children, _ := m.dirs(path.Join(CgroupRoot, rel)) + for _, c := range children { + p := path.Join(rel, c) + if strings.HasSuffix(c, ".slice") { + if depth < 3 { + walk(p, depth+1) + } + continue + } + if !strings.HasSuffix(c, ".service") && !strings.HasSuffix(c, ".scope") { + continue + } + stat, err := m.read(path.Join(CgroupRoot, p, "memory.stat")) + if err != nil { + continue + } + kv := memoryStat(stat) + if kv["file_thp"] == 0 { + continue + } + u := Unit{Cgroup: p, FileTHP: kv["file_thp"], FileDirty: kv["file_dirty"], FileWriteback: kv["file_writeback"], + FileMapped: kv["file_mapped"], CollapsedByKhug: kv["thp_collapse_alloc"]} + if io, err := m.read(path.Join(CgroupRoot, p, "io.stat")); err == nil { + u.Written = mostWritten(io) + } + out = append(out, u) + } + } + walk("", 0) + return out +} + +// memoryStat reads memory.stat's "key value" lines. +func memoryStat(text string) map[string]int64 { + kv := map[string]int64{} + for _, l := range strings.Split(text, "\n") { + f := strings.Fields(l) + if len(f) != 2 { + continue + } + if n, err := strconv.ParseInt(f[1], 10, 64); err == nil { + kv[f[0]] = n + } + } + return kv +} + +// mostWritten is the most bytes any one device of io.stat saw written: a mirror's member disks and the +// mirror itself each count the same write, so the lines are never summed. +func mostWritten(text string) int64 { + var most int64 + for _, l := range strings.Split(text, "\n") { + for _, f := range strings.Fields(l) { + if v, ok := strings.CutPrefix(f, "wbytes="); ok { + if n, err := strconv.ParseInt(v, 10, 64); err == nil && n > most { + most = n + } + } + } + } + return most +} + +// meminfo reads the huge page lines of /proc/meminfo, in MiB. +func meminfo(text string) map[string]float64 { + want := map[string]bool{"AnonHugePages": true, "ShmemHugePages": true, "ShmemPmdMapped": true, "FileHugePages": true, "FilePmdMapped": true} + out := map[string]float64{} + for _, l := range strings.Split(text, "\n") { + k, v, ok := strings.Cut(l, ":") + if !ok || !want[k] { + continue + } + f := strings.Fields(v) + if len(f) == 0 { + continue + } + if n, err := strconv.ParseFloat(f[0], 64); err == nil { + out[k] = n / 1024 + } + } + return out +} + +func kib(size string) int { + n, _ := strconv.Atoi(strings.TrimSuffix(size, "kB")) + return n +} + +func mib(b int64) string { return fmt.Sprintf("%.1f MiB", float64(b)/(1<<20)) } + +func orUnknown(s string) string { + if s == "" { + return "unknown" + } + return s +} + +func orNone(list []string) string { + if len(list) == 0 { + return "none" + } + return strings.Join(list, ", ") +} diff --git a/modules/kernel/cmd/kernel-tools/hugepages_test.go b/modules/kernel/cmd/kernel-tools/hugepages_test.go new file mode 100644 index 00000000..2d6359c7 --- /dev/null +++ b/modules/kernel/cmd/kernel-tools/hugepages_test.go @@ -0,0 +1,113 @@ +package main + +// Over a stand-in tree holding novox's own files of 2026-10-14 23:25 (read through node-reads for +// novox/hq issue 746): the mode, the per-size controls, meminfo's huge page lines, and fail2ban's and +// journald's memory.stat and io.stat, cut to the lines the tool reads. + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func tree(t *testing.T, files map[string]string) *Machine { + t.Helper() + root := t.TempDir() + for p, content := range files { + full := filepath.Join(root, p) + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(full, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + return &Machine{Root: root} +} + +func novox(t *testing.T) *Machine { + thp := "sys/kernel/mm/transparent_hugepage/" + cg := "sys/fs/cgroup/system.slice/" + files := map[string]string{ + thp + "enabled": "[always] madvise never\n", + thp + "defrag": "always defer defer+madvise [madvise] never\n", + thp + "shmem_enabled": "always within_size [advise] never deny force\n", + thp + "hugepages-2048kB/enabled": "always [inherit] madvise never\n", + thp + "hugepages-64kB/enabled": "always inherit madvise [never]\n", + thp + "hugepages-1024kB/enabled": "always inherit madvise [never]\n", + "proc/meminfo": "MemTotal: 65748808 kB\nAnonHugePages: 3579904 kB\nShmemHugePages: 4096 kB\n" + + "ShmemPmdMapped: 4096 kB\nFileHugePages: 1693696 kB\nFilePmdMapped: 882688 kB\nHugepagesize: 2048 kB\n", + cg + "fail2ban.service/memory.stat": "anon 21815296\nfile 204623872\nfile_mapped 130736128\nfile_dirty 27262976\n" + + "file_writeback 4096\nanon_thp 0\nfile_thp 197132288\nthp_fault_alloc 0\nthp_collapse_alloc 550\n", + cg + "fail2ban.service/io.stat": "9:0 \n8:16 rbytes=3223552 wbytes=67723415552 rios=240 wios=281159 dbytes=0 dios=0\n" + + "8:0 rbytes=6152192 wbytes=67723415552 rios=832 wios=281159 dbytes=0 dios=0\n" + + "9:2 rbytes=9375744 wbytes=67723415552 rios=1072 wios=281159 dbytes=0 dios=0\n", + cg + "systemd-journald.service/memory.stat": "anon 3227648\nfile 1032015872\nfile_mapped 131928064\nfile_dirty 622592\n" + + "file_writeback 159744\nanon_thp 2097152\nfile_thp 379584512\nthp_fault_alloc 0\nthp_collapse_alloc 11741\n", + cg + "systemd-journald.service/io.stat": "9:0 \n8:16 rbytes=55209984 wbytes=2498768035840 rios=1317 wios=24719004 dbytes=0 dios=0\n" + + "8:0 rbytes=72953856 wbytes=2498768039936 rios=2194 wios=24719005 dbytes=0 dios=0\n" + + "9:2 rbytes=128163840 wbytes=2498724843520 rios=3511 wios=24707873 dbytes=0 dios=0\n", + cg + "sshd.service/memory.stat": "anon 1000\nfile_thp 0\n", + } + return tree(t, files) +} + +func TestNovoxBeforeTheChange(t *testing.T) { + a, err := novox(t).HugePages(10) + if err != nil { + t.Fatal(err) + } + if a.Mode != "always" || a.Defrag != "madvise" || a.Shmem != "advise" { + t.Fatalf("modes: %+v", a) + } + if len(a.PerSize) != 3 || a.PerSize[0].Size != "64kB" || a.PerSize[2].Size != "2048kB" || + a.PerSize[2].Enabled != "inherit" || a.PerSize[2].Means != "always" || a.PerSize[0].Means != "never" { + t.Fatalf("per size: %+v", a.PerSize) + } + if a.Meminfo["FileHugePages"] != 1654 || a.Meminfo["FilePmdMapped"] != 862 { + t.Fatalf("meminfo: %v", a.Meminfo) + } + if len(a.Units) != 2 || a.Units[0].Cgroup != "system.slice/systemd-journald.service" || + a.Units[1].Cgroup != "system.slice/fail2ban.service" { + t.Fatalf("units: %+v", a.Units) + } + // The mirror and its two members each count the write: the most any one saw, never their sum. + if a.Units[1].Written != 67723415552 || a.Units[1].FileTHP != 197132288 || a.Units[1].CollapsedByKhug != 550 { + t.Fatalf("fail2ban: %+v", a.Units[1]) + } + said := strings.Join(a.Said, "\n") + for _, want := range []string{"transparent huge pages: always, defrag madvise", "sizes not never: 2048kB always", + "file 1654 MiB (mapped at 2 MiB 862 MiB)", "system.slice/fail2ban.service: file in huge folios 188.0 MiB"} { + if !strings.Contains(said, want) { + t.Errorf("said lacks %q:\n%s", want, said) + } + } +} + +func TestUnitsAreBoundedAndNoneIsSaid(t *testing.T) { + a, err := novox(t).HugePages(1) + if err != nil || len(a.Units) != 1 { + t.Fatalf("%v %+v", err, a) + } + b, err := tree(t, map[string]string{"sys/kernel/mm/transparent_hugepage/enabled": "always [madvise] never\n"}).HugePages(10) + if err != nil || b.Mode != "madvise" || len(b.Units) != 0 || !strings.Contains(strings.Join(b.Said, "\n"), "no unit holds") { + t.Fatalf("%v %+v", err, b) + } +} + +func TestAKernelWithoutHugePagesIsAnError(t *testing.T) { + if _, err := tree(t, map[string]string{"proc/meminfo": ""}).HugePages(10); err == nil || + !strings.Contains(err.Error(), "built without transparent huge pages") { + t.Fatalf("%v", err) + } +} + +func TestTheArgumentIsBounded(t *testing.T) { + if _, err := whole(map[string]any{"units": -1.0}, "units", 10, 0, 100); err == nil { + t.Error("a negative count was accepted") + } + if n, _ := whole(map[string]any{"units": 1000.0}, "units", 10, 0, 100); n != 100 { + t.Errorf("not bounded: %d", n) + } +} diff --git a/modules/kernel/cmd/kernel-tools/machine.go b/modules/kernel/cmd/kernel-tools/machine.go new file mode 100644 index 00000000..b5f6c730 --- /dev/null +++ b/modules/kernel/cmd/kernel-tools/machine.go @@ -0,0 +1,65 @@ +package main + +// The machine as the tools see it: the files they read, under a root that tests replace with a stand-in +// tree. Everything read here is readable by the runtime's account; nothing is escalated. + +import ( + "fmt" + "io/fs" + "os" + "path/filepath" +) + +// Machine reads this machine's kernel files under Root ("/" on the machine). +type Machine struct { + Root string +} + +// ThisMachine is the machine the runtime launched this bundle on. +func ThisMachine() *Machine { return &Machine{Root: "/"} } + +func (m *Machine) path(p string) string { return filepath.Join(m.Root, p) } + +func (m *Machine) read(p string) (string, error) { + b, err := os.ReadFile(m.path(p)) + return string(b), err +} + +func (m *Machine) dirs(p string) ([]string, error) { + entries, err := os.ReadDir(m.path(p)) + if err != nil { + return nil, err + } + var out []string + for _, e := range entries { + if e.IsDir() || e.Type()&fs.ModeSymlink != 0 { + out = append(out, e.Name()) + } + } + return out, nil +} + +// whole is a whole-number argument with a default, kept within bounds. +func whole(args map[string]any, key string, def, least, most int) (int, error) { + raw, present := args[key] + if !present || raw == nil { + return def, nil + } + f, ok := raw.(float64) + if !ok || f != float64(int(f)) { + return 0, fmt.Errorf("%s must be a whole number", key) + } + n := int(f) + if n < least { + return 0, fmt.Errorf("%s must be at least %d", key, least) + } + if n > most { + n = most + } + return n, nil +} + +// schema is a tool's input: its properties. +func schema(properties map[string]any) map[string]any { + return map[string]any{"type": "object", "properties": properties} +} diff --git a/modules/kernel/cmd/kernel-tools/main.go b/modules/kernel/cmd/kernel-tools/main.go new file mode 100644 index 00000000..ec15c916 --- /dev/null +++ b/modules/kernel/cmd/kernel-tools/main.go @@ -0,0 +1,50 @@ +// The kernel module's tools bundle (novox/hq to-be 42 Phase 1 row 11, issue 746). +// +// Served by the node's runtime over MCP on stdio through the Go SDK (ADR 0188, ADR 0193). It only reads: +// the transparent huge pages the kernel runs with, and which units hold file pages in huge folios, which +// is what a small write to a mapped file costs a whole huge page of writeback for (issue 746). +package main + +import ( + "fmt" + "os" + + stdio "git.novox.be/novox/mesh-sdk/go" +) + +// binaryName is what the build names this bundle's executable: the manifest's `binary`. +const binaryName = "kernel-tools" + +func main() { + // An empty name serves as the module the runtime names (MESH_SERVED_MODULE): kernel. + if err := stdio.Serve("", tools(ThisMachine())); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} + +func tools(m *Machine) []stdio.Tool { + return []stdio.Tool{ + { + Name: "kernel_huge_pages", + Description: "Transparent huge pages on this machine: the mode in force (always, madvise or never) and defrag, " + + "each per-size control with what inherit resolves to, the machine's huge pages by kind (anonymous, " + + "shared memory, file, file mapped at 2 MiB) from /proc/meminfo, and the units holding the most file " + + "pages in huge folios (memory.stat file_thp) with their dirty and writeback bytes and the bytes their " + + "cgroup has written (io.stat, the most any one device saw). A unit with file_thp that writes far more " + + "than its files grow is paying a whole huge page of writeback per small write (novox/hq issue 746). " + + "Replaces cat /sys/kernel/mm/transparent_hugepage/enabled, grep Huge /proc/meminfo and reading " + + "memory.stat by hand. (r)", + Input: schema(map[string]any{ + "units": map[string]any{"type": "integer", "description": "how many units to name, the most file_thp first (default 10, 0 for none, at most 100)"}, + }), + Run: func(args map[string]any) (any, error) { + n, err := whole(args, "units", 10, 0, 100) + if err != nil { + return nil, err + } + return m.HugePages(n) + }, + }, + } +} diff --git a/modules/kernel/cmd/kernel-tools/manifest_test.go b/modules/kernel/cmd/kernel-tools/manifest_test.go new file mode 100644 index 00000000..fd1a0327 --- /dev/null +++ b/modules/kernel/cmd/kernel-tools/manifest_test.go @@ -0,0 +1,181 @@ +package main + +// The module's shape (novox/hq issue 746): the setting's default is what the kernel boots with, so +// assigning the module changes nothing until a machine sets another value; the script is written into the +// module's own state and run by a oneshot unit that is restarted whenever the script (and so the setting) +// or the unit changes; and the script itself sets, verifies, gives back and refuses as it says. + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +func TestTheDefaultChangesNothingAndTheUnitFollowsTheSetting(t *testing.T) { + m := manifest(t) + s := m.Settings["transparent-huge-pages"] + if s["kind"] != "preference" || s["default"] != "always" || s["why"] == "" { + t.Fatalf("the setting: %v", s) + } + var ids []string + for _, r := range m.Resources { + ids = append(ids, r["id"].(string)) + } + if strings.Join(ids, " ") != "state huge-pages unit apply" { + t.Fatalf("order: %v", ids) + } + script := m.resource(t, "huge-pages") + // The script runs as root, so the setting it is written from is set at the terminal alone (issue 339). + if script["path"] != "${dir:state}/huge-pages" || script["mode"] != "0755" || script["trusted"] != true || + !strings.Contains(script["content"].(string), "want='${setting:transparent-huge-pages}'") { + t.Fatalf("the script: %v", script["path"]) + } + unit := m.resource(t, "unit")["content"].(string) + for _, line := range []string{"Type=oneshot", "RemainAfterExit=yes", "ExecStart=${dir:state}/huge-pages apply", + "ExecStop=${dir:state}/huge-pages restore", "WantedBy=sysinit.target", "DefaultDependencies=no"} { + if !strings.Contains(unit, line+"\n") { + t.Errorf("the unit lacks %q", line) + } + } + apply := m.resource(t, "apply") + if apply["unit"] != "mesh-kernel-huge-pages.service" || apply["state"] != "running" || apply["boot"] != "enabled" { + t.Fatalf("apply: %v", apply) + } + if on := apply["restart-on"].([]any); len(on) != 2 || on[0] != "huge-pages" || on[1] != "unit" { + t.Fatalf("restart-on: %v", on) + } + if h := apply["health"].(map[string]any); h["kind"] != "unit" { + t.Fatalf("health: %v", h) + } +} + +// bed is the script as the mesh writes it for one value of the setting, a stand-in transparent_hugepage +// directory, the module's state, and a PATH whose journalctl records that it was asked to rotate. +type bed struct { + script, sys, state, rotated, path string +} + +func newBed(t *testing.T, want, found string) bed { + t.Helper() + dir := t.TempDir() + b := bed{script: filepath.Join(dir, "huge-pages"), sys: filepath.Join(dir, "thp"), state: filepath.Join(dir, "state"), + rotated: filepath.Join(dir, "rotated")} + content := manifest(t).resource(t, "huge-pages")["content"].(string) + content = strings.ReplaceAll(content, "${setting:transparent-huge-pages}", want) + content = strings.ReplaceAll(content, "${dir:state}", b.state) + if strings.Contains(content, "${") { + t.Fatalf("a substitution the test does not make is left in the script") + } + bin := filepath.Join(dir, "bin") + for _, d := range []string{b.sys, b.state, bin} { + if err := os.MkdirAll(d, 0o755); err != nil { + t.Fatal(err) + } + } + write(t, b.script, content, 0o755) + write(t, filepath.Join(b.sys, "enabled"), found, 0o644) + write(t, filepath.Join(bin, "journalctl"), "#!/bin/sh\necho \"$@\" >>"+b.rotated+"\n", 0o755) + b.path = bin + string(os.PathListSeparator) + os.Getenv("PATH") + return b +} + +func write(t *testing.T, p, content string, mode os.FileMode) { + t.Helper() + if err := os.WriteFile(p, []byte(content), mode); err != nil { + t.Fatal(err) + } +} + +func (b bed) run(t *testing.T, verb string) (string, error) { + t.Helper() + cmd := exec.Command("sh", b.script, verb, b.sys, b.state) + cmd.Env = append(os.Environ(), "PATH="+b.path) + out, err := cmd.CombinedOutput() + return string(out), err +} + +func (b bed) file(t *testing.T, p string) string { + t.Helper() + raw, err := os.ReadFile(p) + if err != nil { + return "" + } + return string(raw) +} + +func TestApplySetsTheModeRotatesTheJournalOnceAndRestoreGivesBackWhatWasFound(t *testing.T) { + // novox's file on 2026-10-14, read through node-reads. + b := newBed(t, "madvise", "[always] madvise never\n") + out, err := b.run(t, "apply") + if err != nil || !strings.Contains(out, "always -> madvise") { + t.Fatalf("apply: %v %s", err, out) + } + if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "madvise\n" { + t.Fatalf("written: %q", got) + } + if got := b.file(t, filepath.Join(b.state, "found")); got != "always\n" { + t.Fatalf("found: %q", got) + } + if got := b.file(t, b.rotated); got != "--rotate\n" { + t.Fatalf("rotation: %q", got) + } + // Applied again (a restart with the same value): nothing written, nothing rotated, found kept. + if out, err := b.run(t, "apply"); err != nil || !strings.Contains(out, "madvise, as set") { + t.Fatalf("again: %v %s", err, out) + } + if got := b.file(t, b.rotated); got != "--rotate\n" { + t.Fatalf("rotated again: %q", got) + } + if out, err := b.run(t, "restore"); err != nil || !strings.Contains(out, "back to always") { + t.Fatalf("restore: %v %s", err, out) + } + if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "always\n" { + t.Fatalf("restored: %q", got) + } +} + +func TestAlwaysIsSetWithoutRotating(t *testing.T) { + b := newBed(t, "always", "always [madvise] never\n") + if out, err := b.run(t, "apply"); err != nil || !strings.Contains(out, "madvise -> always") { + t.Fatalf("apply: %v %s", err, out) + } + if got := b.file(t, b.rotated); got != "" { + t.Fatalf("rotated for always: %q", got) + } +} + +func TestAValueTheKernelDoesNotTakeIsRefusedBeforeAnythingIsWritten(t *testing.T) { + b := newBed(t, "sometimes", "[always] madvise never\n") + out, err := b.run(t, "apply") + if err == nil || !strings.Contains(out, "takes always, madvise or never") { + t.Fatalf("accepted: %v %s", err, out) + } + if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "[always] madvise never\n" { + t.Fatalf("written anyway: %q", got) + } +} + +func TestAWriteThatDoesNotTakeFailsTheUnit(t *testing.T) { + if os.Getuid() == 0 { + t.Skip("root writes a read-only file") + } + b := newBed(t, "madvise", "[always] madvise never\n") + if err := os.Chmod(filepath.Join(b.sys, "enabled"), 0o444); err != nil { + t.Fatal(err) + } + if out, err := b.run(t, "apply"); err == nil { + t.Fatalf("a refused write passed: %s", out) + } +} + +func TestAKernelWithoutHugePagesIsSaid(t *testing.T) { + b := newBed(t, "madvise", "") + if err := os.Remove(filepath.Join(b.sys, "enabled")); err != nil { + t.Fatal(err) + } + if out, err := b.run(t, "apply"); err == nil || !strings.Contains(out, "built without transparent huge pages") { + t.Fatalf("%v %s", err, out) + } +} diff --git a/modules/kernel/cmd/kernel-tools/shape_test.go b/modules/kernel/cmd/kernel-tools/shape_test.go new file mode 100644 index 00000000..c30224f3 --- /dev/null +++ b/modules/kernel/cmd/kernel-tools/shape_test.go @@ -0,0 +1,79 @@ +package main + +import ( + "encoding/json" + "os" + "testing" +) + +type resource map[string]any + +type manifestShape struct { + Module string `json:"module"` + Capabilities []string `json:"capabilities"` + Settings map[string]map[string]any `json:"settings"` + Tools []string `json:"tools"` + Resources []resource `json:"resources"` + Build struct { + Artifacts []map[string]any `json:"artifacts"` + } `json:"build"` +} + +func manifest(t *testing.T) manifestShape { + t.Helper() + raw, err := os.ReadFile("../../module.json") + if err != nil { + t.Fatal(err) + } + var m manifestShape + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + return m +} + +func (m manifestShape) resource(t *testing.T, id string) resource { + t.Helper() + for _, r := range m.Resources { + if r["id"] == id { + return r + } + } + t.Fatalf("no resource %s", id) + return nil +} + +// TestToolsAreTheManifests holds the served tools and the manifest's list to one another, and the +// bundle to the shape the builder compiles and the runtime loads. +func TestToolsAreTheManifests(t *testing.T) { + m := manifest(t) + names := map[string]bool{} + for _, tool := range tools(&Machine{Root: t.TempDir()}) { + if names[tool.Name] { + t.Errorf("%s is served twice", tool.Name) + } + names[tool.Name] = true + } + for _, want := range m.Tools { + if !names[want] { + t.Errorf("the manifest lists %s and the bundle does not serve it", want) + } + delete(names, want) + } + if len(names) != 0 { + t.Errorf("served and not listed: %v", names) + } + var tools map[string]any + for _, a := range m.Build.Artifacts { + if a["name"] == "tools" { + tools = a + } + } + if tools == nil || tools["kind"] != "bundle" || tools["language"] != "go" || tools["system"] != "arch" || + tools["from"] != "cmd/"+binaryName || tools["binary"] != binaryName { + t.Fatalf("the tools artifact: %v", tools) + } + if loads, _ := tools["loads"].([]any); len(loads) != 1 || loads[0] != binaryName { + t.Fatalf("loads: %v", tools["loads"]) + } +} diff --git a/modules/kernel/go.mod b/modules/kernel/go.mod new file mode 100644 index 00000000..748c098c --- /dev/null +++ b/modules/kernel/go.mod @@ -0,0 +1,5 @@ +module kernel + +go 1.22 + +require git.novox.be/novox/mesh-sdk/go v0.1.7 diff --git a/modules/kernel/go.sum b/modules/kernel/go.sum new file mode 100644 index 00000000..b474419e --- /dev/null +++ b/modules/kernel/go.sum @@ -0,0 +1,2 @@ +git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w= +git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY= diff --git a/modules/kernel/module.json b/modules/kernel/module.json new file mode 100644 index 00000000..68645769 --- /dev/null +++ b/modules/kernel/module.json @@ -0,0 +1,69 @@ +{ + "module": "kernel", + "version": "1", + "capabilities": [ + "service-manager" + ], + "settings": { + "transparent-huge-pages": { + "kind": "preference", + "default": "always", + "why": "always is what the distribution's kernel boots with, so assigning the module changes nothing until a machine sets another value; madvise gives huge pages only to programs that ask for them, which stops a small write to a mapped file (the journal) dirtying and writing back a whole 2 MiB page on a machine whose disks are slow (novox/hq issue 746)" + } + }, + "tools": [ + "kernel_huge_pages" + ], + "resources": [ + { + "id": "state", + "type": "directory", + "mode": "0700", + "place": "." + }, + { + "id": "huge-pages", + "type": "file", + "path": "${dir:state}/huge-pages", + "mode": "0755", + "trusted": true, + "content": "#!/bin/sh\n# Transparent huge pages as the mesh sets them on this machine (module kernel, novox/hq issue 746).\n# Written by the mesh from this machine's setting transparent-huge-pages and run by\n# mesh-kernel-huge-pages.service: `apply` at boot and whenever the setting changes, `restore` when the\n# unit stops (the module unassigned, or the machine shutting down). Replaced on every push; change the\n# setting, never this file.\n#\n# huge-pages apply|restore [ []]\n#\n# The two directories are arguments only so the module's tests can run this against a stand-in tree.\nset -eu\n\nwant='${setting:transparent-huge-pages}'\nsys=/sys/kernel/mm/transparent_hugepage\nstate='${dir:state}'\n[ $# -ge 2 ] && sys=$2\n[ $# -ge 3 ] && state=$3\n\ncase \"$want\" in\nalways | madvise | never) ;;\n*)\n\techo \"the setting transparent-huge-pages is '$want'; the kernel takes always, madvise or never\" >&2\n\texit 2\n\t;;\nesac\n[ -f \"$sys/enabled\" ] || {\n\techo \"this kernel has no $sys/enabled: it was built without transparent huge pages, so there is nothing to set\" >&2\n\texit 1\n}\n\n# The mode in force is the word the kernel brackets: \"always [madvise] never\". A file holding one bare\n# word (the stand-in tree of the tests) is read as that word.\nmode() { sed -n -e 's/.*\\[\\([a-z]*\\)\\].*/\\1/p' -e t -e 's/^ *\\([a-z]*\\) *$/\\1/p' \"$sys/enabled\"; }\n\n[ $# -ge 1 ] || set -- usage\ncase \"$1\" in\napply)\n\tnow=$(mode)\n\t# What the machine had before the mesh first set it, kept once, so `restore` gives it back.\n\t[ -s \"$state/found\" ] || printf '%s\\n' \"$now\" >\"$state/found\"\n\tif [ \"$now\" = \"$want\" ]; then\n\t\techo \"transparent huge pages: $want, as set\"\n\t\texit 0\n\tfi\n\tprintf '%s\\n' \"$want\" >\"$sys/enabled\"\n\tgot=$(mode)\n\t[ \"$got\" = \"$want\" ] || {\n\t\techo \"wrote $want to $sys/enabled and the kernel says $got\" >&2\n\t\texit 1\n\t}\n\techo \"transparent huge pages: $now -> $want\"\n\t# A file folio already in the page cache keeps its size: the active journal files, mapped by journald\n\t# and by every reader, stay in 2 MiB folios until they are archived (issue 746). Rotating starts new\n\t# files that are mapped under the new mode. A failed rotation is said and does not fail the setting.\n\tif [ \"$want\" != always ] && command -v journalctl >/dev/null 2>&1; then\n\t\tjournalctl --rotate || echo \"the journal could not be rotated: its active files keep their huge pages until they are archived\" >&2\n\tfi\n\t;;\nrestore)\n\tfound=$(cat \"$state/found\" 2>/dev/null || true)\n\tcase \"$found\" in\n\talways | madvise | never)\n\t\tprintf '%s\\n' \"$found\" >\"$sys/enabled\"\n\t\techo \"transparent huge pages: back to $found, as the machine had them\"\n\t\t;;\n\t*) echo \"transparent huge pages: nothing recorded as found, left at $(mode)\" ;;\n\tesac\n\t;;\n*)\n\techo \"usage: huge-pages apply|restore [ []]\" >&2\n\texit 2\n\t;;\nesac\n" + }, + { + "id": "unit", + "type": "file", + "path": "/etc/systemd/system/mesh-kernel-huge-pages.service", + "mode": "0644", + "content": "[Unit]\nDescription=Transparent huge pages as the mesh sets them on this machine\n# Module kernel, novox/hq issue 746. Starting this unit sets the mode from the setting\n# transparent-huge-pages; stopping it gives back the mode the machine had before. It runs early in boot,\n# before the services that fill the page cache, and again whenever the setting changes.\nDefaultDependencies=no\nAfter=local-fs.target\nBefore=sysinit.target shutdown.target\nConflicts=shutdown.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=${dir:state}/huge-pages apply\nExecStop=${dir:state}/huge-pages restore\n\n[Install]\nWantedBy=sysinit.target\n" + }, + { + "id": "apply", + "type": "service", + "unit": "mesh-kernel-huge-pages.service", + "state": "running", + "boot": "enabled", + "restart-on": [ + "huge-pages", + "unit" + ], + "health": { + "kind": "unit" + } + } + ], + "build": { + "artifacts": [ + { + "name": "tools", + "kind": "bundle", + "language": "go", + "system": "arch", + "from": "cmd/kernel-tools", + "binary": "kernel-tools", + "loads": [ + "kernel-tools" + ] + } + ] + } +} From ab1811d00357a95329870b0c3a7614168a12480b Mon Sep 17 00:00:00 2001 From: jochen Date: Wed, 14 Oct 2026 23:42:49 +0200 Subject: [PATCH 2/2] Reload, not restart, the huge pages unit when the setting changes (hq issue 746) A restart ran restore then apply, so every change passed back through the found mode and rotated the journal again. Reload applies the new value directly; restore also drops its state; no rotation while the machine boots; the unit is sandboxed. --- modules/kernel/README.md | 14 ++-- .../kernel/cmd/kernel-tools/manifest_test.go | 78 ++++++++++++++++--- modules/kernel/module.json | 8 +- 3 files changed, 81 insertions(+), 19 deletions(-) diff --git a/modules/kernel/README.md b/modules/kernel/README.md index faf4cc94..dddcebd4 100644 --- a/modules/kernel/README.md +++ b/modules/kernel/README.md @@ -22,8 +22,8 @@ lost its lease four times in 30 minutes. With `madvise`, only memory a program a |---|---|---| | setting | `transparent-huge-pages` | `always`, `madvise` or `never`. Default `always`, what the distribution's kernel boots with, so assigning the module changes nothing until a machine sets another value | | file | `${dir:state}/huge-pages` | the script, written from the setting: `apply` writes the mode to `/sys/kernel/mm/transparent_hugepage/enabled` and reads it back (a value that does not take fails the unit); `restore` gives back the mode found before the module first set it, kept in `${dir:state}/found` | -| file | `/etc/systemd/system/mesh-kernel-huge-pages.service` | oneshot, early in boot (`sysinit.target`, before the services that fill the page cache); stopping it runs `restore` | -| service | `mesh-kernel-huge-pages.service` | running and enabled, restarted when the script (so the setting) or the unit changes; its health is the unit's | +| file | `/etc/systemd/system/mesh-kernel-huge-pages.service` | oneshot, early in boot (`sysinit.target`, before the services that fill the page cache); reloading it runs `apply`, stopping it runs `restore`, which also removes `found`. Sandboxed: the file system read-only but for the module's state (`/sys` stays writable), no home, no network, no new privileges; the journal's rotation goes over its local socket, which a private network leaves reachable | +| service | `mesh-kernel-huge-pages.service` | running and enabled; **reloaded** when the script (so the setting) changes, so a new value is set directly and never passes back through the found one; restarted only when the unit changes; its health is the unit's | The per-size controls (`hugepages-kB/enabled`) are left alone: on the machines read, 2048 kB says `inherit` (it follows the mode) and every other size `never`. `kernel_huge_pages` shows each one. @@ -31,8 +31,9 @@ The per-size controls (`hugepages-kB/enabled`) are left alone: on the machine **A folio already in the page cache keeps its size.** A new mode applies to pages faulted or collapsed from then on. The active journal files would keep their 2 MiB folios until archived, so `apply` rotates the journal (`journalctl --rotate`) whenever it moves the mode away from `always`: new files are mapped -under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. At -boot the kernel starts at `always`, so a machine set to `madvise` rotates its journal once per boot. +under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. No +rotation is made while the machine is still starting (`systemctl is-system-running` says `initializing` +or `starting`): nothing is in huge pages yet, and the persistent journal opens after this unit. ## Tools @@ -43,8 +44,9 @@ boot the kernel starts at `always`, so a machine set to `madvise` rotates its jo ## Tests `go test ./...`: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the -manifest's shape (the default, the order, the unit, restart and health); and the script as the mesh +manifest's shape (the default, the order, the unit and its sandbox, reload, restart and health); and the script as the mesh writes it, run with `sh` against a stand-in directory: it sets and reads back, keeps what it found once, -rotates the journal once and not for `always`, gives back what it found, refuses a value the kernel +rotates the journal once and not for `always` nor while the machine starts, goes straight to a new value on a +reload without rewriting what it found, gives back what it found and then keeps nothing, refuses a value the kernel does not take before writing anything, fails when the write does not take, and says when the kernel has no transparent huge pages. diff --git a/modules/kernel/cmd/kernel-tools/manifest_test.go b/modules/kernel/cmd/kernel-tools/manifest_test.go index fd1a0327..bcdc42e3 100644 --- a/modules/kernel/cmd/kernel-tools/manifest_test.go +++ b/modules/kernel/cmd/kernel-tools/manifest_test.go @@ -2,8 +2,9 @@ package main // The module's shape (novox/hq issue 746): the setting's default is what the kernel boots with, so // assigning the module changes nothing until a machine sets another value; the script is written into the -// module's own state and run by a oneshot unit that is restarted whenever the script (and so the setting) -// or the unit changes; and the script itself sets, verifies, gives back and refuses as it says. +// module's own state and run by a sandboxed oneshot unit that is reloaded (applied again) when the script, +// and so the setting, changes and restarted only when the unit does; and the script itself sets, verifies, +// gives back and refuses as it says. import ( "os" @@ -34,7 +35,9 @@ func TestTheDefaultChangesNothingAndTheUnitFollowsTheSetting(t *testing.T) { } unit := m.resource(t, "unit")["content"].(string) for _, line := range []string{"Type=oneshot", "RemainAfterExit=yes", "ExecStart=${dir:state}/huge-pages apply", - "ExecStop=${dir:state}/huge-pages restore", "WantedBy=sysinit.target", "DefaultDependencies=no"} { + "ExecReload=${dir:state}/huge-pages apply", "ExecStop=${dir:state}/huge-pages restore", "WantedBy=sysinit.target", + "DefaultDependencies=no", "ProtectSystem=strict", "ReadWritePaths=${dir:state}", "ProtectHome=yes", "PrivateNetwork=yes", + "NoNewPrivileges=yes"} { if !strings.Contains(unit, line+"\n") { t.Errorf("the unit lacks %q", line) } @@ -43,9 +46,14 @@ func TestTheDefaultChangesNothingAndTheUnitFollowsTheSetting(t *testing.T) { if apply["unit"] != "mesh-kernel-huge-pages.service" || apply["state"] != "running" || apply["boot"] != "enabled" { t.Fatalf("apply: %v", apply) } - if on := apply["restart-on"].([]any); len(on) != 2 || on[0] != "huge-pages" || on[1] != "unit" { + // A new value reloads (apply again); only a new unit restarts, whose stop would pass back through the + // mode the machine was found with. + if on := apply["restart-on"].([]any); len(on) != 1 || on[0] != "unit" { t.Fatalf("restart-on: %v", on) } + if on := apply["reload-on"].([]any); len(on) != 1 || on[0] != "huge-pages" { + t.Fatalf("reload-on: %v", on) + } if h := apply["health"].(map[string]any); h["kind"] != "unit" { t.Fatalf("health: %v", h) } @@ -54,12 +62,17 @@ func TestTheDefaultChangesNothingAndTheUnitFollowsTheSetting(t *testing.T) { // bed is the script as the mesh writes it for one value of the setting, a stand-in transparent_hugepage // directory, the module's state, and a PATH whose journalctl records that it was asked to rotate. type bed struct { - script, sys, state, rotated, path string + script, sys, state, rotated, path, bin string } func newBed(t *testing.T, want, found string) bed { + return newBedIn(t, t.TempDir(), want, found) +} + +// newBedIn writes the script for want into dir; a second bed in the same dir shares the machine and the +// state, as a reload with a new value does. +func newBedIn(t *testing.T, dir, want, found string) bed { t.Helper() - dir := t.TempDir() b := bed{script: filepath.Join(dir, "huge-pages"), sys: filepath.Join(dir, "thp"), state: filepath.Join(dir, "state"), rotated: filepath.Join(dir, "rotated")} content := manifest(t).resource(t, "huge-pages")["content"].(string) @@ -75,12 +88,22 @@ func newBed(t *testing.T, want, found string) bed { } } write(t, b.script, content, 0o755) - write(t, filepath.Join(b.sys, "enabled"), found, 0o644) + if found != "" { + write(t, filepath.Join(b.sys, "enabled"), found, 0o644) + } write(t, filepath.Join(bin, "journalctl"), "#!/bin/sh\necho \"$@\" >>"+b.rotated+"\n", 0o755) + b.bin = bin + b.booted(t, "running") b.path = bin + string(os.PathListSeparator) + os.Getenv("PATH") return b } +// booted makes the service manager say state for is-system-running. +func (b bed) booted(t *testing.T, state string) { + t.Helper() + write(t, filepath.Join(b.bin, "systemctl"), "#!/bin/sh\n[ \"$1\" = is-system-running ] && echo "+state+"\n", 0o755) +} + func write(t *testing.T, p, content string, mode os.FileMode) { t.Helper() if err := os.WriteFile(p, []byte(content), mode); err != nil { @@ -134,6 +157,44 @@ func TestApplySetsTheModeRotatesTheJournalOnceAndRestoreGivesBackWhatWasFound(t if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "always\n" { t.Fatalf("restored: %q", got) } + // Nothing of the module is kept once it is stopped. + if _, err := os.Stat(filepath.Join(b.state, "found")); !os.IsNotExist(err) { + t.Fatalf("found kept after restore: %v", err) + } +} + +func TestAReloadWithANewValueGoesStraightThereAndNeverRewritesFound(t *testing.T) { + dir := t.TempDir() + b := newBedIn(t, dir, "madvise", "[always] madvise never\n") + if out, err := b.run(t, "apply"); err != nil { + t.Fatalf("apply: %v %s", err, out) + } + // The setting changes to never: the mesh rewrites the script and reloads the unit, which applies again. + n := newBedIn(t, dir, "never", "") + out, err := n.run(t, "apply") + if err != nil || !strings.Contains(out, "madvise -> never") { + t.Fatalf("reload: %v %s", err, out) + } + if got := n.file(t, filepath.Join(n.state, "found")); got != "always\n" { + t.Fatalf("found rewritten by a reload: %q", got) + } +} + +func TestNoRotationWhileTheMachineIsStarting(t *testing.T) { + for _, state := range []string{"initializing", "starting"} { + b := newBed(t, "madvise", "[always] madvise never\n") + b.booted(t, state) + out, err := b.run(t, "apply") + if err != nil || !strings.Contains(out, "still starting") { + t.Fatalf("%s: %v %s", state, err, out) + } + if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "madvise\n" { + t.Fatalf("%s: not set: %q", state, got) + } + if got := b.file(t, b.rotated); got != "" { + t.Fatalf("%s: rotated at boot: %q", state, got) + } + } } func TestAlwaysIsSetWithoutRotating(t *testing.T) { @@ -172,9 +233,6 @@ func TestAWriteThatDoesNotTakeFailsTheUnit(t *testing.T) { func TestAKernelWithoutHugePagesIsSaid(t *testing.T) { b := newBed(t, "madvise", "") - if err := os.Remove(filepath.Join(b.sys, "enabled")); err != nil { - t.Fatal(err) - } if out, err := b.run(t, "apply"); err == nil || !strings.Contains(out, "built without transparent huge pages") { t.Fatalf("%v %s", err, out) } diff --git a/modules/kernel/module.json b/modules/kernel/module.json index 68645769..15077979 100644 --- a/modules/kernel/module.json +++ b/modules/kernel/module.json @@ -27,14 +27,14 @@ "path": "${dir:state}/huge-pages", "mode": "0755", "trusted": true, - "content": "#!/bin/sh\n# Transparent huge pages as the mesh sets them on this machine (module kernel, novox/hq issue 746).\n# Written by the mesh from this machine's setting transparent-huge-pages and run by\n# mesh-kernel-huge-pages.service: `apply` at boot and whenever the setting changes, `restore` when the\n# unit stops (the module unassigned, or the machine shutting down). Replaced on every push; change the\n# setting, never this file.\n#\n# huge-pages apply|restore [ []]\n#\n# The two directories are arguments only so the module's tests can run this against a stand-in tree.\nset -eu\n\nwant='${setting:transparent-huge-pages}'\nsys=/sys/kernel/mm/transparent_hugepage\nstate='${dir:state}'\n[ $# -ge 2 ] && sys=$2\n[ $# -ge 3 ] && state=$3\n\ncase \"$want\" in\nalways | madvise | never) ;;\n*)\n\techo \"the setting transparent-huge-pages is '$want'; the kernel takes always, madvise or never\" >&2\n\texit 2\n\t;;\nesac\n[ -f \"$sys/enabled\" ] || {\n\techo \"this kernel has no $sys/enabled: it was built without transparent huge pages, so there is nothing to set\" >&2\n\texit 1\n}\n\n# The mode in force is the word the kernel brackets: \"always [madvise] never\". A file holding one bare\n# word (the stand-in tree of the tests) is read as that word.\nmode() { sed -n -e 's/.*\\[\\([a-z]*\\)\\].*/\\1/p' -e t -e 's/^ *\\([a-z]*\\) *$/\\1/p' \"$sys/enabled\"; }\n\n[ $# -ge 1 ] || set -- usage\ncase \"$1\" in\napply)\n\tnow=$(mode)\n\t# What the machine had before the mesh first set it, kept once, so `restore` gives it back.\n\t[ -s \"$state/found\" ] || printf '%s\\n' \"$now\" >\"$state/found\"\n\tif [ \"$now\" = \"$want\" ]; then\n\t\techo \"transparent huge pages: $want, as set\"\n\t\texit 0\n\tfi\n\tprintf '%s\\n' \"$want\" >\"$sys/enabled\"\n\tgot=$(mode)\n\t[ \"$got\" = \"$want\" ] || {\n\t\techo \"wrote $want to $sys/enabled and the kernel says $got\" >&2\n\t\texit 1\n\t}\n\techo \"transparent huge pages: $now -> $want\"\n\t# A file folio already in the page cache keeps its size: the active journal files, mapped by journald\n\t# and by every reader, stay in 2 MiB folios until they are archived (issue 746). Rotating starts new\n\t# files that are mapped under the new mode. A failed rotation is said and does not fail the setting.\n\tif [ \"$want\" != always ] && command -v journalctl >/dev/null 2>&1; then\n\t\tjournalctl --rotate || echo \"the journal could not be rotated: its active files keep their huge pages until they are archived\" >&2\n\tfi\n\t;;\nrestore)\n\tfound=$(cat \"$state/found\" 2>/dev/null || true)\n\tcase \"$found\" in\n\talways | madvise | never)\n\t\tprintf '%s\\n' \"$found\" >\"$sys/enabled\"\n\t\techo \"transparent huge pages: back to $found, as the machine had them\"\n\t\t;;\n\t*) echo \"transparent huge pages: nothing recorded as found, left at $(mode)\" ;;\n\tesac\n\t;;\n*)\n\techo \"usage: huge-pages apply|restore [ []]\" >&2\n\texit 2\n\t;;\nesac\n" + "content": "#!/bin/sh\n# Transparent huge pages as the mesh sets them on this machine (module kernel, novox/hq issue 746).\n# Written by the mesh from this machine's setting transparent-huge-pages and run by\n# mesh-kernel-huge-pages.service: `apply` at boot and whenever the setting changes, `restore` when the\n# unit stops (the module unassigned, or the machine shutting down). Replaced on every push; change the\n# setting, never this file. A change of the setting reloads the unit, which runs `apply` again: the mode\n# moves straight to the new value and never passes back through the one the machine was found with.\n#\n# huge-pages apply|restore [ []]\n#\n# The two directories are arguments only so the module's tests can run this against a stand-in tree.\nset -eu\n\nwant='${setting:transparent-huge-pages}'\nsys=/sys/kernel/mm/transparent_hugepage\nstate='${dir:state}'\n[ $# -ge 2 ] && sys=$2\n[ $# -ge 3 ] && state=$3\n\ncase \"$want\" in\nalways | madvise | never) ;;\n*)\n\techo \"the setting transparent-huge-pages is '$want'; the kernel takes always, madvise or never\" >&2\n\texit 2\n\t;;\nesac\n[ -f \"$sys/enabled\" ] || {\n\techo \"this kernel has no $sys/enabled: it was built without transparent huge pages, so there is nothing to set\" >&2\n\texit 1\n}\n\n# The mode in force is the word the kernel brackets: \"always [madvise] never\". A file holding one bare\n# word (the stand-in tree of the tests) is read as that word.\nmode() { sed -n -e 's/.*\\[\\([a-z]*\\)\\].*/\\1/p' -e t -e 's/^ *\\([a-z]*\\) *$/\\1/p' \"$sys/enabled\"; }\n\n[ $# -ge 1 ] || set -- usage\ncase \"$1\" in\napply)\n\tnow=$(mode)\n\t# What the machine had before the mesh first set it, kept once, so `restore` gives it back.\n\t[ -s \"$state/found\" ] || printf '%s\\n' \"$now\" >\"$state/found\"\n\tif [ \"$now\" = \"$want\" ]; then\n\t\techo \"transparent huge pages: $want, as set\"\n\t\texit 0\n\tfi\n\tprintf '%s\\n' \"$want\" >\"$sys/enabled\"\n\tgot=$(mode)\n\t[ \"$got\" = \"$want\" ] || {\n\t\techo \"wrote $want to $sys/enabled and the kernel says $got\" >&2\n\t\texit 1\n\t}\n\techo \"transparent huge pages: $now -> $want\"\n\t# A file folio already in the page cache keeps its size: the active journal files, mapped by journald\n\t# and by every reader, stay in 2 MiB folios until they are archived (issue 746). Rotating starts new\n\t# files that are mapped under the new mode. A failed rotation is said and does not fail the setting.\n\t# At boot nothing is in huge folios yet, and the persistent journal opens only after this unit: no rotation\n\t# while the service manager is still bringing the machine up.\n\tbooting=$(systemctl is-system-running 2>/dev/null || true)\n\tcase \"$booting\" in\n\tinitializing | starting) echo \"the machine is still starting: nothing to rotate yet\" ;;\n\t*)\n\t\tif [ \"$want\" != always ] && command -v journalctl >/dev/null 2>&1; then\n\t\t\tjournalctl --rotate || echo \"the journal could not be rotated: its active files keep their huge pages until they are archived\" >&2\n\t\tfi\n\t\t;;\n\tesac\n\t;;\nrestore)\n\tfound=$(cat \"$state/found\" 2>/dev/null || true)\n\tcase \"$found\" in\n\talways | madvise | never)\n\t\tprintf '%s\\n' \"$found\" >\"$sys/enabled\"\n\t\techo \"transparent huge pages: back to $found, as the machine had them\"\n\t\t;;\n\t*) echo \"transparent huge pages: nothing recorded as found, left at $(mode)\" ;;\n\tesac\n\t# Unassigned or shut down: nothing of the module is kept; the next start records what it finds again.\n\trm -f \"$state/found\"\n\t;;\n*)\n\techo \"usage: huge-pages apply|restore [ []]\" >&2\n\texit 2\n\t;;\nesac\n" }, { "id": "unit", "type": "file", "path": "/etc/systemd/system/mesh-kernel-huge-pages.service", "mode": "0644", - "content": "[Unit]\nDescription=Transparent huge pages as the mesh sets them on this machine\n# Module kernel, novox/hq issue 746. Starting this unit sets the mode from the setting\n# transparent-huge-pages; stopping it gives back the mode the machine had before. It runs early in boot,\n# before the services that fill the page cache, and again whenever the setting changes.\nDefaultDependencies=no\nAfter=local-fs.target\nBefore=sysinit.target shutdown.target\nConflicts=shutdown.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=${dir:state}/huge-pages apply\nExecStop=${dir:state}/huge-pages restore\n\n[Install]\nWantedBy=sysinit.target\n" + "content": "[Unit]\nDescription=Transparent huge pages as the mesh sets them on this machine\n# Module kernel, novox/hq issue 746. Starting this unit sets the mode from the setting\n# transparent-huge-pages; reloading it (the setting changed) sets the new mode; stopping it gives back the\n# mode the machine had before. It runs early in boot, before the services that fill the page cache.\nDefaultDependencies=no\nAfter=local-fs.target\nBefore=sysinit.target shutdown.target\nConflicts=shutdown.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=${dir:state}/huge-pages apply\nExecReload=${dir:state}/huge-pages apply\nExecStop=${dir:state}/huge-pages restore\n# It writes one file under /sys and its own state, and asks the journal daemon to rotate over its socket.\nProtectSystem=strict\nReadWritePaths=${dir:state}\nProtectHome=yes\nPrivateNetwork=yes\nNoNewPrivileges=yes\n\n[Install]\nWantedBy=sysinit.target\n" }, { "id": "apply", @@ -43,9 +43,11 @@ "state": "running", "boot": "enabled", "restart-on": [ - "huge-pages", "unit" ], + "reload-on": [ + "huge-pages" + ], "health": { "kind": "unit" }