Add the kernel module: transparent huge pages set per machine (hq issue 746, ADR 0369) #310

Merged
mesh-admin merged 2 commits from fix/746-huge-pages-madvise into main 2026-10-14 21:52:50 +00:00
10 changed files with 855 additions and 0 deletions
Showing only changes of commit 3a84044ece - Show all commits
+50
View File
@@ -0,0 +1,50 @@
# kernel
The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). Today it holds one:
**transparent huge pages**, set per machine. The rest of the row (the kernel package, microcode, boot
entries) is not built yet.
## Why (novox/hq issue 746)
With transparent huge pages `always`, the page cache of a file mapped by a program is collapsed into
2 MiB folios (khugepaged), and a small write into such a folio dirties all of it: writeback then writes
2 MiB. journald writes its journal files through mmap, and every journal reader (fail2ban's journal
jails among them) maps them too. Measured on the control node on 2026-10-14: journald's cgroup had
written 2.5 TB in 12 days while the journal grew by about 25-30 KB/s, fail2ban's cgroup 63 GB in 4 h
while its own files did not grow, both holding hundreds of MiB of file pages in huge folios; the bursts
saturated the mirrored spinning root disk that holds the bus store and the store, and the controller
lost its lease four times in 30 minutes. With `madvise`, only memory a program asks to have in huge pages
(`madvise(MADV_HUGEPAGE)`) gets them; a database or a JVM that wants them still asks.
## What it owns
| | what | notes |
|---|---|---|
| setting | `transparent-huge-pages` | `always`, `madvise` or `never`. Default `always`, what the distribution's kernel boots with, so assigning the module changes nothing until a machine sets another value |
| file | `${dir:state}/huge-pages` | the script, written from the setting: `apply` writes the mode to `/sys/kernel/mm/transparent_hugepage/enabled` and reads it back (a value that does not take fails the unit); `restore` gives back the mode found before the module first set it, kept in `${dir:state}/found` |
| file | `/etc/systemd/system/mesh-kernel-huge-pages.service` | oneshot, early in boot (`sysinit.target`, before the services that fill the page cache); stopping it runs `restore` |
| service | `mesh-kernel-huge-pages.service` | running and enabled, restarted when the script (so the setting) or the unit changes; its health is the unit's |
The per-size controls (`hugepages-<n>kB/enabled`) are left alone: on the machines read, 2048 kB says
`inherit` (it follows the mode) and every other size `never`. `kernel_huge_pages` shows each one.
**A folio already in the page cache keeps its size.** A new mode applies to pages faulted or collapsed
from then on. The active journal files would keep their 2 MiB folios until archived, so `apply` rotates
the journal (`journalctl --rotate`) whenever it moves the mode away from `always`: new files are mapped
under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. At
boot the kernel starts at `always`, so a machine set to `madvise` rotates its journal once per boot.
## Tools
| tool | | answers |
|---|---|---|
| `kernel_huge_pages` | r | the mode, defrag and shared-memory mode; each per-size control and what `inherit` resolves to; huge pages by kind from `/proc/meminfo` (MiB); the units holding the most file pages in huge folios (`memory.stat` `file_thp`) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (`io.stat`, the most any one device saw: a mirror and its members each count the same write) |
## Tests
`go test ./...`: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the
manifest's shape (the default, the order, the unit, restart and health); and the script as the mesh
writes it, run with `sh` against a stand-in directory: it sets and reads back, keeps what it found once,
rotates the journal once and not for `always`, gives back what it found, refuses a value the kernel
does not take before writing anything, fails when the write does not take, and says when the kernel has
no transparent huge pages.
@@ -0,0 +1,241 @@
package main
import (
"fmt"
"os"
"path"
"sort"
"strconv"
"strings"
)
// THPDir is where the kernel says and takes its transparent huge pages.
const THPDir = "sys/kernel/mm/transparent_hugepage"
// CgroupRoot is the unified cgroup hierarchy.
const CgroupRoot = "sys/fs/cgroup"
// HugePagesAnswer is what kernel_huge_pages answers.
type HugePagesAnswer struct {
// Mode is the word the kernel brackets in enabled: always, madvise or never.
Mode string `json:"mode"`
Defrag string `json:"defrag,omitempty"`
Shmem string `json:"shmem_enabled,omitempty"`
PerSize []Size `json:"per_size,omitempty"`
// Meminfo is the machine's huge pages by kind, in MiB.
Meminfo map[string]float64 `json:"meminfo_mib,omitempty"`
Units []Unit `json:"units"`
Said []string `json:"said"`
}
// Size is one per-size control (hugepages-<n>kB/enabled).
type Size struct {
Size string `json:"size"`
Enabled string `json:"enabled"`
// Means is what inherit resolves to: the top-level mode.
Means string `json:"means"`
}
// Unit is one cgroup holding file pages in huge folios.
type Unit struct {
Cgroup string `json:"cgroup"`
FileTHP int64 `json:"file_thp_bytes"`
FileDirty int64 `json:"file_dirty_bytes"`
FileWriteback int64 `json:"file_writeback_bytes"`
FileMapped int64 `json:"file_mapped_bytes"`
CollapsedByKhug int64 `json:"thp_collapse_alloc"`
Written int64 `json:"written_bytes"`
}
// bracketed is the word a sysfs choice file brackets: "always [madvise] never" is madvise.
func bracketed(text string) string {
open := strings.IndexByte(text, '[')
end := strings.IndexByte(text, ']')
if open < 0 || end < open {
return ""
}
return text[open+1 : end]
}
// HugePages reads the mode, the per-size controls, meminfo and the top units by file_thp.
func (m *Machine) HugePages(units int) (*HugePagesAnswer, error) {
enabled, err := m.read(THPDir + "/enabled")
if err != nil {
if os.IsNotExist(err) {
return nil, fmt.Errorf("this kernel has no /%s/enabled: it was built without transparent huge pages", THPDir)
}
return nil, fmt.Errorf("reading /%s/enabled: %w", THPDir, err)
}
a := &HugePagesAnswer{Mode: bracketed(enabled), Units: []Unit{}}
if s, err := m.read(THPDir + "/defrag"); err == nil {
a.Defrag = bracketed(s)
}
if s, err := m.read(THPDir + "/shmem_enabled"); err == nil {
a.Shmem = bracketed(s)
}
a.Said = append(a.Said, fmt.Sprintf("transparent huge pages: %s, defrag %s", a.Mode, orUnknown(a.Defrag)))
sizes, _ := m.dirs(THPDir)
for _, d := range sizes {
if !strings.HasPrefix(d, "hugepages-") {
continue
}
s, err := m.read(THPDir + "/" + d + "/enabled")
if err != nil {
continue
}
sz := Size{Size: strings.TrimPrefix(d, "hugepages-"), Enabled: bracketed(s)}
sz.Means = sz.Enabled
if sz.Enabled == "inherit" {
sz.Means = a.Mode
}
a.PerSize = append(a.PerSize, sz)
}
sort.Slice(a.PerSize, func(i, j int) bool { return kib(a.PerSize[i].Size) < kib(a.PerSize[j].Size) })
var on []string
for _, s := range a.PerSize {
if s.Means != "never" {
on = append(on, s.Size+" "+s.Means)
}
}
if len(a.PerSize) > 0 {
a.Said = append(a.Said, "sizes not never: "+orNone(on))
}
if mi, err := m.read("proc/meminfo"); err == nil {
a.Meminfo = meminfo(mi)
a.Said = append(a.Said, fmt.Sprintf("huge pages held: anonymous %.0f MiB, shared memory %.0f MiB, file %.0f MiB (mapped at 2 MiB %.0f MiB)",
a.Meminfo["AnonHugePages"], a.Meminfo["ShmemHugePages"], a.Meminfo["FileHugePages"], a.Meminfo["FilePmdMapped"]))
}
if units > 0 {
all := m.cgroups()
sort.Slice(all, func(i, j int) bool {
if all[i].FileTHP != all[j].FileTHP {
return all[i].FileTHP > all[j].FileTHP
}
return all[i].Cgroup < all[j].Cgroup
})
if len(all) > units {
all = all[:units]
}
a.Units = all
for _, u := range all {
a.Said = append(a.Said, fmt.Sprintf("%s: file in huge folios %s, dirty %s, writeback %s, written %s since it started",
u.Cgroup, mib(u.FileTHP), mib(u.FileDirty), mib(u.FileWriteback), mib(u.Written)))
}
if len(all) == 0 {
a.Said = append(a.Said, "no unit holds file pages in huge folios")
}
}
return a, nil
}
// cgroups walks the leaf services and scopes (to three levels below the root) that hold file_thp.
func (m *Machine) cgroups() []Unit {
var out []Unit
var walk func(rel string, depth int)
walk = func(rel string, depth int) {
children, _ := m.dirs(path.Join(CgroupRoot, rel))
for _, c := range children {
p := path.Join(rel, c)
if strings.HasSuffix(c, ".slice") {
if depth < 3 {
walk(p, depth+1)
}
continue
}
if !strings.HasSuffix(c, ".service") && !strings.HasSuffix(c, ".scope") {
continue
}
stat, err := m.read(path.Join(CgroupRoot, p, "memory.stat"))
if err != nil {
continue
}
kv := memoryStat(stat)
if kv["file_thp"] == 0 {
continue
}
u := Unit{Cgroup: p, FileTHP: kv["file_thp"], FileDirty: kv["file_dirty"], FileWriteback: kv["file_writeback"],
FileMapped: kv["file_mapped"], CollapsedByKhug: kv["thp_collapse_alloc"]}
if io, err := m.read(path.Join(CgroupRoot, p, "io.stat")); err == nil {
u.Written = mostWritten(io)
}
out = append(out, u)
}
}
walk("", 0)
return out
}
// memoryStat reads memory.stat's "key value" lines.
func memoryStat(text string) map[string]int64 {
kv := map[string]int64{}
for _, l := range strings.Split(text, "\n") {
f := strings.Fields(l)
if len(f) != 2 {
continue
}
if n, err := strconv.ParseInt(f[1], 10, 64); err == nil {
kv[f[0]] = n
}
}
return kv
}
// mostWritten is the most bytes any one device of io.stat saw written: a mirror's member disks and the
// mirror itself each count the same write, so the lines are never summed.
func mostWritten(text string) int64 {
var most int64
for _, l := range strings.Split(text, "\n") {
for _, f := range strings.Fields(l) {
if v, ok := strings.CutPrefix(f, "wbytes="); ok {
if n, err := strconv.ParseInt(v, 10, 64); err == nil && n > most {
most = n
}
}
}
}
return most
}
// meminfo reads the huge page lines of /proc/meminfo, in MiB.
func meminfo(text string) map[string]float64 {
want := map[string]bool{"AnonHugePages": true, "ShmemHugePages": true, "ShmemPmdMapped": true, "FileHugePages": true, "FilePmdMapped": true}
out := map[string]float64{}
for _, l := range strings.Split(text, "\n") {
k, v, ok := strings.Cut(l, ":")
if !ok || !want[k] {
continue
}
f := strings.Fields(v)
if len(f) == 0 {
continue
}
if n, err := strconv.ParseFloat(f[0], 64); err == nil {
out[k] = n / 1024
}
}
return out
}
func kib(size string) int {
n, _ := strconv.Atoi(strings.TrimSuffix(size, "kB"))
return n
}
func mib(b int64) string { return fmt.Sprintf("%.1f MiB", float64(b)/(1<<20)) }
func orUnknown(s string) string {
if s == "" {
return "unknown"
}
return s
}
func orNone(list []string) string {
if len(list) == 0 {
return "none"
}
return strings.Join(list, ", ")
}
@@ -0,0 +1,113 @@
package main
// Over a stand-in tree holding novox's own files of 2026-10-14 23:25 (read through node-reads for
// novox/hq issue 746): the mode, the per-size controls, meminfo's huge page lines, and fail2ban's and
// journald's memory.stat and io.stat, cut to the lines the tool reads.
import (
"os"
"path/filepath"
"strings"
"testing"
)
func tree(t *testing.T, files map[string]string) *Machine {
t.Helper()
root := t.TempDir()
for p, content := range files {
full := filepath.Join(root, p)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
return &Machine{Root: root}
}
func novox(t *testing.T) *Machine {
thp := "sys/kernel/mm/transparent_hugepage/"
cg := "sys/fs/cgroup/system.slice/"
files := map[string]string{
thp + "enabled": "[always] madvise never\n",
thp + "defrag": "always defer defer+madvise [madvise] never\n",
thp + "shmem_enabled": "always within_size [advise] never deny force\n",
thp + "hugepages-2048kB/enabled": "always [inherit] madvise never\n",
thp + "hugepages-64kB/enabled": "always inherit madvise [never]\n",
thp + "hugepages-1024kB/enabled": "always inherit madvise [never]\n",
"proc/meminfo": "MemTotal: 65748808 kB\nAnonHugePages: 3579904 kB\nShmemHugePages: 4096 kB\n" +
"ShmemPmdMapped: 4096 kB\nFileHugePages: 1693696 kB\nFilePmdMapped: 882688 kB\nHugepagesize: 2048 kB\n",
cg + "fail2ban.service/memory.stat": "anon 21815296\nfile 204623872\nfile_mapped 130736128\nfile_dirty 27262976\n" +
"file_writeback 4096\nanon_thp 0\nfile_thp 197132288\nthp_fault_alloc 0\nthp_collapse_alloc 550\n",
cg + "fail2ban.service/io.stat": "9:0 \n8:16 rbytes=3223552 wbytes=67723415552 rios=240 wios=281159 dbytes=0 dios=0\n" +
"8:0 rbytes=6152192 wbytes=67723415552 rios=832 wios=281159 dbytes=0 dios=0\n" +
"9:2 rbytes=9375744 wbytes=67723415552 rios=1072 wios=281159 dbytes=0 dios=0\n",
cg + "systemd-journald.service/memory.stat": "anon 3227648\nfile 1032015872\nfile_mapped 131928064\nfile_dirty 622592\n" +
"file_writeback 159744\nanon_thp 2097152\nfile_thp 379584512\nthp_fault_alloc 0\nthp_collapse_alloc 11741\n",
cg + "systemd-journald.service/io.stat": "9:0 \n8:16 rbytes=55209984 wbytes=2498768035840 rios=1317 wios=24719004 dbytes=0 dios=0\n" +
"8:0 rbytes=72953856 wbytes=2498768039936 rios=2194 wios=24719005 dbytes=0 dios=0\n" +
"9:2 rbytes=128163840 wbytes=2498724843520 rios=3511 wios=24707873 dbytes=0 dios=0\n",
cg + "sshd.service/memory.stat": "anon 1000\nfile_thp 0\n",
}
return tree(t, files)
}
func TestNovoxBeforeTheChange(t *testing.T) {
a, err := novox(t).HugePages(10)
if err != nil {
t.Fatal(err)
}
if a.Mode != "always" || a.Defrag != "madvise" || a.Shmem != "advise" {
t.Fatalf("modes: %+v", a)
}
if len(a.PerSize) != 3 || a.PerSize[0].Size != "64kB" || a.PerSize[2].Size != "2048kB" ||
a.PerSize[2].Enabled != "inherit" || a.PerSize[2].Means != "always" || a.PerSize[0].Means != "never" {
t.Fatalf("per size: %+v", a.PerSize)
}
if a.Meminfo["FileHugePages"] != 1654 || a.Meminfo["FilePmdMapped"] != 862 {
t.Fatalf("meminfo: %v", a.Meminfo)
}
if len(a.Units) != 2 || a.Units[0].Cgroup != "system.slice/systemd-journald.service" ||
a.Units[1].Cgroup != "system.slice/fail2ban.service" {
t.Fatalf("units: %+v", a.Units)
}
// The mirror and its two members each count the write: the most any one saw, never their sum.
if a.Units[1].Written != 67723415552 || a.Units[1].FileTHP != 197132288 || a.Units[1].CollapsedByKhug != 550 {
t.Fatalf("fail2ban: %+v", a.Units[1])
}
said := strings.Join(a.Said, "\n")
for _, want := range []string{"transparent huge pages: always, defrag madvise", "sizes not never: 2048kB always",
"file 1654 MiB (mapped at 2 MiB 862 MiB)", "system.slice/fail2ban.service: file in huge folios 188.0 MiB"} {
if !strings.Contains(said, want) {
t.Errorf("said lacks %q:\n%s", want, said)
}
}
}
func TestUnitsAreBoundedAndNoneIsSaid(t *testing.T) {
a, err := novox(t).HugePages(1)
if err != nil || len(a.Units) != 1 {
t.Fatalf("%v %+v", err, a)
}
b, err := tree(t, map[string]string{"sys/kernel/mm/transparent_hugepage/enabled": "always [madvise] never\n"}).HugePages(10)
if err != nil || b.Mode != "madvise" || len(b.Units) != 0 || !strings.Contains(strings.Join(b.Said, "\n"), "no unit holds") {
t.Fatalf("%v %+v", err, b)
}
}
func TestAKernelWithoutHugePagesIsAnError(t *testing.T) {
if _, err := tree(t, map[string]string{"proc/meminfo": ""}).HugePages(10); err == nil ||
!strings.Contains(err.Error(), "built without transparent huge pages") {
t.Fatalf("%v", err)
}
}
func TestTheArgumentIsBounded(t *testing.T) {
if _, err := whole(map[string]any{"units": -1.0}, "units", 10, 0, 100); err == nil {
t.Error("a negative count was accepted")
}
if n, _ := whole(map[string]any{"units": 1000.0}, "units", 10, 0, 100); n != 100 {
t.Errorf("not bounded: %d", n)
}
}
@@ -0,0 +1,65 @@
package main
// The machine as the tools see it: the files they read, under a root that tests replace with a stand-in
// tree. Everything read here is readable by the runtime's account; nothing is escalated.
import (
"fmt"
"io/fs"
"os"
"path/filepath"
)
// Machine reads this machine's kernel files under Root ("/" on the machine).
type Machine struct {
Root string
}
// ThisMachine is the machine the runtime launched this bundle on.
func ThisMachine() *Machine { return &Machine{Root: "/"} }
func (m *Machine) path(p string) string { return filepath.Join(m.Root, p) }
func (m *Machine) read(p string) (string, error) {
b, err := os.ReadFile(m.path(p))
return string(b), err
}
func (m *Machine) dirs(p string) ([]string, error) {
entries, err := os.ReadDir(m.path(p))
if err != nil {
return nil, err
}
var out []string
for _, e := range entries {
if e.IsDir() || e.Type()&fs.ModeSymlink != 0 {
out = append(out, e.Name())
}
}
return out, nil
}
// whole is a whole-number argument with a default, kept within bounds.
func whole(args map[string]any, key string, def, least, most int) (int, error) {
raw, present := args[key]
if !present || raw == nil {
return def, nil
}
f, ok := raw.(float64)
if !ok || f != float64(int(f)) {
return 0, fmt.Errorf("%s must be a whole number", key)
}
n := int(f)
if n < least {
return 0, fmt.Errorf("%s must be at least %d", key, least)
}
if n > most {
n = most
}
return n, nil
}
// schema is a tool's input: its properties.
func schema(properties map[string]any) map[string]any {
return map[string]any{"type": "object", "properties": properties}
}
+50
View File
@@ -0,0 +1,50 @@
// The kernel module's tools bundle (novox/hq to-be 42 Phase 1 row 11, issue 746).
//
// Served by the node's runtime over MCP on stdio through the Go SDK (ADR 0188, ADR 0193). It only reads:
// the transparent huge pages the kernel runs with, and which units hold file pages in huge folios, which
// is what a small write to a mapped file costs a whole huge page of writeback for (issue 746).
package main
import (
"fmt"
"os"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// binaryName is what the build names this bundle's executable: the manifest's `binary`.
const binaryName = "kernel-tools"
func main() {
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): kernel.
if err := stdio.Serve("", tools(ThisMachine())); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
func tools(m *Machine) []stdio.Tool {
return []stdio.Tool{
{
Name: "kernel_huge_pages",
Description: "Transparent huge pages on this machine: the mode in force (always, madvise or never) and defrag, " +
"each per-size control with what inherit resolves to, the machine's huge pages by kind (anonymous, " +
"shared memory, file, file mapped at 2 MiB) from /proc/meminfo, and the units holding the most file " +
"pages in huge folios (memory.stat file_thp) with their dirty and writeback bytes and the bytes their " +
"cgroup has written (io.stat, the most any one device saw). A unit with file_thp that writes far more " +
"than its files grow is paying a whole huge page of writeback per small write (novox/hq issue 746). " +
"Replaces cat /sys/kernel/mm/transparent_hugepage/enabled, grep Huge /proc/meminfo and reading " +
"memory.stat by hand. (r)",
Input: schema(map[string]any{
"units": map[string]any{"type": "integer", "description": "how many units to name, the most file_thp first (default 10, 0 for none, at most 100)"},
}),
Run: func(args map[string]any) (any, error) {
n, err := whole(args, "units", 10, 0, 100)
if err != nil {
return nil, err
}
return m.HugePages(n)
},
},
}
}
@@ -0,0 +1,181 @@
package main
// The module's shape (novox/hq issue 746): the setting's default is what the kernel boots with, so
// assigning the module changes nothing until a machine sets another value; the script is written into the
// module's own state and run by a oneshot unit that is restarted whenever the script (and so the setting)
// or the unit changes; and the script itself sets, verifies, gives back and refuses as it says.
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestTheDefaultChangesNothingAndTheUnitFollowsTheSetting(t *testing.T) {
m := manifest(t)
s := m.Settings["transparent-huge-pages"]
if s["kind"] != "preference" || s["default"] != "always" || s["why"] == "" {
t.Fatalf("the setting: %v", s)
}
var ids []string
for _, r := range m.Resources {
ids = append(ids, r["id"].(string))
}
if strings.Join(ids, " ") != "state huge-pages unit apply" {
t.Fatalf("order: %v", ids)
}
script := m.resource(t, "huge-pages")
// The script runs as root, so the setting it is written from is set at the terminal alone (issue 339).
if script["path"] != "${dir:state}/huge-pages" || script["mode"] != "0755" || script["trusted"] != true ||
!strings.Contains(script["content"].(string), "want='${setting:transparent-huge-pages}'") {
t.Fatalf("the script: %v", script["path"])
}
unit := m.resource(t, "unit")["content"].(string)
for _, line := range []string{"Type=oneshot", "RemainAfterExit=yes", "ExecStart=${dir:state}/huge-pages apply",
"ExecStop=${dir:state}/huge-pages restore", "WantedBy=sysinit.target", "DefaultDependencies=no"} {
if !strings.Contains(unit, line+"\n") {
t.Errorf("the unit lacks %q", line)
}
}
apply := m.resource(t, "apply")
if apply["unit"] != "mesh-kernel-huge-pages.service" || apply["state"] != "running" || apply["boot"] != "enabled" {
t.Fatalf("apply: %v", apply)
}
if on := apply["restart-on"].([]any); len(on) != 2 || on[0] != "huge-pages" || on[1] != "unit" {
t.Fatalf("restart-on: %v", on)
}
if h := apply["health"].(map[string]any); h["kind"] != "unit" {
t.Fatalf("health: %v", h)
}
}
// bed is the script as the mesh writes it for one value of the setting, a stand-in transparent_hugepage
// directory, the module's state, and a PATH whose journalctl records that it was asked to rotate.
type bed struct {
script, sys, state, rotated, path string
}
func newBed(t *testing.T, want, found string) bed {
t.Helper()
dir := t.TempDir()
b := bed{script: filepath.Join(dir, "huge-pages"), sys: filepath.Join(dir, "thp"), state: filepath.Join(dir, "state"),
rotated: filepath.Join(dir, "rotated")}
content := manifest(t).resource(t, "huge-pages")["content"].(string)
content = strings.ReplaceAll(content, "${setting:transparent-huge-pages}", want)
content = strings.ReplaceAll(content, "${dir:state}", b.state)
if strings.Contains(content, "${") {
t.Fatalf("a substitution the test does not make is left in the script")
}
bin := filepath.Join(dir, "bin")
for _, d := range []string{b.sys, b.state, bin} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
write(t, b.script, content, 0o755)
write(t, filepath.Join(b.sys, "enabled"), found, 0o644)
write(t, filepath.Join(bin, "journalctl"), "#!/bin/sh\necho \"$@\" >>"+b.rotated+"\n", 0o755)
b.path = bin + string(os.PathListSeparator) + os.Getenv("PATH")
return b
}
func write(t *testing.T, p, content string, mode os.FileMode) {
t.Helper()
if err := os.WriteFile(p, []byte(content), mode); err != nil {
t.Fatal(err)
}
}
func (b bed) run(t *testing.T, verb string) (string, error) {
t.Helper()
cmd := exec.Command("sh", b.script, verb, b.sys, b.state)
cmd.Env = append(os.Environ(), "PATH="+b.path)
out, err := cmd.CombinedOutput()
return string(out), err
}
func (b bed) file(t *testing.T, p string) string {
t.Helper()
raw, err := os.ReadFile(p)
if err != nil {
return ""
}
return string(raw)
}
func TestApplySetsTheModeRotatesTheJournalOnceAndRestoreGivesBackWhatWasFound(t *testing.T) {
// novox's file on 2026-10-14, read through node-reads.
b := newBed(t, "madvise", "[always] madvise never\n")
out, err := b.run(t, "apply")
if err != nil || !strings.Contains(out, "always -> madvise") {
t.Fatalf("apply: %v %s", err, out)
}
if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "madvise\n" {
t.Fatalf("written: %q", got)
}
if got := b.file(t, filepath.Join(b.state, "found")); got != "always\n" {
t.Fatalf("found: %q", got)
}
if got := b.file(t, b.rotated); got != "--rotate\n" {
t.Fatalf("rotation: %q", got)
}
// Applied again (a restart with the same value): nothing written, nothing rotated, found kept.
if out, err := b.run(t, "apply"); err != nil || !strings.Contains(out, "madvise, as set") {
t.Fatalf("again: %v %s", err, out)
}
if got := b.file(t, b.rotated); got != "--rotate\n" {
t.Fatalf("rotated again: %q", got)
}
if out, err := b.run(t, "restore"); err != nil || !strings.Contains(out, "back to always") {
t.Fatalf("restore: %v %s", err, out)
}
if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "always\n" {
t.Fatalf("restored: %q", got)
}
}
func TestAlwaysIsSetWithoutRotating(t *testing.T) {
b := newBed(t, "always", "always [madvise] never\n")
if out, err := b.run(t, "apply"); err != nil || !strings.Contains(out, "madvise -> always") {
t.Fatalf("apply: %v %s", err, out)
}
if got := b.file(t, b.rotated); got != "" {
t.Fatalf("rotated for always: %q", got)
}
}
func TestAValueTheKernelDoesNotTakeIsRefusedBeforeAnythingIsWritten(t *testing.T) {
b := newBed(t, "sometimes", "[always] madvise never\n")
out, err := b.run(t, "apply")
if err == nil || !strings.Contains(out, "takes always, madvise or never") {
t.Fatalf("accepted: %v %s", err, out)
}
if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "[always] madvise never\n" {
t.Fatalf("written anyway: %q", got)
}
}
func TestAWriteThatDoesNotTakeFailsTheUnit(t *testing.T) {
if os.Getuid() == 0 {
t.Skip("root writes a read-only file")
}
b := newBed(t, "madvise", "[always] madvise never\n")
if err := os.Chmod(filepath.Join(b.sys, "enabled"), 0o444); err != nil {
t.Fatal(err)
}
if out, err := b.run(t, "apply"); err == nil {
t.Fatalf("a refused write passed: %s", out)
}
}
func TestAKernelWithoutHugePagesIsSaid(t *testing.T) {
b := newBed(t, "madvise", "")
if err := os.Remove(filepath.Join(b.sys, "enabled")); err != nil {
t.Fatal(err)
}
if out, err := b.run(t, "apply"); err == nil || !strings.Contains(out, "built without transparent huge pages") {
t.Fatalf("%v %s", err, out)
}
}
@@ -0,0 +1,79 @@
package main
import (
"encoding/json"
"os"
"testing"
)
type resource map[string]any
type manifestShape struct {
Module string `json:"module"`
Capabilities []string `json:"capabilities"`
Settings map[string]map[string]any `json:"settings"`
Tools []string `json:"tools"`
Resources []resource `json:"resources"`
Build struct {
Artifacts []map[string]any `json:"artifacts"`
} `json:"build"`
}
func manifest(t *testing.T) manifestShape {
t.Helper()
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m manifestShape
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
return m
}
func (m manifestShape) resource(t *testing.T, id string) resource {
t.Helper()
for _, r := range m.Resources {
if r["id"] == id {
return r
}
}
t.Fatalf("no resource %s", id)
return nil
}
// TestToolsAreTheManifests holds the served tools and the manifest's list to one another, and the
// bundle to the shape the builder compiles and the runtime loads.
func TestToolsAreTheManifests(t *testing.T) {
m := manifest(t)
names := map[string]bool{}
for _, tool := range tools(&Machine{Root: t.TempDir()}) {
if names[tool.Name] {
t.Errorf("%s is served twice", tool.Name)
}
names[tool.Name] = true
}
for _, want := range m.Tools {
if !names[want] {
t.Errorf("the manifest lists %s and the bundle does not serve it", want)
}
delete(names, want)
}
if len(names) != 0 {
t.Errorf("served and not listed: %v", names)
}
var tools map[string]any
for _, a := range m.Build.Artifacts {
if a["name"] == "tools" {
tools = a
}
}
if tools == nil || tools["kind"] != "bundle" || tools["language"] != "go" || tools["system"] != "arch" ||
tools["from"] != "cmd/"+binaryName || tools["binary"] != binaryName {
t.Fatalf("the tools artifact: %v", tools)
}
if loads, _ := tools["loads"].([]any); len(loads) != 1 || loads[0] != binaryName {
t.Fatalf("loads: %v", tools["loads"])
}
}
+5
View File
@@ -0,0 +1,5 @@
module kernel
go 1.22
require git.novox.be/novox/mesh-sdk/go v0.1.7
+2
View File
@@ -0,0 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w=
git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
+69
View File
@@ -0,0 +1,69 @@
{
"module": "kernel",
"version": "1",
"capabilities": [
"service-manager"
],
"settings": {
"transparent-huge-pages": {
"kind": "preference",
"default": "always",
"why": "always is what the distribution's kernel boots with, so assigning the module changes nothing until a machine sets another value; madvise gives huge pages only to programs that ask for them, which stops a small write to a mapped file (the journal) dirtying and writing back a whole 2 MiB page on a machine whose disks are slow (novox/hq issue 746)"
}
},
"tools": [
"kernel_huge_pages"
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "huge-pages",
"type": "file",
"path": "${dir:state}/huge-pages",
"mode": "0755",
"trusted": true,
"content": "#!/bin/sh\n# Transparent huge pages as the mesh sets them on this machine (module kernel, novox/hq issue 746).\n# Written by the mesh from this machine's setting transparent-huge-pages and run by\n# mesh-kernel-huge-pages.service: `apply` at boot and whenever the setting changes, `restore` when the\n# unit stops (the module unassigned, or the machine shutting down). Replaced on every push; change the\n# setting, never this file.\n#\n# huge-pages apply|restore [<the kernel's transparent_hugepage directory> [<this module's state>]]\n#\n# The two directories are arguments only so the module's tests can run this against a stand-in tree.\nset -eu\n\nwant='${setting:transparent-huge-pages}'\nsys=/sys/kernel/mm/transparent_hugepage\nstate='${dir:state}'\n[ $# -ge 2 ] && sys=$2\n[ $# -ge 3 ] && state=$3\n\ncase \"$want\" in\nalways | madvise | never) ;;\n*)\n\techo \"the setting transparent-huge-pages is '$want'; the kernel takes always, madvise or never\" >&2\n\texit 2\n\t;;\nesac\n[ -f \"$sys/enabled\" ] || {\n\techo \"this kernel has no $sys/enabled: it was built without transparent huge pages, so there is nothing to set\" >&2\n\texit 1\n}\n\n# The mode in force is the word the kernel brackets: \"always [madvise] never\". A file holding one bare\n# word (the stand-in tree of the tests) is read as that word.\nmode() { sed -n -e 's/.*\\[\\([a-z]*\\)\\].*/\\1/p' -e t -e 's/^ *\\([a-z]*\\) *$/\\1/p' \"$sys/enabled\"; }\n\n[ $# -ge 1 ] || set -- usage\ncase \"$1\" in\napply)\n\tnow=$(mode)\n\t# What the machine had before the mesh first set it, kept once, so `restore` gives it back.\n\t[ -s \"$state/found\" ] || printf '%s\\n' \"$now\" >\"$state/found\"\n\tif [ \"$now\" = \"$want\" ]; then\n\t\techo \"transparent huge pages: $want, as set\"\n\t\texit 0\n\tfi\n\tprintf '%s\\n' \"$want\" >\"$sys/enabled\"\n\tgot=$(mode)\n\t[ \"$got\" = \"$want\" ] || {\n\t\techo \"wrote $want to $sys/enabled and the kernel says $got\" >&2\n\t\texit 1\n\t}\n\techo \"transparent huge pages: $now -> $want\"\n\t# A file folio already in the page cache keeps its size: the active journal files, mapped by journald\n\t# and by every reader, stay in 2 MiB folios until they are archived (issue 746). Rotating starts new\n\t# files that are mapped under the new mode. A failed rotation is said and does not fail the setting.\n\tif [ \"$want\" != always ] && command -v journalctl >/dev/null 2>&1; then\n\t\tjournalctl --rotate || echo \"the journal could not be rotated: its active files keep their huge pages until they are archived\" >&2\n\tfi\n\t;;\nrestore)\n\tfound=$(cat \"$state/found\" 2>/dev/null || true)\n\tcase \"$found\" in\n\talways | madvise | never)\n\t\tprintf '%s\\n' \"$found\" >\"$sys/enabled\"\n\t\techo \"transparent huge pages: back to $found, as the machine had them\"\n\t\t;;\n\t*) echo \"transparent huge pages: nothing recorded as found, left at $(mode)\" ;;\n\tesac\n\t;;\n*)\n\techo \"usage: huge-pages apply|restore [<transparent_hugepage directory> [<state directory>]]\" >&2\n\texit 2\n\t;;\nesac\n"
},
{
"id": "unit",
"type": "file",
"path": "/etc/systemd/system/mesh-kernel-huge-pages.service",
"mode": "0644",
"content": "[Unit]\nDescription=Transparent huge pages as the mesh sets them on this machine\n# Module kernel, novox/hq issue 746. Starting this unit sets the mode from the setting\n# transparent-huge-pages; stopping it gives back the mode the machine had before. It runs early in boot,\n# before the services that fill the page cache, and again whenever the setting changes.\nDefaultDependencies=no\nAfter=local-fs.target\nBefore=sysinit.target shutdown.target\nConflicts=shutdown.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=${dir:state}/huge-pages apply\nExecStop=${dir:state}/huge-pages restore\n\n[Install]\nWantedBy=sysinit.target\n"
},
{
"id": "apply",
"type": "service",
"unit": "mesh-kernel-huge-pages.service",
"state": "running",
"boot": "enabled",
"restart-on": [
"huge-pages",
"unit"
],
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/kernel-tools",
"binary": "kernel-tools",
"loads": [
"kernel-tools"
]
}
]
}
}