Add the kernel module: transparent huge pages set per machine (hq issue 746)
With THP 'always' journald's mmapped journal files sit in 2 MiB folios, so each small journal write dirties and writes back 2 MiB; on the control node that saturated the root mirror holding the bus store and the store, and the controller lost its lease. The module sets the mode from a per-machine setting (default always, the kernel's own, so assigning it changes nothing until a machine sets madvise), verifies it took, rotates the journal once so new files are mapped under the new mode, gives back what it found when stopped, and serves kernel_huge_pages to measure file_thp per unit.
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
# kernel
|
||||
|
||||
The kernel's run-time settings on a machine (novox/hq to-be 42 Phase 1, row 11). Today it holds one:
|
||||
**transparent huge pages**, set per machine. The rest of the row (the kernel package, microcode, boot
|
||||
entries) is not built yet.
|
||||
|
||||
## Why (novox/hq issue 746)
|
||||
|
||||
With transparent huge pages `always`, the page cache of a file mapped by a program is collapsed into
|
||||
2 MiB folios (khugepaged), and a small write into such a folio dirties all of it: writeback then writes
|
||||
2 MiB. journald writes its journal files through mmap, and every journal reader (fail2ban's journal
|
||||
jails among them) maps them too. Measured on the control node on 2026-10-14: journald's cgroup had
|
||||
written 2.5 TB in 12 days while the journal grew by about 25-30 KB/s, fail2ban's cgroup 63 GB in 4 h
|
||||
while its own files did not grow, both holding hundreds of MiB of file pages in huge folios; the bursts
|
||||
saturated the mirrored spinning root disk that holds the bus store and the store, and the controller
|
||||
lost its lease four times in 30 minutes. With `madvise`, only memory a program asks to have in huge pages
|
||||
(`madvise(MADV_HUGEPAGE)`) gets them; a database or a JVM that wants them still asks.
|
||||
|
||||
## What it owns
|
||||
|
||||
| | what | notes |
|
||||
|---|---|---|
|
||||
| setting | `transparent-huge-pages` | `always`, `madvise` or `never`. Default `always`, what the distribution's kernel boots with, so assigning the module changes nothing until a machine sets another value |
|
||||
| file | `${dir:state}/huge-pages` | the script, written from the setting: `apply` writes the mode to `/sys/kernel/mm/transparent_hugepage/enabled` and reads it back (a value that does not take fails the unit); `restore` gives back the mode found before the module first set it, kept in `${dir:state}/found` |
|
||||
| file | `/etc/systemd/system/mesh-kernel-huge-pages.service` | oneshot, early in boot (`sysinit.target`, before the services that fill the page cache); stopping it runs `restore` |
|
||||
| service | `mesh-kernel-huge-pages.service` | running and enabled, restarted when the script (so the setting) or the unit changes; its health is the unit's |
|
||||
|
||||
The per-size controls (`hugepages-<n>kB/enabled`) are left alone: on the machines read, 2048 kB says
|
||||
`inherit` (it follows the mode) and every other size `never`. `kernel_huge_pages` shows each one.
|
||||
|
||||
**A folio already in the page cache keeps its size.** A new mode applies to pages faulted or collapsed
|
||||
from then on. The active journal files would keep their 2 MiB folios until archived, so `apply` rotates
|
||||
the journal (`journalctl --rotate`) whenever it moves the mode away from `always`: new files are mapped
|
||||
under the new mode. A failed rotation is said in the unit's journal and does not fail the setting. At
|
||||
boot the kernel starts at `always`, so a machine set to `madvise` rotates its journal once per boot.
|
||||
|
||||
## Tools
|
||||
|
||||
| tool | | answers |
|
||||
|---|---|---|
|
||||
| `kernel_huge_pages` | r | the mode, defrag and shared-memory mode; each per-size control and what `inherit` resolves to; huge pages by kind from `/proc/meminfo` (MiB); the units holding the most file pages in huge folios (`memory.stat` `file_thp`) with their dirty and writeback bytes, khugepaged's collapses and the bytes their cgroup wrote (`io.stat`, the most any one device saw: a mirror and its members each count the same write) |
|
||||
|
||||
## Tests
|
||||
|
||||
`go test ./...`: the tool over a stand-in tree holding the control node's own files of 2026-10-14; the
|
||||
manifest's shape (the default, the order, the unit, restart and health); and the script as the mesh
|
||||
writes it, run with `sh` against a stand-in directory: it sets and reads back, keeps what it found once,
|
||||
rotates the journal once and not for `always`, gives back what it found, refuses a value the kernel
|
||||
does not take before writing anything, fails when the write does not take, and says when the kernel has
|
||||
no transparent huge pages.
|
||||
@@ -0,0 +1,241 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// THPDir is where the kernel says and takes its transparent huge pages.
|
||||
const THPDir = "sys/kernel/mm/transparent_hugepage"
|
||||
|
||||
// CgroupRoot is the unified cgroup hierarchy.
|
||||
const CgroupRoot = "sys/fs/cgroup"
|
||||
|
||||
// HugePagesAnswer is what kernel_huge_pages answers.
|
||||
type HugePagesAnswer struct {
|
||||
// Mode is the word the kernel brackets in enabled: always, madvise or never.
|
||||
Mode string `json:"mode"`
|
||||
Defrag string `json:"defrag,omitempty"`
|
||||
Shmem string `json:"shmem_enabled,omitempty"`
|
||||
PerSize []Size `json:"per_size,omitempty"`
|
||||
// Meminfo is the machine's huge pages by kind, in MiB.
|
||||
Meminfo map[string]float64 `json:"meminfo_mib,omitempty"`
|
||||
Units []Unit `json:"units"`
|
||||
Said []string `json:"said"`
|
||||
}
|
||||
|
||||
// Size is one per-size control (hugepages-<n>kB/enabled).
|
||||
type Size struct {
|
||||
Size string `json:"size"`
|
||||
Enabled string `json:"enabled"`
|
||||
// Means is what inherit resolves to: the top-level mode.
|
||||
Means string `json:"means"`
|
||||
}
|
||||
|
||||
// Unit is one cgroup holding file pages in huge folios.
|
||||
type Unit struct {
|
||||
Cgroup string `json:"cgroup"`
|
||||
FileTHP int64 `json:"file_thp_bytes"`
|
||||
FileDirty int64 `json:"file_dirty_bytes"`
|
||||
FileWriteback int64 `json:"file_writeback_bytes"`
|
||||
FileMapped int64 `json:"file_mapped_bytes"`
|
||||
CollapsedByKhug int64 `json:"thp_collapse_alloc"`
|
||||
Written int64 `json:"written_bytes"`
|
||||
}
|
||||
|
||||
// bracketed is the word a sysfs choice file brackets: "always [madvise] never" is madvise.
|
||||
func bracketed(text string) string {
|
||||
open := strings.IndexByte(text, '[')
|
||||
end := strings.IndexByte(text, ']')
|
||||
if open < 0 || end < open {
|
||||
return ""
|
||||
}
|
||||
return text[open+1 : end]
|
||||
}
|
||||
|
||||
// HugePages reads the mode, the per-size controls, meminfo and the top units by file_thp.
|
||||
func (m *Machine) HugePages(units int) (*HugePagesAnswer, error) {
|
||||
enabled, err := m.read(THPDir + "/enabled")
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("this kernel has no /%s/enabled: it was built without transparent huge pages", THPDir)
|
||||
}
|
||||
return nil, fmt.Errorf("reading /%s/enabled: %w", THPDir, err)
|
||||
}
|
||||
a := &HugePagesAnswer{Mode: bracketed(enabled), Units: []Unit{}}
|
||||
if s, err := m.read(THPDir + "/defrag"); err == nil {
|
||||
a.Defrag = bracketed(s)
|
||||
}
|
||||
if s, err := m.read(THPDir + "/shmem_enabled"); err == nil {
|
||||
a.Shmem = bracketed(s)
|
||||
}
|
||||
a.Said = append(a.Said, fmt.Sprintf("transparent huge pages: %s, defrag %s", a.Mode, orUnknown(a.Defrag)))
|
||||
|
||||
sizes, _ := m.dirs(THPDir)
|
||||
for _, d := range sizes {
|
||||
if !strings.HasPrefix(d, "hugepages-") {
|
||||
continue
|
||||
}
|
||||
s, err := m.read(THPDir + "/" + d + "/enabled")
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
sz := Size{Size: strings.TrimPrefix(d, "hugepages-"), Enabled: bracketed(s)}
|
||||
sz.Means = sz.Enabled
|
||||
if sz.Enabled == "inherit" {
|
||||
sz.Means = a.Mode
|
||||
}
|
||||
a.PerSize = append(a.PerSize, sz)
|
||||
}
|
||||
sort.Slice(a.PerSize, func(i, j int) bool { return kib(a.PerSize[i].Size) < kib(a.PerSize[j].Size) })
|
||||
var on []string
|
||||
for _, s := range a.PerSize {
|
||||
if s.Means != "never" {
|
||||
on = append(on, s.Size+" "+s.Means)
|
||||
}
|
||||
}
|
||||
if len(a.PerSize) > 0 {
|
||||
a.Said = append(a.Said, "sizes not never: "+orNone(on))
|
||||
}
|
||||
|
||||
if mi, err := m.read("proc/meminfo"); err == nil {
|
||||
a.Meminfo = meminfo(mi)
|
||||
a.Said = append(a.Said, fmt.Sprintf("huge pages held: anonymous %.0f MiB, shared memory %.0f MiB, file %.0f MiB (mapped at 2 MiB %.0f MiB)",
|
||||
a.Meminfo["AnonHugePages"], a.Meminfo["ShmemHugePages"], a.Meminfo["FileHugePages"], a.Meminfo["FilePmdMapped"]))
|
||||
}
|
||||
|
||||
if units > 0 {
|
||||
all := m.cgroups()
|
||||
sort.Slice(all, func(i, j int) bool {
|
||||
if all[i].FileTHP != all[j].FileTHP {
|
||||
return all[i].FileTHP > all[j].FileTHP
|
||||
}
|
||||
return all[i].Cgroup < all[j].Cgroup
|
||||
})
|
||||
if len(all) > units {
|
||||
all = all[:units]
|
||||
}
|
||||
a.Units = all
|
||||
for _, u := range all {
|
||||
a.Said = append(a.Said, fmt.Sprintf("%s: file in huge folios %s, dirty %s, writeback %s, written %s since it started",
|
||||
u.Cgroup, mib(u.FileTHP), mib(u.FileDirty), mib(u.FileWriteback), mib(u.Written)))
|
||||
}
|
||||
if len(all) == 0 {
|
||||
a.Said = append(a.Said, "no unit holds file pages in huge folios")
|
||||
}
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// cgroups walks the leaf services and scopes (to three levels below the root) that hold file_thp.
|
||||
func (m *Machine) cgroups() []Unit {
|
||||
var out []Unit
|
||||
var walk func(rel string, depth int)
|
||||
walk = func(rel string, depth int) {
|
||||
children, _ := m.dirs(path.Join(CgroupRoot, rel))
|
||||
for _, c := range children {
|
||||
p := path.Join(rel, c)
|
||||
if strings.HasSuffix(c, ".slice") {
|
||||
if depth < 3 {
|
||||
walk(p, depth+1)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !strings.HasSuffix(c, ".service") && !strings.HasSuffix(c, ".scope") {
|
||||
continue
|
||||
}
|
||||
stat, err := m.read(path.Join(CgroupRoot, p, "memory.stat"))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
kv := memoryStat(stat)
|
||||
if kv["file_thp"] == 0 {
|
||||
continue
|
||||
}
|
||||
u := Unit{Cgroup: p, FileTHP: kv["file_thp"], FileDirty: kv["file_dirty"], FileWriteback: kv["file_writeback"],
|
||||
FileMapped: kv["file_mapped"], CollapsedByKhug: kv["thp_collapse_alloc"]}
|
||||
if io, err := m.read(path.Join(CgroupRoot, p, "io.stat")); err == nil {
|
||||
u.Written = mostWritten(io)
|
||||
}
|
||||
out = append(out, u)
|
||||
}
|
||||
}
|
||||
walk("", 0)
|
||||
return out
|
||||
}
|
||||
|
||||
// memoryStat reads memory.stat's "key value" lines.
|
||||
func memoryStat(text string) map[string]int64 {
|
||||
kv := map[string]int64{}
|
||||
for _, l := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(l)
|
||||
if len(f) != 2 {
|
||||
continue
|
||||
}
|
||||
if n, err := strconv.ParseInt(f[1], 10, 64); err == nil {
|
||||
kv[f[0]] = n
|
||||
}
|
||||
}
|
||||
return kv
|
||||
}
|
||||
|
||||
// mostWritten is the most bytes any one device of io.stat saw written: a mirror's member disks and the
|
||||
// mirror itself each count the same write, so the lines are never summed.
|
||||
func mostWritten(text string) int64 {
|
||||
var most int64
|
||||
for _, l := range strings.Split(text, "\n") {
|
||||
for _, f := range strings.Fields(l) {
|
||||
if v, ok := strings.CutPrefix(f, "wbytes="); ok {
|
||||
if n, err := strconv.ParseInt(v, 10, 64); err == nil && n > most {
|
||||
most = n
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return most
|
||||
}
|
||||
|
||||
// meminfo reads the huge page lines of /proc/meminfo, in MiB.
|
||||
func meminfo(text string) map[string]float64 {
|
||||
want := map[string]bool{"AnonHugePages": true, "ShmemHugePages": true, "ShmemPmdMapped": true, "FileHugePages": true, "FilePmdMapped": true}
|
||||
out := map[string]float64{}
|
||||
for _, l := range strings.Split(text, "\n") {
|
||||
k, v, ok := strings.Cut(l, ":")
|
||||
if !ok || !want[k] {
|
||||
continue
|
||||
}
|
||||
f := strings.Fields(v)
|
||||
if len(f) == 0 {
|
||||
continue
|
||||
}
|
||||
if n, err := strconv.ParseFloat(f[0], 64); err == nil {
|
||||
out[k] = n / 1024
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func kib(size string) int {
|
||||
n, _ := strconv.Atoi(strings.TrimSuffix(size, "kB"))
|
||||
return n
|
||||
}
|
||||
|
||||
func mib(b int64) string { return fmt.Sprintf("%.1f MiB", float64(b)/(1<<20)) }
|
||||
|
||||
func orUnknown(s string) string {
|
||||
if s == "" {
|
||||
return "unknown"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func orNone(list []string) string {
|
||||
if len(list) == 0 {
|
||||
return "none"
|
||||
}
|
||||
return strings.Join(list, ", ")
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package main
|
||||
|
||||
// Over a stand-in tree holding novox's own files of 2026-10-14 23:25 (read through node-reads for
|
||||
// novox/hq issue 746): the mode, the per-size controls, meminfo's huge page lines, and fail2ban's and
|
||||
// journald's memory.stat and io.stat, cut to the lines the tool reads.
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func tree(t *testing.T, files map[string]string) *Machine {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
for p, content := range files {
|
||||
full := filepath.Join(root, p)
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(full, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return &Machine{Root: root}
|
||||
}
|
||||
|
||||
func novox(t *testing.T) *Machine {
|
||||
thp := "sys/kernel/mm/transparent_hugepage/"
|
||||
cg := "sys/fs/cgroup/system.slice/"
|
||||
files := map[string]string{
|
||||
thp + "enabled": "[always] madvise never\n",
|
||||
thp + "defrag": "always defer defer+madvise [madvise] never\n",
|
||||
thp + "shmem_enabled": "always within_size [advise] never deny force\n",
|
||||
thp + "hugepages-2048kB/enabled": "always [inherit] madvise never\n",
|
||||
thp + "hugepages-64kB/enabled": "always inherit madvise [never]\n",
|
||||
thp + "hugepages-1024kB/enabled": "always inherit madvise [never]\n",
|
||||
"proc/meminfo": "MemTotal: 65748808 kB\nAnonHugePages: 3579904 kB\nShmemHugePages: 4096 kB\n" +
|
||||
"ShmemPmdMapped: 4096 kB\nFileHugePages: 1693696 kB\nFilePmdMapped: 882688 kB\nHugepagesize: 2048 kB\n",
|
||||
cg + "fail2ban.service/memory.stat": "anon 21815296\nfile 204623872\nfile_mapped 130736128\nfile_dirty 27262976\n" +
|
||||
"file_writeback 4096\nanon_thp 0\nfile_thp 197132288\nthp_fault_alloc 0\nthp_collapse_alloc 550\n",
|
||||
cg + "fail2ban.service/io.stat": "9:0 \n8:16 rbytes=3223552 wbytes=67723415552 rios=240 wios=281159 dbytes=0 dios=0\n" +
|
||||
"8:0 rbytes=6152192 wbytes=67723415552 rios=832 wios=281159 dbytes=0 dios=0\n" +
|
||||
"9:2 rbytes=9375744 wbytes=67723415552 rios=1072 wios=281159 dbytes=0 dios=0\n",
|
||||
cg + "systemd-journald.service/memory.stat": "anon 3227648\nfile 1032015872\nfile_mapped 131928064\nfile_dirty 622592\n" +
|
||||
"file_writeback 159744\nanon_thp 2097152\nfile_thp 379584512\nthp_fault_alloc 0\nthp_collapse_alloc 11741\n",
|
||||
cg + "systemd-journald.service/io.stat": "9:0 \n8:16 rbytes=55209984 wbytes=2498768035840 rios=1317 wios=24719004 dbytes=0 dios=0\n" +
|
||||
"8:0 rbytes=72953856 wbytes=2498768039936 rios=2194 wios=24719005 dbytes=0 dios=0\n" +
|
||||
"9:2 rbytes=128163840 wbytes=2498724843520 rios=3511 wios=24707873 dbytes=0 dios=0\n",
|
||||
cg + "sshd.service/memory.stat": "anon 1000\nfile_thp 0\n",
|
||||
}
|
||||
return tree(t, files)
|
||||
}
|
||||
|
||||
func TestNovoxBeforeTheChange(t *testing.T) {
|
||||
a, err := novox(t).HugePages(10)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a.Mode != "always" || a.Defrag != "madvise" || a.Shmem != "advise" {
|
||||
t.Fatalf("modes: %+v", a)
|
||||
}
|
||||
if len(a.PerSize) != 3 || a.PerSize[0].Size != "64kB" || a.PerSize[2].Size != "2048kB" ||
|
||||
a.PerSize[2].Enabled != "inherit" || a.PerSize[2].Means != "always" || a.PerSize[0].Means != "never" {
|
||||
t.Fatalf("per size: %+v", a.PerSize)
|
||||
}
|
||||
if a.Meminfo["FileHugePages"] != 1654 || a.Meminfo["FilePmdMapped"] != 862 {
|
||||
t.Fatalf("meminfo: %v", a.Meminfo)
|
||||
}
|
||||
if len(a.Units) != 2 || a.Units[0].Cgroup != "system.slice/systemd-journald.service" ||
|
||||
a.Units[1].Cgroup != "system.slice/fail2ban.service" {
|
||||
t.Fatalf("units: %+v", a.Units)
|
||||
}
|
||||
// The mirror and its two members each count the write: the most any one saw, never their sum.
|
||||
if a.Units[1].Written != 67723415552 || a.Units[1].FileTHP != 197132288 || a.Units[1].CollapsedByKhug != 550 {
|
||||
t.Fatalf("fail2ban: %+v", a.Units[1])
|
||||
}
|
||||
said := strings.Join(a.Said, "\n")
|
||||
for _, want := range []string{"transparent huge pages: always, defrag madvise", "sizes not never: 2048kB always",
|
||||
"file 1654 MiB (mapped at 2 MiB 862 MiB)", "system.slice/fail2ban.service: file in huge folios 188.0 MiB"} {
|
||||
if !strings.Contains(said, want) {
|
||||
t.Errorf("said lacks %q:\n%s", want, said)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnitsAreBoundedAndNoneIsSaid(t *testing.T) {
|
||||
a, err := novox(t).HugePages(1)
|
||||
if err != nil || len(a.Units) != 1 {
|
||||
t.Fatalf("%v %+v", err, a)
|
||||
}
|
||||
b, err := tree(t, map[string]string{"sys/kernel/mm/transparent_hugepage/enabled": "always [madvise] never\n"}).HugePages(10)
|
||||
if err != nil || b.Mode != "madvise" || len(b.Units) != 0 || !strings.Contains(strings.Join(b.Said, "\n"), "no unit holds") {
|
||||
t.Fatalf("%v %+v", err, b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAKernelWithoutHugePagesIsAnError(t *testing.T) {
|
||||
if _, err := tree(t, map[string]string{"proc/meminfo": ""}).HugePages(10); err == nil ||
|
||||
!strings.Contains(err.Error(), "built without transparent huge pages") {
|
||||
t.Fatalf("%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheArgumentIsBounded(t *testing.T) {
|
||||
if _, err := whole(map[string]any{"units": -1.0}, "units", 10, 0, 100); err == nil {
|
||||
t.Error("a negative count was accepted")
|
||||
}
|
||||
if n, _ := whole(map[string]any{"units": 1000.0}, "units", 10, 0, 100); n != 100 {
|
||||
t.Errorf("not bounded: %d", n)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package main
|
||||
|
||||
// The machine as the tools see it: the files they read, under a root that tests replace with a stand-in
|
||||
// tree. Everything read here is readable by the runtime's account; nothing is escalated.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// Machine reads this machine's kernel files under Root ("/" on the machine).
|
||||
type Machine struct {
|
||||
Root string
|
||||
}
|
||||
|
||||
// ThisMachine is the machine the runtime launched this bundle on.
|
||||
func ThisMachine() *Machine { return &Machine{Root: "/"} }
|
||||
|
||||
func (m *Machine) path(p string) string { return filepath.Join(m.Root, p) }
|
||||
|
||||
func (m *Machine) read(p string) (string, error) {
|
||||
b, err := os.ReadFile(m.path(p))
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
func (m *Machine) dirs(p string) ([]string, error) {
|
||||
entries, err := os.ReadDir(m.path(p))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []string
|
||||
for _, e := range entries {
|
||||
if e.IsDir() || e.Type()&fs.ModeSymlink != 0 {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// whole is a whole-number argument with a default, kept within bounds.
|
||||
func whole(args map[string]any, key string, def, least, most int) (int, error) {
|
||||
raw, present := args[key]
|
||||
if !present || raw == nil {
|
||||
return def, nil
|
||||
}
|
||||
f, ok := raw.(float64)
|
||||
if !ok || f != float64(int(f)) {
|
||||
return 0, fmt.Errorf("%s must be a whole number", key)
|
||||
}
|
||||
n := int(f)
|
||||
if n < least {
|
||||
return 0, fmt.Errorf("%s must be at least %d", key, least)
|
||||
}
|
||||
if n > most {
|
||||
n = most
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// schema is a tool's input: its properties.
|
||||
func schema(properties map[string]any) map[string]any {
|
||||
return map[string]any{"type": "object", "properties": properties}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
// The kernel module's tools bundle (novox/hq to-be 42 Phase 1 row 11, issue 746).
|
||||
//
|
||||
// Served by the node's runtime over MCP on stdio through the Go SDK (ADR 0188, ADR 0193). It only reads:
|
||||
// the transparent huge pages the kernel runs with, and which units hold file pages in huge folios, which
|
||||
// is what a small write to a mapped file costs a whole huge page of writeback for (issue 746).
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// binaryName is what the build names this bundle's executable: the manifest's `binary`.
|
||||
const binaryName = "kernel-tools"
|
||||
|
||||
func main() {
|
||||
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): kernel.
|
||||
if err := stdio.Serve("", tools(ThisMachine())); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func tools(m *Machine) []stdio.Tool {
|
||||
return []stdio.Tool{
|
||||
{
|
||||
Name: "kernel_huge_pages",
|
||||
Description: "Transparent huge pages on this machine: the mode in force (always, madvise or never) and defrag, " +
|
||||
"each per-size control with what inherit resolves to, the machine's huge pages by kind (anonymous, " +
|
||||
"shared memory, file, file mapped at 2 MiB) from /proc/meminfo, and the units holding the most file " +
|
||||
"pages in huge folios (memory.stat file_thp) with their dirty and writeback bytes and the bytes their " +
|
||||
"cgroup has written (io.stat, the most any one device saw). A unit with file_thp that writes far more " +
|
||||
"than its files grow is paying a whole huge page of writeback per small write (novox/hq issue 746). " +
|
||||
"Replaces cat /sys/kernel/mm/transparent_hugepage/enabled, grep Huge /proc/meminfo and reading " +
|
||||
"memory.stat by hand. (r)",
|
||||
Input: schema(map[string]any{
|
||||
"units": map[string]any{"type": "integer", "description": "how many units to name, the most file_thp first (default 10, 0 for none, at most 100)"},
|
||||
}),
|
||||
Run: func(args map[string]any) (any, error) {
|
||||
n, err := whole(args, "units", 10, 0, 100)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m.HugePages(n)
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
package main
|
||||
|
||||
// The module's shape (novox/hq issue 746): the setting's default is what the kernel boots with, so
|
||||
// assigning the module changes nothing until a machine sets another value; the script is written into the
|
||||
// module's own state and run by a oneshot unit that is restarted whenever the script (and so the setting)
|
||||
// or the unit changes; and the script itself sets, verifies, gives back and refuses as it says.
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTheDefaultChangesNothingAndTheUnitFollowsTheSetting(t *testing.T) {
|
||||
m := manifest(t)
|
||||
s := m.Settings["transparent-huge-pages"]
|
||||
if s["kind"] != "preference" || s["default"] != "always" || s["why"] == "" {
|
||||
t.Fatalf("the setting: %v", s)
|
||||
}
|
||||
var ids []string
|
||||
for _, r := range m.Resources {
|
||||
ids = append(ids, r["id"].(string))
|
||||
}
|
||||
if strings.Join(ids, " ") != "state huge-pages unit apply" {
|
||||
t.Fatalf("order: %v", ids)
|
||||
}
|
||||
script := m.resource(t, "huge-pages")
|
||||
// The script runs as root, so the setting it is written from is set at the terminal alone (issue 339).
|
||||
if script["path"] != "${dir:state}/huge-pages" || script["mode"] != "0755" || script["trusted"] != true ||
|
||||
!strings.Contains(script["content"].(string), "want='${setting:transparent-huge-pages}'") {
|
||||
t.Fatalf("the script: %v", script["path"])
|
||||
}
|
||||
unit := m.resource(t, "unit")["content"].(string)
|
||||
for _, line := range []string{"Type=oneshot", "RemainAfterExit=yes", "ExecStart=${dir:state}/huge-pages apply",
|
||||
"ExecStop=${dir:state}/huge-pages restore", "WantedBy=sysinit.target", "DefaultDependencies=no"} {
|
||||
if !strings.Contains(unit, line+"\n") {
|
||||
t.Errorf("the unit lacks %q", line)
|
||||
}
|
||||
}
|
||||
apply := m.resource(t, "apply")
|
||||
if apply["unit"] != "mesh-kernel-huge-pages.service" || apply["state"] != "running" || apply["boot"] != "enabled" {
|
||||
t.Fatalf("apply: %v", apply)
|
||||
}
|
||||
if on := apply["restart-on"].([]any); len(on) != 2 || on[0] != "huge-pages" || on[1] != "unit" {
|
||||
t.Fatalf("restart-on: %v", on)
|
||||
}
|
||||
if h := apply["health"].(map[string]any); h["kind"] != "unit" {
|
||||
t.Fatalf("health: %v", h)
|
||||
}
|
||||
}
|
||||
|
||||
// bed is the script as the mesh writes it for one value of the setting, a stand-in transparent_hugepage
|
||||
// directory, the module's state, and a PATH whose journalctl records that it was asked to rotate.
|
||||
type bed struct {
|
||||
script, sys, state, rotated, path string
|
||||
}
|
||||
|
||||
func newBed(t *testing.T, want, found string) bed {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
b := bed{script: filepath.Join(dir, "huge-pages"), sys: filepath.Join(dir, "thp"), state: filepath.Join(dir, "state"),
|
||||
rotated: filepath.Join(dir, "rotated")}
|
||||
content := manifest(t).resource(t, "huge-pages")["content"].(string)
|
||||
content = strings.ReplaceAll(content, "${setting:transparent-huge-pages}", want)
|
||||
content = strings.ReplaceAll(content, "${dir:state}", b.state)
|
||||
if strings.Contains(content, "${") {
|
||||
t.Fatalf("a substitution the test does not make is left in the script")
|
||||
}
|
||||
bin := filepath.Join(dir, "bin")
|
||||
for _, d := range []string{b.sys, b.state, bin} {
|
||||
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
write(t, b.script, content, 0o755)
|
||||
write(t, filepath.Join(b.sys, "enabled"), found, 0o644)
|
||||
write(t, filepath.Join(bin, "journalctl"), "#!/bin/sh\necho \"$@\" >>"+b.rotated+"\n", 0o755)
|
||||
b.path = bin + string(os.PathListSeparator) + os.Getenv("PATH")
|
||||
return b
|
||||
}
|
||||
|
||||
func write(t *testing.T, p, content string, mode os.FileMode) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(p, []byte(content), mode); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func (b bed) run(t *testing.T, verb string) (string, error) {
|
||||
t.Helper()
|
||||
cmd := exec.Command("sh", b.script, verb, b.sys, b.state)
|
||||
cmd.Env = append(os.Environ(), "PATH="+b.path)
|
||||
out, err := cmd.CombinedOutput()
|
||||
return string(out), err
|
||||
}
|
||||
|
||||
func (b bed) file(t *testing.T, p string) string {
|
||||
t.Helper()
|
||||
raw, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return string(raw)
|
||||
}
|
||||
|
||||
func TestApplySetsTheModeRotatesTheJournalOnceAndRestoreGivesBackWhatWasFound(t *testing.T) {
|
||||
// novox's file on 2026-10-14, read through node-reads.
|
||||
b := newBed(t, "madvise", "[always] madvise never\n")
|
||||
out, err := b.run(t, "apply")
|
||||
if err != nil || !strings.Contains(out, "always -> madvise") {
|
||||
t.Fatalf("apply: %v %s", err, out)
|
||||
}
|
||||
if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "madvise\n" {
|
||||
t.Fatalf("written: %q", got)
|
||||
}
|
||||
if got := b.file(t, filepath.Join(b.state, "found")); got != "always\n" {
|
||||
t.Fatalf("found: %q", got)
|
||||
}
|
||||
if got := b.file(t, b.rotated); got != "--rotate\n" {
|
||||
t.Fatalf("rotation: %q", got)
|
||||
}
|
||||
// Applied again (a restart with the same value): nothing written, nothing rotated, found kept.
|
||||
if out, err := b.run(t, "apply"); err != nil || !strings.Contains(out, "madvise, as set") {
|
||||
t.Fatalf("again: %v %s", err, out)
|
||||
}
|
||||
if got := b.file(t, b.rotated); got != "--rotate\n" {
|
||||
t.Fatalf("rotated again: %q", got)
|
||||
}
|
||||
if out, err := b.run(t, "restore"); err != nil || !strings.Contains(out, "back to always") {
|
||||
t.Fatalf("restore: %v %s", err, out)
|
||||
}
|
||||
if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "always\n" {
|
||||
t.Fatalf("restored: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAlwaysIsSetWithoutRotating(t *testing.T) {
|
||||
b := newBed(t, "always", "always [madvise] never\n")
|
||||
if out, err := b.run(t, "apply"); err != nil || !strings.Contains(out, "madvise -> always") {
|
||||
t.Fatalf("apply: %v %s", err, out)
|
||||
}
|
||||
if got := b.file(t, b.rotated); got != "" {
|
||||
t.Fatalf("rotated for always: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAValueTheKernelDoesNotTakeIsRefusedBeforeAnythingIsWritten(t *testing.T) {
|
||||
b := newBed(t, "sometimes", "[always] madvise never\n")
|
||||
out, err := b.run(t, "apply")
|
||||
if err == nil || !strings.Contains(out, "takes always, madvise or never") {
|
||||
t.Fatalf("accepted: %v %s", err, out)
|
||||
}
|
||||
if got := b.file(t, filepath.Join(b.sys, "enabled")); got != "[always] madvise never\n" {
|
||||
t.Fatalf("written anyway: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWriteThatDoesNotTakeFailsTheUnit(t *testing.T) {
|
||||
if os.Getuid() == 0 {
|
||||
t.Skip("root writes a read-only file")
|
||||
}
|
||||
b := newBed(t, "madvise", "[always] madvise never\n")
|
||||
if err := os.Chmod(filepath.Join(b.sys, "enabled"), 0o444); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out, err := b.run(t, "apply"); err == nil {
|
||||
t.Fatalf("a refused write passed: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAKernelWithoutHugePagesIsSaid(t *testing.T) {
|
||||
b := newBed(t, "madvise", "")
|
||||
if err := os.Remove(filepath.Join(b.sys, "enabled")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out, err := b.run(t, "apply"); err == nil || !strings.Contains(out, "built without transparent huge pages") {
|
||||
t.Fatalf("%v %s", err, out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type resource map[string]any
|
||||
|
||||
type manifestShape struct {
|
||||
Module string `json:"module"`
|
||||
Capabilities []string `json:"capabilities"`
|
||||
Settings map[string]map[string]any `json:"settings"`
|
||||
Tools []string `json:"tools"`
|
||||
Resources []resource `json:"resources"`
|
||||
Build struct {
|
||||
Artifacts []map[string]any `json:"artifacts"`
|
||||
} `json:"build"`
|
||||
}
|
||||
|
||||
func manifest(t *testing.T) manifestShape {
|
||||
t.Helper()
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m manifestShape
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func (m manifestShape) resource(t *testing.T, id string) resource {
|
||||
t.Helper()
|
||||
for _, r := range m.Resources {
|
||||
if r["id"] == id {
|
||||
return r
|
||||
}
|
||||
}
|
||||
t.Fatalf("no resource %s", id)
|
||||
return nil
|
||||
}
|
||||
|
||||
// TestToolsAreTheManifests holds the served tools and the manifest's list to one another, and the
|
||||
// bundle to the shape the builder compiles and the runtime loads.
|
||||
func TestToolsAreTheManifests(t *testing.T) {
|
||||
m := manifest(t)
|
||||
names := map[string]bool{}
|
||||
for _, tool := range tools(&Machine{Root: t.TempDir()}) {
|
||||
if names[tool.Name] {
|
||||
t.Errorf("%s is served twice", tool.Name)
|
||||
}
|
||||
names[tool.Name] = true
|
||||
}
|
||||
for _, want := range m.Tools {
|
||||
if !names[want] {
|
||||
t.Errorf("the manifest lists %s and the bundle does not serve it", want)
|
||||
}
|
||||
delete(names, want)
|
||||
}
|
||||
if len(names) != 0 {
|
||||
t.Errorf("served and not listed: %v", names)
|
||||
}
|
||||
var tools map[string]any
|
||||
for _, a := range m.Build.Artifacts {
|
||||
if a["name"] == "tools" {
|
||||
tools = a
|
||||
}
|
||||
}
|
||||
if tools == nil || tools["kind"] != "bundle" || tools["language"] != "go" || tools["system"] != "arch" ||
|
||||
tools["from"] != "cmd/"+binaryName || tools["binary"] != binaryName {
|
||||
t.Fatalf("the tools artifact: %v", tools)
|
||||
}
|
||||
if loads, _ := tools["loads"].([]any); len(loads) != 1 || loads[0] != binaryName {
|
||||
t.Fatalf("loads: %v", tools["loads"])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
module kernel
|
||||
|
||||
go 1.22
|
||||
|
||||
require git.novox.be/novox/mesh-sdk/go v0.1.7
|
||||
@@ -0,0 +1,2 @@
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"module": "kernel",
|
||||
"version": "1",
|
||||
"capabilities": [
|
||||
"service-manager"
|
||||
],
|
||||
"settings": {
|
||||
"transparent-huge-pages": {
|
||||
"kind": "preference",
|
||||
"default": "always",
|
||||
"why": "always is what the distribution's kernel boots with, so assigning the module changes nothing until a machine sets another value; madvise gives huge pages only to programs that ask for them, which stops a small write to a mapped file (the journal) dirtying and writing back a whole 2 MiB page on a machine whose disks are slow (novox/hq issue 746)"
|
||||
}
|
||||
},
|
||||
"tools": [
|
||||
"kernel_huge_pages"
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"id": "state",
|
||||
"type": "directory",
|
||||
"mode": "0700",
|
||||
"place": "."
|
||||
},
|
||||
{
|
||||
"id": "huge-pages",
|
||||
"type": "file",
|
||||
"path": "${dir:state}/huge-pages",
|
||||
"mode": "0755",
|
||||
"trusted": true,
|
||||
"content": "#!/bin/sh\n# Transparent huge pages as the mesh sets them on this machine (module kernel, novox/hq issue 746).\n# Written by the mesh from this machine's setting transparent-huge-pages and run by\n# mesh-kernel-huge-pages.service: `apply` at boot and whenever the setting changes, `restore` when the\n# unit stops (the module unassigned, or the machine shutting down). Replaced on every push; change the\n# setting, never this file.\n#\n# huge-pages apply|restore [<the kernel's transparent_hugepage directory> [<this module's state>]]\n#\n# The two directories are arguments only so the module's tests can run this against a stand-in tree.\nset -eu\n\nwant='${setting:transparent-huge-pages}'\nsys=/sys/kernel/mm/transparent_hugepage\nstate='${dir:state}'\n[ $# -ge 2 ] && sys=$2\n[ $# -ge 3 ] && state=$3\n\ncase \"$want\" in\nalways | madvise | never) ;;\n*)\n\techo \"the setting transparent-huge-pages is '$want'; the kernel takes always, madvise or never\" >&2\n\texit 2\n\t;;\nesac\n[ -f \"$sys/enabled\" ] || {\n\techo \"this kernel has no $sys/enabled: it was built without transparent huge pages, so there is nothing to set\" >&2\n\texit 1\n}\n\n# The mode in force is the word the kernel brackets: \"always [madvise] never\". A file holding one bare\n# word (the stand-in tree of the tests) is read as that word.\nmode() { sed -n -e 's/.*\\[\\([a-z]*\\)\\].*/\\1/p' -e t -e 's/^ *\\([a-z]*\\) *$/\\1/p' \"$sys/enabled\"; }\n\n[ $# -ge 1 ] || set -- usage\ncase \"$1\" in\napply)\n\tnow=$(mode)\n\t# What the machine had before the mesh first set it, kept once, so `restore` gives it back.\n\t[ -s \"$state/found\" ] || printf '%s\\n' \"$now\" >\"$state/found\"\n\tif [ \"$now\" = \"$want\" ]; then\n\t\techo \"transparent huge pages: $want, as set\"\n\t\texit 0\n\tfi\n\tprintf '%s\\n' \"$want\" >\"$sys/enabled\"\n\tgot=$(mode)\n\t[ \"$got\" = \"$want\" ] || {\n\t\techo \"wrote $want to $sys/enabled and the kernel says $got\" >&2\n\t\texit 1\n\t}\n\techo \"transparent huge pages: $now -> $want\"\n\t# A file folio already in the page cache keeps its size: the active journal files, mapped by journald\n\t# and by every reader, stay in 2 MiB folios until they are archived (issue 746). Rotating starts new\n\t# files that are mapped under the new mode. A failed rotation is said and does not fail the setting.\n\tif [ \"$want\" != always ] && command -v journalctl >/dev/null 2>&1; then\n\t\tjournalctl --rotate || echo \"the journal could not be rotated: its active files keep their huge pages until they are archived\" >&2\n\tfi\n\t;;\nrestore)\n\tfound=$(cat \"$state/found\" 2>/dev/null || true)\n\tcase \"$found\" in\n\talways | madvise | never)\n\t\tprintf '%s\\n' \"$found\" >\"$sys/enabled\"\n\t\techo \"transparent huge pages: back to $found, as the machine had them\"\n\t\t;;\n\t*) echo \"transparent huge pages: nothing recorded as found, left at $(mode)\" ;;\n\tesac\n\t;;\n*)\n\techo \"usage: huge-pages apply|restore [<transparent_hugepage directory> [<state directory>]]\" >&2\n\texit 2\n\t;;\nesac\n"
|
||||
},
|
||||
{
|
||||
"id": "unit",
|
||||
"type": "file",
|
||||
"path": "/etc/systemd/system/mesh-kernel-huge-pages.service",
|
||||
"mode": "0644",
|
||||
"content": "[Unit]\nDescription=Transparent huge pages as the mesh sets them on this machine\n# Module kernel, novox/hq issue 746. Starting this unit sets the mode from the setting\n# transparent-huge-pages; stopping it gives back the mode the machine had before. It runs early in boot,\n# before the services that fill the page cache, and again whenever the setting changes.\nDefaultDependencies=no\nAfter=local-fs.target\nBefore=sysinit.target shutdown.target\nConflicts=shutdown.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=${dir:state}/huge-pages apply\nExecStop=${dir:state}/huge-pages restore\n\n[Install]\nWantedBy=sysinit.target\n"
|
||||
},
|
||||
{
|
||||
"id": "apply",
|
||||
"type": "service",
|
||||
"unit": "mesh-kernel-huge-pages.service",
|
||||
"state": "running",
|
||||
"boot": "enabled",
|
||||
"restart-on": [
|
||||
"huge-pages",
|
||||
"unit"
|
||||
],
|
||||
"health": {
|
||||
"kind": "unit"
|
||||
}
|
||||
}
|
||||
],
|
||||
"build": {
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "tools",
|
||||
"kind": "bundle",
|
||||
"language": "go",
|
||||
"system": "arch",
|
||||
"from": "cmd/kernel-tools",
|
||||
"binary": "kernel-tools",
|
||||
"loads": [
|
||||
"kernel-tools"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user