Add the kernel module: transparent huge pages set per machine (hq issue 746, ADR 0369) #310

2026-10-14T21:50:22Z

mesh-admin commented

2026-10-14 21:50:22 +00:00

Fixes the disk starvation on the control node in hq issue 746, as ADR 0369 decides (the operator's word, 2026-10-14 23:25).

Why: with transparent huge pages always, journald's journal files (written through mmap) and fail2ban's journal-backend jails (reading them through mmap) sit in 2 MiB file folios, most collapsed by khugepaged; each small journal write dirties a 2 MiB folio and writeback writes all of it. Measured on the control node 23:25–23:35: fail2ban's cgroup wrote 5.03 GiB and journald's 392 MiB in 10 minutes while the journal grew by a few MiB; md2 (bus store, postgres) 100 % busy, 226 ms write wait; the controller lost its lease four times.

What: a new kernel module (design 42's kernel row) with one setting, transparent-huge-pages (always | madvise | never), defaulting to always, the kernel's boot value, so no machine changes until given another value. A oneshot unit writes the mode at boot and on change, reads it back and fails if it did not take, and gives back the mode it found when stopped. A setting change reloads the unit (reload-on), so the machine never passes back through always. Moving away from always rotates the journal once, so the open journal files are remapped; no rotation while the machine is starting. Sandboxed: ProtectSystem=strict, ReadWritePaths to its state, ProtectHome, PrivateNetwork, NoNewPrivileges. A read tool kernel_huge_pages for the after-measurement.

After merge: assign kernel to novox with transparent-huge-pages=madvise; the after-measurement goes on issue 746 against note 4.

Checks: go vet, go test -race, merge-check.sh with a controller built from main as the gate (kernel ok; resources without health still 62). Reviewed independently before push: APPROVE at ab1811d003.

Fixes the disk starvation on the control node in hq issue 746, as ADR 0369 decides (the operator's word, 2026-10-14 23:25). **Why:** with transparent huge pages `always`, journald's journal files (written through mmap) and fail2ban's journal-backend jails (reading them through mmap) sit in 2 MiB file folios, most collapsed by khugepaged; each small journal write dirties a 2 MiB folio and writeback writes all of it. Measured on the control node 23:25–23:35: fail2ban's cgroup wrote 5.03 GiB and journald's 392 MiB in 10 minutes while the journal grew by a few MiB; md2 (bus store, postgres) 100 % busy, 226 ms write wait; the controller lost its lease four times. **What:** a new `kernel` module (design 42's kernel row) with one setting, `transparent-huge-pages` (always | madvise | never), defaulting to `always`, the kernel's boot value, so no machine changes until given another value. A oneshot unit writes the mode at boot and on change, reads it back and fails if it did not take, and gives back the mode it found when stopped. A setting change reloads the unit (`reload-on`), so the machine never passes back through `always`. Moving away from `always` rotates the journal once, so the open journal files are remapped; no rotation while the machine is starting. Sandboxed: ProtectSystem=strict, ReadWritePaths to its state, ProtectHome, PrivateNetwork, NoNewPrivileges. A read tool `kernel_huge_pages` for the after-measurement. After merge: assign `kernel` to novox with `transparent-huge-pages=madvise`; the after-measurement goes on issue 746 against note 4. Checks: go vet, go test -race, merge-check.sh with a controller built from main as the gate (kernel ok; resources without health still 62). Reviewed independently before push: APPROVE at ab1811d00357a95329870b0c3a7614168a12480b.

mesh-admin added 2 commits 2026-10-14 21:50:22 +00:00

Add the kernel module: transparent huge pages set per machine (hq issue 746) 3a84044ece

With THP 'always' journald's mmapped journal files sit in 2 MiB folios, so each small
journal write dirties and writes back 2 MiB; on the control node that saturated the
root mirror holding the bus store and the store, and the controller lost its lease.
The module sets the mode from a per-machine setting (default always, the kernel's own,
so assigning it changes nothing until a machine sets madvise), verifies it took, rotates
the journal once so new files are mapped under the new mode, gives back what it found
when stopped, and serves kernel_huge_pages to measure file_thp per unit.

Reload, not restart, the huge pages unit when the setting changes (hq issue 746)

mesh/merge-gate head ab1811d00357 merged into trunk c79337579d6b — pass: builds new: modules/kernel, sent nowhere; no bus step; every machine composes with…

Details

mesh/repo-check head ab1811d00357 merged into trunk c79337579d6b — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…

Details

mesh/delivery delivered

Details

ab1811d003

A restart ran restore then apply, so every change passed back through the
found mode and rotated the journal again. Reload applies the new value
directly; restore also drops its state; no rotation while the machine boots;
the unit is sandboxed.

mesh-admin commented

2026-10-14 21:50:35 +00:00

Delivery novox/mesh-catalog@ab1811d00357 — delivered since 2026-10-14T21:53:15Z

Delivery plan — builds new: modules/kernel, sent nowhere; no bus step

Transitions

2026-10-14 21:50 (new) → proposed (announced): novox/mesh-catalog#310's head announced
2026-10-14 21:52 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-14 21:52 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-14 21:53 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-14 21:53 published → delivered (done): nothing for a walk to move

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@ab1811d00357` — **delivered** since 2026-10-14T21:53:15Z **Delivery plan** — builds new: modules/kernel, sent nowhere; no bus step **Transitions** - 2026-10-14 21:50 (new) → proposed (announced): novox/mesh-catalog#310's head announced - 2026-10-14 21:52 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-14 21:52 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-14 21:53 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-14 21:53 published → delivered (done): nothing for a walk to move The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin added the mesh/ready label 2026-10-14 21:52:17 +00:00

mesh-admin commented

2026-10-14 21:52:17 +00:00

ready at ab1811d003 (the operator's word, 2026-10-14; independent review APPROVE at this sha).

ready at ab1811d00357a95329870b0c3a7614168a12480b (the operator's word, 2026-10-14; independent review APPROVE at this sha).

mesh-admin commented

2026-10-14 21:52:18 +00:00

Merge check at ab1811d0
Judged: head ab1811d00357 merged into trunk c79337579d6b; read beside it: mesh-controller 60cc9ec363df, mesh-controller-main 60cc9ec363df, mesh-host 1a2677882380, mesh-tools 5d70eb3e35a9

mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: new:modules/kernel)
mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds new: modules/kernel, sent nowhere; no bus step

Run by the build seat on ace as build-1792014631834903598 (builds --log build-1792014631834903598).

--- the gate: new:modules/kernel
kernel: ok, 1 tool(s), ready: apply by unit every 30s
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T21:39:19Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T21:39:19Z

modules the change moves:
  - kernel (added)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 0 module(s) in 0 tier(s)
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.002s
words: 12 retired words and 70 daemons' lines, none in the 47473 strings of 726 files
ok  	git.novox.be/novox/mesh-catalog/checks/agenthome	0.011s
agent-home: no archive and no file written into under the agent's home
ok  	git.novox.be/novox/mesh-catalog/checks/secrets	0.015s
secrets: no setting can replace a vault secret in a mergeable file (1 known, nested)
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.006s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 62, as kept
ok  	git.novox.be/novox/mesh-catalog/checks/deps	0.198s
testing modules/kernel
ok  	kernel/cmd/kernel-tools	1.049s
--- its merge-check-typescript.sh (typescript toolchain)

**Merge check** at `ab1811d0` Judged: head `ab1811d00357` merged into trunk `c79337579d6b`; read beside it: mesh-controller `60cc9ec363df`, mesh-controller-main `60cc9ec363df`, mesh-host `1a2677882380`, mesh-tools `5d70eb3e35a9` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: new:modules/kernel) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds new: modules/kernel, sent nowhere; no bus step Run by the build seat on ace as `build-1792014631834903598` (`builds --log build-1792014631834903598`). ``` --- the gate: new:modules/kernel kernel: ok, 1 tool(s), ready: apply by unit every 30s long-running resources without health: 0 files asking for a setting without saying whether it is trusted: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T21:39:19Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-14T21:39:19Z modules the change moves: - kernel (added) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 0 module(s) in 0 tier(s) --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.002s words: 12 retired words and 70 daemons' lines, none in the 47473 strings of 726 files ok git.novox.be/novox/mesh-catalog/checks/agenthome 0.011s agent-home: no archive and no file written into under the agent's home ok git.novox.be/novox/mesh-catalog/checks/secrets 0.015s secrets: no setting can replace a vault secret in a mergeable file (1 known, nested) ok git.novox.be/novox/mesh-catalog/checks/paths 0.006s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 62, as kept ok git.novox.be/novox/mesh-catalog/checks/deps 0.198s testing modules/kernel ok kernel/cmd/kernel-tools 1.049s --- its merge-check-typescript.sh (typescript toolchain) ```

mesh-admin merged commit 7913b56d66 into main

2026-10-14 21:52:50 +00:00

mesh-admin deleted branch fix/746-huge-pages-madvise

2026-10-14 21:52:50 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#310