Phase 3: asus-zephyrus-g14 and memory-pressure, with Go tools and long-running code
The laptop model's hardware module and a memory-pressure module for any machine (hq research 027/03, 026/05, to-be 42 phase 3). The predecessor's polling auto-profile and mem-guard user scripts become each module's own Go code launched by the node runtime (ADR 0198): a profile switcher woken by the kernel's power-supply uevents, and a guard that warns on RAM, swap or PSI before systemd-oomd acts, on the desktop over the account's bus and always as an event. supergfxctl and triggerhappy are kept as found (research 027 Q1).
This commit is contained in:
jochen
@@ -0,0 +1,120 @@
|
||||
# memory-pressure
|
||||
|
||||
Compressed swap in RAM, systemd-oomd, and a guard that warns before the machine kills for memory. It
|
||||
can be assigned on any machine (novox/hq research 027/03, 026/05, to-be 42 phase 3). Memory pressure
|
||||
is not the laptop's alone.
|
||||
|
||||
## What it owns
|
||||
|
||||
| | what | notes |
|
||||
|---|---|---|
|
||||
| package | `zram-generator` | |
|
||||
| file | `/etc/systemd/zram-generator.conf` | `zram0`: `min(ram / 2, 16384)` MiB, zstd, swap priority 100. These are the laptop's values, adopted (the path too, ADR 0182). They scale with the machine: 15.3 GiB on 30 GiB of RAM, 16 GiB on the 125 GiB desktop. Read at boot, so a change applies at the next boot. Resizing a live device would mean swapping it off, which pushes what it holds back into RAM |
|
||||
| file | `/etc/sysctl.d/90-memory-pressure.conf` | `vm.page-cluster = 0` (no read-ahead on swap in RAM). Swappiness is left alone on purpose: the comment says why. `systemd-sysctl` is re-run |
|
||||
| file | `/etc/systemd/oomd.conf.d/memory-pressure.conf` | `SwapUsedLimit=90%`, `DefaultMemoryPressureLimit=60%`, `DefaultMemoryPressureDurationSec=20s` |
|
||||
| file | `/etc/systemd/system/-.slice.d/10-oomd.conf` | `ManagedOOMSwap=kill` |
|
||||
| file | `/etc/systemd/system/user@.service.d/10-oomd.conf` | `ManagedOOMMemoryPressure=kill`, limit 80 % |
|
||||
| service | `systemd-oomd` running, enabled | restarted on any of its drop-ins, after a `daemon-reload` |
|
||||
|
||||
The swap on disk is not this module's. Whether there is a swap file or a partition, and how large, is
|
||||
the machine's swap layout (research 027 question 3, the `kernel` module). The two workstations differ:
|
||||
the laptop has a 32 GiB swap file (priority 10) and a 16 GiB partition, and the desktop a 128 GiB
|
||||
partition and no zram. This module adds the zram tier above whatever is there.
|
||||
|
||||
## The long-running code: the guard (ADR 0198)
|
||||
|
||||
The Go bundle serves the tools and runs the guard in the same process, launched by the node's runtime.
|
||||
It replaces the predecessor's `mem-guard` user unit, which checked used RAM every 30 s and called
|
||||
`notify-send`.
|
||||
|
||||
- **Three signals:** RAM used ≥ 95 % (the predecessor's line, which it had raised from 90),
|
||||
swap used ≥ 80 % (oomd kills at 90 %), or memory pressure (PSI `some` avg10) ≥ 40 % (oomd acts at
|
||||
60 %, or 80 % for the user manager, over 20 s). Whichever comes first warns, and the warning says
|
||||
which.
|
||||
- **Every 10 s**, not 30: oomd's window is 20 s, so a check every 30 s could warn after oomd had
|
||||
already acted. Each check reads two files. PSI triggers would wake on pressure alone, but not on RAM
|
||||
or swap filling, so one timer reads all three rather than run two mechanisms.
|
||||
- **Once per episode.** After a warning the guard re-arms only when every value is below its clearing
|
||||
line: 88 % RAM, 70 % swap, 10 % pressure.
|
||||
- **The warning names what to close:** the three largest units by resident plus swapped memory, and
|
||||
in each its largest process.
|
||||
- **The desktop, from outside the session.** The runtime runs as the operator's account but outside
|
||||
the graphical session, with no session bus address and no `XDG_RUNTIME_DIR` (measured on the
|
||||
laptop's runtime unit). The guard names the account's own bus, `/run/user/<uid>/bus`, and calls
|
||||
`org.freedesktop.Notifications.Notify` with `busctl --user`. That client is the service manager's
|
||||
own, so nothing is installed, and a server pulls in no libnotify. Measured on 2026-10-04: from a
|
||||
clean environment as the account's uid, the notification service on that bus answered (dunst 1.13).
|
||||
The account is the runtime's `MESH_OPERATOR_ACCOUNT`. A runtime running as another uid could not
|
||||
authenticate on that bus, and the guard says so rather than try.
|
||||
- **And always an event:** `pressure.high` (reasons, percentages, the largest units) and
|
||||
`pressure.cleared` (how long it lasted) are published through the runtime, whether or not anyone is
|
||||
at a desktop. A machine without one has no session bus, and `memory_guard` says
|
||||
*no session bus*.
|
||||
|
||||
Thresholds are constants until settings exist (issue 168).
|
||||
|
||||
**Known limit.** The runtime restarts a launched bundle that exits on its next tool call, not at once.
|
||||
The guard recovers from a panic and reports it, but a crashed process waits for a call.
|
||||
|
||||
## Tools
|
||||
|
||||
| tool | what |
|
||||
|---|---|
|
||||
| `memory_status` | RAM, swap and each device with its priority, zram and its ratio, PSI some/full, and the guard's verdict on them |
|
||||
| `memory_top` | the largest processes by resident plus swap, with their unit; `by=unit` sums per unit, which is what systemd-oomd chooses among |
|
||||
| `memory_oom_history` | what was killed, newest first, from the journal: the kernel's OOM killer, systemd-oomd, and the service manager's *killed by the OOM killer*. `since` is checked against a short grammar before it reaches `journalctl` |
|
||||
| `memory_zram` | each zram device (algorithm, size, stored, compressed, RAM used, ratio, same-filled and incompressible pages), the generator's configuration, swappiness and page-cluster |
|
||||
| `memory_oomd` | whether oomd runs, and what it watches (`oomctl`) |
|
||||
| `memory_guard` | the guard's thresholds, last verdict, whether a warning stands, and whether the desktop can be reached |
|
||||
|
||||
## Found on 2026-10-04 (read-only)
|
||||
|
||||
- **On the laptop, systemd-oomd guards almost nothing a person runs.** Every desktop application runs
|
||||
in the login session's scope (`session-1.scope`), because the login manager and i3 start them there
|
||||
and not in per-application scopes under `user@.service`. The pressure kill on `user@.service`
|
||||
therefore watches 0.5 GiB. The swap kill on `-.slice`, when it fires, would choose the largest
|
||||
cgroup, which is the whole session scope: X, i3 and every application at once. `memory_top by=unit`
|
||||
shows it. **The fix is the graphical session's** (phase 2): launch applications in their own
|
||||
scopes, for example `systemd-run --user --scope` from the launcher, and oomd then kills one
|
||||
application. This module does not widen oomd's reach onto `user-.slice`, because there it would kill
|
||||
the session.
|
||||
- The desktop has no zram and oomd disabled. Assigning the module there is a change of behaviour:
|
||||
16 GiB of zram at the next boot, and oomd enabled with the same caveat about the session scope.
|
||||
|
||||
## When assigned to the laptop: what changes
|
||||
|
||||
1. Written over found files (originals kept once): `zram-generator.conf` (same values, so nothing
|
||||
until the next boot either), `-.slice.d/10-oomd.conf` and `user@.service.d/10-oomd.conf` (same
|
||||
keys).
|
||||
2. New: `sysctl.d/90-memory-pressure.conf` (the same `vm.page-cluster = 0` already in force) and
|
||||
`oomd.conf.d/memory-pressure.conf` (the same values as the predecessor's file beside it).
|
||||
3. `systemd-sysctl` is re-run (the values are unchanged), and `daemon-reload` and `systemd-oomd` are
|
||||
restarted.
|
||||
4. The node runtime restarts with the bundle, and the guard starts. **Until `mem-guard` is stopped,
|
||||
two notifiers run.**
|
||||
|
||||
## Predecessor files this module makes redundant — the operator removes them once (ADR 0182)
|
||||
|
||||
On the laptop:
|
||||
|
||||
1. `systemctl --user disable --now mem-guard.service`, then delete
|
||||
`~/.config/systemd/user/mem-guard.service` and `~/scripts/mem-guard.sh`.
|
||||
2. `/etc/systemd/oomd.conf.d/g14-oomd.conf`: the same values as the module's drop-in.
|
||||
3. `/etc/sysctl.d/90-g14-zram.conf`: the same key as the module's file.
|
||||
|
||||
The desktop had none of these.
|
||||
|
||||
## Tests
|
||||
|
||||
`go test ./...`. They read a tree standing in for `/proc` and `/sys`, using the laptop's own swaps,
|
||||
zram statistics and PSI lines. They cover:
|
||||
|
||||
- the snapshot;
|
||||
- the guard's three lines, its once-per-episode latch and its clearing, with the largest named;
|
||||
- an unreachable desktop still publishing the event;
|
||||
- the notification being one `busctl` call on the account's bus;
|
||||
- grouping by unit;
|
||||
- the journal's three kinds of kill, with non-UTF-8 messages skipped;
|
||||
- `since` refused when it is an option;
|
||||
- *no match* read as no kills;
|
||||
- the manifest: tools listed equal tools served, events, no machine named, triggers exist.
|
||||
@@ -0,0 +1,286 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The module's long-running code (novox/hq ADR 0198): the guard that warns before the machine kills
|
||||
// something for memory. It replaces the predecessor's `mem-guard`, a user unit that checked used RAM
|
||||
// every thirty seconds and called notify-send.
|
||||
//
|
||||
// What changed, and why:
|
||||
//
|
||||
// - **Three signals, not one.** Used RAM alone misses the two things that decide whether a kill is
|
||||
// coming: swap filling (systemd-oomd kills at SwapUsedLimit) and stall pressure (it kills a unit
|
||||
// whose pressure stays over its limit for twenty seconds). The guard warns on whichever comes
|
||||
// first, and says which.
|
||||
// - **Ten seconds, not thirty.** oomd's own window is twenty seconds; a check every thirty can
|
||||
// notice only after oomd has acted. Reading two files every ten seconds costs nothing measurable.
|
||||
// The kernel's PSI triggers would wake on pressure alone, but not on RAM or swap filling, so the
|
||||
// guard reads all three on one timer rather than run two mechanisms.
|
||||
// - **The notification names what to close.** It lists the largest units, not only a percentage.
|
||||
// - **An event as well as a notification.** `pressure.high` and `pressure.cleared` reach the mesh
|
||||
// whether or not anyone is at the desktop — a server has no desktop at all.
|
||||
// - **No session needed.** The runtime runs as the operator's account but outside the graphical
|
||||
// session, so it has no session bus address. The guard names the account's own bus,
|
||||
// /run/user/<uid>/bus, and speaks to the notification service with busctl, the service manager's
|
||||
// client: nothing is installed for it, and a machine without a desktop simply has no such bus.
|
||||
|
||||
// Thresholds, as constants until settings exist (novox/hq issue 168). Used RAM is the predecessor's,
|
||||
// raised by it from 90 to 95 after the lower line fired during ordinary work.
|
||||
const (
|
||||
WarnUsedPercent = 95.0
|
||||
ClearUsedPercent = 88.0
|
||||
WarnSwapPercent = 80.0 // systemd-oomd's SwapUsedLimit is 90 %
|
||||
ClearSwapPercent = 70.0
|
||||
WarnPressureAvg10 = 40.0 // "some" avg10; oomd acts at 60 % (80 % for the user manager) over 20 s
|
||||
ClearPressureAvg10 = 10.0
|
||||
CheckEvery = 10 * time.Second
|
||||
NotificationID = 9010 // replaces the previous warning rather than stacking
|
||||
)
|
||||
|
||||
// Verdict is what one check of the machine concluded.
|
||||
type Verdict struct {
|
||||
High bool `json:"high"`
|
||||
Reasons []string `json:"reasons"`
|
||||
Clear bool `json:"clear"`
|
||||
}
|
||||
|
||||
// Judge applies the thresholds to a snapshot. High is any warning line crossed; Clear is every value
|
||||
// below its clearing line, which is what re-arms the guard.
|
||||
func Judge(s Snapshot) Verdict {
|
||||
v := Verdict{Reasons: []string{}}
|
||||
clear := true
|
||||
if s.UsedPercent >= WarnUsedPercent {
|
||||
v.Reasons = append(v.Reasons, fmt.Sprintf("RAM %.0f%% used (warns at %.0f%%)", s.UsedPercent, WarnUsedPercent))
|
||||
}
|
||||
if s.UsedPercent > ClearUsedPercent {
|
||||
clear = false
|
||||
}
|
||||
if s.SwapTotalGiB > 0 {
|
||||
if s.SwapPercent >= WarnSwapPercent {
|
||||
v.Reasons = append(v.Reasons, fmt.Sprintf("swap %.0f%% used (warns at %.0f%%; systemd-oomd kills at 90%%)", s.SwapPercent, WarnSwapPercent))
|
||||
}
|
||||
if s.SwapPercent > ClearSwapPercent {
|
||||
clear = false
|
||||
}
|
||||
}
|
||||
if s.PressureSome != nil {
|
||||
if s.PressureSome.Avg10 >= WarnPressureAvg10 {
|
||||
v.Reasons = append(v.Reasons, fmt.Sprintf("tasks stalled on memory %.0f%% of the last 10 s (warns at %.0f%%)", s.PressureSome.Avg10, WarnPressureAvg10))
|
||||
}
|
||||
if s.PressureSome.Avg10 > ClearPressureAvg10 {
|
||||
clear = false
|
||||
}
|
||||
}
|
||||
v.High = len(v.Reasons) > 0
|
||||
v.Clear = clear
|
||||
return v
|
||||
}
|
||||
|
||||
// Guard is the notifier's state, shared with the tools that report it.
|
||||
type Guard struct {
|
||||
m *Machine
|
||||
now func() time.Time
|
||||
emit func(string, any) error
|
||||
notify func(ctx context.Context, summary, body string) error
|
||||
|
||||
mu sync.Mutex
|
||||
latched bool
|
||||
since time.Time
|
||||
last *Verdict
|
||||
lastAt time.Time
|
||||
notified string
|
||||
err string
|
||||
desktop string
|
||||
stopped string
|
||||
}
|
||||
|
||||
func NewGuard(m *Machine, emit func(string, any) error) *Guard {
|
||||
g := &Guard{m: m, now: time.Now, emit: emit}
|
||||
g.notify = g.desktopNotify
|
||||
return g
|
||||
}
|
||||
|
||||
// Check is one wake-up: read, judge, and on crossing a line warn once; on clearing every line, re-arm.
|
||||
func (g *Guard) Check(ctx context.Context) {
|
||||
s, err := g.m.Snapshot()
|
||||
g.mu.Lock()
|
||||
if err != nil {
|
||||
g.err = err.Error()
|
||||
g.mu.Unlock()
|
||||
return
|
||||
}
|
||||
v := Judge(s)
|
||||
now := g.now()
|
||||
g.last, g.lastAt, g.err = &v, now, ""
|
||||
warn := v.High && !g.latched
|
||||
cleared := g.latched && v.Clear
|
||||
if warn {
|
||||
g.latched, g.since = true, now
|
||||
}
|
||||
if cleared {
|
||||
g.latched = false
|
||||
}
|
||||
since := g.since
|
||||
g.mu.Unlock()
|
||||
|
||||
if warn {
|
||||
largest := ByUnit(g.m.Processes(), 3)
|
||||
var names []string
|
||||
for _, u := range largest {
|
||||
names = append(names, fmt.Sprintf("%s %.1f GiB", firstNonEmpty(u.Largest, u.Unit), (u.RSSMiB+u.SwapMiB)/1024))
|
||||
}
|
||||
body := strings.Join(v.Reasons, "; ")
|
||||
if len(names) > 0 {
|
||||
body += ". Largest: " + strings.Join(names, ", ")
|
||||
}
|
||||
body += ". systemd-oomd kills the worst unit if it climbs further."
|
||||
g.publish("pressure.high", map[string]any{"reasons": v.Reasons, "used_percent": s.UsedPercent,
|
||||
"swap_used_percent": s.SwapPercent, "pressure_some": s.PressureSome, "largest": largest})
|
||||
err := g.notify(ctx, "Memory is running low", body)
|
||||
g.mu.Lock()
|
||||
if err != nil {
|
||||
g.desktop = "not reached: " + err.Error()
|
||||
} else {
|
||||
g.desktop, g.notified = "reached", now.Format(time.RFC3339)
|
||||
}
|
||||
g.mu.Unlock()
|
||||
}
|
||||
if cleared {
|
||||
g.publish("pressure.cleared", map[string]any{"used_percent": s.UsedPercent, "swap_used_percent": s.SwapPercent,
|
||||
"lasted_seconds": int(now.Sub(since).Seconds())})
|
||||
}
|
||||
}
|
||||
|
||||
func (g *Guard) publish(eventType string, body map[string]any) {
|
||||
if g.emit == nil {
|
||||
return
|
||||
}
|
||||
if err := g.emit(eventType, body); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "%s not published: %v\n", eventType, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Bus is the operator's session bus socket: the account the runtime names, else the one this process
|
||||
// runs as.
|
||||
func Bus() (string, error) {
|
||||
uid := os.Getuid()
|
||||
if name := os.Getenv("MESH_OPERATOR_ACCOUNT"); name != "" {
|
||||
u, err := user.Lookup(name)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the operator account %s: %w", name, err)
|
||||
}
|
||||
if n, err := strconv.Atoi(u.Uid); err == nil {
|
||||
if n != uid && uid != 0 {
|
||||
return "", fmt.Errorf("this process runs as uid %d and the operator's bus belongs to uid %d", uid, n)
|
||||
}
|
||||
uid = n
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("/run/user/%d/bus", uid), nil
|
||||
}
|
||||
|
||||
// desktopNotify sends one critical notification to the operator's session, replacing the previous one.
|
||||
func (g *Guard) desktopNotify(ctx context.Context, summary, body string) error {
|
||||
bus, err := Bus()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := os.Stat(bus); err != nil {
|
||||
return fmt.Errorf("no session bus at %s (nobody is logged in to a desktop)", bus)
|
||||
}
|
||||
_, err = g.m.Run(ctx, "env", "DBUS_SESSION_BUS_ADDRESS=unix:path="+bus,
|
||||
"busctl", "--user", "call", "org.freedesktop.Notifications", "/org/freedesktop/Notifications",
|
||||
"org.freedesktop.Notifications", "Notify", "susssasa{sv}i",
|
||||
"memory-pressure", strconv.Itoa(NotificationID), "dialog-warning", summary, body, "0", "1", "urgency", "y", "2", "0")
|
||||
return err
|
||||
}
|
||||
|
||||
// Run is the guard's life: a check every CheckEvery.
|
||||
func (g *Guard) Run(ctx context.Context) {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
g.mu.Lock()
|
||||
g.stopped = fmt.Sprintf("the guard stopped on a fault: %v", r)
|
||||
g.mu.Unlock()
|
||||
fmt.Fprintln(os.Stderr, g.stopped)
|
||||
}
|
||||
}()
|
||||
t := time.NewTicker(CheckEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
g.Check(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// GuardReport is the guard as the guard tool shows it.
|
||||
type GuardReport struct {
|
||||
Running bool `json:"running"`
|
||||
Stopped string `json:"stopped,omitempty"`
|
||||
Warned bool `json:"warning_given"`
|
||||
WarnedSince *time.Time `json:"warning_since,omitempty"`
|
||||
LastCheck *time.Time `json:"last_check,omitempty"`
|
||||
LastVerdict *Verdict `json:"last_verdict,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
Desktop string `json:"desktop"`
|
||||
LastNotified string `json:"last_notified,omitempty"`
|
||||
Thresholds map[string]any `json:"thresholds"`
|
||||
}
|
||||
|
||||
func (g *Guard) Report() GuardReport {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
r := GuardReport{
|
||||
Running: g.stopped == "" && !g.lastAt.IsZero(), Stopped: g.stopped, Warned: g.latched, LastCheck: when(g.lastAt),
|
||||
LastVerdict: g.last, LastError: g.err, Desktop: g.desktop, LastNotified: g.notified,
|
||||
Thresholds: map[string]any{
|
||||
"warn_used_percent": WarnUsedPercent, "clear_used_percent": ClearUsedPercent,
|
||||
"warn_swap_percent": WarnSwapPercent, "clear_swap_percent": ClearSwapPercent,
|
||||
"warn_pressure_some_avg10": WarnPressureAvg10, "clear_pressure_some_avg10": ClearPressureAvg10,
|
||||
"check_every": CheckEvery.String(), "set_by": "constants until settings exist (novox/hq issue 168)",
|
||||
},
|
||||
}
|
||||
if g.latched {
|
||||
r.WarnedSince = when(g.since)
|
||||
}
|
||||
if r.Desktop == "" {
|
||||
if bus, err := Bus(); err != nil {
|
||||
r.Desktop = "not reachable: " + err.Error()
|
||||
} else if _, err := os.Stat(bus); err != nil {
|
||||
r.Desktop = "no session bus at " + bus
|
||||
} else {
|
||||
r.Desktop = "reachable at " + bus + " (not yet used)"
|
||||
}
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func firstNonEmpty(ss ...string) string {
|
||||
for _, s := range ss {
|
||||
if s != "" {
|
||||
return s
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// when is a time for a report: absent rather than the zero time.
|
||||
func when(t time.Time) *time.Time {
|
||||
if t.IsZero() {
|
||||
return nil
|
||||
}
|
||||
return &t
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fake is a machine for a test: a tree standing in for /, and a runner answering from a table and
|
||||
// recording every command it was asked to run.
|
||||
type fake struct {
|
||||
t *testing.T
|
||||
root string
|
||||
mu sync.Mutex
|
||||
answers map[string]string
|
||||
fails map[string]error
|
||||
calls []string
|
||||
}
|
||||
|
||||
func newFake(t *testing.T) *fake {
|
||||
t.Helper()
|
||||
return &fake{t: t, root: t.TempDir(), answers: map[string]string{}, fails: map[string]error{}}
|
||||
}
|
||||
|
||||
func (f *fake) machine() *Machine { return &Machine{Root: f.root, Run: f.run} }
|
||||
|
||||
func (f *fake) run(_ context.Context, name string, args ...string) (string, error) {
|
||||
line := strings.TrimSpace(name + " " + strings.Join(args, " "))
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.calls = append(f.calls, line)
|
||||
if err, ok := f.fails[line]; ok {
|
||||
return "", err
|
||||
}
|
||||
if out, ok := f.answers[line]; ok {
|
||||
return out, nil
|
||||
}
|
||||
if err, ok := f.fails[name]; ok {
|
||||
return "", err
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (f *fake) called(line string) bool {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
for _, c := range f.calls {
|
||||
if c == line {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (f *fake) callsLike(prefix string) []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
var out []string
|
||||
for _, c := range f.calls {
|
||||
if strings.HasPrefix(c, prefix) {
|
||||
out = append(out, c)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// file writes a file under the fake root.
|
||||
func (f *fake) file(path, content string) {
|
||||
f.t.Helper()
|
||||
full := filepath.Join(f.root, path)
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(full, []byte(content), 0o644); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// CommandTimeout bounds every command a tool or the guard runs: a journal that takes long to search
|
||||
// must cost a tool call twenty seconds, never the runtime's thirty.
|
||||
const CommandTimeout = 20 * time.Second
|
||||
|
||||
// Runner runs one command and answers its standard output. It is injected so that every tool is
|
||||
// tested against recorded answers rather than this machine's daemons.
|
||||
type Runner func(ctx context.Context, name string, args ...string) (string, error)
|
||||
|
||||
// ExecRunner runs a command on the machine, bounded by CommandTimeout. A failure carries what the
|
||||
// command said on stderr, because "exit status 1" names nothing.
|
||||
func ExecRunner(ctx context.Context, name string, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, CommandTimeout)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &stdout, &stderr
|
||||
err := cmd.Run()
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
return stdout.String(), fmt.Errorf("%s did not answer within %s", name, CommandTimeout)
|
||||
}
|
||||
if err != nil {
|
||||
said := strings.TrimSpace(stderr.String())
|
||||
if said == "" {
|
||||
said = strings.TrimSpace(stdout.String())
|
||||
}
|
||||
if said != "" {
|
||||
return stdout.String(), fmt.Errorf("%s %s: %w: %s", name, strings.Join(args, " "), err, said)
|
||||
}
|
||||
return stdout.String(), fmt.Errorf("%s %s: %w", name, strings.Join(args, " "), err)
|
||||
}
|
||||
return stdout.String(), nil
|
||||
}
|
||||
|
||||
// Machine is what the module reads and acts on: a filesystem root (the real one, or a test's tree of
|
||||
// /proc, /sys and /etc) and a way to run commands.
|
||||
type Machine struct {
|
||||
Root string
|
||||
Run Runner
|
||||
}
|
||||
|
||||
// Here is the machine this process runs on.
|
||||
func Here() *Machine { return &Machine{Root: "/", Run: ExecRunner} }
|
||||
|
||||
func (m *Machine) path(p string) string { return filepath.Join(m.Root, p) }
|
||||
|
||||
// read is a file's content, trimmed; "" when it cannot be read.
|
||||
func (m *Machine) read(p string) string {
|
||||
b, err := os.ReadFile(m.path(p))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
// readInt is a file holding one integer; ok false when it is absent or not a number.
|
||||
func (m *Machine) readInt(p string) (int64, bool) {
|
||||
s := m.read(p)
|
||||
if s == "" {
|
||||
return 0, false
|
||||
}
|
||||
n, err := strconv.ParseInt(s, 10, 64)
|
||||
return n, err == nil
|
||||
}
|
||||
|
||||
func (m *Machine) glob(pattern string) []string {
|
||||
found, _ := filepath.Glob(m.path(pattern))
|
||||
out := make([]string, 0, len(found))
|
||||
for _, f := range found {
|
||||
rel, err := filepath.Rel(m.Root, f)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, "/"+filepath.ToSlash(rel))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// notInstalled says a command failed because it is not on this machine at all.
|
||||
func notInstalled(err error) bool { return errors.Is(err, exec.ErrNotFound) }
|
||||
|
||||
// round to one decimal, for watts and percentages a person reads.
|
||||
func round1(f float64) float64 {
|
||||
return float64(int64(f*10+sign(f)*0.5)) / 10
|
||||
}
|
||||
|
||||
func sign(f float64) float64 {
|
||||
if f < 0 {
|
||||
return -1
|
||||
}
|
||||
return 1
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
// The memory-pressure module's Go bundle (novox/hq ADR 0188, ADR 0193, ADR 0198): one process the
|
||||
// node's runtime launches, serving the module's tools over MCP on stdio and running its long-running
|
||||
// code — the guard that warns before the machine kills for memory — beside them.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
func main() {
|
||||
m := Here()
|
||||
g := NewGuard(m, func(eventType string, body any) error { return stdio.Emit(eventType, body) })
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go g.Run(ctx)
|
||||
if err := stdio.Serve("", Tools(m, g)); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTheManifestNamesExactlyTheToolsTheBundleServesAndNoMachine(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m struct {
|
||||
Tools []string `json:"tools"`
|
||||
Emits []string `json:"emits"`
|
||||
Resources []map[string]any `json:"resources"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var served []string
|
||||
for _, tool := range Tools(newFake(t).machine(), nil) {
|
||||
served = append(served, tool.Name)
|
||||
}
|
||||
sort.Strings(served)
|
||||
sort.Strings(m.Tools)
|
||||
if strings.Join(served, ",") != strings.Join(m.Tools, ",") {
|
||||
t.Fatalf("served %v, listed %v", served, m.Tools)
|
||||
}
|
||||
sort.Strings(m.Emits)
|
||||
if strings.Join(m.Emits, ",") != "pressure.cleared,pressure.high" {
|
||||
t.Fatalf("emits %v", m.Emits)
|
||||
}
|
||||
for _, banned := range []string{"g14", "jochen", "/home/", "shanks", "/run/user/1000"} {
|
||||
if strings.Contains(string(raw), banned) {
|
||||
t.Errorf("the manifest says %q", banned)
|
||||
}
|
||||
}
|
||||
ids := map[string]bool{}
|
||||
for _, r := range m.Resources {
|
||||
ids[r["id"].(string)] = true
|
||||
}
|
||||
for _, r := range m.Resources {
|
||||
list, _ := r["restart-on"].([]any)
|
||||
for _, id := range list {
|
||||
if !ids[id.(string)] {
|
||||
t.Errorf("%s restarts on %v, which is not a resource", r["id"], id)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const kib = 1024
|
||||
|
||||
// GiB turns bytes into gibibytes with one decimal, for a person reading.
|
||||
func GiB(bytes int64) float64 { return round1(float64(bytes) / (1 << 30)) }
|
||||
|
||||
// ParseMeminfo reads /proc/meminfo into bytes by field.
|
||||
func ParseMeminfo(text string) map[string]int64 {
|
||||
out := map[string]int64{}
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
key, rest, ok := strings.Cut(line, ":")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
f := strings.Fields(rest)
|
||||
if len(f) == 0 {
|
||||
continue
|
||||
}
|
||||
n, err := strconv.ParseInt(f[0], 10, 64)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if len(f) > 1 && f[1] == "kB" {
|
||||
n *= kib
|
||||
}
|
||||
out[key] = n
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Pressure is one line of a PSI file: the share of time some (or all) tasks stalled on memory.
|
||||
type Pressure struct {
|
||||
Avg10 float64 `json:"avg10"`
|
||||
Avg60 float64 `json:"avg60"`
|
||||
Avg300 float64 `json:"avg300"`
|
||||
}
|
||||
|
||||
// ParsePSI reads /proc/pressure/memory: its `some` and `full` lines.
|
||||
func ParsePSI(text string) (some, full *Pressure) {
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 4 {
|
||||
continue
|
||||
}
|
||||
p := &Pressure{}
|
||||
for _, kv := range f[1:] {
|
||||
k, v, _ := strings.Cut(kv, "=")
|
||||
n, _ := strconv.ParseFloat(v, 64)
|
||||
switch k {
|
||||
case "avg10":
|
||||
p.Avg10 = n
|
||||
case "avg60":
|
||||
p.Avg60 = n
|
||||
case "avg300":
|
||||
p.Avg300 = n
|
||||
}
|
||||
}
|
||||
switch f[0] {
|
||||
case "some":
|
||||
some = p
|
||||
case "full":
|
||||
full = p
|
||||
}
|
||||
}
|
||||
return some, full
|
||||
}
|
||||
|
||||
// SwapDevice is one line of /proc/swaps.
|
||||
type SwapDevice struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
SizeGiB float64 `json:"size_gib"`
|
||||
UsedGiB float64 `json:"used_gib"`
|
||||
Priority int `json:"priority"`
|
||||
}
|
||||
|
||||
// ParseSwaps reads /proc/swaps (sizes in KiB).
|
||||
func ParseSwaps(text string) []SwapDevice {
|
||||
var out []SwapDevice
|
||||
for i, line := range strings.Split(text, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if i == 0 || len(f) < 5 {
|
||||
continue
|
||||
}
|
||||
size, _ := strconv.ParseInt(f[2], 10, 64)
|
||||
used, _ := strconv.ParseInt(f[3], 10, 64)
|
||||
prio, _ := strconv.Atoi(f[4])
|
||||
out = append(out, SwapDevice{Name: f[0], Type: f[1], SizeGiB: GiB(size * kib), UsedGiB: GiB(used * kib), Priority: prio})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Snapshot is the machine's memory at one moment: what the status tool answers and what the guard
|
||||
// judges.
|
||||
type Snapshot struct {
|
||||
TotalGiB float64 `json:"total_gib"`
|
||||
AvailableGiB float64 `json:"available_gib"`
|
||||
UsedPercent float64 `json:"used_percent"`
|
||||
SwapTotalGiB float64 `json:"swap_total_gib"`
|
||||
SwapUsedGiB float64 `json:"swap_used_gib"`
|
||||
SwapPercent float64 `json:"swap_used_percent"`
|
||||
Swaps []SwapDevice `json:"swap_devices"`
|
||||
Zram []Zram `json:"zram"`
|
||||
PressureSome *Pressure `json:"pressure_some,omitempty"`
|
||||
PressureFull *Pressure `json:"pressure_full,omitempty"`
|
||||
}
|
||||
|
||||
// Snapshot reads /proc and /sys.
|
||||
func (m *Machine) Snapshot() (Snapshot, error) {
|
||||
mem := ParseMeminfo(m.read("/proc/meminfo"))
|
||||
total, avail := mem["MemTotal"], mem["MemAvailable"]
|
||||
if total <= 0 {
|
||||
return Snapshot{}, fmt.Errorf("/proc/meminfo says no MemTotal")
|
||||
}
|
||||
s := Snapshot{
|
||||
TotalGiB: GiB(total), AvailableGiB: GiB(avail),
|
||||
UsedPercent: round1(float64(total-avail) / float64(total) * 100),
|
||||
SwapTotalGiB: GiB(mem["SwapTotal"]),
|
||||
SwapUsedGiB: GiB(mem["SwapTotal"] - mem["SwapFree"]),
|
||||
Swaps: ParseSwaps(m.read("/proc/swaps")),
|
||||
Zram: m.Zram(),
|
||||
}
|
||||
if mem["SwapTotal"] > 0 {
|
||||
s.SwapPercent = round1(float64(mem["SwapTotal"]-mem["SwapFree"]) / float64(mem["SwapTotal"]) * 100)
|
||||
}
|
||||
if s.Swaps == nil {
|
||||
s.Swaps = []SwapDevice{}
|
||||
}
|
||||
s.PressureSome, s.PressureFull = ParsePSI(m.read("/proc/pressure/memory"))
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// Zram is one compressed swap device in RAM.
|
||||
type Zram struct {
|
||||
Device string `json:"device"`
|
||||
Algorithm string `json:"algorithm"`
|
||||
DiskSizeGiB float64 `json:"disk_size_gib"`
|
||||
StoredGiB float64 `json:"stored_gib"`
|
||||
CompressedGiB float64 `json:"compressed_gib"`
|
||||
RAMUsedGiB float64 `json:"ram_used_gib"`
|
||||
Ratio *float64 `json:"compression_ratio,omitempty"`
|
||||
SamePages int64 `json:"same_filled_pages"`
|
||||
HugePages int64 `json:"incompressible_pages"`
|
||||
}
|
||||
|
||||
// Zram reads every zram device's statistics from /sys/block.
|
||||
func (m *Machine) Zram() []Zram {
|
||||
out := []Zram{}
|
||||
for _, dir := range m.glob("/sys/block/zram*") {
|
||||
z := Zram{Device: path.Base(dir), Algorithm: activeAlgorithm(m.read(dir + "/comp_algorithm"))}
|
||||
if v, ok := m.readInt(dir + "/disksize"); ok {
|
||||
z.DiskSizeGiB = GiB(v)
|
||||
}
|
||||
// mm_stat: orig_data_size compr_data_size mem_used_total mem_limit mem_used_max same_pages
|
||||
// pages_compacted huge_pages …
|
||||
f := strings.Fields(m.read(dir + "/mm_stat"))
|
||||
n := func(i int) int64 {
|
||||
if i >= len(f) {
|
||||
return 0
|
||||
}
|
||||
v, _ := strconv.ParseInt(f[i], 10, 64)
|
||||
return v
|
||||
}
|
||||
if len(f) >= 3 {
|
||||
z.StoredGiB, z.CompressedGiB, z.RAMUsedGiB = GiB(n(0)), GiB(n(1)), GiB(n(2))
|
||||
z.SamePages, z.HugePages = n(5), n(7)
|
||||
if n(1) > 0 {
|
||||
r := round1(float64(n(0)) / float64(n(1)))
|
||||
z.Ratio = &r
|
||||
}
|
||||
}
|
||||
out = append(out, z)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// activeAlgorithm is the bracketed one of `lzo lz4 [zstd]`.
|
||||
func activeAlgorithm(s string) string {
|
||||
for _, f := range strings.Fields(s) {
|
||||
if strings.HasPrefix(f, "[") {
|
||||
return strings.Trim(f, "[]")
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Process is one process by the memory it holds.
|
||||
type Process struct {
|
||||
PID int `json:"pid"`
|
||||
Name string `json:"name"`
|
||||
RSSMiB float64 `json:"rss_mib"`
|
||||
SwapMiB float64 `json:"swap_mib"`
|
||||
Unit string `json:"unit,omitempty"`
|
||||
Command string `json:"command,omitempty"`
|
||||
}
|
||||
|
||||
// Group is the memory of every process in one systemd unit or scope.
|
||||
type Group struct {
|
||||
Unit string `json:"unit"`
|
||||
Processes int `json:"processes"`
|
||||
RSSMiB float64 `json:"rss_mib"`
|
||||
SwapMiB float64 `json:"swap_mib"`
|
||||
Largest string `json:"largest"`
|
||||
}
|
||||
|
||||
// unitOf is the last named unit in a process's cgroup path: the scope or service it lives in, which
|
||||
// is what systemd-oomd chooses among.
|
||||
func unitOf(cgroup string) string {
|
||||
line := strings.TrimSpace(strings.SplitN(cgroup, "\n", 2)[0])
|
||||
_, p, _ := strings.Cut(line, "::")
|
||||
parts := strings.Split(p, "/")
|
||||
for i := len(parts) - 1; i >= 0; i-- {
|
||||
if strings.HasSuffix(parts[i], ".scope") || strings.HasSuffix(parts[i], ".service") || strings.HasSuffix(parts[i], ".slice") {
|
||||
return parts[i]
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Processes reads every process's resident and swapped memory. A process that ends while it is read
|
||||
// is skipped.
|
||||
func (m *Machine) Processes() []Process {
|
||||
var out []Process
|
||||
for _, dir := range m.glob("/proc/[0-9]*") {
|
||||
status := m.read(dir + "/status")
|
||||
if status == "" {
|
||||
continue
|
||||
}
|
||||
fields := ParseMeminfo(status)
|
||||
name := ""
|
||||
for _, line := range strings.Split(status, "\n") {
|
||||
if v, ok := strings.CutPrefix(line, "Name:"); ok {
|
||||
name = strings.TrimSpace(v)
|
||||
break
|
||||
}
|
||||
}
|
||||
rss := fields["VmRSS"]
|
||||
swap := fields["VmSwap"]
|
||||
if rss == 0 && swap == 0 {
|
||||
continue // a kernel thread
|
||||
}
|
||||
pid, _ := strconv.Atoi(path.Base(dir))
|
||||
cmd := strings.TrimSpace(strings.ReplaceAll(m.read(dir+"/cmdline"), "\x00", " "))
|
||||
if len(cmd) > 160 {
|
||||
cmd = cmd[:160] + "…"
|
||||
}
|
||||
out = append(out, Process{PID: pid, Name: name, RSSMiB: mib(rss), SwapMiB: mib(swap),
|
||||
Unit: unitOf(m.read(dir + "/cgroup")), Command: cmd})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mib(b int64) float64 { return round1(float64(b) / (1 << 20)) }
|
||||
|
||||
// Top is the largest processes by resident plus swapped memory.
|
||||
func Top(ps []Process, limit int) []Process {
|
||||
sort.Slice(ps, func(i, j int) bool { return ps[i].RSSMiB+ps[i].SwapMiB > ps[j].RSSMiB+ps[j].SwapMiB })
|
||||
if len(ps) > limit {
|
||||
ps = ps[:limit]
|
||||
}
|
||||
return ps
|
||||
}
|
||||
|
||||
// ByUnit sums processes by their unit, largest first.
|
||||
func ByUnit(ps []Process, limit int) []Group {
|
||||
groups := map[string]*Group{}
|
||||
biggest := map[string]float64{}
|
||||
for _, p := range ps {
|
||||
g := groups[p.Unit]
|
||||
if g == nil {
|
||||
g = &Group{Unit: p.Unit}
|
||||
groups[p.Unit] = g
|
||||
}
|
||||
g.Processes++
|
||||
g.RSSMiB = round1(g.RSSMiB + p.RSSMiB)
|
||||
g.SwapMiB = round1(g.SwapMiB + p.SwapMiB)
|
||||
if p.RSSMiB+p.SwapMiB > biggest[p.Unit] {
|
||||
biggest[p.Unit] = p.RSSMiB + p.SwapMiB
|
||||
g.Largest = p.Name
|
||||
}
|
||||
}
|
||||
out := make([]Group, 0, len(groups))
|
||||
for _, g := range groups {
|
||||
out = append(out, *g)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].RSSMiB+out[i].SwapMiB > out[j].RSSMiB+out[j].SwapMiB })
|
||||
if len(out) > limit {
|
||||
out = out[:limit]
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const meminfo = `MemTotal: 32000000 kB
|
||||
MemFree: 905460 kB
|
||||
MemAvailable: 1000000 kB
|
||||
SwapTotal: 10000000 kB
|
||||
SwapFree: 5000000 kB
|
||||
`
|
||||
|
||||
// As /proc/swaps and zram read on the laptop on 2026-10-04.
|
||||
const swaps = `Filename Type Size Used Priority
|
||||
/swapfile file 33554428 0 10
|
||||
/dev/zram0 partition 16059900 7235584 100
|
||||
`
|
||||
|
||||
func (f *fake) machineWith(mem, psi string) *Machine {
|
||||
f.file("/proc/meminfo", mem)
|
||||
f.file("/proc/swaps", swaps)
|
||||
f.file("/proc/pressure/memory", psi)
|
||||
f.file("/sys/block/zram0/comp_algorithm", "lzo-rle lzo lz4 lz4hc [zstd] deflate 842\n")
|
||||
f.file("/sys/block/zram0/disksize", "16445341696\n")
|
||||
f.file("/sys/block/zram0/mm_stat", "14368768 5341499 6680576 0 6680576 2 0 0 0\n")
|
||||
return f.machine()
|
||||
}
|
||||
|
||||
const calm = "some avg10=0.00 avg60=0.02 avg300=0.00 total=14487028\nfull avg10=0.00 avg60=0.01 avg300=0.00 total=14230772\n"
|
||||
|
||||
func TestTheSnapshotReadsRAMSwapZramAndPressure(t *testing.T) {
|
||||
f := newFake(t)
|
||||
s, err := f.machineWith(meminfo, calm).Snapshot()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if s.UsedPercent != 96.9 || s.SwapPercent != 50 || len(s.Swaps) != 2 || s.Swaps[1].Priority != 100 {
|
||||
t.Fatalf("%+v", s)
|
||||
}
|
||||
z := s.Zram[0]
|
||||
if z.Algorithm != "zstd" || *z.Ratio != 2.7 || z.DiskSizeGiB != 15.3 || z.SamePages != 2 {
|
||||
t.Fatalf("%+v", z)
|
||||
}
|
||||
if s.PressureSome == nil || s.PressureSome.Avg60 != 0.02 || s.PressureFull == nil {
|
||||
t.Fatalf("%+v %+v", s.PressureSome, s.PressureFull)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheGuardWarnsOnWhicheverLineIsCrossedFirst(t *testing.T) {
|
||||
for name, c := range map[string]struct {
|
||||
s Snapshot
|
||||
want string
|
||||
}{
|
||||
"ram": {Snapshot{UsedPercent: 96}, "RAM"},
|
||||
"swap": {Snapshot{UsedPercent: 50, SwapTotalGiB: 10, SwapPercent: 85}, "swap"},
|
||||
"pressure": {Snapshot{UsedPercent: 50, PressureSome: &Pressure{Avg10: 45}}, "stalled"},
|
||||
} {
|
||||
v := Judge(c.s)
|
||||
if !v.High || len(v.Reasons) != 1 || !strings.Contains(v.Reasons[0], c.want) || v.Clear {
|
||||
t.Errorf("%s: %+v", name, v)
|
||||
}
|
||||
}
|
||||
if v := Judge(Snapshot{UsedPercent: 90}); v.High || v.Clear {
|
||||
t.Errorf("between the lines is neither a warning nor clear: %+v", v)
|
||||
}
|
||||
if v := Judge(Snapshot{UsedPercent: 50, PressureSome: &Pressure{}}); v.High || !v.Clear {
|
||||
t.Errorf("%+v", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheGuardWarnsOnceNamesTheLargestAndClearsOnlyBelowEveryLine(t *testing.T) {
|
||||
f := newFake(t)
|
||||
m := f.machineWith(meminfo, calm)
|
||||
f.file("/proc/4242/status", "Name:\tfirefox\nVmRSS:\t 6291456 kB\nVmSwap:\t 1048576 kB\n")
|
||||
f.file("/proc/4242/cgroup", "0::/user.slice/user-1000.slice/session-1.scope\n")
|
||||
f.file("/proc/4242/cmdline", "/usr/lib/firefox/firefox\x00")
|
||||
var events []string
|
||||
g := NewGuard(m, func(t string, _ any) error { events = append(events, t); return nil })
|
||||
var notes []string
|
||||
g.notify = func(_ context.Context, summary, body string) error {
|
||||
notes = append(notes, body)
|
||||
return nil
|
||||
}
|
||||
ctx := context.Background()
|
||||
g.Check(ctx)
|
||||
g.Check(ctx)
|
||||
if len(notes) != 1 || !strings.Contains(notes[0], "firefox 7.0 GiB") || !strings.Contains(notes[0], "RAM 97%") {
|
||||
t.Fatalf("%v", notes)
|
||||
}
|
||||
// Between the lines: still latched, nothing new.
|
||||
f.file("/proc/meminfo", strings.Replace(meminfo, "MemAvailable: 1000000", "MemAvailable: 3200000", 1))
|
||||
g.Check(ctx)
|
||||
if len(events) != 1 || !g.Report().Warned {
|
||||
t.Fatalf("%v %+v", events, g.Report())
|
||||
}
|
||||
f.file("/proc/meminfo", "MemTotal: 32000000 kB\nMemAvailable: 20000000 kB\nSwapTotal: 10000000 kB\nSwapFree: 9000000 kB\n")
|
||||
g.Check(ctx)
|
||||
if strings.Join(events, ",") != "pressure.high,pressure.cleared" || g.Report().Warned {
|
||||
t.Fatalf("%v", events)
|
||||
}
|
||||
}
|
||||
|
||||
func TestADesktopThatCannotBeReachedIsSaidAndTheEventStillGoes(t *testing.T) {
|
||||
f := newFake(t)
|
||||
m := f.machineWith(meminfo, calm)
|
||||
var events []string
|
||||
g := NewGuard(m, func(t string, _ any) error { events = append(events, t); return nil })
|
||||
g.notify = func(context.Context, string, string) error { return errors.New("no session bus at /run/user/1000/bus") }
|
||||
g.Check(context.Background())
|
||||
if r := g.Report(); !strings.HasPrefix(r.Desktop, "not reached") || len(events) != 1 {
|
||||
t.Fatalf("%+v %v", r, events)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheNotificationIsOneBusctlCallOnTheAccountsBus(t *testing.T) {
|
||||
f := newFake(t)
|
||||
g := NewGuard(f.machine(), nil)
|
||||
bus, err := Bus()
|
||||
if err != nil {
|
||||
t.Skip(err)
|
||||
}
|
||||
err = g.desktopNotify(context.Background(), "s", "b")
|
||||
calls := f.callsLike("env DBUS_SESSION_BUS_ADDRESS=unix:path=" + bus + " busctl --user call org.freedesktop.Notifications")
|
||||
if err == nil && len(calls) != 1 {
|
||||
t.Fatalf("%v", f.calls)
|
||||
}
|
||||
if err != nil && !strings.Contains(err.Error(), "no session bus") {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheLargestAreSummedByTheUnitOomdChoosesAmong(t *testing.T) {
|
||||
ps := []Process{
|
||||
{PID: 1, Name: "firefox", RSSMiB: 600, Unit: "session-1.scope"},
|
||||
{PID: 2, Name: "Isolated Web Co", RSSMiB: 900, Unit: "session-1.scope"},
|
||||
{PID: 3, Name: "postgres", RSSMiB: 100, Unit: "docker-abc.scope"},
|
||||
}
|
||||
g := ByUnit(ps, 10)
|
||||
if len(g) != 2 || g[0].Unit != "session-1.scope" || g[0].RSSMiB != 1500 || g[0].Largest != "Isolated Web Co" || g[0].Processes != 2 {
|
||||
t.Fatalf("%+v", g)
|
||||
}
|
||||
if top := Top(ps, 1); top[0].PID != 2 {
|
||||
t.Fatalf("%+v", top)
|
||||
}
|
||||
if u := unitOf("0::/user.slice/user-1000.slice/user@1000.service/app.slice/app-foot-123.scope\n"); u != "app-foot-123.scope" {
|
||||
t.Fatal(u)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheJournalIsReadForEveryKindOfKillAndSinceIsChecked(t *testing.T) {
|
||||
journal := strings.Join([]string{
|
||||
`{"__REALTIME_TIMESTAMP":"1759500000000000","MESSAGE":"Out of memory: Killed process 4242 (firefox) total-vm:1kB","_TRANSPORT":"kernel"}`,
|
||||
`{"__REALTIME_TIMESTAMP":"1759600000000000","MESSAGE":"Killed /user.slice/user-1000.slice/session-1.scope due to memory pressure for /user.slice being 84.12% > 80.00% for > 20s with reclaim activity","_SYSTEMD_UNIT":"systemd-oomd.service"}`,
|
||||
`{"__REALTIME_TIMESTAMP":"1759700000000000","MESSAGE":"docker.service: A process of this unit has been killed by the OOM killer.","_SYSTEMD_UNIT":"init.scope"}`,
|
||||
`{"__REALTIME_TIMESTAMP":"1759800000000000","MESSAGE":"Killed something else entirely","_SYSTEMD_UNIT":"bash.service"}`,
|
||||
`{"MESSAGE":[1,2,3]}`,
|
||||
}, "\n")
|
||||
kills := ParseJournal(journal)
|
||||
if len(kills) != 3 || kills[0].By != "service manager" || kills[0].Victim != "docker.service" ||
|
||||
kills[1].By != "systemd-oomd" || kills[2].Victim != "firefox (pid 4242)" || kills[2].Time.Year() != 2025 {
|
||||
t.Fatalf("%+v", kills)
|
||||
}
|
||||
f := newFake(t)
|
||||
m := f.machine()
|
||||
if _, err := m.OOMHistory(context.Background(), "--output=x", 10); err == nil {
|
||||
t.Fatal("an option was passed as since")
|
||||
}
|
||||
f.fails["journalctl --no-pager -q -o json --since -7 days -g Out of memory: Killed process|Killed .* due to|has been killed by the OOM killer"] = errors.New("journalctl: exit status 1")
|
||||
k, err := m.OOMHistory(context.Background(), "7d", 10)
|
||||
if err != nil || len(k) != 0 {
|
||||
t.Fatalf("no match is no kills: %v %v", k, err)
|
||||
}
|
||||
_ = time.Now
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Kill is one thing the machine killed for memory, from the journal.
|
||||
type Kill struct {
|
||||
Time time.Time `json:"time"`
|
||||
By string `json:"by"`
|
||||
Victim string `json:"victim,omitempty"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// The three voices that report a kill: the kernel's OOM killer, systemd-oomd, and the service manager
|
||||
// saying one of its units lost a process to the OOM killer.
|
||||
var (
|
||||
kernelKill = regexp.MustCompile(`Out of memory: Killed process (\d+) \(([^)]*)\)`)
|
||||
oomdKill = regexp.MustCompile(`Killed (\S+) due to (.+)`)
|
||||
unitKilled = regexp.MustCompile(`^(\S+): A process of this unit has been killed by the OOM killer`)
|
||||
)
|
||||
|
||||
// sinceArg is a value journalctl understands after --since; anything else is refused before it
|
||||
// reaches the command line.
|
||||
var sinceArg = regexp.MustCompile(`^(-?\d+[smhdw]|today|yesterday|\d{4}-\d{2}-\d{2}( \d{2}:\d{2}(:\d{2})?)?)$`)
|
||||
|
||||
// ParseJournal reads `journalctl -o json` lines into kills, newest first.
|
||||
func ParseJournal(text string) []Kill {
|
||||
var out []Kill
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
if strings.TrimSpace(line) == "" {
|
||||
continue
|
||||
}
|
||||
var e map[string]any
|
||||
if json.Unmarshal([]byte(line), &e) != nil {
|
||||
continue
|
||||
}
|
||||
msg, ok := e["MESSAGE"].(string) // a message that is not UTF-8 arrives as bytes; not one of these
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
var k Kill
|
||||
if us, err := strconv.ParseInt(fmt.Sprint(e["__REALTIME_TIMESTAMP"]), 10, 64); err == nil {
|
||||
k.Time = time.UnixMicro(us).UTC()
|
||||
}
|
||||
switch {
|
||||
case kernelKill.MatchString(msg):
|
||||
g := kernelKill.FindStringSubmatch(msg)
|
||||
k.By, k.Victim = "kernel", g[2]+" (pid "+g[1]+")"
|
||||
case oomdKill.MatchString(msg) && strings.Contains(fmt.Sprint(e["_SYSTEMD_UNIT"], e["SYSLOG_IDENTIFIER"]), "oomd"):
|
||||
g := oomdKill.FindStringSubmatch(msg)
|
||||
k.By, k.Victim = "systemd-oomd", g[1]
|
||||
case unitKilled.MatchString(msg):
|
||||
k.By, k.Victim = "service manager", unitKilled.FindStringSubmatch(msg)[1]
|
||||
default:
|
||||
continue
|
||||
}
|
||||
k.Message = msg
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].Time.After(out[j].Time) })
|
||||
return out
|
||||
}
|
||||
|
||||
// OOMHistory searches the journal since a time for every kill.
|
||||
func (m *Machine) OOMHistory(ctx context.Context, since string, limit int) ([]Kill, error) {
|
||||
if !sinceArg.MatchString(since) {
|
||||
return nil, fmt.Errorf("since %q is not -7d, 12h, today, yesterday or a date (2026-10-01)", since)
|
||||
}
|
||||
since = expandRelative(since)
|
||||
out, err := m.Run(ctx, "journalctl", "--no-pager", "-q", "-o", "json", "--since", since,
|
||||
"-g", "Out of memory: Killed process|Killed .* due to|has been killed by the OOM killer")
|
||||
if err != nil && strings.TrimSpace(out) == "" {
|
||||
if strings.HasSuffix(err.Error(), "exit status 1") {
|
||||
return []Kill{}, nil // journalctl exits 1 when a grep matches nothing
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
kills := ParseJournal(out)
|
||||
if kills == nil {
|
||||
kills = []Kill{}
|
||||
}
|
||||
if len(kills) > limit {
|
||||
kills = kills[:limit]
|
||||
}
|
||||
return kills, nil
|
||||
}
|
||||
|
||||
// expandRelative turns -7d (or 7d) into the "-7 days" form journalctl's --since reads.
|
||||
func expandRelative(s string) string {
|
||||
g := regexp.MustCompile(`^-?(\d+)([smhdw])$`).FindStringSubmatch(s)
|
||||
if g == nil {
|
||||
return s
|
||||
}
|
||||
unit := map[string]string{"s": "seconds", "m": "minutes", "h": "hours", "d": "days", "w": "weeks"}[g[2]]
|
||||
return "-" + g[1] + " " + unit
|
||||
}
|
||||
|
||||
// Oomd is what systemd-oomd says it watches.
|
||||
type Oomd struct {
|
||||
Active string `json:"active"`
|
||||
Enabled string `json:"enabled"`
|
||||
Report []string `json:"oomctl"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
func (m *Machine) Oomd(ctx context.Context) Oomd {
|
||||
o := Oomd{Report: []string{}}
|
||||
a, _ := m.Run(ctx, "systemctl", "is-active", "systemd-oomd.service")
|
||||
e, _ := m.Run(ctx, "systemctl", "is-enabled", "systemd-oomd.service")
|
||||
o.Active, o.Enabled = strings.TrimSpace(a), strings.TrimSpace(e)
|
||||
out, err := m.Run(ctx, "oomctl")
|
||||
if err != nil {
|
||||
o.Note = "oomctl: " + err.Error()
|
||||
}
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if strings.TrimSpace(line) != "" {
|
||||
o.Report = append(o.Report, strings.TrimRight(strings.ReplaceAll(line, "\t", " "), " "))
|
||||
}
|
||||
}
|
||||
return o
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// Tools is the module's tools over one machine and its guard.
|
||||
func Tools(m *Machine, g *Guard) []stdio.Tool {
|
||||
ctx := context.Background
|
||||
return []stdio.Tool{
|
||||
{
|
||||
Name: "memory_status",
|
||||
Description: "Memory now: RAM total, available and used, swap and each swap device with its priority, the compressed swap in RAM " +
|
||||
"(zram) and its ratio, and pressure stall (PSI some/full, avg10/60/300) — with the guard's verdict on it.",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
s, err := m.Snapshot()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return map[string]any{"memory": s, "verdict": Judge(s)}, nil
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "memory_top",
|
||||
Description: "The largest processes by resident plus swapped memory, with the systemd unit each runs in; by=unit sums them per unit, which is what systemd-oomd chooses among.",
|
||||
Input: map[string]any{
|
||||
"limit": map[string]any{"type": "integer", "description": "how many (default 10, at most 50)"},
|
||||
"by": map[string]any{"type": "string", "enum": []string{"process", "unit"}},
|
||||
},
|
||||
Run: func(args map[string]any) (any, error) {
|
||||
limit, err := bounded(args, "limit", 10, 50)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ps := m.Processes()
|
||||
switch str(args, "by") {
|
||||
case "", "process":
|
||||
return map[string]any{"processes": orEmpty(Top(ps, limit))}, nil
|
||||
case "unit":
|
||||
return map[string]any{"units": orEmpty(ByUnit(ps, limit))}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("by is process or unit")
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "memory_oom_history",
|
||||
Description: "What was killed for memory, newest first, from the journal: the kernel's OOM killer, systemd-oomd, and units the service manager says lost a process to it.",
|
||||
Input: map[string]any{
|
||||
"since": map[string]any{"type": "string", "description": "-7d (default), 12h, today, yesterday or a date like 2026-10-01"},
|
||||
"limit": map[string]any{"type": "integer", "description": "how many (default 20, at most 100)"},
|
||||
},
|
||||
Run: func(args map[string]any) (any, error) {
|
||||
limit, err := bounded(args, "limit", 20, 100)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
since := str(args, "since")
|
||||
if since == "" {
|
||||
since = "-7d"
|
||||
}
|
||||
kills, err := m.OOMHistory(ctx(), since, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return map[string]any{"since": since, "kills": kills}, nil
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "memory_zram",
|
||||
Description: "The compressed swap in RAM: each zram device's algorithm, size, what it stores, what that costs in RAM and the ratio, the generator's configuration, and the swap tunables beside it.",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
return map[string]any{
|
||||
"devices": m.Zram(),
|
||||
"configuration": m.read("/etc/systemd/zram-generator.conf"),
|
||||
"vm": map[string]string{
|
||||
"swappiness": m.read("/proc/sys/vm/swappiness"),
|
||||
"page-cluster": m.read("/proc/sys/vm/page-cluster"),
|
||||
},
|
||||
"note": "a change to the configuration applies at the next boot: swapping the device off to resize it would push what it holds back into RAM",
|
||||
}, nil
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "memory_oomd",
|
||||
Description: "systemd-oomd: whether it runs, and what it watches — the cgroups, their limits and their pressure, as oomctl reports them.",
|
||||
Run: func(map[string]any) (any, error) { return m.Oomd(ctx()), nil },
|
||||
},
|
||||
{
|
||||
Name: "memory_guard",
|
||||
Description: "The module's guard that warns before the machine kills for memory: its thresholds, its last verdict, whether a warning stands, and whether the operator's desktop can be reached.",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
if g == nil {
|
||||
return nil, fmt.Errorf("the guard does not run in this process")
|
||||
}
|
||||
return g.Report(), nil
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func str(args map[string]any, key string) string {
|
||||
s, _ := args[key].(string)
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
// bounded is an integer argument, defaulted, at least 1 and at most most.
|
||||
func bounded(args map[string]any, key string, fallback, most int) (int, error) {
|
||||
v, given := args[key]
|
||||
if !given || v == nil {
|
||||
return fallback, nil
|
||||
}
|
||||
var n int
|
||||
switch x := v.(type) {
|
||||
case float64:
|
||||
if x != math.Trunc(x) {
|
||||
return 0, fmt.Errorf("%s must be a whole number, not %v", key, x)
|
||||
}
|
||||
n = int(x)
|
||||
case string:
|
||||
i, err := strconv.Atoi(strings.TrimSpace(x))
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%s must be a whole number, not %q", key, x)
|
||||
}
|
||||
n = i
|
||||
default:
|
||||
return 0, fmt.Errorf("%s must be a whole number", key)
|
||||
}
|
||||
if n < 1 {
|
||||
return 0, fmt.Errorf("%s must be at least 1", key)
|
||||
}
|
||||
return min(n, most), nil
|
||||
}
|
||||
|
||||
func orEmpty[T any](s []T) []T {
|
||||
if s == nil {
|
||||
return []T{}
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
module memorypressure
|
||||
|
||||
go 1.22
|
||||
|
||||
require git.novox.be/novox/mesh-sdk/go v0.1.6
|
||||
@@ -0,0 +1,2 @@
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.6 h1:9qzdYONYbJdWcu6sxQcq9v1LI0JxcfkiKYkMUzJSkVQ=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.6/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"module": "memory-pressure",
|
||||
"version": "1",
|
||||
"capabilities": [
|
||||
"package-manager",
|
||||
"service-manager"
|
||||
],
|
||||
"emits": [
|
||||
"pressure.high",
|
||||
"pressure.cleared"
|
||||
],
|
||||
"tools": [
|
||||
"memory_status",
|
||||
"memory_top",
|
||||
"memory_oom_history",
|
||||
"memory_zram",
|
||||
"memory_oomd",
|
||||
"memory_guard"
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"id": "zram-generator",
|
||||
"type": "package",
|
||||
"package": "zram-generator"
|
||||
},
|
||||
{
|
||||
"id": "zram",
|
||||
"type": "file",
|
||||
"path": "/etc/systemd/zram-generator.conf",
|
||||
"mode": "0644",
|
||||
"content": "# Managed by the mesh (module memory-pressure). Replaced on every push; edit the catalogue instead.\n#\n# Compressed swap in RAM: fast, at a higher priority than any swap on disk, so it takes the everyday\n# pressure before anything spills to a disk. Half the RAM, at most 16 GiB, compressed with zstd.\n# The swap on disk, its size and whether one exists at all are the machine's, not this module's.\n#\n# Read by the generator at boot: a change applies at the next boot.\n[zram0]\nzram-size = min(ram / 2, 16384)\ncompression-algorithm = zstd\nswap-priority = 100\n"
|
||||
},
|
||||
{
|
||||
"id": "sysctl-drop-ins",
|
||||
"type": "directory",
|
||||
"path": "/etc/sysctl.d",
|
||||
"mode": "0755"
|
||||
},
|
||||
{
|
||||
"id": "swap-tunables",
|
||||
"type": "file",
|
||||
"path": "/etc/sysctl.d/90-memory-pressure.conf",
|
||||
"mode": "0644",
|
||||
"content": "# Managed by the mesh (module memory-pressure). Replaced on every push; edit the catalogue instead.\n#\n# Swap-in reads one page, not eight: read-ahead exists to amortise a disk's seek, and compressed swap in\n# RAM has none — the speculation only costs decompression and memory.\nvm.page-cluster = 0\n#\n# vm.swappiness is deliberately left alone. The usual zram advice (150-180) holds only while the\n# compressed swap has room; once it is full, a higher swappiness moves pages to the disk swap, the thing\n# being avoided. Raise it only together with zram-size.\n"
|
||||
},
|
||||
{
|
||||
"id": "sysctl",
|
||||
"type": "service",
|
||||
"unit": "systemd-sysctl.service",
|
||||
"restart-on": [
|
||||
"swap-tunables"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "oomd-drop-ins",
|
||||
"type": "directory",
|
||||
"path": "/etc/systemd/oomd.conf.d",
|
||||
"mode": "0755"
|
||||
},
|
||||
{
|
||||
"id": "oomd-limits",
|
||||
"type": "file",
|
||||
"path": "/etc/systemd/oomd.conf.d/memory-pressure.conf",
|
||||
"mode": "0644",
|
||||
"content": "# Managed by the mesh (module memory-pressure). Replaced on every push; edit the catalogue instead.\n#\n# systemd-oomd kills the worst unit before the kernel's OOM killer freezes the machine: when swap is\n# 90 % used, or when a watched unit stalls on memory for 20 s above its limit.\n[OOM]\nSwapUsedLimit=90%\nDefaultMemoryPressureLimit=60%\nDefaultMemoryPressureDurationSec=20s\n"
|
||||
},
|
||||
{
|
||||
"id": "root-slice-drop-ins",
|
||||
"type": "directory",
|
||||
"path": "/etc/systemd/system/-.slice.d",
|
||||
"mode": "0755"
|
||||
},
|
||||
{
|
||||
"id": "root-slice",
|
||||
"type": "file",
|
||||
"path": "/etc/systemd/system/-.slice.d/10-oomd.conf",
|
||||
"mode": "0644",
|
||||
"content": "# Managed by the mesh (module memory-pressure). Replaced on every push; edit the catalogue instead.\n#\n# systemd-oomd may act on swap exhaustion across the whole machine.\n[Slice]\nManagedOOMSwap=kill\n"
|
||||
},
|
||||
{
|
||||
"id": "user-manager-drop-ins",
|
||||
"type": "directory",
|
||||
"path": "/etc/systemd/system/user@.service.d",
|
||||
"mode": "0755"
|
||||
},
|
||||
{
|
||||
"id": "user-manager",
|
||||
"type": "file",
|
||||
"path": "/etc/systemd/system/user@.service.d/10-oomd.conf",
|
||||
"mode": "0644",
|
||||
"content": "# Managed by the mesh (module memory-pressure). Replaced on every push; edit the catalogue instead.\n#\n# systemd-oomd may kill the worst unit in a person's service manager when its memory pressure stays\n# above 80 % — instead of the kernel freezing the machine.\n[Service]\nManagedOOMMemoryPressure=kill\nManagedOOMMemoryPressureLimit=80%\n"
|
||||
},
|
||||
{
|
||||
"id": "oomd",
|
||||
"type": "service",
|
||||
"unit": "systemd-oomd.service",
|
||||
"state": "running",
|
||||
"boot": "enabled",
|
||||
"restart-on": [
|
||||
"oomd-limits",
|
||||
"root-slice",
|
||||
"user-manager"
|
||||
]
|
||||
}
|
||||
],
|
||||
"build": {
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "tools-go",
|
||||
"kind": "bundle",
|
||||
"language": "go",
|
||||
"system": "arch",
|
||||
"from": "cmd/memory-pressure",
|
||||
"binary": "memory-pressure",
|
||||
"loads": [
|
||||
"memory-pressure"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user