From c0c7515e06c3683dc74c0e7d82c6b2e091a6b730 Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 10 Oct 2026 01:03:51 +0200 Subject: [PATCH] Issue 361: the control-node's agent account is never judged, because its search for setuid programs cannot finish in fifteen minutes --- .../00-report.md | 61 ++++++++++++++ .../01-diagnosis.md | 79 +++++++++++++++++++ 2 files changed, 140 insertions(+) create mode 100644 04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md create mode 100644 04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md diff --git a/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md new file mode 100644 index 00000000..bddcc81a --- /dev/null +++ b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md @@ -0,0 +1,61 @@ +--- +status: located +opened: 2026-10-10 +located-in: [mesh-host (internal/accounts, rootsearch), mesh-controller (cmd/mesh-controller/agent_account.go, internal/link)] +fixed-by: +amended-design: +--- + +# 361. The control-node's agent account is never judged: its search for setuid programs cannot finish in fifteen minutes + +**Severity: high.** The router refuses to link a messaging channel on a machine that is not root-free +([ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md) §8), +and a machine whose agent account is not judged is not root-free. On the control-node the judgement can never +be given, so linking the operator's phone there is blocked for as long as the search is bounded. + +## Symptom + +On 2026-10-10 at 00:43 (local), after the control-node was given an agent account +([ADR 0266](../../02-DECISIONS/0266-on-the-control-node-agents-run-under-an-account-of-their-own-that-cannot-become-root-without-a-person.md)), +the controller's `root-free` verb for that machine answered: + +> not judged (search incomplete): the search for setuid programs did not finish within 15m0s; it is tried +> again later + +The control-node holds its root and home filesystems on a software RAID of spinning disks, with millions of +files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) walks every local +filesystem mounted without `nosuid`; that walk, cold, takes far longer than fifteen minutes there. Each attempt +was killed at its bound, judged nothing, and was tried again after a back-off, which ends the same way. + +## Why it is a design fault, not one slow disk + +- **A bounded search judges nothing on a machine larger than the bound.** The bound was chosen so the + controller could tell "still running" from "stuck" (`rootsearch.Bound`, read by both). It turned a slow + machine into one that is never judged, failing closed for good rather than for a while. +- **Nothing was kept across a restart.** The search's result lived in the engine's memory only, so every + engine restart (every update of the engine) started from nothing and judged nothing until a whole walk + finished again. +- **The controller's quiet was the same value.** The self-check stayed quiet while a search was "within its + bound" — fifteen minutes — so a search that runs for hours would raise the urgent condition on every + restart even once the search itself was allowed to finish. + +## What is asked (the operator's direction, 2026-10-10) + +1. The search runs to completion, in the background, at the lowest CPU and I/O priority, one at a time, with + no bound on the whole search. +2. It is resumable: its progress is kept every minute beside the engine's state, so a restart, an update or a + reboot carries it on from where it was rather than starting over. +3. Its outcome is kept, with when it started and finished. A verdict is given only from the last **complete** + search while it is younger than a stated age; "not judged yet" before the first one completes; never + "cannot become root" from an incomplete or stale one. +4. A fresh search runs at most once a day, and after any apply that changes an account, a group, a sudo rule or + a package. +5. The controller's own reading of the search (its quiet) changes with it, consistently. + +**Why a day-old search is sound.** Only root can make a setuid- or setgid-root file: the agent account can +neither set the bit on a file root owns nor give a file of its own to root. So a complete search stays true until +root acts, and root's acts through the mesh (applies) start a new search. A file that appears behind a running +walk's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age +bounds. + +The trail and the fix are in [`01-diagnosis.md`](01-diagnosis.md). diff --git a/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md new file mode 100644 index 00000000..4726fb50 --- /dev/null +++ b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md @@ -0,0 +1,79 @@ +# 361 — diagnosis + +## 2026-10-10 — where the bound lives, and who read it + +- **The node-engine** (`mesh-host`, `internal/accounts`): `SetuidCache` ran the search in the background, at most + hourly, under a context with a deadline of `rootsearch.Bound` (15 minutes) for the whole search — `find` over + every mount point with `-xdev`, then the package manager about every path found. A search past the deadline + was an error, said as `not judged (search incomplete)`, and tried again after a back-off doubling from an hour + to six. The result was kept in memory only. +- **The controller** (`mesh-controller`, `cmd/mesh-controller/agent_account.go`): `searchQuietFor` was + `link.RootSearchBound`, which is `rootsearch.Bound`. It decides only how long the self-check raises nothing + while an agent account's verdict says `not judged yet (search running)`, counted from when the controller + first saw it waiting. The `root-free` verb and `judgedConfined` never read it: they need a healthy verdict in + a statement heard within 15 minutes (`verdictFreshFor`, a different value — how often an engine states its + health — which this issue does not change). +- **Nothing else** used the bound: `SetuidCache` had one caller (`moreWays`), and the engine made one cache. +- **Ruled out:** a stuck `find` (each attempt ended at exactly its bound) and a filesystem that should not be + searched (the walk covered only the local root, home and boot filesystems, as ADR 0266 asks). + +## The fix (`mesh-host` pull request 75, `mesh-controller` pull request 190) + +**The walk is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as +`find -xdev` did, with the same prunes and the same match (a regular file, setuid and owned by root, or setgid +and of root's group), never following a link. It visits each directory's entries in lexical order, so its +progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is walked before +`/a-c` although `-` sorts before `/`). Every minute the progress (when the walk started, its filesystems, those +done, the cursor, the matches so far) is written beside the engine's state as a new file renamed into place, +readable by root alone. After a restart the walk carries on from the cursor: what is before it is skipped +without being read, the directories leading back to it are read again. A walk older than the stated age, or one +an apply asked to redo, is thrown away and begun again. + +**Low priority, one at a time, no bound.** The walk runs on a goroutine locked to an operating-system thread of +its own, set to nice 19 and the idle I/O class; on Linux both are a thread's, so the engine's other work keeps +its priority, and the thread ends with the goroutine. The package manager is run the same way. A priority that +cannot be lowered fails the search rather than walk at full priority. There is no deadline: the search ends when +it has walked everything, or with an error (a directory it could not read), which is incomplete. + +**A verdict only from a complete, fresh search.** The last complete search is kept beside the engine's state +with its start and finish, and judges while it is younger than `rootsearch.FreshFor`, counted from its start. +Before one completes, or once it is older, the verdict is `not judged yet (search running)`; after a failure, +`not judged (search incomplete)`. A kept file is believed only as a regular file, opened without following a +link, owned by the engine's account and writable by it alone, in a directory that is the same. + +**When a search runs.** Once a day (`rootsearch.Every`, from the last complete one's start); after an apply +whose outcomes moved a user, a package, an action, or a file of the user and group databases or of sudo's, +doas's or polkit's rules (`apply.MovesRoot`); and when a filesystem appears that the last search did not walk. +The last two at most once an hour. The last complete search keeps judging meanwhile. + +**The values, one place** (`rootsearch`, read by both repositories): + +| Value | What it is | Set to | +|---|---|---| +| `Every` | how often a full search runs at most | 24 hours | +| `Quiet` | how long the controller raises nothing while an agent account waits for a search, from when it first saw it waiting | 12 hours | +| `FreshFor` | how long a complete search judges, from its start | `Every` + `Quiet` = 36 hours | + +`FreshFor` is `Every` plus `Quiet` so that a daily search which finishes within the controller's quiet never +leaves an hour unjudged; the controller's test holds that relation. The operator's direction named 26 hours as an +example; 36 is chosen because a full walk at idle priority on the control-node is expected to take hours, and +with 26 hours a daily search longer than two hours would leave the account unjudged — and phone linking refused — +every day until it finished. + +## Open, after the fix + +- **How long the first walk on the control-node takes is not measured.** Until the first complete search, the + verdict there is `not judged yet`, and linking stays refused. The engine says each search's end and how long + it ran in its journal; that line is the measurement. If a walk takes longer than `Quiet`, the self-check + raises the urgent condition, which is right: it is then not known to finish. +- **The idle I/O class is honoured only by a disk scheduler that has classes** (BFQ). Under another scheduler + the walk still runs at nice 19 but competes for the disk as any reader does. +- **An ask to search again is not kept across a restart.** An apply that moved root's things and a restart + before the next search starts lose the ask; the daily search covers it. The last complete search keeps + judging either way, on the reasoning in the report. +- **ADR 0266 §3 item 10** still says "at most hourly" and "at most 15 minutes". On resolution it gains a + progressive insight naming this issue and the values above. +- **A later option, not built now:** a fanotify filesystem mark with `FAN_ATTRIB` (Linux 5.1 and later) on + every searched filesystem would report each change of mode or owner as it happens, turning the daily walk into + a true incremental search. +- **Resolution** of this core issue needs a replay in mesh-lab's register, or `replay-none:` with the reason.