diff --git a/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md new file mode 100644 index 00000000..bddcc81a --- /dev/null +++ b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md @@ -0,0 +1,61 @@ +--- +status: located +opened: 2026-10-10 +located-in: [mesh-host (internal/accounts, rootsearch), mesh-controller (cmd/mesh-controller/agent_account.go, internal/link)] +fixed-by: +amended-design: +--- + +# 361. The control-node's agent account is never judged: its search for setuid programs cannot finish in fifteen minutes + +**Severity: high.** The router refuses to link a messaging channel on a machine that is not root-free +([ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md) §8), +and a machine whose agent account is not judged is not root-free. On the control-node the judgement can never +be given, so linking the operator's phone there is blocked for as long as the search is bounded. + +## Symptom + +On 2026-10-10 at 00:43 (local), after the control-node was given an agent account +([ADR 0266](../../02-DECISIONS/0266-on-the-control-node-agents-run-under-an-account-of-their-own-that-cannot-become-root-without-a-person.md)), +the controller's `root-free` verb for that machine answered: + +> not judged (search incomplete): the search for setuid programs did not finish within 15m0s; it is tried +> again later + +The control-node holds its root and home filesystems on a software RAID of spinning disks, with millions of +files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) walks every local +filesystem mounted without `nosuid`; that walk, cold, takes far longer than fifteen minutes there. Each attempt +was killed at its bound, judged nothing, and was tried again after a back-off, which ends the same way. + +## Why it is a design fault, not one slow disk + +- **A bounded search judges nothing on a machine larger than the bound.** The bound was chosen so the + controller could tell "still running" from "stuck" (`rootsearch.Bound`, read by both). It turned a slow + machine into one that is never judged, failing closed for good rather than for a while. +- **Nothing was kept across a restart.** The search's result lived in the engine's memory only, so every + engine restart (every update of the engine) started from nothing and judged nothing until a whole walk + finished again. +- **The controller's quiet was the same value.** The self-check stayed quiet while a search was "within its + bound" — fifteen minutes — so a search that runs for hours would raise the urgent condition on every + restart even once the search itself was allowed to finish. + +## What is asked (the operator's direction, 2026-10-10) + +1. The search runs to completion, in the background, at the lowest CPU and I/O priority, one at a time, with + no bound on the whole search. +2. It is resumable: its progress is kept every minute beside the engine's state, so a restart, an update or a + reboot carries it on from where it was rather than starting over. +3. Its outcome is kept, with when it started and finished. A verdict is given only from the last **complete** + search while it is younger than a stated age; "not judged yet" before the first one completes; never + "cannot become root" from an incomplete or stale one. +4. A fresh search runs at most once a day, and after any apply that changes an account, a group, a sudo rule or + a package. +5. The controller's own reading of the search (its quiet) changes with it, consistently. + +**Why a day-old search is sound.** Only root can make a setuid- or setgid-root file: the agent account can +neither set the bit on a file root owns nor give a file of its own to root. So a complete search stays true until +root acts, and root's acts through the mesh (applies) start a new search. A file that appears behind a running +walk's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age +bounds. + +The trail and the fix are in [`01-diagnosis.md`](01-diagnosis.md). diff --git a/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md new file mode 100644 index 00000000..3838ff01 --- /dev/null +++ b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md @@ -0,0 +1,112 @@ +# 361 — diagnosis + +## 2026-10-10 — where the bound lives, and who read it + +- **The node-engine** (`mesh-host`, `internal/accounts`): `SetuidCache` ran the search in the background, at most + hourly, under a context with a deadline of `rootsearch.Bound` (15 minutes) for the whole search — `find` over + every mount point with `-xdev`, then the package manager about every path found. A search past the deadline + was an error, said as `not judged (search incomplete)`, and tried again after a back-off doubling from an hour + to six. The result was kept in memory only. +- **The controller** (`mesh-controller`, `cmd/mesh-controller/agent_account.go`): `searchQuietFor` was + `link.RootSearchBound`, which is `rootsearch.Bound`. It decides only how long the self-check raises nothing + while an agent account's verdict says `not judged yet (search running)`, counted from when the controller + first saw it waiting. The `root-free` verb and `judgedConfined` never read it: they need a healthy verdict in + a statement heard within 15 minutes (`verdictFreshFor`, a different value — how often an engine states its + health — which this issue does not change). +- **Nothing else** used the bound: `SetuidCache` had one caller (`moreWays`), and the engine made one cache. +- **Ruled out:** a stuck `find` (each attempt ended at exactly its bound) and a filesystem that should not be + searched (the walk covered only the local root, home and boot filesystems, as ADR 0266 asks). + +## The fix (`mesh-host` pull request 75, `mesh-controller` pull request 190) + +**The walk is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as +`find -xdev` did, with the same prunes and the same match (a regular file, setuid and owned by root, or setgid +and of root's group), never following a link. It visits each directory's entries in lexical order, so its +progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is walked before +`/a-c` although `-` sorts before `/`). Every minute the progress (when the walk started, its filesystems, those +done, the cursor, the matches so far) is written beside the engine's state as a new file renamed into place, +readable by root alone. After a restart the walk carries on from the cursor: what is before it is skipped +without being read, the directories leading back to it are read again. A walk older than the stated age, or one +an apply asked to redo, is thrown away and begun again. + +**Low priority, one at a time, no bound.** The walk runs on a goroutine locked to an operating-system thread of +its own, set to nice 19 and the idle I/O class; on Linux both are a thread's, so the engine's other work keeps +its priority, and the thread ends with the goroutine. The package manager is run the same way. A priority that +cannot be lowered fails the search rather than walk at full priority. There is no deadline: the search ends when +it has walked everything, or with an error (a directory it could not read), which is incomplete. + +**A verdict only from a complete, fresh search.** The last complete search is kept beside the engine's state +with its start and finish, and judges while it is younger than `rootsearch.FreshFor`, counted from its start. +Before one completes, or once it is older, the verdict is `not judged yet (search running)`; after a failure, +`not judged (search incomplete)`. A kept file is believed only as a regular file, opened without following a +link, owned by the engine's account and writable by it alone, in a directory that is the same. + +**When a search runs.** Once a day (`rootsearch.Every`, from the last complete one's start); after an apply +whose outcomes moved a user, a package, an action, or a file of the user and group databases or of sudo's, +doas's or polkit's rules (`apply.MovesRoot`), at most once an hour from the last search's end, the last complete +search judging meanwhile; and at once when a filesystem appears that the last search did not walk, which that +search does not judge. + +**The values, one place** (`rootsearch`, read by both repositories): + +| Value | What it is | Set to | +|---|---|---| +| `Every` | how often a full search runs at most | 24 hours | +| `Quiet` | how long the controller raises nothing while an agent account waits for a search, from when it first saw it waiting | 12 hours | +| `FreshFor` | how long a complete search judges, from its start | `Every` + `Quiet` = 36 hours | + +`FreshFor` is `Every` plus `Quiet` so that a daily search which finishes within the controller's quiet never +leaves an hour unjudged; the controller's test holds that relation. The operator's direction named 26 hours as an +example; 36 is chosen because a full walk at idle priority on the control-node is expected to take hours, and +with 26 hours a daily search longer than two hours would leave the account unjudged — and phone linking refused — +every day until it finished. + +## 2026-10-10 — the independent review, and what it changed + +An independent review of both pull requests asked four questions: can a stale or incomplete search yield "free"; +can the agent account steer the kept files or the walk; does the walk hurt the machine; do the controller and the +engine agree. What it found, and what was done: + +- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed walk + skipped every name that sorted between the real cursor and its replacement, and could finish "free" without + having read them. Any account can make such a name. The walk's paths are now kept as their bytes. +- **An ask to search again was lost to a restart**, and the restarted engine carried on the walk that started + before root acted. The ask now forgets the walk under way and stops it keeping progress. +- **A fresh search judged a filesystem it never walked.** A complete search now judges only while it walked every + filesystem a search would walk now; a filesystem mounted since makes the verdict "not judged yet" until walked. +- **A mount point inside a walked filesystem was read before it was known to be another filesystem.** A FUSE + mount root cannot read failed the walk, and a network mount whose server is away could hang it. Every mount + point the kernel lists is now left unread. Mount points are read with all the kernel's escapes. +- **A file found and gone before the package manager was asked failed every resumed search.** The walk looks at + what it found again when it ends, and drops what went or is no longer setuid-root. +- **Outcomes that moved nothing started a search**, and a search asked for could follow a walk of hours at once. + Only created, updated, corrected and removed outcomes count now, and an ask waits an hour from the last + search's end. +- **Not changed, because the mesh cannot do it:** the review asked that a file or archive declared setuid start a + search. The engine writes a file's and an archive's permission bits only, so neither carries a setuid bit. + +## Open, after the fix + +- **How long the first walk on the control-node takes is not measured.** Until the first complete search, the + verdict there is `not judged yet`, and linking stays refused. The engine says each search's end and how long + it ran in its journal; that line is the measurement. If a walk takes longer than `Quiet`, the self-check + raises the urgent condition, which is right: it is then not known to finish. +- **The idle I/O class is honoured only by a disk scheduler that has classes** (BFQ). Under another scheduler + the walk still runs at nice 19 but competes for the disk as any reader does. +- **An ask to search again is not kept across a restart.** The walk under way is forgotten when it is asked, + so a restarted engine does not carry an older walk on; the ask itself is lost, and the daily search covers it. + The last complete search keeps judging meanwhile, on the reasoning in the report. +- **The agent account can make the search fail, never pass.** A tree deeper than the longest path the kernel + takes, or a directory unreadable to root through FUSE, ends every walk incomplete; a directory of millions of + entries makes the engine hold them all at once. Each is "not judged" — the urgent condition, and linking + refused — never a pass. Reading by directory handle, as `find` did, would close the first. +- **A filesystem mounted only on use** (an automounted boot partition) makes the verdict "not judged yet" each + time it appears unwalked, until a walk covers it. +- **Part of the walk's cost escapes its priority**: Go's garbage collector runs on threads of normal priority, + and a software RAID's own threads do not carry the idle class to the disks. +- **ADR 0266 §3 item 10** still says "at most hourly" and "at most 15 minutes". On resolution it gains a + progressive insight naming this issue and the values above. +- **A later option, not built now:** a fanotify filesystem mark with `FAN_ATTRIB` (Linux 5.1 and later) on + every searched filesystem would report each change of mode or owner as it happens, turning the daily walk into + a true incremental search. +- **Resolution** of this core issue needs a replay in mesh-lab's register, or `replay-none:` with the reason.