Issue 361: the control-node's agent account is never judged, because its search for setuid programs cannot finish in fifteen minutes

04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md

@@ -0,0 +1,61 @@
+---
+status: located
+opened: 2026-10-10
+located-in: [mesh-host (internal/accounts, rootsearch), mesh-controller (cmd/mesh-controller/agent_account.go, internal/link)]
+fixed-by:
+amended-design:
+---
+
+# 361. The control-node's agent account is never judged: its search for setuid programs cannot finish in fifteen minutes
+
+**Severity: high.** The router refuses to link a messaging channel on a machine that is not root-free
+([ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md) §8),
+and a machine whose agent account is not judged is not root-free. On the control-node the judgement can never
+be given, so linking the operator's phone there is blocked for as long as the search is bounded.
+
+## Symptom
+
+On 2026-10-10 at 00:43 (local), after the control-node was given an agent account
+([ADR 0266](../../02-DECISIONS/0266-on-the-control-node-agents-run-under-an-account-of-their-own-that-cannot-become-root-without-a-person.md)),
+the controller's `root-free` verb for that machine answered:
+
+> not judged (search incomplete): the search for setuid programs did not finish within 15m0s; it is tried
+> again later
+
+The control-node holds its root and home filesystems on a software RAID of spinning disks, with millions of
+files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) walks every local
+filesystem mounted without `nosuid`; that walk, cold, takes far longer than fifteen minutes there. Each attempt
+was killed at its bound, judged nothing, and was tried again after a back-off, which ends the same way.
+
+## Why it is a design fault, not one slow disk
+
+- **A bounded search judges nothing on a machine larger than the bound.** The bound was chosen so the
+  controller could tell "still running" from "stuck" (`rootsearch.Bound`, read by both). It turned a slow
+  machine into one that is never judged, failing closed for good rather than for a while.
+- **Nothing was kept across a restart.** The search's result lived in the engine's memory only, so every
+  engine restart (every update of the engine) started from nothing and judged nothing until a whole walk
+  finished again.
+- **The controller's quiet was the same value.** The self-check stayed quiet while a search was "within its
+  bound" — fifteen minutes — so a search that runs for hours would raise the urgent condition on every
+  restart even once the search itself was allowed to finish.
+
+## What is asked (the operator's direction, 2026-10-10)
+
+1. The search runs to completion, in the background, at the lowest CPU and I/O priority, one at a time, with
+   no bound on the whole search.
+2. It is resumable: its progress is kept every minute beside the engine's state, so a restart, an update or a
+   reboot carries it on from where it was rather than starting over.
+3. Its outcome is kept, with when it started and finished. A verdict is given only from the last **complete**
+   search while it is younger than a stated age; "not judged yet" before the first one completes; never
+   "cannot become root" from an incomplete or stale one.
+4. A fresh search runs at most once a day, and after any apply that changes an account, a group, a sudo rule or
+   a package.
+5. The controller's own reading of the search (its quiet) changes with it, consistently.
+
+**Why a day-old search is sound.** Only root can make a setuid- or setgid-root file: the agent account can
+neither set the bit on a file root owns nor give a file of its own to root. So a complete search stays true until
+root acts, and root's acts through the mesh (applies) start a new search. A file that appears behind a running
+walk's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age
+bounds.
+
+The trail and the fix are in [`01-diagnosis.md`](01-diagnosis.md).

04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md

@@ -0,0 +1,79 @@
+# 361 — diagnosis
+
+## 2026-10-10 — where the bound lives, and who read it
+
+- **The node-engine** (`mesh-host`, `internal/accounts`): `SetuidCache` ran the search in the background, at most
+  hourly, under a context with a deadline of `rootsearch.Bound` (15 minutes) for the whole search — `find` over
+  every mount point with `-xdev`, then the package manager about every path found. A search past the deadline
+  was an error, said as `not judged (search incomplete)`, and tried again after a back-off doubling from an hour
+  to six. The result was kept in memory only.
+- **The controller** (`mesh-controller`, `cmd/mesh-controller/agent_account.go`): `searchQuietFor` was
+  `link.RootSearchBound`, which is `rootsearch.Bound`. It decides only how long the self-check raises nothing
+  while an agent account's verdict says `not judged yet (search running)`, counted from when the controller
+  first saw it waiting. The `root-free` verb and `judgedConfined` never read it: they need a healthy verdict in
+  a statement heard within 15 minutes (`verdictFreshFor`, a different value — how often an engine states its
+  health — which this issue does not change).
+- **Nothing else** used the bound: `SetuidCache` had one caller (`moreWays`), and the engine made one cache.
+- **Ruled out:** a stuck `find` (each attempt ended at exactly its bound) and a filesystem that should not be
+  searched (the walk covered only the local root, home and boot filesystems, as ADR 0266 asks).
+
+## The fix (`mesh-host` pull request 75, `mesh-controller` pull request 190)
+
+**The walk is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as
+`find -xdev` did, with the same prunes and the same match (a regular file, setuid and owned by root, or setgid
+and of root's group), never following a link. It visits each directory's entries in lexical order, so its
+progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is walked before
+`/a-c` although `-` sorts before `/`). Every minute the progress (when the walk started, its filesystems, those
+done, the cursor, the matches so far) is written beside the engine's state as a new file renamed into place,
+readable by root alone. After a restart the walk carries on from the cursor: what is before it is skipped
+without being read, the directories leading back to it are read again. A walk older than the stated age, or one
+an apply asked to redo, is thrown away and begun again.
+
+**Low priority, one at a time, no bound.** The walk runs on a goroutine locked to an operating-system thread of
+its own, set to nice 19 and the idle I/O class; on Linux both are a thread's, so the engine's other work keeps
+its priority, and the thread ends with the goroutine. The package manager is run the same way. A priority that
+cannot be lowered fails the search rather than walk at full priority. There is no deadline: the search ends when
+it has walked everything, or with an error (a directory it could not read), which is incomplete.
+
+**A verdict only from a complete, fresh search.** The last complete search is kept beside the engine's state
+with its start and finish, and judges while it is younger than `rootsearch.FreshFor`, counted from its start.
+Before one completes, or once it is older, the verdict is `not judged yet (search running)`; after a failure,
+`not judged (search incomplete)`. A kept file is believed only as a regular file, opened without following a
+link, owned by the engine's account and writable by it alone, in a directory that is the same.
+
+**When a search runs.** Once a day (`rootsearch.Every`, from the last complete one's start); after an apply
+whose outcomes moved a user, a package, an action, or a file of the user and group databases or of sudo's,
+doas's or polkit's rules (`apply.MovesRoot`); and when a filesystem appears that the last search did not walk.
+The last two at most once an hour. The last complete search keeps judging meanwhile.
+
+**The values, one place** (`rootsearch`, read by both repositories):
+
+| Value | What it is | Set to |
+|---|---|---|
+| `Every` | how often a full search runs at most | 24 hours |
+| `Quiet` | how long the controller raises nothing while an agent account waits for a search, from when it first saw it waiting | 12 hours |
+| `FreshFor` | how long a complete search judges, from its start | `Every` + `Quiet` = 36 hours |
+
+`FreshFor` is `Every` plus `Quiet` so that a daily search which finishes within the controller's quiet never
+leaves an hour unjudged; the controller's test holds that relation. The operator's direction named 26 hours as an
+example; 36 is chosen because a full walk at idle priority on the control-node is expected to take hours, and
+with 26 hours a daily search longer than two hours would leave the account unjudged — and phone linking refused —
+every day until it finished.
+
+## Open, after the fix
+
+- **How long the first walk on the control-node takes is not measured.** Until the first complete search, the
+  verdict there is `not judged yet`, and linking stays refused. The engine says each search's end and how long
+  it ran in its journal; that line is the measurement. If a walk takes longer than `Quiet`, the self-check
+  raises the urgent condition, which is right: it is then not known to finish.
+- **The idle I/O class is honoured only by a disk scheduler that has classes** (BFQ). Under another scheduler
+  the walk still runs at nice 19 but competes for the disk as any reader does.
+- **An ask to search again is not kept across a restart.** An apply that moved root's things and a restart
+  before the next search starts lose the ask; the daily search covers it. The last complete search keeps
+  judging either way, on the reasoning in the report.
+- **ADR 0266 §3 item 10** still says "at most hourly" and "at most 15 minutes". On resolution it gains a
+  progressive insight naming this issue and the values above.
+- **A later option, not built now:** a fanotify filesystem mark with `FAN_ATTRIB` (Linux 5.1 and later) on
+  every searched filesystem would report each change of mode or owner as it happens, turning the daily walk into
+  a true incremental search.
+- **Resolution** of this core issue needs a replay in mesh-lab's register, or `replay-none:` with the reason.