novox/hq
6
0
Fork 0
Code Issues Pull Requests 6 Releases Wiki Activity

Merge pull request 'Issue 361: the control-node's agent account is never judged, because its search for setuid programs cannot finish in fifteen minutes' (#261) from issues/361-the-control-nodes-agent-account-is-never-judged into main

Browse Source
This commit was merged in pull request #261.
This commit is contained in:
mesh-admin
2026-10-09 23:18:48 +00:00
parent 288da29d94 94719d4662
commit 511cea04f2
2 changed files with 173 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md
+61
View File
@@ -0,0 +1,61 @@
---
status: located
opened: 2026-10-10
located-in: [mesh-host (internal/accounts, rootsearch), mesh-controller (cmd/mesh-controller/agent_account.go, internal/link)]
fixed-by:
amended-design:
---
# 361. The control-node's agent account is never judged: its search for setuid programs cannot finish in fifteen minutes
**Severity: high.** The router refuses to link a messaging channel on a machine that is not root-free
([ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md) §8),
and a machine whose agent account is not judged is not root-free. On the control-node the judgement can never
be given, so linking the operator's phone there is blocked for as long as the search is bounded.
## Symptom
On 2026-10-10 at 00:43 (local), after the control-node was given an agent account
([ADR 0266](../../02-DECISIONS/0266-on-the-control-node-agents-run-under-an-account-of-their-own-that-cannot-become-root-without-a-person.md)),
the controller's `root-free` verb for that machine answered:
> not judged (search incomplete): the search for setuid programs did not finish within 15m0s; it is tried
> again later
The control-node holds its root and home filesystems on a software RAID of spinning disks, with millions of
files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) walks every local
filesystem mounted without `nosuid`; that walk, cold, takes far longer than fifteen minutes there. Each attempt
was killed at its bound, judged nothing, and was tried again after a back-off, which ends the same way.
## Why it is a design fault, not one slow disk
- **A bounded search judges nothing on a machine larger than the bound.** The bound was chosen so the
controller could tell "still running" from "stuck" (`rootsearch.Bound`, read by both). It turned a slow
machine into one that is never judged, failing closed for good rather than for a while.
- **Nothing was kept across a restart.** The search's result lived in the engine's memory only, so every
engine restart (every update of the engine) started from nothing and judged nothing until a whole walk
finished again.
- **The controller's quiet was the same value.** The self-check stayed quiet while a search was "within its
bound" — fifteen minutes — so a search that runs for hours would raise the urgent condition on every
restart even once the search itself was allowed to finish.
## What is asked (the operator's direction, 2026-10-10)
1. The search runs to completion, in the background, at the lowest CPU and I/O priority, one at a time, with
no bound on the whole search.
2. It is resumable: its progress is kept every minute beside the engine's state, so a restart, an update or a
reboot carries it on from where it was rather than starting over.
3. Its outcome is kept, with when it started and finished. A verdict is given only from the last **complete**
search while it is younger than a stated age; "not judged yet" before the first one completes; never
"cannot become root" from an incomplete or stale one.
4. A fresh search runs at most once a day, and after any apply that changes an account, a group, a sudo rule or
a package.
5. The controller's own reading of the search (its quiet) changes with it, consistently.
**Why a day-old search is sound.** Only root can make a setuid- or setgid-root file: the agent account can
neither set the bit on a file root owns nor give a file of its own to root. So a complete search stays true until
root acts, and root's acts through the mesh (applies) start a new search. A file that appears behind a running
walk's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age
bounds.
The trail and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md
+112
View File
@@ -0,0 +1,112 @@
# 361 — diagnosis
## 2026-10-10 — where the bound lives, and who read it
- **The node-engine** (`mesh-host`, `internal/accounts`): `SetuidCache` ran the search in the background, at most
hourly, under a context with a deadline of `rootsearch.Bound` (15 minutes) for the whole search — `find` over
every mount point with `-xdev`, then the package manager about every path found. A search past the deadline
was an error, said as `not judged (search incomplete)`, and tried again after a back-off doubling from an hour
to six. The result was kept in memory only.
- **The controller** (`mesh-controller`, `cmd/mesh-controller/agent_account.go`): `searchQuietFor` was
`link.RootSearchBound`, which is `rootsearch.Bound`. It decides only how long the self-check raises nothing
while an agent account's verdict says `not judged yet (search running)`, counted from when the controller
first saw it waiting. The `root-free` verb and `judgedConfined` never read it: they need a healthy verdict in
a statement heard within 15 minutes (`verdictFreshFor`, a different value — how often an engine states its
health — which this issue does not change).
- **Nothing else** used the bound: `SetuidCache` had one caller (`moreWays`), and the engine made one cache.
- **Ruled out:** a stuck `find` (each attempt ended at exactly its bound) and a filesystem that should not be
searched (the walk covered only the local root, home and boot filesystems, as ADR 0266 asks).
## The fix (`mesh-host` pull request 75, `mesh-controller` pull request 190)
**The walk is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as
`find -xdev` did, with the same prunes and the same match (a regular file, setuid and owned by root, or setgid
and of root's group), never following a link. It visits each directory's entries in lexical order, so its
progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is walked before
`/a-c` although `-` sorts before `/`). Every minute the progress (when the walk started, its filesystems, those
done, the cursor, the matches so far) is written beside the engine's state as a new file renamed into place,
readable by root alone. After a restart the walk carries on from the cursor: what is before it is skipped
without being read, the directories leading back to it are read again. A walk older than the stated age, or one
an apply asked to redo, is thrown away and begun again.
**Low priority, one at a time, no bound.** The walk runs on a goroutine locked to an operating-system thread of
its own, set to nice 19 and the idle I/O class; on Linux both are a thread's, so the engine's other work keeps
its priority, and the thread ends with the goroutine. The package manager is run the same way. A priority that
cannot be lowered fails the search rather than walk at full priority. There is no deadline: the search ends when
it has walked everything, or with an error (a directory it could not read), which is incomplete.
**A verdict only from a complete, fresh search.** The last complete search is kept beside the engine's state
with its start and finish, and judges while it is younger than `rootsearch.FreshFor`, counted from its start.
Before one completes, or once it is older, the verdict is `not judged yet (search running)`; after a failure,
`not judged (search incomplete)`. A kept file is believed only as a regular file, opened without following a
link, owned by the engine's account and writable by it alone, in a directory that is the same.
**When a search runs.** Once a day (`rootsearch.Every`, from the last complete one's start); after an apply
whose outcomes moved a user, a package, an action, or a file of the user and group databases or of sudo's,
doas's or polkit's rules (`apply.MovesRoot`), at most once an hour from the last search's end, the last complete
search judging meanwhile; and at once when a filesystem appears that the last search did not walk, which that
search does not judge.
**The values, one place** (`rootsearch`, read by both repositories):
| Value | What it is | Set to |
|---|---|---|
| `Every` | how often a full search runs at most | 24 hours |
| `Quiet` | how long the controller raises nothing while an agent account waits for a search, from when it first saw it waiting | 12 hours |
| `FreshFor` | how long a complete search judges, from its start | `Every` + `Quiet` = 36 hours |
`FreshFor` is `Every` plus `Quiet` so that a daily search which finishes within the controller's quiet never
leaves an hour unjudged; the controller's test holds that relation. The operator's direction named 26 hours as an
example; 36 is chosen because a full walk at idle priority on the control-node is expected to take hours, and
with 26 hours a daily search longer than two hours would leave the account unjudged — and phone linking refused —
every day until it finished.
## 2026-10-10 — the independent review, and what it changed
An independent review of both pull requests asked four questions: can a stale or incomplete search yield "free";
can the agent account steer the kept files or the walk; does the walk hurt the machine; do the controller and the
engine agree. What it found, and what was done:
- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed walk
skipped every name that sorted between the real cursor and its replacement, and could finish "free" without
having read them. Any account can make such a name. The walk's paths are now kept as their bytes.
- **An ask to search again was lost to a restart**, and the restarted engine carried on the walk that started
before root acted. The ask now forgets the walk under way and stops it keeping progress.
- **A fresh search judged a filesystem it never walked.** A complete search now judges only while it walked every
filesystem a search would walk now; a filesystem mounted since makes the verdict "not judged yet" until walked.
- **A mount point inside a walked filesystem was read before it was known to be another filesystem.** A FUSE
mount root cannot read failed the walk, and a network mount whose server is away could hang it. Every mount
point the kernel lists is now left unread. Mount points are read with all the kernel's escapes.
- **A file found and gone before the package manager was asked failed every resumed search.** The walk looks at
what it found again when it ends, and drops what went or is no longer setuid-root.
- **Outcomes that moved nothing started a search**, and a search asked for could follow a walk of hours at once.
Only created, updated, corrected and removed outcomes count now, and an ask waits an hour from the last
search's end.
- **Not changed, because the mesh cannot do it:** the review asked that a file or archive declared setuid start a
search. The engine writes a file's and an archive's permission bits only, so neither carries a setuid bit.
## Open, after the fix
- **How long the first walk on the control-node takes is not measured.** Until the first complete search, the
verdict there is `not judged yet`, and linking stays refused. The engine says each search's end and how long
it ran in its journal; that line is the measurement. If a walk takes longer than `Quiet`, the self-check
raises the urgent condition, which is right: it is then not known to finish.
- **The idle I/O class is honoured only by a disk scheduler that has classes** (BFQ). Under another scheduler
the walk still runs at nice 19 but competes for the disk as any reader does.
- **An ask to search again is not kept across a restart.** The walk under way is forgotten when it is asked,
so a restarted engine does not carry an older walk on; the ask itself is lost, and the daily search covers it.
The last complete search keeps judging meanwhile, on the reasoning in the report.
- **The agent account can make the search fail, never pass.** A tree deeper than the longest path the kernel
takes, or a directory unreadable to root through FUSE, ends every walk incomplete; a directory of millions of
entries makes the engine hold them all at once. Each is "not judged" — the urgent condition, and linking
refused — never a pass. Reading by directory handle, as `find` did, would close the first.
- **A filesystem mounted only on use** (an automounted boot partition) makes the verdict "not judged yet" each
time it appears unwalked, until a walk covers it.
- **Part of the walk's cost escapes its priority**: Go's garbage collector runs on threads of normal priority,
and a software RAID's own threads do not carry the idle class to the disks.
- **ADR 0266 §3 item 10** still says "at most hourly" and "at most 15 minutes". On resolution it gains a
progressive insight naming this issue and the values above.
- **A later option, not built now:** a fanotify filesystem mark with `FAN_ATTRIB` (Linux 5.1 and later) on
every searched filesystem would report each change of mode or owner as it happens, turning the daily walk into
a true incremental search.
- **Resolution** of this core issue needs a replay in mesh-lab's register, or `replay-none:` with the reason.