Fixes issue 433 (the bus on the control-node stalled for half a minute twice, and the controller lost its lease and restarted each time).
What happened
From the node-engine's journal on the control-node: "the search for setuid programs started 2026-10-11 01:27 completed (16m31s this time)". It started right after the engine applied a new controller at 01:26:50 (issue 361 item 4: a fresh search after an apply that changes a package). Both bus stalls fell inside it: the bus logged its JetStream API taking 29-43 s at 01:32:09 and 17-34 s at 01:34:31, including the lease bucket's read (33.9 s), and the controller's renewal failed with "context deadline exceeded" each time.
The walk puts itself in the idle I/O class (lower() in idle_linux.go), whose own comment says it works "where the disk's scheduler honours classes (BFQ does)". The control-node's two disks run mq-deadline, which ignores I/O classes, so the walk read at full priority. Its home and root filesystems are partitions of the same two rotational disks; the bus's store lives on root. A disk sample attributed /home's reads to the engine (nox-mesh-host.service, 141 read IOPS: all a spinning disk does); after the walk ended, /home went idle and I/O pressure fell from about 26 % to 3 %.
What changes
The walk also paces itself by PSI (/proc/pressure/io), which reads the same whatever the scheduler. Before it reads a directory, while some task has waited on I/O for more than 10 % of the last ten seconds (some avg10), it pauses 2 s at a time. It looks at most every 250 ms, and pauses at most 30 s in a row before reading one more directory, so a machine that is never quiet is walked slowly but is still judged — the fault issue 361 repaired must not come back. Its own reads count in the pressure, so on an otherwise idle machine it holds the disk below the bound, which is the yielding the idle class promised. Pressure it cannot read pauses nothing and is said once in the journal. The four bounds are design limits I chose and name here; they are constants in setuid_pace.go.
No field is added to the walk's kept progress, so a walk resumed across this change carries on as before.
Not done here
BFQ. Declaring BFQ for rotational disks would make the idle class work on its own; it changes every machine's disk behaviour, so it is not taken here.
The other two questions 433 asks — that a stall of the bus does not end the controller, and that a stalled bus raises its own condition — stay on the issue.
Checked here
go test -race ./... -count=1: every package passes.
New tests (setuid_pace_test.go): pauses while above the bound and not at it; does not pause on a quiet disk; pauses at most 30 s in a row; looks at most every 250 ms; unreadable pressure pauses nothing and is said once; a paced walk finds exactly what an unpaced one finds and does pause under pressure; a walk whose context ends during a pause is incomplete, never "none"; the kernel's PSI format is parsed and malformed input refused.
Mutation: with the call to the pacer removed from the walk, both walk-level tests fail.
Rollout
A node-engine change, so one at a time and on its own; its apply on the control-node is itself an apply that changes a package, so the walk that follows it is the first paced one there — and is the live proof.
Fixes issue 433 (the bus on the control-node stalled for half a minute twice, and the controller lost its lease and restarted each time).
## What happened
From the node-engine's journal on the control-node: "the search for setuid programs started 2026-10-11 01:27 completed (16m31s this time)". It started right after the engine applied a new controller at 01:26:50 (issue 361 item 4: a fresh search after an apply that changes a package). Both bus stalls fell inside it: the bus logged its JetStream API taking 29-43 s at 01:32:09 and 17-34 s at 01:34:31, including the lease bucket's read (33.9 s), and the controller's renewal failed with "context deadline exceeded" each time.
The walk puts itself in the idle I/O class (`lower()` in idle_linux.go), whose own comment says it works "where the disk's scheduler honours classes (BFQ does)". The control-node's two disks run **mq-deadline**, which ignores I/O classes, so the walk read at full priority. Its home and root filesystems are partitions of the same two rotational disks; the bus's store lives on root. A disk sample attributed /home's reads to the engine (nox-mesh-host.service, 141 read IOPS: all a spinning disk does); after the walk ended, /home went idle and I/O pressure fell from about 26 % to 3 %.
## What changes
The walk also paces itself by PSI (`/proc/pressure/io`), which reads the same whatever the scheduler. Before it reads a directory, while some task has waited on I/O for more than **10 %** of the last ten seconds (`some avg10`), it pauses **2 s** at a time. It looks at most every **250 ms**, and pauses at most **30 s** in a row before reading one more directory, so a machine that is never quiet is walked slowly but is still judged — the fault issue 361 repaired must not come back. Its own reads count in the pressure, so on an otherwise idle machine it holds the disk below the bound, which is the yielding the idle class promised. Pressure it cannot read pauses nothing and is said once in the journal. The four bounds are design limits I chose and name here; they are constants in setuid_pace.go.
No field is added to the walk's kept progress, so a walk resumed across this change carries on as before.
## Not done here
- **BFQ.** Declaring BFQ for rotational disks would make the idle class work on its own; it changes every machine's disk behaviour, so it is not taken here.
- **The other two questions 433 asks** — that a stall of the bus does not end the controller, and that a stalled bus raises its own condition — stay on the issue.
## Checked here
- `go test -race ./... -count=1`: every package passes.
- New tests (setuid_pace_test.go): pauses while above the bound and not at it; does not pause on a quiet disk; pauses at most 30 s in a row; looks at most every 250 ms; unreadable pressure pauses nothing and is said once; a paced walk finds exactly what an unpaced one finds and does pause under pressure; a walk whose context ends during a pause is incomplete, never "none"; the kernel's PSI format is parsed and malformed input refused.
- Mutation: with the call to the pacer removed from the walk, both walk-level tests fail.
## Rollout
A node-engine change, so one at a time and on its own; its apply on the control-node is itself an apply that changes a package, so the walk that follows it is the first paced one there — and is the live proof.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The walk for setuid programs is put in the idle I/O class so it yields
to everything else, but only a scheduler that schedules by class (BFQ)
honours that. The control-node's disks run mq-deadline, which does not,
so the walk read at full priority. On 2026-10-11 the walk started at
01:27, right after the engine applied a new controller, and ran sixteen
minutes; it kept two spinning disks saturated, the bus whose store
shares them stalled for half a minute twice, and the controller lost
its lease and restarted each time.
The walk now also paces itself by PSI, which reads the same whatever
the scheduler: before it reads a directory, while some task has waited
on I/O for more than 10 % of the last ten seconds, it pauses two
seconds at a time. It looks at most every 250 ms, pauses at most 30 s
in a row and then reads one more directory, so a machine that is never
quiet is walked slowly but still judged. Its own reads count in the
pressure, so on an otherwise idle machine it holds the disk below the
bound. Pressure it cannot read pauses nothing, and is said once.
A paced walk finds what an unpaced one finds; a walk ended while it
pauses is an incomplete walk, never one that found nothing.
Deliverynovox/mesh-host@f311fa36dc49 — delivered since 2026-10-11T00:09:11Z
Delivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step
tier 0: mesh-host
ace: receives mesh-host
g14: receives mesh-host
novox: receives mesh-host
shanks: receives mesh-host
Machines
mesh-host on ace: passed — healthy 3 times over 2m34s
mesh-host on the rest running it: sent
Transitions
2026-10-10 23:55 (new) → proposed (announced): novox/mesh-host#79's head announced
2026-10-10 23:58 proposed → checked (checked): the verdict names this commit
2026-10-10 23:58 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-11 00:05 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-11 00:05 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
2026-10-11 00:09 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says
The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.
<!-- mesh-delivery:view -->
**Delivery** `novox/mesh-host@f311fa36dc49` — **delivered** since 2026-10-11T00:09:11Z
**Delivery plan** — builds mesh-host → ace, g14, novox, shanks; no bus step
- tier 0: mesh-host
- ace: receives mesh-host
- g14: receives mesh-host
- novox: receives mesh-host
- shanks: receives mesh-host
**Machines**
- mesh-host on ace: passed — healthy 3 times over 2m34s
- mesh-host on the rest running it: sent
**Transitions**
- 2026-10-10 23:55 (new) → proposed (announced): novox/mesh-host#79's head announced
- 2026-10-10 23:58 proposed → checked (checked): the verdict names this commit
- 2026-10-10 23:58 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
- 2026-10-11 00:05 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
- 2026-10-11 00:05 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
- 2026-10-11 00:09 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says
The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
mesh/repo-check: PASS — its merge-check.sh passed
Delivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step
tier 0: mesh-host
ace: receives mesh-host
g14: receives mesh-host
novox: receives mesh-host
shanks: receives mesh-host
read by no module's build: internal/accounts/setuid_pace_test.go
Run by the build seat on novox as build-1791676098856316090 (builds --log build-1791676098856316090).
--- the gate: mesh-host
mesh-host: ok
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-10T23:44:44Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-10T23:44:44Z
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
notes:
- internal/accounts/setuid_pace_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.032s
ok github.com/novox/mesh-host/cmd/mesh-host 7.382s
ok github.com/novox/mesh-host/examples 1.036s
ok github.com/novox/mesh-host/internal/accounts 3.251s
ok github.com/novox/mesh-host/internal/apply 24.987s
ok github.com/novox/mesh-host/internal/bootstrap 3.144s
ok github.com/novox/mesh-host/internal/bundle 1.021s
ok github.com/novox/mesh-host/internal/declaration 1.065s
ok github.com/novox/mesh-host/internal/firewall 1.852s
ok github.com/novox/mesh-host/internal/identity 1.791s
ok github.com/novox/mesh-host/internal/image 1.030s
ok github.com/novox/mesh-host/internal/inventory 1.030s
ok github.com/novox/mesh-host/internal/link 10.502s
ok github.com/novox/mesh-host/internal/liveness 2.806s
ok github.com/novox/mesh-host/internal/meshcli 1.331s
ok github.com/novox/mesh-host/internal/network 1.031s
ok github.com/novox/mesh-host/internal/outward 1.015s
ok github.com/novox/mesh-host/internal/profile 1.481s
ok github.com/novox/mesh-host/internal/reachable 1.011s
ok github.com/novox/mesh-host/internal/store 1.931s
ok github.com/novox/mesh-host/internal/system 1.012s
ok github.com/novox/mesh-host/internal/tunnel 1.016s
ok github.com/novox/mesh-host/internal/units 1.008s
ok github.com/novox/mesh-host/internal/upgrade 1.805s
ok github.com/novox/mesh-host/internal/witness 26.509s
ok github.com/novox/mesh-host/packaging 1.009s
? github.com/novox/mesh-host/rootsearch [no test files]
ok github.com/novox/mesh-host/validate 1.007s
**Merge check** at `8f7db549`
- `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
- `mesh/repo-check`: **PASS** — its merge-check.sh passed
**Delivery plan** — builds mesh-host → ace, g14, novox, shanks; no bus step
- tier 0: mesh-host
- ace: receives mesh-host
- g14: receives mesh-host
- novox: receives mesh-host
- shanks: receives mesh-host
- read by no module's build: internal/accounts/setuid_pace_test.go
Run by the build seat on novox as `build-1791676098856316090` (`builds --log build-1791676098856316090`).
```
--- the gate: mesh-host
mesh-host: ok
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-10T23:44:44Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-10T23:44:44Z
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
notes:
- internal/accounts/setuid_pace_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.032s
ok github.com/novox/mesh-host/cmd/mesh-host 7.382s
ok github.com/novox/mesh-host/examples 1.036s
ok github.com/novox/mesh-host/internal/accounts 3.251s
ok github.com/novox/mesh-host/internal/apply 24.987s
ok github.com/novox/mesh-host/internal/bootstrap 3.144s
ok github.com/novox/mesh-host/internal/bundle 1.021s
ok github.com/novox/mesh-host/internal/declaration 1.065s
ok github.com/novox/mesh-host/internal/firewall 1.852s
ok github.com/novox/mesh-host/internal/identity 1.791s
ok github.com/novox/mesh-host/internal/image 1.030s
ok github.com/novox/mesh-host/internal/inventory 1.030s
ok github.com/novox/mesh-host/internal/link 10.502s
ok github.com/novox/mesh-host/internal/liveness 2.806s
ok github.com/novox/mesh-host/internal/meshcli 1.331s
ok github.com/novox/mesh-host/internal/network 1.031s
ok github.com/novox/mesh-host/internal/outward 1.015s
ok github.com/novox/mesh-host/internal/profile 1.481s
ok github.com/novox/mesh-host/internal/reachable 1.011s
ok github.com/novox/mesh-host/internal/store 1.931s
ok github.com/novox/mesh-host/internal/system 1.012s
ok github.com/novox/mesh-host/internal/tunnel 1.016s
ok github.com/novox/mesh-host/internal/units 1.008s
ok github.com/novox/mesh-host/internal/upgrade 1.805s
ok github.com/novox/mesh-host/internal/witness 26.509s
ok github.com/novox/mesh-host/packaging 1.009s
? github.com/novox/mesh-host/rootsearch [no test files]
ok github.com/novox/mesh-host/validate 1.007s
```
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The review of #79 found that pacing alone brought back issue 361's
fault. The control-node's own I/O pressure sits near the 10 % bound with
no walk running, so a walk that paused 30 s for every 250 ms it read
would read 0.8 % of the time: a 16-minute walk would take some 32 hours,
outlive the age a search judges for, and be begun again at an engine
restart, leaving the machine unjudged.
A run of pauses is now at most nine times as long as the walk read since
the one before, so the walk keeps at least a tenth of the time whatever
the pressure, and takes at most ten times as long as unpaced: about
2.7 hours for the 16-minute walk, inside the 12 hours the controller
waits quietly. A test ties the bound to rootsearch.Quiet.
The pressure is the share of the time since the last look in which some
task waited on I/O, from PSI's cumulative counter, not its ten-second
average, which the kernel moves only every two seconds. A pause is
500 ms, then the pressure is looked at again.
The engine's walker is built in one place, engineWalker, and a test
fails if it is not paced by /proc/pressure/io. What the walk cannot read
is said through the search's own channel (the cache's Say), as the rest
of the search is, with what it means for the disk.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A file's own details (lstat) may be read from the disk as a directory is,
so a directory of tens of thousands of files was read without a look and
the header's promise that no burst is longer than PaceLook did not hold.
The walk now asks the pacer before each file as well; the 250 ms gate
makes that a clock read per file. A test with one directory of twenty
files fails without it. The clock is read once after a run of pauses,
and the constants test says that it guards the constants only.
mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
mesh/repo-check: PASS — its merge-check.sh passed
Delivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step
tier 0: mesh-host
ace: receives mesh-host
g14: receives mesh-host
novox: receives mesh-host
shanks: receives mesh-host
read by no module's build: internal/accounts/setuid_pace_test.go
Run by the build seat on shanks as build-1791676431655643033 (builds --log build-1791676431655643033).
--- the gate: mesh-host
mesh-host: ok
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-10T23:44:44Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-10T23:44:44Z
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
notes:
- internal/accounts/setuid_pace_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.012s
ok github.com/novox/mesh-host/cmd/mesh-host 5.473s
ok github.com/novox/mesh-host/examples 1.016s
ok github.com/novox/mesh-host/internal/accounts 1.600s
ok github.com/novox/mesh-host/internal/apply 4.737s
ok github.com/novox/mesh-host/internal/bootstrap 1.626s
ok github.com/novox/mesh-host/internal/bundle 1.030s
ok github.com/novox/mesh-host/internal/declaration 1.085s
ok github.com/novox/mesh-host/internal/firewall 1.868s
ok github.com/novox/mesh-host/internal/identity 1.126s
ok github.com/novox/mesh-host/internal/image 1.048s
ok github.com/novox/mesh-host/internal/inventory 1.053s
ok github.com/novox/mesh-host/internal/link 10.481s
ok github.com/novox/mesh-host/internal/liveness 2.141s
ok github.com/novox/mesh-host/internal/meshcli 1.365s
ok github.com/novox/mesh-host/internal/network 1.071s
ok github.com/novox/mesh-host/internal/outward 1.056s
ok github.com/novox/mesh-host/internal/profile 1.543s
ok github.com/novox/mesh-host/internal/reachable 1.053s
ok github.com/novox/mesh-host/internal/store 1.241s
ok github.com/novox/mesh-host/internal/system 1.055s
ok github.com/novox/mesh-host/internal/tunnel 1.056s
ok github.com/novox/mesh-host/internal/units 1.042s
ok github.com/novox/mesh-host/internal/upgrade 1.181s
ok github.com/novox/mesh-host/internal/witness 4.741s
ok github.com/novox/mesh-host/packaging 1.008s
? github.com/novox/mesh-host/rootsearch [no test files]
ok github.com/novox/mesh-host/validate 1.008s
**Merge check** at `30242197`
- `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
- `mesh/repo-check`: **PASS** — its merge-check.sh passed
**Delivery plan** — builds mesh-host → ace, g14, novox, shanks; no bus step
- tier 0: mesh-host
- ace: receives mesh-host
- g14: receives mesh-host
- novox: receives mesh-host
- shanks: receives mesh-host
- read by no module's build: internal/accounts/setuid_pace_test.go
Run by the build seat on shanks as `build-1791676431655643033` (`builds --log build-1791676431655643033`).
```
--- the gate: mesh-host
mesh-host: ok
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-10T23:44:44Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-10T23:44:44Z
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
notes:
- internal/accounts/setuid_pace_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.012s
ok github.com/novox/mesh-host/cmd/mesh-host 5.473s
ok github.com/novox/mesh-host/examples 1.016s
ok github.com/novox/mesh-host/internal/accounts 1.600s
ok github.com/novox/mesh-host/internal/apply 4.737s
ok github.com/novox/mesh-host/internal/bootstrap 1.626s
ok github.com/novox/mesh-host/internal/bundle 1.030s
ok github.com/novox/mesh-host/internal/declaration 1.085s
ok github.com/novox/mesh-host/internal/firewall 1.868s
ok github.com/novox/mesh-host/internal/identity 1.126s
ok github.com/novox/mesh-host/internal/image 1.048s
ok github.com/novox/mesh-host/internal/inventory 1.053s
ok github.com/novox/mesh-host/internal/link 10.481s
ok github.com/novox/mesh-host/internal/liveness 2.141s
ok github.com/novox/mesh-host/internal/meshcli 1.365s
ok github.com/novox/mesh-host/internal/network 1.071s
ok github.com/novox/mesh-host/internal/outward 1.056s
ok github.com/novox/mesh-host/internal/profile 1.543s
ok github.com/novox/mesh-host/internal/reachable 1.053s
ok github.com/novox/mesh-host/internal/store 1.241s
ok github.com/novox/mesh-host/internal/system 1.055s
ok github.com/novox/mesh-host/internal/tunnel 1.056s
ok github.com/novox/mesh-host/internal/units 1.042s
ok github.com/novox/mesh-host/internal/upgrade 1.181s
ok github.com/novox/mesh-host/internal/witness 4.741s
ok github.com/novox/mesh-host/packaging 1.008s
? github.com/novox/mesh-host/rootsearch [no test files]
ok github.com/novox/mesh-host/validate 1.008s
```
mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
mesh/repo-check: PASS — its merge-check.sh passed
Delivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step
tier 0: mesh-host
ace: receives mesh-host
g14: receives mesh-host
novox: receives mesh-host
shanks: receives mesh-host
read by no module's build: internal/accounts/setuid_pace_test.go
Run by the build seat on g14 as build-1791676552596204868 (builds --log build-1791676552596204868).
--- the gate: mesh-host
mesh-host: ok
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-10T23:44:44Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-10T23:44:44Z
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
notes:
- internal/accounts/setuid_pace_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.013s
ok github.com/novox/mesh-host/cmd/mesh-host 5.356s
ok github.com/novox/mesh-host/examples 1.016s
ok github.com/novox/mesh-host/internal/accounts 1.363s
ok github.com/novox/mesh-host/internal/apply 2.348s
ok github.com/novox/mesh-host/internal/bootstrap 1.364s
ok github.com/novox/mesh-host/internal/bundle 1.021s
ok github.com/novox/mesh-host/internal/declaration 1.052s
ok github.com/novox/mesh-host/internal/firewall 1.697s
ok github.com/novox/mesh-host/internal/identity 1.052s
ok github.com/novox/mesh-host/internal/image 1.025s
ok github.com/novox/mesh-host/internal/inventory 1.027s
ok github.com/novox/mesh-host/internal/link 10.473s
ok github.com/novox/mesh-host/internal/liveness 2.064s
ok github.com/novox/mesh-host/internal/meshcli 1.344s
ok github.com/novox/mesh-host/internal/network 1.042s
ok github.com/novox/mesh-host/internal/outward 1.030s
ok github.com/novox/mesh-host/internal/profile 1.527s
ok github.com/novox/mesh-host/internal/reachable 1.028s
ok github.com/novox/mesh-host/internal/store 1.084s
ok github.com/novox/mesh-host/internal/system 1.013s
ok github.com/novox/mesh-host/internal/tunnel 1.016s
ok github.com/novox/mesh-host/internal/units 1.013s
ok github.com/novox/mesh-host/internal/upgrade 1.041s
ok github.com/novox/mesh-host/internal/witness 1.490s
ok github.com/novox/mesh-host/packaging 1.007s
? github.com/novox/mesh-host/rootsearch [no test files]
ok github.com/novox/mesh-host/validate 1.006s
**Merge check** at `f311fa36`
- `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
- `mesh/repo-check`: **PASS** — its merge-check.sh passed
**Delivery plan** — builds mesh-host → ace, g14, novox, shanks; no bus step
- tier 0: mesh-host
- ace: receives mesh-host
- g14: receives mesh-host
- novox: receives mesh-host
- shanks: receives mesh-host
- read by no module's build: internal/accounts/setuid_pace_test.go
Run by the build seat on g14 as `build-1791676552596204868` (`builds --log build-1791676552596204868`).
```
--- the gate: mesh-host
mesh-host: ok
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-10T23:44:44Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-10T23:44:44Z
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
notes:
- internal/accounts/setuid_pace_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.013s
ok github.com/novox/mesh-host/cmd/mesh-host 5.356s
ok github.com/novox/mesh-host/examples 1.016s
ok github.com/novox/mesh-host/internal/accounts 1.363s
ok github.com/novox/mesh-host/internal/apply 2.348s
ok github.com/novox/mesh-host/internal/bootstrap 1.364s
ok github.com/novox/mesh-host/internal/bundle 1.021s
ok github.com/novox/mesh-host/internal/declaration 1.052s
ok github.com/novox/mesh-host/internal/firewall 1.697s
ok github.com/novox/mesh-host/internal/identity 1.052s
ok github.com/novox/mesh-host/internal/image 1.025s
ok github.com/novox/mesh-host/internal/inventory 1.027s
ok github.com/novox/mesh-host/internal/link 10.473s
ok github.com/novox/mesh-host/internal/liveness 2.064s
ok github.com/novox/mesh-host/internal/meshcli 1.344s
ok github.com/novox/mesh-host/internal/network 1.042s
ok github.com/novox/mesh-host/internal/outward 1.030s
ok github.com/novox/mesh-host/internal/profile 1.527s
ok github.com/novox/mesh-host/internal/reachable 1.028s
ok github.com/novox/mesh-host/internal/store 1.084s
ok github.com/novox/mesh-host/internal/system 1.013s
ok github.com/novox/mesh-host/internal/tunnel 1.016s
ok github.com/novox/mesh-host/internal/units 1.013s
ok github.com/novox/mesh-host/internal/upgrade 1.041s
ok github.com/novox/mesh-host/internal/witness 1.490s
ok github.com/novox/mesh-host/packaging 1.007s
? github.com/novox/mesh-host/rootsearch [no test files]
ok github.com/novox/mesh-host/validate 1.006s
```
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes issue 433 (the bus on the control-node stalled for half a minute twice, and the controller lost its lease and restarted each time).
What happened
From the node-engine's journal on the control-node: "the search for setuid programs started 2026-10-11 01:27 completed (16m31s this time)". It started right after the engine applied a new controller at 01:26:50 (issue 361 item 4: a fresh search after an apply that changes a package). Both bus stalls fell inside it: the bus logged its JetStream API taking 29-43 s at 01:32:09 and 17-34 s at 01:34:31, including the lease bucket's read (33.9 s), and the controller's renewal failed with "context deadline exceeded" each time.
The walk puts itself in the idle I/O class (
lower()in idle_linux.go), whose own comment says it works "where the disk's scheduler honours classes (BFQ does)". The control-node's two disks run mq-deadline, which ignores I/O classes, so the walk read at full priority. Its home and root filesystems are partitions of the same two rotational disks; the bus's store lives on root. A disk sample attributed /home's reads to the engine (nox-mesh-host.service, 141 read IOPS: all a spinning disk does); after the walk ended, /home went idle and I/O pressure fell from about 26 % to 3 %.What changes
The walk also paces itself by PSI (
/proc/pressure/io), which reads the same whatever the scheduler. Before it reads a directory, while some task has waited on I/O for more than 10 % of the last ten seconds (some avg10), it pauses 2 s at a time. It looks at most every 250 ms, and pauses at most 30 s in a row before reading one more directory, so a machine that is never quiet is walked slowly but is still judged — the fault issue 361 repaired must not come back. Its own reads count in the pressure, so on an otherwise idle machine it holds the disk below the bound, which is the yielding the idle class promised. Pressure it cannot read pauses nothing and is said once in the journal. The four bounds are design limits I chose and name here; they are constants in setuid_pace.go.No field is added to the walk's kept progress, so a walk resumed across this change carries on as before.
Not done here
Checked here
go test -race ./... -count=1: every package passes.Rollout
A node-engine change, so one at a time and on its own; its apply on the control-node is itself an apply that changes a package, so the walk that follows it is the first paced one there — and is the live proof.
Delivery
novox/mesh-host@f311fa36dc49— delivered since 2026-10-11T00:09:11ZDelivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step
Machines
Transitions
The commit's note under
refs/notes/mesh-plankeeps every transition:git log --notes=mesh-plan.Merge check at
8f7db549mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step
Run by the build seat on novox as
build-1791676098856316090(builds --log build-1791676098856316090).Merge check at
30242197mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step
Run by the build seat on shanks as
build-1791676431655643033(builds --log build-1791676431655643033).Merge check at
f311fa36mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step
Run by the build seat on g14 as
build-1791676552596204868(builds --log build-1791676552596204868).