386 Commits
Author SHA1 Message Date
mesh-admin 8e95c1d9ed Merge pull request 'A machine says what it is made of, and says it once a day unasked (issue 160)' (#92) from fix/160-a-machine-says-what-it-is into main 2026-10-12 21:07:05 +00:00
mesh-admin d8e785bb4b Merge pull request 'Keep libraries under a group of their own, and give each joining module a user in it (hq ADR 0328, issue 567)' (#95) from feat/328-libraries-and-module-users into main 2026-10-12 20:28:16 +00:00
jschoubben 281837847e Put busybox's account removal back, test the sample, and read a held lock as kept only for a running walk (hq ADR 0328 review)
mesh/merge-gate head 281837847e03 merged into trunk 303d4037e84e — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 281837847e03 merged into trunk 303d4037e84e — pass: its merge-check.sh passed
mesh/delivery delivered
An earlier change dropped alpine's RemoveAccount and RemoveGroup, so alpine made no
module identity at all. The sample beneath a root in shape had no test of its own.
And an apply that found the library's lock held read the library as kept even when
what held it was a take act about to be refused for a missing disk; it now does so
only while a take walk is running, and otherwise waits and judges as always.
2026-10-12 22:21:42 +02:00
jschoubben c7551162d6 Give no account a home, keep old data's sub-paths for the warrant, and never hand out setgid (hq ADR 0328 review)
usermod --gid regroups every file under an account's home, and accounts were made
with / as home: they now have /nonexistent, and an older account's home is moved
there (without its files) before its group changes. A sub-path asked for later that
holds data not in the group waits for the operator as the root would. The take act
gives the group to hardlinked files an account owns (the *arr apps' downloads) and
leaves root's; a file's setuid or setgid bit the regroup takes is never put back, so
no setgid file hands out the library's group; each is counted. An apply no longer
waits for a running take walk.
2026-10-12 22:18:16 +02:00
jschoubben 2cffa33bcc Keep a root without asking only when its data beneath is the library's already (hq ADR 0328 §7)
A root in the group with setgid was kept without a warrant even when the data beneath
it was not in the group, which §7 says the operator is asked about. A root is now
kept without asking only when it is empty, or it and a bounded sample beneath it are
all in the group with group write (and setgid on folders). On busybox, an account the
mesh made is given its added groups with addgroup and delgroup; a change of its
primary group is refused, saying why.
2026-10-12 22:14:10 +02:00
jschoubben 2a2be687f3 Keep libraries under groups of their own, and give modules users in them (hq ADR 0328, issue 567)
ADR 0328 supersedes ADR 0325: shared data is a library with one home, kept usable by
its group rather than by ACL entries. The engine now makes each library's group and
each module's user in its libraries' groups; keeps a library's root and requested
sub-paths in its group with group write and setgid on its home, making missing
sub-paths as the root is; waits for the operator's word before touching a root that
holds data, and on the take act walks it by descriptor (no link, same filesystem,
bounded and resumed, files with other names left); holds the containers of a library
whose disk or mount is not there; and puts the tool runner's account in the group.
The ACL keeping, mask repair and watch are gone; an access is again as before ADR 0325.
2026-10-12 22:11:09 +02:00
mesh-admin 303d4037e8 Merge pull request 'Give the controller the contract it can fill: tool runners by name, the take act's access id, wait reasons and ACL support (hq ADR 0325)' (#94) from feat/325-the-contract-the-controller-needs into main 2026-10-12 17:57:41 +00:00
jschoubben 5c8e768ef5 Read an account by its name, never as a number getent takes for a uid (hq ADR 0325)
mesh/merge-gate head 5c8e768ef520 merged into trunk ef26dba2753a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 5c8e768ef520 merged into trunk ef26dba2753a — pass: its merge-check.sh passed
mesh/delivery delivered
getent passwd reads an all-digit key as a uid, so a tool runner's account named
"1000" resolved to whoever has uid 1000 and the entry went to them. Only an answer
naming the account asked for is taken.
2026-10-12 19:51:09 +02:00
jschoubben aad33dfe5c Give the controller the contract it can fill (hq ADR 0325, issue 567)
The controller cannot know the operator's account's uid, names an access by its
declared id (which holds a slash), and should not parse words. A tool runner's
account may now come by name alone and is read here (refused when absent or root's,
its uid said in the access's listing); the take act matches the declared id exactly;
an access wait says its reason (keep or disk) and, for a filesystem that refuses
ACLs, whether the take act can enable them.
2026-10-12 19:51:09 +02:00
mesh-admin ef26dba275 Merge pull request 'Give every module an identity and keep shared folders through ACL entries (hq ADR 0325, issue 567)' (#93) from feat/325-identities-and-kept-folders into main 2026-10-12 17:50:59 +00:00
jschoubben d12bba0109 Give the kept-folder tests' folders an unprivileged owner when they run as root (hq ADR 0325)
mesh/merge-gate head d12bba01090d merged into trunk dd9ab7927d96 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head d12bba01090d merged into trunk dd9ab7927d96 — pass: its merge-check.sh passed
mesh/delivery delivered
The build seat runs merge-check as root, so every folder a test made was root's
data, which is never kept (§12), and 28 tests failed for that alone. A test's
folders now stand for an operator's: run as root, they are given to nobody before
they are judged. The root rule is unchanged.
2026-10-12 19:34:59 +02:00
jschoubben c85c2804a0 Apply an older controller's accesses exactly as before (hq ADR 0325)
mesh/merge-gate head c85c2804a010 merged into trunk dd9ab7927d96 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head c85c2804a010 merged into trunk dd9ab7927d96 — fail: its merge-check.sh failed: --- FAIL: TestAnAccessTheMeshMayNotKeepYetIsSaidAsAWait…
mesh/delivery superseded: a newer head of the same pull request
The engine delivers before the controller half, and today's controller sends no
identities, users or taken. Keeping now starts only for an access whose folder has
users, or a declaration with identities; every other access is mounted when present
and refused when absent as on main, with nothing made, kept, held or judged. A
leftover whose walk could not run says why, not always that its disk is away.
2026-10-12 19:24:24 +02:00
jschoubben 2ff40109ac Read a filesystem that could not be named as unknown, never as a move (hq ADR 0325 sixth review)
A name read that timed out gave an empty name, which differed from the recorded one
and so reset the record: a folder kept while empty asked again, a warranted one was
walked again. An unnamed filesystem now leaves the folder exactly as it is, the
access is said not known for that look, and it resumes once it is named.
2026-10-12 19:20:53 +02:00
jschoubben 46c16e18fe Name a filesystem by its UUID, its kernel id or its source, and keep nothing that has no name (hq ADR 0325 fifth review)
NFS and FUSE report a zero f_fsid, so another export or another FUSE mount at the
same point was named alike and kept without a warrant; xfs's f_fsid follows the
device number. A filesystem is now named by its UUID (its device matched in
/dev/disk/by-uuid), else a nonzero f_fsid, else mountinfo's source; one that cannot
be named is never kept or taken. Entries left on a filesystem not back within 30 days
are forgotten, said once, and the naming read is shared per mount.
2026-10-12 19:14:42 +02:00
jschoubben f00f9c2a2f Name a filesystem by its kernel id, never its device name (hq ADR 0325 fourth review)
An ext4 or xfs library mounted by UUID can come back from a reboot as another device
(/dev/sdb1 as /dev/sdc1); naming the filesystem by mountinfo's source read that as a
move, stripped the warranted library's entries and asked for a new Approve. A
filesystem is now named by statfs's f_fsid (from the UUID, or the ZFS dataset), its
type and its mount point. Entries left on the filesystem a folder is on now are
merged back or held, never walked away, and a TakenOn naming a mount point is not
taken.
2026-10-12 19:10:18 +02:00
jschoubben 21cd036232 Format the third review's tests (hq ADR 0325) 2026-10-12 19:06:57 +02:00
jschoubben f0adb177f0 Leave no leftover for a folder kept only while empty, and name filesystems by more than their mount point (hq ADR 0325 third review)
A folder kept because it was empty, then covered by its disk, was said as unhealthy
leftover entries for ever: its entries sit on a hidden empty folder and grant nothing,
so they are now forgotten, and only entries walked onto data under a warrant are kept
to be taken out. A warrant and a kept folder now name their filesystem by mount point,
source and type, so another pool at the same point is a move; records naming a mount
point are read as its name. A record with no access ids is said under a path-free name.
2026-10-12 19:06:34 +02:00
jschoubben d2d9fbbd7d Hold what mounts a missing disk, and say entries the mesh could not take out (hq ADR 0325 re-review)
A container was still created over an unmounted mount point, so its writes landed on
the filesystem below; a folder moved to another disk was kept on the old warrant; and
entries left by a drop or a move that could not run lived only in a log line. Every
container mounting an access whose disk is away now waits, the warrant holds only for
the filesystem it was carried out on (the wait names the new one, path-free), entries
left on a filesystem are recorded per mount and taken out when it is back, and every
leftover is said as unhealthy until it is. A hung stat is shared per path, and a
widened mask takes from other entries only the bits it adds.
2026-10-12 19:03:02 +02:00
jschoubben 2026ef36f3 Judge a kept folder before every walk, and keep nothing a disk mounted over (hq ADR 0325 review)
A walk run while a dataset was unmounted wrote onto root's empty mount point, or
"dropped" entries from it and forgot the ones on the data; a folder kept while empty
below an unnamed mount was then swept whole once the disk mounted, with no warrant.
Every walk is now judged as an apply is, a changed mount judges the folder afresh, and
systemd .mount units and mounts seen are read for the first-sight rule. Widening a
mask caps the entries it limited, so the owning group gains nothing. The watch lets
go of removed folders, re-arms on lost notice, reads users under its lock and ends
with its context. A drop that cannot read an ACL is counted, a broken folder's words
carry no path, a grant's group that could not be left is tried again, and a folder's
stat is bounded.
2026-10-12 18:55:47 +02:00
jschoubben 51e97a8c14 Give every module an identity and keep shared folders through ACL entries (hq ADR 0325, issue 567)
A module's containers ran as whatever their image chose and shared folders were
made writable by putting accounts in the folder's group (ADR 0312), which ADR 0325
supersedes. The engine now makes each declared module identity as a system account
with the controller's numbers, runs a container declaring `identity` as it, and keeps
each placed shared folder usable through one POSIX ACL entry per user, set through
the kernel's xattrs on descriptors it opened itself. A folder holding data is kept
only after the operator's warrant (the take act); a missing folder is made where its
data lives and waits where its disk is not; root's data is never kept. A watch and a
bounded sweep put back what an app narrows. The group grant is removed, and what it
put an account in is taken back.

Reuses the descriptor walks, the create act, the missing-path wait and the warrant-
backed act message from the paused issue 567 branch.
2026-10-12 18:43:04 +02:00
jochen c7a4bf5c11 Judge the daily report by wall time, and keep one filesystem check at a time (issue 160)
mesh/merge-gate head c7a4bf5c11d6 merged into trunk d8e785bb4b4b — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head c7a4bf5c11d6 merged into trunk d8e785bb4b4b — pass: its merge-check.sh passed
mesh/delivery delivering: 0 machine step(s) passed
Go's monotonic clock stops while a laptop is suspended, so a day judged by it could pass a day and a half
of wall time before the machine said what it is. And a hung statfs left a goroutine and thread blocked per
apply; now one check runs at a time and reports after it say the sizes unknown until it ends.
2026-10-12 17:42:18 +02:00
jochen ec983bfc3d Have a machine say what it is made of, and say it once a day unasked (issue 160)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The mesh knew a machine's architecture and kernel family and nothing it is made of, and heard even
that only when it sent the machine something. The profile now carries the machine's memory, each local
filesystem's size and free space, processor model, cores and threads, distribution and version, kernel
release, virtual or physical, boot time and timezone; a fact it cannot read is named unknown with why,
never zeroed. The reconcile says the profile unasked once the mesh last heard it 23 hours ago, so on a
machine nobody pushed to nothing the mesh holds of it is older than a day.
2026-10-12 17:21:36 +02:00
mesh-admin dd9ab7927d Merge pull request 'Genesis trusts its registry before it pushes to it (issue 92)' (#91) from fix/92-genesis-trusts-its-registry into main 2026-10-12 14:36:16 +00:00
jochen 0818e4c29a Hold the trust-before-push order by a test, and refuse early what would stop it (issue 92)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The fix is an order, and nothing failed if the trust call moved after the
push or went: steps 7 and 8 are now one function a test drives against a
runtime that refuses an untrusted push. Preflight now refuses, before
anything is raised, a runtime file that is no JSON object, a docker.service
that is not loaded and active, and a dockerd started with
--insecure-registry, which would refuse the reload. The comment names the
order the docker module's takeover rests on.
2026-10-12 16:29:14 +02:00
jochen 41e640d668 Have genesis trust its registry before pushing to it (issue 92)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Genesis pushed the control plane's image to the mesh's plain-HTTP registry
before anything told the container runtime to trust it; the docker module
that writes that trust is applied long after. Genesis now writes the trust
itself at the registry step, with the same into-json writer and a reload,
for the address given and the node's mesh name on the node's registry
port, and refuses before any push unless the runtime says it trusts each.
The docker module later finds the mesh name there and takes it over
without a change.
2026-10-12 16:13:11 +02:00
mesh-admin a1f9856b39 Merge pull request 'A package manager that does not answer fails only the packages; the rest of the declaration applies (issue 233)' (#90) from fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 into main 2026-10-12 14:11:11 +00:00
jochen 74ed7cbe0a Say a re-apply whose result changed, refuse whole when the package manager is not there, and install the packages in the apply that made it answer (issue 233 review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 delivered: every member is delivered
2026-10-12 15:39:07 +02:00
mesh-admin 36704b5471 Merge pull request 'Give a mesh directory its mode and owner only through a descriptor (issue 358)' (#87) from fix/358-descriptor-only-directory-mode into main 2026-10-12 13:13:13 +00:00
jochen 6e44a27a63 Fail only the packages when the package manager does not answer, and apply the rest, so a later declaration can restore what an earlier one removed (issue 233)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 15:12:42 +02:00
mesh-admin dce0eef1ca Merge pull request 'Judge a node's directories read-only before the walk refuses any (issue 358)' (#88) from fix/358-judge-directories-read-only into main 2026-10-12 12:48:55 +00:00
jschoubben e3c3bb37d0 Give a mesh directory its mode and owner only through a descriptor, and gate what the walk refuses (issue 358)
mesh/repo-check pass: its merge-check.sh passed
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/delivery delivered
A mesh directory outside every home was chmodded and chowned by path, so a
parent turned into a link between two applies carried root's chmod and chown
to the link's target. It is now made and changed through the descriptor of
the walk from / that follows a link only root alone could have made.

A directory that walk refuses is a refusal like the placement guard's
(review 47): judged before anything is applied, and again when the
directory is applied, used as found or not; refused at either point it gates
its module and refuses what is below it, so no file is written into the
link's target and the report does not say everything was attempted.
2026-10-12 14:00:53 +02:00
jschoubben b92fb68116 Judge a node's directories read-only before the walk refuses any (issue 358)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/358-judge-directories-read-only delivered: every member is delivered
Once a mesh directory's mode and owner go only through a walk that follows no
link another account could make, a directory behind such a link is refused
and its module gated: on the control-node that could stop the controller.
The operator is to be told which directories that walk would refuse on every
node before it lands. So the engine judges the declaration it keeps with the
same walk, making and changing nothing: on a signed ask from the controller
(mesh.node.<node>.ask.judge-directories, held to the hand-over's checks) and
as `nox-mesh-host judge-directories` at the machine.
2026-10-12 13:57:52 +02:00
mesh-admin 3f7f797168 Merge pull request 'A read-write access grants the path's own group, and the path is never changed (hq ADR 0312, issue 537)' (#86) from feat/537-a-read-write-access-grants-the-paths-group into main 2026-10-12 11:22:56 +00:00
jschoubben c433e76de2 Judge a granted group by number, grant nothing a group cannot write, and keep the look bounded (hq ADR 0312, issue 537 review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The engine is built without cgo, so a check by name missed a root group a name
service holds or one number carrying another name: the root groups are resolved
to numbers (now with kmem and shadow) and a number named nothing is refused for
containers too. A group without write is granted nothing (§2). The look reads
each path off the lock within five seconds, so a dead network mount cannot stop
every health statement. A grant's account may not start with a dash. A group a
record leaves for another asker is handed to it, so it is still taken back when
the last asker goes (§5). The scheduler skips paths issue 339's guard refuses.
2026-10-12 12:53:42 +02:00
jschoubben f4efc3cad0 Grant a read-write access's path group to what runs the module's code, so a module can write shared data without the mesh changing it (hq ADR 0312, issue 537)
The engine reads the group owning the accessed path at apply and gives it to the
containers and the tools' account an access's new `grant` names: --group-add for a
container (recreated with it), an account group for the account (recorded, taken
back as ADR 0252 takes back). The path is never changed; an access whose group
cannot write is stated unhealthy for the operator to decide; group 0 and the
root-granting groups are refused by name. The tool runner restarts through the
restart-on the controller gives it, because the access reports updated exactly
when the account's groups moved. Health says GrantContract (4), which tells the
controller it may send `grant`.
2026-10-12 12:23:38 +02:00
mesh-admin a76baa4b47 Merge pull request 'Carry the controller's grant to ask the delivery's owner for its groups in the installer's first user list (hq ADR 0306, issue 525)' (#85) from feat/525-grant-delivery-groups into main 2026-10-12 10:14:32 +00:00
jschoubben 90ce9e9d9f Carry the controller's grant to ask the delivery's owner for its groups in the installer's first user list (hq ADR 0306, issue 525)
mesh/delivery delivered
mesh/repo-check pass: its merge-check.sh passed
mesh/merge-gate pass: the change touches no module of the mesh's graph
The controller is to ask mesh-delivery whether a merge's delivery
group still has a member to merge, so that a merge in no waiting group
is cut at once (ADR 0304, ADR 0306); its bus grant names the verb, and
the installer's first user list must too, as for times (212a3cc).
2026-10-12 12:09:18 +02:00
mesh-admin df5de3bd9f Merge pull request 'Say an unhealthy resource's restart count as what it counts (issue 458)' (#84) from fix/458-restarts-say-what-they-count into main 2026-10-12 09:03:47 +00:00
jschoubben a18d09db75 Say an unhealthy resource's restart count as the engine's own since it began watching
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/458-restarts-say-what-they-count delivered: every member is delivered
Beside the state it read as restarts since then (novox/hq issue 458).
2026-10-12 10:47:07 +02:00
mesh-admin 3333a05a74 Merge pull request 'A refused directory gates its module, and a file meets the placement guard (issue 495)' (#83) from fix/495-a-refused-directory-gates-its-module into main 2026-10-11 17:00:22 +00:00
jschoubben 4695dfc0a2 Say both a gated module and a machine stop, and count skipped apart (review of #83)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A module-less action that stopped the apply after a module was gated was reported as
everything else attempted. The failure now names the gated modules and the stop, counts
skipped resources apart from applied ones, and the tests cover an archive below a refused
path, a gated module tried again by the next apply, and a refused file among the failures.
2026-10-11 18:51:04 +02:00
jschoubben 7233ddbe35 Gate a module behind its refused directory and judge a file by the placement guard (issue 495)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
On ace the engine refused Docker's data root as not placed, said nothing was touched, then
wrote a file into it and ran the module's step. A refused directory now gates the rest of
its module as a failed step does, a file is refused in a forbidden tree or below a refused
path, and the failure names the modules whose rest was not attempted.
2026-10-11 18:19:25 +02:00
jschoubben 1acfed4add Test that a refused directory gates its module and a file meets the same guard (issue 495)
Both fail on main: the shape seen on ace, a refused directory, a file below it and a
step after it, writes the file, runs the step and reports that everything was attempted.
2026-10-11 18:17:59 +02:00
mesh-admin 6defec8f68 Merge pull request 'Refuse a declaration composed from an older assignment generation than the machine applied (issue 234)' (#82) from fix/234-refuse-a-declaration-from-an-older-generation into main 2026-10-11 16:00:01 +00:00
mesh-admin 2833caf3d6 Merge pull request 'A failed run-once process runs again on every apply until it passes (hq issue 467)' (#81) from fix/467-a-failed-step-runs-again into main 2026-10-11 09:30:49 +00:00
jschoubben 20d5af9b2d Refuse a declaration composed from an older assignment generation than the machine applied (hq issue 234)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A newer sequence carrying an older view of the assignments undeclared four
assigned modules on 2026-10-04; the sequence cannot tell it, the generation
can. The engine keeps the highest generation it applied, refuses an older one
before reading the machine unless the declaration is a gate's put-back, and
says the refusal in its report so the controller can name the sender. A
declaration without a generation is applied as before, and every report says
it reads one, so the controller sends it only to engines that do.
2026-10-11 11:22:30 +02:00
jschoubben 0ec96a048e Clear a step's success only when it was started and failed, and say updated when it passes again (novox/hq issue 467)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A bundle that could not be fetched refused nothing; clearing for it would rerun
every step on a node once the artifact store came back from an outage.
2026-10-11 11:12:15 +02:00
jschoubben c957f1ec5e Run a failed run-once process again on every apply until it passes (novox/hq issue 467)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A failure recorded nothing, so the record of the step's last success stayed and
the next apply skipped it, opening ADR 0136's gate on what it had just refused.
The record is kept, for removal, with what it was run as cleared.
2026-10-11 11:07:17 +02:00
mesh-admin 85c51ebb59 Merge pull request 'A refused line's request is read before it is answered, so a busy line reads busy (issue 434)' (#80) from fix/434-a-busy-line-is-answered-busy-whatever-the-load into main 2026-10-11 00:55:47 +00:00
jschoubben f9805b2113 Bound what a refusal costs the engine, and let the busy tests fail instead of hang
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A refusal waiting for its request parsed it into a 64 KiB reader for up to 5 s,
on a socket every account may open. It now drains at most 4 KiB through a small
buffer for at most 1 s and never past the accept-time deadline, only 32 wait at
once, and an unknown account is refused before it takes a slot. The busy tests
wait for the first line with a timeout and say what came of it (novox/hq issue 434).
2026-10-11 02:47:11 +02:00
jschoubben 70cb613614 Read a refused line's request before answering it, so a busy line is read as busy
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The engine answered busy and closed before the client's request had landed;
the client's write then broke and it never read the refusal waiting for it.
Under the whole suite's load this failed the busy test (novox/hq issue 434).
2026-10-11 02:35:20 +02:00
mesh-admin bd5cc69804 Merge pull request 'The setuid walk paces itself by the machine's I/O pressure (issue 433)' (#79) from fix/433-the-setuid-walk-yields-to-a-busy-disk into main 2026-10-11 00:03:46 +00:00
jschoubben f311fa36dc The setuid walk is paced before each file too, so a large directory is not read blind (review of issue 433)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A file's own details (lstat) may be read from the disk as a directory is,
so a directory of tens of thousands of files was read without a look and
the header's promise that no burst is longer than PaceLook did not hold.
The walk now asks the pacer before each file as well; the 250 ms gate
makes that a clock read per file. A test with one directory of twenty
files fails without it. The clock is read once after a run of pauses,
and the constants test says that it guards the constants only.
2026-10-11 01:55:39 +02:00
jschoubben 30242197fa The paced walk keeps a tenth of the time, and reads the pressure since its last look (review of issue 433)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The review of #79 found that pacing alone brought back issue 361's
fault. The control-node's own I/O pressure sits near the 10 % bound with
no walk running, so a walk that paused 30 s for every 250 ms it read
would read 0.8 % of the time: a 16-minute walk would take some 32 hours,
outlive the age a search judges for, and be begun again at an engine
restart, leaving the machine unjudged.

A run of pauses is now at most nine times as long as the walk read since
the one before, so the walk keeps at least a tenth of the time whatever
the pressure, and takes at most ten times as long as unpaced: about
2.7 hours for the 16-minute walk, inside the 12 hours the controller
waits quietly. A test ties the bound to rootsearch.Quiet.

The pressure is the share of the time since the last look in which some
task waited on I/O, from PSI's cumulative counter, not its ten-second
average, which the kernel moves only every two seconds. A pause is
500 ms, then the pressure is looked at again.

The engine's walker is built in one place, engineWalker, and a test
fails if it is not paced by /proc/pressure/io. What the walk cannot read
is said through the search's own channel (the cache's Say), as the rest
of the search is, with what it means for the disk.
2026-10-11 01:53:25 +02:00
jschoubben 8f7db549ae The setuid walk paces itself by the machine's I/O pressure (issue 433)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The walk for setuid programs is put in the idle I/O class so it yields
to everything else, but only a scheduler that schedules by class (BFQ)
honours that. The control-node's disks run mq-deadline, which does not,
so the walk read at full priority. On 2026-10-11 the walk started at
01:27, right after the engine applied a new controller, and ran sixteen
minutes; it kept two spinning disks saturated, the bus whose store
shares them stalled for half a minute twice, and the controller lost
its lease and restarted each time.

The walk now also paces itself by PSI, which reads the same whatever
the scheduler: before it reads a directory, while some task has waited
on I/O for more than 10 % of the last ten seconds, it pauses two
seconds at a time. It looks at most every 250 ms, pauses at most 30 s
in a row and then reads one more directory, so a machine that is never
quiet is walked slowly but still judged. Its own reads count in the
pressure, so on an otherwise idle machine it holds the disk below the
bound. Pressure it cannot read pauses nothing, and is said once.

A paced walk finds what an unpaced one finds; a walk ended while it
pauses is an incomplete walk, never one that found nothing.
2026-10-11 01:47:43 +02:00
mesh-admin 5226d7bcd0 Merge pull request 'Node-engine: state a tool check that waits for the operator as waiting (issue 386, ADR 0283)' (#78) from feat/386-a-wait-for-the-operator into main 2026-10-10 20:03:27 +00:00
jochen 9545121ec9 Clear an old pass on a wait, and say a change of waits (review of #78)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A resource that passed, then waited, then failed once after its grace was
still stated healthy from the old pass; and a change of what it waits for
alone raised no change event, so the controller could miss it.
2026-10-10 21:14:14 +02:00
jochen 3abb07f1e2 State a tool check that waits for the operator as waiting (issue 386, ADR 0283)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
A module whose only fault was a secret the operator had not given yet was
stated unhealthy, failed its first-node gate and was put back. The engine
now reads the tool's answer through mesh-sdk go/health and states such a
resource waiting, with what it waits for, on one look; a malformed wait
stays unhealthy.
2026-10-10 21:11:02 +02:00
mesh-admin 9d8aa9d6f0 Merge pull request 'Carry the controller's grant to ask mesh-delivery for its delivery times in the installer's first user list (hq ADR 0282, issue 382)' (#77) from feat/382-carry-times-grant into main 2026-10-10 18:39:44 +00:00
jochen 212a3cc113 Carry the controller's grant to ask the delivery's owner for its delivery times in the installer's first user list (hq ADR 0282)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 20:32:16 +02:00
mesh-admin cc0b8554b1 Merge pull request 'Show how far the setuid search got, and start a fresh one when the terminal asks (hq issue 361)' (#76) from fix/361-the-setuid-search-shows-its-progress into main 2026-10-10 00:21:51 +00:00
jochen 0b4c86e3be Count a setuid find only where the account can execute it, read at every judgement (hq issue 361)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The two finds on the control-node sit in the operator's home of mode 0700, which the agent account cannot
search. The search keeps every find; each judgement reads the modes on the way again, since an account can
open its own home without root. A find in a home is said by count, its path in the journal alone. Also
from review: a checkpoint racing SearchNow forgets the walk it just kept, and a walk kept by an older engine
says its counts as lower bounds.
2026-10-10 02:00:10 +02:00
jochen 8e79403c21 Show how far the setuid search got, and start a fresh one when the terminal asks (hq issue 361)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestNatsAnEnrolmentWithNoReplyAddressIsNotAnswered (0.00s)
mesh/delivery superseded: a newer head of the same pull request
The walk kept its progress every minute and nobody could read it: root-free said only that a search ran.
The account verdict now carries the progress (counts, times, mount points; never a path) as fields and in
its reason, outside what makes a verdict changed. A signed ask from the controller's terminal throws the last
search away and starts a full one, after the operator removed by hand what it found.
2026-10-10 01:55:58 +02:00
mesh-admin b8c8546118 Merge pull request 'Let the setuid search run to completion, resumably, and judge from the last complete one (hq issue 361)' (#75) from fix/361-the-setuid-search-runs-to-completion into main 2026-10-09 23:18:44 +00:00
jochen a7270f00cb Keep the walk's paths as bytes, forget a walk root superseded, and leave other mounts unread (hq issue 361 review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/361-the-setuid-search-runs-to-completion delivered: every member is delivered
The review found a cursor of a name that is not UTF-8 came back changed
from JSON, so a resumed walk skipped what it stood before; an ask to
search again was lost to a restart, which carried the older walk on; a
fresh search judged a filesystem it never walked; a mount point root
cannot read failed or hung the walk; a found file gone before the package
manager was asked failed every resume; and skipped or kept outcomes
started a search on every reconcile, back to back with a long walk.
2026-10-10 01:10:56 +02:00
jochen 2dbbda51c6 Walk for setuid programs in the engine and carry an interrupted walk on (hq issue 361)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/361-the-setuid-search-runs-to-completion ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
The control-node's walk takes far longer than the engine runs between
updates, so a search that starts over at every restart never completes.
The walk is the engine's own, in lexical order, so its progress is one
cursor per filesystem, kept every minute beside the engine's state and
carried on after a restart while younger than rootsearch.FreshFor. It runs
on a thread of its own at nice 19 and the idle I/O class.
2026-10-10 01:00:47 +02:00
jochen 2410796869 Let the setuid search run to completion and judge from the last complete one (hq issue 361)
A machine with millions of files never finished the walk within fifteen
minutes, so its agent account was never judged and the router refused to
link. Only root can make a setuid-root file, so a complete search stays
sound until root acts: it now runs at idle priority with no bound, is kept
beside the engine's state, judges for rootsearch.FreshFor from its start,
runs again daily and after an apply that moves accounts, packages, actions
or sudo rules, and an incomplete or stale search is never free.
2026-10-10 00:53:05 +02:00
mesh-admin 090b9082eb Merge pull request 'Renew the engine placed by hand to the proved build, and take a hand-over from the controller's terminal (hq issue 356)' (#74) from fix/356-hand-over-by-nox-and-the-engine-on-path-renewed into main 2026-10-09 18:07:21 +00:00
jochen 8fc1d72cd5 Keep the setuid search's time on the caller's clock
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed: a gate on a first machine (what it carried put back), a build, a machine
SetuidCache compared the caller's now with a time read from the wall clock when the search finished. When the caller's clock ran behind the wall clock, a search was never due again: TestTheSetuidSearchRunsInTheBackgroundAndIsKeptForItsInterval, whose clock is a fixed date, has failed on main and every branch since 2026-10-09 18:00 UTC. A search's time is now the now it was started at; a result reads at most one search older than it is, which errs towards searching again.
2026-10-09 20:04:11 +02:00
jochen 9a807de659 Give a handed-over directory through no link on every apply, not only the first (hq issue 356, review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheSetuidSearchRunsInTheBackgroundAndIsKeptForItsInterval (0.00s)
mesh/delivery superseded: a newer head of the same pull request
After the first apply the record is no longer used as found, so the directory read as the mesh's own and later applies set its mode and owner by path: a parent swapped for a link after the hand-over took carried root's chmod and chown to its target. The hand-over is now checked before that shortcut, so a handed-over directory always goes through applyHandedOver. The test swaps the parent and the grandparent after an apply already gave the directory its mode; both cases fail with the old order.
2026-10-09 19:58:27 +02:00
jochen fc6c5a92e6 Believe a hand-over only signed by the mesh, and give it through no link (hq issue 356, review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Two holes the review proved. The ask's subject proved nothing — a responder can reach it through a reply — so the engine now verifies the mesh's signature over a fixed context with the key it verifies declarations with, before reading anything, and refuses an ask for another node, expired, good too long, without a nonce, or replayed. And the apply followed a link the hand-over had refused: a component swapped for a link after the record carried root's chmod and chown to its target. A handed-over directory must now still resolve to itself, and is changed only through one descriptor opened from / with no link followed (fchmod, fchown). Tests for the reply guard, the run loop's hook, the fallback renewal on proof and a non-regular fallback; 27 checks each removed in turn, each failing a test.
2026-10-09 19:45:00 +02:00
jochen e74ee82a76 Renew the engine placed by hand to the proved build, and take a hand-over from the controller's terminal (hq issue 356)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The file on PATH was the engine a person copied at installation, and every delivered engine ran from the versions directory: the operator typed hand-over and a build from before it answered "unexpected argument". A proved engine now copies itself over that file when the bytes differ (never a link). And the hand-over of a directory used as found (issue 339) gets its channel: the controller's terminal asks this node's engine on mesh.node.<node>.ask.hand-over; the engine judges every value — the ask's size and shape, who asked, the path as it states it, a directory and no link, through no link — and records it or refuses and records nothing. Every text the engine addresses to the operator names the nox line (ADR 0272); the engine's own subcommands are install-time and rescue, and its usage says so.
2026-10-09 18:34:10 +02:00
mesh-admin b0d03be7a4 Merge pull request 'mesh-cli: forward the line's standard input to the controller, with the line alone (hq ADR 0259 §10, ADR 0272)' (#73) from feat/a-terminal-line-takes-standard-input into main 2026-10-09 15:58:48 +00:00
jschoubben 4ea6a2f04e mesh-cli: forward the line's standard input to the controller, with the line alone
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/a-terminal-line-takes-standard-input stopped: a member was stopped
A secret given at the controller's terminal through mesh-cli (secret accept … --from -) never reached the
line: the engine forwarded the words only. It now forwards what mesh-cli's standard input held (at most
64 KiB, the SDK's bound) on the same core request to the controller, says it nowhere, and sends none
with a follow of the same call. mesh-sdk moves to f047d0a.
2026-10-09 16:34:45 +02:00
mesh-admin d8ff1540df Merge pull request 'Installer's first user list: the controller's grant to ask the desk for a secret (hq ADR 0259 §10)' (#68) from feat/a-secret-given-at-the-desk into main 2026-10-09 14:17:41 +00:00
mesh-admin cdc2af7b43 Merge pull request 'Installer's first user list: the controller's grants to ask the operator (hq ADR 0259)' (#59) from feat/the-controller-asks-the-operator into main 2026-10-09 14:17:38 +00:00
mesh-admin 7a2ffc0a4c Merge pull request 'Read the session record strictly, and make its tests hold (hq ADR 0272)' (#72) from fix/272-the-session-record-read-strictly into main 2026-10-09 12:12:11 +00:00
jochen b1e7b6d26e Read the session record strictly, and make its tests hold (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
- UID, CLASS and SCOPE are read at their first occurrence: logind writes
  each once, and a later line saying otherwise is not the record's word.
- A comment says /run/systemd/sessions is systemd's private format, and
  that a format change fails closed: no session, so nobody is the terminal.
- Tests that hold, each proved by the mutation it guards failing:
  - a record naming UID=1001 for session-8.scope under user-1000.slice,
    asked as uid 1000 (the record's UID check alone);
  - session-a.b.scope with a passing record (the id's characters alone);
  - session-9.scope/x.scope (the scope being the slice's direct child);
  - a key said twice, either way round (first occurrence);
  - the client sends its line only after the session was read, so reading
    the session in the forward fails the test.
2026-10-09 14:05:53 +02:00
mesh-admin 512b0edb02 Merge pull request 'A login session is the account's own, of class user, read at accept (hq ADR 0272)' (#71) from feat/272-the-session-is-a-users-read-at-accept into main 2026-10-09 12:04:18 +00:00
jschoubben 4ba6d0864e Carry the controller's grant to ask the desk for a secret in the installer's first user list (hq ADR 0259 §10)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/a-secret-given-at-the-desk delivered: every member is delivered
mesh-controller #156 (give) grants the controller mesh.seat.node-launcher.tool.secret.*; the
controller's test holds the installer's list equal to what it composes, and failed without it.
2026-10-09 14:04:01 +02:00
jochen 73e0916968 Carry the controller's grants to ask the operator and act on a warrant in the installer's first user list, so a mesh raised from it is not refused them (hq ADR 0259)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-host@4ba6d0864e3d (merged as d8ff1540 into main, walk plan-17915554870866…
mesh/delivery-group group feat/the-controller-asks-the-operator delivered: every member is delivered
2026-10-09 14:03:58 +02:00
jochen bf3f45e769 A login session is the account's own, of class user, read at accept (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The cgroup names a scope; only logind's record of it says whose session it
is. The engine now takes a session only when /run/systemd/sessions/<id>
says it is this account's (UID), of class user (not a greeter's or the
user manager's), and of this very scope. And it reads the session the
moment the connection is accepted, with the namespaces, before the line,
so the peer is judged as it was when it connected.

Tests: session-9.scope with a user record is taken; x.scope, a different
uid, another account's session, a greeter's, a record naming another
scope, no record, and a path with dots are not; the session is read
before the line is forwarded.
2026-10-09 13:55:45 +02:00
mesh-admin fd13834c0d Merge pull request 'Say the login session a mesh-cli line comes from, and pin the SDK at go/v0.1.9 (hq ADR 0272)' (#70) from feat/272-a-login-session-and-the-tagged-sdk into main 2026-10-09 11:54:31 +00:00
jochen 4a5dcb0209 Say the login session a mesh-cli line comes from, and pin the SDK at go/v0.1.9 (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A service running as the operator — the tool runner, a unit of the
account's user manager — shares the operator's user id, and was as much the
terminal on the control-node as a person typing. The engine now reads the
asking process's cgroup and sends its login session's scope
(session-<id>.scope under the account's own user-<uid>.slice), or nothing;
the controller makes nothing else the terminal. The SDK is pinned at its
tag now that go/v0.1.9 exists.
2026-10-09 13:40:17 +02:00
mesh-admin 1ac7741c63 Merge pull request 'Vouch for the account that runs mesh-cli, and ask the controller its line (hq ADR 0272)' (#69) from feat/272-the-engine-vouches-for-mesh-cli into main 2026-10-09 11:38:39 +00:00
jochen f684967c3b Merge main, and close what the confirmation review found (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
- Any failure of a request the link may have delivered now says the line
  may have run; "nothing ran" is said only of a request the link knows it
  never asked (no link open, nobody listening: link.ErrNotAsked).
- A test holds that a connection saying nothing is answered within its
  deadline, set at accept, and gives its slot back.
- The README says the socket refuses a process outside the machine's own
  namespaces, so flatpak, toolbox and distrobox shells are refused.
2026-10-09 13:09:48 +02:00
jochen 655f515aa7 Merge remote-tracking branch 'origin/main' into feat/272-the-engine-vouches-for-mesh-cli 2026-10-09 13:09:16 +02:00
mesh-admin 8f763ee911 Merge pull request 'Judge whether an account declared never to become root can (hq ADR 0266)' (#61) from feat/an-agent-account-never-becomes-root into main 2026-10-09 10:56:00 +00:00
jochen 564948c2f2 mesh-cli: follow a line to its answer, bound each account, refuse other namespaces (review of hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
- A request that timed out was said as "the controller was not reached,
  nothing ran", though the line may have run. The engine now follows a line
  the controller answers as running, by its call, until it ends; when no
  answer comes in time it says the line may have run and names the call.
  "Nothing ran" is said only of a request no controller took.
- One account could hold the socket: accepting waited for a free slot, and
  a connection could stall before its deadline was set. The deadline is set
  at accept, before any lookup; each user id has two lines at once and the
  machine eight, and one over is answered busy at once.
- A container running with the operator's user id shares the number, not
  the machine: a peer whose mount or user namespace is not the engine's is
  refused, and nothing is asked.

SDK pinned at the go/cli branch's newest commit until go/v0.1.9 is tagged.
2026-10-09 12:48:20 +02:00
jochen 1c61b72f35 Read the homes in one order, so their test does not depend on a map's
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The homes come from a map of the user database, and the test compared them in
the order it happened to iterate: it failed one run in four, the repo check
of 2026-10-09 among them.
2026-10-09 12:36:56 +02:00
jochen 2673e7a2c9 Let the setuid search run to its 15-minute bound, ask pacman once, and call a walk with errors incomplete
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestHomesAreAPersonsOrAnAgentsNotAServicesOrRoots (0.00s)
mesh/delivery superseded: a newer head of the same pull request
The search kept a two-minute limit of its own inside its 15-minute bound, so
a long walk always ended incomplete; pacman was asked once per path; and find
exiting with an error but some output was read as a complete walk. The bound
and the not-judged reasons move to the public rootsearch package so the
controller reads the same values (the review of 2026-10-09).
2026-10-09 12:11:57 +02:00
jochen 021d860ca4 Vouch for the account that runs mesh-cli, and ask the controller its line (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The records send the operator to "the controller's terminal", and nothing
reaches it (hq issue 343). mesh-cli holds no credential: it asks the engine
on its own machine over a socket. The engine reads the asking account from
the kernel (SO_PEERCRED), never from the program, and asks the controller on
mesh.control.<node>.cli, its own node's subject, so the bus server says which
node asked. The controller decides what the line may do; nothing received
on the socket is run or applied here, and the socket has no network address
(ADR 0004 stands).

The local protocol is the SDK's go/cli package, at a pseudo-version of its
branch until that merges and is tagged go/v0.1.8.
2026-10-09 12:05:53 +02:00
jochen b28d7bbcbf Search for setuid programs in the background, and judge each polkit rule alone
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A search over a large disk ran inside the look, holding the judge's lock and
stalling the health statement; it now runs apart, serving its last result,
backing off after a failure, and saying not judged until it has one. And a
rule naming a user, or a comment, no longer hides another rule's
unconditional yes (hq ADR 0266).
2026-10-09 10:10:05 +02:00
jochen e2e8be72fa Keep main's placement guard, and refuse only an archive or a written-into file below an agent's home
Issue 339's guard (#63/#64/#65) now refuses the machine's own trees, another account's
directory below a home, and uses a found directory as found; this branch's own copy of
those rules goes. What stays is the one rule only an account that never becomes root
needs: below its home it owns every parent and could swap a link in between the
engine's check and an unpack or a write-into (hq ADR 0266 §7). One reader of the user
database serves both.
2026-10-09 10:10:05 +02:00
jochen d4af8bba1b Judge an opened file's kind and links below a home, and more ways to root
A hard link swapped in for ~/.claude would have had root chown another
account's file; fstat on the descriptor now refuses a second link, a fifo or
an unexpected kind before anything is changed (hq ADR 0266, the re-review).
The judge also finds polkit rules for every account, a runtime's API on TCP,
setgid-to-root programs whoever owns them, setuid programs on every suid
filesystem, and unprotected links; the rest is listed as not judged.
2026-10-09 10:07:52 +02:00
jochen 02a1c5f753 Refuse to take over a system account as one that never becomes root
The controller cannot read a machine's user database, so a name it is given
for the agents' account (hq ADR 0266) may be a service's own; taking it over
would hand agents that service's files. Refuse it, touching nothing.
2026-10-09 10:07:52 +02:00
jochen b89b143fe7 Follow no link below a home as root, and judge more ways to root
An account could replace ~/.claude with a link to /etc and have the
node-engine chown, chmod or write through it on the next apply. Below a
person's or an agent's home every component is now opened without
following a link, and a link refuses the resource in words.

The root judge also reads doas and polkit rules, the container runtimes'
sockets with their ACLs, ACLs on the secrets, and setuid-root programs no
package owns, in the C locale; Judged and NotJudged write down exactly
what it covers (hq ADR 0266 review).
2026-10-09 10:07:52 +02:00
jochen 3a727011ad Judge whether an account declared never to become root can, so a machine's agents are known confined
An account an agent runs as (novox/hq ADR 0266) is read on every look for a
uid of 0, a group that grants root, any sudo rule and a mesh secret it can
read; any way found is unhealthy and said, a read that fails is unknown.
2026-10-09 10:07:35 +02:00
jochen d93cc86f95 Let a user say it never becomes root, so the agents' account can be judged
A declaration may now state root: never on a user (novox/hq ADR 0266), and a
health statement carries it under contract 3; the judging follows.
2026-10-09 10:07:35 +02:00
mesh-admin a390bce5f2 Merge pull request 'Give a service's held state its own field and key, so main compiles again' (#67) from fix/store-declared-redeclared into main 2026-10-09 01:09:40 +00:00
jochen c2bae14994 Give a service's held state its own field and key, so main compiles again
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
#64/#65 and #66 each added Applied.Declared: a directory's declared owner and
mode (string) and a service's held state (*FoundUnit), both under the JSON key
"declared". The directory's is already on nodes, written by d28ca95, so it
keeps the name and key; the service's becomes DeclaredUnit, "declared_unit".
#66 never deployed, so no node holds a record under the new key.
2026-10-09 03:03:58 +02:00
mesh-admin 690966d847 Merge pull request 'Start again on removal what the mesh's own declaration stopped (hq ADR 0269)' (#66) from feat/give-back-a-unit-the-mesh-stopped into main 2026-10-09 00:54:13 +00:00
mesh-admin d28ca959f0 Merge pull request 'Issue 339 follow-up to #64: ask the running runtimes where they keep their data, and read every form a unit says it in' (#65) from fix/339-runtime-roots-asked into main 2026-10-09 00:15:24 +00:00
mesh-admin b232db4e17 Merge pull request 'Issue 339 follow-ups: a directory used as found is said and can be handed over; the runtime's data and any .ssh refused' (#64) from fix/339-found-directories-are-said into main 2026-10-09 00:15:17 +00:00
jochen dfb14151bd Say in the plan that a unit is restored only if it is still as the mesh held it
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery failed: its walk failed: a gate on a first machine (what it carried put back), a build, a machine
Confirmation pass of #66: the plan is a preview, and the removal reads the unit when it runs; a masked
unit or one changed since is left as it is.
2026-10-09 02:03:51 +02:00
jochen a018ebd1ca Pass -- before every unit, and give a unit back only when it is exactly as the mesh held it
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Third review of mesh-catalog #147.

A unit's name can now come from a setting (a template's instance), so every systemctl call that names a
unit passes `--` before it, and a name can never be read as an option.

On removal, a unit is started or enabled again only when it is exactly the stopped and disabled the mesh
held it to. A masked unit, or one somebody started or enabled since, is left as it is, and the report
says why. A restore that fails (the start refused, or the unit not running after it) puts back what this
attempt changed and is reported as waiting, so no apply is made fatal by it and the next one tries again.

The tests' fake service managers read systemctl's arguments in the shape they were written for
(oldShape); internal/system's own test holds the `--`.
2026-10-09 01:54:26 +02:00
jochen 7c046a8a1d Accept and ignore a file's trusted, and give each runtime its own time to answer
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The fourth review (hq issue 339): `trusted` is the catalogue's word and the
controller takes it out, but a controller that passes it on, an older one or
one rolled back to, must not cost a node its whole declaration. A file now
accepts it and nothing reads it; any other resource still refuses it. And a
podman that hangs no longer eats docker's ten seconds.
2026-10-09 01:42:09 +02:00
jochen 9d48b8baf9 Start again on removal what the mesh's own declaration stopped (hq ADR 0269)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A unit found running and enabled that a declaration held stopped and disabled stayed stopped and
disabled when the declaration went: ADR 0118 never starts a unit on the way out. Unassigning the zfs
module would so have left the home server's pool with no scrub schedule at all, said nowhere.

The host now records what it declared for a service (Declared, beside Found). On removal, a unit found
running and declared stopped is started again, and one found enabled and declared disabled is enabled
again, each read back; the report says restored, and the plan says it first. A record from before the
field starts nothing.
2026-10-09 01:35:27 +02:00
jochen ff2553f4db Ask the running runtimes where they keep their data, and read every form a unit says it in
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The third review of #64 (hq issue 339): an ExecStart continued over lines, a
quoted path, a variable from Environment= or EnvironmentFile= across the unit
and its drop-ins, dockerd's --config-file and podman's TOML literal strings
each hid a data root from the guard. The running runtimes are now asked first
(docker info, podman info) on a runner of the guard's own, and the files are
read beside them in all those forms.
2026-10-09 01:22:11 +02:00
jochen ae1293e73e Read every runtime's data root, keep when a directory was found, and key a hand-over to its resource
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-host@7c046a8a1d84 (merged as d28ca959 into main, walk plan-17915049347305…
The review of #64 asked for containerd's tree, dockerd's --data-root from its
unit and drop-ins, and podman's graphroot to be refused like docker's own
(hq issue 339); and the controller now tells a directory a send found from
one found before it, which needs the time it was first found kept across
applies instead of the time of the last apply. A hand-over now names the
resource it was for, so it lapses if another declaration points at the path.
2026-10-09 00:55:21 +02:00
mesh-admin a2c2c35e85 Merge pull request 'Issue 339: the node-engine places and mounts nothing at the machine's own trees, and uses a found directory as found' (#63) from fix/339-the-engine-places-nothing-in-the-machines-own-trees into main 2026-10-08 22:36:45 +00:00
jochen 07164e40f3 Say every directory used as found, let a person hand it over, and refuse the runtime's data and any .ssh
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A directory the engine used as found kept its owner and mode in silence, so
a module that cannot write it would just fail (hq issue 339 review). Each
one is now a WARNING in every apply's log and its module's resource of kind
directory, stated unhealthy on every look with its owner and what was
declared, which the controller raises as the module's condition; `mesh-host
hand-over <dir>` at the machine hands it to the mesh, and the next apply
gives it the declared owner and mode. The guard also refuses the spool, /opt,
the container runtimes' data (its configured data-root too), and an account's
.ssh as an access, a mount, or a directory anywhere but its own home.
2026-10-09 00:28:01 +02:00
jochen f39dbdbaf0 Give the home test's account a uid of its own, so it holds when the suite runs as root
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
2026-10-09 00:10:28 +02:00
jochen 3a002f6e7b Refuse a placement or mount at the machine's own trees, and use a found directory as found
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestBelowAHomeOnlyThatAccountsDirectoryIsPlaced (0.00s)
A module's places setting could point a directory anywhere, with an owner it
named, and the engine chowned and chmodded whatever it was pointed at as root,
through a link too; its accesses setting could mount any path into a container
(hq issue 339). The engine now refuses, as a failed resource with its reason,
a directory, access or mount source that is or holds one of the machine's
roots, anything placed in the kernel's, the boot loader's, root's or the
engine's own trees, a non-root directory below /etc, /usr or /run, an access
there, another account's directory below a home, and a container mounting
anything it refused, judged with every link resolved. A directory whose last
component is a link is refused, and one found here that the mesh did not make
keeps its owner and mode and is never removed.
2026-10-08 23:58:14 +02:00
mesh-admin ee20388239 Merge pull request 'Hold the removal order a module may rely on, and say what a killed stop leaves (hq ADR 0263)' (#62) from fix/shares-review-followups into main 2026-10-08 20:57:40 +00:00
jochen fedd9dcf35 Hold the removal order a module may rely on, and say what a killed stop leaves (hq ADR 0263)
mesh/delivery-group group fix/shares-review-followups delivered: every member is delivered
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
2026-10-08 21:16:04 +02:00
mesh-admin 29a7de7699 Merge pull request 'Let the first controller keep and deliver again what a consumer gives up on (hq issue 330, ADR 0264)' (#60) from fix/330-a-message-given-up-on-is-kept into main 2026-10-08 18:21:04 +00:00
jochen a31659ed03 Carry no seat queue grant for the first controller
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/330-a-message-given-up-on-is-kept delivered: every member is delivered
mesh/delivery delivered
The controller no longer delivers a seat's ask again (review of hq issue 330).
2026-10-08 18:37:58 +02:00
jochen ccbfa9b58a Let the first controller keep and deliver again what a consumer gives up on (hq issue 330)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The installer's user list must be what the controller composes, or it is
refused on its first copy into DEAD_LETTERS (ADR 0264).
2026-10-08 17:52:59 +02:00
mesh-admin e813391dcf Merge pull request 'Add the replay of issue 331 (R331)' (#58) from replays/331-a-tool-check-on-the-live-link into main 2026-10-08 15:17:20 +00:00
jochen 5684a4579d Add the replay of issue 331, so a tool check refused as no link fails before a merge
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group replays/331-a-tool-check-on-the-live-link delivered: every member is delivered
Written with only what the node-engine had before its fix, so mesh-lab's prover can
lay it over the commit before (R331).
2026-10-08 17:10:29 +02:00
mesh-admin 824911dc50 Merge pull request 'Ask a declared tool check on the NATS link the node-engine holds (hq issue 331)' (#57) from fix/tool-check-asks-on-the-link into main 2026-10-08 15:06:51 +00:00
jochen 5e189c2fca Say a tool check's errors once and as they stand, and test the link on a real bus
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
The evidence read "asking it: asking <subject>": the words are now the link's own.
A deadline is said as the time the check gave it. A refusal is said only when the bus
refused this question, not an earlier one under a grant since widened. merge-check.sh
runs the tests that need a bus against a throwaway nats-server when the toolchain has
one, and says so when it does not (hq issue 331).
2026-10-08 17:01:15 +02:00
jochen 809ab4cf29 Ask a declared tool check on the NATS link the node-engine holds
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The queue asks a tool through a ToolBus assertion that only OverNATS satisfied; the
link the engine actually holds (natsLink) had no Ask, so every tool check failed as
"no link to the bus is open" while the link was up (hq issue 331). Give natsLink Ask,
assert at build time every optional interface the queue expects of it, and tell a link
that cannot ask apart from no link.
2026-10-08 16:44:01 +02:00
mesh-admin 6400e92d19 Merge pull request 'Report whether a battery powers the machine and whether anything measures its draw (hq ADR 0255)' (#56) from feat/the-bar-takes-blocks into main 2026-10-08 12:55:38 +00:00
jochen 41b3392789 Count the same meters the power module's sampler reads (hq ADR 0255)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
An APU's integrated graphics and Intel's second (MMIO) view of a package made a machine
report a power meter the sampler would not read, so its bar could only say not
measured.
2026-10-08 14:44:55 +02:00
jochen e41ab3a505 Report whether a battery powers the machine and whether anything measures its draw (hq ADR 0255)
The power module is on every machine and its bar blocks apply only where the
hardware is. Two capabilities read from the kernel let the controller compose them
there and nowhere else: battery (a Battery supply not scoped to a device) and
power-meter (a system battery, a powercap package counter, or a graphics device
reporting its power).
2026-10-08 14:44:55 +02:00
mesh-admin 800aabed83 Merge pull request 'Name the account whose own manager runs a unit (hq ADR 0254, issue 318)' (#55) from fix/318-a-wait-for-a-person-is-not-a-failure into main 2026-10-08 12:41:05 +00:00
jochen 9d8398206e Name the account whose own manager runs a unit, so a wait for a new login is not read as a fault (hq ADR 0254, issue 318)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/318-a-wait-for-a-person-is-not-a-failure delivered: every member is delivered
2026-10-08 13:41:46 +02:00
mesh-admin 9815796891 Merge pull request 'Give back only the groups the mesh added, and say when a new login is needed (hq ADR 0252, issue 247)' (#54) from feat/module-groups into main 2026-10-08 10:09:22 +00:00
jochen ab4ca44f98 Give back only the groups the mesh added, and say when a new login is needed (hq ADR 0252, issue 247)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/module-groups delivered: every member is delivered
A module puts the operator's account in a group by declaring the account with that group alone.
The node-engine now records each group it added, takes back only those when nothing declared still
asks for them, refuses a group the machine lacks before usermod runs, and states each such account
as its module's resource of kind account: relogin needed while the running session lacks the group.
2026-10-08 12:04:08 +02:00
mesh-admin ef8378a990 Merge pull request 'Say every failed unit, the module's and the machine's (hq issue 315)' (#53) from fix/315-a-failed-unit-is-a-condition into main 2026-10-08 09:40:27 +00:00
jochen 14e5d91c9a Say every failed unit, the module's and the machine's (hq issue 315)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/315-a-failed-unit-is-a-condition delivered: every member is delivered
Liveness judged only what a module runs long-lived, so a module whose
daemon is a package's unit started by D-Bus activation failed at every
start while its machine read healthy, and a degraded service manager was
said by nothing but the profile.

The engine now reads the failed units of the machine's manager and of
every account manager the declaration names, on the two-look rule, and
says whose each is: a declared service or process, a unit file the mesh
writes, or a package the mesh installs makes it that module's, said as
an unhealthy resource of kind unit; anything else is the machine's own,
said in the statement's new units field. It reads; it never acts.
2026-10-08 11:28:51 +02:00
mesh-admin f863adb741 Merge pull request 'A machine makes its tunnel key first and joins through the tunnel (hq ADR 0169)' (#52) from feat/a-machine-joins-through-the-tunnel into main 2026-10-08 08:24:16 +00:00
jochen 86cbebd10f Say controller in the installer's and the token's words, as the glossary does
mesh/delivery delivered
mesh/delivery-group group feat/a-machine-joins-through-the-tunnel delivered: every member is delivered
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
2026-10-08 01:46:21 +02:00
jochen 0e57186bf1 Place the bus's users before the enrolment's wait begins
The installer placed them between making the enrolment's deadline and
using it, so a failure returned without cancelling it, and go vet refused
the package.
2026-10-08 01:40:47 +02:00
jschoubben e30d838395 A joining machine dials the bus once the hub has answered its tunnel
Issuing the token sends the hub its new peer, and the hub applies it on
its own time; a bus dialled before then timed out naming the bus. The
first tunnel now waits for a handshake with the hub, and says so in the
tunnel's words when there is none (novox/hq ADR 0169).
2026-10-08 01:33:30 +02:00
jschoubben 0863695012 The installer lets the first node onto the bus it raised
At genesis the bus's users reach it in no declaration, because the
machine running it has not enrolled. The installer, which raised the
bus from its bundle, places the control plane's composed list beside it
and makes it re-read it: before the machine enrols, for the token's
account, and after, for the node's own (novox/hq issue 146).
2026-10-08 01:33:30 +02:00
jschoubben c38f21bb39 A machine makes its tunnel key first and joins through the tunnel
nox-mesh-host key makes the tunnel key, or reads the one made, and
prints its public half for the token to be issued for. enrol with a
token that carries a tunnel takes that key, refuses another, writes
mesh0 with the hub as its one peer and starts it, then reaches the bus
over it (novox/hq ADR 0169). Tokens without a tunnel enrol as before.
2026-10-08 01:33:30 +02:00
mesh-admin fcca8d9ce8 Merge pull request 'Judge the machine's own networking beside what its modules run (hq ADR 0241)' (#51) from feat/machine-network-health into main 2026-10-07 18:16:08 +00:00
jochen f8ef2de91d Hold the network statement's field names on the engine's side (hq ADR 0241)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/machine-network-health delivered: every member is delivered
2026-10-07 18:53:21 +02:00
jochen babd32cfc7 Take a tick a little early as the network look's tick
The liveness loop's ticker drifts; a look skipped for a few milliseconds
would wait a whole further tick.
2026-10-07 18:52:16 +02:00
jochen 039abeff69 Drill the network judge in a throwaway machine (hq ADR 0241)
The judge's tests run against files and fakes; the drill runs it against a
real resolver file and real resolvers, rewritten as the VPN client does,
and records the statements the controller's replay raises and clears from.
2026-10-07 18:51:45 +02:00
jschoubben 429ea42357 Judge the machine's own networking beside what its modules run (hq ADR 0241)
A VPN client rewrote the laptop's resolver file and every mesh name failed
while each module read healthy: nothing asked the machine. The engine now
looks every 30 s at the resolver file the uplink holder declared (naming
the program that rewrote it), the names through each listed resolver
(NXDOMAIN for a mesh name's IPv6 address is a finding, issue 262), the
tunnel's handshake with the hub, the bus and the default route; a part is
unhealthy on its second failing look, and the statement carries it.
2026-10-07 18:43:39 +02:00
mesh-admin 56e2ebec4b Merge pull request 'Judge how a module says it is ready, beside whether it stays up (hq ADR 0240, to-be 48 Phase B)' (#50) from feat/health-the-field into main 2026-10-07 16:28:34 +00:00
jochen 96bf9415aa Ask the merge check for this pull request: opening it announced nothing to the build seat
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/health-the-field delivered: every member is delivered
2026-10-07 16:17:40 +02:00
jochen bdd44154cc Judge how a module says it is ready, beside whether it stays up (hq ADR 0240, to-be 48 Phase B)
Liveness alone could not see a web application whose port was open and whose
program ran while every request hung for eleven hours (issue 145). A resource
now carries the `health` its module declared: the engine makes http and tcp
looks itself from the machine to the endpoint's published port, reads a unit's
readiness from the show it already makes, hands an exec command or the image's
own check to the runtime as the container's check with the declared timing and
reads its state from the inspect it already makes, and asks a module's tool on
its own node tools. Starting until the check passed, unhealthy once its looks
after the grace fail the declared number of times; never more looks than the
measured budget; nothing restarted. The statement says contract 2, which tells
the controller this engine may be sent the field.
2026-10-07 14:08:32 +02:00
mesh-admin 41f908b803 Merge pull request 'Let a planned maintenance window fail no apply (hq issue 291)' (#49) from fix/a-planned-window-fails-no-apply into main 2026-10-07 11:20:20 +00:00
jochen 3c1ac6aef2 Let a planned maintenance window fail no apply (hq issue 291)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
At 03:30 the store's collector held the registry still while the
node-engine's reconcile on that machine was fetching bundle blobs from
it: every archive failed 'connection refused' and the machine was held
until the next pass. Other machines can meet the same window.

A scheduled step now opens its window only once no apply is in flight
here (the apply lock is taken just to write the record, so a push still
never queues behind the window). An apply whose fetch the store does
not answer waits for a window open on its own machine to close, and
elsewhere retries with backoff within one bounded budget per apply,
well past the window's length; an answer such as 404 still fails at
once.
2026-10-07 13:11:03 +02:00
mesh-admin 038eff5ca0 Merge pull request 'Keep a replaced build reachable, and delete it only once no process runs from it (hq issue 289)' (#48) from fix/a-verb-survives-the-handover into main 2026-10-07 00:55:23 +00:00
jochen 3a117c2d2b Keep a replaced build reachable, and delete it only once no process runs from it (hq issue 289)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The witness moved a running controller's build into a 0700 directory and deleted it
on proof, while the old process could still be serving. Its directories are now
0711, and a build without a reader is retired and swept once /proc shows nothing
runs from it.
2026-10-07 02:45:08 +02:00
mesh-admin 03031a46dd Merge pull request 'Judge whether what a module runs stays up, and say it (hq ADR 0240, to-be 48 Phase A)' (#47) from feat/a-module-says-how-it-is-healthy into main 2026-10-07 00:39:39 +00:00
jochen 1fc1e74cc3 Judge whether what a module runs stays up, and say it (hq ADR 0240, to-be 48 Phase A)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/a-module-says-how-it-is-healthy delivered: every member is delivered
A container that crash-looped after its compose applied passed every check the
gate had: nothing looked at what a module runs. The node-engine now judges every
long-running resource on every look — one read of the runtime, one per service
manager — keeps the restarts it counts across recreates and its own restarts,
and says the state in every report and as an event on change, again every minute
while not healthy. It reads only; nothing is restarted for being unhealthy.
2026-10-07 02:28:15 +02:00
mesh-admin a338c2e581 Merge pull request 'Lay out the publishing test as every gofmt agrees (hq issue 286)' (#46) from fix/gofmt-as-the-toolchain into main 2026-10-07 00:27:48 +00:00
jochen bd30534ab1 Check again, judged by the controller with the gate's fix (novox/hq issue 285)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-07 02:06:51 +02:00
jochen 2a4b521e1b Cite the hq issues by the numbers they were given: 285, 286, 287
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (0 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-07 02:00:13 +02:00
jochen ed43d65bf3 Lay out the publishing test's return as every gofmt agrees, so the build seat's check passes
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (0 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The toolchain the build seat runs merge-check.sh in (Go 1.26) and a newer local Go format a return of
several multi-line composite literals differently; the seat's is the one that judges, and it failed every
pull request on this file (novox/hq issue 283).
2026-10-07 01:29:06 +02:00
mesh-admin 971881d58d Merge pull request 'Let the controller ask the delivery's owner stalled and close (hq ADR 0239)' (#45) from feat/mesh-delivery-waits-said into main 2026-10-06 22:30:57 +00:00
jochen b196cabd8f Let the controller ask the delivery's owner stalled and close (hq ADR 0239)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (0 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: internal/bootstrap/publish_test.go
mesh/delivery delivered
mesh/delivery-group group feat/mesh-delivery-waits-said delivered: every member is delivered
The controller's self-check reads mesh-delivery's stalled and healer H2 calls
its close; a controller raised from genesis must be granted both.
2026-10-07 00:14:19 +02:00
mesh-admin 880e7461f9 Merge pull request 'Carry plan-moved in the installer's first user list (hq ADR 0239)' (#44) from feat/mesh-delivery into main 2026-10-06 22:07:19 +00:00
jochen 94e49c6b2d Carry plan-moved in the installer's first user list (hq ADR 0239)
mesh/merge-gate pass: every machine composes with the change as it did without (0 of 4 compose)
mesh/delivery delivered
mesh/delivery-group group feat/mesh-delivery delivered: every member is delivered
The controller says every walk it keeps as plan-moved; a controller raised
from genesis must be allowed to say it from its first start.
2026-10-06 23:59:19 +02:00
mesh-admin ad812a5888 Merge pull request 'Leave the gate to the build seat; merge-check.sh checks the node-engine's own code (hq ADR 0238)' (#43) from feat/the-graph-decides-what-is-checked into main
mesh/delivery delivered
2026-10-06 21:00:35 +00:00
jochen f823ade868 Leave the gate to the build seat; check the node-engine's own code here (hq ADR 0237)
The build seat now runs the gate itself, judging a node-engine change with the change's
validator in the running controller because the module graph builds mesh-host from here.
This script is the repository's own layer (mesh/repo-check): format, vet, and the suite
under the race detector when the toolchain can.
2026-10-06 21:51:49 +02:00
mesh-admin f39e282fed Merge pull request 'Phase 5: check the node-engine's changes against every machine before they merge (hq ADR 0237)' (#42) from feat/merge-gate into main
mesh/delivery delivered
2026-10-06 19:19:20 +00:00
jochen 2fb7c91eaf Check the node-engine's changes against every machine before they merge (hq ADR 0237)
merge-check.sh runs the suite, then builds the controller the mesh runs with this change's
validator in place of the one it vendors and runs the merge gate: a change to the node-engine
that would refuse what the mesh sends today fails its pull request, naming the machine. The
installer's first user list lets the controller say a check's verdict and hear a pull request's
head, as the controller now composes it.
2026-10-06 21:16:33 +02:00
mesh-admin 7a6f9218a0 Merge pull request 'Core upgrades that roll back: the host side (hq to-be 45 Phase 4, ADR 0227 rule 8)' (#40) from feat/core-upgrades-roll-back into main
mesh/delivery delivered
2026-10-06 18:00:07 +00:00
mesh-admin ad39988bc6 Merge pull request 'Genesis grants for the gate's event and the bus step's snapshot (hq ADR 0236)' (#41) from feat/core-upgrades-that-roll-back into main
mesh/delivery delivered
2026-10-06 17:15:07 +00:00
jochen 7b450ab371 Carry the controller's grants for the gate's event and the bus step's snapshot (hq ADR 0236)
The controller says a rollback as rolled-back and asks the bus machine's backup holder
for a snapshot before the planned bus step; the installer's first user list must grant
both, or a fresh mesh's controller is refused the first time it uses them.
2026-10-06 18:34:41 +02:00
jochen 0c405b70cc Roll a core build back by a witness that is not the new build (hq to-be 45 Phase 4)
The launcher trusted a counter only a by-hand reconcile ever cleared and a
known-good nothing in the daemon wrote, so no machine could roll its host back;
the controller and the node tools were replaced in place with nothing kept.

- The launcher runs a delivered host that is not known-good on trial: one that
  crashes, stops for nothing, or does not report within ten minutes goes back
  to known-good, once per version, recorded in rolled-back. The host proves
  itself when the mesh takes a report under its own build, says every standing
  verdict on its reports, never stands aside for a rolled-back version, and
  restarts its service once when its launcher was replaced on disk.
- The engine keeps the controller's and the node tools' previous build beside
  the new one and judges the new one: the lease taken by the controller it
  started (read-only direct get of mesh-controller_lease/holder), or this
  machine's runtime answering $SRV.PING.node-tools.<node>, within sixty seconds
  of time it could ask. Not healthy: the previous restored, once, said. Proved:
  the previous deleted. A build declared not-reversible is never rolled back.
- Retire never removes a version newer than the running one.
2026-10-06 18:23:56 +02:00
mesh-admin 8d853791b2 Merge pull request 'Genesis grant for D13; a kept directory is never 'removed by hand' (hq ADR 0233)' (#39) from feat/a-module-declares-the-data-it-holds into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 15:00:52 +00:00
jochen 7e9ae9f07e Carry the controller's new self-check grant; stop telling people to delete kept data (hq ADR 0233)
The installer's first user list must match the controller's composed grant, which now reads every
machine's backup holder; and a kept directory's report pointed at the one act that loses data.
2026-10-06 16:47:49 +02:00
mesh-admin 2e884438f9 Merge pull request 'Genesis assigns mesh-wireguard, not the retired networking bundle (hq ADR 0226)' (#31) from feat/genesis-assigns-the-private-network into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 13:04:28 +00:00
jochen 98885e953e Genesis assigns mesh-wireguard, not the retired networking bundle (hq ADR 0226)
A controller that no longer ships networking refuses it, which would stop genesis where the hub
joins the private network.
2026-10-06 14:59:26 +02:00
mesh-admin f274b6effe Merge pull request 'Genesis lock: let the controller hear provisioner.retirement (hq ADR 0230)' (#38) from feat/retired-consumers into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 12:46:32 +00:00
jochen 240b903a04 Let the genesis controller hear what providers retire (hq ADR 0230)
The controller now derives a subscription to provisioner.retirement; the first
user list the installer carries must match it, or the genesis controller is
refused the subject its composition expects.
2026-10-06 14:45:56 +02:00
mesh-admin 280d3b2e1e Merge pull request 'Say again what was last applied when the mesh asks (hq to-be 45 Phase 3, the report verb)' (#37) from feat/a-core-that-cannot-fail-silently-phase-3 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 12:29:14 +00:00
jochen 6e1bcdbde4 Say again what was last applied when the mesh asks (hq to-be 45 Phase 3, the report verb)
The controller's healer H1 answers a send that went unreported by asking the
machine first: a report lost on its way (issue 264) needs no second send. The
node-engine now hears mesh.node.<self>.ask.report on core NATS and enqueues a
reconcile whose account is said whether or not it is news; a delivery waiting
meanwhile is applied and reported instead. The answer is an ordinary report on
its own subject, so the node publishes nothing new and answers nobody's inbox.

The genesis lock grants the controller the healer-acted seat event, which it
now composes; the genesis test in mesh-controller holds the two equal.
2026-10-06 14:05:02 +02:00
mesh-admin b2808549ee Merge pull request 'Grant the genesis controller its lease and secret-replaced, not mesh.control.> (hq to-be 45 Phase 2)' (#36) from feat/a-core-that-cannot-fail-silently-phase-2-grants into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 10:30:28 +00:00
jochen bbe74f3fc2 Grant the genesis controller its lease and the secret-replaced event, and not what the machines say (hq to-be 45 Phase 2)
The controller now composes a publish grant for its lease bucket
($KV.mesh-controller_lease.>), which it must write before it acts, and for
the seat event secret-replaced (hq ADR 0228, mesh-controller #82); and it no
longer publishes mesh.control.>: a machine's report has one writer, its
node-engine, and the writers table refuses a second at composition. The
installer's first user list must say what the controller derives, or a new
mesh's first controller is refused its lease and serves without one.
2026-10-06 12:25:12 +02:00
mesh-admin 3e80b7ae32 Merge pull request 'Apply through one queue, and order what is applied and reported (hq to-be 45 Phase 2)' (#35) from feat/a-core-that-cannot-fail-silently-phase-2 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 09:55:19 +00:00
jochen 31804bd8c2 Apply through one queue, and order what is applied and reported (hq to-be 45 Phase 2)
A delivery and the five-minute reconcile were two paths that applied, ordered only by a lock, and
each order it allowed was met live (issues 257, 261, 267). Now both only enqueue: one worker takes
the newest declaration held when it starts, applies it once and makes one report, and reports leave
in the order they are made.

A declaration may carry the controller's lease epoch beside its sequence; one older than what this
node applied is refused before anything is touched, counted, logged and reported. A report carries
the declaration's epoch and sequence and the host's own report sequence, kept on disk so it goes on
increasing across restarts and self-updates. Without an epoch, today's behaviour stands.
2026-10-06 11:54:10 +02:00
mesh-admin d7d93f57c5 Merge pull request 'Grant the genesis controller the self-check's ban-list question (hq to-be 45 Phase 1, D8)' (#34) from fix/phase-1-d8-grant-and-d10-version into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 08:45:16 +00:00
jochen b9774ea426 Grant the genesis controller the self-check's ban-list question (hq to-be 45 Phase 1, D8)
The controller now composes a publish grant for
mesh.seat.node-intrusion-prevention.tool.banned.*, which D8 asks every
machine; the installer's first user list must say what the controller
derives, or a new mesh's first self-check is refused it.
2026-10-06 10:44:39 +02:00
mesh-admin 1545b00a87 Merge pull request 'Say the heartbeat's interval, export the host's validator, grant the genesis controller Phase 1 (hq to-be 45)' (#33) from feat/a-core-that-cannot-fail-silently-phase-1 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 08:24:39 +00:00
jochen 6953b5bafd Say the heartbeat's interval, export the host's validator, grant the genesis controller Phase 1 (hq to-be 45)
The controller's watchdog of a machine's heartbeat (S1) is bound to three
of its intervals, and a bound the controller guessed would not move when
the interval does: the heartbeat now carries interval_seconds.

Its self-check (D1) must judge every composed declaration as the host
does, and a second validator written from the host's rules would drift
from them: the host's own parsing is exported, unchanged, as
github.com/novox/mesh-host/validate.

The installer's first user list grants what the controller now composes
for itself: its condition buckets, the condition and doctor-heartbeat
events, the bus's two consumer advisories and $SRV.INFO — or the first
controller would be refused them until the broker's machine is pushed.
2026-10-06 10:18:54 +02:00
mesh-admin 93efe41dc9 Merge pull request 'Grant the genesis controller its buckets and cancelled sets (hq to-be 45 Phase 0, issue 269)' (#32) from feat/a-core-that-cannot-fail-silently-phase-0 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 07:13:38 +00:00
jochen 1e463def6b Grant the genesis controller its work queues' cancelled sets (hq issue 269)
The controller now derives the cancelled sets' subjects; the installer's
first user list says the same, or a cancel on a new mesh times out.
2026-10-06 03:01:19 +02:00
jochen a654fa768d Grant the genesis controller its own buckets' subjects (hq to-be 45 Phase 0)
The controller keeps its calls and the hand-act log in two key-value buckets
of its own, and composes a grant to write them. The installer's first user
list must say what the controller derives, or the first controller writes
nothing until the broker's machine is pushed.
2026-10-06 02:59:51 +02:00
mesh-admin 7f98478d6d Merge pull request 'Never say a reconcile's report after a newer apply's (hq issue 267)' (#30) from fix/stale-report-overwrites into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 23:47:10 +00:00
jochen a79972577c Never say a reconcile's report after a newer apply's (hq issue 267)
A reconcile that held the machine to the kept declaration just before a
delivery arrived queued its report while the delivery's apply waited for
it; the link published the apply's report and then the reconcile's, so the
mesh's last word from the machine named the older declaration and the
release plan waited on a report it had already been given. The link now
sets aside an unasked report about a declaration other than the one it
has applied since.
2026-10-06 01:45:34 +02:00
mesh-admin 64defd47c7 Merge pull request 'Say an apply's report even when the apply ends the link (hq issue 264)' (#29) from fix/report-lost-at-self-update into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 22:31:47 +00:00
jochen f62ee0bc75 Say an apply's report even when the apply ends the link
A host that delivered its own successor stood aside before the report of
that apply was published, so it failed with 'context canceled' and the
release plan waited for a report that never came (novox/hq issue 264).
Reports are now published on a context the stand-aside does not cancel,
and the last apply's report is kept until the broker takes it and said
again on the next link, so a crash between apply and report is covered too.
2026-10-06 00:30:15 +02:00
mesh-admin 07430240bb Merge pull request 'Let the genesis controller hear provider standings and its seats' verbs' (#28) from feat/controller-hears-provider-standing into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 22:20:46 +00:00
jochen 6f86b484d0 Let the genesis controller hear provider standings and its seats' verbs
The carried user list must equal what the controller derives (its test reads
this file): add the provisioner.failing/recovered subscriptions (hq ADR 0224)
and the build seats' tool publishes it already derived (hq ADR 0219).
2026-10-06 00:13:23 +02:00
mesh-admin 24bada7a6f Merge pull request 'Hand a whole file to its new owner instead of removing it first (hq ADR 0223)' (#27) from files-forget-when-held into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 21:50:48 +00:00
jochen 762da9e05e Hand a whole file to its new owner instead of removing it first (hq ADR 0223)
The resolver file moves from resolv-conf to the uplink's holder in one apply.
Orphans go first, so the file was deleted or given back its pre-mesh original
until the new owner's turn came. Now the old record is forgotten once the new
one is recorded at the same path, and the kept original goes with it.
2026-10-05 23:35:40 +02:00
mesh-admin a192bccf62 Merge pull request 'Leave a unit alone when another declared service still holds it (hq issue 190)' (#26) from test/190-into-json-member-handover into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 20:31:33 +00:00
jochen f0f5873202 Keep unitKey's comment on unitKey 2026-10-05 22:27:32 +02:00
jochen c9001abdb4 Leave a unit alone when another declared service still holds it (hq issue 190)
Removing one record of a unit gave back what that record found, even while another declared
service holds the unit: when the private network's docker.service record goes and the docker
module's stays, a record that found the runtime stopped would stop it, and every container with
it, only for the docker module to start it again in the same apply. The record is forgotten
instead, and the plan says so. A test pins the registry member moving from the network's record
to the docker module's in one apply without leaving the list.
2026-10-05 22:21:21 +02:00
mesh-admin a566add815 Merge pull request 'A service is restarted only after every file it names is written (hq issue 260)' (#25) from fix/a-service-restarts-after-every-file-it-reads into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 19:57:22 +00:00
jochen 101f0c0d61 Restart a service only after every file it names is written (hq issue 260)
A service was restarted where it was declared, so one declared ahead of a
file in its restart-on was restarted before that file existed (the
resolver's zones file, a fact appended after its module's resources), and
a later change to such a file was never acted on. Each service is now
applied right after the last resource it names under restart-on or
reload-on; nothing else moves.
2026-10-05 21:55:47 +02:00
mesh-admin 89a7796afe Merge pull request 'A reconcile applies what was kept when its turn comes (hq issues 257, 261)' (#24) from fix/a-reconcile-applies-what-was-last-kept-when-its-turn-comes into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 19:43:51 +00:00
jochen ea6fe09e01 A reconcile applies what was kept when its turn comes, not when its timer fired (hq issues 257, 261)
The five-minute reconcile read the kept declaration and then waited for the link's apply; the
link keeps a declaration only once its apply ends, so the reconcile applied the older one over
it. A module assigned a moment earlier was given back, and the report named a declaration the
mesh no longer recorded as sent, which held a plan at its first machine.
2026-10-05 21:43:47 +02:00
mesh-admin 7063b183af Merge pull request 'An apply leaves a container a maintenance window holds still (hq issue 224)' (#23) from fix/an-apply-leaves-a-held-container-held into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 16:27:55 +00:00
jochen f08681f225 An apply leaves a container a maintenance window holds still (hq issue 224)
A while-stopped step stops its module's containers, and an apply arriving
mid-window read the stopped server as broken and recreated it running under
the step - for the store's collector, a registry taking an upload the sweep
then deletes. The scheduler now records each window under the node's state
directory before its first stop and erases it after its last start, so every
apply on the machine (daemon, reconcile by hand, installer) reports a held
container held-still and leaves it for the first apply after the window.
A window whose host died, or past six hours, holds nothing; the report says
which windows are open.
2026-10-05 18:27:04 +02:00
jschoubben 238252e152 Merge pull request 'An undeclared file is deleted only when it holds what the host wrote; otherwise moved aside (hq issue 241)' (#22) from fix/a-file-that-is-not-only-the-meshs-is-moved-aside into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 23:57:56 +00:00
jschoubben b774a0b4f9 An undeclared file is deleted only when it holds what the host wrote; otherwise moved aside (hq issue 241) 2026-10-05 01:30:42 +02:00
mesh-admin 89fce1dae3 Merge pull request 'A file the host wrote over is given back when undeclared (hq ADR 0102, 0118)' (#92) from fix/a-file-written-over-is-given-back into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 10:54:50 +00:00
jochen 3e11d720b4 A file written over is given its kept original back when undeclared (hq ADR 0118, 0102)
The host keeps the original of a file before writing over it (ADR 0102), but
removing the file's record deleted the file and never put the original back,
although ADR 0118 and the comment on meshMadeUnits say it does. A module writing
/etc/pacman.conf, logrotate.conf, locale.conf or vconsole.conf whole would, once
unassigned, leave the machine without the file.

removeWhole now decides, in order: no kept original (the mesh made it) is
removed as before; a file gone since is not brought back; a file changed since
the mesh last wrote it is left as it stands, as a block or JSON write-into stays
the machine's; an unreadable kept copy leaves the mesh's file in place. Otherwise
the original goes back atomically with the mode and owner it was found with,
now recorded beside Kept, and the outcome is "restored". None of it is fatal.
The plan says "restore" for such a file.

A kept original is carried only for the path it was kept from, and a file whose
path moved keeps the original at its new path first, so a moved file is never
given another path's original.
2026-10-04 12:54:16 +02:00
mesh-admin 429672b7ff Merge pull request 'User-scoped units (hq ADR 0177, to-be 38 WP6), with lingering and a manager that is not running' (#91) from feat/user-scoped-units into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 10:43:17 +00:00
jochen d5c365cb2b A user unit waits for its account's manager, and lingering is the account's (hq ADR 0177)
An account's manager runs only while it is logged in or lingers. A user-scoped unit
whose manager is not running is now "waiting" rather than failed, its record kept as
it was; its removal is never fatal (kept recorded, retried) and an account that is
gone is forgotten. Whether the manager runs is asked of user@<uid>.service in the
machine's manager: asking the account's own, through --machine, logs it in.

The user shape gains `linger`, set with loginctl, read back from logind's record,
and given back on removal like the shell. Unit files the mesh writes under
~/.config/systemd/user or /etc/systemd/user make that unit the mesh's, and made,
holds and found units are keyed by manager and name, so an account's unit and the
machine's of one name are two units. A service moved between managers gives the old
one back through the manager it was in. OpenRC refuses both.
2026-10-04 12:41:32 +02:00
jochen 84540e709a A unit may be user-scoped: applied through the account's own manager (hq ADR 0177)
A workstation's per-user daemons — a window manager's reload watcher, an
audio mask, a memory guard — are units in the operator account's own service
manager, and until now had no form the mesh could send (to-be 29). The
`service` shape gains `scope` ("system", the default, or "user") and `user`
(the account, named ${machine:account} by a module); a user-scoped unit
without an account, or a system unit naming one, is refused at parse.

The host reaches the account's manager as `systemctl --user --machine=<account>@`
from its own process: no environment to forge, no user to switch to. Done on
the runner rather than per system, since every system's reading of a unit
already goes through systemctl. Apply, reflect-only and removal all go through
the same manager, and the applied record carries scope and user so removal
gives the unit back to the manager it came from. It answers only while that
manager runs — a login, or lingering enabled for the account; declaring
lingering is a follow-up.

Tests: a user-scoped unit is started and enabled in the account's manager and
recorded with its scope; a system unit never sees --user; the validation of
scope and user.
2026-10-04 12:31:24 +02:00
mesh-admin 6431848342 Merge pull request 'An archive can be undeclared, and undeclaring one no longer stops the apply (hq issue 162)' (#90) from fix/162-an-archive-can-be-undeclared into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 10:22:39 +00:00
jochen 8470dbd8e8 An archive can be undeclared, and undeclaring one no longer stops the apply (hq issue 162)
An archive had no removal, so an unassigned one failed as an orphan and
aborted every apply after: a module with tools could not be unassigned,
and a race between two pushes froze a machine against every change.

The record now keeps what an archive unpacked: its files, the
directories the host made inside its path, whether the host made the
path itself, and the parents it made to reach it. Removal takes exactly
that away, directories only once empty, never one that was there
before; a directory that is the host's alone is renamed aside first so a
reader sees the whole bundle or none of it. Whatever cannot be removed
is said and forgotten, never fatal.

A directory found before the archive is no longer swapped away with
what was in it: the archive is moved in file by file, and one that would
write over a file the mesh did not put there is refused before anything
moves. A record from before this change learns its files from the
archive's bytes on the next apply; one already orphaned is left in
place, said and forgotten. A former target is still left in place: the
version before is what a rollback starts (ADR 0141).
2026-10-04 12:20:41 +02:00
mesh-admin 64b421b6e1 Merge pull request 'A login the mesh set is given back, and directories made inside a home are its account's (hq issue 228, to-be 41 WP1)' (#89) from feat/the-shell-and-its-environment into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 08:49:30 +00:00
jochen f2eda240ec Cite hq issue 228: 225 was taken on main while this branch was open 2026-10-04 10:30:57 +02:00
jochen 2a5f4c8270 Parents the host makes inside an owner's home are the owner's (hq ADR 0182, to-be 41)
A file or archive placed under a fresh account's home with an owner left
the parents it created, such as ~/.config or ~/.local/share, owned by
root, so the person's own programs could not write there. Parents that
already existed, and any outside the owner's home, are left as before.
2026-10-04 04:07:40 +02:00
jochen 5d5dccdd55 A login the mesh set is given back, and undeclaring one no longer stops the apply (hq issue 225)
A user had no removal, so an undeclared one failed as an orphan and
aborted every apply after. Removal now keeps the account, gives back
the shell recorded when the mesh first changed it if it is still the
mesh's and still usable, and says why otherwise (hq ADR 0176 §2).
A shell is refused before it is set unless it is executable and listed
in /etc/shells, since usermod succeeds on a missing one.
2026-10-04 03:57:34 +02:00
mesh-admin 27fb22fddb Merge pull request 'A scheduled step may hold its module's own containers still (hq ADR 0189, issue 108)' (#77) from feat/the-store-keeps-what-the-records-name into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 01:39:07 +00:00
jschoubben b602b223e1 A changed maintenance window is a changed spec (hq ADR 0189)
containerSpec says a changed cadence moves the marker so the install is
reported updated and re-established. Which containers are held still is the
same kind of statement, and a declaration that changed it while the machine
reported no change would be a machine quietly holding yesterday's containers.
2026-10-04 03:27:32 +02:00
jschoubben e8c4824ae2 A scheduled step may hold its module's own containers still (hq ADR 0189)
while-stopped names resource ids of the same module's containers; the host
stops them before the run and starts them again after it, in reverse order,
whatever the step did. The restart is deferred before the first stop and runs
on its own context, because the one real risk of this field is a window that
never closes.

Scheduled steps only: at apply the declaration is applied in order and a
run-once step already gates what follows.
2026-10-04 02:34:33 +02:00
mesh-admin 8c2e76f4c1 Merge pull request 'Read getent's exit code, not its wording (hq issue 213)' (#88) from fix/getent-not-found-is-an-answer into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 00:11:48 +00:00
jochen 8280a82ef8 Read getent's exit code, not its wording (novox/hq issue 213)
A user that does not exist yet was matched as Go's 'exit status 2', while the host's runner says
'getent exited 2', so it read as a user database that did not answer: the controller's account was
never created and the handover to its process stopped there. The runner keeps the exit underneath
its words, and a caller asks the code.
2026-10-04 02:11:43 +02:00
mesh-admin 2d5e76434b Merge pull request 'Raise a process-form controller at genesis as the container it replaces (hq issue 223)' (#87) from fix/issue-223-genesis-pivots-to-the-controllers-container into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 23:50:48 +00:00
jochen d9ea387680 Raise a process-form controller at genesis as the container it replaces (hq issue 223)
The controller's manifest now declares a Go bundle the host runs as a
process (novox/hq issue 213). Genesis cannot run that: the bundle is
fetched from the artifact store and compiled in a toolchain, and the mesh
makes both long after the controller. The builder, asked to build the
manifest at genesis, refuses for lack of the Go toolchain. So genesis
raises the controller as before, as a container, and the first push hands
it over to the process through `replaces` (issue 223, option b).

- Step 3 clones the controller at the commit with the carried builder's
  git and reads its manifest. In the image form (an older controller) it
  builds through the builder as before. In the process form it builds the
  repository's own Dockerfile and hands step 9 a manifest of its own
  shape: the process becomes a container with the id the process
  `replaces`, the image genesis built, host network, and every host path
  the process's env names mounted at that same path read-only. Secrets
  belong to the image's user (65534) until the process's account takes
  them over. `prepares` is dropped: the temporary controller from the
  same commit already migrated the stores, and a pinned image is
  nothing the controller can derive a step from.
- Steps 4 to 9 are unchanged: they take the manifest as they did.
- apply.ForTests lets the bootstrap's test apply a process.

The first composed declaration from the process manifest names
`mesh-controller.server`, which is what the host recorded for the genesis
container, so the first apply hands over and leaves one controller.
2026-10-04 01:47:33 +02:00
mesh-admin 99ad145bec Merge pull request 'Hand a replaced resource over to the process that replaces it (hq issue 213)' (#86) from fix/issue-213-the-controller-is-a-process into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 23:13:52 +00:00
jochen 2ff3b50a84 Hand a replaced resource over to the process that replaces it (hq issue 213)
The controller moves from a container to a process on the one machine
that runs it (novox/hq issue 213). Every orphan is removed before anything
is applied, so the container would go first and nothing would answer the
mesh's verbs while the process was fetched, unpacked and started — and
never again, if it did not start.

- a process may say what it `replaces`: resources the declaration no
  longer declares. Such an orphan is kept through the up-front sweep and
  removed right after the process applied and is up: active and running
  at two looks ten seconds apart, the same main process, no restart in
  between (stricter than ADR 0184's second look, which reads a unit
  waiting to restart as running). If the process failed, was skipped
  behind its module's step, or is not running, the orphan stays running
  and recorded, reported kept, and the next apply hands it over.
  Refused: naming something still declared, itself, an empty id, one
  thing named by two processes, and `replaces` on a step or a schedule.
- beyond #85's oneshot unit for a step: a step written ./name runs its
  own bundle's binary (tested), and is started, never enabled.
- a run-once process that fails gates its module, as a run-once
  container already did, so a version whose preparation failed is not
  started.
- an unchanged run-once process is not run again, and an unchanged
  scheduled one is kept up by its timer: both were "a daemon that had
  stopped" and were started on every apply.
2026-10-04 01:01:52 +02:00
mesh-admin a24670d77c Merge pull request 'Run a run-once process as its oneshot unit (design 38 WP4c)' (#85) from fix/a-run-once-step-runs-where-and-as-declared into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 22:29:10 +00:00
jochen 5fc4052a2b Run a run-once process as its oneshot unit (novox/hq design 38 WP4c)
A step was run directly: in the host's own working directory, without its env, env files or
user. A module step moved out of its container (node bootstrap/index.js) could find neither its
code nor its words. A oneshot unit carries all four as a daemon's does, and starting it waits.
2026-10-04 00:29:03 +02:00
mesh-admin a7bf0f6e39 Merge pull request 'Make an unpacked archive exactly the archive (hq issue 220)' (#84) from fix/issue-220-a-bundle-on-disk-is-exactly-the-artifact into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 22:17:57 +00:00
jochen 1c29168309 Make an unpacked archive exactly the archive (novox/hq issue 220)
Unpacked over the previous tree, a file the new archive no longer has stayed: a bundle rebuilt as
one file per entrypoint kept the old package directory. Unpack into a fresh directory and swap it
in, so a refused archive also leaves the old tree whole.
2026-10-04 00:17:50 +02:00
mesh-admin 483e02e7ef Merge pull request 'The installer's first user list lets the controller answer $SRV.STATS (hq ADR 0197)' (#83) from fix/0197-the-controller-answers-stats-too into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 21:11:25 +00:00
jochen 0650df9414 The installer's first user list lets the controller answer $SRV.STATS (hq ADR 0197) 2026-10-03 22:47:41 +02:00
mesh-admin 47fb924947 Merge pull request 'The installer's first user list lets the controller answer discovery for its seat (hq ADR 0197)' (#82) from feat/0197-the-controller-announces-itself into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 20:11:12 +00:00
jochen 09f84b451d The installer's first user list lets the controller answer discovery for its seat (hq ADR 0197)
The controller announces the mesh-controller seat on $SRV.PING/$SRV.INFO from genesis; the
carried account is the composed one, which the controller's test compares.
2026-10-03 22:11:01 +02:00
mesh-admin 77ae5320ad Merge pull request 'A process runs its own bundle's binary, written ./name (hq ADR 0193)' (#81) from feat/a-process-runs-its-own-binary into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 19:14:04 +00:00
jochen b7ede83c8c A process runs its own bundle's binary, written ./name (hq ADR 0193)
A bundle compiled to a binary runs itself, but only the host knows where it unpacked it, and the
service manager takes no path relative to the working directory. A command written ./name is that
file in the process's own unpacked bundle, made absolute in the unit.
2026-10-03 21:13:55 +02:00
mesh-admin 195fc63f6a Merge pull request 'An unchanged process keeps its record, so the node's runtime is not re-created every other cycle (hq issue 210)' (#80) from fix/issue-210-a-process-is-unchanged-the-second-time into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 11:41:34 +00:00
jochen 482d20737f An unchanged process keeps its record, so the node's runtime is not re-created every other cycle (hq issue 210)
The process applier's unchanged path returned an outcome that said nothing about what was
written; the loop recorded it like any other, erasing the digest. The next cycle found no
record and re-created the daemon, the one after found a record again, and so on: the
node's runtime restarted every ten minutes on every machine since it arrived. The outcome
now carries the digest forward, as a file's does. The test applies one process three
times and asserts the record survives an unchanged apply and no restart is asked.
2026-10-03 13:41:14 +02:00
mesh-admin 09614d94c8 Merge pull request 'A failed install says whether the package database is stale, how old it is, and what fixes it (hq issue 205)' (#79) from fix/issue-205 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 09:44:43 +00:00
jochen c9b963f8b9 A failed install says whether the package database is stale, how old it is, and what fixes it (hq issue 205)
pacman writes its errors to stderr, which the runner folds into the error rather than the output;
the stale-index classifier read the output alone and never saw a single 'failed retrieving file',
so a ten-week-old database on the control node reported as a wall of 404s from the mirrors. The
classifier now reads everything pacman said, knows a failed signature as the same staleness, tells a
mirror outage with a fresh database apart from it, and names the database's date and age beside the
remedy: a full upgrade by the operator, never a one-package sync, which on this distribution is a
partial upgrade. Whose job keeping the database current is stays issue 205's question.
2026-10-03 03:54:17 +02:00
mesh-admin 5a6e51f975 Merge pull request 'The first user list names the build role as node-build-agent (hq ADR 0190)' (#78) from fix/the-build-role-is-node-build-agent into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 00:53:44 +00:00
jochen 3c96683d98 The first user list carries both build seats during the handover (hq ADR 0190)
Until the retired mesh-build-machine row is dropped the controller asks and hears both roles, so
the genesis list grants both; the controller's own test of this list says so.
2026-10-03 02:53:12 +02:00
jochen 8c4e33c165 The first user list names the build role as node-build-agent (hq ADR 0190)
The controller composes the build seat's subjects as node-build-agent's now; the installer's
genesis list must say the same, or the controller's own test of that list disagrees with it.
2026-10-02 22:37:08 +02:00
mesh-admin 2a332b0e6e Merge pull request 'The mesh's own ban chain is a ban wherever it hangs (hq ADR 0186)' (#76) from fix/a-ban-list-never-holds-a-neighbour into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 16:43:09 +00:00
jschoubben b94e0f9a77 The mesh's own ban chain is a ban wherever it hangs; the front end's record is cited as 0180 (hq ADR 0186)
The legacy reader required every path into a chain of refusals to come from a built-in whose policy
accepts. The home server's ban chain hangs off the container runtime's user chain, whose forward
policy the runtime set to DROP, so the machine reported the mesh's own intrusion prevention as a
rule set the mesh did not write. A chain is a ban when every refusal names its sources and the
chain accepts nothing — the rule the nftables side already used. A chain that accepts anything is
still not a ban. Fixture captured from the machine. The citations for the uninstalled front end move
to ADR 0180, which another session's renumber had left pointing at an unrelated record.
2026-10-02 18:42:16 +02:00
mesh-admin b840ce0a77 Merge pull request 'A service the mesh asked to run is still running a moment later (hq ADR 0184)' (#75) from fix/a-service-asked-to-run-is-still-running into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 16:25:25 +00:00
jschoubben 286865dfa7 A service the mesh asked to run is still running a moment later (hq ADR 0184)
The read-back raced the failure: a service manager returns when it has started the process, and a
daemon that refuses its configuration exits a fraction of a second later, so one look saw it alive.
fail2ban took 221ms on the control node and the apply reported "restarted" onto a dead daemon while
both public machines kept no bans at all. The host looks again, after that moment. A unit still
starting is accepted at both looks; a service asked to stop is not waited on.
2026-10-02 18:16:24 +02:00
mesh-admin ca7c4a5915 Merge pull request 'A container may log to the journal (hq ADR 0179)' (#73) from feat/the-intrusion-seat-serves-its-verbs into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 15:04:02 +00:00
jschoubben b30d9c5b5a A container may log to the journal (hq ADR 0179)
A jail reads a log; a container's output went to a file of the runtime's own under a path that
changes on recreate, so no jail could read a container's service. logging: journald runs the
container with the journal as its driver, named in the spec so moving it recreates it; any other
place is refused.
2026-10-02 17:02:49 +02:00
mesh-admin 5b73e04192 Merge pull request 'A package may be declared absent, and an uninstalled front end is retired for good (hq ADR 0175)' (#71) from feat/the-found-front-end-is-uninstalled into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 14:29:17 +00:00
jschoubben f57386cdea A package may be declared absent, and an uninstalled front end is retired for good (hq ADR 0175)
absent: true on a package has the host remove it through the machine's own
package manager when it is installed and leave alone a machine that never had
it; read back either way. Undeclaring a package still removes nothing. A
found firewall whose command is gone is recorded as removed, said once, and
asked nothing of.
2026-10-02 16:28:27 +02:00
mesh-admin f92dd3e286 Merge pull request 'Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered' (#70) from fix/adr-0170-cited into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 12:53:09 +00:00
jschoubben cdbe3ab0a4 Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered after a collision on hq main 2026-10-02 14:52:20 +02:00
mesh-admin a8f1cdb445 Merge pull request 'A machine reports whether its virtualisation daemon runs (hq ADR 0172)' (#69) from jschoubben/the-lab-is-a-module into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 12:47:55 +00:00
jschoubben ecb3003ba4 A machine reports whether its virtualisation daemon runs
The lab module needs the virtualisation daemon (novox/hq ADR 0172); the
capability is detected by asking the daemon about itself, not by finding
a client on disk.
2026-10-02 14:46:18 +02:00
mesh-admin d3861f82d4 Merge pull request 'A container may declare the capabilities it is granted (hq ADR 0169)' (#68) from feat/the-firewall-seat-serves-its-verbs into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 11:28:34 +00:00
jschoubben b6dbe0a7b9 A container may declare the capabilities it is granted (hq ADR 0169)
Exactly the names declared reach the runtime, named in the spec so a change
recreates the container; a name that is not a capability's is refused and a
privileged container stays undeclarable. For a seat holder whose runtime
changes the machine's packet filter.
2026-10-02 13:27:34 +02:00
mesh-admin 07bdad9e94 Merge pull request 'The host says what filters the machine, with owners, and keeps the found firewall retired on every converged apply (hq ADR 0168)' (#67) from feat/one-thing-filters-a-converged-machine into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 09:58:58 +00:00
jschoubben 627ac97d4f The host says what filters the machine, with owners, and keeps the found firewall retired on every converged apply (hq ADR 0168)
Every table and chain that refuses traffic is reported with whose it is:
the mesh's, the found firewall's, the container runtime's own, a ban, or
other — the runtime's user chain is other, which is where both predecessors
kept their rules, in the legacy filter on one machine and invisible to the
mesh. Adoption's threshold does not move; a converged machine's report
grows by its filters and its found firewall's state.

Convergence is a state the host keeps: a found firewall enabled again is
retired again and said; a reconcile that finds it inactive records that it
was found so, never that the mesh did it; a step skipped after a failed
apply is said. A retirement the mesh began and did not finish is finished.

Fixtures are rulesets captured from three machines of the first mesh.
2026-10-02 11:58:16 +02:00
mesh-admin ee2359f29f Merge pull request 'Every physical link faces outside, up or down (hq issue 197)' (#66) from jschoubben/every-physical-link-faces-outside into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 09:54:06 +00:00
jschoubben cbdbf6b7d3 Every physical link faces outside, up or down
The filter accepts what does not arrive on a link the machine names as
outward, and the host named only links carrying a default route. An
unplugged wired port was left unfiltered for whenever it was plugged in
(novox/hq issue 197). A link backed by a physical device is now named
whether or not it is up.
2026-10-02 11:53:53 +02:00
mesh-admin e12ca3f4dd Merge pull request 'A former target of a kind the host cannot remove is left in place and said, never fatal (hq issue 194)' (#65) from fix/a-former-target-without-a-removal-is-left-and-said into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 07:36:57 +00:00
jschoubben c47aa5d9b3 A former target of a kind the host cannot remove is left in place and said, never fatal (hq issue 194)
The host delivers its own successor as an archive whose target is a new
directory each version, and since mesh-host 63 the record keeps a resource's
former target for the next apply to remove. An archive has no removal (issue
162), so the first host that replaced itself under that rule refused its own
former version at the first step of every apply, and all four machines applied
nothing from then on. A former target nobody dropped is forgotten and said;
an archive the declaration dropped still refuses.
2026-10-02 02:43:47 +02:00
mesh-admin 3b9692b3cb Merge pull request 'A taken container keeps a found network, a left-out module is kept, and genesis raises the forge as its module declares (hq ADR 0163)' (#64) from feat/a-take-is-a-comparison-the-rest into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 00:28:17 +00:00
jschoubben fb9c9c3ee8 A taken container keeps a found network, a left-out module is kept, and genesis raises the forge as its module declares (hq ADR 0163)
A container may name networks it also joins once created, for the per-machine
setting that keeps a found network while a neighbour still resolves it there:
joined after the run, part of the spec, refused when it cannot be joined.

A declaration may say which modules the mesh left out because a stored setting
cannot compose with its definition. Absence used to read as removal; a left-out
module's records are kept and said, and its holds are not released.

Genesis raises the bootstrap forge under the gitea module's container name, with
its image digest and its data directory mounted at /data, so the module holds it
by the found rule instead of raising a second forge beside it (issue 090). The
network is the one difference left for a take to say. Before this the forge had
no volume: its repositories were the container's, lost with it.
2026-10-01 23:45:05 +02:00
mesh-admin d53e626366 Merge pull request 'A take is a comparison: the host's facts, former targets, and strays (hq ADR 0163)' (#63) from feat/a-take-is-a-comparison-the-hosts-facts into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 19:25:56 +00:00
jschoubben 83b3d20e68 A take is a comparison: the host's facts, former targets, and strays (hq ADR 0163)
Every held thing carries what a take compares: for a found container its image and the image's date,
the networks it is on and the other containers on each, its mounts and published ports, beside the
declared image (and its date once pulled), ports and volumes, with the downgrade decided when both
dates are known; for a found file whether the declared content differs and how, as lines lost and
lines new. A resource whose target moved keeps the former target on record as an orphan, so the next
apply removes the container or file the host wrote under the old name (issue 097). Every apply reports
the strays: containers the mesh neither wrote nor holds.
2026-10-01 21:24:37 +02:00
mesh-admin e030aa2387 Merge pull request 'The first user list lets the controller publish assignments and hear its seat's tools (hq #251)' (#62) from fix/first-user-list-matches-the-controller into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 15:29:55 +00:00
jschoubben c670bef4e1 The first user list lets the controller publish assignments and hear its seat's tools
The controller's own composition (mesh-controller internal/broker, 2026-10-01)
publishes memberships into the assignments stream after each push and
subscribes to its seat's tool subjects; the list the installer carries did
not say so, and a controller on it is refused on the first thing it tries:
"Permissions Violation for Publish to mesh.assignment.novox.builder" (hq #251),
which is why every build asked through the console was lost. The controller's
test that compares the two (TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose,
run with this checkout beside it) named exactly these two subjects, and passes.
2026-10-01 17:29:48 +02:00
mesh-admin 45529bfec2 Merge pull request 'The profile names the network manager that is running, and travels in every report (hq ADR 0161)' (#61) from feat/the-profile-names-the-uplink-and-travels-in-the-report into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 14:02:16 +00:00
jschoubben b1e9ccff6d The profile names the network manager that is running, and travels in every report (hq ADR 0161)
One capability per manager — uplink-networkmanager, uplink-systemd-networkd, uplink-dhcpcd — from
systemctl is-active, so the uplink seat's holder for a manager this machine does not run is refused
the way any missing capability is, naming it (issue 138). The apply that reports detects the profile
again and sends it, the same shape enrolment sends, so a machine that switched managers reaches the
mesh at its next push.
2026-10-01 15:58:31 +02:00
mesh-admin cbcf0bcc93 Merge pull request 'A node can join the bus the mesh runs on' (#51) from fix/a-node-can-join-the-bus-the-mesh-runs-on into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 11:18:02 +00:00
jschoubben 6910f07d75 Merge pull request 'Say what a container's host entries now are' (#60) from feat/148-names-are-resolved-not-copied into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 12:38:15 +00:00
jschoubben 88037b33b7 Say what a container's host entries now are
novox/hq ADR 0148: the mesh's names are no longer among them, only what
the module declared. Comment only; the digest is unchanged.
2026-09-30 14:38:11 +02:00
jschoubben 422ad516d5 Merge pull request 'A declaration carries its order, and a host refuses an older one' (#59) from feat/107-a-declaration-carries-its-order into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 12:03:10 +00:00
jschoubben 8431ecfb48 A declaration carries its order, and a host refuses an older one
novox/hq 04-ISSUES/107. A declaration's only identity was the digest of
its bytes: a host could say "not the last" and could not say "older". On
the link, nothing refused an older one at all, and the drain picked the
last to arrive — wrong exactly when it mattered, a backlog drained out of
order or a broker that split a burst.

A declaration may now carry a sequence, one higher per send. A host
refuses one lower than what it kept, whole, and says why. The drain keeps
the highest sequence in a batch rather than the last to arrive.

Only when both sides claim an order. Absent reads as zero — "no ordering
claimed", not "first" — so a controller that sends none is still
understood and a host that kept one before it understood them compares
nothing. That is what lets hosts go first and the controller follow, which
is the order 087 says a new field needs.
2026-09-30 14:03:03 +02:00
jschoubben f107d68b2d Merge pull request 'A delivered host knows it is the delivered one' (#58) from fix/163-a-delivered-host-knows-it-is-the-delivered-one into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 11:46:58 +00:00
jschoubben 1028193c8a A delivered host knows it is the delivered one
novox/hq 04-ISSUES/163. The host asks after every apply whether a newer
host is delivered than the one running, and asked with the link-time
version stamp — which every delivered host carries as "development
build", because the version comes from where the binary sits now (0142).
So a delivered host never matched the newest delivered version, stood
aside on every push for ever, and because standing aside cancels the
report, the mesh never heard from it again.

Measured on two machines: each push produced "host <v> is delivered;
standing aside" for the version already running, then "applied, and could
not tell the mesh: reporting: context canceled". A machine restarting its
host on every push and reporting nothing, reading as healthy.

Asked with the running version now. Half of 0142 was applied to the
report and the known-good record and not here; this is the other half.
2026-09-30 13:46:51 +02:00
jschoubben f576287b51 Merge pull request 'A delivered host reads its version from where it sits' (#57) from fix/161-a-delivered-host-reads-its-version-from-its-path into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 10:40:29 +00:00
jschoubben 6c6495f6d9 A delivered host reads its version from where it sits
novox/hq ADR 0142, which decided this and was not implemented: "It is
unpacked into a directory named for its version, so it can read its own
version from its path. The stamp goes, and with it the need for a build to
know what it will be called."

The mesh's toolchain stamps no version, on purpose, so a delivered host
called itself "development build" and the mesh could not tell which host
any machine ran — which is the whole of what 087 added. A delivered host
lives at <libexec>/versions/<version>/<binary>, and that directory is the
answer.

A host placed by hand keeps its stamp, which is the honest answer for one
the mesh did not deliver, and is every machine until a delivery reaches
it. A binary sitting anywhere else is not read as a version at all.

The decision is split from the reading so a test can ask about a path
without being that binary.
2026-09-30 12:40:00 +02:00
jschoubben e6d48cf537 Merge pull request 'The mesh delivers the launcher, which is the last link in self-update' (#56) from feat/142-the-mesh-delivers-the-launcher into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 10:19:07 +00:00
jschoubben b8a766f234 The mesh delivers the launcher, which is the last link in self-update
novox/hq ADR 0141 and 04-ISSUES/142. A version was being delivered to a
machine and nothing started it: the launcher on these machines predates
the versions mechanism and runs the fixed binary path, so the delivery was
correct and inert.

Delivered as a FILE resource, not as part of an archive, and the
difference is the whole reason this is safe. A file is written atomically —
temp file in the same directory, then rename — so the running launcher
keeps the inode it was started from and the next start picks up the new
one. An archive writes in place with truncate, which would cut the file a
running shell is reading halfway through.

The manifest therefore carries a second copy of the script, and a test
refuses any difference between it and packaging/nox-mesh-host-launch.
Proven by drifting one and watching it fail. Two copies of a script is a
bad thing to accept, and the alternative was writing over a running
supervisor.

Together the two resources complete the loop: the version lands, the
running host stands aside because it sees one delivered, and the launcher
that starts next is the one that looks in versions/ and picks the newest
by arrival.
2026-09-30 12:19:00 +02:00
jschoubben 9caea5bc32 Merge pull request 'The delivered binary is named as every machine runs it' (#55) from fix/142-the-delivered-binary-is-named-as-machines-run-it into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 09:41:52 +00:00
jschoubben df27cee7b7 The delivered binary is named as every machine runs it
nox-mesh-host, not mesh-host. The command directory is cmd/mesh-host and
the launcher looks inside a delivered version for nox-mesh-host — the name
this is installed at and the name in its unit. The first delivery landed
the package's name, reported success, and would have been invisible.
2026-09-30 11:41:45 +02:00
jschoubben d275e64ed3 Merge pull request 'The host declares its own successor, as an archive at a versioned path' (#54) from feat/142-the-host-delivers-its-successor into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 09:33:19 +00:00
jschoubben 1a628a4d22 The host declares its own successor, as an archive at a versioned path
novox/hq ADR 0141 and 04-ISSUES/142. The host half of the delivery has
been built and tested since 0141 and has never had a version to work on:
versions side by side, the newest runs, the running one stands aside
between reconciles, rollback picks a directory. This is the declaration
that gives it one.

One archive, unpacked to /usr/lib/nox-mesh-host/versions/${version}. The
version resolves to the artifact's digest, so an unchanged build lands at
the path it already had and re-composing a declaration moves nothing.

Not circular: the host applying this is a different version from the one
being written, and neither writes over the other — the kernel refuses to
truncate a running executable, which is the reason the path carries the
version rather than a link pointing at "current".

**The launcher is deliberately not delivered here.** The one on these
machines predates the versions mechanism and runs the fixed binary path,
so a delivered version is inert until it is replaced — and replacing it
from the mesh means writing over a running shell script, which sh reads
incrementally. That wants a designed swap rather than a file resource, and
it is the last piece rather than this one.
2026-09-30 11:33:12 +02:00
jschoubben b5196e974c Merge pull request 'The host is a module, so the mesh can build it' (#53) from feat/142-the-host-is-a-module into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 07:56:35 +00:00
jschoubben 5162c3b05f The host is a module, so the mesh can build it
novox/hq 04-ISSUES/142 and ADR 0142. Nothing delivered the host because
nothing could compile it, and nothing could compile it partly because the
host was not a thing the mesh builds at all — it had no manifest.

One bundle in Go, for arch, built from cmd/mesh-host. A system is
required for a compiled artifact because a binary is pinned at link time
so a host refuses to touch a machine it was not built for (ADR 0005), and
`arch` is what all four of this mesh's machines report themselves to be.
Another system is another artifact and another build, which is what ADR
0142 means by one per target.

No resources yet. What places a version into a directory named for it
needs an archive resource whose path carries the version, and nothing
interpolates one — the second half of 0141's insight, and the next piece.
2026-09-30 09:56:18 +02:00
jschoubben 98fe8edf35 Merge pull request 'An apply says what it held, not only what it applied' (#52) from fix/125-a-hold-is-a-line-in-the-report into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 06:47:45 +00:00
jschoubben cbf50185d0 An apply says what it held, not only what it applied
novox/hq 04-ISSUES/125. An adopted node keeps what it found until its
module is taken, which is correct and was recorded only in the node's own
state file. On the edge cut-over the mesh sent 346 resources, the journal
said it applied 330, and nothing anywhere said which sixteen or why —
reading it meant opening state.json by hand, and not reading it took every
public name on the machine down.

The line that reports the apply now carries it, grouped by module and
ordered by name, because the sentence an operator needs is "route-proxy is
assigned and not taken" and the module is the thing `take` acts on. An
apply that held nothing says nothing extra: a line that reports "0 held"
on every converged apply is a line that stops being read.
2026-09-30 08:46:19 +02:00
jschoubben 197258c88c A node can join the bus the mesh runs on
novox/hq 04-ISSUES/146, the layers behind the three already fixed.

A new membership says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed, and became a refusal the moment one did: an
enrolled node came up and reconnected for ever against its own record.

The enrolling client takes its inboxes in the space its user may listen in. A
JetStream publish waits for the stream's acknowledgement on an inbox the client
picks, and its default is one this user may not subscribe to — so the enrolment
failed with a permissions violation on a subject nobody had chosen.

And the enrolment publish carries a message id, so the client's own retry is
discarded by the stream rather than enrolling the machine twice. That one is
not finished: the duplicate survives it, and the issue says where the trail
stops.
2026-09-29 17:36:59 +02:00
jschoubben a3b810f1f0 Merge pull request 'A first node gets as far as its own bus: three faults on the way' (#50) from fix/one-foundation-on-the-bus-the-mesh-runs-on into main 2026-09-29 14:06:33 +00:00
jschoubben 971a6d6d03 A first node gets as far as its own bus: three faults on the way
novox/hq 04-ISSUES/146. Each was right while the mesh ran on the previous
broker, and nothing has raised a foundation since it changed.

The bus's certificate is made by the program that needs it rather than by
openssl inside the broker's image — the bus's image is Alpine with a shell and
no openssl, so the step exited 127 and no mesh could be raised. Self-signed as
before and on purpose; --user 0:0 because the volume is root's and the control
plane's image runs as nobody.

Enrolment no longer opens a raw TLS connection to check the pin: NATS speaks
its own protocol and upgrades afterwards, so the handshake met a plaintext
greeting. The client that presents the token carries the same pinned config
and verifies inside its own handshake, so the secret still leaves only after
the certificate is checked. The raw dial stays as what its tests prove, and is
no longer a path anything takes.

And the token says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed and became a refusal the moment one did.

It now stops at the bus's user list, which is the genesis half of 146.
2026-09-29 15:42:43 +02:00
mesh-admin 04a27caa43 Merge pull request 'A host running as a service says what its apply did' (#49) from fix/a-host-running-as-a-service-says-what-it-did into main 2026-09-29 07:16:01 +00:00
jschoubben 6e90c2692d A host running as a service says what its apply did
The serving path passed nil where the apply writes its detail. Nil is silence, so
everything the apply says — a file held, a container replaced, the found firewall
retired — was visible when a person ran the one-shot command and discarded in the
way the host actually runs, which is always.

Measured: after a machine was converged and its found firewall was not retired,
what the host decided was unrecoverable, because it had said it to nobody. That is
why issue 143 has candidates instead of a cause.

say already reaches stdout and the unit sends that to the journal, so this needed
no new mechanism — only for the argument to be passed. Both paths now reach the
apply through one named helper, so a reader asking where the apply's output goes
finds one answer.

The test asserts the log is never nil and cannot catch the fault it was written
for, which is wiring; that is proved by a deployed host whose journal carries the
detail.
2026-09-29 09:15:53 +02:00
mesh-admin ced54d489f Merge pull request 'A converged machine can speak unasked' (#48) from fix/a-converged-machine-can-speak-unasked into main 2026-09-28 23:13:13 +00:00
jschoubben 94a35a39eb A converged machine can speak unasked
A reconcile is otherwise silent, and the condition deciding when it speaks asked
only what an adopted node reports: what it holds, and the firewall it found. A
converged node has neither, so it could speak only in reply to a declaration —
and the mesh composes no declaration for a node that has not said which links
face outside (ADR 0140). A machine waiting for a push that was waiting for the
machine.

Measured, not predicted: after the control plane learnt to read the links, the
control node recorded its own and the three converged machines sat silent while
their filters were refused.

The condition is now a named predicate, because as an inline expression nothing
could test it — which is why the gap shipped.
2026-09-29 01:13:09 +02:00
mesh-admin ec06369101 Merge pull request 'A machine says which links face outside without being asked' (#47) from fix/a-machine-says-unasked-which-links-face-outside into main 2026-09-28 23:00:28 +00:00
jschoubben bb85af1821 A machine says which links face outside without being asked
The mesh composes no filter for a machine that has not said (ADR 0140), and a
converged machine only speaks unasked when its adoption fingerprint changes. The
links were not in that fingerprint, so a machine that had just learnt to say
could only speak when a declaration arrived — and a declaration cannot be
composed until it has spoken. A machine waiting for a push that is waiting for
the machine.

Found before it bit: every machine in this mesh is in exactly that state right
now, having just been given a host that reports the links to a control plane that
does not yet read them.
2026-09-29 01:00:26 +02:00
mesh-admin 0c3928ad19 Merge pull request 'The host delivers its own successor, and versions live side by side' (#46) from feat/the-host-delivers-its-own-successor into main 2026-09-28 22:29:57 +00:00
jschoubben fbf0fb7d63 The host delivers its own successor, and versions live side by side
The supervision was already right: a clean exit means the host stood aside, and
the launcher's next turn runs what is on disk. Two things made it dead code —
nothing told the running host a successor was waiting, and the rollback resolved
its known-good version through pacman, which no machine here uses and which two
of three operating systems do not have.

Keeping a version rather than a path was the clue. Versions now live in
directories named for them:

- the launcher picks the newest delivered one every time round the loop, or the
  one a rollback pinned, or the host placed by hand when nothing is delivered;
- the running host stands aside between reconciles, never inside one, by exiting
  cleanly — and returns nil so the launcher does not count it as a crash;
- a completed reconcile retires what is older than the predecessor, keeping the
  predecessor because that is what a rollback starts, and never the running one;
- rollback pins the predecessor instead of reinstalling a package: no package
  manager, no cache anyone may clean, same script on every operating system;
- the report says which host version produced it, so 'behind' is answerable.

Newest is when it arrived, never how the name sorts: '1.10' orders before '1.9',
and ordering by name would start an older host and call it an upgrade.

novox/hq ADR 0141. The delivery half — a module carrying the next host — follows;
until then nothing delivers a version and every machine takes the fallback, which
is what it does today.
2026-09-29 00:29:36 +02:00
mesh-admin b005d4ef17 Merge pull request 'A machine says which of its links face outside' (#45) from feat/filter-what-arrives-from-outside into main 2026-09-28 21:37:24 +00:00
jschoubben b0d11c2439 A machine says which of its links face outside
The filter blocks everything passing through the machine and then allows the
machine's own containers back by naming the address ranges they sit on — two
ranges fixed in the control plane and the rest typed after a flip had already
cut a workstation off. A range describes one machine and goes stale in silence.

Read the links carrying a default route instead, from /proc rather than by
asking a program, and report them on every apply. A machine with no route off
itself reports nothing, and the mesh composes no filter for it rather than
writing a rule around a link with no name.

novox/hq ADR 0140. The control plane does not read this yet.
2026-09-28 23:37:06 +02:00
mesh-admin 155689672c Merge pull request 'A container's mesh names are part of what it is' (#44) from fix/a-containers-mesh-names-are-part-of-what-it-is into main 2026-09-28 15:19:39 +00:00
jschoubben e82789a322 A container's mesh names are part of what it is
A container resolves every machine and public name through the entries it is given when it is created,
and nothing re-reads them. The host compared everything about a container except those, so one whose
image and files never changed was left alone holding an overlay address five days out of date — it
restarted 2286 times against a database it could no longer find, and the mesh reported the machine as
doing what it was told (novox/hq 04-ISSUES/135, the same fault as 045 in the field left out).

Sorted, so the digest does not move for a reordering nobody made. The first apply after this recreates
every container that carries mesh names, once.
2026-09-28 17:19:38 +02:00
mesh-admin 99562947f3 Merge pull request 'Genesis lets the control plane state its own facts' (#43) from fix/genesis-lets-the-control-plane-state-its-facts into main 2026-09-28 14:07:22 +00:00
jschoubben c171c64d3a Genesis lets the control plane state its own facts
A mesh raised from nothing must be able to say what it applied and what a machine refused (novox/hq
ADR 0134), and the first user list is what permits it. Kept in step with the controller's own
composition by the test that reads this file.
2026-09-28 16:07:20 +02:00
mesh-admin 0dc5515099 Merge pull request 'A resource's owner is read to the last dot, because a module's name may contain one' (#42) from fix/a-resources-owner-is-read-to-the-last-dot into main 2026-09-28 13:45:01 +00:00
jschoubben 58c0e715c7 A resource's owner is read to the last dot, because a module's name may contain one
novox.be is a module on this mesh. Reading a resource's owner to the first dot made its resources
belong to something called "novox", and a step's gate would then skip whatever else happened to start
that way — silently. A resource's own id never contains a dot, which is what makes the last one the
boundary; the mesh's derived step was changed to add a hyphen rather than a dot for the same reason
(mesh-controller #128).
2026-09-28 15:44:59 +02:00
mesh-admin c5b229f95d Merge pull request 'A step gates its module, not the machine' (#41) from fix/a-step-gates-its-module-not-the-machine into main 2026-09-28 13:38:21 +00:00
jschoubben a9c49724b5 A step gates its module, not the machine
A run-once container that does not complete stops the rest of that module's resources; everything
else on the machine is attempted, as every other shape already is (novox/hq ADR 0136). An action
still gates the machine — genesis is a row of them and they belong to no module. What was not
attempted is reported as skipped, because that and 'nothing to do' are different answers.

Without this, ADR 0135's derived preparation would let one module's unreachable database hold a
machine hostage — the fault 04-ISSUES/011 removed for everything else, and the reason the catalogue
migrates itself at start.
2026-09-28 15:38:19 +02:00
mesh-admin 296ec5ece6 Merge pull request 'Genesis lets the controller ask any module's tool' (#40) from fix/genesis-lets-the-controller-ask into main 2026-09-28 02:21:24 +00:00
jschoubben 45b9a507a1 Genesis lets the controller ask any module's tool
The controller is the way in for tool calls (novox/hq ADR 0095); the first user list must say so,
or a mesh raised from nothing refuses its own first ask. Kept in step with the controller's
composition by the test that reads this file.
2026-09-28 04:21:22 +02:00
mesh-admin 5c410aaf54 Merge pull request 'One bus: the AMQP transport is gone from the host' (#39) from feat/one-bus into main 2026-09-28 01:54:23 +00:00
jschoubben 2478b5f127 One bus: the AMQP transport is gone from the host
The mesh runs on the seat's bus alone (novox/hq ADR 0131, design 28 task 5.5). The host's old
dialling and enrolment paths are deleted with the switch that chose between them; a membership or
a token naming another bus is refused before anything is sent, rather than dialled on a transport
that no longer exists.
2026-09-28 03:54:22 +02:00
mesh-admin c82e933268 Merge pull request 'Genesis: the first user list lets the controller hear the forge's merges' (#38) from fix/genesis-follows-the-forge into main 2026-09-28 01:13:18 +00:00
jschoubben e212da6bd2 Genesis: the first user list lets the controller hear the forge's merges 2026-09-28 03:13:16 +02:00
mesh-admin 673912ccdc Merge pull request 'A host adopts a delivered membership at start, not only after a declaration' (#37) from feat/a-host-adopts-a-membership-at-start into main 2026-09-28 00:40:27 +00:00
jschoubben 68a193d6f0 A host adopts a delivered membership at start, not only after a declaration
The rescue path: an operator writes the membership file by hand on a machine no
bus can reach — rotated while it still held the old password — and restarts the
host. Same file, same check as the declaration path.
2026-09-28 02:21:56 +02:00
jschoubben 4fba884d47 Merge pull request 'Genesis: the first user list lets the controller hear its consumers, and the account has JetStream' (#36) from fix/genesis-accounts-hear-and-have-jetstream into main 2026-09-27 23:47:41 +00:00
jschoubben 15f0dabf32 Genesis: the first user list lets the controller hear its consumers, and the account has JetStream
Two things the controller's own composition now derives and the installer's
carried list did not: the delivery subjects of the controller's consumers, and
JetStream enabled on the account — without which the first bound consumer is
refused. Found live on a mesh moved rather than raised; a fresh genesis would
have met both at first start.
2026-09-28 01:46:14 +02:00
jschoubben 4ef41ad5a0 Merge pull request 'A host logs in as node.<name> and hears answers on its own inbox' (#35) from fix/a-host-uses-its-own-inbox into main 2026-09-27 23:18:30 +00:00
jschoubben d749de989c A host hears the server's answers on its own inbox
The mesh grants a machine exactly its own inbox prefix; the client used the
default one and was refused a subscription to it.
2026-09-28 01:16:23 +02:00
jschoubben 14e64844df A host logs in to the new bus as node.<name>
The mesh names a machine's bus user node.<name>; the host dialled with the bare
name and every machine was refused the first time it reached the server:
"authentication error - User". Same string as the composed user list, or nothing
connects.
2026-09-28 01:13:52 +02:00
jschoubben d82fc9a121 Merge pull request 'A machine moves to the bus its declaration tells it to' (#34) from feat/a-machine-moves-to-the-bus-it-is-told into main 2026-09-27 22:09:08 +00:00
jschoubben 9fd3762e93 A machine moves to the bus its declaration tells it to
Until now a membership — bus address, fingerprint, password — was written once,
at enrolment, and nothing ever rewrote it, so a machine already enrolled could not
be moved to another bus at all (novox/hq design 28, task 5.2). The mesh now
delivers a membership for the new bus as a sealed file in the declaration, like
any secret; the host reads it after the declaration has applied, saves it as its
identity, and exits cleanly so the service manager restarts it dialling the bus it
names. The same path a machine takes after a reboot — so nothing new has to be
right for it to work. A membership carries its transport; empty means the bus the
mesh ran on before, so nothing written earlier reads as unset.
2026-09-28 00:08:44 +02:00
jschoubben 587b8aa220 Merge pull request 'A host reaches either bus, and a genesis template that raises the mesh on the new one' (#33) from feat/nats-genesis into main 2026-09-27 17:36:48 +00:00
jschoubben 48e2ff2de5 Merge remote-tracking branch 'origin/main' into feat/nats-genesis 2026-09-27 18:38:45 +02:00
jschoubben 6a3435629e A foundation template that raises the mesh on the bus being built
The same twelve steps, with the difference that matters: the mesh composes its own user
list and at genesis there is none, so this carries the first one — the controller's
account at a bootstrap password, rotated with the store's and replaced by the
controller's own composition from its first start.

The server's settings and the user list are separate files in one directory. Separate
because the settings belong to whoever raises the server and the users belong to the
mesh; in one directory of necessity, because an include path resolves relative to the
including file's own directory, so an absolute one sends the server looking underneath
that directory and it refuses to start.

No `verify` in the TLS block. That makes the server demand a client certificate and
nothing in the mesh presents one — a host pins this server's exact certificate and
authenticates with a password.

The controller's permissions here are checked against what the controller derives, by a
test in its own repository reading this file. They are two statements of one fact, and a
template that granted less than the controller needs would produce a mesh that comes up
and is refused on its first act.
2026-09-27 16:39:32 +02:00
jschoubben 9072f60a30 Enrolment behind a seam, with both transports
The last of the host's link that still named a transport. `Asking` is one
enrolment conversation — a connection made with the token, a question asked, and
an answer waited for — and it is its own seam rather than part of `Link` because
almost nothing about it is the same: the credential is a one-time secret, there
is no declaration to hear, and a node that fails here is not in the mesh at all,
where a node that fails in `Link` has merely lost touch with one it belongs to.

`Enrol`'s thirteen arguments became an `Approach` — where, which certificate,
which bus — and the request it already had. The token says nothing about which
bus, and does not need to: every token names the one the mesh runs on today until
the rollout.

**The reply address is the whole of what changes on the new bus**, and it is
forced rather than preferred. Verified against a running server, both halves: the
answer reaches the node at the address its request carried in the payload, and
the transport's own reply field held something else entirely by the time the
consumer saw it — the consumer's ack address, exactly as design 25 §2 says. The
test asserts the field is *not* the node's inbox, so a future server that stopped
claiming it would fail this rather than let the reason quietly become folklore.

The inbox is under `_INBOX.enrol.<node>.`, which is exactly what the enrolling
user may subscribe and no wider, with a random tail per attempt: a reply left
over from an attempt that timed out is not the answer to this question, which is
what the correlation id does on the other transport. Subscribed before anything
is published, because a node that published first could miss an answer to a
question nobody was listening for.
2026-09-27 01:31:46 +02:00
jschoubben 6e208f7b3e The host's inbound behind a seam, with both transports
The outbound half went behind `Bus` and a node's two statements stopped naming a
transport. This is the other half, and where the transport reached furthest: the
run loop selected on a channel of the client library's own delivery type, so
every part of holding a node in its mesh knew which bus it was on.

`Link` is dialling, hearing and saying in one interface, because dialling is
where the transport is chosen and choosing it twice is how one half of a node
ends up on a different bus from the other. `Declaration` has one way of being
done rather than two: a declaration set aside for a newer one is settled exactly
as an applied one is, on both buses, and the difference is a fact the report
carries.

Four things this settled.

**The host declares nothing on the new bus.** On the bus the mesh has it declares
its own queue, because a queue that is not there means a node that hears
nothing. Here it binds to a consumer the mesh made when the node enrolled, and a
missing one is said as the mesh's to answer rather than quietly created with
whatever this client happens to default to.

**The pin is easier here than in the tool runtime, not harder.** The Go client
takes a *tls.Config, so the same PinnedConfig with the same VerifyPeerCertificate
does the work — the subject-alternative-name constraint recorded against the
runtime's client is that client's, because it takes PEM strings with no verify
hook. A host checks the fingerprint and nothing else.

**Binding needs the subject as well as the consumer.** An empty subject is
refused rather than taken to mean "whatever that consumer delivers", which the
server said plainly and only when asked.

**Reconnection stays the caller's.** Hold already decides when to try again and
how long to wait; a client reconnecting underneath it would make that reasoning
a duplicate of the library's.

The drain keeps its live half and loses its catch-up half, as it said it would:
verified that three declarations pushed to an absent node leave one on the
stream, and it is the newest.

One test-harness lesson worth the comment it got: delete-then-add is not a reset.
A test that did that inherited the previous test's messages, and the symptom was
a declaration counted as delivered twice — which reads as a redelivery bug in the
code under test rather than as a dirty stream.
2026-09-27 01:25:01 +02:00
jschoubben 54f6eb661a Merge pull request 'A taken tunnel's found configuration is retired once the take is proven (hq ADR 0119)' (#32) from feat/a-taken-tunnels-predecessor-is-retired into main 2026-09-26 22:59:51 +00:00
jschoubben 646be4fdf4 Merge pull request 'Undeclaring gives a unit back the state it was found in, and removes a process the mesh made (hq ADR 0118)' (#31) from feat/undeclaring-leaves-the-machines-units into main 2026-09-26 22:59:30 +00:00
jschoubben e688b3b816 Merge pull request 'A file written into a marked block, so a shared hosts file keeps every line that is not the mesh's (hq 128)' (#30) from feat/a-file-written-into-a-marked-block into main 2026-09-26 22:59:21 +00:00
jochen 50776b8613 review: a retired configuration that comes back is retired again on its first original, and only wg-quick's own file is ever removed (hq ADR 0119)
Put back by hand, it was found afresh and its copy became the hold's original, so a machine that
kept restoring it kept growing copies and lost which one was first. The first original now stays
the record's, content that differs is kept once beside it, and the note says a rollback means
unassigning the private network. Nothing is removed unless it is <wireguard dir>/<iface>.conf,
not a path the mesh writes, and not a link, which would leave the key-bearing target behind.
2026-09-27 00:55:50 +02:00
jochen b462f461c6 A taken tunnel's found configuration is retired once the take is proven (hq ADR 0119)
Kept on disk it was the take's fallback; once the mesh's interface is up in its place and a peer
has handshaken with it, it is an unmaintained way back onto the network, held for ever. It is now
removed from where its unit reads it, its kept original verified first and left as it is, and the
hold ends. Until proven — no handshake, or wg not answering — it is kept and the report says why.
The retirement is recorded apart from holds, so later applies, an undeclare, and a reassignment
find it retired rather than missing, and nothing writes it back.
2026-09-27 00:47:57 +02:00
jochen 23a4436499 review: a unit whose file the mesh wrote is stopped when undeclared, and what was found survives a failed first apply (hq ADR 0118)
Records written before Found existed left the adoption guard and the converge filter loaded on
undeclare, then deleted their unit files from under them; a unit whose own file the mesh created
is now the mesh's, whatever its record says. Found is kept apart the moment it is read, so a
first apply that enabled and then failed is not read back as the machine's; boot is found the
first time the mesh sets it; a service once stateless, or moved to another unit, is found afresh
(the old unit given back). The unit is read after the reload that loads a file written in the
same apply, and removal reports what it actually did.
2026-09-27 00:41:02 +02:00
jochen 08c0f40ff0 review: refuse a process named ".", ".." or with a leading dash, so removing one cannot delete the mesh's own directory (hq ADR 0118)
removeProcess deletes filepath.Join(daemonRoot, name) whole; a process named ".." made that
/var/lib/mesh. The declaration and the removal now hold the name to one rule.
2026-09-27 00:41:02 +02:00
jochen 3112c881e4 Undeclaring gives a unit back the state it was found in, and removes a process the mesh made (hq ADR 0118, issue 130)
A service undeclared used to be stopped: unassigning the private network stopped the container
runtime, unassigning sshd would stop ssh, an uplink module would take the machine offline. The
host now records the unit's state when it first applies it and restores that on undeclare —
found running stays running; started by the mesh (the converge filter) is stopped again; nothing
is started on the way out; a pre-existing record leaves the unit alone.

An undeclared process had no removal at all and failed every apply on its node; its unit, timer
and bundle are now removed.
2026-09-27 00:21:25 +02:00
jochen 06aaac0820 A service may omit its state, so unassigning an uplink module never stops the machine's network manager (hq ADR 0117) 2026-09-27 00:11:58 +02:00
jochen fdc768c476 review: rebuild a file the mesh once wrote whole, keep links, give back a missing line end, and release a hold only after the write (hq issue 128) 2026-09-27 00:09:34 +02:00
jschoubben 25449a31c3 The host's outbound behind a seam, with both transports
Step 3.5's first half, mirroring the controller's. A host says exactly two
things unprompted, and the difference between them is the whole interface:
a report must arrive, and a heartbeat must not be insisted on. So a report
goes through JetStream — it is the message the store-window guarantee is
about — and a heartbeat stays on core, because a heartbeat in a stream is
the mesh's least valuable message competing for retention with its most
valuable.

The host still imports nothing of the mesh's own (ADR 0005): this is its
own interface over its own libraries. It agrees with the controller because
a fixture holds both to one envelope, which is the only agreement that
survives two repositories.

Also recorded, where the next person reads it rather than in a plan: the
"newest wins" window narrows at the rollout and does not disappear. Last-
per-subject makes the catch-up half the stream's, and sequence orders them
definitively — but three pushes to a connected node are still three
deliveries. Saying which half goes is worth more than "can probably be
removed", which is how a load-bearing window gets deleted in a hurry.
2026-09-27 00:01:42 +02:00
jochen 1cb895346d Write into a marked block of a text file instead of over it, so a shared hosts file keeps every line that is not the mesh's (hq issue 128) 2026-09-26 23:51:36 +02:00
jochen 3ae999497f Format control_test.go so the gate's fmt step passes on main 2026-09-26 23:51:36 +02:00
jschoubben 32d7234637 Merge pull request 'A host accepts an explicitly-empty declaration (hq 127)' (#29) from feat/an-explicit-empty-declaration into main 2026-09-26 21:39:56 +00:00
jschoubben 2722e7b36e A host accepts an explicitly-empty declaration (hq 127)
The empty-resources guard refused every empty body as a likely mistake,
with no way to say emptiness was meant — so the control plane could
never tell a node to drop its last resource. The envelope gains
owns_nothing: with it, an empty declaration is applied (the node drops
what the mesh owned); without it, empty is still refused, so a
truncated or mis-composed body cannot silently strip a machine. One
test, both directions.
2026-09-26 23:39:32 +02:00
jschoubben 808e93a477 Merge pull request 'A found tunnel carries its MTU to the mesh' (#28) from feat/a-taken-tunnel-carries-its-mtu into main 2026-09-26 20:40:09 +00:00
jschoubben 8e2a1e762d A found tunnel carries its MTU to the mesh
The host parses MTU from the found [Interface] and reports it, so the
mesh's interface can come up with the same MTU when it takes the tunnel
over. A path tuned to 1380 regresses to the 1420 default otherwise —
invisible to ping, fatal to TLS handshakes and transfers over that path
(novox/hq: the mesh had no MTU concept). Zero when the config named
none, and the mesh writes no MTU line then.
2026-09-26 22:39:54 +02:00
jschoubben 5dbc8e5c3c Merge pull request 'the spec names the resolver and the address' (#27) from fix/the-spec-names-the-resolver-and-address into main 2026-09-25 21:51:42 +00:00
jschoubben 260bf0b752 the spec names the resolver and the address
dns and ip were declared, validated, handed to the runtime — and part of
no comparison, so their first deployment compared every container equal
and changed nothing, silently. The same shape as 04-ISSUES/045: a field
that is not in the spec is a field that can never reach a container that
already runs.
2026-09-25 23:51:30 +02:00
jschoubben 93caf26aed Merge pull request 'a container may name its resolvers and its own address' (#26) from feat/a-container-may-name-its-resolver-and-its-address into main 2026-09-25 21:48:44 +00:00
jschoubben 3ac765db65 a container may name its resolvers and its own address
Mailu's 2024.06 admin refuses to serve behind a resolver that does not
validate DNSSEC, and the runtime's own forwarder (127.0.0.11) validates
nothing — so a module shipping its own validating resolver had a
resolver nothing could be pointed at. Found live, blocking a cutover:
the admin sat unhealthy, submission answered 454, and the declaration
language had no words for the fix.

Two fields on a container, both handed to the runtime verbatim: dns —
the resolvers it asks — and ip, its static address on its user-defined
network, which exists for exactly one shape: a container others must
reach before name resolution works, the resolver itself being the case
that forced it. Both take only addresses and are refused on arrival
otherwise — a name here would reach the runtime verbatim and be refused
at create, after the old container was already gone.
2026-09-25 23:48:31 +02:00
jschoubben 7e245dae92 Merge pull request 'inspect by kind, not the ambiguous bare form — a same-named network stops a container from ever being found' (#25) from fix/inspect-by-kind-not-the-ambiguous-bare-form into main 2026-09-24 18:29:44 +00:00
jschoubben 5dd439df50 inspect by kind, not the ambiguous bare form — a same-named network stops a container from ever being found
docker inspect <name> resolves across every object kind, not just
containers. A module regularly names a network the same as the
container that joins it (keycloak does this today, ordinarily) — so
when the container does not exist yet but the same-named network
already does, the bare form answers with the network's JSON instead
of reporting the container absent, and the template these callers use
(.State.Running) fails to execute against it entirely.

Live on novox tonight: minio's LB container, named the same as its
network ("minio"), could never be created — every apply crashed on
"the container runtime could not say whether minio is here", stuck
since first push, because the check itself never got a clean answer.

Fixed at every call site asking a container's state by name
(containerState, inspectFound, NamesFree, raiseGiteaServer,
containerRunning) by scoping to `docker container inspect`, matching
the type-scoped form this codebase already uses correctly for
networks, volumes and images elsewhere. Also scoped the one image
inspect that was still bare (publish.go), for the same reason.

mesh-host runs as a host-level service (nox-mesh-host.service), not a
Docker module — merging this does not redeploy it. The live novox
failure persists until the service itself is rebuilt and updated.
2026-09-24 19:50:16 +02:00
jschoubben f68139c9d1 Merge pull request 'Take over the found tunnel: its key, its port, its peers; stop it, never flush (hq ADR 0105)' (#24) from feat/adopt-the-tunnel into main 2026-09-23 22:38:36 +00:00
jschoubben fc593b9dfd Stop nothing the mesh cannot replace, give the tunnel back on failure, and take it over after enrolment
Review of the ADR 0105 build (hq ADR 0105). The takeover stopped the found
unit and then found out whether the mesh's interface would do; a start that
failed left the machine with no tunnel at all.

Now nothing is stopped until the declared interface listens on the found port
at the found address and the key file it names holds the found key — the
refusal names the remedy — and a mesh interface that fails to start after the
takeover has the found unit started again, with the account saying so. The
account has three states (not taken, taken, down) and is given on every
takeover, failure included. An interface raised by hand is looked at again
for a moment and then refused naming `wg-quick down`. A found unit started
again by hand beside the mesh's is said, not stopped: on the hub it cannot
hold the port, and on a spoke two interfaces with one key would fight.

`mesh-host overlay take --tunnel <iface>` is the path for a node that
enrolled before the mesh knew to take a tunnel over: the found key becomes its
overlay key — identity, sealing and serving keys untouched, so nothing sealed
to the node is remade — and the mesh is told with a rekey signed by the
identity key, over the key left, the key taken and the tunnel. Told first,
written second, so a run again puts right whichever half did not happen.
2026-09-24 00:02:08 +02:00
jschoubben 83306b2dba Merge pull request 'Recreate a container when the content of a file it reads at creation changes (hq issue 103)' (#22) from fix/recreate-on-content-change into main 2026-09-23 21:44:21 +00:00
jschoubben 4840e21405 Say in the plan which file a container would be recreated for
The plan says what an apply would change from the declaration and the
record, before the machine is touched. The apply now recreates a container
when the content of a file it reads at creation changed, and the plan said
"check" for every recorded container — true, but a preview that hides the
one step somebody asked about.

So a recorded container whose record of what it read differs from what this
apply will hand it — a plain file declared here, by its declared content;
otherwise what this host last wrote at that path — is planned as an update
naming the file, the same comparison applyContainer makes. What the record
cannot settle stays a check: a file neither declared nor recorded is read
from the machine by the apply, not by the plan; and a container with no
record of what it read was labelled before the host kept that record and is
accepted as it is.

novox/hq 04-ISSUES/103, 104
2026-09-23 23:42:41 +02:00
jschoubben 982b84310e Look at what a container mounts directly, accept a pre-upgrade label, and write the genesis secret without a newline
Review of the first cut found four things.

A directory mounted into a container is no longer looked inside, not even
for the files this host wrote there. The controller records every
provider's received and contributions file as a plain file under a mounted
directory, so folding those in would have recreated the route proxy — which
re-reads its routes live, by design — on every route change, and killed
every provisioner sidecar, which polls what it receives, mid-reconcile on
every grant. Whether a service reads a file under its directory once or
watches it is the service's; restart-on is how a module says "once", and it
stays the opt-in. Env-files and files mounted directly remain by content.

Genesis wrote the superuser secret as `value\n`; `secret accept` strips the
line ending by design, so the postgres module declared `value` — and with
a mounted file's content in the spec, phase three would have recreated the
store it meant to adopt in place, with the temporary control plane
connected to it. Genesis now writes the value alone. readCredentialFile
tolerated both endings already. Pinned with the bytes the genesis code
path writes, then the module's declaration of the same container: it must
reconcile.

A container carrying a label from before the host folded in what it reads
is accepted rather than recreated, when that label matches the spec as it
used to be computed: what it reads is recorded then, a change is caught
from that record from the next apply on, and the label is renewed at the
next genuine recreate. Recreating them all would have been a restart storm
across the mesh in declaration order, the store first. The trade-off is
stated in the code: a container already stale at upgrade time is not
caught, and could not have been either way.

The record of what a container read is looked up by its name when its
declared id has none — the bundle's `store` becomes `postgres.server` for
the same container — so a change on the day it is adopted still names the
file. The by-target lookup takes the most recently applied record, since
the bundle's record for the same target is never removed by the mesh's.

novox/hq 04-ISSUES/103
2026-09-23 23:40:27 +02:00
jschoubben c60228e719 Recreate a container when the content of a file it reads at creation changes
The host decided whether a container was still the one declared by a digest
of its declaration, and the declaration names an env-file's path and a
mount's path — never what is in them. So when the store was given a new
port, the host rewrote the forge's and the analytics service's environment
files, correctly, and left both containers running with the old port in
their environment: a container reads its env-file when it is CREATED, and
`docker restart` hands it the same environment again. Both looked healthy
until they answered 502.

What a running container takes in at creation is now part of its spec, by
content: every env-file, a file bind-mounted into it, and every file this
host wrote at or under a directory bind-mounted into it — the secrets,
bindings and configs under a module's state directories. The digest is the
one the store already records for a file the host wrote (`wrote`), read
from the state as it stands when the container is reached, so a file
rewritten earlier in the same apply is already the new one; a file the host
has no record of — an env-file a predecessor left, the superuser secret
genesis writes before any declaration names it — is read from disk, which
is what keeps adopting a running store in place a reconcile and not a
recreate.

Deliberately not part of it: what else is in a bind-mounted directory,
which is the service's own data and changes while it runs; a named volume;
a seed created once, which digests as the seed the host wrote and not as
what has grown in it; and a step — a run-once or scheduled container reads
its files when it runs and runs fresh each time. On an adopted node a held
container is held before any of this is looked at.

The host records what each container was created reading, per file, so
the recreate can say which file changed — "recreated: <file> changed" in
the report and, now with its detail, in the log. A container made before
this record existed is recreated once and says so.

novox/hq 04-ISSUES/103
2026-09-23 23:40:27 +02:00
jschoubben 977df39e0d Merge pull request 'Refuse a declaration for the other mode, or older than the mesh's last, and preview before applying (hq issue 104)' (#23) from fix/reconcile-refuses-stale into main 2026-09-23 21:38:33 +00:00
jschoubben f08a8ea3f7 Refuse every file once the mesh has spoken, plan the cutover as one, and let the kept declaration repair the mode
Review of the fix for hq issue 104 found three faults in it. A file applied
on an enrolled node — the mesh's own last declaration included — is applied
as the bundle is, so its resources are recorded as the machine's own and
what the mesh declared reads as undeclared: the plan removed the foundation.
`apply FILE` is for a machine the mesh has not spoken to, and is now refused
saying so whenever declared.json exists. The plan looked at what is held
before what the declaration says is taken, so the one cutover ADR 0100 says
must be previewed read as a hold; it now decides in holdOnAdopted's order,
models a step run inside a held container, and a test holds the plan's
sequence to the apply's outcomes. Genesis wrote the mode on every run, so a
re-run after `converge` left the state saying adopted while the kept,
signed declaration said converged, and the reconcile loop refused every five
minutes with no delivery coming to end it: genesis now writes the mode only
when none is recorded, and where the state and the verified kept declaration
disagree, the kept declaration wins and the repair is said.

Also: a file lock beside the state, taken by the link service, the host's
own commands and the installer alike, so a `reconcile` run by hand no
longer races the loop's save — chosen over refusing while a named service is
active, which would miss a `mesh-host run` started by hand; `--json
--dry-run` emits {plan} like an apply emits {plan, report}; the README's
duplicate flag line; and the bundle refusal is about the digest, not a claim
the carried bytes can never match what genesis applied.
2026-09-23 23:35:49 +02:00
jschoubben 7283924a35 Take over the found tunnel: its key, its port, its peers; stop it, never flush
On an adopted machine the private network takes the predecessor's tunnel
over in place (hq ADR 0105). Genesis finds the one interface up besides the
mesh's own, settles the hub's port and the mesh's range on it, and skips
ADR 0100's non-overlap check for a range that is now the tunnel's; a
--hub-port or --overlay-range that disagrees is refused naming the tunnel's.

At enrolment the found interface's private key becomes this node's overlay
key — the one credential the mesh takes rather than mints — stored where a
generated one is stored, never printed and never sent; the tunnel (port,
address, range, peers) travels with the keys so the mesh composes from it
before the first declaration.

The interface's service may say what it takes over. Before the mesh's unit
starts, the found configuration is kept like any held file and the found
unit is stopped and disabled; nothing is flushed, and an interface still up
after its unit stopped refuses the takeover rather than half-working. The
report says what was carried: interface, port, range, peer count, taken or
not, and where the original was kept.
2026-09-23 23:26:35 +02:00
jschoubben 27c4b765b2 Refuse a declaration for the other mode, or older than the mesh's last, and say what an apply would change first
An operator ran `mesh-host reconcile` on an adopted control-node with twelve
modules assigned. It applied the bundle the host carries — the genesis
declaration, foundation only, converged: recreated the store, failed on the
broker's held port, wrote the converged base filter and started its service,
and stopped at the first failing action. The filter closed the machine for
forty-five minutes. The host reported the node adopted in every report, the
declaration said converged, and nothing compared the two; nothing was printed
before acting (hq issue 104).

The host now records the node's mode — from every declaration the mesh sends,
and at genesis from what the operator said — and refuses, at the point of
application, a declaration that says the other mode, naming both and the act
that changes it. Only a declaration the link delivers, signed, changes the
mode: that is how `converge` and `adopt` arrive, so the flip still works and
nothing else can do it. Genesis marks the bundle consumed, with the digest of
what it applied, so `reconcile` holds a node the mesh has spoken to against
what the mesh last said and never the bundle, and refuses the carried bytes
when they are not what genesis applied. A file is refused when it is not what
the mesh last said: a declaration carries no sequence and no issued-at, so the
host cannot tell older from newer, and says so. Both commands print what they
would change — a hold, a removal, an action named as one — before touching
anything, and --dry-run is that list and nothing more.
2026-09-23 23:15:28 +02:00
jschoubben 9176aea6c4 Merge pull request 'The packages port given at genesis is a node setting, not manifest text (hq issue 085)' (#21) from feat/packages-port into main 2026-09-22 21:59:46 +02:00
jschoubben 744beb6c51 A re-run does not replace the registered forge with whatever checkout it was given
Registering a module is an overwrite. Recording the forge's port ran `module add`
every time, so a genesis re-run pointed at an older catalogue would replace the
manifest of a forge that is built and assigned — with a push a few lines later.
Registering is only here so a settings row has a module row to hang on, and that
row is already there on a mesh that knows the forge. So: ask first, and skip.

Two comments narrowed to what is true. What follows the node's setting is what
the mesh derives from a module's ports — its container mapping, its filter rule,
its opening and what it serves. The forge's own address in its runtime's
environment (hq 088) and its route contribution's port do not, and are already
wrong for any port the mesh assigned. And a settings layer is the module's, not
one resource's: a second mergeable file on the builder would be given `serves`
too.

novox/hq 04-ISSUES/085
2026-09-22 21:56:42 +02:00
jschoubben c4ce57997e The packages port given at genesis is a module's setting, like every other
Every foundation port given at genesis became a per-node setting of the module
that binds it, except the package registry's: that one was fixed by rewriting
the builder's manifest when the installer registered it. Registering the builder
again from the catalogue undid it, and the forge's own module, when it took the
bootstrap forge over, came up on the catalogue's port — which on a machine where
a predecessor holds 3000 points the builder at the predecessor's forge.

So the rewrite is gone, and the port is recorded twice as a setting, both from
the one input:

- the forge's module is registered at genesis — not assigned, nothing of it runs
  — so the controller has something to hold `{"ports": {"3000": <given>}}`
  against. Assigning the forge later raises it on the port this machine was
  given, and its container, its filter rule, its opening, what it serves and
  what consumers are told all read it from there.
- the builder is given `{"serves": {"port": <given>}}`, which merges into the
  binding it carries in place of one nothing can resolve yet.

A genesis on the catalogue's port records nothing and registers nothing, so it
does exactly what it did before.

novox/hq 04-ISSUES/085, ADR 0100
2026-09-22 21:40:02 +02:00
jschoubben 0db1fbdb3b Merge pull request 'Adoption mode: a node in use is adopted before it is converged (hq ADR 0100–0103)' (#20) from feat/adoption-mode into main 2026-09-22 21:01:46 +02:00
jschoubben 6dce63b534 Say plainly where the raised package registry runs, and why an action outside a container still runs 2026-09-22 20:01:26 +02:00
jschoubben c03a31cec5 Count only a record of making something at a path as the mesh's, not an access record (hq ADR 0103) 2026-09-22 20:01:14 +02:00
jschoubben 01e8affa89 Count an unasked report as said only once the broker has taken it (hq ADR 0100) 2026-09-22 20:00:59 +02:00
jschoubben b4c21b4f67 Read a machine's iptables rules when it has no nft, instead of calling it unfiltered (hq ADR 0100) 2026-09-22 19:59:22 +02:00
jschoubben c7ff0b9026 Refuse an endpoint whose port is not the one the private network's hub binds (hq ADR 0100) 2026-09-22 19:58:37 +02:00
jschoubben 04665d36c8 Exempt only the daemons a fresh machine was measured to run from counting as in use (hq ADR 0101) 2026-09-22 19:57:52 +02:00
jschoubben 7beb752be0 Take a key or list member the host may have written itself as the mesh's, so undeclaring gives the file back (hq ADR 0102) 2026-09-22 19:57:15 +02:00
jschoubben 0bd22e50f2 Apply one declaration at a time, so the link and the reconcile do not lose each other's record 2026-09-22 19:56:05 +02:00
jschoubben 9839006d48 Retire the found firewall only once the mesh's own filter is loaded on the machine (hq ADR 0100) 2026-09-22 19:54:43 +02:00
jschoubben 5f126021b7 Keep the originals the carried bundle writes over beside the node's state (hq ADR 0100) 2026-09-22 19:53:50 +02:00
jschoubben eb2f4fcf53 Keep an original by its path and its content, so a second original at one path is not discarded (hq ADR 0100) 2026-09-22 19:53:15 +02:00
jschoubben 232ed74940 gofmt the ufw rule's direction field 2026-09-22 19:52:40 +02:00
jschoubben 40e8ea9fda Hold a unit an administrator installed whatever its state, and a packaged unit only when the machine uses it (hq ADR 0103) 2026-09-22 19:52:37 +02:00
jschoubben d3f2595968 Read a ufw rule's direction: an outgoing rule answers no opening, and incoming is the default ufw merges on (hq ADR 0103) 2026-09-22 19:51:35 +02:00
jschoubben dc861fb4a8 Do not arm a scheduled step of a module held as found on an adopted node (hq ADR 0103) 2026-09-22 19:48:05 +02:00
jschoubben da008460ac Fail a resource when the container runtime cannot answer, instead of reading silence as nothing there (hq ADR 0100) 2026-09-22 19:46:56 +02:00
jschoubben 0ea646b384 Keep the derived filter in force when a guard resource failed on the way back to adopted (hq ADR 0103) 2026-09-22 19:45:33 +02:00
314 changed files with 55252 additions and 1264 deletions
+50 -14
View File
@@ -4,11 +4,16 @@ Tier 0 of the Novox Mesh. The one thing ever installed by hand, and the only thi
a machine.
```
scp mesh-host root@machine:/usr/local/bin/
mesh-host profile
scp nox-mesh-host root@machine:/usr/bin/
nox-mesh-host profile
```
That is the whole installation. One statically linked binary, nothing else present, no runtime
That is the whole installation. **It is a daemon**: the launcher runs it, and the operator's command is
`nox` (`mesh-cli`), on any node, from their own account (`novox/hq` ADR 0272). The subcommands below are
for installing a machine, joining it, and rescuing one no mesh can reach, run as root at the machine; a
line the mesh addresses to the operator names a `nox` line, never this binary. Once a delivered engine
proves itself, it renews the one placed by hand here to its own build, so the command on the machine is
the engine that runs it (`novox/hq` issue 356). One statically linked binary, nothing else present, no runtime
to install first ([`novox/hq` ADR 0005](https://git.novox.be/novox/hq)).
## What it is for
@@ -23,7 +28,7 @@ the host never queries the mesh database. It receives declarations and applies t
## Joining a mesh
```
mesh-host enrol --token <token> --name <what this machine is called>
nox-mesh-host enrol --token <token> --name <what this machine is called>
```
**The node generates its own identity** — an Ed25519 keypair whose private half never leaves the
@@ -52,19 +57,22 @@ and then stops, having saved nothing — so it can be run again unchanged.
connects to nothing and listens on nothing — what it applies comes from a file.
```
mesh-host profile what this machine can be asked to do
mesh-host inventory what this machine is, and what it holds
mesh-host apply FILE make this machine match a declaration from a file
mesh-host reconcile make this machine match the declaration this host carries
mesh-host bundle show what this host carries
mesh-host owned what this host has applied and still owns
nox-mesh-host profile what this machine can be asked to do
nox-mesh-host inventory what this machine is, and what it holds
nox-mesh-host apply FILE make this machine match a declaration from a file
nox-mesh-host reconcile make this machine match what the mesh last told it — or, before
any mesh has, the bundle this host carries
nox-mesh-host bundle show what this host carries
nox-mesh-host owned what this host has applied and still owns
nox-mesh-host hand-over DIR hand a directory used as found to the mesh, on a machine no mesh
can reach; the operator's line is `nox node hand-over <node> DIR`
--json machine-readable
--state where this node keeps what it knows
--dry-run read and check the declaration, change nothing
--dry-run say what applying would change, and change nothing
```
```
$ mesh-host profile
$ nox-mesh-host profile
linux/amd64
yes container-runtime 29.7.2
@@ -114,8 +122,16 @@ A host built for a machine carries its declaration **inside the binary**:
make host BUNDLE=path/to/foundation.lock
```
`mesh-host reconcile` then applies it. That is the first node's path — no mesh present, nothing
fetched, nothing else copied onto the machine. `copy it and run it` stops being true the moment
`nox-mesh-host reconcile` then applies it, on a machine the mesh has told nothing yet. That is the
first node's path — no mesh present, nothing fetched, nothing else copied onto the machine. Once
the mesh has spoken, `reconcile` holds the machine to what it last said and never to the bundle,
which genesis consumed; a bundle or a file is refused when it says the other mode than the node
is in; and `apply FILE` is refused altogether once the mesh has spoken — a file is applied as the
bundle is, its resources recorded as the machine's own, so on an enrolled node it would plan to
remove the foundation. `apply FILE` is for a machine the mesh has not spoken to. (hq's to-be
node lifecycle describes `apply repair.json` as a rescue on an enrolled node; that line is being
amended in hq, and no rescue path exists here yet.) Both commands say what they would change
before changing anything, and `--dry-run` is that alone. `copy it and run it` stops being true the moment
a second file has to arrive with it, which is why the bundle is embedded rather than beside it.
**A default build carries nothing and refuses to reconcile**, saying so. A host that applied
@@ -142,6 +158,26 @@ carries. The claim is not. The foundation is four container services, and:
So `foundation.lock` here is a real bundle with a placeholder's content. Saying that plainly
beats shipping a host that claims a foundation it has never raised.
## mesh-cli's socket (novox/hq ADR 0272)
The engine serves `/run/nox-mesh-host/cli.sock` for `mesh-cli`, the operator's command (`internal/meshcli`). It
reads the asking account from the kernel (`SO_PEERCRED`) and asks the controller on this node's own subject; the
controller decides what the line may do, and the engine runs nothing it receives. One line per connection, each
connection's deadline set the moment it is accepted, two lines at once per account and eight for the machine,
"busy" beyond that.
**Only a login session can be the terminal.** The engine reads the asking process's cgroup and sends its login
session's scope (`session-<id>.scope` under the account's own `user-<uid>.slice`, which logind's own record in
`/run/systemd/sessions/<id>` must say is this account's, of class `user`), or nothing, read the moment the
connection is accepted; the controller makes
nothing else the terminal. A service running as the operator — the tool runner, a unit of the account's user
manager — is at most an ordinary call.
**A process outside the machine's own namespaces is refused**, whatever its user id: the peer's mount and user
namespace must be the engine's. So `mesh-cli` run inside a flatpak, a toolbox or distrobox container, or any other
container — even one running as the operator's own id — is refused with the reason; run it from the machine's own
shell.
## A capability is detected, never assumed
The reason this is the first thing built rather than a detail of it.
+9 -3
View File
@@ -98,8 +98,8 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
everything this mesh will ever run
--source-path the module's directory inside that repository, if not its root
--catalog a checkout of the mesh's catalogue, holding the registry's, the
control plane's and the builder's manifests. Without it this stops
after step 6
control plane's, the builder's and — when a packages port is given
— the forge's manifests. Without it this stops after step 6
--node the name this machine is known by (default: its hostname)
--registry where this mesh keeps its own images (default ` + defaultRegistry + `)
every node pulls the control plane from this, so on a mesh of more
@@ -140,6 +140,10 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
firewall stay as they are, the foundation's filter is not loaded and
the mesh guards its own ports instead, and each module is taken on it
one at a time. Without it, a machine in use is refused
--tunnel adopted: the interface of the tunnel the private network takes over
(its key, port, range and peers); found by itself when one is up, and
needed only when several are. --hub-port and --overlay-range then
follow the tunnel
The installer carries a builder, not a control plane. What raises a mesh is therefore
the same thing that will maintain it, and the control plane a mesh ends up running is
@@ -279,7 +283,7 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
set.StringVar(&opts.Out, "out", opts.Out, "where the produced bundle is written")
set.StringVar(&opts.State, "state", opts.State, "where this node records what it has applied")
set.StringVar(&opts.Catalogue, "catalog", opts.Catalogue,
"a checkout of the mesh's catalogue, for the registry's and the builder's manifests and what phase two installs; without it this stops after the foundation")
"a checkout of the mesh's catalogue, for the registry's, the builder's and the forge's manifests and what phase two installs; without it this stops after the foundation")
set.StringVar(&opts.Source.Repository, "source", opts.Source.Repository,
"the repository the control plane is built from, on a mesh that already exists")
set.StringVar(&opts.Source.Ref, "source-ref", opts.Source.Ref,
@@ -334,6 +338,8 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
"raise this machine adopted: keep what it runs and its firewall until each module is taken")
set.StringVar(&opts.OverlayRange, "overlay-range", opts.OverlayRange,
"the private network's address range; must not overlap a tunnel the machine already runs")
set.StringVar(&opts.Tunnel, "tunnel", "",
"adopted: the found tunnel's interface the private network takes over; found by itself when one is up")
if opts.Answers == nil {
opts.Answers = map[string]string{}
}
+77
View File
@@ -0,0 +1,77 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
"path/filepath"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// **A declaration from an older lease epoch is refused before anything is applied** (novox/hq to-be
// 45 §6, ADR 0227 rule 2). The controller that sent it lost its lease and goes on sending; this node
// applied the holder's. Refused on the host's own apply path — the one the queue's worker runs — before
// the machine is read or touched, with a report naming what was refused and what is held, and the
// kept declaration left as it was.
func TestADeclarationFromAnOlderEpochIsRefusedBeforeAnythingIsApplied(t *testing.T) {
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
_, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "a.conf")
declared := func(epoch, sequence int64) store.Declared {
body, err := json.Marshal(map[string]any{"declaration": 1, "epoch": epoch, "sequence": sequence,
"resources": []any{map[string]any{"id": "a", "type": "file", "path": target, "content": "x\n"}}})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: ed25519.Sign(private, body)}
}
held := declared(57, 3)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), held); err != nil {
t.Fatal(err)
}
stale := declared(41, 12)
report := applyAndKeep(context.Background(), opts, stale.Declaration, &stale, nil, nil)
if report.OlderThan == nil || *report.OlderThan != (link.Order{Epoch: 57, Sequence: 3}) ||
report.Order != (link.Order{Epoch: 41, Sequence: 12}) || report.Declared != digestOf(stale.Declaration) ||
report.Refused == "" {
t.Fatalf("the refusal does not name what was refused and what is held: %+v", report)
}
if _, err := os.Stat(target); !errors.Is(err, os.ErrNotExist) {
t.Fatal("the refused declaration touched the machine")
}
kept, err := store.ReadDeclared(store.DeclaredPath(opts.state))
if err != nil || string(kept.Declaration) != string(held.Declaration) {
t.Fatal("the refused declaration replaced what this node kept")
}
}
// The queue's counts are kept beside the state and read back by the next host; unreadable is an
// error, never zero.
func TestTheNumbersSurviveTheHost(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
if sequence, refused, err := numbersBeside(state).Read(); err != nil || sequence != 0 || refused != 0 {
t.Fatalf("a node that never reported read %d, %d, %v", sequence, refused, err)
}
if err := numbersBeside(state).Save(41, 2); err != nil {
t.Fatal(err)
}
if sequence, refused, err := numbersBeside(state).Read(); err != nil || sequence != 41 || refused != 2 {
t.Fatalf("read back %d, %d, %v", sequence, refused, err)
}
if err := os.WriteFile(store.NumbersPath(state), []byte(`{"report_sequence":`), 0o600); err != nil {
t.Fatal(err)
}
if _, _, err := numbersBeside(state).Read(); err == nil {
t.Fatal("unreadable numbers were read as zero")
}
}
+213
View File
@@ -0,0 +1,213 @@
package main
import (
"errors"
"fmt"
"os"
"os/user"
"path/filepath"
"sort"
"strings"
"sync/atomic"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// A directory used as found is never quiet (novox/hq issue 339).
//
// The engine leaves the owner and mode of a directory it did not make: changing them would be root handing a
// directory it never made to whichever account a declaration named. That is safe and, said nowhere, a module
// whose data directory it cannot write would simply fail. So each such directory is said three ways: a WARNING
// line in every apply's log; a resource of its module of kind `directory`, stated unhealthy on every look,
// which the controller raises as that module's condition naming the directory and its owner; and a way out,
// which is the operator's line at the controller's terminal — `nox node hand-over <node> <path>` on the
// control-node (novox/hq ADR 0272, issue 356) — after which the next apply gives it the declared owner and
// mode like any directory of the mesh's. The controller carries that line to this engine over the bus
// (link.AskHandOverSubject); the engine's own `hand-over` subcommand is the same act for a machine with no
// mesh to reach.
// foundState is where the node's state is, for the looks to read the directories used as found from; empty
// until the engine runs.
var foundState atomic.Value
// foundNode is this node's name in the mesh, for the way out to name it; empty until the engine is enrolled.
var foundNode atomic.Value
// handedOverFrom has the apply read the directories handed over from beside the state at statePath.
func handedOverFrom(statePath string) {
apply.HandedOver = func() map[string]bool {
all, err := store.LoadHandedOver(statePath)
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: %v; no directory counts as handed over until it reads\n", err)
return nil
}
out := map[string]bool{}
for _, h := range all {
out[apply.HandOverKey(h.ID, h.Path)] = true
}
return out
}
}
// foundDirectories is every directory the node's state records as used as found, sorted by id.
func foundDirectories(known store.State) []store.Applied {
var out []store.Applied
for _, r := range known.Resources {
if r.Type == "directory" && r.AsFound {
out = append(out, r)
}
}
sort.Slice(out, func(a, b int) bool { return out[a].ID < out[b].ID })
return out
}
// handOverLine is the operator's line that hands a directory to the mesh, with this node's name when it is
// known (novox/hq ADR 0272): typed on the control-node, from the operator's own account. The directory's
// path is the resource's target beside the reason, never in it: the controller says a reason on the
// operator's channel, which withholds a path (ADR 0234 §6), and keeps the target as evidence.
func handOverLine() string {
node, _ := foundNode.Load().(string)
if node == "" {
node = "<node>"
}
return fmt.Sprintf("`nox node hand-over %s <directory>` on the control-node", node)
}
// withFoundDirectories adds to a statement every directory used as found, as its module's resource of kind
// directory, unhealthy: the reason names its owner and mode as they are now, what was declared, and the line
// that hands it over. The path is the target, which the controller keeps as evidence.
func withFoundDirectories(h *link.Health, found []store.Applied) *link.Health {
for _, r := range found {
at := strings.LastIndex(r.ID, ".")
if at <= 0 {
continue
}
owner, mode := "unknown", "unknown"
if info, err := os.Lstat(r.Target); err == nil {
mode = fmt.Sprintf("%o", info.Mode().Perm())
if uid, gid, ok := apply.OwnerOf(info); ok {
owner = fmt.Sprintf("%d:%d", uid, gid)
}
}
declared := r.Declared
if declared == "" {
declared = "another owner or mode"
}
h.Resources = append(h.Resources, link.ResourceHealth{Module: r.ID[:at], Resource: r.ID,
Kind: link.KindDirectory, Target: r.Target, State: link.StateUnhealthy,
Reason: fmt.Sprintf("%s owned by %s, mode %s, as found; %s was declared and not given it — %s, with its "+
"path, hands it to the mesh", link.ReasonUsedAsFound, owner, mode, declared, handOverLine()),
Since: foundSince(r).UTC()})
}
return h
}
// foundNow is the directories used as found, read from the node's state; none when it cannot be read, which
// the apply says on its own.
func foundNow() []store.Applied {
path, _ := foundState.Load().(string)
if path == "" {
return nil
}
known, err := store.Load(path)
if err != nil {
return nil
}
return foundDirectories(known)
}
// handOverFound hands one directory used as found to the mesh, on the operator's word carried by the
// controller (link.HandOverFunc) or typed at this machine. It changes nothing on disk itself; the next apply
// gives the directory its declared owner and mode, and says so.
//
// **Every value is judged, and a refusal records nothing.** The path is the directory's absolute path exactly
// as the engine states it — clean, so `..` and a doubled separator never name a directory; one this machine's
// state records as used as found; on disk a directory and not a link, reached through no link. The engine
// never follows a path it did not state: a hand-over is root giving a directory to an account at the next
// apply, and a path that resolves elsewhere would give that elsewhere.
func handOverFound(statePath, path, by string) (string, error) {
if !filepath.IsAbs(path) {
return "", fmt.Errorf("hand-over takes the directory's absolute path, and %q is not one; nothing was handed over", path)
}
if filepath.Clean(path) != path {
return "", fmt.Errorf("%q is not the directory's path as the engine states it (no `..`, no doubled or trailing "+
"separator); nothing was handed over", path)
}
known, err := store.Load(statePath)
if err != nil {
return "", err
}
var found *store.Applied
for _, r := range foundDirectories(known) {
if filepath.Clean(r.Target) == path {
r := r
found = &r
break
}
}
if found == nil {
return "", errors.New(path + " is not a directory this machine uses as found (the module's condition names " +
"the ones that are); nothing was handed over")
}
info, err := os.Lstat(path)
switch {
case err != nil:
return "", fmt.Errorf("%s cannot be read on this machine: %v; nothing was handed over", path, err)
case info.Mode()&os.ModeSymlink != 0:
return "", fmt.Errorf("%s is a link, and the mesh hands over no link (novox/hq ADR 0012); nothing was handed over", path)
case !info.IsDir():
return "", fmt.Errorf("%s is not a directory on this machine; nothing was handed over", path)
}
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return "", fmt.Errorf("%s cannot be resolved on this machine: %v; nothing was handed over", path, err)
}
if resolved != path {
return "", fmt.Errorf("%s is reached through a link (it is %s), and the mesh hands over nothing through one "+
"(novox/hq issue 339); nothing was handed over", path, resolved)
}
if err := store.AddHandedOver(statePath, store.HandOver{ID: found.ID, Path: path, At: time.Now().UTC(), By: by}); err != nil {
return "", err
}
return fmt.Sprintf("%s (%s) is handed to the mesh by %s: the next apply gives it %s, and it is the mesh's from "+
"then on — removed when empty and no longer declared", path, found.ID, by, orUnsaid(found.Declared)), nil
}
// handOverFrom is the engine's hand-over for the controller's ask: the state beside statePath, the path as asked.
func handOverFrom(statePath string) link.HandOverFunc {
return func(path, by string) (string, error) { return handOverFound(statePath, path, by) }
}
// handOverCommand is `nox-mesh-host hand-over <dir>` at this machine, as root: the same act, for a machine
// with no mesh to reach. The operator's line is `nox node hand-over <node> <dir>` on the control-node.
func handOverCommand(opts options) error {
by := "a person at the machine"
if u, err := user.Current(); err == nil {
by = u.Username + " at the machine"
}
said, err := handOverFound(opts.state, filepath.Clean(opts.file), by)
if err != nil {
return err
}
fmt.Fprintln(opts.out, said)
return nil
}
// foundSince is when a directory was first found as it is: kept across applies, so the controller can tell one
// a send found from one found before it. A record from before that was kept says when it was last applied.
func foundSince(r store.Applied) time.Time {
if !r.FoundAt.IsZero() {
return r.FoundAt
}
return r.AppliedAt
}
func orUnsaid(s string) string {
if s == "" {
return "the declared owner and mode"
}
return s
}
+177
View File
@@ -0,0 +1,177 @@
package main
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// Every directory used as found is its module's resource of kind directory, unhealthy, naming its owner, on
// every statement (novox/hq issue 339): the controller raises that as the module's condition. The way out it
// names is the operator's line, with this node's name (novox/hq ADR 0272, issue 356).
func TestAFoundDirectoryIsStatedUnhealthy(t *testing.T) {
dir := t.TempDir()
known := store.State{Resources: []store.Applied{
{ID: "notes.data", Type: "directory", Target: dir, AsFound: true, Declared: "root, mode 755", AppliedAt: time.Now()},
{ID: "notes.cache", Type: "directory", Target: dir},
}}
foundNode.Store("laptop")
t.Cleanup(func() { foundNode.Store("") })
h := withFoundDirectories(&link.Health{}, foundDirectories(known))
if len(h.Resources) != 1 {
t.Fatalf("stated: %+v", h.Resources)
}
r := h.Resources[0]
if r.Module != "notes" || r.Kind != link.KindDirectory || r.State != link.StateUnhealthy || r.Target != dir ||
!strings.HasPrefix(r.Reason, link.ReasonUsedAsFound) || !strings.Contains(r.Reason, "owned by ") ||
!strings.Contains(r.Reason, "root, mode 755") {
t.Fatalf("stated as: %+v", r)
}
// The path is the target, kept as evidence, and never in the reason: the operator's channel withholds a path
// (novox/hq ADR 0234 §6), and the controller says the reason there.
if !strings.Contains(r.Reason, "`nox node hand-over laptop <directory>` on the control-node") ||
strings.Contains(r.Reason, "mesh-host") || strings.Contains(r.Reason, dir) {
t.Fatalf("the way out is not the operator's line, or carries the path: %q", r.Reason)
}
foundNode.Store("")
if r := withFoundDirectories(&link.Health{}, foundDirectories(known)).Resources[0]; !strings.Contains(r.Reason,
"`nox node hand-over <node> <directory>`") {
t.Fatalf("before enrolment the line names a placeholder: %q", r.Reason)
}
}
// realDir is a temporary directory with every link in its path resolved, so a judgement of links is about the
// links the test makes.
func realDir(t *testing.T) string {
t.Helper()
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
return dir
}
// A hand-over records a directory used as found, and refuses one that is not — through the engine's own
// subcommand and through the controller's ask alike, which are one judgement.
func TestHandOverRecordsOnlyAFoundDirectory(t *testing.T) {
dir := realDir(t)
statePath := filepath.Join(dir, "state.json")
found := filepath.Join(dir, "found")
if err := os.Mkdir(found, 0o700); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(filepath.Join(dir, "mine"), 0o755); err != nil {
t.Fatal(err)
}
if err := store.Save(statePath, store.State{Resources: []store.Applied{
{ID: "notes.data", Type: "directory", Target: found, AsFound: true, Declared: "root, mode 755"},
{ID: "notes.mine", Type: "directory", Target: filepath.Join(dir, "mine")},
}}); err != nil {
t.Fatal(err)
}
var out bytes.Buffer
if err := handOverCommand(options{state: statePath, file: filepath.Join(dir, "mine"), out: &out}); err == nil {
t.Fatal("a directory the mesh made was handed over")
}
if err := handOverCommand(options{state: statePath, file: found + "/", out: &out}); err != nil {
t.Fatal(err)
}
all, err := store.LoadHandedOver(statePath)
if err != nil || len(all) != 1 || all[0].Path != found || !strings.HasSuffix(all[0].By, " at the machine") {
t.Fatalf("recorded: %+v %v", all, err)
}
handedOverFrom(statePath)
if !apply.HandedOver()[apply.HandOverKey("notes.data", found)] {
t.Fatal("the apply does not see the hand-over")
}
if !strings.Contains(out.String(), "root, mode 755") {
t.Fatalf("said: %s", out.String())
}
// The controller's ask is the same judgement, recording who asked in the controller's words.
_ = os.Remove(store.HandedOverPath(statePath))
said, err := handOverFrom(statePath)(found, "jo through mesh-cli on anchor")
if err != nil || !strings.Contains(said, "by jo through mesh-cli on anchor") || !strings.Contains(said, "root, mode 755") {
t.Fatalf("the ask answered %q, %v", said, err)
}
all, _ = store.LoadHandedOver(statePath)
if len(all) != 1 || all[0].By != "jo through mesh-cli on anchor" || all[0].ID != "notes.data" {
t.Fatalf("recorded: %+v", all)
}
}
// Every value of a hand-over is judged before anything is recorded, and each refusal records nothing: a relative
// path, a path that is not the engine's statement of the directory (`..`, a doubled or trailing separator), one
// not used as found, a link at the path, a link in the path, a file, a directory that is gone.
func TestAHandOverIsRefusedForEveryPathThatIsNotTheFoundDirectory(t *testing.T) {
dir := realDir(t)
statePath := filepath.Join(dir, "state.json")
found := filepath.Join(dir, "found")
if err := os.Mkdir(found, 0o700); err != nil {
t.Fatal(err)
}
// A link at the path of a second found directory, and a found directory below a linked directory.
linked := filepath.Join(dir, "linked")
if err := os.Symlink(found, linked); err != nil {
t.Fatal(err)
}
through := filepath.Join(dir, "via", "data")
if err := os.Symlink(dir, filepath.Join(dir, "via")); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(filepath.Join(dir, "data"), 0o700); err != nil {
t.Fatal(err)
}
file := filepath.Join(dir, "file")
if err := os.WriteFile(file, nil, 0o600); err != nil {
t.Fatal(err)
}
gone := filepath.Join(dir, "gone")
if err := store.Save(statePath, store.State{Resources: []store.Applied{
{ID: "notes.data", Type: "directory", Target: found, AsFound: true},
{ID: "notes.linked", Type: "directory", Target: linked, AsFound: true},
{ID: "notes.through", Type: "directory", Target: through, AsFound: true},
{ID: "notes.file", Type: "directory", Target: file, AsFound: true},
{ID: "notes.gone", Type: "directory", Target: gone, AsFound: true},
}}); err != nil {
t.Fatal(err)
}
// Each refused by its own check, which the reason names.
refused := map[string][2]string{
"a relative path": {"found", "not one"},
"a path with ..": {dir + "/x/../found", "as the engine states it"},
"a doubled separator": {dir + "//found", "as the engine states it"},
"a trailing separator": {found + "/", "as the engine states it"},
"a directory not used as found": {filepath.Join(dir, "other"), "not a directory this machine uses as found"},
"a link at the path": {linked, "is a link"},
"a link in the path": {through, "reached through a link"},
"a file": {file, "is not a directory on this machine"},
"a directory that is gone": {gone, "cannot be read on this machine"},
}
for name, c := range refused {
said, err := handOverFound(statePath, c[0], "jo")
if err == nil || said != "" {
t.Fatalf("%s (%s) was handed over: %q", name, c[0], said)
}
if !strings.Contains(err.Error(), "nothing was handed over") || !strings.Contains(err.Error(), c[1]) {
t.Fatalf("%s's refusal does not say %q and that nothing was handed over: %v", name, c[1], err)
}
}
if all, err := store.LoadHandedOver(statePath); err != nil || len(all) != 0 {
t.Fatalf("a refusal recorded something: %+v %v", all, err)
}
// The found directory itself, stated exactly, is taken.
if _, err := handOverFound(statePath, found, "jo"); err != nil {
t.Fatal(err)
}
if all, _ := store.LoadHandedOver(statePath); len(all) != 1 || all[0].Path != found {
t.Fatalf("recorded: %+v", all)
}
}
+40
View File
@@ -0,0 +1,40 @@
package main
import (
"context"
"testing"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// **The generation applied is kept, and a put-back never lowers it**: after a put-back of generation 5
// over 7, a send from generation 6 is still older than what this machine applied.
func TestThePutBackDoesNotLowerTheGenerationApplied(t *testing.T) {
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
f := machineThatApplied(t, 7)
newer := f.declared(t, map[string]any{"sequence": 9, "generation": 8}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, newer.Declaration, &newer, nil, nil); r.Refused != "" {
t.Fatalf("a newer generation was refused: %s", r.Refused)
}
back := f.declared(t, map[string]any{"sequence": 10, "generation": 5, "put_back": true}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, back.Declaration, &back, nil, nil); r.Refused != "" {
t.Fatalf("a put-back was refused: %s", r.Refused)
}
state, err := store.Load(f.opts.state)
if err != nil || state.Generation != 8 {
t.Fatalf("the generation applied is %d after a put-back of 5 over 8 (%v)", state.Generation, err)
}
stale := f.declared(t, map[string]any{"sequence": 11, "generation": 7}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, stale.Declaration, &stale, nil, nil); r.OlderGeneration == nil {
t.Fatalf("a send from generation 7 after 8 was applied because a put-back came between: %+v", r)
}
}
+170
View File
@@ -0,0 +1,170 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The assignment generation a declaration was composed from (novox/hq issue 234).
//
// On 2026-10-04 a declaration newer in sequence than every other, and composed from a view of the
// assignments older than the one this machine had applied, named four fewer modules — and the host,
// ordering by sequence alone, undeclared all four. The sequence orders arrival; it cannot tell a later
// send carrying earlier content. The generation can: the node-engine refuses a declaration composed from
// an older generation than the one it applied, before anything is read from the machine.
//
// These read what the report says through its JSON, as the controller does, so they run against a
// node-engine that has never heard of a generation and fail there for that reason.
// generationFixture is a machine that applied generation 7: a file the mesh declared, kept in its state,
// and the declaration it kept — signed — that named it.
type generationFixture struct {
opts options
private ed25519.PrivateKey
kept store.Declared
// stays is the file the older generation's declaration does not name.
stays string
}
func machineThatApplied(t *testing.T, generation int64) generationFixture {
t.Helper()
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
_, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
stays := filepath.Join(dir, "docker.conf")
if err := os.WriteFile(stays, []byte("kept\n"), 0o644); err != nil {
t.Fatal(err)
}
// The state as a node-engine that applied generation 7 keeps it: the file it owns, and the highest
// generation it applied. Written as JSON, so the fixture is the same bytes for any build under test.
state := map[string]any{"resources": []any{map[string]any{"id": "docker.conf", "type": "file",
"origin": store.OriginDeclared, "target": stays}}, "generation": generation}
raw, err := json.Marshal(state)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(opts.state, raw, 0o600); err != nil {
t.Fatal(err)
}
f := generationFixture{opts: opts, private: private, stays: stays}
f.kept = f.declared(t, map[string]any{"sequence": 3, "generation": generation}, "docker.conf", stays)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), f.kept); err != nil {
t.Fatal(err)
}
return f
}
// declared is a signed declaration with the envelope's keys given and one file.
func (f generationFixture) declared(t *testing.T, envelope map[string]any, id, path string) store.Declared {
t.Helper()
body := map[string]any{"declaration": 1,
"resources": []any{map[string]any{"id": id, "type": "file", "path": path, "content": "x\n"}}}
for k, v := range envelope {
body[k] = v
}
raw, err := json.Marshal(body)
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: raw, Signature: ed25519.Sign(f.private, raw)}
}
// reportJSON is a report as the controller reads it.
func reportJSON(t *testing.T, r link.Report) map[string]any {
t.Helper()
raw, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
var out map[string]any
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
return out
}
// **A newer sequence from an older generation is refused, and nothing is undeclared** (issue 234).
func TestANewerSequenceFromAnOlderGenerationIsRefusedAndNothingIsUndeclared(t *testing.T) {
f := machineThatApplied(t, 7)
other := filepath.Join(filepath.Dir(f.opts.state), "sudo.conf")
stale := f.declared(t, map[string]any{"sequence": 9, "generation": 6}, "sudo.conf", other)
report := applyAndKeep(context.Background(), f.opts, stale.Declaration, &stale, nil, nil)
said := reportJSON(t, report)
older, _ := said["older_generation"].(map[string]any)
if older == nil || older["generation"] != float64(6) || older["applied"] != float64(7) {
t.Fatalf("the refusal does not name the generation refused and the one applied: %v", said)
}
if report.Refused == "" || !strings.Contains(report.Refused, "generation 6") ||
!strings.Contains(report.Refused, "generation 7") {
t.Errorf("the refusal is not said in words naming both generations: %q", report.Refused)
}
if report.Declared != digestOf(stale.Declaration) || report.Sequence != 9 {
t.Errorf("the refusal does not name the declaration refused, so the mesh cannot name who sent it: %+v", report)
}
if _, err := os.Stat(f.stays); err != nil {
t.Fatalf("a refused declaration undeclared what the machine applied: %v", err)
}
if _, err := os.Stat(other); err == nil {
t.Error("the refused declaration touched the machine")
}
state, err := store.Load(f.opts.state)
if err != nil {
t.Fatal(err)
}
if _, owned := state.Find("docker.conf"); !owned {
t.Error("the refused declaration took a resource out of what this machine owns")
}
kept, err := store.ReadDeclared(store.DeclaredPath(f.opts.state))
if err != nil || string(kept.Declaration) != string(f.kept.Declaration) {
t.Error("the refused declaration replaced what this node kept, so the next reconcile would apply it")
}
}
// refusedForItsGeneration says a report refused its declaration for the generation it came from.
func refusedForItsGeneration(t *testing.T, r link.Report) bool {
t.Helper()
return reportJSON(t, r)["older_generation"] != nil || strings.Contains(r.Refused, "generation")
}
// refusedAsUnread says a report refused its declaration as one this node-engine cannot read.
func refusedAsUnread(r link.Report) bool { return strings.Contains(r.Refused, "not a declaration") }
// **A gate's put-back is applied, whatever generation it carries**: it carries the generation of what it
// puts back, and says it is a put-back (issue 234).
func TestAPutBackFromAnOlderGenerationIsNotRefused(t *testing.T) {
f := machineThatApplied(t, 7)
back := f.declared(t, map[string]any{"sequence": 9, "generation": 5, "put_back": true}, "docker.conf", f.stays)
report := applyAndKeep(context.Background(), f.opts, back.Declaration, &back, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a put-back was refused: %s", report.Refused)
}
}
// **A declaration that carries no generation is applied as it always was**: an older controller sends
// none, and a node-engine that refused it would strand every machine the moment the controller is older.
func TestADeclarationWithoutAGenerationIsNotRefusedForOne(t *testing.T) {
f := machineThatApplied(t, 7)
older := f.declared(t, map[string]any{"sequence": 9}, "docker.conf", f.stays)
report := applyAndKeep(context.Background(), f.opts, older.Declaration, &older, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a declaration without a generation was refused: %s", report.Refused)
}
// And the same generation again is no older.
same := f.declared(t, map[string]any{"sequence": 10, "generation": 7}, "docker.conf", f.stays)
report = applyAndKeep(context.Background(), f.opts, same.Declaration, &same, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a declaration of the generation applied was refused: %s", report.Refused)
}
}
+153
View File
@@ -0,0 +1,153 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/identity"
)
// Joining through the tunnel (novox/hq ADR 0169).
//
// **The bus is never open to the internet, so a joining machine reaches it over the tunnel.** It
// makes its tunnel key first and prints the public half; the token is issued for that key, and the
// hub is told the key before the token is shown; the token carries the one peer this machine needs.
// So the tunnel can come up before the mesh has said anything else — the circle ADR 0004 broke by
// carrying the bus's address in the token is broken here by carrying the hub's.
// tunnelConfigPath and tunnelUnit are where the mesh's own declaration puts the private network, so
// the first tunnel is the same interface and unit the mesh takes over, not a second one beside it.
var (
tunnelConfigPath = "/etc/wireguard/mesh0.conf"
tunnelUnit = "wg-quick@mesh0"
// lookPath finds WireGuard's tools; a variable so a test needs none installed.
lookPath = exec.LookPath
)
// keyCommand makes this machine's tunnel key, or reads the one it already made, and prints the
// public half: what the token is issued for. Making it twice would be a token issued for a key the
// machine no longer has, so an existing key is kept.
func keyCommand(opts options) error {
path := identity.OverlayKeyPath(opts.state)
if key, err := identity.LoadOverlayKey(path); err == nil {
fmt.Println(key.Public)
return nil
} else if !errors.Is(err, os.ErrNotExist) {
return err
}
if _, err := os.Stat(identity.Path(opts.state)); err == nil {
return fmt.Errorf("this machine has joined already (%s), and its tunnel key is its own; "+
"there is no key to make", identity.Path(opts.state))
}
key, err := identity.GenerateOverlayKey()
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return err
}
if err := os.WriteFile(path, []byte(key.Private+"\n"), 0o600); err != nil {
return fmt.Errorf("cannot write this machine's tunnel key: %w", err)
}
fmt.Println(key.Public)
fmt.Fprintln(os.Stderr, "\nthis machine's tunnel key, made here; the private half stays in "+path+".\n"+
"Issue the token for it — `token issue --new <name> --overlay-key <the line above>` — and enrol with that token.")
return nil
}
// tunnelKeyFor is the key a token through the tunnel was issued for, read from where `key` left it.
// Refused when there is none, or it is another: the hub knows only the key the token names.
func tunnelKeyFor(t *identity.TokenTunnel, state string) (identity.OverlayKey, error) {
path := identity.OverlayKeyPath(state)
key, err := identity.LoadOverlayKey(path)
if errors.Is(err, os.ErrNotExist) {
return identity.OverlayKey{}, fmt.Errorf("this token was issued for a tunnel key, and this " +
"machine has none: run `nox-mesh-host key` here first and issue the token for the key it prints")
}
if err != nil {
return identity.OverlayKey{}, err
}
if key.Public != t.Key {
return identity.OverlayKey{}, fmt.Errorf("this token was issued for the tunnel key %s, and this "+
"machine's is %s — it is another machine's token, or the key was made again; issue a new "+
"token for %s", t.Key, key.Public, key.Public)
}
return key, nil
}
// tunnelConfig is the first tunnel: this machine's address, and the hub as its one peer, reaching the
// whole private network through it. The private key is set from its file, as the mesh's own
// declaration does it, so the file holds no secret.
func tunnelConfig(t *identity.TokenTunnel, keyPath string) string {
return fmt.Sprintf(`# Written by nox-mesh-host enrol: the one peer a joining machine needs (novox/hq ADR 0169).
# The mesh's own declaration replaces this once the machine has joined.
[Interface]
Address = %s
PostUp = wg set %%i private-key %s
[Peer]
PublicKey = %s
Endpoint = %s
AllowedIPs = %s
PersistentKeepalive = 25
`, t.Address, keyPath, t.HubKey, t.HubEndpoint, t.Range)
}
// bringTheTunnelUp writes the first tunnel and starts it, so the bus the token names can be reached.
func bringTheTunnelUp(ctx context.Context, t *identity.TokenTunnel, keyPath string, run apply.Runner) error {
if _, err := lookPath("wg-quick"); err != nil {
return errors.New("joining through the tunnel needs WireGuard's tools on this machine " +
"(wireguard-tools), and wg-quick is not here")
}
if err := os.MkdirAll(filepath.Dir(tunnelConfigPath), 0o700); err != nil {
return err
}
if err := os.WriteFile(tunnelConfigPath, []byte(tunnelConfig(t, keyPath)), 0o600); err != nil {
return fmt.Errorf("cannot write the first tunnel: %w", err)
}
if out, err := run(ctx, "systemctl", "restart", tunnelUnit); err != nil {
return fmt.Errorf("the first tunnel would not start (%s): %v %s", tunnelUnit, err, strings.TrimSpace(out))
}
fmt.Printf("the tunnel to the hub is up: %s, through %s\n", t.Address, t.HubEndpoint)
return waitForTheHub(ctx, run, handshakeWithin)
}
// handshakeWithin is how long the hub has to answer the first tunnel. Issuing the token sent the hub
// this machine as a peer; the hub applies that on its own time, and a bus dialled before it has is a
// timeout that names the bus rather than the tunnel.
var handshakeWithin = 90 * time.Second
// waitForTheHub waits until the tunnel has shaken hands with the hub, so the bus is dialled over a
// tunnel that answers — and says so in the tunnel's own words when it does not.
func waitForTheHub(ctx context.Context, run apply.Runner, within time.Duration) error {
deadline := time.Now().Add(within)
for {
out, err := run(ctx, "wg", "show", "mesh0", "latest-handshakes")
if err == nil {
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
fields := strings.Fields(line)
if len(fields) == 2 && fields[1] != "0" {
fmt.Println("the hub answered the tunnel")
return nil
}
}
}
if time.Now().After(deadline) {
return fmt.Errorf("the hub has not answered the tunnel in %s: the token may be another machine's, "+
"the hub may not have been sent this machine as a peer, or its tunnel's port is not reachable "+
"from here", within)
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(2 * time.Second):
}
}
}
+139
View File
@@ -0,0 +1,139 @@
package main
import (
"context"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/identity"
)
// `key` makes the tunnel key once and prints its public half; asked again it prints the same one,
// because a token may already have been issued for it (novox/hq ADR 0169).
func TestKeyMakesTheTunnelKeyOnceAndKeepsIt(t *testing.T) {
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
first := captureStdout(t, func() {
if err := keyCommand(opts); err != nil {
t.Fatal(err)
}
})
second := captureStdout(t, func() {
if err := keyCommand(opts); err != nil {
t.Fatal(err)
}
})
if strings.TrimSpace(first) == "" || strings.TrimSpace(first) != strings.TrimSpace(second) {
t.Fatalf("the key changed between two asks: %q then %q", first, second)
}
info, err := os.Stat(identity.OverlayKeyPath(opts.state))
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm() != 0o600 {
t.Errorf("the private half is readable beyond root: %v", info.Mode().Perm())
}
}
// A token through the tunnel takes the key it was issued for, and says so when this machine has none
// or another.
func TestATokenThroughTheTunnelTakesItsOwnKey(t *testing.T) {
dir := t.TempDir()
state := filepath.Join(dir, "state.json")
tt := &identity.TokenTunnel{Key: "x", Address: "10.42.0.9/32", Range: "10.42.0.0/16", HubKey: "h", HubEndpoint: "198.51.100.1:51820"}
if _, err := tunnelKeyFor(tt, state); err == nil || !strings.Contains(err.Error(), "nox-mesh-host key") {
t.Fatalf("a machine with no key was not told to make one: %v", err)
}
captureStdout(t, func() { _ = keyCommand(options{state: state}) })
if _, err := tunnelKeyFor(tt, state); err == nil || !strings.Contains(err.Error(), "issued for the tunnel key x") {
t.Fatalf("another machine's token was taken: %v", err)
}
mine, _ := identity.LoadOverlayKey(identity.OverlayKeyPath(state))
tt.Key = mine.Public
if got, err := tunnelKeyFor(tt, state); err != nil || got.Public != mine.Public {
t.Fatalf("this machine's own token was refused: %v", err)
}
}
// The first tunnel is the mesh's interface and unit, with the hub as its one peer and no secret in
// the file — the same shape the mesh's declaration replaces it with.
func TestTheFirstTunnelIsTheMeshsInterfaceWithTheHubAsItsPeer(t *testing.T) {
dir := t.TempDir()
tunnelConfigPath = filepath.Join(dir, "wireguard", "mesh0.conf")
lookPath = func(string) (string, error) { return "/usr/bin/wg-quick", nil }
t.Cleanup(func() { tunnelConfigPath = "/etc/wireguard/mesh0.conf" })
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
if name == "wg" {
return "HUBKEY\t1759400000\n", nil
}
ran = append(ran, name+" "+strings.Join(args, " "))
return "", nil
}
tt := &identity.TokenTunnel{Key: "k", Address: "10.42.0.9/32", Range: "10.42.0.0/16", HubKey: "HUBKEY", HubEndpoint: "198.51.100.1:51820"}
captureStdout(t, func() {
if err := bringTheTunnelUp(context.Background(), tt, "/var/lib/mesh-host/overlay.key", run); err != nil {
t.Fatal(err)
}
})
raw, err := os.ReadFile(tunnelConfigPath)
if err != nil {
t.Fatal(err)
}
conf := string(raw)
for _, want := range []string{"Address = 10.42.0.9/32", "PostUp = wg set %i private-key /var/lib/mesh-host/overlay.key",
"PublicKey = HUBKEY", "Endpoint = 198.51.100.1:51820", "AllowedIPs = 10.42.0.0/16", "PersistentKeepalive = 25"} {
if !strings.Contains(conf, want) {
t.Errorf("the first tunnel lacks %q:\n%s", want, conf)
}
}
if strings.Contains(conf, "PrivateKey") {
t.Error("the first tunnel's file holds the private key")
}
if len(ran) != 1 || ran[0] != "systemctl restart wg-quick@mesh0" {
t.Errorf("the tunnel was started as %v", ran)
}
}
// captureStdout is what fn printed to standard output.
func captureStdout(t *testing.T, fn func()) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
fn()
os.Stdout = was
w.Close()
out, _ := io.ReadAll(r)
return string(out)
}
// The bus is dialled only once the hub has answered the tunnel, and a hub that never does is said
// as the tunnel's fault rather than the bus's.
func TestTheBusWaitsForTheHubToAnswer(t *testing.T) {
asked := 0
answersOnThird := func(_ context.Context, name string, args ...string) (string, error) {
asked++
if asked < 3 {
return "HUBKEY\t0\n", nil
}
return "HUBKEY\t1759400000\n", nil
}
captureStdout(t, func() {
if err := waitForTheHub(context.Background(), answersOnThird, time.Minute); err != nil {
t.Fatal(err)
}
})
never := func(context.Context, string, ...string) (string, error) { return "HUBKEY\t0\n", nil }
err := waitForTheHub(context.Background(), never, 0)
if err == nil || !strings.Contains(err.Error(), "has not answered the tunnel") {
t.Fatalf("a hub that never answered was not said: %v", err)
}
}
+44
View File
@@ -0,0 +1,44 @@
package main
import (
"errors"
"fmt"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// judgeDirectories judges every directory of the declaration this node keeps, read-only (novox/hq issue 358): which
// the walk that follows no link another account could make would refuse. Nothing is made, chmodded or chowned.
func judgeDirectories(statePath string) (string, error) {
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
if errors.Is(err, store.ErrNothingDeclared) {
return "", errors.New("this node keeps no declaration, so it is declared no directory")
}
if err != nil {
return "", err
}
d, err := declaration.Parse(kept.Declaration)
if err != nil {
return "", fmt.Errorf("the declaration this node keeps does not read: %w", err)
}
return apply.JudgeDirectories(d), nil
}
// judgeDirectoriesFrom is judgeDirectories as the controller asks it over the link.
func judgeDirectoriesFrom(statePath string) link.JudgeDirectoriesFunc {
return func(string) (string, error) { return judgeDirectories(statePath) }
}
// judgeDirectoriesCommand is `nox-mesh-host judge-directories` at this machine, as root: the same judgement, for a
// machine the controller cannot ask.
func judgeDirectoriesCommand(opts options) error {
said, err := judgeDirectories(opts.state)
if err != nil {
return err
}
fmt.Fprintln(opts.out, said)
return nil
}
+274
View File
@@ -0,0 +1,274 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"sync"
"sync/atomic"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// A library is judged on every look (novox/hq ADR 0328 §8, §9).
//
// Each library this machine is reached by is said as a resource `library:<name>` of kind library, with no module:
// healthy while it is kept (or, off its home, while its mount is there), waiting with an access wait of reason keep
// or disk, unknown while its filesystem could not be named, unhealthy otherwise. The root is the target, never in a
// reason nor in a wait's words: the controller says them on the operator's channel, which withholds a path (ADR
// 0234 §6). An access is judged by nothing, as before ADR 0325.
// libraryJudge reads every library on every look. Safe for the apply and the looking loop at once.
type libraryJudge struct {
mu sync.Mutex
libraries []declaration.Library
declared *declaration.Declaration
// was is each library's last verdict, with since when and how many looks in a row found it unhealthy.
was map[string]link.ResourceHealth
// reading guards inFlight: every library whose look has not returned yet.
reading sync.Mutex
inFlight map[string]bool
}
// accesses is the one library judge on this machine (its name from when it judged accesses).
var accesses = newLibraryJudge()
func newLibraryJudge() *libraryJudge {
return &libraryJudge{was: map[string]link.ResourceHealth{}, inFlight: map[string]bool{}}
}
// set is what the declaration just applied declares: every library. One no longer declared is forgotten.
func (j *libraryJudge) set(d *declaration.Declaration, _ map[string]bool) {
var ls []declaration.Library
if d != nil {
ls = append(ls, d.Libraries...)
}
sort.Slice(ls, func(a, b int) bool { return ls[a].Library < ls[b].Library })
j.mu.Lock()
defer j.mu.Unlock()
j.libraries, j.declared = ls, d
declared := map[string]bool{}
for _, l := range ls {
declared[apply.LibraryAccess(l.Library)] = true
}
for id := range j.was {
if !declared[id] {
delete(j.was, id)
}
}
}
// declaration is the declaration last set.
func (j *libraryJudge) declaration() *declaration.Declaration {
j.mu.Lock()
defer j.mu.Unlock()
return j.declared
}
// lookBound is how long a look waits for one library to answer. A hard network mount whose server is down blocks
// a stat for ever, and one library must never stop every health statement this machine makes.
var lookBound = 5 * time.Second
// verdict is one library's judgement in a look.
type verdict struct {
state, reason string
waits []link.AccessWait
act *link.AccessActRecord
}
// judgeBounded judges one library within lookBound, outside any lock. A look that has not answered is said so;
// while it still hangs, later looks start no other on the same library.
func (j *libraryJudge) judgeBounded(l declaration.Library) (verdict, bool) {
j.reading.Lock()
if j.inFlight[l.Library] {
j.reading.Unlock()
return verdict{}, false
}
j.inFlight[l.Library] = true
j.reading.Unlock()
done := make(chan verdict, 1)
go func() {
v := judgeLibrary(l)
j.reading.Lock()
delete(j.inFlight, l.Library)
j.reading.Unlock()
done <- v
}()
select {
case v := <-done:
return v, true
case <-time.After(lookBound):
return verdict{}, false
}
}
// judgeLibrary is one library's judgement now (ADR 0328 §7, §8, §9).
func judgeLibrary(l declaration.Library) verdict {
v := apply.JudgeLibrary(context.Background(), l)
access := apply.LibraryAccess(l.Library)
var out verdict
if a := v.Act; a != nil {
out.act = &link.AccessActRecord{Act: a.Act, Access: access, Warrant: a.Warrant, By: a.By, State: a.State,
Started: a.Started, Ended: a.Ended, Folders: a.Folders, Files: a.Files, Kept: a.Kept, Links: a.Links,
Other: a.Other, Mounts: a.Mounts, Shared: a.Shared, Linked: a.Linked, Stripped: a.Stripped, Why: a.Why}
}
switch v.Kind {
case apply.VerdictWaitKeep:
out.state, out.reason = link.StateWaiting, "its root holds data the mesh has not taken on the filesystem "+
"it is on now, and waits for the operator's word (novox/hq ADR 0328 §7, §8)"
out.waits = []link.AccessWait{{Kind: link.WaitKindAccess, Access: access, Part: l.Library,
What: apply.KeepWords(l.Library), Placed: v.Placed, Reason: link.WaitKeep}}
case apply.VerdictWaitDisk:
out.state, out.reason = link.StateWaiting, v.Why+" (novox/hq ADR 0328 §9)"
out.waits = []link.AccessWait{{Kind: link.WaitKindAccess, Access: access, Part: l.Library,
What: apply.DiskWords(l.Library), Reason: link.WaitDisk}}
case apply.VerdictUnknown:
// Not a fault and not the operator's: said as not known, and judged again on the next look.
out.state, out.reason = link.StateUnknown, v.Why+"; judged again on the next look"
case apply.VerdictBroken:
out.state, out.reason = link.StateUnhealthy, v.Why
default:
out.state = link.StateHealthy
switch {
case v.Act != nil && v.Act.State == link.ActRunning:
out.state, out.reason = link.StateStarting, "the take act is setting its group under its root"
case v.Act != nil && (v.Act.State == link.ActStopped || v.Act.State == link.ActFailed):
out.state, out.reason = link.StateStarting, "the take act goes on from where it stopped: "+v.Act.Why
}
}
return out
}
// look reads every library now. Whether any verdict changed since the last look, for the loop to say it.
func (j *libraryJudge) look(now time.Time) ([]link.ResourceHealth, bool) {
j.mu.Lock()
ls := append([]declaration.Library(nil), j.libraries...)
j.mu.Unlock()
read := make([]verdict, len(ls))
for i, l := range ls {
v, answered := j.judgeBounded(l)
if !answered {
v = verdict{state: link.StateUnknown, reason: "its root does not answer: judging it did not return " +
"within " + lookBound.String()}
}
read[i] = v
}
j.mu.Lock()
defer j.mu.Unlock()
changed := false
out := make([]link.ResourceHealth, 0, len(ls))
for i, l := range ls {
r := read[i]
id := apply.LibraryAccess(l.Library)
v := link.ResourceHealth{Resource: id, Kind: link.KindLibrary, Target: l.Root, State: r.state,
Reason: r.reason, Since: now.UTC(), AccessWaits: r.waits, Act: r.act}
if was, had := j.was[id]; had && was.State == r.state && was.Reason == r.reason {
v.Since = was.Since
v.Streak = was.Streak
} else {
changed = true
}
if r.state == link.StateUnhealthy {
v.Streak++
}
j.was[id] = v
out = append(out, v)
}
return out, changed
}
// withAccesses adds to a statement every library, as a resource of kind library.
func withAccesses(h *link.Health, vs []link.ResourceHealth) *link.Health {
h.Resources = append(h.Resources, vs...)
return h
}
// accessesHealthy says no library in vs is anything but healthy.
func accessesHealthy(vs []link.ResourceHealth) bool {
for _, v := range vs {
if v.State != link.StateHealthy {
return false
}
}
return true
}
// keeper goes on with take acts stopped at their bound, beside the applies, one at a time (ADR 0328 §7).
type keeper struct {
running atomic.Bool
say func(string)
}
// keeping is the one keeper on this machine.
var keeping = &keeper{say: func(string) {}}
// poke goes on with every stopped take act of the declaration last applied, unless that is running already.
func (k *keeper) poke(ctx context.Context) {
d := accesses.declaration()
if d == nil || len(d.Libraries) == 0 || !k.running.CompareAndSwap(false, true) {
return
}
go func() {
defer k.running.Store(false)
for _, line := range apply.ReconcileLibraries(ctx, d) {
k.say(line)
}
}()
}
// startKeeping has the keeper say what it does.
func startKeeping(ctx context.Context, say func(string)) {
keeping.say = say
keeping.poke(ctx)
}
// accessActFrom is the take act as the controller asks it over the link (ADR 0328 §7): judged against the
// declaration this node keeps, and run beside the applies.
func accessActFrom(statePath string, timeout time.Duration) link.AccessActFunc {
return func(act link.AccessAct) (string, error) {
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
if errors.Is(err, store.ErrNothingDeclared) {
return "", errors.New("this node keeps no declaration, so it is declared no library")
}
if err != nil {
return "", err
}
d, err := declaration.Parse(kept.Declaration)
if err != nil {
return "", fmt.Errorf("the declaration this node keeps does not read: %w", err)
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
_, said, err := apply.TakeLibrary(ctx, d, act.Access, filepath.Clean(act.Path), act.Warrant, act.By)
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: the take act on %s was refused: %v\n", act.Access, err)
return "", err
}
return said, nil
}
}
// withWaiting is held with every id of waiting beside it: what liveness does not judge.
func withWaiting(held, waiting map[string]bool) map[string]bool {
if len(waiting) == 0 {
return held
}
out := map[string]bool{}
for id := range held {
out[id] = true
}
for id := range waiting {
out[id] = true
}
return out
}
+56
View File
@@ -0,0 +1,56 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
)
// A library whose root holds data never taken is said waiting with one access wait: the access `library:<name>`, the
// part its name, words with no path, the filesystem's name, and reason keep (novox/hq ADR 0328 §8); one whose root
// is absent waits with reason disk (§9). Neither carries a module.
func TestALibrarySaysItsWaitWithoutWordsToParse(t *testing.T) {
was := apply.Libraries
apply.Libraries = apply.NewLibraryMemory()
defer func() { apply.Libraries = was }()
dir := t.TempDir()
root := filepath.Join(dir, "media")
if err := os.MkdirAll(filepath.Join(root, "a film"), 0o755); err != nil {
t.Fatal(err)
}
d, err := declaration.Parse([]byte(`{"declaration":1,"groups":[{"name":"mesh-media","gid":62000},` +
`{"name":"mesh-books","gid":62001}],"libraries":[` +
`{"library":"media","root":"` + root + `","group":"mesh-media","gid":62000,"home":true},` +
`{"library":"books","root":"` + filepath.Join(dir, "books") + `","group":"mesh-books","gid":62001,"home":true}],` +
`"owns_nothing":true,"resources":[]}`))
if err != nil {
t.Fatal(err)
}
j := newLibraryJudge()
j.set(d, nil)
vs, _ := j.look(time.Now())
if len(vs) != 2 {
t.Fatalf("the libraries are not said: %+v", vs)
}
books, media := vs[0], vs[1]
if media.Kind != link.KindLibrary || media.Resource != "library:media" || media.Module != "" ||
media.State != link.StateWaiting || len(media.AccessWaits) != 1 {
t.Fatalf("a library holding data is not said waiting: %+v", media)
}
raw, _ := json.Marshal(media.AccessWaits[0])
want := `{"kind":"access","access":"library:media","part":"media","what":"let the mesh keep the library media",` +
`"placed":"` + media.AccessWaits[0].Placed + `","reason":"keep"}`
if string(raw) != want || !strings.HasPrefix(media.AccessWaits[0].Placed, "fs-") {
t.Fatalf("the wait reads %s", raw)
}
if strings.Contains(media.Reason, dir) || len(books.AccessWaits) != 1 || books.AccessWaits[0].Reason != link.WaitDisk {
t.Fatalf("a missing root does not wait for its disk, or a reason names a path: %+v %+v", books, media)
}
}
+107
View File
@@ -0,0 +1,107 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/link"
)
// Defends novox/hq issue 160: a machine nobody pushed to still says what it is made of once a day, so the
// newest of it the mesh holds is no older than a day. Before, the reconcile said it only with something
// else (a hold, a firewall, an outward link changed), and a converged machine's picture was as old as the
// last delivery to it.
func TestAMachineNobodyPushedToSaysWhatItIsOnceADay(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
quiet := link.Report{Outward: []string{"wlan0"},
Profile: map[string]any{"machine": map[string]any{"observed_at": at}}}
if !w.news(quiet) {
t.Fatal("the first reconcile after the engine started did not say what the machine is")
}
w.said(quiet)
for _, after := range []time.Duration{5 * time.Minute, 12 * time.Hour, 22*time.Hour + 55*time.Minute} {
at = time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC).Add(after)
if w.news(quiet) {
t.Errorf("%s after the mesh heard it, an unchanged reconcile was said again", after)
}
}
// The reconcile that falls past 23 hours says it, so it reaches the mesh inside the day.
at = time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC).Add(23*time.Hour + 5*time.Minute)
if !w.news(quiet) {
t.Fatal("a day on, the reconcile did not say what the machine is")
}
w.said(quiet)
at = at.Add(5 * time.Minute)
if w.news(quiet) {
t.Error("once said, it was said again at the next reconcile")
}
}
// A delivery's account carries the profile too: heard, it starts the day again.
func TestADeliveryHeardStartsTheDayAgain(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
report := link.Report{Outward: []string{"eth0"}, Profile: map[string]any{"capabilities": []any{}}}
w.said(report)
at = at.Add(20 * time.Hour)
w.said(report) // a delivery's apply
at = at.Add(20 * time.Hour)
if w.news(report) {
t.Error("said again unasked 20 hours after a delivery told the mesh")
}
}
// A refusal carries nothing about the machine, so it is never said for being a day old.
func TestARefusalIsNotSaidForTheMachinesSake(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
if w.machineDue(link.Report{Refused: "no"}) {
t.Error("a refusal was due for what the machine is made of")
}
}
// A laptop that slept still says what it is within a day of wall time (review of mesh-host#92). Go's monotonic
// clock stops while a machine is suspended, and Sub between two times that both carry a monotonic reading uses
// it, so a watch that kept one would count a day asleep as almost nothing. Simulated with a clock that first
// reads now (a monotonic reading), and after the sleep reads the wall time a day on: the watch must keep no
// monotonic reading, and judge the day as the wall clock says it.
func TestALaptopThatSleptStillSaysWhatItIsWithinADay(t *testing.T) {
before := time.Now() // carries a monotonic reading
asleep := before.Add(3 * time.Second)
if asleep.Sub(before) != 3*time.Second {
t.Fatal("the monotonic reading is not what this test simulates with")
}
calls := 0
w := &adoptionWatch{now: func() time.Time {
calls++
if calls == 1 {
return before
}
// The wall clock a day on, as a machine's clock reads after waking.
return asleep.Round(0).Add(24 * time.Hour)
}}
report := link.Report{Outward: []string{"wlan0"}, Profile: map[string]any{"capabilities": []any{}}}
w.said(report)
if strings.Contains(w.machineSaid.String(), "m=") {
t.Fatalf("the watch kept a monotonic reading (%s): a sleep would not count toward the day", w.machineSaid)
}
if !w.news(report) {
t.Error("a day of wall time on, a laptop that slept did not say what it is")
}
}
// And the watch's own clock never carries a monotonic reading.
func TestTheWatchClockIsWallTime(t *testing.T) {
w := &adoptionWatch{}
if s := w.clock().String(); strings.Contains(s, "m=") {
t.Errorf("the watch's clock carries a monotonic reading: %s", s)
}
w = &adoptionWatch{now: time.Now}
if s := w.clock().String(); strings.Contains(s, "m=") {
t.Errorf("a given clock's time kept its monotonic reading: %s", s)
}
}
+1644 -129
View File
File diff suppressed because it is too large Load Diff
+487 -6
View File
@@ -1,11 +1,24 @@
package main
import (
"bytes"
"context"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/bundle"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// Argument handling gets tests because it already failed silently once: `mesh-host inventory
@@ -158,15 +171,33 @@ func TestAReconcileSpeaksOnlyWhenWhatIsHeldChanged(t *testing.T) {
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a changed firewall was not said")
}
// What filters the machine is part of it (novox/hq ADR 0168): a predecessor's chain removed by
// hand, or the found firewall enabled again, is said without being asked.
filtered := link.Report{Firewall: "none", Held: rewritten.Held,
Filters: []link.Filter{{Where: "chain HAL-MESH-ONLY (iptables-legacy)", Owner: "other", Refuses: "-j DROP"}}}
if !w.changed(filtered) {
t.Error("a filter appearing was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a filter removed was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held, FoundFirewall: &link.FoundFirewall{Kind: "ufw", Active: true}}) {
t.Error("the found firewall coming back was not said")
}
if !worthSaying(link.Report{Filters: filtered.Filters}) {
t.Error("a report carrying only what filters the machine is not worth saying")
}
}
func TestWhatTheLinkPublishedCountsAsSaid(t *testing.T) {
// What the mesh heard — a delivery's account as much as a reconcile's — counts as said: the queue
// tells the watch every report the broker took (link.Queue.Heard).
func TestWhatTheMeshHeardCountsAsSaid(t *testing.T) {
w := &adoptionWatch{}
report := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "a"}}}
applier := w.noting(func(context.Context, []byte, []byte) link.Report { return report })
applier(context.Background(), nil, nil)
heard := w.said
heard(report)
if w.changed(report) {
t.Error("a reconcile repeated what the link had just published")
t.Error("a reconcile repeated what the mesh had just heard")
}
}
@@ -190,3 +221,453 @@ func TestAReconcileSpeaksWhenWhatIsReachableChanged(t *testing.T) {
t.Error("a newly published port was not said")
}
}
// Defends the node's own record: the link and the reconcile loop both apply, and each reads the
// state, acts, and writes it back — so they must not run at the same time, or the last save loses
// what the other recorded.
func TestOnlyOneApplyRunsAtATime(t *testing.T) {
// A host is built for one system at link time, and a test binary has no link time: this asks
// the machine it runs on, and stands aside where the answer is no.
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
raw := []byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"` +
filepath.Join(dir, "a.conf") + `","content":"x\n"}]}`)
// Whatever else is applying — the link, while this is the reconcile — this waits for it.
applying.Lock()
done := make(chan link.Report, 1)
go func() { done <- applyAndKeep(context.Background(), opts, raw, nil, nil, nil) }()
select {
case report := <-done:
applying.Unlock()
t.Fatalf("an apply ran while another held the node: %+v", report)
case <-time.After(50 * time.Millisecond):
}
if _, err := os.Stat(filepath.Join(dir, "a.conf")); !errors.Is(err, os.ErrNotExist) {
applying.Unlock()
t.Fatal("the waiting apply had already touched the machine")
}
applying.Unlock()
select {
case report := <-done:
if report.Refused != "" {
t.Fatalf("refused: %s", report.Refused)
}
case <-time.After(10 * time.Second):
t.Fatal("the apply never ran once the node was free")
}
known, loadErr := store.Load(opts.state)
if loadErr != nil || len(known.Resources) != 1 {
t.Errorf("the apply recorded %d resource(s): %v", len(known.Resources), loadErr)
}
}
// Defends novox/hq ADR 0100: a change is counted as said only once the mesh has been told. Queued
// and lost — the link down when the reconcile spoke — it must be said again.
func TestAChangeThatNeverReachedTheMeshIsSaidAgain(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}}}
if !w.differs(held) {
t.Fatal("the first report of a change was not new")
}
// The link was down: nothing published it, so nothing says it was said.
if !w.differs(held) {
t.Error("a change that never reached the mesh was counted as said")
}
w.said(held)
if w.differs(held) {
t.Error("a change the mesh was told was said again")
}
}
// Defends novox/hq issue 104: a declaration for the other mode than this node is in is refused at
// the point of application, whichever command delivered it, naming both — an adopted control-node
// once applied its converged genesis bundle and closed itself for forty-five minutes.
func stateWithMode(t *testing.T, mode string) options {
t.Helper()
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json"), out: &bytes.Buffer{}}
if err := store.Save(opts.state, store.State{Mode: mode}); err != nil {
t.Fatal(err)
}
return opts
}
func TestAConvergedDeclarationIsRefusedOnAnAdoptedNode(t *testing.T) {
opts := stateWithMode(t, store.ModeAdopted)
path := filepath.Join(filepath.Dir(opts.state), "filter.conf")
raw := []byte(`{"declaration":1,"resources":[{"id":"filter","type":"file","path":"` + path +
`","content":"table inet filter { chain input { policy drop; } }\n"}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
for _, from := range []provenance{fromFile, fromBundle} {
err := runApply(context.Background(), opts, d, raw, from)
if err == nil {
t.Fatalf("a converged declaration was applied to an adopted node (from %d)", from)
}
want := "this node is adopted; the declaration says converged"
if !strings.Contains(err.Error(), want) || !strings.Contains(err.Error(), "`converge`") {
t.Errorf("the refusal does not name both modes and the act that changes it: %v", err)
}
}
if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) {
t.Error("the refused declaration touched the machine")
}
}
func TestTheKeptDeclarationRepairsARecordThatDisagrees(t *testing.T) {
// What a node kept is signed by the mesh and verified on load; the state's note of the mode is
// the host's own. A genesis re-run after `converge`, or a state saved when the kept declaration
// could not be, leaves them apart — and a loop that refused every five minutes would hold the
// node off what the mesh said until a delivery that comes only when something changes. The
// kept declaration wins, and the repair is said.
opts := stateWithMode(t, store.ModeConverged)
raw := []byte(`{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"` +
filepath.Join(filepath.Dir(opts.state), "a.conf") + `","content":"x\n"}]}`)
var said []string
report := applyAndKeep(context.Background(), opts, raw, nil, nil, func(line string) { said = append(said, line) })
if strings.Contains(report.Refused, "the declaration says") {
t.Fatalf("what the node kept was refused against its own note: %s", report.Refused)
}
if len(said) != 1 || !strings.Contains(said[0], "record said converged") ||
!strings.Contains(said[0], "says adopted") || !strings.Contains(said[0], "repaired") {
t.Errorf("the repair was not said: %q", said)
}
}
func TestTheMeshItselfMayChangeTheMode(t *testing.T) {
// The flip is a declaration: `converge` on the controller records the mode and sends the
// first converged declaration. Delivered by the link, signed, it is not held to the record —
// it becomes it. (With no system linked in, the apply is refused later for that; what this
// checks is that the refusal is not the mode's.)
opts := stateWithMode(t, store.ModeAdopted)
raw := []byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"` +
filepath.Join(filepath.Dir(opts.state), "a.conf") + `","content":"x\n"}]}`)
report := applyAndKeep(context.Background(), opts, raw, &store.Declared{Declaration: raw}, nil, nil)
if strings.Contains(report.Refused, "the declaration says") {
t.Errorf("the mesh's own flip was refused for its mode: %s", report.Refused)
}
}
// Defends novox/hq issue 104: a declaration older than what the mesh last said is refused, naming
// both — and `apply FILE` is refused altogether once the mesh has spoken, the last declaration
// itself included: from a file it is applied as the bundle is, which would record the mesh's
// resources as this machine's own and remove the foundation as undeclared.
func TestAnOlderDeclarationIsRefused(t *testing.T) {
opts := stateWithMode(t, "")
genesis := []byte(`{"declaration":1,"resources":[{"id":"g","type":"file","path":"/tmp/g","content":"genesis\n"}]}`)
since := []byte(`{"declaration":1,"resources":[{"id":"g","type":"file","path":"/tmp/g","content":"since\n"}]}`)
other := []byte(`{"declaration":1,"resources":[{"id":"g","type":"file","path":"/tmp/g","content":"other\n"}]}`)
if err := store.Save(opts.state, store.State{Genesis: &store.Genesis{Digest: apply.DigestOf(genesis),
At: time.Now(), Rewritten: true}}); err != nil {
t.Fatal(err)
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state), store.Declared{Declaration: since,
Signature: []byte("unverified here")}); err != nil {
t.Fatal(err)
}
parsed := func(raw []byte) *declaration.Declaration {
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
return d
}
err := runApply(context.Background(), opts, parsed(genesis), genesis, fromFile)
if err == nil {
t.Fatal("the bundle genesis consumed was applied over what the mesh said since")
}
for _, want := range []string{"older", short(apply.DigestOf(genesis)), short(apply.DigestOf(since))} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
err = runApply(context.Background(), opts, parsed(other), other, fromFile)
if err == nil {
t.Fatal("a declaration that is not what the mesh last said was applied")
}
if !strings.Contains(err.Error(), "for a machine the mesh has not spoken to") ||
!strings.Contains(err.Error(), short(apply.DigestOf(since))) {
t.Errorf("the refusal does not say what a file is for and what was last said: %v", err)
}
// The very declaration the mesh last sent, from a file: still a file.
err = runApply(context.Background(), opts, parsed(since), since, fromFile)
if err == nil || !strings.Contains(err.Error(), "applied as the bundle is") {
t.Errorf("the last declaration, from a file, was not refused as a file: %v", err)
}
if known, _ := store.Load(opts.state); len(known.Resources) != 0 {
t.Errorf("a refused file recorded %d resource(s)", len(known.Resources))
}
// A bundle other than the one genesis consumed: what genesis applied was rewritten for this
// machine, so the carried bytes never are.
err = runApply(context.Background(), opts, parsed(other), other, fromBundle)
if err == nil || !strings.Contains(err.Error(), "consumed") ||
!strings.Contains(err.Error(), short(apply.DigestOf(genesis))) {
t.Errorf("a bundle other than the consumed one was not refused naming it: %v", err)
}
}
// Defends novox/hq issue 104: what an apply would change is said before anything is, and
// `--dry-run` is that and nothing else — an action listed as the action it is.
func TestADryRunChangesNothingAndListsTheActions(t *testing.T) {
opts := stateWithMode(t, "")
opts.dryRun = true
out := &bytes.Buffer{}
opts.out = out
path := filepath.Join(filepath.Dir(opts.state), "a.conf")
raw := []byte(`{"declaration":1,"resources":[
{"id":"a","type":"file","path":"` + path + `","content":"x\n"},
{"id":"init","type":"action","command":["createdb","mesh"],"verify":["psql","-c","select 1"]}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
if err := runApply(context.Background(), opts, d, raw, fromFile); err != nil {
t.Fatalf("a dry run failed: %v", err)
}
if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) {
t.Error("a dry run wrote the file")
}
for _, want := range []string{"would change", "create file a", "run action init",
"`createdb mesh`", "--dry-run: nothing applied"} {
if !strings.Contains(out.String(), want) {
t.Errorf("the preview does not say %q:\n%s", want, out.String())
}
}
if strings.Contains(out.String(), "applying:") {
t.Errorf("a dry run went on to apply:\n%s", out.String())
}
// Machine-readable, the same shape as an apply's: the plan, and no report.
out.Reset()
opts.json = true
if err := runApply(context.Background(), opts, d, raw, fromFile); err != nil {
t.Fatal(err)
}
var shape struct {
Plan []apply.Step `json:"plan"`
Report *json.RawMessage `json:"report"`
}
if err := json.Unmarshal(out.Bytes(), &shape); err != nil || len(shape.Plan) != 2 || shape.Report != nil {
t.Errorf("a json dry run is not {plan} alone: %v\n%s", err, out.String())
}
}
// Defends novox/hq issue 104: once the mesh has told this node anything, `reconcile` holds it to
// that — never to the bundle the host carries, which genesis consumed.
func TestReconcileAfterAControllerDeclarationDoesNotReapplyTheBundle(t *testing.T) {
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
opts := stateWithMode(t, store.ModeAdopted)
controllerPublic, controllerPrivate, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
said := []byte(`{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), store.Declared{Declaration: said,
Signature: ed25519.Sign(controllerPrivate, said)}); err != nil {
t.Fatal(err)
}
// Told, and unable to prove by whom: refused, and the bundle is not applied in its place.
_, _, _, err = reconcileSource(opts)
if err == nil || !strings.Contains(err.Error(), "not applied in its place") {
t.Errorf("a node that cannot prove what it was told fell back to something: %v", err)
}
mine, err := identity.Generate("workstation")
if err != nil {
t.Fatal(err)
}
mine.Membership = identity.Membership{Broker: "198.51.100.10:5671", Fingerprint: "sha256:0",
Signer: controllerPublic, Password: "issued"}
if err := identity.Save(identity.Path(opts.state), mine); err != nil {
t.Fatal(err)
}
d, raw, from, err := reconcileSource(opts)
if err != nil {
t.Fatal(err)
}
if from != fromDeclared || !bytes.Equal(raw, said) || d.Adoption == nil {
t.Errorf("reconcile chose %d with %d bytes, not what the mesh last said", from, len(raw))
}
// And before the mesh has said anything: the bundle, as it always was. A test binary carries
// only the placeholder, and asking for it is what proves the path.
if err := os.Remove(store.DeclaredPath(opts.state)); err != nil {
t.Fatal(err)
}
_, _, _, err = reconcileSource(opts)
if !errors.Is(err, bundle.ErrEmpty) {
t.Errorf("with nothing said, reconcile did not reach for the carried bundle: %v", err)
}
}
// **A machine says which links face outside without being asked.**
//
// The mesh composes no filter for a machine that has not said (novox/hq ADR 0140), and a machine only
// speaks unasked when this fingerprint changes. Left out of it, a machine that has just learnt to say
// could speak only when a declaration arrived — and a declaration cannot be composed until it has
// spoken. A machine waiting for a push that is waiting for the machine.
func TestANewOutwardLinkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
first := link.Report{Firewall: "none"}
if !w.differs(first) {
t.Fatal("the first report should differ from nothing")
}
w.said(first)
// Only the links changed, and nothing about adoption.
learnt := link.Report{Firewall: "none", Outward: []string{"eth0"}}
if !w.differs(learnt) {
t.Fatal("a machine that has just learnt which links face outside would never say so, " +
"and could then never be sent a filter")
}
w.said(learnt)
if w.differs(link.Report{Firewall: "none", Outward: []string{"eth0"}}) {
t.Fatal("the same links are reported as a change, so the machine would speak on every reconcile")
}
// And a link that changes — a laptop moving from a cable to a radio — is said too, because the
// filter is written around the old one until it is.
if !w.differs(link.Report{Firewall: "none", Outward: []string{"wlan0"}}) {
t.Fatal("a changed outward link is not said, so the filter stays written around the old one")
}
}
// **A converged machine can say which links face outside, unasked.**
//
// A reconcile is otherwise silent, and the condition deciding when it speaks asked only what an
// adopted node reports — what it holds, and the firewall it found. A converged node has neither, so
// it could speak only in reply to a declaration, and the mesh composes no declaration for a node
// that has not said which links face outside (novox/hq ADR 0140). Measured: three converged machines
// sat silent while the control plane refused to send them a filter.
func TestAConvergedMachineSaysItsOutwardLinksUnasked(t *testing.T) {
// A converged node's reconcile: nothing held, no found firewall, and the links it can see.
converged := link.Report{Outward: []string{"eth0"}}
if !worthSaying(converged) {
t.Fatal("a converged machine cannot say which links face outside, so it can never be " +
"sent a filter — a machine waiting for a push that is waiting for the machine")
}
// An adopted node's reasons still hold, because that is how a predecessor still writing is caught.
if !worthSaying(link.Report{Firewall: "ufw"}) {
t.Fatal("an adopted machine no longer says which firewall it found")
}
if !worthSaying(link.Report{Held: []link.Held{{ID: "a-file"}}}) {
t.Fatal("an adopted machine no longer says what it holds")
}
// And a reconcile with nothing to say stays silent, or every machine speaks every five minutes
// about nothing.
if worthSaying(link.Report{}) {
t.Fatal("a reconcile with nothing to say speaks anyway")
}
// A refused report says nothing about the machine; the refusal is for the console.
if worthSaying(link.Report{Outward: []string{"eth0"}, Refused: "not for this node"}) {
t.Fatal("a refused report is offered as news about the machine")
}
}
// **A host running as a service says what its apply did.**
//
// The serving path passed nil where the apply writes its detail, and nil is silence. The one-shot path
// has always passed a real function, so everything the apply says was visible when a person ran it by
// hand and discarded in the way the host actually runs. Measured before this was written: a machine was
// converged, its found firewall was not retired, and what the host decided was unrecoverable because it
// had been said to nobody (novox/hq 04-ISSUES/143).
//
// This asserts only that the apply's log is never nil and that a line reaches what the caller gave.
// **It cannot catch the fault it was written for** — a call site passing nil directly — because that is
// wiring, and wiring is only proved by running the thing. That proof is a deployed host whose journal
// carries the apply's detail, which is how this fix was verified.
func TestTheApplysLogIsNeverNil(t *testing.T) {
if announceOr(nil) == nil {
t.Fatal("a host with nowhere to say things got a nil log, which the apply will call")
}
announceOr(nil)("this goes nowhere and must not panic")
var said []string
announceOr(func(line string) { said = append(said, line) })(" disabled ufw")
if len(said) != 1 || !strings.Contains(said[0], "disabled ufw") {
t.Fatalf("the apply's detail did not reach the caller's announce: %v", said)
}
}
// Defends novox/hq issues 257 and 261: a reconcile due while the link applies a newer declaration
// applies that newer one once its turn comes — never the one kept when its timer fired. Read before
// waiting, it gave back a module assigned a second earlier, and reported a declaration the mesh no
// longer recorded as sent.
func TestAReconcileAppliesWhatWasKeptWhenItsTurnComes(t *testing.T) {
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "a.conf")
kept := func(content string) store.Declared {
body := []byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"` + target +
`","content":"` + content + `\n"}]}`)
return store.Declared{Declaration: body, Signature: ed25519.Sign(private, body)}
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state), kept("older")); err != nil {
t.Fatal(err)
}
// The link is applying: the reconcile's timer fires and it waits its turn.
applying.Lock()
done := make(chan error, 1)
go func() {
_, err := reapplyKept(context.Background(), opts, public, nil, nil)
done <- err
}()
time.Sleep(50 * time.Millisecond)
// The link's apply ends by keeping the newer declaration, and lets go.
if err := store.SaveDeclared(store.DeclaredPath(opts.state), kept("newer")); err != nil {
applying.Unlock()
t.Fatal(err)
}
applying.Unlock()
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(10 * time.Second):
t.Fatal("the reconcile never ran once the node was free")
}
got, err := os.ReadFile(target)
if err != nil {
t.Fatal(err)
}
if string(got) != "newer\n" {
t.Errorf("the reconcile applied %q, what was kept before the newer declaration", got)
}
}
+25
View File
@@ -0,0 +1,25 @@
package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/cli"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/meshcli"
)
// serveMeshCLI answers mesh-cli on this machine for as long as this engine serves (novox/hq ADR 0272): the account
// from the kernel, the line asked of the controller over the link open at the time. A socket that cannot be served
// is said once, and this engine goes on applying; mesh-cli then says the engine does not answer.
func serveMeshCLI(ctx context.Context, queue *link.Queue, say func(string)) {
if err := meshcli.Serve(ctx, cli.Socket, forwardOver(queue), say); err != nil {
say("mesh-cli is not answered on this machine: " + err.Error())
}
}
// forwardOver asks the controller over the queue's link, and follows the line to what came of it.
func forwardOver(queue *link.Queue) meshcli.Forward {
return func(ctx context.Context, asked meshcli.Asked) cli.Answer {
return meshcli.AskTheController(ctx, queue.AskController, asked)
}
}
+54
View File
@@ -0,0 +1,54 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/network"
)
// The machine's networking in the engine's statement (novox/hq ADR 0241): the file judged is the one the
// uplink holder declares whole, the mesh name asked is the bus's, and the statement carries the parts.
func TestTheResolverFileJudgedIsTheOneAModuleDeclaresWhole(t *testing.T) {
d := &declaration.Declaration{Resources: []declaration.Resource{
&declaration.File{ID: "hosts.file", Type: "file", Path: "/etc/hosts", Content: "x"},
&declaration.File{ID: "networkmanager.resolv", Type: "file", Path: network.ResolvConf, Content: "nameserver 10.10.0.1\n"},
}}
content, owner, ok := declaredResolvConf(d)
if !ok || owner != "networkmanager" || content != "nameserver 10.10.0.1\n" {
t.Fatalf("got %q %q %v", content, owner, ok)
}
d.Resources[1].(*declaration.File).CreateOnce = true
if _, _, ok := declaredResolvConf(d); ok {
t.Fatal("a seed nobody holds the machine to was judged as the declared file")
}
if _, _, ok := declaredResolvConf(nil); ok {
t.Fatal("no declaration declared a file")
}
}
func TestTheMeshNameAskedIsTheBuses(t *testing.T) {
for broker, want := range map[string]string{"anchor.internal:4222": "anchor.internal", "192.0.2.10:5671": "",
"anchor.internal": "anchor.internal", "[2001:db8::1]:4222": ""} {
if got := hostOf(broker); got != want {
t.Errorf("%s: got %q, want %q", broker, got, want)
}
}
}
func TestTheStatementCarriesTheNetwork(t *testing.T) {
at := time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)
h := healthAsReported(liveness.Statement{At: at}, &network.Statement{State: network.Unhealthy, Since: at,
Parts: []network.Part{{Part: network.PartResolvConf, State: network.Unhealthy, Writer: "FortiClient",
Owner: "networkmanager", Reason: "the resolver file was rewritten by another program", Since: at}}})
if h.Network == nil || h.Network.State != network.Unhealthy || len(h.Network.Parts) != 1 ||
h.Network.Parts[0].Writer != "FortiClient" || h.Network.Parts[0].Owner != "networkmanager" {
t.Fatalf("the statement says %+v", h.Network)
}
if h := healthAsReported(liveness.Statement{At: at}, nil); h.Network != nil {
t.Fatal("an engine with no network judge said a network")
}
}
+95
View File
@@ -0,0 +1,95 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// A declaration carries no order, so a host cannot tell an older one from a newer (novox/hq
// 04-ISSUES/107). Its only identity was the digest of its bytes: "not the last" could be said,
// "older" could not.
func keptWith(t *testing.T, sequence int64) store.Declared {
t.Helper()
return keptAt(t, link.Order{Sequence: sequence})
}
func keptAt(t *testing.T, order link.Order) store.Declared {
t.Helper()
body, err := json.Marshal(map[string]any{
"declaration": 1, "resources": []any{}, "owns_nothing": true, "sequence": order.Sequence,
"epoch": order.Epoch,
})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: []byte("x")}
}
func TestAnOlderDeclarationFromTheMeshIsRefused(t *testing.T) {
err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 5})
if err == nil {
t.Fatal("sequence 5 was accepted over a kept 7")
}
if !strings.Contains(err.Error(), "older") || !strings.Contains(err.Error(), "nothing was applied") {
t.Fatalf("the refusal does not say what it is: %v", err)
}
}
func TestANewerOrEqualDeclarationIsNot(t *testing.T) {
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 8}); err != nil {
t.Fatalf("sequence 8 was refused over a kept 7: %v", err)
}
// Equal is the same declaration again, which reconciling is for.
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 7}); err != nil {
t.Fatalf("the same sequence was refused: %v", err)
}
}
func TestNoOrderClaimedMeansNoOrderCompared(t *testing.T) {
// An older controller sends none; a host that received before it understood them kept none.
// Refusing on a guess would strand a node the moment the controller is older than the host.
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 0}); err != nil {
t.Fatalf("a declaration claiming no order was refused: %v", err)
}
if err := refuseOlder(keptWith(t, 0), nil, link.Order{Sequence: 3}); err != nil {
t.Fatalf("a declaration was refused against a kept one that claimed no order: %v", err)
}
if err := refuseOlder(store.Declared{}, store.ErrNothingDeclared, link.Order{Sequence: 3}); err != nil {
t.Fatalf("a first declaration was refused: %v", err)
}
}
// **The lease epoch decides first** (novox/hq to-be 45 §6): a declaration from a controller that lost
// its lease is refused whatever its sequence, and a new lease holder's is taken whatever its sequence.
func TestAnOlderEpochIsRefusedAndANewerTaken(t *testing.T) {
kept := keptAt(t, link.Order{Epoch: 57, Sequence: 3})
err := refuseOlder(kept, nil, link.Order{Epoch: 41, Sequence: 12})
if err == nil || !strings.Contains(err.Error(), "epoch 41") || !strings.Contains(err.Error(), "epoch 57") {
t.Fatalf("an older epoch was not refused by name: %v", err)
}
if err := refuseOlder(kept, nil, link.Order{Epoch: 58, Sequence: 1}); err != nil {
t.Fatalf("a new lease holder's first declaration was refused for its sequence: %v", err)
}
if err := refuseOlder(kept, nil, link.Order{Epoch: 57, Sequence: 2}); err == nil {
t.Fatal("a lower sequence under the same epoch was accepted")
}
// A controller with no lease — older, or rolled back to a build from before it — is today's
// behaviour: no epoch compared.
if err := refuseOlder(kept, nil, link.Order{Sequence: 4}); err != nil {
t.Fatalf("a declaration with no epoch was refused for one: %v", err)
}
if held, older := olderThanKept(kept, nil, link.Order{Epoch: 41, Sequence: 12}); !older ||
held != (link.Order{Epoch: 57, Sequence: 3}) {
t.Fatalf("olderThanKept said %v, holding %+v", older, held)
}
// What is kept cannot be read: no order to compare, so nothing is refused on a guess.
if _, older := olderThanKept(store.Declared{Declaration: []byte("garbled")}, nil,
link.Order{Epoch: 41, Sequence: 1}); older {
t.Fatal("refused against a kept declaration nobody can read")
}
}
+154
View File
@@ -0,0 +1,154 @@
package main
import (
"context"
"errors"
"os/exec"
"sort"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// packageManagers are the tools a system's probe asks, for a system whose probe is its package manager.
var packageManagers = map[string]string{"arch": "pacman", "alpine": "apk"}
// unanswered is a probe of the package manager that failed before an apply: which tool, and its words.
type unanswered struct {
tool string
words string
}
// failedKey is how a report names the probe among what failed: one key, the same on every report, so the
// mesh counts a package manager that keeps not answering as one failure that goes on.
func (u *unanswered) failedKey() string { return "the package manager (" + u.tool + ")" }
// said is the line the apply says before it begins.
func (u *unanswered) said() string {
return u.tool + " does not answer here, so every package in this declaration fails and the rest of it is " +
"applied (novox/hq issue 233): " + u.words
}
// partial is the failure of an apply that applied everything else, when no package was declared to fail.
func (u *unanswered) partial() error {
return errors.New("applied only in part: " + u.words + "\n\nevery resource that does not need " + u.tool +
" was attempted; none here needed it, and " + u.tool + " must answer before this machine applies whole " +
"(novox/hq issue 233)")
}
// probePackageManager proves the machine is the system the host was built for, before anything is applied.
//
// **A package manager that does not answer fails only what needs it** (novox/hq issue 233). The probe was a
// refusal of the whole declaration — right for a host installed on a machine it was not built for, and a
// wedge when what the probe reads is the mesh's own: a declaration that left the package manager's module
// out removed its configuration file, and from then on every declaration was refused, the one that would
// write the file back and the node-engine's own upgrade included. A person on the machine was the only way
// back. Now the rest applies — files, units, accounts, the node-engine itself — and each package fails with
// the probe's words (the system's PackageInstalled asks the same probe first), so a later declaration can
// undo what an earlier one did. Before the rest is applied, a declaration composed from an older view of the
// assignments than this machine applied has already been refused (issue 234's generation).
//
// **A package manager that is not there at all still refuses whole.** That is the host installed on a
// machine it was not built for — the arch host on Alpine — which is what the probe exists to catch, and no
// declaration the mesh sends removes a package manager's program. Only one that is there and errors (a
// configuration file gone, a database locked) is the mesh's own fault and applies in part.
//
// A system with no package manager (android) still refuses whole: what its probe asks is the system itself,
// which no resource the mesh declares can remove, and nothing else stands between it and the wrong machine.
func probePackageManager(ctx context.Context, sys system.System) (*unanswered, error) {
err := sys.Confirm(ctx, apply.ExecRunner)
if err == nil {
return nil, nil
}
tool, ok := packageManagers[sys.Name()]
if !ok || errors.Is(err, exec.ErrNotFound) {
return nil, err
}
return &unanswered{tool: tool, words: err.Error()}, nil
}
// applyPass is one apply of the declaration over what the node's state says.
type applyPass func(known store.State) (apply.Report, store.State, error)
// applyMindingProbe applies, and **once more when the package manager did not answer and this apply wrote
// something** (novox/hq issue 233). What it wrote may be what the package manager needed — the declaration
// that names its module again writes its configuration file back — and then the packages go in now rather
// than at the next reconcile. Asked again only then: an apply that wrote nothing changed nothing the package
// manager reads. The second pass is the same declaration over the state the first left, so what the first
// applied reads unchanged and only what needs the package manager moves.
//
// It returns the last pass's account, the probe as it stands after it (nil once the package manager
// answers), and the first pass's outcomes when a second ran, for whoever counts what an apply moved.
func applyMindingProbe(ctx context.Context, sys system.System, probe *unanswered, known store.State,
pass applyPass, say func(string)) (apply.Report, store.State, error, *unanswered, []apply.Outcome) {
report, updated, err := pass(known)
if probe == nil || !wrote(report, err) {
return report, updated, err, probe, nil
}
again, refused := probePackageManager(ctx, sys)
if refused != nil || again != nil {
return report, updated, err, probe, nil
}
say(probe.tool + " answers now that this apply wrote what it reads; applying again, so its packages are " +
"installed in this apply (novox/hq issue 233)")
first := append([]apply.Outcome(nil), report.Outcomes...)
var partial *apply.Error
if errors.As(err, &partial) {
first = append(first, partial.Done.Outcomes...)
}
report, updated, err = pass(updated)
return report, updated, err, nil, first
}
// wrote is whether an apply moved anything on the machine, failed or not.
func wrote(report apply.Report, err error) bool {
if report.Changed() {
return true
}
var partial *apply.Error
return errors.As(err, &partial) && partial.Done.Changed()
}
// unapplied is which declared resources an apply did not apply, failed or not attempted, sorted; nil for a
// clean one.
func unapplied(d *declaration.Declaration, report apply.Report, err error) []string {
if err == nil {
return nil
}
done := map[string]bool{}
for _, o := range report.Outcomes {
done[o.ID] = o.Action != "skipped"
}
var partial *apply.Error
if errors.As(err, &partial) {
for _, o := range partial.Done.Outcomes {
done[o.ID] = done[o.ID] || o.Action != "skipped"
}
}
var out []string
for _, r := range d.Resources {
if !done[r.Identity()] {
out = append(out, r.Identity())
}
}
sort.Strings(out)
if len(out) == 0 {
// Failed, and every resource has an account: what failed is no resource (the firewall found, a removal).
out = []string{"(apply)"}
}
return out
}
// withUnanswered adds the probe to what a report says failed.
func withUnanswered(failed map[string]string, u *unanswered) map[string]string {
if u == nil {
return failed
}
if failed == nil {
failed = map[string]string{}
}
failed[u.failedKey()] = u.said()
return failed
}
+209
View File
@@ -0,0 +1,209 @@
package main
import (
"context"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/system"
)
// Defends novox/hq issue 233: a package manager that does not answer is said even when nothing declared
// needs it — the apply is partial, never "applied" — and the report names it under one key, so the mesh
// counts it as one failure that goes on.
func TestAPackageManagerThatDoesNotAnswerIsSaidWithNoPackageDeclared(t *testing.T) {
unansweringPacman(t)
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json"), out: &strings.Builder{}}
conf := filepath.Join(dir, "pacman.conf")
raw := []byte(`{"declaration":1,"resources":[{"id":"pacman.conf","type":"file","path":"` + conf +
`","content":"[options]\n"}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
err = runApply(context.Background(), opts, d, raw, fromFile)
if err == nil || !strings.Contains(err.Error(), "applied only in part") || !strings.Contains(err.Error(), "could not be read") {
t.Fatalf("an apply whose package manager did not answer was not said to be partial: %v", err)
}
if _, statErr := os.Stat(conf); statErr != nil {
t.Fatalf("the file was not written: %v", statErr)
}
report := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if report.Refused != "" {
t.Fatalf("refused whole: %s", report.Refused)
}
why, ok := report.Failed["the package manager (pacman)"]
if !ok || !strings.Contains(why, "could not be read") {
t.Errorf("the report does not name the package manager with the probe's words: %v", report.Failed)
}
}
// A system whose probe asks the system itself, not a tool a resource needs, still refuses whole: nothing
// the mesh declares can remove it, and nothing else stands between that host and the wrong machine.
func TestASystemWithNoPackageManagerStillRefusesWhole(t *testing.T) {
t.Setenv("PATH", t.TempDir())
sys, err := system.For("android")
if err != nil {
t.Fatal(err)
}
probe, err := probePackageManager(context.Background(), sys)
if err == nil || probe != nil {
t.Fatalf("the android host's failed probe did not refuse: %v, %+v", err, probe)
}
}
// pacmanAnsweringOnce is a pacman that does not answer until conf exists, as the workstation's did until its
// configuration file was written back, and installs into a directory of its own once it does.
func pacmanAnsweringOnce(t *testing.T, conf string) {
t.Helper()
bin, installed := t.TempDir(), t.TempDir()
script := "#!/bin/sh\n" +
"[ -f '" + conf + "' ] || { echo \"error: config file " + conf + " could not be read: No such file or directory\" >&2; exit 1; }\n" +
"for a; do last=$a; done\n" +
"case \"$1\" in\n" +
"-Q) [ \"$2\" = pacman ] && exit 0; [ -f '" + installed + "'/\"$2\" ] && exit 0; exit 1;;\n" +
"-S) touch '" + installed + "'/\"$last\"; exit 0;;\n" +
"esac\nexit 1\n"
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
}
// Defends the review of novox/hq issue 233: **the apply that writes the package manager's configuration back
// installs its packages too**. The probe is asked again once the partial apply wrote something, and when it
// answers the declaration is applied again over what the first pass left — the packages in this apply, not five
// minutes later, and the report clean.
func TestThePackagesGoInWhenTheApplyThatRestoresThePackageManagerWroteIt(t *testing.T) {
dir := t.TempDir()
conf := filepath.Join(dir, "pacman.conf")
pacmanAnsweringOnce(t, conf)
opts := options{state: filepath.Join(dir, "state.json")}
// The package first: within one pass it is asked about before the file is written, which is the order that
// leaves it to a second pass.
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"pacman.base","type":"package","package":"base"},` +
`{"id":"pacman.conf","type":"file","path":"` + conf + `","content":"[options]\n"}]}`)
report := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if report.Refused != "" || len(report.Failed) != 0 {
t.Fatalf("the apply that made the package manager answer did not install its packages: refused %q, failed %v",
report.Refused, report.Failed)
}
if !slices.Contains(report.Applied, "pacman.base") || !slices.Contains(report.Applied, "pacman.conf") {
t.Errorf("the report does not account for both: %v", report.Applied)
}
}
// Defends decision (1) of the review of novox/hq issue 233: **a package manager that is not there at all still
// refuses whole**. That is the host installed on the wrong machine, which the probe exists to catch.
func TestAPackageManagerThatIsNotThereRefusesWhole(t *testing.T) {
t.Setenv("PATH", t.TempDir())
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
dir := t.TempDir()
conf := filepath.Join(dir, "pacman.conf")
raw := []byte(`{"declaration":1,"resources":[{"id":"pacman.conf","type":"file","path":"` + conf +
`","content":"[options]\n"}]}`)
report := applyAndKeep(context.Background(), options{state: filepath.Join(dir, "state.json")}, raw, nil, nil, nil)
if !strings.Contains(report.Refused, "pacman does not answer") {
t.Fatalf("a machine with no pacman at all was not refused whole: %+v", report)
}
if _, err := os.Stat(conf); err == nil {
t.Error("the refused declaration touched the machine")
}
}
// Defends the review of novox/hq issue 233: **a re-apply whose result changed is said unasked**. A machine
// reconciles every five minutes and speaks only when its account is news; how the apply went was not part of
// it, so a package manager that stopped answering between deliveries was never said, and one that answered
// again never cleared. Now a failure is said once, the same failure again is not, and a clean re-apply after it
// is — which is what clears the mesh's condition.
func TestAReapplyWhoseResultChangedIsSaidUnasked(t *testing.T) {
dir := t.TempDir()
conf := filepath.Join(dir, "elsewhere", "pacman.conf") // not declared: the apply cannot make it answer
pacmanAnsweringOnce(t, conf)
opts := options{state: filepath.Join(dir, "state.json")}
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"motd","type":"file","path":"` + filepath.Join(dir, "motd") + `","content":"hello\n"},` +
`{"id":"pacman.base","type":"package","package":"base"}]}`)
w := &adoptionWatch{}
// Clean once, as it was before: base installed while pacman answered.
if err := os.MkdirAll(filepath.Dir(conf), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(conf, []byte("[options]\n"), 0o644); err != nil {
t.Fatal(err)
}
clean := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if faulty(clean) {
t.Fatalf("the first apply was not clean: %+v", clean)
}
w.said(clean)
if w.news(applyAndKeep(context.Background(), opts, raw, nil, nil, nil)) {
t.Error("a clean re-apply after a clean one was said, so every machine speaks every five minutes")
}
// The configuration goes: the re-apply fails, and that is news once.
if err := os.Remove(conf); err != nil {
t.Fatal(err)
}
failed := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if !faulty(failed) {
t.Fatalf("the re-apply did not fail: %+v", failed)
}
if !w.news(failed) {
t.Fatal("a re-apply that failed after a clean one was not said, so the mesh never hears the machine fails")
}
w.said(failed)
if w.news(applyAndKeep(context.Background(), opts, raw, nil, nil, nil)) {
t.Error("the same failure was said again on the next re-apply")
}
// It comes back: the clean re-apply is news, and is what clears the mesh's condition.
if err := os.WriteFile(conf, []byte("[options]\n"), 0o644); err != nil {
t.Fatal(err)
}
again := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if faulty(again) {
t.Fatalf("the re-apply did not come back clean: %+v", again)
}
if !w.news(again) {
t.Fatal("a clean re-apply after a failed one was not said, so the mesh's condition never clears")
}
}
// A refusal is news too, once, and a reconcile with nothing to say after a clean one stays silent.
func TestARefusalIsSaidOnceAndNothingIsNotSaid(t *testing.T) {
w := &adoptionWatch{}
w.said(link.Report{})
if w.news(link.Report{}) {
t.Error("nothing was said again")
}
refused := link.Report{Refused: "pacman does not answer here"}
if !w.news(refused) {
t.Fatal("a refusal was not said")
}
w.said(refused)
if w.news(link.Report{Refused: "pacman does not answer here, at another time"}) {
t.Error("the same refusal in other words was said again")
}
if !w.news(link.Report{}) {
t.Error("a clean apply after a refusal was not said")
}
}
+94
View File
@@ -0,0 +1,94 @@
package main
import (
"crypto/ed25519"
"strings"
"testing"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/tunnel"
)
// novox/hq ADR 0105: `overlay take` moves this node's overlay key onto the found tunnel's and
// nothing else — identity, sealing and serving keys stay as they were — and tells the mesh with a
// proof signed by the identity key, over the previous key, the new one and the tunnel.
func anEnrolledNode(t *testing.T) identity.Identity {
t.Helper()
mine, err := identity.Generate("anchor")
if err != nil {
t.Fatal(err)
}
mine.Overlay, err = identity.GenerateOverlayKey()
if err != nil {
t.Fatal(err)
}
mine.Membership = identity.Membership{Broker: "198.51.100.1:5671", Fingerprint: "sha256:aa",
Signer: make([]byte, ed25519.PublicKeySize), Password: "p"}
return mine
}
func aFoundTunnel(t *testing.T) tunnel.Found {
t.Helper()
private, err := identity.GenerateOverlayKey()
if err != nil {
t.Fatal(err)
}
found, err := tunnel.Parse([]byte("[Interface]\nPrivateKey = " + private.Private + "\nListenPort = 51900\n" +
"Address = 192.0.2.1/24\n[Peer]\nPublicKey = PEER-A=\nAllowedIPs = 192.0.2.2/32\n"))
if err != nil {
t.Fatal(err)
}
found.Interface, found.Unit, found.Config = "wg0", "wg-quick@wg0", "/etc/wireguard/wg0.conf"
return found
}
func TestTakingATunnelMovesOnlyTheOverlayKeyAndSignsForIt(t *testing.T) {
mine := anEnrolledNode(t)
found := aFoundTunnel(t)
before := mine.Overlay.Public
taken, rekey, err := rekeyOnto(mine, found)
if err != nil {
t.Fatal(err)
}
if taken.Overlay.Public != found.PublicKey || taken.Overlay.Private != found.PrivateKey() {
t.Fatal("the overlay key is not the tunnel's")
}
if string(taken.Public) != string(mine.Public) || string(taken.Private) != string(mine.Private) ||
taken.Node != mine.Node || taken.Membership.Password != mine.Membership.Password ||
taken.Membership.Broker != mine.Membership.Broker {
t.Fatal("something other than the overlay key moved")
}
if taken.OverlayBefore != before {
t.Errorf("the key before the take was not kept: %q", taken.OverlayBefore)
}
if rekey.Previous != before || rekey.OverlayKey != found.PublicKey || rekey.Tunnel == nil ||
rekey.Tunnel.PublicKey != found.PublicKey || len(rekey.Tunnel.Peers) != 1 {
t.Fatalf("the rekey does not say what moved: %+v", rekey)
}
if !ed25519.Verify(ed25519.PublicKey(mine.Public),
link.RekeyProof("anchor", before, found.PublicKey, rekey.Tunnel), rekey.Proof) {
t.Fatal("the rekey is not signed by this node's identity key over what it says")
}
if ed25519.Verify(ed25519.PublicKey(mine.Public),
link.RekeyProof("laptop", before, found.PublicKey, rekey.Tunnel), rekey.Proof) {
t.Fatal("the proof is not bound to the node")
}
for _, said := range []string{rekey.Previous, rekey.OverlayKey, rekey.Tunnel.Interface} {
if strings.Contains(said, found.PrivateKey()) {
t.Fatal("the private key travels")
}
}
// Run again after the take — the mesh not yet told, or told and refused — the previous key it
// names is still the one before the take, so the mesh can tell a repeat from a replay.
again, second, err := rekeyOnto(taken, found)
if err != nil {
t.Fatal(err)
}
if second.Previous != before || again.OverlayBefore != before || again.Overlay.Public != found.PublicKey {
t.Fatalf("a take run again does not name the key before the first: %+v", second)
}
}
+109
View File
@@ -0,0 +1,109 @@
package main
import (
"bytes"
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// unansweringPacman puts a pacman on PATH that answers as one whose configuration file is gone, as the
// workstation's did on 2026-10-04 (novox/hq issue 233), and makes this an arch host.
func unansweringPacman(t *testing.T) {
t.Helper()
bin := t.TempDir()
script := "#!/bin/sh\necho 'error: config file /etc/pacman.conf could not be read: No such file or directory' >&2\nexit 1\n"
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
}
// TestReplay233 replays novox/hq issue 233. The package manager's module wrote pacman's configuration
// file; a declaration that left the module out removed it; from then on pacman did not answer, and the
// node-engine refused every declaration whole — the one that named the module again and would have
// written the file back included. A person restored the file by hand an hour and a half later.
//
// Held to: a declaration that restores the file is applied while pacman does not answer. Only what needs
// pacman fails, with the probe's words; the file is written and recorded. Written only with what the
// node-engine had before the fix, so it lays over the commit before it, where it fails.
func TestReplay233(t *testing.T) {
unansweringPacman(t)
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
conf := filepath.Join(dir, "pacman.conf")
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"pacman.conf","type":"file","path":"` + conf + `","content":"[options]\n"},` +
`{"id":"pacman.base","type":"package","package":"base"}]}`)
report := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if report.Refused != "" {
t.Fatalf("the declaration that restores the package manager's configuration was refused whole: %s", report.Refused)
}
if got, err := os.ReadFile(conf); err != nil || string(got) != "[options]\n" {
t.Fatalf("the file that restores the package manager was not written: %q, %v", got, err)
}
known, err := store.Load(opts.state)
if err != nil {
t.Fatal(err)
}
if _, ok := known.Find("pacman.conf"); !ok {
t.Error("the file written was not recorded, so the next apply would not know it is the mesh's")
}
words := strings.Join(failedWords(report.Failed), "\n")
if !strings.Contains(words, "pacman does not answer") || !strings.Contains(words, "could not be read") {
t.Errorf("the report does not say the package manager did not answer, in the probe's words: %q", words)
}
if !strings.Contains(words, "pacman.base") {
t.Errorf("the report does not name the package that was not applied: %q", words)
}
}
func failedWords(failed map[string]string) []string {
var out []string
for id, why := range failed {
out = append(out, id+": "+why)
}
return out
}
// Defends novox/hq issue 233 at the machine: `mesh-host apply` applies what does not need the package
// manager, and says the apply was partial with the probe's words — a non-zero exit, never "applied".
func TestApplyByHandAppliesWhatDoesNotNeedAPackageManagerThatDoesNotAnswer(t *testing.T) {
unansweringPacman(t)
dir := t.TempDir()
out := &bytes.Buffer{}
opts := options{state: filepath.Join(dir, "state.json"), out: out}
conf := filepath.Join(dir, "pacman.conf")
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"pacman.conf","type":"file","path":"` + conf + `","content":"[options]\n"},` +
`{"id":"pacman.base","type":"package","package":"base"}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
err = runApply(context.Background(), opts, d, raw, fromFile)
if _, statErr := os.Stat(conf); statErr != nil {
t.Fatalf("the file was not written: %v (the apply said %v)", statErr, err)
}
if err == nil {
t.Fatal("an apply whose package manager did not answer said it applied")
}
if !strings.Contains(err.Error(), "pacman does not answer") || !strings.Contains(err.Error(), "pacman.base") {
t.Errorf("the failure does not name the package or the probe's words: %v", err)
}
if !strings.Contains(out.String(), "pacman does not answer") {
t.Errorf("what was said before applying does not say the package manager does not answer:\n%s", out.String())
}
}
+20
View File
@@ -0,0 +1,20 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-host/internal/liveness"
)
// The restart count the engine says is its own since it began watching the resource, never since the state
// began (novox/hq issue 458).
func TestAnUnhealthyResourceSaysWhatItsRestartCountCounts(t *testing.T) {
r := liveness.State{Resource: liveness.Resource{ID: "messenger.own-runtime", Kind: "process",
Target: "messenger-runtime.service"}, State: liveness.Unhealthy, Reason: liveness.ReasonRestarting, Restarts: 3890}
said := unhealthyWords(r)
if !strings.Contains(said, "3890 automatic restart(s) since this engine began watching it") ||
strings.Contains(said, "counted") {
t.Fatalf("the restart count is not said as what it counts: %s", said)
}
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/accounts"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/units"
)
// The machine's failed units in the engine's statement (novox/hq issue 315): a module's is among the
// resources, unhealthy, as that module's unit, naming it; the machine's own beside them; and an engine
// not reading them says nothing of them.
func TestTheStatementCarriesTheFailedUnits(t *testing.T) {
at := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
us := &units.Statement{At: at, State: units.Degraded, Failed: []units.Failed{
{Unit: "openrazer-daemon.service", Scope: units.ScopeUser, Load: "loaded", Result: "exit-code",
User: "operator", Module: "openrazer", Resource: "openrazer.daemon", Via: units.ViaPackage, Since: at, Streak: 2},
{Unit: "storage-media.mount", Scope: units.ScopeSystem, Load: "loaded", Result: "timeout", Since: at, Streak: 2},
}}
h := withUnits(healthAsReported(liveness.Statement{At: at}, nil), us)
if len(h.Resources) != 1 {
t.Fatalf("one module's unit among the resources: %+v", h.Resources)
}
r := h.Resources[0]
if r.Module != "openrazer" || r.Resource != "openrazer.daemon" || r.Kind != link.KindUnit ||
r.Target != "openrazer-daemon.service" || r.State != link.StateUnhealthy ||
!strings.Contains(r.Reason, "account's own service manager") || !strings.Contains(r.Reason, "exit-code") {
t.Fatalf("the module's failed unit: %+v", r)
}
// Whose manager it fails in, so the controller can tell it from a fault (novox/hq ADR 0254).
if r.Account != "operator" {
t.Fatalf("the unit in the account's own manager does not name the account: %q", r.Account)
}
if h.Units == nil || h.Units.State != units.Degraded || len(h.Units.Failed) != 1 ||
h.Units.Failed[0].Unit != "storage-media.mount" || h.Units.Failed[0].Result != "timeout" {
t.Fatalf("the machine's own: %+v", h.Units)
}
if h := withUnits(healthAsReported(liveness.Statement{At: at}, nil), nil); h.Units != nil {
t.Fatal("an engine not reading units said them")
}
if h := withUnits(healthAsReported(liveness.Statement{At: at}, nil), &units.Statement{}); h.Units != nil {
t.Fatal("a judge not yet given a declaration said units")
}
}
// Every resource that runs in an account's own service manager names the account, and so does the account
// a module put in a group (novox/hq ADR 0254): the controller reads a unit that cannot run before a new login
// by the two together. What the machine's own manager runs names none.
func TestTheStatementNamesTheAccountsManager(t *testing.T) {
at := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
st := liveness.Statement{At: at, Resources: []liveness.State{
{Resource: liveness.Resource{Module: "openrazer", ID: "openrazer.daemon", Kind: "service",
Target: "openrazer-daemon.service", Scope: declaration.ScopeUser, User: "operator"}, State: liveness.Unhealthy,
Reason: "down", Since: at},
{Resource: liveness.Resource{Module: "docker", ID: "docker.engine", Kind: "service", Target: "docker.service",
Scope: declaration.ScopeSystem}, State: liveness.Healthy, Since: at},
}}
as := &accounts.Statement{At: at, Accounts: []accounts.Verdict{{Account: accounts.Account{Module: "openrazer",
ID: "openrazer.operator", Name: "operator", Groups: []string{"openrazer"}}, State: accounts.Unhealthy,
Reason: accounts.ReasonRelogin + ": operator is in the group openrazer", Since: at}}}
h := withAccounts(healthAsReported(st, nil), as)
got := map[string]string{}
for _, r := range h.Resources {
got[r.Resource] = r.Account
}
want := map[string]string{"openrazer.daemon": "operator", "docker.engine": "", "openrazer.operator": "operator"}
for id, account := range want {
if a, said := got[id]; !said || a != account {
t.Errorf("%s names the account %q (said %v); want %q", id, a, said, account)
}
}
}
// An account declared never to become root (novox/hq ADR 0266) is said with root never, under the contract
// that tells the controller this engine judges it; an account judged for its groups alone is not.
func TestTheStatementSaysAnAccountJudgedForRoot(t *testing.T) {
at := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
as := &accounts.Statement{At: at, Accounts: []accounts.Verdict{
{Account: accounts.Account{Module: "claude-code", ID: "claude-code.agent", Name: "agent", Root: true},
State: accounts.Unhealthy, Reason: accounts.ReasonRoot + ": in the group docker, which grants root", Since: at},
{Account: accounts.Account{Module: "openrazer", ID: "openrazer.account", Name: "operator",
Groups: []string{"openrazer"}}, State: accounts.Healthy, Since: at},
}}
h := withAccounts(healthAsReported(liveness.Statement{At: at}, nil), as)
// At RootContract or later: an engine that also reads libraries says LibraryContract (novox/hq ADR 0328), one
// above ADR 0325's IdentityContract.
if h.Contract < link.RootContract || link.RootContract != 3 || h.Contract != link.LibraryContract ||
link.LibraryContract != link.IdentityContract+1 {
t.Fatalf("contract %d", h.Contract)
}
got := map[string]string{}
for _, r := range h.Resources {
got[r.Resource] = r.Root
}
if got["claude-code.agent"] != "never" || got["openrazer.account"] != "" {
t.Fatalf("root said: %v", got)
}
}
// A verdict waiting for the setuid search says how far it got, in the reason the root-free verb repeats and as
// fields (novox/hq issue 361); a mount point, never a path the walk visited or found.
func TestTheStatementSaysHowFarTheSetuidSearchGot(t *testing.T) {
at := time.Date(2026, 10, 10, 1, 40, 0, 0, time.UTC)
sp := &accounts.SearchProgress{Started: at, Filesystems: 3, Done: 1, Walking: "/home", Visited: 42, Found: 2,
Kept: at, Finished: []accounts.SearchedFilesystem{{Mount: "/", Took: 90 * time.Second}}}
as := &accounts.Statement{At: at, Accounts: []accounts.Verdict{{Account: accounts.Account{Module: "claude-code",
ID: "claude-code.agent", Name: "agent", Root: true}, State: accounts.Unknown,
Reason: accounts.ReasonPending + ": the search for setuid programs, started 2026-10-10 01:27, has not finished yet",
Since: at, Search: sp}}}
h := withAccounts(healthAsReported(liveness.Statement{At: at}, nil), as)
r := h.Resources[0]
if !strings.HasPrefix(r.Reason, accounts.ReasonPending) || !strings.Contains(r.Reason, "1 of 3 filesystems walked, walking /home, 42 entries visited") {
t.Fatalf("the reason: %s", r.Reason)
}
s := r.Search
if s == nil || s.Filesystems != 3 || s.Done != 1 || s.Walking != "/home" || s.Visited != 42 || s.Found != 2 ||
s.Kept == nil || len(s.Finished) != 1 || s.Finished[0].Mount != "/" || *s.Finished[0].TookSeconds != 90 {
t.Fatalf("the fields: %+v", s)
}
}
+64
View File
@@ -0,0 +1,64 @@
package main
import (
"os"
"path/filepath"
"reflect"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The last apply's report, kept beside the state until the mesh takes it (novox/hq issue 264).
// What a host that died between applying and reporting kept is read back by the next one exactly —
// the digest and the outcome, never a new account of the machine.
func TestTheKeptReportSurvivesTheHost(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
made := link.Report{Declared: "sha256:d1", Applied: []string{"a"}, Failed: map[string]string{"b": "no room"},
Host: "v2"}
if err := unsaidBeside(state).Keep(made); err != nil {
t.Fatal(err)
}
kept, ok, err := unsaidBeside(state).Pending()
if err != nil || !ok {
t.Fatalf("nothing kept: %v", err)
}
if !reflect.DeepEqual(kept, made) {
t.Fatalf("kept %+v, made %+v", kept, made)
}
// Taken about another declaration, it stays; taken about this one, it goes.
if err := unsaidBeside(state).Said("sha256:other"); err != nil {
t.Fatal(err)
}
if _, ok, _ := unsaidBeside(state).Pending(); !ok {
t.Fatal("a report about another declaration cleared this one")
}
if err := unsaidBeside(state).Said("sha256:d1"); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(store.UnsaidPath(state)); !os.IsNotExist(err) {
t.Fatalf("the report was taken and is still kept: %v", err)
}
}
// A node that never applied anything has nothing to say again; and an apply refused before its
// declaration was read clears an older report, which would otherwise be re-said after it.
func TestNothingKeptWhenNothingWasApplied(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
if _, ok, err := unsaidBeside(state).Pending(); ok || err != nil {
t.Fatalf("a node that applied nothing has a report to say: %v %v", ok, err)
}
if err := unsaidBeside(state).Keep(link.Report{Declared: "sha256:d1"}); err != nil {
t.Fatal(err)
}
if err := unsaidBeside(state).Keep(link.Report{Refused: "forged"}); err != nil {
t.Fatal(err)
}
if _, ok, _ := unsaidBeside(state).Pending(); ok {
t.Fatal("an older report is kept after a later declaration was refused")
}
}
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// A component's version comes from where it sits, not from its linker (novox/hq ADR 0142). The mesh's
// toolchain stamps no version — a build does not know what it will be called — so a delivered host
// read as "development build" and the mesh could not tell which host a machine ran (04-ISSUES/161).
func TestADeliveredHostReadsItsVersionFromItsPath(t *testing.T) {
// A delivered host lives at <libexec>/versions/<version>/<binary>.
dir := t.TempDir()
versioned := filepath.Join(dir, "versions", "637f65559d16")
if err := os.MkdirAll(versioned, 0o755); err != nil {
t.Fatal(err)
}
self := filepath.Join(versioned, "nox-mesh-host")
if err := os.WriteFile(self, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
if got := versionAt(self, "development build"); got != "637f65559d16" {
t.Fatalf("a delivered host read its version as %q", got)
}
}
func TestAHostPlacedByHandKeepsItsStamp(t *testing.T) {
// The honest answer for one the mesh did not deliver — and every machine is in that state until
// a delivery reaches it.
if got := versionAt("/usr/bin/nox-mesh-host", "04a27ca"); got != "04a27ca" {
t.Fatalf("a hand-placed host read its version as %q", got)
}
}
func TestADirectoryThatIsNotAVersionIsNotReadAsOne(t *testing.T) {
// A binary sitting anywhere else must not have its parent directory's name read as a version.
for _, path := range []string{
"/opt/somewhere/nox-mesh-host",
"/usr/lib/nox-mesh-host/launch",
"/home/someone/build/nox-mesh-host",
} {
if got := versionAt(path, "the stamp"); got != "the stamp" {
t.Fatalf("%s read its version as %q", path, got)
}
}
}
+166
View File
@@ -0,0 +1,166 @@
package main
import (
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/upgrade"
"github.com/novox/mesh-host/internal/witness"
)
// The node-engine's part in core upgrades that roll back (novox/hq to-be 45 §8, ADR 0227 rule 8):
// proving its own build to the launcher that witnesses it, witnessing the controller and the node
// tools it places, and saying every verdict on its reports.
// standingRollbacks is every verdict that still stands on this machine: the launcher's about this
// host, and the engine's about the processes it witnesses. Said on every report.
func standingRollbacks(statePath string) []link.Rollback {
var out []link.Rollback
verdicts, err := upgrade.ReadRolledBack(upgrade.RolledBackPath(statePath))
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: %v\n", err)
}
for _, v := range verdicts {
out = append(out, link.Rollback{Component: link.ComponentEngine, From: v.From, To: v.To,
Outcome: v.Outcome, Why: v.Why, At: v.At})
}
return append(out, witness.Standing(apply.DaemonRoot())...)
}
// rolledBackHosts are the host versions the launcher will not start again, so this host never stands
// aside for one.
func rolledBackHosts(statePath string) []string {
verdicts, _ := upgrade.ReadRolledBack(upgrade.RolledBackPath(statePath))
return upgrade.RolledBackVersions(verdicts)
}
// proving is this host waiting for the mesh to take a report it made under its own build — the
// evidence its launcher's trial waits for — and acting on it once.
type proving struct {
statePath string
version string
say func(string)
once sync.Once
}
// unproved says whether this host's version is not yet known-good here: a host on trial.
func unproved(statePath, version string) bool {
known, err := upgrade.ReadKnownGood(upgrade.KnownGoodPath(statePath))
return err != nil || known != version
}
// heard is told every account of the machine the mesh has taken. The first one about a declaration,
// made by this build, proves it.
func (p *proving) heard(r link.Report) {
if r.Declared == "" || r.Refused != "" || r.Host != p.version {
return
}
p.once.Do(func() {
retired, err := upgrade.Proved(p.statePath, upgrade.VersionsDir(""), p.version)
if err != nil {
p.say(fmt.Sprintf("this host reported under its own build %s, and proving it was not complete: %v", p.version, err))
}
if len(retired) > 0 {
p.say(fmt.Sprintf("host %s is proved; retired the host version(s) %s", p.version, strings.Join(retired, ", ")))
}
renewFallback(p.version, p.say)
renewLauncher(p.say)
})
}
// renewFallback makes the engine placed by hand — the launcher's fallback, and the `nox-mesh-host` on this
// machine's PATH — a copy of this proved build when it is not one (novox/hq issue 356). On the same
// evidence known-good is written on, and said when it happens or cannot: a stale command that answers
// for a build that is not running would otherwise stay until somebody typed it.
func renewFallback(version string, say func(string)) {
self, err := os.Executable()
if err != nil {
say("the engine placed by hand could not be renewed: this build does not know its own path: " + err.Error())
return
}
fallback := upgrade.FallbackPath()
renewed, err := upgrade.RenewFallback(self, fallback)
switch {
case err != nil:
say(fmt.Sprintf("the engine placed by hand at %s is not this build, and could not be renewed: %v", fallback, err))
case renewed:
say(fmt.Sprintf("renewed the engine placed by hand at %s to this build, %s: the command on this machine is the "+
"engine that runs it", fallback, version))
}
}
// The launcher, as the service manager runs it.
const (
launcherPath = upgrade.DefaultLibexec + "/launch"
hostUnit = "nox-mesh-host.service"
)
// renewLauncher asks the service manager to restart this host's service when the launcher running
// it was replaced on disk since it started (novox/hq to-be 45 §8): a delivered launcher otherwise
// takes effect only at the next boot, and the witness it carries with it. A launcher from
// this one on runs its successor itself, at the host's next clean exit; this is for the launcher
// before it. Said, and only when it is certain: the parent is the launcher, and the file it reads
// is gone from under it.
func renewLauncher(say func(string)) {
if _, err := os.Stat("/run/systemd/system"); err != nil {
return
}
if !launcherReplaced(os.Getppid(), "/proc", launcherPath) {
return
}
say("the launcher running this host was replaced on disk; asking the service manager to run the new one")
if _, err := apply.ExecRunner(context.Background(), "systemctl", "restart", "--no-block", hostUnit); err != nil {
say("could not ask for the new launcher, so it runs from the next boot: " + err.Error())
}
}
// launcherReplaced is whether process pid is a shell reading the launcher at path, from a file that
// is no longer there — the one a delivery renamed a new launcher over.
func launcherReplaced(pid int, proc, path string) bool {
if pid <= 1 {
return false
}
base := filepath.Join(proc, strconv.Itoa(pid))
cmdline, err := os.ReadFile(filepath.Join(base, "cmdline"))
if err != nil || !strings.Contains(string(cmdline), path) {
return false
}
fds, err := os.ReadDir(filepath.Join(base, "fd"))
if err != nil {
return false
}
for _, fd := range fds {
target, err := os.Readlink(filepath.Join(base, "fd", fd.Name()))
if err == nil && target == path+" (deleted)" {
return true
}
}
return false
}
// watchWhatThisHostPlaced judges the core builds this host placed, until ctx ends (to-be 45 §8).
func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Queue, say func(string)) {
host, _ := os.Hostname()
w := &witness.Watcher{
Root: apply.DaemonRoot(), Node: node, Host: host,
Asker: func() link.Asker { return queue.Asker() },
Run: witness.Runner(apply.ExecRunner),
Hold: func(f func()) {
applying.Lock()
defer applying.Unlock()
f()
},
// Said now, not at the next report: the verdict is on every report from here, and the mesh
// is asked for one at once.
Concluded: func(link.Rollback) { queue.ReportAsked() },
Say: say,
}
w.Watch(ctx)
}
+120
View File
@@ -0,0 +1,120 @@
package main
import (
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/upgrade"
)
// The first account the mesh takes from this build, about a declaration, proves it to the launcher;
// nothing else does — a refusal, a report about no declaration, another build's report (to-be 45 §8).
func TestOnlyAnAccountUnderThisBuildProvesIt(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
t.Setenv("MESH_HOST_LIBEXEC", t.TempDir())
// Never the machine's own engine placed by hand: proving renews it (issue 356).
fallback := filepath.Join(t.TempDir(), "nox-mesh-host")
if err := os.WriteFile(fallback, []byte("a stale engine placed by hand"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv(upgrade.FallbackVar, fallback)
var said []string
p := &proving{statePath: state, version: "2.0", say: func(s string) { said = append(said, s) }}
for _, r := range []link.Report{
{Host: "2.0", Refused: "no"},
{Host: "2.0"},
{Host: "1.0", Declared: "abc"},
} {
p.heard(r)
if !unproved(state, "2.0") {
t.Fatalf("%+v proved this build", r)
}
if got, _ := os.ReadFile(fallback); string(got) != "a stale engine placed by hand" {
t.Fatalf("%+v renewed the engine placed by hand", r)
}
}
p.heard(link.Report{Host: "2.0", Declared: "abc"})
if unproved(state, "2.0") {
t.Fatalf("an account of a declaration under this build did not prove it (%v)", said)
}
if got, _ := upgrade.ReadKnownGood(upgrade.KnownGoodPath(state)); got != "2.0" {
t.Fatalf("known-good is %q", got)
}
// **Proved, the engine placed by hand becomes this build** (issue 356): the command on the machine's PATH is
// the engine that runs it, and that is said.
self, err := os.Executable()
if err != nil {
t.Fatal(err)
}
want, _ := os.ReadFile(self)
if got, _ := os.ReadFile(fallback); len(want) == 0 || string(got) != string(want) {
t.Fatalf("the engine placed by hand was not renewed to this build (%d bytes, want %d)", len(got), len(want))
}
renewed := false
for _, s := range said {
renewed = renewed || strings.Contains(s, "renewed the engine placed by hand at "+fallback)
}
if !renewed {
t.Fatalf("the renewal was not said: %q", said)
}
}
// Every verdict that stands is said, and a reconcile that has one says it unasked.
func TestAStandingRollbackIsSaidOnEveryReport(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
line := "2.0\t1.0\t1759744800\trolled-back\tit did not report within 600s of starting\n"
if err := os.WriteFile(upgrade.RolledBackPath(state), []byte(line), 0o644); err != nil {
t.Fatal(err)
}
got := standingRollbacks(state)
if len(got) == 0 || got[0].Component != link.ComponentEngine || got[0].From != "2.0" || got[0].To != "1.0" ||
got[0].Outcome != link.RolledBack {
t.Fatalf("what the report says is %+v", got)
}
if !worthSaying(link.Report{Rollbacks: got}) {
t.Fatal("a reconcile with a rollback standing does not say it")
}
if adoptionFingerprint(link.Report{Rollbacks: got}) == adoptionFingerprint(link.Report{}) {
t.Fatal("a rollback reached or ended is not news to the reconcile")
}
}
// The launcher running this host is known to have been replaced only when it certainly was: a shell
// reading the launcher, from a file renamed over. Asked of a real process, not a model of one.
func TestAReplacedLauncherIsSeen(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "launch")
if err := os.WriteFile(path, []byte("#!/bin/sh\nsleep 5\necho done\n"), 0o755); err != nil {
t.Fatal(err)
}
shell := exec.Command("sh", path)
if err := shell.Start(); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = shell.Process.Kill(); _ = shell.Wait() })
time.Sleep(200 * time.Millisecond)
if launcherReplaced(shell.Process.Pid, "/proc", path) {
t.Fatal("a launcher still on disk reads as replaced")
}
// Delivered as the mesh writes a file: a new one renamed over it.
if err := os.WriteFile(path+".new", []byte("#!/bin/sh\necho new\n"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Rename(path+".new", path); err != nil {
t.Fatal(err)
}
if !launcherReplaced(shell.Process.Pid, "/proc", path) {
fds, _ := os.ReadDir(filepath.Join("/proc", strconv.Itoa(shell.Process.Pid), "fd"))
t.Skipf("this shell does not keep its script open (%d fds), so a replaced launcher cannot be seen here", len(fds))
}
if launcherReplaced(shell.Process.Pid, "/proc", filepath.Join(dir, "other")) {
t.Fatal("a process reading another script reads as this launcher")
}
}
+200
View File
@@ -0,0 +1,200 @@
// foundation-first-node-nats.lock — what a machine must be before a mesh exists, on the bus being
// built (novox/hq ADR 0106, design 25).
//
// The same twelve steps as foundation-first-node.lock, with one difference that matters: **the mesh
// writes its own user list, and at genesis there is no mesh yet to write it.** So this carries the
// first one — the controller's own account, at a well-known bootstrap password, exactly as the store
// is reached at `postgres:bootstrap` and the old bus at `guest:guest`. It is rotated with those, and
// from the controller's first composition onward the file is the controller's to write.
//
// The accounts file is its own file beside the server's configuration, because the server's own
// settings belong to whoever raises it and the users belong to the mesh (design 25 §4). Both live in
// one directory, of necessity: an include path is resolved relative to the including file's own
// directory, so a server given an absolute one looks for it underneath that directory and refuses to
// start.
//
// No `verify` on the TLS block, deliberately — that setting makes the server demand a *client*
// certificate, and nothing in the mesh presents one: a host pins this server's exact certificate and
// authenticates with a password (ADR 0004, design 25 §4).
{
"declaration": 1,
"resources": [
{
"id": "container-runtime",
"type": "package",
"package": "docker"
},
{
"id": "container-runtime-running",
"type": "service",
"unit": "docker.service",
"state": "running",
"boot": "enabled"
},
// **A filter before anything listens** (novox/hq issue 054, ADR 0088). The store and the
// broker are adopted as modules later and so bind to every interface from the moment they
// start; the packet filter that governs who may reach them is a module too, installed a
// dozen steps later. Between the two, a control-node facing the network had its store and
// its bus open to anyone who could reach the machine. So the foundation carries a filter of
// its own — the same table the filter module will replace wholesale once it can derive one:
// drop by default, keep loopback, replies, ssh and the mesh's own ports (the bus a node
// enrols over, the registry a node pulls from), and let the container runtime's own
// networks through the forward chain so containers keep working. A published container port
// is forwarded, never input (issue 047), which is why the forward chain is where the store's
// and broker's ports are refused from outside — and a container on this machine dialling a
// port this machine publishes reaches it through the runtime's proxy, which IS input, which
// is why the bus and the registry are opened in both chains, exactly as the derived ruleset
// does.
{
"id": "base-filter-package",
"type": "package",
"package": "nftables"
},
{
"id": "base-filter",
"type": "file",
"path": "/etc/nftables.conf",
"mode": "0644",
"content": "#!/usr/sbin/nft -f\n# the foundation's own filter, until the mesh derives one (novox/hq issue 054)\ntable inet mesh {}\ndelete table inet mesh\n\ntable inet mesh {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\tiif lo accept\n\t\ticmp type echo-request accept\n\t\ticmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept\n\t\t# ssh, from anywhere — never closed\n\t\ttcp dport 22 accept\n\t\t# the mesh's own, from anywhere: the bus a node enrols over and a container on this machine reaches through the proxy, the registry a node pulls from\n\t\ttcp dport 5671 accept\n\t\ttcp dport 5000 accept\n\t}\n\tchain output {\n\t\ttype filter hook output priority filter; policy accept;\n\t}\n\tchain forward {\n\t\ttype filter hook forward priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\t# the container runtime's bridge networks, and the networks its compose files are given\n\t\tip saddr 172.16.0.0/12 accept\n\t\tip saddr 192.168.128.0/17 accept\n\t\t# the mesh's own: the bus a node enrols over, the registry a node pulls from\n\t\tct original proto-dst 5671 accept\n\t\tct original proto-dst 5000 accept\n\t}\n}\n"
},
{
"id": "base-filter-loaded",
"type": "service",
"unit": "nftables.service",
"state": "running",
"boot": "enabled",
"restart-on": ["base-filter"]
},
{
"id": "store",
"type": "container",
"name": "mesh-store",
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
"env": {
"POSTGRES_PASSWORD": "bootstrap",
"PGDATA": "/var/lib/postgresql/data/pgdata"
},
"ports": ["5432:5432"],
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
},
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
// exits happy, and the verify a moment later lands in the gap and fails. The action and its
// verify must ask the same question, or the action can succeed into a state verify rejects.
{
"id": "store-ready",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
},
{
"id": "inventory-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
},
{
"id": "identity-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
},
// Each context owns its own database (novox/hq ADR 0008). A third one is a third database,
// created the same way and named the same way — which is the whole of adding a context to the
// bootstrap, and is why the count is not something the foundation has an opinion about.
{
"id": "licences-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
},
{
"id": "context-schemas",
"type": "action",
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"migrate"],
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
},
{
"id": "bus-certificate",
"type": "action",
// **The mesh makes its own** (novox/hq 04-ISSUES/146). This ran `openssl` inside the
// broker's image while the broker was one that carried it; the bus that replaced it has a
// shell and no openssl, and no other image the bundle names has one either. So the program
// that needs the certificate writes it — already on this machine, since the schema step ran
// it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
// this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
// to ask an authority of.
// `--user 0:0` because the volume is root's and this image runs as nobody, which is right
// for the long-running control plane and wrong for a one-shot writing into a fresh volume.
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--into", "/tls"],
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--check", "--into", "/tls"]
},
{
"id": "bus-conf-dir",
"type": "directory",
"path": "/var/lib/mesh-bus-conf",
"mode": "0700"
},
{
"id": "bus-conf",
"type": "file",
"path": "/var/lib/mesh-bus-conf/nats.conf",
"mode": "0644",
"content": "port: 4222\nhttp: 127.0.0.1:8222\n\ntls {\n cert_file: \"/tls/tls.crt\"\n key_file: \"/tls/tls.key\"\n}\n\njetstream {\n store_dir: \"/data\"\n}\n\ninclude accounts.conf\n"
},
{
"id": "bus-accounts",
"type": "file",
"path": "/var/lib/mesh-bus-conf/accounts.conf",
"mode": "0600",
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
{
"id": "broker",
"type": "container",
"name": "mesh-broker",
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
"volumes": ["mesh-broker-data:/data", "mesh-broker-tls:/tls:ro", "/var/lib/mesh-bus-conf:/etc/nats:ro"],
"args": ["-c", "/etc/nats/nats.conf", "-js"]
},
{
"id": "broker-ready",
"type": "action",
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
},
{
"id": "control-plane",
"type": "container",
"name": "mesh-controller",
"image": "192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"network": "host",
"args": ["serve"],
"volumes": ["mesh-broker-tls:/broker-tls:ro"],
"env": {
"MESH_STORE_INVENTORY": "postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"MESH_STORE_IDENTITY": "postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"MESH_STORE_LICENCES": "postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"MESH_BUS_NATS": "nats://controller:bootstrap@127.0.0.1:5671",
"MESH_BROKER_ADDRESS": "192.0.2.10:5671",
"MESH_BROKER_CERTIFICATE": "/broker-tls/tls.crt"
}
}
]
}
+11 -4
View File
@@ -1,9 +1,16 @@
module github.com/novox/mesh-host
go 1.25.0
go 1.26.0
require (
github.com/rabbitmq/amqp091-go v1.14.0 // indirect
golang.org/x/crypto v0.55.0 // indirect
golang.org/x/sys v0.47.0 // indirect
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
golang.org/x/sys v0.48.0
)
require (
github.com/klauspost/compress v1.20.0 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
)
+18 -6
View File
@@ -1,6 +1,18 @@
github.com/rabbitmq/amqp091-go v1.14.0 h1:RSaT7aOKt/OrkVUyswPDW29lnRz9psuGmfZFBmLqLek=
github.com/rabbitmq/amqp091-go v1.14.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
git.novox.be/novox/mesh-sdk/go v0.1.11-0.20261009143344-f047d0a4a970 h1:9tFDQsgmI+4X7/BpZGXIr+HemPKE7YddYGqWV0lINAI=
git.novox.be/novox/mesh-sdk/go v0.1.11-0.20261009143344-f047d0a4a970/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010190644-622b6b4f556d h1:BR8uwv4BqyLaBwITuLylrd9DRUnu0D5DjJw0RuQSXbk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010190644-622b6b4f556d/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c h1:l5onJwoIeH8yE/PjVlEeoPyXBsHp2NQiWLllz2fwX7s=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+320
View File
@@ -0,0 +1,320 @@
// Package accounts is the node-engine reading whether an account has, where it runs, the groups a module
// put it in (novox/hq ADR 0252, issue 247).
//
// **A group takes effect at the next login.** The apply puts the account in a group in the machine's
// database at once. Every process already running keeps the groups it started with: the account's own
// service manager, and every unit it starts. The lighting daemon's module puts the operator's account in
// `openrazer`, and the daemon, started by that manager, still refuses to start, for as long as the
// session that began before lasts. Nothing said why.
//
// On every look the engine reads, for each account a module declares groups for:
//
// 1. whether the user database lists the account in each group. One it does not is the apply's to put
// right, and is said as `not in the group`;
// 2. whether the account's own service manager runs, and if it does, the groups that process holds,
// read from the process itself. A group the database lists and the running manager lacks is said as
// **relogin needed**, with what to do.
//
// No running manager is healthy: nobody is logged in, and the next login takes the groups.
//
// Each verdict is the declaring module's, as a resource of kind `account`, beside what liveness says: the
// controller raises it as the module's condition on two statements in a row, and clears it on the first
// healthy one.
//
// **And an account declared never to become root without a person** (`root: never`, novox/hq ADR 0266) —
// the account agent sessions run as on a machine where they must not reach root by themselves — is judged
// on every look for whether it can anyway: by its uid, by a group that grants root to its members, by any
// sudo rule at all, or by reading a secret the mesh placed for another account (the tool runner's bus
// credential among them, which carries every co-hosted module's grants). Judged first, whether or not
// anybody is logged in: a way to root does not wait for a session. Any way found is unhealthy, its reason
// starting ReasonRoot and naming every way; a question the machine did not answer is unknown, never
// healthy. This is the fact the controller reads to tell a machine where no agent can become root from one
// where an agent can.
//
// **It reads; it never acts** (ADR 0240 rule 6): the user and group databases, the machine's own service
// manager's `show` of the account's manager unit, and that process's status file. It never starts the
// account's manager, which asking that manager itself would.
package accounts
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
"github.com/novox/mesh-host/internal/declaration"
)
// The states, in the words liveness says them (the controller reads them alike).
const (
Healthy = "healthy"
Unhealthy = "unhealthy"
Unknown = "unknown"
)
// ReasonRelogin starts the reason of an account whose running session lacks a group it is in.
const ReasonRelogin = "relogin needed"
// ReasonRoot starts the reason of an account declared never to become root without a person that can
// (novox/hq ADR 0266); every way found follows it.
const ReasonRoot = "can become root without a person"
// RootGroups are the groups whose members become root by membership alone, without a sudo rule naming
// them: the administrators' groups a distribution's own rules or polkit treat as root, the container
// runtime's socket (a container with the host's root mounted), the raw disks, and the virtualisation
// daemons that start a guest with the host's devices — and the kernel's memory (kmem) and the password hashes
// (shadow), either of which is root a step away (novox/hq ADR 0312 §4 review). A closed list, so what the judge
// calls "grants root" is written down and reviewable rather than guessed per machine.
var RootGroups = []string{"root", "wheel", "sudo", "admin", "docker", "disk", "lxd", "incus-admin", "libvirt",
"kmem", "shadow"}
// Account is one user resource of a module that declares groups.
type Account struct {
Module string
// ID is the user resource's id; Name the account.
ID string
Name string
Groups []string
// Root is true for an account declared never to become root without a person (novox/hq ADR 0266).
Root bool
// Secrets are the paths of every secret the declaration places for another account, judged for
// whether this one can read them; only for a Root account.
Secrets []string
}
// Of is every account a declaration has a module put in a group. held is every resource an adopted
// machine holds as found, by id: its groups are not the mesh's yet. A resource the mesh declares in its
// own right (no module) is not judged here.
func Of(d *declaration.Declaration, held map[string]bool) []Account {
if d == nil {
return nil
}
var out []Account
for _, r := range d.Resources {
u, ok := r.(*declaration.User)
root := ok && u.Root == declaration.RootNever
if !ok || (len(u.Groups) == 0 && !root) || held[u.ID] || u.Name == "" {
continue
}
at := strings.LastIndex(u.ID, ".")
if at <= 0 || strings.HasPrefix(u.ID, declaration.AdoptionPrefix) {
continue
}
a := Account{Module: u.ID[:at], ID: u.ID, Name: u.Name, Groups: append([]string(nil), u.Groups...), Root: root}
if root {
a.Secrets = secretsOf(d, u.Name)
}
out = append(out, a)
}
sort.Slice(out, func(a, b int) bool { return out[a].ID < out[b].ID })
return out
}
// secretsOf is the path of every secret file the declaration places for anybody but account: sealed whole,
// or with sealed values in its content. One the account owns is its own to read.
func secretsOf(d *declaration.Declaration, account string) []string {
var out []string
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || f.Path == "" || (f.Sealed == "" && len(f.Secrets) == 0) || f.Owner == account {
continue
}
out = append(out, f.Path)
}
sort.Strings(out)
return out
}
// Session is what an account's own service manager holds.
type Session struct {
// Running is whether the manager runs.
Running bool
// Has is, of the groups asked about, those the running manager holds.
Has map[string]bool
}
// Reader is what a look asks the machine. An error is "could not be read": said unknown, never healthy.
type Reader interface {
// InDatabase is every group the user database lists an account in.
InDatabase(ctx context.Context, account string) ([]string, error)
// Session is the account's own service manager: whether it runs, and which of groups it holds.
Session(ctx context.Context, account string, groups []string) (Session, error)
// Escalation is every way the account can become root without a person, in words — its uid, a group
// of RootGroups, a sudo rule, a secret of secrets it can read — and none when there is none. Reads
// only (novox/hq ADR 0266).
Escalation(ctx context.Context, account string, secrets []string) ([]string, error)
}
// Verdict is one account's state as a statement says it.
type Verdict struct {
Account
State string
Reason string
Since time.Time
Streak int
// Search is how far the setuid search got, while the verdict waits for it (novox/hq issue 361); nil
// otherwise. Never part of Reason: it moves every minute, and a verdict that changed every minute would
// be said in the journal every minute.
Search *SearchProgress
}
// Statement is one look at every account.
type Statement struct {
At time.Time
Accounts []Verdict
}
// Healthy says no account in it is anything but healthy.
func (s Statement) Healthy() bool {
for _, v := range s.Accounts {
if v.State != Healthy {
return false
}
}
return true
}
// Judge reads every account's groups on every look. Safe for the apply and the looking loop at once.
type Judge struct {
reader Reader
Now func() time.Time
mu sync.Mutex
accounts []Account
kept map[string]*Verdict
last Statement
}
// New is a judge reading through r.
func New(r Reader) *Judge {
return &Judge{reader: r, Now: time.Now, kept: map[string]*Verdict{}}
}
// Set is what the declaration just applied asks. An account no longer asked for is forgotten.
func (j *Judge) Set(as []Account) {
j.mu.Lock()
defer j.mu.Unlock()
j.accounts = append([]Account(nil), as...)
declared := map[string]bool{}
for _, a := range as {
declared[a.ID] = true
}
for id := range j.kept {
if !declared[id] {
delete(j.kept, id)
}
}
}
// Last is the statement of the last look.
func (j *Judge) Last() Statement {
j.mu.Lock()
defer j.mu.Unlock()
return j.last
}
// Look reads every account once, and answers the statement and whether any verdict changed since the
// last look.
func (j *Judge) Look(ctx context.Context) (Statement, bool) {
j.mu.Lock()
defer j.mu.Unlock()
now := j.Now()
st := Statement{At: now}
changed := len(j.last.Accounts) != len(j.accounts)
for _, a := range j.accounts {
state, reason, search := j.judge(ctx, a)
k := j.kept[a.ID]
if k == nil || k.State != state || k.Reason != reason {
changed = true
k = &Verdict{Account: a, State: state, Reason: reason, Since: now}
j.kept[a.ID] = k
}
k.Account, k.Search = a, search
if state == Unhealthy {
k.Streak++
} else {
k.Streak = 0
}
st.Accounts = append(st.Accounts, *k)
}
j.last = st
return st, changed
}
// judge is one account's verdict on one look, and how far the setuid search got while it waits for one.
func (j *Judge) judge(ctx context.Context, a Account) (string, string, *SearchProgress) {
if a.Root {
ways, err := j.reader.Escalation(ctx, a.Name, a.Secrets)
var notJudged *SetuidNotJudged
switch {
case len(ways) > 0:
// A way found is a way found, whatever else could not be judged; said beside it.
reason := ReasonRoot + ": " + strings.Join(ways, "; ")
if errors.As(err, &notJudged) {
reason += "; " + firstLine(err.Error())
}
var progress *SearchProgress
if notJudged != nil {
progress = notJudged.Progress
}
return Unhealthy, reason, progress
case errors.As(err, &notJudged):
// The setuid search has no result yet: not judged, said as such — the same words on every look
// until it has one, never a search that stalls the look (novox/hq ADR 0266).
return Unknown, firstLine(err.Error()), notJudged.Progress
case err != nil:
return Unknown, "whether the account can become root could not be read: " + firstLine(err.Error()), nil
}
if len(a.Groups) == 0 {
// Declared for root alone: nothing of a session to read.
return Healthy, "", nil
}
}
in, err := j.reader.InDatabase(ctx, a.Name)
if err != nil {
return Unknown, "the user database could not be read: " + firstLine(err.Error()), nil
}
listed := map[string]bool{}
for _, g := range in {
listed[g] = true
}
var missing, inDB []string
for _, g := range a.Groups {
if listed[g] {
inDB = append(inDB, g)
} else {
missing = append(missing, g)
}
}
if len(missing) > 0 {
return Unhealthy, fmt.Sprintf("not in the group %s: the apply has not put %s there; its outcome says why",
strings.Join(missing, ", "), a.Name), nil
}
s, err := j.reader.Session(ctx, a.Name, inDB)
if err != nil {
return Unknown, "the account's own service manager could not be read: " + firstLine(err.Error()), nil
}
if !s.Running {
// Nobody is logged in, and the account does not linger: the next login takes every group.
return Healthy, "", nil
}
var lacking []string
for _, g := range inDB {
if !s.Has[g] {
lacking = append(lacking, g)
}
}
if len(lacking) > 0 {
return Unhealthy, fmt.Sprintf("%s: %s is in the group %s, and its running session began before it was; "+
"log out of every session and in again, or reboot", ReasonRelogin, a.Name, strings.Join(lacking, ", ")), nil
}
return Healthy, "", nil
}
func firstLine(s string) string {
line, _, _ := strings.Cut(strings.TrimSpace(s), "\n")
return line
}
+165
View File
@@ -0,0 +1,165 @@
package accounts
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
)
// The account judge (novox/hq ADR 0252, "how it is checked"): an account in a group its running session
// lacks is unhealthy with "relogin needed"; the same account with no session, or a session that has the
// group, is healthy; one the database does not list in the group is unhealthy saying so; and everything
// asked of the machine is a read.
// machine is a fake: the database's groups, the manager's process and its status file.
type machine struct {
t *testing.T
proc string
inDB string
pid string
held string // the Groups line of the manager's status
asked []string
failsID bool
}
func (m *machine) run(_ context.Context, name string, args ...string) (string, error) {
m.asked = append(m.asked, name+" "+strings.Join(args, " "))
switch {
case name == "id":
if m.failsID {
return "", errors.New("id exited 1")
}
return m.inDB + "\n", nil
case name == "getent" && args[0] == "passwd":
return "operator:x:1500:1500::/home/operator:/bin/zsh\n", nil
case name == "getent" && args[0] == "group":
switch args[1] {
case "openrazer":
return "openrazer:x:964:operator\n", nil
case "wheel":
return "wheel:x:998:operator\n", nil
}
return "", errors.New("getent exited 2: ")
case name == "systemctl":
return m.pid + "\n", nil
}
return "", errors.New("not a command the judge may run")
}
func (m *machine) status() {
m.t.Helper()
if m.pid == "" || m.pid == "0" {
return
}
dir := filepath.Join(m.proc, m.pid)
if err := os.MkdirAll(dir, 0o755); err != nil {
m.t.Fatal(err)
}
body := "Name:\tsystemd\nUid:\t1500\t1500\t1500\t1500\nGid:\t1500\t1500\t1500\t1500\nGroups:\t" + m.held + "\n"
if err := os.WriteFile(filepath.Join(dir, "status"), []byte(body), 0o644); err != nil {
m.t.Fatal(err)
}
}
var razer = Account{Module: "openrazer", ID: "openrazer.account", Name: "operator", Groups: []string{"openrazer"}}
func look(t *testing.T, m *machine) Verdict {
t.Helper()
m.status()
j := New(Exec{Run: m.run, Proc: m.proc})
j.Now = func() time.Time { return time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC) }
j.Set([]Account{razer})
st, _ := j.Look(t.Context())
if len(st.Accounts) != 1 {
t.Fatalf("the statement: %+v", st)
}
return st.Accounts[0]
}
func TestAGroupTheRunningSessionLacksSaysReloginNeeded(t *testing.T) {
m := &machine{t: t, proc: t.TempDir(), inDB: "operator wheel openrazer", pid: "4242", held: "998"}
v := look(t, m)
if v.State != Unhealthy || !strings.HasPrefix(v.Reason, ReasonRelogin) || !strings.Contains(v.Reason, "openrazer") ||
!strings.Contains(v.Reason, "log out") {
t.Fatalf("a session without the group: %+v", v)
}
if v.Module != "openrazer" || v.ID != "openrazer.account" {
t.Errorf("the verdict is not the module's: %+v", v)
}
for _, a := range m.asked {
read := strings.HasPrefix(a, "id -nG ") || strings.HasPrefix(a, "getent ") ||
strings.HasPrefix(a, "systemctl show --property=MainPID --value user@")
if !read {
t.Errorf("the judge asked something that is not a read: %q", a)
}
}
}
func TestASessionThatHasTheGroupIsHealthy(t *testing.T) {
m := &machine{t: t, proc: t.TempDir(), inDB: "operator wheel openrazer", pid: "4242", held: "998 964"}
if v := look(t, m); v.State != Healthy || v.Reason != "" {
t.Fatalf("a session with the group: %+v", v)
}
}
func TestNoSessionIsHealthyTheNextLoginTakesTheGroup(t *testing.T) {
m := &machine{t: t, proc: t.TempDir(), inDB: "operator openrazer", pid: "0"}
if v := look(t, m); v.State != Healthy {
t.Fatalf("no session: %+v", v)
}
}
func TestAnAccountTheDatabaseDoesNotListIsUnhealthySayingSo(t *testing.T) {
m := &machine{t: t, proc: t.TempDir(), inDB: "operator wheel", pid: "4242", held: "998"}
v := look(t, m)
if v.State != Unhealthy || !strings.HasPrefix(v.Reason, "not in the group openrazer") {
t.Fatalf("not in the group: %+v", v)
}
}
func TestADatabaseThatDoesNotAnswerIsUnknownNeverHealthy(t *testing.T) {
m := &machine{t: t, proc: t.TempDir(), failsID: true}
if v := look(t, m); v.State != Unknown {
t.Fatalf("an unread database: %+v", v)
}
}
func TestOnlyAModulesAccountWithGroupsIsJudged(t *testing.T) {
d, err := declaration.ParseTrusted([]byte(`{"declaration":1,"resources":[
{"id":"zsh.login","type":"user","name":"operator","shell":"/bin/zsh"},
{"id":"openrazer.account","type":"user","name":"operator","groups":["openrazer"]},
{"id":"docker.account","type":"user","name":"operator","groups":["docker"]},
{"id":"meshown","type":"user","name":"mesh","groups":["x"]}
]}`))
if err != nil {
t.Fatal(err)
}
got := Of(d, map[string]bool{"docker.account": true})
if len(got) != 1 || got[0].ID != "openrazer.account" || got[0].Module != "openrazer" {
t.Fatalf("judged: %+v", got)
}
}
func TestAVerdictThatHoldsCountsItsStreakAndAChangeIsSaid(t *testing.T) {
m := &machine{t: t, proc: t.TempDir(), inDB: "operator openrazer", pid: "4242", held: "998"}
m.status()
j := New(Exec{Run: m.run, Proc: m.proc})
j.Set([]Account{razer})
_, changed := j.Look(t.Context())
st, again := j.Look(t.Context())
if !changed || again || st.Accounts[0].Streak != 2 {
t.Fatalf("first look changed %v, second %v, streak %d", changed, again, st.Accounts[0].Streak)
}
m.held = "998 964"
m.status()
st, changed = j.Look(t.Context())
if !changed || st.Accounts[0].State != Healthy || st.Accounts[0].Streak != 0 {
t.Fatalf("after the new login: changed %v, %+v", changed, st.Accounts[0])
}
}
+316
View File
@@ -0,0 +1,316 @@
package accounts
import (
"context"
"errors"
"fmt"
"io/fs"
"os"
"os/exec"
"path/filepath"
"slices"
"strconv"
"strings"
"syscall"
"time"
)
// Runner runs a command and answers what it printed — the apply's own (apply.ExecRunner).
type Runner func(ctx context.Context, name string, args ...string) (string, error)
// Exec reads the machine through its command lines and the process table. **Reads only**: `id`, `getent`,
// the machine's own manager's `systemctl show`, `sudo -l -U` (which lists, as root, what sudo would let an
// account run, and runs nothing), a status file under /proc and a secret's owner and mode (a test holds
// both).
type Exec struct {
Run Runner
// Proc is where the process table is; empty is /proc. A test points it at a directory of its own.
Proc string
// Stat is a file's owner and mode; nil is the machine's own (os.Stat). A test gives files of its own.
Stat func(path string) (FileMode, error)
// Lstat is a path's own owner and mode, never through a link, with its type bits in Perm (fs.ModeSymlink,
// fs.ModeDir); nil is the machine's own (os.Lstat). Whether an account can reach a setuid program is read
// with it at every judgement (novox/hq issue 361).
Lstat func(path string) (FileMode, error)
// ReadFile, ReadDir and ACL read doas's and polkit's rules and a file's POSIX ACL; nil is the machine's.
ReadFile func(path string) ([]byte, error)
ReadDir func(path string) ([]fs.DirEntry, error)
ACL func(path string) ([]ACLEntry, error)
// Cache keeps the search for setuid programs between looks; nil searches on every look.
Cache *SetuidCache
// SearchRun runs the setuid search's commands (the package manager it asks); nil is Run. A
// serving engine gives Idle, so they yield to everything else on the machine, as the walk does.
SearchRun Runner
// Walk carries a walk for setuid programs on; nil is the engine's own (walker). A test gives its own.
Walk func(ctx context.Context, p *SetuidProgress, checkpoint func()) error
// Now is the clock the cache is kept by; nil is the machine's.
Now func() time.Time
}
// FileMode is what decides whether an account reads a file: its owner, its group and its permission bits.
type FileMode struct {
UID, GID int
Perm fs.FileMode
}
// Escalation is every way account can become root without a person (novox/hq ADR 0266) that the judge
// looks for — Judged lists them, NotJudged what it does not: its uid, a group of RootGroups the user database
// lists it in, any sudo rule naming it or a group of it, any of secrets it can read by owner, group or other
// bits, and the ways of ways.go: doas, polkit, a runtime's socket, an ACL, a setuid program no package owns.
// sudo absent is no sudo rule; doas and polkit (pkexec's grants) are judged from their own rules. A secret not there yet is skipped: there is nothing to read.
// The parent directories are not walked, so a file the bits allow and a directory hides is still said:
// the judge errs toward saying a way that is not, never toward missing one that is.
func (e Exec) Escalation(ctx context.Context, account string, secrets []string) ([]string, error) {
var ways []string
uidOut, err := e.Run(ctx, "id", "-u", account)
if err != nil {
return nil, fmt.Errorf("the user database did not answer about %q: %w", account, err)
}
uid, err := strconv.Atoi(strings.TrimSpace(uidOut))
if err != nil {
return nil, fmt.Errorf("the user database gave %q as %q's number", strings.TrimSpace(uidOut), account)
}
if uid == 0 {
ways = append(ways, "its uid is 0")
}
names, err := e.InDatabase(ctx, account)
if err != nil {
return nil, err
}
for _, g := range names {
if slices.Contains(RootGroups, g) {
ways = append(ways, "in the group "+g+", which grants root")
}
}
listed, err := e.Run(ctx, "sudo", "-l", "-U", account)
rules, err := SudoRules(listed, err)
if err != nil {
return nil, err
}
if len(rules) > 0 {
ways = append(ways, "sudo grants it: "+strings.Join(rules, ", "))
}
gidsOut, err := e.Run(ctx, "id", "-G", account)
if err != nil {
return nil, fmt.Errorf("the user database did not answer about %q's groups: %w", account, err)
}
gids := map[int]bool{}
for _, f := range strings.Fields(gidsOut) {
if n, err := strconv.Atoi(f); err == nil {
gids[n] = true
}
}
if len(secrets) > 0 {
stat := e.Stat
if stat == nil {
stat = statOf
}
for _, path := range secrets {
m, err := stat(path)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("the secret %s could not be read for its owner and mode: %w", path, err)
}
if Readable(m, uid, gids) {
ways = append(ways, "it can read the secret "+path)
}
}
}
more, err := e.moreWays(ctx, account, uid, names, gids, secrets)
var notJudged *SetuidNotJudged
if err != nil && !errors.As(err, &notJudged) {
return nil, err
}
// Every way found, and — when the setuid search has no result yet — that it is not judged.
return append(ways, more...), err
}
// Readable is whether an account of uid, in the groups gids, reads a file of m by its permission bits, as
// the kernel decides it: the owner's bits for the owner (root reads everything), the group's for a member,
// the others' for anybody else.
func Readable(m FileMode, uid int, gids map[int]bool) bool {
switch {
case uid == 0:
return true
case m.UID == uid:
return m.Perm&0o400 != 0
case gids[m.GID]:
return m.Perm&0o040 != 0
default:
return m.Perm&0o004 != 0
}
}
// SudoRules is the rules `sudo -l -U <account>` lists, from what it printed and how it ended: none when
// the account "is not allowed to run sudo" or sudo is not on the machine; every indented line after "may
// run the following commands" otherwise. Any rule counts — the decision is no sudo for the account at
// all, so a rule that asks for a password the account was never given is still a rule somebody can give
// it one for. Output that says neither is an error: unread is never none.
func SudoRules(out string, err error) ([]string, error) {
if err != nil && (errors.Is(err, exec.ErrNotFound) || errors.Is(err, fs.ErrNotExist)) {
return nil, nil
}
text := out
if err != nil {
text += "\n" + err.Error()
}
if strings.Contains(text, "is not allowed to run sudo") {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("sudo did not list the account's rules: %w", err)
}
var rules []string
listing := false
for _, line := range strings.Split(out, "\n") {
if strings.Contains(line, "may run the following commands") {
listing = true
continue
}
if listing && (strings.HasPrefix(line, " ") || strings.HasPrefix(line, "\t")) && strings.TrimSpace(line) != "" {
rules = append(rules, strings.TrimSpace(line))
}
}
if !listing {
return nil, fmt.Errorf("sudo listed neither rules nor a refusal: %q", firstLine(out))
}
return rules, nil
}
func statOf(path string) (FileMode, error) {
info, err := os.Stat(path)
if err != nil {
return FileMode{}, err
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return FileMode{}, fmt.Errorf("%s has no owner this platform reports", path)
}
return FileMode{UID: int(st.Uid), GID: int(st.Gid), Perm: info.Mode().Perm()}, nil
}
// lstatOf is a path's own owner and mode, its type bits kept.
func lstatOf(path string) (FileMode, error) {
info, err := os.Lstat(path)
if err != nil {
return FileMode{}, err
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return FileMode{}, fmt.Errorf("%s has no owner this platform reports", path)
}
return FileMode{UID: int(st.Uid), GID: int(st.Gid), Perm: info.Mode()}, nil
}
// InDatabase is `id -nG`: every group the user database lists the account in.
func (e Exec) InDatabase(ctx context.Context, account string) ([]string, error) {
out, err := e.Run(ctx, "id", "-nG", account)
if err != nil {
return nil, err
}
return strings.Fields(out), nil
}
// Session reads the account's own manager, user@<uid>.service, from the machine's manager — never from
// the account's, which asking would start — and the groups its process holds, from its status file.
func (e Exec) Session(ctx context.Context, account string, groups []string) (Session, error) {
passwd, err := e.Run(ctx, "getent", "passwd", account)
if err != nil {
return Session{}, fmt.Errorf("the user database did not answer about %q: %w", account, err)
}
fields := strings.Split(strings.TrimSpace(passwd), ":")
if len(fields) < 7 || fields[2] == "" {
return Session{}, fmt.Errorf("the user database gave no number for %q", account)
}
shown, err := e.Run(ctx, "systemctl", "show", "--property=MainPID", "--value", "user@"+fields[2]+".service")
if err != nil {
return Session{}, fmt.Errorf("the machine's service manager did not say whether %q's own runs: %w", account, err)
}
pid := strings.TrimSpace(shown)
if pid == "" || pid == "0" {
return Session{}, nil
}
held, err := e.heldBy(pid)
if err != nil {
return Session{}, err
}
s := Session{Running: true, Has: map[string]bool{}}
for _, g := range groups {
entry, err := e.Run(ctx, "getent", "group", g)
if err != nil {
return Session{}, fmt.Errorf("the group database did not answer about %q: %w", g, err)
}
parts := strings.Split(strings.TrimSpace(entry), ":")
if len(parts) < 3 {
return Session{}, fmt.Errorf("the group database gave %q for %q, which is not a group entry", entry, g)
}
s.Has[g] = held[parts[2]]
}
return s, nil
}
// heldBy is every group id a process holds, from the Groups line of its status file.
func (e Exec) heldBy(pid string) (map[string]bool, error) {
proc := e.Proc
if proc == "" {
proc = "/proc"
}
raw, err := os.ReadFile(filepath.Join(proc, pid, "status"))
if errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("the account's manager, process %s, ended while it was read", pid)
}
if err != nil {
return nil, err
}
// Its supplementary groups, and its own group, which the supplementary list need not repeat.
held := map[string]bool{}
named := false
for _, line := range strings.Split(string(raw), "\n") {
if rest, ok := strings.CutPrefix(line, "Groups:"); ok {
named = true
for _, gid := range strings.Fields(rest) {
held[gid] = true
}
}
if rest, ok := strings.CutPrefix(line, "Gid:"); ok {
if f := strings.Fields(rest); len(f) > 0 {
held[f[0]] = true
}
}
}
if !named {
return nil, fmt.Errorf("process %s's status names no groups", pid)
}
return held, nil
}
// IDs is `getent passwd`: an account's uid and primary gid, and whether the user database names it at all. A
// database that does not answer is an error, never "no such account".
func (e Exec) IDs(ctx context.Context, account string) (uid, gid int, found bool, err error) {
out, err := e.Run(ctx, "getent", "passwd", account)
if err != nil {
if strings.Contains(err.Error(), "exited 2") || strings.Contains(err.Error(), "exit status 2") {
return 0, 0, false, nil
}
return 0, 0, false, fmt.Errorf("the user database did not answer about %q: %w", account, err)
}
fields := strings.Split(strings.TrimSpace(out), ":")
if len(fields) < 4 {
return 0, 0, false, fmt.Errorf("the user database gave %q for %q, which is not a passwd entry",
strings.TrimSpace(out), account)
}
// getent reads an all-digit key as a uid: "1000" answers with whoever has uid 1000. Only the account named is
// the one asked for.
if fields[0] != account {
return 0, 0, false, fmt.Errorf("the user database answered %q with the account %q: an account is named, "+
"never numbered", account, fields[0])
}
uid, uerr := strconv.Atoi(fields[2])
gid, gerr := strconv.Atoi(fields[3])
if uerr != nil || gerr != nil {
return 0, 0, false, fmt.Errorf("the user database gave no numbers for %q", account)
}
return uid, gid, true, nil
}
+78
View File
@@ -0,0 +1,78 @@
//go:build linux
package accounts
import (
"bytes"
"context"
"errors"
"fmt"
"os"
"os/exec"
"runtime"
"strings"
"syscall"
"golang.org/x/sys/unix"
)
// ioprioIdle is the idle I/O class (IOPRIO_CLASS_IDLE, 3, shifted by IOPRIO_CLASS_SHIFT, 13): the process gets
// the disk only when nothing else asks for it, where the disk's scheduler honours classes (BFQ does).
const ioprioIdle = 3 << 13
// Idle runs a command as CLocale does, at the lowest priority the machine has (novox/hq issue 361): CPU nice
// 19 and the idle I/O class, set on the process as soon as it has started — so the search for setuid programs,
// which walks every file and runs as long as that takes, yields to everything else. It dies with the engine
// (the parent-death signal), so a restarted engine never leaves a walk running beside its own. A priority it
// could not lower is a failure: the command is killed rather than walk the disks at full priority.
func Idle(ctx context.Context, name string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
cmd.Env = append(os.Environ(), "LC_ALL=C", "LANG=C", "LANGUAGE=C")
cmd.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL}
var stdout, stderr bytes.Buffer
cmd.Stdout, cmd.Stderr = &stdout, &stderr
// The parent-death signal follows the thread that started the child, not the process: held on this
// thread until the child has ended, so the runtime retiring a thread never kills a walk midway.
runtime.LockOSThread()
defer runtime.UnlockOSThread()
if err := cmd.Start(); err != nil {
return "", fmt.Errorf("%s: %w", name, err)
}
pid := cmd.Process.Pid
if err := lower(pid); err != nil {
_ = cmd.Process.Kill()
_ = cmd.Wait()
return "", fmt.Errorf("%s: its priority could not be lowered, so it was stopped: %w", name, err)
}
err := cmd.Wait()
out := stdout.String()
if err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
return out, fmt.Errorf("%s exited %d: %s", name, exit.ExitCode(), strings.TrimSpace(stderr.String()))
}
return out, fmt.Errorf("%s: %w", name, err)
}
return out, nil
}
// lower sets a process's CPU nice to 19 and its I/O class to idle.
func lower(pid int) error {
if err := unix.Setpriority(unix.PRIO_PROCESS, pid, 19); err != nil {
return fmt.Errorf("nice: %w", err)
}
if _, _, errno := unix.Syscall(unix.SYS_IOPRIO_SET, 1 /* IOPRIO_WHO_PROCESS */, uintptr(pid), ioprioIdle); errno != 0 {
return fmt.Errorf("I/O class: %w", errno)
}
return nil
}
// LowerThisThread puts the calling goroutine on a thread of its own at nice 19 and the idle I/O class, for the
// walk for setuid programs, which runs in the engine (novox/hq issue 361): on Linux both are a thread's, so the
// engine's other work keeps its priority. The goroutine is locked to the thread and never unlocked, so the
// runtime ends the thread with it rather than hand a lowered thread to other work; a command it starts inherits
// the priority too.
func LowerThisThread() error {
runtime.LockOSThread()
return lower(unix.Gettid())
}
+13
View File
@@ -0,0 +1,13 @@
//go:build !linux
package accounts
import "context"
// Idle is CLocale where the machine offers no idle class this engine sets (novox/hq issue 361).
func Idle(ctx context.Context, name string, args ...string) (string, error) {
return CLocale(ctx, name, args...)
}
// LowerThisThread is nothing where the machine offers no per-thread priority this engine sets.
func LowerThisThread() error { return nil }
+32
View File
@@ -0,0 +1,32 @@
package accounts
import (
"context"
"errors"
"strings"
"testing"
)
// An account is read by its name (novox/hq ADR 0325 §4): getent reads an all-digit key as a uid, so "1000" would
// answer with whoever has uid 1000 — refused, never taken for the account named.
func TestAnAccountIsReadByItsNameNeverItsNumber(t *testing.T) {
db := map[string]string{
"nox": "nox:x:1003:1003::/home/nox:/bin/sh",
"1000": "jochen:x:1000:1000::/home/jochen:/bin/zsh", // what getent answers for a uid
}
e := Exec{Run: func(_ context.Context, name string, args ...string) (string, error) {
if line, ok := db[args[len(args)-1]]; ok {
return line + "\n", nil
}
return "", errors.New("getent exited 2")
}}
if uid, gid, found, err := e.IDs(context.Background(), "nox"); err != nil || !found || uid != 1003 || gid != 1003 {
t.Fatalf("nox: %d %d %v %v", uid, gid, found, err)
}
if _, _, found, err := e.IDs(context.Background(), "1000"); found || err == nil || !strings.Contains(err.Error(), "jochen") {
t.Fatalf("an all-digit name was read as a uid: found=%v err=%v", found, err)
}
if _, _, found, err := e.IDs(context.Background(), "ghost"); found || err != nil {
t.Fatalf("an absent account: found=%v err=%v", found, err)
}
}
+302
View File
@@ -0,0 +1,302 @@
package accounts
import (
"context"
"errors"
"fmt"
"io/fs"
"os/exec"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
// An account declared never to become root without a person (novox/hq ADR 0266, "how it is checked"): each
// way to root is found and said, none is healthy whether or not anybody is logged in, an unanswered question
// is unknown, and the judge only reads.
// agentMachine is a fake machine for the escalation question: the account's uid, its groups by name and
// number, what sudo lists, and the secrets' owners and modes.
type agentMachine struct {
uid string
groups string
gids string
sudo string
sudoErr error
files map[string]FileMode
failsID bool
asked []string
// texts are files' contents (doas's, polkit's); dirs a directory's files by name; acls a file's ACL;
// setuid what the walk finds, one per line, and packaged the paths a package owns.
texts map[string]string
dirs map[string][]string
acls map[string][]ACLEntry
setuid string
findErr error
packaged map[string]bool
// proc is /proc's files, aSafeProc unless a test changes them; findArgs sees the filesystems walked.
proc map[string]string
findArgs func([]string)
// pacmanCalls counts the package manager's calls: one for every path found.
pacmanCalls int
// nodes are paths' own owners and modes (Lstat); any other is root's, a directory searchable by all, or a
// file the walk found, setuid and executable by all.
nodes map[string]FileMode
}
func (m *agentMachine) lstat(path string) (FileMode, error) {
if n, ok := m.nodes[path]; ok {
return n, nil
}
if strings.Contains(m.setuid, path+"\n") {
return FileMode{Perm: fs.ModeSetuid | 0o755}, nil
}
return FileMode{Perm: fs.ModeDir | 0o755}, nil
}
func (m *agentMachine) readFile(path string) ([]byte, error) {
if t, ok := m.texts[path]; ok {
return []byte(t), nil
}
if t, ok := m.proc[path]; ok {
return []byte(t), nil
}
return nil, fs.ErrNotExist
}
// aSafeProc is what a machine with nothing to say has in /proc: links protected, nothing on the runtimes'
// ports, one root filesystem.
func aSafeProc() map[string]string {
return map[string]string{
"/proc/sys/fs/protected_hardlinks": "1\n",
"/proc/sys/fs/protected_symlinks": "1\n",
"/proc/net/tcp": " sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n" +
" 0: 0100007F:1092 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 1 1\n",
"/proc/mounts": "/dev/sda2 / ext4 rw,relatime 0 0\nproc /proc proc rw,nosuid 0 0\ntmpfs /tmp tmpfs rw,nosuid 0 0\n",
}
}
func (m *agentMachine) readDir(path string) ([]fs.DirEntry, error) {
names, ok := m.dirs[path]
if !ok {
return nil, fs.ErrNotExist
}
var out []fs.DirEntry
for _, n := range names {
out = append(out, fakeEntry(n))
}
return out, nil
}
type fakeEntry string
func (f fakeEntry) Name() string { return string(f) }
func (f fakeEntry) IsDir() bool { return false }
func (f fakeEntry) Type() fs.FileMode { return 0 }
func (f fakeEntry) Info() (fs.FileInfo, error) { return nil, fs.ErrNotExist }
func (m *agentMachine) acl(path string) ([]ACLEntry, error) {
if _, ok := m.files[path]; !ok {
return nil, fs.ErrNotExist
}
return m.acls[path], nil
}
// walk is the walk for setuid programs on this machine: m.setuid found, then m.findErr.
func (m *agentMachine) walk(_ context.Context, p *SetuidProgress, _ func()) error {
m.asked = append(m.asked, "walk "+strings.Join(p.Mounts, " "))
if m.findArgs != nil {
m.findArgs(p.Mounts)
}
for _, l := range strings.Split(m.setuid, "\n") {
if l = strings.TrimSpace(l); l != "" {
p.Found = append(p.Found, l)
}
}
return m.findErr
}
func (m *agentMachine) run(_ context.Context, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
m.asked = append(m.asked, line)
switch {
case m.failsID && name == "id":
return "", errors.New("id exited 1: no such user")
case line == "id -u agent":
return m.uid + "\n", nil
case line == "id -nG agent":
return m.groups + "\n", nil
case line == "id -G agent":
return m.gids + "\n", nil
case line == "sudo -l -U agent":
return m.sudo, m.sudoErr
case name == "pacman" && len(args) >= 2 && args[0] == "-Qo":
m.pacmanCalls++
var out, errs []string
for _, p := range args[1:] {
if m.packaged[p] {
out = append(out, p+" is owned by somepackage 1.0-1")
} else {
errs = append(errs, "error: No package owns "+p)
}
}
said := strings.Join(out, "\n")
if len(errs) > 0 {
return said, errors.New("pacman exited 1: " + strings.Join(errs, "\n"))
}
return said, nil
}
return "", errors.New("not a command the judge may run: " + line)
}
func (m *agentMachine) stat(path string) (FileMode, error) {
if f, ok := m.files[path]; ok {
return f, nil
}
return FileMode{}, fs.ErrNotExist
}
const notAllowed = "User agent is not allowed to run sudo on box.\n"
var agent = Account{Module: "claude-code", ID: "claude-code.agent", Name: "agent", Root: true,
Secrets: []string{"/var/lib/mesh/node-tools/broker"}}
func clean() *agentMachine {
return &agentMachine{uid: "1600", groups: "agent", gids: "1600", sudo: notAllowed,
files: map[string]FileMode{"/var/lib/mesh/node-tools/broker": {UID: 1500, GID: 1500, Perm: 0o600}},
proc: aSafeProc()}
}
func lookAgent(t *testing.T, m *agentMachine) Verdict {
t.Helper()
j := New(Exec{Run: m.run, Walk: m.walk, Stat: m.stat, ReadFile: m.readFile, ReadDir: m.readDir, ACL: m.acl, Lstat: m.lstat})
j.Set([]Account{agent})
st, _ := j.Look(t.Context())
if len(st.Accounts) != 1 {
t.Fatalf("the statement: %+v", st)
}
for _, a := range m.asked {
if !strings.HasPrefix(a, "id ") && a != "sudo -l -U agent" && !strings.HasPrefix(a, "walk /") &&
!strings.HasPrefix(a, "pacman -Qo ") {
t.Errorf("the judge asked something that is not a read: %q", a)
}
}
return st.Accounts[0]
}
func TestAnAgentAccountWithNoWayToRootIsHealthyWithNobodyLoggedIn(t *testing.T) {
if v := lookAgent(t, clean()); v.State != Healthy || v.Reason != "" || !v.Root {
t.Fatalf("no way to root: %+v", v)
}
}
func TestEachWayToRootIsSaid(t *testing.T) {
for _, c := range []struct {
name string
set func(*agentMachine)
said string
}{
{"uid 0", func(m *agentMachine) { m.uid = "0" }, "its uid is 0"},
{"docker", func(m *agentMachine) { m.groups = "agent docker" }, "in the group docker, which grants root"},
{"wheel", func(m *agentMachine) { m.groups = "agent wheel" }, "in the group wheel, which grants root"},
{"sudo", func(m *agentMachine) {
m.sudo = "Matching Defaults entries for agent on box:\n env_reset\n\nUser agent may run the following commands on box:\n (ALL) NOPASSWD: ALL\n"
}, "sudo grants it: (ALL) NOPASSWD: ALL"},
{"secret by others", func(m *agentMachine) {
m.files["/var/lib/mesh/node-tools/broker"] = FileMode{UID: 1500, GID: 1500, Perm: 0o644}
}, "it can read the secret /var/lib/mesh/node-tools/broker"},
{"secret by group", func(m *agentMachine) {
m.gids = "1600 1500"
m.files["/var/lib/mesh/node-tools/broker"] = FileMode{UID: 1500, GID: 1500, Perm: 0o640}
}, "it can read the secret /var/lib/mesh/node-tools/broker"},
} {
t.Run(c.name, func(t *testing.T) {
m := clean()
c.set(m)
v := lookAgent(t, m)
if v.State != Unhealthy || !strings.HasPrefix(v.Reason, ReasonRoot+": ") || !strings.Contains(v.Reason, c.said) {
t.Fatalf("want %q said: %+v", c.said, v)
}
})
}
}
func TestEveryWayIsSaidAtOnce(t *testing.T) {
m := clean()
m.groups = "agent docker"
m.sudo = "User agent may run the following commands on box:\n (ALL) ALL\n"
v := lookAgent(t, m)
if !strings.Contains(v.Reason, "docker") || !strings.Contains(v.Reason, "(ALL) ALL") {
t.Fatalf("both ways: %+v", v)
}
}
func TestAnUnansweredQuestionIsUnknownNeverHealthy(t *testing.T) {
m := clean()
m.failsID = true
if v := lookAgent(t, m); v.State != Unknown {
t.Fatalf("an unread database: %+v", v)
}
m = clean()
m.sudo = "something sudo never says\n"
if v := lookAgent(t, m); v.State != Unknown {
t.Fatalf("an unread sudo: %+v", v)
}
}
func TestASecretNotThereYetIsNoWay(t *testing.T) {
m := clean()
delete(m.files, "/var/lib/mesh/node-tools/broker")
if v := lookAgent(t, m); v.State != Healthy {
t.Fatalf("no secret placed: %+v", v)
}
}
func TestSudoRules(t *testing.T) {
none := []struct {
out string
err error
}{
{notAllowed, nil},
{notAllowed, errors.New("sudo exited 1: ")},
{"", fmt.Errorf("sudo: %w", exec.ErrNotFound)},
}
for _, c := range none {
if rules, err := SudoRules(c.out, c.err); err != nil || len(rules) != 0 {
t.Errorf("%q, %v: rules %v, err %v", c.out, c.err, rules, err)
}
}
rules, err := SudoRules("Matching Defaults entries for agent on box:\n env_reset\n\nUser agent may run the following commands on box:\n (ALL) /usr/bin/systemctl\n (root) NOPASSWD: /usr/bin/true\n", nil)
if err != nil || len(rules) != 2 || rules[0] != "(ALL) /usr/bin/systemctl" {
t.Fatalf("two rules: %v, %v", rules, err)
}
if _, err := SudoRules("", errors.New("sudo exited 1: sudo: unable to resolve host")); err == nil {
t.Error("a failing sudo that said neither was read as no rules")
}
}
func TestReadable(t *testing.T) {
m := FileMode{UID: 1500, GID: 1500, Perm: 0o600}
if Readable(m, 1600, map[int]bool{1600: true}) || !Readable(m, 1500, nil) || !Readable(m, 0, nil) {
t.Fatal("owner bits")
}
}
func TestOfJudgesARootNeverAccountWithNoGroupsAndItsSecrets(t *testing.T) {
d, err := declaration.ParseTrusted([]byte(`{"declaration":1,"resources":[
{"id":"claude-code.agent","type":"user","name":"agent","root":"never"},
{"id":"zsh.login","type":"user","name":"operator","shell":"/bin/zsh"},
{"id":"node-tools.need-broker","type":"file","path":"/var/lib/mesh/broker","sealed":"x","owner":"operator"},
{"id":"claude-code.own","type":"file","path":"/home/agent/own","sealed":"x","owner":"agent"},
{"id":"claude-code.plain","type":"file","path":"/etc/plain","content":"x"}
]}`))
if err != nil {
t.Fatal(err)
}
got := Of(d, nil)
if len(got) != 1 || got[0].ID != "claude-code.agent" || !got[0].Root ||
len(got[0].Secrets) != 1 || got[0].Secrets[0] != "/var/lib/mesh/broker" {
t.Fatalf("judged: %+v", got)
}
}
+635
View File
@@ -0,0 +1,635 @@
package accounts
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"syscall"
"time"
"github.com/novox/mesh-host/rootsearch"
)
// SetuidCache keeps the search for setuid programs, which walks every local filesystem, and runs it in the
// background (novox/hq ADR 0266, issue 361): a look never waits for it, and the search itself is never cut
// short.
//
// - **A verdict comes only from a complete search, while it is fresh**: one that walked everything and had the
// package manager answer for every path found, started less than FreshFor ago. Before the first completes,
// or once the last is older, the answer is "not judged yet" (ReasonPending) while a search runs, and "not
// judged" (ReasonIncomplete) after one failed. Never "none found" from a search that did not finish.
// - **It runs to completion**, once at a time, with no bound, on a thread of its own at idle priority (Lower):
// a fresh one when the last complete one is Every old, when SearchAgain asks (an apply changed what root
// controls), or when a filesystem appeared that the last one did not walk. One that fails is tried again
// after a back-off that doubles from FailedBackoff up to MaxBackoff.
// - **Resumable**: the walk's progress is kept in ProgressPath every CheckpointEvery, and a search started
// again — after an engine restart, an update, a reboot or a failure — carries on from it while it is
// younger than FreshFor. A search SearchAgain asked for never resumes an older walk: it starts from nothing.
// - **Kept** in Path, beside the engine's other state, with when it started and finished: an engine restart
// serves the last complete search rather than judging nothing until a new walk ends.
//
// Why a complete search stays sound for a day and more, and a resumed walk is sound, is said in the rootsearch
// package and setuid_walk.go: only root makes a setuid-root file, never the agent account.
type SetuidCache struct {
// Every and FreshFor are rootsearch's when zero.
Every, FreshFor time.Duration
// Path is the file the last complete search is kept in, and ProgressPath the walk under way; empty keeps
// either in memory only. The engine's own state directory, which only root writes: a file there that is
// not a regular file the engine's account owns and alone may write is not read (keptFile).
Path, ProgressPath string
// Lower is called on the search's own goroutine before it walks (LowerThisThread on a serving engine); a
// priority it cannot lower fails the search rather than walk the disks at full priority. Nil lowers nothing.
Lower func() error
// Say is where a search's end and a kept file refused are said; nil says nothing.
Say func(string)
mu sync.Mutex
wg sync.WaitGroup
loaded bool
last *setuidResult // the last complete search
unfinished *SetuidProgress // a walk that did not finish, to resume
progress *SearchProgress // how far the running search got, as of its last checkpoint
running bool
gen int // counts SearchNow: a search started under an older one is thrown away
cancel context.CancelFunc // ends the running search's walk
startedAt time.Time // when the search running, or the last one, was started (a resumed one: its start)
attemptAt time.Time // when the running or last search was started or resumed
endedAt time.Time // when the last search ended, on the same clock
again bool // asked to search again since the running or last one started
failedAt time.Time
failed error
backoff time.Duration
starts int // how many searches were started or resumed, for a test
}
// setuidResult is one complete search, as kept.
type setuidResult struct {
// Started and Finished are on the caller's clock: Finished is Started plus how long the search took.
Started time.Time `json:"started"`
Finished time.Time `json:"finished"`
// Mounts are the filesystems it walked; Found the setuid-root programs no package owns.
Mounts Paths `json:"mounts"`
Found Paths `json:"found"`
}
const (
// FailedBackoff is the first wait after a search failed, doubling to MaxBackoff.
FailedBackoff = time.Hour
MaxBackoff = 6 * time.Hour
// AgainAfter is the least time between the starts of two searches that SearchAgain or a new filesystem
// asked for: a run of applies does not walk the disks back to back.
AgainAfter = time.Hour
// clockSkew is how far in the future a kept search may say it started and still judge: a clock set back
// further makes it no verdict, never a fresh one.
clockSkew = 5 * time.Minute
// SetuidSearchFile and SetuidProgressFile are the files' names beside the engine's state: the last
// complete search, and the walk under way.
SetuidSearchFile = "setuid-search.json"
SetuidProgressFile = "setuid-search-progress.json"
)
// SetuidNotJudged is the search's answer when no complete search judges: none has finished yet, the last one is
// older than FreshFor, or every one since failed. The judge says the way is not judged, never that there is none.
type SetuidNotJudged struct {
// Pending is true while a search runs; false after one failed.
Pending bool
Since time.Time
Err error
// Last is when the last complete search started, when there is one too old to judge.
Last time.Time
// Progress is how far the running search got, as of its last checkpoint: nil when none runs.
Progress *SearchProgress
}
// SearchProgress is how far a running search got, as of its last checkpoint, for anybody to read (novox/hq
// issue 361): counts, times and mount points only. **Never a path the walk visited or found**: the cursor and
// the files found are under accounts' homes as often as not, and stay in the engine's state and journal.
type SearchProgress struct {
// Started is when the search began, on the cache's clock.
Started time.Time
// Filesystems is how many it walks; Done how many it has walked to their end; Walking the mount point of
// the one it walks now, empty when every one is walked.
Filesystems, Done int
Walking string
// Visited is how many entries it has gone past; Found how many setuid- or setgid-root files it found so
// far, before the package manager is asked about them.
Visited int64
Found int
// AtLeast says Visited is a lower bound: the walk was begun by an engine that did not count.
AtLeast bool
// Kept is when its progress was last kept, zero before the first time.
Kept time.Time
// Finished is every filesystem walked to its end, in order, and how long its walk took.
Finished []SearchedFilesystem
}
// SearchedFilesystem is one filesystem a search walked to its end. Took is negative when not known: walked by
// an engine before it counted.
type SearchedFilesystem struct {
Mount string
Took time.Duration
}
// progressOf is p as anybody may read it. Counts and mount points, no I/O.
func progressOf(p *SetuidProgress, kept time.Time) *SearchProgress {
sp := &SearchProgress{Started: p.Started, Filesystems: len(p.Mounts), Done: len(p.Done), Visited: p.Visited,
Found: len(p.Found), Kept: kept, AtLeast: !p.Counted}
for i, m := range p.Done {
took := time.Duration(-1)
if i < len(p.Took) {
took = p.Took[i]
}
sp.Finished = append(sp.Finished, SearchedFilesystem{Mount: m, Took: took})
}
for _, m := range p.Mounts {
if !slices.Contains(p.Done, m) {
sp.Walking = m
break
}
}
return sp
}
// Words are the progress in one line, as the verdict says it beside "not judged yet".
func (sp *SearchProgress) Words() string {
if sp == nil {
return ""
}
s := fmt.Sprintf("progress: %d of %d filesystems walked", sp.Done, sp.Filesystems)
if sp.Walking != "" {
s += ", walking " + sp.Walking
}
atLeast := ""
if sp.AtLeast {
atLeast = "at least "
}
s += fmt.Sprintf(", %s%d entries visited, %d setuid- or setgid-root files found so far", atLeast, sp.Visited, sp.Found)
if sp.Kept.IsZero() {
s += ", not kept yet"
} else {
s += ", kept " + sp.Kept.Local().Format("2006-01-02 15:04:05")
}
var took []string
for _, f := range sp.Finished {
if f.Took < 0 {
took = append(took, f.Mount+" in a time not known")
} else {
took = append(took, f.Mount+" in "+f.Took.Round(time.Second).String())
}
}
if len(took) > 0 {
s += "; walked " + strings.Join(took, ", ")
}
return s
}
func (e *SetuidNotJudged) Error() string {
stale := ""
if !e.Last.IsZero() {
stale = fmt.Sprintf("; the last complete one, started %s, is too old to judge", e.Last.Local().Format("2006-01-02 15:04"))
}
if e.Pending {
return fmt.Sprintf("%s: the search for setuid programs, started %s, has not finished yet%s", ReasonPending,
e.Since.Local().Format("2006-01-02 15:04"), stale)
}
return fmt.Sprintf("%s: the search for setuid programs did not complete (last tried %s: %v); it is tried "+
"again later%s", ReasonIncomplete, e.Since.Local().Format("2006-01-02 15:04"), e.Err, stale)
}
// The reasons of a root verdict the setuid search could not answer.
const (
ReasonPending = rootsearch.ReasonPending
ReasonIncomplete = rootsearch.ReasonIncomplete
)
func (c *SetuidCache) every() time.Duration {
if c.Every > 0 {
return c.Every
}
return rootsearch.Every
}
func (c *SetuidCache) freshFor() time.Duration {
if c.FreshFor > 0 {
return c.FreshFor
}
return rootsearch.FreshFor
}
// SearchAgain asks for a new search at the next look: an apply changed what root controls (an account, a group,
// a sudo rule, a package, an action). The last complete search keeps judging until it is too old: only root acts
// through an apply, and the agent account cannot make a setuid-root file. Nil c is nothing to ask.
func (c *SetuidCache) SearchAgain() {
if c == nil {
return
}
c.mu.Lock()
defer c.mu.Unlock()
c.again = true
// The walk under way, or kept, started before root acted: never carried on, even by an engine that
// restarts before the next search starts (the review of 2026-10-10). It keeps no progress from now on.
c.unfinished = nil
c.forgetProgress()
}
// SearchNow throws away the last complete search, kept or in memory, and any walk under way or kept, and has
// the next look start a full search at once, past AgainAfter and any back-off (novox/hq issue 361): the
// operator changed by hand what the last one found, which no apply says. Until the new one completes the
// verdict is "not judged yet". Asked at the controller's terminal alone (`nox node setuid-search <node>`): a
// search can never make a machine free wrongly, but asked again and again it would keep a machine unjudged
// and its disks busy. by is who asked, for the journal. Nil c has nothing to throw away.
func (c *SetuidCache) SearchNow(by string) string {
if c == nil {
return "this engine keeps no search for setuid programs"
}
c.mu.Lock()
defer c.mu.Unlock()
c.loaded = true // nothing kept is read after this: it is thrown away
what := "no complete search was kept"
if c.last != nil {
what = "the complete search started " + c.last.Started.Local().Format("2006-01-02 15:04") + " is thrown away"
}
if c.running {
what += ", and the search running since " + c.startedAt.Local().Format("2006-01-02 15:04") + " is ended"
if c.cancel != nil {
c.cancel()
}
}
c.gen++
c.last, c.unfinished, c.progress = nil, nil, nil
c.failed, c.failedAt, c.backoff, c.again = nil, time.Time{}, 0, false
c.forgetProgress()
if c.Path != "" {
if err := os.Remove(c.Path); err != nil && !errors.Is(err, fs.ErrNotExist) {
c.say("the search for setuid programs could not forget the complete search it kept: " + err.Error())
}
}
said := "a new search for setuid programs starts at the next look, asked by " + by + ": " + what +
"; until it completes, whether an account can become root through one is not judged"
c.say(said)
return said
}
// SetuidSearch is one search: it carries p on to its end — walking what p has not walked, calling checkpoint
// as it goes so its progress can be kept — and answers the setuid-root programs no package owns.
type SetuidSearch func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error)
// Look answers the setuid-root programs no package owns as the last complete search found them, and when it
// started — while it is fresh — and starts a search in the background when one is due; it never waits for one.
// mounts are the filesystems a new search walks. Nil c searches in place, for a caller that wants to.
//
// **One clock: the caller's.** A search's time is the `now` it was started at, never the wall clock read when it
// finished: due-ness compares a later `now` with it, and mixing the two clocks made a search never due again
// whenever the caller's clock ran behind the wall clock (found while checking issue 356). Measured from the
// start, a result reads at most one search older than it is, which errs towards not judging.
func (c *SetuidCache) Look(now time.Time, mounts []string, search SetuidSearch) ([]string, time.Time, error) {
if c == nil {
p := &SetuidProgress{Started: now, Mounts: slices.Clone(mounts), Counted: true}
found, err := search(context.Background(), p, func() {})
return found, now, err
}
c.mu.Lock()
defer c.mu.Unlock()
if !c.loaded {
c.loaded = true
c.last = c.loadResult()
c.unfinished = c.loadProgress()
}
var age time.Duration
if c.last != nil {
age = now.Sub(c.last.Started)
}
// Fresh only while it walked every filesystem a search would walk now: one mounted since is not judged
// by a search that never saw it.
fresh := c.last != nil && age > -clockSkew && age < c.freshFor() && covers(c.last.Mounts, mounts)
// Asked for (an apply, a new filesystem): at most once an AgainAfter, counted from when the last search
// ended, so a walk of hours is not followed by another at once, apply after apply.
gap := now.Sub(c.attemptAt)
if !c.endedAt.IsZero() {
gap = now.Sub(c.endedAt)
}
due := !c.running && (c.failed == nil || now.Sub(c.failedAt) >= c.backoff) &&
(!fresh || age >= c.every() || (c.again && (c.attemptAt.IsZero() || gap >= AgainAfter)))
if due {
c.start(now, mounts, search)
}
switch {
case fresh:
return slices.Clone(c.last.Found), c.last.Started, nil
case c.running:
return nil, time.Time{}, &SetuidNotJudged{Pending: true, Since: c.startedAt, Last: c.lastStarted(),
Progress: c.progress}
case c.failed != nil:
return nil, time.Time{}, &SetuidNotJudged{Since: c.failedAt, Err: c.failed, Last: c.lastStarted()}
default:
// Not fresh, not running, never failed: cannot be, as not fresh is due. Never a verdict.
return nil, time.Time{}, &SetuidNotJudged{Pending: true, Since: now, Last: c.lastStarted()}
}
}
func (c *SetuidCache) lastStarted() time.Time {
if c.last == nil {
return time.Time{}
}
return c.last.Started
}
// resumable is the walk that did not finish, when a search may carry it on: younger than FreshFor, and no
// SearchAgain since — an apply that moved root's things wants a walk that starts after it. Anything else is
// thrown away. c.mu held.
func (c *SetuidCache) resumable(now time.Time) *SetuidProgress {
p := c.unfinished
c.unfinished = nil
if p == nil {
return nil
}
age := now.Sub(p.Started)
if c.again || age <= -clockSkew || age >= c.freshFor() {
c.forgetProgress()
return nil
}
return p
}
// start runs one search in the background, with no bound: a walk that did not finish carried on, or a new one.
// c.mu held.
func (c *SetuidCache) start(now time.Time, mounts []string, search SetuidSearch) {
p := c.resumable(now)
resumed := p != nil
if p == nil {
p = &SetuidProgress{ID: fmt.Sprintf("%x", now.UnixNano()), Started: now, Mounts: slices.Clone(mounts),
Cursor: map[string]string{}, Counted: true}
}
c.running, c.startedAt, c.attemptAt, c.again = true, p.Started, now, false
c.starts++
// What a look says until the first checkpoint: where the kept walk was, or nothing walked.
kept := p.Kept
c.progress = progressOf(p, kept)
if resumed {
c.say(fmt.Sprintf("the search for setuid programs started %s carries on where it was (%d of %d "+
"filesystems walked)", p.Started.Local().Format("2006-01-02 15:04"), len(p.Done), len(p.Mounts)))
}
began := time.Now()
gen := c.gen
ctx, cancel := context.WithCancel(context.Background())
c.cancel = cancel
c.wg.Add(1)
go func() {
defer c.wg.Done()
var found []string
var err error
if c.Lower != nil {
// The goroutine stays on its lowered thread until it ends, and the thread ends with it.
if lerr := c.Lower(); lerr != nil {
err = fmt.Errorf("its priority could not be lowered, so it did not walk: %w", lerr)
}
}
if err == nil {
unkept := false
found, err = search(ctx, p, func() {
c.mu.Lock()
superseded := c.again || c.gen != gen
c.mu.Unlock()
if !superseded { // root acted since this walk started: its progress is not to be carried on
at := time.Now()
p.Kept = at
kerr := c.keepProgress(p)
// Superseded while it wrote (SearchAgain, SearchNow), after they forgot the kept walk: forgotten
// again here, so no engine restarted before this walk ends carries it on (the review of
// 2026-10-10). Written outside the lock, so a look never waits on the disk.
c.mu.Lock()
if c.again || c.gen != gen {
c.forgetProgress()
}
c.mu.Unlock()
if kerr != nil {
p.Kept = kept // not kept: the file still says the time before
if !unkept {
unkept = true
c.say("the search for setuid programs could not keep how far it got, and starts over " +
"if the engine stops: " + kerr.Error())
}
} else {
kept = at
}
}
// What a look says from now: computed from p here, on the walk's own goroutine, never read
// by a look while the walk changes it.
sp := progressOf(p, kept)
c.mu.Lock()
if c.gen == gen {
c.progress = sp
}
c.mu.Unlock()
})
}
took := time.Since(began)
cancel()
c.mu.Lock()
defer c.mu.Unlock()
c.running, c.progress, c.cancel = false, nil, nil
c.endedAt = now.Add(took)
if c.gen != gen {
// Thrown away by SearchNow while it ran: neither a result nor a failure, and nothing to carry on. The
// next look starts the search asked for.
c.endedAt = time.Time{}
// A checkpoint that read its generation just before SearchNow may have kept this walk again.
c.forgetProgress()
c.say(fmt.Sprintf("the search for setuid programs started %s was ended, as asked (%s this time)",
p.Started.Local().Format("2006-01-02 15:04"), took.Round(time.Second)))
return
}
if err != nil {
// The back-off counts from when it failed.
c.failed, c.failedAt = err, c.endedAt
c.backoff = min(max(2*c.backoff, FailedBackoff), MaxBackoff)
if !c.again { // superseded by an ask to search again: never carried on
c.unfinished = p
}
c.say(fmt.Sprintf("the search for setuid programs started %s did not complete (%s this time): %v",
p.Started.Local().Format("2006-01-02 15:04"), took.Round(time.Second), err))
return
}
r := &setuidResult{Started: p.Started, Finished: now.Add(took), Mounts: p.Mounts, Found: found}
c.last, c.failed, c.backoff = r, nil, 0
if err := c.keepResult(r); err != nil {
c.say("the search for setuid programs could not be kept, and judges from memory until the engine " +
"restarts: " + err.Error())
}
c.forgetProgress()
// The paths in the engine's journal alone: the verdict says one in a home by count.
named := ""
if len(found) > 0 {
named = ": " + strings.Join(found, ", ")
}
c.say(fmt.Sprintf("the search for setuid programs started %s completed (%s this time): %d no package "+
"owns%s", p.Started.Local().Format("2006-01-02 15:04"), took.Round(time.Second), len(found), named))
}()
}
func (c *SetuidCache) say(s string) {
if c.Say != nil {
c.Say(s)
}
}
// covers says every filesystem a search would walk now was walked.
func covers(walked, now []string) bool {
for _, m := range now {
if !slices.Contains(walked, m) {
return false
}
}
return true
}
// Wait is for a test: until the search running, if any, has finished.
func (c *SetuidCache) Wait() { c.wg.Wait() }
func (c *SetuidCache) keepResult(r *setuidResult) error { return keepFile(c.Path, r) }
func (c *SetuidCache) keepProgress(p *SetuidProgress) error { return keepFile(c.ProgressPath, p) }
// forgetProgress removes the kept walk: finished, or thrown away.
func (c *SetuidCache) forgetProgress() {
if c.ProgressPath == "" {
return
}
if err := os.Remove(c.ProgressPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
c.say("the search for setuid programs could not forget the walk it kept: " + err.Error())
}
}
// keepFile writes v to path: a new file beside it, readable by its owner alone, renamed over it — a link at
// path is replaced, never followed. Empty path keeps nothing.
func keepFile(path string, v any) error {
if path == "" {
return nil
}
raw, err := json.Marshal(v)
if err != nil {
return err
}
f, err := os.CreateTemp(filepath.Dir(path), "."+filepath.Base(path)+".*")
if err != nil {
return err
}
defer os.Remove(f.Name())
if _, err := f.Write(raw); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
return err
}
return os.Rename(f.Name(), path)
}
// loadResult reads the kept complete search, or nil: none kept, or one that cannot be believed, said why.
func (c *SetuidCache) loadResult() *setuidResult {
var r setuidResult
if !c.loadFile(c.Path, "complete search", &r) {
return nil
}
if r.Started.IsZero() || r.Finished.Before(r.Started) || len(r.Mounts) == 0 {
c.say("the kept search for setuid programs says no complete search, and a new one runs")
return nil
}
return &r
}
// loadProgress reads the kept walk, or nil.
func (c *SetuidCache) loadProgress() *SetuidProgress {
var p SetuidProgress
if !c.loadFile(c.ProgressPath, "walk under way", &p) {
return nil
}
if p.Started.IsZero() || len(p.Mounts) == 0 {
c.say("the kept walk for setuid programs says no walk, and it starts over")
c.forgetProgress()
return nil
}
if p.Cursor == nil {
p.Cursor = map[string]string{}
}
return &p
}
func (c *SetuidCache) loadFile(path, what string, v any) bool {
if path == "" {
return false
}
raw, err := keptFile(path)
if errors.Is(err, fs.ErrNotExist) {
return false
}
if err == nil {
if err = json.Unmarshal(raw, v); err != nil {
err = fmt.Errorf("%s is not JSON: %w", path, err)
}
}
if err != nil {
c.say(fmt.Sprintf("the kept %s for setuid programs is not read, and a new one runs: %v", what, err))
return false
}
return true
}
// keptFile reads a kept file, believing it only as a regular file, not followed through a link, owned by the
// engine's own account and writable by it alone, in a directory that is the same: anything else could have been
// written by somebody other than root, and a search it says is no search.
func keptFile(path string) ([]byte, error) {
if err := ownedAlone(filepath.Dir(path), true); err != nil {
return nil, err
}
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NOFOLLOW, 0)
if err != nil {
return nil, err
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return nil, err
}
if err := ownedAloneInfo(path, info, false); err != nil {
return nil, err
}
return io.ReadAll(io.LimitReader(f, 64<<20))
}
func ownedAlone(path string, dir bool) error {
info, err := os.Lstat(path)
if err != nil {
return err
}
return ownedAloneInfo(path, info, dir)
}
func ownedAloneInfo(path string, info fs.FileInfo, dir bool) error {
if dir && !info.IsDir() || !dir && !info.Mode().IsRegular() {
return fmt.Errorf("%s is not a plain %s", path, map[bool]string{true: "directory", false: "file"}[dir])
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return fmt.Errorf("%s's owner could not be read", path)
}
if int(st.Uid) != os.Geteuid() {
return fmt.Errorf("%s is owned by uid %d, not the engine's %d", path, st.Uid, os.Geteuid())
}
if info.Mode().Perm()&0o022 != 0 {
return fmt.Errorf("%s may be written by others than its owner (mode %o)", path, info.Mode().Perm())
}
return nil
}
+447
View File
@@ -0,0 +1,447 @@
package accounts
import (
"context"
"encoding/json"
"errors"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/rootsearch"
)
// The search for setuid programs runs to completion and judges from its last complete result while it is fresh
// (novox/hq issue 361): never a bound that ends it, never a verdict from a search that did not finish or is too
// old, a new search after root acted, and the result kept beside the engine's state, believed only as root's.
var at = time.Date(2026, 10, 10, 1, 0, 0, 0, time.UTC)
var root = []string{"/"}
// gate is a search that waits until released; each search started hands its context on started. found and
// err are read when it is released.
type gate struct {
release chan struct{}
started chan context.Context
found []string
err error
}
func newGate() *gate {
return &gate{release: make(chan struct{}, 16), started: make(chan context.Context, 16)}
}
func (g *gate) search(ctx context.Context, p *SetuidProgress, _ func()) ([]string, error) {
g.started <- ctx
<-g.release
return g.found, g.err
}
func (g *gate) finish(c *SetuidCache) { g.release <- struct{}{}; c.Wait() }
// starts is how many searches c started.
func starts(c *SetuidCache) int {
c.mu.Lock()
defer c.mu.Unlock()
return c.starts
}
func notJudged(t *testing.T, err error, pending bool) *SetuidNotJudged {
t.Helper()
var nj *SetuidNotJudged
if !errors.As(err, &nj) || nj.Pending != pending {
t.Fatalf("want not judged (pending %v), got %v", pending, err)
}
want := ReasonIncomplete
if pending {
want = ReasonPending
}
if !strings.HasPrefix(err.Error(), want) {
t.Fatalf("the words start %q: %v", want, err)
}
return nj
}
func TestTheSearchRunsToCompletionWithNoBound(t *testing.T) {
c := &SetuidCache{}
g := newGate()
if _, _, err := c.Look(at, root, g.search); err == nil {
t.Fatal("a look before the first search finished judged")
}
ctx := <-g.started
if _, ok := ctx.Deadline(); ok {
t.Fatal("the search was given a deadline: it must run until it has walked everything")
}
// Hours later, still running: still pending, never killed, never a second walk beside it.
_, _, err := c.Look(at.Add(5*time.Hour), root, g.search)
notJudged(t, err, true)
if starts(c) != 1 || ctx.Err() != nil {
t.Fatalf("a running search was started again or stopped: %d searches, %v", starts(c), ctx.Err())
}
g.found = []string{"/opt/x"}
g.finish(c)
found, when, err := c.Look(at.Add(5*time.Hour), root, g.search)
if err != nil || len(found) != 1 || !when.Equal(at) {
t.Fatalf("the complete search judges, from its start: %v %v %v", found, when, err)
}
}
func TestAFreshCompleteSearchJudgesUntilItIsDue(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
if _, _, err := c.Look(at.Add(rootsearch.Every-time.Minute), root, g.search); err != nil {
t.Fatalf("a complete search within a day: %v", err)
}
if starts(c) != 1 {
t.Fatalf("searched again before it was due: %d", starts(c))
}
// Due: a new search starts, and the last complete one judges meanwhile.
if _, _, err := c.Look(at.Add(rootsearch.Every), root, g.search); err != nil {
t.Fatalf("while the daily search runs, the last complete one still judges: %v", err)
}
if starts(c) != 2 {
t.Fatalf("the daily search did not start: %d", starts(c))
}
g.finish(c)
}
func TestAStaleSearchIsNotJudgedNeverFree(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
c.Look(at.Add(rootsearch.Every), root, g.search) // the daily one, which does not finish
_, _, err := c.Look(at.Add(rootsearch.FreshFor), root, g.search)
nj := notJudged(t, err, true)
if !nj.Last.Equal(at) || !strings.Contains(err.Error(), "too old to judge") {
t.Fatalf("a stale search says when the last complete one started: %v", err)
}
g.finish(c)
if _, _, err := c.Look(at.Add(rootsearch.FreshFor), root, g.search); err != nil {
t.Fatalf("once the new search completed it judges again: %v", err)
}
}
func TestAnIncompleteSearchIsNeverFree(t *testing.T) {
c := &SetuidCache{}
g := newGate()
g.err = errors.New("the search for setuid programs did not complete its walk")
c.Look(at, root, g.search)
g.finish(c)
_, _, err := c.Look(at.Add(time.Minute), root, g.search)
notJudged(t, err, false)
if starts(c) != 1 {
t.Fatalf("tried again within its back-off: %d", starts(c))
}
c.Look(at.Add(FailedBackoff+time.Second), root, g.search)
if starts(c) != 2 {
t.Fatalf("not tried again after its back-off: %d", starts(c))
}
g.finish(c)
// A stale complete search and a failed new one: still not judged, never the stale one.
c = &SetuidCache{}
g = newGate()
c.Look(at, root, g.search)
g.finish(c)
g.err = errors.New("pacman could not say who owns the setuid programs found")
c.Look(at.Add(rootsearch.Every), root, g.search)
g.finish(c)
if _, _, err := c.Look(at.Add(rootsearch.Every+time.Hour), root, g.search); err != nil {
t.Fatalf("a failed search leaves the fresh complete one judging: %v", err)
}
_, _, err = c.Look(at.Add(rootsearch.FreshFor), root, g.search)
if err == nil {
t.Fatal("a stale search judged after the newer one failed")
}
g.finish(c)
}
func TestAnApplyThatMovedRootSearchesAgain(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
c.SearchAgain()
if _, _, err := c.Look(at.Add(time.Minute), root, g.search); err != nil || starts(c) != 1 {
t.Fatalf("asked again within AgainAfter of the last search's end: %v, %d searches", err, starts(c))
}
if _, _, err := c.Look(at.Add(AgainAfter+time.Second), root, g.search); err != nil {
t.Fatalf("the last complete search judges while the new one runs: %v", err)
}
if starts(c) != 2 {
t.Fatalf("an apply that moved root did not search again: %d", starts(c))
}
// Asked again while one runs: one more after it, never two at once.
c.SearchAgain()
c.Look(at.Add(AgainAfter+time.Minute), root, g.search)
if starts(c) != 2 {
t.Fatalf("a second search started beside the running one: %d", starts(c))
}
g.finish(c) // ends, on the cache's clock, at about AgainAfter+1s
c.Look(at.Add(2*AgainAfter), root, g.search)
if starts(c) != 2 {
t.Fatal("a search started again less than AgainAfter after the last one ended")
}
c.Look(at.Add(2*AgainAfter+2*time.Second), root, g.search)
if starts(c) != 3 {
t.Fatalf("the ask made while a search ran was dropped: %d", starts(c))
}
g.finish(c)
var nilCache *SetuidCache
nilCache.SearchAgain()
}
// A filesystem the last complete search did not walk is not judged by it: not judged yet, and a search starts.
func TestAFilesystemTheSearchDidNotWalkIsNotJudgedByIt(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
_, _, err := c.Look(at.Add(time.Hour), []string{"/", "/srv"}, g.search)
notJudged(t, err, true)
if starts(c) != 2 {
t.Fatalf("a new filesystem was not searched: %d", starts(c))
}
g.finish(c)
if _, _, err := c.Look(at.Add(time.Hour), []string{"/", "/srv"}, g.search); err != nil {
t.Fatalf("once walked it judges: %v", err)
}
}
// An ask to search again forgets the walk under way, so an engine restarted before the next search does not
// carry on a walk that started before root acted.
func TestAnAskToSearchAgainIsNotLostToARestart(t *testing.T) {
dir := t.TempDir()
paths := func() *SetuidCache {
return &SetuidCache{Path: filepath.Join(dir, SetuidSearchFile), ProgressPath: filepath.Join(dir, SetuidProgressFile)}
}
seen := make(chan *SetuidProgress, 1)
first := paths()
first.Look(at, root, interrupted(seen))
<-seen
first.SearchAgain() // an apply moved root's things; then the engine restarts
restarted := paths()
var with SetuidProgress
restarted.Look(at.Add(time.Hour), root, func(ctx context.Context, p *SetuidProgress, _ func()) ([]string, error) {
with = *p
return nil, nil
})
restarted.Wait()
if !with.Started.Equal(at.Add(time.Hour)) || len(with.Found) != 0 {
t.Fatalf("the walk from before the apply was carried on: %+v", with)
}
}
// A cursor whose name is not UTF-8 is kept as its bytes: JSON strings would change it, and the resumed walk
// would skip what it stood before.
func TestACursorOfAnyBytesIsKeptExactly(t *testing.T) {
p := &SetuidProgress{Started: at, Mounts: Paths{"/"}, Cursor: Cursors{"/": "/home/agent/\x80/x"},
Found: Paths{"/opt/\xff"}}
var back SetuidProgress
fromJSON(t, mustJSON(t, p), &back)
if back.Cursor["/"] != "/home/agent/\x80/x" || back.Found[0] != "/opt/\xff" {
t.Fatalf("kept as %q, %q", back.Cursor["/"], back.Found[0])
}
}
func TestTheCompleteSearchIsKeptAndReadAfterARestart(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, SetuidSearchFile)
c := &SetuidCache{Path: path}
g := newGate()
g.found = []string{"/opt/x"}
c.Look(at, root, g.search)
g.finish(c)
info, err := os.Stat(path)
if err != nil || info.Mode().Perm() != 0o600 {
t.Fatalf("kept readable by its owner alone: %v %v", info, err)
}
restarted := &SetuidCache{Path: path}
found, when, err := restarted.Look(at.Add(time.Hour), root, newGate().search)
if err != nil || len(found) != 1 || !when.Equal(at) {
t.Fatalf("a restarted engine judges from the kept search: %v %v %v", found, when, err)
}
restarted.Wait()
// And not once it is too old.
again := &SetuidCache{Path: path}
g2 := newGate()
if _, _, err := again.Look(at.Add(rootsearch.FreshFor), root, g2.search); err == nil {
t.Fatal("a kept search too old to judge judged")
}
g2.finish(again)
}
func TestAKeptSearchNotRootsAloneIsNotRead(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, SetuidSearchFile)
c := &SetuidCache{Path: path}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
for name, spoil := range map[string]func() error{
"writable by others": func() error { return os.Chmod(path, 0o666) },
"a link": func() error {
other := filepath.Join(dir, "elsewhere.json")
if err := os.Rename(path, other); err != nil {
return err
}
return os.Symlink(other, path)
},
"its directory writable by others": func() error { return os.Chmod(dir, 0o777) },
} {
c = &SetuidCache{Path: path}
g = newGate()
c.Look(at, root, g.search)
g.finish(c) // a fresh, good one kept again
os.Chmod(dir, 0o700)
if err := spoil(); err != nil {
t.Fatal(err)
}
var said []string
r := &SetuidCache{Path: path, Say: func(s string) { said = append(said, s) }}
gr := newGate()
if _, _, err := r.Look(at.Add(time.Minute), root, gr.search); err == nil {
t.Fatalf("%s: a kept search that is not root's alone judged", name)
}
if len(said) == 0 {
t.Fatalf("%s: refusing the kept search said nothing", name)
}
gr.finish(r)
os.Chmod(dir, 0o700)
os.Remove(path)
}
}
func TestAKeptSearchFromTheFutureIsNotFresh(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
if _, _, err := c.Look(at.Add(-time.Hour), root, g.search); err == nil {
t.Fatal("a clock set back an hour read a search as fresh")
}
g.finish(c)
}
func TestTheIdleRunnerLowersItsCommand(t *testing.T) {
if _, err := exec.LookPath("nice"); err != nil {
t.Skip("no nice here")
}
out, err := Idle(t.Context(), "sh", "-c", "sleep 0.3; nice")
if err != nil {
t.Fatal(err)
}
if strings.TrimSpace(out) != "19" {
t.Fatalf("the search's commands run at nice %q, want 19", strings.TrimSpace(out))
}
if _, err := Idle(t.Context(), "sh", "-c", "echo gone >&2; exit 1"); err == nil ||
!strings.HasPrefix(err.Error(), "sh exited 1: gone") {
t.Fatalf("its failure is worded as CLocale's: %v", err)
}
}
func mustJSON(t *testing.T, v any) []byte {
t.Helper()
raw, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return raw
}
func fromJSON(t *testing.T, raw []byte, v any) {
t.Helper()
if err := json.Unmarshal(raw, v); err != nil {
t.Fatal(err)
}
}
// interrupted is a search that walks to a cursor, keeps it, and never returns: the engine stopped there.
func interrupted(seen chan<- *SetuidProgress) SetuidSearch {
return func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error) {
p.Found = append(p.Found, "/opt/a")
p.Cursor["/"] = "/opt/a"
checkpoint()
seen <- p
select {} // the engine stops here
}
}
func TestASearchInterruptedByARestartCarriesOnAndIsDatedByItsStart(t *testing.T) {
dir := t.TempDir()
paths := func() *SetuidCache {
return &SetuidCache{Path: filepath.Join(dir, SetuidSearchFile), ProgressPath: filepath.Join(dir, SetuidProgressFile)}
}
seen := make(chan *SetuidProgress, 1)
first := paths()
first.Look(at, root, interrupted(seen))
<-seen
// The engine restarts two hours later.
restarted := paths()
var resumedWith SetuidProgress
done := make(chan struct{})
_, _, err := restarted.Look(at.Add(2*time.Hour), root, func(ctx context.Context, p *SetuidProgress, _ func()) ([]string, error) {
resumedWith = *p
close(done)
return p.Found, nil
})
notJudged(t, err, true)
<-done
restarted.Wait()
if !resumedWith.Started.Equal(at) || resumedWith.Cursor["/"] != "/opt/a" || len(resumedWith.Found) != 1 {
t.Fatalf("the restarted search did not carry on the walk: %+v", resumedWith)
}
found, when, err := restarted.Look(at.Add(2*time.Hour), root, newGate().search)
if err != nil || !when.Equal(at) || len(found) != 1 {
t.Fatalf("the complete search is dated by its start, not its resumption: %v %v %v", found, when, err)
}
if _, err := os.Stat(filepath.Join(dir, SetuidProgressFile)); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("the finished walk's progress is still kept: %v", err)
}
}
func TestAWalkTooOldOrAskedAgainStartsOver(t *testing.T) {
for name, c := range map[string]struct {
after time.Duration
again bool
}{"older than it may judge": {rootsearch.FreshFor, false}, "an apply asked for a new one": {time.Hour, true}} {
dir := t.TempDir()
paths := func() *SetuidCache {
return &SetuidCache{Path: filepath.Join(dir, SetuidSearchFile), ProgressPath: filepath.Join(dir, SetuidProgressFile)}
}
seen := make(chan *SetuidProgress, 1)
paths().Look(at, root, interrupted(seen))
<-seen
restarted := paths()
if c.again {
restarted.SearchAgain()
}
var with SetuidProgress
restarted.Look(at.Add(c.after), root, func(ctx context.Context, p *SetuidProgress, _ func()) ([]string, error) {
with = *p
return nil, nil
})
restarted.Wait()
if !with.Started.Equal(at.Add(c.after)) || len(with.Cursor) != 0 || len(with.Found) != 0 {
t.Fatalf("%s: the old walk was carried on: %+v", name, with)
}
}
}
func TestASearchWhosePriorityCannotBeLoweredDoesNotWalk(t *testing.T) {
c := &SetuidCache{Lower: func() error { return errors.New("not permitted") }}
walked := false
c.Look(at, root, func(context.Context, *SetuidProgress, func()) ([]string, error) { walked = true; return nil, nil })
c.Wait()
_, _, err := c.Look(at.Add(time.Minute), root, newGate().search)
notJudged(t, err, false)
if walked {
t.Fatal("it walked at full priority")
}
}
+175
View File
@@ -0,0 +1,175 @@
package accounts
// The walk for setuid programs yields to a busy disk (novox/hq issue 433). It asks before each directory and each
// file it reads, so no burst between two looks is longer than PaceLook, however large a directory. The walk is put in the idle I/O class
// (idle_linux.go), but only a disk scheduler that schedules by class honours that (BFQ does). The control-node's
// disks run mq-deadline, which does not, and there the walk read at full priority: on 2026-10-11 it kept two
// spinning disks saturated for sixteen minutes, the bus whose store shares those disks stalled for half a minute
// twice, and the controller lost its lease each time. So the walk also paces itself by the machine's I/O pressure
// (PSI), which reads the same whatever the scheduler.
//
// It looks every PaceLook at the share of the time since its last look in which some task waited on I/O — from
// the kernel's cumulative stall counter, not its ten-second average, which moves only every two seconds and would
// let the walk read blind that long. Above PaceAbove it pauses, PaceStep at a time, until the share falls below.
//
// **The walk keeps at least a tenth of the time, whatever the pressure** (the review of #79): a run of pauses is
// at most PaceRatio times as long as the walk read since the one before. The control-node's own load sits near
// PaceAbove with no walk running, so a walk that paused for as long as the pressure stayed high would take days,
// outlive the age a search judges for, and leave the machine unjudged — the fault issue 361 repaired. With the
// floor a walk takes at most 1 + PaceRatio times as long as it would unpaced, and its bursts are PaceLook long.
import (
"bufio"
"bytes"
"context"
"fmt"
"os"
"strconv"
"strings"
"time"
)
const (
// PaceAbove is the share of the time since the last look, in percent, in which some task waited on I/O, above
// which the walk pauses.
PaceAbove = 10.0
// PaceLook is how long the walk reads between two looks at the pressure: the longest it reads blind.
PaceLook = 250 * time.Millisecond
// PaceStep is one pause, after which the pressure is looked at again.
PaceStep = 500 * time.Millisecond
// PaceRatio bounds a run of pauses to this many times the time the walk read since the run before: the walk
// keeps at least 1/(1+PaceRatio) of the time.
PaceRatio = 9
// PaceAtMost is the longest run of pauses, whatever the walk read before it.
PaceAtMost = 30 * time.Second
// ioPressureFile is where the kernel says the machine's I/O pressure.
ioPressureFile = "/proc/pressure/io"
)
// pacer is what the walk asks before it reads a directory. A nil pacer never pauses.
type pacer struct {
// path is the pressure file read, kept so a test can see the engine's own walker reads the machine's.
path string
// stalled is the cumulative time some task waited on I/O (PSI's "some total").
stalled func() (time.Duration, error)
sleep func(context.Context, time.Duration) error
now func() time.Time
say func(string)
lastLook time.Time // when the pressure was last looked at
readSince time.Time // when the walk last began to read, after a run of pauses or at its start
prevAt time.Time // the time of the last sample of the counter
prevTotal time.Duration // the counter at that sample
saidBlind bool
}
// newPacer paces by the machine's own I/O pressure, and says through say what it cannot do.
func newPacer(say func(string)) *pacer {
return &pacer{
path: ioPressureFile,
stalled: func() (time.Duration, error) { return readIOStalled(ioPressureFile) },
sleep: sleepCtx,
now: time.Now,
say: say,
}
}
// yield pauses while the machine's I/O pressure is above PaceAbove: looking at most every PaceLook, pausing
// PaceStep at a time, a run of pauses at most PaceRatio times what the walk read since the last run and never more
// than PaceAtMost. It ends early only with ctx, whose error ends the walk as incomplete. Pressure that cannot be
// read pauses nothing, and is said once.
func (pc *pacer) yield(ctx context.Context) error {
if pc == nil || pc.stalled == nil {
return nil
}
now := pc.now()
if pc.readSince.IsZero() {
pc.readSince = now
}
if !pc.lastLook.IsZero() && now.Sub(pc.lastLook) < PaceLook {
return nil
}
pc.lastLook = now
share, known, err := pc.sample(now)
if err != nil || !known || share <= PaceAbove {
return nil
}
budget := min(PaceAtMost, PaceRatio*now.Sub(pc.readSince))
for paused := time.Duration(0); paused < budget; {
step := min(PaceStep, budget-paused)
if err := pc.sleep(ctx, step); err != nil {
return err
}
paused += step
share, known, err = pc.sample(pc.now())
if err != nil || (known && share <= PaceAbove) {
break
}
}
resumed := pc.now()
pc.readSince, pc.lastLook = resumed, resumed
return nil
}
// sample is the share of the time since the last sample, in percent, in which some task waited on I/O; known is
// false for the first sample, which only sets where the next one counts from.
func (pc *pacer) sample(now time.Time) (float64, bool, error) {
total, err := pc.stalled()
if err != nil {
if !pc.saidBlind && pc.say != nil {
pc.say(fmt.Sprintf("the walk for setuid programs cannot read the machine's I/O pressure (%v), so it does "+
"not pace itself: where the disk's scheduler ignores the idle class, it may load the disk at full "+
"priority", err))
}
pc.saidBlind = true
return 0, false, err
}
prevAt, prevTotal := pc.prevAt, pc.prevTotal
pc.prevAt, pc.prevTotal = now, total
elapsed := now.Sub(prevAt)
if prevAt.IsZero() || elapsed <= 0 || total < prevTotal {
return 0, false, nil
}
return 100 * float64(total-prevTotal) / float64(elapsed), true, nil
}
// readIOStalled is the "some total" of a PSI file: the time, cumulative since boot, in which some task waited on I/O.
func readIOStalled(path string) (time.Duration, error) {
b, err := os.ReadFile(path)
if err != nil {
return 0, err
}
return parseIOStalled(b)
}
func parseIOStalled(b []byte) (time.Duration, error) {
sc := bufio.NewScanner(bytes.NewReader(b))
for sc.Scan() {
fields := strings.Fields(sc.Text())
if len(fields) == 0 || fields[0] != "some" {
continue
}
for _, f := range fields[1:] {
if v, ok := strings.CutPrefix(f, "total="); ok {
us, err := strconv.ParseUint(v, 10, 64)
if err != nil {
return 0, fmt.Errorf("the I/O pressure's total is not a count of microseconds: %q", v)
}
return time.Duration(us) * time.Microsecond, nil
}
}
}
return 0, fmt.Errorf("the I/O pressure names no \"some\" line with a total")
}
// sleepCtx waits d, or until ctx ends.
func sleepCtx(ctx context.Context, d time.Duration) error {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-t.C:
return nil
}
}
+302
View File
@@ -0,0 +1,302 @@
package accounts
import (
"context"
"errors"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/rootsearch"
)
// The walk for setuid programs yields to a busy disk by the machine's I/O pressure (novox/hq issue 433), because
// the idle I/O class it is put in is ignored by a disk scheduler that does not schedule by class, as the
// control-node's mq-deadline does not — and it keeps a tenth of the time whatever the pressure, so it still ends.
// fakePace is a pacer whose stall counter, sleeps and clock a test controls.
type fakePace struct {
// shares is the stall share, in percent, of each interval between two reads of the counter, in turn; the last
// is said again once they run out. The first read only sets the counter's start.
shares []float64
i int
total time.Duration
lastRead time.Time
reads int
clock time.Time
// tick moves the clock on at every look at it, as a walk reading directories takes time; zero does not.
tick time.Duration
slept []time.Duration
said []string
fail error
}
func newFake(shares ...float64) *fakePace {
return &fakePace{shares: shares, clock: time.Unix(1_000_000, 0)}
}
func (f *fakePace) pacer() *pacer {
return &pacer{
stalled: func() (time.Duration, error) {
if f.fail != nil {
return 0, f.fail
}
f.reads++
if !f.lastRead.IsZero() {
s := f.shares[min(f.i, len(f.shares)-1)]
f.i++
f.total += time.Duration(float64(f.clock.Sub(f.lastRead)) * s / 100)
}
f.lastRead = f.clock
return f.total, nil
},
sleep: func(ctx context.Context, d time.Duration) error {
if err := ctx.Err(); err != nil {
return err
}
f.slept = append(f.slept, d)
f.clock = f.clock.Add(d)
return nil
},
now: func() time.Time {
f.clock = f.clock.Add(f.tick)
return f.clock
},
say: func(s string) { f.said = append(f.said, s) },
}
}
func (f *fakePace) paused() time.Duration {
var total time.Duration
for _, d := range f.slept {
total += d
}
return total
}
func TestTheWalkPausesWhileTheDiskIsUnderPressureAndGoesOnWhenItEases(t *testing.T) {
f := newFake(60, 40, 3)
pc := f.pacer()
if err := pc.yield(t.Context()); err != nil { // the first look sets where the counter counts from
t.Fatal(err)
}
f.clock = f.clock.Add(300 * time.Millisecond) // the walk reads
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
// 60 % since the first look: pause; 40 % during the first pause: pause again; 3 %: go on.
if want := []time.Duration{PaceStep, PaceStep}; !slices.Equal(f.slept, want) {
t.Fatalf("slept %v, want %v", f.slept, want)
}
}
func TestTheFirstLookOnlySetsWhereTheNextCountsFrom(t *testing.T) {
f := newFake(99)
if err := f.pacer().yield(t.Context()); err != nil {
t.Fatal(err)
}
if len(f.slept) != 0 {
t.Fatalf("slept %v on the first look, which has nothing to compare", f.slept)
}
}
func TestTheWalkDoesNotPauseOnAQuietDisk(t *testing.T) {
f := newFake(0.4)
pc := f.pacer()
for range 10 {
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(PaceLook)
}
if len(f.slept) != 0 {
t.Fatalf("slept %v on a quiet disk", f.slept)
}
}
// The control-node's own load sits near the bound with no walk running: the walk must still end (issue 361).
func TestTheWalkKeepsATenthOfTheTimeWhateverThePressure(t *testing.T) {
f := newFake(100)
pc := f.pacer()
var read time.Duration
for range 400 {
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(PaceLook)
read += PaceLook
}
if f.paused() == 0 {
t.Fatal("never paused under constant pressure")
}
if f.paused() > PaceRatio*read {
t.Fatalf("paused %v while it read %v: more than %d times as long", f.paused(), read, PaceRatio)
}
}
// What the floor promises against the ages a search is judged by: the walk that took 16m31s unpaced on the
// control-node on 2026-10-11, slowed to its worst, still completes well inside the time the controller stays quiet
// while it waits for a search. A guard on the constants only: that the code keeps to the ratio is proven by
// TestTheWalkKeepsATenthOfTheTimeWhateverThePressure.
func TestTheSlowestPacedWalkStillJudgesInTime(t *testing.T) {
const unpaced = 16*time.Minute + 31*time.Second
worst := (1 + PaceRatio) * unpaced
if worst >= rootsearch.Quiet {
t.Fatalf("a walk paced to its worst takes %v, not less than the %v the controller waits quietly", worst,
rootsearch.Quiet)
}
}
func TestARunOfPausesIsAtMostPaceAtMost(t *testing.T) {
f := newFake(100)
pc := f.pacer()
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(10 * time.Minute) // read a long time without a look: the ratio would allow 90 minutes
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
if f.paused() != PaceAtMost {
t.Fatalf("paused %v in a row, want %v", f.paused(), PaceAtMost)
}
}
func TestTheWalkLooksAtThePressureAtMostEveryPaceLook(t *testing.T) {
f := newFake(1)
pc := f.pacer()
for range 5 {
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(PaceLook / 10)
}
if f.reads != 1 {
t.Fatalf("read the pressure %d times within %v, want once", f.reads, PaceLook)
}
f.clock = f.clock.Add(PaceLook)
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
if f.reads != 2 {
t.Fatalf("read the pressure %d times after %v, want twice", f.reads, PaceLook)
}
}
// Pressure that cannot be read pauses nothing, and is said once, with what it means, rather than at every look.
func TestAPressureThatCannotBeReadIsSaidOnce(t *testing.T) {
f := newFake(0)
f.fail = errors.New("no such file")
pc := f.pacer()
for range 3 {
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(PaceLook)
}
if len(f.slept) != 0 || len(f.said) != 1 {
t.Fatalf("slept %v, said %q; want no pause and one line", f.slept, f.said)
}
if !strings.Contains(f.said[0], "full priority") {
t.Errorf("the line does not say what it means for the disk: %q", f.said[0])
}
}
// The engine's own walk is paced by the machine's pressure, and says through the search's own channel.
func TestTheEnginesWalkIsPacedByTheMachinesPressure(t *testing.T) {
var said []string
w := engineWalker(nil, func(s string) { said = append(said, s) })
if w.pace == nil || w.pace.path != ioPressureFile || w.pace.stalled == nil {
t.Fatalf("the engine's walker is not paced by %s: %+v", ioPressureFile, w.pace)
}
w.pace.say("x")
if len(said) != 1 {
t.Fatal("the engine's walker does not say through the search's channel")
}
}
// many is a tree of enough directories that a walk reaches a look at the pressure partway.
func many(t *testing.T) string {
names := []string{"bin/", "bin/su!", "bin/ls", "sbin/", "sbin/mount!", "usr/lib/helper%"}
for _, d := range []string{"a", "b", "c", "d", "e", "f", "g", "h"} {
names = append(names, "usr/share/"+d+"/")
}
return tree(t, names...)
}
// A walk ended while it pauses is an incomplete walk, never one that found nothing.
func TestAWalkEndedWhileItPausesIsIncomplete(t *testing.T) {
top := many(t)
f := newFake(80)
f.tick = 100 * time.Millisecond
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
pc := f.pacer()
// The engine stops while the walk waits for the disk: the context ends inside the pause, not before the walk.
pc.sleep = func(ctx context.Context, d time.Duration) error {
f.slept = append(f.slept, d)
cancel()
return ctx.Err()
}
w := mine()
w.pace = pc
err := w.walk(ctx, &SetuidProgress{Mounts: []string{top}}, func() {})
if len(f.slept) == 0 {
t.Fatal("the walk never paused, so this proves nothing about a walk ended while it pauses")
}
if !errors.Is(err, context.Canceled) {
t.Fatalf("a walk ended while it paused answered %v", err)
}
}
// Paced, the walk finds what it finds unpaced: pausing changes when, never what.
func TestAPacedWalkFindsWhatAnUnpacedWalkFinds(t *testing.T) {
top := many(t)
plain := walkAll(t, mine(), top)
f := newFake(80, 2)
f.tick = 100 * time.Millisecond
w := mine()
w.pace = f.pacer()
paced := walkAll(t, w, top)
if !slices.Equal(plain, paced) {
t.Fatalf("paced walk found %v, unpaced %v", paced, plain)
}
if len(f.slept) == 0 {
t.Fatal("the walk read its directories without asking the pacer: it never paused under pressure")
}
}
func TestTheIOPressureIsReadAsTheKernelWritesIt(t *testing.T) {
got, err := parseIOStalled([]byte("some avg10=26.24 avg60=20.07 avg300=27.47 total=123456789\n" +
"full avg10=21.90 avg60=15.77 avg300=22.47 total=98765432\n"))
if err != nil || got != 123456789*time.Microsecond {
t.Fatalf("read %v, %v; want 123456789 µs", got, err)
}
for _, bad := range []string{"", "full avg10=1.00 avg60=0 avg300=0 total=1\n", "some avg10=1 total=x\n"} {
if _, err := parseIOStalled([]byte(bad)); err == nil {
t.Errorf("%q was read as a pressure", bad)
}
}
}
// One directory of many files is paced as it is read, not only between directories: a file's own details may be
// read from the disk too.
func TestADirectoryOfManyFilesIsPacedAsItIsRead(t *testing.T) {
names := []string{"bin/"}
for i := range 20 {
names = append(names, "bin/f"+strings.Repeat("x", i+1))
}
top := tree(t, names...)
f := newFake(80, 2)
f.tick = 100 * time.Millisecond
w := mine()
w.pace = f.pacer()
walkAll(t, w, top)
// Two directories alone would be looked at once, 200 ms apart, within one look: no pause at all.
if len(f.slept) == 0 {
t.Fatal("a directory of twenty files was read without a pause under pressure")
}
}
+218
View File
@@ -0,0 +1,218 @@
package accounts
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// How far a running search got is readable through the verdict (novox/hq issue 361): counts, times and mount
// points, counted once across a resume, and never a path the walk visited or found.
// A walk stopped midway and resumed counts every entry once, as the walk that was never stopped does, and says
// how long each filesystem took.
func TestTheProgressCountsEachEntryOnceAcrossAResume(t *testing.T) {
top := tree(t, "a/1!", "a/2", "a-b/3!", "b/c/4!", "b/c/5", "b/d!", "c/6!", "d/7!")
other := tree(t, "x/1", "y!")
whole := &SetuidProgress{Mounts: []string{top, other}}
if err := mine().walk(t.Context(), whole, func() {}); err != nil {
t.Fatal(err)
}
// top and the 14 entries below it, and other and its 3.
if whole.Visited != 15+4 || len(whole.Took) != 2 || whole.Took[0] < 0 || whole.Walking != 0 {
t.Fatalf("the whole walk: visited %d, took %v, walking %v", whole.Visited, whole.Took, whole.Walking)
}
w := mine()
w.every = 0
ctx, stop := context.WithCancel(t.Context())
p := &SetuidProgress{Mounts: []string{top, other}, Counted: true}
var kept []byte
err := w.walk(ctx, p, func() {
kept = mustJSON(t, p)
if strings.HasSuffix(p.Cursor[top], "/b/c/4") {
stop()
}
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("the walk was stopped: %v", err)
}
var resumed SetuidProgress
fromJSON(t, kept, &resumed)
if resumed.Visited == 0 || resumed.Visited >= whole.Visited || resumed.Walking < 0 {
t.Fatalf("the kept walk counted %d of %d", resumed.Visited, whole.Visited)
}
if err := mine().walk(t.Context(), &resumed, func() {}); err != nil {
t.Fatal(err)
}
if resumed.Visited != whole.Visited || len(resumed.Took) != 2 || resumed.Took[0] < 0 {
t.Fatalf("the resumed walk counted %d entries (the whole one %d), took %v", resumed.Visited,
whole.Visited, resumed.Took)
}
}
// A walk kept before the times were counted has them not known, never shifted onto the next filesystem, and
// its entries visited are a lower bound; the filesystem it was part-way through took a time not known.
func TestAWalkKeptWithoutTimesSaysThemNotKnown(t *testing.T) {
top, other := tree(t, "a"), tree(t, "b")
p := &SetuidProgress{Mounts: []string{top, other}, Done: Paths{top}}
if err := mine().walk(t.Context(), p, func() {}); err != nil {
t.Fatal(err)
}
if len(p.Took) != 2 || p.Took[0] >= 0 || p.Took[1] < 0 {
t.Fatalf("took %v", p.Took)
}
if w := progressOf(p, time.Time{}).Words(); !strings.Contains(w, top+" in a time not known") ||
!strings.Contains(w, "at least ") {
t.Fatalf("the words: %s", w)
}
}
// A look while the search runs says how far it got at its last checkpoint, with no path it visited or found.
func TestALookSaysHowFarTheRunningSearchGotAndNoPath(t *testing.T) {
dir := t.TempDir()
c := &SetuidCache{ProgressPath: filepath.Join(dir, SetuidProgressFile)}
secret := "/home/someone/private/rootshell"
checkpointed, release := make(chan struct{}), make(chan struct{})
search := func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error) {
p.Mounts = Paths{"/", "/home", "/boot"}
p.Done, p.Took = Paths{"/"}, []time.Duration{90 * time.Minute}
p.Cursor["/home"] = "/home/someone/private"
p.Found = Paths{"/usr/bin/odd", secret}
p.Visited = 1234567
checkpoint()
close(checkpointed)
<-release
return nil, nil
}
_, _, err := c.Look(at, root, search)
nj := notJudged(t, err, true)
if nj.Progress == nil || nj.Progress.Visited != 0 || nj.Progress.Walking != "/" {
t.Fatalf("before the first checkpoint: %+v", nj.Progress)
}
<-checkpointed
_, _, err = c.Look(at.Add(time.Minute), root, search)
nj = notJudged(t, err, true)
sp := nj.Progress
if sp == nil || sp.Filesystems != 3 || sp.Done != 1 || sp.Walking != "/home" || sp.Visited != 1234567 ||
sp.Found != 2 || sp.Kept.IsZero() || len(sp.Finished) != 1 || sp.Finished[0].Took != 90*time.Minute {
t.Fatalf("the progress: %+v", sp)
}
words := sp.Words()
for _, want := range []string{"1 of 3 filesystems walked", "walking /home", "1234567 entries visited",
"2 setuid- or setgid-root files found so far", "kept " + sp.Kept.Local().Format("2006-01-02 15:04:05"),
"walked / in 1h30m0s"} {
if !strings.Contains(words, want) {
t.Errorf("the words lack %q: %s", want, words)
}
}
for _, leak := range []string{"someone", "private", "odd", "rootshell"} {
if strings.Contains(words, leak) || strings.Contains(err.Error(), leak) {
t.Errorf("a path leaks (%q): %s / %v", leak, words, err)
}
}
close(release)
c.Wait()
if _, _, err := c.Look(at.Add(2*time.Minute), root, search); err != nil {
t.Fatalf("the complete search judges: %v", err)
}
}
// The progress moves every minute; the verdict does not change for it, so the journal says it once.
type waiting struct{ progress *SearchProgress }
func (w *waiting) InDatabase(context.Context, string) ([]string, error) { return nil, nil }
func (w *waiting) Session(context.Context, string, []string) (Session, error) {
return Session{}, nil
}
func (w *waiting) Escalation(context.Context, string, []string) ([]string, error) {
return nil, &SetuidNotJudged{Pending: true, Since: at, Progress: w.progress}
}
func TestTheProgressMovesWithoutChangingTheVerdict(t *testing.T) {
r := &waiting{progress: &SearchProgress{Filesystems: 2, Visited: 10}}
j := New(r)
j.Set([]Account{{Module: "claude-code", ID: "claude-code.agent", Name: "agent", Root: true}})
st, changed := j.Look(t.Context())
if !changed || st.Accounts[0].State != Unknown || st.Accounts[0].Search.Visited != 10 {
t.Fatalf("first look: changed %v, %+v", changed, st.Accounts[0])
}
r.progress = &SearchProgress{Filesystems: 2, Visited: 20}
st, changed = j.Look(t.Context())
if changed || st.Accounts[0].Search.Visited != 20 || strings.Contains(st.Accounts[0].Reason, "progress") {
t.Fatalf("second look: changed %v, %+v", changed, st.Accounts[0])
}
}
// Asked at the terminal, a search starts afresh at the next look: the last complete one is thrown away, kept or
// in memory, and a running one is ended and neither judges nor is carried on.
func TestSearchNowThrowsAwayTheLastSearchAndStartsAFullOne(t *testing.T) {
dir := t.TempDir()
c := &SetuidCache{Path: filepath.Join(dir, SetuidSearchFile), ProgressPath: filepath.Join(dir, SetuidProgressFile)}
g := newGate()
g.found = []string{"/home/someone/rootshell"}
c.Look(at, root, g.search)
<-g.started
g.finish(c)
if found, _, err := c.Look(at.Add(time.Hour), root, g.search); err != nil || len(found) != 1 {
t.Fatalf("the complete search judges: %v %v", found, err)
}
said := c.SearchNow("jo at the controller's terminal")
if !strings.Contains(said, "thrown away") || strings.Contains(said, "someone") {
t.Fatalf("said: %s", said)
}
if _, err := os.Stat(c.Path); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("the kept search is still there: %v", err)
}
// The next look starts a full search at once, though the last ended only an hour ago.
ended := make(chan error, 1)
running := func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error) {
if len(p.Done) != 0 || len(p.Cursor) != 0 {
t.Errorf("not a full search: %+v", p)
}
checkpoint()
<-ctx.Done()
ended <- ctx.Err()
return nil, ctx.Err()
}
_, _, err := c.Look(at.Add(time.Hour+time.Minute), root, running)
notJudged(t, err, true)
if starts(c) != 2 {
t.Fatalf("searches started: %d", starts(c))
}
// Asked again while it runs: it is ended, judges nothing, fails nothing, and is not carried on.
c.SearchNow("jo")
if err := <-ended; !errors.Is(err, context.Canceled) {
t.Fatalf("the running search was not ended: %v", err)
}
c.Wait()
if _, err := os.Stat(c.ProgressPath); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("the ended walk is still kept: %v", err)
}
g2 := newGate()
_, _, err = c.Look(at.Add(time.Hour+2*time.Minute), root, g2.search)
notJudged(t, err, true)
<-g2.started
g2.finish(c)
if starts(c) != 3 {
t.Fatalf("searches started: %d", starts(c))
}
if _, _, err := c.Look(at.Add(time.Hour+3*time.Minute), root, g2.search); err != nil {
t.Fatalf("the new search judges: %v", err)
}
}
func TestAFilesystemPartWalkedByAnEngineThatDidNotCountTookATimeNotKnown(t *testing.T) {
top := tree(t, "a", "b")
p := &SetuidProgress{Mounts: []string{top}, Cursor: Cursors{top: filepath.Join(top, "a")}}
if err := mine().walk(t.Context(), p, func() {}); err != nil {
t.Fatal(err)
}
if len(p.Took) != 1 || p.Took[0] >= 0 {
t.Fatalf("took %v", p.Took)
}
}
+115
View File
@@ -0,0 +1,115 @@
package accounts
import (
"io/fs"
"os"
"path/filepath"
"strings"
"testing"
)
// A setuid program no package owns counts against an account only where it can execute it, read at every
// judgement from the modes on the way (novox/hq issue 361): the operator's copies in a home of mode 0700 are no
// way for the agent account, and become one the moment the home is opened, without a new search.
const homeFind = "/home/op/usr/bin/mount"
func homeMachine(home FileMode) *agentMachine {
m := clean()
m.setuid = homeFind + "\n"
m.nodes = map[string]FileMode{"/home/op": home}
return m
}
func TestReachabilityDecidesWhetherAFindCounts(t *testing.T) {
op := func(perm fs.FileMode) FileMode { return FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | perm} }
for _, c := range []struct {
name string
set func(*agentMachine)
counted bool
}{
{"a home of 0700", func(m *agentMachine) {}, false},
{"a home of 0755", func(m *agentMachine) { m.nodes["/home/op"] = op(0o755) }, true},
{"a home its group may search, the agent in it", func(m *agentMachine) {
m.nodes["/home/op"], m.gids = op(0o710), "1600 1500"
}, true},
{"a home its group may search, the agent not in it", func(m *agentMachine) { m.nodes["/home/op"] = op(0o710) }, false},
{"an ACL on the home", func(m *agentMachine) {
m.files["/home/op"] = FileMode{UID: 1500, GID: 1500, Perm: 0o700}
m.acls = map[string][]ACLEntry{"/home/op": {{}}}
}, true},
{"a link on the way", func(m *agentMachine) {
m.nodes["/home/op"] = FileMode{UID: 1500, GID: 1500, Perm: fs.ModeSymlink | 0o777}
}, true},
{"a file the agent may not execute", func(m *agentMachine) {
m.nodes["/home/op"] = op(0o755)
m.nodes[homeFind] = FileMode{Perm: fs.ModeSetuid | 0o4750}
}, false},
} {
t.Run(c.name, func(t *testing.T) {
m := homeMachine(FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | 0o700})
c.set(m)
v := lookAgent(t, m)
if c.counted != (v.State == Unhealthy) {
t.Fatalf("counted %v: %+v", c.counted, v)
}
if strings.Contains(v.Reason, "/home/op") {
t.Fatalf("a path in a home was said: %s", v.Reason)
}
if c.counted && !strings.Contains(v.Reason, "a setuid-root program no package owns in an account's home, which agent can execute") {
t.Fatalf("the reason: %s", v.Reason)
}
})
}
}
// The search keeps every find; whether one counts is read again at every look.
func TestAHomeOpenedAfterTheSearchCountsWithoutANewSearch(t *testing.T) {
m := homeMachine(FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | 0o700})
c := &SetuidCache{}
j := New(Exec{Run: m.run, Walk: m.walk, Stat: m.stat, ReadFile: m.readFile, ReadDir: m.readDir, ACL: m.acl,
Lstat: m.lstat, Cache: c})
j.Set([]Account{agent})
j.Look(t.Context())
c.Wait()
if st, _ := j.Look(t.Context()); st.Accounts[0].State != Healthy {
t.Fatalf("a find in a closed home: %+v", st.Accounts[0])
}
m.nodes["/home/op"] = FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | 0o755}
st, _ := j.Look(t.Context())
if st.Accounts[0].State != Unhealthy || starts(c) != 1 {
t.Fatalf("the home opened: %+v, %d searches", st.Accounts[0], starts(c))
}
m.nodes["/home/op"] = FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | 0o700}
if st, _ := j.Look(t.Context()); st.Accounts[0].State != Healthy || starts(c) != 1 {
t.Fatalf("the home closed again: %+v", st.Accounts[0])
}
}
// On the machine's own modes, as their owner (the directories above a test's are its own): a directory without
// its owner's search bit hides what is in it.
func TestExecutableReadsTheMachinesModes(t *testing.T) {
if os.Getuid() == 0 {
t.Skip("root searches every directory")
}
top := tree(t, "home/bin/x!")
home := filepath.Join(top, "home")
me := os.Getuid()
noACL := func(string) ([]ACLEntry, error) { return nil, nil }
if err := os.Chmod(home, 0o600); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = os.Chmod(home, 0o700) })
if Executable(filepath.Join(home, "bin/x"), me, map[int]bool{}, lstatOf, noACL) {
t.Fatal("a file in a directory its account may not search was executable")
}
if err := os.Chmod(home, 0o700); err != nil {
t.Fatal(err)
}
if !Executable(filepath.Join(home, "bin/x"), me, map[int]bool{}, lstatOf, noACL) {
t.Fatal("a file every directory opens was not executable")
}
if Executable(filepath.Join(home, "bin/gone"), me, map[int]bool{}, lstatOf, noACL) {
t.Fatal("a file gone was executable")
}
}
+313
View File
@@ -0,0 +1,313 @@
package accounts
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
"syscall"
"time"
)
// The walk for setuid programs, resumable (novox/hq issue 361).
//
// The walk is the engine's own, in Go rather than find, so it can say how far it got: filepath.WalkDir visits
// every filesystem's tree depth first, each directory's entries in lexical order, so the walk's progress is one
// cursor per filesystem — the last path it visited — and everything before the cursor in that order is done.
// The progress is kept every CheckpointEvery beside the engine's state (SetuidCache), so an engine restarted,
// updated or rebooted midway carries on from its cursor rather than starting over: on the control-node the
// whole walk takes far longer than an engine runs between updates.
//
// **Why a resumed walk is sound.** Only root can make a setuid- or setgid-root file: the agent account can
// neither set the bit on root's file nor give its own file to root. A file that appears behind the cursor while
// the walk runs, or while it waits to be resumed, was made by root — and root's acts through the mesh start a
// new search (apply.MovesRoot), whose walk starts from nothing. A walk is dated by its start, and one resumed
// more than FreshFor after its start is thrown away and begun again.
// SetuidProgress is one walk, as far as it got.
type SetuidProgress struct {
// ID names the walk; Started is when it began, on the cache's clock: the search is dated by it.
ID string `json:"id"`
Started time.Time `json:"started"`
// Mounts are the filesystems it walks, root first; Done those walked to their end.
Mounts Paths `json:"mounts"`
Done Paths `json:"done"`
// Cursor is, for a filesystem being walked, the last path visited.
Cursor Cursors `json:"cursor"`
// Found is every setuid- or setgid-root regular file found so far, before the package manager is asked.
Found Paths `json:"found"`
// Visited is how many entries the walk has gone past: each once, across a resume, as it is kept with the
// cursor it matches.
Visited int64 `json:"visited"`
// Took is how long the walk spent in each of Done, in its order (negative: not known, a walk kept by an
// engine before it counted); Walking how long it has spent in the filesystem it walks now. Time the
// engine did not run is in neither.
Took []time.Duration `json:"took"`
Walking time.Duration `json:"walking"`
// Kept is when this progress was kept, as the cache wrote it; zero before the first time.
Kept time.Time `json:"kept"`
// Counted is a walk counted from its start: false for one kept by an engine before it counted, whose
// Visited is then a lower bound and whose filesystem under way took a time not known (the review of
// 2026-10-10).
Counted bool `json:"counted"`
}
// Paths are paths kept as their bytes. **Never as JSON strings**: encoding/json replaces bytes that are not
// UTF-8 with U+FFFD, and a cursor so changed sorts after the names it stood before — a resumed walk would skip
// them unread (the review of 2026-10-10). A name of any bytes is any account's to make.
type Paths []string
func (ps Paths) MarshalJSON() ([]byte, error) {
raw := make([][]byte, len(ps))
for i, p := range ps {
raw[i] = []byte(p)
}
return json.Marshal(raw)
}
func (ps *Paths) UnmarshalJSON(b []byte) error {
var raw [][]byte
if err := json.Unmarshal(b, &raw); err != nil {
return err
}
*ps = make(Paths, len(raw))
for i, r := range raw {
(*ps)[i] = string(r)
}
return nil
}
// Cursors are each filesystem's cursor, kept as bytes as Paths are.
type Cursors map[string]string
type keptCursor struct {
Mount []byte `json:"mount"`
Path []byte `json:"path"`
}
func (cs Cursors) MarshalJSON() ([]byte, error) {
raw := []keptCursor{}
for m, p := range cs {
raw = append(raw, keptCursor{Mount: []byte(m), Path: []byte(p)})
}
slices.SortFunc(raw, func(a, b keptCursor) int { return strings.Compare(string(a.Mount), string(b.Mount)) })
return json.Marshal(raw)
}
func (cs *Cursors) UnmarshalJSON(b []byte) error {
var raw []keptCursor
if err := json.Unmarshal(b, &raw); err != nil {
return err
}
*cs = Cursors{}
for _, r := range raw {
(*cs)[string(r.Mount)] = string(r.Path)
}
return nil
}
// CheckpointEvery is how often a walk's progress is kept.
const CheckpointEvery = time.Minute
// prunedPaths are never walked: the kernel's own trees, and the container runtimes' layers.
var prunedPaths = []string{"/proc", "/sys", "/var/lib/docker", "/var/lib/containers"}
// walker walks for setuid programs. rootUID and rootGID are the owner and group a match has: root's (0), but
// for a test. mountPoints are every filesystem mounted on the machine (MountPoints): a directory that is one is
// another filesystem, left without being read — a FUSE mount root cannot read, or a network mount whose
// server is away, would fail or hang the walk (the review of 2026-10-10).
type walker struct {
rootUID, rootGID uint32
every time.Duration
mountPoints map[string]bool
// pace is asked before each directory is read (setuid_pace.go); nil never pauses.
pace *pacer
}
// walk carries p on to its end, calling checkpoint every w.every and after each filesystem. It ends early only
// with an error — a directory it could not read, or ctx ended — which is an incomplete walk, never "none".
// A file or directory that vanished while it walked is not an error: what still exists was walked.
func (w walker) walk(ctx context.Context, p *SetuidProgress, checkpoint func()) error {
if p.Cursor == nil {
p.Cursor = Cursors{}
}
// One Took for each of Done: a walk kept before they were counted has them not known.
for len(p.Took) < len(p.Done) {
p.Took = append(p.Took, -1)
}
p.Took = p.Took[:len(p.Done)]
if !p.Counted && len(p.Cursor) > 0 {
p.Walking = -1 // walked partly by an engine that did not count its time
}
last := time.Now()
for _, m := range p.Mounts {
if slices.Contains(p.Done, m) {
continue
}
if err := w.walkMount(ctx, p, m, checkpoint, &last); err != nil {
return err
}
p.Done = append(p.Done, m)
p.Took = append(p.Took, p.Walking)
p.Walking = 0
delete(p.Cursor, m)
checkpoint()
last = time.Now()
}
// Looked at again: a file found that is gone, or no longer setuid-root, is no answer the package manager
// can give, and would fail every search that resumes this walk (the review of 2026-10-10).
var still Paths
for _, f := range p.Found {
info, err := os.Lstat(f)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return fmt.Errorf("the walk for setuid programs could not read %s again: %w", f, err)
}
if w.setuidRoot(info) {
still = append(still, f)
}
}
slices.Sort(still)
p.Found = slices.Compact(still)
return nil
}
func (w walker) walkMount(ctx context.Context, p *SetuidProgress, mount string, checkpoint func(), last *time.Time) error {
top, err := os.Lstat(mount)
if err != nil {
return fmt.Errorf("the filesystem at %s could not be read: %w", mount, err)
}
dev, ok := deviceOf(top)
if !ok || !top.IsDir() {
return fmt.Errorf("the filesystem at %s is not a directory whose device could be read", mount)
}
var cursor []string
if c := p.Cursor[mount]; c != "" {
cursor = components(c)
}
// How long it walks this filesystem, carried on from what a resumed walk had spent in it.
began, before := time.Now(), p.Walking
spent := func() time.Duration {
if before < 0 {
return -1
}
return before + time.Since(began)
}
defer func() { p.Walking = spent() }()
return filepath.WalkDir(mount, func(path string, d fs.DirEntry, err error) error {
if ctxErr := ctx.Err(); ctxErr != nil {
return ctxErr
}
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return nil // vanished while it walked
}
return fmt.Errorf("the walk for setuid programs could not read %s: %w", path, err)
}
at := components(path)
if cursor != nil && walkOrder(at, cursor) < 0 && !isPrefix(at, cursor) {
// Before the cursor and not on its way: visited by the walk this resumes.
if d.IsDir() {
return filepath.SkipDir
}
return nil
}
switch {
case d.IsDir():
if path != mount && (slices.Contains(prunedPaths, path) || w.mountPoints[path]) {
return filepath.SkipDir
}
info, err := os.Lstat(path)
if errors.Is(err, fs.ErrNotExist) {
return filepath.SkipDir
}
if err != nil {
return fmt.Errorf("the walk for setuid programs could not read %s: %w", path, err)
}
if d, ok := deviceOf(info); !ok || d != dev {
return filepath.SkipDir // another filesystem, walked as its own or not at all (find's -xdev)
}
// Before its entries are read: a busy disk is waited for here (novox/hq issue 433).
if err := w.pace.yield(ctx); err != nil {
return err
}
case d.Type().IsRegular():
// A file's own details may be read from the disk too: a directory of many files is paced as it is read.
if err := w.pace.yield(ctx); err != nil {
return err
}
info, err := d.Info() // the entry's own, never through a link
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
return fmt.Errorf("the walk for setuid programs could not read %s: %w", path, err)
}
if w.setuidRoot(info) {
p.Found = append(p.Found, path)
}
}
// The cursor only moves on: the directories on the way back to it are visited again, and are before it.
// Counted here alone, so an entry visited again on the way back to the cursor is counted once.
if cursor == nil || walkOrder(at, cursor) > 0 {
p.Cursor[mount], cursor = path, at
p.Visited++
}
if time.Since(*last) >= w.every {
p.Walking = spent()
checkpoint()
*last = time.Now()
}
return nil
})
}
// setuidRoot is a regular file setuid and owned by root, or setgid to root's group.
func (w walker) setuidRoot(info fs.FileInfo) bool {
st, ok := info.Sys().(*syscall.Stat_t)
if !ok || !info.Mode().IsRegular() {
return false
}
return uint32(st.Uid) == w.rootUID && info.Mode()&fs.ModeSetuid != 0 ||
uint32(st.Gid) == w.rootGID && info.Mode()&fs.ModeSetgid != 0
}
func deviceOf(info fs.FileInfo) (uint64, bool) {
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return 0, false
}
return uint64(st.Dev), true
}
// components are a path's names, in order; "/" has none.
func components(path string) []string {
path = strings.Trim(filepath.Clean(path), "/")
if path == "" {
return nil
}
return strings.Split(path, "/")
}
// walkOrder compares two paths in the order WalkDir visits them: name by name, each in lexical order, a
// directory before what is in it.
func walkOrder(a, b []string) int {
for i := 0; i < len(a) && i < len(b); i++ {
if c := strings.Compare(a[i], b[i]); c != 0 {
return c
}
}
return len(a) - len(b)
}
// isPrefix says a is b or a directory b is in.
func isPrefix(a, b []string) bool {
return len(a) <= len(b) && slices.Equal(a, b[:len(a)])
}
+213
View File
@@ -0,0 +1,213 @@
package accounts
import (
"context"
"errors"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
)
// The engine's own walk for setuid programs (novox/hq issue 361): what it finds, what it never follows, and that
// a walk interrupted carries on from its cursor.
// tree makes files under a directory of its own: a name ending in "/" a directory, "!" setuid, "%" setgid.
func tree(t *testing.T, names ...string) string {
t.Helper()
top := t.TempDir()
for _, n := range names {
mode := os.FileMode(0o755)
switch {
case strings.HasSuffix(n, "/"):
if err := os.MkdirAll(filepath.Join(top, n), 0o755); err != nil {
t.Fatal(err)
}
continue
case strings.HasSuffix(n, "!"):
n, mode = strings.TrimSuffix(n, "!"), mode|os.ModeSetuid
case strings.HasSuffix(n, "%"):
n, mode = strings.TrimSuffix(n, "%"), mode|os.ModeSetgid
}
p := filepath.Join(top, n)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chmod(p, mode); err != nil {
t.Fatal(err)
}
}
return top
}
// mine is a walker for which this test's own account is root.
func mine() walker {
return walker{rootUID: uint32(os.Getuid()), rootGID: uint32(os.Getgid()), every: time.Hour}
}
func walkAll(t *testing.T, w walker, top string) []string {
t.Helper()
p := &SetuidProgress{Mounts: []string{top}}
if err := w.walk(t.Context(), p, func() {}); err != nil {
t.Fatal(err)
}
return rel(top, p.Found)
}
func rel(top string, paths []string) []string {
var out []string
for _, p := range paths {
out = append(out, strings.TrimPrefix(p, top+"/"))
}
return out
}
func TestTheWalkFindsSetuidAndSetgidRootFilesAndFollowsNoLink(t *testing.T) {
top := tree(t, "bin/plain", "bin/rootshell!", "sbin/group%", "deep/a/b/c/x!")
if err := os.Symlink(filepath.Join(top, "bin/rootshell"), filepath.Join(top, "link")); err != nil {
t.Fatal(err)
}
if err := os.Symlink(filepath.Join(top, "bin"), filepath.Join(top, "linkdir")); err != nil {
t.Fatal(err)
}
found := walkAll(t, mine(), top)
if strings.Join(found, ",") != "bin/rootshell,deep/a/b/c/x,sbin/group" {
t.Fatalf("found %v", found)
}
if got := walkAll(t, walker{rootUID: 4242424, rootGID: 4242424, every: time.Hour}, top); len(got) != 0 {
t.Fatalf("files nobody's root owns were found: %v", got)
}
}
func TestAWalkThatCannotReadADirectoryIsIncomplete(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads every directory")
}
top := tree(t, "open/x!", "closed/y!")
if err := os.Chmod(filepath.Join(top, "closed"), 0); err != nil {
t.Fatal(err)
}
defer os.Chmod(filepath.Join(top, "closed"), 0o755)
p := &SetuidProgress{Mounts: []string{top}}
if err := mine().walk(t.Context(), p, func() {}); err == nil {
t.Fatal("a walk that could not read a directory was complete")
}
}
func TestWalkOrderIsNameByName(t *testing.T) {
// "/a/b" is walked before "/a-c" although '-' sorts before '/': the walk goes name by name.
if walkOrder(components("/a/b"), components("/a-c")) >= 0 {
t.Fatal("/a/b is walked before /a-c")
}
if walkOrder(components("/a"), components("/a/b")) >= 0 || !isPrefix(components("/a"), components("/a/b")) {
t.Fatal("a directory is walked before what is in it")
}
if len(components("/")) != 0 {
t.Fatal("/ has no names")
}
}
// A walk stopped midway carries on from its cursor: what was before the cursor is not walked again, what is after
// it is, and every file of the first walk is found.
func TestAnInterruptedWalkCarriesOnFromItsCursor(t *testing.T) {
top := tree(t, "a/1!", "a/2", "a-b/3!", "b/c/4!", "b/c/5", "b/d!", "c/6!", "d/7!")
w := mine()
w.every = 0 // a checkpoint at every path
ctx, stop := context.WithCancel(t.Context())
p := &SetuidProgress{Mounts: []string{top}}
var kept []byte
visits := 0
err := w.walk(ctx, p, func() {
visits++
kept = mustJSON(t, p)
if strings.HasSuffix(p.Cursor[top], "/b/c/4") {
stop() // the engine stops here
}
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("the walk was stopped: %v", err)
}
var resumed SetuidProgress
fromJSON(t, kept, &resumed)
if !strings.HasSuffix(resumed.Cursor[top], "/b/c/4") {
t.Fatalf("the kept cursor is %q", resumed.Cursor[top])
}
// Meanwhile, a file appears before the cursor and one after it.
for _, n := range []string{"a/late!", "e/late!"} {
tree2(t, top, n)
}
w.every = time.Hour
if err := w.walk(t.Context(), &resumed, func() {}); err != nil {
t.Fatal(err)
}
got := rel(top, resumed.Found)
want := []string{"a-b/3", "a/1", "b/c/4", "b/d", "c/6", "d/7", "e/late"}
if !slices.Equal(got, want) {
t.Fatalf("the resumed walk found %v, want %v: the first walk's finds kept, nothing before the cursor "+
"walked again, everything after it walked", got, want)
}
}
func tree2(t *testing.T, top, n string) {
t.Helper()
p := filepath.Join(top, strings.TrimSuffix(n, "!"))
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chmod(p, 0o755|os.ModeSetuid); err != nil {
t.Fatal(err)
}
}
// A search's commands and walk run on a lowered thread while the rest of the engine keeps its priority.
func TestTheSearchRunsOnALoweredThreadOfItsOwn(t *testing.T) {
var wg sync.WaitGroup
wg.Add(1)
var err error
go func() {
defer wg.Done()
err = LowerThisThread()
}()
wg.Wait()
if err != nil {
t.Fatalf("the thread could not be lowered: %v", err)
}
}
// Another filesystem's mount point is left without being read, and a file found that is gone, or no longer
// setuid-root, when the walk ends is not asked about.
func TestTheWalkLeavesMountPointsAndDropsWhatWentMeanwhile(t *testing.T) {
top := tree(t, "fuse/secret!", "bin/x!", "bin/y!")
w := mine()
w.mountPoints = map[string]bool{filepath.Join(top, "fuse"): true}
w.every = 0
p := &SetuidProgress{Mounts: []string{top}}
removed := false
if err := w.walk(t.Context(), p, func() {
if !removed && strings.HasSuffix(p.Cursor[top], "/bin/y") {
removed = true
os.Remove(filepath.Join(top, "bin/x"))
}
}); err != nil {
t.Fatal(err)
}
if got := rel(top, p.Found); strings.Join(got, ",") != "bin/y" {
t.Fatalf("found %v", got)
}
}
func TestAMountPointIsReadAsTheKernelEscapesIt(t *testing.T) {
mp := MountPoints("/dev/a /mnt/a\\040b ext4 rw 0 0\n/dev/b /mnt/back\\134slash ext4 rw 0 0\n")
if !mp["/mnt/a b"] || !mp[`/mnt/back\slash`] {
t.Fatalf("read %v", mp)
}
}
+810
View File
@@ -0,0 +1,810 @@
package accounts
// The ways to root the judge looks for beyond uid, groups and sudo (novox/hq ADR 0266, the review of
// 2026-10-08), and the ones it does not. Judged and NotJudged are the one written list: the record quotes it,
// and a way added here is a line added there.
import (
"bytes"
"context"
"encoding/binary"
"errors"
"fmt"
"io/fs"
"os"
"os/exec"
"path/filepath"
"slices"
"sort"
"strings"
"time"
"golang.org/x/sys/unix"
)
// Judged is every way to root the judge looks for, in the words its reasons use.
var Judged = []string{
"its uid is 0",
"membership of a group that grants root by membership: " + strings.Join(RootGroups, ", "),
"any sudo rule for it or one of its groups, as `sudo -l -U` lists it (with or without a password)",
"any doas rule permitting it or one of its groups (/etc/doas.conf, /etc/opendoas.conf)",
"any polkit rule naming it or one of its groups (/etc/polkit-1/rules.d, /usr/share/polkit-1/rules.d, " +
"/etc/polkit-1/localauthority), which is how pkexec and systemd's own actions are granted",
"a polkit rule that grants every account: a rule of a .rules file that answers polkit.Result.YES and names " +
"no user and no group, or a .pkla section whose Identity is unix-user:* or unix-group:* with a Result of " +
"yes — each rule and section judged on its own, comments taken out",
"write access to a container runtime's socket (docker, podman, containerd), by owner, group, other or " +
"POSIX ACL",
"a container runtime's API listening on TCP port 2375 or 2376 (/proc/net/tcp, /proc/net/tcp6), which any " +
"local account reaches",
"read access to a secret the mesh placed for another account, by owner, group, other or POSIX ACL",
"a program that no installed package owns and is setuid with owner root, or setgid with group root, " +
"whoever owns it, on every local filesystem mounted without nosuid (container and image layers, " +
"network and pseudo filesystems excepted) — searched in the background at most hourly, its last result " +
"served with when it was found; until a search has finished, the verdict says not judged",
"hard links or symbolic links to other accounts' files left unprotected by the kernel " +
"(fs.protected_hardlinks or fs.protected_symlinks is 0)",
}
// NotJudged is what the judge does not look for: each is a way to root it would miss.
var NotJudged = []string{
"a root-run unit, timer, cron entry or script the account can write",
"a directory on root's PATH, or in /etc/profile.d, the account can write",
"root's or the operator's ssh keys or authorized_keys readable or writable by it",
"a file root writes or reads, in a directory the account owns, other than below its home (the node-engine " +
"refuses links there)",
"file capabilities (setcap) on a program",
"a setuid program a package installed that has a flaw of its own",
"a terminal the account shares with a root process (TIOCSTI, sudo without use_pty)",
"a polkit rule that grants every account by logic the text does not show (a JavaScript condition " +
"other than a named user or group), or every account with an active local session",
"a container runtime's API on a TCP port other than 2375 and 2376, or on a unix socket not named here",
"a setuid program on a filesystem not judged: a container or image layer (overlay, squashfs), a network " +
"or FUSE filesystem",
}
// RuntimeSockets are the container runtimes' sockets: write access to one runs a container as root.
var RuntimeSockets = []string{"/run/docker.sock", "/var/run/docker.sock", "/run/podman/podman.sock",
"/run/containerd/containerd.sock"}
// DoasConfigs and PolkitDirs are where those grants live.
var (
DoasConfigs = []string{"/etc/doas.conf", "/etc/opendoas.conf"}
PolkitDirs = []string{"/etc/polkit-1/rules.d", "/usr/share/polkit-1/rules.d", "/etc/polkit-1/localauthority"}
)
// ACLEntry is one named entry of a POSIX ACL: a user's or a group's, and whether it grants read and write
// once the ACL's mask is applied.
type ACLEntry struct {
User bool
ID int
Read, Write bool
}
// The tags and bits of the kernel's ACL xattr.
const (
aclUser = 0x02
aclGroup = 0x08
aclMask = 0x10
)
// ParseACL reads a system.posix_acl_access value: a version, then entries of tag, permission and id; the
// named users' and groups' entries are answered with the mask applied.
func ParseACL(raw []byte) ([]ACLEntry, error) {
if len(raw) < 4 || (len(raw)-4)%8 != 0 {
return nil, fmt.Errorf("an ACL of %d bytes is not one", len(raw))
}
type entry struct {
tag, perm uint16
id uint32
}
var es []entry
mask := uint16(7)
for at := 4; at < len(raw); at += 8 {
e := entry{binary.LittleEndian.Uint16(raw[at:]), binary.LittleEndian.Uint16(raw[at+2:]),
binary.LittleEndian.Uint32(raw[at+4:])}
if e.tag == aclMask {
mask = e.perm
}
es = append(es, e)
}
var out []ACLEntry
for _, e := range es {
if e.tag != aclUser && e.tag != aclGroup {
continue
}
p := e.perm & mask
out = append(out, ACLEntry{User: e.tag == aclUser, ID: int(e.id), Read: p&4 != 0, Write: p&2 != 0})
}
return out, nil
}
// aclOf is a file's ACL from the machine: none when it has none.
func aclOf(path string) ([]ACLEntry, error) {
buf := make([]byte, 1024)
n, err := unix.Getxattr(path, "system.posix_acl_access", buf)
if errors.Is(err, unix.ENODATA) || errors.Is(err, unix.EOPNOTSUPP) {
return nil, nil
}
if err != nil {
return nil, &os.PathError{Op: "getxattr", Path: path, Err: err}
}
return ParseACL(buf[:n])
}
// grantsByACL says whether an ACL entry gives the account read (or write).
func grantsByACL(acl []ACLEntry, uid int, gids map[int]bool, write bool) bool {
for _, e := range acl {
if (e.User && e.ID == uid) || (!e.User && gids[e.ID]) {
if (write && e.Write) || (!write && e.Read) {
return true
}
}
}
return false
}
// Writable is Readable's twin for the write bits.
func Writable(m FileMode, uid int, gids map[int]bool) bool {
switch {
case uid == 0:
return true
case m.UID == uid:
return m.Perm&0o200 != 0
case gids[m.GID]:
return m.Perm&0o020 != 0
default:
return m.Perm&0o002 != 0
}
}
// DoasRules is every line of a doas configuration that permits the account or one of its groups.
func DoasRules(conf string, account string, groups []string) []string {
var out []string
for _, line := range strings.Split(conf, "\n") {
if i := strings.IndexByte(line, '#'); i >= 0 {
line = line[:i]
}
f := strings.Fields(line)
if len(f) < 2 || f[0] != "permit" {
continue
}
i := 1
for i < len(f) && (f[i] == "nopass" || f[i] == "persist" || f[i] == "keepenv" || f[i] == "nolog" ||
f[i] == "setenv" || strings.HasPrefix(f[i], "{")) {
if strings.HasPrefix(f[i], "{") {
for i < len(f) && !strings.HasSuffix(f[i], "}") {
i++
}
}
i++
}
if i >= len(f) {
continue
}
who := f[i]
if who == account || (strings.HasPrefix(who, ":") && contains(groups, who[1:])) {
out = append(out, strings.TrimSpace(line))
}
}
return out
}
// PolkitNames says whether a polkit rule or authority file names the account or one of its groups.
func PolkitNames(text, account string, groups []string) bool {
needles := []string{"unix-user:" + account, `"` + account + `"`, `'` + account + `'`}
for _, g := range groups {
needles = append(needles, "unix-group:"+g, `isInGroup("`+g+`")`, `isInGroup('`+g+`')`)
}
for _, n := range needles {
if strings.Contains(text, n) {
return true
}
}
return false
}
// PolkitGrantsEveryone says whether a polkit rule or authority file grants every account, naming none: a .rules
// file one of whose rules answers polkit.Result.YES with no condition on a user or a group, or a .pkla one of
// whose sections has an Identity of every user or every group with a Result of yes.
//
// **Each rule is judged on its own, comments taken out first** (the third review of 2026-10-08): a rule naming a
// user elsewhere in the file, or a user named only in a comment, must not hide another rule's unconditional
// yes. A rule's text is read, not run: a condition the text does not show is listed under NotJudged.
func PolkitGrantsEveryone(path, text string) bool {
if strings.HasSuffix(path, ".pkla") {
var kept []string
for _, line := range strings.Split(text, "\n") {
if t := strings.TrimSpace(line); strings.HasPrefix(t, "#") || strings.HasPrefix(t, ";") {
continue
}
kept = append(kept, line)
}
for _, section := range strings.Split(strings.Join(kept, "\n"), "[") {
var every, yes bool
for _, line := range strings.Split(section, "\n") {
k, v, ok := strings.Cut(strings.TrimSpace(line), "=")
if !ok {
continue
}
v = strings.TrimSpace(v)
switch strings.TrimSpace(k) {
case "Identity":
for _, id := range strings.Split(v, ";") {
if id = strings.TrimSpace(id); id == "unix-user:*" || id == "unix-group:*" {
every = true
}
}
case "ResultAny", "ResultActive", "ResultInactive":
if v == "yes" {
yes = true
}
}
}
if every && yes {
return true
}
}
return false
}
for _, rule := range polkitRules(stripJSComments(text)) {
if !strings.Contains(rule, "polkit.Result.YES") {
continue
}
// A condition on a user or a group names whom it grants; one on an active local session grants only an
// account with a seat, which an agent started through sudo has not (listed under NotJudged).
conditioned := false
for _, condition := range []string{".user", "isInGroup", "unix-user:", "unix-group:", ".active", ".local"} {
if strings.Contains(rule, condition) {
conditioned = true
break
}
}
if !conditioned {
return true
}
}
return false
}
// polkitRules is each rule of a .rules file: the text from one polkit.addRule to the next. Text before the first
// is no rule.
func polkitRules(text string) []string {
parts := strings.Split(text, "addRule(")
if len(parts) < 2 {
return nil
}
return parts[1:]
}
// stripJSComments takes /* … */ and // … comments out of a rule file, leaving what is in strings alone.
func stripJSComments(text string) string {
var b strings.Builder
var quote byte
for i := 0; i < len(text); i++ {
c := text[i]
switch {
case quote != 0:
b.WriteByte(c)
if c == '\\' && i+1 < len(text) {
i++
b.WriteByte(text[i])
} else if c == quote {
quote = 0
}
case c == '"' || c == '\'' || c == '`':
quote = c
b.WriteByte(c)
case c == '/' && i+1 < len(text) && text[i+1] == '*':
end := strings.Index(text[i+2:], "*/")
if end < 0 {
return b.String()
}
i += end + 3
case c == '/' && i+1 < len(text) && text[i+1] == '/':
end := strings.IndexByte(text[i:], '\n')
if end < 0 {
return b.String()
}
i += end - 1
default:
b.WriteByte(c)
}
}
return b.String()
}
// RuntimeAPIPorts are the ports a container runtime's API listens on by convention: 2375 plain, 2376 TLS.
var RuntimeAPIPorts = []int{2375, 2376}
// ListeningPorts is every local TCP port in the listening state, from /proc/net/tcp's or tcp6's text.
func ListeningPorts(text string) []int {
var out []int
for i, line := range strings.Split(text, "\n") {
f := strings.Fields(line)
if i == 0 || len(f) < 4 || f[3] != "0A" {
continue
}
_, port, ok := strings.Cut(f[1], ":")
if !ok {
continue
}
var n int
if _, err := fmt.Sscanf(port, "%X", &n); err == nil {
out = append(out, n)
}
}
return out
}
// pseudoFS are filesystem types no setuid program is installed on, or that are a container's or an image's
// own layers: not searched.
var pseudoFS = map[string]bool{"proc": true, "sysfs": true, "devtmpfs": true, "devpts": true, "tmpfs": true,
"cgroup": true, "cgroup2": true, "securityfs": true, "pstore": true, "bpf": true, "debugfs": true,
"tracefs": true, "mqueue": true, "hugetlbfs": true, "configfs": true, "fusectl": true, "autofs": true,
"binfmt_misc": true, "efivarfs": true, "overlay": true, "squashfs": true, "nsfs": true, "ramfs": true,
"nfs": true, "nfs4": true, "cifs": true, "smb3": true, "9p": true, "virtiofs": true}
// SuidMounts is every local filesystem a setuid program could run from: the mount points of /proc/mounts whose
// type is not pseudo, network or a container's layer, and that are not mounted nosuid. Root first.
func SuidMounts(text string) []string {
seen := map[string]bool{}
var out []string
for _, line := range strings.Split(text, "\n") {
f := strings.Fields(line)
if len(f) < 4 || pseudoFS[f[2]] || strings.HasPrefix(f[2], "fuse") {
continue
}
at := unescapeMount(f[1])
if strings.HasPrefix(at, "/var/lib/docker") || strings.HasPrefix(at, "/var/lib/containers") ||
strings.HasPrefix(at, "/proc") || strings.HasPrefix(at, "/sys") {
continue
}
if contains(strings.Split(f[3], ","), "nosuid") || seen[at] {
continue
}
seen[at] = true
out = append(out, at)
}
sort.Slice(out, func(i, j int) bool { return out[i] == "/" || (out[j] != "/" && out[i] < out[j]) })
return out
}
// MountPoints is every mount point /proc/mounts lists, of any type: the walk leaves each it meets inside
// another filesystem without reading it.
func MountPoints(text string) map[string]bool {
out := map[string]bool{}
for _, line := range strings.Split(text, "\n") {
if f := strings.Fields(line); len(f) >= 2 {
out[unescapeMount(f[1])] = true
}
}
return out
}
// unescapeMount is a mount point as /proc/mounts escapes it — a space, a tab, a newline and a backslash as
// three octal digits after a backslash — made a path again.
func unescapeMount(s string) string {
var b strings.Builder
for i := 0; i < len(s); i++ {
if s[i] == '\\' && i+3 < len(s) && isOctal(s[i+1:i+4]) {
b.WriteByte((s[i+1]-'0')<<6 | (s[i+2]-'0')<<3 | (s[i+3] - '0'))
i += 3
continue
}
b.WriteByte(s[i])
}
return b.String()
}
func isOctal(s string) bool {
if len(s) != 3 {
return false
}
for i := 0; i < 3; i++ {
if s[i] < '0' || s[i] > '7' {
return false
}
}
return true
}
func contains(xs []string, s string) bool {
for _, x := range xs {
if x == s {
return true
}
}
return false
}
// setuidSearch is every setuid- or setgid-root regular file on the filesystems of p that no installed package
// owns: walked by the engine (setuid_walk.go, resumable from p), then asked of the package manager through the
// search's runner (Exec.SearchRun). **No bound of its own** (novox/hq issue 361): it runs until it has walked
// everything. A walk that could not complete — a directory it could not read — is an unanswered question, never
// "none"; a file that vanished while it walked is not.
func (e Exec) setuidSearch(ctx context.Context, p *SetuidProgress, checkpoint func(), mountPoints map[string]bool) ([]string, error) {
walk := e.Walk
if walk == nil {
var say func(string)
if e.Cache != nil {
say = e.Cache.Say
}
walk = engineWalker(mountPoints, say).walk
}
if err := walk(ctx, p, checkpoint); err != nil {
return nil, fmt.Errorf("the walk for setuid programs did not complete: %w", err)
}
if e.SearchRun != nil {
e.Run = e.SearchRun
}
found := slices.Clone(p.Found)
slices.Sort(found)
return e.unpackaged(ctx, slices.Compact(found))
}
// engineWalker is the walk the engine runs: checkpointed every CheckpointEvery, leaving other filesystems unread,
// and paced by the machine's I/O pressure (setuid_pace.go, novox/hq issue 433), saying through say what it cannot do.
func engineWalker(mountPoints map[string]bool, say func(string)) walker {
return walker{every: CheckpointEvery, mountPoints: mountPoints, pace: newPacer(say)}
}
// errorLines are the lines a command said on its error stream, as the runner words a failure: "<name> exited N:
// <its words>", the prefix taken off the first line. Empty lines are dropped.
func errorLines(err error, name string) []string {
words := err.Error()
if strings.HasPrefix(words, name+" exited") {
if i := strings.Index(words, ": "); i >= 0 {
words = words[i+2:]
}
}
var out []string
for _, l := range strings.Split(words, "\n") {
if l = strings.TrimSpace(l); l != "" {
out = append(out, l)
}
}
return out
}
// unpackaged is every path no installed package owns: pacman asked about all of them in one call, else apk
// about each. Never a guess: a path neither said owned nor said unowned is an unanswered question.
func (e Exec) unpackaged(ctx context.Context, paths []string) ([]string, error) {
if len(paths) == 0 {
return nil, nil
}
out, err := e.Run(ctx, "pacman", append([]string{"-Qo"}, paths...)...)
if err == nil || !errors.Is(err, exec.ErrNotFound) && !errors.Is(err, fs.ErrNotExist) {
owned := map[string]bool{}
for _, l := range strings.Split(out, "\n") {
if p, _, ok := strings.Cut(strings.TrimSpace(l), " is owned by "); ok {
owned[p] = true
}
}
unowned := map[string]bool{}
if err != nil {
// pacman exits 1 when a path is owned by nothing, and says which on its error stream.
for _, l := range errorLines(err, "pacman") {
if _, p, ok := strings.Cut(l, "No package owns "); ok {
unowned[strings.TrimSpace(p)] = true
} else {
return nil, fmt.Errorf("pacman could not say who owns the setuid programs found: %s", l)
}
}
}
var out []string
for _, p := range paths {
switch {
case owned[p]:
case unowned[p]:
out = append(out, p)
default:
return nil, fmt.Errorf("pacman said nothing of who owns %s", p)
}
}
return out, nil
}
var none []string
for _, p := range paths {
said, err := e.Run(ctx, "apk", "info", "-W", p)
if err != nil {
return nil, fmt.Errorf("no package manager could say who owns %s: %w", p, err)
}
if !strings.Contains(said, " is owned by ") {
none = append(none, p)
}
}
return none, nil
}
// moreWays is every way beyond uid, groups and sudo; an unanswered question is an error, never none.
func (e Exec) moreWays(ctx context.Context, account string, uid int, groups []string, gids map[int]bool,
secrets []string) ([]string, error) {
read := e.ReadFile
if read == nil {
read = os.ReadFile
}
readDir := e.ReadDir
if readDir == nil {
readDir = os.ReadDir
}
acl := e.ACL
if acl == nil {
acl = aclOf
}
stat := e.Stat
if stat == nil {
stat = statOf
}
var ways []string
// doas.
for _, conf := range DoasConfigs {
raw, err := read(conf)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("doas's rules in %s could not be read: %w", conf, err)
}
for _, r := range DoasRules(string(raw), account, groups) {
ways = append(ways, "doas permits it: "+r)
}
}
// polkit.
for _, dir := range PolkitDirs {
var named, everyone []string
err := walkFiles(readDir, dir, func(path string) error {
raw, err := read(path)
if err != nil {
return err
}
if PolkitNames(string(raw), account, groups) {
named = append(named, path)
} else if PolkitGrantsEveryone(path, string(raw)) {
everyone = append(everyone, path)
}
return nil
})
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return nil, fmt.Errorf("polkit's rules in %s could not be read: %w", dir, err)
}
for _, p := range named {
ways = append(ways, "a polkit rule names it or a group of it: "+p)
}
for _, p := range everyone {
ways = append(ways, "a polkit rule grants every account: "+p)
}
}
// The container runtimes' sockets.
seen := map[string]bool{}
for _, s := range RuntimeSockets {
// Two names of one socket are one socket. A test that gives its own files names them as they are.
real, err := filepath.EvalSymlinks(s)
if e.Stat != nil {
real, err = s, nil
}
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("the socket %s could not be read: %w", s, err)
}
if seen[real] {
continue
}
seen[real] = true
m, err := stat(real)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("the socket %s could not be read for its owner and mode: %w", s, err)
}
a, err := acl(real)
if err != nil {
return nil, err
}
if Writable(m, uid, gids) || grantsByACL(a, uid, gids, true) {
ways = append(ways, "it can write the container runtime's socket "+s+", which runs a container as root")
}
}
// The secrets' ACLs: the bits were judged already.
for _, path := range secrets {
a, err := acl(path)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("the secret %s's ACL could not be read: %w", path, err)
}
if grantsByACL(a, uid, gids, false) {
ways = append(ways, "an ACL lets it read the secret "+path)
}
}
// setuid programs no package owns.
now := time.Now
if e.Now != nil {
now = e.Now
}
mountsText, err := read("/proc/mounts")
if err != nil {
return nil, fmt.Errorf("the mounted filesystems could not be read: %w", err)
}
mounts := SuidMounts(string(mountsText))
if len(mounts) == 0 {
return nil, errors.New("no filesystem a setuid program could run from was found in /proc/mounts")
}
// In the background, the last complete result served while it is fresh (SetuidCache): never a stall. No
// fresh complete result is said beside the other ways, which are judged all the same.
mountPoints := MountPoints(string(mountsText))
unowned, searchedAt, searchErr := e.Cache.Look(now(), mounts,
func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error) {
return e.setuidSearch(ctx, p, checkpoint, mountPoints)
})
var notJudged *SetuidNotJudged
if searchErr != nil && !errors.As(searchErr, &notJudged) {
return nil, searchErr
}
// Counted only where the account can execute it, read now from the paths' modes, never kept with the
// search: an account may open its own home to others later without root, and no search would see it.
lstat := e.Lstat
if lstat == nil {
lstat = lstatOf
}
started := " (search started " + searchedAt.Local().Format("2006-01-02 15:04") + ")"
inHomes := 0
for _, p := range unowned {
if !Executable(p, uid, gids, lstat, acl) {
continue
}
if inAHome(p) {
inHomes++ // its path stays in the engine's journal: a home's names are its owner's
continue
}
ways = append(ways, "a setuid-root program no package owns, which "+account+" can execute: "+p+started)
}
switch {
case inHomes == 1:
ways = append(ways, "a setuid-root program no package owns in an account's home, which "+account+
" can execute"+started)
case inHomes > 1:
ways = append(ways, fmt.Sprintf("%d setuid-root programs no package owns in accounts' homes, which %s can "+
"execute%s", inHomes, account, started))
}
// The kernel's protection of links: off, any account may link another's file where root then acts on it.
for _, sysctl := range []string{"protected_hardlinks", "protected_symlinks"} {
raw, err := read("/proc/sys/fs/" + sysctl)
if err != nil {
return nil, fmt.Errorf("fs.%s could not be read: %w", sysctl, err)
}
if strings.TrimSpace(string(raw)) == "0" {
what := map[string]string{"protected_hardlinks": "hard links", "protected_symlinks": "symbolic links"}[sysctl]
ways = append(ways, fmt.Sprintf("%s to other accounts' files are not protected (fs.%s=0)", what, sysctl))
}
}
// A container runtime's API on TCP, which any account on the machine reaches.
for _, table := range []string{"/proc/net/tcp", "/proc/net/tcp6"} {
raw, err := read(table)
if errors.Is(err, fs.ErrNotExist) && table == "/proc/net/tcp6" {
continue // no IPv6 on this machine
}
if err != nil {
return nil, fmt.Errorf("the listening ports in %s could not be read: %w", table, err)
}
for _, port := range ListeningPorts(string(raw)) {
for _, api := range RuntimeAPIPorts {
if port == api {
ways = append(ways, fmt.Sprintf("a container runtime's API listens on TCP port %d, which any "+
"local account reaches", port))
}
}
}
}
if notJudged != nil {
// The other ways are judged; this one is said as not judged, never as none.
return ways, notJudged
}
return ways, nil
}
// Executable says whether an account of uid, in the groups gids, can execute path as the kernel decides it:
// every directory from the root down searchable for it, and the file executable for it, each by the owner's
// bits for its owner, the group's for a member, the others' for anybody else (novox/hq issue 361). It errs
// towards a way: a link on the way, an ACL on any of them, or a mode that could not be read is executable.
// A path gone is not: there is nothing to execute.
func Executable(path string, uid int, gids map[int]bool, lstat func(string) (FileMode, error),
acl func(string) ([]ACLEntry, error)) bool {
if uid == 0 {
return true
}
at := "/"
steps := []string{at}
for _, name := range components(path) {
at = filepath.Join(at, name)
steps = append(steps, at)
}
for i, p := range steps {
m, err := lstat(p)
if errors.Is(err, fs.ErrNotExist) {
return false
}
if err != nil || m.Perm&fs.ModeSymlink != 0 {
return true
}
if a, err := acl(p); (err != nil && !errors.Is(err, fs.ErrNotExist)) || len(a) > 0 {
return true
}
bit := fs.FileMode(0o001)
switch {
case m.UID == uid:
bit = 0o100
case gids[m.GID]:
bit = 0o010
}
if m.Perm&bit == 0 {
return false
}
if i < len(steps)-1 && !m.Perm.IsDir() {
return false // a file on the way: the path names nothing
}
}
return true
}
// homes are where accounts' homes are: a path under one is said by count alone, its name its owner's.
var homes = []string{"/home/", "/root/", "/var/home/"}
func inAHome(path string) bool {
for _, h := range homes {
if strings.HasPrefix(path, h) {
return true
}
}
return false
}
// walkFiles calls fn for every regular file below dir, through readDir.
func walkFiles(readDir func(string) ([]fs.DirEntry, error), dir string, fn func(string) error) error {
entries, err := readDir(dir)
if err != nil {
return err
}
for _, en := range entries {
p := filepath.Join(dir, en.Name())
if en.IsDir() {
if err := walkFiles(readDir, p, fn); err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
continue
}
if err := fn(p); err != nil {
return err
}
}
return nil
}
// CLocale runs a command in the C locale, so what the judge parses — sudo's listing above all — is one
// language whatever the machine's is; stdin closed, output captured.
func CLocale(ctx context.Context, name string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
cmd.Env = append(os.Environ(), "LC_ALL=C", "LANG=C", "LANGUAGE=C")
var stderr bytes.Buffer
cmd.Stderr = &stderr
out, err := cmd.Output()
if err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
return string(out), fmt.Errorf("%s exited %d: %s", name, exit.ExitCode(), strings.TrimSpace(stderr.String()))
}
return string(out), fmt.Errorf("%s: %w", name, err)
}
return string(out), nil
}
+305
View File
@@ -0,0 +1,305 @@
package accounts
import (
"context"
"encoding/binary"
"errors"
"fmt"
"os/exec"
"strings"
"testing"
"time"
)
// The ways beyond uid, groups and sudo (novox/hq ADR 0266, the review of 2026-10-08): doas, polkit, a
// runtime's socket, an ACL on a secret, a setuid program no package owns — each said; each unread question
// unknown; and the judge's commands in the C locale.
func TestEachFurtherWayToRootIsSaid(t *testing.T) {
const broker = "/var/lib/mesh/node-tools/broker"
for _, c := range []struct {
name string
set func(*agentMachine)
said string
}{
{"doas by name", func(m *agentMachine) {
m.texts = map[string]string{"/etc/doas.conf": "permit persist operator\npermit nopass agent as root\n"}
}, "doas permits it: permit nopass agent as root"},
{"doas by group", func(m *agentMachine) {
m.groups = "agent builders"
m.texts = map[string]string{"/etc/opendoas.conf": "permit :builders # the builders\n"}
}, "doas permits it: permit :builders"},
{"polkit by name", func(m *agentMachine) {
m.dirs = map[string][]string{"/etc/polkit-1/rules.d": {"10-agent.rules"}}
m.texts = map[string]string{"/etc/polkit-1/rules.d/10-agent.rules": `polkit.addRule(function(a, s) { if (s.user == "agent") return polkit.Result.YES; });`}
}, "a polkit rule names it or a group of it: /etc/polkit-1/rules.d/10-agent.rules"},
{"polkit by group", func(m *agentMachine) {
m.groups = "agent network"
m.dirs = map[string][]string{"/usr/share/polkit-1/rules.d": {"50-nm.rules"}}
m.texts = map[string]string{"/usr/share/polkit-1/rules.d/50-nm.rules": `if (subject.isInGroup("network")) return polkit.Result.YES;`}
}, "a polkit rule names it or a group of it: /usr/share/polkit-1/rules.d/50-nm.rules"},
{"docker socket by group bits", func(m *agentMachine) {
m.gids = "1600 970"
m.files["/run/docker.sock"] = FileMode{UID: 0, GID: 970, Perm: 0o660}
}, "it can write the container runtime's socket /run/docker.sock"},
{"docker socket by ACL", func(m *agentMachine) {
m.files["/run/docker.sock"] = FileMode{UID: 0, GID: 970, Perm: 0o660}
m.acls = map[string][]ACLEntry{"/run/docker.sock": {{User: true, ID: 1600, Read: true, Write: true}}}
}, "it can write the container runtime's socket /run/docker.sock"},
{"secret by ACL", func(m *agentMachine) {
m.acls = map[string][]ACLEntry{broker: {{User: false, ID: 1600, Read: true}}}
}, "an ACL lets it read the secret " + broker},
{"polkit for every account", func(m *agentMachine) {
m.dirs = map[string][]string{"/etc/polkit-1/rules.d": {"00-all.rules"}}
m.texts = map[string]string{"/etc/polkit-1/rules.d/00-all.rules": `polkit.addRule(function(action, subject) { return polkit.Result.YES; });`}
}, "a polkit rule grants every account: /etc/polkit-1/rules.d/00-all.rules"},
{"pkla for every account", func(m *agentMachine) {
m.dirs = map[string][]string{"/etc/polkit-1/localauthority": {"all.pkla"}}
m.texts = map[string]string{"/etc/polkit-1/localauthority/all.pkla": "[all]\nIdentity=unix-user:*\nAction=*\nResultAny=yes\n"}
}, "a polkit rule grants every account: /etc/polkit-1/localauthority/all.pkla"},
{"docker on TCP", func(m *agentMachine) {
m.proc["/proc/net/tcp6"] = " sl local_address rem_address st\n 0: 00000000000000000000000000000000:0947 00000000000000000000000000000000:0000 0A 0\n"
}, "a container runtime's API listens on TCP port 2375"},
{"hard links unprotected", func(m *agentMachine) {
m.proc["/proc/sys/fs/protected_hardlinks"] = "0\n"
}, "hard links to other accounts' files are not protected (fs.protected_hardlinks=0)"},
{"setuid on a second filesystem", func(m *agentMachine) {
m.proc["/proc/mounts"] += "/dev/sdb1 /srv xfs rw 0 0\n/dev/sdc1 /data ext4 rw,nosuid 0 0\n"
m.setuid = "/srv/bin/rootshell\n"
m.findArgs = func(args []string) {
if len(args) != 2 || args[0] != "/" || args[1] != "/srv" {
panic(fmt.Sprintf("searched %v", args))
}
}
}, "a setuid-root program no package owns, which agent can execute: /srv/bin/rootshell"},
{"setuid no package owns", func(m *agentMachine) {
m.setuid = "/usr/bin/sudo\n/usr/local/bin/rootshell\n"
m.packaged = map[string]bool{"/usr/bin/sudo": true}
}, "a setuid-root program no package owns, which agent can execute: /usr/local/bin/rootshell"},
} {
t.Run(c.name, func(t *testing.T) {
m := clean()
c.set(m)
v := lookAgent(t, m)
if v.State != Unhealthy || !strings.HasPrefix(v.Reason, ReasonRoot+": ") || !strings.Contains(v.Reason, c.said) {
t.Fatalf("want %q said: %+v", c.said, v)
}
})
}
}
func TestWhatGrantsSomebodyElseIsNoWay(t *testing.T) {
m := clean()
m.texts = map[string]string{"/etc/doas.conf": "permit operator\npermit :wheel\n# permit agent\n",
"/usr/share/polkit-1/rules.d/50-default.rules": `polkit.addAdminRule(function(a, s) { return ["unix-group:wheel"]; });`}
m.dirs = map[string][]string{"/usr/share/polkit-1/rules.d": {"50-default.rules"}}
m.files["/run/docker.sock"] = FileMode{UID: 0, GID: 970, Perm: 0o660}
m.setuid = "/usr/bin/sudo\n/usr/bin/passwd\n"
m.packaged = map[string]bool{"/usr/bin/sudo": true, "/usr/bin/passwd": true}
if v := lookAgent(t, m); v.State != Healthy {
t.Fatalf("the operator's and wheel's grants, a socket of another group, packaged setuid programs: %+v", v)
}
}
func TestDoasWithoutSudoIsJudgedFromItsRules(t *testing.T) {
m := clean()
m.sudo, m.sudoErr = "", fmt.Errorf("sudo: %w", exec.ErrNotFound)
m.texts = map[string]string{"/etc/doas.conf": "permit nopass agent\n"}
v := lookAgent(t, m)
if v.State != Unhealthy || !strings.Contains(v.Reason, "doas permits it") {
t.Fatalf("no sudo is no sudo rule, and doas is read for itself: %+v", v)
}
}
func TestAFurtherQuestionUnansweredIsUnknown(t *testing.T) {
m := clean()
m.findErr = errors.New("find: interrupted")
if v := lookAgent(t, m); v.State != Unknown {
t.Fatalf("a search that did not finish: %+v", v)
}
m = clean()
m.setuid = "/usr/local/bin/x\n"
j := New(Exec{Run: func(ctx context.Context, name string, args ...string) (string, error) {
if name == "pacman" || name == "apk" {
return "", fmt.Errorf("%s: %w", name, exec.ErrNotFound)
}
return m.run(ctx, name, args...)
}, Walk: m.walk, Stat: m.stat, ReadFile: m.readFile, ReadDir: m.readDir, ACL: m.acl, Lstat: m.lstat})
j.Set([]Account{agent})
if st, _ := j.Look(t.Context()); st.Accounts[0].State != Unknown {
t.Fatalf("no package manager to ask: %+v", st.Accounts[0])
}
}
func TestANotJudgedSearchStillSaysAWayFound(t *testing.T) {
j := New(fakeReader{ways: []string{"in the group docker, which grants root"}, err: &SetuidNotJudged{Pending: true, Since: time.Now()}})
j.Set([]Account{{Module: "m", ID: "m.a", Name: "agent", Root: true}})
st, _ := j.Look(context.Background())
if v := st.Accounts[0]; v.State != Unhealthy || !strings.Contains(v.Reason, "docker") || !strings.Contains(v.Reason, ReasonPending) {
t.Fatalf("a way found is unhealthy, with the search's state beside it: %+v", v)
}
j = New(fakeReader{err: &SetuidNotJudged{Since: time.Now(), Err: errors.New("timeout")}})
j.Set([]Account{{Module: "m", ID: "m.a", Name: "agent", Root: true}})
st, _ = j.Look(context.Background())
if v := st.Accounts[0]; v.State != Unknown || !strings.HasPrefix(v.Reason, ReasonIncomplete) {
t.Fatalf("nothing found and the search incomplete: unknown, said so: %+v", v)
}
}
type fakeReader struct {
ways []string
err error
}
func (f fakeReader) InDatabase(context.Context, string) ([]string, error) { return nil, nil }
func (f fakeReader) Session(context.Context, string, []string) (Session, error) {
return Session{}, nil
}
func (f fakeReader) Escalation(context.Context, string, []string) ([]string, error) {
return f.ways, f.err
}
func TestParseACL(t *testing.T) {
entry := func(tag, perm uint16, id uint32) []byte {
b := make([]byte, 8)
binary.LittleEndian.PutUint16(b, tag)
binary.LittleEndian.PutUint16(b[2:], perm)
binary.LittleEndian.PutUint32(b[4:], id)
return b
}
raw := []byte{2, 0, 0, 0}
raw = append(raw, entry(0x01, 6, 0xffffffff)...)
raw = append(raw, entry(0x02, 6, 1600)...) // user agent rw
raw = append(raw, entry(0x04, 4, 0xffffffff)...)
raw = append(raw, entry(0x08, 6, 970)...) // group 970 rw
raw = append(raw, entry(0x10, 4, 0xffffffff)...) // mask r--
raw = append(raw, entry(0x20, 0, 0xffffffff)...)
acl, err := ParseACL(raw)
if err != nil || len(acl) != 2 {
t.Fatalf("%v %v", acl, err)
}
if !acl[0].User || acl[0].ID != 1600 || !acl[0].Read || acl[0].Write {
t.Fatalf("the mask takes write away: %+v", acl[0])
}
if grantsByACL(acl, 1600, nil, true) || !grantsByACL(acl, 1601, map[int]bool{970: true}, false) {
t.Fatal("grants")
}
if _, err := ParseACL([]byte{2, 0, 0}); err == nil {
t.Fatal("a short ACL")
}
}
func TestTheJudgesCommandsRunInTheCLocale(t *testing.T) {
t.Setenv("LC_ALL", "de_DE.UTF-8")
t.Setenv("LANG", "de_DE.UTF-8")
out, err := CLocale(t.Context(), "sh", "-c", `echo "$LC_ALL $LANG"`)
if err != nil || strings.TrimSpace(out) != "C C" {
t.Fatalf("%q %v", out, err)
}
if _, err := CLocale(t.Context(), "sh", "-c", "echo nope >&2; exit 3"); err == nil ||
!strings.Contains(err.Error(), "exited 3: nope") {
t.Fatalf("an exit is said with its words: %v", err)
}
}
func TestTheJudgedListIsWrittenDown(t *testing.T) {
if len(Judged) < 8 || len(NotJudged) == 0 {
t.Fatal("what is judged and what is not are both written down")
}
}
func TestDistributionRulesForASeatOrForSomebodyAreNotEveryone(t *testing.T) {
for _, rule := range []string{
`polkit.addRule(function(a, s) { if (s.local && s.active) return polkit.Result.YES; });`,
`polkit.addRule(function(a, s) { if (s.isInGroup("wheel")) return polkit.Result.YES; });`,
`polkit.addAdminRule(function(a, s) { return ["unix-group:wheel"]; });`,
} {
if PolkitGrantsEveryone("/x.rules", rule) {
t.Errorf("not every account: %s", rule)
}
}
if PolkitGrantsEveryone("/x.pkla", "[a]\nIdentity=unix-group:wheel\nResultAny=yes\n") {
t.Error("a pkla for wheel is not every account")
}
}
func TestAMachineMissingWhatTheJudgeReadsIsUnknown(t *testing.T) {
for _, file := range []string{"/proc/sys/fs/protected_hardlinks", "/proc/net/tcp", "/proc/mounts"} {
m := clean()
delete(m.proc, file)
if v := lookAgent(t, m); v.State != Unknown {
t.Errorf("%s unread: %+v", file, v)
}
}
}
func TestSuidMountsLeaveOutNosuidPseudoAndContainerLayers(t *testing.T) {
got := SuidMounts("/dev/sdb1 /srv xfs rw 0 0\n/dev/sda2 / ext4 rw 0 0\noverlay /var/lib/docker/overlay2/x/merged overlay rw 0 0\n" +
"/dev/sdc1 /data ext4 rw,nosuid 0 0\nproc /proc proc rw 0 0\nhost:/x /mnt/nfs nfs4 rw 0 0\n/dev/sdd1 /my\\040disk ext4 rw 0 0\n")
if strings.Join(got, "|") != "/|/my disk|/srv" {
t.Fatalf("got %q", got)
}
}
// Each polkit rule is judged on its own, comments taken out (the third review of 2026-10-08).
func TestAnUnconditionalPolkitYesIsNotMaskedByAnotherRuleOrAComment(t *testing.T) {
cases := []struct {
name, text string
every bool
}{
{"another rule names a user", `polkit.addRule(function(a, s) { if (s.user == "operator") return polkit.Result.YES; });
polkit.addRule(function(a, s) { return polkit.Result.YES; });`, true},
{"a user named only in a comment", `// subject.user == "x"
polkit.addRule(function(a, s) { /* isInGroup("wheel") */ return polkit.Result.YES; });`, true},
{"the yes only in a comment", `polkit.addRule(function(a, s) { // return polkit.Result.YES;
return polkit.Result.NO; });`, false},
{"a condition in the same rule", `polkit.addRule(function(a, s) { if (s.isInGroup("wheel")) { return polkit.Result.YES; } });`, false},
{"a comment marker inside a string", `polkit.addRule(function(a, s) { var u = "http://x"; return polkit.Result.YES; });`, true},
}
for _, c := range cases {
if got := PolkitGrantsEveryone("/etc/polkit-1/rules.d/x.rules", c.text); got != c.every {
t.Errorf("%s: grants every account %v, want %v", c.name, got, c.every)
}
}
if PolkitGrantsEveryone("/x.pkla", "# [a]\n# Identity=unix-user:*\n# ResultAny=yes\n") {
t.Error("a .pkla section only in comments grants nobody")
}
if !PolkitGrantsEveryone("/x.pkla", "[a]\nIdentity=unix-group:wheel\nResultAny=yes\n[b]\nIdentity=unix-user:*\nResultActive=yes\n") {
t.Error("each .pkla section on its own")
}
}
// The package manager is asked about every path found in one call, and answers each: owned, or owned by nothing.
func TestThePackageManagerIsAskedOnceForEveryPathFound(t *testing.T) {
m := clean()
m.setuid = "/usr/bin/sudo\n/usr/bin/passwd\n/usr/local/bin/x\n/opt/y\n"
m.packaged = map[string]bool{"/usr/bin/sudo": true, "/usr/bin/passwd": true}
search := func(e Exec) ([]string, error) {
return e.setuidSearch(t.Context(), &SetuidProgress{Mounts: []string{"/"}}, func() {}, nil)
}
found, err := search(Exec{Run: m.run, Walk: m.walk})
if err != nil {
t.Fatal(err)
}
if m.pacmanCalls != 1 {
t.Errorf("pacman was asked %d times for four paths", m.pacmanCalls)
}
if strings.Join(found, ",") != "/opt/y,/usr/local/bin/x" {
t.Errorf("unowned: %v", found)
}
// An error pacman says that is not "owned by nothing" is no answer.
odd := func(ctx context.Context, name string, args ...string) (string, error) {
if name == "pacman" {
return "", errors.New("pacman exited 1: error: could not open database")
}
return m.run(ctx, name, args...)
}
if _, err := search(Exec{Run: odd, Walk: m.walk}); err == nil {
t.Error("a package manager that could not answer was read as an answer")
}
// A walk that did not complete is no answer either.
m.findErr = errors.New("could not read /mnt/data/private: permission denied")
if _, err := search(Exec{Run: m.run, Walk: m.walk}); err == nil {
t.Error("a walk that did not complete was read as an answer")
}
}
-23
View File
@@ -3,7 +3,6 @@ package apply
import (
"context"
"os"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
@@ -32,28 +31,6 @@ func TestAnAccessPresentIsConfirmedAndNothingIsChanged(t *testing.T) {
}
}
// Absent is refused, not created. A bind mount whose source does not exist is made by the
// container runtime as root, with whatever mode it picks — the silent wrong-ownership
// 04-ISSUES/026 records. So the host says plainly that the operator must provide the path, rather
// than conjuring a directory it does not own.
func TestAnAccessThatIsAbsentIsRefusedClearlyAndNotCreated(t *testing.T) {
missing := t.TempDir() + "/media/library" // named, never created
d := declare(t, `{"id":"lib","type":"access","path":"`+missing+`"}`)
_, _, err := Apply(context.Background(), archHost(t), d, store.State{},
store.OriginDeclared, noServices, nil, nil)
if err == nil {
t.Fatal("an absent operator-owned path was accepted, and would be created as root by the runtime")
}
if !strings.Contains(err.Error(), "the operator must provide") ||
!strings.Contains(err.Error(), "does not own") {
t.Fatalf("the refusal does not say whose the path is: %v", err)
}
if _, statErr := os.Stat(missing); statErr == nil {
t.Fatal("the host created the path it does not own")
}
}
// Undeclaring an access never removes the path. Unassigning the module that reached the media
// library must not delete the library — that is the data loss ADR 0030 exists to prevent, on a
// directory the mesh never made. The record is dropped; the operator's data is left exactly as it
+76
View File
@@ -0,0 +1,76 @@
package apply
// What is never placed below an agent's home (novox/hq ADR 0266 §7).
//
// An account that never becomes root owns every directory in its home, so it can swap a link into the path
// between the engine's check and the engine's write. The engine writes a whole file there through a
// descriptor that followed no link (home_links.go), but it unpacks an archive, and writes into an existing
// file, by path. So below the home of an account the declaration says never becomes root, neither is placed.
// The catalogue's own check (checks/agenthome) refuses either under `${machine:agent-home}`, so no module
// asks; this is the engine refusing it again where it applies. The machine's own trees, other accounts'
// homes and a directory's owner below a home are placement_guard.go's (novox/hq issue 339).
import (
"fmt"
"path/filepath"
"github.com/novox/mesh-host/internal/declaration"
)
// AgentHomeRefusedError is an archive or a written-into file the engine will not place below an agent's home.
type AgentHomeRefusedError struct {
Path, What, Home string
}
func (e *AgentHomeRefusedError) Error() string {
return fmt.Sprintf("%s is not placed: %s below %s, the home of an account that never becomes root, which "+
"owns every parent there and could swap a link in between the check and the write (novox/hq ADR 0266); "+
"nothing was touched", e.Path, e.What, e.Home)
}
// refuseUnderAgentHome says why a resource is not placed below one of agentHomes; nil when it may be.
func refuseUnderAgentHome(r declaration.Resource, agentHomes []string) error {
what := ""
switch res := r.(type) {
case *declaration.Archive:
what = "an archive unpacked"
case *declaration.File:
if res.Into != "" {
what = "a file written into (" + res.Into + ")"
}
}
if what == "" {
return nil
}
path := filepath.Clean(r.Target())
for _, p := range []string{path, resolved(path)} {
for _, home := range agentHomes {
if atOrBelow(p, home) {
return &AgentHomeRefusedError{Path: path, What: what, Home: home}
}
}
}
return nil
}
// rootNeverHomes is the home of every account the declaration says never becomes root, from the user
// database; an account not made yet has no home to protect.
func rootNeverHomes(d *declaration.Declaration) []string {
if d == nil {
return nil
}
homes := accountsOfHomes()
var out []string
for _, r := range d.Resources {
u, ok := r.(*declaration.User)
if !ok || u.Root != declaration.RootNever {
continue
}
for home, a := range homes {
if a.Name == u.Name {
out = append(out, home)
}
}
}
return out
}
+36
View File
@@ -0,0 +1,36 @@
package apply
// Defends novox/hq ADR 0266 §7: no archive is unpacked and no file is written into below the home of an account
// the declaration says never becomes root; a whole file there still passes.
import (
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
func TestNothingIsUnpackedOrWrittenIntoBelowAnAgentsHome(t *testing.T) {
was := accountsOfHomes
accountsOfHomes = func() map[string]homeAccount {
return map[string]homeAccount{"/home/agent": {"agent", 1001}, "/home/operator": {"operator", 1000}}
}
t.Cleanup(func() { accountsOfHomes = was })
agent := []string{"/home/agent"}
if err := refuseUnderAgentHome(&declaration.Archive{ID: "a.x", Type: declaration.TypeArchive, Path: "/home/agent/.local/x", Owner: "agent"}, agent); err == nil {
t.Error("an archive below an agent's home is refused")
}
if err := refuseUnderAgentHome(&declaration.File{ID: "a.f", Type: declaration.TypeFile, Path: "/home/agent/.claude.json", Owner: "agent", Into: "json", Content: "{}"}, agent); err == nil {
t.Error("a file written into below an agent's home is refused")
}
if err := refuseUnderAgentHome(&declaration.File{ID: "a.f", Type: declaration.TypeFile, Path: "/home/agent/x", Owner: "agent", Content: "x"}, agent); err != nil {
t.Errorf("a whole file there passes: %v", err)
}
if err := refuseUnderAgentHome(&declaration.Archive{ID: "a.x", Type: declaration.TypeArchive, Path: "/home/operator/x", Owner: "operator"}, agent); err != nil {
t.Errorf("an archive in a home whose account may become root is placement_guard.go's to judge, not this: %v", err)
}
d := parse(t, `{"declaration":1,"resources":[{"id":"c.agent","type":"user","name":"agent","root":"never"},`+
`{"id":"c.op","type":"user","name":"operator"}]}`)
if got := rootNeverHomes(d); len(got) != 1 || got[0] != "/home/agent" {
t.Errorf("the agent's home is known by the user database: %v", got)
}
}
+2237 -133
View File
File diff suppressed because it is too large Load Diff
+142 -32
View File
@@ -313,6 +313,7 @@ func TestAServiceIsReadBackNotAssumed(t *testing.T) {
// running. A unit that starts and immediately dies satisfies the command.
started := false
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if args[0] == "show" {
if started {
return "LoadState=loaded\nActiveState=failed\n", nil // started, then died
@@ -337,6 +338,7 @@ func TestAServiceIsReadBackNotAssumed(t *testing.T) {
func TestAnUnknownServiceStateIsRefusedNotGuessed(t *testing.T) {
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "LoadState=loaded\nActiveState=reticent\n", nil
}
d := parse(t, `{"declaration":1,"resources":[
@@ -348,33 +350,113 @@ func TestAnUnknownServiceStateIsRefusedNotGuessed(t *testing.T) {
}
}
func TestADroppedServiceIsStoppedNotDeleted(t *testing.T) {
// The host did not install the unit and does not own the unit file — only the state it put
// the unit into.
var commands []string
func TestADroppedServiceIsGivenBackTheStateItWasFoundIn(t *testing.T) {
// novox/hq ADR 0118: undeclaring removes what the mesh made, gives back what it changed, and
// leaves what was the machine's. A service resource never installs a unit — so undeclaring it
// undoes what the mesh did to the unit, and nothing more. Stopping every undeclared unit is
// how unassigning the private network stopped the container runtime (novox/hq issue 130).
cases := []struct {
name string
found *store.FoundUnit
action string
stop bool
disable bool
}{
{"recorded before the host kept what it found", nil, "forgotten", false, false},
{"running before the mesh", &store.FoundUnit{State: "running"}, "forgotten", false, false},
{"running and enabled before the mesh", &store.FoundUnit{State: "running", Boot: "enabled"}, "forgotten", false, false},
{"started by the mesh", &store.FoundUnit{State: "stopped"}, "restored", true, false},
{"started and enabled by the mesh", &store.FoundUnit{State: "stopped", Boot: "disabled"}, "restored", true, true},
{"enabled by the mesh, running before it", &store.FoundUnit{State: "running", Boot: "disabled"}, "restored", false, true},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
var commands []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
commands = append(commands, strings.Join(args, " "))
if args[0] == "show" {
return "LoadState=loaded\nActiveState=active\n", nil
}
return "", nil
}
state := store.State{Resources: []store.Applied{
{ID: "s", Type: "service", Target: "unit.service", Found: c.found},
}}
d := parse(t, `{"declaration":1,"resources":[
{"id":"other","type":"file","path":"`+filepath.Join(t.TempDir(), "a")+`","content":"a\n"}
]}`)
report, after, err := Apply(context.Background(), archHost(t), d, state, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
joined := strings.Join(commands, "; ")
if stopped := strings.Contains(joined, "stop unit.service"); stopped != c.stop {
t.Errorf("stopped %v, want %v: %s", stopped, c.stop, joined)
}
if disabled := strings.Contains(joined, "disable unit.service"); disabled != c.disable {
t.Errorf("disabled %v, want %v: %s", disabled, c.disable, joined)
}
if strings.Contains(joined, "start unit.service") || strings.Contains(joined, "mask") {
t.Errorf("the host started or masked a unit on its way out: %s", joined)
}
if o := outcomeOf(report, "s"); o.Action != c.action {
t.Errorf("outcome %+v, want %s", o, c.action)
}
if _, still := after.Find("s"); still {
t.Error("the host still believes it owns the undeclared service")
}
})
}
}
func TestAServiceRecordsTheStateItWasFoundInOnceAndCarriesIt(t *testing.T) {
// Read the first time the host applies the unit — before it starts or enables anything —
// and never again: by the next apply, the unit's state is the mesh's doing.
active := "inactive"
enabled := "disabled"
run := func(ctx context.Context, name string, args ...string) (string, error) {
commands = append(commands, strings.Join(args, " "))
if args[0] == "show" {
return "LoadState=loaded\nActiveState=active\n", nil
args = oldShape(name, args)
switch args[0] {
case "show":
return "LoadState=loaded\nActiveState=" + active + "\n", nil
case "is-enabled":
return enabled, nil
case "start":
active = "active"
case "enable":
enabled = "enabled"
}
return "", nil
}
state := store.State{Resources: []store.Applied{
{ID: "s", Type: "service", Target: "gone.service"},
}}
d := parse(t, `{"declaration":1,"resources":[
{"id":"other","type":"file","path":"`+filepath.Join(t.TempDir(), "a")+`","content":"a\n"}
{"id":"s","type":"service","unit":"filter.service","state":"running","boot":"enabled"}
]}`)
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginCarried, run, nil, nil); err != nil {
_, first, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
joined := strings.Join(commands, "; ")
if !strings.Contains(joined, "stop gone.service") {
t.Errorf("the dropped service was not stopped: %s", joined)
rec, _ := first.Find("s")
if rec.Found == nil || rec.Found.State != "stopped" || rec.Found.Boot != "disabled" {
t.Fatalf("first apply recorded %+v, want stopped and disabled", rec.Found)
}
if strings.Contains(joined, "disable") || strings.Contains(joined, "mask") {
t.Errorf("the host did more than stop a unit it does not own: %s", joined)
_, second, err := Apply(context.Background(), archHost(t), d, first, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
rec, _ = second.Find("s")
if rec.Found == nil || rec.Found.State != "stopped" {
t.Errorf("a later apply replaced what was found with what the mesh made: %+v", rec.Found)
}
// A record from before the host kept what it found is not given one later.
old := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "filter.service"}}}
_, third, err := Apply(context.Background(), archHost(t), d, old, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if rec, _ = third.Find("s"); rec.Found != nil {
t.Errorf("a record that predates the field was given one after the mesh had acted: %+v", rec.Found)
}
}
@@ -386,6 +468,7 @@ func TestAUnitThatDoesNotExistIsNotStopped(t *testing.T) {
// Absence read as satisfaction — 04-ISSUES/007 wearing a different hat, and the mirror of
// the degraded-init bug the capability detector had.
absent := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "LoadState=not-found\nActiveState=inactive\n", nil
}
d := parse(t, `{"declaration":1,"resources":[
@@ -408,6 +491,7 @@ func TestAMaskedUnitIsRefused(t *testing.T) {
// Masked means someone deliberately made it unstartable. Applying over that would undo a
// decision the host did not make and cannot see the reason for.
masked := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "LoadState=masked\nActiveState=inactive\n", nil
}
d := parse(t, `{"declaration":1,"resources":[
@@ -428,6 +512,7 @@ func TestForgettingAUnitThatIsGoneDoesNotStrandTheNode(t *testing.T) {
// already true.
var stopped bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if args[0] == "show" {
return "LoadState=not-found\nActiveState=inactive\n", nil
}
@@ -479,6 +564,7 @@ func TestABrokenPackageDatabaseIsNotReadAsNotInstalled(t *testing.T) {
// answer would silently reinstall on a machine whose package manager is broken, or report
// "installed nothing" as success. The apply must fail instead.
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "", errors.New("pacman: error: could not lock database")
}
d := parseTrusted(t, `{"declaration":1,"resources":[
@@ -497,6 +583,7 @@ func TestABrokenPackageDatabaseIsNotReadAsNotInstalled(t *testing.T) {
func TestAnInstalledPackageIsNotReinstalled(t *testing.T) {
var installed bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if args[0] == "-S" {
installed = true
}
@@ -524,6 +611,7 @@ func TestAPackageIsNeverUninstalled(t *testing.T) {
// machine may have had it before the mesh ever saw it. Undeclaring is not "remove it".
var uninstalled bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if len(args) > 0 && (args[0] == "-R" || args[0] == "-Rs") {
uninstalled = true
}
@@ -557,6 +645,7 @@ func TestAnActionThatIsAlreadyTrueDoesNotRun(t *testing.T) {
// database is, so "is it already there" is a question only the declaration can ask.
var ran bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "create-db" {
ran = true
}
@@ -583,6 +672,7 @@ func TestAnActionThatSucceedsAndDoesNothingFails(t *testing.T) {
// this repository's most expensive failure shape. Here the command "succeeds" every time
// and verify never passes.
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "has-db" {
return "", errors.New("no such database")
}
@@ -609,6 +699,7 @@ func TestAnActionRunsInsideTheContainerItNames(t *testing.T) {
// there is any mesh to ask.
var sawExec bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "docker" && args[0] == "exec" && args[1] == "store" {
sawExec = true
return "", nil
@@ -632,10 +723,11 @@ func TestAContainerThatExitsImmediatelyFailsTheApply(t *testing.T) {
// still running. A container whose entrypoint dies satisfies the command exactly as one
// that came up does — which is the read-back rule, in the place it matters most.
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch {
case args[0] == "info":
return "27.0\n", nil
case args[0] == "inspect":
case args[0] == "container":
return "false\t" + "", nil // exists, not running
case args[0] == "run":
return "deadbeef\n", nil
@@ -666,14 +758,15 @@ func TestAContainerWhoseDeclarationChangedIsReplaced(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"store","image":"`+pinned+`","env":{"PGDATA":"/data"}}
]}`)
want := containerSpec(d.Resources[0].(*declaration.Container), nil)
want := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
if created {
return "true\t" + want, nil
}
@@ -704,14 +797,15 @@ func TestAContainerThatMatchesIsLeftAlone(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"store","image":"`+pinned+`","env":{"PGDATA":"/data"}}
]}`)
spec := containerSpec(d.Resources[0].(*declaration.Container), nil)
spec := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
var touched bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
return "true\t" + spec, nil
}
touched = true
@@ -748,15 +842,16 @@ func TestAContainerIsRecreatedWhenARestartOnResourceChanged(t *testing.T) {
// what a container reads is part of what it is, so the old content yields a different spec.
was := map[string]string{"config": declaredDigest(&declaration.File{Content: "{\"token\":\"old\"}\n"})}
now := map[string]string{"config": declaredDigest(d.Resources[0].(*declaration.File))}
stale := containerSpec(d.Resources[1].(*declaration.Container), was)
fresh := containerSpec(d.Resources[1].(*declaration.Container), now)
stale := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: was})
fresh := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: now})
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
// Already there and running, created against the file as it was. After the host
// recreates it, the runtime holds the one it just made — as a real one would.
if created {
@@ -808,6 +903,7 @@ func TestRestartOnFiresOnlyForResourcesThatChanged(t *testing.T) {
func systemctlStub(t *testing.T, load, active, enabled string, verbs *[]string) Runner {
t.Helper()
return func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "show":
return "LoadState=" + load + "\nActiveState=" + active + "\n", nil
@@ -965,6 +1061,7 @@ func TestTheRuntimeProbeIsPerRuntime(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
var probedWith string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != tc.present {
return "", errors.New("not installed")
}
@@ -993,6 +1090,7 @@ func TestAContainerUsesTheRuntimeTheMachineHas(t *testing.T) {
// machine already has (novox/hq research 012), so hardcoding one contradicts it.
var calledWith []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "docker" {
return "", errors.New("not installed")
}
@@ -1000,7 +1098,7 @@ func TestAContainerUsesTheRuntimeTheMachineHas(t *testing.T) {
switch args[0] {
case "info":
return "6.1.0\n", nil
case "inspect":
case "container":
return "false\t\n", errors.New("no such container")
case "run":
return "deadbeef\n", nil
@@ -1029,6 +1127,7 @@ func TestNoRuntimeIsSaidPlainly(t *testing.T) {
// Naming what was tried, because "docker: command not found" on a machine that deliberately
// runs podman sends the reader looking for the wrong thing.
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "", errors.New("not installed")
}
d := parseTrusted(t, `{"declaration":1,"resources":[
@@ -1172,6 +1271,7 @@ func TestAServiceIsNotRestartedByAChangeItDoesNotName(t *testing.T) {
// before and after one.
func recordingServices(commands *[]string) Runner {
return func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
line := name + " " + strings.Join(args, " ")
*commands = append(*commands, line)
switch {
@@ -1320,13 +1420,14 @@ func TestAFailedActionStopsWhatFollows(t *testing.T) {
func TestAContainerIsGivenTheMeshsNames(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("not installed")
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "inspect":
case "container":
return "false\t\n", errors.New("no such container")
case "run":
ran = args
@@ -1355,13 +1456,14 @@ func TestAContainerIsGivenTheMeshsNames(t *testing.T) {
func TestAContainerGivenNoNamesIsRunAsBefore(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("not installed")
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "inspect":
case "container":
return "false\t\n", errors.New("no such container")
case "run":
ran = args
@@ -1399,6 +1501,7 @@ func TestResourcesAreAppliedInTheOrderTheyWereDeclared(t *testing.T) {
var order []string
done := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
order = append(order, name+" "+strings.Join(args, " "))
// `verify` is the idempotency check, so it has to fail before the step and pass after —
// a stub that always succeeds means every action is already done and nothing ever runs,
@@ -1487,6 +1590,12 @@ func TestADirectoryHoldingAnythingTheMeshDidNotPutThereIsKept(t *testing.T) {
for _, o := range report.Outcomes {
if o.Action == "kept" && strings.Contains(o.Detail, "did not put there") {
said = true
// And it says how data goes, which is never by hand on this line's word (novox/hq ADR
// 0233): a person's `cleanup delete`, after a last restore point.
if strings.Contains(o.Detail, "by hand") || !strings.Contains(o.Detail, "never deletes data") ||
!strings.Contains(o.Detail, "cleanup delete") {
t.Errorf("the kept directory is said as %q", o.Detail)
}
}
}
if !said {
@@ -1538,15 +1647,16 @@ func TestAContainerStaleFromAnEarlierApplyIsReplaced(t *testing.T) {
// The container was created when the file said something else.
was := map[string]string{"env": declaredDigest(&declaration.File{Content: "PASSWORD=old\n"})}
now := map[string]string{"env": declaredDigest(d.Resources[0].(*declaration.File))}
stale := containerSpec(d.Resources[1].(*declaration.Container), was)
fresh := containerSpec(d.Resources[1].(*declaration.Container), now)
stale := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: was})
fresh := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: now})
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
if created {
return "true\t" + fresh, nil
}
+120 -13
View File
@@ -35,11 +35,11 @@ import (
// enough for a desktop theme and small enough to notice.
const maxArchive = 512 << 20
func applyArchive(ctx context.Context, r *declaration.Archive, previous store.Applied) (Outcome, error) {
func applyArchive(ctx context.Context, r *declaration.Archive, previous store.Applied, away *storeAway) (Outcome, error) {
out := begin(r)
out.Action = "unchanged"
body, err := fetch(ctx, r.Source)
body, err := fetchMinding(ctx, r.Source, away)
if err != nil {
return out, err
}
@@ -56,26 +56,28 @@ func applyArchive(ctx context.Context, r *declaration.Archive, previous store.Ap
out.wrote = got
// Already what it should be. The digest is the whole identity of an archive, so a matching
// record means the unpacked tree came from these exact bytes.
if previous.Wrote == got {
// record means the unpacked tree came from these exact bytes — at this path: a record of the
// same bytes somewhere else says nothing about what is here.
if previous.Wrote == got && previous.Target == r.Path {
if _, err := os.Stat(r.Path); err == nil {
owned, err := ownedBy(r.Path, r.Owner)
owned, err := ownedBy(ownershipProbe(r.Path, previous.Unpacked), r.Owner)
if err == nil && owned {
// What it unpacked is carried, or — on a record from before the host kept it — read
// from the archive now, so the record can say it from here on (novox/hq issue 162).
out.unpacked, err = stillUnpacked(body, r.Path, previous.Unpacked)
if err != nil {
return out, err
}
return out, nil
}
}
}
if err := os.MkdirAll(r.Path, 0o755); err != nil {
return out, err
}
written, err := unpack(body, r.Path)
unpacked, written, err := replaceWith(body, r.Path, r.Owner, oursFrom(previous, r.Path))
if err != nil {
return out, err
}
if err := ownAll(r.Path, r.Owner); err != nil {
return out, err
}
out.unpacked = &unpacked
out.Action = "updated"
if previous.Wrote == "" {
out.Action = "created"
@@ -84,6 +86,111 @@ func applyArchive(ctx context.Context, r *declaration.Archive, previous store.Ap
return out, nil
}
// replaceWith makes the directory exactly the archive (novox/hq issue 220), and says what it put
// there (novox/hq issue 162).
//
// **The tree on disk is the archive and nothing else — of what the mesh put there.** The digest is
// the whole identity of what is unpacked here, so a file the previous archive had and this one does
// not must go. Unpacked over the old tree, it stayed: a bundle rebuilt as one file per entrypoint
// kept the package directory of the version before, which code could still import, and a fix that
// removed a file worked on a fresh machine only. So the archive is unpacked into a fresh directory
// beside the old one, owned, and swapped in by rename. A running process keeps the files it has open,
// and the old tree is removed only once the new one is in place. A failed unpack leaves the old tree
// untouched.
//
// **What the mesh did not put there is never swapped away** (novox/hq issue 162, ADR 0030). The
// swap is for a directory that is the host's own: one it made, holding nothing but what the mesh
// put there. A directory that was there before the archive, or that something else has written
// into since, is the machine's: the archive's files are moved into it one by one, what the previous
// archive placed and this one does not is taken out, and everything else is left as it is. A file
// the archive would write over that the mesh did not put there refuses the archive before anything
// is moved — unless it already holds exactly the archive's bytes.
func replaceWith(body []byte, path, owner string, o ours) (store.Unpacked, int, error) {
parent := filepath.Dir(path)
madeParents, err := makeDirsSaying(parent, 0o755, owner)
if err != nil {
return store.Unpacked{}, 0, err
}
parents := joinParents(madeParents, o.parents)
fresh := path + ".unpacking"
replaced := path + ".replaced"
// What an interrupted earlier attempt — or removal — left beside the directory.
for _, leftover := range []string{fresh, replaced, path + ".removing"} {
if err := os.RemoveAll(leftover); err != nil {
return store.Unpacked{}, 0, err
}
}
if err := os.Mkdir(fresh, 0o755); err != nil {
return store.Unpacked{}, 0, err
}
written, err := unpack(body, fresh)
if err == nil {
err = ownAll(fresh, owner)
}
var files, dirs []string
if err == nil {
files, dirs, err = treeOf(fresh)
}
if err != nil {
os.RemoveAll(fresh)
return store.Unpacked{}, written, err
}
info, err := os.Lstat(path)
existed := err == nil
if err != nil && !os.IsNotExist(err) {
os.RemoveAll(fresh)
return store.Unpacked{}, written, err
}
if existed && !info.IsDir() {
// Swapped, it would be deleted: a file at the path is nothing an archive put there.
os.RemoveAll(fresh)
return store.Unpacked{}, written, fmt.Errorf(
"%s is there and is not a directory, and the mesh did not put it there; nothing was unpacked", path)
}
foreign := 0
if existed && !o.all {
if foreign, err = foreignIn(path, o.paths); err != nil {
os.RemoveAll(fresh)
return store.Unpacked{}, written, err
}
}
if existed && (foreign > 0 || !(o.made || o.all)) {
u, err := mergeInto(fresh, path, owner, files, dirs, o)
os.RemoveAll(fresh)
if err != nil {
return store.Unpacked{}, written, err
}
u.Parents = parents
return u, written, nil
}
hadOne := true
if err := os.Rename(path, replaced); err != nil {
if !os.IsNotExist(err) {
os.RemoveAll(fresh)
return store.Unpacked{}, written, err
}
hadOne = false
}
if err := os.Rename(fresh, path); err != nil {
if hadOne {
// Put the old tree back rather than leave nothing at the path.
os.Rename(replaced, path)
}
os.RemoveAll(fresh)
return store.Unpacked{}, written, err
}
// The directory is the host's own: it made it, now or before, and nothing else is in it.
u := store.Unpacked{Files: files, Dirs: dirs, Made: true, Parents: parents}
if hadOne {
if err := os.RemoveAll(replaced); err != nil {
return u, written, fmt.Errorf("%s is in place, and the tree it replaced could not be removed: %w", path, err)
}
}
return u, written, nil
}
func fetch(ctx context.Context, source string) ([]byte, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, source, nil)
if err != nil {
@@ -95,7 +202,7 @@ func fetch(ctx context.Context, source string) ([]byte, error) {
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return nil, fmt.Errorf("%s answered %s", source, response.Status)
return nil, statusError{source: source, code: response.StatusCode, status: response.Status}
}
body, err := io.ReadAll(io.LimitReader(response.Body, maxArchive+1))
if err != nil {
+261
View File
@@ -0,0 +1,261 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq issue 162: an archive can be undeclared. What it unpacked is gone, anything that
// was in its directory beforehand is still there, and the apply that removed it applied everything
// else in the same declaration.
func exists(path string) bool {
_, err := os.Lstat(path)
return err == nil
}
func archiveDecl(t *testing.T, id, path string, files map[string]string, more string) (string, string) {
t.Helper()
body, digest := anArchive(t, files)
return `{"id":"` + id + `","type":"archive","source":"` + serving(t, body) + `","digest":"` + digest +
`","path":"` + path + `"}` + more, digest
}
func TestUnassigningAnArchiveRemovesExactlyWhatItUnpacked(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "bundles", "notes", "tools")
archive, _ := archiveDecl(t, "notes.tools", target,
map[string]string{"index.js": "x", "lib/one.js": "1", "lib/two.js": "2"},
`,{"id":"notes.conf","type":"file","path":"`+dir+`/notes.conf","content":"a"}`)
_, state, err := Apply(context.Background(), archHost(t), declare(t, archive), store.State{},
store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if rec, _ := state.Find("notes.tools"); rec.Unpacked == nil || !rec.Unpacked.Made ||
len(rec.Unpacked.Files) != 3 || len(rec.Unpacked.Parents) != 2 {
t.Fatalf("what the archive unpacked was not recorded: %+v", rec.Unpacked)
}
// Unassigned, and the same push declares something new: both happen.
d := declare(t, `{"id":"notes.conf","type":"file","path":"`+dir+`/notes.conf","content":"a"},
{"id":"other.conf","type":"file","path":"`+dir+`/other.conf","content":"b"}`)
report, state, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatalf("undeclaring an archive stopped the apply: %v", err)
}
if o := outcomeOf(report, "notes.tools"); o.Action != "removed" {
t.Fatalf("the archive: %+v", o)
}
if o := outcomeOf(report, "other.conf"); o.Action != "created" {
t.Fatalf("the rest of the declaration: %+v", o)
}
if _, still := state.Find("notes.tools"); still {
t.Fatal("the archive is still on record")
}
// The directory it was unpacked into and the parents the host made to reach it, all gone.
if exists(filepath.Join(dir, "bundles")) {
t.Fatal("what the host made for the archive is still there")
}
if !exists(filepath.Join(dir, "notes.conf")) {
t.Fatal("a file the archive did not place was removed")
}
if entries, _ := os.ReadDir(dir); len(entries) != 2 {
t.Fatalf("%d entries left in the directory", len(entries))
}
}
func TestADirectoryFoundBeforeTheArchiveKeepsWhatWasInIt(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "powerlevel10k")
if err := os.MkdirAll(filepath.Join(target, "lib"), 0o755); err != nil {
t.Fatal(err)
}
os.WriteFile(filepath.Join(target, "mine.zsh"), []byte("somebody's"), 0o644)
os.WriteFile(filepath.Join(target, "lib", "mine.zsh"), []byte("somebody's"), 0o644)
archive, _ := archiveDecl(t, "shell.theme", target,
map[string]string{"p10k.zsh": "theme", "lib/theme.zsh": "lib", "gitstatus/gs": "gs"}, "")
_, state, err := Apply(context.Background(), archHost(t), declare(t, archive), store.State{},
store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
// Applying over a directory that was there keeps what was in it.
for _, kept := range []string{"mine.zsh", "lib/mine.zsh"} {
if got, _ := os.ReadFile(filepath.Join(target, kept)); string(got) != "somebody's" {
t.Fatalf("%s after the apply: %q", kept, got)
}
}
if got, _ := os.ReadFile(filepath.Join(target, "lib", "theme.zsh")); string(got) != "lib" {
t.Fatalf("lib/theme.zsh is %q", got)
}
rec, _ := state.Find("shell.theme")
if rec.Unpacked == nil || rec.Unpacked.Made || strings.Join(rec.Unpacked.Dirs, ",") != "gitstatus" {
t.Fatalf("recorded as %+v", rec.Unpacked)
}
// A new version without one of its files: that file goes, nothing else does.
archive, _ = archiveDecl(t, "shell.theme", target, map[string]string{"p10k.zsh": "theme 2"}, "")
if _, state, err = Apply(context.Background(), archHost(t), declare(t, archive), state,
store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
if exists(filepath.Join(target, "lib", "theme.zsh")) || exists(filepath.Join(target, "gitstatus")) {
t.Fatal("the previous version's files are still there")
}
if !exists(filepath.Join(target, "lib", "mine.zsh")) {
t.Fatal("a file the mesh did not put there went with the previous version")
}
report, _, err := Apply(context.Background(), archHost(t), declare(t, `{"id":"other","type":"file","path":"`+dir+`/other","content":"o"}`), state,
store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "shell.theme"); o.Action != "removed" || !strings.Contains(o.Detail, "there before") {
t.Fatalf("the archive: %+v", o)
}
if exists(filepath.Join(target, "p10k.zsh")) {
t.Fatal("the archive's file is still there")
}
for _, kept := range []string{"mine.zsh", "lib/mine.zsh"} {
if got, _ := os.ReadFile(filepath.Join(target, kept)); string(got) != "somebody's" {
t.Fatalf("%s after the removal: %q", kept, got)
}
}
}
// A file the archive would write over that the mesh did not put there refuses the archive, and the
// directory is left exactly as it was.
func TestAnArchiveDoesNotWriteOverAFileItFound(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "theme")
os.MkdirAll(target, 0o755)
os.WriteFile(filepath.Join(target, "p10k.zsh"), []byte("somebody's"), 0o644)
archive, _ := archiveDecl(t, "shell.theme", target, map[string]string{"p10k.zsh": "theme", "x": "x"}, "")
if _, _, err := Apply(context.Background(), archHost(t), declare(t, archive), store.State{},
store.OriginDeclared, noServices, nil, nil); err == nil || !strings.Contains(err.Error(), "did not put there") {
t.Fatalf("written over: %v", err)
}
if got, _ := os.ReadFile(filepath.Join(target, "p10k.zsh")); string(got) != "somebody's" {
t.Fatalf("p10k.zsh is %q", got)
}
if exists(filepath.Join(target, "x")) {
t.Fatal("part of a refused archive was moved in")
}
if entries, _ := os.ReadDir(dir); len(entries) != 1 {
t.Fatalf("%d entries beside the directory", len(entries))
}
}
func TestADirectoryTheHostMadeGoesOnlyWhenEmpty(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "theme")
archive, _ := archiveDecl(t, "shell.theme", target, map[string]string{"p10k.zsh": "t", "lib/a.zsh": "a"}, "")
_, state, err := Apply(context.Background(), archHost(t), declare(t, archive), store.State{},
store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
// Somebody writes into the directory the host made.
os.WriteFile(filepath.Join(target, "lib", "local.zsh"), []byte("mine"), 0o644)
report, _, err := Apply(context.Background(), archHost(t), declare(t, `{"id":"other","type":"file","path":"`+dir+`/other","content":"o"}`), state,
store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
o := outcomeOf(report, "shell.theme")
if o.Action != "removed" || !strings.Contains(o.Detail, "did not put there") {
t.Fatalf("the archive: %+v", o)
}
if exists(filepath.Join(target, "p10k.zsh")) || exists(filepath.Join(target, "lib", "a.zsh")) {
t.Fatal("the archive's files are still there")
}
if got, _ := os.ReadFile(filepath.Join(target, "lib", "local.zsh")); string(got) != "mine" {
t.Fatalf("a file the archive did not place: %q", got)
}
}
// An archive recorded before the host kept what it unpacked cannot be told from anything else in
// its directory: it is left in place, said and forgotten, and the apply goes on.
func TestAnArchiveRecordedBeforeItsFilesWereKeptIsLeftAndForgotten(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "tools")
os.MkdirAll(target, 0o755)
os.WriteFile(filepath.Join(target, "index.js"), []byte("x"), 0o644)
known := store.State{Resources: []store.Applied{
{ID: "notes.tools", Type: "archive", Target: target, Wrote: "sha256:old", Origin: store.OriginDeclared},
}}
d := declare(t, `{"id":"notes.conf","type":"file","path":"`+dir+`/notes.conf","content":"a"}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatalf("an archive from before stopped the apply: %v", err)
}
if o := outcomeOf(report, "notes.tools"); o.Action != "forgotten" || !strings.Contains(o.Detail, "left in place") {
t.Fatalf("the archive: %+v", o)
}
if o := outcomeOf(report, "notes.conf"); o.Action != "created" {
t.Fatalf("the rest of the declaration: %+v", o)
}
if _, still := state.Find("notes.tools"); still {
t.Fatal("still on record")
}
if !exists(filepath.Join(target, "index.js")) {
t.Fatal("removed without a record of what it unpacked")
}
}
// One recorded before, still declared and unchanged, is read from its own bytes on the next apply,
// so it can be undeclared from then on: the whole directory when it holds exactly the archive, only
// the archive's files when it holds anything else.
func TestAnArchiveRecordedBeforeLearnsWhatItUnpacked(t *testing.T) {
for _, extra := range []bool{false, true} {
dir := t.TempDir()
target := filepath.Join(dir, "tools")
files := map[string]string{"index.js": "x", "lib/a.js": "a"}
archive, digest := archiveDecl(t, "notes.tools", target, files, "")
for name, body := range files {
os.MkdirAll(filepath.Dir(filepath.Join(target, name)), 0o755)
os.WriteFile(filepath.Join(target, name), []byte(body), 0o644)
}
if extra {
os.WriteFile(filepath.Join(target, "lib", "local.js"), []byte("mine"), 0o644)
}
known := store.State{Resources: []store.Applied{
{ID: "notes.tools", Type: "archive", Target: target, Wrote: digest, Origin: store.OriginDeclared},
}}
report, state, err := Apply(context.Background(), archHost(t), declare(t, archive), known,
store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "notes.tools"); o.Action != "unchanged" {
t.Fatalf("an unchanged archive was %s", o.Action)
}
rec, _ := state.Find("notes.tools")
if rec.Unpacked == nil || len(rec.Unpacked.Files) != 2 || rec.Unpacked.Made == extra {
t.Fatalf("extra=%v: learned %+v", extra, rec.Unpacked)
}
if _, _, err := Apply(context.Background(), archHost(t), declare(t, `{"id":"other","type":"file","path":"`+dir+`/other","content":"o"}`), state,
store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
if exists(filepath.Join(target, "index.js")) || exists(filepath.Join(target, "lib", "a.js")) {
t.Fatalf("extra=%v: the archive's files are still there", extra)
}
if exists(target) != extra {
t.Fatalf("extra=%v: the directory is there: %v", extra, exists(target))
}
if extra && !exists(filepath.Join(target, "lib", "local.js")) {
t.Fatal("a file the archive did not place was removed")
}
}
}
+68
View File
@@ -0,0 +1,68 @@
package apply
import (
"context"
"os"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// novox/hq issue 220: the tree on disk is exactly the archive. A file the previous archive had and
// this one does not is gone, and nothing is left beside the directory.
func TestAnArchiveReplacesTheTreeItWasUnpackedOver(t *testing.T) {
dir := t.TempDir()
first, firstDigest := anArchive(t, map[string]string{"index.js": "old", "node_modules/dep/index.js": "dep"})
d := declare(t, `{"id":"bundle-tools","type":"archive","source":"`+serving(t, first)+
`","digest":"`+firstDigest+`","path":"`+dir+`/tools"}`)
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
second, secondDigest := anArchive(t, map[string]string{"index.js": "one file"})
d = declare(t, `{"id":"bundle-tools","type":"archive","source":"`+serving(t, second)+
`","digest":"`+secondDigest+`","path":"`+dir+`/tools"}`)
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginCarried, noServices, nil, nil); err != nil {
t.Fatal(err)
}
if got, _ := os.ReadFile(dir + "/tools/index.js"); string(got) != "one file" {
t.Fatalf("index.js is %q", got)
}
if _, err := os.Stat(dir + "/tools/node_modules"); err == nil {
t.Fatal("the previous archive's directory is still there")
}
entries, _ := os.ReadDir(dir)
if len(entries) != 1 {
names := []string{}
for _, e := range entries {
names = append(names, e.Name())
}
t.Fatalf("beside the tree: %v", names)
}
}
// A tree is replaced only by one that unpacked whole: an archive refused halfway leaves the old
// tree as it was.
func TestARefusedArchiveLeavesTheTreeItWouldHaveReplaced(t *testing.T) {
dir := t.TempDir()
first, firstDigest := anArchive(t, map[string]string{"index.js": "old"})
d := declare(t, `{"id":"bundle-tools","type":"archive","source":"`+serving(t, first)+
`","digest":"`+firstDigest+`","path":"`+dir+`/tools"}`)
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
bad, badDigest := anArchive(t, map[string]string{"index.js": "new", "../../escaped": "no"})
d = declare(t, `{"id":"bundle-tools","type":"archive","source":"`+serving(t, bad)+
`","digest":"`+badDigest+`","path":"`+dir+`/tools"}`)
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginCarried, noServices, nil, nil); err == nil {
t.Fatal("an escaping archive was accepted")
}
if got, _ := os.ReadFile(dir + "/tools/index.js"); string(got) != "old" {
t.Fatalf("index.js is %q after a refused archive", got)
}
if entries, _ := os.ReadDir(dir); len(entries) != 1 {
t.Fatalf("%d entries beside the tree after a refused archive", len(entries))
}
}
+506
View File
@@ -0,0 +1,506 @@
package apply
import (
"archive/tar"
"bytes"
"compress/gzip"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
"github.com/novox/mesh-host/internal/store"
)
// What an archive put on the machine, and taking exactly that away (novox/hq issue 162).
//
// An archive unpacks many files into a directory the mesh did not necessarily make, so undeclaring
// one has a real question in it: remove what the archive put there, or remove the directory? The
// second deletes whatever else lives there — for the host's own versions directory, every other
// delivered version. So the host records what each archive unpacked and whether it made the
// directory, and removal takes away exactly that: the files the archive placed, then the
// directories the host made for them once they are empty. Never a file the archive did not place,
// never a directory that was there before, never one that still holds anything else. It is the
// rule every other kind follows: the mesh gives back what it found (ADR 0118), and data outlives
// the mesh that declared it (ADR 0030).
// ours is what of the tree at an archive's path the record says is the mesh's.
type ours struct {
// all is a record from before the host kept what an archive unpacked: since the swap of issue
// 220 the tree at the path was the archive and nothing else, so the whole of it is taken for
// the mesh's, as the swap that follows has always taken it.
all bool
// paths are the files and directories the previous archive put there, relative to the path.
paths map[string]bool
files []string
dirs []string
made bool
// parents are the directories above the path the host made to reach it, deepest first.
parents []string
}
// oursFrom reads the record of the archive before this apply, for this path only: a record of the
// same archive at a path it has moved from says nothing about what is at the new one.
func oursFrom(previous store.Applied, path string) ours {
if previous.Wrote == "" || previous.Target != path {
return ours{}
}
u := previous.Unpacked
if u == nil {
return ours{all: true, made: true}
}
o := ours{paths: map[string]bool{}, files: u.Files, dirs: u.Dirs, made: u.Made, parents: u.Parents}
for _, rel := range append(append([]string{}, u.Files...), u.Dirs...) {
o.paths[rel] = true
}
return o
}
// ownershipProbe is what says whether an archive is still its owner's. The directory, when the host
// made it; one of the archive's own files when the directory was there before — that one is held as
// found (ADR 0182), so its owner is never the archive's to judge.
func ownershipProbe(path string, u *store.Unpacked) string {
if u != nil && !u.Made && len(u.Files) > 0 {
return filepath.Join(path, u.Files[0])
}
return path
}
// stillUnpacked is what an unchanged archive has on the machine. The record's, when it has one; on
// a record from before the host kept it, read from the archive's own bytes now — and the directory
// is taken for the host's only when it holds exactly the archive and nothing else, which is what the
// swap of issue 220 leaves. Otherwise the directory is kept for somebody's, and only the archive's
// files are recorded as its.
func stillUnpacked(body []byte, path string, recorded *store.Unpacked) (*store.Unpacked, error) {
if recorded != nil {
kept := *recorded
return &kept, nil
}
files, dirs, err := listArchive(body)
if err != nil {
return nil, err
}
u := &store.Unpacked{Files: pathsOf(files)}
if exactly, err := holdsExactly(path, files, dirs); err == nil && exactly {
u.Dirs = pathsOf(dirs)
u.Made = true
}
return u, nil
}
// listArchive reads what an archive holds without unpacking it: each file's digest by its path, and
// every directory, named or implied, relative to where it unpacks. Refused on the same terms as
// unpack, so a listing never names a path an unpack would not write.
func listArchive(body []byte) (map[string]string, map[string]bool, error) {
zipped, err := gzip.NewReader(bytes.NewReader(body))
if err != nil {
return nil, nil, fmt.Errorf("this is not a gzipped tar: %w", err)
}
defer zipped.Close()
files, dirs := map[string]string{}, map[string]bool{}
reader := tar.NewReader(zipped)
for {
header, err := reader.Next()
if err == io.EOF {
return files, dirs, nil
}
if err != nil {
return nil, nil, err
}
rel := filepath.Clean(header.Name)
if rel == "." {
continue
}
if !insideRel(rel) {
return nil, nil, fmt.Errorf("%s names a path outside the archive", header.Name)
}
for d := filepath.Dir(rel); d != "."; d = filepath.Dir(d) {
dirs[filepath.ToSlash(d)] = true
}
switch header.Typeflag {
case tar.TypeDir:
dirs[filepath.ToSlash(rel)] = true
case tar.TypeReg:
sum := sha256.New()
if _, err := io.Copy(sum, io.LimitReader(reader, maxArchive)); err != nil {
return nil, nil, err
}
files[filepath.ToSlash(rel)] = hex.EncodeToString(sum.Sum(nil))
default:
return nil, nil, fmt.Errorf("%s is a %c, and this host unpacks only files and directories",
header.Name, header.Typeflag)
}
}
}
// holdsExactly is whether a directory holds the archive's files with the archive's bytes, its
// directories, and nothing else.
func holdsExactly(root string, files map[string]string, dirs map[string]bool) (bool, error) {
seen := 0
exact := true
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, err := filepath.Rel(root, path)
if err != nil {
return err
}
if rel == "." {
return nil
}
rel = filepath.ToSlash(rel)
switch {
case d.IsDir():
if !dirs[rel] {
exact = false
return filepath.SkipAll
}
case d.Type().IsRegular():
want, ok := files[rel]
if !ok || digestOfFile(path) != want {
exact = false
return filepath.SkipAll
}
seen++
default:
exact = false
return filepath.SkipAll
}
return nil
})
if err != nil {
return false, err
}
return exact && seen == len(files), nil
}
func digestOfFile(path string) string {
file, err := os.Open(path)
if err != nil {
return ""
}
defer file.Close()
sum := sha256.New()
if _, err := io.Copy(sum, file); err != nil {
return ""
}
return hex.EncodeToString(sum.Sum(nil))
}
// treeOf is every file and directory under root, relative to it, slash-separated and sorted.
func treeOf(root string) (files, dirs []string, err error) {
err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, err := filepath.Rel(root, path)
if err != nil || rel == "." {
return err
}
if d.IsDir() {
dirs = append(dirs, filepath.ToSlash(rel))
} else {
files = append(files, filepath.ToSlash(rel))
}
return nil
})
sort.Strings(files)
sort.Strings(dirs)
if files == nil {
files = []string{}
}
return files, dirs, err
}
// foreignIn counts what under root the mesh did not put there. A directory that is not the mesh's
// counts once, with everything in it.
func foreignIn(root string, mine map[string]bool) (int, error) {
count := 0
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, err := filepath.Rel(root, path)
if err != nil || rel == "." {
return err
}
if !mine[filepath.ToSlash(rel)] {
count++
if d.IsDir() {
return filepath.SkipDir
}
}
return nil
})
return count, err
}
// mergeInto moves a freshly unpacked archive into a directory that is not only the mesh's, one entry
// at a time, and takes out what the previous archive placed that this one does not. Everything the
// mesh did not put there stays as it is. Collisions are looked for before anything is moved, so a
// refused archive leaves the directory exactly as it was.
func mergeInto(fresh, path, owner string, files, dirs []string, o ours) (store.Unpacked, error) {
var collisions []string
for _, rel := range dirs {
info, err := os.Lstat(filepath.Join(path, filepath.FromSlash(rel)))
if err == nil && !info.IsDir() && !o.paths[rel] {
collisions = append(collisions, rel)
}
}
for _, rel := range files {
dest := filepath.Join(path, filepath.FromSlash(rel))
info, err := os.Lstat(dest)
switch {
case err != nil:
case o.paths[rel] && !info.IsDir():
case info.IsDir():
if !o.paths[rel] {
collisions = append(collisions, rel)
} else if n, err := foreignIn(dest, o.paths); err != nil || n > 0 {
collisions = append(collisions, rel)
}
case !info.Mode().IsRegular() ||
digestOfFile(dest) != digestOfFile(filepath.Join(fresh, filepath.FromSlash(rel))):
// Already holding exactly the archive's bytes is not a collision: it is what an
// interrupted earlier apply of this same archive left, or the same file either way.
collisions = append(collisions, rel)
}
}
if len(collisions) > 0 {
shown := collisions
if len(shown) > 5 {
shown = shown[:5]
}
return store.Unpacked{}, fmt.Errorf("%s already holds %d path(s) the archive would write over "+
"and the mesh did not put there (%s); nothing was unpacked, and what is there is left as it "+
"is (novox/hq issue 162)", path, len(collisions), strings.Join(shown, ", "))
}
var made []string
for _, rel := range dirs {
dest := filepath.Join(path, filepath.FromSlash(rel))
info, err := os.Lstat(dest)
if err == nil && info.IsDir() {
if o.paths[rel] {
made = append(made, rel)
}
continue
}
if err == nil {
// The previous archive's file where this one has a directory.
if err := os.Remove(dest); err != nil {
return store.Unpacked{}, err
}
}
mode := os.FileMode(0o755)
if from, err := os.Stat(filepath.Join(fresh, filepath.FromSlash(rel))); err == nil {
mode = from.Mode().Perm()
}
if err := os.Mkdir(dest, mode); err != nil {
return store.Unpacked{}, err
}
if err := own(dest, owner); err != nil {
return store.Unpacked{}, err
}
made = append(made, rel)
}
for _, rel := range files {
dest := filepath.Join(path, filepath.FromSlash(rel))
if info, err := os.Lstat(dest); err == nil && info.IsDir() {
// The previous archive's directory where this one has a file, holding nothing else.
if err := os.RemoveAll(dest); err != nil {
return store.Unpacked{}, err
}
}
if err := os.Rename(filepath.Join(fresh, filepath.FromSlash(rel)), dest); err != nil {
return store.Unpacked{}, err
}
}
// What the previous archive placed and this one does not.
now := map[string]bool{}
for _, rel := range append(append([]string{}, files...), dirs...) {
now[rel] = true
}
for _, rel := range o.files {
if now[rel] {
continue
}
if err := os.Remove(filepath.Join(path, filepath.FromSlash(rel))); err != nil && !os.IsNotExist(err) {
return store.Unpacked{}, err
}
}
for _, rel := range deepestFirst(o.dirs) {
if now[rel] {
continue
}
dest := filepath.Join(path, filepath.FromSlash(rel))
if err := os.Remove(dest); err != nil && !os.IsNotExist(err) {
// Still holding something the mesh did not put there: kept, and still the host's to
// take away once it is empty.
made = append(made, rel)
}
}
sort.Strings(made)
return store.Unpacked{Files: files, Dirs: made, Made: o.made}, nil
}
// removeArchive is what undeclaring an archive does (novox/hq issue 162): exactly the files it
// unpacked, then the directories the host made for them once they are empty.
//
// **Never fatal.** An archive that could not be removed stopped the whole apply, on every apply
// after, until it was declared again — so every module with tools was un-unassignable, and a race
// between two pushes froze a machine against every other change. Whatever cannot be taken away is
// said, left in place, and forgotten, as a former target is (issue 194).
//
// **Removed whole when it is the host's own, and in one step.** A directory the host made that
// holds nothing but the archive is renamed aside and then removed: a reader — the runtime serving a
// module's tools from its bundle — sees the whole tree or none of it, never half, and a file it has
// open stays readable until it closes it. The runtime is told the module went by its own membership,
// not by the files disappearing.
func removeArchive(a store.Applied) (string, string, error) {
if store.IsFormer(a.ID) {
// The version before is what a rollback starts (ADR 0141) and what a reader may still have
// open; the launcher retires the host's own versions, not the apply (issue 194).
return "forgotten", "a former target left in place: only an archive the declaration dropped is " +
"taken away (novox/hq issues 162, 194)", nil
}
u := a.Unpacked
if u == nil {
return "forgotten", "left in place: recorded before the host kept what an archive unpacked, so " +
"its files cannot be told from anything else there (novox/hq issue 162)", nil
}
root := filepath.Clean(a.Target)
mine := map[string]bool{}
for _, rel := range append(append([]string{}, u.Files...), u.Dirs...) {
if !insideRel(filepath.FromSlash(rel)) {
return "forgotten", fmt.Sprintf("left in place: its record names %q, which is not inside %s",
rel, root), nil
}
mine[rel] = true
}
info, err := os.Lstat(root)
if os.IsNotExist(err) {
removeParents(root, u.Parents)
return "forgotten", "no longer there", nil
}
if err != nil {
return "forgotten", fmt.Sprintf("left in place: %v", err), nil
}
if !info.IsDir() {
return "forgotten", "left in place: no longer a directory, so not what the archive was unpacked into", nil
}
foreign, err := foreignIn(root, mine)
if err != nil {
return "forgotten", fmt.Sprintf("left in place: cannot read what is in it: %v", err), nil
}
if u.Made && foreign == 0 {
aside := root + ".removing"
if err := os.RemoveAll(aside); err == nil {
if err := os.Rename(root, aside); err == nil {
if err := os.RemoveAll(aside); err != nil {
return "forgotten", fmt.Sprintf("taken out of place, and what it unpacked could not be "+
"removed from %s: %v — remove it by hand", aside, err), nil
}
removeParents(root, u.Parents)
return "removed", fmt.Sprintf("no longer declared; the %d file(s) it unpacked, and the "+
"directory the host made for them", len(u.Files)), nil
}
}
// A rename that could not be made is taken file by file instead.
}
removed := 0
var failed []string
for _, rel := range u.Files {
err := os.Remove(filepath.Join(root, filepath.FromSlash(rel)))
switch {
case err == nil:
removed++
case os.IsNotExist(err):
default:
failed = append(failed, err.Error())
}
}
for _, rel := range deepestFirst(u.Dirs) {
// Only once empty: what is still inside is somebody's.
_ = os.Remove(filepath.Join(root, filepath.FromSlash(rel)))
}
detail := fmt.Sprintf("no longer declared; %d file(s) it unpacked removed", removed)
switch {
case u.Made && os.Remove(root) == nil:
removeParents(root, u.Parents)
detail += ", and the directory the host made for them"
case u.Made:
left, _ := os.ReadDir(root)
detail += fmt.Sprintf("; the directory is kept: %d item(s) inside that the mesh did not put there",
len(left))
default:
detail += "; the directory is kept: it was there before the archive"
}
if len(failed) > 0 {
// Said and not fatal: fatal, the record would stay and fail the same way on every apply
// after — the very wedge this removal exists to end.
return "forgotten", detail + "; could not remove, and left in place: " + strings.Join(failed, "; "), nil
}
return "removed", detail, nil
}
// removeParents takes away the directories above an archive the host made to reach it, deepest
// first, each only once it is empty and only if it is above the archive's directory.
func removeParents(root string, parents []string) {
for _, p := range parents {
clean := filepath.Clean(p)
if !filepath.IsAbs(clean) || !strings.HasPrefix(root, clean+string(os.PathSeparator)) {
continue
}
_ = os.Remove(clean)
}
}
// joinParents is the parents made now and those recorded before, deepest first, once each.
func joinParents(now, before []string) []string {
seen := map[string]bool{}
var out []string
for _, p := range append(append([]string{}, now...), before...) {
if !seen[p] {
seen[p] = true
out = append(out, p)
}
}
sort.Slice(out, func(i, j int) bool { return len(out[i]) > len(out[j]) })
return out
}
// insideRel is whether a relative path stays inside the directory it is relative to.
func insideRel(rel string) bool {
clean := filepath.Clean(rel)
return clean != "." && !filepath.IsAbs(clean) && clean != ".." &&
!strings.HasPrefix(clean, ".."+string(os.PathSeparator))
}
func deepestFirst(rels []string) []string {
out := append([]string{}, rels...)
sort.Slice(out, func(i, j int) bool {
return strings.Count(out[i], "/") > strings.Count(out[j], "/") ||
(strings.Count(out[i], "/") == strings.Count(out[j], "/") && out[i] > out[j])
})
return out
}
func pathsOf[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
+433
View File
@@ -0,0 +1,433 @@
package apply
import (
"errors"
"fmt"
"net"
"os"
"path/filepath"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A file written into a marked block, never over (novox/hq issue 128, ADR 0102).
//
// **The file is the machine's; the mesh owns lines in it.** The machine's hosts file is the case
// that needed it. The mesh wrote it whole — its own header, localhost, the machine's name and every
// name in the mesh — and on a workstation that file is shared: the distribution's lines, a local
// development tool's own marked blocks rewritten whenever its projects change, the operator's
// hand-added names. Written whole, all of those went at the next change to the mesh's names, with
// no failure anywhere: the tool believed it had written its block, and the mesh believed it owned
// the file. It is ADR 0102's failure exactly, in a file ADR 0102's JSON verb cannot speak.
//
// So the host finds the lines between `# BEGIN mesh <id>` and `# END mesh <id>`, rewrites those and
// nothing else, and records what they held before. Every line outside the markers is kept byte for
// byte — including another tool's `# BEGIN …` blocks, which are that tool's. Undeclared, the region
// is given back what it held, or taken out with its markers when it held nothing, and a file the
// mesh created goes only if nothing but whitespace is left.
// applyBlock writes a file's declared lines into its region of the file already at its path.
//
// **A link stays a link.** Where the path is a symbolic link — a hosts file some distributions keep
// elsewhere and link into /etc — the file read, written and renamed over is the one it points to,
// so the link and whatever manages it are left as they were. A file written whole, or into JSON,
// still replaces a link with a file; that is unchanged here.
func applyBlock(r *declaration.File, previous store.Applied) (Outcome, error) {
out := begin(r)
opening, closing := declaration.BlockMarkers(r.ID)
want := blockBody(r.Content)
real, err := realPath(r.Path)
if err != nil {
return out, err
}
raw, err := os.ReadFile(real)
existed := err == nil
if err != nil && !errors.Is(err, os.ErrNotExist) {
return out, err
}
// What the file is, taken once with what it holds: its mode and owner are the machine's and
// go back onto what is written. A file read and then not there to stat is a failure, never a
// file with no owner.
var info os.FileInfo
if existed {
if info, err = os.Stat(real); err != nil {
return out, fmt.Errorf("read %s and cannot see it: %w", r.Path, err)
}
}
existing := string(raw)
rec := store.Into{Format: declaration.IntoBlock}
var note string
rebuilt := false
// **A file the mesh once wrote whole** (novox/hq issue 128). The resource keeps its id when its
// module moves from writing the file whole to writing into it, and the file on the machine is
// then the mesh's own old write — its header, its loopback lines, its names. Adding the region
// after that would leave the old names above the new ones, and a resolver takes the first
// line that answers: the region would be shadowed by what it replaced. So the file is rebuilt:
// the original the mesh kept before its first write, with the region in it; or, where the mesh
// made the file itself, the loopback lines every machine needs, kept as the machine's, with the
// region beside them. Changed since the mesh wrote it, the file is somebody's again and is
// written into as it stands, and the outcome says so.
if existed && previous.Into == nil && previous.Wrote != "" {
if digestOf(existing) == previous.Wrote {
if previous.Kept != "" {
original, err := os.ReadFile(previous.Kept)
if err != nil {
return out, fmt.Errorf("%s was written whole by the mesh over an original kept at %s, "+
"which cannot be read to give it back: %w; it was left as it is", r.Path, previous.Kept, err)
}
existing = string(original)
note = "the mesh's old whole file replaced by the original kept at " + previous.Kept + ", with the region in it"
} else {
existing = loopbackOf(existing)
rec.Created = true
note = "the mesh's old whole file replaced by its loopback lines and the region"
}
// Not what was read: the whole of it was the mesh's, and the file is written afresh.
rebuilt = true
} else {
note = "a file the mesh once wrote whole, changed since; its old lines were kept"
}
}
lines := linesOf(existing)
at, found, err := regionIn(lines, opening, closing)
if err != nil {
// Refused, never guessed at: markers the host cannot pair are markers it cannot write
// between without risking lines that are not the mesh's.
return out, fmt.Errorf("%s: %w; it was left as it is", r.Path, err)
}
// A record of a block is carried; anything else — no record, a file once written whole, one
// once written into as JSON — is a file the host is seeing for the first time as a block.
recorded := previous.Into != nil && previous.Into.Format == declaration.IntoBlock
if recorded && existed {
rec.Created = previous.Into.Created
rec.Region = previous.Into.Region
rec.Separated = previous.Into.Separated
rec.At = previous.Into.At
rec.Ended = previous.Into.Ended
} else if !rebuilt {
// A file gone since the last apply is made again, and made by the mesh: what it held
// before went with it, so there is nothing to give back but the file's absence.
rec.Created = !existed
if found {
// **What the host may have written itself is not the machine's** — the same reasoning
// as a key in a JSON file (novox/hq ADR 0102). With no record, a region already holding
// exactly the declared lines cannot be told from one this host wrote a moment ago and
// died before saving; remembered as the machine's, it would be put back on undeclare
// for ever. So it is the mesh's, and undeclaring takes it out.
if held := at.body(lines); held != want {
rec.Region = &held
}
}
}
// Drift: the machine no longer holds, between the mesh's markers, what this host last put
// there. Judged only against a record of a block: a digest of a whole file says nothing about
// a region of it.
drifted := recorded && previous.Wrote != "" && existed &&
(!found || digestOf(at.body(lines)) != previous.Wrote)
var next string
switch {
case !existed:
next = regionOf(opening, closing, want)
case found:
// Where it is, whatever At says: the region is never moved, because moving it moves the
// machine's lines around it.
next = strings.Join(lines[:at.begin+1], "") + want + strings.Join(lines[at.end:], "")
case r.At == declaration.AtStart:
// Above everything, and one blank line between the region and the machine's first line
// unless there is one already — a line in some files means what the lines above it say.
rec.At, rec.Separated, rec.Ended = declaration.AtStart, false, false
next = regionOf(opening, closing, want)
if existing != "" && !strings.HasPrefix(existing, "\n") {
next += "\n"
rec.Separated = true
}
next += existing
default:
// At the end, apart from whatever is there: the file's last line is ended if it was not,
// and one blank line separates the region from the machine's lines unless there is one.
rec.At, rec.Separated, rec.Ended = "", false, false
next = existing
if next != "" && !strings.HasSuffix(next, "\n") {
next += "\n"
rec.Ended = true
}
if next != "" && next != "\n" && !strings.HasSuffix(next, "\n\n") {
next += "\n"
rec.Separated = true
}
next += regionOf(opening, closing, want)
}
// What was not the mesh's is what it was. By construction — and checked, because a slip in
// splicing lines is exactly the fault this mode exists to prevent, and it must never be written.
if found {
after := linesOf(next)
if where, ok, err := regionIn(after, opening, closing); err != nil || !ok || outside(after, where) != outside(lines, at) {
return out, fmt.Errorf("%s: writing the region would change lines outside it; it was left as it is", r.Path)
}
}
same := existed && next == string(raw)
if !same {
mode := os.FileMode(0o644)
if info != nil {
mode = info.Mode().Perm() // the machine's file keeps the machine's mode
} else if mode, err = modeOf(r.Mode, mode); err != nil {
return out, err
}
if err := makeDirs(filepath.Dir(real), 0o755, r.Owner); err != nil {
return out, err
}
if err := writeAtomically(real, []byte(next), mode); err != nil {
return out, err
}
if info != nil {
// The write is a new file renamed over the old, so it belongs to whoever wrote it. The
// machine's file keeps the machine's owner, as it keeps its mode.
if err := keepOwner(real, info); err != nil {
return out, err
}
} else if err := own(real, r.Owner); err != nil {
return out, err
}
}
// Read back: the region holds what was declared. Only the region — another tool writing its
// own lines in the moment after the rename is not a failed write. What remains is the moment
// between reading the file and renaming over it: a line another tool writes there is lost, and
// found again at its next write. Nothing short of a lock every writer honours closes that, and
// the other writers of a hosts file honour none.
written, err := os.ReadFile(real)
if err != nil {
return out, fmt.Errorf("wrote into %s and cannot read it back: %w", r.Path, err)
}
back := linesOf(string(written))
if where, ok, err := regionIn(back, opening, closing); err != nil || !ok || where.body(back) != want {
return out, fmt.Errorf("%s does not hold the mesh's region after writing into it", r.Path)
}
out.into = &rec
out.wrote = digestOf(want)
switch {
case note != "" && !same:
out.Action = "updated"
out.Detail = note
case !existed:
out.Action = "created"
out.Detail = "written into; the file was not there"
case same:
out.Action = "unchanged"
case drifted:
out.Action = "corrected"
out.Detail = "the mesh's region had been changed on the machine; every line outside it was kept"
case !found:
out.Action = "updated"
where := "end"
if rec.At == declaration.AtStart {
where = "start"
}
out.Detail = "the mesh's region added at the " + where + "; every other line kept as it was"
default:
out.Action = "updated"
out.Detail = "the mesh's region rewritten; every line outside it kept as it was"
}
return out, nil
}
// removeBlock gives back what a file written into a block held before the mesh's region.
func removeBlock(a store.Applied) (string, string, error) {
real, err := realPath(a.Target)
if err != nil {
return "", "", err
}
raw, err := os.ReadFile(real)
if errors.Is(err, os.ErrNotExist) {
return "forgotten", "no longer there", nil
}
if err != nil {
return "", "", err
}
info, err := os.Stat(real)
if err != nil {
return "", "", fmt.Errorf("read %s and cannot see it: %w", a.Target, err)
}
opening, closing := declaration.BlockMarkers(a.ID)
lines := linesOf(string(raw))
at, found, err := regionIn(lines, opening, closing)
if err != nil {
return "kept", err.Error() + ", so nothing was taken out of it; remove the mesh's region by hand", nil
}
next, action, detail := string(raw), "forgotten", "the mesh's region was no longer in it"
switch {
case found && a.Into.Region != nil:
next = strings.Join(lines[:at.begin+1], "") + *a.Into.Region + strings.Join(lines[at.end:], "")
action, detail = "restored", "no longer declared; the region was given back what it held"
case found:
from, to := at.begin, at.end+1
// The blank line the host added beside the region, when a blank line still stands there.
// Whether it is the same one the host added cannot be known from the file; a blank line
// is the one line whose going changes nothing any program reads, so it is taken. A line
// that is not blank is never taken, whoever put it there.
if a.Into.Separated {
if a.Into.At == declaration.AtStart {
if to < len(lines) && lines[to] == "\n" {
to++
}
} else if from > 0 && lines[from-1] == "\n" {
from--
}
}
next = strings.Join(lines[:from], "") + strings.Join(lines[to:], "")
// And the line end the host gave the machine's last line, if that line is still last.
if a.Into.Ended && strings.Join(lines[to:], "") == "" {
next = strings.TrimSuffix(next, "\n")
}
action, detail = "restored", "no longer declared; the mesh's region was taken out and every other line kept"
}
if a.Into.Created && strings.TrimSpace(next) == "" && real == a.Target {
if err := os.Remove(real); err != nil {
return "", "", err
}
return "removed", "no longer declared; the mesh had created it and nothing else was in it", nil
}
if next == string(raw) {
return action, detail, nil
}
if err := writeAtomically(real, []byte(next), info.Mode().Perm()); err != nil {
return "", "", err
}
if err := keepOwner(real, info); err != nil {
return "", "", err
}
return action, detail, nil
}
// realPath is the file a path names, through any links; a path that is not there yet is itself.
// A link to nothing is refused: writing through it would replace the link with a file.
func realPath(path string) (string, error) {
real, err := filepath.EvalSymlinks(path)
if err == nil {
return real, nil
}
if _, lerr := os.Lstat(path); errors.Is(lerr, os.ErrNotExist) {
return path, nil
}
return "", fmt.Errorf("%s is a link the host cannot follow to a file: %w; it was left as it is", path, err)
}
// loopbackOf is the lines of a file that answer for the machine itself — localhost, its own name on
// 127.0.1.1, ::1 — and nothing else: what the mesh's old whole hosts file carried that the machine
// needs, without the mesh's header or its names.
func loopbackOf(text string) string {
var b strings.Builder
for _, line := range linesOf(text) {
fields := strings.Fields(line)
if len(fields) < 2 {
continue
}
if ip := net.ParseIP(fields[0]); ip != nil && ip.IsLoopback() {
b.WriteString(strings.TrimSuffix(line, "\n") + "\n")
}
}
return b.String()
}
// outside is every line of a file but the mesh's region, markers included, as one string.
func outside(lines []string, at region) string {
end := at.end + 1
if end > len(lines) {
end = len(lines)
}
return strings.Join(lines[:at.begin], "") + "\x00" + strings.Join(lines[end:], "")
}
// blockBody is the declared lines as they stand in the region: ending in exactly one line end, or
// nothing at all when there are no lines.
func blockBody(content string) string {
trimmed := strings.TrimRight(content, "\n")
if trimmed == "" {
return ""
}
return trimmed + "\n"
}
func regionOf(begin, end, body string) string {
return begin + "\n" + body + end + "\n"
}
// linesOf splits text into lines that keep their line ends, so joining them again gives back
// exactly the bytes that were read — a last line without one included.
func linesOf(text string) []string {
return strings.SplitAfter(text, "\n")
}
// region is where the mesh's markers stand, as indices into the lines of a file.
type region struct{ begin, end int }
// body is what stands between the markers.
func (r region) body(lines []string) string {
return strings.Join(lines[r.begin+1:r.end], "")
}
// regionIn finds the mesh's markers for one resource. A line is a marker only if it is exactly the
// marker, so another tool's block and another resource's region are never it. Markers that do not
// form one pair — a begin with no end, an end before its begin, either twice — are an error rather
// than a best guess, because a guess is how the host would rewrite lines that are not its own.
func regionIn(lines []string, begin, end string) (region, bool, error) {
at := region{begin: -1, end: -1}
for i, line := range lines {
switch strings.TrimSuffix(line, "\n") {
case begin:
if at.begin >= 0 {
return at, false, fmt.Errorf("%q is in it more than once", begin)
}
at.begin = i
case end:
if at.end >= 0 {
return at, false, fmt.Errorf("%q is in it more than once", end)
}
at.end = i
}
}
switch {
case at.begin < 0 && at.end < 0:
return at, false, nil
case at.begin < 0:
return at, false, fmt.Errorf("%q is in it with no %q before it", end, begin)
case at.end < 0:
return at, false, fmt.Errorf("%q is in it with no %q after it", begin, end)
case at.end < at.begin:
return at, false, fmt.Errorf("%q stands before %q", end, begin)
}
return at, true, nil
}
// keepOwner gives a file rewritten through a new one back to whoever owned what it replaced.
// Changed only where it differs, so a host that is not root can still write a file it owns.
func keepOwner(path string, was os.FileInfo) error {
uid, gid, ok := ownerOf(was)
if !ok {
return nil
}
now, err := os.Stat(path)
if err != nil {
return err
}
if u, g, ok := ownerOf(now); ok && u == uid && g == gid {
return nil
}
if err := os.Chown(path, uid, gid); err != nil {
return fmt.Errorf("cannot give %s back to its owner %d:%d: %w", path, uid, gid, err)
}
return nil
}
+618
View File
@@ -0,0 +1,618 @@
package apply
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq issue 128 and ADR 0102: a text file the mesh shares with software it did not
// install is written into a marked block, never over — every line outside the mesh's markers is
// the machine's and is kept byte for byte, and undeclaring gives the file back.
const namesID = "mesh-wireguard.fact-node-names"
func blockDecl(t *testing.T, path, content string, extra ...string) string {
t.Helper()
more := ""
for _, e := range extra {
more += "," + e
}
return fmt.Sprintf(`{"declaration":1,"resources":[
{"id":%q,"type":"file","path":%q,"into":"block","content":%q%s}
]}`, namesID, path, content, more)
}
func applyBlockDecl(t *testing.T, raw string, known store.State) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), parse(t, raw), known, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
return report, state
}
func undeclare(t *testing.T, known store.State) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), somethingElse(t), known, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
return report, state
}
func readText(t *testing.T, path string) string {
t.Helper()
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
return string(raw)
}
func marked(id, body string) string {
return "# BEGIN mesh " + id + "\n" + body + "# END mesh " + id + "\n"
}
// A workstation's hosts file, the way issue 128 found it: the distribution's lines, a development
// tool's own marked blocks, and the operator's hand-added names.
const workstationHosts = "127.0.0.1\tlocalhost\n" +
"127.0.1.1\tg14.localdomain g14\n" +
"\n" +
"# BEGIN devtool project-a\n" +
"127.0.0.1 a.test api.a.test\n" +
"# END devtool project-a\n" +
"# BEGIN devtool project-b\n" +
"127.0.0.1 b.test\n" +
"# END devtool project-b\n" +
"192.168.1.20 printer # the operator's\n"
const meshNames = "10.42.0.1 ace\n10.42.0.2 novox\n"
func TestABlockKeepsEveryLineOutsideItsMarkers(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
if err := os.WriteFile(path, []byte(workstationHosts), 0o640); err != nil {
t.Fatal(err)
}
report, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
want := workstationHosts + "\n" + marked(namesID, meshNames)
if got := readText(t, path); got != want {
t.Fatalf("the file after writing into it:\n%q\nwant\n%q", got, want)
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("adding the region was %q", got)
}
if info, _ := os.Stat(path); info.Mode().Perm() != 0o640 {
t.Errorf("the machine's file mode was changed to %o", info.Mode().Perm())
}
rec, _ := state.Find(namesID)
if rec.Into == nil || rec.Into.Format != "block" || rec.Into.Region != nil || rec.Into.Created {
t.Fatalf("recorded as %+v", rec.Into)
}
// Again, with nothing changed: nothing written.
report, state = applyBlockDecl(t, blockDecl(t, path, meshNames), state)
if got := report.Outcomes[0].Action; got != "unchanged" {
t.Errorf("a second apply was %q", got)
}
// The development tool rewrites its block, and the operator adds a line after the mesh's
// region; the mesh's names change. Only the region moves.
edited := strings.Replace(readText(t, path), "127.0.0.1 b.test\n", "127.0.0.1 b.test c.test\n", 1) +
"10.0.0.5 nas # added after\n"
_ = os.WriteFile(path, []byte(edited), 0o640)
changed := meshNames + "10.42.0.3 shanks\n"
report, state = applyBlockDecl(t, blockDecl(t, path, changed), state)
want = strings.Replace(edited, marked(namesID, meshNames), marked(namesID, changed), 1)
if got := readText(t, path); got != want {
t.Fatalf("rewriting the region moved something else:\n%q\nwant\n%q", got, want)
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("rewriting the region was %q", got)
}
// Undeclared: the region, its markers and the blank line the host put before it go; every
// other line is where it was.
report, _ = undeclare(t, state)
want = strings.Replace(edited, "\n"+marked(namesID, meshNames), "", 1)
if got := readText(t, path); got != want {
t.Fatalf("undeclaring left:\n%q\nwant\n%q", got, want)
}
if got := report.Outcomes[0].Action; got != "restored" {
t.Errorf("undeclaring was %q", got)
}
}
func TestUndeclaringABlockAddedAtTheEndGivesTheFileBackExactly(t *testing.T) {
for name, original := range map[string]string{
"ending in a line": "127.0.0.1 localhost\n",
"ending in a blank line": "127.0.0.1 localhost\n\n",
"empty": "",
} {
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(original), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
undeclare(t, state)
if got := readText(t, path); got != original {
t.Errorf("undeclaring left %q, the machine had %q", got, original)
}
})
}
}
func TestABlockAddedAtTheEndIsSetApartFromTheMachinesLines(t *testing.T) {
for name, c := range map[string]struct{ before, after string }{
"no line end": {"127.0.0.1 localhost", "127.0.0.1 localhost\n\n" + marked(namesID, meshNames)},
"a line end": {"127.0.0.1 localhost\n", "127.0.0.1 localhost\n\n" + marked(namesID, meshNames)},
"a blank line already": {"127.0.0.1 localhost\n\n", "127.0.0.1 localhost\n\n" + marked(namesID, meshNames)},
"empty": {"", marked(namesID, meshNames)},
"only a blank line": {"\n", "\n" + marked(namesID, meshNames)},
"content without an end": {"x\n\n", "x\n\n" + marked(namesID, meshNames)},
} {
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(c.before), 0o644)
applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if got := readText(t, path); got != c.after {
t.Errorf("got %q, want %q", got, c.after)
}
})
}
}
func TestTheRegionEndsInExactlyOneLineEnd(t *testing.T) {
for content, body := range map[string]string{
"10.42.0.1 ace": "10.42.0.1 ace\n",
"10.42.0.1 ace\n": "10.42.0.1 ace\n",
"10.42.0.1 ace\n\n\n": "10.42.0.1 ace\n",
"": "",
"\n\n": "",
} {
path := filepath.Join(t.TempDir(), "hosts")
_, state := applyBlockDecl(t, blockDecl(t, path, content), store.State{})
if got := readText(t, path); got != marked(namesID, body) {
t.Errorf("content %q was written as %q", content, got)
}
// And the same content again is not a change.
report, _ := applyBlockDecl(t, blockDecl(t, path, content), state)
if got := report.Outcomes[0].Action; got != "unchanged" {
t.Errorf("content %q applied twice was %q", content, got)
}
}
}
func TestARegionAlreadyThereIsRewrittenInPlaceAndGivenBack(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
before := "127.0.0.1 localhost\n"
after := "# BEGIN devtool x\n127.0.0.1 x.test\n# END devtool x\n192.168.1.20 printer\n"
found := "10.42.0.9 old-name\n"
original := before + marked(namesID, found) + after
_ = os.WriteFile(path, []byte(original), 0o644)
report, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if got := readText(t, path); got != before+marked(namesID, meshNames)+after {
t.Fatalf("the region was not rewritten in place: %q", got)
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("rewriting a found region was %q", got)
}
rec, _ := state.Find(namesID)
if rec.Into.Region == nil || *rec.Into.Region != found {
t.Fatalf("what the region held before was recorded as %v", rec.Into.Region)
}
// Undeclared: what the region held goes back, where it was.
report, _ = undeclare(t, state)
if got := readText(t, path); got != original {
t.Errorf("undeclaring left %q, the machine had %q", got, original)
}
if got := report.Outcomes[0].Action; got != "restored" {
t.Errorf("undeclaring was %q", got)
}
}
func TestARegionWithNoRecordHoldingExactlyTheDeclaredLinesIsTheMeshs(t *testing.T) {
// A host that wrote the region and died before saving its state: what is between the markers
// is exactly what the mesh declares, and remembered as the machine's it would never go.
path := filepath.Join(t.TempDir(), "hosts")
original := "127.0.0.1 localhost\n"
_ = os.WriteFile(path, []byte(original+"\n"+marked(namesID, meshNames)), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if rec, _ := state.Find(namesID); rec.Into.Region != nil {
t.Fatalf("the mesh's own lines were recorded as the machine's: %q", *rec.Into.Region)
}
undeclare(t, state)
if got := readText(t, path); !strings.HasPrefix(got, original) || strings.Contains(got, "BEGIN mesh") {
t.Errorf("undeclaring left the mesh's region behind: %q", got)
}
}
func TestADriftedRegionIsCorrected(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
written := readText(t, path)
_ = os.WriteFile(path, []byte(strings.Replace(written, "10.42.0.2 novox\n", "10.42.0.2 novox\n6.6.6.6 evil\n", 1)), 0o644)
report, _ := applyBlockDecl(t, blockDecl(t, path, meshNames), state)
if got := report.Outcomes[0].Action; got != "corrected" {
t.Errorf("a region edited on the machine was %q", got)
}
if got := readText(t, path); got != written {
t.Errorf("the region was not put back: %q", got)
}
// The region taken out by hand is drift too, and it is put back.
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
report, _ = applyBlockDecl(t, blockDecl(t, path, meshNames), state)
if got := report.Outcomes[0].Action; got != "corrected" {
t.Errorf("a region removed on the machine was %q", got)
}
if got := readText(t, path); got != written {
t.Errorf("the region was not put back: %q", got)
}
}
func TestTwoRegionsInOneFileAreEachTheirOwn(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
raw := fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"a.names","type":"file","path":%q,"into":"block","content":"10.42.0.1 ace\n"},
{"id":"b.names","type":"file","path":%q,"into":"block","content":"10.43.0.1 lab\n"}
]}`, path, path)
_, state := applyBlockDecl(t, raw, store.State{})
want := workstationHosts + "\n" + marked("a.names", "10.42.0.1 ace\n") + "\n" + marked("b.names", "10.43.0.1 lab\n")
if got := readText(t, path); got != want {
t.Fatalf("two regions:\n%q\nwant\n%q", got, want)
}
report, state := applyBlockDecl(t, raw, state)
for _, o := range report.Outcomes {
if o.Action != "unchanged" {
t.Errorf("%s applied twice was %q", o.ID, o.Action)
}
}
// One undeclared: only its region goes.
only := fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"b.names","type":"file","path":%q,"into":"block","content":"10.43.0.1 lab\n"}
]}`, path)
applyBlockDecl(t, only, state)
want = workstationHosts + "\n" + marked("b.names", "10.43.0.1 lab\n")
if got := readText(t, path); got != want {
t.Errorf("undeclaring one region:\n%q\nwant\n%q", got, want)
}
}
func TestABlockInAFileThatWasNotThereIsCreatedAndRemovedWithIt(t *testing.T) {
path := filepath.Join(t.TempDir(), "conf.d", "mesh.conf")
report, state := applyBlockDecl(t, blockDecl(t, path, meshNames, `"mode":"0600"`), store.State{})
if got := readText(t, path); got != marked(namesID, meshNames) {
t.Fatalf("a created file holds %q", got)
}
if info, _ := os.Stat(path); info.Mode().Perm() != 0o600 {
t.Errorf("a created file is mode %o, declared 0600", info.Mode().Perm())
}
if got := report.Outcomes[0].Action; got != "created" {
t.Errorf("creating was %q", got)
}
if rec, _ := state.Find(namesID); !rec.Into.Created {
t.Error("the mesh creating the file was not recorded")
}
report, _ = undeclare(t, state)
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Errorf("a file the mesh created, holding only its region, was left behind")
}
if got := report.Outcomes[0].Action; got != "removed" {
t.Errorf("undeclaring was %q", got)
}
// Somebody else wrote into it meanwhile: it is no longer only the mesh's, and it stays.
_, state = applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
_ = os.WriteFile(path, []byte(readText(t, path)+"their = line\n"), 0o600)
undeclare(t, state)
if got := readText(t, path); got != "their = line\n" {
t.Errorf("undeclaring a created file somebody wrote into left %q", got)
}
}
func TestMarkersThatDoNotPairAreRefusedAndLeftAlone(t *testing.T) {
for name, text := range map[string]string{
"a begin with no end": "a\n# BEGIN mesh " + namesID + "\nb\n",
"an end with no begin": "a\n# END mesh " + namesID + "\n",
"an end before a begin": "# END mesh " + namesID + "\n# BEGIN mesh " + namesID + "\n",
"a begin twice": marked(namesID, "x\n") + "# BEGIN mesh " + namesID + "\n",
} {
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(text), 0o644)
if _, _, err := Apply(context.Background(), archHost(t), parse(t, blockDecl(t, path, meshNames)),
store.State{}, store.OriginDeclared, nil, nil, nil); err == nil {
t.Fatal("markers that do not pair were written between")
}
if got := readText(t, path); got != text {
t.Errorf("the file was changed: %q", got)
}
})
}
}
func TestAMarkerOfAnotherIDIsNotThisRegion(t *testing.T) {
// The id is part of the marker: a region whose id merely starts with this one is not it.
path := filepath.Join(t.TempDir(), "hosts")
other := marked(namesID+"-extra", "10.9.9.9 other\n")
_ = os.WriteFile(path, []byte(other), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if got := readText(t, path); got != other+"\n"+marked(namesID, meshNames) {
t.Fatalf("got %q", got)
}
undeclare(t, state)
if got := readText(t, path); got != other {
t.Errorf("undeclaring touched the other region: %q", got)
}
}
func TestABlockAtTheStartStandsAboveEverything(t *testing.T) {
// dhcpcd scopes every line after `interface X` to that interface, so the mesh's global options
// go above all of it.
dhcpcd := "hostname\nduid\n\ninterface enp6s0\nstatic ip_address=192.168.1.5/24\n"
opts := "nohook resolv.conf\ndenyinterfaces mesh0\n"
for name, c := range map[string]struct{ before, after string }{
"a file with content": {dhcpcd, marked(namesID, opts) + "\n" + dhcpcd},
"an empty file": {"", marked(namesID, opts)},
"a file opening blank": {"\n" + dhcpcd, marked(namesID, opts) + "\n" + dhcpcd},
"a last line with no end": {"interface enp6s0", marked(namesID, opts) + "\ninterface enp6s0"},
} {
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "dhcpcd.conf")
_ = os.WriteFile(path, []byte(c.before), 0o644)
report, state := applyBlockDecl(t, blockDecl(t, path, opts, `"at":"start"`), store.State{})
if got := readText(t, path); got != c.after {
t.Fatalf("got %q, want %q", got, c.after)
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("adding the region was %q", got)
}
report, state = applyBlockDecl(t, blockDecl(t, path, opts, `"at":"start"`), state)
if got := report.Outcomes[0].Action; got != "unchanged" {
t.Errorf("a second apply was %q", got)
}
undeclare(t, state)
if got := readText(t, path); got != c.before {
t.Errorf("undeclaring left %q, the machine had %q", got, c.before)
}
})
}
t.Run("a file that was not there", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "dhcpcd.conf")
applyBlockDecl(t, blockDecl(t, path, opts, `"at":"start"`), store.State{})
if got := readText(t, path); got != marked(namesID, opts) {
t.Errorf("got %q", got)
}
})
}
func TestARegionAlreadyThereIsNotMovedWhereverAtSaysItGoes(t *testing.T) {
for _, at := range []string{`"at":"start"`, `"at":"end"`} {
path := filepath.Join(t.TempDir(), "dhcpcd.conf")
original := "hostname\n" + marked(namesID, "old\n") + "interface enp6s0\n"
_ = os.WriteFile(path, []byte(original), 0o644)
applyBlockDecl(t, blockDecl(t, path, "nohook resolv.conf\n", at), store.State{})
want := "hostname\n" + marked(namesID, "nohook resolv.conf\n") + "interface enp6s0\n"
if got := readText(t, path); got != want {
t.Errorf("%s: a found region was moved: %q", at, got)
}
}
}
func TestAFileWrittenIntoABlockIsNeverHeldOnAnAdoptedNode(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
resource := fmt.Sprintf(`{"id":%q,"type":"file","path":%q,"into":"block","content":%q}`, namesID, path, meshNames)
d := adopted(t, `{"taken":[],"untaken":{"mesh-wireguard":["`+namesID+`"]}}`, resource)
report, state := applyAdopted(t, d, store.State{}, &machine{}, t.TempDir())
if got := outcomeOf(report, namesID).Action; got == "held" {
t.Fatal("a file written into a block was held, though it replaces nothing that was found")
}
if len(state.Held) != 0 {
t.Errorf("something was held: %+v", state.Held)
}
if got := readText(t, path); got != workstationHosts+"\n"+marked(namesID, meshNames) {
t.Errorf("the adopted node's file was not written into: %q", got)
}
// Held from when it was declared whole, the hold does not keep the region out.
path2 := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path2, []byte(workstationHosts), 0o644)
known := store.State{Held: []store.Held{{ID: namesID, Module: "mesh-wireguard", Kind: "file", Target: path2}}}
resource2 := fmt.Sprintf(`{"id":%q,"type":"file","path":%q,"into":"block","content":%q}`, namesID, path2, meshNames)
d2 := adopted(t, `{"taken":[],"untaken":{"mesh-wireguard":["`+namesID+`"]}}`, resource2)
report, state = applyAdopted(t, d2, known, &machine{}, t.TempDir())
if got := outcomeOf(report, namesID).Action; got != "updated" {
t.Errorf("the file was %q, not written into", got)
}
if len(state.Held) != 0 {
t.Errorf("the old hold outlived the block declaration: %+v", state.Held)
}
// And the preview says the same: written into, not held.
for _, s := range Plan(d2, known, store.OriginDeclared) {
if s.ID == namesID && s.Verb == "hold" {
t.Errorf("the preview holds a file written into a block: %+v", s)
}
}
}
func TestTheRecordOfABlockSurvivesTheStateFile(t *testing.T) {
// What undeclaring needs is in the state a host saves, not only in memory.
path := filepath.Join(t.TempDir(), "hosts")
original := "a\n" + marked(namesID, "old\n")
_ = os.WriteFile(path, []byte(original), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames, `"at":"start"`), store.State{})
raw, err := json.Marshal(state)
if err != nil {
t.Fatal(err)
}
var back store.State
if err := json.Unmarshal(raw, &back); err != nil {
t.Fatal(err)
}
undeclare(t, back)
if got := readText(t, path); got != original {
t.Errorf("undeclaring from a saved state left %q", got)
}
}
// The mesh's old whole hosts file, as the controller composed it before issue 128.
const oldWholeHosts = "# Generated by the mesh. Do not edit — this file is replaced whenever a machine\n" +
"# joins or leaves, and an edit would survive until then and vanish.\n\n" +
"127.0.0.1\tlocalhost\n" +
"::1\t\tlocalhost ip6-localhost ip6-loopback\n" +
"127.0.1.1\tg14\n" +
"\n" +
"10.42.0.1\tace.internal\tace\n" +
"10.42.0.9\tg14.internal\tg14\t# this machine\n"
func wholeDecl(path, content string) string {
return fmt.Sprintf(`{"declaration":1,"resources":[
{"id":%q,"type":"file","path":%q,"content":%q}
]}`, namesID, path, content)
}
func applyKeepingIn(t *testing.T, raw string, known store.State, keepDir string) (Report, store.State) {
t.Helper()
report, state, err := ApplyKeeping(context.Background(), archHost(t), parse(t, raw), known,
store.OriginDeclared, (&machine{}).run, nil, nil, KeepIn(keepDir))
if err != nil {
t.Fatalf("apply failed: %v", err)
}
return report, state
}
func TestAFileTheMeshWroteWholeAndMadeItselfKeepsOnlyItsLoopbackLines(t *testing.T) {
// Written whole into a file that was not there, then declared as a block under the same id:
// the old names must not stay above the region, where a resolver would answer from them first.
path := filepath.Join(t.TempDir(), "hosts")
_, state := applyKeepingIn(t, wholeDecl(path, oldWholeHosts), store.State{}, t.TempDir())
report, state := applyKeepingIn(t, blockDecl(t, path, meshNames), state, t.TempDir())
floor := "127.0.0.1\tlocalhost\n::1\t\tlocalhost ip6-localhost ip6-loopback\n127.0.1.1\tg14\n"
if got := readText(t, path); got != floor+"\n"+marked(namesID, meshNames) {
t.Fatalf("the old whole file became:\n%q", got)
}
o := outcomeOf(report, namesID)
if o.Action != "updated" || !strings.Contains(o.Detail, "loopback lines") {
t.Errorf("the rebuild was reported as %q: %s", o.Action, o.Detail)
}
if rec, _ := state.Find(namesID); !rec.Into.Created {
t.Error("a file the mesh made itself was not recorded as the mesh's")
}
report, _ = applyKeepingIn(t, blockDecl(t, path, meshNames), state, t.TempDir())
if got := outcomeOf(report, namesID).Action; got != "unchanged" {
t.Errorf("applied again, the rebuilt file was %q", got)
}
undeclare(t, state)
if got := readText(t, path); got != floor {
t.Errorf("undeclared, the file holds %q", got)
}
}
func TestAFileTheMeshWroteWholeOverAnOriginalGetsTheOriginalBack(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
keep := t.TempDir()
_, state := applyKeepingIn(t, wholeDecl(path, oldWholeHosts), store.State{}, keep)
if rec, _ := state.Find(namesID); rec.Kept == "" {
t.Fatal("where the original was kept was not recorded")
}
report, state := applyKeepingIn(t, blockDecl(t, path, meshNames), state, keep)
if got := readText(t, path); got != workstationHosts+"\n"+marked(namesID, meshNames) {
t.Fatalf("the old whole file became:\n%q", got)
}
if d := outcomeOf(report, namesID).Detail; !strings.Contains(d, "original kept at") {
t.Errorf("the rebuild was reported as: %s", d)
}
undeclare(t, state)
if got := readText(t, path); got != workstationHosts {
t.Errorf("undeclared, the machine did not get its original back: %q", got)
}
}
func TestAFileTheMeshWroteWholeAndSomebodyChangedIsWrittenIntoAsItStands(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_, state := applyKeepingIn(t, wholeDecl(path, oldWholeHosts), store.State{}, t.TempDir())
edited := oldWholeHosts + "192.168.1.20 printer\n"
_ = os.WriteFile(path, []byte(edited), 0o644)
report, _ := applyKeepingIn(t, blockDecl(t, path, meshNames), state, t.TempDir())
if got := readText(t, path); got != edited+"\n"+marked(namesID, meshNames) {
t.Fatalf("an edited whole file became:\n%q", got)
}
if d := outcomeOf(report, namesID).Detail; !strings.Contains(d, "changed since; its old lines were kept") {
t.Errorf("the outcome does not say so: %s", d)
}
}
func TestALinkedFileStaysALink(t *testing.T) {
dir := t.TempDir()
real := filepath.Join(dir, "static", "hosts")
_ = os.MkdirAll(filepath.Dir(real), 0o755)
_ = os.WriteFile(real, []byte(workstationHosts), 0o644)
link := filepath.Join(dir, "hosts")
if err := os.Symlink(real, link); err != nil {
t.Fatal(err)
}
_, state := applyBlockDecl(t, blockDecl(t, link, meshNames), store.State{})
if info, err := os.Lstat(link); err != nil || info.Mode()&os.ModeSymlink == 0 {
t.Fatalf("the link was replaced by a file")
}
if got := readText(t, real); got != workstationHosts+"\n"+marked(namesID, meshNames) {
t.Errorf("the file the link names holds %q", got)
}
undeclare(t, state)
if info, err := os.Lstat(link); err != nil || info.Mode()&os.ModeSymlink == 0 {
t.Fatalf("undeclaring replaced the link with a file")
}
if got := readText(t, real); got != workstationHosts {
t.Errorf("undeclared, the file the link names holds %q", got)
}
}
func TestALastLineWithNoEndIsGivenBackWithNone(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte("x"), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if got := readText(t, path); got != "x\n\n"+marked(namesID, meshNames) {
t.Fatalf("got %q", got)
}
undeclare(t, state)
if got := readText(t, path); got != "x" {
t.Errorf("undeclaring left %q, the machine had %q", got, "x")
}
}
func TestAFailedBlockWriteKeepsItsHold(t *testing.T) {
// Held from when it was declared whole, then declared as a block into a file whose markers do
// not pair: the write is refused, and the hold — with where its original is — stays.
path := filepath.Join(t.TempDir(), "hosts")
broken := "a\n# BEGIN mesh " + namesID + "\n"
_ = os.WriteFile(path, []byte(broken), 0o644)
known := store.State{Held: []store.Held{{ID: namesID, Module: "mesh-wireguard", Kind: "file",
Target: path, Kept: "/var/lib/mesh/kept/hosts"}}}
resource := fmt.Sprintf(`{"id":%q,"type":"file","path":%q,"into":"block","content":%q}`, namesID, path, meshNames)
d := adopted(t, `{"taken":[],"untaken":{"mesh-wireguard":["`+namesID+`"]}}`, resource)
_, state, err := ApplyKeeping(context.Background(), archHost(t), d, known,
store.OriginDeclared, (&machine{}).run, nil, nil, KeepIn(t.TempDir()))
if err == nil {
t.Fatal("a write into unpaired markers was not refused")
}
h, held := state.HeldAt(namesID)
if !held || h.Kept != "/var/lib/mesh/kept/hosts" {
t.Errorf("a failed write released the hold: %+v", state.Held)
}
}
+97
View File
@@ -0,0 +1,97 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A take is a comparison (novox/hq ADR 0163): while a module's container is held, the host reports
// the found image and its age beside the declared one, the networks and who else is on them, the
// mounts and the ports — and says when the declared image is the older.
func TestAHeldContainerCarriesTheFactsATakeCompares(t *testing.T) {
dir, page, m := predecessor(t)
m.containers["hello-web"].image = "web:1.27"
m.containers["hello-web"].imageID = "sha256:found"
m.containers["hello-web"].networks = []string{"predecessor_default"}
m.containers["hello-web"].mounts = []string{"/srv/web:/data"}
m.containers["hello-web"].ports = []string{"80/tcp>0.0.0.0:8080"}
m.images = map[string]string{"sha256:found": "2026-09-17T10:00:00Z", pinned: "2026-08-20T10:00:00Z"}
m.members = map[string][]string{"predecessor_default": {"hello-web", "office", "db"}}
_, state := applyAdopted(t, adopted(t, untakenWeb, webResources(page)), store.State{}, m, dir)
h, ok := state.HeldAt("hello-web.server")
if !ok || h.Facts == nil {
t.Fatalf("a held container carries no facts: %+v", h)
}
f := h.Facts
if f.Image != "web:1.27" || f.ImageCreated != "2026-09-17T10:00:00Z" {
t.Errorf("the found image and its age: %+v", f)
}
if f.DeclaredImage != pinned || f.DeclaredImageCreated != "2026-08-20T10:00:00Z" || !f.Downgrade {
t.Errorf("the declared image, its age, and that it is a downgrade: %+v", f)
}
if got := f.Networks["predecessor_default"]; len(got) != 2 || got[0] != "db" || got[1] != "office" {
t.Errorf("the neighbours on the found network, without the container itself: %v", f.Networks)
}
if len(f.Mounts) != 1 || f.Mounts[0] != "/srv/web:/data" || len(f.Ports) != 1 || f.Ports[0] != "80/tcp>0.0.0.0:8080" {
t.Errorf("mounts and ports as found: %+v", f)
}
// And the held file carries how the declared content differs from what was found.
p, ok := state.HeldAt("hello-web.page")
if !ok || p.Facts == nil || !p.Facts.Differs {
t.Fatalf("a held file that differs from the declared content does not say so: %+v", p)
}
joined := strings.Join(p.Facts.Difference, "\n")
if !strings.Contains(joined, "- the predecessor's page") || !strings.Contains(joined, "+ the mesh's page") {
t.Errorf("the difference does not show what is lost and what is new: %q", joined)
}
_ = os.Remove(filepath.Join(dir, "unused"))
}
// A declared image not yet on the machine leaves its age unknown and the comparison undecided.
func TestAnImageNotYetPulledLeavesTheDowngradeUndecided(t *testing.T) {
dir, page, m := predecessor(t)
m.containers["hello-web"].image = "web:1.27"
m.containers["hello-web"].imageID = "sha256:found"
m.images = map[string]string{"sha256:found": "2026-09-17T10:00:00Z"}
_, state := applyAdopted(t, adopted(t, untakenWeb, webResources(page)), store.State{}, m, dir)
h, _ := state.HeldAt("hello-web.server")
if h.Facts == nil || h.Facts.DeclaredImageCreated != "" || h.Facts.Downgrade {
t.Fatalf("an unknown declared age decided a downgrade: %+v", h.Facts)
}
}
func TestTheDifferenceIsWhatIsLostAndWhatIsNew(t *testing.T) {
differs, lines := differenceOf("a\nprivate scope: local\nb\n", "a\nb\nupstream: public\n")
if !differs || len(lines) != 2 || lines[0] != "- private scope: local" || lines[1] != "+ upstream: public" {
t.Fatalf("got %v %v", differs, lines)
}
if differs, lines := differenceOf("same\n", "same\n"); differs || lines != nil {
t.Fatalf("identical content differs: %v %v", differs, lines)
}
}
// What runs on the machine that the mesh neither wrote nor holds is reported (ADR 0163).
func TestStraysAreWhatRunsHereThatNobodyAsked(t *testing.T) {
m := &machine{containers: map[string]*fakeContainer{
"hello-web": {id: "ours", running: true, image: "web:1"},
"gitea-old": {id: "left-behind", running: true, image: "gitea:1.22"},
"held-thing": {id: "found", running: true, image: "x:1"},
}}
known := store.State{
Resources: []store.Applied{{ID: "hello-web.server", Type: "container", Target: "hello-web"}},
Held: []store.Held{{ID: "other.server", Kind: "container", Target: "held-thing"}},
}
strays, err := Strays(context.Background(), m.run, known)
if err != nil {
t.Fatal(err)
}
if len(strays) != 1 || strays[0].Name != "gitea-old" || !strings.Contains(strays[0].Detail, "gitea:1.22") {
t.Fatalf("strays: %+v", strays)
}
}
+180
View File
@@ -0,0 +1,180 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// novox/hq ADR 0223: the machine's resolver file moves from the module that wrote it to the uplink's
// holder, and the two records meet in one apply — the old undeclared, the new declared at the same
// path. Every orphan is removed before anything is applied, so without a handover the file was
// deleted (or given back the original the mesh once wrote over) and every resource applied before the
// new one ran on a machine without it. Handed over, the file is never absent, the record going is
// forgotten only once the new one is recorded, and what was kept of the machine's original goes with
// the file to its new owner.
const (
theOriginal = "nameserver 192.0.2.53\n"
firstOwners = "# the mesh, as the first module wrote it\nnameserver 10.42.0.1\n"
newOwners = "# the mesh, as the uplink writes it\nnameserver 10.42.0.1\nnameserver 10.42.0.3\n"
)
// heldBy is a declaration whose file at path is the module's, with a service declared ahead of it
// so the apply does something on the machine before the file's turn comes.
func heldBy(t *testing.T, path, module, content string) string {
t.Helper()
return fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"%[1]s.service","type":"service","unit":"%[1]s.service","state":"running"},
{"id":"%[1]s.fact-resolvers","type":"file","path":%[2]q,"mode":"0644","content":%[3]q}
]}`, module, path, content)
}
// watching is a service manager that answers every unit running, and fails the test whenever the
// file is not there while it is asked something — the moment between an orphan's removal and the
// new owner's turn.
func watching(t *testing.T, path string) Runner {
t.Helper()
var commands []string
services := recordingServices(&commands)
return func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if _, err := os.Stat(path); err != nil {
t.Errorf("the file was not there while %s %s ran: %v", name, strings.Join(args, " "), err)
}
return services(ctx, name, args...)
}
}
func TestAWholeFileIsHandedToItsNewOwnerNotRemoved(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "resolv.conf")
if err := os.WriteFile(path, []byte(theOriginal), 0o644); err != nil {
t.Fatal(err)
}
keep := KeepIn(filepath.Join(dir, "kept"))
apply := func(raw string, known store.State) (Report, store.State) {
t.Helper()
report, state, err := ApplyKeeping(context.Background(), archHost(t), parse(t, raw), known,
store.OriginDeclared, watching(t, path), nil, nil, keep)
if err != nil {
t.Fatalf("apply failed: %v", err)
}
return report, state
}
// The first owner writes over the machine's file, keeping the original.
_, state := apply(heldBy(t, path, "resolv-conf", firstOwners), store.State{})
first, _ := state.Find("resolv-conf.fact-resolvers")
if first.Kept == "" {
t.Fatal("the machine's original was not kept before the first write")
}
// The uplink takes it over in one apply.
report, state := apply(heldBy(t, path, "networkmanager", newOwners), state)
if got, _ := os.ReadFile(path); string(got) != newOwners {
t.Fatalf("after the handover the file holds %q", got)
}
for _, o := range report.Outcomes {
if o.ID == "resolv-conf.fact-resolvers" && o.Action != "forgotten" {
t.Errorf("the record going was %q, not forgotten: %s", o.Action, o.Detail)
}
}
if _, still := state.Find("resolv-conf.fact-resolvers"); still {
t.Error("the record going is still recorded after its file was handed over")
}
now, _ := state.Find("networkmanager.fact-resolvers")
if now.Kept != first.Kept {
t.Errorf("the new owner keeps the original at %q; the first kept it at %q", now.Kept, first.Kept)
}
if kept, _ := os.ReadFile(now.Kept); string(kept) != theOriginal {
t.Errorf("what the new owner would give back is %q, not the machine's original", kept)
}
if aside, _ := filepath.Glob(path + ".removed-*"); len(aside) > 0 {
t.Errorf("the file was moved aside on the way: %v", aside)
}
// Steady from here, and undeclared the machine's original comes back — not the first owner's.
report, state = apply(heldBy(t, path, "networkmanager", newOwners), state)
for _, o := range report.Outcomes {
if o.ID == "networkmanager.fact-resolvers" && o.Action != "unchanged" {
t.Errorf("a second apply was %q: %s", o.Action, o.Detail)
}
}
if _, _, err := ApplyKeeping(context.Background(), archHost(t), somethingElse(t), state,
store.OriginDeclared, recordingServices(new([]string)), nil, nil, keep); err != nil {
t.Fatal(err)
}
if got, _ := os.ReadFile(path); string(got) != theOriginal {
t.Errorf("undeclaring the new owner left %q; the machine's original goes back", got)
}
}
// A file the mesh made where there was none is handed over the same way, and is still the mesh's —
// undeclared, it goes.
func TestAFileTheMeshMadeIsHandedOverAndStillGoesWithItsLastOwner(t *testing.T) {
path := filepath.Join(t.TempDir(), "resolv.conf")
run := watching(t, path)
_, state, err := Apply(context.Background(), archHost(t), parse(t, heldBy(t, path, "resolv-conf", firstOwners)),
store.State{}, store.OriginDeclared, recordingServices(new([]string)), nil, nil)
if err != nil {
t.Fatal(err)
}
_, state, err = Apply(context.Background(), archHost(t), parse(t, heldBy(t, path, "dhcpcd", newOwners)),
state, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if got, _ := os.ReadFile(path); string(got) != newOwners {
t.Fatalf("after the handover the file holds %q", got)
}
if _, _, err := Apply(context.Background(), archHost(t), somethingElse(t), state, store.OriginDeclared,
recordingServices(new([]string)), nil, nil); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Errorf("the mesh's own file stayed after its last owner was undeclared: %v", err)
}
}
// A file edited on the machine since the first owner wrote it is still corrected and said so by the
// new owner: the handover judges drift against what this host last wrote there.
func TestAHandedOverFileChangedOnTheMachineIsSaidCorrected(t *testing.T) {
path := filepath.Join(t.TempDir(), "resolv.conf")
_, state, err := Apply(context.Background(), archHost(t), parse(t, heldBy(t, path, "resolv-conf", firstOwners)),
store.State{}, store.OriginDeclared, recordingServices(new([]string)), nil, nil)
if err != nil {
t.Fatal(err)
}
_ = os.WriteFile(path, []byte("nameserver 1.1.1.1\n"), 0o644)
report, _, err := Apply(context.Background(), archHost(t), parse(t, heldBy(t, path, "systemd-networkd", newOwners)),
state, store.OriginDeclared, recordingServices(new([]string)), nil, nil)
if err != nil {
t.Fatal(err)
}
for _, o := range report.Outcomes {
if o.ID == "systemd-networkd.fact-resolvers" && o.Action != "corrected" {
t.Errorf("a handed-over file changed on the machine was %q: %s", o.Action, o.Detail)
}
}
}
// The preview says the same before it happens.
func TestThePlanHandsAFileOverRatherThanRemovingIt(t *testing.T) {
path := filepath.Join(t.TempDir(), "resolv.conf")
_, state, err := Apply(context.Background(), archHost(t), parse(t, heldBy(t, path, "resolv-conf", firstOwners)),
store.State{}, store.OriginDeclared, recordingServices(new([]string)), nil, nil)
if err != nil {
t.Fatal(err)
}
for _, step := range Plan(parse(t, heldBy(t, path, "networkmanager", newOwners)), state, store.OriginDeclared) {
if step.ID == "resolv-conf.fact-resolvers" && (step.Verb != "forget" || !strings.Contains(step.Why, "networkmanager.fact-resolvers")) {
t.Errorf("the plan would %s the file being handed over: %s", step.Verb, step.Why)
}
}
}
+69
View File
@@ -0,0 +1,69 @@
package apply
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// The host's own former archive stops nothing (novox/hq issue 194). A new host's first apply finds
// the version before it as a former target of the archive that delivered it; an archive has no
// removal then (issue 162), and the refusal stopped every machine applying anything. A former
// target of an archive is left in place, said, and forgotten: the version before is what a rollback
// starts (ADR 0141).
func TestTheHostsOwnFormerArchiveIsLeftInPlaceNotFatal(t *testing.T) {
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "docker" && args[0] == "info" {
return "29.0.0\n", nil
}
return "", nil
}
dir := t.TempDir()
former := store.FormerID("mesh-host.next", dir+"/versions/old")
known := store.State{Resources: []store.Applied{
{ID: "mesh-host.next", Type: "archive", Target: dir + "/versions/new", Origin: store.OriginDeclared},
{ID: former, Type: "archive", Target: dir + "/versions/old", Origin: store.OriginDeclared},
}}
body, digest := anArchive(t, map[string]string{"nox-mesh-host": "#!/bin/sh\n"})
d := parse(t, `{"declaration":1,"resources":[
{"id":"mesh-host.next","type":"archive","path":"`+dir+`/versions/new","source":"`+serving(t, body)+`","digest":"`+digest+`"},
{"id":"notes.conf","type":"file","path":"`+dir+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatalf("the apply failed: %v", err)
}
if _, still := state.HeldAt(former); still {
t.Fatal("held?")
}
for _, r := range state.Resources {
if r.ID == former {
t.Fatal("the former archive is still on record")
}
}
said := false
for _, o := range report.Outcomes {
if o.ID == former && o.Action == "forgotten" && strings.Contains(o.Detail, "left in place") {
said = true
}
}
if !said {
t.Fatalf("leaving the former archive was not said: %+v", report.Outcomes)
}
applied := false
for _, o := range report.Outcomes {
if o.ID == "notes.conf" && o.Action == "created" {
applied = true
}
}
if !applied {
t.Fatalf("the rest of the declaration was not applied: %+v", report.Outcomes)
}
// An archive the declaration dropped is taken away since issue 162; one recorded before the
// host kept what it unpacked is left in place and forgotten, never fatal — that case is
// archive_removal_test.go's.
}
+257
View File
@@ -0,0 +1,257 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/store"
)
// A directory used as found is never quiet, and a person at the machine can hand it over (novox/hq issue 339).
func TestAFoundDirectoryIsSaidAndCanBeHandedOver(t *testing.T) {
dir := t.TempDir()
found := filepath.Join(dir, "found")
if err := os.Mkdir(found, 0o700); err != nil {
t.Fatal(err)
}
decl := `{"id":"notes.data","type":"directory","path":"` + found + `","mode":"0755"}`
var said []string
log := func(line string) { said = append(said, line) }
_, state, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, log, nil)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(said, "\n"), "WARNING notes.data") {
t.Fatalf("the apply did not warn of a directory used as found: %q", said)
}
if a, _ := state.Find("notes.data"); !a.AsFound || !strings.Contains(a.Declared, "755") {
t.Fatalf("the record does not say what was declared: %+v", a)
}
// Said again at the next apply: a warning once is a warning missed.
said = nil
if _, state, err = Apply(context.Background(), archHost(t), declare(t, decl), state, store.OriginDeclared,
noServices, log, nil); err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(said, "\n"), "WARNING notes.data") {
t.Fatalf("the next apply did not warn again: %q", said)
}
// Handed over: the next apply gives it the declared mode and it is the mesh's.
before := HandedOver
HandedOver = func() map[string]bool { return map[string]bool{handOverKey("notes.data", found): true} }
t.Cleanup(func() { HandedOver = before })
if _, state, err = Apply(context.Background(), archHost(t), declare(t, decl), state, store.OriginDeclared,
noServices, nil, nil); err != nil {
t.Fatal(err)
}
if info, _ := os.Stat(found); info.Mode().Perm() != 0o755 {
t.Fatalf("a directory handed over was not given its declared mode: %o", info.Mode().Perm())
}
if a, _ := state.Find("notes.data"); a.AsFound {
t.Fatal("a directory handed over is still recorded as found")
}
}
// The review's additions to the guard: the spool, /opt, the container runtimes' data wherever the runtime keeps
// it, and an account's .ssh as an access or a mount, or as a directory anywhere but its own home.
func TestTheRuntimesDataAndAnySSHAreRefused(t *testing.T) {
cfg := filepath.Join(t.TempDir(), "daemon.json")
if err := os.WriteFile(cfg, []byte(`{"data-root":"/storage/docker"}`), 0o644); err != nil {
t.Fatal(err)
}
before := runtimeConfigs
runtimeConfigs = runtimeFiles{daemonJSON: cfg}
t.Cleanup(func() { runtimeConfigs = before })
for _, kind := range []placing{placingDirectory, placingAccess, placingMount} {
for _, path := range []string{"/var/spool/cron", "/opt/app", "/var/lib/docker/volumes/x",
"/var/lib/containers/storage", "/storage/docker", "/storage/docker/overlay2"} {
if err := refusePath(path, kind, "notes", ""); err == nil {
t.Errorf("%v at %s was taken", kind, path)
}
}
}
for _, path := range []string{"/srv/backup/.ssh", "/srv/backup/.ssh/keys"} {
for _, kind := range []placing{placingDirectory, placingAccess, placingMount} {
if err := refusePath(path, kind, "notes", ""); err == nil {
t.Errorf("%v at %s was taken", kind, path)
}
}
}
if err := refusePath("/storage/dockerish", placingAccess, "notes", ""); err != nil {
t.Errorf("a path beside the data root: %v", err)
}
}
// When a directory was found is kept from the first apply that found it (novox/hq issue 339): the controller
// tells a directory a send found from one found before it by that time, so it must not move with every apply.
func TestWhenADirectoryWasFoundIsKept(t *testing.T) {
dir := t.TempDir()
found := filepath.Join(dir, "found")
if err := os.Mkdir(found, 0o700); err != nil {
t.Fatal(err)
}
decl := `{"id":"notes.data","type":"directory","path":"` + found + `","mode":"0755"}`
_, state, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
first, _ := state.Find("notes.data")
if first.FoundAt.IsZero() {
t.Fatal("no time the directory was found")
}
time.Sleep(10 * time.Millisecond)
_, state, err = Apply(context.Background(), archHost(t), declare(t, decl), state, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if again, _ := state.Find("notes.data"); !again.FoundAt.Equal(first.FoundAt) {
t.Fatalf("found at %v, then %v: the time moved with the apply", first.FoundAt, again.FoundAt)
}
}
// A hand-over is for the resource that was used as found there: another declaration that later points another
// resource at the same path is not handed anything (novox/hq issue 339).
func TestAHandOverIsForTheResourceThatAskedForIt(t *testing.T) {
dir := t.TempDir()
found := filepath.Join(dir, "found")
if err := os.Mkdir(found, 0o700); err != nil {
t.Fatal(err)
}
before := HandedOver
HandedOver = func() map[string]bool { return map[string]bool{handOverKey("notes.data", found): true} }
t.Cleanup(func() { HandedOver = before })
other := `{"id":"other.data","type":"directory","path":"` + found + `","mode":"0755"}`
if _, _, err := Apply(context.Background(), archHost(t), declare(t, other), store.State{}, store.OriginDeclared,
noServices, nil, nil); err != nil {
t.Fatal(err)
}
if info, _ := os.Stat(found); info.Mode().Perm() != 0o700 {
t.Fatalf("a hand-over for notes.data gave other.data the directory: mode %o", info.Mode().Perm())
}
}
// **A handed-over directory is changed through no link, at the moment of use** (review of issue 356). The
// hand-over refused a link when it was recorded; between then and the apply its parent is swapped for a link to
// a victim tree. The apply refuses, and the victim's directory keeps its mode — and so does a directory whose own
// name became a link, and one whose grandparent did.
func TestAHandedOverDirectoryIsChangedThroughNoLink(t *testing.T) {
for _, c := range []struct {
swap string
applyOnce bool
}{{"parent", false}, {"grandparent", false}, {"itself", false}, {"parent", true}, {"grandparent", true}} {
swap, applyOnce := c.swap, c.applyOnce
name := swap
if applyOnce {
name += " after an apply gave it its owner"
}
t.Run(name, func(t *testing.T) {
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
found := filepath.Join(dir, "top", "parent", "data")
if err := os.MkdirAll(found, 0o700); err != nil {
t.Fatal(err)
}
decl := `{"id":"notes.data","type":"directory","path":"` + found + `","mode":"0755"}`
_, state, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if a, _ := state.Find("notes.data"); !a.AsFound {
t.Fatalf("not used as found: %+v", a)
}
before := HandedOver
HandedOver = func() map[string]bool { return map[string]bool{handOverKey("notes.data", found): true} }
t.Cleanup(func() { HandedOver = before })
if applyOnce {
// The hand-over takes: the directory is given its mode, and the record is no longer as found.
if _, state, err = Apply(context.Background(), archHost(t), declare(t, decl), state, store.OriginDeclared,
noServices, nil, nil); err != nil {
t.Fatal(err)
}
if a, _ := state.Find("notes.data"); a.AsFound {
t.Fatalf("still as found after the hand-over took: %+v", a)
}
if info, _ := os.Stat(found); info.Mode().Perm() != 0o755 {
t.Fatalf("the hand-over did not take: %o", info.Mode().Perm())
}
}
// The victim: a tree of the same shape somewhere else, its data directory mode 0700.
victim := filepath.Join(dir, "victim", "top", "parent", "data")
if err := os.MkdirAll(victim, 0o700); err != nil {
t.Fatal(err)
}
var swapped, target string
switch swap {
case "parent":
swapped, target = filepath.Join(dir, "top", "parent"), filepath.Join(dir, "victim", "top", "parent")
case "grandparent":
swapped, target = filepath.Join(dir, "top"), filepath.Join(dir, "victim", "top")
case "itself":
swapped, target = found, victim
}
if err := os.Rename(swapped, swapped+".real"); err != nil {
t.Fatal(err)
}
if err := os.Symlink(target, swapped); err != nil {
t.Fatal(err)
}
_, _, err = Apply(context.Background(), archHost(t), declare(t, decl), state, store.OriginDeclared,
noServices, nil, nil)
if err == nil {
t.Fatal("a handed-over directory reached through a link was applied")
}
// Refused by the path's re-check before any descriptor is opened (a link at the name itself is refused
// earlier still, by the directory applier's own look at it).
want := "reached through a link"
if swap == "itself" {
want = "symbolic link"
}
if !strings.Contains(err.Error(), want) {
t.Fatalf("refused as %v; want %q", err, want)
}
if info, _ := os.Stat(victim); info.Mode().Perm() != 0o700 {
t.Fatalf("the victim's directory was given the declared mode through the link: %o", info.Mode().Perm())
}
})
}
}
// The check at the moment of use, without the path's re-check: a component swapped for a link is refused by the
// descriptor walk itself, whatever was resolved a moment before.
func TestGiveHandedOverFollowsNoLink(t *testing.T) {
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
victim := filepath.Join(dir, "victim", "data")
if err := os.MkdirAll(victim, 0o700); err != nil {
t.Fatal(err)
}
if err := os.Symlink(filepath.Join(dir, "victim"), filepath.Join(dir, "parent")); err != nil {
t.Fatal(err)
}
if _, _, _, err := giveHandedOver(filepath.Join(dir, "parent", "data"), 0o755, 0, 0, false); err == nil {
t.Fatal("a directory reached through a link was changed")
}
if info, _ := os.Stat(victim); info.Mode().Perm() != 0o700 {
t.Fatalf("the victim was changed: %o", info.Mode().Perm())
}
mode, _, _, err := giveHandedOver(victim, 0o750, 0, 0, false)
if err != nil || mode.Perm() != 0o700 {
t.Fatalf("the directory itself: before %o, %v", mode, err)
}
if info, _ := os.Stat(victim); info.Mode().Perm() != 0o750 {
t.Fatalf("the directory itself was not given its mode: %o", info.Mode().Perm())
}
}
+388
View File
@@ -0,0 +1,388 @@
package apply
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0118: undeclaring removes what the mesh made, gives back what it changed,
// and leaves what was the machine's — which needs what was found kept exactly, and a unit the mesh
// made known for the mesh's whatever its record says.
func unitsMachine(units map[string]*fakeUnit) *machine {
return &machine{containers: map[string]*fakeContainer{}, units: units}
}
// nothingButA is a declaration of one unrelated file, so everything recorded is undeclared.
func nothingButA(t *testing.T) string {
return `{"declaration":1,"resources":[
{"id":"other","type":"file","path":"` + filepath.Join(t.TempDir(), "a") + `","content":"a\n"}]}`
}
// copyOf is a state as a later load of it would be: sharing nothing with the one it came from.
func copyOf(t *testing.T, s store.State) store.State {
t.Helper()
raw, err := json.Marshal(s)
if err != nil {
t.Fatal(err)
}
var out store.State
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
return out
}
func applyOn(t *testing.T, raw string, known store.State, m *machine) (Report, store.State, error) {
t.Helper()
return Apply(context.Background(), archHost(t), parse(t, raw), known, store.OriginDeclared, m.run, nil, nil)
}
func TestAUnitWhoseFileTheMeshWroteIsStoppedWhateverItsRecordSays(t *testing.T) {
// The adoption guard's unit file is the mesh's own file resource, written where there was none.
// Its service recorded before the host kept what it found has no Found, and forgetting it left
// the guard's table loaded on a converged node and its unit file deleted from under it.
units := t.TempDir()
was := unitDir
unitDir = units
t.Cleanup(func() { unitDir = was })
unitFile := filepath.Join(units, "mesh-guard.service")
if err := os.WriteFile(unitFile, []byte("[Unit]\n"), 0o644); err != nil {
t.Fatal(err)
}
m := unitsMachine(map[string]*fakeUnit{"mesh-guard.service": {active: "active", enabled: "enabled"}})
fileThereAtStop := false
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "systemctl" && args[0] == "stop" {
_, err := os.Stat(unitFile)
fileThereAtStop = err == nil
}
return m.run(ctx, name, args...)
}
// As a host before this change recorded them: the unit file first, then the service it
// starts, and no Found on the service.
known := store.State{Resources: []store.Applied{
{ID: "adoption.guard-unit", Type: "file", Target: unitFile, Origin: store.OriginDeclared},
{ID: "adoption.guard-running", Type: "service", Target: "mesh-guard.service", Origin: store.OriginDeclared},
}}
report, after, err := applyWith(t, parse(t, nothingButA(t)), known, run)
if err != nil {
t.Fatal(err)
}
if u := m.units["mesh-guard.service"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("the mesh's own unit was left %s and %s: %v", u.active, u.enabled, m.asked)
}
if !fileThereAtStop {
t.Error("the unit was stopped after its file was deleted, or not at all")
}
if o := outcomeOf(report, "adoption.guard-running"); o.Action != "removed" || !strings.Contains(o.Detail, "unit file") {
t.Errorf("outcome %+v", o)
}
if _, err := os.Stat(unitFile); !os.IsNotExist(err) {
t.Error("the unit file outlived its record")
}
if len(after.Resources) != 1 {
t.Errorf("still recorded: %v", after.IDs())
}
// A unit file the host wrote OVER — its original kept — is the machine's unit, and a record
// with no Found leaves it as it is.
if err := os.WriteFile(unitFile, []byte("[Unit]\n"), 0o644); err != nil {
t.Fatal(err)
}
m = unitsMachine(map[string]*fakeUnit{"mesh-guard.service": {active: "active", enabled: "enabled"}})
known.Resources[0].Kept = filepath.Join(t.TempDir(), "original")
if err := os.WriteFile(known.Resources[0].Kept, []byte("[Unit]\n"), 0o600); err != nil {
t.Fatal(err)
}
if _, _, err := applyWith(t, parse(t, nothingButA(t)), known, m.run); err != nil {
t.Fatal(err)
}
if m.did("systemctl stop") || m.did("systemctl disable") {
t.Errorf("a unit whose file the mesh only wrote over was stopped: %v", m.asked)
}
}
func TestOnlyAUnitsOwnFileTheMeshCreatedMakesItTheMeshs(t *testing.T) {
known := store.State{Resources: []store.Applied{
{ID: "a", Type: "file", Target: "/etc/systemd/system/made.service"},
{ID: "b", Type: "file", Target: "/run/systemd/system/runtime.service"},
{ID: "c", Type: "file", Target: "/etc/systemd/system/kept.service", Kept: "/var/lib/mesh-host/kept/x"},
{ID: "d", Type: "file", Target: "/etc/systemd/system/into.service", Into: &store.Into{Format: "block"}},
{ID: "e", Type: "file", Target: "/etc/systemd/system/docker.service.d/mesh.conf"},
{ID: "f", Type: "file", Target: "/etc/mesh/elsewhere.service"},
{ID: "g", Type: "directory", Target: "/etc/systemd/system/dir.service"},
}}
made := meshMadeUnits(known)
for unit, want := range map[string]bool{"made.service": true, "runtime.service": true, "kept.service": false,
"into.service": false, "mesh.conf": false, "docker.service.d": false, "elsewhere.service": false, "dir.service": false} {
if made[unitKey("", "", unit)] != want {
t.Errorf("%s: made %v, want %v", unit, made[unitKey("", "", unit)], want)
}
}
}
func TestWhatWasFoundOutlivesAFirstApplyThatFailed(t *testing.T) {
// Enabled, then it would not start: no record. The next apply must not read the enable as
// the machine's — or undeclaring leaves enabled a unit the mesh enabled.
m := unitsMachine(map[string]*fakeUnit{"filter.service": {active: "inactive", enabled: "disabled", wontStart: true}})
declared := `{"declaration":1,"resources":[
{"id":"s","type":"service","unit":"filter.service","state":"running","boot":"enabled"}]}`
_, first, err := applyOn(t, declared, store.State{}, m)
if err == nil || !m.did("systemctl enable filter.service") {
t.Fatalf("the fixture did not enable and then fail: %v, %v", err, m.asked)
}
if _, ok := first.Find("s"); ok {
t.Fatal("a failed apply was recorded")
}
if p := first.FoundFirst["s"]; p.State != "stopped" || p.Boot != "disabled" || p.Unit != "filter.service" {
t.Fatalf("what was found was not kept through the failure: %+v", first.FoundFirst)
}
failed := copyOf(t, first)
m.units["filter.service"].wontStart = false
_, second, err := applyOn(t, declared, first, m)
if err != nil {
t.Fatal(err)
}
if rec, _ := second.Find("s"); rec.Found == nil || rec.Found.State != "stopped" || rec.Found.Boot != "disabled" {
t.Errorf("the record carries the mesh's own effect as found: %+v", rec.Found)
}
if second.FoundFirst != nil {
t.Errorf("kept apart after the record carried it: %+v", second.FoundFirst)
}
if _, _, err := applyOn(t, nothingButA(t), second, m); err != nil {
t.Fatal(err)
}
if u := m.units["filter.service"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("undeclared, the unit was left %s and %s", u.active, u.enabled)
}
// Undeclared before it was ever recorded, what was found goes with it — only for its origin.
if _, kept, _ := Apply(context.Background(), archHost(t), parse(t, nothingButA(t)), copyOf(t, failed),
store.OriginCarried, m.run, nil, nil); kept.FoundFirst["s"].State == "" {
t.Error("a carried apply dropped what the mesh's declaration found")
}
if _, dropped, err := applyOn(t, nothingButA(t), copyOf(t, failed), m); err != nil || dropped.FoundFirst != nil {
t.Errorf("kept for a service no longer declared: %+v, %v", dropped.FoundFirst, err)
}
}
func TestBootIsFoundTheFirstTimeTheMeshSetsIt(t *testing.T) {
// The declaration said nothing about boot at first, so the mesh never touched it: what is
// there when a declaration first does is still the machine's.
m := unitsMachine(map[string]*fakeUnit{"web.service": {active: "active", enabled: "disabled"}})
known := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "web.service",
Origin: store.OriginDeclared, Found: &store.FoundUnit{Unit: "web.service", State: "running"}}}}
_, after, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"s","type":"service","unit":"web.service","state":"running","boot":"enabled"}]}`, known, m)
if err != nil {
t.Fatal(err)
}
rec, _ := after.Find("s")
if rec.Found == nil || rec.Found.State != "running" || rec.Found.Boot != "disabled" {
t.Fatalf("found %+v, want running and disabled", rec.Found)
}
report, _, err := applyOn(t, nothingButA(t), after, m)
if err != nil {
t.Fatal(err)
}
if u := m.units["web.service"]; u.active != "active" || u.enabled != "disabled" {
t.Errorf("undeclared, the unit is %s and %s; want running, and disabled again", u.active, u.enabled)
}
if o := outcomeOf(report, "s"); o.Action != "restored" || o.Detail != "disabled at boot, as the host found it" {
t.Errorf("outcome %+v", o)
}
}
func TestAServiceOnceDeclaredWithNoStateIsFoundWhenFirstGivenOne(t *testing.T) {
// Declared with no state (novox/hq ADR 0117), the mesh never started or stopped it — so what
// is there when a declaration first gives it one is what the machine had.
m := unitsMachine(map[string]*fakeUnit{"net.service": {active: "inactive", enabled: "disabled"}})
known := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "net.service",
Origin: store.OriginDeclared, Stateless: true}}}
_, after, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"s","type":"service","unit":"net.service","state":"running"}]}`, known, m)
if err != nil {
t.Fatal(err)
}
if rec, _ := after.Find("s"); rec.Found == nil || rec.Found.State != "stopped" {
t.Fatalf("found %+v, want stopped", rec.Found)
}
if _, _, err := applyOn(t, nothingButA(t), after, m); err != nil {
t.Fatal(err)
}
if m.units["net.service"].active != "inactive" {
t.Error("the unit the mesh started outlived its declaration")
}
}
func TestAUnitWrittenInTheSameApplyIsLoadedBeforeItIsRead(t *testing.T) {
// Read before the service manager is told about its new file, the unit is not there to find.
dir := t.TempDir()
m := unitsMachine(map[string]*fakeUnit{"fresh.service": {active: "inactive", enabled: "disabled"}})
_, _, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"unit","type":"file","path":"`+filepath.Join(dir, "fresh.service")+`","content":"[Unit]\n"},
{"id":"s","type":"service","unit":"fresh.service","state":"running","restart-on":["unit"]}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
reload, show := -1, -1
for i, a := range m.asked {
if a == "systemctl daemon-reload" && reload < 0 {
reload = i
}
if strings.HasPrefix(a, "systemctl show fresh.service") && show < 0 {
show = i
}
}
if reload < 0 || show < 0 || reload > show {
t.Errorf("the unit was read before its file was loaded: %v", m.asked)
}
}
func TestAServiceMovedToAnotherUnitGivesTheOldOneBackAndFindsTheNewOne(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{
"old.service": {active: "active", enabled: "enabled"},
"new.service": {active: "inactive", enabled: "disabled"},
})
known := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "old.service",
Origin: store.OriginDeclared, Found: &store.FoundUnit{Unit: "old.service", State: "stopped"}}}}
report, after, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"s","type":"service","unit":"new.service","state":"running"}]}`, known, m)
if err != nil {
t.Fatal(err)
}
if m.units["old.service"].active != "inactive" {
t.Error("the unit the mesh started is still running though nothing declares it")
}
if m.units["new.service"].active != "active" {
t.Error("the unit now declared was not started")
}
rec, _ := after.Find("s")
if rec.Found == nil || rec.Found.Unit != "new.service" || rec.Found.State != "stopped" {
t.Errorf("what was found about the old unit was carried to the new one: %+v", rec.Found)
}
if o := outcomeOf(report, "s"); !strings.Contains(o.Detail, "old.service stopped, as the host found it") {
t.Errorf("giving the old unit back went unsaid: %+v", o)
}
}
func TestARemovalSaysWhatItDid(t *testing.T) {
cases := []struct {
name string
found *store.FoundUnit
unit *fakeUnit
action, detail string
}{
{"found stopped and still stopped", &store.FoundUnit{State: "stopped"},
&fakeUnit{active: "inactive", enabled: "disabled"}, "forgotten", "already as the host found it (stopped)"},
{"started by the mesh", &store.FoundUnit{State: "stopped"},
&fakeUnit{active: "active", enabled: "disabled"}, "restored", "stopped, as the host found it"},
{"started and enabled by the mesh", &store.FoundUnit{State: "stopped", Boot: "disabled"},
&fakeUnit{active: "active", enabled: "enabled"}, "restored", "stopped, disabled at boot, as the host found it"},
{"found running, stopped by the mesh", &store.FoundUnit{State: "running"},
&fakeUnit{active: "inactive", enabled: "disabled"}, "forgotten",
"left as it is; the mesh stopped it and does not start anything on the way out"},
{"found running and enabled, disabled by the mesh", &store.FoundUnit{State: "running", Boot: "enabled"},
&fakeUnit{active: "active", enabled: "disabled"}, "forgotten",
"left as it is; the mesh disabled it at boot and does not start anything on the way out"},
{"found running, still running", &store.FoundUnit{State: "running"},
&fakeUnit{active: "active", enabled: "enabled"}, "forgotten", "it was running before the mesh; left as it is"},
// Found says to stop it, so the machine is asked about it — and it is gone.
{"started by the mesh, since uninstalled", &store.FoundUnit{State: "stopped"},
nil, "forgotten", "the unit no longer exists"},
{"recorded before the host kept what it found", nil,
&fakeUnit{active: "active", enabled: "enabled"}, "forgotten", "recorded before the host kept what it found; left as it is"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
units := map[string]*fakeUnit{}
if c.unit != nil {
units["unit.service"] = c.unit
}
m := unitsMachine(units)
known := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "unit.service",
Origin: store.OriginDeclared, Found: c.found}}}
report, after, err := applyOn(t, nothingButA(t), known, m)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "s"); o.Action != c.action || o.Detail != c.detail {
t.Errorf("said %s: %s; want %s: %s", o.Action, o.Detail, c.action, c.detail)
}
if c.unit == nil && !m.did("systemctl show unit.service") {
t.Errorf("the machine was never asked whether the unit is there: %v", m.asked)
}
if m.did("systemctl start") || m.did("systemctl enable") {
t.Errorf("something was started on the way out: %v", m.asked)
}
if _, still := after.Find("s"); still {
t.Error("still recorded")
}
})
}
}
func TestAUnitTheMeshStartedIsStoppedWhenUndeclaredAndOneFoundRunningIsNot(t *testing.T) {
// End to end: found by the first apply, carried, given back.
m := unitsMachine(map[string]*fakeUnit{
"filter.service": {active: "inactive", enabled: "disabled"},
"runtime.service": {active: "active", enabled: "enabled"},
})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"filter","type":"service","unit":"filter.service","state":"running","boot":"enabled"},
{"id":"runtime","type":"service","unit":"runtime.service","state":"running","boot":"enabled"}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
if m.units["filter.service"].active != "active" {
t.Fatal("the fixture did not start the filter")
}
if _, _, err := applyOn(t, nothingButA(t), state, m); err != nil {
t.Fatal(err)
}
if u := m.units["filter.service"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("the filter the mesh started and enabled is %s and %s", u.active, u.enabled)
}
if u := m.units["runtime.service"]; u.active != "active" || u.enabled != "enabled" {
t.Errorf("the runtime that was running before the mesh is %s and %s", u.active, u.enabled)
}
}
func TestAUnitHeldAndThenTakenIsFoundAsThePredecessorLeftIt(t *testing.T) {
// Held while its module was untaken, nothing was applied and nothing found; taken, the first
// apply finds the predecessor's unit — stopped, but started at boot — before starting it.
dir := t.TempDir()
m := unitsMachine(map[string]*fakeUnit{"hello.service": {active: "inactive", enabled: "enabled"}})
service := `{"id":"hello-web.unit","type":"service","unit":"hello.service","state":"running","boot":"enabled"}`
_, held := applyAdopted(t, adopted(t, untaken("hello-web.unit"), service), store.State{}, m, dir)
if _, ok := held.HeldAt("hello-web.unit"); !ok {
t.Fatal("the fixture's unit was not held")
}
_, taken := applyAdopted(t, adopted(t, `{"taken":["hello-web"]}`, service), held, m, dir)
rec, _ := taken.Find("hello-web.unit")
if rec.Found == nil || rec.Found.State != "stopped" || rec.Found.Boot != "enabled" {
t.Fatalf("found %+v, want the predecessor's stopped and enabled", rec.Found)
}
if m.units["hello.service"].active != "active" {
t.Fatal("the taken unit was not started")
}
if _, _, err := applyOn(t, nothingButA(t), taken, m); err != nil {
t.Fatal(err)
}
if u := m.units["hello.service"]; u.active != "inactive" || u.enabled != "enabled" {
t.Errorf("given back as %s and %s; the predecessor left it stopped and enabled", u.active, u.enabled)
}
}
+171
View File
@@ -0,0 +1,171 @@
package apply
import (
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A unit the mesh found running and enabled, and held stopped and disabled by its own declaration, is
// given back running and enabled when that declaration goes (the zfs module holds the distribution's
// scrub timer stopped while its own timer scrubs the pool; unassigning it must not leave the pool with no
// schedule at all). The report says so.
func TestAUnitTheMeshStoppedAndDisabledIsGivenBackRunningAndEnabled(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{
"zfs-scrub-weekly@storage.timer": {active: "active", enabled: "enabled"},
})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"zfs.distribution-weekly","type":"service","unit":"zfs-scrub-weekly@storage.timer","state":"stopped","boot":"disabled"}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
if u := m.units["zfs-scrub-weekly@storage.timer"]; u.active != "inactive" || u.enabled != "disabled" {
t.Fatalf("the fixture did not stop and disable it: %s and %s", u.active, u.enabled)
}
report, _, err := applyOn(t, nothingButA(t), copyOf(t, state), m)
if err != nil {
t.Fatal(err)
}
if u := m.units["zfs-scrub-weekly@storage.timer"]; u.active != "active" || u.enabled != "enabled" {
t.Errorf("given back as %s and %s; it was found running and enabled", u.active, u.enabled)
}
said := ""
for _, r := range report.Outcomes {
if r.ID == "zfs.distribution-weekly" {
said = r.Action + ": " + r.Detail
}
}
if !strings.HasPrefix(said, "restored: ") || !strings.Contains(said, "started") || !strings.Contains(said, "enabled at boot") {
t.Errorf("the report says %q", said)
}
}
// Only what the mesh's own declaration did is undone: a unit the mesh held stopped that was found stopped
// stays stopped, and one found running that the mesh never stopped is left running as before.
func TestOnlyWhatTheMeshsDeclarationStoppedIsStartedAgain(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{
"found-stopped.timer": {active: "inactive", enabled: "disabled"},
"left-running.timer": {active: "active", enabled: "enabled"},
})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"a","type":"service","unit":"found-stopped.timer","state":"stopped","boot":"disabled"},
{"id":"b","type":"service","unit":"left-running.timer","state":"running","boot":"enabled"}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
if _, _, err := applyOn(t, nothingButA(t), state, m); err != nil {
t.Fatal(err)
}
if u := m.units["found-stopped.timer"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("found stopped and disabled, given back as %s and %s", u.active, u.enabled)
}
if u := m.units["left-running.timer"]; u.active != "active" || u.enabled != "enabled" {
t.Errorf("found running, given back as %s and %s", u.active, u.enabled)
}
}
// A record from before the host kept what it declared says nothing about who stopped the unit: it is
// left as it is, never started.
func TestARecordWithoutWhatWasDeclaredStartsNothing(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{"old.timer": {active: "inactive", enabled: "disabled"}})
state := store.State{}
state.Record(store.Applied{ID: "old", Type: "service", Target: "old.timer", Origin: store.OriginDeclared,
Found: &store.FoundUnit{Unit: "old.timer", State: "running", Boot: "enabled"}})
if _, _, err := applyOn(t, nothingButA(t), state, m); err != nil {
t.Fatal(err)
}
if u := m.units["old.timer"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("an old record started %s and %s", u.active, u.enabled)
}
}
// The plan says it before the apply does it.
func TestThePlanSaysAStoppedUnitIsStartedAgain(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{"zfs-scrub-weekly@storage.timer": {active: "active", enabled: "enabled"}})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"w","type":"service","unit":"zfs-scrub-weekly@storage.timer","state":"stopped","boot":"disabled"}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
said := ""
for _, s := range Plan(parse(t, nothingButA(t)), state, store.OriginDeclared) {
if s.ID == "w" {
said = s.Verb + ": " + s.Why
}
}
if !strings.HasPrefix(said, "restore: ") || !strings.Contains(said, "restore if still as the mesh held it") ||
strings.Contains(said, "so it is started") {
t.Errorf("the plan says %q", said)
}
}
// Third review: on removal a unit is started again only when it is exactly as the mesh held it, stopped and
// disabled. A masked unit, or one somebody started or enabled since, is left alone, and the report says why.
func TestAMaskedOrChangedUnitIsLeftAloneOnRemoval(t *testing.T) {
for _, c := range []struct {
name, active, enabled, says string
}{
{"masked", "inactive", "masked", "masked"},
{"enabled since", "inactive", "enabled", "not the stopped and disabled"},
{"started since", "active", "disabled", "not the stopped and disabled"},
} {
m := unitsMachine(map[string]*fakeUnit{"w.timer": {active: "active", enabled: "enabled"}})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"w","type":"service","unit":"w.timer","state":"stopped","boot":"disabled"}]}`, store.State{}, m)
if err != nil {
t.Fatal(err)
}
m.units["w.timer"].active, m.units["w.timer"].enabled = c.active, c.enabled
report, _, err := applyOn(t, nothingButA(t), state, m)
if err != nil {
t.Fatalf("%s: %v", c.name, err)
}
if u := m.units["w.timer"]; u.active != c.active || u.enabled != c.enabled {
t.Errorf("%s: changed to %s and %s", c.name, u.active, u.enabled)
}
said := ""
for _, o := range report.Outcomes {
if o.ID == "w" {
said = o.Action + ": " + o.Detail
}
}
if !strings.HasPrefix(said, "forgotten: ") || !strings.Contains(said, c.says) {
t.Errorf("%s: the report says %q", c.name, said)
}
}
}
// A restore that fails — the start refused, or the unit not running after it — waits: the record stays,
// the report says so, and no apply is made fatal by it.
func TestAFailedRestoreWaitsAndIsNeverFatal(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{"w.timer": {active: "active", enabled: "enabled"}})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"w","type":"service","unit":"w.timer","state":"stopped","boot":"disabled"}]}`, store.State{}, m)
if err != nil {
t.Fatal(err)
}
m.units["w.timer"].wontStart = true
for i := 0; i < 2; i++ {
report, next, err := applyOn(t, nothingButA(t), state, m)
if err != nil {
t.Fatalf("apply %d failed: %v", i+1, err)
}
said := ""
for _, o := range report.Outcomes {
if o.ID == "w" {
said = o.Action + ": " + o.Detail
}
}
if !strings.Contains(said, "wait") || !strings.Contains(said, "not running") {
t.Errorf("apply %d: the report says %q", i+1, said)
}
if _, kept := next.Find("w"); !kept {
t.Errorf("apply %d: the record went, so nothing would try again", i+1)
}
state = next
}
}
+230
View File
@@ -0,0 +1,230 @@
package apply
import (
"context"
"fmt"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// An account's groups, from any module (novox/hq ADR 0252, issue 247).
//
// **A module that needs the account in a group declares the account with that group, and nothing else
// of it.** The shell's module sets the account's shell; the lighting daemon's module puts the same
// account in `openrazer`; the container runtime's in `docker`. A shell is one value and stays one
// module's (the controller refuses two); a group is added, so several modules' groups never contradict.
//
// **The mesh takes back only what it gave.** A group the account was in before is the machine's or the
// operator's, and stays when every resource that named it goes. A group the mesh put the account in is
// recorded on the resource that put it there, and given back when that resource stops asking for it —
// unless another declared resource still asks for the same group, which keeps it.
//
// **A group takes effect at the next login.** The machine's group database changes at once; every
// process already running, the account's own service manager and everything it started included, keeps
// the groups it started with. So the outcome says a new login is needed, and the node-engine's account
// judge (internal/accounts) says so on every look until the account's running manager has the group.
// Wanted is every group a declaration asks an account to be in, and which resources ask: account →
// group → resource ids.
type Wanted map[string]map[string][]string
// groupsWanted reads every user resource's groups from a declaration.
func groupsWanted(resources []declaration.Resource) Wanted {
w := Wanted{}
for _, r := range resources {
u, ok := r.(*declaration.User)
if !ok || u.Name == "" {
continue
}
for _, g := range u.Groups {
if w[u.Name] == nil {
w[u.Name] = map[string][]string{}
}
w[u.Name][g] = append(w[u.Name][g], u.ID)
}
}
return w
}
// othersAsk is every resource other than one that asks for an account to be in a group.
func (w Wanted) othersAsk(account, group, except string) []string {
var others []string
for _, id := range w[account][group] {
if id != except {
others = append(others, id)
}
}
return others
}
// applyGroups makes the account be in every group the resource declares, and takes it out of every group
// this resource put it in and no longer asks for. What the mesh put the account in is recorded on out.
func applyGroups(ctx context.Context, sys system.System, r *declaration.User, run Runner,
previous store.Applied, wanted Wanted, handed Handed, out *Outcome) error {
// What this resource put the account in before, for this account only: a declaration that renamed its
// user says nothing about the new one's groups.
var added []string
if previous.Target == r.Name {
added = append(added, previous.Groups...)
}
if len(r.Groups) == 0 && len(added) == 0 {
return nil
}
in, err := system.GroupsOf(ctx, system.Runner(run), r.Name)
if err != nil {
return err
}
already := setOf(in)
declared := setOf(r.Groups)
var put []string
for _, want := range r.Groups {
if already[want] {
// A record that went handed it here: the mesh's, and this resource's to take back (novox/hq ADR 0312 §5).
if handed.has(r.Name, want) && !contains(added, want) {
added = append(added, want)
}
continue
}
exists, err := system.GroupExists(ctx, system.Runner(run), want)
if err != nil {
return err
}
if !exists {
return fmt.Errorf("%q was not put in the group %q: this machine has no such group yet. The package "+
"that makes it is not installed, or is declared after the account", r.Name, want)
}
if err := sys.AddUserToGroup(ctx, system.Runner(run), r.Name, want); err != nil {
return err
}
already[want] = true
put = append(put, want)
if !contains(added, want) {
added = append(added, want)
}
}
// What this resource put the account in and no longer asks for.
var kept, said []string
for _, g := range added {
switch {
case declared[g]:
kept = append(kept, g)
case !already[g]:
// Taken out since, by a person: nothing to give back.
case len(wanted.othersAsk(r.Name, g, r.ID)) > 0:
said = append(said, fmt.Sprintf("left in %s, which %s still asks for", g,
strings.Join(wanted.othersAsk(r.Name, g, r.ID), ", ")))
default:
gone, why := leaveGroup(ctx, sys, r.Name, g, run)
if !gone {
kept = append(kept, g)
}
said = append(said, why)
}
}
out.groups = kept
if len(put) > 0 {
said = append([]string{fmt.Sprintf("put in %s; a session that began before has %s only after a new "+
"login", strings.Join(put, ", "), them(len(put)))}, said...)
}
if len(put) > 0 || len(said) > 0 {
if out.Action == "unchanged" {
out.Action = "updated"
}
out.Detail = joinDetail(out.Detail, strings.Join(said, "; "))
}
return nil
}
// giveGroupsBack is removeUser's groups: every group the mesh put the account in, taken back unless
// another declared resource still asks for it. Never fatal, for the shell's reason: a removal that failed
// would stay recorded and fail the same way on every apply after. Empty when there was nothing to say.
//
// A group left because another declared resource still asks for it is handed to that resource (handed), which
// records it as the mesh's, so it is taken back once nobody asks (novox/hq ADR 0312 §5).
func giveGroupsBack(ctx context.Context, sys system.System, a store.Applied, run Runner, wanted Wanted,
handed Handed) (bool, string) {
if len(a.Groups) == 0 {
return false, ""
}
in, err := system.GroupsOf(ctx, system.Runner(run), a.Target)
if err != nil {
return false, fmt.Sprintf("the groups the mesh put it in (%s) were not given back: %v",
strings.Join(a.Groups, ", "), err)
}
already := setOf(in)
gave := false
var said []string
for _, g := range a.Groups {
if !already[g] {
continue
}
if others := wanted.othersAsk(a.Target, g, a.ID); len(others) > 0 {
said = append(said, fmt.Sprintf("left in %s, which %s still asks for", g, strings.Join(others, ", ")))
handed.hand(a.Target, g)
continue
}
gone, why := leaveGroup(ctx, sys, a.Target, g, run)
gave = gave || gone
said = append(said, why)
}
return gave, strings.Join(said, "; ")
}
// leaveGroup takes an account out of one group the mesh put it in, read back from the machine, and says
// what came of it.
func leaveGroup(ctx context.Context, sys system.System, account, group string, run Runner) (bool, string) {
l, ok := sys.(system.GroupLeaver)
if !ok {
return false, fmt.Sprintf("left in %s: this machine's system cannot take an account out of one group", group)
}
if err := l.RemoveUserFromGroup(ctx, system.Runner(run), account, group); err != nil {
return false, fmt.Sprintf("left in %s: %v", group, err)
}
in, err := system.GroupsOf(ctx, system.Runner(run), account)
if err != nil {
return false, fmt.Sprintf("taken out of %s, and the group database could not be read back: %v", group, err)
}
if setOf(in)[group] {
return false, fmt.Sprintf("taken out of %s, and the group database still lists it there", group)
}
return true, fmt.Sprintf("taken out of %s, which the mesh had put it in", group)
}
func setOf(items []string) map[string]bool {
s := map[string]bool{}
for _, i := range items {
s[i] = true
}
return s
}
func contains(items []string, want string) bool {
for _, i := range items {
if i == want {
return true
}
}
return false
}
func them(n int) string {
if n == 1 {
return "it"
}
return "them"
}
func joinDetail(a, b string) string {
switch {
case a == "":
return b
case b == "":
return a
}
return a + "; " + b
}
+233
View File
@@ -0,0 +1,233 @@
package apply
import (
"context"
"errors"
"sort"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0252 and issue 247: a module puts the operator's account in a group by declaring
// the account with that group alone; the account's other groups are never touched; a new login is said;
// and the mesh gives back only a group it put the account in, and only when nobody declared still asks.
// groupDB is a fake user and group database: the account's groups, the groups the machine has, and every
// command it was asked.
type groupDB struct {
account string
in map[string]bool
exists map[string]bool
asked []string
}
func newGroupDB(in []string, exists ...string) *groupDB {
g := &groupDB{account: "operator", in: map[string]bool{}, exists: map[string]bool{}}
for _, x := range in {
g.in[x], g.exists[x] = true, true
}
for _, x := range exists {
g.exists[x] = true
}
return g
}
func (g *groupDB) run(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
g.asked = append(g.asked, name+" "+strings.Join(args, " "))
switch {
case name == "getent" && args[0] == "passwd":
if args[1] == g.account {
return g.account + ":x:1500:1500::/home/" + g.account + ":/bin/bash\n", nil
}
return "", errors.New("getent exited 2: ")
case name == "getent" && args[0] == "group":
if g.exists[args[1]] {
return args[1] + ":x:900:\n", nil
}
return "", errors.New("getent exited 2: ")
case name == "id":
var out []string
for x := range g.in {
out = append(out, x)
}
sort.Strings(out)
return strings.Join(out, " ") + "\n", nil
case name == "usermod" && args[0] == "--append":
if !g.exists[args[2]] {
return "", errors.New("usermod exited 6: group '" + args[2] + "' does not exist")
}
g.in[args[2]] = true
case name == "gpasswd" && args[0] == "--delete":
delete(g.in, args[2])
}
return "", nil
}
func (g *groupDB) groups() string {
var out []string
for x := range g.in {
out = append(out, x)
}
sort.Strings(out)
return strings.Join(out, " ")
}
func applyGroupsOf(t *testing.T, g *groupDB, known store.State, resources ...string) (Report, store.State, error) {
t.Helper()
if len(resources) == 0 {
resources = []string{`{"id":"other.dir","type":"directory","path":"` + t.TempDir() + `/other"}`}
}
return Apply(context.Background(), archHost(t), parse(t, `{"declaration":1,"resources":[`+
strings.Join(resources, ",")+`]}`), known, store.OriginDeclared, g.run, nil, nil)
}
const (
razer = `{"id":"openrazer.account","type":"user","name":"operator","groups":["openrazer"]}`
docker = `{"id":"docker.account","type":"user","name":"operator","groups":["docker"]}`
shell = `{"id":"zsh.login","type":"user","name":"operator","groups":[]}`
)
func TestAModulePutsTheAccountInAGroupAndSaysANewLoginIsNeeded(t *testing.T) {
g := newGroupDB([]string{"wheel", "plugdev"}, "openrazer")
report, state, err := applyGroupsOf(t, g, store.State{}, shell, razer)
if err != nil {
t.Fatal(err)
}
if got := g.groups(); got != "openrazer plugdev wheel" {
t.Fatalf("the account's groups are %q", got)
}
o := outcomeOf(report, "openrazer.account")
if o.Action != "updated" || !strings.Contains(o.Detail, "put in openrazer") || !strings.Contains(o.Detail, "new login") {
t.Errorf("the outcome did not say the group and the new login: %+v", o)
}
a, _ := state.Find("openrazer.account")
if strings.Join(a.Groups, " ") != "openrazer" {
t.Errorf("the record holds %v, want the one group the mesh added", a.Groups)
}
for _, asked := range g.asked {
if strings.HasPrefix(asked, "usermod") && !strings.HasPrefix(asked, "usermod --append --groups openrazer ") {
t.Errorf("usermod was asked something other than appending the one group: %q", asked)
}
}
// Applied again: nothing to do, and the record still says the mesh added it.
report, state, err = applyGroupsOf(t, g, state, shell, razer)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "openrazer.account"); o.Action != "unchanged" {
t.Errorf("a second apply changed something: %+v", o)
}
if a, _ := state.Find("openrazer.account"); strings.Join(a.Groups, " ") != "openrazer" {
t.Errorf("a second apply lost what the mesh added: %v", a.Groups)
}
}
func TestAGroupTheAccountWasAlreadyInIsNeverTakenBack(t *testing.T) {
g := newGroupDB([]string{"wheel", "openrazer"})
_, state, err := applyGroupsOf(t, g, store.State{}, razer)
if err != nil {
t.Fatal(err)
}
if a, _ := state.Find("openrazer.account"); len(a.Groups) != 0 {
t.Fatalf("a group found was recorded as the mesh's: %v", a.Groups)
}
if _, _, err := applyGroupsOf(t, g, state); err != nil {
t.Fatal(err)
}
if got := g.groups(); got != "openrazer wheel" {
t.Errorf("undeclaring took a found group: %q", got)
}
}
func TestTheGroupTheMeshAddedIsGivenBackWhenItsModuleGoes(t *testing.T) {
g := newGroupDB([]string{"wheel"}, "openrazer")
_, state, err := applyGroupsOf(t, g, store.State{}, razer)
if err != nil {
t.Fatal(err)
}
report, state, err := applyGroupsOf(t, g, state)
if err != nil {
t.Fatal(err)
}
if got := g.groups(); got != "wheel" {
t.Errorf("the account's groups after its module went: %q, want wheel alone", got)
}
o := outcomeOf(report, "openrazer.account")
if o.Action != "restored" || !strings.Contains(o.Detail, "taken out of openrazer") {
t.Errorf("the removal did not say the group was given back: %+v", o)
}
if _, still := state.Find("openrazer.account"); still {
t.Error("the record stayed")
}
}
func TestAGroupAnotherResourceStillAsksForStays(t *testing.T) {
both := `{"id":"openrazer.account","type":"user","name":"operator","groups":["openrazer","video"]}`
video := `{"id":"media.account","type":"user","name":"operator","groups":["video"]}`
g := newGroupDB(nil, "openrazer", "video")
_, state, err := applyGroupsOf(t, g, store.State{}, both, video)
if err != nil {
t.Fatal(err)
}
report, _, err := applyGroupsOf(t, g, state, video)
if err != nil {
t.Fatal(err)
}
if got := g.groups(); got != "video" {
t.Errorf("the account's groups: %q, want video kept for media and openrazer given back", got)
}
if o := outcomeOf(report, "openrazer.account"); !strings.Contains(o.Detail, "media.account still asks for") {
t.Errorf("the removal did not say why video stayed: %+v", o)
}
}
func TestAGroupNoLongerDeclaredIsGivenBackWhileTheAccountStaysDeclared(t *testing.T) {
g := newGroupDB(nil, "openrazer", "input")
two := `{"id":"openrazer.account","type":"user","name":"operator","groups":["openrazer","input"]}`
_, state, err := applyGroupsOf(t, g, store.State{}, two)
if err != nil {
t.Fatal(err)
}
_, state, err = applyGroupsOf(t, g, state, razer)
if err != nil {
t.Fatal(err)
}
if got := g.groups(); got != "openrazer" {
t.Errorf("the account's groups: %q, want input given back", got)
}
if a, _ := state.Find("openrazer.account"); strings.Join(a.Groups, " ") != "openrazer" {
t.Errorf("the record holds %v", a.Groups)
}
}
func TestAGroupThatDoesNotExistYetFailsTheResourceSayingSo(t *testing.T) {
g := newGroupDB([]string{"wheel"})
_, _, err := applyGroupsOf(t, g, store.State{}, razer)
if err == nil || !strings.Contains(err.Error(), "no such group yet") {
t.Fatalf("a missing group was not said: %v", err)
}
for _, asked := range g.asked {
if strings.HasPrefix(asked, "usermod") {
t.Errorf("usermod was run for a group the machine does not have: %q", asked)
}
}
}
func TestAGroupAPersonTookTheAccountOutOfSinceIsPutBackWhileDeclared(t *testing.T) {
g := newGroupDB(nil, "openrazer")
_, state, err := applyGroupsOf(t, g, store.State{}, razer)
if err != nil {
t.Fatal(err)
}
delete(g.in, "openrazer")
if _, _, err := applyGroupsOf(t, g, state, razer); err != nil {
t.Fatal(err)
}
if got := g.groups(); got != "openrazer" {
t.Errorf("a declared group was not put back: %q", got)
}
}
+321
View File
@@ -0,0 +1,321 @@
package apply
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// novox/hq issue 213: the controller moves from a container to a process on the one machine that
// runs it. Every orphan is removed before anything is applied, so without a handover the container
// went first and nothing answered the mesh's verbs while the process was fetched, unpacked and
// started — and for ever, if it did not start. A process that `replaces` the container is applied
// first; the container goes only once the process is running a moment later.
// aMachine fakes the service manager and the container runtime: it records every command, answers
// `systemctl show` with whether the process is running, and has a container until it is removed.
type aMachine struct {
commands []string
running bool // what `systemctl show` says of the process once it was started
started bool
container bool
timer bool // whether a timer, once started, is up
crashing bool // up at the first look, waiting to restart at the next
looks int
}
func (m *aMachine) run(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
line := name + " " + strings.Join(args, " ")
m.commands = append(m.commands, line)
switch {
case name == "systemctl" && len(args) > 0 && (args[0] == "restart" || args[0] == "start"):
m.started = true
case name == "systemctl" && len(args) > 0 && args[0] == "is-active" && strings.HasSuffix(args[len(args)-1], ".timer"):
if m.timer {
return "active", nil
}
return "inactive", errors.New("inactive")
case name == "systemctl" && len(args) > 0 && args[0] == "is-active":
if m.started && m.running {
return "active", nil
}
return "inactive", errors.New("inactive")
case name == "systemctl" && len(args) > 0 && args[0] == "show":
m.looks++
if m.crashing && m.started {
// Up at the first look; waiting to be started again, a new process, at the second.
if m.looks == 1 {
return "ActiveState=active\nSubState=running\nMainPID=42\nNRestarts=0\n", nil
}
return "ActiveState=activating\nSubState=auto-restart\nMainPID=0\nNRestarts=1\n", nil
}
if m.started && m.running {
return "ActiveState=active\nSubState=running\nMainPID=42\nNRestarts=0\n", nil
}
return "ActiveState=inactive\nSubState=dead\nMainPID=0\nNRestarts=0\n", nil
case name == "docker" && len(args) > 1 && args[0] == "rm":
m.container = false
case name == "docker" && len(args) > 1 && args[0] == "container" && args[1] == "inspect":
if !m.container {
return "", errors.New("no such container")
}
return "true\t", nil
}
return "", nil
}
func (m *aMachine) index(prefix string) int {
for i, c := range m.commands {
if strings.HasPrefix(c, prefix) {
return i
}
}
return -1
}
// theController is a machine whose controller ran as a container, recorded, and a declaration that
// runs it as a process from a bundle served here instead.
func theController(t *testing.T, digest, source, extra string) (store.State, string) {
t.Helper()
known := store.State{Resources: []store.Applied{{
Origin: store.OriginDeclared, ID: "mesh-controller.server", Type: "container", Target: "mesh-controller",
}}}
return known, `{"declaration":1,"resources":[
{"id":"mesh-controller.controller","type":"process","name":"mesh-controller","source":"` + source +
`","digest":"` + digest + `","run":["./mesh-controller","serve"]` + extra + `}]}`
}
func onAMachine(t *testing.T) {
t.Helper()
serviceSettle, handoverSettle = 0, 0
wasUnits, wasBundles := unitDir, daemonRoot
unitDir, daemonRoot = t.TempDir(), t.TempDir()
t.Cleanup(func() { unitDir, daemonRoot = wasUnits, wasBundles })
}
func TestAContainerAProcessReplacesGoesOnlyOnceTheProcessRuns(t *testing.T) {
onAMachine(t)
body, digest := anArchive(t, map[string]string{"mesh-controller": "#!/bin/sh\n"})
known, raw := theController(t, digest, serving(t, body), `,"replaces":["mesh-controller.server"]`)
m := &aMachine{running: true, container: true}
report, after, err := Apply(context.Background(), archHost(t), parse(t, raw), known,
store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatalf("the handover failed: %v", err)
}
started, removed := m.index("systemctl restart mesh-controller.service"), m.index("docker rm -f mesh-controller")
if started < 0 || removed < 0 {
t.Fatalf("the process was not started or the container not removed: %v", m.commands)
}
if removed < started {
t.Fatalf("the container was removed before its replacement was started — a window with "+
"nothing answering: %v", m.commands)
}
if looked := m.index("systemctl show mesh-controller.service"); looked < 0 || looked > removed {
t.Errorf("the container was removed without looking whether the process runs: %v", m.commands)
}
if o := outcomeOf(report, "mesh-controller.server"); o.Action != "removed" {
t.Errorf("the container's outcome is %+v, want removed", o)
}
if _, still := after.Find("mesh-controller.server"); still {
t.Error("the host still records the container it removed")
}
if _, has := after.Find("mesh-controller.controller"); !has {
t.Error("the process was not recorded")
}
}
// The case the handover exists for: the replacement does not stay up. The container keeps
// answering, stays recorded so a later apply hands it over, and the apply says why it failed.
func TestAContainerIsKeptWhenItsReplacementDoesNotRun(t *testing.T) {
onAMachine(t)
body, digest := anArchive(t, map[string]string{"mesh-controller": "#!/bin/sh\n"})
known, raw := theController(t, digest, serving(t, body), `,"replaces":["mesh-controller.server"]`)
m := &aMachine{running: false, container: true}
report, after, err := Apply(context.Background(), archHost(t), parse(t, raw), known,
store.OriginDeclared, m.run, nil, nil)
if err == nil {
t.Fatal("a replacement that is not running was reported as a clean apply")
}
if !strings.Contains(err.Error(), "mesh-controller.server") {
t.Errorf("the failure does not name what was kept: %v", err)
}
if m.index("docker rm") >= 0 {
t.Fatalf("the container was removed though its replacement is not running: %v", m.commands)
}
if o := outcomeOf(report, "mesh-controller.server"); o.Action != "kept" {
t.Errorf("the container's outcome is %+v, want kept", o)
}
if _, still := after.Find("mesh-controller.server"); !still {
t.Fatal("the container was forgotten, so no later apply would ever remove it")
}
// The next apply finds the process up and finishes the handover.
m.running, m.commands = true, nil
report, after, err = Apply(context.Background(), archHost(t), parse(t, raw), after,
store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatalf("the second apply failed: %v", err)
}
if o := outcomeOf(report, "mesh-controller.server"); o.Action != "removed" {
t.Errorf("the second apply did not hand over: %+v (%v)", o, m.commands)
}
if _, still := after.Find("mesh-controller.server"); still {
t.Error("the container is still recorded after the handover")
}
}
// A replacement that never applied — its bundle is not what was declared — touches nothing, and
// what it replaces keeps running.
func TestAContainerIsKeptWhenItsReplacementFailsToApply(t *testing.T) {
onAMachine(t)
body, _ := anArchive(t, map[string]string{"mesh-controller": "#!/bin/sh\n"})
known, raw := theController(t, "sha256:"+strings.Repeat("b", 64), serving(t, body),
`,"replaces":["mesh-controller.server"]`)
m := &aMachine{running: true, container: true}
report, after, err := Apply(context.Background(), archHost(t), parse(t, raw), known,
store.OriginDeclared, m.run, nil, nil)
if err == nil {
t.Fatal("a replacement whose bundle did not match was reported applied")
}
if m.index("docker rm") >= 0 {
t.Fatalf("the container was removed though nothing replaced it: %v", m.commands)
}
if o := outcomeOf(report, "mesh-controller.server"); o.Action != "kept" {
t.Errorf("the container's outcome is %+v, want kept", o)
}
if _, still := after.Find("mesh-controller.server"); !still {
t.Fatal("the container was forgotten")
}
}
// Without `replaces` nothing changes: an orphan goes before anything is applied, as it always has.
// Kept as a test because it is the window the field exists to close.
func TestWithoutReplacesAnOrphanStillGoesFirst(t *testing.T) {
onAMachine(t)
body, digest := anArchive(t, map[string]string{"mesh-controller": "#!/bin/sh\n"})
known, raw := theController(t, digest, serving(t, body), ``)
m := &aMachine{running: true, container: true}
if _, _, err := Apply(context.Background(), archHost(t), parse(t, raw), known,
store.OriginDeclared, m.run, nil, nil); err != nil {
t.Fatal(err)
}
if removed, started := m.index("docker rm -f mesh-controller"), m.index("systemctl restart mesh-controller.service"); removed < 0 || removed > started {
t.Fatalf("an orphan nothing replaces was not removed first: %v", m.commands)
}
}
// novox/hq issue 213, beyond what the oneshot unit (process_step_test.go) already holds: a step
// written `./name` runs its own bundle's binary — the controller's preparation is its own binary —
// and a step is started, never enabled.
func TestAStepRunsItsOwnBundlesBinaryAndIsNotEnabled(t *testing.T) {
onAMachine(t)
body, digest := anArchive(t, map[string]string{"mesh-controller": "#!/bin/sh\n"})
m := &aMachine{running: true}
d := declare(t, `{"id":"mesh-controller.controller-prepare","type":"process","name":"mesh-controller-prepare",
"source":"`+serving(t, body)+`","digest":"`+digest+`","run":["./mesh-controller","prepare"],"run-once":true}`)
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil); err != nil {
t.Fatalf("the step failed: %v", err)
}
for _, c := range m.commands {
if strings.HasPrefix(c, "./") || strings.HasPrefix(c, "systemctl enable") {
t.Errorf("the step was run directly or enabled: %v", m.commands)
}
}
unit, err := os.ReadFile(filepath.Join(unitDir, "mesh-controller-prepare.service"))
if err != nil {
t.Fatal(err)
}
want := "ExecStart=" + filepath.Join(daemonRoot, "mesh-controller-prepare", "mesh-controller") + " prepare"
if !strings.Contains(string(unit), want) {
t.Errorf("the step does not run its own bundle's binary (%q):\n%s", want, unit)
}
}
func TestAStepThatFailsGatesItsModule(t *testing.T) {
onAMachine(t)
body, digest := anArchive(t, map[string]string{"mesh-controller": "#!/bin/sh\n"})
src := serving(t, body)
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "systemctl" && len(args) > 1 && args[0] == "start" {
return "", errors.New("Job for mesh-controller-prepare.service failed")
}
if name == "systemctl" && len(args) > 0 && args[0] == "restart" {
t.Errorf("the module's process was started after its step failed")
}
return "", nil
}
d := declare(t, `{"id":"mesh-controller.controller-prepare","type":"process","name":"mesh-controller-prepare",
"source":"`+src+`","digest":"`+digest+`","run":["./mesh-controller","prepare"],"run-once":true},
{"id":"mesh-controller.controller","type":"process","name":"mesh-controller",
"source":"`+src+`","digest":"`+digest+`","run":["./mesh-controller","serve"]}`)
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, run, nil, nil)
if err == nil {
t.Fatal("a failed step was reported as a clean apply")
}
if o := outcomeOf(report, "mesh-controller.controller"); o.Action != "skipped" {
t.Errorf("the process after a failed step was %+v, want skipped", o)
}
}
// A completed step is done: applied again unchanged, it is not run again — its service is never up
// between runs, and reading that as "a daemon that stopped" re-ran the controller's preparation on
// every apply. Nor is a scheduled run started off its cadence; its timer is what is kept up.
func TestACompletedStepIsNotRunAgainAndAScheduleIsItsTimer(t *testing.T) {
onAMachine(t)
body, digest := anArchive(t, map[string]string{"job": "#!/bin/sh\n"})
src := serving(t, body)
for _, mode := range []string{`"run-once":true`, `"schedule":"0 3 * * *"`} {
// The service of either is never up between runs; a scheduled one's timer is.
m := &aMachine{running: false, timer: true}
d := declare(t, `{"id":"m.job","type":"process","name":"m-job","source":"`+src+`","digest":"`+digest+
`","run":["./job"],`+mode+`}`)
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatalf("%s: first apply: %v", mode, err)
}
m.commands = nil
report, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatalf("%s: second apply: %v", mode, err)
}
if m.index("systemctl start m-job.service") >= 0 || m.index("systemctl restart m-job.service") >= 0 {
t.Errorf("%s: an unchanged apply ran the job again: %v", mode, m.commands)
}
if o := outcomeOf(report, "m.job"); o.Action != "unchanged" {
t.Errorf("%s: an unchanged apply reported %+v", mode, o)
}
}
}
// A replacement crash-looping between its restarts is not up, though the service manager calls it
// "activating" — the state the host's ordinary second look accepts as running. Removing the
// container on that reading would leave nothing answering.
func TestAContainerIsKeptWhenItsReplacementIsCrashLooping(t *testing.T) {
onAMachine(t)
body, digest := anArchive(t, map[string]string{"mesh-controller": "#!/bin/sh\n"})
known, raw := theController(t, digest, serving(t, body), `,"replaces":["mesh-controller.server"]`)
m := &aMachine{crashing: true, container: true}
_, after, err := Apply(context.Background(), archHost(t), parse(t, raw), known,
store.OriginDeclared, m.run, nil, nil)
if err == nil || !strings.Contains(err.Error(), "auto-restart") {
t.Fatalf("a crash-looping replacement was accepted: %v", err)
}
if m.index("docker rm") >= 0 {
t.Fatalf("the container was removed for a replacement that keeps dying: %v", m.commands)
}
if _, still := after.Find("mesh-controller.server"); !still {
t.Fatal("the container was forgotten")
}
}
+84
View File
@@ -0,0 +1,84 @@
package apply
import (
"context"
"slices"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// The node-engine owns every verdict, and nothing else sets a container's check (novox/hq ADR 0240 rule 3,
// "how it is checked"): a declared command becomes the container's check with the declared timing, an
// adopted image check keeps the image's command under the declared timing, and a check the engine makes
// itself — http, tcp — sets nothing on the container and recreates nothing.
func TestADeclaredCommandBecomesTheContainersCheckAndNothingElseSetsOne(t *testing.T) {
pinned := "postgres@sha256:" + strings.Repeat("a", 64)
ranWith := func(health string) []string {
t.Helper()
var ran []string
run := func(_ context.Context, cmd string, args ...string) (string, error) {
if cmd == "docker" && len(args) > 0 && args[0] == "run" {
ran = args
return "deadbeef\n", nil
}
return "", nil
}
field := ""
if health != "" {
field = `,"health":` + health
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"db","type":"container","name":"db","image":"`+pinned+`","ports":["31001:5432"]`+field+`}
]}`)
_, _, _ = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if ran == nil {
t.Fatal("the container was not run")
}
return ran
}
healthFlags := func(args []string) []string {
var out []string
for i, a := range args {
if strings.HasPrefix(a, "--health") || a == "--no-healthcheck" {
out = append(out, a)
if i+1 < len(args) {
out = append(out, args[i+1])
}
}
}
return out
}
exec := healthFlags(ranWith(`{"kind":"exec","command":"pg_isready -q","interval":"30s","timeout":"5s","looks":3,"grace":"1m0s"}`))
want := []string{"--health-cmd", "pg_isready -q", "--health-interval", "30s", "--health-timeout", "5s",
"--health-retries", "3", "--health-start-period", "1m0s"}
if !slices.Equal(exec, want) {
t.Errorf("a declared command was run as %v, want %v", exec, want)
}
adopted := healthFlags(ranWith(`{"kind":"runtime","interval":"20s","timeout":"3s","looks":2,"grace":"30s"}`))
if slices.Contains(adopted, "--health-cmd") || !slices.Contains(adopted, "20s") || !slices.Contains(adopted, "2") {
t.Errorf("an adopted image check was run as %v: the image's command under the declared timing", adopted)
}
for _, h := range []string{"", `{"kind":"http","endpoint":"web","port":31001,"path":"/","interval":"30s","timeout":"5s","looks":3,"grace":"1m0s"}`,
`{"kind":"tcp","port":31001,"interval":"30s","timeout":"5s","looks":3,"grace":"1m0s"}`} {
if got := healthFlags(ranWith(h)); len(got) > 0 {
t.Errorf("a container whose check the engine makes itself (%q) was given the runtime's: %v", h, got)
}
}
// Declaring an http check recreates nothing; declaring a command does.
plain := &declaration.Container{ID: "db", Name: "db", Image: pinned}
withHTTP := *plain
withHTTP.Health = &declaration.Health{Kind: "http", Port: 31001, Path: "/", Interval: "30s", Timeout: "5s", Looks: 3, Grace: "1m0s"}
withExec := *plain
withExec.Health = &declaration.Health{Kind: "exec", Command: "true", Interval: "30s", Timeout: "5s", Looks: 3, Grace: "1m0s"}
if containerSpec(plain, inputs{}) != containerSpec(&withHTTP, inputs{}) {
t.Error("declaring an http check would recreate the container")
}
if containerSpec(plain, inputs{}) == containerSpec(&withExec, inputs{}) {
t.Error("declaring a command would not reach a running container: it is fixed when the container is made")
}
}
+316 -39
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"syscall"
"time"
@@ -22,13 +23,19 @@ import (
// first copy is the one that was there before the mesh.
type Keep func(path string, content []byte, mode os.FileMode) (string, error)
// KeepIn keeps originals under dir/kept, each named for the path it came from, readable by root
// alone — a predecessor's configuration may carry its credentials.
// KeepIn keeps originals under dir/kept, each named for the path it came from AND for what was in
// it, readable by root alone — a predecessor's configuration may carry its credentials.
//
// **By content as well as path, because a path has more than one original.** A file held, let go
// when its module was unassigned, rewritten by the predecessor and found again is a second
// original; named by path alone the second copy was silently discarded while the report said it
// was kept (novox/hq ADR 0100). The same content at the same path is kept once.
func KeepIn(dir string) Keep {
return func(path string, content []byte, _ os.FileMode) (string, error) {
sum := sha256.Sum256([]byte(path))
kept := filepath.Join(dir, "kept",
hex.EncodeToString(sum[:])[:16]+"-"+filepath.Base(path))
where := sha256.Sum256([]byte(path))
what := sha256.Sum256(content)
kept := filepath.Join(dir, "kept", hex.EncodeToString(where[:])[:12]+"-"+
hex.EncodeToString(what[:])[:12]+"-"+filepath.Base(path))
if _, err := os.Lstat(kept); err == nil {
return kept, nil
}
@@ -51,11 +58,21 @@ func KeepIn(dir string) Keep {
// record (novox/hq ADR 0103). Looked at first, because the apply itself makes such things — a
// file's parent directory, a unit file a module writes, a package that brings its unit — and what
// the mesh made in this apply was not found.
type foundBefore map[string]bool
type foundBefore struct {
is map[string]bool
// trouble is what could not be asked about, by the same key, so the resource that would need
// the answer fails rather than proceeding as if the machine had nothing there.
trouble map[string]string
}
func (f foundBefore) has(key string) bool { return f.is[key] }
// why is the reason a key could not be settled, and empty when there was none.
func (f foundBefore) why(key string) string { return f.trouble[key] }
func lookBefore(ctx context.Context, sys system.System, d *declaration.Declaration, known store.State,
run Runner) foundBefore {
seen := foundBefore{}
seen := foundBefore{is: map[string]bool{}, trouble: map[string]string{}}
if d.Adoption == nil {
return seen
}
@@ -70,42 +87,77 @@ func lookBefore(ctx context.Context, sys system.System, d *declaration.Declarati
switch res := r.(type) {
case *declaration.Directory:
if present(res.Path) && !recordedPath(known, res.Path) {
seen["path:"+res.Path] = true
seen.is["path:"+res.Path] = true
}
case *declaration.Archive:
// Unpacking over it, and re-owning it recursively, would change the predecessor's
// files.
if present(res.Path) && !recordedPath(known, res.Path) {
seen["path:"+res.Path] = true
seen.is["path:"+res.Path] = true
}
case *declaration.Process:
// Its unit would be written over and restarted.
if !known.Recorded(string(declaration.TypeProcess), res.Name) &&
present(filepath.Join(unitDir, res.Name+".service")) {
seen["unit-file:"+res.Name] = true
seen.is["unit-file:"+res.Name] = true
}
case *declaration.User:
// Its shell and groups would be changed.
if !known.Recorded(string(declaration.TypeUser), res.Name) {
if _, exists, err := system.LookUpUser(ctx, run, res.Name); err == nil && exists {
seen["user:"+res.Name] = true
seen.is["user:"+res.Name] = true
}
}
case *declaration.Service:
if known.Recorded(string(declaration.TypeService), res.Unit) {
if res.Stateless() || recordedUnit(known, res) {
continue
}
// Found is what the machine runs: a unit that is running or starts at boot. A unit
// file a package merely ships — a template instance nothing ever started, say the
// private network's own wg-quick@mesh0 — is not a predecessor's service, and holding
// it kept the private network from ever coming up (found by the adoption bed).
key := "unit:" + unitKey(res.Scope, res.User, res.Unit)
run := run
if res.UserScoped() {
// In the account's own manager (novox/hq ADR 0177), and only asked while it runs. With
// it not running, what can be known is whether somebody put the unit's file where that
// manager loads an administrator's units from — and the mesh's own file there is not
// somebody's. An account the machine does not have has no unit to find.
away, err := managerAway(ctx, sys, res.Scope, res.User, run)
if err != nil {
continue
}
if away != "" {
for _, path := range userUnitFiles(res.User, res.Unit) {
if present(path) && !meshWroteWhole(known, path) {
seen.is[key] = true
}
}
continue
}
run = managerFor(res.Scope, res.User, run)
}
// **Found is a unit somebody put on this machine, or one the machine uses.**
//
// Where it comes from first: a unit the service manager loads from outside /usr —
// /etc/systemd/system or /run/systemd/system — was installed by an administrator, so
// it is a predecessor's whatever state it is in, and one deliberately stopped and
// disabled must stay that way (novox/hq ADR 0103).
//
// A unit a package ships, under /usr, is not held by its mere presence: the private
// network's own wg-quick@mesh0 is an instance of a template the tunnel package ships,
// nothing had ever run it, and holding it kept the private network from ever coming up
// (found by the adoption bed). Such a unit is held only if the machine actually uses
// it — running, or started at boot.
state, err := sys.ServiceState(ctx, run, res.Unit)
if err != nil {
continue
}
if from, ok := sys.(unitFiles); ok {
if path, err := from.ServiceUnitFile(ctx, run, res.Unit); err == nil && installedByHand(known, path) {
seen.is[key] = true
continue
}
}
boot, _ := sys.ServiceBoot(ctx, run, res.Unit)
if state == "running" || boot == "enabled" {
seen["unit:"+res.Unit] = true
seen.is[key] = true
}
case *declaration.Container:
if known.Recorded(string(declaration.TypeContainer), res.Name) {
@@ -117,7 +169,7 @@ func lookBefore(ctx context.Context, sys system.System, d *declaration.Declarati
case src == "":
case strings.HasPrefix(src, "/"):
if !systemPath(src) && present(src) && !recordedPath(known, src) {
seen["path:"+src] = true
seen.is["path:"+src] = true
}
default:
if !asked {
@@ -128,7 +180,10 @@ func lookBefore(ctx context.Context, sys system.System, d *declaration.Declarati
continue
}
if _, err := run(ctx, cri, "volume", "inspect", src); err == nil {
seen["volume:"+src] = true
seen.is["volume:"+src] = true
} else if !absent(err) {
seen.trouble["volume:"+src] = fmt.Sprintf(
"the container runtime could not say whether the volume %s is here: %v", src, err)
}
}
}
@@ -137,15 +192,46 @@ func lookBefore(ctx context.Context, sys system.System, d *declaration.Declarati
return seen
}
// unitFiles is a service manager that can say where it loads a unit from.
type unitFiles interface {
ServiceUnitFile(ctx context.Context, run Runner, unit string) (string, error)
}
// installedByHand is whether a unit file is one somebody put on this machine rather than one a
// package ships: anywhere but /usr, where distributions keep what they install — and not one this
// host wrote whole. That covers an account's units (novox/hq ADR 0177): one under the account's
// ~/.config/systemd/user or /etc/systemd/user is somebody's, unless the mesh wrote it there.
func installedByHand(known store.State, path string) bool {
if path == "" {
return false
}
return !strings.HasPrefix(filepath.Clean(path), "/usr/") && !meshWroteWhole(known, path)
}
// recordedUnit is whether this host has a record of a service in the same manager as res, under the
// same name (novox/hq ADR 0177): an account's unit and the machine's of one name are two units.
func recordedUnit(known store.State, res *declaration.Service) bool {
want := unitKey(res.Scope, res.User, res.Unit)
for _, r := range known.Resources {
if r.Type == string(declaration.TypeService) && unitKey(r.Scope, r.User, r.Target) == want {
return true
}
}
return false
}
func present(path string) bool {
_, err := os.Lstat(path)
return err == nil
}
// recordedPath is whether this host has a record of making something at a path.
// recordedPath is whether this host has a record of MAKING something at a path — a directory it
// created, a file it wrote, an archive it unpacked. An access record is not one of those: it says
// the mesh set permissions on a path it does not own, which is exactly what it does to a path
// somebody else's software made, so a path it only has access for is still found (novox/hq ADR 0103).
func recordedPath(known store.State, path string) bool {
for _, kind := range []declaration.Type{declaration.TypeDirectory, declaration.TypeFile,
declaration.TypeArchive, declaration.TypeAccess} {
declaration.TypeArchive} {
if known.Recorded(string(kind), path) {
return true
}
@@ -184,7 +270,9 @@ func mountSource(mapping string) string {
}
// runsIn is the container a resource runs inside, if any: an action's `in`, or a run-once step
// sharing a container's namespace.
// sharing a container's namespace. An action with no `in` runs on the machine itself and is not
// held for a container: it reaches nothing a predecessor holds by running there, and holding every
// action of an untaken module would stop a module preparing itself before its cutover.
func runsIn(r declaration.Resource) string {
switch res := r.(type) {
case *declaration.Action:
@@ -209,6 +297,19 @@ func heldContainer(known store.State, name string) (store.Held, bool) {
return store.Held{}, false
}
// replacesNothing is a resource that takes nothing found on the machine from it, so on an adopted
// node it is never held and never previewed as replacing what was found: a file written into
// (novox/hq ADR 0102), and a service whose unit's lifecycle is the machine's (novox/hq ADR 0117).
func replacesNothing(r declaration.Resource) bool {
switch res := r.(type) {
case *declaration.File:
return res.Into != ""
case *declaration.Service:
return res.Stateless()
}
return false
}
// holdOnAdopted decides whether a resource of an adopted node is held rather than applied, and
// holds it (novox/hq ADR 0100, ADR 0103). For a module not yet taken, what is present with no
// record is kept as it is: a file or a container under its name, a directory, a service's unit,
@@ -243,11 +344,11 @@ func holdOnAdopted(ctx context.Context, sys system.System, r declaration.Resourc
}
// A file written into replaces nothing that was found, so it is never held (novox/hq ADR
// 0102) — and a hold from when it was declared whole must not keep the mesh's keys out.
if f, ok := r.(*declaration.File); ok && f.Into != "" {
if already {
known.Release(r.Identity())
}
// 0102) — and a hold from when it was declared whole must not keep the mesh's keys out. That
// hold is released by the apply once the write has worked, not here: a write that fails keeps
// it, and with it where the original was kept. A service whose lifecycle is the machine's
// replaces nothing either (novox/hq ADR 0117).
if replacesNothing(r) {
return false, false, out, nil
}
@@ -287,21 +388,28 @@ func holdOnAdopted(ctx context.Context, sys system.System, r declaration.Resourc
if strings.HasPrefix(src, "/") {
key = "path:" + src
}
if src != "" && before[key] {
if src == "" {
continue
}
if trouble := before.why(key); trouble != "" {
return false, false, begin(r), fmt.Errorf(
"%s, so it is not safe to create a container that would mount it", trouble)
}
if before.has(key) {
isFound, why = true, "would mount "+src+", found on the machine"
break
}
}
case *declaration.Directory:
isFound = before["path:"+res.Path]
isFound = before.has("path:" + res.Path)
case *declaration.Archive:
isFound = before["path:"+res.Path]
isFound = before.has("path:" + res.Path)
case *declaration.Process:
isFound = before["unit-file:"+res.Name]
isFound = before.has("unit-file:" + res.Name)
case *declaration.User:
isFound = before["user:"+res.Name]
isFound = before.has("user:" + res.Name)
case *declaration.Service:
isFound = before["unit:"+res.Unit]
isFound = before.has("unit:" + unitKey(res.Scope, res.User, res.Unit))
}
}
if !isFound {
@@ -315,7 +423,11 @@ func holdOnAdopted(ctx context.Context, sys system.System, r declaration.Resourc
// A held service is not started, stopped, enabled or restarted — but a reload stops nothing,
// so one the module names still happens (novox/hq ADR 0102, ADR 0103).
if svc, ok := r.(*declaration.Service); ok && svc.State == "running" {
if which := restartedBy(svc.ReloadOn, changed); len(which) > 0 {
// In the unit's own manager, and nothing to reload in one that is not running (novox/hq ADR
// 0177): the state read below then fails, and the reload is not attempted.
away, awayErr := managerAway(ctx, sys, svc.Scope, svc.User, run)
run := managerFor(svc.Scope, svc.User, run)
if which := restartedBy(svc.ReloadOn, changed); len(which) > 0 && awayErr == nil && away == "" {
if state, err := sys.ServiceState(ctx, run, svc.Unit); err == nil && state == "running" {
reloader, can := sys.(serviceReloader)
if !can {
@@ -361,29 +473,171 @@ type foundContainer struct {
id string
running bool
spec string
// What a take compares (novox/hq ADR 0163): the image and its id, the networks the container
// is on, its mounts and its published ports — empty from a runtime (or a test's fake) that
// answers the short form.
image string
imageID string
networks []string
mounts []string
ports []string
}
// foundFormat is what inspectFound asks the runtime for, tab-separated: the three a hold has
// always needed, then the facts a take compares.
const foundFormat = "{{.Id}}\t{{.State.Running}}\t{{index .Config.Labels \"" + specLabel + "\"}}" +
"\t{{.Config.Image}}\t{{.Image}}" +
"\t{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}" +
"\t{{range .Mounts}}{{.Source}}:{{.Destination}},{{end}}" +
"\t{{range $p, $b := .NetworkSettings.Ports}}{{$p}}{{range $b}}>{{.HostIp}}:{{.HostPort}}{{end}},{{end}}"
func splitList(s string) []string {
var out []string
for _, part := range strings.Split(s, ",") {
if part = strings.TrimSpace(part); part != "" {
out = append(out, part)
}
}
return out
}
// inspectFound reads a container by name the way a hold needs it: its id, whether it runs, and
// whether a host made it.
//
// **`container inspect`, not the bare form.** A name is not unique across object kinds — a
// module regularly names a network the same as the container that joins it (`keycloak` names
// both, and it is ordinary). The bare form resolves across every kind and returns whichever it
// finds, so a container that does not exist yet but a same-named network does answers with the
// network's JSON — no `.State` field at all — and the template below fails to execute rather
// than failing to find anything. That reads as "the runtime could not say", which this function's
// caller correctly refuses to build on (novox/hq ADR 0100) — but there was something to say, a
// question of kind, not of ambiguity that should have stopped anything.
func inspectFound(ctx context.Context, name string, run Runner) (foundContainer, bool, error) {
cri, err := containerRuntime(ctx, run)
if err != nil {
return foundContainer{}, false, fmt.Errorf("%w, so nothing can be said about %q", err, name)
}
out, err := run(ctx, cri, "inspect", "--format",
"{{.Id}}\t{{.State.Running}}\t{{index .Config.Labels \""+specLabel+"\"}}", name)
out, err := run(ctx, cri, "container", "inspect", "--format", foundFormat, name)
if err != nil {
return foundContainer{}, false, nil
if absent(err) {
return foundContainer{}, false, nil
}
// **A runtime that could not answer is not a machine with nothing there.** Read as
// absence, a daemon that is down or a permission denied would let the mesh create its own
// container over a predecessor's — the one thing an adopted node must never do
// (novox/hq ADR 0100).
return foundContainer{}, false, fmt.Errorf(
"the container runtime could not say whether %s is here, so it is not safe to make one: %w",
name, err)
}
parts := strings.Split(strings.TrimSpace(out), "\t")
for len(parts) < 3 {
for len(parts) < 8 {
parts = append(parts, "")
}
spec := strings.TrimSpace(parts[2])
if spec == "<no value>" {
spec = ""
}
return foundContainer{id: strings.TrimSpace(parts[0]), running: parts[1] == "true", spec: spec}, true, nil
return foundContainer{id: strings.TrimSpace(parts[0]), running: parts[1] == "true", spec: spec,
image: strings.TrimSpace(parts[3]), imageID: strings.TrimSpace(parts[4]),
networks: splitList(parts[5]), mounts: splitList(parts[6]), ports: splitList(parts[7])}, true, nil
}
// factsOf is what a take would compare for a found container (novox/hq ADR 0163): the found
// image and when it was made, the networks and who else is on them, mounts and ports — beside
// what the module declares, and the declared image's date when that image is on the machine.
// Every question the runtime cannot answer leaves its fact empty; a preview says so rather than
// guesses.
func factsOf(ctx context.Context, seen foundContainer, res *declaration.Container, run Runner) *Facts {
cri, err := containerRuntime(ctx, run)
if err != nil {
return nil
}
f := &store.Facts{Image: seen.image, Mounts: seen.mounts, Ports: seen.ports,
DeclaredImage: res.Image, DeclaredPorts: res.Ports, DeclaredVolumes: res.Volumes}
if seen.imageID != "" {
if out, err := run(ctx, cri, "image", "inspect", "--format", "{{.Created}}", seen.imageID); err == nil {
f.ImageCreated = strings.TrimSpace(out)
}
}
if res.Image != "" {
if out, err := run(ctx, cri, "image", "inspect", "--format", "{{.Created}}", res.Image); err == nil {
f.DeclaredImageCreated = strings.TrimSpace(out)
}
}
if found, err := time.Parse(time.RFC3339Nano, f.ImageCreated); err == nil {
if declared, err := time.Parse(time.RFC3339Nano, f.DeclaredImageCreated); err == nil {
f.Downgrade = declared.Before(found)
}
}
for _, network := range seen.networks {
if f.Networks == nil {
f.Networks = map[string][]string{}
}
var members []string
if out, err := run(ctx, cri, "network", "inspect", "--format",
"{{range .Containers}}{{.Name}},{{end}}", network); err == nil {
for _, m := range splitList(out) {
if m != res.Name {
members = append(members, m)
}
}
}
sort.Strings(members)
f.Networks[network] = members
}
return (*Facts)(f)
}
// Facts is store.Facts, named here so hold's callers read as one vocabulary.
type Facts = store.Facts
// differenceOf is how a found file differs from the declared content: the lines only the found
// file has, marked -, then the lines only the declared content has, marked +, in their own order,
// bounded so a report stays a report. Not a diff tool's output: the question a take answers is
// "what would be lost and what would be new", and that is these two lists.
func differenceOf(found, declared string) (bool, []string) {
if found == declared {
return false, nil
}
const bound = 40
count := func(s string) map[string]int {
out := map[string]int{}
for _, line := range strings.Split(s, "\n") {
out[line]++
}
return out
}
inFound, inDeclared := count(found), count(declared)
var out []string
add := func(mark, s string, other map[string]int) {
seen := map[string]int{}
for _, line := range strings.Split(s, "\n") {
seen[line]++
if seen[line] > other[line] && len(out) < bound {
out = append(out, mark+" "+line)
}
}
}
add("-", found, inDeclared)
add("+", declared, inFound)
if len(out) >= bound {
out = append(out, "… and more")
}
return true, out
}
// absent is whether a runtime said the thing is not there, rather than failing to answer. Its own
// words: docker and podman both say "No such object", "No such container" or "No such volume".
func absent(err error) bool {
said := strings.ToLower(err.Error())
for _, missing := range []string{"no such object", "no such container", "no such volume",
"no such image"} {
if strings.Contains(said, missing) {
return true
}
}
return false
}
// hold keeps a found file or container as it is, and reports it — the first time by recording
@@ -437,6 +691,12 @@ func hold(ctx context.Context, sys system.System, r declaration.Resource, module
} else if digestOf(string(content)) != h.Digest {
changed = "rewritten"
}
// What a take would replace it with, and how that differs (novox/hq ADR 0163): a
// file declared whole is compared whole; one written into is not replaced at all.
if res.Into == "" {
differs, lines := differenceOf(string(content), res.Content)
h.Facts = &Facts{Differs: differs, Difference: lines}
}
}
case *declaration.Directory:
info, err := os.Lstat(res.Path)
@@ -490,6 +750,20 @@ func hold(ctx context.Context, sys system.System, r declaration.Resource, module
}
detail = "the user was found on the machine; its shell and groups are kept until " + module + " is taken"
case *declaration.Service:
if res.UserScoped() {
// Read in the account's own manager, and not at all while it is not running (novox/hq
// ADR 0177): held as found, with nothing to compare until it runs.
away, err := managerAway(ctx, sys, res.Scope, res.User, run)
if err != nil {
return out, h, err
}
if away != "" {
detail = "its unit was found in " + res.User + "'s own manager, which is not running; " +
"kept as found until " + module + " is taken"
break
}
run = managerFor(res.Scope, res.User, run)
}
state, err := sys.ServiceState(ctx, run, res.Unit)
switch {
case err != nil && !already:
@@ -525,6 +799,9 @@ func hold(ctx context.Context, sys system.System, r declaration.Resource, module
case h.Running && !seen.running:
changed = "stopped"
}
if exists {
h.Facts = factsOf(ctx, seen, res, run)
}
default:
return out, h, fmt.Errorf("a %s cannot be held", r.Kind())
}
+304 -8
View File
@@ -5,6 +5,7 @@ import (
"errors"
"os"
"path/filepath"
"sort"
"strings"
"testing"
@@ -18,7 +19,17 @@ import (
// label when a host made it. Every command it is asked is written down.
type machine struct {
containers map[string]*fakeContainer
asked []string
// images is what `image inspect --format {{.Created}}` answers per image or id; members is
// what `network inspect` lists per network (ADR 0163).
images map[string]string
members map[string][]string
asked []string
// wgUp is what `wg show interfaces` answers: the tunnels up on the machine.
wgUp string
// handshakes is what `wg show <interface> latest-handshakes` answers, and handshakesFail the
// error it fails with instead — a machine with no `wg`, say (novox/hq ADR 0119).
handshakes string
handshakesFail error
// units are service units by name, as systemd would report them; volumes are the runtime's
// named volumes.
@@ -29,6 +40,11 @@ type machine struct {
type fakeUnit struct {
active, enabled string
// wontStart is a unit that accepts `start` and stays inactive — one that starts and dies.
wontStart bool
// fragment is where systemd loads the unit from; empty means /etc/systemd/system, where an
// administrator installs one.
fragment string
}
// systemctl answers as systemd does for the units the machine has, and "not-found" for any other.
@@ -41,8 +57,18 @@ func (m *machine) systemctl(args []string) (string, error) {
switch args[0] {
case "show":
if !ok {
if len(args) > 2 && strings.Contains(args[2], "FragmentPath") {
return "FragmentPath=\n", nil
}
return "LoadState=not-found\nActiveState=inactive\nType=simple\n", nil
}
if len(args) > 2 && strings.Contains(args[2], "FragmentPath") {
from := u.fragment
if from == "" {
from = "/etc/systemd/system/" + unit
}
return "FragmentPath=" + from + "\n", nil
}
return "LoadState=loaded\nActiveState=" + u.active + "\nType=simple\nRemainAfterExit=no\n", nil
case "is-enabled":
if !ok {
@@ -50,7 +76,9 @@ func (m *machine) systemctl(args []string) (string, error) {
}
return u.enabled + "\n", nil
case "start":
u.active = "active"
if !u.wontStart {
u.active = "active"
}
case "stop":
u.active = "inactive"
case "enable":
@@ -74,13 +102,43 @@ type fakeContainer struct {
id string
running bool
spec string
// What a take compares (ADR 0163), answered in the long inspect form when set.
image, imageID string
networks, mounts, ports []string
}
// oldShape is a systemctl call as these fakes were written to read it: the verb, the unit, then the
// options. The host passes `--` before every unit (third review of mesh-catalog #147), which
// internal/system's own test holds; the fakes read the unit where they always did.
func oldShape(name string, args []string) []string {
if name != "systemctl" || len(args) < 2 || args[len(args)-2] != "--" {
return args
}
v := 0
for v < len(args) && strings.HasPrefix(args[v], "-") {
v++
}
if v >= len(args)-2 {
return args
}
out := append([]string{}, args[:v+1]...)
out = append(out, args[len(args)-1])
return append(out, args[v+1:len(args)-2]...)
}
func (m *machine) run(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
m.asked = append(m.asked, name+" "+strings.Join(args, " "))
if name == "systemctl" {
return m.systemctl(args)
}
if name == "wg" {
if len(args) > 0 && args[len(args)-1] == "latest-handshakes" {
return m.handshakes, m.handshakesFail
}
return m.wgUp, nil
}
if name == "getent" {
if m.users[args[len(args)-1]] {
return args[len(args)-1] + ":x:1500:1500::/home/" + args[len(args)-1] + ":/bin/bash\n", nil
@@ -98,7 +156,10 @@ func (m *machine) run(_ context.Context, name string, args ...string) (string, e
return "", errors.New("no such volume")
case "info":
return "27.0\n", nil
case "inspect":
case "container":
if args[1] != "inspect" {
return "", errors.New("unexpected docker container command")
}
c, ok := m.containers[args[len(args)-1]]
if !ok {
return "", errors.New("no such container")
@@ -107,10 +168,39 @@ func (m *machine) run(_ context.Context, name string, args ...string) (string, e
if c.running {
running = "true"
}
if strings.HasPrefix(args[2], "{{.Id}}") {
return c.id + "\t" + running + "\t" + c.spec + "\n", nil
if strings.HasPrefix(args[3], "{{.Id}}") {
line := c.id + "\t" + running + "\t" + c.spec
if c.image != "" {
line += "\t" + c.image + "\t" + c.imageID + "\t" + strings.Join(c.networks, ",") + "," +
"\t" + strings.Join(c.mounts, ",") + "," + "\t" + strings.Join(c.ports, ",") + ","
}
return line + "\n", nil
}
return running + "\t" + c.spec + "\n", nil
case "image":
if len(args) > 1 && args[1] == "inspect" {
if created, ok := m.images[args[len(args)-1]]; ok {
return created + "\n", nil
}
return "", errors.New("no such image")
}
return "", nil
case "network":
if len(args) > 1 && args[1] == "inspect" {
return strings.Join(m.members[args[len(args)-1]], ",") + ",\n", nil
}
return "", nil
case "ps":
var lines []string
for name, c := range m.containers {
state := "exited"
if c.running {
state = "running"
}
lines = append(lines, name+"\t"+c.image+"\t"+state)
}
sort.Strings(lines)
return strings.Join(lines, "\n") + "\n", nil
case "rm":
delete(m.containers, args[len(args)-1])
return "", nil
@@ -369,8 +459,13 @@ func TestAHoldNoLongerDeclaredIsLetGoAndFoundAgainIfItsModuleReturns(t *testing.
t.Fatalf("a returning module's found file was not held again: %+v", o)
}
again, _ := state.HeldAt("hello-web.page")
if kept, _ := os.ReadFile(again.Kept); again.Kept != first.Kept || string(kept) != "the predecessor's page\n" {
t.Errorf("the first kept original was lost: %s %q", again.Kept, kept)
// The predecessor rewrote it while nothing held it, so that is a second original, kept beside
// the first rather than in place of it (novox/hq ADR 0100).
if kept, _ := os.ReadFile(again.Kept); string(kept) != "the predecessor wrote again\n" {
t.Errorf("what is named as kept is %q", kept)
}
if kept, _ := os.ReadFile(first.Kept); string(kept) != "the predecessor's page\n" {
t.Errorf("the first kept original was lost: %q", kept)
}
if got, _ := os.ReadFile(page); string(got) != "the predecessor wrote again\n" {
t.Errorf("the found file was touched: %q", got)
@@ -617,7 +712,8 @@ func TestAUnitAPackageOnlyShipsIsNotFound(t *testing.T) {
// the private network never came up. Found is what the machine runs.
dir := t.TempDir()
m := &machine{containers: map[string]*fakeContainer{},
units: map[string]*fakeUnit{"wg-quick@mesh0.service": {active: "inactive", enabled: "disabled"}}}
units: map[string]*fakeUnit{"wg-quick@mesh0.service": {active: "inactive", enabled: "disabled",
fragment: "/usr/lib/systemd/system/wg-quick@.service"}}}
report, state := applyAdopted(t, adopted(t, untaken("mesh-wireguard.overlay-up"),
`{"id":"mesh-wireguard.overlay-up","type":"service","unit":"wg-quick@mesh0.service","state":"running","boot":"enabled"}`),
store.State{}, m, dir)
@@ -825,3 +921,203 @@ func TestMountingTheMachinesOwnPlumbingIsNotFoundData(t *testing.T) {
t.Errorf("held: %+v", state.Held)
}
}
// Defends novox/hq ADR 0100: a runtime that cannot answer is not a machine with nothing there.
// unreachable is a machine whose container runtime answers everything with a daemon that is down.
type unreachable struct{ asked []string }
func (u *unreachable) run(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
u.asked = append(u.asked, name+" "+strings.Join(args, " "))
if name == "docker" && args[0] == "info" {
return "27.0\n", nil
}
if name == "docker" {
return "", errors.New("docker exited 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock")
}
return "", nil
}
func TestARuntimeThatCannotAnswerNeverLetsTheMeshCreateOverAFoundContainer(t *testing.T) {
dir := t.TempDir()
u := &unreachable{}
d := adopted(t, untaken("hello-web.server"),
`{"id":"hello-web.server","type":"container","name":"hello-web","image":"`+pinned+`"}`)
_, state, err := ApplyKeeping(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
u.run, nil, nil, KeepIn(dir))
if err == nil || !strings.Contains(err.Error(), "not safe to make one") {
t.Fatalf("a runtime that could not answer was read as nothing there: %v", err)
}
for _, a := range u.asked {
if strings.HasPrefix(a, "docker run") || strings.HasPrefix(a, "docker rm") {
t.Errorf("the mesh acted on a container it could not ask about: %s", a)
}
}
if len(state.Held) != 0 {
t.Errorf("something was held on an answer the machine never gave: %+v", state.Held)
}
}
func TestAHeldContainerStaysHeldWhenTheRuntimeCannotAnswer(t *testing.T) {
dir, page, m := predecessor(t)
d := adopted(t, untaken("hello-web.page", "hello-web.server"), webResources(page))
_, state := applyAdopted(t, d, store.State{}, m, dir)
if _, held := state.HeldAt("hello-web.server"); !held {
t.Fatal("the found container was not held to begin with")
}
u := &unreachable{}
_, state, err := ApplyKeeping(context.Background(), archHost(t), d, state, store.OriginDeclared,
u.run, nil, nil, KeepIn(dir))
if err == nil {
t.Fatal("a runtime that could not answer reported success")
}
if h, held := state.HeldAt("hello-web.server"); !held || h.Changed != "" {
t.Errorf("a hold was let go or called changed on an answer the machine never gave: %+v", h)
}
}
func TestAVolumeTheRuntimeCannotBeAskedAboutStopsTheContainer(t *testing.T) {
dir := t.TempDir()
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch {
case name == "docker" && args[0] == "info":
return "27.0\n", nil
case name == "docker" && args[0] == "volume":
return "", errors.New("docker exited 1: Cannot connect to the Docker daemon")
case name == "docker" && args[0] == "container":
return "", errors.New("Error: No such object: hello-web")
case name == "docker":
return "", errors.New("docker run must not happen")
}
return "", nil
}
d := adopted(t, untaken("hello-web.server"),
`{"id":"hello-web.server","type":"container","name":"hello-web","image":"`+pinned+`",
"volumes":["predecessor-data:/data"]}`)
_, _, err := ApplyKeeping(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
run, nil, nil, KeepIn(dir))
if err == nil || !strings.Contains(err.Error(), "could not say whether the volume") {
t.Fatalf("a volume the runtime could not be asked about did not stop the container: %v", err)
}
}
func TestAUnitSomebodyInstalledIsHeldWhateverStateItIsIn(t *testing.T) {
// A predecessor's unit under /etc, deliberately stopped and disabled: starting it would put
// back a service somebody took down on purpose (novox/hq ADR 0103).
dir := t.TempDir()
m := &machine{containers: map[string]*fakeContainer{},
units: map[string]*fakeUnit{"hello.service": {active: "inactive", enabled: "disabled",
fragment: "/etc/systemd/system/hello.service"}}}
report, state := applyAdopted(t, adopted(t, untaken("hello-web.unit"),
`{"id":"hello-web.unit","type":"service","unit":"hello.service","state":"running","boot":"enabled"}`),
store.State{}, m, dir)
if o := outcomeOf(report, "hello-web.unit"); o.Action != "held" {
t.Fatalf("a unit an administrator installed was not held: %+v", o)
}
if m.did("systemctl start") || m.did("systemctl enable") {
t.Errorf("a unit somebody had stopped and disabled was started: %v", m.asked)
}
if _, ok := state.HeldAt("hello-web.unit"); !ok {
t.Error("the hold was not recorded")
}
}
func TestAPackagedUnitTheMachineUsesIsStillHeld(t *testing.T) {
// The predecessor's own service from a package, running: not the mesh's to restart.
dir := t.TempDir()
conf := filepath.Join(dir, "hello.conf")
m := &machine{containers: map[string]*fakeContainer{},
units: map[string]*fakeUnit{"nginx.service": {active: "active", enabled: "enabled",
fragment: "/usr/lib/systemd/system/nginx.service"}}}
report, _ := applyAdopted(t, adopted(t, untaken("hello-web.conf", "hello-web.unit"),
`{"id":"hello-web.conf","type":"file","path":"`+conf+`","content":"x\n"},
{"id":"hello-web.unit","type":"service","unit":"nginx.service","state":"running","boot":"enabled",
"restart-on":["hello-web.conf"]}`), store.State{}, m, dir)
if o := outcomeOf(report, "hello-web.unit"); o.Action != "held" {
t.Fatalf("a packaged unit the machine runs was not held: %+v", o)
}
if m.did("systemctl stop") || m.did("systemctl restart") {
t.Errorf("the predecessor's service was restarted: %v", m.asked)
}
}
func TestASecondOriginalAtTheSamePathIsKeptToo(t *testing.T) {
// Held, let go when the module was unassigned, rewritten by the predecessor, found again:
// both originals are kept, and the report names the one it kept (novox/hq ADR 0100).
dir := t.TempDir()
page := filepath.Join(dir, "index.html")
keep := KeepIn(dir)
first, err := keep(page, []byte("the predecessor's page\n"), 0o640)
if err != nil {
t.Fatal(err)
}
same, err := keep(page, []byte("the predecessor's page\n"), 0o640)
if err != nil || same != first {
t.Errorf("the same original was kept twice: %s %s %v", first, same, err)
}
second, err := keep(page, []byte("the predecessor wrote again\n"), 0o640)
if err != nil {
t.Fatal(err)
}
if second == first {
t.Fatal("a second original was kept under the first's name")
}
if got, _ := os.ReadFile(first); string(got) != "the predecessor's page\n" {
t.Errorf("the first original is %q", got)
}
if got, _ := os.ReadFile(second); string(got) != "the predecessor wrote again\n" {
t.Errorf("the second original is %q", got)
}
}
func TestAHoldLetGoAndFoundAgainKeepsBothOriginals(t *testing.T) {
dir, page, m := predecessor(t)
_, state := applyAdopted(t, adopted(t, untaken("hello-web.page", "hello-web.server"), webResources(page)),
store.State{}, m, dir)
first, _ := state.HeldAt("hello-web.page")
// Unassigned, then the predecessor writes again, then assigned once more.
_, state = applyAdopted(t, adopted(t, `{"taken":[]}`, withConf(dir)), state, m, dir)
if err := os.WriteFile(page, []byte("the predecessor wrote again\n"), 0o640); err != nil {
t.Fatal(err)
}
report, state := applyAdopted(t, adopted(t, untaken("hello-web.page", "hello-web.server"), webResources(page)),
state, m, dir)
again, _ := state.HeldAt("hello-web.page")
if again.Kept == first.Kept {
t.Fatalf("the second original was kept under the first's name: %s", again.Kept)
}
if got, _ := os.ReadFile(again.Kept); string(got) != "the predecessor wrote again\n" {
t.Errorf("what the report names as kept is %q", got)
}
if got, _ := os.ReadFile(first.Kept); string(got) != "the predecessor's page\n" {
t.Errorf("the first original was lost: %q", got)
}
if o := outcomeOf(report, "hello-web.page"); o.Action != "held" {
t.Errorf("the file found again was not held: %+v", o)
}
}
func TestAPathTheMeshOnlySetAccessOnIsStillFound(t *testing.T) {
// An access record says the mesh set permissions on a path it does not own — which is what it
// does to somebody else's directory. It is not a record of making it (novox/hq ADR 0103).
dir := t.TempDir()
data := filepath.Join(dir, "data")
if err := os.Mkdir(data, 0o700); err != nil {
t.Fatal(err)
}
known := store.State{Resources: []store.Applied{
{ID: "hello-web.readable", Type: "access", Target: data, Origin: store.OriginDeclared}}}
m := &machine{containers: map[string]*fakeContainer{}}
report, _ := applyAdopted(t, adopted(t, untaken("hello-web.data"),
`{"id":"hello-web.data","type":"directory","path":"`+data+`","mode":"0755"}`), known, m, dir)
if o := outcomeOf(report, "hello-web.data"); o.Action != "held" {
t.Errorf("a directory the mesh only has access for was not held: %+v", o)
}
if info, _ := os.Stat(data); info.Mode().Perm() != 0o700 {
t.Errorf("it was re-moded to %o", info.Mode().Perm())
}
}
+161
View File
@@ -0,0 +1,161 @@
package apply
// A link under a person's home is never followed as root (novox/hq ADR 0266, the review of 2026-10-08).
//
// The node-engine runs as root and places files and directories under homes: the operator's shell
// configuration, the agent's `~/.claude`. Everything below a home is the account's to change. An account that
// replaces `~/.claude` with a symbolic link to /etc, between two applies, would have root change the owner and
// mode of /etc, or write a file into it, on the next one — and on a machine whose agents run as an account of
// their own, that account is exactly the one that must not become root.
//
// So for a path strictly below a home:
//
// - **a symbolic link in any component below the home, the path's own included, refuses the resource**, said
// in words, and nothing is done to it — before anything is read, written, chowned or chmodded;
// - **the owner and mode are set through a descriptor opened without following a link**, each component
// opened from the one above it (homes_linux.go), so a link put in place after the check is not followed
// either; a link that is itself the thing to own is owned as a link, never its target;
// - **a file is written through its directory's descriptor**, made with O_EXCL|O_NOFOLLOW and renamed within
// that directory, so neither the file nor a parent can be swapped for a link between check and write.
//
// The home itself is the machine's, made by root, and is followed as it is. A path outside every home is
// handled exactly as before: a module's system paths may be links the machine set up (a resolver's file is).
//
// Which homes: every account the user database lists with a uid of 1000 or more, other than the overflow
// account, and every home under /home. A service account's home under /var/lib is a module's own directory,
// and is left as it was.
import (
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
)
// homes is every home of a person's or an agent's account on this machine, as placement_guard.go reads them
// from the user database. A variable so a test can name the homes it made.
var homes = func() []string {
var out []string
for home := range accountsOfHomes() {
out = append(out, home)
}
// In one order: the user database is read into a map, and a map's order is not one.
sort.Strings(out)
return out
}
// homeAbove is the home a path lies strictly below, the deepest when homes nest; "" when none.
func homeAbove(path string) string {
path = filepath.Clean(path)
hs := homes()
sort.Slice(hs, func(i, j int) bool { return len(hs[i]) > len(hs[j]) })
for _, h := range hs {
if strings.HasPrefix(path, h+string(os.PathSeparator)) {
return h
}
}
return ""
}
// LinkUnderHomeError is a resource refused because a component below a home is a symbolic link.
type LinkUnderHomeError struct {
Path, Link, Home string
}
func (e *LinkUnderHomeError) Error() string {
return fmt.Sprintf("%s is a symbolic link inside the home %s, and the node-engine runs as root: it never "+
"follows a link an account can put there, so nothing was done to %s (novox/hq ADR 0266). Make %s a "+
"directory or file again, or ask why something replaced it", e.Link, e.Home, e.Path, e.Link)
}
// refuseLinksUnderHome refuses a path below a home when any component below the home, its own included, is a
// symbolic link. Components that do not exist yet end the walk: what is missing is made without following.
func refuseLinksUnderHome(path string) error {
home := homeAbove(path)
if home == "" {
return nil
}
rel, err := filepath.Rel(home, filepath.Clean(path))
if err != nil {
return err
}
at := home
for _, part := range strings.Split(rel, string(os.PathSeparator)) {
at = filepath.Join(at, part)
info, err := os.Lstat(at)
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
return err
}
if info.Mode()&os.ModeSymlink != 0 {
return &LinkUnderHomeError{Path: path, Link: at, Home: home}
}
}
return nil
}
// HardLinkUnderHomeError is a regular file below a home with more than one link: the account that owns the
// home may have linked a file it does not own there, and root acting on it by name would act on that file
// (novox/hq ADR 0266). Refused, whatever the resource.
type HardLinkUnderHomeError struct {
Path string
Links uint64
}
func (e *HardLinkUnderHomeError) Error() string {
return fmt.Sprintf("%s has %d links: below a home a file with more than one is not owned, chmodded or read "+
"as root, since one of its names may be a file the home's account does not own", e.Path, e.Links)
}
// chmodDirPath is chmodPath for a directory resource: below a home, what is there must be a directory.
func chmodDirPath(path string, mode os.FileMode) error {
home := homeAbove(path)
if home == "" {
return os.Chmod(path, mode)
}
return chmodUnderAs(home, path, mode, kindDir)
}
// statPath is os.Stat, except below a home, where it never follows a link.
func statPath(path string) (os.FileInfo, error) {
if homeAbove(path) != "" {
return os.Lstat(path)
}
return os.Stat(path)
}
// chmodPath sets a mode; below a home through a descriptor that followed no link.
func chmodPath(path string, mode os.FileMode) error {
home := homeAbove(path)
if home == "" {
return os.Chmod(path, mode)
}
return chmodUnder(home, path, mode)
}
// chownPath sets an owner; below a home through a descriptor that followed no link, and a link itself is
// owned as a link.
func chownPath(path string, uid, gid int) error {
home := homeAbove(path)
if home == "" {
return os.Chown(path, uid, gid)
}
if info, err := os.Lstat(path); err == nil && info.Mode()&os.ModeSymlink != 0 {
return os.Lchown(path, uid, gid)
}
return chownUnder(home, path, uid, gid)
}
// readPath reads a file; below a home without following a link.
func readPath(path string) ([]byte, error) {
home := homeAbove(path)
if home == "" {
return os.ReadFile(path)
}
return readUnder(home, path)
}
+255
View File
@@ -0,0 +1,255 @@
//go:build linux
package apply
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0266 (the review of 2026-10-08): below a home, the node-engine — root — follows no
// symbolic link an account can put there: not to chown or chmod, not to write, not to make a directory.
// aLinkedHome is a home the test names as one, with `.claude` replaced by a link to a directory outside it,
// as an account would to have root change /etc.
func aLinkedHome(t *testing.T) (home, outside string) {
t.Helper()
root := t.TempDir()
home = filepath.Join(root, "home", "agent")
outside = filepath.Join(root, "etc")
for _, d := range []string{home, outside} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
if err := os.WriteFile(filepath.Join(outside, "shadow"), []byte("root's"), 0o600); err != nil {
t.Fatal(err)
}
was := homes
homes = func() []string { return []string{home} }
t.Cleanup(func() { homes = was })
return home, outside
}
func link(t *testing.T, target, at string) {
t.Helper()
if err := os.Symlink(target, at); err != nil {
t.Fatal(err)
}
}
func applyOneResource(t *testing.T, resource string) error {
t.Helper()
d := declare(t, resource)
_, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices, nil, nil)
return err
}
func mustBeLinkRefusal(t *testing.T, err error) {
t.Helper()
var refused *LinkUnderHomeError
if !errors.As(err, &refused) {
t.Fatalf("refused as a link under a home, got %v", err)
}
if !strings.Contains(err.Error(), "never follows a link") {
t.Fatalf("said in words: %v", err)
}
}
func modeOfPath(t *testing.T, p string) os.FileMode {
t.Helper()
info, err := os.Stat(p)
if err != nil {
t.Fatal(err)
}
return info.Mode().Perm()
}
func TestADirectoryUnderAHomeThatIsALinkIsRefusedAndItsTargetUntouched(t *testing.T) {
home, outside := aLinkedHome(t)
link(t, outside, filepath.Join(home, ".claude"))
err := applyOneResource(t, `{"id":"claude-code.agent-home","type":"directory","path":"`+
filepath.Join(home, ".claude")+`","mode":"0700"}`)
mustBeLinkRefusal(t, err)
if m := modeOfPath(t, outside); m != 0o755 {
t.Fatalf("the link's target was chmodded to %o", m)
}
}
func TestAFileThroughALinkedParentUnderAHomeIsRefused(t *testing.T) {
home, outside := aLinkedHome(t)
link(t, outside, filepath.Join(home, ".claude"))
err := applyOneResource(t, `{"id":"m.f","type":"file","path":"`+filepath.Join(home, ".claude", "shadow")+
`","content":"the mesh's\n","mode":"0644"}`)
mustBeLinkRefusal(t, err)
if got, _ := os.ReadFile(filepath.Join(outside, "shadow")); string(got) != "root's" {
t.Fatalf("written through the link: %q", got)
}
if m := modeOfPath(t, filepath.Join(outside, "shadow")); m != 0o600 {
t.Fatalf("chmodded through the link: %o", m)
}
}
func TestAFileThatIsItselfALinkUnderAHomeIsRefused(t *testing.T) {
home, outside := aLinkedHome(t)
link(t, filepath.Join(outside, "shadow"), filepath.Join(home, ".zshrc"))
err := applyOneResource(t, `{"id":"m.f","type":"file","path":"`+filepath.Join(home, ".zshrc")+
`","content":"x\n"}`)
mustBeLinkRefusal(t, err)
if got, _ := os.ReadFile(filepath.Join(outside, "shadow")); string(got) != "root's" {
t.Fatalf("written through the link: %q", got)
}
}
func TestADirectoryAndAFileUnderAHomeAreMadeAsBefore(t *testing.T) {
home, _ := aLinkedHome(t)
dir := filepath.Join(home, ".claude")
if err := applyOneResource(t, `{"id":"m.d","type":"directory","path":"`+dir+`","mode":"0700"}`); err != nil {
t.Fatal(err)
}
if m := modeOfPath(t, dir); m != 0o700 {
t.Fatalf("mode %o", m)
}
file := filepath.Join(home, ".config", "mesh", "env.sh")
if err := applyOneResource(t, `{"id":"m.f","type":"file","path":"`+file+`","content":"A=1\n","mode":"0640"}`); err != nil {
t.Fatal(err)
}
if got, _ := os.ReadFile(file); string(got) != "A=1\n" || modeOfPath(t, file) != 0o640 {
t.Fatalf("written %q mode %o", got, modeOfPath(t, file))
}
}
// The descriptor half: a link put in place after any check is still not followed.
func TestTheDescriptorCallsFollowNoLinkBelowAHome(t *testing.T) {
home, outside := aLinkedHome(t)
link(t, outside, filepath.Join(home, ".claude"))
mustBeLinkRefusal(t, chmodPath(filepath.Join(home, ".claude"), 0o777))
mustBeLinkRefusal(t, chmodPath(filepath.Join(home, ".claude", "shadow"), 0o777))
if m := modeOfPath(t, filepath.Join(outside, "shadow")); m != 0o600 {
t.Fatalf("chmodded through the link: %o", m)
}
mustBeLinkRefusal(t, writeAtomically(filepath.Join(home, ".claude", "new"), []byte("x"), 0o644))
if _, err := os.Stat(filepath.Join(outside, "new")); !os.IsNotExist(err) {
t.Fatal("written through the link")
}
if _, err := makeDirsSaying(filepath.Join(home, ".claude", "a", "b"), 0o755, ""); err == nil {
t.Fatal("made directories through the link")
}
if _, err := os.Stat(filepath.Join(outside, "a")); !os.IsNotExist(err) {
t.Fatal("made a directory through the link")
}
if _, err := readPath(filepath.Join(home, ".claude", "shadow")); err == nil {
t.Fatal("read through the link")
}
me := os.Getuid()
// A link itself is owned as a link, never its target.
if err := chownPath(filepath.Join(home, ".claude"), me, os.Getgid()); err != nil {
t.Fatalf("a link is owned as a link: %v", err)
}
}
func TestRemovalThroughALinkUnderAHomeIsRefused(t *testing.T) {
home, outside := aLinkedHome(t)
link(t, outside, filepath.Join(home, ".claude"))
_, _, err := remove(context.Background(), archHost(t), store.Applied{ID: "m.f", Type: "file",
Target: filepath.Join(home, ".claude", "shadow")}, nil, nil)
mustBeLinkRefusal(t, err)
if _, err := os.Stat(filepath.Join(outside, "shadow")); err != nil {
t.Fatal("removed through the link")
}
}
func TestAPathOutsideEveryHomeIsHandledAsBefore(t *testing.T) {
_, outside := aLinkedHome(t)
elsewhere := filepath.Join(filepath.Dir(outside), "srv")
if err := os.MkdirAll(elsewhere, 0o755); err != nil {
t.Fatal(err)
}
link(t, outside, filepath.Join(elsewhere, "linked"))
if homeAbove(filepath.Join(elsewhere, "linked", "x")) != "" {
t.Fatal("not below a home")
}
if err := refuseLinksUnderHome(filepath.Join(elsewhere, "linked", "x")); err != nil {
t.Fatal("a system path may be a link the machine set up; only a home's are refused")
}
}
func TestHomesAreAPersonsOrAnAgentsNotAServicesOrRoots(t *testing.T) {
dir := t.TempDir()
passwd := filepath.Join(dir, "passwd")
if err := os.WriteFile(passwd, []byte("root:x:0:0::/root:/bin/bash\n"+
"postgres:x:968:968::/var/lib/postgres:/usr/bin/nologin\n"+
"nobody:x:65534:65534::/:/usr/bin/nologin\n"+
"operator:x:1000:1000::/home/operator:/bin/zsh\n"+
"agent:x:1001:1001::/home/agent:/bin/bash\n"+
"svc:x:990:990::/home/svc:/usr/bin/nologin\n"), 0o644); err != nil {
t.Fatal(err)
}
was := passwdFile
passwdFile = passwd
t.Cleanup(func() { passwdFile = was })
got := strings.Join(homes(), ",")
if got != "/home/agent,/home/operator,/home/svc" {
t.Fatalf("homes %s", got)
}
if homeAbove("/home/agent/.claude") != "/home/agent" || homeAbove("/home/agent") != "" ||
homeAbove("/var/lib/postgres/data") != "" || homeAbove("/root/.ssh") != "" {
t.Fatal("strictly below a person's or an agent's home, and nothing else")
}
}
// A hard link below a home to a file the home's account does not own is never owned, chmodded or read by
// name: fstat on the opened descriptor counts its links first (the re-review of 2026-10-08).
func TestAHardLinkedFileBelowAHomeIsRefused(t *testing.T) {
home, outside := aLinkedHome(t)
shadow := filepath.Join(outside, "shadow")
linked := filepath.Join(home, ".claude")
if err := os.Link(shadow, linked); err != nil {
t.Skipf("no hard link here: %v", err)
}
var hard *HardLinkUnderHomeError
if err := chmodPath(linked, 0o777); !errors.As(err, &hard) {
t.Fatalf("chmod of a hard-linked file below a home is refused, got %v", err)
}
if err := chownPath(linked, os.Getuid(), os.Getgid()); !errors.As(err, &hard) {
t.Fatalf("chown of it is refused, got %v", err)
}
if _, err := readPath(linked); !errors.As(err, &hard) {
t.Fatalf("reading it is refused, got %v", err)
}
if m := modeOfPath(t, shadow); m != 0o600 {
t.Fatalf("the other file's mode changed: %o", m)
}
// As the directory resource sees it: a file where ~/.claude should be is left alone.
if err := chmodDirPath(linked, 0o700); err == nil || !strings.Contains(err.Error(), "where a directory was expected") {
t.Fatalf("a file where a directory was expected is refused, got %v", err)
}
}
func TestADirectoryWhereAFileWasExpectedAndAFifoAreRefused(t *testing.T) {
home, _ := aLinkedHome(t)
dir := filepath.Join(home, "d")
if err := os.Mkdir(dir, 0o755); err != nil {
t.Fatal(err)
}
if _, err := readPath(dir); err == nil || !strings.Contains(err.Error(), "where a file was expected") {
t.Fatalf("a directory where a file was expected is refused, got %v", err)
}
fifo := filepath.Join(home, "f")
if err := syscall.Mkfifo(fifo, 0o600); err != nil {
t.Skipf("no fifo here: %v", err)
}
if err := chmodPath(fifo, 0o644); err == nil || !strings.Contains(err.Error(), "neither a file nor a directory") {
t.Fatalf("a fifo is refused, got %v", err)
}
if err := chmodDirPath(dir, 0o700); err != nil {
t.Fatalf("a directory as a directory passes: %v", err)
}
}
+126
View File
@@ -0,0 +1,126 @@
package apply
import (
"context"
"os"
osuser "os/user"
"path/filepath"
"sort"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0182 and to-be 41: a parent the host makes inside an owner's home is the
// owner's, one that was there is held as found, and one outside the home is made as before.
// aHome gives the account running the test a home in a directory the test owns, and writes down
// every directory the host gives to whom. The account's own name, so what the host chowns resolves
// without being root; the record, so what was given is told apart from what was merely made.
func aHome(t *testing.T) (home, owner string, given map[string]string) {
t.Helper()
me, err := osuser.Current()
if err != nil {
t.Skip("no current user to own anything")
}
home = t.TempDir()
given = map[string]string{}
wasHome, wasOwn := homeOf, ownMade
homeOf = func(name string) (string, error) {
if name == me.Username {
return home, nil
}
return wasHome(name)
}
ownMade = func(path, owner string) error {
given[path] = owner
return wasOwn(path, owner)
}
t.Cleanup(func() { homeOf, ownMade = wasHome, wasOwn })
return home, me.Username, given
}
func givenPaths(given map[string]string) []string {
var paths []string
for p := range given {
paths = append(paths, p)
}
sort.Strings(paths)
return paths
}
func TestAFileUnderAHomeGivesTheParentsItMadeToItsOwner(t *testing.T) {
home, owner, given := aHome(t)
target := filepath.Join(home, ".config", "mesh", "environment.sh")
d := parse(t, `{"declaration":1,"resources":[{"id":"shell.env","type":"file","path":"`+target+
`","content":"export A=1\n","owner":"`+owner+`"}]}`)
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
noServices, nil, nil); err != nil {
t.Fatal(err)
}
want := []string{filepath.Join(home, ".config"), filepath.Join(home, ".config", "mesh")}
if got := givenPaths(given); strings.Join(got, ",") != strings.Join(want, ",") {
t.Errorf("given to the owner: %v, want %v", got, want)
}
for _, p := range want {
if given[p] != owner {
t.Errorf("%s given to %q", p, given[p])
}
}
}
func TestAnArchiveUnderAHomeGivesTheParentsItMadeToItsOwner(t *testing.T) {
home, owner, given := aHome(t)
body, digest := anArchive(t, map[string]string{"p10k.zsh": "theme"})
target := filepath.Join(home, ".local", "share", "powerlevel10k")
d := declare(t, `{"id":"shell.theme","type":"archive","source":"`+serving(t, body)+
`","digest":"`+digest+`","path":"`+target+`","owner":"`+owner+`"}`)
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
noServices, nil, nil); err != nil {
t.Fatal(err)
}
for _, p := range []string{filepath.Join(home, ".local"), filepath.Join(home, ".local", "share")} {
if given[p] != owner {
t.Errorf("%s, made by the host, was not given to the owner: %v", p, givenPaths(given))
}
}
}
func TestAParentThatWasThereIsHeldAsFound(t *testing.T) {
home, owner, given := aHome(t)
config := filepath.Join(home, ".config")
if err := os.Mkdir(config, 0o700); err != nil {
t.Fatal(err)
}
target := filepath.Join(config, "mesh", "environment.sh")
d := parse(t, `{"declaration":1,"resources":[{"id":"shell.env","type":"file","path":"`+target+
`","content":"export A=1\n","owner":"`+owner+`"}]}`)
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
noServices, nil, nil); err != nil {
t.Fatal(err)
}
if _, touched := given[config]; touched {
t.Error("a parent that was already there was given to the owner")
}
if info, _ := os.Stat(config); info.Mode().Perm() != 0o700 {
t.Errorf("a parent that was already there changed mode: %o", info.Mode().Perm())
}
if given[filepath.Join(config, "mesh")] != owner {
t.Errorf("the parent the host made was not given to the owner: %v", givenPaths(given))
}
}
func TestAParentOutsideTheHomeIsMadeAsBefore(t *testing.T) {
_, owner, given := aHome(t)
target := filepath.Join(t.TempDir(), "var", "lib", "module", "settings.conf")
d := parse(t, `{"declaration":1,"resources":[{"id":"module.conf","type":"file","path":"`+target+
`","content":"a=1\n","owner":"`+owner+`"}]}`)
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
noServices, nil, nil); err != nil {
t.Fatal(err)
}
if len(given) != 0 {
t.Errorf("parents outside the owner's home were given to it: %v", givenPaths(given))
}
}
+269
View File
@@ -0,0 +1,269 @@
//go:build linux
package apply
// The descriptor half of home_links.go: below a home, every component is opened from the one above it with
// O_NOFOLLOW, so a link is refused by the kernel at the moment of use, not only at the check before it.
import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"golang.org/x/sys/unix"
)
// openDirUnder opens the directory rel names below home, following no link below the home.
func openDirUnder(home, dir string) (int, error) {
rel, err := filepath.Rel(home, filepath.Clean(dir))
if err != nil || strings.HasPrefix(rel, "..") {
return -1, fmt.Errorf("%s is not below %s", dir, home)
}
fd, err := unix.Open(home, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
if err != nil {
return -1, &os.PathError{Op: "open", Path: home, Err: err}
}
if rel == "." {
return fd, nil
}
at := home
for _, part := range strings.Split(rel, string(os.PathSeparator)) {
at = filepath.Join(at, part)
next, err := unix.Openat(fd, part, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)
unix.Close(fd)
if err != nil {
return -1, linkOr(at, home, dir, "open", err)
}
fd = next
}
return fd, nil
}
// linkOr says a refused link in the mesh's words, and any other failure as the system's.
func linkOr(at, home, path, op string, err error) error {
if errors.Is(err, unix.ELOOP) || errors.Is(err, unix.ENOTDIR) {
if info, lerr := os.Lstat(at); lerr == nil && info.Mode()&os.ModeSymlink != 0 {
return &LinkUnderHomeError{Path: path, Link: at, Home: home}
}
}
return &os.PathError{Op: op, Path: at, Err: err}
}
// What a caller expects to find at a path below a home: either, a directory, or a regular file.
const (
kindAny = iota
kindDir
kindFile
)
// openUnder opens the file or directory at path below home, following no link, for its metadata.
func openUnder(home, path string) (int, error) { return openUnderAs(home, path, kindAny) }
// openUnderAs opens path below home as what the caller expects, and refuses what root must not act on
// (novox/hq ADR 0266): a directory is opened with O_DIRECTORY|O_NOFOLLOW; whatever was opened is judged by
// fstat on the descriptor itself, before any fchown, fchmod or read — so a name swapped after a check is
// judged as what it now is. Refused: anything but a directory or a regular file (a device, a fifo, a
// socket), a kind other than the one expected, and a regular file with more than one link. The account that
// owns the home can hard-link a file it does not own (root's, where fs.protected_hardlinks is off) into its
// home; owned or chmodded by name, root would hand that file over.
func openUnderAs(home, path string, want int) (int, error) {
dir, err := openDirUnder(home, filepath.Dir(path))
if err != nil {
return -1, err
}
defer unix.Close(dir)
base := filepath.Base(path)
flags := unix.O_RDONLY | unix.O_NOFOLLOW | unix.O_NONBLOCK | unix.O_CLOEXEC
fd := -1
if want != kindFile {
fd, err = unix.Openat(dir, base, flags|unix.O_DIRECTORY, 0)
if errors.Is(err, unix.ENOTDIR) && want == kindAny {
fd, err = unix.Openat(dir, base, flags, 0)
}
} else {
fd, err = unix.Openat(dir, base, flags, 0)
}
if err != nil {
err = linkOr(path, home, path, "open", err)
var link *LinkUnderHomeError
if want == kindDir && !errors.As(err, &link) && errors.Is(err, unix.ENOTDIR) {
return -1, fmt.Errorf("%s is not a directory where a directory was expected: below a home it is left alone", path)
}
return -1, err
}
if err := judgeOpened(fd, path, want); err != nil {
unix.Close(fd)
return -1, err
}
return fd, nil
}
// judgeOpened is openUnderAs's verdict on what the descriptor holds.
func judgeOpened(fd int, path string, want int) error {
var st unix.Stat_t
if err := unix.Fstat(fd, &st); err != nil {
return &os.PathError{Op: "fstat", Path: path, Err: err}
}
switch st.Mode & unix.S_IFMT {
case unix.S_IFDIR:
if want == kindFile {
return fmt.Errorf("%s is a directory where a file was expected: below a home it is left alone", path)
}
case unix.S_IFREG:
if want == kindDir {
return fmt.Errorf("%s is a file where a directory was expected: below a home it is left alone", path)
}
if st.Nlink > 1 {
return &HardLinkUnderHomeError{Path: path, Links: uint64(st.Nlink)}
}
default:
return fmt.Errorf("%s is neither a file nor a directory: below a home it is left alone", path)
}
return nil
}
func chmodUnder(home, path string, mode os.FileMode) error {
return chmodUnderAs(home, path, mode, kindAny)
}
func chmodUnderAs(home, path string, mode os.FileMode, want int) error {
fd, err := openUnderAs(home, path, want)
if err != nil {
return err
}
defer unix.Close(fd)
if err := unix.Fchmod(fd, uint32(mode.Perm())); err != nil {
return &os.PathError{Op: "chmod", Path: path, Err: err}
}
return nil
}
func chownUnder(home, path string, uid, gid int) error {
fd, err := openUnder(home, path)
if err != nil {
return err
}
defer unix.Close(fd)
if err := unix.Fchown(fd, uid, gid); err != nil {
return &os.PathError{Op: "chown", Path: path, Err: err}
}
return nil
}
func readUnder(home, path string) ([]byte, error) {
fd, err := openUnderAs(home, path, kindFile)
if err != nil {
return nil, err
}
f := os.NewFile(uintptr(fd), path)
defer f.Close()
var st unix.Stat_t
if err := unix.Fstat(fd, &st); err == nil && st.Mode&unix.S_IFMT != unix.S_IFREG {
return nil, fmt.Errorf("%s is not a regular file", path)
}
return io.ReadAll(f)
}
// mkdirAllUnder makes the directories missing below home down to dir, each made in its parent's descriptor
// and opened without following a link; answers those it made, deepest first, as makeDirsSaying does.
func mkdirAllUnder(home, dir string, mode os.FileMode) ([]string, error) {
rel, err := filepath.Rel(home, filepath.Clean(dir))
if err != nil || strings.HasPrefix(rel, "..") {
return nil, fmt.Errorf("%s is not below %s", dir, home)
}
fd, err := unix.Open(home, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
if err != nil {
return nil, &os.PathError{Op: "open", Path: home, Err: err}
}
defer func() { unix.Close(fd) }()
var made []string
if rel == "." {
return nil, nil
}
at := home
for _, part := range strings.Split(rel, string(os.PathSeparator)) {
at = filepath.Join(at, part)
if err := unix.Mkdirat(fd, part, uint32(mode.Perm())); err == nil {
made = append([]string{at}, made...)
} else if !errors.Is(err, unix.EEXIST) {
return made, &os.PathError{Op: "mkdir", Path: at, Err: err}
}
next, err := unix.Openat(fd, part, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)
if err != nil {
return made, linkOr(at, home, dir, "open", err)
}
unix.Close(fd)
fd = next
}
return made, nil
}
// writeUnder writes a file below home atomically, through its directory's descriptor: made O_EXCL|O_NOFOLLOW
// under a name of its own, given its mode, and renamed over the file within that directory.
func writeUnder(home, path string, content []byte, mode os.FileMode) error {
dir, err := openDirUnder(home, filepath.Dir(path))
if err != nil {
return err
}
defer unix.Close(dir)
name := ".mesh-host-" + strconv.FormatInt(time.Now().UnixNano(), 36) + "-" + strconv.Itoa(os.Getpid())
fd, err := unix.Openat(dir, name, unix.O_WRONLY|unix.O_CREAT|unix.O_EXCL|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0o600)
if err != nil {
return &os.PathError{Op: "create", Path: filepath.Join(filepath.Dir(path), name), Err: err}
}
f := os.NewFile(uintptr(fd), name)
_, werr := f.Write(content)
if werr == nil {
werr = f.Sync()
}
if werr == nil {
if err := unix.Fchmod(fd, uint32(mode.Perm())); err != nil {
werr = &os.PathError{Op: "chmod", Path: path, Err: err}
}
}
if cerr := f.Close(); werr == nil {
werr = cerr
}
if werr == nil {
if err := unix.Renameat(dir, name, dir, filepath.Base(path)); err != nil {
werr = &os.PathError{Op: "rename", Path: path, Err: err}
}
}
if werr != nil {
_ = unix.Unlinkat(dir, name, 0)
}
return werr
}
// giveHandedOver gives a directory the operator handed to the mesh its declared mode and owner (novox/hq issue
// 356, issue 339) through one descriptor opened from / with no link followed on any component (openUnderAs with /
// as the home), judged a directory by fstat on that descriptor, and changed with fchmod and fchown on it. The
// hand-over refused a link when it was recorded; this refuses one at the moment of use, so a component swapped for
// a link between the record and the apply cannot carry root's chmod and chown to wherever it points. Returns the
// mode and owner the directory had before.
func giveHandedOver(path string, mode os.FileMode, uid, gid int, chown bool) (before os.FileMode, beforeUID, beforeGID int, err error) {
fd, err := openUnderAs("/", path, kindDir)
if err != nil {
return 0, 0, 0, err
}
defer unix.Close(fd)
var st unix.Stat_t
if err := unix.Fstat(fd, &st); err != nil {
return 0, 0, 0, &os.PathError{Op: "fstat", Path: path, Err: err}
}
before, beforeUID, beforeGID = os.FileMode(st.Mode&0o777), int(st.Uid), int(st.Gid)
if err := unix.Fchmod(fd, uint32(mode.Perm())); err != nil {
return 0, 0, 0, &os.PathError{Op: "chmod", Path: path, Err: err}
}
if chown && (beforeUID != uid || beforeGID != gid) {
if err := unix.Fchown(fd, uid, gid); err != nil {
return 0, 0, 0, &os.PathError{Op: "chown", Path: path, Err: err}
}
}
return before, beforeUID, beforeGID, nil
}
+84
View File
@@ -0,0 +1,84 @@
//go:build !linux
package apply
// Where there is no openat with O_NOFOLLOW to rely on, the check before use is all: a link below a home is
// refused, and the call is made by path. The node-engine runs on Linux; this keeps the package building.
import (
"os"
"path/filepath"
"github.com/novox/mesh-host/internal/declaration"
)
func chmodUnder(home, path string, mode os.FileMode) error {
if err := refuseLinksUnderHome(path); err != nil {
return err
}
return os.Chmod(path, mode)
}
func chmodUnderAs(home, path string, mode os.FileMode, _ int) error {
return chmodUnder(home, path, mode)
}
const kindDir = 1
func chownUnder(home, path string, uid, gid int) error {
if err := refuseLinksUnderHome(path); err != nil {
return err
}
return os.Lchown(path, uid, gid)
}
func readUnder(home, path string) ([]byte, error) {
if err := refuseLinksUnderHome(path); err != nil {
return nil, err
}
return os.ReadFile(path)
}
func mkdirAllUnder(home, dir string, mode os.FileMode) ([]string, error) {
if err := refuseLinksUnderHome(dir); err != nil {
return nil, err
}
var made []string
for d := filepath.Clean(dir); d != home; d = filepath.Dir(d) {
if _, err := os.Lstat(d); err == nil {
break
}
made = append(made, d)
}
return made, os.MkdirAll(dir, mode)
}
func writeUnder(home, path string, content []byte, mode os.FileMode) error {
if err := refuseLinksUnderHome(path); err != nil {
return err
}
return writeAtomicallyByPath(path, content, mode)
}
// giveHandedOver is refused where there is no openat with O_NOFOLLOW: a handed-over directory is never changed by
// path (novox/hq issue 356). The node-engine runs on Linux.
func giveHandedOver(path string, _ os.FileMode, _, _ int, _ bool) (os.FileMode, int, int, error) {
return 0, 0, 0, &PlacementRefusedError{Path: path, Why: "a handed-over directory is changed only through a " +
"descriptor that followed no link, which this system cannot open"}
}
// judgeMeshDir refuses where there is no openat with O_NOFOLLOW (novox/hq issue 358). The node-engine runs on
// Linux.
func judgeMeshDir(path string) error {
return &PlacementRefusedError{Path: path, Why: "a mesh directory is judged only by a walk that follows no link " +
"another account could make, which this system cannot do"}
}
// meshDirsByDescriptor is false where there is no openat with O_NOFOLLOW: a mesh directory is changed by path, as
// before novox/hq issue 358. The node-engine runs on Linux.
const meshDirsByDescriptor = false
func applyMeshDirectory(r *declaration.Directory, out Outcome, _ os.FileMode, _ bool) (Outcome, error) {
return out, &PlacementRefusedError{Path: r.Path, Why: "a mesh directory is changed only through a descriptor " +
"that followed no link another account could make, which this system cannot open"}
}
+382
View File
@@ -0,0 +1,382 @@
package apply
import (
"context"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// Every module is given an identity of its own (novox/hq ADR 0328 §3, §4, §15, issue 567).
//
// The controller allocates each module a user and a group number once, from a range of its own, the same on every
// machine, and declares them as `mesh-<module>`. The engine makes the group and the account with exactly those
// numbers — a system account: no login shell, no home, no password — before anything of the declaration is
// applied, so a directory the module owns can be given to it and a container told to run as it.
//
// **Never root** (§11): a number that is 0, or a gid a root group has here (by its number and by its name), is
// refused for the whole declaration before anything is made or started, naming why — for an account and for an
// image alike. **Never another's**: a name that exists with other numbers, or numbers another name holds, is
// refused for that identity alone, naming whose, and its module is not attempted; the rest of the machine is.
//
// **What it made, and only that, goes** when the identity is no longer declared: the record says whether the
// engine made the account, the group or both, and an account or a group that was there before is never removed.
// accountsTool is how the engine reads and changes the machine's accounts: the system's own tools through the
// runner, which is what a test stands in for.
type accountsTool interface {
CreateSystemAccount(ctx context.Context, run system.Runner, name string, uid, gid int, groups []string) error
ModifyAccount(ctx context.Context, run system.Runner, name string, gid int, groups []string) error
CreateSystemGroup(ctx context.Context, run system.Runner, name string, gid int) error
RemoveAccount(ctx context.Context, run system.Runner, name string) error
RemoveGroup(ctx context.Context, run system.Runner, name string) error
}
// entry is one line the user or group database gives for a key: its name and its numbers.
type entry struct {
name string
id, prim int
}
// lookUp asks the database (passwd or group) for key, a name or a number. Absent is (entry{}, false, nil); a
// database that does not answer is an error, never read as absent (system.LookUpUser's rule).
func lookUp(ctx context.Context, run Runner, database, key string) (entry, bool, error) {
out, err := run(ctx, "getent", database, key)
if err != nil {
if code, ok := system.ExitCode(err); ok && code == 2 {
return entry{}, false, nil
}
return entry{}, false, fmt.Errorf("the %s database did not answer about %s: %w", database, key, err)
}
fields := strings.Split(strings.TrimSpace(strings.SplitN(out, "\n", 2)[0]), ":")
if len(fields) < 3 {
return entry{}, false, fmt.Errorf("the %s database gave %q for %s", database, strings.TrimSpace(out), key)
}
e := entry{name: fields[0], prim: -1}
if e.id, err = strconv.Atoi(fields[2]); err != nil {
return entry{}, false, fmt.Errorf("the %s database gave %q for %s", database, strings.TrimSpace(out), key)
}
if database == "passwd" && len(fields) >= 4 {
if e.prim, err = strconv.Atoi(fields[3]); err != nil {
e.prim = -1
}
}
return e, true, nil
}
// refuseRootIdentities is every identity and every container's run-as that is root's (§11), judged before
// anything of the declaration is applied: one is enough to refuse it whole, so nothing is made or started.
func refuseRootIdentities(d *declaration.Declaration) error {
var problems []string
for _, id := range d.Identities {
// Its user's number, its own group's (the same number), its primary group and every added group.
if err := rootIdentity(id.UID, id.UID); err != nil {
problems = append(problems, fmt.Sprintf("the identity %s: %v", id.Name, err))
}
for _, gid := range append([]int{id.GID}, id.Groups...) {
if why := rootGroup(gid); why != "" {
problems = append(problems, fmt.Sprintf("the identity %s is given gid %s, and a module's identity "+
"is never in a root group (novox/hq ADR 0328 §15)", id.Name, why))
}
}
}
for _, g := range d.Groups {
if why := rootGroup(g.GID); why != "" {
problems = append(problems, fmt.Sprintf("the library group %s is gid %s, and a library's group is never a "+
"root group (novox/hq ADR 0328 §15)", g.Name, why))
}
}
for _, r := range d.Resources {
if c, ok := r.(*declaration.Container); ok && c.RunAs != nil {
if err := rootIdentity(c.RunAs.UID, c.RunAs.GID); err != nil {
problems = append(problems, fmt.Sprintf("the container %s is told to run as uid %d, gid %d: %v",
c.ID, c.RunAs.UID, c.RunAs.GID, err))
}
}
}
if len(problems) == 0 {
return nil
}
sort.Strings(problems)
return fmt.Errorf("%s; nothing of this declaration was made or started", strings.Join(problems, "; "))
}
// applyIdentity makes one identity as declared (ADR 0328 §4): its own group `mesh-<module>` with the number of its
// uid, and its user with that uid, its primary group gid (its own, or its first library's) and its other libraries'
// groups as added groups — or refuses it naming whose the name or the numbers are. A user the engine made is given
// a changed primary or added groups; one that was there before is never changed, and is refused if it is not as
// declared. What it made is recorded on known; an identity already right is unchanged.
func applyIdentity(ctx context.Context, tool accountsTool, id declaration.Identity, names map[int]string,
known *store.State, run Runner) (Outcome, error) {
out := Outcome{ID: "identity " + id.Name, Type: "identity", Target: id.Name, Action: "unchanged"}
user, hasUser, err := lookUp(ctx, run, "passwd", id.Name)
if err != nil {
return out, err
}
group, hasGroup, err := lookUp(ctx, run, "group", id.Name)
if err != nil {
return out, err
}
byUID, uidUsed, err := lookUp(ctx, run, "passwd", strconv.Itoa(id.UID))
if err != nil {
return out, err
}
byGID, gidUsed, err := lookUp(ctx, run, "group", strconv.Itoa(id.UID))
if err != nil {
return out, err
}
rec := madeIdentity(known, id.Name)
const never = "; nothing was made, and nothing of the module was attempted (novox/hq ADR 0328 §4)"
switch {
case hasUser && user.id != id.UID:
return out, fmt.Errorf("the account %s is here with uid %d, and the mesh allocated uid %d%s", id.Name,
user.id, id.UID, never)
case hasUser && user.prim != id.GID && !rec.MadeUser:
return out, fmt.Errorf("the account %s is here with group %d, was not made by the mesh, and the mesh "+
"gives it group %d%s", id.Name, user.prim, id.GID, never)
case hasGroup && group.id != id.UID:
return out, fmt.Errorf("the group %s is here with gid %d, and a module's own group has its uid, %d%s",
id.Name, group.id, id.UID, never)
case uidUsed && byUID.name != id.Name:
return out, fmt.Errorf("uid %d is the account %s's here, and the mesh allocated it to %s%s", id.UID,
byUID.name, id.Name, never)
case gidUsed && byGID.name != id.Name:
return out, fmt.Errorf("gid %d is the group %s's here, and the mesh allocated it to %s%s", id.UID,
byGID.name, id.Name, never)
}
added := make([]string, 0, len(id.Groups))
for _, gid := range id.Groups {
added = append(added, names[gid])
}
sort.Strings(added)
rec.Module, rec.Name, rec.UID, rec.GID = id.Module, id.Name, id.UID, id.GID
var made []string
if !hasGroup {
if err := tool.CreateSystemGroup(ctx, run, id.Name, id.UID); err != nil {
return out, err
}
rec.MadeGroup = true
made = append(made, fmt.Sprintf("its own group (gid %d)", id.UID))
}
if !hasUser {
if err := tool.CreateSystemAccount(ctx, run, id.Name, id.UID, id.GID, added); err != nil {
recordIdentity(known, rec)
return out, err
}
rec.MadeUser = true
made = append(made, fmt.Sprintf("the account (uid %d, group %d%s), with no login, no home and no password",
id.UID, id.GID, alsoIn(added)))
} else if rec.MadeUser {
// The mesh's own account: its primary and added groups follow what is declared.
in, err := system.GroupsOf(ctx, system.Runner(run), id.Name)
if err != nil {
return out, err
}
have := map[string]bool{}
for _, g := range in {
if g != id.Name && g != names[user.prim] {
have[g] = true
}
}
same := user.prim == id.GID && len(have) == len(added)
for _, g := range added {
same = same && have[g]
}
if !same {
if err := tool.ModifyAccount(ctx, run, id.Name, id.GID, added); err != nil {
return out, err
}
out.Action = "updated"
out.Detail = fmt.Sprintf("gave the account group %d%s", id.GID, alsoIn(added))
}
}
if rec.At.IsZero() {
rec.At = time.Now().UTC()
}
recordIdentity(known, rec)
if len(made) > 0 {
out.Action, out.Detail = "created", "made "+strings.Join(made, " and ")
} else if out.Detail == "" {
out.Detail = "here with its numbers"
}
return out, nil
}
func alsoIn(groups []string) string {
if len(groups) == 0 {
return ""
}
return ", also in " + strings.Join(groups, ", ")
}
// applyGroup makes one library's group with the number the controller allocated (ADR 0328 §3), or refuses it naming
// whose the name or the number is, or that the number is a root group's (§15). What it made is recorded on known.
func applyGroup(ctx context.Context, tool accountsTool, g declaration.Group, known *store.State,
run Runner) (Outcome, error) {
out := Outcome{ID: "group " + g.Name, Type: "group", Target: g.Name, Action: "unchanged"}
if why := rootGroup(g.GID); why != "" {
return out, fmt.Errorf("its gid is %s, and a library's group is never a root group (novox/hq ADR 0328 §15); "+
"nothing was made", why)
}
byName, hasName, err := lookUp(ctx, run, "group", g.Name)
if err != nil {
return out, err
}
byGID, gidUsed, err := lookUp(ctx, run, "group", strconv.Itoa(g.GID))
if err != nil {
return out, err
}
switch {
case hasName && byName.id != g.GID:
return out, fmt.Errorf("the group %s is here with gid %d, and the mesh allocated gid %d; nothing was made",
g.Name, byName.id, g.GID)
case gidUsed && byGID.name != g.Name:
return out, fmt.Errorf("gid %d is the group %s's here, and the mesh allocated it to %s; nothing was made",
g.GID, byGID.name, g.Name)
case hasName:
out.Detail = "here with its number"
return out, nil
}
if err := tool.CreateSystemGroup(ctx, run, g.Name, g.GID); err != nil {
return out, err
}
known.Groups = append(known.Groups, store.MadeGroup{Name: g.Name, GID: g.GID, At: time.Now().UTC()})
out.Action, out.Detail = "created", fmt.Sprintf("made the library's group (gid %d)", g.GID)
return out, nil
}
// removeGroups removes every library group the engine made that is no longer declared, while it still has the
// number the engine gave it. Never fatal: one that could not be removed (still a user's primary group) stays
// recorded, said, for the next apply.
func removeGroups(ctx context.Context, tool accountsTool, declared []declaration.Group, known *store.State,
run Runner) []Outcome {
wanted := map[string]bool{}
for _, g := range declared {
wanted[g.Name] = true
}
var out []Outcome
var keep []store.MadeGroup
for _, m := range known.Groups {
if wanted[m.Name] {
keep = append(keep, m)
continue
}
o := Outcome{ID: "group " + m.Name, Type: "group", Target: m.Name, Action: "removed",
Detail: "no longer declared: removed the group"}
g, has, err := lookUp(ctx, run, "group", m.Name)
switch {
case err != nil || (has && g.id != m.GID):
o.Action, o.Detail = "forgotten", "no longer declared: the group was left, it no longer has the gid the "+
"mesh gave it, or the database did not answer"
case has:
if err := tool.RemoveGroup(ctx, run, m.Name); err != nil {
o.Action, o.Detail = "waiting", "no longer declared: the group could not be removed: "+err.Error()
keep = append(keep, m)
}
default:
o.Action, o.Detail = "forgotten", "no longer declared, and no longer here"
}
out = append(out, o)
}
known.Groups = keep
return out
}
// madeIdentity is what the engine recorded of name, or an empty record.
func madeIdentity(known *store.State, name string) store.MadeIdentity {
for _, m := range known.Identities {
if m.Name == name {
return m
}
}
return store.MadeIdentity{}
}
// recordIdentity records what was made of one identity, in place of any record before.
func recordIdentity(known *store.State, rec store.MadeIdentity) {
for i, m := range known.Identities {
if m.Name == rec.Name {
known.Identities[i] = rec
return
}
}
if rec.MadeUser || rec.MadeGroup {
known.Identities = append(known.Identities, rec)
}
}
// removeIdentities removes what the engine made of every identity no longer declared — the account, then the
// group — and only while it still has the numbers the engine gave it. Never fatal: what could not be removed
// stays recorded, said, for the next apply.
func removeIdentities(ctx context.Context, tool accountsTool, declared []declaration.Identity, known *store.State,
run Runner) []Outcome {
wanted := map[string]bool{}
for _, id := range declared {
wanted[id.Name] = true
}
var out []Outcome
var keep []store.MadeIdentity
for _, m := range known.Identities {
if wanted[m.Name] {
keep = append(keep, m)
continue
}
o := Outcome{ID: "identity " + m.Name, Type: "identity", Target: m.Name, Action: "removed"}
var said []string
if m.MadeUser {
if u, has, err := lookUp(ctx, run, "passwd", m.Name); err != nil || (has && u.id != m.UID) {
said = append(said, "the account was left: it no longer has the uid the mesh gave it, or the "+
"database did not answer")
m.MadeUser = false
} else if has {
if err := tool.RemoveAccount(ctx, run, m.Name); err != nil {
said = append(said, "the account could not be removed: "+err.Error())
} else {
m.MadeUser = false
said = append(said, "removed the account")
}
} else {
m.MadeUser = false
}
}
if m.MadeGroup && !m.MadeUser {
if g, has, err := lookUp(ctx, run, "group", m.Name); err != nil || (has && g.id != m.GID) {
said = append(said, "the group was left: it no longer has the gid the mesh gave it, or the "+
"database did not answer")
m.MadeGroup = false
} else if has {
if err := tool.RemoveGroup(ctx, run, m.Name); err != nil {
said = append(said, "the group could not be removed: "+err.Error())
} else {
m.MadeGroup = false
said = append(said, "removed the group")
}
} else {
m.MadeGroup = false
}
}
if m.MadeUser || m.MadeGroup {
o.Action = "waiting"
keep = append(keep, m)
}
o.Detail = "no longer declared: " + strings.Join(said, "; ")
out = append(out, o)
}
known.Identities = keep
return out
}
// toolFor is the accounts tool of a system, or why it has none.
func toolFor(sys system.System) (accountsTool, error) {
if t, ok := sys.(accountsTool); ok {
return t, nil
}
return nil, errors.New("this system makes no module identity: it has no way to make a system account")
}
+385
View File
@@ -0,0 +1,385 @@
package apply
import (
"context"
"errors"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// The engine's half of ADR 0325 §3 and §11 (issue 567): every module given an account and a group of its own with
// the numbers the controller allocated, a container told to run as it, and never a root number.
// accountsMachine is a user and group database the runner answers from, and every command that changed it.
type accountsMachine struct {
users, groups map[string][2]int // name → id, primary gid (a group's second is unused)
// in is each account's added groups, by name; homes each account's home, "/" when not set.
in map[string][]string
homes map[string]string
ran []string
}
// groupNamed is the name of gid in m, or the number.
func (m *accountsMachine) groupNamed(gid int) string {
for n, ids := range m.groups {
if ids[0] == gid {
return n
}
}
return strconv.Itoa(gid)
}
func (m *accountsMachine) run(_ context.Context, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
switch name {
case "getent":
db, key := args[0], args[1]
table := m.users
if db == "group" {
table = m.groups
}
for n, ids := range table {
if n == key || strconv.Itoa(ids[0]) == key {
if db == "group" {
return n + ":x:" + strconv.Itoa(ids[0]) + ":\n", nil
}
home := m.homes[n]
if home == "" {
home = "/"
}
return n + ":x:" + strconv.Itoa(ids[0]) + ":" + strconv.Itoa(ids[1]) + "::" + home + ":/usr/bin/nologin\n", nil
}
}
return "", errors.New("getent exited 2")
case "id":
name := args[len(args)-1]
u, ok := m.users[name]
if !ok {
return "", errors.New("id exited 1")
}
return strings.Join(append([]string{m.groupNamed(u[1])}, m.in[name]...), " ") + "\n", nil
case "groupadd":
m.groups[args[len(args)-1]] = [2]int{mustAtoi(args[2]), 0}
case "useradd":
name := args[len(args)-1]
m.users[name] = [2]int{mustAtoi(args[2]), mustAtoi(args[4])}
if args[5] == "--groups" {
m.setIn(name, strings.Split(args[6], ","))
}
if m.homes == nil {
m.homes = map[string]string{}
}
m.homes[name] = "/nonexistent"
case "usermod":
name := args[len(args)-1]
if args[0] == "--append" {
m.setIn(name, append(m.in[name], args[2]))
} else if args[0] == "--home" {
if m.homes == nil {
m.homes = map[string]string{}
}
m.homes[name] = args[1]
} else {
u := m.users[name]
m.users[name] = [2]int{u[0], mustAtoi(args[1])}
m.setIn(name, strings.Split(args[3], ","))
}
case "gpasswd":
var kept []string
for _, g := range m.in[args[1]] {
if g != args[2] {
kept = append(kept, g)
}
}
m.setIn(args[1], kept)
case "userdel":
delete(m.users, args[0])
case "groupdel":
delete(m.groups, args[0])
default:
return "", nil
}
m.ran = append(m.ran, line)
return "", nil
}
func (m *accountsMachine) setIn(name string, groups []string) {
if m.in == nil {
m.in = map[string][]string{}
}
var out []string
for _, g := range groups {
if g != "" {
out = append(out, g)
}
}
m.in[name] = out
}
func mustAtoi(s string) int {
n, _ := strconv.Atoi(s)
return n
}
// withGroups is a declaration with library groups, identities and resources.
func withGroups(t *testing.T, groups, identities, resources string) *declaration.Declaration {
t.Helper()
nothing := ""
if resources == "" {
nothing = `"owns_nothing":true,`
}
d, err := declaration.Parse([]byte(`{"declaration":1,"groups":[` + groups + `],"identities":[` + identities +
`],` + nothing + `"resources":[` + resources + `]}`))
if err != nil {
t.Fatal(err)
}
return d
}
func withIdentities(t *testing.T, identities, resources string) *declaration.Declaration {
t.Helper()
d, err := declaration.Parse([]byte(`{"declaration":1,"identities":[` + identities + `],"resources":[` +
resources + `]}`))
if err != nil {
t.Fatal(err)
}
return d
}
// Row 3: the account made as a system account with exactly its numbers, recorded, and removed — only what was
// made — when no longer declared.
func TestAModulesIdentityIsMadeWithItsNumbersAndRemovedWhenUndeclared(t *testing.T) {
m := &accountsMachine{users: map[string][2]int{}, groups: map[string][2]int{}}
dir := t.TempDir()
d := withIdentities(t, `{"module":"radarr","name":"mesh-radarr","uid":61001,"gid":61001}`,
`{"id":"radarr.config","type":"directory","path":"`+dir+`/config"}`)
report, known, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatal(err)
}
want := []string{
"groupadd --system --gid 61001 mesh-radarr",
"useradd --system --uid 61001 --gid 61001 --no-create-home --home-dir /nonexistent --shell /usr/bin/nologin mesh-radarr",
}
if strings.Join(m.ran, "\n") != strings.Join(want, "\n") {
t.Fatalf("made with:\n%s\nwant:\n%s", strings.Join(m.ran, "\n"), strings.Join(want, "\n"))
}
if o := outcomeOf(report, "identity mesh-radarr"); o.Action != "created" {
t.Errorf("the identity's outcome: %+v", o)
}
if len(known.Identities) != 1 || !known.Identities[0].MadeUser || !known.Identities[0].MadeGroup {
t.Fatalf("what was made is not recorded: %+v", known.Identities)
}
// Applied again: nothing is made twice.
m.ran = nil
if _, known, err = Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, m.run, nil, nil); err != nil {
t.Fatal(err)
}
if len(m.ran) != 0 {
t.Fatalf("an identity already here was made again: %v", m.ran)
}
// Undeclared, with a group the machine had before for another identity: only what was made goes.
m.groups["mesh-sonarr"] = [2]int{61002, 0}
other := withIdentities(t, `{"module":"sonarr","name":"mesh-sonarr","uid":61002,"gid":61002}`,
`{"id":"x.dir","type":"directory","path":"`+dir+`/x"}`)
if _, known, err = Apply(context.Background(), archHost(t), other, known, store.OriginDeclared, m.run, nil, nil); err != nil {
t.Fatal(err)
}
if _, has := m.users["mesh-radarr"]; has {
t.Error("an identity no longer declared kept its account")
}
if _, has := m.groups["mesh-radarr"]; has {
t.Error("an identity no longer declared kept its group")
}
if rec := madeIdentity(&known, "mesh-sonarr"); rec.MadeGroup {
t.Error("a group that was there before is recorded as the mesh's")
}
}
// Row 3: a name that exists with other numbers, or numbers another name holds, is refused naming whose, and the
// module is not attempted; the rest of the machine is.
func TestAnIdentityWhoseNameOrNumbersAreAnothersIsRefused(t *testing.T) {
dir := t.TempDir()
for _, tc := range []struct {
users, groups map[string][2]int
says string
}{
{users: map[string][2]int{"mesh-radarr": {1500, 1500}}, groups: map[string][2]int{}, says: "uid 1500"},
{users: map[string][2]int{"someone": {61001, 100}}, groups: map[string][2]int{}, says: "someone"},
{users: map[string][2]int{}, groups: map[string][2]int{"media": {61001, 0}}, says: "media"},
} {
m := &accountsMachine{users: tc.users, groups: tc.groups}
d := withIdentities(t, `{"module":"radarr","name":"mesh-radarr","uid":61001,"gid":61001}`,
`{"id":"radarr.config","type":"directory","path":"`+dir+`/radarr"},`+
`{"id":"plex.config","type":"directory","path":"`+dir+`/plex"}`)
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err == nil || !strings.Contains(err.Error(), tc.says) {
t.Errorf("not refused naming %s: %v", tc.says, err)
}
if len(m.ran) != 0 {
t.Errorf("something was made for an identity that is another's: %v", m.ran)
}
if o := outcomeOf(report, "radarr.config"); o.Action != "skipped" {
t.Errorf("the module of a refused identity was attempted: %+v", o)
}
if _, err := os.Stat(filepath.Join(dir, "plex")); err != nil {
t.Errorf("the rest of the machine was not applied: %v", err)
}
os.RemoveAll(filepath.Join(dir, "plex"))
}
}
// Row 11: a declaration naming a root group for an account or an image is refused with its reason, and nothing is
// made or started. A number 0 is refused on arrival, by the parser.
func TestNoModuleIdentityIsRoot(t *testing.T) {
was := RootGroupIDs
RootGroupIDs = func() map[int]string { return map[int]string{10: "wheel"} }
defer func() { RootGroupIDs = was }()
dir := t.TempDir()
for _, d := range []*declaration.Declaration{
withGroups(t, `{"name":"mesh-media","gid":10}`, `{"module":"radarr","name":"mesh-radarr","uid":61001,"gid":10}`,
`{"id":"radarr.config","type":"directory","path":"`+dir+`/config"}`),
withGroups(t, `{"name":"mesh-media","gid":10}`, ``,
`{"id":"radarr.config","type":"directory","path":"`+dir+`/config"}`),
withIdentities(t, ``, `{"id":"radarr.config","type":"directory","path":"`+dir+`/config"},`+
`{"id":"radarr.app","type":"container","name":"radarr","image":"`+pinned+`",`+
`"identity":{"mode":"run","uid":61001,"gid":10}}`),
} {
m := &accountsMachine{users: map[string][2]int{}, groups: map[string][2]int{}}
_, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err == nil || !strings.Contains(err.Error(), "wheel") || !strings.Contains(err.Error(), "never") {
t.Errorf("a root group was not refused naming why: %v", err)
}
if len(m.ran) != 0 {
t.Errorf("something was made or started: %v", m.ran)
}
if _, err := os.Stat(filepath.Join(dir, "config")); err == nil {
t.Error("a resource of a declaration naming a root group was applied")
}
}
for _, raw := range []string{
`{"declaration":1,"identities":[{"module":"m","name":"mesh-m","uid":0,"gid":61001}],"resources":[]}`,
`{"declaration":1,"resources":[{"id":"m.app","type":"container","name":"m","image":"` + pinned + `",` +
`"identity":{"mode":"env","user_env":"PUID","group_env":"PGID","uid":61001,"gid":0}}]}`,
} {
if _, err := declaration.Parse([]byte(raw)); err == nil || !strings.Contains(err.Error(), "never root") {
t.Errorf("a number 0 was not refused on arrival: %v", err)
}
}
}
// Row 3: a container declaring identity is told it — by the variables its image reads, or the runtime's user
// option — and a changed identity is a container recreated.
func TestAContainerIsToldItsModulesIdentity(t *testing.T) {
env := &declaration.Container{ID: "radarr.app", Name: "radarr", Image: pinned, RunAs: &declaration.ContainerIdentity{
Mode: declaration.IdentityByEnv, UserEnv: "PUID", GroupEnv: "PGID", UID: 61001, GID: 61001}}
run := &declaration.Container{ID: "plex.app", Name: "plex", Image: pinned, RunAs: &declaration.ContainerIdentity{
Mode: declaration.IdentityByRun, UID: 61002, GID: 61002}}
if got := strings.Join(foregroundRunArgs(env, "x"), " "); !strings.Contains(got, "--env PUID=61001 --env PGID=61001") {
t.Errorf("an image told by environment was not: %s", got)
}
if got := strings.Join(foregroundRunArgs(run, "x"), " "); !strings.Contains(got, "--user 61002:61002") {
t.Errorf("an image told by the runtime's user option was not: %s", got)
}
plain := *env
plain.RunAs = nil
moved := *env
moved.RunAs = &declaration.ContainerIdentity{Mode: declaration.IdentityByEnv, UserEnv: "PUID", GroupEnv: "PGID",
UID: 61009, GID: 61009}
if containerSpec(env, inputs{}) == containerSpec(&plain, inputs{}) ||
containerSpec(env, inputs{}) == containerSpec(&moved, inputs{}) {
t.Error("an identity told, or renumbered, does not recreate the container")
}
}
// ADR 0325 §5 supersedes ADR 0312's group grant: what a grant put an account in is taken back on the first apply of
// the access, and nothing is granted again.
func TestWhatAGroupGrantPutAnAccountInIsTakenBack(t *testing.T) {
dir := t.TempDir()
groups := "nox media"
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
switch name {
case "id":
return groups + "\n", nil
case "gpasswd":
groups = "nox"
}
return "", nil
}
known := store.State{Resources: []store.Applied{{ID: "importer.books", Type: "access", Target: dir,
Origin: store.OriginDeclared, User: "nox", Groups: []string{"media"}}}}
d := declare(t, `{"id":"importer.books","type":"access","path":"`+dir+`","mode":"read-write","grant":{"account":"nox"}}`)
report, known, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if groups != "nox" || !strings.Contains(strings.Join(ran, "\n"), "gpasswd --delete nox media") {
t.Fatalf("the grant's group was not taken back: %v", ran)
}
if o := outcomeOf(report, "importer.books"); o.Action != "updated" || !strings.Contains(o.Detail, "taken back") {
t.Errorf("the take-back is not said: %+v", o)
}
if rec, _ := known.Find("importer.books"); len(rec.Groups) != 0 {
t.Errorf("the record still holds the grant's groups: %+v", rec)
}
}
// Review L7: a group the grant put an account in that could not be left stays recorded, and is tried again.
func TestAGrantsGroupThatCouldNotBeLeftIsTriedAgain(t *testing.T) {
dir := t.TempDir()
refuse := true
groups := "nox media"
run := func(_ context.Context, name string, args ...string) (string, error) {
switch name {
case "id":
return groups + "\n", nil
case "gpasswd":
if refuse {
return "", errors.New("gpasswd exited 1: locked")
}
groups = "nox"
}
return "", nil
}
known := store.State{Resources: []store.Applied{{ID: "importer.books", Type: "access", Target: dir,
Origin: store.OriginDeclared, User: "nox", Groups: []string{"media"}}}}
d := declare(t, `{"id":"importer.books","type":"access","path":"`+dir+`","mode":"read-write"}`)
_, known, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if rec, _ := known.Find("importer.books"); rec.User != "nox" || len(rec.Groups) != 1 {
t.Fatalf("a group that could not be left was forgotten: %+v", rec)
}
refuse = false
if _, known, err = Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil); err != nil {
t.Fatal(err)
}
if rec, _ := known.Find("importer.books"); len(rec.Groups) != 0 || groups != "nox" {
t.Fatalf("the group was not taken back on the next apply: %+v (%s)", rec, groups)
}
}
// Every system the engine runs on that makes accounts makes module identities: arch and alpine alike.
func TestArchAndAlpineMakeModuleIdentities(t *testing.T) {
for _, name := range []string{"arch", "alpine"} {
sys, err := system.For(name)
if err != nil {
t.Fatal(err)
}
if _, err := toolFor(sys); err != nil {
t.Errorf("%s makes no module identity: %v", name, err)
}
}
}
+67
View File
@@ -0,0 +1,67 @@
package apply
import (
"context"
"errors"
"testing"
)
// A name is not unique across object kinds: a module regularly names a network the same as the
// container that joins it (keycloak does this today, ordinarily). `docker inspect <name>`, unlike
// `docker container inspect <name>`, resolves across every kind — so when the container does not
// exist yet but a same-named network does, the bare form answers with the network's JSON instead
// of reporting the container absent. containerState and inspectFound must ask by kind, or a
// same-named network makes them unable to tell "not here yet" from "the runtime is broken"
// (novox/hq ADR 0100's refusal, tripped by nothing wrong).
//
// dockerLikeByKind is Docker's real behaviour, not the bug: `container inspect` only ever
// answers from the container namespace. A fake that also answered the bare, unscoped form would
// not catch a regression back to it — this one refuses to, on purpose.
func dockerLikeByKind(containers map[string]bool) func(context.Context, string, ...string) (string, error) {
return func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("unexpected program: " + name)
}
if len(args) > 0 && args[0] == "info" {
// containerRuntime's probe, answered so inspectFound gets past it to the check under
// test.
return "27.0\n", nil
}
if len(args) < 2 || args[0] != "container" || args[1] != "inspect" {
return "", errors.New("unexpected command: only `docker container inspect` is modelled here")
}
target := args[len(args)-1]
if !containers[target] {
return "", errors.New("Error: No such container: " + target)
}
return "false\t\n", nil
}
}
func TestContainerStateAsksTheContainerNamespaceNotTheBareForm(t *testing.T) {
// "minio" exists only as a network in this scenario — never in `containers` — matching the
// live failure this guards: a module's network and its container share a name, and the
// container does not exist yet.
run := dockerLikeByKind(map[string]bool{"keycloak": true})
if _, err := containerState(context.Background(), "minio", run); err == nil {
t.Fatal("a container that does not exist should report absent, not be mistaken for found")
}
if state, err := containerState(context.Background(), "keycloak", run); err != nil {
t.Fatalf("a container that does exist should be found: %v", err)
} else if state.Running {
t.Errorf("the fake said not running; containerState disagreed: %+v", state)
}
}
func TestInspectFoundAsksTheContainerNamespaceNotTheBareForm(t *testing.T) {
run := dockerLikeByKind(map[string]bool{"keycloak": true})
if _, exists, err := inspectFound(context.Background(), "minio", run); err != nil || exists {
t.Fatalf("a container that does not exist should be reported absent cleanly, not refused: exists=%v err=%v", exists, err)
}
if _, exists, err := inspectFound(context.Background(), "keycloak", run); err != nil || !exists {
t.Fatalf("a container that does exist should be found: exists=%v err=%v", exists, err)
}
}
+20 -8
View File
@@ -93,7 +93,7 @@ func applyInto(r *declaration.File, previous store.Applied) (Outcome, error) {
if !tracked(k) && !had {
rec.Absent = append(rec.Absent, k)
}
merged, added, err := addMembers(current, declared[k], rec.Added[k])
merged, added, err := addMembers(current, declared[k], rec.Added[k], previous.Into == nil)
if err != nil {
return out, fmt.Errorf("%s: %q: %w", r.Path, k, err)
}
@@ -102,7 +102,14 @@ func applyInto(r *declaration.File, previous store.Applied) (Outcome, error) {
continue
}
if !tracked(k) {
if v, had := object[k]; had {
v, had := object[k]
// **What the host may have written itself is not the machine's.** With no record of
// this file — the first apply, or a host that wrote and died before saving its state —
// a key already holding exactly what the mesh declares cannot be told from one the
// mesh set a moment ago. Remembered as the machine's, it would never be given back:
// undeclaring would leave the mesh's own value behind for ever. So it is the mesh's,
// and undeclaring takes it out (novox/hq ADR 0102).
if had && !(previous.Into == nil && canonical(v) == canonical(declared[k])) {
rec.Before[k] = v
} else {
rec.Absent = append(rec.Absent, k)
@@ -125,7 +132,7 @@ func applyInto(r *declaration.File, previous store.Applied) (Outcome, error) {
mode = m
}
}
if err := os.MkdirAll(filepath.Dir(r.Path), 0o755); err != nil {
if err := makeDirs(filepath.Dir(r.Path), 0o755, r.Owner); err != nil {
return out, err
}
if err := writeAtomically(r.Path, want, mode); err != nil {
@@ -320,8 +327,11 @@ func listOf(members []json.RawMessage) json.RawMessage {
// addMembers adds the declared members to the machine's list, dropping only members the mesh
// added before and no longer declares. It returns the list and exactly which members the mesh
// added — a declared member the machine already had is the machine's, and is never recorded.
func addMembers(current, declared json.RawMessage, addedBefore []json.RawMessage) (json.RawMessage,
[]json.RawMessage, error) {
// unrecorded says there is no record of this file yet, in which case a declared member already in
// the list may be one the host itself wrote before it could save its state, and is taken as the
// mesh's.
func addMembers(current, declared json.RawMessage, addedBefore []json.RawMessage,
unrecorded bool) (json.RawMessage, []json.RawMessage, error) {
have, err := membersOf(current)
if err != nil {
return nil, nil, err
@@ -341,9 +351,11 @@ func addMembers(current, declared json.RawMessage, addedBefore []json.RawMessage
for _, m := range want {
if !hasMember(have, m) {
have = append(have, m)
if !hasMember(added, m) {
added = append(added, m)
}
} else if !unrecorded {
continue // the machine's own, and never the mesh's to take out
}
if !hasMember(added, m) {
added = append(added, m)
}
}
return listOf(have), added, nil
+129
View File
@@ -0,0 +1,129 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// novox/hq issue 190, ADR 0222: the private network stops writing the mesh's registry into the
// container runtime's file, and the runtime's own module writes the same member. Both are moved in
// one apply, and the machine never loses the member: the network's record goes first (its member
// leaves the list), the runtime's module is applied after (the member is added back, now recorded
// as the runtime's), and the daemon is reloaded once, never stopped, disabled or restarted — even
// though the record going says the unit was found stopped and disabled.
const theRegistry = "anchor.internal:5100"
func runtimeDecl(t *testing.T, path string, network bool) *declaration.Declaration {
t.Helper()
var resources []string
if network {
resources = append(resources,
fmt.Sprintf(`{"id":"networking.registry-trust","type":"file","path":%q,"into":"json","content":%q}`,
path, `{"insecure-registries":["`+theRegistry+`"]}`),
`{"id":"networking.registry-trust-reload","type":"service","unit":"docker.service","state":"running",
"reload-on":["networking.registry-trust"]}`)
}
resources = append(resources,
fmt.Sprintf(`{"id":"docker.daemon","type":"file","path":%q,"into":"json","content":%q}`,
path, `{"live-restore":true,"insecure-registries":["`+theRegistry+`"]}`),
`{"id":"docker.runtime","type":"service","unit":"docker.service","state":"running","boot":"enabled",
"reload-on":["docker.daemon"]}`)
return parse(t, `{"declaration":1,"resources":[`+strings.Join(resources, ",")+`]}`)
}
func TestTheRegistryMovesToTheRuntimesModuleWithoutLeavingTheList(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
if err := os.WriteFile(path, []byte(`{"insecure-registries":["192.0.2.7:5000"]}`), 0o644); err != nil {
t.Fatal(err)
}
var commands []string
run := recordingServices(&commands)
// Both declare it: the network's record added the member, so the runtime's finds it there.
_, state, err := Apply(context.Background(), archHost(t), runtimeDecl(t, path, true), store.State{},
store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
// Say the network's record found the runtime stopped and disabled: giving that back would stop
// every container on the machine.
for i := range state.Resources {
if state.Resources[i].ID == "networking.registry-trust-reload" {
state.Resources[i].Found = &store.FoundUnit{Unit: "docker.service", State: "stopped", Boot: "disabled"}
}
}
commands = nil
report, state, err := Apply(context.Background(), archHost(t), runtimeDecl(t, path, false), state,
store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := fmt.Sprint(readObject(t, path)["insecure-registries"]); got != "[192.0.2.7:5000 "+theRegistry+"]" {
t.Fatalf("the list after the move: %s", got)
}
rec, _ := state.Find("docker.daemon")
if added := rec.Into.Added["insecure-registries"]; len(added) != 1 || canonical(added[0]) != `"`+theRegistry+`"` {
t.Errorf("the member is not recorded as the runtime's module's: %s", added)
}
reloads := 0
for _, c := range commands {
if strings.Contains(c, "docker.service") && (strings.Contains(c, " stop ") || strings.Contains(c, " restart ") ||
strings.Contains(c, " disable ")) {
t.Errorf("the runtime was given back as the network's record found it: %q", c)
}
if strings.Contains(c, "reload docker.service") {
reloads++
}
}
if reloads != 1 {
t.Errorf("the runtime was reloaded %d times; commands were %v", reloads, commands)
}
for _, o := range report.Outcomes {
if o.ID == "networking.registry-trust-reload" && o.Action != "forgotten" {
t.Errorf("the network's record of the unit was %q: %s", o.Action, o.Detail)
}
}
// Steady from here on, and undeclaring the runtime's module takes only what it added.
report, state, err = Apply(context.Background(), archHost(t), runtimeDecl(t, path, false), state,
store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
for _, o := range report.Outcomes {
if o.ID == "docker.daemon" && o.Action != "unchanged" {
t.Errorf("a second apply was %q", o.Action)
}
}
if _, _, err := Apply(context.Background(), archHost(t), somethingElse(t), state, store.OriginDeclared, run, nil, nil); err != nil {
t.Fatal(err)
}
if got := fmt.Sprint(readObject(t, path)["insecure-registries"]); got != "[192.0.2.7:5000]" {
t.Errorf("undeclaring the runtime's module left %s", got)
}
}
// And the preview says the same before it happens.
func TestThePlanForgetsARecordOfAUnitStillHeld(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
var commands []string
_, state, err := Apply(context.Background(), archHost(t), runtimeDecl(t, path, true), store.State{},
store.OriginDeclared, recordingServices(&commands), nil, nil)
if err != nil {
t.Fatal(err)
}
for _, step := range Plan(runtimeDecl(t, path, false), state, store.OriginDeclared) {
if step.ID == "networking.registry-trust-reload" && step.Verb != "forget" {
t.Errorf("the plan would %s the network's record of a unit docker.runtime holds: %s", step.Verb, step.Why)
}
}
}
+40 -2
View File
@@ -217,9 +217,15 @@ func TestAListIsAddedToNeverReplaced(t *testing.T) {
// takes back only what it added (novox/hq ADR 0102).
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte(`{"insecure-registries":["192.0.2.7:5000","10.42.0.9:5000"]}`), 0o644)
// 10.42.0.9 is declared too, and was already the machine's: it is never the mesh's to remove.
// First the mesh's own member alone, so there is a record of this file.
first := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000"]}`))
_, state, err := Apply(context.Background(), archHost(t), first, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
// Now 10.42.0.9 is declared too, and was already the machine's: it is never the mesh's to remove.
d := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000","10.42.0.9:5000"]}`))
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil)
_, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
@@ -302,3 +308,35 @@ func TestAHoldFromAWholeFileDoesNotKeepOutAnIntoWrite(t *testing.T) {
t.Errorf("the mesh's member was not written in: %v", readObject(t, path))
}
}
func TestAWriteWithNoRecordOfItIsTheMeshsOwn(t *testing.T) {
// A host that wrote into the file and died before saving its state comes back with no record
// of it. What is there is then exactly what the mesh declares — and remembered as the
// machine's it would never be given back (novox/hq ADR 0102).
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte(`{"data-root":"/srv/docker","insecure-registries":["192.0.2.7:5000"]}`), 0o644)
d := parse(t, intoDecl(t, path, `{"live-restore":true,"insecure-registries":["10.42.0.1:5000"]}`))
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
// The crash: the state was never saved, so the next apply knows nothing of this file.
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
rec, _ := state.Find("networking.registry-trust")
if _, asTheMachines := rec.Into.Before["live-restore"]; asTheMachines {
t.Error("the mesh's own key was remembered as the machine's")
}
if _, _, err := Apply(context.Background(), archHost(t), somethingElse(t), state, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
o := readObject(t, path)
if _, still := o["live-restore"]; still {
t.Errorf("undeclaring left the mesh's key behind: %v", o)
}
if fmt.Sprint(o["insecure-registries"]) != "[192.0.2.7:5000]" || o["data-root"] != "/srv/docker" {
t.Errorf("the machine did not get its file back: %v", o)
}
}
+72
View File
@@ -0,0 +1,72 @@
//go:build linux
package apply
import (
"os"
"path/filepath"
"strings"
"testing"
)
// The read-only judgement of a node's directories (novox/hq issue 358): one line each — placed, would be refused
// naming the link and who could replace it, or not there yet (placed: an apply makes it) — and nothing made or
// changed.
func TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing(t *testing.T) {
base := t.TempDir()
plain := filepath.Join(base, "plain")
if err := os.Mkdir(plain, 0o755); err != nil {
t.Fatal(err)
}
elsewhere := filepath.Join(base, "elsewhere", "state")
if err := os.MkdirAll(elsewhere, 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink(filepath.Join(base, "elsewhere"), filepath.Join(base, "linked")); err != nil {
t.Fatal(err)
}
missing := filepath.Join(base, "missing", "deeper")
d := declare(t, `{"id":"a.plain","type":"directory","path":"`+plain+`","mode":"0700"},`+
`{"id":"a.linked","type":"directory","path":"`+filepath.Join(base, "linked", "state")+`","mode":"0700"},`+
`{"id":"a.missing","type":"directory","path":"`+missing+`","mode":"0700"}`)
said := JudgeDirectories(d)
lines := strings.Split(said, "\n")
if len(lines) != 4 || !strings.HasPrefix(lines[0], "3 directories declared: 2 placed, 1 would be refused") {
t.Fatalf("the judgement: %q", said)
}
if lines[1] != plain+": placed" || lines[3] != missing+": placed" {
t.Errorf("a directory the walk reaches is not said placed: %q", said)
}
if !strings.Contains(lines[2], "would be refused") || !strings.Contains(lines[2], filepath.Join(base, "linked")) ||
!strings.Contains(lines[2], "every account") {
t.Errorf("the linked directory is not said refused naming the link and who: %q", lines[2])
}
if _, err := os.Stat(filepath.Join(base, "missing")); !os.IsNotExist(err) {
t.Error("the judgement made a directory")
}
for _, p := range []string{plain, elsewhere} {
if info, _ := os.Stat(p); info.Mode().Perm() != 0o755 {
t.Errorf("the judgement changed %s to %o", p, info.Mode().Perm())
}
}
}
// A link only root could have made is followed by the judgement as by the apply.
func TestJudgeDirectoriesFollowsARootOnlyLink(t *testing.T) {
wasDirs, wasLinks := couldReplace, linkReplacer
couldReplace = func(string, uint32, uint32, uint32) string { return "" }
linkReplacer = func(string, uint32) string { return "" }
t.Cleanup(func() { couldReplace, linkReplacer = wasDirs, wasLinks })
base := t.TempDir()
if err := os.MkdirAll(filepath.Join(base, "real", "state"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink("real", filepath.Join(base, "run")); err != nil {
t.Fatal(err)
}
path := filepath.Join(base, "run", "state")
if said := JudgeDirectories(declare(t, `{"id":"a.s","type":"directory","path":"`+path+`"}`)); !strings.Contains(said, path+": placed") {
t.Fatalf("a directory behind a root-only link is not said placed: %q", said)
}
}
+244
View File
@@ -0,0 +1,244 @@
package apply
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"os"
"regexp"
"strings"
"sync"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// An access is an operator's path a module mounts and the mesh does not own (novox/hq ADR 0051): mounted when
// present, refused when absent, nothing changed on it. ADR 0325 kept a shared folder through ACL entries; ADR 0328
// supersedes it with libraries (library.go), and an access is again exactly what it was before: an access's `users`
// and `taken` are accepted and not acted on.
// applyAccess is one access, applied as ADR 0051 has it; what ADR 0312's grant put an account in is taken back
// first, so no account is in a group for data but a library's (ADR 0328 §11).
func applyAccess(ctx context.Context, sys system.System, r *declaration.Access, run Runner, previous store.Applied,
in inputs) (Outcome, error) {
gave, left, said := takeBackGranted(ctx, sys, previous, run, in.groups, in.handed)
out, err := legacyAccess(r)
if len(left) > 0 {
out.user, out.groups = previous.User, left
}
if said != "" {
out.Detail = joinDetail(out.Detail, said)
if gave && out.Action == "unchanged" {
// Updated, so what names the access under restart-on — the tool runner — holds the groups it has now.
out.Action = "updated"
}
}
return out, err
}
// takeBackGranted takes an account out of every group ADR 0312's grant put it in for an access, as removeUser takes
// back a user's (ADR 0252 §3): unless another declared resource still asks for it, which is then handed the group.
// Whether any was taken back, the groups still to take back — one whose leaving failed stays recorded, so the next
// apply tries again — and what was said; nothing for an access that granted nothing.
func takeBackGranted(ctx context.Context, sys system.System, previous store.Applied, run Runner, wanted Wanted,
handed Handed) (bool, []string, string) {
if previous.User == "" || len(previous.Groups) == 0 {
return false, nil, ""
}
account := previous.User
in, err := system.GroupsOf(ctx, system.Runner(run), account)
if err != nil {
return false, previous.Groups, fmt.Sprintf("the groups ADR 0312's grant put %s in (%s) were not taken "+
"back, and are tried again: %v", account, strings.Join(previous.Groups, ", "), err)
}
already := setOf(in)
gave := false
var left, said []string
for _, g := range previous.Groups {
switch {
case !already[g]:
continue
case len(wanted.othersAsk(account, g, previous.ID)) > 0:
said = append(said, fmt.Sprintf("left in %s, which %s still asks for", g,
strings.Join(wanted.othersAsk(account, g, previous.ID), ", ")))
handed.hand(account, g)
continue
}
gone, why := leaveGroup(ctx, sys, account, g, run)
if !gone {
left = append(left, g)
why += "; tried again on the next apply"
}
gave = gave || gone
said = append(said, why)
}
return gave, left, "the group grant ADR 0312 made is taken back: " + account + " " + strings.Join(said, "; ")
}
// legacyAccess is an access from a controller older than ADR 0325, applied exactly as before it (ADR 0051 §7).
//
// **Absent is refused, not created.** A bind mount whose source does not exist is made for you by the container
// runtime, as root, with whatever mode it picks — which is exactly the silent wrong-ownership 04-ISSUES/026
// records. So the host checks first and says plainly that the operator must provide the path, rather than
// conjuring a directory it does not own and cannot give the right owner. Nothing is written, so this never reports
// a change: the machine did not move, the host merely confirmed a fact about it.
func legacyAccess(r *declaration.Access) (Outcome, error) {
out := begin(r)
info, err := os.Stat(r.Path)
if errors.Is(err, os.ErrNotExist) {
return out, fmt.Errorf(
"%s is not there, and the mesh does not own it — the operator must provide it. It is "+
"shared, pre-existing data (novox/hq ADR 0051): the host mounts it and creates "+
"nothing, so a missing one is said here rather than made as root by the container "+
"runtime", r.Path)
}
if err != nil {
return out, err
}
if !info.IsDir() {
return out, fmt.Errorf(
"%s is not a directory, and an access is a shared directory the operator provides",
r.Path)
}
mode := declaration.AccessRead
if r.Mode != "" {
mode = r.Mode
}
out.Action = "unchanged"
out.Detail = "operator-owned; present, " + mode + ", nothing managed"
return out, nil
}
// PlacedID names the filesystem at mount for a wait, with no path in it: the controller asks again when the
// folder's filesystem is not the one a warrant was given for. Empty when the filesystem cannot be named, and a folder
// on it is then never kept (it fails closed, and waits).
func PlacedID(mount string) string { return placedFrom(mount, filesystemKey(mount)) }
// placedFrom is PlacedID from a filesystem key already read.
func placedFrom(mount, key string) string {
if mount == "" || key == "" {
return ""
}
sum := sha256.Sum256([]byte(mount + "\x00" + key))
return "fs-" + hex.EncodeToString(sum[:6])
}
// filesystemKey is what names the filesystem at mount across reboots, in this order:
//
// - a filesystem on a block device: its UUID, found by matching the mount's device against the devices in
// /dev/disk/by-uuid (never blkid). xfs's statfs id follows the device number, so the UUID comes first;
// - else the kernel's filesystem id (statfs f_fsid) when it is not zero: ZFS's from its dataset, btrfs's from its
// UUID;
// - else mountinfo's source with the filesystem's type: an NFS export's server:/export, a FUSE filesystem's own
// source string (both report f_fsid 0).
//
// Never the mount point and type alone, and never a device's name, which a reboot may renumber. Empty when none of
// these answers within statBound: then nothing is kept there. Read once per mount at a time.
func filesystemKey(mount string) string {
if mount == "" {
return ""
}
v, ok := boundedOnce("fs "+mount, func() any { return readFilesystemKey(mount) })
if !ok {
return ""
}
return v.(string)
}
func readFilesystemKey(mount string) string {
if u := MountUUID(mount); u != "" {
return "uuid:" + u
}
if id := MountFsid(mount); strings.Trim(id, "0") != "" {
return "fsid:" + id
}
if source, fstype := mountSourceOf(mount); source != "" && source != "none" {
return "source:" + fstype + ":" + source
}
return ""
}
// statBound is how long a read of a folder may take before it is said not to answer: a hard network mount whose
// server is down blocks a stat for ever, and one access must never hold the apply.
var statBound = 5 * time.Second
// errNoAnswer is a folder whose read did not return within statBound.
var errNoAnswer = errors.New("it does not answer: a read of it did not return within its bound (a mount that is down)")
// lstatWithin is os.Lstat within statBound, and statWithin os.Stat. A read that hangs is left to return on its own.
func lstatWithin(path string) (os.FileInfo, error) { return within("lstat", os.Lstat, path) }
func statWithin(path string) (os.FileInfo, error) { return within("stat", os.Stat, path) }
// flight is one read in flight, shared by everyone who asks while it is: a read that hangs on a dead mount holds
// one goroutine per path, never one per apply or look.
type flight struct {
done chan struct{}
v any
}
var inFlight = struct {
sync.Mutex
reads map[string]*flight
}{reads: map[string]*flight{}}
// boundedOnce is read's answer within statBound, sharing a read of key already in flight; false when it did not
// answer in time.
func boundedOnce(key string, read func() any) (any, bool) {
inFlight.Lock()
f, running := inFlight.reads[key]
if !running {
f = &flight{done: make(chan struct{})}
inFlight.reads[key] = f
go func() {
f.v = read()
inFlight.Lock()
delete(inFlight.reads, key)
inFlight.Unlock()
close(f.done)
}()
}
inFlight.Unlock()
select {
case <-f.done:
return f.v, true
case <-time.After(statBound):
return nil, false
}
}
func within(op string, stat func(string) (os.FileInfo, error), path string) (os.FileInfo, error) {
type read struct {
info os.FileInfo
err error
}
v, ok := boundedOnce(op+" "+path, func() any {
info, err := stat(path)
return read{info, err}
})
if !ok {
return nil, errNoAnswer
}
r := v.(read)
return r.info, r.err
}
// absPath is an absolute path in words: what pathless takes out.
var absPath = regexp.MustCompile(`(^|[\s"'(=])/[^\s"',;:)]*`)
// Pathless is an error in words with no path (ADR 0234 §6): a path error's operation and cause, and any other
// path said as "a path".
func Pathless(err error) string { return pathless(err) }
func pathless(err error) string {
var pe *os.PathError
if errors.As(err, &pe) {
return pe.Op + ": " + pe.Err.Error()
}
return absPath.ReplaceAllString(err.Error(), "${1}a path")
}
+218
View File
@@ -0,0 +1,218 @@
package apply
import (
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A taken container keeps a found network by a per-machine setting (novox/hq ADR 0163, rule 4):
// joined once it runs, part of its spec, and refused when it cannot be joined.
func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
var ran []string
connectFails := false
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("not installed")
}
ran = append(ran, strings.Join(args, " "))
switch args[0] {
case "info":
return "29.0.0\n", nil
case "container":
if len(ran) > 2 {
return "true\t" + specOfLast, nil
}
return "false\t\n", errors.New("no such container")
case "run":
return "deadbeef\n", nil
case "network":
if connectFails {
return "", errors.New("network predecessor_default not found")
}
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"`+pinned+`",
"networks":["predecessor_default"]}
]}`)
specOfLast = containerSpec(d.Resources[0].(*declaration.Container), inputs{})
alone := *d.Resources[0].(*declaration.Container)
alone.Networks = nil
if specOfLast == containerSpec(&alone, inputs{}) {
t.Fatal("the kept network is not part of the container's spec: kept or let go, the container would be left alone")
}
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
joined := false
for i, line := range ran {
if line == "network connect predecessor_default app" {
joined = true
if ran[i-1] != "container inspect --format {{.State.Running}}\t{{index .Config.Labels \""+specLabel+"\"}} app" &&
!strings.HasPrefix(ran[i-1], "container inspect") {
t.Errorf("joined before the container was read back as running: %v", ran)
}
}
}
if !joined || report.Outcomes[0].Action != "created" {
t.Fatalf("the container did not join the kept network: %v\n%+v", ran, report.Outcomes)
}
connectFails, ran = true, nil
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil); err == nil ||
!strings.Contains(err.Error(), "could not join the kept network predecessor_default") {
t.Fatalf("a network that cannot be joined was passed over: %v", err)
}
}
var specOfLast string
// A module the mesh left out of a declaration is not a module the mesh removed (novox/hq ADR 0163,
// rule 6): what the host wrote for it stays, recorded and said; what it holds for it stays held.
// A module simply absent is removed as it always was.
func TestALeftOutModuleIsNeitherRemovedNorForgotten(t *testing.T) {
var removed []string
gone := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", nil
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "rm":
removed = append(removed, args[len(args)-1])
gone[args[len(args)-1]] = true
case "container":
if gone[args[len(args)-1]] {
return "", errors.New("no such container")
}
return "true\tspec", nil
}
return "", nil
}
known := store.State{
Resources: []store.Applied{
{ID: "web.server", Type: "container", Target: "web", Origin: store.OriginDeclared},
{ID: "old.server", Type: "container", Target: "old", Origin: store.OriginDeclared},
},
Held: []store.Held{{ID: "web.page", Module: "web", Kind: "file", Target: "/srv/web/index.html"}},
}
d := parse(t, `{"declaration":1,"left_out":["web"],"resources":[
{"id":"notes.conf","type":"file","path":"`+t.TempDir()+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if len(removed) != 1 || removed[0] != "old" {
t.Fatalf("removed %v; only the module that is absent goes", removed)
}
if _, kept := state.At("container", "web"); !kept {
t.Fatal("the left-out module's record was forgotten")
}
if _, held := state.HeldAt("web.page"); !held {
t.Fatal("the left-out module's hold was released")
}
said := false
for _, o := range report.Outcomes {
if o.ID == "web.server" && o.Action == "unchanged" && strings.Contains(o.Detail, "web was left out of this declaration by the mesh") {
said = true
}
if o.ID == "web.server" && o.Action != "unchanged" {
t.Errorf("the left-out module's container was %s", o.Action)
}
}
if !said {
t.Fatalf("keeping the left-out module's container was not said: %+v", report.Outcomes)
}
}
// A container's capabilities reach the runtime and are part of its spec (novox/hq ADR 0170).
func TestACapabilityReachesTheRuntimeAndTheSpec(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("not installed")
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "container":
return "false\t\n", errors.New("no such container")
case "run":
ran = args
return "deadbeef\n", nil
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"fw","type":"container","name":"fw","image":"`+pinned+`","network":"host","capabilities":["NET_ADMIN"]}
]}`)
_, _, _ = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
granted := false
for i, a := range ran {
if a == "--cap-add" && i+1 < len(ran) && ran[i+1] == "NET_ADMIN" {
granted = true
}
}
if !granted {
t.Fatalf("the capability was not granted: %v", ran)
}
with := d.Resources[0].(*declaration.Container)
without := *with
without.Capabilities = nil
if containerSpec(with, inputs{}) == containerSpec(&without, inputs{}) {
t.Fatal("a capability is not part of the container's spec")
}
}
// A package may be declared absent (novox/hq ADR 0180): removed when it is installed, read back,
// left alone when it is not.
func TestAPackageDeclaredAbsentIsRemovedWhenPresentAndLeftWhenNot(t *testing.T) {
installed := true
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
ran = append(ran, name+" "+strings.Join(args, " "))
if name != "pacman" {
return "", nil
}
switch args[0] {
case "-Q":
if args[1] == "pacman" || installed {
return args[1] + " 1.0\n", nil
}
return "", errors.New("package not found")
case "-R":
installed = false
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[{"id":"front-end","type":"package","package":"ufw","absent":true}]}`)
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "removed" || !strings.Contains(strings.Join(ran, "\n"), "pacman -R --noconfirm ufw") {
t.Fatalf("an installed package declared absent was not removed: %+v\n%v", report.Outcomes[0], ran)
}
ran = nil
report, _, err = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "unchanged" || strings.Contains(strings.Join(ran, "\n"), "-R") {
t.Fatalf("a package already absent was touched: %+v\n%v", report.Outcomes[0], ran)
}
}
+506
View File
@@ -0,0 +1,506 @@
package apply
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// A library is kept usable by its group (novox/hq ADR 0328, issue 567).
//
// A library is a named tree of shared data with one home (§1). On its home the engine keeps its root and every
// sub-path its modules ask for — those folders themselves, never what is in them — in the library's group with
// group read, write and enter and the setgid bit, so what is made beneath keeps the group (§6); a missing sub-path
// is made with the root's owner and the library's group (§9). The owner, the owner's bits and others' bits are
// never changed. What the root already holds joins only on the operator's word (§7): until then the library waits
// with reason keep and nothing under it is walked. A root that is absent, or whose disk is not there, makes
// nothing, holds every container that mounts the library, and waits with reason disk (§9). Off its home a library is
// a mount the mounts module makes: nothing is kept there, and a mount that is not there holds the containers too.
//
// **When a root is the mesh's to keep without asking.** When it holds nothing yet, or is already in the library's
// group with group write and setgid (kept so before, by hand or by the mesh). Every other root waits for the take
// act, which walks it whole (TakeLibrary). Kept, it stays kept on the filesystem it was kept on — named by UUID,
// filesystem id or source (PlacedID) — and another filesystem at its root is judged afresh.
// libraryRecords is where the engine keeps what it knows of each library it is home to: the file beside the state
// in the engine, memory in a test.
type libraryRecords interface {
All() (map[string]store.LibraryRecord, error)
Update(name string, change func(r *store.LibraryRecord, had bool) bool) error
}
// Libraries is the engine's record of the libraries it is home to. The engine sets the file beside its state.
var Libraries libraryRecords = NewLibraryMemory()
type libraryMemory struct {
mu sync.Mutex
m map[string]store.LibraryRecord
}
// NewLibraryMemory is an empty record held in memory: a test's, or a one-shot apply's.
func NewLibraryMemory() libraryRecords { return &libraryMemory{m: map[string]store.LibraryRecord{}} }
func (k *libraryMemory) All() (map[string]store.LibraryRecord, error) {
k.mu.Lock()
defer k.mu.Unlock()
out := map[string]store.LibraryRecord{}
for n, r := range k.m {
out[n] = r
}
return out, nil
}
func (k *libraryMemory) Update(name string, change func(r *store.LibraryRecord, had bool) bool) error {
k.mu.Lock()
defer k.mu.Unlock()
r, had := k.m[name]
r.Library = name
if change(&r, had) {
k.m[name] = r
} else {
delete(k.m, name)
}
return nil
}
// libraryRecord is the record of one library, and whether there is one. A record that cannot be read is none: the
// library is judged afresh, which waits rather than keeps.
func libraryRecord(name string) (store.LibraryRecord, bool) {
all, err := Libraries.All()
if err != nil {
return store.LibraryRecord{}, false
}
r, had := all[name]
return r, had
}
// libraryLocks is one lock per library, held by every change to its root and every walk under it.
var libraryLocks sync.Map
func libraryLock(name string) *sync.Mutex {
l, _ := libraryLocks.LoadOrStore(name, &sync.Mutex{})
return l.(*sync.Mutex)
}
// libraryWalks is every library a take walk is running under now, with the filesystem it walks: set once the walk
// has been judged and started, cleared when it ends.
var libraryWalks sync.Map
// LibraryAccess is the access a library's wait and its take act name.
func LibraryAccess(name string) string { return declaration.LibraryAccessPrefix + name }
// KeepWords and DiskWords are what a library's wait says the operator gives (ADR 0328 §8, §9): never a path.
func KeepWords(name string) string { return "let the mesh keep the library " + name }
func DiskWords(name string) string { return "mount the disk that holds the library " + name }
// The judgements of a library.
const (
// libraryKeep is a library the mesh keeps: taken, empty when first seen, or found in shape.
libraryKeep = "keep"
// libraryWaitKeep is a root that holds data never taken: it waits for the operator's word.
libraryWaitKeep = "wait-keep"
// libraryWaitDisk is a root that is absent, or whose disk (or, off its home, whose mount) is not there.
libraryWaitDisk = "wait-disk"
// libraryUnknown is a root whose filesystem could not be named now: left as it is until it is.
libraryUnknown = "unknown"
// libraryMounted is a library off its home whose mount is there: nothing to keep.
libraryMounted = "mounted"
)
// libraryVerdict is the judgement of one library, with what it read.
type libraryVerdict struct {
kind string
// how is why a kept library is the mesh's: store.KeptByWarrant, store.KeptEmpty or store.KeptInShape.
how string
// mount is the mount point that holds the root; placed and fs name its filesystem.
mount, placed, fs string
// moved is that another filesystem than the one the library was kept on is at its root.
moved bool
why string
err error
}
// assessLibrary judges a library as an apply, a look and the take act all do (ADR 0328 §7, §9, §15): its group
// first, then its disk, then whether the mesh may keep it. Its words carry no path: they reach the operator's
// channel. Reads only.
func assessLibrary(ctx context.Context, l declaration.Library, rec store.LibraryRecord) libraryVerdict {
if why := rootGroup(l.GID); why != "" {
return libraryVerdict{err: fmt.Errorf("its group is %s, and a library's group is never a root group "+
"(novox/hq ADR 0328 §15); nothing was made or changed", why)}
}
// Nowhere the machine's own trees are, as for an access (novox/hq issue 339): a library at /etc would be
// the machine's configuration given a group.
if err := refusePath(l.Root, placingAccess, "", ""); err != nil {
return libraryVerdict{err: errors.New(pathless(err) + "; nothing was made or changed")}
}
if !l.Home {
// Off its home the mounts module mounts it: there when its root is a mount point now.
for _, p := range MountPoints() {
if p == l.Root {
return libraryVerdict{kind: libraryMounted, mount: p}
}
}
return libraryVerdict{kind: libraryWaitDisk, why: "its mount is not there"}
}
info, err := lstatWithin(l.Root)
switch {
case errors.Is(err, os.ErrNotExist):
return libraryVerdict{kind: libraryWaitDisk, why: "its root is not there"}
case errors.Is(err, errNoAnswer):
return libraryVerdict{kind: libraryUnknown, why: "its root does not answer: a read of it did not return " +
"within its bound"}
case err != nil:
return libraryVerdict{err: errors.New("its root cannot be read: " + pathless(err))}
case !info.IsDir():
return libraryVerdict{err: errors.New("its root is not a folder (a file, or a link), and a library's root " +
"is a folder the engine reaches through no link")}
}
pl := placedOn(ctx, l.Root, l.Root, rec.Mount, recordedLibraryMounts(l.Library))
if pl.Away {
return libraryVerdict{kind: libraryWaitDisk, mount: pl.Mount, why: "the disk that holds its root is not there"}
}
fs := filesystemKey(pl.Mount)
here := placedFrom(pl.Mount, fs)
// **A filesystem that cannot be named now is unknown, never a move**: the library holds what it is — nothing
// walked, nothing newly kept, nothing reset — and is judged again when it is named. One never kept is so never
// kept: it fails closed.
if here == "" {
return libraryVerdict{kind: libraryUnknown, mount: pl.Mount,
why: "its filesystem could not be named this time (no UUID, filesystem id or source answered)"}
}
moved := pl.Moved || (rec.Fs != "" && rec.Mount == pl.Mount && rec.Placed != here)
v := libraryVerdict{mount: pl.Mount, placed: here, fs: fs, moved: moved}
// The controller's word that the library was taken holds for the filesystem the warrant was carried out on.
takenHere := l.Taken && (rec.KeptOn == here || (rec.KeptOn == "" && !moved))
switch {
case rec.Kept && rec.KeptOn == here && !moved && rec.How == store.KeptByWarrant, takenHere:
// Taken whole on the operator's word: every sub-path is the library's.
v.kind, v.how = libraryKeep, store.KeptByWarrant
return v
case rec.Kept && rec.KeptOn == here && !moved:
// Kept without a warrant (empty, or in shape): a sub-path asked for since that holds data not in the group
// waits for the operator's word as the root would (§7).
v.kind, v.how = libraryKeep, rec.How
if !pathsInShape(l, rec.Paths) {
v.kind, v.how = libraryWaitKeep, ""
}
return v
}
facts, err := readFolder(l.Root)
if err != nil {
return libraryVerdict{err: errors.New("its root cannot be read: " + pathless(err))}
}
switch {
case facts.Empty:
// Nothing there, so no sub-path either: every one is made.
v.kind, v.how = libraryKeep, store.KeptEmpty
case int(facts.GID) == l.GID && facts.Mode&0o2070 == 0o2070 && sampleInShape(l.Root, l.GID) &&
pathsInShape(l, nil):
// The root and a bounded sample beneath it already in the group with group write and setgid: data that
// is the library's already, kept without asking. One entry of the sample out of shape asks (§7).
v.kind, v.how = libraryKeep, store.KeptInShape
default:
// What is there holds data the mesh never took — a disk mounted over a root kept while it was empty below
// it included (a move): it waits for the operator's word, and nothing under it is walked.
v.kind = libraryWaitKeep
}
return v
}
// recordedLibraryMounts is every mount point recorded for another library: where a disk was once seen.
func recordedLibraryMounts(except string) []string {
all, err := Libraries.All()
if err != nil {
return nil
}
var out []string
for name, r := range all {
if name != except && r.Mount != "" && r.Mount != "/" {
out = append(out, r.Mount)
}
}
return out
}
// applyLibrary is one library, applied (ADR 0328 §6, §7, §9): kept on its home when the mesh may keep it, waiting
// otherwise, and failing for a root group. Nothing is changed on the root but the group, group rwx and setgid of
// the root and its requested sub-paths, and the sub-paths made.
func applyLibrary(ctx context.Context, l declaration.Library) (Outcome, libraryVerdict, error) {
out := Outcome{ID: LibraryAccess(l.Library), Type: "library", Target: l.Root}
lock := libraryLock(l.Library)
if !lock.TryLock() {
// **A take walk is running (up to its bound, half an hour): the apply does not wait for it.** The library is
// the walk's until it is through; it keeps it, and its containers are not held for it. Only a walk that
// has started, on the filesystem it judged: anything else holding the lock (a take act about to be
// refused, say) is short, and the apply waits for it and judges the library as always.
if on, walking := libraryWalks.Load(l.Library); walking {
out.Action = "unchanged"
out.Detail = "the take act is setting its group under its root now; this apply leaves the library to it"
return out, libraryVerdict{kind: libraryKeep, how: store.KeptByWarrant, placed: on.(string)}, nil
}
lock.Lock()
}
defer lock.Unlock()
rec, _ := libraryRecord(l.Library)
v := assessLibrary(ctx, l, rec)
if v.err != nil {
return out, v, v.err
}
switch v.kind {
case libraryMounted:
out.Action, out.Detail = "unchanged", "not its home: mounted, and nothing is kept here"
return out, v, nil
case libraryWaitDisk:
out.Action = "waiting"
out.Detail = v.why + ": nothing is made, and no container that mounts it is created until it is there " +
"(novox/hq ADR 0328 §9); waits for the operator to " + DiskWords(l.Library)
return out, v, nil
case libraryUnknown:
out.Action = "waiting"
out.Detail = v.why + ": the library is left exactly as it is, and judged again on the next apply"
return out, v, nil
case libraryWaitKeep:
_ = Libraries.Update(l.Library, func(r *store.LibraryRecord, had bool) bool {
if !had || v.moved {
r.Root, r.Mount, r.Placed, r.Fs = l.Root, v.mount, v.placed, v.fs
}
if v.moved {
// Kept on another filesystem: what was kept there no longer holds here.
r.Kept, r.How, r.Cursor = false, "", nil
}
return true
})
out.Action = "waiting"
out.Detail = "its root holds data the mesh has not taken: nothing is changed under it until the operator's " +
"warrant (novox/hq ADR 0328 §7); waits for the operator to " + KeepWords(l.Library)
return out, v, nil
}
changed, made, err := keepFolders(l.Root, l.Paths, l.GID)
if err != nil {
return out, v, err
}
_ = Libraries.Update(l.Library, func(r *store.LibraryRecord, _ bool) bool {
if !r.Kept || r.KeptOn != v.placed {
r.How, r.At = v.how, time.Now().UTC()
}
r.Root, r.Mount, r.Placed, r.Fs, r.GID = l.Root, v.mount, v.placed, v.fs, l.GID
r.Kept, r.KeptOn = true, v.placed
r.Paths = unionPaths(r.Paths, l.Paths)
return true
})
out.Action = "unchanged"
if changed+made > 0 {
out.Action = "updated"
}
out.Detail = fmt.Sprintf("kept in %s: its root and %d sub-path(s) with group rwx and setgid, %d made; owners "+
"and everyone else's rights unchanged (novox/hq ADR 0328 §6)", l.Group, len(l.Paths), made)
return out, v, nil
}
// heldRoots is every library root whose containers are held: its disk, or its mount, is not there (§9).
func heldRoots(verdicts map[string]libraryVerdict, libs []declaration.Library) map[string]string {
out := map[string]string{}
for _, l := range libs {
if v, ok := verdicts[l.Library]; ok && v.kind == libraryWaitDisk {
out[l.Root] = l.Library
}
}
return out
}
// mountsHeld is the library a container waits for: one whose root it mounts, or mounts below, is held.
func mountsHeld(c *declaration.Container, held map[string]string) string {
for _, v := range c.Volumes {
src := mountSource(v)
if !strings.HasPrefix(src, "/") {
continue
}
for root, name := range held {
if atOrBelow(filepath.Clean(src), root) {
return name
}
}
}
return ""
}
// WaitingOnLibrary is every container of the declaration that waits for a library whose disk or mount is not
// there, read now: not created by the apply, so not judged alive or dead — the library says why.
func WaitingOnLibrary(d *declaration.Declaration) map[string]bool {
out := map[string]bool{}
if d == nil || len(d.Libraries) == 0 {
return out
}
verdicts := map[string]libraryVerdict{}
for _, l := range d.Libraries {
rec, _ := libraryRecord(l.Library)
verdicts[l.Library] = assessLibrary(context.Background(), l, rec)
}
held := heldRoots(verdicts, d.Libraries)
for _, r := range d.Resources {
if c, ok := r.(*declaration.Container); ok && mountsHeld(c, held) != "" {
out[c.ID] = true
}
}
return out
}
// The judgements of a library as a look says them.
const (
VerdictKeep = "keep"
VerdictMounted = "mounted"
VerdictWaitKeep = "wait-keep"
VerdictWaitDisk = "wait-disk"
VerdictUnknown = "unknown"
VerdictBroken = "broken"
)
// LibraryJudgement is a look's judgement of a library, and why, in words with no path; Placed names its root's
// filesystem for a keep wait, with no path (PlacedID).
type LibraryJudgement struct {
Kind, Why, Placed string
// Act is the last walk under its root.
Act *store.KeptAct
}
// JudgeLibrary judges a library for a look, read only.
func JudgeLibrary(ctx context.Context, l declaration.Library) LibraryJudgement {
rec, _ := libraryRecord(l.Library)
v := assessLibrary(ctx, l, rec)
j := LibraryJudgement{Act: rec.Act}
switch {
case v.err != nil:
j.Kind, j.Why = VerdictBroken, pathless(v.err)
case v.kind == libraryWaitKeep:
j.Kind, j.Placed = VerdictWaitKeep, v.placed
case v.kind == libraryWaitDisk:
j.Kind, j.Why = VerdictWaitDisk, v.why
case v.kind == libraryUnknown:
j.Kind, j.Why = VerdictUnknown, v.why
case v.kind == libraryMounted:
j.Kind = VerdictMounted
default:
j.Kind = VerdictKeep
}
return j
}
// joinRunner puts a library's tool runner's account in its group (ADR 0328 §11), as ADR 0252 puts an account in a
// group: recorded only when the mesh put it there, so one it was in before is never taken back. Whether it moved.
func joinRunner(ctx context.Context, sys system.System, l declaration.Library, known *store.State,
run Runner) (bool, string, error) {
if l.Runner == "" {
return false, "", nil
}
in, err := system.GroupsOf(ctx, system.Runner(run), l.Runner)
if err != nil {
return false, "", fmt.Errorf("the tool runner's account %s could not be read: %w", l.Runner, err)
}
if setOf(in)[l.Group] {
return false, "", nil
}
if err := sys.AddUserToGroup(ctx, system.Runner(run), l.Runner, l.Group); err != nil {
return false, "", err
}
for _, m := range known.Members {
if m.Account == l.Runner && m.Group == l.Group {
return true, fmt.Sprintf("put the tool runner's %s back in %s", l.Runner, l.Group), nil
}
}
known.Members = append(known.Members, store.Membership{Account: l.Runner, Group: l.Group, Library: l.Library})
return true, fmt.Sprintf("put the tool runner's %s in %s; it holds it once restarted", l.Runner, l.Group), nil
}
// leaveRunners takes every tool runner's account the mesh put in a library's group out of it once no declared
// library puts it there and no declared user resource asks for it (ADR 0252 §3). Never fatal: what could not be
// left stays recorded, said, for the next apply.
func leaveRunners(ctx context.Context, sys system.System, libs []declaration.Library, known *store.State, run Runner,
wanted Wanted) []Outcome {
declared := map[string]bool{}
for _, l := range libs {
if l.Runner != "" {
declared[l.Runner+" "+l.Group] = true
}
}
var out []Outcome
var keep []store.Membership
for _, m := range known.Members {
if declared[m.Account+" "+m.Group] {
keep = append(keep, m)
continue
}
o := Outcome{ID: LibraryAccess(m.Library) + " runner", Type: "library", Target: m.Account}
if others := wanted.othersAsk(m.Account, m.Group, ""); len(others) > 0 {
o.Action, o.Detail = "forgotten", fmt.Sprintf("left %s in %s, which %s asks for", m.Account, m.Group,
strings.Join(others, ", "))
out = append(out, o)
continue
}
gone, why := leaveGroup(ctx, sys, m.Account, m.Group, run)
o.Action, o.Detail = "restored", m.Account+" "+why
if !gone {
o.Action = "waiting"
keep = append(keep, m)
}
out = append(out, o)
}
known.Members = keep
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out
}
// pathsInShape is whether every requested sub-path not kept before is absent (it is made), empty, or already in the
// group with group write and setgid, it and a bounded sample beneath it: what may be kept without the operator's
// word (§7). One that cannot be read is not.
func pathsInShape(l declaration.Library, kept []string) bool {
done := map[string]bool{}
for _, p := range kept {
done[p] = true
}
for _, p := range l.Paths {
if done[p] {
continue
}
path := filepath.Join(l.Root, p)
if _, err := lstatWithin(path); errors.Is(err, os.ErrNotExist) {
continue
}
facts, err := readFolder(path)
if err != nil {
return false
}
if facts.Empty {
continue
}
if int(facts.GID) != l.GID || facts.Mode&0o2070 != 0o2070 || !sampleInShape(path, l.GID) {
return false
}
}
return true
}
// unionPaths is a with every path of b not in it.
func unionPaths(a, b []string) []string {
have := map[string]bool{}
for _, p := range a {
have[p] = true
}
for _, p := range b {
if !have[p] {
a = append(a, p)
have[p] = true
}
}
sort.Strings(a)
return a
}
+161
View File
@@ -0,0 +1,161 @@
//go:build linux
package apply
import (
"context"
"errors"
"fmt"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// The take act on a library, and the walk that goes on with one stopped at its bound (novox/hq ADR 0328 §7).
// actRecord is a walk's counts as the record keeps them.
func actRecord(warrant, by, on string, started time.Time, w *treeWalk, err error) *store.KeptAct {
c := w.counts
rec := &store.KeptAct{Act: "take", Warrant: warrant, By: by, On: on, State: "done", Started: started,
Ended: time.Now().UTC(), Folders: c.Folders, Files: c.Files, Kept: c.Kept, Links: c.Links, Other: c.Other,
Mounts: c.Mounts, Shared: c.Shared, Linked: c.Linked, Stripped: c.Stripped}
if c.Shared+c.Linked+c.Stripped > 0 {
rec.Why = fmt.Sprintf("%d file(s) with another name given the group, %d root's left as they are, %d whose "+
"setuid or setgid bit the group change took", c.Linked, c.Shared, c.Stripped)
}
switch {
case errors.Is(err, ErrWalkBound):
rec.State, rec.Why = "stopped", "it met its bound of entries or time; it goes on from where it stopped"
case err != nil:
rec.State, rec.Why = "failed", err.Error()
case w.Failed != "":
rec.State = "failed"
rec.Why = fmt.Sprintf("%d entries could not be given the group: %s; walked again on the next look", c.Failed,
w.Failed)
}
return rec
}
// TakeLibrary is the take act (ADR 0328 §7): on the operator's warrant, the library's group, group read and write,
// and on folders group enter and setgid, set on its root and everything under it — no link followed, no other
// filesystem entered, the owner and the owner's and others' bits unchanged, a file with another name elsewhere left
// as it is. It judges the ask against the declaration last applied and the root as it is, and starts the walk: what
// it answers at once, and a channel closed when the walk is through or stopped at its bound. Refused, nothing is
// changed.
func TakeLibrary(ctx context.Context, d *declaration.Declaration, access, path, warrant, by string) (<-chan struct{},
string, error) {
if d == nil {
return nil, "", errors.New("this engine has applied no declaration, and takes no library it was not told of")
}
var lib *declaration.Library
for i := range d.Libraries {
if LibraryAccess(d.Libraries[i].Library) == access {
lib = &d.Libraries[i]
}
}
switch {
case lib == nil:
return nil, "", fmt.Errorf("the access %s names no library declared on this machine", access)
case !lib.Home:
return nil, "", fmt.Errorf("this machine is not the home of the library %s, and a library is taken on its "+
"home", lib.Library)
case lib.Root != path:
return nil, "", fmt.Errorf("the library %s is declared at another root than the one asked", lib.Library)
}
lock := libraryLock(lib.Library)
if !lock.TryLock() {
return nil, "", errors.New("a walk of the library is running; asked again once it is through, the act goes on")
}
rec, _ := libraryRecord(lib.Library)
v := assessLibrary(ctx, *lib, rec)
var refused error
switch {
case v.err != nil:
refused = v.err
case v.kind == libraryWaitDisk:
refused = errors.New(v.why)
case v.kind == libraryUnknown:
refused = errors.New(v.why + ", so a warrant could name no filesystem it was given for")
}
if refused != nil {
lock.Unlock()
return nil, "", refused
}
if _, _, err := keepFolders(lib.Root, lib.Paths, lib.GID); err != nil {
lock.Unlock()
return nil, "", err
}
started := time.Now().UTC()
_ = Libraries.Update(lib.Library, func(r *store.LibraryRecord, _ bool) bool {
r.Root, r.Mount, r.Placed, r.Fs, r.GID = lib.Root, v.mount, v.placed, v.fs, lib.GID
r.Kept, r.KeptOn, r.How, r.At = true, v.placed, store.KeptByWarrant, started
r.Warrant, r.By, r.Cursor = warrant, by, nil
r.Act = &store.KeptAct{Act: "take", Warrant: warrant, By: by, On: v.placed, State: "running", Started: started}
return true
})
done := make(chan struct{})
go func() {
defer close(done)
defer lock.Unlock()
walkLibrary(lib.Library, lib.Root, lib.GID, warrant, by, v.placed, nil)
}()
return done, fmt.Sprintf("the take act started under the warrant %s: %s's group, group read and write, and "+
"setgid on its folders are set on everything under its root, following no link and entering no other "+
"filesystem", warrant, lib.Library), nil
}
// walkLibrary walks a library's root from resume with its group, and records what it did: a walk stopped at its
// bound keeps its cursor for the next. The caller holds the library's lock.
func walkLibrary(name, root string, gid int, warrant, by, on string, resume []string) {
libraryWalks.Store(name, on)
defer libraryWalks.Delete(name)
started := time.Now().UTC()
w, err := walkRoot(root, ActLimits, machineStats, resume, groupVisit(gid))
rec := actRecord(warrant, by, on, started, w, err)
_ = Libraries.Update(name, func(r *store.LibraryRecord, _ bool) bool {
r.Act = rec
r.Cursor = nil
if errors.Is(err, ErrWalkBound) {
r.Cursor = w.Cursor
}
return true
})
}
// ReconcileLibraries goes on with every take act that stopped at its bound or left entries it could not change,
// from where it stopped — only while the library is still kept on the filesystem it was taken on (judged as an apply
// judges it). What it did, in words, for the engine to say.
func ReconcileLibraries(ctx context.Context, d *declaration.Declaration) []string {
if d == nil {
return nil
}
var said []string
for _, l := range d.Libraries {
if ctx.Err() != nil {
break
}
if !l.Home {
continue
}
rec, _ := libraryRecord(l.Library)
if rec.Act == nil || (rec.Act.State != "stopped" && rec.Act.State != "failed") || rec.How != store.KeptByWarrant {
continue
}
lock := libraryLock(l.Library)
if !lock.TryLock() {
continue
}
v := assessLibrary(ctx, l, rec)
if v.err != nil || v.kind != libraryKeep || v.placed != rec.KeptOn || v.moved {
lock.Unlock()
continue
}
walkLibrary(l.Library, l.Root, l.GID, rec.Warrant, rec.By, rec.KeptOn, rec.Cursor)
lock.Unlock()
after, _ := libraryRecord(l.Library)
said = append(said, fmt.Sprintf("the library %s: the take act went on, %s (%d folders and %d files changed)",
l.Library, after.Act.State, after.Act.Folders, after.Act.Files))
}
return said
}
+834
View File
@@ -0,0 +1,834 @@
//go:build linux
package apply
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"golang.org/x/sys/unix"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// The engine's half of ADR 0328's checks (novox/hq ADR 0328, "How it is checked", issue 567). The engine runs as
// root; a test may not, and a non-root test gives its folders a group it is in itself.
// testGID is a group the test may give its folders: any number as root, else one of its own added groups that is
// no root group's.
func testGID(t *testing.T) int {
t.Helper()
if os.Geteuid() == 0 {
return 61000
}
groups, _ := os.Getgroups()
for _, g := range groups {
if g != os.Getgid() && g != 0 && rootGroup(g) == "" {
return g
}
}
t.Skip("the test runs as no root and in no group of its own beside its primary one, so it can give no " +
"folder a library's group")
return 0
}
// libraries has the test keep its own record of libraries and judge mounts as the test says.
func libraries(t *testing.T, points, targets []string) {
t.Helper()
was, wasPoints, wasTargets := Libraries, MountPoints, MountTargets
Libraries = NewLibraryMemory()
forgetMountHistory()
MountPoints = func() []string { return append([]string{"/"}, points...) }
MountTargets = func(context.Context) []string { return targets }
t.Cleanup(func() {
Libraries, MountPoints, MountTargets = was, wasPoints, wasTargets
forgetMountHistory()
})
}
func forgetMountHistory() {
mountHistory.Lock()
mountHistory.seen = map[string]bool{}
mountHistory.Unlock()
}
// libraryDecl is a declaration with one library, its group, and resources.
func libraryDecl(t *testing.T, lib string, resources ...string) *declaration.Declaration {
t.Helper()
var l declaration.Library
if err := jsonInto(lib, &l); err != nil {
t.Fatal(err)
}
raw := fmt.Sprintf(`{"declaration":1,"groups":[{"name":%q,"gid":%d}],"libraries":[%s],"resources":[%s]}`,
l.Group, l.GID, lib, strings.Join(resources, ","))
if len(resources) == 0 {
raw = strings.Replace(raw, `"resources":[]`, `"owns_nothing":true,"resources":[]`, 1)
}
d, err := declaration.Parse([]byte(raw))
if err != nil {
t.Fatal(err)
}
return d
}
// libJSON is a library's declaration.
func libJSON(name, root string, gid int, home, taken bool, paths ...string) string {
quoted := make([]string, len(paths))
for i, p := range paths {
quoted[i] = strconv.Quote(p)
}
return fmt.Sprintf(`{"library":%q,"root":%q,"group":"mesh-%s","gid":%d,"home":%v,"taken":%v,"paths":[%s]}`,
name, root, name, gid, home, taken, strings.Join(quoted, ","))
}
// applyLib applies d with a runner that answers the group database for the library group (already there) and runs
// nothing else.
func applyLib(t *testing.T, d *declaration.Declaration, known store.State) (Report, store.State, error) {
t.Helper()
run := func(_ context.Context, name string, args ...string) (string, error) {
if name == "getent" && args[0] == "group" {
for _, g := range d.Groups {
if args[1] == g.Name || args[1] == strconv.Itoa(g.GID) {
return fmt.Sprintf("%s:x:%d:\n", g.Name, g.GID), nil
}
}
return "", errors.New("getent exited 2")
}
return "", nil
}
return Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
}
func statOf(t *testing.T, path string) unix.Stat_t {
t.Helper()
var st unix.Stat_t
if err := unix.Lstat(path, &st); err != nil {
t.Fatal(err)
}
return st
}
// Row 6: the root and every requested sub-path kept in the library's group with group rwx and setgid on every apply —
// those folders alone, never what is in them; owner, owner's and others' bits unchanged.
func TestALibraryIsKeptInItsGroup(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o755); err != nil {
t.Fatal(err)
}
d := libraryDecl(t, libJSON("media", root, gid, true, false, "movies"))
if _, _, err := applyLib(t, d, store.State{}); err != nil {
t.Fatal(err)
}
// Kept since it was empty; an app then narrows a requested folder and writes beneath it.
movies := filepath.Join(root, "movies")
if err := os.Chown(movies, -1, os.Getgid()); err != nil {
t.Fatal(err)
}
if err := unix.Chmod(movies, 0o705); err != nil {
t.Fatal(err)
}
film := filepath.Join(movies, "film")
if err := os.WriteFile(film, []byte("data"), 0o640); err != nil {
t.Fatal(err)
}
filmBefore := statOf(t, film)
before := statOf(t, movies)
report, _, err := applyLib(t, d, store.State{})
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "library:media"); o.Action != "updated" {
t.Fatalf("a kept library was not put back: %+v", o)
}
after := statOf(t, movies)
if int(after.Gid) != gid || after.Mode&0o7777 != 0o2775 {
t.Fatalf("a requested path is not kept in the group with group rwx and setgid: gid %d, mode %o", after.Gid,
after.Mode&0o7777)
}
if after.Uid != before.Uid || after.Mode&0o707 != before.Mode&0o707 {
t.Errorf("the owner or the owner's and others' bits changed: %o to %o", before.Mode&0o7777, after.Mode&0o7777)
}
if f := statOf(t, film); f.Mode != filmBefore.Mode || f.Gid != filmBefore.Gid {
t.Error("a file under a kept path was changed without the take act")
}
if rec, _ := libraryRecord("media"); !rec.Kept || rec.How != store.KeptEmpty {
t.Errorf("not recorded kept as empty: %+v", rec)
}
}
// Row 9: a missing sub-path is made with the root's owner, the library's group, group rwx and setgid.
func TestAMissingSubPathIsMadeAsTheRootIs(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o751); err != nil {
t.Fatal(err)
}
if err := os.Chmod(root, 0o751); err != nil {
t.Fatal(err)
}
d := libraryDecl(t, libJSON("media", root, gid, true, false, "tv/shows"))
if _, _, err := applyLib(t, d, store.State{}); err != nil {
t.Fatalf("an empty root was not kept: %v", err)
}
rst := statOf(t, root)
for _, p := range []string{filepath.Join(root, "tv"), filepath.Join(root, "tv", "shows")} {
st := statOf(t, p)
if st.Uid != rst.Uid || int(st.Gid) != gid || st.Mode&0o7777 != 0o2771 {
t.Errorf("%s made as uid %d gid %d mode %o, and the root's owner, the group and 2771 were wanted", p,
st.Uid, st.Gid, st.Mode&0o7777)
}
}
if rec, _ := libraryRecord("media"); rec.How != store.KeptEmpty {
t.Errorf("an empty root is not kept as empty: %+v", rec)
}
}
// Row 7 (keep wait) and §8: a root that holds data not in the group waits with reason keep, and nothing is changed.
func TestARootThatHoldsDataWaitsForTheOperatorsWord(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.MkdirAll(filepath.Join(root, "movies"), 0o755); err != nil {
t.Fatal(err)
}
before := statOf(t, root)
d := libraryDecl(t, libJSON("media", root, gid, true, false, "movies"))
report, _, err := applyLib(t, d, store.State{})
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "library:media"); o.Action != "waiting" || !strings.Contains(o.Detail, KeepWords("media")) {
t.Fatalf("a root holding data did not wait for the operator: %+v", o)
}
if after := statOf(t, root); after.Gid != before.Gid || after.Mode != before.Mode {
t.Fatal("the root was changed without a warrant")
}
j := JudgeLibrary(context.Background(), d.Libraries[0])
if j.Kind != VerdictWaitKeep || j.Placed == "" || strings.Contains(KeepWords("media")+DiskWords("media"), "/") {
t.Fatalf("the look does not wait for the operator, naming the filesystem with no path: %+v", j)
}
}
// Row 7, the act: group, group write and setgid set under the root; no link followed; no other filesystem entered;
// a file with another name elsewhere left; owner, owner's and others' bits unchanged; nothing without a warrant
// (the act's ask, internal/link); an undeclared library refused; a walk over its bound goes on from its cursor.
func TestTheTakeActSetsTheGroupUnderTheRoot(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
base := t.TempDir()
root := filepath.Join(base, "media")
outside := filepath.Join(base, "outside")
for _, d := range []string{filepath.Join(root, "movies", "a film"), outside} {
if err := os.MkdirAll(d, 0o705); err != nil {
t.Fatal(err)
}
}
file := filepath.Join(root, "movies", "a film", "film.mkv")
if err := os.WriteFile(file, []byte("data"), 0o604); err != nil {
t.Fatal(err)
}
if err := unix.Chmod(file, 0o4604); err != nil { // its own setuid bit, which a group change drops
t.Fatal(err)
}
secret := filepath.Join(outside, "secret")
if err := os.WriteFile(secret, nil, 0o600); err != nil {
t.Fatal(err)
}
if err := os.Link(secret, filepath.Join(root, "movies", "hardlink")); err != nil {
t.Fatal(err)
}
if os.Geteuid() == 0 { // an account's file, as an *arr app's hardlinked download is
if err := os.Chown(secret, 65534, -1); err != nil {
t.Fatal(err)
}
}
if err := os.Symlink(outside, filepath.Join(root, "link")); err != nil {
t.Fatal(err)
}
d := libraryDecl(t, libJSON("media", root, gid, true, false, "movies"))
if _, _, err := applyLib(t, d, store.State{}); err != nil {
t.Fatal(err)
}
if _, _, err := TakeLibrary(context.Background(), d, "library:books", root, "w", "op"); err == nil {
t.Fatal("an undeclared library was taken")
}
// Bounded: three entries a walk, gone on from the cursor until done.
was := ActLimits
ActLimits = WalkLimits{Entries: 3, Time: time.Minute}
defer func() { ActLimits = was }()
done, said, err := TakeLibrary(context.Background(), d, "library:media", root, "w-1", "the operator")
if err != nil {
t.Fatalf("the take act was refused: %v", err)
}
<-done
if !strings.Contains(said, "w-1") {
t.Errorf("the act does not say its warrant: %s", said)
}
rec, _ := libraryRecord("media")
if rec.Act == nil || rec.Act.State != "stopped" || len(rec.Cursor) == 0 {
t.Fatalf("a walk over its bound does not say where it stopped: %+v %+v", rec.Act, rec.Cursor)
}
for i := 0; i < 10; i++ {
ReconcileLibraries(context.Background(), d)
if rec, _ = libraryRecord("media"); rec.Act.State == "done" {
break
}
}
if rec.Act.State != "done" || rec.Act.Linked != 1 || rec.Act.Stripped != 1 || !rec.Kept || rec.How != store.KeptByWarrant ||
rec.KeptOn == "" {
t.Fatalf("the act is not recorded done under its warrant on its filesystem: %+v %+v", rec, rec.Act)
}
for _, p := range []string{root, filepath.Join(root, "movies"), filepath.Join(root, "movies", "a film")} {
if st := statOf(t, p); int(st.Gid) != gid || st.Mode&0o7777 != 0o2775 {
t.Errorf("%s: gid %d mode %o, and the group with 2775 (705 plus group rwx and setgid) was wanted", p,
st.Gid, st.Mode&0o7777)
}
}
if st := statOf(t, file); int(st.Gid) != gid || st.Mode&0o7777 != 0o0664 {
t.Errorf("a file: gid %d mode %o, and the group with 0664 (its setuid bit taken by the group change and "+
"never put back) was wanted", st.Gid, st.Mode&0o7777)
}
if st := statOf(t, secret); int(st.Gid) != gid || st.Mode&0o7777 != 0o660 {
t.Error("an account's file with another name (a hardlinked download) was not given the group")
}
if st := statOf(t, outside); int(st.Gid) == gid {
t.Error("the act followed a link out of the library")
}
}
// Row 7, the act: another filesystem is not entered — a folder whose device is not the root's is passed by.
func TestTheTakeActEntersNoOtherFilesystem(t *testing.T) {
gid := testGID(t)
root := t.TempDir()
mount := filepath.Join(root, "elsewhere")
if err := os.Mkdir(mount, 0o755); err != nil {
t.Fatal(err)
}
stats := walkStats{
at: func(dirfd int, name string, st *unix.Stat_t) error {
err := unix.Fstatat(dirfd, name, st, unix.AT_SYMLINK_NOFOLLOW)
if name == "elsewhere" {
st.Dev++
}
return err
},
fd: unix.Fstat,
}
w, err := walkRoot(root, ActLimits, stats, nil, groupVisit(gid))
if err != nil {
t.Fatal(err)
}
if w.counts.Mounts != 1 || int(statOf(t, mount).Gid) == gid {
t.Fatalf("the walk entered another filesystem: %+v", w.counts)
}
}
// Row 9: a root that is absent, or whose disk is not mounted, makes nothing, holds the containers that mount it,
// and waits with reason disk; off its home, a mount that is not there holds them too.
func TestALibraryWhoseDiskIsNotThereHoldsItsContainers(t *testing.T) {
gid := testGID(t)
base := t.TempDir()
disk := filepath.Join(base, "disk")
if err := os.Mkdir(disk, 0o755); err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
why, root string
home bool
targets []string
}{
{"its root absent", filepath.Join(base, "gone", "media"), true, nil},
{"its disk not mounted", filepath.Join(disk, "media"), true, []string{disk}},
{"its mount not there", disk, false, nil},
} {
libraries(t, nil, tc.targets)
d := libraryDecl(t, libJSON("media", tc.root, gid, tc.home, false, "movies"),
`{"id":"radarr.app","type":"container","name":"radarr","image":"`+pinned+`","volumes":["`+
filepath.Join(tc.root, "movies")+`:/movies"]}`)
var started bool
run := func(_ context.Context, name string, args ...string) (string, error) {
if len(args) > 0 && args[0] == "run" {
started = true
}
if name == "getent" && args[0] == "group" {
return fmt.Sprintf("mesh-media:x:%d:\n", gid), nil
}
return "", nil
}
report, _, _ := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, run, nil, nil)
if o := outcomeOf(report, "library:media"); o.Action != "waiting" || !strings.Contains(o.Detail, DiskWords("media")) {
t.Errorf("%s: the library does not wait for its disk: %+v", tc.why, o)
}
if o := outcomeOf(report, "radarr.app"); o.Action != "waiting" || started {
t.Errorf("%s: a container that mounts it was not held: %+v", tc.why, o)
}
if !WaitingOnLibrary(d)["radarr.app"] {
t.Errorf("%s: liveness is not told the container waits", tc.why)
}
if j := JudgeLibrary(context.Background(), d.Libraries[0]); j.Kind != VerdictWaitDisk {
t.Errorf("%s: the look judges it %s", tc.why, j.Kind)
}
if _, err := os.Stat(filepath.Join(tc.root, "movies")); !errors.Is(err, os.ErrNotExist) {
t.Errorf("%s: something was made", tc.why)
}
}
}
// Row 3: the library's group made with its number; a name or number that is another's refused; what was made, and
// only that, removed when no longer declared.
func TestALibrarysGroupIsMadeWithItsNumber(t *testing.T) {
m := &accountsMachine{users: map[string][2]int{}, groups: map[string][2]int{"media": {2000, 0}}}
d := withGroups(t, `{"name":"mesh-media","gid":62000}`, ``, ``)
report, known, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatal(err)
}
if strings.Join(m.ran, "\n") != "groupadd --system --gid 62000 mesh-media" {
t.Fatalf("made with: %v", m.ran)
}
if o := outcomeOf(report, "group mesh-media"); o.Action != "created" || len(known.Groups) != 1 {
t.Fatalf("not recorded as made: %+v %+v", o, known.Groups)
}
clash := withGroups(t, `{"name":"mesh-books","gid":2000}`, ``, ``)
if _, _, err := Apply(context.Background(), archHost(t), clash, known, store.OriginDeclared, m.run, nil, nil); err == nil ||
!strings.Contains(err.Error(), "media") {
t.Errorf("a number that is another group's was not refused naming it: %v", err)
}
m.ran = nil
none := withGroups(t, ``, ``, `{"id":"x.d","type":"directory","path":"`+t.TempDir()+`/x"}`)
if _, known, err = Apply(context.Background(), archHost(t), none, known, store.OriginDeclared, m.run, nil, nil); err != nil {
t.Fatal(err)
}
if _, still := m.groups["mesh-media"]; still || len(known.Groups) != 0 {
t.Fatalf("a library group no longer declared was not removed: %v", m.ran)
}
if _, kept := m.groups["media"]; !kept {
t.Fatal("a group the mesh did not make was removed")
}
}
// Row 4: a module's user with its own group (its uid's number), its first library's group as primary and its other
// libraries' as added groups.
func TestAModulesUserIsInItsLibrariesGroups(t *testing.T) {
m := &accountsMachine{users: map[string][2]int{}, groups: map[string][2]int{}}
d := withGroups(t, `{"name":"mesh-media","gid":62000},{"name":"mesh-books","gid":62001}`,
`{"module":"radarr","name":"mesh-radarr","uid":61001,"gid":62000,"groups":[62001]}`, ``)
if _, known, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil,
nil); err != nil {
t.Fatal(err)
} else if len(known.Identities) != 1 {
t.Fatalf("the identity is not recorded: %+v", known.Identities)
}
want := []string{
"groupadd --system --gid 62000 mesh-media",
"groupadd --system --gid 62001 mesh-books",
"groupadd --system --gid 61001 mesh-radarr",
"useradd --system --uid 61001 --gid 62000 --groups mesh-books --no-create-home --home-dir /nonexistent --shell " +
"/usr/bin/nologin mesh-radarr",
}
if strings.Join(m.ran, "\n") != strings.Join(want, "\n") {
t.Fatalf("made with:\n%s\nwant:\n%s", strings.Join(m.ran, "\n"), strings.Join(want, "\n"))
}
}
// Row 11: the tool runner's account is put in each library's group once, its unit restarted for it (a change of
// library:<name>), and taken out once no library puts it there.
func TestTheToolRunnerIsInItsLibrariesGroups(t *testing.T) {
gid := testGID(t)
libraries(t, []string{"/srv/media"}, nil)
m := &accountsMachine{users: map[string][2]int{"nox": {1003, 1003}}, groups: map[string][2]int{"nox": {1003, 0}}}
lib := `{"library":"media","root":"/srv/media","group":"mesh-media","gid":` + strconv.Itoa(gid) +
`,"home":false,"runner":"nox"}`
d := libraryDecl(t, lib)
report, known, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(m.ran, "\n"), "usermod --append --groups mesh-media nox") {
t.Fatalf("the tool runner was not put in the library's group: %v", m.ran)
}
if o := outcomeOf(report, "library:media"); o.Action != "updated" || len(known.Members) != 1 {
t.Fatalf("its joining is not a change of library:media, or not recorded: %+v %+v", o, known.Members)
}
m.ran = nil
if _, known, err = Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, m.run, nil, nil); err != nil {
t.Fatal(err)
}
if strings.Contains(strings.Join(m.ran, "\n"), "usermod") {
t.Fatal("the tool runner was put in its group again")
}
none := withGroups(t, ``, ``, `{"id":"x.d","type":"directory","path":"`+t.TempDir()+`/x"}`)
if _, known, err = Apply(context.Background(), archHost(t), none, known, store.OriginDeclared, m.run, nil, nil); err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(m.ran, "\n"), "gpasswd --delete nox mesh-media") || len(known.Members) != 0 {
t.Fatalf("the tool runner was not taken out of a group no library puts it in: %v", m.ran)
}
}
// Row 15: a library group, an identity's added group or an image told a root group's number is refused with its
// reason, and nothing is made or started.
func TestNoLibraryGroupIsRoot(t *testing.T) {
was := RootGroupIDs
RootGroupIDs = func() map[int]string { return map[int]string{10: "wheel"} }
defer func() { RootGroupIDs = was }()
dir := t.TempDir()
for _, d := range []*declaration.Declaration{
withGroups(t, `{"name":"mesh-media","gid":10}`, ``, `{"id":"m.d","type":"directory","path":"`+dir+`/d"}`),
withGroups(t, `{"name":"mesh-media","gid":10}`,
`{"module":"radarr","name":"mesh-radarr","uid":61001,"gid":61001,"groups":[10]}`,
`{"id":"m.d","type":"directory","path":"`+dir+`/d"}`),
} {
m := &accountsMachine{users: map[string][2]int{}, groups: map[string][2]int{}}
_, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err == nil || !strings.Contains(err.Error(), "wheel") {
t.Errorf("a root group was not refused naming why: %v", err)
}
if len(m.ran) != 0 {
t.Errorf("something was made: %v", m.ran)
}
if _, err := os.Stat(filepath.Join(dir, "d")); err == nil {
t.Error("a resource of a declaration naming a root group was applied")
}
}
if _, err := declaration.Parse([]byte(`{"declaration":1,"groups":[{"name":"mesh-media","gid":0}],` +
`"owns_nothing":true,"resources":[]}`)); err == nil || !strings.Contains(err.Error(), "never root") {
t.Errorf("a library group of number 0 was not refused on arrival: %v", err)
}
}
// A library's filesystem that cannot be named this time is unknown, never a move; another filesystem at the root is.
func TestALibraryNotNamedIsUnknownAndAnotherFilesystemIsAMove(t *testing.T) {
gid := testGID(t)
base := t.TempDir()
pool := filepath.Join(base, "pool")
root := filepath.Join(pool, "media")
if err := os.MkdirAll(filepath.Join(root, "old"), 0o755); err != nil {
t.Fatal(err)
}
libraries(t, []string{pool}, nil)
uuid := "6f1c0a4e-1b2d-4c3e-8f00-aa11bb22cc33"
wasUUID := MountUUID
MountUUID = func(p string) string {
if p == pool {
return uuid
}
return "x"
}
defer func() { MountUUID = wasUUID }()
d := libraryDecl(t, libJSON("media", root, gid, true, false))
if _, _, err := applyLib(t, d, store.State{}); err != nil {
t.Fatal(err)
}
done, _, err := TakeLibrary(context.Background(), d, "library:media", root, "w-1", "op")
if err != nil {
t.Fatal(err)
}
<-done
taken := libraryDecl(t, libJSON("media", root, gid, true, true))
before, _ := libraryRecord("media")
uuid = "" // nothing names it now
wasFsid := MountFsid
MountFsid = func(string) string { return "" }
wasSource := MountSource
MountSource = func(string) (string, string) { return "", "" }
report, _, _ := applyLib(t, taken, store.State{})
MountFsid, MountSource = wasFsid, wasSource
if o := outcomeOf(report, "library:media"); o.Action != "waiting" || !strings.Contains(o.Detail, "could not be named") {
t.Fatalf("an unnamed filesystem is not said so: %+v", o)
}
if after, _ := libraryRecord("media"); !after.Kept || after.KeptOn != before.KeptOn {
t.Fatalf("a name not read reset the record: %+v", after)
}
uuid = "00000000-1111-2222-3333-444444444444" // another pool at the same point, its root not in the group
if err := os.Chown(root, -1, os.Getgid()); err != nil {
t.Fatal(err)
}
if err := unix.Chmod(root, 0o755); err != nil {
t.Fatal(err)
}
report, _, _ = applyLib(t, taken, store.State{})
if o := outcomeOf(report, "library:media"); o.Action != "waiting" || !strings.Contains(o.Detail, KeepWords("media")) {
t.Fatalf("another filesystem at the root was kept on the old warrant: %+v", o)
}
}
// Old-shape parity: a controller older than ADR 0328 sends no groups, identities or libraries, and an access is
// applied exactly as before ADR 0325 — mounted when present, refused when absent, nothing kept, made or held.
func TestAnOlderControllersDeclarationIsAppliedAsBefore(t *testing.T) {
libraries(t, nil, nil)
dir := t.TempDir()
present := filepath.Join(dir, "library")
absent := filepath.Join(dir, "media", "movies")
if err := os.Mkdir(present, 0o755); err != nil {
t.Fatal(err)
}
before := statOf(t, present)
resources := []string{
`{"id":"radarr.library","type":"access","path":"` + present + `","mode":"read-write","users":[` +
`{"identity":"mesh-radarr","uid":61001,"use":"read-write","from":"container"}],"taken":true}`,
`{"id":"plex.movies","type":"access","path":"` + absent + `"}`,
}
report, _, err := Apply(context.Background(), archHost(t), declare(t, strings.Join(resources, ",")), store.State{},
store.OriginDeclared, noServices, nil, nil)
if err == nil || !strings.Contains(err.Error(), absent+" is not there, and the mesh does not own it — the operator "+
"must provide it") {
t.Fatalf("an absent access is not refused as before: %v", err)
}
if o := outcomeOf(report, "radarr.library"); o.Action != "unchanged" ||
o.Detail != "operator-owned; present, read-write, nothing managed" {
t.Errorf("a present access is not as before: %+v", o)
}
if after := statOf(t, present); after.Gid != before.Gid || after.Mode != before.Mode {
t.Error("a present access was changed")
}
if n, _ := unix.Getxattr(present, "system.posix_acl_access", nil); n > 0 {
t.Error("an ACL was set on an access")
}
if _, err := os.Stat(absent); !errors.Is(err, os.ErrNotExist) {
t.Error("an absent access was made")
}
}
func jsonInto(raw string, into any) error { return json.Unmarshal([]byte(raw), into) }
// Row 4: a module that joins a library later is given its group as primary on the account the mesh made; an account
// the mesh did not make is never changed, and is refused if it is not as declared.
func TestAModuleThatJoinsALibraryLaterIsGivenItsGroup(t *testing.T) {
m := &accountsMachine{users: map[string][2]int{}, groups: map[string][2]int{}}
own := withGroups(t, `{"name":"mesh-media","gid":62000}`,
`{"module":"radarr","name":"mesh-radarr","uid":61001,"gid":61001}`, ``)
_, known, err := Apply(context.Background(), archHost(t), own, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatal(err)
}
m.ran = nil
joined := withGroups(t, `{"name":"mesh-media","gid":62000}`,
`{"module":"radarr","name":"mesh-radarr","uid":61001,"gid":62000}`, ``)
if _, known, err = Apply(context.Background(), archHost(t), joined, known, store.OriginDeclared, m.run, nil, nil); err != nil {
t.Fatal(err)
}
if strings.Join(m.ran, "\n") != "usermod --gid 62000 --groups mesh-radarr" || m.users["mesh-radarr"][1] != 62000 {
t.Fatalf("the mesh's account was not given its library's group: %v", m.ran)
}
theirs := &accountsMachine{users: map[string][2]int{"mesh-radarr": {61001, 61001}},
groups: map[string][2]int{"mesh-radarr": {61001, 0}, "mesh-media": {62000, 0}}}
if _, _, err := Apply(context.Background(), archHost(t), joined, store.State{}, store.OriginDeclared, theirs.run,
nil, nil); err == nil || !strings.Contains(err.Error(), "was not made by the mesh") {
t.Fatalf("an account the mesh did not make was changed, or not refused: %v %v", err, theirs.ran)
}
}
// A library's root is never in the machine's own trees, as an access's never is (novox/hq issue 339).
func TestALibraryIsNeverTheMachinesOwnTree(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
for _, root := range []string{"/etc", "/usr/share", "/var/lib/mesh-host/x"} {
d := libraryDecl(t, libJSON("sys", root, gid, true, true))
if _, _, err := applyLib(t, d, store.State{}); err == nil {
t.Errorf("a library at %s was not refused", root)
}
if _, _, err := TakeLibrary(context.Background(), d, "library:sys", root, "w", "op"); err == nil {
t.Errorf("a library at %s was taken", root)
}
}
}
// §7: a root in shape with old data beneath it not in the group is asked about, never kept without a warrant; a root
// and sample all in shape are kept; once taken, never asked again on that filesystem.
func TestARootInShapeWithDataBelowNotInTheGroupWaits(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
movies := filepath.Join(root, "movies")
if err := os.MkdirAll(movies, 0o755); err != nil {
t.Fatal(err)
}
old := filepath.Join(movies, "old film")
if err := os.WriteFile(old, []byte("data"), 0o644); err != nil {
t.Fatal(err)
}
for _, p := range []string{root, movies} {
if err := os.Chown(p, -1, gid); err != nil {
t.Fatal(err)
}
if err := unix.Chmod(p, 0o2775); err != nil {
t.Fatal(err)
}
}
d := libraryDecl(t, libJSON("media", root, gid, true, false, "movies"))
report, _, err := applyLib(t, d, store.State{})
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "library:media"); o.Action != "waiting" || !strings.Contains(o.Detail, KeepWords("media")) {
t.Fatalf("a root in shape with an old file below not in the group was kept without asking: %+v", o)
}
// The old file in the group too: the sample is in shape, and the library is kept.
if err := os.Chown(old, -1, gid); err != nil {
t.Fatal(err)
}
if err := unix.Chmod(old, 0o664); err != nil {
t.Fatal(err)
}
if report, _, _ = applyLib(t, d, store.State{}); outcomeOf(report, "library:media").Action == "waiting" {
t.Fatal("a root and sample in shape were not kept")
}
// Once kept, a file an app later leaves out of the group asks nothing again on that filesystem.
if err := os.WriteFile(filepath.Join(movies, "new"), nil, 0o600); err != nil {
t.Fatal(err)
}
if report, _, _ = applyLib(t, d, store.State{}); outcomeOf(report, "library:media").Action == "waiting" {
t.Fatal("a kept library asked again")
}
}
// Review: a file with another name that root owns is left as it is (its other name may be anywhere), and a setgid
// file never keeps its setgid bit once regrouped.
func TestRootsLinkedFilesAreLeftAndSetgidIsNeverKept(t *testing.T) {
gid := testGID(t)
f, err := os.CreateTemp(t.TempDir(), "file")
if err != nil {
t.Fatal(err)
}
defer f.Close()
if err := unix.Chmod(f.Name(), 0o2755); err != nil {
t.Fatal(err)
}
st := statOf(t, f.Name())
rootLinked := st
rootLinked.Uid, rootLinked.Nlink = 0, 2
if did, err := groupVisit(gid)(int(f.Fd()), rootLinked, false); err != nil || did != visitShared {
t.Fatalf("a root-owned file with another name was not left: %d %v", did, err)
}
if after := statOf(t, f.Name()); after.Gid != st.Gid || after.Mode != st.Mode {
t.Fatal("a root-owned file with another name was changed")
}
did, err := groupVisit(gid)(int(f.Fd()), st, false)
if err != nil || did&visitStripped == 0 {
t.Fatalf("a setgid file's bit is not said taken: %d %v", did, err)
}
if after := statOf(t, f.Name()); after.Mode&(unix.S_ISGID|unix.S_ISUID) != 0 || int(after.Gid) != gid {
t.Fatalf("a setgid file kept its bit in the library's group: %o", after.Mode&0o7777)
}
}
// Review: a sub-path asked for after the library was kept without a warrant, which holds data not in the group,
// waits for the operator's word as the root would, and nothing of it is changed; an empty one is kept.
func TestASubPathHoldingOldDataWaitsForTheOperatorsWord(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o755); err != nil {
t.Fatal(err)
}
if _, _, err := applyLib(t, libraryDecl(t, libJSON("media", root, gid, true, false, "movies")), store.State{}); err != nil {
t.Fatal(err)
}
// Someone puts old data at a sub-path the library is asked for only now.
music := filepath.Join(root, "music")
if err := os.MkdirAll(filepath.Join(music, "album"), 0o755); err != nil {
t.Fatal(err)
}
before := statOf(t, music)
report, _, err := applyLib(t, libraryDecl(t, libJSON("media", root, gid, true, false, "movies", "music")), store.State{})
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "library:media"); o.Action != "waiting" || !strings.Contains(o.Detail, KeepWords("media")) {
t.Fatalf("a sub-path holding old data was kept without a warrant: %+v", o)
}
if after := statOf(t, music); after.Gid != before.Gid || after.Mode != before.Mode {
t.Fatal("a sub-path holding old data was changed")
}
// An empty one asked for is kept.
if err := os.RemoveAll(filepath.Join(music, "album")); err != nil {
t.Fatal(err)
}
report, _, _ = applyLib(t, libraryDecl(t, libJSON("media", root, gid, true, false, "movies", "music")), store.State{})
if o := outcomeOf(report, "library:media"); o.Action == "waiting" {
t.Fatalf("an empty sub-path was not kept: %+v", o)
}
}
// Review: an apply does not wait for a take walk that is running: it leaves the library to it.
func TestAnApplyDoesNotWaitForATakeWalk(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o755); err != nil {
t.Fatal(err)
}
lock := libraryLock("media")
lock.Lock()
defer lock.Unlock()
libraryWalks.Store("media", "fs-test")
defer libraryWalks.Delete("media")
done := make(chan Report, 1)
go func() {
r, _, _ := applyLib(t, libraryDecl(t, libJSON("media", root, gid, true, false)), store.State{})
done <- r
}()
select {
case r := <-done:
if o := outcomeOf(r, "library:media"); !strings.Contains(o.Detail, "take act") {
t.Fatalf("the apply does not say it left the library to the walk: %+v", o)
}
case <-time.After(2 * time.Second):
t.Fatal("the apply waited for the take walk")
}
}
// Review: a root in shape whose data beneath — in no requested sub-path — is not in the group waits: the sample, not
// the root alone, says whether the data is the library's already (§7).
func TestTheSampleBeneathARootInShapeIsJudged(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.MkdirAll(filepath.Join(root, "old"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, "old", "film"), []byte("data"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chown(root, -1, gid); err != nil {
t.Fatal(err)
}
if err := unix.Chmod(root, 0o2775); err != nil {
t.Fatal(err)
}
report, _, err := applyLib(t, libraryDecl(t, libJSON("media", root, gid, true, false)), store.State{})
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "library:media"); o.Action != "waiting" {
t.Fatalf("a root in shape with data beneath not in the group was kept: %+v", o)
}
}
// Review: an apply that finds the library's lock held by something that is no take walk (a take act about to be
// refused) waits for it and judges the library as always: a library whose disk is not there still holds.
func TestAnApplyWaitsForALockThatIsNoWalk(t *testing.T) {
gid := testGID(t)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "gone", "media")
lock := libraryLock("media")
lock.Lock()
go func() { time.Sleep(50 * time.Millisecond); lock.Unlock() }()
report, _, _ := applyLib(t, libraryDecl(t, libJSON("media", root, gid, true, false)), store.State{})
if o := outcomeOf(report, "library:media"); o.Action != "waiting" || !strings.Contains(o.Detail, DiskWords("media")) {
t.Fatalf("a library whose root is not there was read as kept while its lock was held: %+v", o)
}
}
+43
View File
@@ -0,0 +1,43 @@
package apply
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container declared to log to the journal is run with the journal as its log driver, and the
// place it logs is part of its spec, so moving it recreates the container (novox/hq ADR 0179).
func TestAContainerLoggingToTheJournalIsRunThatWayAndRecreatedWhenMoved(t *testing.T) {
pinned := "postgres@sha256:" + strings.Repeat("a", 64)
var ran []string
run := func(_ context.Context, cmd string, args ...string) (string, error) {
if cmd == "docker" && len(args) > 0 && args[0] == "run" {
ran = args
return "deadbeef\n", nil
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"front","type":"container","name":"front","image":"`+pinned+`","logging":"journald"}
]}`)
_, _, _ = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
sent := false
for i, a := range ran {
if a == "--log-driver" && i+1 < len(ran) && ran[i+1] == "journald" {
sent = true
}
}
if !sent {
t.Fatalf("the container's output was not sent to the journal: %v", ran)
}
with := d.Resources[0].(*declaration.Container)
without := *with
without.Logging = ""
if containerSpec(with, inputs{}) == containerSpec(&without, inputs{}) {
t.Fatal("where a container logs is not part of its spec, so moving it would not recreate it")
}
}
+248
View File
@@ -0,0 +1,248 @@
package apply
// What a maintenance window is holding still, written where every apply on this machine can read it
// (novox/hq issue 224, ADR 0189).
//
// A scheduled step declared `while-stopped` stops its module's containers, runs, and starts them
// again. Until this, nothing told the **apply** that a window was open, and the apply's rule for a
// container it finds stopped is the right rule everywhere else: stopped is broken, so remove it and
// create it running. In the middle of a window that rule reopens the window — for the store's
// collector, a registry recreated mid-collection accepts an upload the sweep then deletes, and the
// build that made the image reported success.
//
// **The apply learns which containers a window holds; the window does not take the apply lock.**
// The issue weighed both. Holding the lock for a window makes the race impossible and makes a push
// that arrives mid-window wait for minutes, which is what issue 185's outages looked like from
// outside. Telling the apply blocks nothing: a held container is reported as held and left exactly
// as it is — even when its declaration changed — and the first apply after the window converges it
// by the ordinary rule. The cost, stated in the issue, is a second source for "is this container
// meant to be running"; it is kept honest by being short-lived and by expiring on its own.
//
// **A file under the node's state directory, not a variable in the scheduler.** The scheduler lives
// in the daemon, but the daemon is not the only thing that applies here: `mesh-host reconcile` and
// `mesh-host apply` run as their own processes beside it, and the installer applies the carried
// bundle — which is where the store's registry comes from — from a third. A map shared in memory
// would protect only the daemon's own applies, which is to say it would leave open exactly the case
// a person running `reconcile` by hand at 03:31 creates. The state directory is already where every
// one of them meets (the apply lock, the node's state, the kept originals), it is root's alone, and
// each window is one small file written atomically, so no reader ever sees half of one.
//
// **A window that is never closed must not hold for ever** — the same risk the field itself carries,
// one level up. Three things close it:
//
// - the scheduler removes the file in a defer that runs after the containers are started again,
// whatever the step did;
// - a window whose process is gone is closed: the file names the process that opened it, and a
// host that crashed or was replaced mid-window is no longer holding anything, so the apply is
// free to bring back what it left stopped — which is then the apply doing the job the dead
// process could not;
// - and every window carries an end, windowAtMost after it opened, past which it is read as
// closed whoever is still alive. A step that genuinely runs longer than that is pathological, and
// the apply then does what it did before this record existed, which is the lesser harm next to a
// service the mesh can never bring back.
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"syscall"
"time"
"github.com/novox/mesh-host/internal/store"
)
// WindowsName is the directory beside the node's state that holds the open windows, one file each.
const WindowsName = "windows"
// windowAtMost is the longest a window is believed. The store's collector is minutes over a store of
// fifty-odd repositories; six hours is far past any collection this mesh will do and short enough
// that a window nothing closed is over by the next working day (novox/hq issue 224).
const windowAtMost = 6 * time.Hour
// Window is one scheduled step holding containers still: which step, which containers by runtime
// name, since when, and the latest moment it is believed.
type Window struct {
Step string `json:"step"`
Holds []string `json:"holds"`
Opened time.Time `json:"opened"`
Until time.Time `json:"until"`
// PID is the process that opened it. Not reported: it is how a window whose host died is read
// as closed.
PID int `json:"pid"`
}
// Windows is where this machine's open windows are recorded. A nil *Windows records nothing and
// holds nothing — a test, or a caller that has no state directory.
type Windows struct {
dir string
// alive says whether a process is still running. Replaced in tests; the real one asks the kernel.
alive func(pid int) bool
}
// WindowsIn records windows under stateDir/windows — the directory the node's state lives in, which
// every process that applies on this machine already shares.
func WindowsIn(stateDir string) *Windows {
return &Windows{dir: filepath.Join(stateDir, WindowsName), alive: processAlive}
}
// processAlive is whether pid names a running process. Signal 0 delivers nothing and only asks; a
// process this one may not signal is still a process (EPERM), which can only happen across users.
func processAlive(pid int) bool {
if pid <= 0 {
return false
}
err := syscall.Kill(pid, 0)
return err == nil || errors.Is(err, syscall.EPERM)
}
// fileFor is where one step's window is written. A step id is the declaration's, which may carry a
// dot or a slash; the file name keeps what is safe and replaces the rest.
func (w *Windows) fileFor(step string) string {
safe := strings.Map(func(r rune) rune {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_', r == '.':
return r
}
return '_'
}, step)
return filepath.Join(w.dir, "window-"+safe+".json")
}
// open records that step holds these containers still from now, before the first of them is
// stopped — so no apply can find one stopped and not know why.
func (w *Windows) open(step string, holds []string, now time.Time) error {
if w == nil {
return nil
}
if err := os.MkdirAll(w.dir, 0o700); err != nil {
return err
}
raw, err := json.Marshal(Window{Step: step, Holds: holds, Opened: now.UTC(),
Until: now.UTC().Add(windowAtMost), PID: os.Getpid()})
if err != nil {
return err
}
return writeAtomically(w.fileFor(step), raw, 0o600)
}
// How long a step about to open a window waits for an apply in flight on this machine to end, and how
// often it looks (novox/hq issue 291). Past the bound it opens the window anyway, said: the applies
// then wait for the window instead (store_away.go), which is the slower order, never a failed one.
var (
applyWaitAtMost = 15 * time.Minute
applyPoll = time.Second
)
// openAfterApplies opens step's window only once no apply is in flight on this machine (novox/hq issue
// 291): an apply that started before the window would otherwise reach the store's archives with its
// server already held still. The apply lock is held only while the record is written — every apply
// that starts afterwards reads the window and waits for the store — so a push arriving mid-window
// still never queues behind the window itself (issue 224's reason for not taking the lock).
func (w *Windows) openAfterApplies(ctx context.Context, step string, holds []string, log func(string)) error {
if w == nil {
return nil
}
stateDir := filepath.Dir(w.dir)
deadline := time.Now().Add(applyWaitAtMost)
waited := false
for {
release, took, err := store.TryLockIn(stateDir)
if err != nil {
return err
}
if took {
defer release()
if waited {
log(fmt.Sprintf("scheduled step %s: the apply in flight ended; its window opens now", step))
}
return w.open(step, holds, time.Now())
}
if !waited {
waited = true
log(fmt.Sprintf("scheduled step %s: an apply is in flight on this machine — the window opens once "+
"it ends, so nothing it fetches finds %s held still (novox/hq issue 291)", step, strings.Join(holds, ", ")))
}
if time.Now().After(deadline) {
log(fmt.Sprintf("scheduled step %s: the apply in flight did not end within %s; the window opens "+
"beside it, and it waits for the window", step, applyWaitAtMost))
return w.open(step, holds, time.Now())
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(applyPoll):
}
}
}
// close removes the record of step's window. Called after its containers are started again; a file
// already gone is the state wanted.
func (w *Windows) close(step string) error {
if w == nil {
return nil
}
if err := os.Remove(w.fileFor(step)); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
return nil
}
// Open is the windows open at now, by step. One that has ended, or whose process is gone, is not
// open — and its file is removed on the way, since nothing else will (novox/hq issue 224). A file
// that cannot be read is not a window either: the alternative is a machine whose containers are
// never converged because of a byte nobody can see.
func (w *Windows) Open(now time.Time) []Window {
if w == nil {
return nil
}
entries, err := os.ReadDir(w.dir)
if err != nil {
return nil
}
var open []Window
for _, e := range entries {
if e.IsDir() || !strings.HasPrefix(e.Name(), "window-") || !strings.HasSuffix(e.Name(), ".json") {
continue
}
path := filepath.Join(w.dir, e.Name())
raw, err := os.ReadFile(path)
if err != nil {
continue
}
var win Window
if err := json.Unmarshal(raw, &win); err != nil {
_ = os.Remove(path)
continue
}
if !now.Before(win.Until) || !w.alive(win.PID) {
_ = os.Remove(path)
continue
}
open = append(open, win)
}
sort.Slice(open, func(i, j int) bool { return open[i].Step < open[j].Step })
return open
}
// holding is the open window that holds the container named name, if one does.
func (w *Windows) holding(name string, now time.Time) (Window, bool) {
for _, win := range w.Open(now) {
for _, held := range win.Holds {
if held == name {
return win, true
}
}
}
return Window{}, false
}
// String is the window as a person reads it in a log line.
func (win Window) String() string {
return fmt.Sprintf("%s holds %s still since %s", win.Step, strings.Join(win.Holds, ", "),
win.Opened.Format(time.RFC3339))
}
+542
View File
@@ -0,0 +1,542 @@
package apply
import (
"context"
"errors"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A scheduled step may hold its module's own containers still while it runs (novox/hq ADR 0189,
// issue 108).
//
// What it exists for: the artifact store's collector walks the storage and requires every writer
// stopped. A run-once step runs beside containers and a scheduled one is the same container again,
// so the mesh had no way to say it — which is why the store it inherited has never collected
// anything. The risk the field brings is one shape only: a window that opens and never closes.
// Every test here is about that shape.
// windowRun records the order of stop / run / start, which is the whole of what is being asserted.
type windowRun struct {
mu sync.Mutex
order []string
failAt string // the arg[0] that should fail ("run" makes the step fail)
wontGo string // a container name that refuses to start again
}
func (w *windowRun) run(_ context.Context, _ string, args ...string) (string, error) {
w.mu.Lock()
defer w.mu.Unlock()
switch args[0] {
case "info":
return "27.0\n", nil
case "stop", "start":
w.order = append(w.order, args[0]+" "+args[1])
if args[0] == "start" && args[1] == w.wontGo {
return "", errors.New("the runtime refused")
}
case "run":
w.order = append(w.order, "run")
if w.failAt == "run" {
return "", errors.New("the step exited non-zero")
}
}
return "", nil
}
func (w *windowRun) seen() []string {
w.mu.Lock()
defer w.mu.Unlock()
return append([]string{}, w.order...)
}
// aStoreWithACollector is a module in the shape distribution has: a server that must not be
// writing, and a nightly step that walks its storage with the server held still.
func aStoreWithACollector(t *testing.T) *declaration.Declaration {
t.Helper()
return parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"mesh-registry","image":"`+pinned+`"},
{"id":"collect","type":"container","name":"mesh-registry-collect","image":"`+pinned+`",
"schedule":"30 3 * * *","while-stopped":["store"]}
]}`)
}
func fireOnce(t *testing.T, d *declaration.Declaration, w *windowRun) {
t.Helper()
clock := &fixedClock{now: time.Date(2026, 10, 2, 3, 29, 0, 0, time.UTC)}
s := NewScheduler(clock, w.run, func(string) {})
s.Sync(d, nil)
s.Advance(context.Background(), time.Date(2026, 10, 2, 3, 30, 5, 0, time.UTC))
s.Wait()
}
func TestAScheduledStepHoldsItsModulesContainerStillAndStartsItAgain(t *testing.T) {
w := &windowRun{}
fireOnce(t, aStoreWithACollector(t), w)
got := w.seen()
want := []string{"stop mesh-registry", "run", "start mesh-registry"}
var kept []string
for _, line := range got {
if strings.HasPrefix(line, "stop mesh-registry-collect") {
// Clearing the step's own exited container by name; not part of the window.
continue
}
kept = append(kept, line)
}
if len(kept) != len(want) {
t.Fatalf("the window was not stop, run, start: %v", got)
}
for i := range want {
if kept[i] != want[i] {
t.Fatalf("the window was %v, want %v", kept, want)
}
}
}
// The one that matters: a step that fails must leave the service running.
func TestAFailedStepStillClosesTheWindow(t *testing.T) {
w := &windowRun{failAt: "run"}
fireOnce(t, aStoreWithACollector(t), w)
var started bool
for _, line := range w.seen() {
if line == "start mesh-registry" {
started = true
}
}
if !started {
t.Fatalf("the step failed and the container it held still was never started again: %v", w.seen())
}
}
// A container that will not come back is said loudly: it is down, and nothing else notices until
// the next apply compares it.
func TestAContainerThatWillNotStartAgainIsSaidLoudly(t *testing.T) {
w := &windowRun{wontGo: "mesh-registry"}
var said []string
clock := &fixedClock{now: time.Date(2026, 10, 2, 3, 29, 0, 0, time.UTC)}
s := NewScheduler(clock, w.run, func(line string) { said = append(said, line) })
s.Sync(aStoreWithACollector(t), nil)
s.Advance(context.Background(), time.Date(2026, 10, 2, 3, 30, 5, 0, time.UTC))
s.Wait()
var loud bool
for _, line := range said {
if strings.Contains(line, "WILL NOT START AGAIN") && strings.Contains(line, "mesh-registry") {
loud = true
}
}
if !loud {
t.Fatalf("a service left stopped by a maintenance window was not said loudly: %v", said)
}
}
// Several containers come back in the reverse of the order they were stopped: a module names the
// dependant first, and starting it before what it depends on is not bringing it back.
func TestTheWindowClosesInTheReverseOfTheOrderItOpened(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"web","type":"container","name":"web","image":"`+pinned+`"},
{"id":"db","type":"container","name":"db","image":"`+pinned+`"},
{"id":"collect","type":"container","name":"collect","image":"`+pinned+`",
"schedule":"30 3 * * *","while-stopped":["web","db"]}
]}`)
w := &windowRun{}
fireOnce(t, d, w)
var stops, starts []string
for _, line := range w.seen() {
switch {
case line == "stop web" || line == "stop db":
stops = append(stops, line)
case strings.HasPrefix(line, "start "):
starts = append(starts, line)
}
}
if len(stops) != 2 || stops[0] != "stop web" || stops[1] != "stop db" {
t.Fatalf("stopped in %v, want the order the step named them", stops)
}
if len(starts) != 2 || starts[0] != "start db" || starts[1] != "start web" {
t.Fatalf("started in %v, want the reverse", starts)
}
}
// And the refusals, each for what it says rather than that it says something.
func TestAMaintenanceWindowIsRefusedWhereItCannotMean(t *testing.T) {
for _, c := range []struct{ name, body, says string }{
{
"a window with no schedule",
`{"id":"collect","type":"container","name":"c","image":"` + pinned + `","while-stopped":["store"]}`,
"needs a schedule",
},
{
"a window naming itself",
`{"id":"collect","type":"container","name":"c","image":"` + pinned + `","schedule":"30 3 * * *","while-stopped":["collect"]}`,
"this step itself",
},
{
"a window naming something that is not a container here",
`{"id":"collect","type":"container","name":"c","image":"` + pinned + `","schedule":"30 3 * * *","while-stopped":["elsewhere"]}`,
"no container by that id",
},
} {
_, err := declaration.ParseTrusted([]byte(`{"declaration":1,"resources":[` + c.body + `]}`))
if err == nil {
t.Errorf("%s was accepted", c.name)
continue
}
if !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: the refusal does not say %q: %v", c.name, c.says, err)
}
}
}
// A changed window is a changed declaration, and the install says so.
//
// The cadence already works this way: "a changed schedule is a changed spec — the marker moves and
// the install is reported updated and re-established" (containerSpec). Which containers are held
// still for the run is the same kind of statement, and a declaration that changed it while the
// machine reported no change would be a machine quietly running the old window.
func TestAChangedWindowMovesTheSpec(t *testing.T) {
one := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"mesh-registry","image":"`+pinned+`"},
{"id":"other","type":"container","name":"other","image":"`+pinned+`"},
{"id":"collect","type":"container","name":"collect","image":"`+pinned+`",
"schedule":"30 3 * * *","while-stopped":["store"]}
]}`)
two := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"mesh-registry","image":"`+pinned+`"},
{"id":"other","type":"container","name":"other","image":"`+pinned+`"},
{"id":"collect","type":"container","name":"collect","image":"`+pinned+`",
"schedule":"30 3 * * *","while-stopped":["store","other"]}
]}`)
stepOf := func(d *declaration.Declaration) *declaration.Container {
for _, r := range d.Resources {
if c, ok := r.(*declaration.Container); ok && c.ID == "collect" {
return c
}
}
t.Fatal("no step in the fixture")
return nil
}
if containerSpec(stepOf(one), inputs{}) == containerSpec(stepOf(two), inputs{}) {
t.Fatal("the window changed and the spec did not; the machine would report no change " +
"and keep holding the containers it held yesterday")
}
// And a container with no window is untouched by the field existing at all.
plain := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"mesh-registry","image":"`+pinned+`"}
]}`)
spec := containerSpec(plain.Resources[0].(*declaration.Container), inputs{})
if strings.Contains(spec, "while-stopped") || strings.Contains(spec, "held") {
t.Errorf("an ordinary container's spec mentions a field it does not set:\n%s", spec)
}
}
// ---- An apply arriving during a window (novox/hq issue 224) ----
//
// A window holds a container stopped on purpose; the apply's rule for a stopped container is to
// replace it. Until the apply could see the window, a push landing at 03:30 recreated the registry
// in the middle of its collection — the collector, already past its mark phase, then swept a blob
// that an upload had just written. These drive the scheduler and the apply against one pretend
// runtime, so what is asserted is what one does to the other.
// aRegistryMachine is a runtime holding one long-running container, mesh-registry, and running the
// step in the foreground until the test lets it finish.
type aRegistryMachine struct {
mu sync.Mutex
exists bool
running bool
spec string
calls []string
wontGo bool // the registry refuses to start again after the window
failRun bool // the step exits non-zero
stepIn chan struct{} // the step has started: the window is open
stepGo chan struct{} // let the step finish
}
func newRegistryMachine(spec string) *aRegistryMachine {
return &aRegistryMachine{exists: true, running: true, spec: spec,
stepIn: make(chan struct{}, 4), stepGo: make(chan struct{})}
}
func (m *aRegistryMachine) run(_ context.Context, _ string, args ...string) (string, error) {
m.mu.Lock()
switch args[0] {
case "info", "image":
m.mu.Unlock()
return "27.0\n", nil
case "stop":
m.calls = append(m.calls, "stop "+args[1])
if args[1] == "mesh-registry" {
m.running = false
}
case "start":
m.calls = append(m.calls, "start "+args[1])
if args[1] == "mesh-registry" {
if m.wontGo {
m.mu.Unlock()
return "", errors.New("the runtime refused")
}
m.running = true
}
case "container": // inspect
name := args[len(args)-1]
if name != "mesh-registry" || !m.exists {
m.mu.Unlock()
return "", errors.New("no such container")
}
out := "false\t" + m.spec
if m.running {
out = "true\t" + m.spec
}
m.mu.Unlock()
return out, nil
case "rm":
name := args[len(args)-1]
m.calls = append(m.calls, "rm "+name)
if name == "mesh-registry" {
m.exists, m.running = false, false
}
case "run":
if !slices.Contains(args, "--detach") {
// The step, in the foreground, for as long as the test says.
m.calls = append(m.calls, "step")
fail := m.failRun
m.mu.Unlock()
m.stepIn <- struct{}{}
<-m.stepGo
if fail {
return "", errors.New("the step exited non-zero")
}
return "", nil
}
m.calls = append(m.calls, "create "+args[slices.Index(args, "--name")+1])
for i, a := range args {
if a == "--label" && strings.HasPrefix(args[i+1], specLabel+"=") {
m.spec = strings.TrimPrefix(args[i+1], specLabel+"=")
}
}
m.exists, m.running = true, true
}
m.mu.Unlock()
return "", nil
}
// touched is every call since mark that recreated or removed the registry.
func (m *aRegistryMachine) touched(mark int) []string {
m.mu.Lock()
defer m.mu.Unlock()
var out []string
for _, c := range m.calls[mark:] {
if c == "rm mesh-registry" || c == "create mesh-registry" {
out = append(out, c)
}
}
return out
}
func (m *aRegistryMachine) mark() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.calls)
}
// theStoreMoved is the store's module with the server's declaration changed — a push that landed
// at 03:30 with a new registry image, say.
func theStoreMoved(t *testing.T) *declaration.Declaration {
t.Helper()
return parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"mesh-registry","image":"`+pinned+`","env":{"NEW":"1"}},
{"id":"collect","type":"container","name":"mesh-registry-collect","image":"`+pinned+`",
"schedule":"30 3 * * *","while-stopped":["store"]}
]}`)
}
// openAWindow fires the collector and returns once it is running, the registry held still.
func openAWindow(t *testing.T, d *declaration.Declaration, m *aRegistryMachine, windows *Windows) *Scheduler {
t.Helper()
clock := &fixedClock{now: time.Date(2026, 10, 2, 3, 29, 0, 0, time.UTC)}
s := NewScheduler(clock, m.run, func(string) {})
s.RecordWindowsIn(windows)
s.Sync(d, nil)
s.Advance(context.Background(), time.Date(2026, 10, 2, 3, 30, 5, 0, time.UTC))
select {
case <-m.stepIn:
case <-time.After(5 * time.Second):
t.Fatal("the step never started")
}
return s
}
func TestAnApplyDuringAWindowLeavesWhatItHoldsAlone(t *testing.T) {
d := aStoreWithACollector(t)
want := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
m := newRegistryMachine(want)
windows := WindowsIn(t.TempDir())
s := openAWindow(t, d, m, windows)
mark := m.mark()
// The push lands mid-window, and with it a registry whose declaration moved: the very shape
// that, before this, removed the server and created it running under the collector.
report, state, err := ApplyMindingWindows(context.Background(), archHost(t), theStoreMoved(t),
store.State{}, store.OriginDeclared, m.run, nil, nil, nil, windows)
if err != nil {
t.Fatalf("the apply failed: %v", err)
}
if got := m.touched(mark); len(got) > 0 {
t.Fatalf("an apply during the window touched the container it holds still: %v", got)
}
o := outcomeOf(report, "store")
if o.Action != heldStill {
t.Fatalf("a held container was reported %q, want %q: %+v", o.Action, heldStill, o)
}
for _, says := range []string{"collect", "maintenance window", "first apply after", "declaration moved"} {
if !strings.Contains(o.Detail, says) {
t.Errorf("the outcome does not say %q: %s", says, o.Detail)
}
}
// Held still is not a change: nothing about the machine moved (the step's own install, first
// time here, is).
if (Report{Outcomes: []Outcome{o}}).Changed() {
t.Errorf("a container left alone reads as a change: %+v", o)
}
// The report says a window is open, and which.
if len(report.Windows) != 1 || report.Windows[0].Step != "collect" ||
len(report.Windows[0].Holds) != 1 || report.Windows[0].Holds[0] != "mesh-registry" {
t.Errorf("the report does not say the collector's window is open: %+v", report.Windows)
}
// The window closes: the registry is running again and the record is gone.
close(m.stepGo)
s.Wait()
if open := windows.Open(time.Now()); len(open) != 0 {
t.Fatalf("the window closed and is still recorded open: %+v", open)
}
// And the next apply converges it by the ordinary rule — the changed declaration recreates it.
mark = m.mark()
report, _, err = ApplyMindingWindows(context.Background(), archHost(t), theStoreMoved(t),
state, store.OriginDeclared, m.run, nil, nil, nil, windows)
if err != nil {
t.Fatalf("the apply after the window failed: %v", err)
}
if got := m.touched(mark); len(got) != 2 || got[0] != "rm mesh-registry" || got[1] != "create mesh-registry" {
t.Fatalf("the apply after the window did not converge the container: %v", got)
}
if o := outcomeOf(report, "store"); o.Action != "updated" {
t.Errorf("the container was reported %q after the window, want updated: %+v", o.Action, o)
}
if len(report.Windows) != 0 {
t.Errorf("a closed window is still reported open: %+v", report.Windows)
}
}
// A step that fails releases its hold like one that succeeds: the record is erased in the same
// defer that starts the server again, and an apply afterwards does what it always did — here, to a
// registry that would not come back, the one thing that brings it back.
func TestAFailedStepReleasesItsHold(t *testing.T) {
d := aStoreWithACollector(t)
m := newRegistryMachine(containerSpec(d.Resources[0].(*declaration.Container), inputs{}))
m.failRun, m.wontGo = true, true
windows := WindowsIn(t.TempDir())
s := openAWindow(t, d, m, windows)
if open := windows.Open(time.Now()); len(open) != 1 {
t.Fatalf("the window is not recorded while the step runs: %+v", open)
}
close(m.stepGo)
s.Wait()
if open := windows.Open(time.Now()); len(open) != 0 {
t.Fatalf("a failed step left its window recorded open: %+v", open)
}
mark := m.mark()
report, _, err := ApplyMindingWindows(context.Background(), archHost(t), d, store.State{},
store.OriginDeclared, m.run, nil, nil, nil, windows)
if err != nil {
t.Fatalf("the apply after the failed step failed: %v", err)
}
if got := m.touched(mark); len(got) != 2 {
t.Fatalf("the registry left stopped by a failed window was not brought back: %v", got)
}
if o := outcomeOf(report, "store"); o.Action == heldStill {
t.Errorf("the hold outlived the step: %+v", o)
}
}
// A window whose host died mid-step — crashed, or stood aside for a successor — holds nothing:
// the process that would start the server again is gone, so the apply must be free to. And a window
// nothing closed ends on its own.
func TestAWindowNothingClosesDoesNotHoldForEver(t *testing.T) {
dir := t.TempDir()
alive := true
windows := &Windows{dir: dir, alive: func(int) bool { return alive }}
now := time.Now()
if err := windows.open("collect", []string{"mesh-registry"}, now); err != nil {
t.Fatal(err)
}
if _, held := windows.holding("mesh-registry", now); !held {
t.Fatal("an open window with a live host does not hold its container")
}
if _, held := windows.holding("mesh-registry", now.Add(windowAtMost)); held {
t.Error("a window past its end still holds")
}
if err := windows.open("collect", []string{"mesh-registry"}, now); err != nil {
t.Fatal(err)
}
alive = false
if _, held := windows.holding("mesh-registry", now); held {
t.Error("a window whose host is gone still holds")
}
if entries, _ := os.ReadDir(dir); len(entries) != 0 {
t.Errorf("a window read as closed was left on disk: %v", entries)
}
// The real liveness check: this process is alive, and a pid nothing can have is not.
if !processAlive(os.Getpid()) || processAlive(0) {
t.Error("processAlive does not tell a live process from none")
}
}
// A window that cannot be recorded is not opened: an apply would not know, and would recreate the
// server under the step. One night's collection is the cheaper loss.
func TestAWindowThatCannotBeRecordedIsNotOpened(t *testing.T) {
notADir := filepath.Join(t.TempDir(), "state")
if err := os.WriteFile(notADir, nil, 0o600); err != nil {
t.Fatal(err)
}
d := aStoreWithACollector(t)
m := newRegistryMachine(containerSpec(d.Resources[0].(*declaration.Container), inputs{}))
var said []string
var mu sync.Mutex
clock := &fixedClock{now: time.Date(2026, 10, 2, 3, 29, 0, 0, time.UTC)}
s := NewScheduler(clock, m.run, func(line string) { mu.Lock(); said = append(said, line); mu.Unlock() })
s.RecordWindowsIn(WindowsIn(notADir))
s.Sync(d, nil)
s.Advance(context.Background(), time.Date(2026, 10, 2, 3, 30, 5, 0, time.UTC))
s.Wait()
for _, c := range m.calls {
if c == "stop mesh-registry" || c == "step" {
t.Fatalf("a window that could not be recorded was opened anyway: %v", m.calls)
}
}
mu.Lock()
defer mu.Unlock()
if len(said) == 0 || !strings.Contains(strings.Join(said, "\n"), "not run") {
t.Errorf("the step not running was not said: %v", said)
}
}
+125
View File
@@ -0,0 +1,125 @@
package apply
// A mesh directory outside a home is given its mode and owner only through a descriptor (novox/hq issue 358).
//
// Below a home every component is opened without following a link (ADR 0266, home_links.go), and a directory the
// operator handed to the mesh is changed through one descriptor opened from / with no link followed (issue 356).
// Every other directory the mesh owns — a module's state under /var/lib, a drop-in directory under /etc — was
// given its mode and owner by path, with chmod and chown, which follow a link anywhere in the path: whoever could
// turn a parent into a link between two applies directed root's chmod and chown at the link's target.
//
// The danger is a link another account can make or replace, not a link as such: /var/run is a link to /run on
// every node, and a disk is sometimes reached through one. So (the decision recorded on issue 358, 2026-10-11):
//
// - **the directory is opened component by component from /**, each with O_DIRECTORY|O_NOFOLLOW, and its mode
// and owner are changed with fchmod and fchown on that descriptor; a directory the walk makes is made in its
// parent's descriptor. A link put in place after any check is refused by the kernel at the moment of use;
// - **a link in a parent is followed only when root alone could have put it there**: the link is root's, and
// every directory from / down to the one holding it is root's and writable by no other account (no write for
// others, no write for a group other than root's). Its target is then walked the same way, from /;
// - **any other link refuses that directory**, naming the link and the account that could replace it; nothing
// is changed there, and the rest of the declaration applies;
// - **the directory itself is never a link** (issue 339, applyDirectory).
//
// A directory below a home keeps the walk of ADR 0266, which follows no link below the home at all.
import (
"fmt"
osuser "os/user"
"path/filepath"
"strconv"
"strings"
"github.com/novox/mesh-host/internal/declaration"
)
// LinkInPathError is a mesh directory refused because a link in its path is one another account could replace.
type LinkInPathError struct {
// Path is the directory declared, Link the link in its path, Who the account that could replace it.
Path, Link, Who string
}
func (e *LinkInPathError) Error() string { return e.Path + " " + e.why() }
// why is the refusal without the path it refuses, as a PlacementRefusedError says it.
func (e *LinkInPathError) why() string {
return fmt.Sprintf("is reached through the link %s, which %s could replace: the node-engine runs as root and "+
"follows a link to a directory it gives a mode and owner only when root alone could have made it (novox/hq "+
"issue 358). Make %s a directory, or make it and every directory above it root's and writable by no other "+
"account", e.Link, e.Who, e.Link)
}
// couldReplace says who besides root could replace an entry of a directory with this owner, group and mode: ""
// when nobody can. A variable so a test, which runs below a directory every account may write (/tmp), can name
// which of its own directories count as root's alone.
var couldReplace = func(path string, uid, gid, mode uint32) string {
if uid != 0 {
return "the account " + accountNamed(uid) + " (it owns " + path + ")"
}
if mode&0o002 != 0 {
return "every account (" + path + " is writable by all)"
}
if mode&0o020 != 0 && gid != 0 {
return "the group " + groupNamed(gid) + " (it may write " + path + ")"
}
return ""
}
// linkReplacer says who besides root owns a link: "" when root does. A variable for the same reason.
var linkReplacer = func(link string, uid uint32) string {
if uid != 0 {
return "the account " + accountNamed(uid) + " (it owns the link)"
}
return ""
}
func accountNamed(uid uint32) string {
if u, err := osuser.LookupId(strconv.FormatUint(uint64(uid), 10)); err == nil {
return u.Username
}
return "with uid " + strconv.FormatUint(uint64(uid), 10)
}
func groupNamed(gid uint32) string {
if g, err := osuser.LookupGroupId(strconv.FormatUint(uint64(gid), 10)); err == nil {
return g.Name
}
return "with gid " + strconv.FormatUint(uint64(gid), 10)
}
// JudgeDirectories is the walk's judgement over every directory a declaration names, read-only (novox/hq issue
// 358): nothing is made, chmodded or chowned. One line per directory, in the declaration's order, after one line
// that counts them: placed, would be refused and why, refused already by the placement guard (issue 339), or
// below a home, where ADR 0266's walk judges it as before.
func JudgeDirectories(d *declaration.Declaration) string {
var lines []string
var placed, refused, guarded, homed int
for _, r := range d.Resources {
dir, ok := r.(*declaration.Directory)
if !ok {
continue
}
path := filepath.Clean(dir.Path)
if err := refusePath(path, placingDirectory, moduleOfID(dir.ID), dir.Owner); err != nil {
guarded++
lines = append(lines, path+": refused already by the placement guard: "+err.Error())
continue
}
if home := homeAbove(path); home != "" {
homed++
lines = append(lines, path+": below the home "+home+", walked without following any link as before (ADR 0266)")
continue
}
if err := judgeMeshDir(path); err != nil {
refused++
lines = append(lines, path+": would be refused: "+err.Error())
continue
}
placed++
lines = append(lines, path+": placed")
}
head := fmt.Sprintf("%d directories declared: %d placed, %d would be refused, %d refused already by the "+
"placement guard, %d below a home (novox/hq issue 358; judged read-only, nothing changed)",
placed+refused+guarded+homed, placed, refused, guarded, homed)
return strings.Join(append([]string{head}, lines...), "\n")
}
+114
View File
@@ -0,0 +1,114 @@
//go:build linux
package apply
// The apply half of mesh_dirs.go (novox/hq issue 358): a mesh directory outside every home made and given its mode
// and owner through the walk's descriptor.
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"golang.org/x/sys/unix"
"github.com/novox/mesh-host/internal/declaration"
)
// meshDirsByDescriptor says this system can open a directory component by component without following a link.
const meshDirsByDescriptor = true
// openMeshDir opens the directory at path, from /, one component at a time with O_DIRECTORY|O_NOFOLLOW; a missing
// component is made in its parent's descriptor with mode, and made is called with each one made and its descriptor.
// A link in a parent is followed only when couldReplace and linkReplacer find that root alone could have made it,
// and its target is walked again from /; any other link is a LinkInPathError, and the last component being one is
// refused as not placed. Answers the directory's descriptor, which the caller closes, with the walk: what it made
// and the path it reached the directory at.
func openMeshDir(path string, mode os.FileMode, made func(dir string, fd int) error) (int, *meshWalk, error) {
w := &meshWalk{want: filepath.Clean(path), create: true, mode: mode, made: made}
fd, err := w.open()
return fd, w, err
}
// applyMeshDirectory gives a mesh directory outside every home its declared mode and owner through a descriptor
// (novox/hq issue 358): opened by openMeshDir, made there when missing, judged a directory by fstat on it, changed
// with fchmod and fchown on it, and read back from it. The parents it makes inside the owner's home are the owner's,
// as makeDirs gives them (novox/hq ADR 0182).
func applyMeshDirectory(r *declaration.Directory, out Outcome, mode os.FileMode, existed bool) (Outcome, error) {
uid, gid, chown := 0, 0, r.Owner != ""
var err error
if chown {
if uid, gid, err = idsOf(r.Owner); err != nil {
return out, fmt.Errorf("%s should belong to %q: %w", r.Path, r.Owner, err)
}
}
clean := filepath.Clean(r.Path)
home := ""
if chown {
if h, herr := homeOf(r.Owner); herr == nil && h != "" {
home = filepath.Clean(h)
}
}
giveMade := func(dir string, fd int) error {
if dir == clean || home == "" || (dir != home && !strings.HasPrefix(dir, home+string(os.PathSeparator))) {
return nil
}
if err := unix.Fchown(fd, uid, gid); err != nil {
return fmt.Errorf("cannot give %s to %q: %w", dir, r.Owner, &os.PathError{Op: "chown", Path: dir, Err: err})
}
return nil
}
fd, walk, err := openMeshDir(clean, mode, giveMade)
if err != nil {
var link *LinkInPathError
if errors.As(err, &link) {
return out, &PlacementRefusedError{Path: r.Path, Why: "it " + link.why()}
}
return out, err
}
defer unix.Close(fd)
// Made by this walk when its last component is: at the path the walk reached it at, which root-only links in its
// path may have moved from the path declared.
created := len(walk.Made) > 0 && walk.Made[len(walk.Made)-1] == walk.At
var st unix.Stat_t
if err := unix.Fstat(fd, &st); err != nil {
return out, &os.PathError{Op: "fstat", Path: r.Path, Err: err}
}
beforeMode, beforeUID, beforeGID := os.FileMode(st.Mode&0o777), int(st.Uid), int(st.Gid)
// Set explicitly even when it existed: a mode given at creation is not a mode maintained.
if err := unix.Fchmod(fd, uint32(mode.Perm())); err != nil {
return out, &os.PathError{Op: "chmod", Path: r.Path, Err: err}
}
ownedAlready := !chown || (beforeUID == uid && beforeGID == gid)
if !ownedAlready {
if err := unix.Fchown(fd, uid, gid); err != nil {
return out, fmt.Errorf("cannot give %s to %q: %w", r.Path, r.Owner, &os.PathError{Op: "chown", Path: r.Path, Err: err})
}
}
// Read back, from the descriptor that was changed.
if err := unix.Fstat(fd, &st); err != nil {
return out, fmt.Errorf("made %s and cannot stat it: %w", r.Path, err)
}
if st.Mode&unix.S_IFMT != unix.S_IFDIR {
return out, fmt.Errorf("%s is not a directory after applying", r.Path)
}
if got := os.FileMode(st.Mode & 0o777); got != mode.Perm() {
return out, fmt.Errorf("%s is mode %o after setting %o", r.Path, got, mode.Perm())
}
out.Action = "unchanged"
switch {
case created || !existed:
out.Action = "created"
case !ownedAlready:
out.Action = "updated"
case beforeMode != mode.Perm():
out.Action = "updated"
out.Detail = fmt.Sprintf("mode %o to %o", beforeMode, mode.Perm())
}
return out, nil
}
+95
View File
@@ -0,0 +1,95 @@
//go:build linux
package apply
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Review 47 of novox/hq issue 358: a directory the walk refuses gates its module, and nothing below it is written,
// whether the link was there when the apply began or appeared while it ran.
// aLinkedModule is a module's directory at <base>/parent/state, its parent a link to <base>/elsewhere, and a file of
// the module below the directory; and another module's file elsewhere.
func aLinkedModule(t *testing.T) (target, other, decl string) {
t.Helper()
base := t.TempDir()
target = filepath.Join(base, "elsewhere", "state")
if err := os.MkdirAll(target, 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink(filepath.Join(base, "elsewhere"), filepath.Join(base, "parent")); err != nil {
t.Fatal(err)
}
dir := filepath.Join(base, "parent", "state")
other = filepath.Join(base, "other.conf")
decl = `{"id":"mod.state","type":"directory","path":"` + dir + `","mode":"0700"},` +
`{"id":"mod.conf","type":"file","path":"` + dir + `/settings.conf","content":"a=1\n"},` +
`{"id":"other.conf","type":"file","path":"` + other + `","content":"kept apart\n"}`
return target, other, decl
}
func holdsTheGate(t *testing.T, target, other string, report Report, err error) {
t.Helper()
var refusal *PlacementRefusedError
if !errors.As(err, &refusal) {
t.Fatalf("the directory behind the link was not refused: %v", err)
}
if _, statErr := os.Stat(filepath.Join(target, "settings.conf")); statErr == nil {
t.Error("a file was written into the link's target, below the directory the engine refused")
}
if info, _ := os.Stat(target); info.Mode().Perm() != 0o755 {
t.Errorf("the link's target was given mode %o (was 755)", info.Mode().Perm())
}
if o := outcomeOf(report, "mod.conf"); o.Action != "skipped" {
t.Errorf("the file below the refused directory is not reported skipped: %+v", o)
}
if strings.Contains(err.Error(), "everything was attempted") || !strings.Contains(err.Error(), "the rest of mod was not attempted") {
t.Errorf("the report does not say the module was gated: %v", err)
}
if got, readErr := os.ReadFile(other); readErr != nil || string(got) != "kept apart\n" {
t.Errorf("another module's file was not applied: %q, %v", got, readErr)
}
}
// The link there when the apply begins: refused by the guard before anything is applied.
func TestADirectoryBehindALinkGatesItsModuleThroughApply(t *testing.T) {
target, other, decl := aLinkedModule(t)
report, _, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, nil, nil)
holdsTheGate(t, target, other, report, err)
}
// The link found only at the moment of use: refused by the walk during the apply, and still a refusal that gates its
// module and what is below it. Simulated by trusting / for the guard's walk (two walks of /: one to the link, one
// from its target) and not for the apply's.
func TestADirectoryRefusedDuringTheApplyGatesItsModule(t *testing.T) {
target, other, decl := aLinkedModule(t)
wasDirs, wasLinks := couldReplace, linkReplacer
walksOfRoot := 0
couldReplace = func(path string, _, _, _ uint32) string {
if path == "/" {
walksOfRoot++
if walksOfRoot > 2 {
return "every account (a test's /)"
}
}
return ""
}
linkReplacer = func(string, uint32) string { return "" }
t.Cleanup(func() { couldReplace, linkReplacer = wasDirs, wasLinks })
report, _, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, nil, nil)
if walksOfRoot < 3 {
t.Fatalf("the apply did not walk again at the moment of use (%d walks)", walksOfRoot)
}
holdsTheGate(t, target, other, report, err)
}
+186
View File
@@ -0,0 +1,186 @@
//go:build linux
package apply
// The descriptor half of mesh_dirs.go (novox/hq issue 358).
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"golang.org/x/sys/unix"
)
// maxLinksFollowed bounds the root-only links one walk follows, as the kernel bounds a lookup's (MAXSYMLINKS).
const maxLinksFollowed = 40
// errNotThereYet is a read-only walk meeting a component that does not exist: an apply would make it there.
var errNotThereYet = errors.New("not there yet")
// meshWalk is one walk of a mesh directory's path from /.
type meshWalk struct {
// want is the directory as declared.
want string
// create makes a missing component, with mode, in its parent's descriptor; without it the walk stops there
// with errNotThereYet and makes nothing.
create bool
mode os.FileMode
// made is called with each directory the walk made and its descriptor.
made func(dir string, fd int) error
// Made is every directory the walk made, outermost first, and At the path the directory was reached at:
// want, or where the root-only links in its path led.
Made []string
At string
}
// judgeMeshDir is openMeshDir's judgement and nothing else (novox/hq issue 358): the same walk, making nothing
// and changing nothing; nil when an apply would give the directory its mode and owner, also when a part of its
// path is not there yet (an apply makes it, as root, in a directory it reached).
func judgeMeshDir(path string) error {
w := &meshWalk{want: filepath.Clean(path)}
fd, err := w.open()
if errors.Is(err, errNotThereYet) {
return nil
}
if err != nil {
return err
}
unix.Close(fd)
return nil
}
func (w *meshWalk) open() (int, error) {
if !filepath.IsAbs(w.want) {
return -1, fmt.Errorf("%s is not an absolute path", w.want)
}
p := w.want
for followed := 0; ; followed++ {
if followed > maxLinksFollowed {
return -1, &os.PathError{Op: "open", Path: w.want, Err: unix.ELOOP}
}
fd, next, err := w.fromRoot(p)
if err != nil {
return -1, err
}
if next == "" {
w.At = p
return fd, nil
}
p = next
}
}
// fromRoot is one walk down p: the directory's descriptor, or the path to walk next when a root-only link was
// met (with -1), or the refusal.
func (w *meshWalk) fromRoot(p string) (int, string, error) {
fd, err := unix.Open("/", unix.O_RDONLY|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
if err != nil {
return -1, "", &os.PathError{Op: "open", Path: "/", Err: err}
}
// Who could replace an entry of the directories walked so far: once one could, every link below it is
// theirs to make, whoever owns the link.
who, err := replacerOf(fd, "/")
if err != nil {
unix.Close(fd)
return -1, "", err
}
at := "/"
var parts []string
if p != "/" {
parts = strings.Split(strings.TrimPrefix(p, "/"), "/")
}
const dirFlags = unix.O_RDONLY | unix.O_DIRECTORY | unix.O_NOFOLLOW | unix.O_CLOEXEC
for i, part := range parts {
here := filepath.Join(at, part)
last := i == len(parts)-1
next, err := unix.Openat(fd, part, dirFlags, 0)
if errors.Is(err, unix.ENOENT) {
if !w.create {
unix.Close(fd)
return -1, "", errNotThereYet
}
mkErr := unix.Mkdirat(fd, part, uint32(w.mode.Perm()))
if mkErr != nil && !errors.Is(mkErr, unix.EEXIST) {
unix.Close(fd)
return -1, "", &os.PathError{Op: "mkdir", Path: here, Err: mkErr}
}
next, err = unix.Openat(fd, part, dirFlags, 0)
if mkErr == nil && err == nil {
w.Made = append(w.Made, here)
if w.made != nil {
if gErr := w.made(here, next); gErr != nil {
unix.Close(next)
unix.Close(fd)
return -1, "", gErr
}
}
}
}
if errors.Is(err, unix.ELOOP) || errors.Is(err, unix.ENOTDIR) {
var st unix.Stat_t
if lerr := unix.Fstatat(fd, part, &st, unix.AT_SYMLINK_NOFOLLOW); lerr == nil && st.Mode&unix.S_IFMT == unix.S_IFLNK {
to, rerr := w.followable(fd, part, here, st, who, last)
unix.Close(fd)
if rerr != nil {
return -1, "", rerr
}
if !filepath.IsAbs(to) {
to = filepath.Join(at, to)
}
return -1, filepath.Join("/", to, filepath.Join(parts[i+1:]...)), nil
}
}
unix.Close(fd)
if err != nil {
switch {
case errors.Is(err, unix.ENOTDIR) && last:
return -1, "", fmt.Errorf("%s exists and is not a directory", here)
case errors.Is(err, unix.ENOTDIR):
return -1, "", fmt.Errorf("%s is not a directory, and %s lies below it", here, w.want)
}
return -1, "", &os.PathError{Op: "open", Path: here, Err: err}
}
fd = next
at = here
if who == "" {
if who, err = replacerOf(fd, here); err != nil {
unix.Close(fd)
return -1, "", err
}
}
}
return fd, "", nil
}
// replacerOf is couldReplace for the directory open on fd.
func replacerOf(fd int, path string) (string, error) {
var st unix.Stat_t
if err := unix.Fstat(fd, &st); err != nil {
return "", &os.PathError{Op: "fstat", Path: path, Err: err}
}
return couldReplace(path, st.Uid, st.Gid, st.Mode&0o7777), nil
}
// followable judges the link at here, held by the directory open on fd: its target when root alone could have
// made it, else the refusal. who is who could replace an entry of a directory above it ("" when nobody).
func (w *meshWalk) followable(fd int, name, here string, st unix.Stat_t, who string, last bool) (string, error) {
if last {
return "", &PlacementRefusedError{Path: w.want, Why: "it is a symbolic link, and the node-engine follows no " +
"link to a directory it owns"}
}
if who == "" {
who = linkReplacer(here, st.Uid)
}
if who != "" {
return "", &LinkInPathError{Path: w.want, Link: here, Who: who}
}
buf := make([]byte, unix.PathMax)
n, err := unix.Readlinkat(fd, name, buf)
if err != nil {
return "", &os.PathError{Op: "readlink", Path: here, Err: err}
}
return string(buf[:n]), nil
}
+91
View File
@@ -0,0 +1,91 @@
//go:build linux
package apply
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq issue 358: a mesh directory outside every home is given its mode and owner only through a
// descriptor opened from /, so a parent swapped for a link between two applies cannot carry root's chmod and
// chown to the link's target — unless root alone could have made that link (/var/run is one on every node).
// A link only root could have made — itself root's, in directories no other account may write — is followed, as
// /var/run to /run is on every node; relative or absolute.
func TestALinkOnlyRootCouldMakeIsFollowed(t *testing.T) {
for name, to := range map[string]func(base string) string{
"absolute": func(base string) string { return filepath.Join(base, "elsewhere") },
"relative": func(string) string { return "elsewhere" },
} {
t.Run(name, func(t *testing.T) {
rootAlone(t, "")
base, r, previous := aMeshDirectory(t)
target := swapParentForLink(t, base, to(base))
out, err := applyDirectory(r, previous, true)
if err != nil {
t.Fatalf("a directory behind a root-only link was refused: %v", err)
}
if got := permOf(t, target); got != 0o700 {
t.Errorf("the directory behind a root-only link is mode %o, want 700", got)
}
if out.Action != "updated" {
t.Errorf("the apply said %q, want updated", out.Action)
}
})
}
}
// A link another account owns refuses the directory even in directories root alone may write, naming the account.
func TestALinkAnotherAccountOwnsIsRefused(t *testing.T) {
rootAlone(t, "the account mallory (it owns the link)")
base, r, previous := aMeshDirectory(t)
target := swapParentForLink(t, base, filepath.Join(base, "elsewhere"))
_, err := applyDirectory(r, previous, true)
if err == nil || !strings.Contains(err.Error(), "mallory") {
t.Fatalf("a link another account owns was not refused naming it: %v", err)
}
if got := permOf(t, target); got != 0o755 {
t.Errorf("the link's target was given mode %o (was 755)", got)
}
}
// A directory that is missing is made through the walk, below a link root alone could make.
func TestAMissingDirectoryIsMadeBehindARootOnlyLink(t *testing.T) {
rootAlone(t, "")
base := t.TempDir()
if err := os.Mkdir(filepath.Join(base, "real"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink("real", filepath.Join(base, "link")); err != nil {
t.Fatal(err)
}
r := &declaration.Directory{ID: "mod.state", Path: filepath.Join(base, "link", "a", "b"), Mode: "0750"}
out, err := applyDirectory(r, store.Applied{}, false)
if err != nil {
t.Fatal(err)
}
if out.Action != "created" {
t.Errorf("the apply said %q, want created", out.Action)
}
if got := permOf(t, filepath.Join(base, "real", "a", "b")); got != 0o750 {
t.Errorf("the made directory is mode %o, want 750", got)
}
}
// rootAlone has every directory count as writable by root alone, as /, /var and /run are on a node, and a link's
// owner be linkOwner ("" for root): a test runs below /tmp, which every account may write.
func rootAlone(t *testing.T, linkOwner string) {
t.Helper()
wasDirs, wasLinks := couldReplace, linkReplacer
couldReplace = func(string, uint32, uint32, uint32) string { return "" }
linkReplacer = func(string, uint32) string { return linkOwner }
t.Cleanup(func() { couldReplace, linkReplacer = wasDirs, wasLinks })
}
+54
View File
@@ -0,0 +1,54 @@
package apply
import (
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
// **A container's mesh names are part of what it is** (novox/hq 04-ISSUES/135).
//
// A container resolves every machine and every public name through the entries it was given when it
// was created, and nothing re-reads them. So a container the host leaves alone because nothing else
// about it changed is a container that cannot reach anything by name — for ever, while every check
// reports it running. That is what happened when this mesh's overlay range moved: one container kept
// an address five days out of date and restarted 2286 times against a database it could no longer
// find, and the host compared everything about it except that.
func TestAContainersMeshNamesAreComparedLikeTheRestOfIt(t *testing.T) {
was := &declaration.Container{
Name: "umami", Image: "ghcr.io/example/umami@sha256:" + zeros(64),
Hosts: []string{"novox.internal:10.42.0.1", "umami.novox.be:10.42.0.1"},
}
moved := &declaration.Container{
Name: was.Name, Image: was.Image,
Hosts: []string{"novox.internal:10.10.0.1", "umami.novox.be:10.10.0.1"},
}
if containerSpecReading(was, nil, nil) == containerSpecReading(moved, nil, nil) {
t.Fatal("a container whose mesh names moved compares equal, so it is never recreated")
}
// And the order they arrive in is not a change: the digest must not move for a reordering
// nobody made.
reordered := &declaration.Container{
Name: moved.Name, Image: moved.Image,
Hosts: []string{moved.Hosts[1], moved.Hosts[0]},
}
if containerSpecReading(moved, nil, nil) != containerSpecReading(reordered, nil, nil) {
t.Fatal("the same names in another order read as a different container")
}
// A container the mesh gives no names is unaffected, so nothing is recreated for a field it
// does not set.
plain := &declaration.Container{Name: "plex", Image: was.Image}
if containerSpecReading(plain, nil, nil) == containerSpecReading(was, nil, nil) {
return // different for other reasons, which is fine
}
}
func zeros(n int) string {
out := make([]byte, n)
for i := range out {
out[i] = '0'
}
return string(out)
}
+55
View File
@@ -0,0 +1,55 @@
package apply
import (
"fmt"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A node is adopted or converged, and a declaration says which it is for (novox/hq ADR 0100).
//
// **The two are not interchangeable, and the host knows which it is.** A converged declaration
// carries the mesh's drop-by-default filter and retires the firewall the node was found with; on
// an adopted node that closes the machine to everything the predecessor still serves — which is
// what happened when an operator ran `reconcile` on an adopted control-node and it applied the
// converged genesis bundle (novox/hq issue 104). The host reported "adopted" in every report and
// compared nothing. Now it compares, at the point of application, whichever command delivered
// the declaration.
//
// Only the mesh changes a node's mode, and it does so by sending a declaration: the flip arrives
// over the link as the first converged declaration after adopted ones (`converge` on the
// controller), and the way back as the first adopted one (`adopt`). So a declaration the link
// delivers, signed and verified, is the mode's authority and is not checked against the record —
// it becomes the record. Everything else — the carried bundle, a file, the kept declaration a
// disconnected node re-applies — is held to the mode already recorded.
// ModeOf says which mode a declaration is for.
func ModeOf(d *declaration.Declaration) string {
if d.Adoption != nil {
return store.ModeAdopted
}
return store.ModeConverged
}
// CheckMode refuses a declaration whose mode is not the one this node has recorded. A node with
// no recorded mode — a machine nothing has said a mode to yet — takes either.
func CheckMode(known store.State, d *declaration.Declaration) error {
if known.Mode == "" || known.Mode == ModeOf(d) {
return nil
}
act := "`converge`"
if ModeOf(d) == store.ModeAdopted {
act = "`adopt`"
}
return fmt.Errorf("this node is %s; the declaration says %s — %s declaration is not applied "+
"to %s node; %s on the controller is the act that changes it, and it sends the "+
"declaration that does", known.Mode, ModeOf(d), article(ModeOf(d)), article(known.Mode), act)
}
func article(mode string) string {
if mode == store.ModeAdopted {
return "an adopted"
}
return "a converged"
}
+32
View File
@@ -0,0 +1,32 @@
package apply
import (
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq issue 104: a declaration is refused for the other mode than the node is in,
// naming both and the act that changes it; a node no mode has been said to takes either.
func TestADeclarationForTheOtherModeIsRefused(t *testing.T) {
converged := parse(t, `{"declaration":1,"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
adopted := parse(t, `{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
err := CheckMode(store.State{Mode: store.ModeAdopted}, converged)
if err == nil || !strings.Contains(err.Error(), "this node is adopted; the declaration says converged") ||
!strings.Contains(err.Error(), "`converge`") {
t.Errorf("a converged declaration on an adopted node: %v", err)
}
err = CheckMode(store.State{Mode: store.ModeConverged}, adopted)
if err == nil || !strings.Contains(err.Error(), "this node is converged; the declaration says adopted") ||
!strings.Contains(err.Error(), "`adopt`") {
t.Errorf("an adopted declaration on a converged node: %v", err)
}
if err := CheckMode(store.State{Mode: store.ModeAdopted}, adopted); err != nil {
t.Errorf("the node's own mode was refused: %v", err)
}
if err := CheckMode(store.State{}, converged); err != nil {
t.Errorf("a node in no mode yet refused a declaration: %v", err)
}
}
+62 -10
View File
@@ -54,20 +54,66 @@ func foundFirewall(ctx context.Context, d *declaration.Declaration, known *store
return kind, nil
}
// retireFirewall disables the found firewall once a converged declaration has applied cleanly,
// which is when the mesh's derived filter has taken its place. Disabled, never flushed: its
// configuration stays on disk for a return to adopted, and the container runtime's rules are not
// its to take.
// retireFirewall keeps the found firewall retired on a converged machine (novox/hq ADR 0100, ADR
// 0168): disabled, never flushed, its configuration left on disk for a return to adopted, and the
// container runtime's rules not its to take.
//
// **Convergence is a state the host keeps, not a step it takes once.** Every converged apply reads
// whether the front end is in force; enabled again by a package, a boot or a hand, it is retired
// again and said. The record says how it came to be inactive — the mesh disabled it, or a reconcile
// found it so — and the two are never confused: a flip that did not take, followed by a hand that
// did, used to be recorded as the mesh's doing (issue 143).
//
// Only a declaration from the mesh converges a node. A carried bundle never says a node is adopted
// — it cannot — so its silence is not the controller's word that the node was converged, and an
// adopted node re-applying its bundle keeps the firewall it was found with.
//
// Returned is what this apply did about the found firewall, for the report; empty when the machine
// has none or is not converged.
func retireFirewall(ctx context.Context, d *declaration.Declaration, origin string, known *store.State,
run Runner, log func(string)) error {
run Runner, log func(string)) (string, error) {
rec := known.Firewall
if origin != store.OriginDeclared || d.Adoption != nil || rec == nil || rec.Kind != string(firewall.UFW) || !rec.WasActive ||
rec.DisabledByMesh {
return nil
if origin != store.OriginDeclared || d.Adoption != nil || rec == nil || rec.Kind != string(firewall.UFW) || !rec.WasActive {
return "", nil
}
if !firewall.Installed(ctx, run) {
// Uninstalled (novox/hq ADR 0180): retired for good, by the module that replaced it. Said
// once, and nothing is asked of a command that is not there.
if rec.RetiredBy != firewall.RetiredRemoved {
rec.RetiredBy = firewall.RetiredRemoved
log(" the found firewall (ufw) is no longer installed; the mesh's filter is what filters this machine")
return "removed: ufw is no longer installed; the mesh's filter is what filters this machine", nil
}
return "", nil
}
active := firewall.Active(ctx, run)
if !active && !(rec.Forward != nil && !rec.DisabledByMesh) {
// Inactive, and either the mesh's doing already or nobody's recorded here: said as found,
// never as done (issue 143's second fault). A retirement the mesh began and did not finish —
// the forward policy recorded, ufw down, the restore failed — is the one inactive state that
// is still the mesh's to complete, below.
if rec.RetiredBy == "" {
if rec.DisabledByMesh {
rec.RetiredBy = firewall.RetiredByMesh
} else {
rec.RetiredBy = firewall.RetiredFoundSo
log(" ufw is inactive on this converged node, and not by the mesh; recorded as found so")
}
}
return "", nil
}
// **Nothing is retired until what replaces it is in force** (novox/hq ADR 0100). The flip
// loads the mesh's derived filter in ufw's place; disabling ufw before that table is actually
// loaded — a filter module not assigned, or a unit that did not load — leaves the machine with
// no filter at all.
loaded, err := firewall.MeshTableLoaded(ctx, run)
if err != nil {
return "", err
}
if !loaded {
return "", fmt.Errorf("this node is converged and the mesh's own filter (table %s) is not loaded on "+
"this machine, so ufw was left in force: retiring it would leave the machine filtering "+
"nothing. Assign a filter module to this node, or return it to adopted", firewall.MeshTable)
}
if rec.Forward == nil {
// Recorded before ufw is touched: disabling it opens the forward policy, and a retry
@@ -75,11 +121,17 @@ func retireFirewall(ctx context.Context, d *declaration.Declaration, origin stri
rec.Forward = firewall.ForwardPolicies(ctx, run)
}
if err := firewall.Disable(ctx, run, rec.Forward); err != nil {
return err
return "", err
}
again := rec.DisabledByMesh || rec.RetiredBy != ""
rec.DisabledByMesh = true
rec.RetiredBy = firewall.RetiredByMesh
if again {
log(" disabled ufw again: it had been enabled since the mesh retired it; this node is converged and filtered by the mesh")
return "disabled again: ufw had been enabled since the mesh retired it", nil
}
log(" disabled ufw: this node is converged and filtered by the mesh; ufw's configuration is left on disk")
return nil
return "disabled: this node is converged and filtered by the mesh; ufw's configuration is left on disk", nil
}
// applyOpening makes one opening true through the firewall found here.

Some files were not shown because too many files have changed in this diff Show More