Author SHA1 Message Date
jschoubben 1a190090ab The engine states its key again until it hears the vault's key, and at a slow interval after, so a vault delivered after it or a changed vault key is heard (review of the adoption handing, ADR 0547 clause 9) 2026-10-26 12:03:12 +01:00
jschoubben fd381567f4 The bus node hands the loaded list's hashes to the vault, sealed to its key, in the bus module's have (ADR 0547 clause 9, issue 1945)
The vault holds no value for the controller's own account, the node accounts and the enrolled devices, so it cannot
hash them, and take-over waits on their hashes. The bus node's engine is the one part that has them, in the list the
bus loaded. Its follower of the user list now puts every loaded user's bcrypt hash in the bus module's have, sealed to
the vault's own key (mesh-sdk delivery.AccountsHanded), so of every reader of the have bucket the vault alone opens
it; the clear part says only the list's digest, the key's fingerprint and how many users. The vault's key is heard from
its answer to node-key, which the engine already asks at start; a first or changed key looks at the list again. It is
sealed again only when the list on the machine or the vault's key changes; a refused list hands the list kept on the
machine. The vault takes from it only the users it holds no value for.

Also: the user list follower's test sent its later puts before the watch had handed the keys there before it, so two
of its cases failed on main whenever the goroutine started late; the feed now says when the watch began.
2026-10-26 11:32:36 +01:00
mesh-tools 1195cf4597 Merge pull request 'The node-engine retries an apply that met a passing bus error, so a reload does not leave a module failed (issue 1945)' (#144) from fix/apply-retries-transient into main 2026-10-25 22:42:41 +00:00
jschoubben 99285cb080 Retry an apply, a fetch by digest and a have write that met the bus reloading, and keep the cause in the have
mesh/delivery merged: source.main moved to novox/mesh-host@1195cf45971e
mesh/check pass 99285cb080db on 1195cf45971e | unit 303 run, 1080 kept, 5 skipped | integration 306 run, 1082 kept, 4 skipped | over bound: unit (slow…
The bus node's engine writes the bus user list (ADR 0547 bridge); each write makes the bus reload its accounts, and
for about three seconds every JetStream request meets no responders. An apply that failed then was kept failed until a
key changed (mesh-vault on novox, 2026-10-25 23:04:24), holding every module that needs the secret seat. A transient
bus failure is now retried with a backoff (2 s, 5 s, 15 s, 60 s, then every 5 min), the have saying retrying meanwhile;
after the engine writes the user list, the transiently failed modules are tried again once the reload is over. A have
write that met it is written again the same way. shorten keeps the head and the tail, the cause.
2026-10-25 23:37:02 +01:00
mesh-tools d75c66dc7e Merge pull request 'The bus node writes the newest signed bus user list by itself, without a want (ADR 0547 bridge, issue 1945)' (#143) from fix/1945-user-list-follows into main 2026-10-25 21:16:06 +00:00
mesh-tools e196058f6f Merge pull request 'The node-engine wakes the provisioner when an apply lets a file it refused be written (issue 924)' (#142) from fix/provisioner-retries-on-declaration into main 2026-10-25 21:08:52 +00:00
jschoubben 5e71ab6ae6 Count the bus's users from the list a digest names, take only a composition of this node and module, and refuse a list that would lock the mesh out (review of the bridge, ADR 0547 §8)
mesh/delivery merged: source.main moved to novox/mesh-host@d75c66dc7e27
mesh/delivery-group group fix/1945-user-list-follows merged: every member is merged
mesh/check pass 5e71ab6ae612 on d75c66dc7e27 | unit 303 run, 1077 kept, 5 skipped | integration 306 run, 1079 kept, 4 skipped | over bound: unit, inte…
B1: the controller sends the user list by digest (over 64 KiB), so its users are counted from the list as resolved, not
from the composition's inline content, which is empty then.
R1: a composition signed for another node or module than its key names is no source.
R2: a list without the controller's own user, or one that drops more than half the users of the file on the machine,
is not written; the have says why and the file on the machine is kept.
2026-10-25 22:00:05 +01:00
jschoubben 3f02bd7ff6 The bus node writes the bus's user list alone from the newest composition of the bus module, with no want, and says it in the bus module's have (ADR 0547 §8, issue 1945)
The list reached the bus only through a want of the bus module, which nobody writes (ADR 0539 §15): on 2026-10-25 it
stood from 11:23 to 20:50 and was put back by hand. The node agent now follows its composition bucket for the newest
signed composition carrying the user list, writes that file alone through the apply's file step when it differs
(mode 0600, read back, recorded as the module's), moves nothing that reads it (the bus module's entrypoint reloads the
server in place), never goes back to an older list, looks again after every apply of the bus module, and says source,
revision, wanted and written digests, users and any error in the bus module's have (mesh-sdk UserListWritten).

Read with mesh-delivery-view.node_haves (module nats); the bus's own nats_user_can says in_force once it reloaded.
2026-10-25 21:50:49 +01:00
jschoubben 338797a527 Knock under the apply's lock, and read what is owed first (review of this fix)
mesh/delivery merged: source.main moved to novox/mesh-host@e196058f6fb0
mesh/delivery-group group fix/provisioner-retries-on-declaration merged: every member is merged
mesh/check pass 338797a52703 on d75c66dc7e27 | unit 2301 run, 3410 kept, 10 skipped | integration 2326 run, 3431 kept, 13 skipped | over bound: unit, …
A write judged by the declaration before the apply settled could record its
refusal just after Knock found nothing owed: the original fault in a small
window. Knock now holds the apply's lock as a write does, reads no declaration
while nothing is owed, and its comment names Settled as its caller.
2026-10-25 21:49:23 +01:00
jschoubben 2c93f79429 Tell the provisioner's client to fill again once a settled apply may let through a write the filling refused (ace, 2026-10-25 21:17)
ace's filling refused mesh-build-go's files as not named as filled 32 s before
its want arrived; after the apply that named them nothing asked again, and the
runtime crash-looped on its missing broker credential. The filling now keeps
what it refused or failed and, when the node agent says a module's apply or
removal settled (agent.Env.Settled, called once what runs holds it), touches its
write socket if the declaration now names such a path, names the provisioner's
account, or a write had failed; the client watches that socket and fills every
module again. Said in the journal each time.
2026-10-25 21:42:41 +01:00
mesh-tools 5f16cf507a Merge pull request 'Say a module's health is not yet judged while it changes, and judge it as soon as its change ends (issue 1945)' (#141) from fix/r6-health-unknown into main 2026-10-25 15:04:54 +00:00
jschoubben b32a27c115 Owe a look when a module's change ends unjudged, so a module with nothing long-running is ok within a look of its apply
mesh/delivery merged: source.main moved to novox/mesh-host@5f16cf507a0f
mesh/check pass b32a27c115a2 on 0dfbe1a179f2 | unit 292 run, 1060 kept, 5 skipped | integration 295 run, 1062 kept, 4 skipped | over bound: unit (slow…
The facts look inside an apply runs while the module is still applying and judges nothing; the
running write then left xorg running and unknown until the five-minute statement, holding every
x11-display consumer behind it. The running write now nudges the health loop, which looks and says
at once. A declaration's report gets the same motion and composition check, and a removed module
leaves nothing marked unjudged (review of fix 15).
2026-10-25 15:46:05 +01:00
jschoubben c16e625a08 Say a module's health is not yet judged while its change is under way and until a look after it
A have said applying and failed for the controller on novox at 15:27:36, from a look at the
container being replaced, two seconds before its launch said healthy; the assigner held the
canary on it and set novox back. A look now judges only a module that is not in motion, and a
module whose composition moved carries unknown until a look after the move.
2026-10-25 15:32:59 +01:00
jschoubben 1cdb4d1a29 Say a module found running on its own have again at the agent's start, so its health is this engine's and not an older engine's unknown
xorg's have on g14 and shanks kept the unknown an engine before 958ccbd wrote: the
baseline believed the have and never handed the module to the session, so no look
rewrote its health, and the assigner held every module requiring x11-display back.
2026-10-25 15:27:23 +01:00
mesh-tools 0dfbe1a179 Merge pull request 'Drop the node's own copy of a resource a module composition declares, and give the kept declaration's filled paths to their modules (issue 1945)' (#140) from fix/r6-own-composition-dupes into main 2026-10-25 13:51:00 +00:00
jschoubben 1bdbf47ffa Leave out the node's own copy only by kind and id, never by a shared target (issue 1945, review)
mesh/delivery merged: source.main moved to novox/mesh-host@0dfbe1a179f2
mesh/check pass 1bdbf47ffa83 on 0dfbe1a179f2 | unit 302 run, 1044 kept, 5 skipped | integration 305 run, 1046 kept, 4 skipped | over bound: unit (slow…
A unit or container of the node's own that shares only its name with a
module's resource is not the module's: dropped, its record became an orphan
the apply removes, stopping the module's unit (1945/AC17).
2026-10-25 14:29:06 +01:00
jschoubben 16f84b53da Leave the node's own copy of a module's resource out, and carry the kept declaration's filled paths (issue 1945)
The node's own composition and a module's both declared the same files, and
the joined declaration was refused whole ("2 resources declare X"): the
node's own copy is now left out when joining and at the node's own apply, and
said once (1945/AC17). A module still running from the kept declaration now
carries its filled paths, so the filling no longer refuses them as not named
as filled (1945/AC18).
2026-10-25 14:10:52 +01:00
mesh-tools 57b553ac5a Merge pull request 'Apply the revision of a composition its want names, read from the key's history, when the latest has another digest (issue 1945)' (#139) from fix/r6-want-names-revision into main 2026-10-25 12:30:40 +00:00
jschoubben 11a635796d Bound a key's history read to 10 s and test it on a real bus; the witness test waits for its process to run from its image (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-host@57b553ac5a57
mesh/check pass 11a635796df9 on 57b553ac5a57 | unit 299 run, 1043 kept, 5 skipped | integration 302 run, 1045 kept, 4 skipped | over bound: unit (slow…
2026-10-25 12:51:00 +01:00
jschoubben 8343c4fb36 Apply the revision of a composition key whose digest the want names, from the revisions the bucket keeps (issue 1945) 2026-10-25 12:42:20 +01:00
mesh-tools 3e3e43de52 Merge pull request 'Never lose the bus to a replace of its own container (issue 1945)' (#138) from fix/r6-bus-replace into main 2026-10-25 11:25:11 +00:00
jschoubben 175fb45c19 Take a container left set aside as the next apply's, and bound an apply to 30 minutes (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-host@3e3e43de5214
mesh/check pass 175fb45c19e3 on 3e3e43de5214 | unit 299 run, 1042 kept, 5 skipped | integration 302 run, 1043 kept, 4 skipped | over bound: unit (slow…
2026-10-25 12:18:50 +01:00
jschoubben 219169dbd9 Set a running container aside until its replacement runs, and put it back when the replacement does not come up (issue 1945) 2026-10-25 12:14:41 +01:00
jschoubben b99d0b45bf Run an apply in a context the bus's loss cannot end, and say why a killed command was killed (issue 1945) 2026-10-25 12:09:28 +01:00
jschoubben f06058390c Merge pull request 'Apply again a module whose resources are gone, move a module with nothing to back up, and report ok for a running module with nothing to watch (issue 1944)' (#137) from fix/r6-reapply-missing into main
Reviewed-on: #137
2026-10-25 10:11:19 +00:00
jschoubben 958ccbde3e Say a running module ok once looked at when nothing of it is judged, so mesh-host's canary can be healthy and its other nodes get the new engine (issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 958ccbde3ed4 on f06058390c0f | unit 297 run, 1038 kept, 5 skipped | integration 300 run, 1039 kept, 4 skipped | over bound: unit (slow…
2026-10-25 11:08:10 +01:00
jschoubben 43d1c2456f Apply a module again only for a newly missing resource, leave filled files out of the record check, and say when the applied state cannot be read (issue 1945) 2026-10-25 11:04:08 +01:00
jschoubben c46937da04 Apply a module again when the engine holds no record of its resources or its health says one is missing, and move a module with nothing to back up without a backup (issue 1944, issue 1945) 2026-10-25 10:59:21 +01:00
jschoubben cf455bc4fe Merge pull request 'Never remove a module's resources while its own composition waits to be applied (issue 1944)' (#136) from fix/r6-node-own-keeps-moved into main
Reviewed-on: #136
2026-10-25 09:23:28 +00:00
jschoubben ecf142e451 Let only a module's wanted or newest composition hold the node's own apply back, never a stale refused one (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-host@cf455bc4fe3e
mesh/check pass ecf142e4513f on cf455bc4fe3e | unit 292 run, 1033 kept, 5 skipped | integration 295 run, 1034 kept, 4 skipped | over bound: unit (slow…
2026-10-25 02:01:24 +01:00
jschoubben fa9e4483c8 Hold back the node's own apply while a module's composition cannot be read or is refused, naming it (issue 1944, issue 1945) 2026-10-25 02:56:20 +02:00
jschoubben e62c32c8c3 Leave a record another module's composition now holds, whatever its type, instead of removing it in the node's own apply; a removed module holds nothing back (issue 1944, issue 1945) 2026-10-25 02:54:58 +02:00
jschoubben 2bfde7d659 Hold out a module whose own composition waits to be applied, so the node's own apply never removes the resources it moved there (issue 1944, issue 1945) 2026-10-25 02:52:10 +02:00
jschoubben db06feed2e Merge pull request 'Refuse an apply that would write a provisioner file empty, naming the file (issue 1944)' (#135) from fix/r6-node-composition into main
Reviewed-on: #135
2026-10-25 00:17:56 +00:00
jschoubben 4e6b756bd2 Refuse an apply that would write a provisioner's file empty, read the node's own filled paths, and name the node's own composition when its apply fails (issue 1944, issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
mesh/check checking 4e6b756bd225 on b833d09d2872 | novox/mesh-host#135 in group fix/r6-node-composition
2026-10-25 01:49:00 +02:00
jschoubben b833d09d28 Merge pull request 'R6: the node-engine applies from its wants and signed compositions, one module at a time; operator-signers (issue 1944)' (#134) from feat/r6 into main
Reviewed-on: #134
2026-10-24 23:17:39 +00:00
jschoubben 81758b24a8 Pin mesh-sdk to feat/r6 748c564, which carries main's composer (issue 1943)
mesh/delivery delivering: 0 machine step(s) passed
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/check checking 81758b24a8e4 on 285395d523e1 | novox/mesh-host#134 in group feat/r6
2026-10-24 23:06:00 +02:00
jschoubben 25ad5ce42c Merge remote-tracking branch 'origin/main' into feat/r6 2026-10-24 23:00:09 +02:00
jschoubben 285395d523 Merge pull request 'R4g finish: merge-check.sh gone, reads-outside markers (issues 1295, 1935)' (#133) from feat/r4g-finish into main
Reviewed-on: #133
2026-10-24 20:57:23 +00:00
jschoubben b47df31834 Pin mesh-sdk to d7a9b96 2026-10-24 21:18:33 +02:00
jschoubben fd05c8235d Pin mesh-sdk to 7f9cc0e 2026-10-24 20:43:47 +02:00
jschoubben 98d1f165b7 Pin mesh-sdk to e45bdc0 2026-10-24 20:22:00 +02:00
jschoubben 332d8f161b The carried genesis template no longer lists the controller's plan-moved event 2026-10-24 18:32:58 +02:00
jschoubben 5088755e5e gofmt 2026-10-24 18:28:22 +02:00
jschoubben e72c682450 Pin mesh-sdk to a523a8c 2026-10-24 18:18:06 +02:00
jschoubben be5f31ce66 What runs is the agent's compositions plus the kept declaration's rest: health, filling, take act, directories and the others held out of an apply read it; a restart trusts the node-engine's own haves; the bed tests run off Arch (review round 1) 2026-10-24 18:13:40 +02:00
jschoubben 67db003813 Pin mesh-sdk to the head of feat/r6 (5da87a0) 2026-10-24 17:50:20 +02:00
jschoubben e5a06cc73d The node agent waits for a backup's end on the holder's event, not by polling; the manifest consumes it (R6-A A5) 2026-10-24 17:03:30 +02:00
jschoubben 20d0998cdd The operator's public keys reach the node as the file operator-keys, fed by the node-engine's setting operator-signers (ADR 0539 clause 3) 2026-10-24 16:57:15 +02:00
jschoubben 83f16cff8d apply FILE is no hand path: mesh-compose and a want are (ADR 0539 clause 10); genesis keeps the carried bundle 2026-10-24 16:55:23 +02:00
jschoubben 96d0322498 The node-engine reports reads-wants; a failed apply is said; the have's fingerprint and digest follow its health (R6-A A6) 2026-10-24 16:43:26 +02:00
jschoubben b2003e7a38 A composition is applied only when the controller or the operator signed it; the node agent's integration tests on an embedded bus (R6-A A9, A1, A5) 2026-10-24 16:42:11 +02:00
jschoubben 3303797bfb The node agent runs in the node-engine: watches the wants and compositions on the link, applies one module's composition, asks the backup, writes the haves (R6-A A1, A5, A6) 2026-10-24 16:39:51 +02:00
jschoubben d5dd7761c6 The node agent: follows the wants and the node's compositions, applies one module at a time, backs a data module up first, holds it on a failed backup (R6-A A1, A2, A5, A6) 2026-10-24 16:36:21 +02:00
jschoubben 5f5f37b0b2 Pin mesh-sdk to feat/r6 S1-S3 (d6cef58): the composition bucket, the want's composition and backup answer, the have's composition and held phase 2026-10-24 16:31:20 +02:00
jschoubben c30b86ae22 Proved: the evidence is a have under the running build (comment) 2026-10-24 16:26:39 +02:00
jschoubben 1598156752 The node-engine hears no declaration: the queue, orders, put-backs, unsaid, numbers and news go; a session says the facts in the node-engine's own order, and the first have proves a build (R6-A A7, A8) 2026-10-24 16:25:11 +02:00
jschoubben e2b6f37168 Apply one module at a time: a scope of the module and what the others hold, and an owner record, so a removal is the module's own (R6-A A3, A4) 2026-10-24 16:16:32 +02:00
jschoubben 13bdcb694b Mark the packages whose tests read files outside their directory, found by the review (issue 1935)
mesh/delivery delivered
mesh/delivery-group group feat/r4g-finish merged: every member is merged
mesh/check checking 13bdcb694be0 on 3c3628c0ce46 | novox/mesh-host#133 in group feat/r4g-finish
2026-10-24 04:39:01 +02:00
jschoubben 89ebd0eefe Merge remote-tracking branch 'origin/feat/r4g-finish-1935' into HEAD 2026-10-24 04:26:19 +02:00
jschoubben b4f6537f80 Mark the packages whose tests read files outside their directory, so mesh-build-go never keeps their verdict (issue 1935) 2026-10-24 04:01:02 +02:00
jschoubben 7618f661ea Fail, not skip, when the checker places no controller beside the node-engine (issue 1295) 2026-10-24 03:43:49 +02:00
jschoubben 3c3628c0ce Merge pull request 'R4g (mesh-host): merge-check.sh goes; the checker judges the repository (design 47)' (#132) from feat/r4g into main
Reviewed-on: #132
2026-10-23 17:50:55 +00:00
jschoubben 4b4bb09a05 Make the apply tests fit the checker's isolated step: accept any who in the refused line, skip aloud where no group can be set (issue 1919, R4g)
mesh/delivery delivered
mesh/delivery-group group feat/r4g merged: every member is merged
mesh/check checking 4b4bb09a05a0 on 48c3e3834c69 | novox/mesh-host#132 in group feat/r4g
2026-10-23 18:54:53 +02:00
jschoubben 11b0a1fb29 Skip aloud the controller's step-nine check where the running checker places no controller (issue 1295, R4g)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: github.com/novox/mesh-host/internal/apply.TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing: judge_directories_…
2026-10-23 18:35:32 +02:00
jschoubben 6ea77844d6 Judge the other genesis modules' toolchains; set aside only build-agent (issue 1152, R4g)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: github.com/novox/mesh-host/internal/bootstrap.TestTheControllersOwnManifestPassesStepNine: genesis_form_test.go:249: the cont…
2026-10-23 18:20:50 +02:00
jschoubben 2633573012 Skip aloud the tests whose subject R4g removes: genesis's build agent (issue 1152)
The build agent left mesh-controller with R4g (ADR 0524 section 5); genesis still installs it until issue 1152 decides where a new mesh's builder comes from. The tests that read its manifest say NOT JUDGED and are judged again with 1152.
2026-10-23 18:20:11 +02:00
jschoubben 976c9998d6 Delete merge-check.sh; the checker's levels run what it ran (issue 1295, S3; ADR 0529)
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/check failed: unit: github.com/novox/mesh-host/internal/bootstrap.TestGenesisInstallsTheCataloguesBuildAgent: builder_test.go:180: mesh-controlle…
2026-10-23 15:26:03 +02:00
mesh-tools 48c3e3834c Merge pull request 'Take a root-made directory in the mesh's own tree, and hold a container until the file it mounts exists (issue 1873)' (#131) from fix/1873-mesh-tree-dirs into main 2026-10-22 20:41:08 +00:00
jschoubben 0d12fb11a4 Make no container before a file it mounts is there, and clear the empty directory a runtime left at a file's path (issue 1873)
mesh/merge-gate head 0d12fb11a48b merged into trunk 59e6a6fa2a4a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 0d12fb11a48b merged into trunk 59e6a6fa2a4a — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: mesh-build-go: no go build node qualifies: builders disagree: go1.27.1-X:nodwarf5 recipe 74744709d277ded9 on ace, g14, shanks; go1.…
On ace the postgres container was replaced at 22:11:51 mounting superuser.secret, a
file the provisioner's client fills only after the apply. The runtime made the
missing source a directory, and every fill after it failed with "is a directory".

A container whose mount source is a file the declaration declares, not yet on the
machine, now waits and is neither created nor replaced. The write of that file
(WriteFilled) applies what reads it, as before, so the container is made once the
file is there.

What the runtime already left is cured by the mesh: a declared file whose path is
an empty directory in the mesh's own tree (by its path and by where it leads),
owned by the engine's account, has that directory removed before the file is
written, and the outcome says so. A directory with anything in it, or outside the
tree, is never removed.
2026-10-22 22:29:39 +02:00
jschoubben 03b9c47af4 Judge the mesh's tree by where a path leads, not only by its text (issue 1873, review)
A root-only link inside /var/lib/mesh pointing out of it would have made a
directory outside the tree, owned by the engine's account, the mesh's to give
away: the fault of issue 1873 again. The path and the tree's root are now both
resolved, links followed, and the directory is the tree's only when the resolved
path lies below the resolved root as well.
2026-10-22 22:27:45 +02:00
jschoubben cdbff1777e Take a directory in the mesh's own tree that the engine's account owns as the mesh's, not as found (issue 1873)
postgres's data directory moved to /var/lib/mesh/postgres. On ace and novox that
directory was there, root:root 0700, before the directory resource was applied, so
the engine used it as found (issue 339's rule) and left it to root: the postgres
provider, running as the operator's account, cannot reach its own secrets,
consumers and grants there.

/var/lib/mesh is the engine's tree (root, 0755): only the engine's account makes a
directory in it. A directory below it owned by that account is now the mesh's and
is given its declared owner and mode. Anywhere else, or owned by another account,
issue 339's rule stands.
2026-10-22 22:23:04 +02:00
mesh-tools 59e6a6fa2a Merge pull request 'Write the store's superuser secret where the postgres module now keeps it, and read the bus's release from its upstream image (issues 1873, 1875)' (#130) from fix/1873-system-state-dirs into main 2026-10-22 20:04:32 +00:00
jschoubben 8c0bb9d453 Read the bus's release from the nats module's pinned server image, not a NATS_BASE build argument (issue 1875)
mesh/repo-check head 8c0bb9d4531d merged into trunk 4065b45ae84f beside mesh-catalog#453@e0bf87c2 — pass: its merge-check.sh passed
mesh/merge-gate head 8c0bb9d4531d merged into trunk 4065b45ae84f — pass: group fix/1873-system-state-dirs composed: every machine composes with the change …
mesh/delivery delivered
mesh/delivery-group group fix/1873-system-state-dirs merged: every member is merged
mesh/check failed: unit: github.com/novox/mesh-host/internal/bootstrap.TestGenesisInstallsTheCataloguesBuildAgent: builder_test.go:180: mesh-controlle…
The nats module takes its server as an upstream artifact, pinned by digest
(issue 1641), and builds no image on a NATS_BASE argument any more. The test of
the bundle's bus read that argument, found none, and failed every mesh-host check
against the catalogue's main. Bootstrap's own code never read it: the bundle's
bus image comes from the bundle, so genesis was not affected. The test now
compares the bundle's image with the module's server by repository and digest.
2026-10-22 21:46:34 +02:00
jschoubben de809d7e0a Keep the store's superuser file in the postgres module's own directory, not in /var/lib/postgres (issue 1873)
The postgres module's own secrets move from /var/lib/postgres, the postgresql
package's directory, to its directory under /var/lib/mesh (mesh-catalog,
fix/1873-system-state-dirs). Genesis writes the store's superuser file at the
module's own-secret path, so the store it raises is the one the module adopts in
place: StoreSuperuserFile follows, to /var/lib/mesh/postgres/superuser.secret.

A directory genesis makes on the way is 0755, the mode of the mesh's directories,
and only the credential's own is 0700: MkdirAll at 0700 would have closed
/var/lib/mesh on a machine that had none yet.
2026-10-22 21:36:38 +02:00
mesh-tools 4065b45ae8 Merge pull request 'Have a replacement image before removing a running container (issue 1742)' (#129) from fix/1732-held-moves-asked into main 2026-10-21 19:07:56 +00:00
jschoubben 09c77d57b8 Have a replacement container's image before removing the running one, so an unobtainable image leaves the service up (issue 1742)
mesh/repo-check head 09c77d57b823 merged into trunk c231d1031fc7 beside mesh-controller#428@d6db3348 — pass: its merge-check.sh passed
mesh/merge-gate head 09c77d57b823 merged into trunk c231d1031fc7 — pass: group fix/1732-held-moves-asked composed: every machine composes with the change a…
mesh/delivery delivered
mesh/delivery-group group fix/1732-held-moves-asked merged: every member is merged
2026-10-21 19:16:32 +02:00
mesh-tools c231d1031f Merge pull request 'Read a declared file from the node's own object bucket, declaration-files-<node> (issue 1695, ADR 0488)' (#128) from feat/1696-declaration-files into main 2026-10-21 16:31:25 +00:00
jschoubben e597f9791c Merge main (the container-groups change) into the bucket-per-node branch, keeping the mesh-sdk pin of #64
mesh/merge-gate head e597f9791c62 merged into trunk 7e83137aa822 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head e597f9791c62 merged into trunk 7e83137aa822 — pass: its merge-check.sh passed
mesh/delivery superseded: its walk was superseded and names no walk that took it over: a walk that took it over answers its merge, and it follows that wa…
2026-10-21 18:05:17 +02:00
jschoubben e3a595d8fe Pin mesh-sdk to the merge of #64 (a bucket per node) 2026-10-21 18:03:34 +02:00
mesh-tools 7e83137aa8 Merge pull request 'Pass a container's groups to the runtime as supplementary groups and report container-groups (issue 1637, ADR 0480 clause 7)' (#127) from feat/r4d-groups into main 2026-10-21 14:45:04 +00:00
jschoubben f9e1cd183a Pin mesh-sdk to the tip of fix/1694-bucket-per-node (issue 1695) 2026-10-21 15:49:44 +02:00
jschoubben d9f8bbec0c Pin mesh-sdk to the merge of #60 (container-groups capability)
mesh/repo-check head d9f8bbec0c0b merged into trunk fd3259a3f5d7 beside mesh-controller#422@2aa291f3 — pass: its merge-check.sh passed
mesh/merge-gate head d9f8bbec0c0b merged into trunk fd3259a3f5d7 — pass: group feat/r4d-groups composed: every machine composes with the change as it did w…
mesh/delivery delivered
mesh/delivery-group group feat/r4d-groups merged: every member is merged
2026-10-21 15:48:44 +02:00
jschoubben d5eeb860e3 Read a file's object from the node's own bucket, declaration-files-<node>, never a shared one (issue 1695, ADR 0486 §2)
mesh-sdk is pinned to fix/1694-bucket-per-node 36d9eba51db9 until it merges.
2026-10-21 15:41:09 +02:00
jschoubben ef30c12291 Pass a container's groups to the runtime as supplementary groups and report container-groups (issues 1637, ADR 0480 clause 7)
mesh-sdk is pinned to feat/r4d-groups f0f66d5b8d42 (mesh-sdk #58) until it merges.
2026-10-21 15:34:37 +02:00
mesh-tools fd3259a3f5 Merge pull request 'Fetch a file the declaration names by digest from the bus's object bucket and verify it before writing, and run the integration tests in the merge check (issue 1695, ADR 0486 step 2)' (#126) from fix/1695-fetch-by-digest into main 2026-10-21 13:33:54 +00:00
jschoubben fdb59bcf4e Correct the grant list above FileObject to the subjects it calls, flow control and consumer delete included (issue 1695)
mesh/merge-gate head fdb59bcf4eb7 merged into trunk c5c4caa2a4ab — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head fdb59bcf4eb7 merged into trunk c5c4caa2a4ab — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
2026-10-21 15:01:31 +02:00
jschoubben d0298a99da Pin mesh-sdk to the merge of #59 (file references) 2026-10-21 14:59:57 +02:00
jschoubben 4cfa0469fd Run the object-bucket test in the trunk's tagless suite too: the trunk's merge check judges this change (issue 1695) 2026-10-21 14:56:41 +02:00
jschoubben 45f437eea6 Run the integration tests in the merge check, resolve file references in the hand apply, refuse an unresolved one, and read an object by name without listing the bucket (issue 1695) 2026-10-21 14:50:34 +02:00
jschoubben a457b38968 Fetch a declaration's files given by digest from the bus's object bucket, verify them and report file-references (issue 1695, ADR 0486)
mesh-sdk is pinned to fix/1694-file-references 6c857e1c025f until it merges.
2026-10-21 14:34:29 +02:00
mesh-tools c5c4caa2a4 Merge pull request 'Count the home of an account a declaration makes for its own .ssh, never taking over another account's home (issue 1542)' (#125) from fix/1542-declared-account-is-a-home into main 2026-10-20 23:37:59 +00:00
mesh-tools 5a2bc03910 Let a declared account only add a home, never take one over, and match it by name alone, so no declaration hands another account's .ssh away (issue 1542, review round 1)
mesh/merge-gate head 5a2bc0391054 merged into trunk ce1b97ad973d — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 5a2bc0391054 merged into trunk ce1b97ad973d — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-20 23:32:27 +00:00
mesh-tools 60371aadad Count the home of an account a declaration makes as a home for the directory placement guard, so its own .ssh is placed (issue 1542) 2026-10-20 23:25:37 +00:00
mesh-tools ce1b97ad97 Merge pull request 'Take the node-engine's parser and validator from mesh-sdk and delete mesh-host's copies (design 47 R4b, issue 1392)' (#124) from feat/r4b-sdk-users into main 2026-10-20 12:51:54 +00:00
mesh-tools 52600ec7bd Take the node-engine's parser from mesh-sdk: remaining imports (issue 1392)
mesh/merge-gate head 52600ec7bd61 merged into trunk cdcba9ca7375 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 52600ec7bd61 merged into trunk cdcba9ca7375 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-20 11:17:14 +00:00
mesh-tools 278a94d59b Take the node-engine's parser from mesh-sdk: internal/apply imports (issue 1392) 2026-10-20 11:16:26 +00:00
mesh-tools 9d76759e04 Take the node-engine's parser and validator from mesh-sdk: pin, README and cmd imports (issue 1392) 2026-10-20 11:15:23 +00:00
mesh-tools a8eb24b49c Delete mesh-host's copy of the node-engine's parser and validator, which mesh-sdk holds (issue 1392), rest 2026-10-20 11:14:38 +00:00
mesh-tools da26b7f130 Delete mesh-host's copy of the node-engine's parser and validator, which mesh-sdk holds (issue 1392) 2026-10-20 11:14:30 +00:00
mesh-tools cdcba9ca73 Merge pull request 'The node-engine reports whether an unprivileged user can make a user namespace (design 47 R4a, issue 1308)' (#123) from feat/r4a-sdk-first into main 2026-10-20 07:15:53 +00:00
mesh-tools 454be9b269 Merge pull request 'Write a module that leaves a node as removed in its have key, so a reader can tell it stopped (issue 1369)' (#121) from fix/r3e-before-the-gate into main 2026-10-20 06:01:59 +00:00
mesh-tools 619e14aa8e Merge pull request 'Close the test bus only after the enrolment stand-in has removed its consumer, so the next test can subscribe (issue 604)' (#122) from fix/604-the-enrolment-stand-in-unbinds into main 2026-10-20 05:57:07 +00:00
mesh-tools d6f8b00bf8 Do not let the asking's Close shut the stand-in's connection before its cleanup (issue 604)
mesh/merge-gate head d6f8b00bf822 merged into trunk 2474bf28330f — pass: the change touches no module of the mesh's graph
mesh/repo-check head d6f8b00bf822 merged into trunk 2474bf28330f — pass: its merge-check.sh passed
mesh/delivery delivered
The asking helper registered a.Close, which closes the shared test connection; the stand-in's cleanup ran after it on a closed connection, so its consumer was never deleted and blocked the next test on the workqueue stream.
2026-10-20 01:34:27 +00:00
mesh-tools a9789b5125 Give each enrolment stand-in its own durable and wait for it to be gone (issue 604)
The stand-in shared one durable name across tests and let go with Unsubscribe alone, so a test starting before the server dropped the binding was refused with "consumer is already bound to a subscription".
2026-10-20 01:30:20 +00:00
mesh-tools 150fadfe8b Write a removal after a restart from the declaration the apply replaced (issue 1369)
mesh/repo-check head 150fadfe8bed merged into trunk 619e14aa8e57 — pass: its merge-check.sh passed
mesh/merge-gate head 150fadfe8bed merged into trunk 619e14aa8e57 — pass: group fix/r3e-before-the-gate composed: every machine composes with the change as …
mesh/delivery delivered
mesh/delivery-group group fix/r3e-before-the-gate merged: every member is merged
The set of modules this process wrote lives in memory, so a node-engine restarted before the apply that drops a module never wrote that module's removal. The report now names the modules the kept declaration held before, and they are removal candidates.
2026-10-20 01:26:33 +00:00
mesh-tools 6a2fb63bfc Write a module's have key as removed when the applied declaration no longer holds it (issue 1369)
A module that left a node kept its have key at running for ever, so nothing could tell it stopped and the gate would never take the seat. The key is written once with the phase removed over the revision read, never deleted. Pins mesh-sdk to the commit that adds the phase.
2026-10-20 01:20:52 +00:00
mesh-tools 64dfa90c3a Report whether an unprivileged user namespace can be made (issue 1308)
mesh/repo-check head 64dfa90c3a6c merged into trunk 454be9b2693d — pass: its merge-check.sh passed
mesh/merge-gate head 64dfa90c3a6c merged into trunk 454be9b2693d — pass: group feat/r4a-sdk-first composed: every machine composes with the change as it di…
mesh/delivery delivered
mesh/delivery-group group feat/r4a-sdk-first merged: every member is merged
A build module's sandbox needs one, and a module can require only a capability the node-engine detects (ADR 0448 section 5). The probe drops to an unprivileged uid first when run as root, since root always can.
2026-10-20 01:12:24 +00:00
mesh-tools 2474bf2833 Merge pull request 'The node-engine sends no alive message: presence replaces it (design 47 R2d, issue 1158)' (#120) from feat/alive-gone into main 2026-10-19 07:07:56 +00:00
jschoubben 671f284004 Delete the node-engine's alive message, presence replaces it (issue 1158)
mesh/merge-gate head 671f2840046f merged into trunk 89025ad004d4 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 671f2840046f merged into trunk 89025ad004d4 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-19 02:15:00 +02:00
mesh-tools 89025ad004 Merge pull request 'Judge a filled write with the roots the last apply read, asking no runtime per file (issue 1138)' (#119) from fix/1138-filled-write-asks-no-runtime into main 2026-10-18 22:20:54 +00:00
jochen 3798673821 Judge a filled write with the roots the last apply read, asking no runtime per file (issue 1138)
mesh/merge-gate out of date: judged against trunk 8119c470d393, the trunk is now 89025ad004d4; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 8119c470d393, the trunk is now 89025ad004d4; judged again on a recheck or when it is merged
mesh/delivery delivered
2026-10-19 00:11:58 +02:00
mesh-tools 8119c470d3 Merge pull request 'Write each module's have and the node's own facts beside the report (issue 1132); the check reads build-agent from the controller (issue 1142)' (#118) from feat/delivery-have into main 2026-10-18 18:36:14 +00:00
jschoubben aab65b81f6 Read the build agent's manifest from the controller's repository beside the check, where it moved (issue 1151)
mesh/repo-check head aab65b81f65c merged into trunk 6469e8238443 — pass: its merge-check.sh passed
mesh/merge-gate head aab65b81f65c merged into trunk 6469e8238443 — pass: group feat/delivery-have composed: every machine composes with the change as it di…
mesh/delivery delivered
mesh/delivery-group group feat/delivery-have merged: every member is merged
2026-10-18 19:28:07 +02:00
jschoubben 9d9e3a46c4 Bind the have bucket once per pass, bound by the queue's timeout, and stop the pass at the first error (issue 1132) 2026-10-18 19:18:08 +02:00
jschoubben 0b391f49a2 Write each module's have and the node's own facts beside the report, grouped per module and keyed by the module the controller names (issue 1132)
The node-engine reads a resource's module and a declaration's builds, says it does in every report (reads_modules), groups the apply's outcome per module with every failing resource named, and writes have.<node>.<module> and have.<node>.node to the bucket only when a value differs, never ahead of or in place of the report (novox/hq ADR 0429 section 4).
2026-10-18 19:10:05 +02:00
mesh-tools 6469e82384 Merge pull request 'A directory says how it reacts to files it does not know; the node-engine moves out what a move-out directory does not know (issue 1024)' (#117) from feat/owned-paths into main 2026-10-17 16:20:06 +00:00
mesh-tools dae79c7643 Prove move-out on a machine that is not Arch (issue 1024)
mesh/merge-gate head dae79c76434c merged into trunk cd13f7726ef0 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head dae79c76434c merged into trunk cd13f7726ef0 — pass: its merge-check.sh passed
mesh/delivery delivered
The merge check's build container has no pacman, so the acceptance test now stubs it and names the host it was built for, as other command tests do; nothing in the node-engine itself changes.
2026-10-17 16:16:02 +00:00
mesh-tools d237dee4c9 Let a directory say how it reacts to files it does not know, and move out what a move-out directory does not know (issue 1024, ADR 0422)
mesh/merge-gate head d237dee4c9e3 merged into trunk cd13f7726ef0 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head d237dee4c9e3 merged into trunk cd13f7726ef0 — fail: its merge-check.sh failed: --- FAIL: TestAMoveOutDirectoryKeepsOnlyWhatItKnows (0.…
mesh/delivery superseded: a newer head of the same pull request
A directory or archive resource carries `unknown` (move-out, module, program) and `knows`. For move-out under a home, every look moves each path the declaration does not know into ~/.local/state/mesh-moved/<module>/<time>/, the list written before each move, without following a link, a young path waiting, a socket or pipe left and reported, nothing made when nothing moved, and another file system refused. Reviewed in two rounds.
2026-10-17 16:09:33 +00:00
mesh-tools cd13f7726e Merge pull request 'Feature 57 (config and provisioning): release mesh-host' (#116) from release/57-config-and-provisioning into main 2026-10-16 09:32:15 +00:00
mesh-tools bff1ac14a1 Feature 57 (config and provisioning): release mesh-host, the reviewed release branch on top of main
mesh/merge-gate head bff1ac14a198 merged into trunk 49c9002c04d3 beside mesh-controller#355@8c381172 — pass: builds mesh-host → ace, g14, novox, shanks; no…
mesh/repo-check head bff1ac14a198 merged into trunk 49c9002c04d3 beside mesh-controller#355@8c381172 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group release/57-config-and-provisioning merged: every member is merged
2026-10-16 08:24:13 +00:00
mesh-tools 49c9002c04 Merge pull request 'Grant the genesis controller user operator-channel.tell (issue 835)' (#113) from feat/mail-seats into main 2026-10-15 17:09:30 +00:00
jochen c3b2fce0a2 Grant the controller's new operator-channel.tell publish to the genesis user (issue 835)
mesh/merge-gate head c3b2fce0a29f merged into trunk 198fd34ec497 — pass: the change touches no module of the mesh's graph
mesh/repo-check head c3b2fce0a29f merged into trunk 198fd34ec497 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/mail-seats merged: every member is merged
ADR 0380's secret-given announcement asks operator-channel.tell over the
bus; the genesis bundle's first user list must already carry that
publish subject, in the same place the controller composes it
(internal/broker, mesh-controller ms-835 435468c1), or a first node's
controller starts without the grant until its own first composition.
2026-10-15 17:49:51 +02:00
mesh-admin 198fd34ec4 Merge pull request 'Start no account's manager to look at its units (hq issue 764)' (#112) from fix/764-a-look-starts-no-user-manager into main 2026-10-14 22:38:02 +00:00
jschoubben 62af89f85d Start no account's manager to look at its units (hq issue 764)
mesh/merge-gate head 62af89f85d0d merged into trunk 1a2677882380 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 62af89f85d0d merged into trunk 1a2677882380 — pass: its merge-check.sh passed
mesh/delivery delivered
Every 15 s the failed-units look read the own manager of every account
the declaration creates, and reading one is a login that starts it: six
module accounts' managers on the control node started and stopped all
day, most of its journal. A look only reads (ADR 0240 rule 6) and never
asks an account's manager (ADR 0177).

An account's manager is read now only when the declaration places a
user-scoped unit in it, and only after the machine's own manager says
user@<uid>.service is active, as the apply asks before it. A manager not
running is said away in the units statement, with no failed units; in
liveness its units read unknown with the apply's words, never down. The
operator's account on a desktop is still read while logged in, since a
module places a user unit there (issue 315's Razer daemon stays said).
2026-10-15 00:33:34 +02:00
mesh-admin 1a26778823 Merge pull request 'A busy container is left running until it says idle, within 15 minutes (issue 594, hq ADR 0335)' (#98) from feat/335-a-busy-container-is-replaced-when-idle into main 2026-10-13 22:20:54 +00:00
jschoubben b9a293f8c8 Merge remote-tracking branch 'origin/main' into feat/335-a-busy-container-is-replaced-when-idle
mesh/merge-gate head b9a293f8c8bc merged into trunk 0e92ef3d1b21 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head b9a293f8c8bc merged into trunk 0e92ef3d1b21 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle merged: every member is merged
# Conflicts:
#	cmd/mesh-host/main.go
2026-10-14 00:00:09 +02:00
jschoubben 695e022991 Keep a replacement owed after a busy wait, and clamp a wait from a later clock
A restart-on trigger that is no file counts as changed only in its own pass,
so after a busy wait the container read unchanged and was never replaced,
while the report stopped listing it (review of #98).
2026-10-13 23:59:58 +02:00
mesh-admin 0e92ef3d1b Merge pull request 'Report the networks the outward links are on, for the local network (issue 154, ADR 0340)' (#107) from fix/154-local-network into main 2026-10-13 13:07:08 +00:00
mesh-admin 12bc6a56a3 Merge pull request 'Adopt genesis's bus through the bus's planned step, its streams snapshotted first (issue 590, hq ADR 0333 §3)' (#108) from fix/590-adopt-the-bus-as-its-planned-step into main 2026-10-13 11:11:41 +00:00
jochen cdcfcd9380 Adopt genesis's bus through the bus's planned step, its streams snapshotted first (hq issue 590, ADR 0333 §3)
mesh/merge-gate head cdcfcd938016 merged into trunk 130c5d98a601 — pass: the change touches no module of the mesh's graph
mesh/repo-check head cdcfcd938016 merged into trunk 130c5d98a601 — pass: its merge-check.sh passed
mesh/delivery delivered
The broker step sent the nats module with a plain push, and the controller refuses a push that would
replace the bus: that is a planned step, bus upgrade, which snapshots the streams first and checks
them after (ADR 0236). Genesis stopped at step 18.

The step now snapshots every stream with the module's own snapshot program (no backup holder exists
yet to be asked), asks bus upgrade with that snapshot and why, says the one restart, and waits until
the bus runs the module's image, answers, and the controller ends the step done. The bundle's bus runs
the release the module's image is built on (2.11.17, by the same digest, where it ran 2.10.29), so the
step changes no server release and is said reversible.
2026-10-13 11:47:10 +02:00
mesh-admin 130c5d98a6 Merge pull request 'Build the Go toolchain at genesis's base step, beside the base (issue 616)' (#106) from fix/616-the-base-builds-every-genesis-toolchain into main 2026-10-13 09:23:04 +00:00
jochen cb7b71dcc5 Report the networks the outward links are on, for the local network (issue 154, ADR 0340)
mesh/merge-gate head cb7b71dcc568 merged into trunk 12bc6a56a38a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head cb7b71dcc568 merged into trunk 12bc6a56a38a — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/154-local-network merged: every member is merged
A port reached from local-network is admitted from the private networks the machine is attached to, and only the machine can say which those are. Every network on a link facing outside is said among its facts, and a change is said at the next reconcile.
2026-10-13 11:20:21 +02:00
jochen 8108dd5c23 Build the Go toolchain at genesis's base step, beside the base (hq issue 616)
mesh/merge-gate head 8108dd5c23c6 merged into trunk f0d39b4a4ab0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 8108dd5c23c6 merged into trunk f0d39b4a4ab0 — pass: its merge-check.sh passed
mesh/delivery delivered
Genesis's base step built only mesh-tools, the TypeScript toolchain. The store's and the bus's modules
carry Go bundles, compiled in mesh-tools-go, so genesis stopped at the store step: "this mesh holds no
copy of it. Build mesh-tools-go first". The base step now builds mesh-tools-go too, from
--tools-go-source or the repository beside --tools-source, and a test holds every toolchain a genesis
module's bundles compile in, read from the catalogue, to the ones the step builds.
2026-10-13 11:13:44 +02:00
mesh-admin f0d39b4a4a Merge pull request 'Wait until the node applied the genesis document before the sdk step asks for a build (issue 615)' (#105) from fix/615-wait-for-the-genesis-document into main 2026-10-13 09:08:59 +00:00
jochen 89fd0c06e1 Wait until the node applied the genesis document before the sdk step asks for a build (hq issue 615)
mesh/merge-gate head 89fd0c06e102 merged into trunk 17038b2a074b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 89fd0c06e102 merged into trunk 17038b2a074b — pass: its merge-check.sh passed
mesh/delivery delivered
The packages step sealed the build agent its genesis document and pushed, and the push returned as soon
as it was sent. The sdk step then asked for the SDK's build at once, and the agent reads the document
from the file the node writes when it applies that push: one genesis found it in time, the next was
refused for having neither a binding nor the document. That one push now waits until the node reports
it applied it (push --wait); every other push still returns on sending.
2026-10-13 10:59:45 +02:00
mesh-admin 17038b2a07 Merge pull request 'Raise genesis's store as the postgres module's server, so the store step adopts it as it runs (issue 609, hq ADR 0338)' (#104) from fix/609-genesis-raises-the-store-as-postgres into main 2026-10-13 08:55:41 +00:00
jochen 4dde21f0ab Raise genesis's store as the postgres module's server, so the store step adopts it as it runs (hq issue 609, ADR 0338)
mesh/merge-gate head 4dde21f0ab20 merged into trunk 7d1d6bffe30c — pass: the change touches no module of the mesh's graph
mesh/repo-check head 4dde21f0ab20 merged into trunk 7d1d6bffe30c — pass: its merge-check.sh passed
mesh/delivery delivered
The bundle raised the store as mesh-store, on upstream postgres, in a docker volume; the postgres
module keeps it as postgres, on its pgvector image, in /var/lib/mesh-store owned 999:70, reading its
superuser password from a file. The store step refused, and genesis stopped at step 17 of 24.

The bundle now raises the store with the module's name, image, data directory and password file, and
every step of the bundle and the installer reaches it by that name. The refusal to raise a store over
data it cannot open looks for a cluster in the store's directory, no longer for the old volume.
2026-10-13 10:47:53 +02:00
mesh-admin 7d1d6bffe3 Merge pull request 'Place the bus's users after every push until the bus is adopted (issue 608)' (#103) from fix/608-place-the-bus-users-after-every-push into main 2026-10-13 05:00:42 +00:00
jochen 7283209d1a Place the bus's users after every push until the bus is adopted (hq issue 608)
mesh/merge-gate head 7283209d1abc merged into trunk 64f0ab405101 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 7283209d1abc merged into trunk 64f0ab405101 — pass: its merge-check.sh passed
mesh/delivery delivered
The bus reads its user list from a file that, until the broker step adopts it as the nats module,
only the installer writes, and the installer wrote it only at enrolment. The build agent's account,
minted with step 13's push, never reached the bus; the agent was refused (Authorization Violation)
and the sdk step's build waited until it was killed. Every push genesis makes now places the composed
list and has the bus re-read it, until the nats module is registered (ADR 0331 §5, ADR 0333 §4).
2026-10-13 06:58:39 +02:00
mesh-admin 64f0ab4051 Merge pull request 'genesis: seal the build agent the forge it raised, in place of the builder's npm-password (hq issue 121, ADR 0336)' (#102) from fix/121-genesis-package-registry into main 2026-10-13 03:37:15 +00:00
jochen 5f147fe6b9 Seal the build agent the forge genesis raised, in place of the builder's npm-password (hq issue 121, ADR 0336)
mesh/merge-gate head 5f147fe6b9c7 merged into trunk 0d11b4fc930d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5f147fe6b9c7 merged into trunk 0d11b4fc930d — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/121-genesis-package-registry merged: every member is merged
The build agent declares neither the retired builder's npm-password own-secret
nor a serves setting, so the packages step handed its credential to nothing
and genesis's sdk step had no way to publish. The packages step now seals the
genesis-package-registry own-secret to build-agent: the forge on this node's
loopback, the port given for the package registry, its npm path, and the
mesh-builder account with its password. setFoundationSettings no longer gives
the builder a serves.port.
2026-10-13 03:01:35 +02:00
mesh-admin 0d11b4fc93 Merge pull request 'Install the build agent at genesis, the module that builds since ADR 0190 (issue 596)' (#101) from fix/596-genesis-installs-the-build-agent into main 2026-10-13 00:52:22 +00:00
mesh-admin d34b4f32e1 Merge pull request 'Leave a file the controller's manifest writes itself out of genesis's secrets (issue 597)' (#100) from fix/597-a-file-the-module-writes-is-not-a-secret into main 2026-10-13 00:40:33 +00:00
jochen d941d798e9 Install the build agent at genesis, the module that builds since ADR 0190 (hq issue 596)
mesh/merge-gate head d941d798e9df merged into trunk d34b4f32e17f — pass: the change touches no module of the mesh's graph
mesh/repo-check head d941d798e9df merged into trunk d34b4f32e17f — pass: its merge-check.sh passed
mesh/delivery delivered
Genesis's builder step read modules/builder from the catalogue, which retired it on 2026-10-03 for
the build agent (node-build-agent on every machine), so no genesis got past step 13. The step now
installs build-agent from the catalogue, puts the builder image it carried and published where the
manifest names the image it builds, and issues its bus account as any module's.
2026-10-13 02:25:10 +02:00
jochen b7f70bc4c0 Leave a file the controller's manifest writes itself out of genesis's secrets (hq issue 597)
mesh/merge-gate head b7f70bc4c01d merged into trunk 8431ac96562e — pass: the change touches no module of the mesh's graph
mesh/repo-check head b7f70bc4c01d merged into trunk 8431ac96562e — pass: its merge-check.sh passed
mesh/delivery delivered
The controller reads its merge window's settings from a file its manifest declares from its
settings (MESH_MERGE_WINDOW_FILE, ADR 0276). Step 11 took every MESH_..._FILE for a secret and
refused this one as a file nobody writes, so no genesis got past step 11. A _FILE naming a file
resource of the module is now the module's own; one naming neither an own-secret nor a file of the
module is still refused.
2026-10-13 02:23:21 +02:00
mesh-admin 8431ac9656 Merge pull request 'Raise genesis's bus where the nats module keeps it, so adopting it keeps it (issue 590, hq ADR 0333)' (#99) from fix/590-genesis-bus-is-the-nats-module into main 2026-10-13 00:22:01 +00:00
jochen bc2a448512 Raise genesis's bus where the nats module keeps it, so adopting it keeps it (hq issue 590, ADR 0333)
mesh/merge-gate head bc2a4485122c merged into trunk 7a41e120635b — pass: the change touches no module of the mesh's graph
mesh/repo-check head bc2a4485122c merged into trunk 7a41e120635b — pass: its merge-check.sh passed
mesh/delivery delivered
The bundle raised the bus as mesh-broker on docker volumes; the catalogue's nats module keeps it as
mesh-broker-nats on directories of the machine, so the broker step refused rather than raise a second
bus, and genesis stopped at step 18 of 24.

The bundle now raises mesh-broker-nats with its configuration and user list in /var/lib/nats-module/conf,
its streams in /var/lib/mesh-broker-nats and its certificate in /var/lib/mesh-broker-tls, published on the
node's bus port. The broker step holds that layout to the module's manifest field by field before
registering anything, says the one restart onto the module's image, and waits until the bus runs that image
and its monitor answers. The installer's bridge writes the composed users at the module's bus-users path and
signals mesh-broker-nats. The bundle's bus configuration and user list are seeds, so a re-run after the
adoption does not put the bundle's one user back over the module's.
2026-10-13 02:09:03 +02:00
jschoubben bb60e7b584 Leave a busy container running until it says idle, within 15 minutes
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle rejected: its order contradicts itself: novox/mesh-catalog@190608193764, novox/mesh-c…
mesh/merge-gate out of date: judged against trunk 7a41e120635b, the trunk is now 8431ac96562e; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 7a41e120635b, the trunk is now 8431ac96562e; judged again on a recheck or when it is merged
A send that replaced the build agent killed the check it ran, and the queue
started it again elsewhere (issue 594). A container may now declare busy; the
node-engine asks it before replacing it, applies everything else, says the
wait, and replaces it on the first idle pass or at the bound, kept across
restarts (novox/hq ADR 0335).
2026-10-13 00:41:32 +02:00
mesh-admin 7a41e12063 Merge pull request 'The tool runner joins the own groups of the modules whose tools it runs, and own directories may keep their group (hq ADR 0334, issue 567)' (#97) from feat/334-the-runner-joins-own-groups into main 2026-10-12 22:22:14 +00:00
jschoubben c7b534a08c Keep a module directory's setgid bit, warn about a refused runner, and restart the runner when it leaves (hq ADR 0334 review)
mesh/merge-gate head c7b534a08c98 merged into trunk 6fe82c61bd32 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head c7b534a08c98 merged into trunk 6fe82c61bd32 — pass: its merge-check.sh passed
mesh/delivery delivered
A module's own directories need setgid so what is made beneath keeps its own group,
and a mode of 2750 refused the whole declaration while every apply dropped the bit:
a directory's mode now carries setgid (never setuid or sticky), set, kept and said.
A tool runner's account that is missing or root's failed the whole apply and dropped
the library's outcome: it is now a warning, said as runner:<module> or
runner:library:<name>. And a runner that leaves a group is taken out before the
resources apply, so its unit is restarted in that apply to lose it.
2026-10-13 00:11:50 +02:00
jschoubben ccc86d3594 Put the tool runner in the own group of each module whose tools it runs (hq ADR 0334 §2)
A joining module's own directories belong to its own group, and its tools run in the
tool runner, which could not read them. An identity may now name the tool runner's
account: the engine puts it in mesh-<module> as it puts it in a library's group,
says the change as identity:<module> for the runner's unit to restart once, takes
it back when no identity names it, and refuses an account that is missing or root's.
The engine states contract 7.
2026-10-13 00:06:29 +02:00
mesh-admin 6fe82c61bd Merge pull request 'Raise genesis on the bus the mesh runs on, from one bundle (issue 146, ADR 0331)' (#96) from fix/146-genesis-on-the-bus into main 2026-10-12 21:11:52 +00:00
mesh-admin 8e95c1d9ed Merge pull request 'A machine says what it is made of, and says it once a day unasked (issue 160)' (#92) from fix/160-a-machine-says-what-it-is into main 2026-10-12 21:07:05 +00:00
jochen cb55a46400 Place the bus's users again on a re-run until the bus is adopted (hq issue 146, review 102)
mesh/merge-gate head cb55a4640050 merged into trunk 8e95c1d9ed2d — pass: the change touches no module of the mesh's graph
mesh/repo-check head cb55a4640050 merged into trunk 8e95c1d9ed2d — pass: its merge-check.sh passed
mesh/delivery delivered
A re-run's apply writes the bundle's one-entry user list over the list placed at
enrolment, and an enrolled machine skipped placing it: enrolled was taken for adopted,
against ADR 0331 §5. Until the controller knows the nats module, the installer places
the composed list again. Also drop the previous broker's AMQP and management ports,
which nothing raises any more, and say why the bus always carries a ports setting.
2026-10-12 23:06:33 +02:00
jochen 37947ca4ee Raise genesis on the bus the mesh runs on, from one bundle (hq issue 146, ADR 0331)
mesh/merge-gate head 37947ca4eed1 merged into trunk d8e785bb4b4b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 37947ca4eed1 merged into trunk d8e785bb4b4b — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The installer could not raise a first node: the old bundle's control plane had no
MESH_BUS_NATS and crash-looped, and the NATS bundle was refused because rootsecrets
only knew the previous broker's guest:guest@ template.

One bundle now, the NATS one. rootsecrets makes the controller's bus account and puts
it in both placeholders (the address and the bcrypt hash in the carried user list),
kept with its hash so a re-run writes the same bundle. Preflight refuses a template
naming no bus connection it knows, naming the placeholder, instead of waiting out the
verify step. The broker step adopts the bus as the nats module and refuses, by name,
a catalogue without it or a server that is not the foundation's container.
2026-10-12 22:49:49 +02:00
mesh-admin d8e785bb4b Merge pull request 'Keep libraries under a group of their own, and give each joining module a user in it (hq ADR 0328, issue 567)' (#95) from feat/328-libraries-and-module-users into main 2026-10-12 20:28:16 +00:00
jschoubben 281837847e Put busybox's account removal back, test the sample, and read a held lock as kept only for a running walk (hq ADR 0328 review)
mesh/merge-gate head 281837847e03 merged into trunk 303d4037e84e — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 281837847e03 merged into trunk 303d4037e84e — pass: its merge-check.sh passed
mesh/delivery delivered
An earlier change dropped alpine's RemoveAccount and RemoveGroup, so alpine made no
module identity at all. The sample beneath a root in shape had no test of its own.
And an apply that found the library's lock held read the library as kept even when
what held it was a take act about to be refused for a missing disk; it now does so
only while a take walk is running, and otherwise waits and judges as always.
2026-10-12 22:21:42 +02:00
jschoubben c7551162d6 Give no account a home, keep old data's sub-paths for the warrant, and never hand out setgid (hq ADR 0328 review)
usermod --gid regroups every file under an account's home, and accounts were made
with / as home: they now have /nonexistent, and an older account's home is moved
there (without its files) before its group changes. A sub-path asked for later that
holds data not in the group waits for the operator as the root would. The take act
gives the group to hardlinked files an account owns (the *arr apps' downloads) and
leaves root's; a file's setuid or setgid bit the regroup takes is never put back, so
no setgid file hands out the library's group; each is counted. An apply no longer
waits for a running take walk.
2026-10-12 22:18:16 +02:00
jschoubben 2cffa33bcc Keep a root without asking only when its data beneath is the library's already (hq ADR 0328 §7)
A root in the group with setgid was kept without a warrant even when the data beneath
it was not in the group, which §7 says the operator is asked about. A root is now
kept without asking only when it is empty, or it and a bounded sample beneath it are
all in the group with group write (and setgid on folders). On busybox, an account the
mesh made is given its added groups with addgroup and delgroup; a change of its
primary group is refused, saying why.
2026-10-12 22:14:10 +02:00
jschoubben 2a2be687f3 Keep libraries under groups of their own, and give modules users in them (hq ADR 0328, issue 567)
ADR 0328 supersedes ADR 0325: shared data is a library with one home, kept usable by
its group rather than by ACL entries. The engine now makes each library's group and
each module's user in its libraries' groups; keeps a library's root and requested
sub-paths in its group with group write and setgid on its home, making missing
sub-paths as the root is; waits for the operator's word before touching a root that
holds data, and on the take act walks it by descriptor (no link, same filesystem,
bounded and resumed, files with other names left); holds the containers of a library
whose disk or mount is not there; and puts the tool runner's account in the group.
The ACL keeping, mask repair and watch are gone; an access is again as before ADR 0325.
2026-10-12 22:11:09 +02:00
mesh-admin 303d4037e8 Merge pull request 'Give the controller the contract it can fill: tool runners by name, the take act's access id, wait reasons and ACL support (hq ADR 0325)' (#94) from feat/325-the-contract-the-controller-needs into main 2026-10-12 17:57:41 +00:00
jschoubben 5c8e768ef5 Read an account by its name, never as a number getent takes for a uid (hq ADR 0325)
mesh/merge-gate head 5c8e768ef520 merged into trunk ef26dba2753a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 5c8e768ef520 merged into trunk ef26dba2753a — pass: its merge-check.sh passed
mesh/delivery delivered
getent passwd reads an all-digit key as a uid, so a tool runner's account named
"1000" resolved to whoever has uid 1000 and the entry went to them. Only an answer
naming the account asked for is taken.
2026-10-12 19:51:09 +02:00
jschoubben aad33dfe5c Give the controller the contract it can fill (hq ADR 0325, issue 567)
The controller cannot know the operator's account's uid, names an access by its
declared id (which holds a slash), and should not parse words. A tool runner's
account may now come by name alone and is read here (refused when absent or root's,
its uid said in the access's listing); the take act matches the declared id exactly;
an access wait says its reason (keep or disk) and, for a filesystem that refuses
ACLs, whether the take act can enable them.
2026-10-12 19:51:09 +02:00
mesh-admin ef26dba275 Merge pull request 'Give every module an identity and keep shared folders through ACL entries (hq ADR 0325, issue 567)' (#93) from feat/325-identities-and-kept-folders into main 2026-10-12 17:50:59 +00:00
jschoubben d12bba0109 Give the kept-folder tests' folders an unprivileged owner when they run as root (hq ADR 0325)
mesh/merge-gate head d12bba01090d merged into trunk dd9ab7927d96 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head d12bba01090d merged into trunk dd9ab7927d96 — pass: its merge-check.sh passed
mesh/delivery delivered
The build seat runs merge-check as root, so every folder a test made was root's
data, which is never kept (§12), and 28 tests failed for that alone. A test's
folders now stand for an operator's: run as root, they are given to nobody before
they are judged. The root rule is unchanged.
2026-10-12 19:34:59 +02:00
jschoubben c85c2804a0 Apply an older controller's accesses exactly as before (hq ADR 0325)
mesh/merge-gate head c85c2804a010 merged into trunk dd9ab7927d96 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head c85c2804a010 merged into trunk dd9ab7927d96 — fail: its merge-check.sh failed: --- FAIL: TestAnAccessTheMeshMayNotKeepYetIsSaidAsAWait…
mesh/delivery superseded: a newer head of the same pull request
The engine delivers before the controller half, and today's controller sends no
identities, users or taken. Keeping now starts only for an access whose folder has
users, or a declaration with identities; every other access is mounted when present
and refused when absent as on main, with nothing made, kept, held or judged. A
leftover whose walk could not run says why, not always that its disk is away.
2026-10-12 19:24:24 +02:00
jschoubben 2ff40109ac Read a filesystem that could not be named as unknown, never as a move (hq ADR 0325 sixth review)
A name read that timed out gave an empty name, which differed from the recorded one
and so reset the record: a folder kept while empty asked again, a warranted one was
walked again. An unnamed filesystem now leaves the folder exactly as it is, the
access is said not known for that look, and it resumes once it is named.
2026-10-12 19:20:53 +02:00
jschoubben 46c16e18fe Name a filesystem by its UUID, its kernel id or its source, and keep nothing that has no name (hq ADR 0325 fifth review)
NFS and FUSE report a zero f_fsid, so another export or another FUSE mount at the
same point was named alike and kept without a warrant; xfs's f_fsid follows the
device number. A filesystem is now named by its UUID (its device matched in
/dev/disk/by-uuid), else a nonzero f_fsid, else mountinfo's source; one that cannot
be named is never kept or taken. Entries left on a filesystem not back within 30 days
are forgotten, said once, and the naming read is shared per mount.
2026-10-12 19:14:42 +02:00
jschoubben f00f9c2a2f Name a filesystem by its kernel id, never its device name (hq ADR 0325 fourth review)
An ext4 or xfs library mounted by UUID can come back from a reboot as another device
(/dev/sdb1 as /dev/sdc1); naming the filesystem by mountinfo's source read that as a
move, stripped the warranted library's entries and asked for a new Approve. A
filesystem is now named by statfs's f_fsid (from the UUID, or the ZFS dataset), its
type and its mount point. Entries left on the filesystem a folder is on now are
merged back or held, never walked away, and a TakenOn naming a mount point is not
taken.
2026-10-12 19:10:18 +02:00
jschoubben 21cd036232 Format the third review's tests (hq ADR 0325) 2026-10-12 19:06:57 +02:00
jschoubben f0adb177f0 Leave no leftover for a folder kept only while empty, and name filesystems by more than their mount point (hq ADR 0325 third review)
A folder kept because it was empty, then covered by its disk, was said as unhealthy
leftover entries for ever: its entries sit on a hidden empty folder and grant nothing,
so they are now forgotten, and only entries walked onto data under a warrant are kept
to be taken out. A warrant and a kept folder now name their filesystem by mount point,
source and type, so another pool at the same point is a move; records naming a mount
point are read as its name. A record with no access ids is said under a path-free name.
2026-10-12 19:06:34 +02:00
jschoubben d2d9fbbd7d Hold what mounts a missing disk, and say entries the mesh could not take out (hq ADR 0325 re-review)
A container was still created over an unmounted mount point, so its writes landed on
the filesystem below; a folder moved to another disk was kept on the old warrant; and
entries left by a drop or a move that could not run lived only in a log line. Every
container mounting an access whose disk is away now waits, the warrant holds only for
the filesystem it was carried out on (the wait names the new one, path-free), entries
left on a filesystem are recorded per mount and taken out when it is back, and every
leftover is said as unhealthy until it is. A hung stat is shared per path, and a
widened mask takes from other entries only the bits it adds.
2026-10-12 19:03:02 +02:00
jschoubben 2026ef36f3 Judge a kept folder before every walk, and keep nothing a disk mounted over (hq ADR 0325 review)
A walk run while a dataset was unmounted wrote onto root's empty mount point, or
"dropped" entries from it and forgot the ones on the data; a folder kept while empty
below an unnamed mount was then swept whole once the disk mounted, with no warrant.
Every walk is now judged as an apply is, a changed mount judges the folder afresh, and
systemd .mount units and mounts seen are read for the first-sight rule. Widening a
mask caps the entries it limited, so the owning group gains nothing. The watch lets
go of removed folders, re-arms on lost notice, reads users under its lock and ends
with its context. A drop that cannot read an ACL is counted, a broken folder's words
carry no path, a grant's group that could not be left is tried again, and a folder's
stat is bounded.
2026-10-12 18:55:47 +02:00
jschoubben 51e97a8c14 Give every module an identity and keep shared folders through ACL entries (hq ADR 0325, issue 567)
A module's containers ran as whatever their image chose and shared folders were
made writable by putting accounts in the folder's group (ADR 0312), which ADR 0325
supersedes. The engine now makes each declared module identity as a system account
with the controller's numbers, runs a container declaring `identity` as it, and keeps
each placed shared folder usable through one POSIX ACL entry per user, set through
the kernel's xattrs on descriptors it opened itself. A folder holding data is kept
only after the operator's warrant (the take act); a missing folder is made where its
data lives and waits where its disk is not; root's data is never kept. A watch and a
bounded sweep put back what an app narrows. The group grant is removed, and what it
put an account in is taken back.

Reuses the descriptor walks, the create act, the missing-path wait and the warrant-
backed act message from the paused issue 567 branch.
2026-10-12 18:43:04 +02:00
jochen c7a4bf5c11 Judge the daily report by wall time, and keep one filesystem check at a time (issue 160)
mesh/merge-gate head c7a4bf5c11d6 merged into trunk d8e785bb4b4b — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head c7a4bf5c11d6 merged into trunk d8e785bb4b4b — pass: its merge-check.sh passed
mesh/delivery delivered
Go's monotonic clock stops while a laptop is suspended, so a day judged by it could pass a day and a half
of wall time before the machine said what it is. And a hung statfs left a goroutine and thread blocked per
apply; now one check runs at a time and reports after it say the sizes unknown until it ends.
2026-10-12 17:42:18 +02:00
jochen ec983bfc3d Have a machine say what it is made of, and say it once a day unasked (issue 160)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The mesh knew a machine's architecture and kernel family and nothing it is made of, and heard even
that only when it sent the machine something. The profile now carries the machine's memory, each local
filesystem's size and free space, processor model, cores and threads, distribution and version, kernel
release, virtual or physical, boot time and timezone; a fact it cannot read is named unknown with why,
never zeroed. The reconcile says the profile unasked once the mesh last heard it 23 hours ago, so on a
machine nobody pushed to nothing the mesh holds of it is older than a day.
2026-10-12 17:21:36 +02:00
mesh-admin dd9ab7927d Merge pull request 'Genesis trusts its registry before it pushes to it (issue 92)' (#91) from fix/92-genesis-trusts-its-registry into main 2026-10-12 14:36:16 +00:00
jochen 0818e4c29a Hold the trust-before-push order by a test, and refuse early what would stop it (issue 92)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The fix is an order, and nothing failed if the trust call moved after the
push or went: steps 7 and 8 are now one function a test drives against a
runtime that refuses an untrusted push. Preflight now refuses, before
anything is raised, a runtime file that is no JSON object, a docker.service
that is not loaded and active, and a dockerd started with
--insecure-registry, which would refuse the reload. The comment names the
order the docker module's takeover rests on.
2026-10-12 16:29:14 +02:00
jochen 41e640d668 Have genesis trust its registry before pushing to it (issue 92)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Genesis pushed the control plane's image to the mesh's plain-HTTP registry
before anything told the container runtime to trust it; the docker module
that writes that trust is applied long after. Genesis now writes the trust
itself at the registry step, with the same into-json writer and a reload,
for the address given and the node's mesh name on the node's registry
port, and refuses before any push unless the runtime says it trusts each.
The docker module later finds the mesh name there and takes it over
without a change.
2026-10-12 16:13:11 +02:00
mesh-admin a1f9856b39 Merge pull request 'A package manager that does not answer fails only the packages; the rest of the declaration applies (issue 233)' (#90) from fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 into main 2026-10-12 14:11:11 +00:00
jochen 74ed7cbe0a Say a re-apply whose result changed, refuse whole when the package manager is not there, and install the packages in the apply that made it answer (issue 233 review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 merged: every member is merged
2026-10-12 15:39:07 +02:00
mesh-admin 36704b5471 Merge pull request 'Give a mesh directory its mode and owner only through a descriptor (issue 358)' (#87) from fix/358-descriptor-only-directory-mode into main 2026-10-12 13:13:13 +00:00
jochen 6e44a27a63 Fail only the packages when the package manager does not answer, and apply the rest, so a later declaration can restore what an earlier one removed (issue 233)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/233-a-failed-probe-fails-only-what-needs-the-tool-2 ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 15:12:42 +02:00
mesh-admin dce0eef1ca Merge pull request 'Judge a node's directories read-only before the walk refuses any (issue 358)' (#88) from fix/358-judge-directories-read-only into main 2026-10-12 12:48:55 +00:00
jschoubben e3c3bb37d0 Give a mesh directory its mode and owner only through a descriptor, and gate what the walk refuses (issue 358)
mesh/repo-check pass: its merge-check.sh passed
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/delivery delivered
A mesh directory outside every home was chmodded and chowned by path, so a
parent turned into a link between two applies carried root's chmod and chown
to the link's target. It is now made and changed through the descriptor of
the walk from / that follows a link only root alone could have made.

A directory that walk refuses is a refusal like the placement guard's
(review 47): judged before anything is applied, and again when the
directory is applied, used as found or not; refused at either point it gates
its module and refuses what is below it, so no file is written into the
link's target and the report does not say everything was attempted.
2026-10-12 14:00:53 +02:00
jschoubben b92fb68116 Judge a node's directories read-only before the walk refuses any (issue 358)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/358-judge-directories-read-only merged: every member is merged
Once a mesh directory's mode and owner go only through a walk that follows no
link another account could make, a directory behind such a link is refused
and its module gated: on the control-node that could stop the controller.
The operator is to be told which directories that walk would refuse on every
node before it lands. So the engine judges the declaration it keeps with the
same walk, making and changing nothing: on a signed ask from the controller
(mesh.node.<node>.ask.judge-directories, held to the hand-over's checks) and
as `nox-mesh-host judge-directories` at the machine.
2026-10-12 13:57:52 +02:00
mesh-admin 3f7f797168 Merge pull request 'A read-write access grants the path's own group, and the path is never changed (hq ADR 0312, issue 537)' (#86) from feat/537-a-read-write-access-grants-the-paths-group into main 2026-10-12 11:22:56 +00:00
jschoubben c433e76de2 Judge a granted group by number, grant nothing a group cannot write, and keep the look bounded (hq ADR 0312, issue 537 review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The engine is built without cgo, so a check by name missed a root group a name
service holds or one number carrying another name: the root groups are resolved
to numbers (now with kmem and shadow) and a number named nothing is refused for
containers too. A group without write is granted nothing (§2). The look reads
each path off the lock within five seconds, so a dead network mount cannot stop
every health statement. A grant's account may not start with a dash. A group a
record leaves for another asker is handed to it, so it is still taken back when
the last asker goes (§5). The scheduler skips paths issue 339's guard refuses.
2026-10-12 12:53:42 +02:00
jschoubben f4efc3cad0 Grant a read-write access's path group to what runs the module's code, so a module can write shared data without the mesh changing it (hq ADR 0312, issue 537)
The engine reads the group owning the accessed path at apply and gives it to the
containers and the tools' account an access's new `grant` names: --group-add for a
container (recreated with it), an account group for the account (recorded, taken
back as ADR 0252 takes back). The path is never changed; an access whose group
cannot write is stated unhealthy for the operator to decide; group 0 and the
root-granting groups are refused by name. The tool runner restarts through the
restart-on the controller gives it, because the access reports updated exactly
when the account's groups moved. Health says GrantContract (4), which tells the
controller it may send `grant`.
2026-10-12 12:23:38 +02:00
mesh-admin a76baa4b47 Merge pull request 'Carry the controller's grant to ask the delivery's owner for its groups in the installer's first user list (hq ADR 0306, issue 525)' (#85) from feat/525-grant-delivery-groups into main 2026-10-12 10:14:32 +00:00
jschoubben 90ce9e9d9f Carry the controller's grant to ask the delivery's owner for its groups in the installer's first user list (hq ADR 0306, issue 525)
mesh/delivery delivered
mesh/repo-check pass: its merge-check.sh passed
mesh/merge-gate pass: the change touches no module of the mesh's graph
The controller is to ask mesh-delivery whether a merge's delivery
group still has a member to merge, so that a merge in no waiting group
is cut at once (ADR 0304, ADR 0306); its bus grant names the verb, and
the installer's first user list must too, as for times (212a3cc).
2026-10-12 12:09:18 +02:00
mesh-admin df5de3bd9f Merge pull request 'Say an unhealthy resource's restart count as what it counts (issue 458)' (#84) from fix/458-restarts-say-what-they-count into main 2026-10-12 09:03:47 +00:00
jschoubben a18d09db75 Say an unhealthy resource's restart count as the engine's own since it began watching
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/458-restarts-say-what-they-count merged: every member is merged
Beside the state it read as restarts since then (novox/hq issue 458).
2026-10-12 10:47:07 +02:00
mesh-admin 3333a05a74 Merge pull request 'A refused directory gates its module, and a file meets the placement guard (issue 495)' (#83) from fix/495-a-refused-directory-gates-its-module into main 2026-10-11 17:00:22 +00:00
jschoubben 4695dfc0a2 Say both a gated module and a machine stop, and count skipped apart (review of #83)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A module-less action that stopped the apply after a module was gated was reported as
everything else attempted. The failure now names the gated modules and the stop, counts
skipped resources apart from applied ones, and the tests cover an archive below a refused
path, a gated module tried again by the next apply, and a refused file among the failures.
2026-10-11 18:51:04 +02:00
jschoubben 7233ddbe35 Gate a module behind its refused directory and judge a file by the placement guard (issue 495)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
On ace the engine refused Docker's data root as not placed, said nothing was touched, then
wrote a file into it and ran the module's step. A refused directory now gates the rest of
its module as a failed step does, a file is refused in a forbidden tree or below a refused
path, and the failure names the modules whose rest was not attempted.
2026-10-11 18:19:25 +02:00
jschoubben 1acfed4add Test that a refused directory gates its module and a file meets the same guard (issue 495)
Both fail on main: the shape seen on ace, a refused directory, a file below it and a
step after it, writes the file, runs the step and reports that everything was attempted.
2026-10-11 18:17:59 +02:00
mesh-admin 6defec8f68 Merge pull request 'Refuse a declaration composed from an older assignment generation than the machine applied (issue 234)' (#82) from fix/234-refuse-a-declaration-from-an-older-generation into main 2026-10-11 16:00:01 +00:00
mesh-admin 2833caf3d6 Merge pull request 'A failed run-once process runs again on every apply until it passes (hq issue 467)' (#81) from fix/467-a-failed-step-runs-again into main 2026-10-11 09:30:49 +00:00
jschoubben 20d5af9b2d Refuse a declaration composed from an older assignment generation than the machine applied (hq issue 234)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A newer sequence carrying an older view of the assignments undeclared four
assigned modules on 2026-10-04; the sequence cannot tell it, the generation
can. The engine keeps the highest generation it applied, refuses an older one
before reading the machine unless the declaration is a gate's put-back, and
says the refusal in its report so the controller can name the sender. A
declaration without a generation is applied as before, and every report says
it reads one, so the controller sends it only to engines that do.
2026-10-11 11:22:30 +02:00
jschoubben 0ec96a048e Clear a step's success only when it was started and failed, and say updated when it passes again (novox/hq issue 467)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A bundle that could not be fetched refused nothing; clearing for it would rerun
every step on a node once the artifact store came back from an outage.
2026-10-11 11:12:15 +02:00
jschoubben c957f1ec5e Run a failed run-once process again on every apply until it passes (novox/hq issue 467)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A failure recorded nothing, so the record of the step's last success stayed and
the next apply skipped it, opening ADR 0136's gate on what it had just refused.
The record is kept, for removal, with what it was run as cleared.
2026-10-11 11:07:17 +02:00
mesh-admin 85c51ebb59 Merge pull request 'A refused line's request is read before it is answered, so a busy line reads busy (issue 434)' (#80) from fix/434-a-busy-line-is-answered-busy-whatever-the-load into main 2026-10-11 00:55:47 +00:00
jschoubben f9805b2113 Bound what a refusal costs the engine, and let the busy tests fail instead of hang
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A refusal waiting for its request parsed it into a 64 KiB reader for up to 5 s,
on a socket every account may open. It now drains at most 4 KiB through a small
buffer for at most 1 s and never past the accept-time deadline, only 32 wait at
once, and an unknown account is refused before it takes a slot. The busy tests
wait for the first line with a timeout and say what came of it (novox/hq issue 434).
2026-10-11 02:47:11 +02:00
jschoubben 70cb613614 Read a refused line's request before answering it, so a busy line is read as busy
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The engine answered busy and closed before the client's request had landed;
the client's write then broke and it never read the refusal waiting for it.
Under the whole suite's load this failed the busy test (novox/hq issue 434).
2026-10-11 02:35:20 +02:00
mesh-admin bd5cc69804 Merge pull request 'The setuid walk paces itself by the machine's I/O pressure (issue 433)' (#79) from fix/433-the-setuid-walk-yields-to-a-busy-disk into main 2026-10-11 00:03:46 +00:00
jschoubben f311fa36dc The setuid walk is paced before each file too, so a large directory is not read blind (review of issue 433)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A file's own details (lstat) may be read from the disk as a directory is,
so a directory of tens of thousands of files was read without a look and
the header's promise that no burst is longer than PaceLook did not hold.
The walk now asks the pacer before each file as well; the 250 ms gate
makes that a clock read per file. A test with one directory of twenty
files fails without it. The clock is read once after a run of pauses,
and the constants test says that it guards the constants only.
2026-10-11 01:55:39 +02:00
jschoubben 30242197fa The paced walk keeps a tenth of the time, and reads the pressure since its last look (review of issue 433)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The review of #79 found that pacing alone brought back issue 361's
fault. The control-node's own I/O pressure sits near the 10 % bound with
no walk running, so a walk that paused 30 s for every 250 ms it read
would read 0.8 % of the time: a 16-minute walk would take some 32 hours,
outlive the age a search judges for, and be begun again at an engine
restart, leaving the machine unjudged.

A run of pauses is now at most nine times as long as the walk read since
the one before, so the walk keeps at least a tenth of the time whatever
the pressure, and takes at most ten times as long as unpaced: about
2.7 hours for the 16-minute walk, inside the 12 hours the controller
waits quietly. A test ties the bound to rootsearch.Quiet.

The pressure is the share of the time since the last look in which some
task waited on I/O, from PSI's cumulative counter, not its ten-second
average, which the kernel moves only every two seconds. A pause is
500 ms, then the pressure is looked at again.

The engine's walker is built in one place, engineWalker, and a test
fails if it is not paced by /proc/pressure/io. What the walk cannot read
is said through the search's own channel (the cache's Say), as the rest
of the search is, with what it means for the disk.
2026-10-11 01:53:25 +02:00
jschoubben 8f7db549ae The setuid walk paces itself by the machine's I/O pressure (issue 433)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The walk for setuid programs is put in the idle I/O class so it yields
to everything else, but only a scheduler that schedules by class (BFQ)
honours that. The control-node's disks run mq-deadline, which does not,
so the walk read at full priority. On 2026-10-11 the walk started at
01:27, right after the engine applied a new controller, and ran sixteen
minutes; it kept two spinning disks saturated, the bus whose store
shares them stalled for half a minute twice, and the controller lost
its lease and restarted each time.

The walk now also paces itself by PSI, which reads the same whatever
the scheduler: before it reads a directory, while some task has waited
on I/O for more than 10 % of the last ten seconds, it pauses two
seconds at a time. It looks at most every 250 ms, pauses at most 30 s
in a row and then reads one more directory, so a machine that is never
quiet is walked slowly but still judged. Its own reads count in the
pressure, so on an otherwise idle machine it holds the disk below the
bound. Pressure it cannot read pauses nothing, and is said once.

A paced walk finds what an unpaced one finds; a walk ended while it
pauses is an incomplete walk, never one that found nothing.
2026-10-11 01:47:43 +02:00
mesh-admin 5226d7bcd0 Merge pull request 'Node-engine: state a tool check that waits for the operator as waiting (issue 386, ADR 0283)' (#78) from feat/386-a-wait-for-the-operator into main 2026-10-10 20:03:27 +00:00
jochen 9545121ec9 Clear an old pass on a wait, and say a change of waits (review of #78)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A resource that passed, then waited, then failed once after its grace was
still stated healthy from the old pass; and a change of what it waits for
alone raised no change event, so the controller could miss it.
2026-10-10 21:14:14 +02:00
jochen 3abb07f1e2 State a tool check that waits for the operator as waiting (issue 386, ADR 0283)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
A module whose only fault was a secret the operator had not given yet was
stated unhealthy, failed its first-node gate and was put back. The engine
now reads the tool's answer through mesh-sdk go/health and states such a
resource waiting, with what it waits for, on one look; a malformed wait
stays unhealthy.
2026-10-10 21:11:02 +02:00
mesh-admin 9d8aa9d6f0 Merge pull request 'Carry the controller's grant to ask mesh-delivery for its delivery times in the installer's first user list (hq ADR 0282, issue 382)' (#77) from feat/382-carry-times-grant into main 2026-10-10 18:39:44 +00:00
jochen 212a3cc113 Carry the controller's grant to ask the delivery's owner for its delivery times in the installer's first user list (hq ADR 0282)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 20:32:16 +02:00
mesh-admin cc0b8554b1 Merge pull request 'Show how far the setuid search got, and start a fresh one when the terminal asks (hq issue 361)' (#76) from fix/361-the-setuid-search-shows-its-progress into main 2026-10-10 00:21:51 +00:00
jochen 0b4c86e3be Count a setuid find only where the account can execute it, read at every judgement (hq issue 361)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The two finds on the control-node sit in the operator's home of mode 0700, which the agent account cannot
search. The search keeps every find; each judgement reads the modes on the way again, since an account can
open its own home without root. A find in a home is said by count, its path in the journal alone. Also
from review: a checkpoint racing SearchNow forgets the walk it just kept, and a walk kept by an older engine
says its counts as lower bounds.
2026-10-10 02:00:10 +02:00
jochen 8e79403c21 Show how far the setuid search got, and start a fresh one when the terminal asks (hq issue 361)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestNatsAnEnrolmentWithNoReplyAddressIsNotAnswered (0.00s)
mesh/delivery superseded: a newer head of the same pull request
The walk kept its progress every minute and nobody could read it: root-free said only that a search ran.
The account verdict now carries the progress (counts, times, mount points; never a path) as fields and in
its reason, outside what makes a verdict changed. A signed ask from the controller's terminal throws the last
search away and starts a full one, after the operator removed by hand what it found.
2026-10-10 01:55:58 +02:00
mesh-admin b8c8546118 Merge pull request 'Let the setuid search run to completion, resumably, and judge from the last complete one (hq issue 361)' (#75) from fix/361-the-setuid-search-runs-to-completion into main 2026-10-09 23:18:44 +00:00
jochen a7270f00cb Keep the walk's paths as bytes, forget a walk root superseded, and leave other mounts unread (hq issue 361 review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/361-the-setuid-search-runs-to-completion merged: every member is merged
The review found a cursor of a name that is not UTF-8 came back changed
from JSON, so a resumed walk skipped what it stood before; an ask to
search again was lost to a restart, which carried the older walk on; a
fresh search judged a filesystem it never walked; a mount point root
cannot read failed or hung the walk; a found file gone before the package
manager was asked failed every resume; and skipped or kept outcomes
started a search on every reconcile, back to back with a long walk.
2026-10-10 01:10:56 +02:00
jochen 2dbbda51c6 Walk for setuid programs in the engine and carry an interrupted walk on (hq issue 361)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/361-the-setuid-search-runs-to-completion ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
The control-node's walk takes far longer than the engine runs between
updates, so a search that starts over at every restart never completes.
The walk is the engine's own, in lexical order, so its progress is one
cursor per filesystem, kept every minute beside the engine's state and
carried on after a restart while younger than rootsearch.FreshFor. It runs
on a thread of its own at nice 19 and the idle I/O class.
2026-10-10 01:00:47 +02:00
jochen 2410796869 Let the setuid search run to completion and judge from the last complete one (hq issue 361)
A machine with millions of files never finished the walk within fifteen
minutes, so its agent account was never judged and the router refused to
link. Only root can make a setuid-root file, so a complete search stays
sound until root acts: it now runs at idle priority with no bound, is kept
beside the engine's state, judges for rootsearch.FreshFor from its start,
runs again daily and after an apply that moves accounts, packages, actions
or sudo rules, and an incomplete or stale search is never free.
2026-10-10 00:53:05 +02:00
mesh-admin 090b9082eb Merge pull request 'Renew the engine placed by hand to the proved build, and take a hand-over from the controller's terminal (hq issue 356)' (#74) from fix/356-hand-over-by-nox-and-the-engine-on-path-renewed into main 2026-10-09 18:07:21 +00:00
jochen 8fc1d72cd5 Keep the setuid search's time on the caller's clock
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1791657606681868190 (novox/mesh-host at 9d8aa9d6 into main, do…
SetuidCache compared the caller's now with a time read from the wall clock when the search finished. When the caller's clock ran behind the wall clock, a search was never due again: TestTheSetuidSearchRunsInTheBackgroundAndIsKeptForItsInterval, whose clock is a fixed date, has failed on main and every branch since 2026-10-09 18:00 UTC. A search's time is now the now it was started at; a result reads at most one search older than it is, which errs towards searching again.
2026-10-09 20:04:11 +02:00
jochen 9a807de659 Give a handed-over directory through no link on every apply, not only the first (hq issue 356, review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheSetuidSearchRunsInTheBackgroundAndIsKeptForItsInterval (0.00s)
mesh/delivery superseded: a newer head of the same pull request
After the first apply the record is no longer used as found, so the directory read as the mesh's own and later applies set its mode and owner by path: a parent swapped for a link after the hand-over took carried root's chmod and chown to its target. The hand-over is now checked before that shortcut, so a handed-over directory always goes through applyHandedOver. The test swaps the parent and the grandparent after an apply already gave the directory its mode; both cases fail with the old order.
2026-10-09 19:58:27 +02:00
jochen fc6c5a92e6 Believe a hand-over only signed by the mesh, and give it through no link (hq issue 356, review)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Two holes the review proved. The ask's subject proved nothing — a responder can reach it through a reply — so the engine now verifies the mesh's signature over a fixed context with the key it verifies declarations with, before reading anything, and refuses an ask for another node, expired, good too long, without a nonce, or replayed. And the apply followed a link the hand-over had refused: a component swapped for a link after the record carried root's chmod and chown to its target. A handed-over directory must now still resolve to itself, and is changed only through one descriptor opened from / with no link followed (fchmod, fchown). Tests for the reply guard, the run loop's hook, the fallback renewal on proof and a non-regular fallback; 27 checks each removed in turn, each failing a test.
2026-10-09 19:45:00 +02:00
jochen e74ee82a76 Renew the engine placed by hand to the proved build, and take a hand-over from the controller's terminal (hq issue 356)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The file on PATH was the engine a person copied at installation, and every delivered engine ran from the versions directory: the operator typed hand-over and a build from before it answered "unexpected argument". A proved engine now copies itself over that file when the bytes differ (never a link). And the hand-over of a directory used as found (issue 339) gets its channel: the controller's terminal asks this node's engine on mesh.node.<node>.ask.hand-over; the engine judges every value — the ask's size and shape, who asked, the path as it states it, a directory and no link, through no link — and records it or refuses and records nothing. Every text the engine addresses to the operator names the nox line (ADR 0272); the engine's own subcommands are install-time and rescue, and its usage says so.
2026-10-09 18:34:10 +02:00
mesh-admin b0d03be7a4 Merge pull request 'mesh-cli: forward the line's standard input to the controller, with the line alone (hq ADR 0259 §10, ADR 0272)' (#73) from feat/a-terminal-line-takes-standard-input into main 2026-10-09 15:58:48 +00:00
jschoubben 4ea6a2f04e mesh-cli: forward the line's standard input to the controller, with the line alone
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/a-terminal-line-takes-standard-input closed: a member was closed before the group merged
A secret given at the controller's terminal through mesh-cli (secret accept … --from -) never reached the
line: the engine forwarded the words only. It now forwards what mesh-cli's standard input held (at most
64 KiB, the SDK's bound) on the same core request to the controller, says it nowhere, and sends none
with a follow of the same call. mesh-sdk moves to f047d0a.
2026-10-09 16:34:45 +02:00
mesh-admin d8ff1540df Merge pull request 'Installer's first user list: the controller's grant to ask the desk for a secret (hq ADR 0259 §10)' (#68) from feat/a-secret-given-at-the-desk into main 2026-10-09 14:17:41 +00:00
mesh-admin cdc2af7b43 Merge pull request 'Installer's first user list: the controller's grants to ask the operator (hq ADR 0259)' (#59) from feat/the-controller-asks-the-operator into main 2026-10-09 14:17:38 +00:00
mesh-admin 7a2ffc0a4c Merge pull request 'Read the session record strictly, and make its tests hold (hq ADR 0272)' (#72) from fix/272-the-session-record-read-strictly into main 2026-10-09 12:12:11 +00:00
jochen b1e7b6d26e Read the session record strictly, and make its tests hold (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
- UID, CLASS and SCOPE are read at their first occurrence: logind writes
  each once, and a later line saying otherwise is not the record's word.
- A comment says /run/systemd/sessions is systemd's private format, and
  that a format change fails closed: no session, so nobody is the terminal.
- Tests that hold, each proved by the mutation it guards failing:
  - a record naming UID=1001 for session-8.scope under user-1000.slice,
    asked as uid 1000 (the record's UID check alone);
  - session-a.b.scope with a passing record (the id's characters alone);
  - session-9.scope/x.scope (the scope being the slice's direct child);
  - a key said twice, either way round (first occurrence);
  - the client sends its line only after the session was read, so reading
    the session in the forward fails the test.
2026-10-09 14:05:53 +02:00
mesh-admin 512b0edb02 Merge pull request 'A login session is the account's own, of class user, read at accept (hq ADR 0272)' (#71) from feat/272-the-session-is-a-users-read-at-accept into main 2026-10-09 12:04:18 +00:00
jschoubben 4ba6d0864e Carry the controller's grant to ask the desk for a secret in the installer's first user list (hq ADR 0259 §10)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/a-secret-given-at-the-desk merged: every member is merged
mesh-controller #156 (give) grants the controller mesh.seat.node-launcher.tool.secret.*; the
controller's test holds the installer's list equal to what it composes, and failed without it.
2026-10-09 14:04:01 +02:00
jochen 73e0916968 Carry the controller's grants to ask the operator and act on a warrant in the installer's first user list, so a mesh raised from it is not refused them (hq ADR 0259)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-host@4ba6d0864e3d (merged as d8ff1540 into main, walk plan-17915554870866…
mesh/delivery-group group feat/the-controller-asks-the-operator merged: every member is merged
2026-10-09 14:03:58 +02:00
jochen bf3f45e769 A login session is the account's own, of class user, read at accept (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The cgroup names a scope; only logind's record of it says whose session it
is. The engine now takes a session only when /run/systemd/sessions/<id>
says it is this account's (UID), of class user (not a greeter's or the
user manager's), and of this very scope. And it reads the session the
moment the connection is accepted, with the namespaces, before the line,
so the peer is judged as it was when it connected.

Tests: session-9.scope with a user record is taken; x.scope, a different
uid, another account's session, a greeter's, a record naming another
scope, no record, and a path with dots are not; the session is read
before the line is forwarded.
2026-10-09 13:55:45 +02:00
mesh-admin fd13834c0d Merge pull request 'Say the login session a mesh-cli line comes from, and pin the SDK at go/v0.1.9 (hq ADR 0272)' (#70) from feat/272-a-login-session-and-the-tagged-sdk into main 2026-10-09 11:54:31 +00:00
jochen 4a5dcb0209 Say the login session a mesh-cli line comes from, and pin the SDK at go/v0.1.9 (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A service running as the operator — the tool runner, a unit of the
account's user manager — shares the operator's user id, and was as much the
terminal on the control-node as a person typing. The engine now reads the
asking process's cgroup and sends its login session's scope
(session-<id>.scope under the account's own user-<uid>.slice), or nothing;
the controller makes nothing else the terminal. The SDK is pinned at its
tag now that go/v0.1.9 exists.
2026-10-09 13:40:17 +02:00
mesh-admin 1ac7741c63 Merge pull request 'Vouch for the account that runs mesh-cli, and ask the controller its line (hq ADR 0272)' (#69) from feat/272-the-engine-vouches-for-mesh-cli into main 2026-10-09 11:38:39 +00:00
jochen f684967c3b Merge main, and close what the confirmation review found (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
- Any failure of a request the link may have delivered now says the line
  may have run; "nothing ran" is said only of a request the link knows it
  never asked (no link open, nobody listening: link.ErrNotAsked).
- A test holds that a connection saying nothing is answered within its
  deadline, set at accept, and gives its slot back.
- The README says the socket refuses a process outside the machine's own
  namespaces, so flatpak, toolbox and distrobox shells are refused.
2026-10-09 13:09:48 +02:00
jochen 655f515aa7 Merge remote-tracking branch 'origin/main' into feat/272-the-engine-vouches-for-mesh-cli 2026-10-09 13:09:16 +02:00
mesh-admin 8f763ee911 Merge pull request 'Judge whether an account declared never to become root can (hq ADR 0266)' (#61) from feat/an-agent-account-never-becomes-root into main 2026-10-09 10:56:00 +00:00
jochen 564948c2f2 mesh-cli: follow a line to its answer, bound each account, refuse other namespaces (review of hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
- A request that timed out was said as "the controller was not reached,
  nothing ran", though the line may have run. The engine now follows a line
  the controller answers as running, by its call, until it ends; when no
  answer comes in time it says the line may have run and names the call.
  "Nothing ran" is said only of a request no controller took.
- One account could hold the socket: accepting waited for a free slot, and
  a connection could stall before its deadline was set. The deadline is set
  at accept, before any lookup; each user id has two lines at once and the
  machine eight, and one over is answered busy at once.
- A container running with the operator's user id shares the number, not
  the machine: a peer whose mount or user namespace is not the engine's is
  refused, and nothing is asked.

SDK pinned at the go/cli branch's newest commit until go/v0.1.9 is tagged.
2026-10-09 12:48:20 +02:00
jochen 1c61b72f35 Read the homes in one order, so their test does not depend on a map's
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The homes come from a map of the user database, and the test compared them in
the order it happened to iterate: it failed one run in four, the repo check
of 2026-10-09 among them.
2026-10-09 12:36:56 +02:00
jochen 2673e7a2c9 Let the setuid search run to its 15-minute bound, ask pacman once, and call a walk with errors incomplete
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestHomesAreAPersonsOrAnAgentsNotAServicesOrRoots (0.00s)
mesh/delivery superseded: a newer head of the same pull request
The search kept a two-minute limit of its own inside its 15-minute bound, so
a long walk always ended incomplete; pacman was asked once per path; and find
exiting with an error but some output was read as a complete walk. The bound
and the not-judged reasons move to the public rootsearch package so the
controller reads the same values (the review of 2026-10-09).
2026-10-09 12:11:57 +02:00
jochen 021d860ca4 Vouch for the account that runs mesh-cli, and ask the controller its line (hq ADR 0272)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The records send the operator to "the controller's terminal", and nothing
reaches it (hq issue 343). mesh-cli holds no credential: it asks the engine
on its own machine over a socket. The engine reads the asking account from
the kernel (SO_PEERCRED), never from the program, and asks the controller on
mesh.control.<node>.cli, its own node's subject, so the bus server says which
node asked. The controller decides what the line may do; nothing received
on the socket is run or applied here, and the socket has no network address
(ADR 0004 stands).

The local protocol is the SDK's go/cli package, at a pseudo-version of its
branch until that merges and is tagged go/v0.1.8.
2026-10-09 12:05:53 +02:00
jochen b28d7bbcbf Search for setuid programs in the background, and judge each polkit rule alone
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A search over a large disk ran inside the look, holding the judge's lock and
stalling the health statement; it now runs apart, serving its last result,
backing off after a failure, and saying not judged until it has one. And a
rule naming a user, or a comment, no longer hides another rule's
unconditional yes (hq ADR 0266).
2026-10-09 10:10:05 +02:00
jochen e2e8be72fa Keep main's placement guard, and refuse only an archive or a written-into file below an agent's home
Issue 339's guard (#63/#64/#65) now refuses the machine's own trees, another account's
directory below a home, and uses a found directory as found; this branch's own copy of
those rules goes. What stays is the one rule only an account that never becomes root
needs: below its home it owns every parent and could swap a link in between the
engine's check and an unpack or a write-into (hq ADR 0266 §7). One reader of the user
database serves both.
2026-10-09 10:10:05 +02:00
jochen d4af8bba1b Judge an opened file's kind and links below a home, and more ways to root
A hard link swapped in for ~/.claude would have had root chown another
account's file; fstat on the descriptor now refuses a second link, a fifo or
an unexpected kind before anything is changed (hq ADR 0266, the re-review).
The judge also finds polkit rules for every account, a runtime's API on TCP,
setgid-to-root programs whoever owns them, setuid programs on every suid
filesystem, and unprotected links; the rest is listed as not judged.
2026-10-09 10:07:52 +02:00
jochen 02a1c5f753 Refuse to take over a system account as one that never becomes root
The controller cannot read a machine's user database, so a name it is given
for the agents' account (hq ADR 0266) may be a service's own; taking it over
would hand agents that service's files. Refuse it, touching nothing.
2026-10-09 10:07:52 +02:00
jochen b89b143fe7 Follow no link below a home as root, and judge more ways to root
An account could replace ~/.claude with a link to /etc and have the
node-engine chown, chmod or write through it on the next apply. Below a
person's or an agent's home every component is now opened without
following a link, and a link refuses the resource in words.

The root judge also reads doas and polkit rules, the container runtimes'
sockets with their ACLs, ACLs on the secrets, and setuid-root programs no
package owns, in the C locale; Judged and NotJudged write down exactly
what it covers (hq ADR 0266 review).
2026-10-09 10:07:52 +02:00
jochen 3a727011ad Judge whether an account declared never to become root can, so a machine's agents are known confined
An account an agent runs as (novox/hq ADR 0266) is read on every look for a
uid of 0, a group that grants root, any sudo rule and a mesh secret it can
read; any way found is unhealthy and said, a read that fails is unknown.
2026-10-09 10:07:35 +02:00
jochen d93cc86f95 Let a user say it never becomes root, so the agents' account can be judged
A declaration may now state root: never on a user (novox/hq ADR 0266), and a
health statement carries it under contract 3; the judging follows.
2026-10-09 10:07:35 +02:00
mesh-admin a390bce5f2 Merge pull request 'Give a service's held state its own field and key, so main compiles again' (#67) from fix/store-declared-redeclared into main 2026-10-09 01:09:40 +00:00
jochen c2bae14994 Give a service's held state its own field and key, so main compiles again
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
#64/#65 and #66 each added Applied.Declared: a directory's declared owner and
mode (string) and a service's held state (*FoundUnit), both under the JSON key
"declared". The directory's is already on nodes, written by d28ca95, so it
keeps the name and key; the service's becomes DeclaredUnit, "declared_unit".
#66 never deployed, so no node holds a record under the new key.
2026-10-09 03:03:58 +02:00
mesh-admin 690966d847 Merge pull request 'Start again on removal what the mesh's own declaration stopped (hq ADR 0269)' (#66) from feat/give-back-a-unit-the-mesh-stopped into main 2026-10-09 00:54:13 +00:00
jochen dfb14151bd Say in the plan that a unit is restored only if it is still as the mesh held it
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1791657606681868190 (novox/mesh-host at 9d8aa9d6 into main, do…
Confirmation pass of #66: the plan is a preview, and the removal reads the unit when it runs; a masked
unit or one changed since is left as it is.
2026-10-09 02:03:51 +02:00
jochen a018ebd1ca Pass -- before every unit, and give a unit back only when it is exactly as the mesh held it
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Third review of mesh-catalog #147.

A unit's name can now come from a setting (a template's instance), so every systemctl call that names a
unit passes `--` before it, and a name can never be read as an option.

On removal, a unit is started or enabled again only when it is exactly the stopped and disabled the mesh
held it to. A masked unit, or one somebody started or enabled since, is left as it is, and the report
says why. A restore that fails (the start refused, or the unit not running after it) puts back what this
attempt changed and is reported as waiting, so no apply is made fatal by it and the next one tries again.

The tests' fake service managers read systemctl's arguments in the shape they were written for
(oldShape); internal/system's own test holds the `--`.
2026-10-09 01:54:26 +02:00
jochen 9d48b8baf9 Start again on removal what the mesh's own declaration stopped (hq ADR 0269)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A unit found running and enabled that a declaration held stopped and disabled stayed stopped and
disabled when the declaration went: ADR 0118 never starts a unit on the way out. Unassigning the zfs
module would so have left the home server's pool with no scrub schedule at all, said nowhere.

The host now records what it declared for a service (Declared, beside Found). On removal, a unit found
running and declared stopped is started again, and one found enabled and declared disabled is enabled
again, each read back; the report says restored, and the plan says it first. A record from before the
field starts nothing.
2026-10-09 01:35:27 +02:00
341 changed files with 36879 additions and 8827 deletions
+54 -14
View File
@@ -4,11 +4,16 @@ Tier 0 of the Novox Mesh. The one thing ever installed by hand, and the only thi
a machine.
```
scp mesh-host root@machine:/usr/local/bin/
mesh-host profile
scp nox-mesh-host root@machine:/usr/bin/
nox-mesh-host profile
```
That is the whole installation. One statically linked binary, nothing else present, no runtime
That is the whole installation. **It is a daemon**: the launcher runs it, and the operator's command is
`nox` (`mesh-cli`), on any node, from their own account (`novox/hq` ADR 0272). The subcommands below are
for installing a machine, joining it, and rescuing one no mesh can reach, run as root at the machine; a
line the mesh addresses to the operator names a `nox` line, never this binary. Once a delivered engine
proves itself, it renews the one placed by hand here to its own build, so the command on the machine is
the engine that runs it (`novox/hq` issue 356). One statically linked binary, nothing else present, no runtime
to install first ([`novox/hq` ADR 0005](https://git.novox.be/novox/hq)).
## What it is for
@@ -23,7 +28,7 @@ the host never queries the mesh database. It receives declarations and applies t
## Joining a mesh
```
mesh-host enrol --token <token> --name <what this machine is called>
nox-mesh-host enrol --token <token> --name <what this machine is called>
```
**The node generates its own identity** — an Ed25519 keypair whose private half never leaves the
@@ -52,20 +57,22 @@ and then stops, having saved nothing — so it can be run again unchanged.
connects to nothing and listens on nothing — what it applies comes from a file.
```
mesh-host profile what this machine can be asked to do
mesh-host inventory what this machine is, and what it holds
mesh-host apply FILE make this machine match a declaration from a file
mesh-host reconcile make this machine match what the mesh last told it — or, before
any mesh has, the bundle this host carries
mesh-host bundle show what this host carries
mesh-host owned what this host has applied and still owns
nox-mesh-host profile what this machine can be asked to do
nox-mesh-host inventory what this machine is, and what it holds
nox-mesh-host apply FILE make this machine match a declaration from a file
nox-mesh-host reconcile make this machine match what the mesh last told it — or, before
any mesh has, the bundle this host carries
nox-mesh-host bundle show what this host carries
nox-mesh-host owned what this host has applied and still owns
nox-mesh-host hand-over DIR hand a directory used as found to the mesh, on a machine no mesh
can reach; the operator's line is `nox node hand-over <node> DIR`
--json machine-readable
--state where this node keeps what it knows
--dry-run say what applying would change, and change nothing
```
```
$ mesh-host profile
$ nox-mesh-host profile
linux/amd64
yes container-runtime 29.7.2
@@ -95,6 +102,19 @@ what it did not understand would apply most of a declaration and report success.
]}
```
**A directory or archive says how it treats files it did not write** (issue 1024,
[`novox/hq` ADR 0422](https://git.novox.be/novox/hq)). `unknown` is `move-out`, `module` or `program`;
absent, nothing about unknown paths is done. `knows` lists paths inside it, relative to its `path`, that
are not declared but belong there; a trailing `/` names a folder and everything in it. `knows` without
`unknown` is refused. For `move-out` the node-engine, after every apply and so on every reconcile, moves
each path inside the directory that nothing declared, no archive unpacked and `knows` does not name,
whole, into `<home>/.local/state/mesh-moved/<module>/<UTC time>/` beside a `<UTC time>.json` list. The
list names each move before it is made. Nothing is deleted. A path younger than a minute waits for the
next look; a socket, pipe or device is left and said. It is only for a directory under an account's
home on one file system with the kept folder, and otherwise the resource fails and nothing moves.
`module` and `program` are read by the module and the program that owns the path, and the node-engine
does nothing for them.
**It converges rather than executes.** Applying twice changes nothing the second time; applying
to a drifted machine returns it. A mode is *maintained*, not merely set — a permission applied
at creation is not a permission held.
@@ -115,7 +135,7 @@ A host built for a machine carries its declaration **inside the binary**:
make host BUNDLE=path/to/foundation.lock
```
`mesh-host reconcile` then applies it, on a machine the mesh has told nothing yet. That is the
`nox-mesh-host reconcile` then applies it, on a machine the mesh has told nothing yet. That is the
first node's path — no mesh present, nothing fetched, nothing else copied onto the machine. Once
the mesh has spoken, `reconcile` holds the machine to what it last said and never to the bundle,
which genesis consumed; a bundle or a file is refused when it says the other mode than the node
@@ -151,6 +171,26 @@ carries. The claim is not. The foundation is four container services, and:
So `foundation.lock` here is a real bundle with a placeholder's content. Saying that plainly
beats shipping a host that claims a foundation it has never raised.
## mesh-cli's socket (novox/hq ADR 0272)
The engine serves `/run/nox-mesh-host/cli.sock` for `mesh-cli`, the operator's command (`internal/meshcli`). It
reads the asking account from the kernel (`SO_PEERCRED`) and asks the controller on this node's own subject; the
controller decides what the line may do, and the engine runs nothing it receives. One line per connection, each
connection's deadline set the moment it is accepted, two lines at once per account and eight for the machine,
"busy" beyond that.
**Only a login session can be the terminal.** The engine reads the asking process's cgroup and sends its login
session's scope (`session-<id>.scope` under the account's own `user-<uid>.slice`, which logind's own record in
`/run/systemd/sessions/<id>` must say is this account's, of class `user`), or nothing, read the moment the
connection is accepted; the controller makes
nothing else the terminal. A service running as the operator — the tool runner, a unit of the account's user
manager — is at most an ordinary call.
**A process outside the machine's own namespaces is refused**, whatever its user id: the peer's mount and user
namespace must be the engine's. So `mesh-cli` run inside a flatpak, a toolbox or distrobox container, or any other
container — even one running as the operator's own id — is refused with the reason; run it from the machine's own
shell.
## A capability is detected, never assumed
The reason this is the first thing built rather than a detail of it.
@@ -217,7 +257,7 @@ without its neighbour checked out is a repository nobody can build.
```
mesh-controller: ./build/mesh-controller plan <node> --json > /tmp/d.json
mesh-host: MESH_EMITTED=/tmp/d.json go test ./internal/declaration/ -v
mesh-sdk: MESH_EMITTED=/tmp/d.json go test ./declaration/ -v (run in mesh-sdk's go/ directory)
```
**What this node says when it joins, read by the mesh:**
+6
View File
@@ -0,0 +1,6 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
+8 -4
View File
@@ -116,6 +116,8 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
--tools-source the repository the shared base is built from
--tools-ref what of it to build (default main)
--tools-go-source the repository the Go toolchain is built from (default: mesh-tools-go beside --tools-source)
--tools-go-ref what of it to build (default main)
--catalog-source the catalogue REPOSITORY, for building its modules;
--catalog is the checkout that says what they are
--catalog-ref what of it to build (default main)
@@ -130,7 +132,7 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
The foundation's ports are this machine's, each checked free before anything is
raised and kept as the node's setting for the module that binds it:
--store-port 5432 --bus-port 5671 --amqp-port 5672 --management-port 15672
--store-port 5432 --bus-port 5671
--registry-port 5000 (follows --registry, and must agree with it)
--packages-port 3000 --hub-port 51820/udp
--overlay-range the private network's range (default 10.42.0.0/16); refused
@@ -308,6 +310,10 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
"the repository the shared base is built from")
set.StringVar(&opts.ToolsSource.Ref, "tools-ref", opts.ToolsSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.ToolsGoSource.Repository, "tools-go-source", opts.ToolsGoSource.Repository,
"the repository the Go toolchain is built from (default: mesh-tools-go beside --tools-source)")
set.StringVar(&opts.ToolsGoSource.Ref, "tools-go-ref", opts.ToolsGoSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.CatalogSource.Repository, "catalog-source", opts.CatalogSource.Repository,
"the catalogue REPOSITORY, for building its modules — --catalog is the checkout that says what they are")
set.StringVar(&opts.CatalogSource.Ref, "catalog-ref", opts.CatalogSource.Ref,
@@ -325,9 +331,7 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
into *int
}{
{"store-port", "the store", &opts.Ports.Store},
{"bus-port", "the bus (amqps)", &opts.Ports.Bus},
{"amqp-port", "the broker's AMQP", &opts.Ports.AMQP},
{"management-port", "the broker's management, on loopback", &opts.Ports.Management},
{"bus-port", "the bus", &opts.Ports.Bus},
{"registry-port", "the registry", &opts.Ports.Registry},
{"packages-port", "the package registry", &opts.Ports.Packages},
{"hub-port", "the private network's hub (udp)", &opts.Ports.Hub},
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"context"
"testing"
"time"
)
// An apply runs in a context the bus's loss cannot end: the link's context ends when the bus goes, and the bus may go
// because of the apply itself (it replaces the bus's own container), which then must finish (issue 1945, live on
// 2026-10-25 at 11:23: novox's node-engine removed the bus container, its link's context ended, and the docker run that
// would have made the new one was never made).
func TestAnApplyOutlivesTheLinksContext(t *testing.T) {
type key struct{}
link, cancel := context.WithCancel(context.WithValue(context.Background(), key{}, "v"))
ctx, done := applyContext(link)
defer done()
cancel()
if ctx.Err() != nil {
t.Fatalf("the apply's context ended with the link's: %v", ctx.Err())
}
if ctx.Value(key{}) != "v" {
t.Fatal("the apply's context lost the link's values")
}
}
// An apply is bounded all the same: a step that hangs (a docker call that never returns) cannot hold the node agent for
// ever (the review of issue 1945).
func TestAnApplyIsBounded(t *testing.T) {
ctx, done := applyContext(context.Background())
defer done()
at, ok := ctx.Deadline()
if !ok {
t.Fatal("an apply has no bound")
}
if left := time.Until(at); left > applyBound || left < applyBound-time.Minute {
t.Fatalf("the bound is %s away, want about %s", left, applyBound)
}
}
+41
View File
@@ -0,0 +1,41 @@
package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// applyAndKeep applies a whole declaration the way the carried bundle is applied, and says what came of it as the facts
// the node says: what the tests of the apply's steps read. The node-engine applies no whole declaration on its own now
// (novox/hq ADR 0539 §9); this is the test's way to put a machine in a state.
func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.Declared, sched *apply.Scheduler,
say link.Announce) link.Report {
declared, err := declaration.Parse(raw)
if err != nil {
return link.Report{Refused: err.Error()}
}
got := applyModuleHeld(ctx, opts, declared, nil, sched, say)
if got.Refused != "" {
return link.Report{Refused: got.Refused}
}
held := map[string]bool{}
for _, h := range got.Updated.Held {
held[h.ID] = true
}
report := factsOf(ctx, opts, declared, got.Updated, got.Outcome, held)
if signed != nil {
if err := store.SaveDeclared(store.DeclaredPath(opts.state), *signed); err != nil {
report.Failed = map[string]string{"keeping the declaration": err.Error()}
}
}
if got.Err != nil {
report.Failed = map[string]string{"apply": got.Err.Error()}
}
report.Failed = withUnanswered(report.Failed, got.Probe)
report.Unapplied = unapplied(declared, got.Outcome, got.Err)
return report
}
+42
View File
@@ -0,0 +1,42 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
)
func backupEvent(node, module string, ok bool, at time.Time, why string) []byte {
raw, _ := json.Marshal(map[string]any{"node": node, "module": module, "ok": ok, "at": at, "error": why})
return raw
}
// A backup is done when the holder says so for this node and module, held on its words when it says it failed, and held
// when it says nothing within the bound; another node's or module's event, or one from before the ask, is not the answer
// (R6-A A5, novox/hq ADR 0429 §14).
func TestABackupEndsWhenItsEventSaysSo(t *testing.T) {
since := time.Now()
run := func(within time.Duration, events ...[]byte) (done bool, words string) {
ch := make(chan []byte, len(events))
for _, e := range events {
ch <- e
}
got := awaitBackup(context.Background(), ch, "n1", "db", since, within)
return got.Done, got.Words
}
if done, _ := run(time.Second,
backupEvent("n2", "db", true, since, ""), backupEvent("n1", "web", true, since, ""),
backupEvent("n1", "db", false, since.Add(-time.Hour), "an old night"),
backupEvent("n1", "db", true, since.Add(time.Second), "")); !done {
t.Error("the event of this node and module was not the answer")
}
if done, words := run(time.Second, backupEvent("n1", "db", false, since, "the repository is locked")); done ||
!strings.Contains(words, "repository is locked") {
t.Errorf("a failed backup: done=%v words=%q", done, words)
}
if done, words := run(50 * time.Millisecond); done || !strings.Contains(words, "did not say it was done") {
t.Errorf("silence within the bound: done=%v words=%q", done, words)
}
}
+29
View File
@@ -0,0 +1,29 @@
package main
import (
"encoding/json"
"testing"
)
// A module that declares nothing to back up on this machine is not backed up before it moves, and so never held for a
// backup that cannot run (issue 1945, live on 2026-10-25: redis and searxng on ace were held: "redis declares nothing to
// back up on this machine"). What node-backup.backed-up answers decides it.
func TestAModuleWithNothingToBackUpIsNotBackedUp(t *testing.T) {
for _, c := range []struct {
said string
nothing bool
}{
{`[{"module":"postgres","runs":1,"paths":["/var/lib/postgres"]}]`, true},
{`[{"module":"redis","runs":0,"paths":null,"data":[{"item":"cache"}]}]`, true},
{`[]`, true},
{`[{"module":"redis","runs":1,"paths":["/var/lib/redis"]}]`, false},
} {
nothing, ok := declaresNothingToBackUp(json.RawMessage(c.said), "redis")
if !ok || nothing != c.nothing {
t.Errorf("%s: nothing %v (read %v), want %v", c.said, nothing, ok, c.nothing)
}
}
if _, ok := declaresNothingToBackUp(json.RawMessage(`{"error":"x"}`), "redis"); ok {
t.Error("an answer that is not the list was read")
}
}
+34
View File
@@ -0,0 +1,34 @@
package main
import (
"context"
"sync/atomic"
"testing"
"time"
)
// While a container waits, a pass is asked for every so often, not at some long interval; with nothing waiting, none
// (novox/hq ADR 0335 §2, ADR 0360 §11).
func TestAWaitIsAskedAgainSoon(t *testing.T) {
if BusyAskEvery > time.Minute {
t.Fatalf("a waiting container is asked every %s", BusyAskEvery)
}
var asked atomic.Int32
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
busyWaiting.Store(false)
t.Cleanup(func() { busyWaiting.Store(false) })
go busyPasses(ctx, func() { asked.Add(1) }, 10*time.Millisecond)
time.Sleep(100 * time.Millisecond)
if n := asked.Load(); n != 0 {
t.Fatalf("%d passes were asked with nothing waiting", n)
}
busyWaiting.Store(true)
deadline := time.Now().Add(2 * time.Second)
for asked.Load() == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if asked.Load() == 0 {
t.Fatal("no pass was asked while a container waits")
}
}
+33
View File
@@ -0,0 +1,33 @@
package main
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A composition that names a file the provisioner fills without saying it is filled would have this node-engine write
// that file empty: its apply is refused, loudly, before anything is written (issue 1945, live on 2026-10-25 at 01:22,
// when ace, g14 and shanks wrote claude-code's, node-env's and power's files empty).
func TestAFileItCannotProduceIsNeverWrittenEmpty(t *testing.T) {
unit := json.RawMessage(`{"id":"power.after-boot-unit","type":"file","path":"/etc/systemd/system/mesh-power-after-boot.service","mode":"0644","owner-at-runtime":"owned"}`)
c := delivery.Composed{Node: "ace", Resources: []json.RawMessage{unit}}
d, err := declarationOf(c)
if err != nil {
t.Fatal(err)
}
if err := refuseEmptyWrites(d); err == nil || !strings.Contains(err.Error(), "power.after-boot-unit") {
t.Fatalf("a file the provisioner fills, not filled here: %v", err)
}
c.Filled = []string{"/etc/systemd/system/mesh-power-after-boot.service"}
if d, _ = declarationOf(c); refuseEmptyWrites(d) != nil {
t.Fatalf("a filled file was refused: %v", refuseEmptyWrites(d))
}
plain := delivery.Composed{Node: "ace", Resources: []json.RawMessage{json.RawMessage(
`{"id":"x.f","type":"file","path":"/etc/x","mode":"0644","content":"x"}`)}}
if d, _ = declarationOf(plain); refuseEmptyWrites(d) != nil {
t.Fatalf("a file with its content was refused: %v", refuseEmptyWrites(d))
}
}
-77
View File
@@ -1,77 +0,0 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
"path/filepath"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// **A declaration from an older lease epoch is refused before anything is applied** (novox/hq to-be
// 45 §6, ADR 0227 rule 2). The controller that sent it lost its lease and goes on sending; this node
// applied the holder's. Refused on the host's own apply path — the one the queue's worker runs — before
// the machine is read or touched, with a report naming what was refused and what is held, and the
// kept declaration left as it was.
func TestADeclarationFromAnOlderEpochIsRefusedBeforeAnythingIsApplied(t *testing.T) {
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
_, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "a.conf")
declared := func(epoch, sequence int64) store.Declared {
body, err := json.Marshal(map[string]any{"declaration": 1, "epoch": epoch, "sequence": sequence,
"resources": []any{map[string]any{"id": "a", "type": "file", "path": target, "content": "x\n"}}})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: ed25519.Sign(private, body)}
}
held := declared(57, 3)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), held); err != nil {
t.Fatal(err)
}
stale := declared(41, 12)
report := applyAndKeep(context.Background(), opts, stale.Declaration, &stale, nil, nil)
if report.OlderThan == nil || *report.OlderThan != (link.Order{Epoch: 57, Sequence: 3}) ||
report.Order != (link.Order{Epoch: 41, Sequence: 12}) || report.Declared != digestOf(stale.Declaration) ||
report.Refused == "" {
t.Fatalf("the refusal does not name what was refused and what is held: %+v", report)
}
if _, err := os.Stat(target); !errors.Is(err, os.ErrNotExist) {
t.Fatal("the refused declaration touched the machine")
}
kept, err := store.ReadDeclared(store.DeclaredPath(opts.state))
if err != nil || string(kept.Declaration) != string(held.Declaration) {
t.Fatal("the refused declaration replaced what this node kept")
}
}
// The queue's counts are kept beside the state and read back by the next host; unreadable is an
// error, never zero.
func TestTheNumbersSurviveTheHost(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
if sequence, refused, err := numbersBeside(state).Read(); err != nil || sequence != 0 || refused != 0 {
t.Fatalf("a node that never reported read %d, %d, %v", sequence, refused, err)
}
if err := numbersBeside(state).Save(41, 2); err != nil {
t.Fatal(err)
}
if sequence, refused, err := numbersBeside(state).Read(); err != nil || sequence != 41 || refused != 2 {
t.Fatalf("read back %d, %d, %v", sequence, refused, err)
}
if err := os.WriteFile(store.NumbersPath(state), []byte(`{"report_sequence":`), 0o600); err != nil {
t.Fatal(err)
}
if _, _, err := numbersBeside(state).Read(); err == nil {
t.Fatal("unreadable numbers were read as zero")
}
}
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/link"
)
// aFilesLink is a link that serves objects by name, or fails every read with err.
type aFilesLink struct {
objects map[string][]byte
err error
}
func (a aFilesLink) FileObject(_ context.Context, name string, _ int64) ([]byte, error) {
if a.err != nil {
return nil, a.err
}
if b, ok := a.objects[name]; ok {
return b, nil
}
return nil, os.ErrNotExist
}
func referenceDeclaration(path string, body []byte) []byte {
return []byte(`{"declaration":1,"resources":[{"id":"m.big","type":"file","path":"` + path + `","digest":"` +
declaration.DigestOf(body) + `","size":` + strconv.Itoa(len(body)) + `}]}`)
}
// 1695/AC1 and AC2, through the whole apply: a reference is fetched, verified and written; a differing object
// refuses the declaration whole and writes nothing.
func TestADeclarationWithAFileByDigestAppliesFromTheObject(t *testing.T) {
unansweringPacman(t)
body := []byte(strings.Repeat("a line of a big file\n", 4000))
name, _ := declaration.ObjectName(declaration.DigestOf(body))
dir := t.TempDir()
target := filepath.Join(dir, "big")
opts := options{state: filepath.Join(dir, "state.json"), out: &strings.Builder{}}
raw := referenceDeclaration(target, body)
bad := append([]byte(nil), body...)
bad[10] ^= 1
ctx := link.WithFiles(context.Background(), aFilesLink{objects: map[string][]byte{name: bad}})
report := applyAndKeep(ctx, opts, raw, nil, nil, nil)
if report.Refused == "" || !strings.Contains(report.Refused, "m.big") {
t.Errorf("1695/AC2: a differing object did not refuse the declaration: %+v", report)
}
if _, err := os.Stat(target); err == nil {
t.Error("1695/AC2: a file was written from an object that differs")
}
if failedWith(report.Refused).Transient {
t.Errorf("an object that differs is a real failure, and was taken as the bus not answering: %s", report.Refused)
}
// The bus answers no JetStream request while it reloads its accounts (2026-10-25 23:04:24, mesh-vault on the bus
// node): the failure is transient, so the agent tries it again, and the detail kept is its cause, the tail.
noResponders := fmt.Errorf("fetching the object 2bd98ebd of this node's own object bucket: reading the chunks of "+
"object %s: %w", strings.Repeat("x", 300), nats.ErrNoResponders)
ctx = link.WithFiles(context.Background(), aFilesLink{err: noResponders})
report = applyAndKeep(ctx, opts, raw, nil, nil, nil)
failed := failedWith("a file of this composition could not be had by its digest: file mesh-vault.kept: " + report.Refused)
if !failed.Transient || !strings.HasSuffix(failed.Detail, nats.ErrNoResponders.Error()) || len(failed.Detail) > 310 ||
!strings.HasPrefix(failed.Detail, "a file of this composition") {
t.Errorf("a fetch that met no responders: transient %v, detail %q (%d bytes); want transient, its head and its cause",
failed.Transient, failed.Detail, len(failed.Detail))
}
ctx = link.WithFiles(context.Background(), aFilesLink{objects: map[string][]byte{name: body}})
report = applyAndKeep(ctx, opts, raw, nil, nil, nil)
if report.Refused != "" {
t.Fatalf("1695/AC1: refused: %s", report.Refused)
}
got, err := os.ReadFile(target)
if err != nil || string(got) != string(body) {
t.Errorf("1695/AC1: the file holds %d bytes (%v), want %d", len(got), err, len(body))
}
}
// 1695/AC8: the hand-run `apply <file>` and `reconcile` have no bus: a reference whose file here verifies is applied
// as it is, any other refuses and writes nothing — never an empty file.
func TestTheHandApplyNeverWritesAReferenceEmpty(t *testing.T) {
unansweringPacman(t)
body := []byte(strings.Repeat("a line of a big file\n", 4000))
dir := t.TempDir()
target := filepath.Join(dir, "big")
opts := options{state: filepath.Join(dir, "state.json"), out: &strings.Builder{}}
raw := referenceDeclaration(target, body)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
if err := runApply(context.Background(), opts, d, raw, fromFile); err == nil || !strings.Contains(err.Error(), "digest") {
t.Errorf("1695/AC8: a reference with no file and no bus was not refused: %v", err)
}
if _, err := os.Stat(target); err == nil {
t.Error("1695/AC8: a file was written for a reference that could not be had")
}
if err := os.WriteFile(target, body, 0o644); err != nil {
t.Fatal(err)
}
d, _ = declaration.ParseFileTrusted(raw)
if err := runApply(context.Background(), opts, d, raw, fromFile); err != nil && !strings.Contains(err.Error(), "applied only in part") {
t.Fatalf("1695/AC8: %v", err)
}
if got, _ := os.ReadFile(target); string(got) != string(body) {
t.Errorf("1695/AC8: the file holds %d bytes, want %d", len(got), len(body))
}
}
+95
View File
@@ -0,0 +1,95 @@
package main
import (
"context"
"crypto/ed25519"
"errors"
"fmt"
"path/filepath"
"sync/atomic"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// serveFilling writes filled files on the provisioner client's request for as long as this engine serves (novox/hq
// ADR 0349 §4). A socket that cannot be served is said once, and this engine goes on applying.
func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, say func(string)) {
e := fillingEngine(opts, signer, say)
fillingNow.Store(e)
defer fillingNow.CompareAndSwap(e, nil)
if err := filling.Serve(ctx, filling.Socket, e, say); err != nil {
say("filled files are not written on request on this machine: " + err.Error())
}
}
// fillingNow is the filling this engine serves, for knockFilling; nil while none is served.
var fillingNow atomic.Pointer[filling.Engine]
// knockFilling tells the provisioner's client to fill again what the filling refused or failed and the declaration
// applied now may let through (filling.Engine.Knock), once a module's apply or removal settled (agent.Env.Settled).
// What it told, or why it could not, is said.
func knockFilling(module string, say func(string)) {
e := fillingNow.Load()
if e == nil {
return
}
told, err := e.Knock(filling.Socket)
switch {
case err != nil:
say(fmt.Sprintf("after %s settled: %v", module, err))
case told != "":
say(fmt.Sprintf("after %s settled, the provisioner's client is told to fill again: %s", module, told))
}
}
// fillingEngine is the write of a filled file as this engine does it: the declaration it keeps, verified, read under
// the apply's lock — the same two locks every apply takes, the process's and the one beside the state — and the file
// written through the apply (apply.WriteFilled), its outcome recorded in the node's state as an apply's is.
func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *filling.Engine {
return &filling.Engine{
Declared: func() (*declaration.Declaration, error) {
return declaredNow(opts.state, signer)
},
UIDOf: filling.UIDOf,
Lock: func() (func(), error) {
applying.Lock()
unlock, err := store.Lock(opts.state, nil)
if err != nil {
applying.Unlock()
return nil, err
}
return func() { unlock(); applying.Unlock() }, nil
},
Place: func(ctx context.Context, d *declaration.Declaration, path string, content []byte) (string, string, error) {
built, err := system.For(builtFor)
if err != nil {
return "", "", err
}
known, err := store.Load(opts.state)
if err != nil {
return "", "", err
}
report, updated, writeErr := apply.WriteFilled(ctx, built, d, known, path, content, apply.ExecRunner,
announceOr(say), apply.KeepIn(filepath.Dir(opts.state)), apply.WindowsIn(filepath.Dir(opts.state)))
// Saved whichever way it went, as an apply's: what was written is on the machine either way.
if err := store.Save(opts.state, updated); err != nil {
return "", "", errors.Join(writeErr, errors.New("written, and the node's state could not be saved: "+
err.Error()))
}
action, digest := "", ""
for _, o := range report.Outcomes {
if o.Target == path && o.Type == string(declaration.TypeFile) {
action = o.Action
}
}
if rec, ok := updated.At(string(declaration.TypeFile), path); ok {
digest = rec.Wrote
}
return action, digest, writeErr
},
}
}
+105
View File
@@ -0,0 +1,105 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"os"
"os/user"
"path/filepath"
"testing"
"time"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §4 (issue 668): a write of a filled file and an apply are one at a time — the write
// takes the lock the apply holds, and writes only once the apply let it go — and the write is recorded in the
// node's state as an apply's outcome is.
func TestAWriteAndAnApplyOfTheSameFileSerialise(t *testing.T) {
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
me, err := user.Current()
if err != nil {
t.Fatal(err)
}
if me.Uid == "0" {
t.Skip("run as root, whose account is never the provisioner's")
}
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(dir, "m.conf")
raw, err := json.Marshal(map[string]any{"declaration": 1, "provisioner_account": me.Username,
"filled_files": []string{path}, "resources": []any{map[string]any{"id": "m.conf", "type": "file",
"path": path, "content": "from the declaration\n", "owner-at-runtime": "owned", "owner": me.Username}}})
if err != nil {
t.Fatal(err)
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state),
store.Declared{Declaration: raw, Signature: ed25519.Sign(private, raw)}); err != nil {
t.Fatal(err)
}
e := fillingEngine(opts, public, func(string) {})
uid := uint32(os.Getuid())
if err := e.Accepts(uid); err != nil {
t.Fatal(err)
}
// An apply holds the lock.
applying.Lock()
done := make(chan filling.Answer)
go func() {
done <- e.Write(context.Background(), uid, filling.Request{Path: path, Content: []byte("filled\n")})
}()
select {
case a := <-done:
applying.Unlock()
t.Fatalf("written while an apply held the lock: %+v", a)
case <-time.After(200 * time.Millisecond):
}
if _, err := os.Stat(path); err == nil {
applying.Unlock()
t.Fatal("the file was written while an apply held the lock")
}
applying.Unlock()
a := <-done
if a.Refused != "" || a.Failed != "" || a.Action != "created" {
t.Fatalf("the write after the apply: %+v", a)
}
got, err := os.ReadFile(path)
if err != nil || string(got) != "filled\n" {
t.Fatalf("the file holds %q (%v)", got, err)
}
known, err := store.Load(opts.state)
if err != nil {
t.Fatal(err)
}
if rec, ok := known.Find("m.conf"); !ok || rec.Wrote != a.Digest || rec.Target != path {
t.Errorf("the write is not recorded in the node's state: %+v, answered %q", rec, a.Digest)
}
// And the cross-process lock beside the state too: a write waits for a `reconcile` run by hand.
unlock, err := store.Lock(opts.state, nil)
if err != nil {
t.Fatal(err)
}
go func() {
done <- e.Write(context.Background(), uid, filling.Request{Path: path, Content: []byte("again\n")})
}()
select {
case a := <-done:
unlock()
t.Fatalf("written while another process held the apply's lock: %+v", a)
case <-time.After(200 * time.Millisecond):
}
unlock()
if a := <-done; a.Action != "updated" {
t.Fatalf("the second write: %+v", a)
}
}
+95 -31
View File
@@ -22,14 +22,20 @@ import (
// directory it never made to whichever account a declaration named. That is safe and, said nowhere, a module
// whose data directory it cannot write would simply fail. So each such directory is said three ways: a WARNING
// line in every apply's log; a resource of its module of kind `directory`, stated unhealthy on every look,
// which the controller raises as that module's condition naming the directory and its owner; and a way out at
// the machine's terminal, `mesh-host hand-over <path>`, after which the next apply gives it the declared owner
// and mode like any directory of the mesh's.
// which the controller raises as that module's condition naming the directory and its owner; and a way out,
// which is the operator's line at the controller's terminal — `nox node hand-over <node> <path>` on the
// control-node (novox/hq ADR 0272, issue 356) — after which the next apply gives it the declared owner and
// mode like any directory of the mesh's. The controller carries that line to this engine over the bus
// (link.AskHandOverSubject); the engine's own `hand-over` subcommand is the same act for a machine with no
// mesh to reach.
// foundState is where the node's state is, for the looks to read the directories used as found from; empty
// until the engine runs.
var foundState atomic.Value
// foundNode is this node's name in the mesh, for the way out to name it; empty until the engine is enrolled.
var foundNode atomic.Value
// handedOverFrom has the apply read the directories handed over from beside the state at statePath.
func handedOverFrom(statePath string) {
apply.HandedOver = func() map[string]bool {
@@ -58,9 +64,21 @@ func foundDirectories(known store.State) []store.Applied {
return out
}
// handOverLine is the operator's line that hands a directory to the mesh, with this node's name when it is
// known (novox/hq ADR 0272): typed on the control-node, from the operator's own account. The directory's
// path is the resource's target beside the reason, never in it: the controller says a reason on the
// operator's channel, which withholds a path (ADR 0234 §6), and keeps the target as evidence.
func handOverLine() string {
node, _ := foundNode.Load().(string)
if node == "" {
node = "<node>"
}
return fmt.Sprintf("`nox node hand-over %s <directory>` on the control-node", node)
}
// withFoundDirectories adds to a statement every directory used as found, as its module's resource of kind
// directory, unhealthy: the reason names its owner and mode as they are now, what was declared, and how to hand
// it over. The path is the target, which the controller keeps as evidence.
// directory, unhealthy: the reason names its owner and mode as they are now, what was declared, and the line
// that hands it over. The path is the target, which the controller keeps as evidence.
func withFoundDirectories(h *link.Health, found []store.Applied) *link.Health {
for _, r := range found {
at := strings.LastIndex(r.ID, ".")
@@ -80,9 +98,8 @@ func withFoundDirectories(h *link.Health, found []store.Applied) *link.Health {
}
h.Resources = append(h.Resources, link.ResourceHealth{Module: r.ID[:at], Resource: r.ID,
Kind: link.KindDirectory, Target: r.Target, State: link.StateUnhealthy,
Reason: fmt.Sprintf("%s owned by %s, mode %s, as found; %s was declared and not given it — "+
"`mesh-host hand-over` at the machine hands it to the mesh", link.ReasonUsedAsFound, owner, mode,
declared),
Reason: fmt.Sprintf("%s owned by %s, mode %s, as found; %s was declared and not given it — %s, with its "+
"path, hands it to the mesh", link.ReasonUsedAsFound, owner, mode, declared, handOverLine()),
Since: foundSince(r).UTC()})
}
return h
@@ -102,34 +119,81 @@ func foundNow() []store.Applied {
return foundDirectories(known)
}
// handOverCommand hands one directory used as found to the mesh: run by a person, as root, at the machine.
// It changes nothing on disk itself; the next apply gives the directory its declared owner and mode, and says so.
func handOverCommand(opts options) error {
path := filepath.Clean(opts.file)
// handOverFound hands one directory used as found to the mesh, on the operator's word carried by the
// controller (link.HandOverFunc) or typed at this machine. It changes nothing on disk itself; the next apply
// gives the directory its declared owner and mode, and says so.
//
// **Every value is judged, and a refusal records nothing.** The path is the directory's absolute path exactly
// as the engine states it — clean, so `..` and a doubled separator never name a directory; one this machine's
// state records as used as found; on disk a directory and not a link, reached through no link. The engine
// never follows a path it did not state: a hand-over is root giving a directory to an account at the next
// apply, and a path that resolves elsewhere would give that elsewhere.
func handOverFound(statePath, path, by string) (string, error) {
if !filepath.IsAbs(path) {
return fmt.Errorf("hand-over takes the directory's absolute path, and %q is not one", opts.file)
return "", fmt.Errorf("hand-over takes the directory's absolute path, and %q is not one; nothing was handed over", path)
}
known, err := store.Load(opts.state)
if filepath.Clean(path) != path {
return "", fmt.Errorf("%q is not the directory's path as the engine states it (no `..`, no doubled or trailing "+
"separator); nothing was handed over", path)
}
known, err := store.Load(statePath)
if err != nil {
return "", err
}
var found *store.Applied
for _, r := range foundDirectories(known) {
if filepath.Clean(r.Target) == path {
r := r
found = &r
break
}
}
if found == nil {
return "", errors.New(path + " is not a directory this machine uses as found (the module's condition names " +
"the ones that are); nothing was handed over")
}
info, err := os.Lstat(path)
switch {
case err != nil:
return "", fmt.Errorf("%s cannot be read on this machine: %v; nothing was handed over", path, err)
case info.Mode()&os.ModeSymlink != 0:
return "", fmt.Errorf("%s is a link, and the mesh hands over no link (novox/hq ADR 0012); nothing was handed over", path)
case !info.IsDir():
return "", fmt.Errorf("%s is not a directory on this machine; nothing was handed over", path)
}
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return "", fmt.Errorf("%s cannot be resolved on this machine: %v; nothing was handed over", path, err)
}
if resolved != path {
return "", fmt.Errorf("%s is reached through a link (it is %s), and the mesh hands over nothing through one "+
"(novox/hq issue 339); nothing was handed over", path, resolved)
}
if err := store.AddHandedOver(statePath, store.HandOver{ID: found.ID, Path: path, At: time.Now().UTC(), By: by}); err != nil {
return "", err
}
return fmt.Sprintf("%s (%s) is handed to the mesh by %s: the next apply gives it %s, and it is the mesh's from "+
"then on — removed when empty and no longer declared", path, found.ID, by, orUnsaid(found.Declared)), nil
}
// handOverFrom is the engine's hand-over for the controller's ask: the state beside statePath, the path as asked.
func handOverFrom(statePath string) link.HandOverFunc {
return func(path, by string) (string, error) { return handOverFound(statePath, path, by) }
}
// handOverCommand is `nox-mesh-host hand-over <dir>` at this machine, as root: the same act, for a machine
// with no mesh to reach. The operator's line is `nox node hand-over <node> <dir>` on the control-node.
func handOverCommand(opts options) error {
by := "a person at the machine"
if u, err := user.Current(); err == nil {
by = u.Username + " at the machine"
}
said, err := handOverFound(opts.state, filepath.Clean(opts.file), by)
if err != nil {
return err
}
for _, r := range foundDirectories(known) {
if filepath.Clean(r.Target) != path {
continue
}
by := "a person at the machine"
if u, err := user.Current(); err == nil {
by = u.Username
}
if err := store.AddHandedOver(opts.state, store.HandOver{ID: r.ID, Path: path, At: time.Now().UTC(), By: by}); err != nil {
return err
}
fmt.Fprintf(opts.out, "%s (%s) is handed to the mesh: the next apply gives it %s, and it is the mesh's from "+
"then on — removed when empty and no longer declared\n", path, r.ID, orUnsaid(r.Declared))
return nil
}
return errors.New(path + " is not a directory this machine uses as found (`mesh-host owned` lists what the " +
"engine applied); nothing was handed over")
fmt.Fprintln(opts.out, said)
return nil
}
// foundSince is when a directory was first found as it is: kept across applies, so the controller can tell one
+117 -6
View File
@@ -14,13 +14,16 @@ import (
)
// Every directory used as found is its module's resource of kind directory, unhealthy, naming its owner, on
// every statement (novox/hq issue 339): the controller raises that as the module's condition.
// every statement (novox/hq issue 339): the controller raises that as the module's condition. The way out it
// names is the operator's line, with this node's name (novox/hq ADR 0272, issue 356).
func TestAFoundDirectoryIsStatedUnhealthy(t *testing.T) {
dir := t.TempDir()
known := store.State{Resources: []store.Applied{
{ID: "notes.data", Type: "directory", Target: dir, AsFound: true, Declared: "root, mode 755", AppliedAt: time.Now()},
{ID: "notes.cache", Type: "directory", Target: dir},
}}
foundNode.Store("laptop")
t.Cleanup(func() { foundNode.Store("") })
h := withFoundDirectories(&link.Health{}, foundDirectories(known))
if len(h.Resources) != 1 {
t.Fatalf("stated: %+v", h.Resources)
@@ -28,16 +31,45 @@ func TestAFoundDirectoryIsStatedUnhealthy(t *testing.T) {
r := h.Resources[0]
if r.Module != "notes" || r.Kind != link.KindDirectory || r.State != link.StateUnhealthy || r.Target != dir ||
!strings.HasPrefix(r.Reason, link.ReasonUsedAsFound) || !strings.Contains(r.Reason, "owned by ") ||
!strings.Contains(r.Reason, "root, mode 755") || !strings.Contains(r.Reason, "hand-over") {
!strings.Contains(r.Reason, "root, mode 755") {
t.Fatalf("stated as: %+v", r)
}
// The path is the target, kept as evidence, and never in the reason: the operator's channel withholds a path
// (novox/hq ADR 0234 §6), and the controller says the reason there.
if !strings.Contains(r.Reason, "`nox node hand-over laptop <directory>` on the control-node") ||
strings.Contains(r.Reason, "mesh-host") || strings.Contains(r.Reason, dir) {
t.Fatalf("the way out is not the operator's line, or carries the path: %q", r.Reason)
}
foundNode.Store("")
if r := withFoundDirectories(&link.Health{}, foundDirectories(known)).Resources[0]; !strings.Contains(r.Reason,
"`nox node hand-over <node> <directory>`") {
t.Fatalf("before enrolment the line names a placeholder: %q", r.Reason)
}
}
// `mesh-host hand-over` records a directory used as found, and refuses one that is not.
// realDir is a temporary directory with every link in its path resolved, so a judgement of links is about the
// links the test makes.
func realDir(t *testing.T) string {
t.Helper()
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
return dir
}
// A hand-over records a directory used as found, and refuses one that is not — through the engine's own
// subcommand and through the controller's ask alike, which are one judgement.
func TestHandOverRecordsOnlyAFoundDirectory(t *testing.T) {
dir := t.TempDir()
dir := realDir(t)
statePath := filepath.Join(dir, "state.json")
found := filepath.Join(dir, "found")
if err := os.Mkdir(found, 0o700); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(filepath.Join(dir, "mine"), 0o755); err != nil {
t.Fatal(err)
}
if err := store.Save(statePath, store.State{Resources: []store.Applied{
{ID: "notes.data", Type: "directory", Target: found, AsFound: true, Declared: "root, mode 755"},
{ID: "notes.mine", Type: "directory", Target: filepath.Join(dir, "mine")},
@@ -52,7 +84,7 @@ func TestHandOverRecordsOnlyAFoundDirectory(t *testing.T) {
t.Fatal(err)
}
all, err := store.LoadHandedOver(statePath)
if err != nil || len(all) != 1 || all[0].Path != found {
if err != nil || len(all) != 1 || all[0].Path != found || !strings.HasSuffix(all[0].By, " at the machine") {
t.Fatalf("recorded: %+v %v", all, err)
}
handedOverFrom(statePath)
@@ -62,5 +94,84 @@ func TestHandOverRecordsOnlyAFoundDirectory(t *testing.T) {
if !strings.Contains(out.String(), "root, mode 755") {
t.Fatalf("said: %s", out.String())
}
_ = os.Remove(statePath)
// The controller's ask is the same judgement, recording who asked in the controller's words.
_ = os.Remove(store.HandedOverPath(statePath))
said, err := handOverFrom(statePath)(found, "jo through mesh-cli on anchor")
if err != nil || !strings.Contains(said, "by jo through mesh-cli on anchor") || !strings.Contains(said, "root, mode 755") {
t.Fatalf("the ask answered %q, %v", said, err)
}
all, _ = store.LoadHandedOver(statePath)
if len(all) != 1 || all[0].By != "jo through mesh-cli on anchor" || all[0].ID != "notes.data" {
t.Fatalf("recorded: %+v", all)
}
}
// Every value of a hand-over is judged before anything is recorded, and each refusal records nothing: a relative
// path, a path that is not the engine's statement of the directory (`..`, a doubled or trailing separator), one
// not used as found, a link at the path, a link in the path, a file, a directory that is gone.
func TestAHandOverIsRefusedForEveryPathThatIsNotTheFoundDirectory(t *testing.T) {
dir := realDir(t)
statePath := filepath.Join(dir, "state.json")
found := filepath.Join(dir, "found")
if err := os.Mkdir(found, 0o700); err != nil {
t.Fatal(err)
}
// A link at the path of a second found directory, and a found directory below a linked directory.
linked := filepath.Join(dir, "linked")
if err := os.Symlink(found, linked); err != nil {
t.Fatal(err)
}
through := filepath.Join(dir, "via", "data")
if err := os.Symlink(dir, filepath.Join(dir, "via")); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(filepath.Join(dir, "data"), 0o700); err != nil {
t.Fatal(err)
}
file := filepath.Join(dir, "file")
if err := os.WriteFile(file, nil, 0o600); err != nil {
t.Fatal(err)
}
gone := filepath.Join(dir, "gone")
if err := store.Save(statePath, store.State{Resources: []store.Applied{
{ID: "notes.data", Type: "directory", Target: found, AsFound: true},
{ID: "notes.linked", Type: "directory", Target: linked, AsFound: true},
{ID: "notes.through", Type: "directory", Target: through, AsFound: true},
{ID: "notes.file", Type: "directory", Target: file, AsFound: true},
{ID: "notes.gone", Type: "directory", Target: gone, AsFound: true},
}}); err != nil {
t.Fatal(err)
}
// Each refused by its own check, which the reason names.
refused := map[string][2]string{
"a relative path": {"found", "not one"},
"a path with ..": {dir + "/x/../found", "as the engine states it"},
"a doubled separator": {dir + "//found", "as the engine states it"},
"a trailing separator": {found + "/", "as the engine states it"},
"a directory not used as found": {filepath.Join(dir, "other"), "not a directory this machine uses as found"},
"a link at the path": {linked, "is a link"},
"a link in the path": {through, "reached through a link"},
"a file": {file, "is not a directory on this machine"},
"a directory that is gone": {gone, "cannot be read on this machine"},
}
for name, c := range refused {
said, err := handOverFound(statePath, c[0], "jo")
if err == nil || said != "" {
t.Fatalf("%s (%s) was handed over: %q", name, c[0], said)
}
if !strings.Contains(err.Error(), "nothing was handed over") || !strings.Contains(err.Error(), c[1]) {
t.Fatalf("%s's refusal does not say %q and that nothing was handed over: %v", name, c[1], err)
}
}
if all, err := store.LoadHandedOver(statePath); err != nil || len(all) != 0 {
t.Fatalf("a refusal recorded something: %+v %v", all, err)
}
// The found directory itself, stated exactly, is taken.
if _, err := handOverFound(statePath, found, "jo"); err != nil {
t.Fatal(err)
}
if all, _ := store.LoadHandedOver(statePath); len(all) != 1 || all[0].Path != found {
t.Fatalf("recorded: %+v", all)
}
}
+90
View File
@@ -0,0 +1,90 @@
package main
import (
"context"
"errors"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
)
// owedBus is a link that takes reports and statements and keeps a have bucket in memory.
type owedBus struct {
mu sync.Mutex
kv map[string][]byte
rv map[string]uint64
}
func (b *owedBus) Report(context.Context, string, []byte) error { return nil }
func (b *owedBus) Health(context.Context, string, []byte) error { return nil }
func (b *owedBus) HaveBind(context.Context) (link.HaveBucket, error) { return b, nil }
func (b *owedBus) Get(_ context.Context, key string) ([]byte, uint64, error) {
b.mu.Lock()
defer b.mu.Unlock()
v, ok := b.kv[key]
if !ok {
return nil, 0, link.ErrHaveAbsent
}
return v, b.rv[key], nil
}
func (b *owedBus) Set(_ context.Context, key string, value []byte, revision uint64) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.rv[key] != revision {
return errors.New("wrong last sequence")
}
b.kv[key], b.rv[key] = value, revision+1
return nil
}
func (b *owedBus) have(key string) (delivery.Have, bool) {
b.mu.Lock()
defer b.mu.Unlock()
h, err := delivery.DecodeHave(b.kv[key])
return h, err == nil
}
// A module with nothing long-running (xorg) is ok within one look of its apply, not at the five-minute statement: the
// facts look inside the apply runs while it is applying and judges nothing, and the running write then owes a look,
// which the health loop makes at once (review of fix 15, B1).
func TestAModuleWithNothingLongRunningIsOkWithinALookOfItsApply(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
bus := &owedBus{kv: map[string][]byte{}, rv: map[string]uint64{}}
queue := &link.Session{Membership: link.Membership{Node: "n1"}, Timeout: time.Second}
queue.Attach(ctx, bus)
j, err := liveness.Open(filepath.Join(t.TempDir(), "liveness.json"), nil)
if err != nil {
t.Fatal(err)
}
fact := link.ModuleFact{Module: "xorg", FP: "src2:" + strings.Repeat("a", 64), Composed: "sha256:" + strings.Repeat("b", 64)}
applying := fact
applying.Phase = string(delivery.PhaseApplying)
queue.NoteModule(ctx, applying)
go judgeWhatRuns(ctx, j, queue, func(string) {}, nil)
// The facts look inside the apply, while the module is still applying.
queue.SayFacts(ctx, link.Report{Health: &link.Health{Contract: link.LivenessContract, At: time.Now().UTC()}})
if h, _ := bus.have("n1.xorg"); h.Health != delivery.HealthUnknown {
t.Fatalf("judged while applying: %+v", h)
}
queue.NoteModule(ctx, fact)
deadline := time.Now().Add(2 * time.Second) // well within one look (liveness.LookEvery, 15 s)
for {
h, _ := bus.have("n1.xorg")
if h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
return
}
if time.Now().After(deadline) {
t.Fatalf("not ok within a look of its apply: %+v", h)
}
time.Sleep(10 * time.Millisecond)
}
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"errors"
"fmt"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// judgeDirectories judges every directory of the declaration this node keeps, read-only (novox/hq issue 358): which
// the walk that follows no link another account could make would refuse. Nothing is made, chmodded or chowned.
func judgeDirectories(statePath string) (string, error) {
d, err := declaredNow(statePath, nil)
if errors.Is(err, store.ErrNothingDeclared) {
return "", errors.New("this node keeps no declaration, so it is declared no directory")
}
if err != nil {
return "", fmt.Errorf("what runs on this node does not read: %w", err)
}
return apply.JudgeDirectories(d), nil
}
// judgeDirectoriesFrom is judgeDirectories as the controller asks it over the link.
func judgeDirectoriesFrom(statePath string) link.JudgeDirectoriesFunc {
return func(string) (string, error) { return judgeDirectories(statePath) }
}
// judgeDirectoriesCommand is `nox-mesh-host judge-directories` at this machine, as root: the same judgement, for a
// machine the controller cannot ask.
func judgeDirectoriesCommand(opts options) error {
said, err := judgeDirectories(opts.state)
if err != nil {
return err
}
fmt.Fprintln(opts.out, said)
return nil
}
+285
View File
@@ -0,0 +1,285 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"sync"
"sync/atomic"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// A library is judged on every look (novox/hq ADR 0328 §8, §9).
//
// Each library this machine is reached by is said as a resource `library:<name>` of kind library, with no module:
// healthy while it is kept (or, off its home, while its mount is there), waiting with an access wait of reason keep
// or disk, unknown while its filesystem could not be named, unhealthy otherwise. The root is the target, never in a
// reason nor in a wait's words: the controller says them on the operator's channel, which withholds a path (ADR
// 0234 §6). An access is judged by nothing, as before ADR 0325.
// libraryJudge reads every library on every look. Safe for the apply and the looking loop at once.
type libraryJudge struct {
mu sync.Mutex
libraries []declaration.Library
declared *declaration.Declaration
// was is each library's last verdict, with since when and how many looks in a row found it unhealthy.
was map[string]link.ResourceHealth
// reading guards inFlight: every library whose look has not returned yet.
reading sync.Mutex
inFlight map[string]bool
}
// accesses is the one library judge on this machine (its name from when it judged accesses).
var accesses = newLibraryJudge()
func newLibraryJudge() *libraryJudge {
return &libraryJudge{was: map[string]link.ResourceHealth{}, inFlight: map[string]bool{}}
}
// set is what the declaration just applied declares: every library. One no longer declared is forgotten.
func (j *libraryJudge) set(d *declaration.Declaration, _ map[string]bool) {
var ls []declaration.Library
if d != nil {
ls = append(ls, d.Libraries...)
}
sort.Slice(ls, func(a, b int) bool { return ls[a].Library < ls[b].Library })
j.mu.Lock()
defer j.mu.Unlock()
j.libraries, j.declared = ls, d
declared := map[string]bool{}
for _, l := range ls {
declared[apply.LibraryAccess(l.Library)] = true
}
for id := range j.was {
if !declared[id] {
delete(j.was, id)
}
}
}
// declaration is the declaration last set.
func (j *libraryJudge) declaration() *declaration.Declaration {
j.mu.Lock()
defer j.mu.Unlock()
return j.declared
}
// lookBound is how long a look waits for one library to answer. A hard network mount whose server is down blocks
// a stat for ever, and one library must never stop every health statement this machine makes.
var lookBound = 5 * time.Second
// verdict is one library's judgement in a look.
type verdict struct {
state, reason string
waits []link.AccessWait
act *link.AccessActRecord
}
// judgeBounded judges one library within lookBound, outside any lock. A look that has not answered is said so;
// while it still hangs, later looks start no other on the same library.
func (j *libraryJudge) judgeBounded(l declaration.Library) (verdict, bool) {
j.reading.Lock()
if j.inFlight[l.Library] {
j.reading.Unlock()
return verdict{}, false
}
j.inFlight[l.Library] = true
j.reading.Unlock()
done := make(chan verdict, 1)
go func() {
v := judgeLibrary(l)
j.reading.Lock()
delete(j.inFlight, l.Library)
j.reading.Unlock()
done <- v
}()
select {
case v := <-done:
return v, true
case <-time.After(lookBound):
return verdict{}, false
}
}
// judgeLibrary is one library's judgement now (ADR 0328 §7, §8, §9).
func judgeLibrary(l declaration.Library) verdict {
v := apply.JudgeLibrary(context.Background(), l)
access := apply.LibraryAccess(l.Library)
var out verdict
if a := v.Act; a != nil {
out.act = &link.AccessActRecord{Act: a.Act, Access: access, Warrant: a.Warrant, By: a.By, State: a.State,
Started: a.Started, Ended: a.Ended, Folders: a.Folders, Files: a.Files, Kept: a.Kept, Links: a.Links,
Other: a.Other, Mounts: a.Mounts, Shared: a.Shared, Linked: a.Linked, Stripped: a.Stripped, Why: a.Why}
}
switch v.Kind {
case apply.VerdictWaitKeep:
out.state, out.reason = link.StateWaiting, "its root holds data the mesh has not taken on the filesystem "+
"it is on now, and waits for the operator's word (novox/hq ADR 0328 §7, §8)"
out.waits = []link.AccessWait{{Kind: link.WaitKindAccess, Access: access, Part: l.Library,
What: apply.KeepWords(l.Library), Placed: v.Placed, Reason: link.WaitKeep}}
case apply.VerdictWaitDisk:
out.state, out.reason = link.StateWaiting, v.Why+" (novox/hq ADR 0328 §9)"
out.waits = []link.AccessWait{{Kind: link.WaitKindAccess, Access: access, Part: l.Library,
What: apply.DiskWords(l.Library), Reason: link.WaitDisk}}
case apply.VerdictUnknown:
// Not a fault and not the operator's: said as not known, and judged again on the next look.
out.state, out.reason = link.StateUnknown, v.Why+"; judged again on the next look"
case apply.VerdictBroken:
out.state, out.reason = link.StateUnhealthy, v.Why
default:
out.state = link.StateHealthy
switch {
case v.Act != nil && v.Act.State == link.ActRunning:
out.state, out.reason = link.StateStarting, "the take act is setting its group under its root"
case v.Act != nil && (v.Act.State == link.ActStopped || v.Act.State == link.ActFailed):
out.state, out.reason = link.StateStarting, "the take act goes on from where it stopped: "+v.Act.Why
}
}
return out
}
// look reads every library now. Whether any verdict changed since the last look, for the loop to say it.
func (j *libraryJudge) look(now time.Time) ([]link.ResourceHealth, bool) {
j.mu.Lock()
ls := append([]declaration.Library(nil), j.libraries...)
j.mu.Unlock()
read := make([]verdict, len(ls))
for i, l := range ls {
v, answered := j.judgeBounded(l)
if !answered {
v = verdict{state: link.StateUnknown, reason: "its root does not answer: judging it did not return " +
"within " + lookBound.String()}
}
read[i] = v
}
// And every tool runner's account the last apply could not put in a group, as a resource of its own (ADR 0334
// §2): unhealthy, never failing the apply.
warnings := apply.RunnerWarnings()
j.mu.Lock()
defer j.mu.Unlock()
changed := false
out := make([]link.ResourceHealth, 0, len(ls)+len(warnings))
for _, w := range warnings {
v := link.ResourceHealth{Module: w.Module, Resource: w.Resource, Kind: link.KindRunner, State: link.StateUnhealthy,
Reason: "the tool runner's account joins no group: " + w.Why, Since: now.UTC(), Streak: 1}
if was, had := j.was[w.Resource]; had && was.Reason == v.Reason {
v.Since, v.Streak = was.Since, was.Streak+1
} else {
changed = true
}
j.was[w.Resource] = v
out = append(out, v)
}
for i, l := range ls {
r := read[i]
id := apply.LibraryAccess(l.Library)
v := link.ResourceHealth{Resource: id, Kind: link.KindLibrary, Target: l.Root, State: r.state,
Reason: r.reason, Since: now.UTC(), AccessWaits: r.waits, Act: r.act}
if was, had := j.was[id]; had && was.State == r.state && was.Reason == r.reason {
v.Since = was.Since
v.Streak = was.Streak
} else {
changed = true
}
if r.state == link.StateUnhealthy {
v.Streak++
}
j.was[id] = v
out = append(out, v)
}
return out, changed
}
// withAccesses adds to a statement every library, as a resource of kind library.
func withAccesses(h *link.Health, vs []link.ResourceHealth) *link.Health {
h.Resources = append(h.Resources, vs...)
return h
}
// accessesHealthy says no library in vs is anything but healthy.
func accessesHealthy(vs []link.ResourceHealth) bool {
for _, v := range vs {
if v.State != link.StateHealthy {
return false
}
}
return true
}
// keeper goes on with take acts stopped at their bound, beside the applies, one at a time (ADR 0328 §7).
type keeper struct {
running atomic.Bool
say func(string)
}
// keeping is the one keeper on this machine.
var keeping = &keeper{say: func(string) {}}
// poke goes on with every stopped take act of the declaration last applied, unless that is running already.
func (k *keeper) poke(ctx context.Context) {
d := accesses.declaration()
if d == nil || len(d.Libraries) == 0 || !k.running.CompareAndSwap(false, true) {
return
}
go func() {
defer k.running.Store(false)
for _, line := range apply.ReconcileLibraries(ctx, d) {
k.say(line)
}
}()
}
// startKeeping has the keeper say what it does.
func startKeeping(ctx context.Context, say func(string)) {
keeping.say = say
keeping.poke(ctx)
}
// accessActFrom is the take act as the controller asks it over the link (ADR 0328 §7): judged against the
// declaration this node keeps, and run beside the applies.
func accessActFrom(statePath string, timeout time.Duration) link.AccessActFunc {
return func(act link.AccessAct) (string, error) {
d, err := declaredNow(statePath, nil)
if errors.Is(err, store.ErrNothingDeclared) {
return "", errors.New("this node keeps no declaration, so it is declared no library")
}
if err != nil {
return "", fmt.Errorf("what runs on this node does not read: %w", err)
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
_, said, err := apply.TakeLibrary(ctx, d, act.Access, filepath.Clean(act.Path), act.Warrant, act.By)
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: the take act on %s was refused: %v\n", act.Access, err)
return "", err
}
return said, nil
}
}
// withWaiting is held with every id of waiting beside it: what liveness does not judge.
func withWaiting(held, waiting map[string]bool) map[string]bool {
if len(waiting) == 0 {
return held
}
out := map[string]bool{}
for id := range held {
out[id] = true
}
for id := range waiting {
out[id] = true
}
return out
}
+56
View File
@@ -0,0 +1,56 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
)
// A library whose root holds data never taken is said waiting with one access wait: the access `library:<name>`, the
// part its name, words with no path, the filesystem's name, and reason keep (novox/hq ADR 0328 §8); one whose root
// is absent waits with reason disk (§9). Neither carries a module.
func TestALibrarySaysItsWaitWithoutWordsToParse(t *testing.T) {
was := apply.Libraries
apply.Libraries = apply.NewLibraryMemory()
defer func() { apply.Libraries = was }()
dir := t.TempDir()
root := filepath.Join(dir, "media")
if err := os.MkdirAll(filepath.Join(root, "a film"), 0o755); err != nil {
t.Fatal(err)
}
d, err := declaration.Parse([]byte(`{"declaration":1,"groups":[{"name":"mesh-media","gid":62000},` +
`{"name":"mesh-books","gid":62001}],"libraries":[` +
`{"library":"media","root":"` + root + `","group":"mesh-media","gid":62000,"home":true},` +
`{"library":"books","root":"` + filepath.Join(dir, "books") + `","group":"mesh-books","gid":62001,"home":true}],` +
`"owns_nothing":true,"resources":[]}`))
if err != nil {
t.Fatal(err)
}
j := newLibraryJudge()
j.set(d, nil)
vs, _ := j.look(time.Now())
if len(vs) != 2 {
t.Fatalf("the libraries are not said: %+v", vs)
}
books, media := vs[0], vs[1]
if media.Kind != link.KindLibrary || media.Resource != "library:media" || media.Module != "" ||
media.State != link.StateWaiting || len(media.AccessWaits) != 1 {
t.Fatalf("a library holding data is not said waiting: %+v", media)
}
raw, _ := json.Marshal(media.AccessWaits[0])
want := `{"kind":"access","access":"library:media","part":"media","what":"let the mesh keep the library media",` +
`"placed":"` + media.AccessWaits[0].Placed + `","reason":"keep"}`
if string(raw) != want || !strings.HasPrefix(media.AccessWaits[0].Placed, "fs-") {
t.Fatalf("the wait reads %s", raw)
}
if strings.Contains(media.Reason, dir) || len(books.AccessWaits) != 1 || books.AccessWaits[0].Reason != link.WaitDisk {
t.Fatalf("a missing root does not wait for its disk, or a reason names a path: %+v %+v", books, media)
}
}
+457 -538
View File
File diff suppressed because it is too large Load Diff
+5 -296
View File
@@ -3,7 +3,6 @@ package main
import (
"bytes"
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
@@ -12,10 +11,8 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/bundle"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
@@ -98,26 +95,6 @@ func TestNoCommandIsNotAnError(t *testing.T) {
}
}
func TestApplyNeedsExactlyOneDeclaration(t *testing.T) {
// `apply` takes a file where every other command takes nothing, so the leftover-argument
// rule has an exception — and an exception is where a parser stops refusing things it
// should. Both directions are checked.
if _, _, err := parseArgs([]string{"apply"}); err == nil {
t.Error("apply with no file was accepted")
}
if _, _, err := parseArgs([]string{"apply", "a.json", "b.json"}); err == nil {
t.Error("apply with two files was accepted")
}
command, opts, err := parseArgs([]string{"apply", "decl.json", "--dry-run"})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if command != "apply" || opts.file != "decl.json" || !opts.dryRun {
t.Errorf("parsed as command=%q file=%q dry-run=%v", command, opts.file, opts.dryRun)
}
}
func TestTheStateHasADocumentedDefault(t *testing.T) {
// A host that wrote its state somewhere unexpected would forget what it owns on the next
// run, and then leave everything it had applied behind forever.
@@ -135,93 +112,22 @@ func TestAFlagAfterAPositionalIsRead(t *testing.T) {
// off the front fixed the flag after the COMMAND and not the flag after its ARGUMENT: the
// standard library stops at the first non-flag argument wherever that argument is.
for _, args := range [][]string{
{"apply", "decl.json", "--dry-run", "--json"},
{"apply", "--dry-run", "decl.json", "--json"},
{"apply", "--dry-run", "--json", "decl.json"},
{"hand-over", "decl.json", "--dry-run", "--json"},
{"hand-over", "--dry-run", "decl.json", "--json"},
{"hand-over", "--dry-run", "--json", "decl.json"},
} {
command, opts, err := parseArgs(args)
if err != nil {
t.Errorf("%v: unexpected error: %v", args, err)
continue
}
if command != "apply" || opts.file != "decl.json" || !opts.dryRun || !opts.json {
if command != "hand-over" || opts.file != "decl.json" || !opts.dryRun || !opts.json {
t.Errorf("%v parsed as file=%q dry-run=%v json=%v",
args, opts.file, opts.dryRun, opts.json)
}
}
}
// Defends novox/hq ADR 0100: a reconcile on an adopted node speaks unasked only when what it holds
// or its firewall changed — which is how a predecessor still writing is caught, without a report
// every five minutes saying nothing new.
func TestAReconcileSpeaksOnlyWhenWhatIsHeldChanged(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page"}, {ID: "hello-web.server"}}}
if !w.changed(held) {
t.Fatal("the first report of a hold was not said")
}
again := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.server"}, {ID: "hello-web.page"}}}
if w.changed(again) {
t.Error("the same holds in another order were said again")
}
rewritten := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}, {ID: "hello-web.server"}}}
if !w.changed(rewritten) {
t.Error("a held file rewritten by something else was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a changed firewall was not said")
}
// What filters the machine is part of it (novox/hq ADR 0168): a predecessor's chain removed by
// hand, or the found firewall enabled again, is said without being asked.
filtered := link.Report{Firewall: "none", Held: rewritten.Held,
Filters: []link.Filter{{Where: "chain HAL-MESH-ONLY (iptables-legacy)", Owner: "other", Refuses: "-j DROP"}}}
if !w.changed(filtered) {
t.Error("a filter appearing was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a filter removed was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held, FoundFirewall: &link.FoundFirewall{Kind: "ufw", Active: true}}) {
t.Error("the found firewall coming back was not said")
}
if !worthSaying(link.Report{Filters: filtered.Filters}) {
t.Error("a report carrying only what filters the machine is not worth saying")
}
}
// What the mesh heard — a delivery's account as much as a reconcile's — counts as said: the queue
// tells the watch every report the broker took (link.Queue.Heard).
func TestWhatTheMeshHeardCountsAsSaid(t *testing.T) {
w := &adoptionWatch{}
report := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "a"}}}
heard := w.said
heard(report)
if w.changed(report) {
t.Error("a reconcile repeated what the mesh had just heard")
}
}
func TestAReconcileSpeaksWhenWhatIsReachableChanged(t *testing.T) {
// The controller previews a flip from what the node last said is reachable; a port that opened
// since must reach it without waiting for the next delivery (novox/hq ADR 0100).
w := &adoptionWatch{}
before := link.Report{Firewall: "ufw", Reachable: []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}}
if !w.changed(before) {
t.Fatal("the first report was not said")
}
reordered := link.Report{Firewall: "ufw", Reachable: []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}}
if w.changed(reordered) {
t.Error("the same reachable set was said again")
}
opened := link.Report{Firewall: "ufw", Reachable: append(before.Reachable,
link.Reach{Protocol: "tcp", Address: "0.0.0.0", Port: 8080, By: "hello-web", Published: true, ContainerPort: 80})}
if !w.changed(opened) {
t.Error("a newly published port was not said")
}
}
// Defends the node's own record: the link and the reconcile loop both apply, and each reads the
// state, acts, and writes it back — so they must not run at the same time, or the last save loses
// what the other recorded.
@@ -270,24 +176,6 @@ func TestOnlyOneApplyRunsAtATime(t *testing.T) {
}
}
// Defends novox/hq ADR 0100: a change is counted as said only once the mesh has been told. Queued
// and lost — the link down when the reconcile spoke — it must be said again.
func TestAChangeThatNeverReachedTheMeshIsSaidAgain(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}}}
if !w.differs(held) {
t.Fatal("the first report of a change was not new")
}
// The link was down: nothing published it, so nothing says it was said.
if !w.differs(held) {
t.Error("a change that never reached the mesh was counted as said")
}
w.said(held)
if w.differs(held) {
t.Error("a change the mesh was told was said again")
}
}
// Defends novox/hq issue 104: a declaration for the other mode than this node is in is refused at
// the point of application, whichever command delivered it, naming both — an adopted control-node
// once applied its converged genesis bundle and closed itself for forty-five minutes.
@@ -468,125 +356,6 @@ func TestADryRunChangesNothingAndListsTheActions(t *testing.T) {
}
}
// Defends novox/hq issue 104: once the mesh has told this node anything, `reconcile` holds it to
// that — never to the bundle the host carries, which genesis consumed.
func TestReconcileAfterAControllerDeclarationDoesNotReapplyTheBundle(t *testing.T) {
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
opts := stateWithMode(t, store.ModeAdopted)
controllerPublic, controllerPrivate, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
said := []byte(`{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), store.Declared{Declaration: said,
Signature: ed25519.Sign(controllerPrivate, said)}); err != nil {
t.Fatal(err)
}
// Told, and unable to prove by whom: refused, and the bundle is not applied in its place.
_, _, _, err = reconcileSource(opts)
if err == nil || !strings.Contains(err.Error(), "not applied in its place") {
t.Errorf("a node that cannot prove what it was told fell back to something: %v", err)
}
mine, err := identity.Generate("workstation")
if err != nil {
t.Fatal(err)
}
mine.Membership = identity.Membership{Broker: "198.51.100.10:5671", Fingerprint: "sha256:0",
Signer: controllerPublic, Password: "issued"}
if err := identity.Save(identity.Path(opts.state), mine); err != nil {
t.Fatal(err)
}
d, raw, from, err := reconcileSource(opts)
if err != nil {
t.Fatal(err)
}
if from != fromDeclared || !bytes.Equal(raw, said) || d.Adoption == nil {
t.Errorf("reconcile chose %d with %d bytes, not what the mesh last said", from, len(raw))
}
// And before the mesh has said anything: the bundle, as it always was. A test binary carries
// only the placeholder, and asking for it is what proves the path.
if err := os.Remove(store.DeclaredPath(opts.state)); err != nil {
t.Fatal(err)
}
_, _, _, err = reconcileSource(opts)
if !errors.Is(err, bundle.ErrEmpty) {
t.Errorf("with nothing said, reconcile did not reach for the carried bundle: %v", err)
}
}
// **A machine says which links face outside without being asked.**
//
// The mesh composes no filter for a machine that has not said (novox/hq ADR 0140), and a machine only
// speaks unasked when this fingerprint changes. Left out of it, a machine that has just learnt to say
// could speak only when a declaration arrived — and a declaration cannot be composed until it has
// spoken. A machine waiting for a push that is waiting for the machine.
func TestANewOutwardLinkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
first := link.Report{Firewall: "none"}
if !w.differs(first) {
t.Fatal("the first report should differ from nothing")
}
w.said(first)
// Only the links changed, and nothing about adoption.
learnt := link.Report{Firewall: "none", Outward: []string{"eth0"}}
if !w.differs(learnt) {
t.Fatal("a machine that has just learnt which links face outside would never say so, " +
"and could then never be sent a filter")
}
w.said(learnt)
if w.differs(link.Report{Firewall: "none", Outward: []string{"eth0"}}) {
t.Fatal("the same links are reported as a change, so the machine would speak on every reconcile")
}
// And a link that changes — a laptop moving from a cable to a radio — is said too, because the
// filter is written around the old one until it is.
if !w.differs(link.Report{Firewall: "none", Outward: []string{"wlan0"}}) {
t.Fatal("a changed outward link is not said, so the filter stays written around the old one")
}
}
// **A converged machine can say which links face outside, unasked.**
//
// A reconcile is otherwise silent, and the condition deciding when it speaks asked only what an
// adopted node reports — what it holds, and the firewall it found. A converged node has neither, so
// it could speak only in reply to a declaration, and the mesh composes no declaration for a node
// that has not said which links face outside (novox/hq ADR 0140). Measured: three converged machines
// sat silent while the control plane refused to send them a filter.
func TestAConvergedMachineSaysItsOutwardLinksUnasked(t *testing.T) {
// A converged node's reconcile: nothing held, no found firewall, and the links it can see.
converged := link.Report{Outward: []string{"eth0"}}
if !worthSaying(converged) {
t.Fatal("a converged machine cannot say which links face outside, so it can never be " +
"sent a filter — a machine waiting for a push that is waiting for the machine")
}
// An adopted node's reasons still hold, because that is how a predecessor still writing is caught.
if !worthSaying(link.Report{Firewall: "ufw"}) {
t.Fatal("an adopted machine no longer says which firewall it found")
}
if !worthSaying(link.Report{Held: []link.Held{{ID: "a-file"}}}) {
t.Fatal("an adopted machine no longer says what it holds")
}
// And a reconcile with nothing to say stays silent, or every machine speaks every five minutes
// about nothing.
if worthSaying(link.Report{}) {
t.Fatal("a reconcile with nothing to say speaks anyway")
}
// A refused report says nothing about the machine; the refusal is for the console.
if worthSaying(link.Report{Outward: []string{"eth0"}, Refused: "not for this node"}) {
t.Fatal("a refused report is offered as news about the machine")
}
}
// **A host running as a service says what its apply did.**
//
// The serving path passed nil where the apply writes its detail, and nil is silence. The one-shot path
@@ -611,63 +380,3 @@ func TestTheApplysLogIsNeverNil(t *testing.T) {
t.Fatalf("the apply's detail did not reach the caller's announce: %v", said)
}
}
// Defends novox/hq issues 257 and 261: a reconcile due while the link applies a newer declaration
// applies that newer one once its turn comes — never the one kept when its timer fired. Read before
// waiting, it gave back a module assigned a second earlier, and reported a declaration the mesh no
// longer recorded as sent.
func TestAReconcileAppliesWhatWasKeptWhenItsTurnComes(t *testing.T) {
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "a.conf")
kept := func(content string) store.Declared {
body := []byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"` + target +
`","content":"` + content + `\n"}]}`)
return store.Declared{Declaration: body, Signature: ed25519.Sign(private, body)}
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state), kept("older")); err != nil {
t.Fatal(err)
}
// The link is applying: the reconcile's timer fires and it waits its turn.
applying.Lock()
done := make(chan error, 1)
go func() {
_, err := reapplyKept(context.Background(), opts, public, nil, nil)
done <- err
}()
time.Sleep(50 * time.Millisecond)
// The link's apply ends by keeping the newer declaration, and lets go.
if err := store.SaveDeclared(store.DeclaredPath(opts.state), kept("newer")); err != nil {
applying.Unlock()
t.Fatal(err)
}
applying.Unlock()
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(10 * time.Second):
t.Fatal("the reconcile never ran once the node was free")
}
got, err := os.ReadFile(target)
if err != nil {
t.Fatal(err)
}
if string(got) != "newer\n" {
t.Errorf("the reconcile applied %q, what was kept before the newer declaration", got)
}
}
+25
View File
@@ -0,0 +1,25 @@
package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/cli"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/meshcli"
)
// serveMeshCLI answers mesh-cli on this machine for as long as this engine serves (novox/hq ADR 0272): the account
// from the kernel, the line asked of the controller over the link open at the time. A socket that cannot be served
// is said once, and this engine goes on applying; mesh-cli then says the engine does not answer.
func serveMeshCLI(ctx context.Context, queue *link.Session, say func(string)) {
if err := meshcli.Serve(ctx, cli.Socket, forwardOver(queue), say); err != nil {
say("mesh-cli is not answered on this machine: " + err.Error())
}
}
// forwardOver asks the controller over the queue's link, and follows the line to what came of it.
func forwardOver(queue *link.Session) meshcli.Forward {
return func(ctx context.Context, asked meshcli.Asked) cli.Answer {
return meshcli.AskTheController(ctx, queue.AskController, asked)
}
}
+38
View File
@@ -0,0 +1,38 @@
package main
import (
"errors"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
)
// 1132/AC1, AC3: the report carries one fact per module the declaration names, with the build the controller sent for
// it; a module not yet taken on an adopted node is not one.
func TestTheReportCarriesOneFactPerModuleWithItsBuild(t *testing.T) {
d, err := declaration.Parse([]byte(`{"declaration":1,"builds":{"a":"b-1","b":"b-2"},"resources":[
{"id":"a.dir","type":"directory","path":"/srv/a","module":"a"},
{"id":"b.dir","type":"directory","path":"/srv/b","module":"b"}]}`))
if err != nil {
t.Fatal(err)
}
failure := &apply.Error{Resource: "b.dir", Err: errors.New("no space"), Failures: []apply.Failure{{Resource: "b.dir", Err: errors.New("no space")}}}
facts := moduleFacts(d, apply.Report{}, failure)
if len(facts) != 2 || facts[0].Module != "a" || facts[0].Build != "b-1" || facts[0].Failed ||
facts[1].Module != "b" || facts[1].Build != "b-2" || !facts[1].Failed || facts[1].Detail == "" {
t.Fatalf("facts %+v", facts)
}
adopted, err := declaration.Parse([]byte(`{"declaration":1,"adoption":{"taken":["a"],"untaken":{"b":["b.dir"]}},"resources":[
{"id":"a.dir","type":"directory","path":"/srv/a","module":"a"},
{"id":"b.dir","type":"directory","path":"/srv/b","module":"b"}]}`))
if err != nil {
t.Fatal(err)
}
if facts := moduleFacts(adopted, apply.Report{}, nil); len(facts) != 1 || facts[0].Module != "a" {
t.Fatalf("an untaken module is a fact: %+v", facts)
}
if moduleFacts(nil, apply.Report{}, nil) != nil {
t.Fatal("facts without a declaration")
}
}
+113
View File
@@ -0,0 +1,113 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
osuser "os/user"
"path/filepath"
"strconv"
"strings"
"syscall"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
)
// TestAMoveOutDirectoryKeepsOnlyWhatItKnows drives the node-engine whole through its command (issue 1024, novox/hq
// ADR 0422): a directory under an account's home that says move-out keeps the file the mesh declared and moves a
// file nobody declared into the kept folder, named in a list, and the run says it moved it. It needs no bus: the
// apply is the one a delivery and the five-minute reconcile both run.
func TestAMoveOutDirectoryKeepsOnlyWhatItKnows(t *testing.T) {
me, err := osuser.Current()
if err != nil {
t.Fatal(err)
}
// The user database lists an account as a person's only at uid 1000 or more; root, as the forge's merge check
// runs, is listed under a number of its own (it can give files to it), by the same name.
listedUID := me.Uid
if uid, _ := strconv.Atoi(me.Uid); uid < 1000 || uid == 65534 {
listedUID = "4242"
}
root := t.TempDir()
home := filepath.Join(root, "home", "agent")
dir := filepath.Join(home, ".ssh")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
passwd := filepath.Join(root, "passwd")
line := fmt.Sprintf("%s:x:%s:%s::%s:/bin/sh\n", me.Username, listedUID, me.Gid, home)
if err := os.WriteFile(passwd, []byte(line), 0o644); err != nil {
t.Fatal(err)
}
t.Cleanup(apply.UseAccountsFileForTest(passwd))
// Independent of the machine: an arch host with a pacman on PATH that answers and holds nothing, as
// TestReplay233 puts one there (it answers with an error); nothing here declares a package.
bin := t.TempDir()
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
wasBuilt := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = wasBuilt })
stray := filepath.Join(dir, "id_stray")
if err := os.WriteFile(stray, []byte("somebody else's key"), 0o600); err != nil {
t.Fatal(err)
}
old := time.Now().Add(-time.Hour)
if err := os.Chtimes(stray, old, old); err != nil {
t.Fatal(err)
}
pipe := filepath.Join(dir, "agent.sock")
if err := syscall.Mkfifo(pipe, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chtimes(pipe, old, old); err != nil {
t.Fatal(err)
}
opts := options{state: filepath.Join(root, "state.json")}
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"ssh.dir","type":"directory","path":"` + dir + `","owner":"` + me.Username + `","unknown":"move-out"},` +
`{"id":"ssh.config","type":"file","path":"` + dir + `/config","content":"Host *\n","owner":"` + me.Username + `"}]}`)
var said []string
report := applyAndKeep(context.Background(), opts, raw, nil, nil, func(s string) { said = append(said, s) })
if report.Refused != "" || len(report.Failed) > 0 {
t.Fatalf("refused or failed: %q %v", report.Refused, report.Failed)
}
if _, err := os.Stat(filepath.Join(dir, "config")); err != nil {
t.Errorf("the declared file is not there: %v", err)
}
if _, err := os.Lstat(stray); err == nil {
t.Error("the file nobody declared stayed in the directory")
}
lists, _ := filepath.Glob(filepath.Join(home, ".local", "state", "mesh-moved", "ssh", "*.json"))
if len(lists) != 1 {
t.Fatalf("one list expected: %v", lists)
}
rawList, err := os.ReadFile(lists[0])
if err != nil {
t.Fatal(err)
}
var entries []struct{ From, To string }
if err := json.Unmarshal(rawList, &entries); err != nil || len(entries) != 1 {
t.Fatalf("the list: %s %v", rawList, err)
}
if got, err := os.ReadFile(entries[0].To); err != nil || string(got) != "somebody else's key" {
t.Errorf("the moved file is not whole in the kept folder: %q %v", got, err)
}
if len(report.MovedOut) != 1 || report.MovedOut[0].From != stray || report.MovedOut[0].To != entries[0].To {
t.Errorf("the report does not carry the move: %+v", report.MovedOut)
}
if len(report.Left) != 1 || report.Left[0].Path != pipe {
t.Errorf("the report does not list the pipe left in place: %+v", report.Left)
}
if !strings.Contains(strings.Join(said, "\n"), "moved "+stray) {
t.Errorf("the run does not say it moved the file: %v", said)
}
}
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"context"
"crypto/rand"
"crypto/subtle"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"time"
"golang.org/x/crypto/nacl/box"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// The move of a value sealed to this machine into the vault (design 57's V3, issue 676; internal/link
// movesecret.go). The controller cannot open what it holds; this machine can open what was sealed to it, and so
// hands it to the vault, sealed to a key the vault made for the call. The value is in this process for that one
// call and written nowhere.
// vaultAsk asks one of the vault's verbs in this engine's own name and answers its result.
type vaultAsk func(verb string, args any) (json.RawMessage, error)
// moveSecretFrom is the engine's move: this machine's sealing key, read for each ask, and the vault asked over
// the link open now.
func moveSecretFrom(statePath string, queue *link.Session) link.MoveSecretFunc {
return func(m link.MoveSecret) (link.MovedSecret, error) {
key, err := identity.LoadSealingKey(identity.SealingKeyPath(statePath))
if err != nil {
return link.MovedSecret{}, err
}
ask := func(verb string, args any) (json.RawMessage, error) {
return queue.AskVault(context.Background(), verb, args)
}
moved, err := moveIntoTheVault(ask, key, m)
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: a value was not moved into the vault: %v\n", err)
}
return moved, err
}
}
// moveIntoTheVault opens the row's value with this machine's sealing key, hands it to the vault's move-in in its two
// steps, and answers what the vault kept once its read-back — sealed to a key made here for this one call — opens
// to the same value.
func moveIntoTheVault(ask vaultAsk, key identity.SealingKey, m link.MoveSecret) (link.MovedSecret, error) {
if m.SealedTo != "" && m.SealedTo != key.Public {
return link.MovedSecret{}, errors.New("the controller's store says this value was sealed to a sealing key this " +
"machine no longer has, so it cannot be opened here")
}
value, err := key.Unseal(m.Sealed)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the value does not open with this machine's sealing key: %w", err)
}
defer wipe(value)
// This machine's sealing key, stated in its own name: the key the vault seals the core's credentials to for the
// controller to carry here (novox/hq ADR 0349 §16, issue 763), never a key the controller names.
first := map[string]any{"origin": m.Origin, "generation": m.Generation, "sealing_key": key.Public}
if !m.MadeAt.IsZero() {
first["made_at"] = m.MadeAt.UTC().Format(time.RFC3339Nano)
}
if m.Own() {
first["node"], first["module"], first["name"] = m.Node, m.Module, m.Name
} else {
first["provider"], first["provision"] = m.Provider, m.Provision
first["consumer"], first["consumer_module"] = m.Consumer, m.ConsumerModule
if m.Local != "" {
first["local"] = m.Local
}
}
raw, err := ask("move-in", first)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault did not open the move: %w", err)
}
var opened struct {
Call string `json:"call"`
SealTo string `json:"seal_to"`
Secret string `json:"secret"`
}
if err := json.Unmarshal(raw, &opened); err != nil || opened.Call == "" || opened.SealTo == "" {
return link.MovedSecret{}, errors.New("the vault answered no call and no key to seal to")
}
sealed, err := identity.Seal(opened.SealTo, value)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault's key for the call is not a sealing key: %w", err)
}
backPub, backPriv, err := box.GenerateKey(rand.Reader)
if err != nil {
return link.MovedSecret{}, err
}
defer wipe(backPriv[:])
raw, err = ask("move-in", map[string]any{"call": opened.Call, "sealed": sealed,
"read_back_to": base64.StdEncoding.EncodeToString(backPub[:])})
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault did not take %s: %w", opened.Secret, err)
}
var kept struct {
Name string `json:"name"`
Generation int `json:"generation"`
Unchanged bool `json:"unchanged"`
ReadBack string `json:"read_back"`
}
if err := json.Unmarshal(raw, &kept); err != nil || kept.Name == "" || kept.ReadBack == "" {
return link.MovedSecret{}, fmt.Errorf("the vault answered %s with no read-back", opened.Secret)
}
blob, err := base64.StdEncoding.DecodeString(kept.ReadBack)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault's read-back of %s is not sealed", kept.Name)
}
back, ok := box.OpenAnonymous(nil, blob, backPub, backPriv)
if !ok {
return link.MovedSecret{}, fmt.Errorf("the vault's read-back of %s does not open with the key made for it", kept.Name)
}
defer wipe(back)
if subtle.ConstantTimeCompare(back, value) != 1 {
return link.MovedSecret{}, fmt.Errorf("the vault keeps %s, but it read back another value than the one handed "+
"in: the move of it did not hold", kept.Name)
}
return link.MovedSecret{Secret: kept.Name, Generation: kept.Generation, Unchanged: kept.Unchanged}, nil
}
func wipe(b []byte) {
for i := range b {
b[i] = 0
}
}
+134
View File
@@ -0,0 +1,134 @@
package main
import (
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"strings"
"testing"
"golang.org/x/crypto/nacl/box"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// standInVault answers move-in's two steps as the vault does: a key made for the call, the value opened with it,
// and the value kept read back sealed to the key the second step sent. lie makes its read-back another value.
type standInVault struct {
t *testing.T
pub *[32]byte
priv *[32]byte
opened map[string]any
kept string
lie string
refuse string
carried []string
}
func (v *standInVault) ask(verb string, args any) (json.RawMessage, error) {
raw, _ := json.Marshal(args)
v.carried = append(v.carried, string(raw))
if verb != "move-in" {
v.t.Fatalf("asked %s", verb)
}
if v.refuse != "" {
return nil, errors.New(v.refuse)
}
var a map[string]any
_ = json.Unmarshal(raw, &a)
if a["call"] == nil {
v.opened = a
pub, priv, _ := box.GenerateKey(rand.Reader)
v.pub, v.priv = pub, priv
return json.Marshal(map[string]string{"call": "c1", "seal_to": base64.StdEncoding.EncodeToString(pub[:]),
"secret": "own/anchor/postgres/superuser-password"})
}
blob, _ := base64.StdEncoding.DecodeString(a["sealed"].(string))
value, ok := box.OpenAnonymous(nil, blob, v.pub, v.priv)
if !ok {
v.t.Fatal("what the engine sealed does not open with the call's key")
}
v.kept = string(value)
back := v.kept
if v.lie != "" {
back = v.lie
}
key, _ := base64.StdEncoding.DecodeString(a["read_back_to"].(string))
var to [32]byte
copy(to[:], key)
boxed, _ := box.SealAnonymous(nil, []byte(back), &to, rand.Reader)
return json.Marshal(map[string]any{"name": "own/anchor/postgres/superuser-password", "generation": 1,
"read_back": base64.StdEncoding.EncodeToString(boxed)})
}
const theValue = "the-superusers-password"
func aMoveOf(t *testing.T, key identity.SealingKey) link.MoveSecret {
t.Helper()
sealed, err := identity.Seal(key.Public, []byte(theValue))
if err != nil {
t.Fatal(err)
}
return link.MoveSecret{Node: "anchor", By: "the move", MoveSecretAsk: link.MoveSecretAsk{Module: "postgres",
Name: "superuser-password", Origin: "made", Generation: "g-1", Sealed: sealed, SealedTo: key.Public}}
}
// The engine opens what was sealed to its machine, hands it to the vault sealed to the call's key — never in the
// clear — and answers what the vault kept only once the read-back opens to the same value.
func TestTheEngineMovesAValueSealedToItAndChecksTheReadBack(t *testing.T) {
key, err := identity.GenerateSealingKey()
if err != nil {
t.Fatal(err)
}
vault := &standInVault{t: t}
moved, err := moveIntoTheVault(vault.ask, key, aMoveOf(t, key))
if err != nil {
t.Fatal(err)
}
if moved.Secret != "own/anchor/postgres/superuser-password" || moved.Generation != 1 || vault.kept != theValue {
t.Fatalf("moved %+v, the vault kept %q", moved, vault.kept)
}
if vault.opened["module"] != "postgres" || vault.opened["name"] != "superuser-password" || vault.opened["node"] != "anchor" ||
vault.opened["origin"] != "made" || vault.opened["generation"] != "g-1" ||
vault.opened["sealing_key"] != key.Public {
t.Errorf("the first step named %v", vault.opened)
}
if strings.Contains(strings.Join(vault.carried, " "), theValue) {
t.Fatal("the value crossed the bus in the clear")
}
}
// A read-back that is not the value handed in is a failed move, said; so is a value this machine cannot open, and
// the vault's refusal.
func TestTheEngineSaysAMoveThatDidNotHold(t *testing.T) {
key, _ := identity.GenerateSealingKey()
vault := &standInVault{t: t, lie: "something-else"}
if _, err := moveIntoTheVault(vault.ask, key, aMoveOf(t, key)); err == nil ||
!strings.Contains(err.Error(), "read back") {
t.Fatalf("a read-back of another value: %v", err)
}
other, _ := identity.GenerateSealingKey()
if _, err := moveIntoTheVault((&standInVault{t: t}).ask, key, aMoveOf(t, other)); err == nil ||
!strings.Contains(err.Error(), "sealing key") {
t.Fatalf("a value sealed to another key: %v", err)
}
refusing := &standInVault{t: t, refuse: "the move never replaces a value the vault holds"}
if _, err := moveIntoTheVault(refusing.ask, key, aMoveOf(t, key)); err == nil ||
!strings.Contains(err.Error(), "never replaces") {
t.Fatalf("the vault's refusal: %v", err)
}
// A pair credential names both its ends.
pair := aMoveOf(t, key)
pair.Module, pair.Name = "", ""
pair.Provider, pair.Provision, pair.Consumer, pair.ConsumerModule = "anchor", "database", "shanks", "gitea"
ends := &standInVault{t: t}
if _, err := moveIntoTheVault(ends.ask, key, pair); err != nil {
t.Fatal(err)
}
if ends.opened["provider"] != "anchor" || ends.opened["consumer"] != "shanks" || ends.opened["consumer_module"] != "gitea" ||
ends.opened["node"] != nil {
t.Errorf("the pair's first step named %v", ends.opened)
}
}
+1 -1
View File
@@ -4,7 +4,7 @@ import (
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/network"
)
+15
View File
@@ -0,0 +1,15 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// The checker runs gofmt, go vet and both test levels by Go's convention, and the build module starts the bus the
// integration level needs (novox/hq ADR 0529): no script of the repository's own runs beside them.
func TestTheRepositoryHoldsNoCheckScript(t *testing.T) {
if _, err := os.Stat(filepath.Join("..", "..", "merge-check.sh")); err == nil {
t.Fatal("merge-check.sh is back: the checker's levels run the tests")
}
}
+750
View File
@@ -0,0 +1,750 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The node agent's wiring (novox/hq ADR 0429 §4, ADR 0539 §9): the buckets it watches, the haves it writes, the backup
// it asks and the apply of one module's composition.
// bucketWatch is a bucket of the bus as the agent watches it, through the session's link.
type bucketWatch struct {
session *link.Session
bucket string
}
func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Update, error) {
in, err := w.session.KVWatch(ctx, w.bucket, pattern)
if err != nil {
return nil, err
}
out := make(chan agent.Update, 64)
go func() {
defer close(out)
for u := range in {
select {
case out <- agent.Update{Key: u.Key, Value: u.Value, Revision: u.Revision, Deleted: u.Deleted, Caught: u.Caught}:
case <-ctx.Done():
return
}
}
}()
return out, nil
}
// sessionHaves writes the haves the agent says, through the session.
type sessionHaves struct{ session *link.Session }
func (h sessionHaves) Read(ctx context.Context, module string) (agent.Have, bool) {
got, ok := h.session.ReadHave(ctx, module)
if !ok {
return agent.Have{}, false
}
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail}, true
}
func (h sessionHaves) Write(ctx context.Context, x agent.Have) {
if x.Module == agent.NodeModule {
// The node's own key holds its facts and the digest of the composition it runs.
if x.Phase == "" || x.Phase == string(delivery.PhaseRunning) {
h.session.NoteNode(ctx, x.Composed)
}
return
}
phase := x.Phase
if phase == string(delivery.PhaseRunning) {
phase = ""
}
h.session.NoteModule(ctx, link.ModuleFact{Module: x.Module, FP: x.FP, Composed: x.Composed, Phase: phase,
Failed: phase == string(delivery.PhaseFailed), Detail: shorten(x.Detail)})
}
func (h sessionHaves) Removed(ctx context.Context, module string) {
h.session.NoteRemoved(ctx, module, "")
}
// shorten keeps a have's detail within what a have value holds: its head, which says what failed, and its tail, which is
// the cause (an error wrapped five times names the bus's "no responders" only at its end: 2026-10-25 23:04, mesh-vault on
// the bus node said only which object it read).
func shorten(s string) string {
const most, head = 300, 100
if len(s) <= most {
return s
}
const gap = " … "
tail := s[len(s)-(most-head-len(gap)):]
return strings.ToValidUTF8(s[:head], "") + gap + strings.ToValidUTF8(tail, "")
}
// failedWith is a failed apply that says why, transient when the words are the bus not answering for a moment
// (link.TransientWords): the agent then tries it again without a new revision.
func failedWith(why string) agent.Result {
return agent.Result{Failed: true, Transient: link.TransientWords(why), Detail: shorten(why)}
}
// backupWithin is how long a backup is given from its ask to say it is done (ADR 0429 §14).
var backupWithin = 10 * time.Minute
// backedUpEvent is where the holder of node-backup says a backup taken by hand ended, also one that could not start.
const backedUpEvent = "mesh.mod.restic.event.backed-up"
// backupNode asks the holder of node-backup on this node to back a module up now, and waits for its event to say how it
// went: the holder answers `now` when it has started, and says `backed-up` when the backup ends.
func backupNode(session *link.Session) func(ctx context.Context, module string) agent.Backup {
return func(ctx context.Context, module string) agent.Backup {
// Heard before the ask, so an event that comes at once is not lost.
events, stop, err := session.Hear(backedUpEvent)
if err != nil {
return agent.Backup{Words: "the end of the backup cannot be heard: " + err.Error()}
}
defer stop()
// A module this machine backs nothing of up moves without a backup: never held for one that cannot run (issue
// 1945). An answer that cannot be had or read asks for the backup as before.
if said, err := session.AskSeat(ctx, "node-backup", "backed-up", map[string]any{"module": module}, 30*time.Second); err == nil {
if nothing, read := declaresNothingToBackUp(said, module); read && nothing {
return agent.Backup{Done: true, Words: module + " declares nothing to back up on this machine"}
}
}
asked := time.Now()
if _, err := session.AskSeat(ctx, "node-backup", "now", map[string]any{"module": module}, 30*time.Second); err != nil {
return agent.Backup{Words: err.Error()}
}
return awaitBackup(ctx, events, session.Membership.Node, module, asked, backupWithin)
}
}
// awaitBackup waits for the event that says the backup of a module on this node ended, within the bound: done when it
// says ok, held on its words when it does not, and held when it says nothing within the bound. An event of another node
// or module, or from before the ask, is not the answer.
func awaitBackup(ctx context.Context, events <-chan []byte, node, module string, since time.Time, within time.Duration) agent.Backup {
bound := time.NewTimer(within)
defer bound.Stop()
for {
select {
case <-ctx.Done():
return agent.Backup{Words: "the node agent was stopped while the backup ran"}
case <-bound.C:
return agent.Backup{Words: fmt.Sprintf("the backup did not say it was done within %s", within)}
case raw := <-events:
var e struct {
Node string `json:"node"`
Module string `json:"module"`
OK bool `json:"ok"`
At time.Time `json:"at"`
Error string `json:"error"`
}
if json.Unmarshal(raw, &e) != nil || e.Node != node || e.Module != module || e.At.Before(since.Add(-time.Second)) {
continue
}
if e.OK {
return agent.Backup{Done: true}
}
words := "the backup failed"
if e.Error != "" {
words += ": " + e.Error
}
return agent.Backup{Words: words}
}
}
}
// declarationOf is the declaration the compositions make together: their resources, and the node's own parts when the
// node's composition is among them. It is what the node judges and says its health by, and what one module's apply is
// made of (with a single composition).
//
// **A resource the node's own composition shares with a module's is the module's** (issue 1945): from 2026-10-25 12:43
// the controller put modules it knew no fingerprint of the new form for into the node's own composition while their
// wants named their own, and the declaration joined of both declared each such file twice, which the filling refuses
// ("2 resources declare X") on every node. The node's own copy is left out, and which were left out is said once.
func declarationOf(cs ...delivery.Composed) (*declaration.Declaration, error) {
d := &declaration.Declaration{Version: declaration.Version}
var modules []delivery.Composed
for _, c := range cs {
if c.Module != "" {
modules = append(modules, c)
}
}
if len(modules) > 0 {
joined := make([]delivery.Composed, 0, len(cs))
for _, c := range cs {
if c.Module == "" {
var dropped []string
c, dropped = ownWithout(c, modules)
sayDroppedOnce(dropped)
}
joined = append(joined, c)
}
cs = joined
}
for _, c := range cs {
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
return nil, fmt.Errorf("a resource of %q: %w", c.Module, err)
}
d.Resources = append(d.Resources, r)
}
d.Filled = append(d.Filled, c.Filled...)
if c.Module == "" {
d.Adoption = c.Own.Adoption
d.Identities = append(d.Identities, c.Own.Identities...)
d.Groups = append(d.Groups, c.Own.Groups...)
d.Libraries = append(d.Libraries, c.Own.Libraries...)
d.Provisioner = c.Own.Provisioner
}
}
return d, nil
}
// ownWithout is the node's own composition without the resources a module's composition among modules also declares:
// the same kind and id. A resource that shares only its target (a unit or container of the same name, another id) is
// not the module's, and stays: dropped, its record would be an orphan the apply removes, stopping the module's unit. It
// answers what it left out, as "<id> (<module>)".
func ownWithout(own delivery.Composed, modules []delivery.Composed) (delivery.Composed, []string) {
byID := map[string]string{}
for _, c := range modules {
if c.Module == "" {
continue
}
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
continue
}
byID[string(r.Kind())+"\x00"+r.Identity()] = c.Module
}
}
if len(byID) == 0 {
return own, nil
}
var dropped []string
kept := make([]json.RawMessage, 0, len(own.Resources))
for _, raw := range own.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
kept = append(kept, raw) // refused where it is parsed, with its words
continue
}
module, dup := byID[string(r.Kind())+"\x00"+r.Identity()]
if dup {
dropped = append(dropped, fmt.Sprintf("%s (%s)", r.Identity(), module))
continue
}
kept = append(kept, raw)
}
if len(dropped) == 0 {
return own, nil
}
own.Resources = kept
return own, dropped
}
// droppedSaid is every set of the node's own resources left out that was said, so each is said once.
var droppedSaid sync.Map
// sayDroppedOnce says once which of the node's own resources were left out because a module's composition declares them.
func sayDroppedOnce(dropped []string) {
if len(dropped) == 0 {
return
}
sort.Strings(dropped)
key := strings.Join(dropped, ", ")
if _, said := droppedSaid.LoadOrStore(key, true); said {
return
}
fmt.Fprintf(os.Stderr, "mesh-host: the node's own composition also declares %d resource(s) a module's composition "+
"declares, and they are left out of it: %s (issue 1945)\n", len(dropped), key)
}
func scopeName(c delivery.Composed) string {
if c.Module == "" {
return agent.NodeModule
}
return c.Module
}
func resourcesOf(cs []delivery.Composed) []declaration.Resource {
var out []declaration.Resource
for _, c := range cs {
for _, raw := range c.Resources {
if r, err := declaration.ParseResource(raw); err == nil {
out = append(out, r)
}
}
}
return out
}
// operatorKeysFile is the file beside the node's state that holds the operator's public keys, one ed25519 key in base64 a
// line (a line that starts with # is a note). The switch's last declaration writes it as a file resource, beside the
// controller's key the node was enrolled with (novox/hq ADR 0539 §3, §11); a node adopted later has it at its adoption.
const operatorKeysFile = "operator-keys"
// trustedKeys are the keys a composition may be signed by: the controller's, which the node was enrolled with, and the
// operator's, read from operatorKeysFile each time it is asked, so a key that arrives while the agent runs is trusted at
// the next composition. A file that cannot be read trusts the controller's alone, and says so.
func trustedKeys(opts options, controller ed25519.PublicKey) func() []ed25519.PublicKey {
return func() []ed25519.PublicKey {
keys := []ed25519.PublicKey{}
if len(controller) == ed25519.PublicKeySize {
keys = append(keys, controller)
}
raw, err := os.ReadFile(filepath.Join(filepath.Dir(opts.state), operatorKeysFile))
if err != nil {
return keys
}
for _, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
key, err := base64.StdEncoding.DecodeString(line)
if err != nil || len(key) != ed25519.PublicKeySize {
fmt.Fprintf(os.Stderr, "mesh-host: a line of %s is not an ed25519 public key in base64, and is not trusted\n", operatorKeysFile)
continue
}
keys = append(keys, ed25519.PublicKey(key))
}
return keys
}
}
// keptOnce reads the kept declaration once and splits it by module: its resources (the node's own are those that say
// none, ADR 0539 §9) and its filled paths.
func keptOnce(opts options, signer ed25519.PublicKey) func() (map[string][]json.RawMessage, map[string][]string) {
var once sync.Once
var by map[string][]json.RawMessage
var filled map[string][]string
return func() (map[string][]json.RawMessage, map[string][]string) {
once.Do(func() {
raw, err := store.LoadDeclared(store.DeclaredPath(opts.state), signer)
if err != nil {
return
}
by, filled = splitKept(raw)
})
return by, filled
}
}
// splitKept is a kept declaration's resources by the module they say (the node's own under agent.NodeModule) and its
// filled paths by the module whose file is at each path; nil for a declaration that does not read.
func splitKept(raw []byte) (map[string][]json.RawMessage, map[string][]string) {
var kept struct {
Resources []json.RawMessage `json:"resources"`
Filled []string `json:"filled_files"`
}
if json.Unmarshal(raw, &kept) != nil {
return nil, nil
}
isFilled := make(map[string]bool, len(kept.Filled))
for _, p := range kept.Filled {
isFilled[p] = true
}
by, filled := map[string][]json.RawMessage{}, map[string][]string{}
named := map[string]bool{}
for _, r := range kept.Resources {
var head struct {
Module string `json:"module"`
Type string `json:"type"`
Path string `json:"path"`
}
_ = json.Unmarshal(r, &head)
name := head.Module
if name == "" {
name = agent.NodeModule
}
by[name] = append(by[name], r)
if head.Type == string(declaration.TypeFile) && isFilled[head.Path] && !named[name+"\x00"+head.Path] {
named[name+"\x00"+head.Path] = true
filled[name] = append(filled[name], head.Path)
}
}
return by, filled
}
// nodeAgent runs the node agent under the session's links for as long as ctx lasts: it waits for a link, follows the
// node's wants and compositions on it, and begins again with the next link. A watch it cannot make (a bucket this node's
// account may not read) is said, and tried again with a growing wait, as the link is.
func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, session *link.Session, sched *apply.Scheduler,
say link.Announce, afterApply func(), current *agentNow, sayFacts func(ctx context.Context, ag *agent.Agent)) {
wait := 2 * time.Second
for ctx.Err() == nil {
linked, cancel, err := session.WhenLinked(ctx)
if err != nil {
return
}
began := time.Now()
node := session.Membership.Node
composedBucket, err := delivery.ComposedBucketFor(node)
if err != nil {
cancel()
say("the node agent cannot name this node's composition bucket: " + err.Error())
return
}
var ag *agent.Agent
kept := keptOnce(opts, signer)
// The bus's user list, written alone from the newest composition of the bus module (ADR 0547 §8): on every
// node, and a node whose compositions carry no list writes nothing.
users := userListFollower(opts, session, composedBucket, trustedKeys(opts, signer), say, func() {
// This engine wrote the list, so the bus reloads its accounts and answers no JetStream request for a few
// seconds: what failed in them is tried again once that is over, not at its next wait.
time.AfterFunc(reloadSettles, func() {
if now := current.get(); now != nil {
now.RetryTransient()
}
})
})
// A vault key heard (or heard again changed) looks at the list again, so its hashes are handed (ADR 0547 §9).
vaultKey.following(users.Again)
go func() {
if err := users.Run(linked); err != nil && linked.Err() == nil {
say("the bus's user list is not followed on this link: " + err.Error())
}
}()
env := agent.Env{
Node: node,
Wants: bucketWatch{session, delivery.AssignerWantsBucket},
Composed: bucketWatch{session, composedBucket},
Haves: sessionHaves{session},
Kept: func() map[string][]json.RawMessage { by, _ := kept(); return by },
// The same kept declaration's filled paths, so a module still running from it keeps its files filled.
KeptFilled: func() map[string][]string { _, filled := kept(); return filled },
Recorded: recordedIn(opts.state, say),
Revisions: func(ctx context.Context, key string) ([]agent.Update, error) {
got, err := session.KVHistory(ctx, composedBucket, key)
if err != nil {
return nil, err
}
out := make([]agent.Update, 0, len(got))
for _, u := range got {
out = append(out, agent.Update{Key: u.Key, Value: u.Value, Revision: u.Revision, Deleted: u.Deleted})
}
return out, nil
},
Trusted: trustedKeys(opts, signer),
Backup: backupNode(session),
// What the filling refused before this apply may be named now: the provisioner's client is told to fill
// again (ace, 2026-10-25 21:17).
Settled: func(module string) { knockFilling(module, say) },
Say: say,
// A failure of the bus not answering for a moment is tried again: 2 s, 5 s, 15 s, 60 s, then every 5 min.
Backoff: link.Backoff,
Apply: func(ctx context.Context, c delivery.Composed, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := applyComposition(actx, opts, c, others, sched, say)
done()
// An apply of the bus module may write an older list than the newest composition holds: looked at again.
if module, _ := users.State(); module != "" && module == c.Module {
users.Again()
}
afterApply()
sayFacts(ctx, ag)
return res
},
Remove: func(ctx context.Context, module string, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := removeComposition(actx, opts, module, others, sched, say)
done()
afterApply()
sayFacts(ctx, ag)
return res
},
}
ag = agent.New(env)
current.set(ag)
err = ag.Run(linked)
cancel()
current.set(nil)
if ctx.Err() != nil {
return
}
if time.Since(began) > 30*time.Second {
wait = 2 * time.Second
}
if err != nil {
say(fmt.Sprintf("the node agent stopped: %v — beginning again in %s", err, wait))
}
select {
case <-ctx.Done():
return
case <-time.After(wait):
}
if wait *= 2; wait > 2*time.Minute {
wait = 2 * time.Minute
}
}
}
// reloadSettles is how long after this engine wrote the bus's user list the bus is taken to have reloaded its accounts:
// the bus module's entrypoint looks at the list every 5 s and then signals the server, whose reload of the accounts
// answers no JetStream request for about three seconds (on the bus node, 2026-10-25).
var reloadSettles = 10 * time.Second
// agentNow is the agent running now, which the node's facts are built from.
type agentNow struct {
mu sync.Mutex
ag *agent.Agent
}
func (c *agentNow) set(a *agent.Agent) {
c.mu.Lock()
c.ag = a
c.mu.Unlock()
}
func (c *agentNow) get() *agent.Agent {
c.mu.Lock()
defer c.mu.Unlock()
return c.ag
}
// applyComposition applies one composition: a module's, or the node's own.
func applyComposition(ctx context.Context, opts options, c delivery.Composed, others []delivery.Composed,
sched *apply.Scheduler, say link.Announce) agent.Result {
name := scopeName(c)
if c.Module == "" {
// What another module's composition declares is that module's, never the node's own apply's (issue 1945).
var dropped []string
c, dropped = ownWithout(c, others)
sayDroppedOnce(dropped)
}
d, err := declarationOf(c)
if err != nil {
return failedWith(err.Error())
}
if err := refuseEmptyWrites(d); err != nil {
return failedWith(err.Error())
}
scope := apply.Scope{Module: name, Others: resourcesOf(others)}
got := applyModuleHeld(ctx, opts, d, &scope, sched, say)
if got.Refused != "" {
return failedWith(got.Refused)
}
results := apply.ModuleResults(d, got.Outcome, got.Err)
if r, ok := results[name]; ok && r.Failed {
return failedWith(r.Detail)
}
if got.Err != nil {
return failedWith(got.Err.Error())
}
if got.Probe != nil {
return failedWith(got.Probe.partial().Error())
}
return agent.Result{}
}
// removeComposition removes one module's own resources and nothing of another's.
func removeComposition(ctx context.Context, opts options, module string, others []delivery.Composed,
sched *apply.Scheduler, say link.Announce) agent.Result {
scope := apply.Scope{Module: module, Others: resourcesOf(others)}
got := applyModuleHeld(ctx, opts, &declaration.Declaration{Version: declaration.Version}, &scope, sched, say)
switch {
case got.Refused != "":
return agent.Result{Failed: true, Detail: shorten(got.Refused)}
case got.Err != nil:
return agent.Result{Failed: true, Detail: shorten(got.Err.Error())}
}
return agent.Result{}
}
// busyModules are the modules whose last apply left a container running because it said busy.
func busyModules() []string {
busyMu.Lock()
defer busyMu.Unlock()
var out []string
for m, w := range busyBy {
if w && m != "" {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// keptDeclaration is the declaration the node kept before the switch, verified with the mesh's key when it is given.
func keptDeclaration(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
var raw []byte
if len(signer) > 0 {
var err error
if raw, err = store.LoadDeclared(store.DeclaredPath(statePath), signer); err != nil {
return nil, err
}
} else {
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
if err != nil {
return nil, err
}
raw = kept.Declaration
}
return declaration.Parse(raw)
}
// whatRuns is what runs on this node as the declaration of it (novox/hq ADR 0539 §9): the compositions the agent runs
// and, for every module the agent does not hold yet, what the kept declaration has for it, with the node's own parts
// (adoption, identities, groups, libraries) from the node's composition once the agent holds it and from the kept
// declaration until then. What the node judges its health by, and what the filling, the take act and the judgement of
// directories read.
func whatRuns(everything []delivery.Composed, holdsNode bool, kept func() (*declaration.Declaration, error)) (*declaration.Declaration, error) {
d, err := declarationOf(everything...)
if err != nil {
return nil, err
}
if holdsNode {
return d, nil
}
old, err := kept()
if err != nil {
if len(everything) > 0 {
return d, nil
}
return nil, err
}
d.Adoption, d.Identities, d.Groups, d.Libraries = old.Adoption, old.Identities, old.Groups, old.Libraries
d.Provisioner, d.Builds = old.Provisioner, old.Builds
d.Filled = append(d.Filled, old.Filled...)
return d, nil
}
// runsHere says what runs on this node; the running serve sets it to the agent's view, and a command run by hand reads
// the kept declaration.
var runsHere struct {
mu sync.Mutex
fn func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error)
}
func setRunsHere(fn func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error)) {
runsHere.mu.Lock()
runsHere.fn = fn
runsHere.mu.Unlock()
}
// declaredNow is the declaration of what runs on this node now.
func declaredNow(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
runsHere.mu.Lock()
fn := runsHere.fn
runsHere.mu.Unlock()
if fn != nil {
return fn(statePath, signer)
}
return keptDeclaration(statePath, signer)
}
// runsFromAgent is what runs now as the agent in hand sees it.
func runsFromAgent(current *agentNow) func(string, ed25519.PublicKey) (*declaration.Declaration, error) {
return func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
kept := func() (*declaration.Declaration, error) { return keptDeclaration(statePath, signer) }
ag := current.get()
if ag == nil {
return kept()
}
return whatRuns(ag.Everything(), ag.HoldsNode(), kept)
}
}
// refuseEmptyWrites refuses a declaration that names a file the provisioner fills (owned at runtime, no content, no
// bytes, no digest, no sealed value) without saying it is filled: applied, the file would be written empty over what the
// provisioner wrote (issue 1945, live on 2026-10-25 at 01:22 on ace, g14 and shanks). The apply fails, naming the file,
// and nothing is written.
func refuseEmptyWrites(d *declaration.Declaration) error {
filled := make(map[string]bool, len(d.Filled))
for _, p := range d.Filled {
filled[p] = true
}
var refused []string
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || f.OwnerAtRuntime != declaration.RuntimeOwned || filled[f.Path] {
continue
}
if f.Content == "" && f.Bytes == "" && f.Digest == "" && f.Sealed == "" && f.VaultSealed == nil {
refused = append(refused, fmt.Sprintf("%s (%s)", f.ID, f.Path))
}
}
if len(refused) > 0 {
return fmt.Errorf("the composition names %s with no content and does not say the provisioner fills it, so it "+
"would be written empty; nothing was applied", strings.Join(refused, ", "))
}
return nil
}
// recordedIn says the engine's applied state holds a record of every resource of a composition: what the agent believes
// of a have or the kept declaration only when it does, so a resource removed since is made again (issue 1945). A state
// that cannot be read says yes: the agent then believes what it was told, as before.
func recordedIn(statePath string, say func(string)) func(ctx context.Context, c delivery.Composed) bool {
return func(_ context.Context, c delivery.Composed) bool {
known, err := store.Load(statePath)
if err != nil {
if say != nil {
say(fmt.Sprintf("the applied state cannot be read, so what runs is believed as told: %v", err))
}
return true
}
filled := make(map[string]bool, len(c.Filled))
for _, p := range c.Filled {
filled[p] = true
}
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
return true
}
// A file the provisioner client fills is left to it, and has no record until it writes it.
if f, ok := r.(*declaration.File); ok && filled[f.Path] {
continue
}
if _, ok := known.Find(r.Identity()); !ok {
return false
}
}
return true
}
}
// declaresNothingToBackUp reads node-backup.backed-up's answer for a module: true when this machine backs nothing of it
// up (no runs, no paths), and whether the answer could be read at all.
func declaresNothingToBackUp(said json.RawMessage, module string) (nothing bool, read bool) {
var list []struct {
Module string `json:"module"`
Runs int `json:"runs"`
Paths []string `json:"paths"`
}
if err := json.Unmarshal(said, &list); err != nil {
return false, false
}
for _, m := range list {
if m.Module == module && (m.Runs > 0 || len(m.Paths) > 0) {
return false, true
}
}
return true, true
}
// applyBound is how long one apply or removal may take: long enough for the slowest image pull and backup step, short
// enough that a step that hangs cannot hold the node agent for ever.
const applyBound = 30 * time.Minute
// applyContext is the context an apply or a removal runs in: the link's values, never its end, and a bound of its own.
// The link's context ends when the bus goes, and the bus may go because of the apply itself (it replaces the bus's own
// container), which must then finish: a container removed and never made again took the mesh down (issue 1945, live on
// 2026-10-25 at 11:23). applyBound keeps a hung step from blocking the agent for ever.
func applyContext(ctx context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(context.WithoutCancel(ctx), applyBound)
}
+313
View File
@@ -0,0 +1,313 @@
//go:build integration
package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/link"
)
// The node agent whole, against a real bus with JetStream started in the test: the wants bucket, the node's composition
// bucket and the have bucket, a session on the bus, and the real apply of a module's composition into a directory of the
// test. R6-A AC1, AC5 and AC9 (novox/hq ADR 0539).
const itNode = "it1"
type itBed struct {
t *testing.T
conn *nats.Conn
js nats.JetStreamContext
wants nats.KeyValue
composed nats.KeyValue
haves nats.KeyValue
dir string
opts options
session *link.Session
controll ed25519.PrivateKey
operator ed25519.PrivateKey
said []string
mu sync.Mutex
}
func aServer(t *testing.T) *nats.Conn {
t.Helper()
srv, err := server.NewServer(&server.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true})
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the embedded server did not start")
}
t.Cleanup(srv.Shutdown)
conn, err := nats.Connect(srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
return conn
}
func newItBed(t *testing.T) *itBed {
t.Helper()
// Not skipped off Arch: a package manager that answers is put on PATH, so the apply of files runs on any build node.
bin := t.TempDir()
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
b := &itBed{t: t, conn: aServer(t), dir: t.TempDir()}
var err error
if b.js, err = b.conn.JetStream(); err != nil {
t.Fatal(err)
}
composedBucket, _ := delivery.ComposedBucketFor(itNode)
for _, c := range []struct {
name string
into *nats.KeyValue
history uint8
}{{delivery.AssignerWantsBucket, &b.wants, 8}, {composedBucket, &b.composed, delivery.ComposedHistory},
{delivery.HaveBucket, &b.haves, delivery.HaveHistory}} {
kv, err := b.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: c.name, History: c.history})
if err != nil {
t.Fatal(err)
}
*c.into = kv
}
b.opts = options{state: filepath.Join(b.dir, "state", "state.json"), timeout: 2 * time.Second}
if err := os.MkdirAll(filepath.Dir(b.opts.state), 0o755); err != nil {
t.Fatal(err)
}
_, b.controll, _ = ed25519.GenerateKey(nil)
_, b.operator, _ = ed25519.GenerateKey(nil)
return b
}
// start runs the node agent under a session on the bus, trusting the controller's key, and the operator's when asked.
func (b *itBed) start(trustOperator bool) (stop func()) {
b.t.Helper()
if trustOperator {
pub := b.operator.Public().(ed25519.PublicKey)
if err := os.WriteFile(filepath.Join(filepath.Dir(b.opts.state), operatorKeysFile),
[]byte("# the operator\n"+base64.StdEncoding.EncodeToString(pub)+"\n"), 0o600); err != nil {
b.t.Fatal(err)
}
}
b.session = &link.Session{Membership: link.Membership{Node: itNode}, Timeout: 2 * time.Second,
Say: b.say}
b.session.Attach(context.Background(), link.OverNATS{Conn: b.conn, JS: b.js, Node: itNode})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
defer close(done)
nodeAgent(ctx, b.opts, b.controll.Public().(ed25519.PublicKey), b.session, nil, b.say, func() {},
&agentNow{}, func(context.Context, *agent.Agent) {})
}()
return func() { cancel(); <-done }
}
func (b *itBed) say(line string) {
b.mu.Lock()
b.said = append(b.said, line)
b.mu.Unlock()
b.t.Log(line)
}
func fpIt(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
// compose writes a module's signed composition of one file into the node's bucket and returns its digest.
func (b *itBed) compose(module, fp, content string, data bool, signer ed25519.PrivateKey) string {
b.t.Helper()
path := filepath.Join(b.dir, module+".conf")
res, _ := json.Marshal(map[string]any{"id": module + ".conf", "type": "file", "path": path, "content": content, "module": module})
c := delivery.Composed{Node: itNode, Module: module, FP: fp, Data: data, Resources: []json.RawMessage{res}}
signed, err := delivery.SignComposed(c, signer)
if err != nil {
b.t.Fatal(err)
}
raw, err := delivery.EncodeComposed(signed)
if err != nil {
b.t.Fatal(err)
}
dig, _ := delivery.ComposedDigestOf(signed)
key, _ := delivery.ComposedKey(itNode, module, fp)
if _, err := b.composed.Put(key, raw); err != nil {
b.t.Fatal(err)
}
return dig
}
func (b *itBed) want(module, fp, digest, backup string) {
b.t.Helper()
raw, err := delivery.EncodeWant(delivery.Want{FP: fp, Composed: digest, Backup: backup})
if err != nil {
b.t.Fatal(err)
}
key, _ := delivery.WantKey(itNode, module)
if _, err := b.wants.Put(key, raw); err != nil {
b.t.Fatal(err)
}
}
func (b *itBed) file(module string) string {
got, _ := os.ReadFile(filepath.Join(b.dir, module+".conf"))
return string(got)
}
func (b *itBed) have(module string) delivery.Have {
key, _ := delivery.HaveKey(itNode, module)
e, err := b.haves.Get(key)
if err != nil {
return delivery.Have{}
}
h, _ := delivery.DecodeHave(e.Value())
return h
}
func within(t *testing.T, d time.Duration, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(d)
for !ok() {
if time.Now().After(deadline) {
t.Fatal(what)
}
time.Sleep(10 * time.Millisecond)
}
}
// R6-A AC1: given a running agent on a bus, when a want naming a digest and that composition are written, the module is
// applied within one second and its have names the fingerprint and the digest; a composition whose digest no want names
// is not applied.
func TestNatsAWantAndItsCompositionAreAppliedWithinASecond(t *testing.T) {
b := newItBed(t)
stop := b.start(false)
defer stop()
time.Sleep(300 * time.Millisecond) // the watches are up
fp := fpIt("a")
dig := b.compose("web", fp, "one\n", false, b.controll)
time.Sleep(300 * time.Millisecond)
if b.file("web") != "" {
t.Fatal("a composition no want names was applied")
}
began := time.Now()
b.want("web", fp, dig, "")
within(t, time.Second, "the module was not applied within one second of its want", func() bool { return b.file("web") == "one\n" })
t.Logf("applied %s after the want", time.Since(began))
within(t, 2*time.Second, "the have does not name the fingerprint and the digest", func() bool {
h := b.have("web")
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
})
// A want naming a digest the composition at its fingerprint does not have applies nothing.
fp2 := fpIt("b")
b.compose("web", fp2, "two\n", false, b.controll)
b.want("web", fp2, "sha256:"+strings.Repeat("0", 64), "")
time.Sleep(300 * time.Millisecond)
if b.file("web") != "one\n" {
t.Fatalf("a want naming another digest changed the module: %q", b.file("web"))
}
}
// R6-A AC5: a data module that ran before asks its backup before any resource changes; a backup that fails holds the
// module (nothing applied, the have says held with the backup's words); when its want then says skip, it is applied.
func TestNatsADataModuleAsksItsBackupFirstAndIsHeldWhenItFails(t *testing.T) {
b := newItBed(t)
var mu sync.Mutex
var asked []string
var onDiskWhenAsked string
if _, err := b.conn.Subscribe("mesh.seat.node-backup.tool.now."+itNode, func(m *nats.Msg) {
mu.Lock()
asked = append(asked, "now")
onDiskWhenAsked = b.file("db")
mu.Unlock()
_ = m.Respond([]byte(`{"result":{"started":"db"}}`))
// The holder says the backup ended on its event.
ev, _ := json.Marshal(map[string]any{"node": itNode, "module": "db", "ok": false, "at": time.Now().UTC(),
"error": "restic: the repository is locked"})
_ = b.conn.Publish(backedUpEvent, ev)
}); err != nil {
t.Fatal(err)
}
stop := b.start(false)
defer stop()
time.Sleep(300 * time.Millisecond)
fp1, fp2 := fpIt("a"), fpIt("b")
b.want("db", fp1, b.compose("db", fp1, "v1\n", true, b.controll), "")
within(t, 3*time.Second, "the first composition was not applied", func() bool { return b.file("db") == "v1\n" })
mu.Lock()
if len(asked) != 0 {
t.Fatalf("a module that never ran was backed up: %v", asked)
}
mu.Unlock()
dig2 := b.compose("db", fp2, "v2\n", true, b.controll)
b.want("db", fp2, dig2, "")
within(t, 3*time.Second, "the module was not held behind the failed backup", func() bool {
h := b.have("db")
return h.Phase == delivery.PhaseHeld && strings.Contains(h.Detail, "repository is locked")
})
mu.Lock()
if len(asked) != 1 || onDiskWhenAsked != "v1\n" {
t.Fatalf("the backup was asked %v, with %q on disk: it is asked before any resource changes", asked, onDiskWhenAsked)
}
mu.Unlock()
if b.file("db") != "v1\n" {
t.Fatalf("something was applied behind a failed backup: %q", b.file("db"))
}
if h := b.have("db"); h.FP != fp1 {
t.Fatalf("a held module should still name what it runs: %+v", h)
}
b.want("db", fp2, dig2, delivery.BackupSkip)
within(t, 3*time.Second, "the module was not applied once its want said skip", func() bool { return b.file("db") == "v2\n" })
within(t, 2*time.Second, "the have does not say running at the new fingerprint", func() bool {
h := b.have("db")
return h.Phase == delivery.PhaseRunning && h.FP == fp2 && h.Composed == dig2
})
}
// R6-A AC9, the hand path end to end: with no controller running, a composition signed by the operator's key and a want
// naming it are applied and the have written; the same signed by a key nobody trusts is not.
func TestNatsTheHandPathAppliesACompositionTheOperatorSigned(t *testing.T) {
b := newItBed(t)
stop := b.start(true)
defer stop()
time.Sleep(300 * time.Millisecond)
_, stranger, _ := ed25519.GenerateKey(nil)
fpS := fpIt("c")
b.want("evil", fpS, b.compose("evil", fpS, "no\n", false, stranger), "")
time.Sleep(300 * time.Millisecond)
if b.file("evil") != "" {
t.Fatal("a composition signed by a key nobody trusts was applied")
}
fp := fpIt("d")
dig := b.compose("rescue", fp, "back\n", false, b.operator)
b.want("rescue", fp, dig, "")
within(t, 2*time.Second, "the operator's composition was not applied", func() bool { return b.file("rescue") == "back\n" })
within(t, 2*time.Second, "the have was not written", func() bool {
h := b.have("rescue")
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
})
}
+42
View File
@@ -0,0 +1,42 @@
package main
import (
"io/fs"
"os"
"path/filepath"
"strings"
"testing"
)
// R6-A AC8: nothing in the node-engine receives, orders or reconciles a whole declaration (novox/hq ADR 0539 §9, ADR
// 0429 §8): no package subscribes to a node's declare subject or names the deleted symbols.
func TestNoPackageHearsOrOrdersAWholeDeclaration(t *testing.T) {
deleted := []string{
"DeclareSubject", "mesh.node.\" + node + \".declare", "ReconcileDue", "ReconcileEvery", "olderThanKept",
"refuseOlder", "olderGeneration", "appliedGeneration", "reconcileKept", "reapplyKept", "adoptionWatch",
"unsaidFile", "numbersFile", "func gather(", "func handleBody(", "Supersedes(", "ReportSequence",
"RefusedOlder", "ReadsGeneration", "ReadsModules", "GenerationRefusal", "LeftOutModuleOf",
}
root := filepath.Join("..", "..")
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "nodeclare_test.go") {
return nil
}
body, err := os.ReadFile(path)
if err != nil {
return err
}
for _, word := range deleted {
if strings.Contains(string(body), word) {
t.Errorf("%s still names %q", path, word)
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
}
+142
View File
@@ -0,0 +1,142 @@
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"time"
"github.com/novox/mesh-host/internal/identity"
)
// This machine's sealing key, stated to the vault in the engine's own name (novox/hq ADR 0372, issue 786). The vault
// records it from the machine's own account alone and, when it is a first or a changed key, says the machine's core
// secrets changed so the controller asks for them again sealed to it. Without this a machine that never moved a
// secret in (adopted after feature 57) has no key at the vault. Nothing but the public half crosses.
const (
// nodeKeyRetryFrom and nodeKeyRetryUpTo bound the wait between tries while the vault does not answer: it starts
// after the bus and may not serve the verb yet.
nodeKeyRetryFrom = 5 * time.Second
nodeKeyRetryUpTo = 5 * time.Minute
// nodeKeyLook is how often the key file is read to see whether it changed.
nodeKeyLook = 30 * time.Second
)
// heardKey is the vault's own public key as its answer to node-key gave it (ADR 0547 clause 9, issue 1945): the key the
// bus node hands the loaded list's hashes sealed to. One for the process, as the key stated is; kick is the follower of
// the user list on the link open now, told when the key is first heard or changes.
type heardKey struct {
mu sync.Mutex
pem string
kick func()
}
// vaultKey is the vault's key this engine heard.
var vaultKey = &heardKey{}
func (h *heardKey) get() string {
h.mu.Lock()
defer h.mu.Unlock()
return h.pem
}
// heard keeps the key a node-key answer gave, and tells the follower when it is a first or changed one. An answer
// with no key (a vault before this one) changes nothing.
func (h *heardKey) heard(answer json.RawMessage) {
var a struct {
VaultKey string `json:"vault_key"`
}
if json.Unmarshal(answer, &a) != nil || !strings.Contains(a.VaultKey, "PUBLIC KEY") {
return
}
h.mu.Lock()
changed := a.VaultKey != h.pem
h.pem = a.VaultKey
kick := h.kick
h.mu.Unlock()
if changed && kick != nil {
kick()
}
}
// following names the follower told when the key changes, and tells it at once when a key is known.
func (h *heardKey) following(kick func()) {
h.mu.Lock()
h.kick = kick
known := h.pem != ""
h.mu.Unlock()
if known && kick != nil {
kick()
}
}
// nodeKeyReask is how often the key is stated again though it did not change, so a changed vault key is heard: the
// vault takes the same key again and changes nothing (ADR 0547 clause 9).
const nodeKeyReask = 10 * time.Minute
// keyHearing is what the keeper does with the vault's own key in node-key's answers (ADR 0547 clause 9): heard takes
// it from an answer; held says whether one is held, and while none is the key is stated again at every look, so a vault
// delivered after this engine is heard within a look; every is how often it is stated again all the same, so a changed
// vault key is heard.
type keyHearing struct {
heard func(json.RawMessage)
held func() bool
// look is how often it is stated again while no vault key is held; every how often once one is.
look, every time.Duration
}
// keepNodeKeyStated states the sealing key load answers to the vault at start, retrying with a growing wait from
// retryFrom to retryUpTo while it does not answer, and states it again whenever the key changes. It says one line
// when a key is stated and one when a try fails for another reason than the last, and returns when ctx ends. With
// hearing, each answer goes to it, and the key is stated again while no vault key is held and at its interval.
func keepNodeKeyStated(ctx context.Context, load func() (identity.SealingKey, error), ask vaultAsk, say func(string),
retryFrom, retryUpTo time.Duration, hearing *keyHearing) {
stated, lastFailure := "", ""
var askedAt time.Time
wait := retryFrom
fail := func(why string) {
if why != lastFailure {
say("mesh-host: the sealing key was not stated to the vault, trying again: " + why)
lastFailure = why
}
wait = min(wait*2, retryUpTo)
}
look := min(nodeKeyLook, retryFrom)
for {
delay := look
key, err := load()
again := false
if hearing != nil && stated != "" {
since := time.Since(askedAt)
held := hearing.held == nil || hearing.held()
again = (!held && since >= hearing.look) || (held && hearing.every > 0 && since >= hearing.every)
}
switch {
case err != nil:
fail(err.Error())
delay = wait
case key.Public != stated || again:
if answer, err := ask("node-key", map[string]string{"key": key.Public}); err != nil {
fail(err.Error())
delay = wait
} else {
askedAt = time.Now()
if hearing != nil && hearing.heard != nil {
hearing.heard(answer)
}
if key.Public != stated {
say(fmt.Sprintf("mesh-host: this machine's sealing key %s is stated to the vault", key.Public))
}
stated, lastFailure, wait = key.Public, "", retryFrom
}
}
select {
case <-ctx.Done():
return
case <-time.After(delay):
}
}
}
+190
View File
@@ -0,0 +1,190 @@
package main
import (
"context"
"encoding/json"
"errors"
"sync"
"testing"
"time"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// Issue 786 (novox/hq ADR 0372): the engine states its own sealing key to the vault's node-key at start, retries
// while the vault does not answer, and states it again when the key changes; the same key is not stated twice.
type keyVault struct {
mu sync.Mutex
stated []string
fail int
verbs []string
// keyless is how many answers carry no vault key (a vault before ADR 0547 clause 9's, or not yet started); key the
// vault's key the others carry.
keyless int
key string
}
func (v *keyVault) ask(verb string, args any) (json.RawMessage, error) {
v.mu.Lock()
defer v.mu.Unlock()
v.verbs = append(v.verbs, verb)
if v.fail > 0 {
v.fail--
return nil, errors.New("the vault does not answer")
}
raw, _ := json.Marshal(args)
var a struct {
Key string `json:"key"`
}
_ = json.Unmarshal(raw, &a)
v.stated = append(v.stated, a.Key)
if v.key == "" || v.keyless > 0 {
v.keyless--
return json.RawMessage(`{"recorded":true}`), nil
}
answer, _ := json.Marshal(map[string]any{"recorded": true, "vault_key": v.key})
return answer, nil
}
func (v *keyVault) said() []string {
v.mu.Lock()
defer v.mu.Unlock()
return append([]string(nil), v.stated...)
}
func waitFor(t *testing.T, what string, ok func() bool) {
t.Helper()
for i := 0; i < 400; i++ {
if ok() {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatalf("timed out: %s", what)
}
func TestTheEngineStatesItsSealingKeyToTheVaultAtStartRetriesAndRestatesAChange(t *testing.T) {
first, _ := identity.GenerateSealingKey()
second, _ := identity.GenerateSealingKey()
var mu sync.Mutex
current := first
load := func() (identity.SealingKey, error) {
mu.Lock()
defer mu.Unlock()
return current, nil
}
vault := &keyVault{fail: 2}
var lines []string
var lmu sync.Mutex
say := func(l string) { lmu.Lock(); lines = append(lines, l); lmu.Unlock() }
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
var hmu sync.Mutex
var answers []string
heard := func(a json.RawMessage) { hmu.Lock(); answers = append(answers, string(a)); hmu.Unlock() }
go keepNodeKeyStated(ctx, load, vault.ask, say, time.Millisecond, 5*time.Millisecond, &keyHearing{heard: heard})
waitFor(t, "the first key stated after two refusals", func() bool { return len(vault.said()) == 1 })
// Its answer is handed on, so the vault's own key in it is heard (ADR 0547 clause 9); refusals hand nothing.
waitFor(t, "the answer heard", func() bool { hmu.Lock(); defer hmu.Unlock(); return len(answers) == 1 })
if vault.said()[0] != first.Public {
t.Errorf("stated %q, want the engine's own key %q", vault.said()[0], first.Public)
}
vault.mu.Lock()
verbs := append([]string(nil), vault.verbs...)
vault.mu.Unlock()
for _, v := range verbs {
if v != "node-key" {
t.Errorf("asked %s", v)
}
}
// The same key again is not stated again.
time.Sleep(60 * time.Millisecond)
if n := len(vault.said()); n != 1 {
t.Errorf("the same key was stated %d times", n)
}
mu.Lock()
current = second
mu.Unlock()
waitFor(t, "the changed key stated", func() bool { return len(vault.said()) == 2 })
if vault.said()[1] != second.Public {
t.Errorf("stated %q after the change, want %q", vault.said()[1], second.Public)
}
lmu.Lock()
defer lmu.Unlock()
if len(lines) == 0 {
t.Error("the engine said nothing")
}
asksAgainUntilItHearsTheVaultsKey(t)
}
// ADR 0547 clause 9: the vault's key is heard though the engine's own key never changes. While none is held the key
// is stated again at each look, so a vault delivered after the engine (or answering no key yet) is heard; once one is
// held, again only at the slow interval, so a changed vault key is heard; never more.
func asksAgainUntilItHearsTheVaultsKey(t *testing.T) {
sealing, _ := identity.GenerateSealingKey()
load := func() (identity.SealingKey, error) { return sealing, nil }
pem := func(c string) string { return "-----BEGIN PUBLIC KEY-----\n" + c + "\n-----END PUBLIC KEY-----\n" }
for _, c := range []struct {
name string
keyless int
every time.Duration
change bool
wantAtLeast int
wantAtMost int
}{
{name: "a vault with no key yet is asked again until it answers one, then not", keyless: 3, wantAtLeast: 4, wantAtMost: 4},
{name: "a held key is asked again at the slow interval, and a changed one heard", every: 20 * time.Millisecond,
change: true, wantAtLeast: 3, wantAtMost: 1 << 20},
} {
t.Run(c.name, func(t *testing.T) {
vault := &keyVault{keyless: c.keyless, key: pem("AAAA")}
h := &heardKey{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go keepNodeKeyStated(ctx, load, vault.ask, func(string) {}, time.Millisecond, 5*time.Millisecond,
&keyHearing{heard: h.heard, held: func() bool { return h.get() != "" }, look: time.Millisecond, every: c.every})
waitFor(t, "the vault's key heard", func() bool { return h.get() == pem("AAAA") })
if c.change {
vault.mu.Lock()
vault.key = pem("BBBB")
vault.mu.Unlock()
waitFor(t, "the changed vault key heard", func() bool { return h.get() == pem("BBBB") })
}
time.Sleep(60 * time.Millisecond)
n := len(vault.said())
if n < c.wantAtLeast || n > c.wantAtMost {
t.Errorf("node-key asked %d times, want %d to %d", n, c.wantAtLeast, c.wantAtMost)
}
})
}
}
// The statement a host makes names this contract (issue 786), one above the vault's box. And the vault's key its
// answer gives is kept, told to the follower once when first heard and again only when changed; an answer without one
// (an older vault) or with no public key changes nothing (ADR 0547 clause 9).
func TestTheNodeKeyContractIsThirteen(t *testing.T) {
if link.NodeKeyContract != 13 {
t.Errorf("NodeKeyContract %d", link.NodeKeyContract)
}
h := &heardKey{}
kicks := 0
h.following(func() { kicks++ })
h.heard(json.RawMessage(`{"recorded":true}`))
h.heard(json.RawMessage(`{"recorded":true,"vault_key":"not a key"}`))
if h.get() != "" || kicks != 0 {
t.Fatalf("an answer with no key was kept: %q, %d kicks", h.get(), kicks)
}
pem := "-----BEGIN PUBLIC KEY-----\nAAAA\n-----END PUBLIC KEY-----\n"
answer, _ := json.Marshal(map[string]any{"recorded": true, "vault_key": pem})
h.heard(answer)
h.heard(answer)
if h.get() != pem || kicks != 1 {
t.Fatalf("key %q, %d kicks, want the key and one", h.get(), kicks)
}
h.following(func() { kicks += 10 })
if kicks != 11 {
t.Fatalf("a follower named after the key was heard was not told: %d", kicks)
}
}
-95
View File
@@ -1,95 +0,0 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// A declaration carries no order, so a host cannot tell an older one from a newer (novox/hq
// 04-ISSUES/107). Its only identity was the digest of its bytes: "not the last" could be said,
// "older" could not.
func keptWith(t *testing.T, sequence int64) store.Declared {
t.Helper()
return keptAt(t, link.Order{Sequence: sequence})
}
func keptAt(t *testing.T, order link.Order) store.Declared {
t.Helper()
body, err := json.Marshal(map[string]any{
"declaration": 1, "resources": []any{}, "owns_nothing": true, "sequence": order.Sequence,
"epoch": order.Epoch,
})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: []byte("x")}
}
func TestAnOlderDeclarationFromTheMeshIsRefused(t *testing.T) {
err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 5})
if err == nil {
t.Fatal("sequence 5 was accepted over a kept 7")
}
if !strings.Contains(err.Error(), "older") || !strings.Contains(err.Error(), "nothing was applied") {
t.Fatalf("the refusal does not say what it is: %v", err)
}
}
func TestANewerOrEqualDeclarationIsNot(t *testing.T) {
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 8}); err != nil {
t.Fatalf("sequence 8 was refused over a kept 7: %v", err)
}
// Equal is the same declaration again, which reconciling is for.
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 7}); err != nil {
t.Fatalf("the same sequence was refused: %v", err)
}
}
func TestNoOrderClaimedMeansNoOrderCompared(t *testing.T) {
// An older controller sends none; a host that received before it understood them kept none.
// Refusing on a guess would strand a node the moment the controller is older than the host.
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 0}); err != nil {
t.Fatalf("a declaration claiming no order was refused: %v", err)
}
if err := refuseOlder(keptWith(t, 0), nil, link.Order{Sequence: 3}); err != nil {
t.Fatalf("a declaration was refused against a kept one that claimed no order: %v", err)
}
if err := refuseOlder(store.Declared{}, store.ErrNothingDeclared, link.Order{Sequence: 3}); err != nil {
t.Fatalf("a first declaration was refused: %v", err)
}
}
// **The lease epoch decides first** (novox/hq to-be 45 §6): a declaration from a controller that lost
// its lease is refused whatever its sequence, and a new lease holder's is taken whatever its sequence.
func TestAnOlderEpochIsRefusedAndANewerTaken(t *testing.T) {
kept := keptAt(t, link.Order{Epoch: 57, Sequence: 3})
err := refuseOlder(kept, nil, link.Order{Epoch: 41, Sequence: 12})
if err == nil || !strings.Contains(err.Error(), "epoch 41") || !strings.Contains(err.Error(), "epoch 57") {
t.Fatalf("an older epoch was not refused by name: %v", err)
}
if err := refuseOlder(kept, nil, link.Order{Epoch: 58, Sequence: 1}); err != nil {
t.Fatalf("a new lease holder's first declaration was refused for its sequence: %v", err)
}
if err := refuseOlder(kept, nil, link.Order{Epoch: 57, Sequence: 2}); err == nil {
t.Fatal("a lower sequence under the same epoch was accepted")
}
// A controller with no lease — older, or rolled back to a build from before it — is today's
// behaviour: no epoch compared.
if err := refuseOlder(kept, nil, link.Order{Sequence: 4}); err != nil {
t.Fatalf("a declaration with no epoch was refused for one: %v", err)
}
if held, older := olderThanKept(kept, nil, link.Order{Epoch: 41, Sequence: 12}); !older ||
held != (link.Order{Epoch: 57, Sequence: 3}) {
t.Fatalf("olderThanKept said %v, holding %+v", older, held)
}
// What is kept cannot be read: no order to compare, so nothing is refused on a guess.
if _, older := olderThanKept(store.Declared{Declaration: []byte("garbled")}, nil,
link.Order{Epoch: 41, Sequence: 1}); older {
t.Fatal("refused against a kept declaration nobody can read")
}
}
+154
View File
@@ -0,0 +1,154 @@
package main
import (
"context"
"errors"
"os/exec"
"sort"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// packageManagers are the tools a system's probe asks, for a system whose probe is its package manager.
var packageManagers = map[string]string{"arch": "pacman", "alpine": "apk"}
// unanswered is a probe of the package manager that failed before an apply: which tool, and its words.
type unanswered struct {
tool string
words string
}
// failedKey is how a report names the probe among what failed: one key, the same on every report, so the
// mesh counts a package manager that keeps not answering as one failure that goes on.
func (u *unanswered) failedKey() string { return "the package manager (" + u.tool + ")" }
// said is the line the apply says before it begins.
func (u *unanswered) said() string {
return u.tool + " does not answer here, so every package in this declaration fails and the rest of it is " +
"applied (novox/hq issue 233): " + u.words
}
// partial is the failure of an apply that applied everything else, when no package was declared to fail.
func (u *unanswered) partial() error {
return errors.New("applied only in part: " + u.words + "\n\nevery resource that does not need " + u.tool +
" was attempted; none here needed it, and " + u.tool + " must answer before this machine applies whole " +
"(novox/hq issue 233)")
}
// probePackageManager proves the machine is the system the host was built for, before anything is applied.
//
// **A package manager that does not answer fails only what needs it** (novox/hq issue 233). The probe was a
// refusal of the whole declaration — right for a host installed on a machine it was not built for, and a
// wedge when what the probe reads is the mesh's own: a declaration that left the package manager's module
// out removed its configuration file, and from then on every declaration was refused, the one that would
// write the file back and the node-engine's own upgrade included. A person on the machine was the only way
// back. Now the rest applies — files, units, accounts, the node-engine itself — and each package fails with
// the probe's words (the system's PackageInstalled asks the same probe first), so a later declaration can
// undo what an earlier one did. Before the rest is applied, a declaration composed from an older view of the
// assignments than this machine applied has already been refused (issue 234's generation).
//
// **A package manager that is not there at all still refuses whole.** That is the host installed on a
// machine it was not built for — the arch host on Alpine — which is what the probe exists to catch, and no
// declaration the mesh sends removes a package manager's program. Only one that is there and errors (a
// configuration file gone, a database locked) is the mesh's own fault and applies in part.
//
// A system with no package manager (android) still refuses whole: what its probe asks is the system itself,
// which no resource the mesh declares can remove, and nothing else stands between it and the wrong machine.
func probePackageManager(ctx context.Context, sys system.System) (*unanswered, error) {
err := sys.Confirm(ctx, apply.ExecRunner)
if err == nil {
return nil, nil
}
tool, ok := packageManagers[sys.Name()]
if !ok || errors.Is(err, exec.ErrNotFound) {
return nil, err
}
return &unanswered{tool: tool, words: err.Error()}, nil
}
// applyPass is one apply of the declaration over what the node's state says.
type applyPass func(known store.State) (apply.Report, store.State, error)
// applyMindingProbe applies, and **once more when the package manager did not answer and this apply wrote
// something** (novox/hq issue 233). What it wrote may be what the package manager needed — the declaration
// that names its module again writes its configuration file back — and then the packages go in now rather
// than at the next reconcile. Asked again only then: an apply that wrote nothing changed nothing the package
// manager reads. The second pass is the same declaration over the state the first left, so what the first
// applied reads unchanged and only what needs the package manager moves.
//
// It returns the last pass's account, the probe as it stands after it (nil once the package manager
// answers), and the first pass's outcomes when a second ran, for whoever counts what an apply moved.
func applyMindingProbe(ctx context.Context, sys system.System, probe *unanswered, known store.State,
pass applyPass, say func(string)) (apply.Report, store.State, error, *unanswered, []apply.Outcome) {
report, updated, err := pass(known)
if probe == nil || !wrote(report, err) {
return report, updated, err, probe, nil
}
again, refused := probePackageManager(ctx, sys)
if refused != nil || again != nil {
return report, updated, err, probe, nil
}
say(probe.tool + " answers now that this apply wrote what it reads; applying again, so its packages are " +
"installed in this apply (novox/hq issue 233)")
first := append([]apply.Outcome(nil), report.Outcomes...)
var partial *apply.Error
if errors.As(err, &partial) {
first = append(first, partial.Done.Outcomes...)
}
report, updated, err = pass(updated)
return report, updated, err, nil, first
}
// wrote is whether an apply moved anything on the machine, failed or not.
func wrote(report apply.Report, err error) bool {
if report.Changed() {
return true
}
var partial *apply.Error
return errors.As(err, &partial) && partial.Done.Changed()
}
// unapplied is which declared resources an apply did not apply, failed or not attempted, sorted; nil for a
// clean one.
func unapplied(d *declaration.Declaration, report apply.Report, err error) []string {
if err == nil {
return nil
}
done := map[string]bool{}
for _, o := range report.Outcomes {
done[o.ID] = o.Action != "skipped"
}
var partial *apply.Error
if errors.As(err, &partial) {
for _, o := range partial.Done.Outcomes {
done[o.ID] = done[o.ID] || o.Action != "skipped"
}
}
var out []string
for _, r := range d.Resources {
if !done[r.Identity()] {
out = append(out, r.Identity())
}
}
sort.Strings(out)
if len(out) == 0 {
// Failed, and every resource has an account: what failed is no resource (the firewall found, a removal).
out = []string{"(apply)"}
}
return out
}
// withUnanswered adds the probe to what a report says failed.
func withUnanswered(failed map[string]string, u *unanswered) map[string]string {
if u == nil {
return failed
}
if failed == nil {
failed = map[string]string{}
}
failed[u.failedKey()] = u.said()
return failed
}
+128
View File
@@ -0,0 +1,128 @@
package main
import (
"context"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/system"
)
// Defends novox/hq issue 233: a package manager that does not answer is said even when nothing declared
// needs it — the apply is partial, never "applied" — and the report names it under one key, so the mesh
// counts it as one failure that goes on.
func TestAPackageManagerThatDoesNotAnswerIsSaidWithNoPackageDeclared(t *testing.T) {
unansweringPacman(t)
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json"), out: &strings.Builder{}}
conf := filepath.Join(dir, "pacman.conf")
raw := []byte(`{"declaration":1,"resources":[{"id":"pacman.conf","type":"file","path":"` + conf +
`","content":"[options]\n"}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
err = runApply(context.Background(), opts, d, raw, fromFile)
if err == nil || !strings.Contains(err.Error(), "applied only in part") || !strings.Contains(err.Error(), "could not be read") {
t.Fatalf("an apply whose package manager did not answer was not said to be partial: %v", err)
}
if _, statErr := os.Stat(conf); statErr != nil {
t.Fatalf("the file was not written: %v", statErr)
}
report := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if report.Refused != "" {
t.Fatalf("refused whole: %s", report.Refused)
}
why, ok := report.Failed["the package manager (pacman)"]
if !ok || !strings.Contains(why, "could not be read") {
t.Errorf("the report does not name the package manager with the probe's words: %v", report.Failed)
}
}
// A system whose probe asks the system itself, not a tool a resource needs, still refuses whole: nothing
// the mesh declares can remove it, and nothing else stands between that host and the wrong machine.
func TestASystemWithNoPackageManagerStillRefusesWhole(t *testing.T) {
t.Setenv("PATH", t.TempDir())
sys, err := system.For("android")
if err != nil {
t.Fatal(err)
}
probe, err := probePackageManager(context.Background(), sys)
if err == nil || probe != nil {
t.Fatalf("the android host's failed probe did not refuse: %v, %+v", err, probe)
}
}
// pacmanAnsweringOnce is a pacman that does not answer until conf exists, as the workstation's did until its
// configuration file was written back, and installs into a directory of its own once it does.
func pacmanAnsweringOnce(t *testing.T, conf string) {
t.Helper()
bin, installed := t.TempDir(), t.TempDir()
script := "#!/bin/sh\n" +
"[ -f '" + conf + "' ] || { echo \"error: config file " + conf + " could not be read: No such file or directory\" >&2; exit 1; }\n" +
"for a; do last=$a; done\n" +
"case \"$1\" in\n" +
"-Q) [ \"$2\" = pacman ] && exit 0; [ -f '" + installed + "'/\"$2\" ] && exit 0; exit 1;;\n" +
"-S) touch '" + installed + "'/\"$last\"; exit 0;;\n" +
"esac\nexit 1\n"
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
}
// Defends the review of novox/hq issue 233: **the apply that writes the package manager's configuration back
// installs its packages too**. The probe is asked again once the partial apply wrote something, and when it
// answers the declaration is applied again over what the first pass left — the packages in this apply, not five
// minutes later, and the report clean.
func TestThePackagesGoInWhenTheApplyThatRestoresThePackageManagerWroteIt(t *testing.T) {
dir := t.TempDir()
conf := filepath.Join(dir, "pacman.conf")
pacmanAnsweringOnce(t, conf)
opts := options{state: filepath.Join(dir, "state.json")}
// The package first: within one pass it is asked about before the file is written, which is the order that
// leaves it to a second pass.
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"pacman.base","type":"package","package":"base"},` +
`{"id":"pacman.conf","type":"file","path":"` + conf + `","content":"[options]\n"}]}`)
report := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if report.Refused != "" || len(report.Failed) != 0 {
t.Fatalf("the apply that made the package manager answer did not install its packages: refused %q, failed %v",
report.Refused, report.Failed)
}
if !slices.Contains(report.Applied, "pacman.base") || !slices.Contains(report.Applied, "pacman.conf") {
t.Errorf("the report does not account for both: %v", report.Applied)
}
}
// Defends decision (1) of the review of novox/hq issue 233: **a package manager that is not there at all still
// refuses whole**. That is the host installed on the wrong machine, which the probe exists to catch.
func TestAPackageManagerThatIsNotThereRefusesWhole(t *testing.T) {
t.Setenv("PATH", t.TempDir())
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
dir := t.TempDir()
conf := filepath.Join(dir, "pacman.conf")
raw := []byte(`{"declaration":1,"resources":[{"id":"pacman.conf","type":"file","path":"` + conf +
`","content":"[options]\n"}]}`)
report := applyAndKeep(context.Background(), options{state: filepath.Join(dir, "state.json")}, raw, nil, nil, nil)
if !strings.Contains(report.Refused, "pacman does not answer") {
t.Fatalf("a machine with no pacman at all was not refused whole: %+v", report)
}
if _, err := os.Stat(conf); err == nil {
t.Error("the refused declaration touched the machine")
}
}
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+94
View File
@@ -0,0 +1,94 @@
package main
import (
"context"
"encoding/json"
"path/filepath"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/store"
)
// The record check reads the engine's applied state: a composition one of whose resources has no record there is not
// recorded (issue 1945).
func TestACompositionIsRecordedOnlyWhenEveryResourceIs(t *testing.T) {
path := filepath.Join(t.TempDir(), "state.json")
var s store.State
s.Record(store.Applied{ID: "ca-trust.trust", Type: "file", Target: "/etc/ca"})
if err := store.Save(path, s); err != nil {
t.Fatal(err)
}
c := delivery.Composed{Node: "ace", Module: "ca-trust", Resources: []json.RawMessage{
json.RawMessage(`{"id":"ca-trust.trust","type":"file","module":"ca-trust","path":"/etc/ca","mode":"0644","content":"x"}`)}}
recorded := recordedIn(path, func(string) {})
if !recorded(context.Background(), c) {
t.Fatal("a composition whose every resource has a record was not recorded")
}
c.Resources = append(c.Resources, json.RawMessage(
`{"id":"ca-trust.more","type":"file","module":"ca-trust","path":"/etc/ca2","mode":"0644","content":"y"}`))
if recorded(context.Background(), c) {
t.Fatal("a composition with a resource the state holds no record of was recorded")
}
}
// The modules applied again are those with a missing resource, each once; one down but there is not among them.
func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
st := liveness.Statement{Resources: []liveness.State{
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.homeserver"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.db"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "ca-trust", ID: "ca-trust.trust"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "web", ID: "web.server"}, State: liveness.Unhealthy},
}}
if got := missingModules(st); len(got) != 2 || got[0] != "ca-trust" || got[1] != "matrix" {
t.Fatalf("applied again: %v", got)
}
}
// A module is applied again only for a resource newly missing: one still missing at the next change of the statement is
// not acted on again, so a resource that cannot be made is not applied (and backed up) at every unrelated change. One
// that comes back and goes again is newly missing again (the review of issue 1945).
func TestOnlyANewlyMissingResourceAppliesItsModuleAgain(t *testing.T) {
missing := func(ids ...string) liveness.Statement {
var st liveness.Statement
for _, id := range ids {
st.Resources = append(st.Resources, liveness.State{Resource: liveness.Resource{Module: "matrix", ID: id},
State: liveness.Unhealthy, Missing: true})
}
return st
}
var last map[string]bool
var got []string
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 1 {
t.Fatalf("first missing: %v", got)
}
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 0 {
t.Fatalf("still missing was acted on again: %v", got)
}
got, last = newlyMissing(last, missing())
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 1 {
t.Fatalf("missing again after it came back: %v", got)
}
}
// A file the provisioner client fills is left to it by the apply and has no record until the client writes it: it does
// not make its composition unrecorded (the review of issue 1945).
func TestAFilledFileDoesNotMakeItsCompositionUnrecorded(t *testing.T) {
path := filepath.Join(t.TempDir(), "state.json")
var s store.State
s.Record(store.Applied{ID: "ca-trust.trust", Type: "file", Target: "/etc/ca"})
if err := store.Save(path, s); err != nil {
t.Fatal(err)
}
c := delivery.Composed{Node: "ace", Module: "ca-trust", Filled: []string{"/etc/ca-filled"}, Resources: []json.RawMessage{
json.RawMessage(`{"id":"ca-trust.trust","type":"file","module":"ca-trust","path":"/etc/ca","mode":"0644","content":"x"}`),
json.RawMessage(`{"id":"ca-trust.filled","type":"file","module":"ca-trust","path":"/etc/ca-filled","mode":"0644","owner-at-runtime":"owned"}`)}}
if !recordedIn(path, func(string) {})(context.Background(), c) {
t.Fatal("a filled file the client has not written made its composition unrecorded")
}
}
+109
View File
@@ -0,0 +1,109 @@
package main
import (
"bytes"
"context"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// unansweringPacman puts a pacman on PATH that answers as one whose configuration file is gone, as the
// workstation's did on 2026-10-04 (novox/hq issue 233), and makes this an arch host.
func unansweringPacman(t *testing.T) {
t.Helper()
bin := t.TempDir()
script := "#!/bin/sh\necho 'error: config file /etc/pacman.conf could not be read: No such file or directory' >&2\nexit 1\n"
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
}
// TestReplay233 replays novox/hq issue 233. The package manager's module wrote pacman's configuration
// file; a declaration that left the module out removed it; from then on pacman did not answer, and the
// node-engine refused every declaration whole — the one that named the module again and would have
// written the file back included. A person restored the file by hand an hour and a half later.
//
// Held to: a declaration that restores the file is applied while pacman does not answer. Only what needs
// pacman fails, with the probe's words; the file is written and recorded. Written only with what the
// node-engine had before the fix, so it lays over the commit before it, where it fails.
func TestReplay233(t *testing.T) {
unansweringPacman(t)
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
conf := filepath.Join(dir, "pacman.conf")
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"pacman.conf","type":"file","path":"` + conf + `","content":"[options]\n"},` +
`{"id":"pacman.base","type":"package","package":"base"}]}`)
report := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if report.Refused != "" {
t.Fatalf("the declaration that restores the package manager's configuration was refused whole: %s", report.Refused)
}
if got, err := os.ReadFile(conf); err != nil || string(got) != "[options]\n" {
t.Fatalf("the file that restores the package manager was not written: %q, %v", got, err)
}
known, err := store.Load(opts.state)
if err != nil {
t.Fatal(err)
}
if _, ok := known.Find("pacman.conf"); !ok {
t.Error("the file written was not recorded, so the next apply would not know it is the mesh's")
}
words := strings.Join(failedWords(report.Failed), "\n")
if !strings.Contains(words, "pacman does not answer") || !strings.Contains(words, "could not be read") {
t.Errorf("the report does not say the package manager did not answer, in the probe's words: %q", words)
}
if !strings.Contains(words, "pacman.base") {
t.Errorf("the report does not name the package that was not applied: %q", words)
}
}
func failedWords(failed map[string]string) []string {
var out []string
for id, why := range failed {
out = append(out, id+": "+why)
}
return out
}
// Defends novox/hq issue 233 at the machine: `mesh-host apply` applies what does not need the package
// manager, and says the apply was partial with the probe's words — a non-zero exit, never "applied".
func TestApplyByHandAppliesWhatDoesNotNeedAPackageManagerThatDoesNotAnswer(t *testing.T) {
unansweringPacman(t)
dir := t.TempDir()
out := &bytes.Buffer{}
opts := options{state: filepath.Join(dir, "state.json"), out: out}
conf := filepath.Join(dir, "pacman.conf")
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"pacman.conf","type":"file","path":"` + conf + `","content":"[options]\n"},` +
`{"id":"pacman.base","type":"package","package":"base"}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
err = runApply(context.Background(), opts, d, raw, fromFile)
if _, statErr := os.Stat(conf); statErr != nil {
t.Fatalf("the file was not written: %v (the apply said %v)", statErr, err)
}
if err == nil {
t.Fatal("an apply whose package manager did not answer said it applied")
}
if !strings.Contains(err.Error(), "pacman does not answer") || !strings.Contains(err.Error(), "pacman.base") {
t.Errorf("the failure does not name the package or the probe's words: %v", err)
}
if !strings.Contains(out.String(), "pacman does not answer") {
t.Errorf("what was said before applying does not say the package manager does not answer:\n%s", out.String())
}
}
+20
View File
@@ -0,0 +1,20 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-host/internal/liveness"
)
// The restart count the engine says is its own since it began watching the resource, never since the state
// began (novox/hq issue 458).
func TestAnUnhealthyResourceSaysWhatItsRestartCountCounts(t *testing.T) {
r := liveness.State{Resource: liveness.Resource{ID: "messenger.own-runtime", Kind: "process",
Target: "messenger-runtime.service"}, State: liveness.Unhealthy, Reason: liveness.ReasonRestarting, Restarts: 3890}
said := unhealthyWords(r)
if !strings.Contains(said, "3890 automatic restart(s) since this engine began watching it") ||
strings.Contains(said, "counted") {
t.Fatalf("the restart count is not said as what it counts: %s", said)
}
}
+52 -1
View File
@@ -5,8 +5,8 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/accounts"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/units"
@@ -75,3 +75,54 @@ func TestTheStatementNamesTheAccountsManager(t *testing.T) {
}
}
}
// An account declared never to become root (novox/hq ADR 0266) is said with root never, under the contract
// that tells the controller this engine judges it; an account judged for its groups alone is not.
func TestTheStatementSaysAnAccountJudgedForRoot(t *testing.T) {
at := time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC)
as := &accounts.Statement{At: at, Accounts: []accounts.Verdict{
{Account: accounts.Account{Module: "claude-code", ID: "claude-code.agent", Name: "agent", Root: true},
State: accounts.Unhealthy, Reason: accounts.ReasonRoot + ": in the group docker, which grants root", Since: at},
{Account: accounts.Account{Module: "openrazer", ID: "openrazer.account", Name: "operator",
Groups: []string{"openrazer"}}, State: accounts.Healthy, Since: at},
}}
h := withAccounts(healthAsReported(liveness.Statement{At: at}, nil), as)
// At RootContract or later: an engine that states its sealing key to the vault says NodeKeyContract (ADR 0372, issue 786), one above
// VaultBoxContract (it opens the vault's box for the core's credentials, ADR 0349 §16, issue 755), one above HandedContract (design 57 P7), one above ProvisionerContract (novox/hq ADR
// 0349 §4), one above ADR 0349 §5's OwnershipContract.
if h.Contract < link.RootContract || link.RootContract != 3 || h.Contract != link.NodeKeyContract ||
link.NodeKeyContract != link.VaultBoxContract+1 || link.VaultBoxContract != link.HandedContract+1 ||
link.HandedContract != link.ProvisionerContract+1 || link.ProvisionerContract != link.OwnershipContract+1 || link.OwnershipContract != link.BusyContract+1 ||
link.BusyContract != link.OwnGroupContract+1 {
t.Fatalf("contract %d", h.Contract)
}
got := map[string]string{}
for _, r := range h.Resources {
got[r.Resource] = r.Root
}
if got["claude-code.agent"] != "never" || got["openrazer.account"] != "" {
t.Fatalf("root said: %v", got)
}
}
// A verdict waiting for the setuid search says how far it got, in the reason the root-free verb repeats and as
// fields (novox/hq issue 361); a mount point, never a path the walk visited or found.
func TestTheStatementSaysHowFarTheSetuidSearchGot(t *testing.T) {
at := time.Date(2026, 10, 10, 1, 40, 0, 0, time.UTC)
sp := &accounts.SearchProgress{Started: at, Filesystems: 3, Done: 1, Walking: "/home", Visited: 42, Found: 2,
Kept: at, Finished: []accounts.SearchedFilesystem{{Mount: "/", Took: 90 * time.Second}}}
as := &accounts.Statement{At: at, Accounts: []accounts.Verdict{{Account: accounts.Account{Module: "claude-code",
ID: "claude-code.agent", Name: "agent", Root: true}, State: accounts.Unknown,
Reason: accounts.ReasonPending + ": the search for setuid programs, started 2026-10-10 01:27, has not finished yet",
Since: at, Search: sp}}}
h := withAccounts(healthAsReported(liveness.Statement{At: at}, nil), as)
r := h.Resources[0]
if !strings.HasPrefix(r.Reason, accounts.ReasonPending) || !strings.Contains(r.Reason, "1 of 3 filesystems walked, walking /home, 42 entries visited") {
t.Fatalf("the reason: %s", r.Reason)
}
s := r.Search
if s == nil || s.Filesystems != 3 || s.Done != 1 || s.Walking != "/home" || s.Visited != 42 || s.Found != 2 ||
s.Kept == nil || len(s.Finished) != 1 || s.Finished[0].Mount != "/" || *s.Finished[0].TookSeconds != 90 {
t.Fatalf("the fields: %+v", s)
}
}
-64
View File
@@ -1,64 +0,0 @@
package main
import (
"os"
"path/filepath"
"reflect"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The last apply's report, kept beside the state until the mesh takes it (novox/hq issue 264).
// What a host that died between applying and reporting kept is read back by the next one exactly —
// the digest and the outcome, never a new account of the machine.
func TestTheKeptReportSurvivesTheHost(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
made := link.Report{Declared: "sha256:d1", Applied: []string{"a"}, Failed: map[string]string{"b": "no room"},
Host: "v2"}
if err := unsaidBeside(state).Keep(made); err != nil {
t.Fatal(err)
}
kept, ok, err := unsaidBeside(state).Pending()
if err != nil || !ok {
t.Fatalf("nothing kept: %v", err)
}
if !reflect.DeepEqual(kept, made) {
t.Fatalf("kept %+v, made %+v", kept, made)
}
// Taken about another declaration, it stays; taken about this one, it goes.
if err := unsaidBeside(state).Said("sha256:other"); err != nil {
t.Fatal(err)
}
if _, ok, _ := unsaidBeside(state).Pending(); !ok {
t.Fatal("a report about another declaration cleared this one")
}
if err := unsaidBeside(state).Said("sha256:d1"); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(store.UnsaidPath(state)); !os.IsNotExist(err) {
t.Fatalf("the report was taken and is still kept: %v", err)
}
}
// A node that never applied anything has nothing to say again; and an apply refused before its
// declaration was read clears an older report, which would otherwise be re-said after it.
func TestNothingKeptWhenNothingWasApplied(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
if _, ok, err := unsaidBeside(state).Pending(); ok || err != nil {
t.Fatalf("a node that applied nothing has a report to say: %v %v", ok, err)
}
if err := unsaidBeside(state).Keep(link.Report{Declared: "sha256:d1"}); err != nil {
t.Fatal(err)
}
if err := unsaidBeside(state).Keep(link.Report{Refused: "forged"}); err != nil {
t.Fatal(err)
}
if _, ok, _ := unsaidBeside(state).Pending(); ok {
t.Fatal("an older report is kept after a later declaration was refused")
}
}
+127
View File
@@ -0,0 +1,127 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"fmt"
"os"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
"github.com/novox/mesh-host/internal/userlist"
)
// The bridge of novox/hq ADR 0547 §8, wired: the bus node's engine writes the bus's user list alone from the newest
// composition of the bus module, under the apply's two locks as a filled file is written (filling.go), and says what
// it wrote in the bus module's have.
// userListFollower is the follower of the bus's user list for one link: the node's composition bucket, the keys the
// agent trusts, the write alone, and the have.
func userListFollower(opts options, session *link.Session, composedBucket string, trusted func() []ed25519.PublicKey,
say link.Announce, wrote func()) *userlist.Follower {
return userlist.New(userlist.Env{
Node: session.Membership.Node,
Composed: bucketWatch{session, composedBucket},
Trusted: trusted,
Resolve: func(ctx context.Context, raw json.RawMessage) ([]byte, error) {
return resolveUserList(session.WithLink(ctx), raw)
},
OnDisk: userListOnDisk,
Write: func(ctx context.Context, module string, raw json.RawMessage, body []byte) (string, bool, error) {
digest, changed, err := writeUserListAlone(session.WithLink(ctx), opts, raw, body, say)
if changed && wrote != nil {
wrote()
}
return digest, changed, err
},
Report: func(ctx context.Context, module string, w userlist.Written) {
session.NoteUserList(ctx, module, w)
},
// The vault's key from its answer to node-key: the loaded list's hashes are handed sealed to it (ADR 0547
// clause 9).
VaultKey: vaultKey.get,
Say: announceOr(say),
})
}
// userListFile is a user-list resource as the file it declares.
func userListFile(raw []byte) (*declaration.File, error) {
r, err := declaration.ParseResource(raw)
if err != nil {
return nil, fmt.Errorf("the user list is not a resource: %w", err)
}
f, ok := r.(*declaration.File)
if !ok {
return nil, fmt.Errorf("the user list is a %s, not a file", r.Kind())
}
return f, nil
}
// resolveUserList is the list a user-list resource holds: its content, or the object its digest names in this node's
// file bucket, fetched and verified (apply.ResolveFileReferences, ADR 0486 §1).
func resolveUserList(ctx context.Context, raw []byte) ([]byte, error) {
f, err := userListFile(raw)
if err != nil {
return nil, err
}
d := &declaration.Declaration{Version: declaration.Version, Resources: []declaration.Resource{f}}
if err := apply.ResolveFileReferences(ctx, d, link.FilesFrom(ctx)); err != nil {
return nil, fmt.Errorf("the user list could not be had by its digest: %w", err)
}
return []byte(d.Resources[0].(*declaration.File).Content), nil
}
// userListOnDisk is what the user list's file holds on this machine now; nil when there is none.
func userListOnDisk(raw json.RawMessage) ([]byte, error) {
f, err := userListFile(raw)
if err != nil {
return nil, err
}
body, err := os.ReadFile(f.Path)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
return body, err
}
// writeUserListAlone writes one user-list resource alone (apply.WriteAlone) with body as its content, the node's state
// read and saved under the apply's locks, and answers the digest of the file on the machine after.
func writeUserListAlone(ctx context.Context, opts options, raw []byte, body []byte, say link.Announce) (string, bool, error) {
declared, err := userListFile(raw)
if err != nil {
return "", false, err
}
file := *declared
file.Content, file.Digest, file.Size = string(body), "", 0
applying.Lock()
defer applying.Unlock()
unlock, err := store.Lock(opts.state, nil)
if err != nil {
return "", false, err
}
defer unlock()
known, err := store.Load(opts.state)
if err != nil {
return "", false, err
}
built, err := system.For(builtFor)
if err != nil {
return "", false, err
}
out, updated, writeErr := apply.WriteAlone(ctx, built, &file, known, apply.ExecRunner, announceOr(say))
// Saved whichever way it went, as an apply's: what was written is on the machine either way.
if err := store.Save(opts.state, updated); err != nil {
writeErr = errors.Join(writeErr, errors.New("written, and the node's state could not be saved: "+err.Error()))
}
digest := ""
if body, err := os.ReadFile(file.Path); err == nil {
digest = userlist.Digest(body)
}
return digest, writeErr == nil && out.Action != "unchanged", writeErr
}
+153
View File
@@ -0,0 +1,153 @@
package main
// Issue 755's proof on a real bus (novox/hq ADR 0349 §16): a core credential the vault sealed to this machine crosses
// the bus inside the declaration, which the controller carries and cannot open, and this engine opens it with its
// sealing key and writes the value. A listener on every subject keeps every byte that crossed; the value is never
// among them. Run against the throwaway bus the build module starts for the integration level (MESH_TEST_NATS).
import (
"context"
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/seal"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
const coreCredential = "the-provisioner-clients-bus-password-2025"
func TestNatsTheEngineFillsACoreCredentialFromTheVaultsBoxWithNoValueInTheClear(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
// This machine's sealing key, where the engine reads it.
state := filepath.Join(t.TempDir(), "state.json")
key, err := identity.GenerateSealingKey()
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(identity.SealingKeyPath(state), []byte(key.Private), 0o600); err != nil {
t.Fatal(err)
}
// Every byte that crosses.
tap, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer tap.Close()
var mu sync.Mutex
var crossed []string
if _, err := tap.Subscribe(">", func(m *nats.Msg) {
mu.Lock()
crossed = append(crossed, m.Subject+" "+string(m.Data))
mu.Unlock()
}); err != nil {
t.Fatal(err)
}
_ = tap.Flush()
// The engine hears the composition on a subject of the test (the node agent reads it from its bucket).
const subject = "mesh.test.vaultbox.composition"
engine, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer engine.Close()
heard, err := engine.SubscribeSync(subject)
if err != nil {
t.Fatal(err)
}
_ = engine.Flush()
// The vault's answer to the controller: the value in the ADR 0206 box, sealed to this machine's key. The
// controller carries it in the declaration as it is.
pub, err := key.PublicPEM()
if err != nil {
t.Fatal(err)
}
box, err := seal.Seal(coreCredential, pub)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "broker")
body, _ := json.Marshal(map[string]any{"declaration": 1, "resources": []map[string]any{
{"id": "need-provisioner-broker", "type": "file", "path": path, "vault-sealed": box}}})
controller, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer controller.Close()
if err := controller.Publish(subject, body); err != nil {
t.Fatal(err)
}
_ = controller.Flush()
msg, err := heard.NextMsg(5 * time.Second)
if err != nil {
t.Fatalf("the declaration did not reach the engine: %v", err)
}
d, err := declaration.Parse(msg.Data)
if err != nil {
t.Fatal(err)
}
sys, err := system.For("arch")
if err != nil {
t.Fatal(err)
}
if _, _, err := apply.Apply(context.Background(), sys, d, store.State{}, store.OriginCarried,
func(context.Context, string, ...string) (string, error) { return "", nil }, nil, sealOpener(state)); err != nil {
t.Fatal(err)
}
on, err := os.ReadFile(path)
if err != nil || string(on) != coreCredential {
t.Fatalf("the engine wrote %q, %v", on, err)
}
_ = tap.Flush()
time.Sleep(100 * time.Millisecond)
mu.Lock()
all := strings.Join(crossed, "\n")
mu.Unlock()
if !strings.Contains(all, subject) || !strings.Contains(all, "vault-sealed") {
t.Fatalf("the listener heard nothing of the declaration:\n%s", all)
}
for _, form := range []string{coreCredential, base64.StdEncoding.EncodeToString([]byte(coreCredential))} {
if strings.Contains(all, form) {
t.Fatal("the core credential crossed the bus in the clear")
}
}
}
// The engine opens a value the controller sealed (the NaCl box) as before, and the vault's box, with one key.
func TestTheEnginesOpenerOpensBothBoxes(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
key, _ := identity.GenerateSealingKey()
if err := os.WriteFile(identity.SealingKeyPath(state), []byte(key.Private), 0o600); err != nil {
t.Fatal(err)
}
open := sealOpener(state)
nacl, _ := identity.Seal(key.Public, []byte("old"))
if got, err := open(nacl); err != nil || string(got) != "old" {
t.Errorf("the controller's box opened to %q, %v", got, err)
}
pub, _ := key.PublicPEM()
box, _ := seal.Seal("new", pub)
raw, _ := json.Marshal(box)
if got, err := open(string(raw)); err != nil || string(got) != "new" {
t.Errorf("the vault's box opened to %q, %v", got, err)
}
}
+139
View File
@@ -0,0 +1,139 @@
package main
import (
"crypto/ed25519"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/agent"
)
func aFileOf(module, id string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": module + "." + id, "type": "file", "path": "/tmp/r6/" + module + "/" + id,
"content": "x", "module": module})
return raw
}
// What runs is the agent's compositions and the kept declaration's resources for each module the agent does not hold;
// the node's own parts are the kept declaration's until the agent holds the node's composition (ADR 0539 §9). One module
// under the agent must not make the rest invisible to the liveness, units, accounts, libraries, filling and the take act.
func TestWhatRunsIsTheAgentsCompositionsAndTheKeptRest(t *testing.T) {
kept := &declaration.Declaration{Identities: []declaration.Identity{{Module: "db", Name: "mesh-db"}},
Libraries: []declaration.Library{{Library: "media"}}}
everything := []delivery.Composed{
{Node: "n1", Module: "web", Resources: []json.RawMessage{aFileOf("web", "a")}},
{Node: "n1", Module: "db", Resources: []json.RawMessage{aFileOf("db", "a")}}, // still the kept declaration's
}
d, err := whatRuns(everything, false, func() (*declaration.Declaration, error) { return kept, nil })
if err != nil {
t.Fatal(err)
}
if len(d.Resources) != 2 || len(d.Identities) != 1 || len(d.Libraries) != 1 {
t.Fatalf("%d resources, %d identities, %d libraries: the kept modules and the node's kept parts are missing",
len(d.Resources), len(d.Identities), len(d.Libraries))
}
// Once the agent holds the node's composition, its parts are the node's.
node := delivery.Composed{Node: "n1", Own: delivery.NodeOwn{Identities: []declaration.Identity{{Module: "web", Name: "mesh-web"}}}}
d, err = whatRuns(append(everything, node), true, func() (*declaration.Declaration, error) { return nil, errors.New("not read") })
if err != nil {
t.Fatal(err)
}
if len(d.Identities) != 1 || d.Identities[0].Name != "mesh-web" || len(d.Libraries) != 0 {
t.Fatalf("the node's parts: %+v %+v", d.Identities, d.Libraries)
}
}
// What the take act, the judgement of directories and the filling read is what runs, not the frozen kept declaration.
func TestDeclaredNowReadsWhatRunsWhenTheAgentIsRunning(t *testing.T) {
setRunsHere(func(string, ed25519.PublicKey) (*declaration.Declaration, error) {
return &declaration.Declaration{Version: 1, Resources: nil, Provisioner: "from-agent"}, nil
})
defer setRunsHere(nil)
d, err := declaredNow("/nowhere", nil)
if err != nil || d.Provisioner != "from-agent" {
t.Fatalf("declaredNow read %+v (%v), not what runs", d, err)
}
}
// 1945/AC17 (unit): a resource the node's own composition shares with a running module's composition is declared once,
// the module's: the declaration joined of both names each file once, so the filling writes it; the node's own resources
// no module declares stay. (Live from 2026-10-25 12:43: "2 resources declare X" on every node.)
func TestTheNodesOwnCopyOfAModulesResourceIsLeftOut(t *testing.T) {
own := delivery.Composed{Node: "n1", Resources: []json.RawMessage{aFileOf("zsh", "rc"), aFileOf("mesh-wireguard", "conf")},
Filled: []string{"/tmp/r6/zsh/rc"}}
zsh := delivery.Composed{Node: "n1", Module: "zsh", FP: "src2:aa", Resources: []json.RawMessage{aFileOf("zsh", "rc")},
Filled: []string{"/tmp/r6/zsh/rc"}}
d, err := declarationOf(own, zsh)
if err != nil {
t.Fatal(err)
}
at := map[string]int{}
for _, r := range d.Resources {
at[r.Target()]++
}
if at["/tmp/r6/zsh/rc"] != 1 || at["/tmp/r6/mesh-wireguard/conf"] != 1 || len(d.Resources) != 2 {
t.Errorf("the joined declaration declares %v, want each path once", at)
}
if !d.Fills("/tmp/r6/zsh/rc") {
t.Errorf("the filled path is no longer named")
}
// The same at the node's own apply: what the others declare is theirs.
trimmed, dropped := ownWithout(own, []delivery.Composed{zsh})
if len(trimmed.Resources) != 1 || len(dropped) != 1 || dropped[0] != "zsh.rc (zsh)" {
t.Errorf("the node's own without zsh's: %d resources, dropped %v", len(trimmed.Resources), dropped)
}
if len(own.Resources) != 2 {
t.Errorf("ownWithout changed its argument")
}
// A node's own composition alone is all of it.
if d, err := declarationOf(own); err != nil || len(d.Resources) != 2 {
t.Errorf("the node's own alone: %v (%v)", d, err)
}
}
// 1945/AC18 (unit): the kept declaration's filled paths go to the module whose file is at each path, the node's own to
// the node's; and what runs from it, joined with the node's composition, names them filled.
func TestTheKeptDeclarationsFilledPathsGoToTheirModules(t *testing.T) {
raw := []byte(`{"resources":[
{"id":"claude-code.settings","type":"file","module":"claude-code","path":"/etc/cc.json"},
{"id":"screen-lock.conf","type":"file","module":"screen-lock","path":"/etc/lock.conf"},
{"id":"host.motd","type":"file","path":"/etc/motd"},
{"id":"zsh.rc","type":"file","module":"zsh","path":"/etc/zshrc"}],
"filled_files":["/etc/cc.json","/etc/lock.conf","/etc/motd"]}`)
by, filled := splitKept(raw)
if len(by["claude-code"]) != 1 || len(by[agent.NodeModule]) != 1 {
t.Fatalf("resources by module: %v", by)
}
if strings.Join(filled["claude-code"], ",") != "/etc/cc.json" || strings.Join(filled["screen-lock"], ",") != "/etc/lock.conf" ||
strings.Join(filled[agent.NodeModule], ",") != "/etc/motd" || len(filled["zsh"]) != 0 {
t.Fatalf("filled by module: %v", filled)
}
cc := delivery.Composed{Node: "n1", Module: "claude-code", Resources: by["claude-code"], Filled: filled["claude-code"]}
node := delivery.Composed{Node: "n1"}
d, err := whatRuns([]delivery.Composed{cc, node}, true, func() (*declaration.Declaration, error) { return nil, errors.New("not read") })
if err != nil || !d.Fills("/etc/cc.json") {
t.Fatalf("what runs does not name claude-code's file filled: %v (%v)", d, err)
}
if splitBy, _ := splitKept([]byte("not json")); splitBy != nil {
t.Errorf("a kept declaration that does not read split into %v", splitBy)
}
}
// 1945/AC17 (unit, review): a resource of the node's own that shares only its target with a module's (another id: a
// unit or a container of the same name) is not the module's, and is neither dropped nor left to be removed.
func TestASharedTargetWithAnotherIDIsNotDropped(t *testing.T) {
ownFile, _ := json.Marshal(map[string]any{"id": "host.zshrc", "type": "file", "path": "/tmp/r6/zsh/rc", "content": "x"})
own := delivery.Composed{Node: "n1", Resources: []json.RawMessage{ownFile}}
zsh := delivery.Composed{Node: "n1", Module: "zsh", FP: "src2:aa", Resources: []json.RawMessage{aFileOf("zsh", "rc")}}
trimmed, dropped := ownWithout(own, []delivery.Composed{zsh})
if len(dropped) != 0 || len(trimmed.Resources) != 1 {
t.Errorf("a resource sharing only its target was dropped: %v", dropped)
}
}
+32 -11
View File
@@ -41,8 +41,9 @@ func rolledBackHosts(statePath string) []string {
return upgrade.RolledBackVersions(verdicts)
}
// proving is this host waiting for the mesh to take a report it made under its own build — the
// evidence its launcher's trial waits for — and acting on it once.
// proving is this host waiting to write a have key under its own build — the evidence its launcher's trial waits for
// (novox/hq ADR 0539 §9, ADR 0227 §8: the component replaced is never the only witness of its successor) — and acting
// on it once.
type proving struct {
statePath string
version string
@@ -56,24 +57,44 @@ func unproved(statePath, version string) bool {
return err != nil || known != version
}
// heard is told every account of the machine the mesh has taken. The first one about a declaration,
// made by this build, proves it.
func (p *proving) heard(r link.Report) {
if r.Declared == "" || r.Refused != "" || r.Host != p.version {
return
}
// haveWritten is told when this build has written a have key, or found the bucket already holding what it would write.
// The first proves it. **A declaration's report does not**: a node-engine that receives none is not on trial for it, and
// one that cannot write its haves is what the launcher keeps rolling back.
func (p *proving) haveWritten() {
p.once.Do(func() {
retired, err := upgrade.Proved(p.statePath, upgrade.VersionsDir(""), p.version)
if err != nil {
p.say(fmt.Sprintf("this host reported under its own build %s, and proving it was not complete: %v", p.version, err))
p.say(fmt.Sprintf("this host wrote its first have under its own build %s, and proving it was not complete: %v", p.version, err))
}
if len(retired) > 0 {
p.say(fmt.Sprintf("host %s is proved; retired the host version(s) %s", p.version, strings.Join(retired, ", ")))
}
renewFallback(p.version, p.say)
renewLauncher(p.say)
})
}
// renewFallback makes the engine placed by hand — the launcher's fallback, and the `nox-mesh-host` on this
// machine's PATH — a copy of this proved build when it is not one (novox/hq issue 356). On the same
// evidence known-good is written on, and said when it happens or cannot: a stale command that answers
// for a build that is not running would otherwise stay until somebody typed it.
func renewFallback(version string, say func(string)) {
self, err := os.Executable()
if err != nil {
say("the engine placed by hand could not be renewed: this build does not know its own path: " + err.Error())
return
}
fallback := upgrade.FallbackPath()
renewed, err := upgrade.RenewFallback(self, fallback)
switch {
case err != nil:
say(fmt.Sprintf("the engine placed by hand at %s is not this build, and could not be renewed: %v", fallback, err))
case renewed:
say(fmt.Sprintf("renewed the engine placed by hand at %s to this build, %s: the command on this machine is the "+
"engine that runs it", fallback, version))
}
}
// The launcher, as the service manager runs it.
const (
launcherPath = upgrade.DefaultLibexec + "/launch"
@@ -124,7 +145,7 @@ func launcherReplaced(pid int, proc, path string) bool {
}
// watchWhatThisHostPlaced judges the core builds this host placed, until ctx ends (to-be 45 §8).
func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Queue, say func(string)) {
func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Session, say func(string)) {
host, _ := os.Hostname()
w := &witness.Watcher{
Root: apply.DaemonRoot(), Node: node, Host: host,
@@ -137,7 +158,7 @@ func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Queue
},
// Said now, not at the next report: the verdict is on every report from here, and the mesh
// is asked for one at once.
Concluded: func(link.Rollback) { queue.ReportAsked() },
Concluded: func(link.Rollback) { queue.FactsAsked(ctx) },
Say: say,
}
w.Watch(ctx)
+39 -20
View File
@@ -5,6 +5,7 @@ import (
"os/exec"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
@@ -12,31 +13,55 @@ import (
"github.com/novox/mesh-host/internal/upgrade"
)
// The first account the mesh takes from this build, about a declaration, proves it to the launcher;
// nothing else does — a refusal, a report about no declaration, another build's report (to-be 45 §8).
func TestOnlyAnAccountUnderThisBuildProvesIt(t *testing.T) {
// The first have this build writes proves it to the launcher; nothing else does — not a report about a declaration, not
// a facts event, not the haves of another build (novox/hq ADR 0539 §9, ADR 0227 §8). R6-A AC7.
func TestOnlyAHaveUnderThisBuildProvesIt(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
t.Setenv("MESH_HOST_LIBEXEC", t.TempDir())
// Never the machine's own engine placed by hand: proving renews it (issue 356).
fallback := filepath.Join(t.TempDir(), "nox-mesh-host")
if err := os.WriteFile(fallback, []byte("a stale engine placed by hand"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv(upgrade.FallbackVar, fallback)
var said []string
p := &proving{statePath: state, version: "2.0", say: func(s string) { said = append(said, s) }}
for _, r := range []link.Report{
{Host: "2.0", Refused: "no"},
{Host: "2.0"},
{Host: "1.0", Declared: "abc"},
} {
p.heard(r)
if !unproved(state, "2.0") {
t.Fatalf("%+v proved this build", r)
}
// Facts said, a link opened, a declaration of the old kind reported: none of it is a have.
if !unproved(state, "2.0") {
t.Fatal("a build that wrote no have is proved")
}
p.heard(link.Report{Host: "2.0", Declared: "abc"})
if got, _ := os.ReadFile(fallback); string(got) != "a stale engine placed by hand" {
t.Fatal("the engine placed by hand was renewed before any have")
}
// The session tells the proof when it has a have key saying what it would write.
session := &link.Session{HaveWritten: p.haveWritten}
_ = session
p.haveWritten()
if unproved(state, "2.0") {
t.Fatalf("an account of a declaration under this build did not prove it (%v)", said)
t.Fatalf("the first have under this build did not prove it (%v)", said)
}
if got, _ := upgrade.ReadKnownGood(upgrade.KnownGoodPath(state)); got != "2.0" {
t.Fatalf("known-good is %q", got)
}
// **Proved, the engine placed by hand becomes this build** (issue 356): the command on the machine's PATH is
// the engine that runs it, and that is said.
self, err := os.Executable()
if err != nil {
t.Fatal(err)
}
want, _ := os.ReadFile(self)
if got, _ := os.ReadFile(fallback); len(want) == 0 || string(got) != string(want) {
t.Fatalf("the engine placed by hand was not renewed to this build (%d bytes, want %d)", len(got), len(want))
}
renewed := false
for _, s := range said {
renewed = renewed || strings.Contains(s, "renewed the engine placed by hand at "+fallback)
}
if !renewed {
t.Fatalf("the renewal was not said: %q", said)
}
}
// Every verdict that stands is said, and a reconcile that has one says it unasked.
@@ -51,12 +76,6 @@ func TestAStandingRollbackIsSaidOnEveryReport(t *testing.T) {
got[0].Outcome != link.RolledBack {
t.Fatalf("what the report says is %+v", got)
}
if !worthSaying(link.Report{Rollbacks: got}) {
t.Fatal("a reconcile with a rollback standing does not say it")
}
if adoptionFingerprint(link.Report{Rollbacks: got}) == adoptionFingerprint(link.Report{}) {
t.Fatal("a rollback reached or ended is not news to the reconcile")
}
}
// The launcher running this host is known to have been replaced only when it certainly was: a shell
+26 -33
View File
@@ -2,51 +2,44 @@
## `foundation-first-node.lock`
What a machine must be before a mesh exists — the bootstrap in
[novox/hq `07-the-foundation.md`](https://git.novox.be/novox/hq), whole:
The one genesis bundle (novox/hq ADR 0331): what a machine must be before a mesh exists — the
bootstrap in [novox/hq `07-the-foundation.md`](https://git.novox.be/novox/hq), whole, on the bus the
mesh runs on:
```
0 a container runtime
0 a container runtime, and a filter before anything listens
1 the store runs
2 a database per context `inventory` and `identity`
2 a database per context `inventory`, `identity` and `licences`
3 those contexts' schemas mesh-controller migrate
4 the broker runs with a certificate it generated itself
5 the control plane runs mesh-controller serve
4 the bus's certificate mesh-controller broker certificate
5 the bus runs NATS as mesh-broker-nats, where the nats module keeps it
(ADR 0333), with the controller's account as its first user
6 the control plane runs mesh-controller serve, given the bus (MESH_BUS_NATS)
```
**A machine that applies this is a mesh** — one node, with nothing joined to it yet, which is
exactly what the first node is (novox/hq ADR 0004). From here it hands out tokens and everything
else joins the ordinary way.
This file said it stopped at step 4 for longer than that was true, which is its own small lesson:
a comment about what something does not do is a comment nobody updates.
It is a **template**: `mesh-bootstrap` reads it, makes the store's superuser and the controller's
account on the bus, and writes them where the template says `bootstrap` (internal/bootstrap,
rootsecrets.go). A template whose bus connection it does not recognise is refused at preflight,
naming the placeholder it looked for. The bundle that raised the previous broker was deleted with
ADR 0331; there is one bundle, and every lab bed names it.
Build a host carrying it:
**The bus is raised where the `nats` module keeps it** (novox/hq ADR 0333): the container
`mesh-broker-nats`, its configuration and user list in `/var/lib/nats-module/conf`, its streams in
`/var/lib/mesh-broker-nats`, its certificate in `/var/lib/mesh-broker-tls`, published on the node's bus
port. At the installer's broker step the module adopts it: its own image replaces the upstream one, once,
on the same streams, users and certificate. `internal/bootstrap` holds the bundle to the module's manifest.
Build an installer carrying a builder, and point it at this:
```
make host SYSTEM=arch BUNDLE=examples/foundation-first-node.lock
make bootstrap IMAGE=<builder image>
mesh-bootstrap --bundle examples/foundation-first-node.lock …
```
**The registry address and digests have to be replaced before this is useful.** They are written
as `192.0.2.250:5000/…@sha256:…` because a digest belongs to whatever registry serves it — here,
one a lab scenario raises, which reports its digests when it comes up. That is not a placeholder
to be tidied away: a bundle is built *for a target*, and which registry that target pulls from is
part of the target.
### What was verified, and how
On a lab machine confirmed sealed — `curl https://example.com` times out, the lab registry answers
200 — the whole bundle applied from bare: eight resources, `inventory` created and `mesh` nowhere,
the `node` table present with its indexes, the migration recorded, and LavinMQ answering
`lavinmqctl status` with AMQP listening on 5672.
Three consecutive reconciles after that: **already matches — 8 resource(s) checked**, each time.
Then the machine was **rebooted**, and everything came back: docker from `boot: enabled`, both
containers because the host creates every container `--restart unless-stopped`
(`internal/apply/apply.go`), the schema intact in its named volume, and reconcile still finding
nothing to do.
The reboot is worth doing rather than assuming. Nothing in the declaration asks for a container to
return, so that it does is a property of the host, and the only way to know it holds is to take
the machine away and give it back.
**The registry address is the lab's.** The images are written as `192.0.2.250:5000/…@sha256:…`; the
lab rewrites the third-party ones to upstream references by the same digest, and the installer
replaces the control plane's with the image it built.
+32 -2
View File
@@ -7,11 +7,12 @@ package examples
import (
"os"
"path/filepath"
"sort"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
func bundle(t *testing.T) *declaration.Declaration {
@@ -48,7 +49,7 @@ func TestTheBundleCarriesTheFoundationAndTheControlPlaneAndNothingElse(t *testin
}
sort.Strings(images)
want := []string{"lavinmq", "mesh-controller", "postgres"}
want := []string{"mesh-controller", "nats", "pgvector"}
if strings.Join(images, ",") != strings.Join(want, ",") {
t.Fatalf("the bundle carries %v; expected exactly %v.\n\n"+
"Adding one is a change to what every first node becomes, and to ADR 0006's "+
@@ -67,3 +68,32 @@ func TestTheControlPlaneIsCarriedToo(t *testing.T) {
t.Fatal("nothing in the bundle starts the control plane, so the machine would raise a " +
"foundation and stop")
}
// One genesis bundle, on the bus the mesh runs on (novox/hq ADR 0331 §2). Two bundles let every bed
// and README name the one that could not raise a mesh, for weeks (issue 146).
func TestThereIsOneGenesisBundle(t *testing.T) {
bundles, err := filepath.Glob("*.lock")
if err != nil {
t.Fatal(err)
}
if len(bundles) != 1 || bundles[0] != "foundation-first-node.lock" {
t.Fatalf("examples/ holds %v; it holds one genesis bundle, foundation-first-node.lock", bundles)
}
}
// The control plane the bundle raises is given the bus, with the account the installer fills and
// the certificate it pins (ADR 0331 §4); without it, it crash-loops for ever (issue 146).
func TestTheControlPlaneIsGivenTheBus(t *testing.T) {
for _, r := range bundle(t).Resources {
c, ok := r.(*declaration.Container)
if !ok || !strings.Contains(c.Image, "mesh-controller") {
continue
}
if !strings.HasPrefix(c.Env["MESH_BUS_NATS"], "nats://controller:bootstrap@") || c.Env["MESH_BROKER_CERTIFICATE"] == "" {
t.Fatalf("the control plane is given the bus %q and the certificate %q",
c.Env["MESH_BUS_NATS"], c.Env["MESH_BROKER_CERTIFICATE"])
}
return
}
t.Fatal("nothing in the bundle starts the control plane")
}
-200
View File
@@ -1,200 +0,0 @@
// foundation-first-node-nats.lock — what a machine must be before a mesh exists, on the bus being
// built (novox/hq ADR 0106, design 25).
//
// The same twelve steps as foundation-first-node.lock, with one difference that matters: **the mesh
// writes its own user list, and at genesis there is no mesh yet to write it.** So this carries the
// first one — the controller's own account, at a well-known bootstrap password, exactly as the store
// is reached at `postgres:bootstrap` and the old bus at `guest:guest`. It is rotated with those, and
// from the controller's first composition onward the file is the controller's to write.
//
// The accounts file is its own file beside the server's configuration, because the server's own
// settings belong to whoever raises it and the users belong to the mesh (design 25 §4). Both live in
// one directory, of necessity: an include path is resolved relative to the including file's own
// directory, so a server given an absolute one looks for it underneath that directory and refuses to
// start.
//
// No `verify` on the TLS block, deliberately — that setting makes the server demand a *client*
// certificate, and nothing in the mesh presents one: a host pins this server's exact certificate and
// authenticates with a password (ADR 0004, design 25 §4).
{
"declaration": 1,
"resources": [
{
"id": "container-runtime",
"type": "package",
"package": "docker"
},
{
"id": "container-runtime-running",
"type": "service",
"unit": "docker.service",
"state": "running",
"boot": "enabled"
},
// **A filter before anything listens** (novox/hq issue 054, ADR 0088). The store and the
// broker are adopted as modules later and so bind to every interface from the moment they
// start; the packet filter that governs who may reach them is a module too, installed a
// dozen steps later. Between the two, a control-node facing the network had its store and
// its bus open to anyone who could reach the machine. So the foundation carries a filter of
// its own — the same table the filter module will replace wholesale once it can derive one:
// drop by default, keep loopback, replies, ssh and the mesh's own ports (the bus a node
// enrols over, the registry a node pulls from), and let the container runtime's own
// networks through the forward chain so containers keep working. A published container port
// is forwarded, never input (issue 047), which is why the forward chain is where the store's
// and broker's ports are refused from outside — and a container on this machine dialling a
// port this machine publishes reaches it through the runtime's proxy, which IS input, which
// is why the bus and the registry are opened in both chains, exactly as the derived ruleset
// does.
{
"id": "base-filter-package",
"type": "package",
"package": "nftables"
},
{
"id": "base-filter",
"type": "file",
"path": "/etc/nftables.conf",
"mode": "0644",
"content": "#!/usr/sbin/nft -f\n# the foundation's own filter, until the mesh derives one (novox/hq issue 054)\ntable inet mesh {}\ndelete table inet mesh\n\ntable inet mesh {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\tiif lo accept\n\t\ticmp type echo-request accept\n\t\ticmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept\n\t\t# ssh, from anywhere — never closed\n\t\ttcp dport 22 accept\n\t\t# the mesh's own, from anywhere: the bus a node enrols over and a container on this machine reaches through the proxy, the registry a node pulls from\n\t\ttcp dport 5671 accept\n\t\ttcp dport 5000 accept\n\t}\n\tchain output {\n\t\ttype filter hook output priority filter; policy accept;\n\t}\n\tchain forward {\n\t\ttype filter hook forward priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\t# the container runtime's bridge networks, and the networks its compose files are given\n\t\tip saddr 172.16.0.0/12 accept\n\t\tip saddr 192.168.128.0/17 accept\n\t\t# the mesh's own: the bus a node enrols over, the registry a node pulls from\n\t\tct original proto-dst 5671 accept\n\t\tct original proto-dst 5000 accept\n\t}\n}\n"
},
{
"id": "base-filter-loaded",
"type": "service",
"unit": "nftables.service",
"state": "running",
"boot": "enabled",
"restart-on": ["base-filter"]
},
{
"id": "store",
"type": "container",
"name": "mesh-store",
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
"env": {
"POSTGRES_PASSWORD": "bootstrap",
"PGDATA": "/var/lib/postgresql/data/pgdata"
},
"ports": ["5432:5432"],
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
},
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
// exits happy, and the verify a moment later lands in the gap and fails. The action and its
// verify must ask the same question, or the action can succeed into a state verify rejects.
{
"id": "store-ready",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
},
{
"id": "inventory-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
},
{
"id": "identity-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
},
// Each context owns its own database (novox/hq ADR 0008). A third one is a third database,
// created the same way and named the same way — which is the whole of adding a context to the
// bootstrap, and is why the count is not something the foundation has an opinion about.
{
"id": "licences-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
},
{
"id": "context-schemas",
"type": "action",
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"migrate"],
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
},
{
"id": "bus-certificate",
"type": "action",
// **The mesh makes its own** (novox/hq 04-ISSUES/146). This ran `openssl` inside the
// broker's image while the broker was one that carried it; the bus that replaced it has a
// shell and no openssl, and no other image the bundle names has one either. So the program
// that needs the certificate writes it — already on this machine, since the schema step ran
// it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
// this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
// to ask an authority of.
// `--user 0:0` because the volume is root's and this image runs as nobody, which is right
// for the long-running control plane and wrong for a one-shot writing into a fresh volume.
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--into", "/tls"],
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--check", "--into", "/tls"]
},
{
"id": "bus-conf-dir",
"type": "directory",
"path": "/var/lib/mesh-bus-conf",
"mode": "0700"
},
{
"id": "bus-conf",
"type": "file",
"path": "/var/lib/mesh-bus-conf/nats.conf",
"mode": "0644",
"content": "port: 4222\nhttp: 127.0.0.1:8222\n\ntls {\n cert_file: \"/tls/tls.crt\"\n key_file: \"/tls/tls.key\"\n}\n\njetstream {\n store_dir: \"/data\"\n}\n\ninclude accounts.conf\n"
},
{
"id": "bus-accounts",
"type": "file",
"path": "/var/lib/mesh-bus-conf/accounts.conf",
"mode": "0600",
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
{
"id": "broker",
"type": "container",
"name": "mesh-broker",
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
"volumes": ["mesh-broker-data:/data", "mesh-broker-tls:/tls:ro", "/var/lib/mesh-bus-conf:/etc/nats:ro"],
"args": ["-c", "/etc/nats/nats.conf", "-js"]
},
{
"id": "broker-ready",
"type": "action",
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
},
{
"id": "control-plane",
"type": "container",
"name": "mesh-controller",
"image": "192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"network": "host",
"args": ["serve"],
"volumes": ["mesh-broker-tls:/broker-tls:ro"],
"env": {
"MESH_STORE_INVENTORY": "postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"MESH_STORE_IDENTITY": "postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"MESH_STORE_LICENCES": "postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"MESH_BUS_NATS": "nats://controller:bootstrap@127.0.0.1:5671",
"MESH_BROKER_ADDRESS": "192.0.2.10:5671",
"MESH_BROKER_CERTIFICATE": "/broker-tls/tls.crt"
}
}
]
}
+134 -54
View File
@@ -1,36 +1,40 @@
// foundation-first-node.lock — what a machine must be before a mesh exists.
// foundation-first-node.lock — what a machine must be before a mesh exists: the one genesis bundle
// (novox/hq ADR 0331 §2), on the bus the mesh runs on (ADR 0106, ADR 0131, design 25).
//
// The whole bootstrap (novox/hq 03-DESIGN/01-to-be/07-the-foundation.md): a container runtime, a
// store, a database per context, those contexts' schemas, the broker, and the control plane
// running on top of them.
// filter, a store, a database per context, those contexts' schemas, the bus with its certificate
// and its first user list, and the control plane running on top of them.
//
// It stopped before the control plane once, and this comment said so for longer than it was true.
// **The mesh writes its own user list, and at genesis there is no mesh yet to write it.** So this
// carries the first one — the controller's own account — and the control plane's address names the
// same account. Both say `bootstrap`, and both are placeholders: the installer makes the account's
// password, puts it in the address and its hash in the list (mesh-host internal/bootstrap
// rootsecrets.go), and refuses a bundle whose bus connection it does not recognise. Until the first
// node has enrolled the installer places the list the controller composes; from the bus's adoption
// as the `nats` module on, the module receives it like any module (ADR 0331 §1, §5).
//
// The broker generates its OWN certificate, in its own image, into a volume it then mounts read
// only. Self-signed, because at this moment there is no mesh to issue one and no public name to
// obtain one for -- and it does not matter, because what a joining node checks is the fingerprint
// pinned in its token, not a chain or a name (novox/hq ADR 0004). The subject is decoration.
// The accounts file is its own file beside the server's configuration, because the server's own
// settings belong to whoever raises it and the users belong to the mesh (design 25 §4). Both live in
// one directory, of necessity: an include path is resolved relative to the including file's own
// directory, so a server given an absolute one looks for it underneath that directory and refuses to
// start.
//
// No `verify` on the TLS block, deliberately — that setting makes the server demand a *client*
// certificate, and nothing in the mesh presents one: a host pins this server's exact certificate and
// authenticates with a password (ADR 0004, design 25 §4). The control plane pins it too, through
// MESH_BROKER_CERTIFICATE, and dials the bus on the machine's own loopback.
//
// PINNED BY DIGEST, and the digest is not decoration: a tag can be made to point at a different
// image, and this file is applied on a machine with no mesh to ask about anything. These digests
// belong to the registry the lab raises, which is what a real node pulls from anyway — what is
// required is a reference that is exact and cannot move (novox/hq ADR 0006).
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
// registry prefix is the lab's; the digests are upstream's.
//
// The store waits up to three minutes rather than one. A machine that has just pulled the
// image and is running initdb for the first time can take longer than sixty seconds, and it
// failed that way three times in the lab -- a flaky bootstrap that a second run always fixed,
// which is the worst kind because it teaches people to run things twice.
//
// It failed a fourth time on 2026-08-30, on a loaded machine, and the report was `docker exited
// 1:` with nothing after the colon. Raising the timeout again would treat the symptom; what makes
// a retry the only available response is a timeout that reports nothing. So the wait now says
// what it saw before giving up.
//
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
// creates is owned by root, and the database runs as somebody else inside the container — so it
// could not write, and the container crash-looped. A named volume lets the image set up its own
// ownership, and outlives the container, which is what you want for the thing holding the mesh's
// state.
// **The store is raised where the `postgres` module keeps it** (novox/hq ADR 0338): the container
// `postgres`, on the module's pgvector image, its data in the directory /var/lib/mesh-store owned by
// the image's postgres user, its superuser password read from the module's own-secret file — so the
// store step adopts it as it runs. Pinned upstream by the module's own digest, not through the lab's
// registry prefix, because the two must be the same reference (mesh-host internal/bootstrap holds
// them to each other against the module's manifest).
{
"declaration": 1,
"resources": [
@@ -80,17 +84,28 @@
"boot": "enabled",
"restart-on": ["base-filter"]
},
// The store's data is a directory owned by the image's postgres user (999), made before the
// container starts: a directory the runtime made would be root's, and the database could not
// write it. Measured on the module's image 2026-10-13: it initialises a new store on a directory
// made so (novox/hq ADR 0338 §5).
{
"id": "store-data",
"type": "directory",
"path": "/var/lib/mesh-store",
"mode": "0700",
"owner": "999:70"
},
{
"id": "store",
"type": "container",
"name": "mesh-store",
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
"name": "postgres",
"image": "pgvector/pgvector@sha256:cf134a767f474095eeba57e0117be8e568e011a63f33fbf252f14c9b760f8e6f",
"env": {
"POSTGRES_PASSWORD": "bootstrap",
"PGDATA": "/var/lib/postgresql/data/pgdata"
},
"ports": ["5432:5432"],
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
"volumes": ["/var/lib/mesh-store:/var/lib/postgresql/data"]
},
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
@@ -99,21 +114,21 @@
{
"id": "store-ready",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
},
{
"id": "inventory-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
},
{
"id": "identity-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
},
@@ -123,46 +138,112 @@
{
"id": "licences-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
},
{
"id": "context-schemas",
"type": "action",
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
"command": ["docker", "run", "--rm", "--network", "container:postgres",
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"migrate"],
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
"verify": ["sh", "-c", "docker exec postgres psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec postgres psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec postgres psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
},
// **The bus is raised where the `nats` module keeps it** (novox/hq ADR 0333 §1): the container
// `mesh-broker-nats`, its configuration and user list in /var/lib/nats-module/conf, its streams in
// /var/lib/mesh-broker-nats and its certificate in /var/lib/mesh-broker-tls — the names and paths the
// catalogue's module declares. So adopting it as that module at the broker step keeps it: the
// module's built image takes the place of the upstream one below, once, on the same streams, users
// and certificate, and nothing else moves. mesh-host's internal/bootstrap holds the two to each other
// against the module's own manifest (busLayout), at the broker step and in its tests.
//
// Directories on the machine, not volumes the runtime names: the module mounts these paths, and the
// control plane, once it runs as the mesh's process, reads the certificate at its path on the machine.
// The certificate's directory is readable by everyone because the certificate is public and the control
// plane runs as nobody; its key is written 0600 by the program that makes it.
{
"id": "bus-tls-dir",
"type": "directory",
"path": "/var/lib/mesh-broker-tls",
"mode": "0755"
},
{
"id": "broker-certificate",
"type": "action",
"command": ["docker", "run", "--rm", "--entrypoint", "sh", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/cloudamqp/lavinmq@sha256:b117c254e6e269a29db479e6b410ca4e46e035b4981e49d24b159673ef09d336",
"-c", "test -f /tls/tls.crt || (openssl req -x509 -newkey rsa:2048 -nodes -keyout /tls/tls.key -out /tls/tls.crt -days 3650 -subj '/CN=mesh-broker' >/dev/null 2>&1 && chmod 644 /tls/tls.crt && chmod 600 /tls/tls.key)"],
"verify": ["docker", "run", "--rm", "--entrypoint", "sh", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/cloudamqp/lavinmq@sha256:b117c254e6e269a29db479e6b410ca4e46e035b4981e49d24b159673ef09d336",
"-c", "test -s /tls/tls.crt && openssl x509 -in /tls/tls.crt -noout"]
"id": "bus-data-dir",
"type": "directory",
"path": "/var/lib/mesh-broker-nats",
"mode": "0700"
},
{
"id": "bus-certificate",
"type": "action",
// **The mesh makes its own** (novox/hq 04-ISSUES/146). This ran `openssl` inside the
// broker's image while the broker was one that carried it; the bus that replaced it has a
// shell and no openssl, and no other image the bundle names has one either. So the program
// that needs the certificate writes it — already on this machine, since the schema step ran
// it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
// this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
// to ask an authority of.
// `--user 0:0` because the directory is root's and this image runs as nobody, which is right
// for the long-running control plane and wrong for a one-shot writing into a fresh directory.
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "/var/lib/mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--into", "/tls"],
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "/var/lib/mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--check", "--into", "/tls"]
},
// **Seeds, written only where nothing is** (create-once). The bus's configuration and user list are
// the bundle's until the bus is adopted, and the module's from then on (ADR 0333 §4): an installer
// run again after the adoption applies this bundle again, and an ordinary file would put the
// bundle's one user back over every user the mesh composed, and leave the node unable to reach its
// bus. Until the adoption the installer places the composed list over the seed (enrol.go).
{
"id": "bus-conf-dir",
"type": "directory",
"path": "/var/lib/nats-module/conf",
"mode": "0700"
},
{
"id": "bus-conf",
"type": "file",
"path": "/var/lib/nats-module/conf/nats.conf",
"mode": "0644",
"create-once": true,
"content": "port: 4222\nhttp: 0.0.0.0:8222\n\ntls {\n cert_file: \"/tls/tls.crt\"\n key_file: \"/tls/tls.key\"\n}\n\njetstream {\n store_dir: \"/data\"\n}\n\ninclude accounts.conf\n"
},
{
"id": "bus-accounts",
"type": "file",
"path": "/var/lib/nats-module/conf/accounts.conf",
"mode": "0600",
"create-once": true,
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_assignments.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_declared-config.>\", \"$KV.mesh-controller_lease.>\", \"$KV.mesh-controller_predefined.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.assignment-changed\", \"mesh.seat.mesh-controller.event.declared-config-changed\", \"mesh.seat.mesh-controller.event.predefined-changed\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.operator-channel.tool.tell\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
// The upstream server, until the broker step adopts it: the `nats` module's own image, built by the
// mesh, takes its place there once, on these same directories (ADR 0333 §3), as the bus's planned
// step (ADR 0236). The release is the one the module's image is built on (its NATS_BASE, 2.11.17),
// pinned upstream by that same digest: the adoption changes no server release, so it can be undone
// by putting the old image back and is said reversible. The published port is the
// node's bus port, 5671 unless given, which every enrolment token names; the installer gives the
// module the setting that maps its declared 4222 to it (ADR 0333 §2).
{
"id": "broker",
"type": "container",
"name": "mesh-broker",
"image": "192.0.2.250:5000/cloudamqp/lavinmq@sha256:b117c254e6e269a29db479e6b410ca4e46e035b4981e49d24b159673ef09d336",
"ports": ["5671:5671", "5672:5672", "127.0.0.1:15672:15672"],
"volumes": ["mesh-broker-data:/var/lib/lavinmq", "mesh-broker-tls:/tls:ro"],
"args": ["--amqps-port=5671", "--cert=/tls/tls.crt", "--key=/tls/tls.key"]
"name": "mesh-broker-nats",
"image": "nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00",
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
"volumes": ["/var/lib/mesh-broker-nats:/data", "/var/lib/nats-module/conf:/etc/nats:ro", "/var/lib/mesh-broker-tls:/tls:ro"],
"args": ["-c", "/etc/nats/nats.conf", "-js"]
},
{
"id": "broker-ready",
"type": "action",
"in": "mesh-broker",
"command": ["sh", "-c", "for i in $(seq 1 60); do lavinmqctl status >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["lavinmqctl", "status"]
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671'"]
},
{
"id": "control-plane",
@@ -171,13 +252,12 @@
"image": "192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"network": "host",
"args": ["serve"],
"volumes": ["mesh-broker-tls:/broker-tls:ro"],
"volumes": ["/var/lib/mesh-broker-tls:/broker-tls:ro"],
"env": {
"MESH_STORE_INVENTORY": "postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"MESH_STORE_IDENTITY": "postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"MESH_STORE_LICENCES": "postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"MESH_BROKER_AMQP": "amqp://guest:guest@127.0.0.1:5672/",
"MESH_BROKER_MANAGEMENT": "http://guest:guest@127.0.0.1:15672",
"MESH_BUS_NATS": "nats://controller:bootstrap@127.0.0.1:5671",
"MESH_BROKER_ADDRESS": "192.0.2.10:5671",
"MESH_BROKER_CERTIFICATE": "/broker-tls/tls.crt"
}
+8 -1
View File
@@ -3,13 +3,20 @@ module github.com/novox/mesh-host
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
golang.org/x/sys v0.48.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+21
View File
@@ -1,5 +1,23 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f h1:lUEyNPPF3w4Q0LIgcUUD6ofkKVhCF0n7dVjMoixQmXo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456 h1:He2o9vSKjgKbWOCOgOJY0WlaDh1n8RcGhvGY/dwXsBQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e h1:k/1NjjW5HuuuHW2ldyWTvjp+1u3fN0RO18Q57Iup6Dk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936 h1:HGBoWimZXYZp+Qn9VCstYDitOP84t9EUQYd1yM9j2eg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
@@ -8,5 +26,8 @@ github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+97 -14
View File
@@ -21,6 +21,16 @@
// controller raises it as the module's condition on two statements in a row, and clears it on the first
// healthy one.
//
// **And an account declared never to become root without a person** (`root: never`, novox/hq ADR 0266) —
// the account agent sessions run as on a machine where they must not reach root by themselves — is judged
// on every look for whether it can anyway: by its uid, by a group that grants root to its members, by any
// sudo rule at all, or by reading a secret the mesh placed for another account (the tool runner's bus
// credential among them, which carries every co-hosted module's grants). Judged first, whether or not
// anybody is logged in: a way to root does not wait for a session. Any way found is unhealthy, its reason
// starting ReasonRoot and naming every way; a question the machine did not answer is unknown, never
// healthy. This is the fact the controller reads to tell a machine where no agent can become root from one
// where an agent can.
//
// **It reads; it never acts** (ADR 0240 rule 6): the user and group databases, the machine's own service
// manager's `show` of the account's manager unit, and that process's status file. It never starts the
// account's manager, which asking that manager itself would.
@@ -28,13 +38,14 @@ package accounts
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// The states, in the words liveness says them (the controller reads them alike).
@@ -47,6 +58,19 @@ const (
// ReasonRelogin starts the reason of an account whose running session lacks a group it is in.
const ReasonRelogin = "relogin needed"
// ReasonRoot starts the reason of an account declared never to become root without a person that can
// (novox/hq ADR 0266); every way found follows it.
const ReasonRoot = "can become root without a person"
// RootGroups are the groups whose members become root by membership alone, without a sudo rule naming
// them: the administrators' groups a distribution's own rules or polkit treat as root, the container
// runtime's socket (a container with the host's root mounted), the raw disks, and the virtualisation
// daemons that start a guest with the host's devices — and the kernel's memory (kmem) and the password hashes
// (shadow), either of which is root a step away (novox/hq ADR 0312 §4 review). A closed list, so what the judge
// calls "grants root" is written down and reviewable rather than guessed per machine.
var RootGroups = []string{"root", "wheel", "sudo", "admin", "docker", "disk", "lxd", "incus-admin", "libvirt",
"kmem", "shadow"}
// Account is one user resource of a module that declares groups.
type Account struct {
Module string
@@ -54,6 +78,11 @@ type Account struct {
ID string
Name string
Groups []string
// Root is true for an account declared never to become root without a person (novox/hq ADR 0266).
Root bool
// Secrets are the paths of every secret the declaration places for another account, judged for
// whether this one can read them; only for a Root account.
Secrets []string
}
// Of is every account a declaration has a module put in a group. held is every resource an adopted
@@ -66,20 +95,39 @@ func Of(d *declaration.Declaration, held map[string]bool) []Account {
var out []Account
for _, r := range d.Resources {
u, ok := r.(*declaration.User)
if !ok || len(u.Groups) == 0 || held[u.ID] || u.Name == "" {
root := ok && u.Root == declaration.RootNever
if !ok || (len(u.Groups) == 0 && !root) || held[u.ID] || u.Name == "" {
continue
}
at := strings.LastIndex(u.ID, ".")
if at <= 0 || strings.HasPrefix(u.ID, declaration.AdoptionPrefix) {
continue
}
out = append(out, Account{Module: u.ID[:at], ID: u.ID, Name: u.Name,
Groups: append([]string(nil), u.Groups...)})
a := Account{Module: u.ID[:at], ID: u.ID, Name: u.Name, Groups: append([]string(nil), u.Groups...), Root: root}
if root {
a.Secrets = secretsOf(d, u.Name)
}
out = append(out, a)
}
sort.Slice(out, func(a, b int) bool { return out[a].ID < out[b].ID })
return out
}
// secretsOf is the path of every secret file the declaration places for anybody but account: sealed whole,
// or with sealed values in its content. One the account owns is its own to read.
func secretsOf(d *declaration.Declaration, account string) []string {
var out []string
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || f.Path == "" || (!f.Secret() && len(f.Secrets) == 0) || f.Owner == account {
continue
}
out = append(out, f.Path)
}
sort.Strings(out)
return out
}
// Session is what an account's own service manager holds.
type Session struct {
// Running is whether the manager runs.
@@ -94,6 +142,10 @@ type Reader interface {
InDatabase(ctx context.Context, account string) ([]string, error)
// Session is the account's own service manager: whether it runs, and which of groups it holds.
Session(ctx context.Context, account string, groups []string) (Session, error)
// Escalation is every way the account can become root without a person, in words — its uid, a group
// of RootGroups, a sudo rule, a secret of secrets it can read — and none when there is none. Reads
// only (novox/hq ADR 0266).
Escalation(ctx context.Context, account string, secrets []string) ([]string, error)
}
// Verdict is one account's state as a statement says it.
@@ -103,6 +155,10 @@ type Verdict struct {
Reason string
Since time.Time
Streak int
// Search is how far the setuid search got, while the verdict waits for it (novox/hq issue 361); nil
// otherwise. Never part of Reason: it moves every minute, and a verdict that changed every minute would
// be said in the journal every minute.
Search *SearchProgress
}
// Statement is one look at every account.
@@ -169,14 +225,14 @@ func (j *Judge) Look(ctx context.Context) (Statement, bool) {
st := Statement{At: now}
changed := len(j.last.Accounts) != len(j.accounts)
for _, a := range j.accounts {
state, reason := j.judge(ctx, a)
state, reason, search := j.judge(ctx, a)
k := j.kept[a.ID]
if k == nil || k.State != state || k.Reason != reason {
changed = true
k = &Verdict{Account: a, State: state, Reason: reason, Since: now}
j.kept[a.ID] = k
}
k.Account = a
k.Account, k.Search = a, search
if state == Unhealthy {
k.Streak++
} else {
@@ -188,11 +244,38 @@ func (j *Judge) Look(ctx context.Context) (Statement, bool) {
return st, changed
}
// judge is one account's verdict on one look.
func (j *Judge) judge(ctx context.Context, a Account) (string, string) {
// judge is one account's verdict on one look, and how far the setuid search got while it waits for one.
func (j *Judge) judge(ctx context.Context, a Account) (string, string, *SearchProgress) {
if a.Root {
ways, err := j.reader.Escalation(ctx, a.Name, a.Secrets)
var notJudged *SetuidNotJudged
switch {
case len(ways) > 0:
// A way found is a way found, whatever else could not be judged; said beside it.
reason := ReasonRoot + ": " + strings.Join(ways, "; ")
if errors.As(err, &notJudged) {
reason += "; " + firstLine(err.Error())
}
var progress *SearchProgress
if notJudged != nil {
progress = notJudged.Progress
}
return Unhealthy, reason, progress
case errors.As(err, &notJudged):
// The setuid search has no result yet: not judged, said as such — the same words on every look
// until it has one, never a search that stalls the look (novox/hq ADR 0266).
return Unknown, firstLine(err.Error()), notJudged.Progress
case err != nil:
return Unknown, "whether the account can become root could not be read: " + firstLine(err.Error()), nil
}
if len(a.Groups) == 0 {
// Declared for root alone: nothing of a session to read.
return Healthy, "", nil
}
}
in, err := j.reader.InDatabase(ctx, a.Name)
if err != nil {
return Unknown, "the user database could not be read: " + firstLine(err.Error())
return Unknown, "the user database could not be read: " + firstLine(err.Error()), nil
}
listed := map[string]bool{}
for _, g := range in {
@@ -208,15 +291,15 @@ func (j *Judge) judge(ctx context.Context, a Account) (string, string) {
}
if len(missing) > 0 {
return Unhealthy, fmt.Sprintf("not in the group %s: the apply has not put %s there; its outcome says why",
strings.Join(missing, ", "), a.Name)
strings.Join(missing, ", "), a.Name), nil
}
s, err := j.reader.Session(ctx, a.Name, inDB)
if err != nil {
return Unknown, "the account's own service manager could not be read: " + firstLine(err.Error())
return Unknown, "the account's own service manager could not be read: " + firstLine(err.Error()), nil
}
if !s.Running {
// Nobody is logged in, and the account does not linger: the next login takes every group.
return Healthy, ""
return Healthy, "", nil
}
var lacking []string
for _, g := range inDB {
@@ -226,9 +309,9 @@ func (j *Judge) judge(ctx context.Context, a Account) (string, string) {
}
if len(lacking) > 0 {
return Unhealthy, fmt.Sprintf("%s: %s is in the group %s, and its running session began before it was; "+
"log out of every session and in again, or reboot", ReasonRelogin, a.Name, strings.Join(lacking, ", "))
"log out of every session and in again, or reboot", ReasonRelogin, a.Name, strings.Join(lacking, ", ")), nil
}
return Healthy, ""
return Healthy, "", nil
}
func firstLine(s string) string {
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// The account judge (novox/hq ADR 0252, "how it is checked"): an account in a group its running session
+214 -1
View File
@@ -4,20 +4,204 @@ import (
"context"
"errors"
"fmt"
"io/fs"
"os"
"os/exec"
"path/filepath"
"slices"
"strconv"
"strings"
"syscall"
"time"
)
// Runner runs a command and answers what it printed — the apply's own (apply.ExecRunner).
type Runner func(ctx context.Context, name string, args ...string) (string, error)
// Exec reads the machine through its command lines and the process table. **Reads only**: `id`, `getent`,
// the machine's own manager's `systemctl show`, and a status file under /proc (a test holds it).
// the machine's own manager's `systemctl show`, `sudo -l -U` (which lists, as root, what sudo would let an
// account run, and runs nothing), a status file under /proc and a secret's owner and mode (a test holds
// both).
type Exec struct {
Run Runner
// Proc is where the process table is; empty is /proc. A test points it at a directory of its own.
Proc string
// Stat is a file's owner and mode; nil is the machine's own (os.Stat). A test gives files of its own.
Stat func(path string) (FileMode, error)
// Lstat is a path's own owner and mode, never through a link, with its type bits in Perm (fs.ModeSymlink,
// fs.ModeDir); nil is the machine's own (os.Lstat). Whether an account can reach a setuid program is read
// with it at every judgement (novox/hq issue 361).
Lstat func(path string) (FileMode, error)
// ReadFile, ReadDir and ACL read doas's and polkit's rules and a file's POSIX ACL; nil is the machine's.
ReadFile func(path string) ([]byte, error)
ReadDir func(path string) ([]fs.DirEntry, error)
ACL func(path string) ([]ACLEntry, error)
// Cache keeps the search for setuid programs between looks; nil searches on every look.
Cache *SetuidCache
// SearchRun runs the setuid search's commands (the package manager it asks); nil is Run. A
// serving engine gives Idle, so they yield to everything else on the machine, as the walk does.
SearchRun Runner
// Walk carries a walk for setuid programs on; nil is the engine's own (walker). A test gives its own.
Walk func(ctx context.Context, p *SetuidProgress, checkpoint func()) error
// Now is the clock the cache is kept by; nil is the machine's.
Now func() time.Time
}
// FileMode is what decides whether an account reads a file: its owner, its group and its permission bits.
type FileMode struct {
UID, GID int
Perm fs.FileMode
}
// Escalation is every way account can become root without a person (novox/hq ADR 0266) that the judge
// looks for — Judged lists them, NotJudged what it does not: its uid, a group of RootGroups the user database
// lists it in, any sudo rule naming it or a group of it, any of secrets it can read by owner, group or other
// bits, and the ways of ways.go: doas, polkit, a runtime's socket, an ACL, a setuid program no package owns.
// sudo absent is no sudo rule; doas and polkit (pkexec's grants) are judged from their own rules. A secret not there yet is skipped: there is nothing to read.
// The parent directories are not walked, so a file the bits allow and a directory hides is still said:
// the judge errs toward saying a way that is not, never toward missing one that is.
func (e Exec) Escalation(ctx context.Context, account string, secrets []string) ([]string, error) {
var ways []string
uidOut, err := e.Run(ctx, "id", "-u", account)
if err != nil {
return nil, fmt.Errorf("the user database did not answer about %q: %w", account, err)
}
uid, err := strconv.Atoi(strings.TrimSpace(uidOut))
if err != nil {
return nil, fmt.Errorf("the user database gave %q as %q's number", strings.TrimSpace(uidOut), account)
}
if uid == 0 {
ways = append(ways, "its uid is 0")
}
names, err := e.InDatabase(ctx, account)
if err != nil {
return nil, err
}
for _, g := range names {
if slices.Contains(RootGroups, g) {
ways = append(ways, "in the group "+g+", which grants root")
}
}
listed, err := e.Run(ctx, "sudo", "-l", "-U", account)
rules, err := SudoRules(listed, err)
if err != nil {
return nil, err
}
if len(rules) > 0 {
ways = append(ways, "sudo grants it: "+strings.Join(rules, ", "))
}
gidsOut, err := e.Run(ctx, "id", "-G", account)
if err != nil {
return nil, fmt.Errorf("the user database did not answer about %q's groups: %w", account, err)
}
gids := map[int]bool{}
for _, f := range strings.Fields(gidsOut) {
if n, err := strconv.Atoi(f); err == nil {
gids[n] = true
}
}
if len(secrets) > 0 {
stat := e.Stat
if stat == nil {
stat = statOf
}
for _, path := range secrets {
m, err := stat(path)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("the secret %s could not be read for its owner and mode: %w", path, err)
}
if Readable(m, uid, gids) {
ways = append(ways, "it can read the secret "+path)
}
}
}
more, err := e.moreWays(ctx, account, uid, names, gids, secrets)
var notJudged *SetuidNotJudged
if err != nil && !errors.As(err, &notJudged) {
return nil, err
}
// Every way found, and — when the setuid search has no result yet — that it is not judged.
return append(ways, more...), err
}
// Readable is whether an account of uid, in the groups gids, reads a file of m by its permission bits, as
// the kernel decides it: the owner's bits for the owner (root reads everything), the group's for a member,
// the others' for anybody else.
func Readable(m FileMode, uid int, gids map[int]bool) bool {
switch {
case uid == 0:
return true
case m.UID == uid:
return m.Perm&0o400 != 0
case gids[m.GID]:
return m.Perm&0o040 != 0
default:
return m.Perm&0o004 != 0
}
}
// SudoRules is the rules `sudo -l -U <account>` lists, from what it printed and how it ended: none when
// the account "is not allowed to run sudo" or sudo is not on the machine; every indented line after "may
// run the following commands" otherwise. Any rule counts — the decision is no sudo for the account at
// all, so a rule that asks for a password the account was never given is still a rule somebody can give
// it one for. Output that says neither is an error: unread is never none.
func SudoRules(out string, err error) ([]string, error) {
if err != nil && (errors.Is(err, exec.ErrNotFound) || errors.Is(err, fs.ErrNotExist)) {
return nil, nil
}
text := out
if err != nil {
text += "\n" + err.Error()
}
if strings.Contains(text, "is not allowed to run sudo") {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("sudo did not list the account's rules: %w", err)
}
var rules []string
listing := false
for _, line := range strings.Split(out, "\n") {
if strings.Contains(line, "may run the following commands") {
listing = true
continue
}
if listing && (strings.HasPrefix(line, " ") || strings.HasPrefix(line, "\t")) && strings.TrimSpace(line) != "" {
rules = append(rules, strings.TrimSpace(line))
}
}
if !listing {
return nil, fmt.Errorf("sudo listed neither rules nor a refusal: %q", firstLine(out))
}
return rules, nil
}
func statOf(path string) (FileMode, error) {
info, err := os.Stat(path)
if err != nil {
return FileMode{}, err
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return FileMode{}, fmt.Errorf("%s has no owner this platform reports", path)
}
return FileMode{UID: int(st.Uid), GID: int(st.Gid), Perm: info.Mode().Perm()}, nil
}
// lstatOf is a path's own owner and mode, its type bits kept.
func lstatOf(path string) (FileMode, error) {
info, err := os.Lstat(path)
if err != nil {
return FileMode{}, err
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return FileMode{}, fmt.Errorf("%s has no owner this platform reports", path)
}
return FileMode{UID: int(st.Uid), GID: int(st.Gid), Perm: info.Mode()}, nil
}
// InDatabase is `id -nG`: every group the user database lists the account in.
@@ -101,3 +285,32 @@ func (e Exec) heldBy(pid string) (map[string]bool, error) {
}
return held, nil
}
// IDs is `getent passwd`: an account's uid and primary gid, and whether the user database names it at all. A
// database that does not answer is an error, never "no such account".
func (e Exec) IDs(ctx context.Context, account string) (uid, gid int, found bool, err error) {
out, err := e.Run(ctx, "getent", "passwd", account)
if err != nil {
if strings.Contains(err.Error(), "exited 2") || strings.Contains(err.Error(), "exit status 2") {
return 0, 0, false, nil
}
return 0, 0, false, fmt.Errorf("the user database did not answer about %q: %w", account, err)
}
fields := strings.Split(strings.TrimSpace(out), ":")
if len(fields) < 4 {
return 0, 0, false, fmt.Errorf("the user database gave %q for %q, which is not a passwd entry",
strings.TrimSpace(out), account)
}
// getent reads an all-digit key as a uid: "1000" answers with whoever has uid 1000. Only the account named is
// the one asked for.
if fields[0] != account {
return 0, 0, false, fmt.Errorf("the user database answered %q with the account %q: an account is named, "+
"never numbered", account, fields[0])
}
uid, uerr := strconv.Atoi(fields[2])
gid, gerr := strconv.Atoi(fields[3])
if uerr != nil || gerr != nil {
return 0, 0, false, fmt.Errorf("the user database gave no numbers for %q", account)
}
return uid, gid, true, nil
}
+78
View File
@@ -0,0 +1,78 @@
//go:build linux
package accounts
import (
"bytes"
"context"
"errors"
"fmt"
"os"
"os/exec"
"runtime"
"strings"
"syscall"
"golang.org/x/sys/unix"
)
// ioprioIdle is the idle I/O class (IOPRIO_CLASS_IDLE, 3, shifted by IOPRIO_CLASS_SHIFT, 13): the process gets
// the disk only when nothing else asks for it, where the disk's scheduler honours classes (BFQ does).
const ioprioIdle = 3 << 13
// Idle runs a command as CLocale does, at the lowest priority the machine has (novox/hq issue 361): CPU nice
// 19 and the idle I/O class, set on the process as soon as it has started — so the search for setuid programs,
// which walks every file and runs as long as that takes, yields to everything else. It dies with the engine
// (the parent-death signal), so a restarted engine never leaves a walk running beside its own. A priority it
// could not lower is a failure: the command is killed rather than walk the disks at full priority.
func Idle(ctx context.Context, name string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
cmd.Env = append(os.Environ(), "LC_ALL=C", "LANG=C", "LANGUAGE=C")
cmd.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL}
var stdout, stderr bytes.Buffer
cmd.Stdout, cmd.Stderr = &stdout, &stderr
// The parent-death signal follows the thread that started the child, not the process: held on this
// thread until the child has ended, so the runtime retiring a thread never kills a walk midway.
runtime.LockOSThread()
defer runtime.UnlockOSThread()
if err := cmd.Start(); err != nil {
return "", fmt.Errorf("%s: %w", name, err)
}
pid := cmd.Process.Pid
if err := lower(pid); err != nil {
_ = cmd.Process.Kill()
_ = cmd.Wait()
return "", fmt.Errorf("%s: its priority could not be lowered, so it was stopped: %w", name, err)
}
err := cmd.Wait()
out := stdout.String()
if err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
return out, fmt.Errorf("%s exited %d: %s", name, exit.ExitCode(), strings.TrimSpace(stderr.String()))
}
return out, fmt.Errorf("%s: %w", name, err)
}
return out, nil
}
// lower sets a process's CPU nice to 19 and its I/O class to idle.
func lower(pid int) error {
if err := unix.Setpriority(unix.PRIO_PROCESS, pid, 19); err != nil {
return fmt.Errorf("nice: %w", err)
}
if _, _, errno := unix.Syscall(unix.SYS_IOPRIO_SET, 1 /* IOPRIO_WHO_PROCESS */, uintptr(pid), ioprioIdle); errno != 0 {
return fmt.Errorf("I/O class: %w", errno)
}
return nil
}
// LowerThisThread puts the calling goroutine on a thread of its own at nice 19 and the idle I/O class, for the
// walk for setuid programs, which runs in the engine (novox/hq issue 361): on Linux both are a thread's, so the
// engine's other work keeps its priority. The goroutine is locked to the thread and never unlocked, so the
// runtime ends the thread with it rather than hand a lowered thread to other work; a command it starts inherits
// the priority too.
func LowerThisThread() error {
runtime.LockOSThread()
return lower(unix.Gettid())
}
+13
View File
@@ -0,0 +1,13 @@
//go:build !linux
package accounts
import "context"
// Idle is CLocale where the machine offers no idle class this engine sets (novox/hq issue 361).
func Idle(ctx context.Context, name string, args ...string) (string, error) {
return CLocale(ctx, name, args...)
}
// LowerThisThread is nothing where the machine offers no per-thread priority this engine sets.
func LowerThisThread() error { return nil }
+32
View File
@@ -0,0 +1,32 @@
package accounts
import (
"context"
"errors"
"strings"
"testing"
)
// An account is read by its name (novox/hq ADR 0325 §4): getent reads an all-digit key as a uid, so "1000" would
// answer with whoever has uid 1000 — refused, never taken for the account named.
func TestAnAccountIsReadByItsNameNeverItsNumber(t *testing.T) {
db := map[string]string{
"nox": "nox:x:1003:1003::/home/nox:/bin/sh",
"1000": "jochen:x:1000:1000::/home/jochen:/bin/zsh", // what getent answers for a uid
}
e := Exec{Run: func(_ context.Context, name string, args ...string) (string, error) {
if line, ok := db[args[len(args)-1]]; ok {
return line + "\n", nil
}
return "", errors.New("getent exited 2")
}}
if uid, gid, found, err := e.IDs(context.Background(), "nox"); err != nil || !found || uid != 1003 || gid != 1003 {
t.Fatalf("nox: %d %d %v %v", uid, gid, found, err)
}
if _, _, found, err := e.IDs(context.Background(), "1000"); found || err == nil || !strings.Contains(err.Error(), "jochen") {
t.Fatalf("an all-digit name was read as a uid: found=%v err=%v", found, err)
}
if _, _, found, err := e.IDs(context.Background(), "ghost"); found || err != nil {
t.Fatalf("an absent account: found=%v err=%v", found, err)
}
}
+302
View File
@@ -0,0 +1,302 @@
package accounts
import (
"context"
"errors"
"fmt"
"io/fs"
"os/exec"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// An account declared never to become root without a person (novox/hq ADR 0266, "how it is checked"): each
// way to root is found and said, none is healthy whether or not anybody is logged in, an unanswered question
// is unknown, and the judge only reads.
// agentMachine is a fake machine for the escalation question: the account's uid, its groups by name and
// number, what sudo lists, and the secrets' owners and modes.
type agentMachine struct {
uid string
groups string
gids string
sudo string
sudoErr error
files map[string]FileMode
failsID bool
asked []string
// texts are files' contents (doas's, polkit's); dirs a directory's files by name; acls a file's ACL;
// setuid what the walk finds, one per line, and packaged the paths a package owns.
texts map[string]string
dirs map[string][]string
acls map[string][]ACLEntry
setuid string
findErr error
packaged map[string]bool
// proc is /proc's files, aSafeProc unless a test changes them; findArgs sees the filesystems walked.
proc map[string]string
findArgs func([]string)
// pacmanCalls counts the package manager's calls: one for every path found.
pacmanCalls int
// nodes are paths' own owners and modes (Lstat); any other is root's, a directory searchable by all, or a
// file the walk found, setuid and executable by all.
nodes map[string]FileMode
}
func (m *agentMachine) lstat(path string) (FileMode, error) {
if n, ok := m.nodes[path]; ok {
return n, nil
}
if strings.Contains(m.setuid, path+"\n") {
return FileMode{Perm: fs.ModeSetuid | 0o755}, nil
}
return FileMode{Perm: fs.ModeDir | 0o755}, nil
}
func (m *agentMachine) readFile(path string) ([]byte, error) {
if t, ok := m.texts[path]; ok {
return []byte(t), nil
}
if t, ok := m.proc[path]; ok {
return []byte(t), nil
}
return nil, fs.ErrNotExist
}
// aSafeProc is what a machine with nothing to say has in /proc: links protected, nothing on the runtimes'
// ports, one root filesystem.
func aSafeProc() map[string]string {
return map[string]string{
"/proc/sys/fs/protected_hardlinks": "1\n",
"/proc/sys/fs/protected_symlinks": "1\n",
"/proc/net/tcp": " sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n" +
" 0: 0100007F:1092 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 1 1\n",
"/proc/mounts": "/dev/sda2 / ext4 rw,relatime 0 0\nproc /proc proc rw,nosuid 0 0\ntmpfs /tmp tmpfs rw,nosuid 0 0\n",
}
}
func (m *agentMachine) readDir(path string) ([]fs.DirEntry, error) {
names, ok := m.dirs[path]
if !ok {
return nil, fs.ErrNotExist
}
var out []fs.DirEntry
for _, n := range names {
out = append(out, fakeEntry(n))
}
return out, nil
}
type fakeEntry string
func (f fakeEntry) Name() string { return string(f) }
func (f fakeEntry) IsDir() bool { return false }
func (f fakeEntry) Type() fs.FileMode { return 0 }
func (f fakeEntry) Info() (fs.FileInfo, error) { return nil, fs.ErrNotExist }
func (m *agentMachine) acl(path string) ([]ACLEntry, error) {
if _, ok := m.files[path]; !ok {
return nil, fs.ErrNotExist
}
return m.acls[path], nil
}
// walk is the walk for setuid programs on this machine: m.setuid found, then m.findErr.
func (m *agentMachine) walk(_ context.Context, p *SetuidProgress, _ func()) error {
m.asked = append(m.asked, "walk "+strings.Join(p.Mounts, " "))
if m.findArgs != nil {
m.findArgs(p.Mounts)
}
for _, l := range strings.Split(m.setuid, "\n") {
if l = strings.TrimSpace(l); l != "" {
p.Found = append(p.Found, l)
}
}
return m.findErr
}
func (m *agentMachine) run(_ context.Context, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
m.asked = append(m.asked, line)
switch {
case m.failsID && name == "id":
return "", errors.New("id exited 1: no such user")
case line == "id -u agent":
return m.uid + "\n", nil
case line == "id -nG agent":
return m.groups + "\n", nil
case line == "id -G agent":
return m.gids + "\n", nil
case line == "sudo -l -U agent":
return m.sudo, m.sudoErr
case name == "pacman" && len(args) >= 2 && args[0] == "-Qo":
m.pacmanCalls++
var out, errs []string
for _, p := range args[1:] {
if m.packaged[p] {
out = append(out, p+" is owned by somepackage 1.0-1")
} else {
errs = append(errs, "error: No package owns "+p)
}
}
said := strings.Join(out, "\n")
if len(errs) > 0 {
return said, errors.New("pacman exited 1: " + strings.Join(errs, "\n"))
}
return said, nil
}
return "", errors.New("not a command the judge may run: " + line)
}
func (m *agentMachine) stat(path string) (FileMode, error) {
if f, ok := m.files[path]; ok {
return f, nil
}
return FileMode{}, fs.ErrNotExist
}
const notAllowed = "User agent is not allowed to run sudo on box.\n"
var agent = Account{Module: "claude-code", ID: "claude-code.agent", Name: "agent", Root: true,
Secrets: []string{"/var/lib/mesh/node-tools/broker"}}
func clean() *agentMachine {
return &agentMachine{uid: "1600", groups: "agent", gids: "1600", sudo: notAllowed,
files: map[string]FileMode{"/var/lib/mesh/node-tools/broker": {UID: 1500, GID: 1500, Perm: 0o600}},
proc: aSafeProc()}
}
func lookAgent(t *testing.T, m *agentMachine) Verdict {
t.Helper()
j := New(Exec{Run: m.run, Walk: m.walk, Stat: m.stat, ReadFile: m.readFile, ReadDir: m.readDir, ACL: m.acl, Lstat: m.lstat})
j.Set([]Account{agent})
st, _ := j.Look(t.Context())
if len(st.Accounts) != 1 {
t.Fatalf("the statement: %+v", st)
}
for _, a := range m.asked {
if !strings.HasPrefix(a, "id ") && a != "sudo -l -U agent" && !strings.HasPrefix(a, "walk /") &&
!strings.HasPrefix(a, "pacman -Qo ") {
t.Errorf("the judge asked something that is not a read: %q", a)
}
}
return st.Accounts[0]
}
func TestAnAgentAccountWithNoWayToRootIsHealthyWithNobodyLoggedIn(t *testing.T) {
if v := lookAgent(t, clean()); v.State != Healthy || v.Reason != "" || !v.Root {
t.Fatalf("no way to root: %+v", v)
}
}
func TestEachWayToRootIsSaid(t *testing.T) {
for _, c := range []struct {
name string
set func(*agentMachine)
said string
}{
{"uid 0", func(m *agentMachine) { m.uid = "0" }, "its uid is 0"},
{"docker", func(m *agentMachine) { m.groups = "agent docker" }, "in the group docker, which grants root"},
{"wheel", func(m *agentMachine) { m.groups = "agent wheel" }, "in the group wheel, which grants root"},
{"sudo", func(m *agentMachine) {
m.sudo = "Matching Defaults entries for agent on box:\n env_reset\n\nUser agent may run the following commands on box:\n (ALL) NOPASSWD: ALL\n"
}, "sudo grants it: (ALL) NOPASSWD: ALL"},
{"secret by others", func(m *agentMachine) {
m.files["/var/lib/mesh/node-tools/broker"] = FileMode{UID: 1500, GID: 1500, Perm: 0o644}
}, "it can read the secret /var/lib/mesh/node-tools/broker"},
{"secret by group", func(m *agentMachine) {
m.gids = "1600 1500"
m.files["/var/lib/mesh/node-tools/broker"] = FileMode{UID: 1500, GID: 1500, Perm: 0o640}
}, "it can read the secret /var/lib/mesh/node-tools/broker"},
} {
t.Run(c.name, func(t *testing.T) {
m := clean()
c.set(m)
v := lookAgent(t, m)
if v.State != Unhealthy || !strings.HasPrefix(v.Reason, ReasonRoot+": ") || !strings.Contains(v.Reason, c.said) {
t.Fatalf("want %q said: %+v", c.said, v)
}
})
}
}
func TestEveryWayIsSaidAtOnce(t *testing.T) {
m := clean()
m.groups = "agent docker"
m.sudo = "User agent may run the following commands on box:\n (ALL) ALL\n"
v := lookAgent(t, m)
if !strings.Contains(v.Reason, "docker") || !strings.Contains(v.Reason, "(ALL) ALL") {
t.Fatalf("both ways: %+v", v)
}
}
func TestAnUnansweredQuestionIsUnknownNeverHealthy(t *testing.T) {
m := clean()
m.failsID = true
if v := lookAgent(t, m); v.State != Unknown {
t.Fatalf("an unread database: %+v", v)
}
m = clean()
m.sudo = "something sudo never says\n"
if v := lookAgent(t, m); v.State != Unknown {
t.Fatalf("an unread sudo: %+v", v)
}
}
func TestASecretNotThereYetIsNoWay(t *testing.T) {
m := clean()
delete(m.files, "/var/lib/mesh/node-tools/broker")
if v := lookAgent(t, m); v.State != Healthy {
t.Fatalf("no secret placed: %+v", v)
}
}
func TestSudoRules(t *testing.T) {
none := []struct {
out string
err error
}{
{notAllowed, nil},
{notAllowed, errors.New("sudo exited 1: ")},
{"", fmt.Errorf("sudo: %w", exec.ErrNotFound)},
}
for _, c := range none {
if rules, err := SudoRules(c.out, c.err); err != nil || len(rules) != 0 {
t.Errorf("%q, %v: rules %v, err %v", c.out, c.err, rules, err)
}
}
rules, err := SudoRules("Matching Defaults entries for agent on box:\n env_reset\n\nUser agent may run the following commands on box:\n (ALL) /usr/bin/systemctl\n (root) NOPASSWD: /usr/bin/true\n", nil)
if err != nil || len(rules) != 2 || rules[0] != "(ALL) /usr/bin/systemctl" {
t.Fatalf("two rules: %v, %v", rules, err)
}
if _, err := SudoRules("", errors.New("sudo exited 1: sudo: unable to resolve host")); err == nil {
t.Error("a failing sudo that said neither was read as no rules")
}
}
func TestReadable(t *testing.T) {
m := FileMode{UID: 1500, GID: 1500, Perm: 0o600}
if Readable(m, 1600, map[int]bool{1600: true}) || !Readable(m, 1500, nil) || !Readable(m, 0, nil) {
t.Fatal("owner bits")
}
}
func TestOfJudgesARootNeverAccountWithNoGroupsAndItsSecrets(t *testing.T) {
d, err := declaration.ParseTrusted([]byte(`{"declaration":1,"resources":[
{"id":"claude-code.agent","type":"user","name":"agent","root":"never"},
{"id":"zsh.login","type":"user","name":"operator","shell":"/bin/zsh"},
{"id":"node-tools.need-broker","type":"file","path":"/var/lib/mesh/broker","sealed":"x","owner":"operator"},
{"id":"claude-code.own","type":"file","path":"/home/agent/own","sealed":"x","owner":"agent"},
{"id":"claude-code.plain","type":"file","path":"/etc/plain","content":"x"}
]}`))
if err != nil {
t.Fatal(err)
}
got := Of(d, nil)
if len(got) != 1 || got[0].ID != "claude-code.agent" || !got[0].Root ||
len(got[0].Secrets) != 1 || got[0].Secrets[0] != "/var/lib/mesh/broker" {
t.Fatalf("judged: %+v", got)
}
}
+635
View File
@@ -0,0 +1,635 @@
package accounts
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"syscall"
"time"
"github.com/novox/mesh-host/rootsearch"
)
// SetuidCache keeps the search for setuid programs, which walks every local filesystem, and runs it in the
// background (novox/hq ADR 0266, issue 361): a look never waits for it, and the search itself is never cut
// short.
//
// - **A verdict comes only from a complete search, while it is fresh**: one that walked everything and had the
// package manager answer for every path found, started less than FreshFor ago. Before the first completes,
// or once the last is older, the answer is "not judged yet" (ReasonPending) while a search runs, and "not
// judged" (ReasonIncomplete) after one failed. Never "none found" from a search that did not finish.
// - **It runs to completion**, once at a time, with no bound, on a thread of its own at idle priority (Lower):
// a fresh one when the last complete one is Every old, when SearchAgain asks (an apply changed what root
// controls), or when a filesystem appeared that the last one did not walk. One that fails is tried again
// after a back-off that doubles from FailedBackoff up to MaxBackoff.
// - **Resumable**: the walk's progress is kept in ProgressPath every CheckpointEvery, and a search started
// again — after an engine restart, an update, a reboot or a failure — carries on from it while it is
// younger than FreshFor. A search SearchAgain asked for never resumes an older walk: it starts from nothing.
// - **Kept** in Path, beside the engine's other state, with when it started and finished: an engine restart
// serves the last complete search rather than judging nothing until a new walk ends.
//
// Why a complete search stays sound for a day and more, and a resumed walk is sound, is said in the rootsearch
// package and setuid_walk.go: only root makes a setuid-root file, never the agent account.
type SetuidCache struct {
// Every and FreshFor are rootsearch's when zero.
Every, FreshFor time.Duration
// Path is the file the last complete search is kept in, and ProgressPath the walk under way; empty keeps
// either in memory only. The engine's own state directory, which only root writes: a file there that is
// not a regular file the engine's account owns and alone may write is not read (keptFile).
Path, ProgressPath string
// Lower is called on the search's own goroutine before it walks (LowerThisThread on a serving engine); a
// priority it cannot lower fails the search rather than walk the disks at full priority. Nil lowers nothing.
Lower func() error
// Say is where a search's end and a kept file refused are said; nil says nothing.
Say func(string)
mu sync.Mutex
wg sync.WaitGroup
loaded bool
last *setuidResult // the last complete search
unfinished *SetuidProgress // a walk that did not finish, to resume
progress *SearchProgress // how far the running search got, as of its last checkpoint
running bool
gen int // counts SearchNow: a search started under an older one is thrown away
cancel context.CancelFunc // ends the running search's walk
startedAt time.Time // when the search running, or the last one, was started (a resumed one: its start)
attemptAt time.Time // when the running or last search was started or resumed
endedAt time.Time // when the last search ended, on the same clock
again bool // asked to search again since the running or last one started
failedAt time.Time
failed error
backoff time.Duration
starts int // how many searches were started or resumed, for a test
}
// setuidResult is one complete search, as kept.
type setuidResult struct {
// Started and Finished are on the caller's clock: Finished is Started plus how long the search took.
Started time.Time `json:"started"`
Finished time.Time `json:"finished"`
// Mounts are the filesystems it walked; Found the setuid-root programs no package owns.
Mounts Paths `json:"mounts"`
Found Paths `json:"found"`
}
const (
// FailedBackoff is the first wait after a search failed, doubling to MaxBackoff.
FailedBackoff = time.Hour
MaxBackoff = 6 * time.Hour
// AgainAfter is the least time between the starts of two searches that SearchAgain or a new filesystem
// asked for: a run of applies does not walk the disks back to back.
AgainAfter = time.Hour
// clockSkew is how far in the future a kept search may say it started and still judge: a clock set back
// further makes it no verdict, never a fresh one.
clockSkew = 5 * time.Minute
// SetuidSearchFile and SetuidProgressFile are the files' names beside the engine's state: the last
// complete search, and the walk under way.
SetuidSearchFile = "setuid-search.json"
SetuidProgressFile = "setuid-search-progress.json"
)
// SetuidNotJudged is the search's answer when no complete search judges: none has finished yet, the last one is
// older than FreshFor, or every one since failed. The judge says the way is not judged, never that there is none.
type SetuidNotJudged struct {
// Pending is true while a search runs; false after one failed.
Pending bool
Since time.Time
Err error
// Last is when the last complete search started, when there is one too old to judge.
Last time.Time
// Progress is how far the running search got, as of its last checkpoint: nil when none runs.
Progress *SearchProgress
}
// SearchProgress is how far a running search got, as of its last checkpoint, for anybody to read (novox/hq
// issue 361): counts, times and mount points only. **Never a path the walk visited or found**: the cursor and
// the files found are under accounts' homes as often as not, and stay in the engine's state and journal.
type SearchProgress struct {
// Started is when the search began, on the cache's clock.
Started time.Time
// Filesystems is how many it walks; Done how many it has walked to their end; Walking the mount point of
// the one it walks now, empty when every one is walked.
Filesystems, Done int
Walking string
// Visited is how many entries it has gone past; Found how many setuid- or setgid-root files it found so
// far, before the package manager is asked about them.
Visited int64
Found int
// AtLeast says Visited is a lower bound: the walk was begun by an engine that did not count.
AtLeast bool
// Kept is when its progress was last kept, zero before the first time.
Kept time.Time
// Finished is every filesystem walked to its end, in order, and how long its walk took.
Finished []SearchedFilesystem
}
// SearchedFilesystem is one filesystem a search walked to its end. Took is negative when not known: walked by
// an engine before it counted.
type SearchedFilesystem struct {
Mount string
Took time.Duration
}
// progressOf is p as anybody may read it. Counts and mount points, no I/O.
func progressOf(p *SetuidProgress, kept time.Time) *SearchProgress {
sp := &SearchProgress{Started: p.Started, Filesystems: len(p.Mounts), Done: len(p.Done), Visited: p.Visited,
Found: len(p.Found), Kept: kept, AtLeast: !p.Counted}
for i, m := range p.Done {
took := time.Duration(-1)
if i < len(p.Took) {
took = p.Took[i]
}
sp.Finished = append(sp.Finished, SearchedFilesystem{Mount: m, Took: took})
}
for _, m := range p.Mounts {
if !slices.Contains(p.Done, m) {
sp.Walking = m
break
}
}
return sp
}
// Words are the progress in one line, as the verdict says it beside "not judged yet".
func (sp *SearchProgress) Words() string {
if sp == nil {
return ""
}
s := fmt.Sprintf("progress: %d of %d filesystems walked", sp.Done, sp.Filesystems)
if sp.Walking != "" {
s += ", walking " + sp.Walking
}
atLeast := ""
if sp.AtLeast {
atLeast = "at least "
}
s += fmt.Sprintf(", %s%d entries visited, %d setuid- or setgid-root files found so far", atLeast, sp.Visited, sp.Found)
if sp.Kept.IsZero() {
s += ", not kept yet"
} else {
s += ", kept " + sp.Kept.Local().Format("2006-01-02 15:04:05")
}
var took []string
for _, f := range sp.Finished {
if f.Took < 0 {
took = append(took, f.Mount+" in a time not known")
} else {
took = append(took, f.Mount+" in "+f.Took.Round(time.Second).String())
}
}
if len(took) > 0 {
s += "; walked " + strings.Join(took, ", ")
}
return s
}
func (e *SetuidNotJudged) Error() string {
stale := ""
if !e.Last.IsZero() {
stale = fmt.Sprintf("; the last complete one, started %s, is too old to judge", e.Last.Local().Format("2006-01-02 15:04"))
}
if e.Pending {
return fmt.Sprintf("%s: the search for setuid programs, started %s, has not finished yet%s", ReasonPending,
e.Since.Local().Format("2006-01-02 15:04"), stale)
}
return fmt.Sprintf("%s: the search for setuid programs did not complete (last tried %s: %v); it is tried "+
"again later%s", ReasonIncomplete, e.Since.Local().Format("2006-01-02 15:04"), e.Err, stale)
}
// The reasons of a root verdict the setuid search could not answer.
const (
ReasonPending = rootsearch.ReasonPending
ReasonIncomplete = rootsearch.ReasonIncomplete
)
func (c *SetuidCache) every() time.Duration {
if c.Every > 0 {
return c.Every
}
return rootsearch.Every
}
func (c *SetuidCache) freshFor() time.Duration {
if c.FreshFor > 0 {
return c.FreshFor
}
return rootsearch.FreshFor
}
// SearchAgain asks for a new search at the next look: an apply changed what root controls (an account, a group,
// a sudo rule, a package, an action). The last complete search keeps judging until it is too old: only root acts
// through an apply, and the agent account cannot make a setuid-root file. Nil c is nothing to ask.
func (c *SetuidCache) SearchAgain() {
if c == nil {
return
}
c.mu.Lock()
defer c.mu.Unlock()
c.again = true
// The walk under way, or kept, started before root acted: never carried on, even by an engine that
// restarts before the next search starts (the review of 2026-10-10). It keeps no progress from now on.
c.unfinished = nil
c.forgetProgress()
}
// SearchNow throws away the last complete search, kept or in memory, and any walk under way or kept, and has
// the next look start a full search at once, past AgainAfter and any back-off (novox/hq issue 361): the
// operator changed by hand what the last one found, which no apply says. Until the new one completes the
// verdict is "not judged yet". Asked at the controller's terminal alone (`nox node setuid-search <node>`): a
// search can never make a machine free wrongly, but asked again and again it would keep a machine unjudged
// and its disks busy. by is who asked, for the journal. Nil c has nothing to throw away.
func (c *SetuidCache) SearchNow(by string) string {
if c == nil {
return "this engine keeps no search for setuid programs"
}
c.mu.Lock()
defer c.mu.Unlock()
c.loaded = true // nothing kept is read after this: it is thrown away
what := "no complete search was kept"
if c.last != nil {
what = "the complete search started " + c.last.Started.Local().Format("2006-01-02 15:04") + " is thrown away"
}
if c.running {
what += ", and the search running since " + c.startedAt.Local().Format("2006-01-02 15:04") + " is ended"
if c.cancel != nil {
c.cancel()
}
}
c.gen++
c.last, c.unfinished, c.progress = nil, nil, nil
c.failed, c.failedAt, c.backoff, c.again = nil, time.Time{}, 0, false
c.forgetProgress()
if c.Path != "" {
if err := os.Remove(c.Path); err != nil && !errors.Is(err, fs.ErrNotExist) {
c.say("the search for setuid programs could not forget the complete search it kept: " + err.Error())
}
}
said := "a new search for setuid programs starts at the next look, asked by " + by + ": " + what +
"; until it completes, whether an account can become root through one is not judged"
c.say(said)
return said
}
// SetuidSearch is one search: it carries p on to its end — walking what p has not walked, calling checkpoint
// as it goes so its progress can be kept — and answers the setuid-root programs no package owns.
type SetuidSearch func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error)
// Look answers the setuid-root programs no package owns as the last complete search found them, and when it
// started — while it is fresh — and starts a search in the background when one is due; it never waits for one.
// mounts are the filesystems a new search walks. Nil c searches in place, for a caller that wants to.
//
// **One clock: the caller's.** A search's time is the `now` it was started at, never the wall clock read when it
// finished: due-ness compares a later `now` with it, and mixing the two clocks made a search never due again
// whenever the caller's clock ran behind the wall clock (found while checking issue 356). Measured from the
// start, a result reads at most one search older than it is, which errs towards not judging.
func (c *SetuidCache) Look(now time.Time, mounts []string, search SetuidSearch) ([]string, time.Time, error) {
if c == nil {
p := &SetuidProgress{Started: now, Mounts: slices.Clone(mounts), Counted: true}
found, err := search(context.Background(), p, func() {})
return found, now, err
}
c.mu.Lock()
defer c.mu.Unlock()
if !c.loaded {
c.loaded = true
c.last = c.loadResult()
c.unfinished = c.loadProgress()
}
var age time.Duration
if c.last != nil {
age = now.Sub(c.last.Started)
}
// Fresh only while it walked every filesystem a search would walk now: one mounted since is not judged
// by a search that never saw it.
fresh := c.last != nil && age > -clockSkew && age < c.freshFor() && covers(c.last.Mounts, mounts)
// Asked for (an apply, a new filesystem): at most once an AgainAfter, counted from when the last search
// ended, so a walk of hours is not followed by another at once, apply after apply.
gap := now.Sub(c.attemptAt)
if !c.endedAt.IsZero() {
gap = now.Sub(c.endedAt)
}
due := !c.running && (c.failed == nil || now.Sub(c.failedAt) >= c.backoff) &&
(!fresh || age >= c.every() || (c.again && (c.attemptAt.IsZero() || gap >= AgainAfter)))
if due {
c.start(now, mounts, search)
}
switch {
case fresh:
return slices.Clone(c.last.Found), c.last.Started, nil
case c.running:
return nil, time.Time{}, &SetuidNotJudged{Pending: true, Since: c.startedAt, Last: c.lastStarted(),
Progress: c.progress}
case c.failed != nil:
return nil, time.Time{}, &SetuidNotJudged{Since: c.failedAt, Err: c.failed, Last: c.lastStarted()}
default:
// Not fresh, not running, never failed: cannot be, as not fresh is due. Never a verdict.
return nil, time.Time{}, &SetuidNotJudged{Pending: true, Since: now, Last: c.lastStarted()}
}
}
func (c *SetuidCache) lastStarted() time.Time {
if c.last == nil {
return time.Time{}
}
return c.last.Started
}
// resumable is the walk that did not finish, when a search may carry it on: younger than FreshFor, and no
// SearchAgain since — an apply that moved root's things wants a walk that starts after it. Anything else is
// thrown away. c.mu held.
func (c *SetuidCache) resumable(now time.Time) *SetuidProgress {
p := c.unfinished
c.unfinished = nil
if p == nil {
return nil
}
age := now.Sub(p.Started)
if c.again || age <= -clockSkew || age >= c.freshFor() {
c.forgetProgress()
return nil
}
return p
}
// start runs one search in the background, with no bound: a walk that did not finish carried on, or a new one.
// c.mu held.
func (c *SetuidCache) start(now time.Time, mounts []string, search SetuidSearch) {
p := c.resumable(now)
resumed := p != nil
if p == nil {
p = &SetuidProgress{ID: fmt.Sprintf("%x", now.UnixNano()), Started: now, Mounts: slices.Clone(mounts),
Cursor: map[string]string{}, Counted: true}
}
c.running, c.startedAt, c.attemptAt, c.again = true, p.Started, now, false
c.starts++
// What a look says until the first checkpoint: where the kept walk was, or nothing walked.
kept := p.Kept
c.progress = progressOf(p, kept)
if resumed {
c.say(fmt.Sprintf("the search for setuid programs started %s carries on where it was (%d of %d "+
"filesystems walked)", p.Started.Local().Format("2006-01-02 15:04"), len(p.Done), len(p.Mounts)))
}
began := time.Now()
gen := c.gen
ctx, cancel := context.WithCancel(context.Background())
c.cancel = cancel
c.wg.Add(1)
go func() {
defer c.wg.Done()
var found []string
var err error
if c.Lower != nil {
// The goroutine stays on its lowered thread until it ends, and the thread ends with it.
if lerr := c.Lower(); lerr != nil {
err = fmt.Errorf("its priority could not be lowered, so it did not walk: %w", lerr)
}
}
if err == nil {
unkept := false
found, err = search(ctx, p, func() {
c.mu.Lock()
superseded := c.again || c.gen != gen
c.mu.Unlock()
if !superseded { // root acted since this walk started: its progress is not to be carried on
at := time.Now()
p.Kept = at
kerr := c.keepProgress(p)
// Superseded while it wrote (SearchAgain, SearchNow), after they forgot the kept walk: forgotten
// again here, so no engine restarted before this walk ends carries it on (the review of
// 2026-10-10). Written outside the lock, so a look never waits on the disk.
c.mu.Lock()
if c.again || c.gen != gen {
c.forgetProgress()
}
c.mu.Unlock()
if kerr != nil {
p.Kept = kept // not kept: the file still says the time before
if !unkept {
unkept = true
c.say("the search for setuid programs could not keep how far it got, and starts over " +
"if the engine stops: " + kerr.Error())
}
} else {
kept = at
}
}
// What a look says from now: computed from p here, on the walk's own goroutine, never read
// by a look while the walk changes it.
sp := progressOf(p, kept)
c.mu.Lock()
if c.gen == gen {
c.progress = sp
}
c.mu.Unlock()
})
}
took := time.Since(began)
cancel()
c.mu.Lock()
defer c.mu.Unlock()
c.running, c.progress, c.cancel = false, nil, nil
c.endedAt = now.Add(took)
if c.gen != gen {
// Thrown away by SearchNow while it ran: neither a result nor a failure, and nothing to carry on. The
// next look starts the search asked for.
c.endedAt = time.Time{}
// A checkpoint that read its generation just before SearchNow may have kept this walk again.
c.forgetProgress()
c.say(fmt.Sprintf("the search for setuid programs started %s was ended, as asked (%s this time)",
p.Started.Local().Format("2006-01-02 15:04"), took.Round(time.Second)))
return
}
if err != nil {
// The back-off counts from when it failed.
c.failed, c.failedAt = err, c.endedAt
c.backoff = min(max(2*c.backoff, FailedBackoff), MaxBackoff)
if !c.again { // superseded by an ask to search again: never carried on
c.unfinished = p
}
c.say(fmt.Sprintf("the search for setuid programs started %s did not complete (%s this time): %v",
p.Started.Local().Format("2006-01-02 15:04"), took.Round(time.Second), err))
return
}
r := &setuidResult{Started: p.Started, Finished: now.Add(took), Mounts: p.Mounts, Found: found}
c.last, c.failed, c.backoff = r, nil, 0
if err := c.keepResult(r); err != nil {
c.say("the search for setuid programs could not be kept, and judges from memory until the engine " +
"restarts: " + err.Error())
}
c.forgetProgress()
// The paths in the engine's journal alone: the verdict says one in a home by count.
named := ""
if len(found) > 0 {
named = ": " + strings.Join(found, ", ")
}
c.say(fmt.Sprintf("the search for setuid programs started %s completed (%s this time): %d no package "+
"owns%s", p.Started.Local().Format("2006-01-02 15:04"), took.Round(time.Second), len(found), named))
}()
}
func (c *SetuidCache) say(s string) {
if c.Say != nil {
c.Say(s)
}
}
// covers says every filesystem a search would walk now was walked.
func covers(walked, now []string) bool {
for _, m := range now {
if !slices.Contains(walked, m) {
return false
}
}
return true
}
// Wait is for a test: until the search running, if any, has finished.
func (c *SetuidCache) Wait() { c.wg.Wait() }
func (c *SetuidCache) keepResult(r *setuidResult) error { return keepFile(c.Path, r) }
func (c *SetuidCache) keepProgress(p *SetuidProgress) error { return keepFile(c.ProgressPath, p) }
// forgetProgress removes the kept walk: finished, or thrown away.
func (c *SetuidCache) forgetProgress() {
if c.ProgressPath == "" {
return
}
if err := os.Remove(c.ProgressPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
c.say("the search for setuid programs could not forget the walk it kept: " + err.Error())
}
}
// keepFile writes v to path: a new file beside it, readable by its owner alone, renamed over it — a link at
// path is replaced, never followed. Empty path keeps nothing.
func keepFile(path string, v any) error {
if path == "" {
return nil
}
raw, err := json.Marshal(v)
if err != nil {
return err
}
f, err := os.CreateTemp(filepath.Dir(path), "."+filepath.Base(path)+".*")
if err != nil {
return err
}
defer os.Remove(f.Name())
if _, err := f.Write(raw); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
return err
}
return os.Rename(f.Name(), path)
}
// loadResult reads the kept complete search, or nil: none kept, or one that cannot be believed, said why.
func (c *SetuidCache) loadResult() *setuidResult {
var r setuidResult
if !c.loadFile(c.Path, "complete search", &r) {
return nil
}
if r.Started.IsZero() || r.Finished.Before(r.Started) || len(r.Mounts) == 0 {
c.say("the kept search for setuid programs says no complete search, and a new one runs")
return nil
}
return &r
}
// loadProgress reads the kept walk, or nil.
func (c *SetuidCache) loadProgress() *SetuidProgress {
var p SetuidProgress
if !c.loadFile(c.ProgressPath, "walk under way", &p) {
return nil
}
if p.Started.IsZero() || len(p.Mounts) == 0 {
c.say("the kept walk for setuid programs says no walk, and it starts over")
c.forgetProgress()
return nil
}
if p.Cursor == nil {
p.Cursor = map[string]string{}
}
return &p
}
func (c *SetuidCache) loadFile(path, what string, v any) bool {
if path == "" {
return false
}
raw, err := keptFile(path)
if errors.Is(err, fs.ErrNotExist) {
return false
}
if err == nil {
if err = json.Unmarshal(raw, v); err != nil {
err = fmt.Errorf("%s is not JSON: %w", path, err)
}
}
if err != nil {
c.say(fmt.Sprintf("the kept %s for setuid programs is not read, and a new one runs: %v", what, err))
return false
}
return true
}
// keptFile reads a kept file, believing it only as a regular file, not followed through a link, owned by the
// engine's own account and writable by it alone, in a directory that is the same: anything else could have been
// written by somebody other than root, and a search it says is no search.
func keptFile(path string) ([]byte, error) {
if err := ownedAlone(filepath.Dir(path), true); err != nil {
return nil, err
}
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NOFOLLOW, 0)
if err != nil {
return nil, err
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return nil, err
}
if err := ownedAloneInfo(path, info, false); err != nil {
return nil, err
}
return io.ReadAll(io.LimitReader(f, 64<<20))
}
func ownedAlone(path string, dir bool) error {
info, err := os.Lstat(path)
if err != nil {
return err
}
return ownedAloneInfo(path, info, dir)
}
func ownedAloneInfo(path string, info fs.FileInfo, dir bool) error {
if dir && !info.IsDir() || !dir && !info.Mode().IsRegular() {
return fmt.Errorf("%s is not a plain %s", path, map[bool]string{true: "directory", false: "file"}[dir])
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return fmt.Errorf("%s's owner could not be read", path)
}
if int(st.Uid) != os.Geteuid() {
return fmt.Errorf("%s is owned by uid %d, not the engine's %d", path, st.Uid, os.Geteuid())
}
if info.Mode().Perm()&0o022 != 0 {
return fmt.Errorf("%s may be written by others than its owner (mode %o)", path, info.Mode().Perm())
}
return nil
}
+447
View File
@@ -0,0 +1,447 @@
package accounts
import (
"context"
"encoding/json"
"errors"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/rootsearch"
)
// The search for setuid programs runs to completion and judges from its last complete result while it is fresh
// (novox/hq issue 361): never a bound that ends it, never a verdict from a search that did not finish or is too
// old, a new search after root acted, and the result kept beside the engine's state, believed only as root's.
var at = time.Date(2026, 10, 10, 1, 0, 0, 0, time.UTC)
var root = []string{"/"}
// gate is a search that waits until released; each search started hands its context on started. found and
// err are read when it is released.
type gate struct {
release chan struct{}
started chan context.Context
found []string
err error
}
func newGate() *gate {
return &gate{release: make(chan struct{}, 16), started: make(chan context.Context, 16)}
}
func (g *gate) search(ctx context.Context, p *SetuidProgress, _ func()) ([]string, error) {
g.started <- ctx
<-g.release
return g.found, g.err
}
func (g *gate) finish(c *SetuidCache) { g.release <- struct{}{}; c.Wait() }
// starts is how many searches c started.
func starts(c *SetuidCache) int {
c.mu.Lock()
defer c.mu.Unlock()
return c.starts
}
func notJudged(t *testing.T, err error, pending bool) *SetuidNotJudged {
t.Helper()
var nj *SetuidNotJudged
if !errors.As(err, &nj) || nj.Pending != pending {
t.Fatalf("want not judged (pending %v), got %v", pending, err)
}
want := ReasonIncomplete
if pending {
want = ReasonPending
}
if !strings.HasPrefix(err.Error(), want) {
t.Fatalf("the words start %q: %v", want, err)
}
return nj
}
func TestTheSearchRunsToCompletionWithNoBound(t *testing.T) {
c := &SetuidCache{}
g := newGate()
if _, _, err := c.Look(at, root, g.search); err == nil {
t.Fatal("a look before the first search finished judged")
}
ctx := <-g.started
if _, ok := ctx.Deadline(); ok {
t.Fatal("the search was given a deadline: it must run until it has walked everything")
}
// Hours later, still running: still pending, never killed, never a second walk beside it.
_, _, err := c.Look(at.Add(5*time.Hour), root, g.search)
notJudged(t, err, true)
if starts(c) != 1 || ctx.Err() != nil {
t.Fatalf("a running search was started again or stopped: %d searches, %v", starts(c), ctx.Err())
}
g.found = []string{"/opt/x"}
g.finish(c)
found, when, err := c.Look(at.Add(5*time.Hour), root, g.search)
if err != nil || len(found) != 1 || !when.Equal(at) {
t.Fatalf("the complete search judges, from its start: %v %v %v", found, when, err)
}
}
func TestAFreshCompleteSearchJudgesUntilItIsDue(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
if _, _, err := c.Look(at.Add(rootsearch.Every-time.Minute), root, g.search); err != nil {
t.Fatalf("a complete search within a day: %v", err)
}
if starts(c) != 1 {
t.Fatalf("searched again before it was due: %d", starts(c))
}
// Due: a new search starts, and the last complete one judges meanwhile.
if _, _, err := c.Look(at.Add(rootsearch.Every), root, g.search); err != nil {
t.Fatalf("while the daily search runs, the last complete one still judges: %v", err)
}
if starts(c) != 2 {
t.Fatalf("the daily search did not start: %d", starts(c))
}
g.finish(c)
}
func TestAStaleSearchIsNotJudgedNeverFree(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
c.Look(at.Add(rootsearch.Every), root, g.search) // the daily one, which does not finish
_, _, err := c.Look(at.Add(rootsearch.FreshFor), root, g.search)
nj := notJudged(t, err, true)
if !nj.Last.Equal(at) || !strings.Contains(err.Error(), "too old to judge") {
t.Fatalf("a stale search says when the last complete one started: %v", err)
}
g.finish(c)
if _, _, err := c.Look(at.Add(rootsearch.FreshFor), root, g.search); err != nil {
t.Fatalf("once the new search completed it judges again: %v", err)
}
}
func TestAnIncompleteSearchIsNeverFree(t *testing.T) {
c := &SetuidCache{}
g := newGate()
g.err = errors.New("the search for setuid programs did not complete its walk")
c.Look(at, root, g.search)
g.finish(c)
_, _, err := c.Look(at.Add(time.Minute), root, g.search)
notJudged(t, err, false)
if starts(c) != 1 {
t.Fatalf("tried again within its back-off: %d", starts(c))
}
c.Look(at.Add(FailedBackoff+time.Second), root, g.search)
if starts(c) != 2 {
t.Fatalf("not tried again after its back-off: %d", starts(c))
}
g.finish(c)
// A stale complete search and a failed new one: still not judged, never the stale one.
c = &SetuidCache{}
g = newGate()
c.Look(at, root, g.search)
g.finish(c)
g.err = errors.New("pacman could not say who owns the setuid programs found")
c.Look(at.Add(rootsearch.Every), root, g.search)
g.finish(c)
if _, _, err := c.Look(at.Add(rootsearch.Every+time.Hour), root, g.search); err != nil {
t.Fatalf("a failed search leaves the fresh complete one judging: %v", err)
}
_, _, err = c.Look(at.Add(rootsearch.FreshFor), root, g.search)
if err == nil {
t.Fatal("a stale search judged after the newer one failed")
}
g.finish(c)
}
func TestAnApplyThatMovedRootSearchesAgain(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
c.SearchAgain()
if _, _, err := c.Look(at.Add(time.Minute), root, g.search); err != nil || starts(c) != 1 {
t.Fatalf("asked again within AgainAfter of the last search's end: %v, %d searches", err, starts(c))
}
if _, _, err := c.Look(at.Add(AgainAfter+time.Second), root, g.search); err != nil {
t.Fatalf("the last complete search judges while the new one runs: %v", err)
}
if starts(c) != 2 {
t.Fatalf("an apply that moved root did not search again: %d", starts(c))
}
// Asked again while one runs: one more after it, never two at once.
c.SearchAgain()
c.Look(at.Add(AgainAfter+time.Minute), root, g.search)
if starts(c) != 2 {
t.Fatalf("a second search started beside the running one: %d", starts(c))
}
g.finish(c) // ends, on the cache's clock, at about AgainAfter+1s
c.Look(at.Add(2*AgainAfter), root, g.search)
if starts(c) != 2 {
t.Fatal("a search started again less than AgainAfter after the last one ended")
}
c.Look(at.Add(2*AgainAfter+2*time.Second), root, g.search)
if starts(c) != 3 {
t.Fatalf("the ask made while a search ran was dropped: %d", starts(c))
}
g.finish(c)
var nilCache *SetuidCache
nilCache.SearchAgain()
}
// A filesystem the last complete search did not walk is not judged by it: not judged yet, and a search starts.
func TestAFilesystemTheSearchDidNotWalkIsNotJudgedByIt(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
_, _, err := c.Look(at.Add(time.Hour), []string{"/", "/srv"}, g.search)
notJudged(t, err, true)
if starts(c) != 2 {
t.Fatalf("a new filesystem was not searched: %d", starts(c))
}
g.finish(c)
if _, _, err := c.Look(at.Add(time.Hour), []string{"/", "/srv"}, g.search); err != nil {
t.Fatalf("once walked it judges: %v", err)
}
}
// An ask to search again forgets the walk under way, so an engine restarted before the next search does not
// carry on a walk that started before root acted.
func TestAnAskToSearchAgainIsNotLostToARestart(t *testing.T) {
dir := t.TempDir()
paths := func() *SetuidCache {
return &SetuidCache{Path: filepath.Join(dir, SetuidSearchFile), ProgressPath: filepath.Join(dir, SetuidProgressFile)}
}
seen := make(chan *SetuidProgress, 1)
first := paths()
first.Look(at, root, interrupted(seen))
<-seen
first.SearchAgain() // an apply moved root's things; then the engine restarts
restarted := paths()
var with SetuidProgress
restarted.Look(at.Add(time.Hour), root, func(ctx context.Context, p *SetuidProgress, _ func()) ([]string, error) {
with = *p
return nil, nil
})
restarted.Wait()
if !with.Started.Equal(at.Add(time.Hour)) || len(with.Found) != 0 {
t.Fatalf("the walk from before the apply was carried on: %+v", with)
}
}
// A cursor whose name is not UTF-8 is kept as its bytes: JSON strings would change it, and the resumed walk
// would skip what it stood before.
func TestACursorOfAnyBytesIsKeptExactly(t *testing.T) {
p := &SetuidProgress{Started: at, Mounts: Paths{"/"}, Cursor: Cursors{"/": "/home/agent/\x80/x"},
Found: Paths{"/opt/\xff"}}
var back SetuidProgress
fromJSON(t, mustJSON(t, p), &back)
if back.Cursor["/"] != "/home/agent/\x80/x" || back.Found[0] != "/opt/\xff" {
t.Fatalf("kept as %q, %q", back.Cursor["/"], back.Found[0])
}
}
func TestTheCompleteSearchIsKeptAndReadAfterARestart(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, SetuidSearchFile)
c := &SetuidCache{Path: path}
g := newGate()
g.found = []string{"/opt/x"}
c.Look(at, root, g.search)
g.finish(c)
info, err := os.Stat(path)
if err != nil || info.Mode().Perm() != 0o600 {
t.Fatalf("kept readable by its owner alone: %v %v", info, err)
}
restarted := &SetuidCache{Path: path}
found, when, err := restarted.Look(at.Add(time.Hour), root, newGate().search)
if err != nil || len(found) != 1 || !when.Equal(at) {
t.Fatalf("a restarted engine judges from the kept search: %v %v %v", found, when, err)
}
restarted.Wait()
// And not once it is too old.
again := &SetuidCache{Path: path}
g2 := newGate()
if _, _, err := again.Look(at.Add(rootsearch.FreshFor), root, g2.search); err == nil {
t.Fatal("a kept search too old to judge judged")
}
g2.finish(again)
}
func TestAKeptSearchNotRootsAloneIsNotRead(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, SetuidSearchFile)
c := &SetuidCache{Path: path}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
for name, spoil := range map[string]func() error{
"writable by others": func() error { return os.Chmod(path, 0o666) },
"a link": func() error {
other := filepath.Join(dir, "elsewhere.json")
if err := os.Rename(path, other); err != nil {
return err
}
return os.Symlink(other, path)
},
"its directory writable by others": func() error { return os.Chmod(dir, 0o777) },
} {
c = &SetuidCache{Path: path}
g = newGate()
c.Look(at, root, g.search)
g.finish(c) // a fresh, good one kept again
os.Chmod(dir, 0o700)
if err := spoil(); err != nil {
t.Fatal(err)
}
var said []string
r := &SetuidCache{Path: path, Say: func(s string) { said = append(said, s) }}
gr := newGate()
if _, _, err := r.Look(at.Add(time.Minute), root, gr.search); err == nil {
t.Fatalf("%s: a kept search that is not root's alone judged", name)
}
if len(said) == 0 {
t.Fatalf("%s: refusing the kept search said nothing", name)
}
gr.finish(r)
os.Chmod(dir, 0o700)
os.Remove(path)
}
}
func TestAKeptSearchFromTheFutureIsNotFresh(t *testing.T) {
c := &SetuidCache{}
g := newGate()
c.Look(at, root, g.search)
g.finish(c)
if _, _, err := c.Look(at.Add(-time.Hour), root, g.search); err == nil {
t.Fatal("a clock set back an hour read a search as fresh")
}
g.finish(c)
}
func TestTheIdleRunnerLowersItsCommand(t *testing.T) {
if _, err := exec.LookPath("nice"); err != nil {
t.Skip("no nice here")
}
out, err := Idle(t.Context(), "sh", "-c", "sleep 0.3; nice")
if err != nil {
t.Fatal(err)
}
if strings.TrimSpace(out) != "19" {
t.Fatalf("the search's commands run at nice %q, want 19", strings.TrimSpace(out))
}
if _, err := Idle(t.Context(), "sh", "-c", "echo gone >&2; exit 1"); err == nil ||
!strings.HasPrefix(err.Error(), "sh exited 1: gone") {
t.Fatalf("its failure is worded as CLocale's: %v", err)
}
}
func mustJSON(t *testing.T, v any) []byte {
t.Helper()
raw, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return raw
}
func fromJSON(t *testing.T, raw []byte, v any) {
t.Helper()
if err := json.Unmarshal(raw, v); err != nil {
t.Fatal(err)
}
}
// interrupted is a search that walks to a cursor, keeps it, and never returns: the engine stopped there.
func interrupted(seen chan<- *SetuidProgress) SetuidSearch {
return func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error) {
p.Found = append(p.Found, "/opt/a")
p.Cursor["/"] = "/opt/a"
checkpoint()
seen <- p
select {} // the engine stops here
}
}
func TestASearchInterruptedByARestartCarriesOnAndIsDatedByItsStart(t *testing.T) {
dir := t.TempDir()
paths := func() *SetuidCache {
return &SetuidCache{Path: filepath.Join(dir, SetuidSearchFile), ProgressPath: filepath.Join(dir, SetuidProgressFile)}
}
seen := make(chan *SetuidProgress, 1)
first := paths()
first.Look(at, root, interrupted(seen))
<-seen
// The engine restarts two hours later.
restarted := paths()
var resumedWith SetuidProgress
done := make(chan struct{})
_, _, err := restarted.Look(at.Add(2*time.Hour), root, func(ctx context.Context, p *SetuidProgress, _ func()) ([]string, error) {
resumedWith = *p
close(done)
return p.Found, nil
})
notJudged(t, err, true)
<-done
restarted.Wait()
if !resumedWith.Started.Equal(at) || resumedWith.Cursor["/"] != "/opt/a" || len(resumedWith.Found) != 1 {
t.Fatalf("the restarted search did not carry on the walk: %+v", resumedWith)
}
found, when, err := restarted.Look(at.Add(2*time.Hour), root, newGate().search)
if err != nil || !when.Equal(at) || len(found) != 1 {
t.Fatalf("the complete search is dated by its start, not its resumption: %v %v %v", found, when, err)
}
if _, err := os.Stat(filepath.Join(dir, SetuidProgressFile)); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("the finished walk's progress is still kept: %v", err)
}
}
func TestAWalkTooOldOrAskedAgainStartsOver(t *testing.T) {
for name, c := range map[string]struct {
after time.Duration
again bool
}{"older than it may judge": {rootsearch.FreshFor, false}, "an apply asked for a new one": {time.Hour, true}} {
dir := t.TempDir()
paths := func() *SetuidCache {
return &SetuidCache{Path: filepath.Join(dir, SetuidSearchFile), ProgressPath: filepath.Join(dir, SetuidProgressFile)}
}
seen := make(chan *SetuidProgress, 1)
paths().Look(at, root, interrupted(seen))
<-seen
restarted := paths()
if c.again {
restarted.SearchAgain()
}
var with SetuidProgress
restarted.Look(at.Add(c.after), root, func(ctx context.Context, p *SetuidProgress, _ func()) ([]string, error) {
with = *p
return nil, nil
})
restarted.Wait()
if !with.Started.Equal(at.Add(c.after)) || len(with.Cursor) != 0 || len(with.Found) != 0 {
t.Fatalf("%s: the old walk was carried on: %+v", name, with)
}
}
}
func TestASearchWhosePriorityCannotBeLoweredDoesNotWalk(t *testing.T) {
c := &SetuidCache{Lower: func() error { return errors.New("not permitted") }}
walked := false
c.Look(at, root, func(context.Context, *SetuidProgress, func()) ([]string, error) { walked = true; return nil, nil })
c.Wait()
_, _, err := c.Look(at.Add(time.Minute), root, newGate().search)
notJudged(t, err, false)
if walked {
t.Fatal("it walked at full priority")
}
}
+175
View File
@@ -0,0 +1,175 @@
package accounts
// The walk for setuid programs yields to a busy disk (novox/hq issue 433). It asks before each directory and each
// file it reads, so no burst between two looks is longer than PaceLook, however large a directory. The walk is put in the idle I/O class
// (idle_linux.go), but only a disk scheduler that schedules by class honours that (BFQ does). The control-node's
// disks run mq-deadline, which does not, and there the walk read at full priority: on 2026-10-11 it kept two
// spinning disks saturated for sixteen minutes, the bus whose store shares those disks stalled for half a minute
// twice, and the controller lost its lease each time. So the walk also paces itself by the machine's I/O pressure
// (PSI), which reads the same whatever the scheduler.
//
// It looks every PaceLook at the share of the time since its last look in which some task waited on I/O — from
// the kernel's cumulative stall counter, not its ten-second average, which moves only every two seconds and would
// let the walk read blind that long. Above PaceAbove it pauses, PaceStep at a time, until the share falls below.
//
// **The walk keeps at least a tenth of the time, whatever the pressure** (the review of #79): a run of pauses is
// at most PaceRatio times as long as the walk read since the one before. The control-node's own load sits near
// PaceAbove with no walk running, so a walk that paused for as long as the pressure stayed high would take days,
// outlive the age a search judges for, and leave the machine unjudged — the fault issue 361 repaired. With the
// floor a walk takes at most 1 + PaceRatio times as long as it would unpaced, and its bursts are PaceLook long.
import (
"bufio"
"bytes"
"context"
"fmt"
"os"
"strconv"
"strings"
"time"
)
const (
// PaceAbove is the share of the time since the last look, in percent, in which some task waited on I/O, above
// which the walk pauses.
PaceAbove = 10.0
// PaceLook is how long the walk reads between two looks at the pressure: the longest it reads blind.
PaceLook = 250 * time.Millisecond
// PaceStep is one pause, after which the pressure is looked at again.
PaceStep = 500 * time.Millisecond
// PaceRatio bounds a run of pauses to this many times the time the walk read since the run before: the walk
// keeps at least 1/(1+PaceRatio) of the time.
PaceRatio = 9
// PaceAtMost is the longest run of pauses, whatever the walk read before it.
PaceAtMost = 30 * time.Second
// ioPressureFile is where the kernel says the machine's I/O pressure.
ioPressureFile = "/proc/pressure/io"
)
// pacer is what the walk asks before it reads a directory. A nil pacer never pauses.
type pacer struct {
// path is the pressure file read, kept so a test can see the engine's own walker reads the machine's.
path string
// stalled is the cumulative time some task waited on I/O (PSI's "some total").
stalled func() (time.Duration, error)
sleep func(context.Context, time.Duration) error
now func() time.Time
say func(string)
lastLook time.Time // when the pressure was last looked at
readSince time.Time // when the walk last began to read, after a run of pauses or at its start
prevAt time.Time // the time of the last sample of the counter
prevTotal time.Duration // the counter at that sample
saidBlind bool
}
// newPacer paces by the machine's own I/O pressure, and says through say what it cannot do.
func newPacer(say func(string)) *pacer {
return &pacer{
path: ioPressureFile,
stalled: func() (time.Duration, error) { return readIOStalled(ioPressureFile) },
sleep: sleepCtx,
now: time.Now,
say: say,
}
}
// yield pauses while the machine's I/O pressure is above PaceAbove: looking at most every PaceLook, pausing
// PaceStep at a time, a run of pauses at most PaceRatio times what the walk read since the last run and never more
// than PaceAtMost. It ends early only with ctx, whose error ends the walk as incomplete. Pressure that cannot be
// read pauses nothing, and is said once.
func (pc *pacer) yield(ctx context.Context) error {
if pc == nil || pc.stalled == nil {
return nil
}
now := pc.now()
if pc.readSince.IsZero() {
pc.readSince = now
}
if !pc.lastLook.IsZero() && now.Sub(pc.lastLook) < PaceLook {
return nil
}
pc.lastLook = now
share, known, err := pc.sample(now)
if err != nil || !known || share <= PaceAbove {
return nil
}
budget := min(PaceAtMost, PaceRatio*now.Sub(pc.readSince))
for paused := time.Duration(0); paused < budget; {
step := min(PaceStep, budget-paused)
if err := pc.sleep(ctx, step); err != nil {
return err
}
paused += step
share, known, err = pc.sample(pc.now())
if err != nil || (known && share <= PaceAbove) {
break
}
}
resumed := pc.now()
pc.readSince, pc.lastLook = resumed, resumed
return nil
}
// sample is the share of the time since the last sample, in percent, in which some task waited on I/O; known is
// false for the first sample, which only sets where the next one counts from.
func (pc *pacer) sample(now time.Time) (float64, bool, error) {
total, err := pc.stalled()
if err != nil {
if !pc.saidBlind && pc.say != nil {
pc.say(fmt.Sprintf("the walk for setuid programs cannot read the machine's I/O pressure (%v), so it does "+
"not pace itself: where the disk's scheduler ignores the idle class, it may load the disk at full "+
"priority", err))
}
pc.saidBlind = true
return 0, false, err
}
prevAt, prevTotal := pc.prevAt, pc.prevTotal
pc.prevAt, pc.prevTotal = now, total
elapsed := now.Sub(prevAt)
if prevAt.IsZero() || elapsed <= 0 || total < prevTotal {
return 0, false, nil
}
return 100 * float64(total-prevTotal) / float64(elapsed), true, nil
}
// readIOStalled is the "some total" of a PSI file: the time, cumulative since boot, in which some task waited on I/O.
func readIOStalled(path string) (time.Duration, error) {
b, err := os.ReadFile(path)
if err != nil {
return 0, err
}
return parseIOStalled(b)
}
func parseIOStalled(b []byte) (time.Duration, error) {
sc := bufio.NewScanner(bytes.NewReader(b))
for sc.Scan() {
fields := strings.Fields(sc.Text())
if len(fields) == 0 || fields[0] != "some" {
continue
}
for _, f := range fields[1:] {
if v, ok := strings.CutPrefix(f, "total="); ok {
us, err := strconv.ParseUint(v, 10, 64)
if err != nil {
return 0, fmt.Errorf("the I/O pressure's total is not a count of microseconds: %q", v)
}
return time.Duration(us) * time.Microsecond, nil
}
}
}
return 0, fmt.Errorf("the I/O pressure names no \"some\" line with a total")
}
// sleepCtx waits d, or until ctx ends.
func sleepCtx(ctx context.Context, d time.Duration) error {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-t.C:
return nil
}
}
+302
View File
@@ -0,0 +1,302 @@
package accounts
import (
"context"
"errors"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/rootsearch"
)
// The walk for setuid programs yields to a busy disk by the machine's I/O pressure (novox/hq issue 433), because
// the idle I/O class it is put in is ignored by a disk scheduler that does not schedule by class, as the
// control-node's mq-deadline does not — and it keeps a tenth of the time whatever the pressure, so it still ends.
// fakePace is a pacer whose stall counter, sleeps and clock a test controls.
type fakePace struct {
// shares is the stall share, in percent, of each interval between two reads of the counter, in turn; the last
// is said again once they run out. The first read only sets the counter's start.
shares []float64
i int
total time.Duration
lastRead time.Time
reads int
clock time.Time
// tick moves the clock on at every look at it, as a walk reading directories takes time; zero does not.
tick time.Duration
slept []time.Duration
said []string
fail error
}
func newFake(shares ...float64) *fakePace {
return &fakePace{shares: shares, clock: time.Unix(1_000_000, 0)}
}
func (f *fakePace) pacer() *pacer {
return &pacer{
stalled: func() (time.Duration, error) {
if f.fail != nil {
return 0, f.fail
}
f.reads++
if !f.lastRead.IsZero() {
s := f.shares[min(f.i, len(f.shares)-1)]
f.i++
f.total += time.Duration(float64(f.clock.Sub(f.lastRead)) * s / 100)
}
f.lastRead = f.clock
return f.total, nil
},
sleep: func(ctx context.Context, d time.Duration) error {
if err := ctx.Err(); err != nil {
return err
}
f.slept = append(f.slept, d)
f.clock = f.clock.Add(d)
return nil
},
now: func() time.Time {
f.clock = f.clock.Add(f.tick)
return f.clock
},
say: func(s string) { f.said = append(f.said, s) },
}
}
func (f *fakePace) paused() time.Duration {
var total time.Duration
for _, d := range f.slept {
total += d
}
return total
}
func TestTheWalkPausesWhileTheDiskIsUnderPressureAndGoesOnWhenItEases(t *testing.T) {
f := newFake(60, 40, 3)
pc := f.pacer()
if err := pc.yield(t.Context()); err != nil { // the first look sets where the counter counts from
t.Fatal(err)
}
f.clock = f.clock.Add(300 * time.Millisecond) // the walk reads
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
// 60 % since the first look: pause; 40 % during the first pause: pause again; 3 %: go on.
if want := []time.Duration{PaceStep, PaceStep}; !slices.Equal(f.slept, want) {
t.Fatalf("slept %v, want %v", f.slept, want)
}
}
func TestTheFirstLookOnlySetsWhereTheNextCountsFrom(t *testing.T) {
f := newFake(99)
if err := f.pacer().yield(t.Context()); err != nil {
t.Fatal(err)
}
if len(f.slept) != 0 {
t.Fatalf("slept %v on the first look, which has nothing to compare", f.slept)
}
}
func TestTheWalkDoesNotPauseOnAQuietDisk(t *testing.T) {
f := newFake(0.4)
pc := f.pacer()
for range 10 {
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(PaceLook)
}
if len(f.slept) != 0 {
t.Fatalf("slept %v on a quiet disk", f.slept)
}
}
// The control-node's own load sits near the bound with no walk running: the walk must still end (issue 361).
func TestTheWalkKeepsATenthOfTheTimeWhateverThePressure(t *testing.T) {
f := newFake(100)
pc := f.pacer()
var read time.Duration
for range 400 {
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(PaceLook)
read += PaceLook
}
if f.paused() == 0 {
t.Fatal("never paused under constant pressure")
}
if f.paused() > PaceRatio*read {
t.Fatalf("paused %v while it read %v: more than %d times as long", f.paused(), read, PaceRatio)
}
}
// What the floor promises against the ages a search is judged by: the walk that took 16m31s unpaced on the
// control-node on 2026-10-11, slowed to its worst, still completes well inside the time the controller stays quiet
// while it waits for a search. A guard on the constants only: that the code keeps to the ratio is proven by
// TestTheWalkKeepsATenthOfTheTimeWhateverThePressure.
func TestTheSlowestPacedWalkStillJudgesInTime(t *testing.T) {
const unpaced = 16*time.Minute + 31*time.Second
worst := (1 + PaceRatio) * unpaced
if worst >= rootsearch.Quiet {
t.Fatalf("a walk paced to its worst takes %v, not less than the %v the controller waits quietly", worst,
rootsearch.Quiet)
}
}
func TestARunOfPausesIsAtMostPaceAtMost(t *testing.T) {
f := newFake(100)
pc := f.pacer()
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(10 * time.Minute) // read a long time without a look: the ratio would allow 90 minutes
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
if f.paused() != PaceAtMost {
t.Fatalf("paused %v in a row, want %v", f.paused(), PaceAtMost)
}
}
func TestTheWalkLooksAtThePressureAtMostEveryPaceLook(t *testing.T) {
f := newFake(1)
pc := f.pacer()
for range 5 {
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(PaceLook / 10)
}
if f.reads != 1 {
t.Fatalf("read the pressure %d times within %v, want once", f.reads, PaceLook)
}
f.clock = f.clock.Add(PaceLook)
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
if f.reads != 2 {
t.Fatalf("read the pressure %d times after %v, want twice", f.reads, PaceLook)
}
}
// Pressure that cannot be read pauses nothing, and is said once, with what it means, rather than at every look.
func TestAPressureThatCannotBeReadIsSaidOnce(t *testing.T) {
f := newFake(0)
f.fail = errors.New("no such file")
pc := f.pacer()
for range 3 {
if err := pc.yield(t.Context()); err != nil {
t.Fatal(err)
}
f.clock = f.clock.Add(PaceLook)
}
if len(f.slept) != 0 || len(f.said) != 1 {
t.Fatalf("slept %v, said %q; want no pause and one line", f.slept, f.said)
}
if !strings.Contains(f.said[0], "full priority") {
t.Errorf("the line does not say what it means for the disk: %q", f.said[0])
}
}
// The engine's own walk is paced by the machine's pressure, and says through the search's own channel.
func TestTheEnginesWalkIsPacedByTheMachinesPressure(t *testing.T) {
var said []string
w := engineWalker(nil, func(s string) { said = append(said, s) })
if w.pace == nil || w.pace.path != ioPressureFile || w.pace.stalled == nil {
t.Fatalf("the engine's walker is not paced by %s: %+v", ioPressureFile, w.pace)
}
w.pace.say("x")
if len(said) != 1 {
t.Fatal("the engine's walker does not say through the search's channel")
}
}
// many is a tree of enough directories that a walk reaches a look at the pressure partway.
func many(t *testing.T) string {
names := []string{"bin/", "bin/su!", "bin/ls", "sbin/", "sbin/mount!", "usr/lib/helper%"}
for _, d := range []string{"a", "b", "c", "d", "e", "f", "g", "h"} {
names = append(names, "usr/share/"+d+"/")
}
return tree(t, names...)
}
// A walk ended while it pauses is an incomplete walk, never one that found nothing.
func TestAWalkEndedWhileItPausesIsIncomplete(t *testing.T) {
top := many(t)
f := newFake(80)
f.tick = 100 * time.Millisecond
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
pc := f.pacer()
// The engine stops while the walk waits for the disk: the context ends inside the pause, not before the walk.
pc.sleep = func(ctx context.Context, d time.Duration) error {
f.slept = append(f.slept, d)
cancel()
return ctx.Err()
}
w := mine()
w.pace = pc
err := w.walk(ctx, &SetuidProgress{Mounts: []string{top}}, func() {})
if len(f.slept) == 0 {
t.Fatal("the walk never paused, so this proves nothing about a walk ended while it pauses")
}
if !errors.Is(err, context.Canceled) {
t.Fatalf("a walk ended while it paused answered %v", err)
}
}
// Paced, the walk finds what it finds unpaced: pausing changes when, never what.
func TestAPacedWalkFindsWhatAnUnpacedWalkFinds(t *testing.T) {
top := many(t)
plain := walkAll(t, mine(), top)
f := newFake(80, 2)
f.tick = 100 * time.Millisecond
w := mine()
w.pace = f.pacer()
paced := walkAll(t, w, top)
if !slices.Equal(plain, paced) {
t.Fatalf("paced walk found %v, unpaced %v", paced, plain)
}
if len(f.slept) == 0 {
t.Fatal("the walk read its directories without asking the pacer: it never paused under pressure")
}
}
func TestTheIOPressureIsReadAsTheKernelWritesIt(t *testing.T) {
got, err := parseIOStalled([]byte("some avg10=26.24 avg60=20.07 avg300=27.47 total=123456789\n" +
"full avg10=21.90 avg60=15.77 avg300=22.47 total=98765432\n"))
if err != nil || got != 123456789*time.Microsecond {
t.Fatalf("read %v, %v; want 123456789 µs", got, err)
}
for _, bad := range []string{"", "full avg10=1.00 avg60=0 avg300=0 total=1\n", "some avg10=1 total=x\n"} {
if _, err := parseIOStalled([]byte(bad)); err == nil {
t.Errorf("%q was read as a pressure", bad)
}
}
}
// One directory of many files is paced as it is read, not only between directories: a file's own details may be
// read from the disk too.
func TestADirectoryOfManyFilesIsPacedAsItIsRead(t *testing.T) {
names := []string{"bin/"}
for i := range 20 {
names = append(names, "bin/f"+strings.Repeat("x", i+1))
}
top := tree(t, names...)
f := newFake(80, 2)
f.tick = 100 * time.Millisecond
w := mine()
w.pace = f.pacer()
walkAll(t, w, top)
// Two directories alone would be looked at once, 200 ms apart, within one look: no pause at all.
if len(f.slept) == 0 {
t.Fatal("a directory of twenty files was read without a pause under pressure")
}
}
+218
View File
@@ -0,0 +1,218 @@
package accounts
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// How far a running search got is readable through the verdict (novox/hq issue 361): counts, times and mount
// points, counted once across a resume, and never a path the walk visited or found.
// A walk stopped midway and resumed counts every entry once, as the walk that was never stopped does, and says
// how long each filesystem took.
func TestTheProgressCountsEachEntryOnceAcrossAResume(t *testing.T) {
top := tree(t, "a/1!", "a/2", "a-b/3!", "b/c/4!", "b/c/5", "b/d!", "c/6!", "d/7!")
other := tree(t, "x/1", "y!")
whole := &SetuidProgress{Mounts: []string{top, other}}
if err := mine().walk(t.Context(), whole, func() {}); err != nil {
t.Fatal(err)
}
// top and the 14 entries below it, and other and its 3.
if whole.Visited != 15+4 || len(whole.Took) != 2 || whole.Took[0] < 0 || whole.Walking != 0 {
t.Fatalf("the whole walk: visited %d, took %v, walking %v", whole.Visited, whole.Took, whole.Walking)
}
w := mine()
w.every = 0
ctx, stop := context.WithCancel(t.Context())
p := &SetuidProgress{Mounts: []string{top, other}, Counted: true}
var kept []byte
err := w.walk(ctx, p, func() {
kept = mustJSON(t, p)
if strings.HasSuffix(p.Cursor[top], "/b/c/4") {
stop()
}
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("the walk was stopped: %v", err)
}
var resumed SetuidProgress
fromJSON(t, kept, &resumed)
if resumed.Visited == 0 || resumed.Visited >= whole.Visited || resumed.Walking < 0 {
t.Fatalf("the kept walk counted %d of %d", resumed.Visited, whole.Visited)
}
if err := mine().walk(t.Context(), &resumed, func() {}); err != nil {
t.Fatal(err)
}
if resumed.Visited != whole.Visited || len(resumed.Took) != 2 || resumed.Took[0] < 0 {
t.Fatalf("the resumed walk counted %d entries (the whole one %d), took %v", resumed.Visited,
whole.Visited, resumed.Took)
}
}
// A walk kept before the times were counted has them not known, never shifted onto the next filesystem, and
// its entries visited are a lower bound; the filesystem it was part-way through took a time not known.
func TestAWalkKeptWithoutTimesSaysThemNotKnown(t *testing.T) {
top, other := tree(t, "a"), tree(t, "b")
p := &SetuidProgress{Mounts: []string{top, other}, Done: Paths{top}}
if err := mine().walk(t.Context(), p, func() {}); err != nil {
t.Fatal(err)
}
if len(p.Took) != 2 || p.Took[0] >= 0 || p.Took[1] < 0 {
t.Fatalf("took %v", p.Took)
}
if w := progressOf(p, time.Time{}).Words(); !strings.Contains(w, top+" in a time not known") ||
!strings.Contains(w, "at least ") {
t.Fatalf("the words: %s", w)
}
}
// A look while the search runs says how far it got at its last checkpoint, with no path it visited or found.
func TestALookSaysHowFarTheRunningSearchGotAndNoPath(t *testing.T) {
dir := t.TempDir()
c := &SetuidCache{ProgressPath: filepath.Join(dir, SetuidProgressFile)}
secret := "/home/someone/private/rootshell"
checkpointed, release := make(chan struct{}), make(chan struct{})
search := func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error) {
p.Mounts = Paths{"/", "/home", "/boot"}
p.Done, p.Took = Paths{"/"}, []time.Duration{90 * time.Minute}
p.Cursor["/home"] = "/home/someone/private"
p.Found = Paths{"/usr/bin/odd", secret}
p.Visited = 1234567
checkpoint()
close(checkpointed)
<-release
return nil, nil
}
_, _, err := c.Look(at, root, search)
nj := notJudged(t, err, true)
if nj.Progress == nil || nj.Progress.Visited != 0 || nj.Progress.Walking != "/" {
t.Fatalf("before the first checkpoint: %+v", nj.Progress)
}
<-checkpointed
_, _, err = c.Look(at.Add(time.Minute), root, search)
nj = notJudged(t, err, true)
sp := nj.Progress
if sp == nil || sp.Filesystems != 3 || sp.Done != 1 || sp.Walking != "/home" || sp.Visited != 1234567 ||
sp.Found != 2 || sp.Kept.IsZero() || len(sp.Finished) != 1 || sp.Finished[0].Took != 90*time.Minute {
t.Fatalf("the progress: %+v", sp)
}
words := sp.Words()
for _, want := range []string{"1 of 3 filesystems walked", "walking /home", "1234567 entries visited",
"2 setuid- or setgid-root files found so far", "kept " + sp.Kept.Local().Format("2006-01-02 15:04:05"),
"walked / in 1h30m0s"} {
if !strings.Contains(words, want) {
t.Errorf("the words lack %q: %s", want, words)
}
}
for _, leak := range []string{"someone", "private", "odd", "rootshell"} {
if strings.Contains(words, leak) || strings.Contains(err.Error(), leak) {
t.Errorf("a path leaks (%q): %s / %v", leak, words, err)
}
}
close(release)
c.Wait()
if _, _, err := c.Look(at.Add(2*time.Minute), root, search); err != nil {
t.Fatalf("the complete search judges: %v", err)
}
}
// The progress moves every minute; the verdict does not change for it, so the journal says it once.
type waiting struct{ progress *SearchProgress }
func (w *waiting) InDatabase(context.Context, string) ([]string, error) { return nil, nil }
func (w *waiting) Session(context.Context, string, []string) (Session, error) {
return Session{}, nil
}
func (w *waiting) Escalation(context.Context, string, []string) ([]string, error) {
return nil, &SetuidNotJudged{Pending: true, Since: at, Progress: w.progress}
}
func TestTheProgressMovesWithoutChangingTheVerdict(t *testing.T) {
r := &waiting{progress: &SearchProgress{Filesystems: 2, Visited: 10}}
j := New(r)
j.Set([]Account{{Module: "claude-code", ID: "claude-code.agent", Name: "agent", Root: true}})
st, changed := j.Look(t.Context())
if !changed || st.Accounts[0].State != Unknown || st.Accounts[0].Search.Visited != 10 {
t.Fatalf("first look: changed %v, %+v", changed, st.Accounts[0])
}
r.progress = &SearchProgress{Filesystems: 2, Visited: 20}
st, changed = j.Look(t.Context())
if changed || st.Accounts[0].Search.Visited != 20 || strings.Contains(st.Accounts[0].Reason, "progress") {
t.Fatalf("second look: changed %v, %+v", changed, st.Accounts[0])
}
}
// Asked at the terminal, a search starts afresh at the next look: the last complete one is thrown away, kept or
// in memory, and a running one is ended and neither judges nor is carried on.
func TestSearchNowThrowsAwayTheLastSearchAndStartsAFullOne(t *testing.T) {
dir := t.TempDir()
c := &SetuidCache{Path: filepath.Join(dir, SetuidSearchFile), ProgressPath: filepath.Join(dir, SetuidProgressFile)}
g := newGate()
g.found = []string{"/home/someone/rootshell"}
c.Look(at, root, g.search)
<-g.started
g.finish(c)
if found, _, err := c.Look(at.Add(time.Hour), root, g.search); err != nil || len(found) != 1 {
t.Fatalf("the complete search judges: %v %v", found, err)
}
said := c.SearchNow("jo at the controller's terminal")
if !strings.Contains(said, "thrown away") || strings.Contains(said, "someone") {
t.Fatalf("said: %s", said)
}
if _, err := os.Stat(c.Path); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("the kept search is still there: %v", err)
}
// The next look starts a full search at once, though the last ended only an hour ago.
ended := make(chan error, 1)
running := func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error) {
if len(p.Done) != 0 || len(p.Cursor) != 0 {
t.Errorf("not a full search: %+v", p)
}
checkpoint()
<-ctx.Done()
ended <- ctx.Err()
return nil, ctx.Err()
}
_, _, err := c.Look(at.Add(time.Hour+time.Minute), root, running)
notJudged(t, err, true)
if starts(c) != 2 {
t.Fatalf("searches started: %d", starts(c))
}
// Asked again while it runs: it is ended, judges nothing, fails nothing, and is not carried on.
c.SearchNow("jo")
if err := <-ended; !errors.Is(err, context.Canceled) {
t.Fatalf("the running search was not ended: %v", err)
}
c.Wait()
if _, err := os.Stat(c.ProgressPath); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("the ended walk is still kept: %v", err)
}
g2 := newGate()
_, _, err = c.Look(at.Add(time.Hour+2*time.Minute), root, g2.search)
notJudged(t, err, true)
<-g2.started
g2.finish(c)
if starts(c) != 3 {
t.Fatalf("searches started: %d", starts(c))
}
if _, _, err := c.Look(at.Add(time.Hour+3*time.Minute), root, g2.search); err != nil {
t.Fatalf("the new search judges: %v", err)
}
}
func TestAFilesystemPartWalkedByAnEngineThatDidNotCountTookATimeNotKnown(t *testing.T) {
top := tree(t, "a", "b")
p := &SetuidProgress{Mounts: []string{top}, Cursor: Cursors{top: filepath.Join(top, "a")}}
if err := mine().walk(t.Context(), p, func() {}); err != nil {
t.Fatal(err)
}
if len(p.Took) != 1 || p.Took[0] >= 0 {
t.Fatalf("took %v", p.Took)
}
}
+115
View File
@@ -0,0 +1,115 @@
package accounts
import (
"io/fs"
"os"
"path/filepath"
"strings"
"testing"
)
// A setuid program no package owns counts against an account only where it can execute it, read at every
// judgement from the modes on the way (novox/hq issue 361): the operator's copies in a home of mode 0700 are no
// way for the agent account, and become one the moment the home is opened, without a new search.
const homeFind = "/home/op/usr/bin/mount"
func homeMachine(home FileMode) *agentMachine {
m := clean()
m.setuid = homeFind + "\n"
m.nodes = map[string]FileMode{"/home/op": home}
return m
}
func TestReachabilityDecidesWhetherAFindCounts(t *testing.T) {
op := func(perm fs.FileMode) FileMode { return FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | perm} }
for _, c := range []struct {
name string
set func(*agentMachine)
counted bool
}{
{"a home of 0700", func(m *agentMachine) {}, false},
{"a home of 0755", func(m *agentMachine) { m.nodes["/home/op"] = op(0o755) }, true},
{"a home its group may search, the agent in it", func(m *agentMachine) {
m.nodes["/home/op"], m.gids = op(0o710), "1600 1500"
}, true},
{"a home its group may search, the agent not in it", func(m *agentMachine) { m.nodes["/home/op"] = op(0o710) }, false},
{"an ACL on the home", func(m *agentMachine) {
m.files["/home/op"] = FileMode{UID: 1500, GID: 1500, Perm: 0o700}
m.acls = map[string][]ACLEntry{"/home/op": {{}}}
}, true},
{"a link on the way", func(m *agentMachine) {
m.nodes["/home/op"] = FileMode{UID: 1500, GID: 1500, Perm: fs.ModeSymlink | 0o777}
}, true},
{"a file the agent may not execute", func(m *agentMachine) {
m.nodes["/home/op"] = op(0o755)
m.nodes[homeFind] = FileMode{Perm: fs.ModeSetuid | 0o4750}
}, false},
} {
t.Run(c.name, func(t *testing.T) {
m := homeMachine(FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | 0o700})
c.set(m)
v := lookAgent(t, m)
if c.counted != (v.State == Unhealthy) {
t.Fatalf("counted %v: %+v", c.counted, v)
}
if strings.Contains(v.Reason, "/home/op") {
t.Fatalf("a path in a home was said: %s", v.Reason)
}
if c.counted && !strings.Contains(v.Reason, "a setuid-root program no package owns in an account's home, which agent can execute") {
t.Fatalf("the reason: %s", v.Reason)
}
})
}
}
// The search keeps every find; whether one counts is read again at every look.
func TestAHomeOpenedAfterTheSearchCountsWithoutANewSearch(t *testing.T) {
m := homeMachine(FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | 0o700})
c := &SetuidCache{}
j := New(Exec{Run: m.run, Walk: m.walk, Stat: m.stat, ReadFile: m.readFile, ReadDir: m.readDir, ACL: m.acl,
Lstat: m.lstat, Cache: c})
j.Set([]Account{agent})
j.Look(t.Context())
c.Wait()
if st, _ := j.Look(t.Context()); st.Accounts[0].State != Healthy {
t.Fatalf("a find in a closed home: %+v", st.Accounts[0])
}
m.nodes["/home/op"] = FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | 0o755}
st, _ := j.Look(t.Context())
if st.Accounts[0].State != Unhealthy || starts(c) != 1 {
t.Fatalf("the home opened: %+v, %d searches", st.Accounts[0], starts(c))
}
m.nodes["/home/op"] = FileMode{UID: 1500, GID: 1500, Perm: fs.ModeDir | 0o700}
if st, _ := j.Look(t.Context()); st.Accounts[0].State != Healthy || starts(c) != 1 {
t.Fatalf("the home closed again: %+v", st.Accounts[0])
}
}
// On the machine's own modes, as their owner (the directories above a test's are its own): a directory without
// its owner's search bit hides what is in it.
func TestExecutableReadsTheMachinesModes(t *testing.T) {
if os.Getuid() == 0 {
t.Skip("root searches every directory")
}
top := tree(t, "home/bin/x!")
home := filepath.Join(top, "home")
me := os.Getuid()
noACL := func(string) ([]ACLEntry, error) { return nil, nil }
if err := os.Chmod(home, 0o600); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = os.Chmod(home, 0o700) })
if Executable(filepath.Join(home, "bin/x"), me, map[int]bool{}, lstatOf, noACL) {
t.Fatal("a file in a directory its account may not search was executable")
}
if err := os.Chmod(home, 0o700); err != nil {
t.Fatal(err)
}
if !Executable(filepath.Join(home, "bin/x"), me, map[int]bool{}, lstatOf, noACL) {
t.Fatal("a file every directory opens was not executable")
}
if Executable(filepath.Join(home, "bin/gone"), me, map[int]bool{}, lstatOf, noACL) {
t.Fatal("a file gone was executable")
}
}
+313
View File
@@ -0,0 +1,313 @@
package accounts
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
"syscall"
"time"
)
// The walk for setuid programs, resumable (novox/hq issue 361).
//
// The walk is the engine's own, in Go rather than find, so it can say how far it got: filepath.WalkDir visits
// every filesystem's tree depth first, each directory's entries in lexical order, so the walk's progress is one
// cursor per filesystem — the last path it visited — and everything before the cursor in that order is done.
// The progress is kept every CheckpointEvery beside the engine's state (SetuidCache), so an engine restarted,
// updated or rebooted midway carries on from its cursor rather than starting over: on the control-node the
// whole walk takes far longer than an engine runs between updates.
//
// **Why a resumed walk is sound.** Only root can make a setuid- or setgid-root file: the agent account can
// neither set the bit on root's file nor give its own file to root. A file that appears behind the cursor while
// the walk runs, or while it waits to be resumed, was made by root — and root's acts through the mesh start a
// new search (apply.MovesRoot), whose walk starts from nothing. A walk is dated by its start, and one resumed
// more than FreshFor after its start is thrown away and begun again.
// SetuidProgress is one walk, as far as it got.
type SetuidProgress struct {
// ID names the walk; Started is when it began, on the cache's clock: the search is dated by it.
ID string `json:"id"`
Started time.Time `json:"started"`
// Mounts are the filesystems it walks, root first; Done those walked to their end.
Mounts Paths `json:"mounts"`
Done Paths `json:"done"`
// Cursor is, for a filesystem being walked, the last path visited.
Cursor Cursors `json:"cursor"`
// Found is every setuid- or setgid-root regular file found so far, before the package manager is asked.
Found Paths `json:"found"`
// Visited is how many entries the walk has gone past: each once, across a resume, as it is kept with the
// cursor it matches.
Visited int64 `json:"visited"`
// Took is how long the walk spent in each of Done, in its order (negative: not known, a walk kept by an
// engine before it counted); Walking how long it has spent in the filesystem it walks now. Time the
// engine did not run is in neither.
Took []time.Duration `json:"took"`
Walking time.Duration `json:"walking"`
// Kept is when this progress was kept, as the cache wrote it; zero before the first time.
Kept time.Time `json:"kept"`
// Counted is a walk counted from its start: false for one kept by an engine before it counted, whose
// Visited is then a lower bound and whose filesystem under way took a time not known (the review of
// 2026-10-10).
Counted bool `json:"counted"`
}
// Paths are paths kept as their bytes. **Never as JSON strings**: encoding/json replaces bytes that are not
// UTF-8 with U+FFFD, and a cursor so changed sorts after the names it stood before — a resumed walk would skip
// them unread (the review of 2026-10-10). A name of any bytes is any account's to make.
type Paths []string
func (ps Paths) MarshalJSON() ([]byte, error) {
raw := make([][]byte, len(ps))
for i, p := range ps {
raw[i] = []byte(p)
}
return json.Marshal(raw)
}
func (ps *Paths) UnmarshalJSON(b []byte) error {
var raw [][]byte
if err := json.Unmarshal(b, &raw); err != nil {
return err
}
*ps = make(Paths, len(raw))
for i, r := range raw {
(*ps)[i] = string(r)
}
return nil
}
// Cursors are each filesystem's cursor, kept as bytes as Paths are.
type Cursors map[string]string
type keptCursor struct {
Mount []byte `json:"mount"`
Path []byte `json:"path"`
}
func (cs Cursors) MarshalJSON() ([]byte, error) {
raw := []keptCursor{}
for m, p := range cs {
raw = append(raw, keptCursor{Mount: []byte(m), Path: []byte(p)})
}
slices.SortFunc(raw, func(a, b keptCursor) int { return strings.Compare(string(a.Mount), string(b.Mount)) })
return json.Marshal(raw)
}
func (cs *Cursors) UnmarshalJSON(b []byte) error {
var raw []keptCursor
if err := json.Unmarshal(b, &raw); err != nil {
return err
}
*cs = Cursors{}
for _, r := range raw {
(*cs)[string(r.Mount)] = string(r.Path)
}
return nil
}
// CheckpointEvery is how often a walk's progress is kept.
const CheckpointEvery = time.Minute
// prunedPaths are never walked: the kernel's own trees, and the container runtimes' layers.
var prunedPaths = []string{"/proc", "/sys", "/var/lib/docker", "/var/lib/containers"}
// walker walks for setuid programs. rootUID and rootGID are the owner and group a match has: root's (0), but
// for a test. mountPoints are every filesystem mounted on the machine (MountPoints): a directory that is one is
// another filesystem, left without being read — a FUSE mount root cannot read, or a network mount whose
// server is away, would fail or hang the walk (the review of 2026-10-10).
type walker struct {
rootUID, rootGID uint32
every time.Duration
mountPoints map[string]bool
// pace is asked before each directory is read (setuid_pace.go); nil never pauses.
pace *pacer
}
// walk carries p on to its end, calling checkpoint every w.every and after each filesystem. It ends early only
// with an error — a directory it could not read, or ctx ended — which is an incomplete walk, never "none".
// A file or directory that vanished while it walked is not an error: what still exists was walked.
func (w walker) walk(ctx context.Context, p *SetuidProgress, checkpoint func()) error {
if p.Cursor == nil {
p.Cursor = Cursors{}
}
// One Took for each of Done: a walk kept before they were counted has them not known.
for len(p.Took) < len(p.Done) {
p.Took = append(p.Took, -1)
}
p.Took = p.Took[:len(p.Done)]
if !p.Counted && len(p.Cursor) > 0 {
p.Walking = -1 // walked partly by an engine that did not count its time
}
last := time.Now()
for _, m := range p.Mounts {
if slices.Contains(p.Done, m) {
continue
}
if err := w.walkMount(ctx, p, m, checkpoint, &last); err != nil {
return err
}
p.Done = append(p.Done, m)
p.Took = append(p.Took, p.Walking)
p.Walking = 0
delete(p.Cursor, m)
checkpoint()
last = time.Now()
}
// Looked at again: a file found that is gone, or no longer setuid-root, is no answer the package manager
// can give, and would fail every search that resumes this walk (the review of 2026-10-10).
var still Paths
for _, f := range p.Found {
info, err := os.Lstat(f)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return fmt.Errorf("the walk for setuid programs could not read %s again: %w", f, err)
}
if w.setuidRoot(info) {
still = append(still, f)
}
}
slices.Sort(still)
p.Found = slices.Compact(still)
return nil
}
func (w walker) walkMount(ctx context.Context, p *SetuidProgress, mount string, checkpoint func(), last *time.Time) error {
top, err := os.Lstat(mount)
if err != nil {
return fmt.Errorf("the filesystem at %s could not be read: %w", mount, err)
}
dev, ok := deviceOf(top)
if !ok || !top.IsDir() {
return fmt.Errorf("the filesystem at %s is not a directory whose device could be read", mount)
}
var cursor []string
if c := p.Cursor[mount]; c != "" {
cursor = components(c)
}
// How long it walks this filesystem, carried on from what a resumed walk had spent in it.
began, before := time.Now(), p.Walking
spent := func() time.Duration {
if before < 0 {
return -1
}
return before + time.Since(began)
}
defer func() { p.Walking = spent() }()
return filepath.WalkDir(mount, func(path string, d fs.DirEntry, err error) error {
if ctxErr := ctx.Err(); ctxErr != nil {
return ctxErr
}
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return nil // vanished while it walked
}
return fmt.Errorf("the walk for setuid programs could not read %s: %w", path, err)
}
at := components(path)
if cursor != nil && walkOrder(at, cursor) < 0 && !isPrefix(at, cursor) {
// Before the cursor and not on its way: visited by the walk this resumes.
if d.IsDir() {
return filepath.SkipDir
}
return nil
}
switch {
case d.IsDir():
if path != mount && (slices.Contains(prunedPaths, path) || w.mountPoints[path]) {
return filepath.SkipDir
}
info, err := os.Lstat(path)
if errors.Is(err, fs.ErrNotExist) {
return filepath.SkipDir
}
if err != nil {
return fmt.Errorf("the walk for setuid programs could not read %s: %w", path, err)
}
if d, ok := deviceOf(info); !ok || d != dev {
return filepath.SkipDir // another filesystem, walked as its own or not at all (find's -xdev)
}
// Before its entries are read: a busy disk is waited for here (novox/hq issue 433).
if err := w.pace.yield(ctx); err != nil {
return err
}
case d.Type().IsRegular():
// A file's own details may be read from the disk too: a directory of many files is paced as it is read.
if err := w.pace.yield(ctx); err != nil {
return err
}
info, err := d.Info() // the entry's own, never through a link
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
return fmt.Errorf("the walk for setuid programs could not read %s: %w", path, err)
}
if w.setuidRoot(info) {
p.Found = append(p.Found, path)
}
}
// The cursor only moves on: the directories on the way back to it are visited again, and are before it.
// Counted here alone, so an entry visited again on the way back to the cursor is counted once.
if cursor == nil || walkOrder(at, cursor) > 0 {
p.Cursor[mount], cursor = path, at
p.Visited++
}
if time.Since(*last) >= w.every {
p.Walking = spent()
checkpoint()
*last = time.Now()
}
return nil
})
}
// setuidRoot is a regular file setuid and owned by root, or setgid to root's group.
func (w walker) setuidRoot(info fs.FileInfo) bool {
st, ok := info.Sys().(*syscall.Stat_t)
if !ok || !info.Mode().IsRegular() {
return false
}
return uint32(st.Uid) == w.rootUID && info.Mode()&fs.ModeSetuid != 0 ||
uint32(st.Gid) == w.rootGID && info.Mode()&fs.ModeSetgid != 0
}
func deviceOf(info fs.FileInfo) (uint64, bool) {
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return 0, false
}
return uint64(st.Dev), true
}
// components are a path's names, in order; "/" has none.
func components(path string) []string {
path = strings.Trim(filepath.Clean(path), "/")
if path == "" {
return nil
}
return strings.Split(path, "/")
}
// walkOrder compares two paths in the order WalkDir visits them: name by name, each in lexical order, a
// directory before what is in it.
func walkOrder(a, b []string) int {
for i := 0; i < len(a) && i < len(b); i++ {
if c := strings.Compare(a[i], b[i]); c != 0 {
return c
}
}
return len(a) - len(b)
}
// isPrefix says a is b or a directory b is in.
func isPrefix(a, b []string) bool {
return len(a) <= len(b) && slices.Equal(a, b[:len(a)])
}
+213
View File
@@ -0,0 +1,213 @@
package accounts
import (
"context"
"errors"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
)
// The engine's own walk for setuid programs (novox/hq issue 361): what it finds, what it never follows, and that
// a walk interrupted carries on from its cursor.
// tree makes files under a directory of its own: a name ending in "/" a directory, "!" setuid, "%" setgid.
func tree(t *testing.T, names ...string) string {
t.Helper()
top := t.TempDir()
for _, n := range names {
mode := os.FileMode(0o755)
switch {
case strings.HasSuffix(n, "/"):
if err := os.MkdirAll(filepath.Join(top, n), 0o755); err != nil {
t.Fatal(err)
}
continue
case strings.HasSuffix(n, "!"):
n, mode = strings.TrimSuffix(n, "!"), mode|os.ModeSetuid
case strings.HasSuffix(n, "%"):
n, mode = strings.TrimSuffix(n, "%"), mode|os.ModeSetgid
}
p := filepath.Join(top, n)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chmod(p, mode); err != nil {
t.Fatal(err)
}
}
return top
}
// mine is a walker for which this test's own account is root.
func mine() walker {
return walker{rootUID: uint32(os.Getuid()), rootGID: uint32(os.Getgid()), every: time.Hour}
}
func walkAll(t *testing.T, w walker, top string) []string {
t.Helper()
p := &SetuidProgress{Mounts: []string{top}}
if err := w.walk(t.Context(), p, func() {}); err != nil {
t.Fatal(err)
}
return rel(top, p.Found)
}
func rel(top string, paths []string) []string {
var out []string
for _, p := range paths {
out = append(out, strings.TrimPrefix(p, top+"/"))
}
return out
}
func TestTheWalkFindsSetuidAndSetgidRootFilesAndFollowsNoLink(t *testing.T) {
top := tree(t, "bin/plain", "bin/rootshell!", "sbin/group%", "deep/a/b/c/x!")
if err := os.Symlink(filepath.Join(top, "bin/rootshell"), filepath.Join(top, "link")); err != nil {
t.Fatal(err)
}
if err := os.Symlink(filepath.Join(top, "bin"), filepath.Join(top, "linkdir")); err != nil {
t.Fatal(err)
}
found := walkAll(t, mine(), top)
if strings.Join(found, ",") != "bin/rootshell,deep/a/b/c/x,sbin/group" {
t.Fatalf("found %v", found)
}
if got := walkAll(t, walker{rootUID: 4242424, rootGID: 4242424, every: time.Hour}, top); len(got) != 0 {
t.Fatalf("files nobody's root owns were found: %v", got)
}
}
func TestAWalkThatCannotReadADirectoryIsIncomplete(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads every directory")
}
top := tree(t, "open/x!", "closed/y!")
if err := os.Chmod(filepath.Join(top, "closed"), 0); err != nil {
t.Fatal(err)
}
defer os.Chmod(filepath.Join(top, "closed"), 0o755)
p := &SetuidProgress{Mounts: []string{top}}
if err := mine().walk(t.Context(), p, func() {}); err == nil {
t.Fatal("a walk that could not read a directory was complete")
}
}
func TestWalkOrderIsNameByName(t *testing.T) {
// "/a/b" is walked before "/a-c" although '-' sorts before '/': the walk goes name by name.
if walkOrder(components("/a/b"), components("/a-c")) >= 0 {
t.Fatal("/a/b is walked before /a-c")
}
if walkOrder(components("/a"), components("/a/b")) >= 0 || !isPrefix(components("/a"), components("/a/b")) {
t.Fatal("a directory is walked before what is in it")
}
if len(components("/")) != 0 {
t.Fatal("/ has no names")
}
}
// A walk stopped midway carries on from its cursor: what was before the cursor is not walked again, what is after
// it is, and every file of the first walk is found.
func TestAnInterruptedWalkCarriesOnFromItsCursor(t *testing.T) {
top := tree(t, "a/1!", "a/2", "a-b/3!", "b/c/4!", "b/c/5", "b/d!", "c/6!", "d/7!")
w := mine()
w.every = 0 // a checkpoint at every path
ctx, stop := context.WithCancel(t.Context())
p := &SetuidProgress{Mounts: []string{top}}
var kept []byte
visits := 0
err := w.walk(ctx, p, func() {
visits++
kept = mustJSON(t, p)
if strings.HasSuffix(p.Cursor[top], "/b/c/4") {
stop() // the engine stops here
}
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("the walk was stopped: %v", err)
}
var resumed SetuidProgress
fromJSON(t, kept, &resumed)
if !strings.HasSuffix(resumed.Cursor[top], "/b/c/4") {
t.Fatalf("the kept cursor is %q", resumed.Cursor[top])
}
// Meanwhile, a file appears before the cursor and one after it.
for _, n := range []string{"a/late!", "e/late!"} {
tree2(t, top, n)
}
w.every = time.Hour
if err := w.walk(t.Context(), &resumed, func() {}); err != nil {
t.Fatal(err)
}
got := rel(top, resumed.Found)
want := []string{"a-b/3", "a/1", "b/c/4", "b/d", "c/6", "d/7", "e/late"}
if !slices.Equal(got, want) {
t.Fatalf("the resumed walk found %v, want %v: the first walk's finds kept, nothing before the cursor "+
"walked again, everything after it walked", got, want)
}
}
func tree2(t *testing.T, top, n string) {
t.Helper()
p := filepath.Join(top, strings.TrimSuffix(n, "!"))
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chmod(p, 0o755|os.ModeSetuid); err != nil {
t.Fatal(err)
}
}
// A search's commands and walk run on a lowered thread while the rest of the engine keeps its priority.
func TestTheSearchRunsOnALoweredThreadOfItsOwn(t *testing.T) {
var wg sync.WaitGroup
wg.Add(1)
var err error
go func() {
defer wg.Done()
err = LowerThisThread()
}()
wg.Wait()
if err != nil {
t.Fatalf("the thread could not be lowered: %v", err)
}
}
// Another filesystem's mount point is left without being read, and a file found that is gone, or no longer
// setuid-root, when the walk ends is not asked about.
func TestTheWalkLeavesMountPointsAndDropsWhatWentMeanwhile(t *testing.T) {
top := tree(t, "fuse/secret!", "bin/x!", "bin/y!")
w := mine()
w.mountPoints = map[string]bool{filepath.Join(top, "fuse"): true}
w.every = 0
p := &SetuidProgress{Mounts: []string{top}}
removed := false
if err := w.walk(t.Context(), p, func() {
if !removed && strings.HasSuffix(p.Cursor[top], "/bin/y") {
removed = true
os.Remove(filepath.Join(top, "bin/x"))
}
}); err != nil {
t.Fatal(err)
}
if got := rel(top, p.Found); strings.Join(got, ",") != "bin/y" {
t.Fatalf("found %v", got)
}
}
func TestAMountPointIsReadAsTheKernelEscapesIt(t *testing.T) {
mp := MountPoints("/dev/a /mnt/a\\040b ext4 rw 0 0\n/dev/b /mnt/back\\134slash ext4 rw 0 0\n")
if !mp["/mnt/a b"] || !mp[`/mnt/back\slash`] {
t.Fatalf("read %v", mp)
}
}
+810
View File
@@ -0,0 +1,810 @@
package accounts
// The ways to root the judge looks for beyond uid, groups and sudo (novox/hq ADR 0266, the review of
// 2026-10-08), and the ones it does not. Judged and NotJudged are the one written list: the record quotes it,
// and a way added here is a line added there.
import (
"bytes"
"context"
"encoding/binary"
"errors"
"fmt"
"io/fs"
"os"
"os/exec"
"path/filepath"
"slices"
"sort"
"strings"
"time"
"golang.org/x/sys/unix"
)
// Judged is every way to root the judge looks for, in the words its reasons use.
var Judged = []string{
"its uid is 0",
"membership of a group that grants root by membership: " + strings.Join(RootGroups, ", "),
"any sudo rule for it or one of its groups, as `sudo -l -U` lists it (with or without a password)",
"any doas rule permitting it or one of its groups (/etc/doas.conf, /etc/opendoas.conf)",
"any polkit rule naming it or one of its groups (/etc/polkit-1/rules.d, /usr/share/polkit-1/rules.d, " +
"/etc/polkit-1/localauthority), which is how pkexec and systemd's own actions are granted",
"a polkit rule that grants every account: a rule of a .rules file that answers polkit.Result.YES and names " +
"no user and no group, or a .pkla section whose Identity is unix-user:* or unix-group:* with a Result of " +
"yes — each rule and section judged on its own, comments taken out",
"write access to a container runtime's socket (docker, podman, containerd), by owner, group, other or " +
"POSIX ACL",
"a container runtime's API listening on TCP port 2375 or 2376 (/proc/net/tcp, /proc/net/tcp6), which any " +
"local account reaches",
"read access to a secret the mesh placed for another account, by owner, group, other or POSIX ACL",
"a program that no installed package owns and is setuid with owner root, or setgid with group root, " +
"whoever owns it, on every local filesystem mounted without nosuid (container and image layers, " +
"network and pseudo filesystems excepted) — searched in the background at most hourly, its last result " +
"served with when it was found; until a search has finished, the verdict says not judged",
"hard links or symbolic links to other accounts' files left unprotected by the kernel " +
"(fs.protected_hardlinks or fs.protected_symlinks is 0)",
}
// NotJudged is what the judge does not look for: each is a way to root it would miss.
var NotJudged = []string{
"a root-run unit, timer, cron entry or script the account can write",
"a directory on root's PATH, or in /etc/profile.d, the account can write",
"root's or the operator's ssh keys or authorized_keys readable or writable by it",
"a file root writes or reads, in a directory the account owns, other than below its home (the node-engine " +
"refuses links there)",
"file capabilities (setcap) on a program",
"a setuid program a package installed that has a flaw of its own",
"a terminal the account shares with a root process (TIOCSTI, sudo without use_pty)",
"a polkit rule that grants every account by logic the text does not show (a JavaScript condition " +
"other than a named user or group), or every account with an active local session",
"a container runtime's API on a TCP port other than 2375 and 2376, or on a unix socket not named here",
"a setuid program on a filesystem not judged: a container or image layer (overlay, squashfs), a network " +
"or FUSE filesystem",
}
// RuntimeSockets are the container runtimes' sockets: write access to one runs a container as root.
var RuntimeSockets = []string{"/run/docker.sock", "/var/run/docker.sock", "/run/podman/podman.sock",
"/run/containerd/containerd.sock"}
// DoasConfigs and PolkitDirs are where those grants live.
var (
DoasConfigs = []string{"/etc/doas.conf", "/etc/opendoas.conf"}
PolkitDirs = []string{"/etc/polkit-1/rules.d", "/usr/share/polkit-1/rules.d", "/etc/polkit-1/localauthority"}
)
// ACLEntry is one named entry of a POSIX ACL: a user's or a group's, and whether it grants read and write
// once the ACL's mask is applied.
type ACLEntry struct {
User bool
ID int
Read, Write bool
}
// The tags and bits of the kernel's ACL xattr.
const (
aclUser = 0x02
aclGroup = 0x08
aclMask = 0x10
)
// ParseACL reads a system.posix_acl_access value: a version, then entries of tag, permission and id; the
// named users' and groups' entries are answered with the mask applied.
func ParseACL(raw []byte) ([]ACLEntry, error) {
if len(raw) < 4 || (len(raw)-4)%8 != 0 {
return nil, fmt.Errorf("an ACL of %d bytes is not one", len(raw))
}
type entry struct {
tag, perm uint16
id uint32
}
var es []entry
mask := uint16(7)
for at := 4; at < len(raw); at += 8 {
e := entry{binary.LittleEndian.Uint16(raw[at:]), binary.LittleEndian.Uint16(raw[at+2:]),
binary.LittleEndian.Uint32(raw[at+4:])}
if e.tag == aclMask {
mask = e.perm
}
es = append(es, e)
}
var out []ACLEntry
for _, e := range es {
if e.tag != aclUser && e.tag != aclGroup {
continue
}
p := e.perm & mask
out = append(out, ACLEntry{User: e.tag == aclUser, ID: int(e.id), Read: p&4 != 0, Write: p&2 != 0})
}
return out, nil
}
// aclOf is a file's ACL from the machine: none when it has none.
func aclOf(path string) ([]ACLEntry, error) {
buf := make([]byte, 1024)
n, err := unix.Getxattr(path, "system.posix_acl_access", buf)
if errors.Is(err, unix.ENODATA) || errors.Is(err, unix.EOPNOTSUPP) {
return nil, nil
}
if err != nil {
return nil, &os.PathError{Op: "getxattr", Path: path, Err: err}
}
return ParseACL(buf[:n])
}
// grantsByACL says whether an ACL entry gives the account read (or write).
func grantsByACL(acl []ACLEntry, uid int, gids map[int]bool, write bool) bool {
for _, e := range acl {
if (e.User && e.ID == uid) || (!e.User && gids[e.ID]) {
if (write && e.Write) || (!write && e.Read) {
return true
}
}
}
return false
}
// Writable is Readable's twin for the write bits.
func Writable(m FileMode, uid int, gids map[int]bool) bool {
switch {
case uid == 0:
return true
case m.UID == uid:
return m.Perm&0o200 != 0
case gids[m.GID]:
return m.Perm&0o020 != 0
default:
return m.Perm&0o002 != 0
}
}
// DoasRules is every line of a doas configuration that permits the account or one of its groups.
func DoasRules(conf string, account string, groups []string) []string {
var out []string
for _, line := range strings.Split(conf, "\n") {
if i := strings.IndexByte(line, '#'); i >= 0 {
line = line[:i]
}
f := strings.Fields(line)
if len(f) < 2 || f[0] != "permit" {
continue
}
i := 1
for i < len(f) && (f[i] == "nopass" || f[i] == "persist" || f[i] == "keepenv" || f[i] == "nolog" ||
f[i] == "setenv" || strings.HasPrefix(f[i], "{")) {
if strings.HasPrefix(f[i], "{") {
for i < len(f) && !strings.HasSuffix(f[i], "}") {
i++
}
}
i++
}
if i >= len(f) {
continue
}
who := f[i]
if who == account || (strings.HasPrefix(who, ":") && contains(groups, who[1:])) {
out = append(out, strings.TrimSpace(line))
}
}
return out
}
// PolkitNames says whether a polkit rule or authority file names the account or one of its groups.
func PolkitNames(text, account string, groups []string) bool {
needles := []string{"unix-user:" + account, `"` + account + `"`, `'` + account + `'`}
for _, g := range groups {
needles = append(needles, "unix-group:"+g, `isInGroup("`+g+`")`, `isInGroup('`+g+`')`)
}
for _, n := range needles {
if strings.Contains(text, n) {
return true
}
}
return false
}
// PolkitGrantsEveryone says whether a polkit rule or authority file grants every account, naming none: a .rules
// file one of whose rules answers polkit.Result.YES with no condition on a user or a group, or a .pkla one of
// whose sections has an Identity of every user or every group with a Result of yes.
//
// **Each rule is judged on its own, comments taken out first** (the third review of 2026-10-08): a rule naming a
// user elsewhere in the file, or a user named only in a comment, must not hide another rule's unconditional
// yes. A rule's text is read, not run: a condition the text does not show is listed under NotJudged.
func PolkitGrantsEveryone(path, text string) bool {
if strings.HasSuffix(path, ".pkla") {
var kept []string
for _, line := range strings.Split(text, "\n") {
if t := strings.TrimSpace(line); strings.HasPrefix(t, "#") || strings.HasPrefix(t, ";") {
continue
}
kept = append(kept, line)
}
for _, section := range strings.Split(strings.Join(kept, "\n"), "[") {
var every, yes bool
for _, line := range strings.Split(section, "\n") {
k, v, ok := strings.Cut(strings.TrimSpace(line), "=")
if !ok {
continue
}
v = strings.TrimSpace(v)
switch strings.TrimSpace(k) {
case "Identity":
for _, id := range strings.Split(v, ";") {
if id = strings.TrimSpace(id); id == "unix-user:*" || id == "unix-group:*" {
every = true
}
}
case "ResultAny", "ResultActive", "ResultInactive":
if v == "yes" {
yes = true
}
}
}
if every && yes {
return true
}
}
return false
}
for _, rule := range polkitRules(stripJSComments(text)) {
if !strings.Contains(rule, "polkit.Result.YES") {
continue
}
// A condition on a user or a group names whom it grants; one on an active local session grants only an
// account with a seat, which an agent started through sudo has not (listed under NotJudged).
conditioned := false
for _, condition := range []string{".user", "isInGroup", "unix-user:", "unix-group:", ".active", ".local"} {
if strings.Contains(rule, condition) {
conditioned = true
break
}
}
if !conditioned {
return true
}
}
return false
}
// polkitRules is each rule of a .rules file: the text from one polkit.addRule to the next. Text before the first
// is no rule.
func polkitRules(text string) []string {
parts := strings.Split(text, "addRule(")
if len(parts) < 2 {
return nil
}
return parts[1:]
}
// stripJSComments takes /* … */ and // … comments out of a rule file, leaving what is in strings alone.
func stripJSComments(text string) string {
var b strings.Builder
var quote byte
for i := 0; i < len(text); i++ {
c := text[i]
switch {
case quote != 0:
b.WriteByte(c)
if c == '\\' && i+1 < len(text) {
i++
b.WriteByte(text[i])
} else if c == quote {
quote = 0
}
case c == '"' || c == '\'' || c == '`':
quote = c
b.WriteByte(c)
case c == '/' && i+1 < len(text) && text[i+1] == '*':
end := strings.Index(text[i+2:], "*/")
if end < 0 {
return b.String()
}
i += end + 3
case c == '/' && i+1 < len(text) && text[i+1] == '/':
end := strings.IndexByte(text[i:], '\n')
if end < 0 {
return b.String()
}
i += end - 1
default:
b.WriteByte(c)
}
}
return b.String()
}
// RuntimeAPIPorts are the ports a container runtime's API listens on by convention: 2375 plain, 2376 TLS.
var RuntimeAPIPorts = []int{2375, 2376}
// ListeningPorts is every local TCP port in the listening state, from /proc/net/tcp's or tcp6's text.
func ListeningPorts(text string) []int {
var out []int
for i, line := range strings.Split(text, "\n") {
f := strings.Fields(line)
if i == 0 || len(f) < 4 || f[3] != "0A" {
continue
}
_, port, ok := strings.Cut(f[1], ":")
if !ok {
continue
}
var n int
if _, err := fmt.Sscanf(port, "%X", &n); err == nil {
out = append(out, n)
}
}
return out
}
// pseudoFS are filesystem types no setuid program is installed on, or that are a container's or an image's
// own layers: not searched.
var pseudoFS = map[string]bool{"proc": true, "sysfs": true, "devtmpfs": true, "devpts": true, "tmpfs": true,
"cgroup": true, "cgroup2": true, "securityfs": true, "pstore": true, "bpf": true, "debugfs": true,
"tracefs": true, "mqueue": true, "hugetlbfs": true, "configfs": true, "fusectl": true, "autofs": true,
"binfmt_misc": true, "efivarfs": true, "overlay": true, "squashfs": true, "nsfs": true, "ramfs": true,
"nfs": true, "nfs4": true, "cifs": true, "smb3": true, "9p": true, "virtiofs": true}
// SuidMounts is every local filesystem a setuid program could run from: the mount points of /proc/mounts whose
// type is not pseudo, network or a container's layer, and that are not mounted nosuid. Root first.
func SuidMounts(text string) []string {
seen := map[string]bool{}
var out []string
for _, line := range strings.Split(text, "\n") {
f := strings.Fields(line)
if len(f) < 4 || pseudoFS[f[2]] || strings.HasPrefix(f[2], "fuse") {
continue
}
at := unescapeMount(f[1])
if strings.HasPrefix(at, "/var/lib/docker") || strings.HasPrefix(at, "/var/lib/containers") ||
strings.HasPrefix(at, "/proc") || strings.HasPrefix(at, "/sys") {
continue
}
if contains(strings.Split(f[3], ","), "nosuid") || seen[at] {
continue
}
seen[at] = true
out = append(out, at)
}
sort.Slice(out, func(i, j int) bool { return out[i] == "/" || (out[j] != "/" && out[i] < out[j]) })
return out
}
// MountPoints is every mount point /proc/mounts lists, of any type: the walk leaves each it meets inside
// another filesystem without reading it.
func MountPoints(text string) map[string]bool {
out := map[string]bool{}
for _, line := range strings.Split(text, "\n") {
if f := strings.Fields(line); len(f) >= 2 {
out[unescapeMount(f[1])] = true
}
}
return out
}
// unescapeMount is a mount point as /proc/mounts escapes it — a space, a tab, a newline and a backslash as
// three octal digits after a backslash — made a path again.
func unescapeMount(s string) string {
var b strings.Builder
for i := 0; i < len(s); i++ {
if s[i] == '\\' && i+3 < len(s) && isOctal(s[i+1:i+4]) {
b.WriteByte((s[i+1]-'0')<<6 | (s[i+2]-'0')<<3 | (s[i+3] - '0'))
i += 3
continue
}
b.WriteByte(s[i])
}
return b.String()
}
func isOctal(s string) bool {
if len(s) != 3 {
return false
}
for i := 0; i < 3; i++ {
if s[i] < '0' || s[i] > '7' {
return false
}
}
return true
}
func contains(xs []string, s string) bool {
for _, x := range xs {
if x == s {
return true
}
}
return false
}
// setuidSearch is every setuid- or setgid-root regular file on the filesystems of p that no installed package
// owns: walked by the engine (setuid_walk.go, resumable from p), then asked of the package manager through the
// search's runner (Exec.SearchRun). **No bound of its own** (novox/hq issue 361): it runs until it has walked
// everything. A walk that could not complete — a directory it could not read — is an unanswered question, never
// "none"; a file that vanished while it walked is not.
func (e Exec) setuidSearch(ctx context.Context, p *SetuidProgress, checkpoint func(), mountPoints map[string]bool) ([]string, error) {
walk := e.Walk
if walk == nil {
var say func(string)
if e.Cache != nil {
say = e.Cache.Say
}
walk = engineWalker(mountPoints, say).walk
}
if err := walk(ctx, p, checkpoint); err != nil {
return nil, fmt.Errorf("the walk for setuid programs did not complete: %w", err)
}
if e.SearchRun != nil {
e.Run = e.SearchRun
}
found := slices.Clone(p.Found)
slices.Sort(found)
return e.unpackaged(ctx, slices.Compact(found))
}
// engineWalker is the walk the engine runs: checkpointed every CheckpointEvery, leaving other filesystems unread,
// and paced by the machine's I/O pressure (setuid_pace.go, novox/hq issue 433), saying through say what it cannot do.
func engineWalker(mountPoints map[string]bool, say func(string)) walker {
return walker{every: CheckpointEvery, mountPoints: mountPoints, pace: newPacer(say)}
}
// errorLines are the lines a command said on its error stream, as the runner words a failure: "<name> exited N:
// <its words>", the prefix taken off the first line. Empty lines are dropped.
func errorLines(err error, name string) []string {
words := err.Error()
if strings.HasPrefix(words, name+" exited") {
if i := strings.Index(words, ": "); i >= 0 {
words = words[i+2:]
}
}
var out []string
for _, l := range strings.Split(words, "\n") {
if l = strings.TrimSpace(l); l != "" {
out = append(out, l)
}
}
return out
}
// unpackaged is every path no installed package owns: pacman asked about all of them in one call, else apk
// about each. Never a guess: a path neither said owned nor said unowned is an unanswered question.
func (e Exec) unpackaged(ctx context.Context, paths []string) ([]string, error) {
if len(paths) == 0 {
return nil, nil
}
out, err := e.Run(ctx, "pacman", append([]string{"-Qo"}, paths...)...)
if err == nil || !errors.Is(err, exec.ErrNotFound) && !errors.Is(err, fs.ErrNotExist) {
owned := map[string]bool{}
for _, l := range strings.Split(out, "\n") {
if p, _, ok := strings.Cut(strings.TrimSpace(l), " is owned by "); ok {
owned[p] = true
}
}
unowned := map[string]bool{}
if err != nil {
// pacman exits 1 when a path is owned by nothing, and says which on its error stream.
for _, l := range errorLines(err, "pacman") {
if _, p, ok := strings.Cut(l, "No package owns "); ok {
unowned[strings.TrimSpace(p)] = true
} else {
return nil, fmt.Errorf("pacman could not say who owns the setuid programs found: %s", l)
}
}
}
var out []string
for _, p := range paths {
switch {
case owned[p]:
case unowned[p]:
out = append(out, p)
default:
return nil, fmt.Errorf("pacman said nothing of who owns %s", p)
}
}
return out, nil
}
var none []string
for _, p := range paths {
said, err := e.Run(ctx, "apk", "info", "-W", p)
if err != nil {
return nil, fmt.Errorf("no package manager could say who owns %s: %w", p, err)
}
if !strings.Contains(said, " is owned by ") {
none = append(none, p)
}
}
return none, nil
}
// moreWays is every way beyond uid, groups and sudo; an unanswered question is an error, never none.
func (e Exec) moreWays(ctx context.Context, account string, uid int, groups []string, gids map[int]bool,
secrets []string) ([]string, error) {
read := e.ReadFile
if read == nil {
read = os.ReadFile
}
readDir := e.ReadDir
if readDir == nil {
readDir = os.ReadDir
}
acl := e.ACL
if acl == nil {
acl = aclOf
}
stat := e.Stat
if stat == nil {
stat = statOf
}
var ways []string
// doas.
for _, conf := range DoasConfigs {
raw, err := read(conf)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("doas's rules in %s could not be read: %w", conf, err)
}
for _, r := range DoasRules(string(raw), account, groups) {
ways = append(ways, "doas permits it: "+r)
}
}
// polkit.
for _, dir := range PolkitDirs {
var named, everyone []string
err := walkFiles(readDir, dir, func(path string) error {
raw, err := read(path)
if err != nil {
return err
}
if PolkitNames(string(raw), account, groups) {
named = append(named, path)
} else if PolkitGrantsEveryone(path, string(raw)) {
everyone = append(everyone, path)
}
return nil
})
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return nil, fmt.Errorf("polkit's rules in %s could not be read: %w", dir, err)
}
for _, p := range named {
ways = append(ways, "a polkit rule names it or a group of it: "+p)
}
for _, p := range everyone {
ways = append(ways, "a polkit rule grants every account: "+p)
}
}
// The container runtimes' sockets.
seen := map[string]bool{}
for _, s := range RuntimeSockets {
// Two names of one socket are one socket. A test that gives its own files names them as they are.
real, err := filepath.EvalSymlinks(s)
if e.Stat != nil {
real, err = s, nil
}
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("the socket %s could not be read: %w", s, err)
}
if seen[real] {
continue
}
seen[real] = true
m, err := stat(real)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("the socket %s could not be read for its owner and mode: %w", s, err)
}
a, err := acl(real)
if err != nil {
return nil, err
}
if Writable(m, uid, gids) || grantsByACL(a, uid, gids, true) {
ways = append(ways, "it can write the container runtime's socket "+s+", which runs a container as root")
}
}
// The secrets' ACLs: the bits were judged already.
for _, path := range secrets {
a, err := acl(path)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("the secret %s's ACL could not be read: %w", path, err)
}
if grantsByACL(a, uid, gids, false) {
ways = append(ways, "an ACL lets it read the secret "+path)
}
}
// setuid programs no package owns.
now := time.Now
if e.Now != nil {
now = e.Now
}
mountsText, err := read("/proc/mounts")
if err != nil {
return nil, fmt.Errorf("the mounted filesystems could not be read: %w", err)
}
mounts := SuidMounts(string(mountsText))
if len(mounts) == 0 {
return nil, errors.New("no filesystem a setuid program could run from was found in /proc/mounts")
}
// In the background, the last complete result served while it is fresh (SetuidCache): never a stall. No
// fresh complete result is said beside the other ways, which are judged all the same.
mountPoints := MountPoints(string(mountsText))
unowned, searchedAt, searchErr := e.Cache.Look(now(), mounts,
func(ctx context.Context, p *SetuidProgress, checkpoint func()) ([]string, error) {
return e.setuidSearch(ctx, p, checkpoint, mountPoints)
})
var notJudged *SetuidNotJudged
if searchErr != nil && !errors.As(searchErr, &notJudged) {
return nil, searchErr
}
// Counted only where the account can execute it, read now from the paths' modes, never kept with the
// search: an account may open its own home to others later without root, and no search would see it.
lstat := e.Lstat
if lstat == nil {
lstat = lstatOf
}
started := " (search started " + searchedAt.Local().Format("2006-01-02 15:04") + ")"
inHomes := 0
for _, p := range unowned {
if !Executable(p, uid, gids, lstat, acl) {
continue
}
if inAHome(p) {
inHomes++ // its path stays in the engine's journal: a home's names are its owner's
continue
}
ways = append(ways, "a setuid-root program no package owns, which "+account+" can execute: "+p+started)
}
switch {
case inHomes == 1:
ways = append(ways, "a setuid-root program no package owns in an account's home, which "+account+
" can execute"+started)
case inHomes > 1:
ways = append(ways, fmt.Sprintf("%d setuid-root programs no package owns in accounts' homes, which %s can "+
"execute%s", inHomes, account, started))
}
// The kernel's protection of links: off, any account may link another's file where root then acts on it.
for _, sysctl := range []string{"protected_hardlinks", "protected_symlinks"} {
raw, err := read("/proc/sys/fs/" + sysctl)
if err != nil {
return nil, fmt.Errorf("fs.%s could not be read: %w", sysctl, err)
}
if strings.TrimSpace(string(raw)) == "0" {
what := map[string]string{"protected_hardlinks": "hard links", "protected_symlinks": "symbolic links"}[sysctl]
ways = append(ways, fmt.Sprintf("%s to other accounts' files are not protected (fs.%s=0)", what, sysctl))
}
}
// A container runtime's API on TCP, which any account on the machine reaches.
for _, table := range []string{"/proc/net/tcp", "/proc/net/tcp6"} {
raw, err := read(table)
if errors.Is(err, fs.ErrNotExist) && table == "/proc/net/tcp6" {
continue // no IPv6 on this machine
}
if err != nil {
return nil, fmt.Errorf("the listening ports in %s could not be read: %w", table, err)
}
for _, port := range ListeningPorts(string(raw)) {
for _, api := range RuntimeAPIPorts {
if port == api {
ways = append(ways, fmt.Sprintf("a container runtime's API listens on TCP port %d, which any "+
"local account reaches", port))
}
}
}
}
if notJudged != nil {
// The other ways are judged; this one is said as not judged, never as none.
return ways, notJudged
}
return ways, nil
}
// Executable says whether an account of uid, in the groups gids, can execute path as the kernel decides it:
// every directory from the root down searchable for it, and the file executable for it, each by the owner's
// bits for its owner, the group's for a member, the others' for anybody else (novox/hq issue 361). It errs
// towards a way: a link on the way, an ACL on any of them, or a mode that could not be read is executable.
// A path gone is not: there is nothing to execute.
func Executable(path string, uid int, gids map[int]bool, lstat func(string) (FileMode, error),
acl func(string) ([]ACLEntry, error)) bool {
if uid == 0 {
return true
}
at := "/"
steps := []string{at}
for _, name := range components(path) {
at = filepath.Join(at, name)
steps = append(steps, at)
}
for i, p := range steps {
m, err := lstat(p)
if errors.Is(err, fs.ErrNotExist) {
return false
}
if err != nil || m.Perm&fs.ModeSymlink != 0 {
return true
}
if a, err := acl(p); (err != nil && !errors.Is(err, fs.ErrNotExist)) || len(a) > 0 {
return true
}
bit := fs.FileMode(0o001)
switch {
case m.UID == uid:
bit = 0o100
case gids[m.GID]:
bit = 0o010
}
if m.Perm&bit == 0 {
return false
}
if i < len(steps)-1 && !m.Perm.IsDir() {
return false // a file on the way: the path names nothing
}
}
return true
}
// homes are where accounts' homes are: a path under one is said by count alone, its name its owner's.
var homes = []string{"/home/", "/root/", "/var/home/"}
func inAHome(path string) bool {
for _, h := range homes {
if strings.HasPrefix(path, h) {
return true
}
}
return false
}
// walkFiles calls fn for every regular file below dir, through readDir.
func walkFiles(readDir func(string) ([]fs.DirEntry, error), dir string, fn func(string) error) error {
entries, err := readDir(dir)
if err != nil {
return err
}
for _, en := range entries {
p := filepath.Join(dir, en.Name())
if en.IsDir() {
if err := walkFiles(readDir, p, fn); err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
continue
}
if err := fn(p); err != nil {
return err
}
}
return nil
}
// CLocale runs a command in the C locale, so what the judge parses — sudo's listing above all — is one
// language whatever the machine's is; stdin closed, output captured.
func CLocale(ctx context.Context, name string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
cmd.Env = append(os.Environ(), "LC_ALL=C", "LANG=C", "LANGUAGE=C")
var stderr bytes.Buffer
cmd.Stderr = &stderr
out, err := cmd.Output()
if err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
return string(out), fmt.Errorf("%s exited %d: %s", name, exit.ExitCode(), strings.TrimSpace(stderr.String()))
}
return string(out), fmt.Errorf("%s: %w", name, err)
}
return string(out), nil
}
+305
View File
@@ -0,0 +1,305 @@
package accounts
import (
"context"
"encoding/binary"
"errors"
"fmt"
"os/exec"
"strings"
"testing"
"time"
)
// The ways beyond uid, groups and sudo (novox/hq ADR 0266, the review of 2026-10-08): doas, polkit, a
// runtime's socket, an ACL on a secret, a setuid program no package owns — each said; each unread question
// unknown; and the judge's commands in the C locale.
func TestEachFurtherWayToRootIsSaid(t *testing.T) {
const broker = "/var/lib/mesh/node-tools/broker"
for _, c := range []struct {
name string
set func(*agentMachine)
said string
}{
{"doas by name", func(m *agentMachine) {
m.texts = map[string]string{"/etc/doas.conf": "permit persist operator\npermit nopass agent as root\n"}
}, "doas permits it: permit nopass agent as root"},
{"doas by group", func(m *agentMachine) {
m.groups = "agent builders"
m.texts = map[string]string{"/etc/opendoas.conf": "permit :builders # the builders\n"}
}, "doas permits it: permit :builders"},
{"polkit by name", func(m *agentMachine) {
m.dirs = map[string][]string{"/etc/polkit-1/rules.d": {"10-agent.rules"}}
m.texts = map[string]string{"/etc/polkit-1/rules.d/10-agent.rules": `polkit.addRule(function(a, s) { if (s.user == "agent") return polkit.Result.YES; });`}
}, "a polkit rule names it or a group of it: /etc/polkit-1/rules.d/10-agent.rules"},
{"polkit by group", func(m *agentMachine) {
m.groups = "agent network"
m.dirs = map[string][]string{"/usr/share/polkit-1/rules.d": {"50-nm.rules"}}
m.texts = map[string]string{"/usr/share/polkit-1/rules.d/50-nm.rules": `if (subject.isInGroup("network")) return polkit.Result.YES;`}
}, "a polkit rule names it or a group of it: /usr/share/polkit-1/rules.d/50-nm.rules"},
{"docker socket by group bits", func(m *agentMachine) {
m.gids = "1600 970"
m.files["/run/docker.sock"] = FileMode{UID: 0, GID: 970, Perm: 0o660}
}, "it can write the container runtime's socket /run/docker.sock"},
{"docker socket by ACL", func(m *agentMachine) {
m.files["/run/docker.sock"] = FileMode{UID: 0, GID: 970, Perm: 0o660}
m.acls = map[string][]ACLEntry{"/run/docker.sock": {{User: true, ID: 1600, Read: true, Write: true}}}
}, "it can write the container runtime's socket /run/docker.sock"},
{"secret by ACL", func(m *agentMachine) {
m.acls = map[string][]ACLEntry{broker: {{User: false, ID: 1600, Read: true}}}
}, "an ACL lets it read the secret " + broker},
{"polkit for every account", func(m *agentMachine) {
m.dirs = map[string][]string{"/etc/polkit-1/rules.d": {"00-all.rules"}}
m.texts = map[string]string{"/etc/polkit-1/rules.d/00-all.rules": `polkit.addRule(function(action, subject) { return polkit.Result.YES; });`}
}, "a polkit rule grants every account: /etc/polkit-1/rules.d/00-all.rules"},
{"pkla for every account", func(m *agentMachine) {
m.dirs = map[string][]string{"/etc/polkit-1/localauthority": {"all.pkla"}}
m.texts = map[string]string{"/etc/polkit-1/localauthority/all.pkla": "[all]\nIdentity=unix-user:*\nAction=*\nResultAny=yes\n"}
}, "a polkit rule grants every account: /etc/polkit-1/localauthority/all.pkla"},
{"docker on TCP", func(m *agentMachine) {
m.proc["/proc/net/tcp6"] = " sl local_address rem_address st\n 0: 00000000000000000000000000000000:0947 00000000000000000000000000000000:0000 0A 0\n"
}, "a container runtime's API listens on TCP port 2375"},
{"hard links unprotected", func(m *agentMachine) {
m.proc["/proc/sys/fs/protected_hardlinks"] = "0\n"
}, "hard links to other accounts' files are not protected (fs.protected_hardlinks=0)"},
{"setuid on a second filesystem", func(m *agentMachine) {
m.proc["/proc/mounts"] += "/dev/sdb1 /srv xfs rw 0 0\n/dev/sdc1 /data ext4 rw,nosuid 0 0\n"
m.setuid = "/srv/bin/rootshell\n"
m.findArgs = func(args []string) {
if len(args) != 2 || args[0] != "/" || args[1] != "/srv" {
panic(fmt.Sprintf("searched %v", args))
}
}
}, "a setuid-root program no package owns, which agent can execute: /srv/bin/rootshell"},
{"setuid no package owns", func(m *agentMachine) {
m.setuid = "/usr/bin/sudo\n/usr/local/bin/rootshell\n"
m.packaged = map[string]bool{"/usr/bin/sudo": true}
}, "a setuid-root program no package owns, which agent can execute: /usr/local/bin/rootshell"},
} {
t.Run(c.name, func(t *testing.T) {
m := clean()
c.set(m)
v := lookAgent(t, m)
if v.State != Unhealthy || !strings.HasPrefix(v.Reason, ReasonRoot+": ") || !strings.Contains(v.Reason, c.said) {
t.Fatalf("want %q said: %+v", c.said, v)
}
})
}
}
func TestWhatGrantsSomebodyElseIsNoWay(t *testing.T) {
m := clean()
m.texts = map[string]string{"/etc/doas.conf": "permit operator\npermit :wheel\n# permit agent\n",
"/usr/share/polkit-1/rules.d/50-default.rules": `polkit.addAdminRule(function(a, s) { return ["unix-group:wheel"]; });`}
m.dirs = map[string][]string{"/usr/share/polkit-1/rules.d": {"50-default.rules"}}
m.files["/run/docker.sock"] = FileMode{UID: 0, GID: 970, Perm: 0o660}
m.setuid = "/usr/bin/sudo\n/usr/bin/passwd\n"
m.packaged = map[string]bool{"/usr/bin/sudo": true, "/usr/bin/passwd": true}
if v := lookAgent(t, m); v.State != Healthy {
t.Fatalf("the operator's and wheel's grants, a socket of another group, packaged setuid programs: %+v", v)
}
}
func TestDoasWithoutSudoIsJudgedFromItsRules(t *testing.T) {
m := clean()
m.sudo, m.sudoErr = "", fmt.Errorf("sudo: %w", exec.ErrNotFound)
m.texts = map[string]string{"/etc/doas.conf": "permit nopass agent\n"}
v := lookAgent(t, m)
if v.State != Unhealthy || !strings.Contains(v.Reason, "doas permits it") {
t.Fatalf("no sudo is no sudo rule, and doas is read for itself: %+v", v)
}
}
func TestAFurtherQuestionUnansweredIsUnknown(t *testing.T) {
m := clean()
m.findErr = errors.New("find: interrupted")
if v := lookAgent(t, m); v.State != Unknown {
t.Fatalf("a search that did not finish: %+v", v)
}
m = clean()
m.setuid = "/usr/local/bin/x\n"
j := New(Exec{Run: func(ctx context.Context, name string, args ...string) (string, error) {
if name == "pacman" || name == "apk" {
return "", fmt.Errorf("%s: %w", name, exec.ErrNotFound)
}
return m.run(ctx, name, args...)
}, Walk: m.walk, Stat: m.stat, ReadFile: m.readFile, ReadDir: m.readDir, ACL: m.acl, Lstat: m.lstat})
j.Set([]Account{agent})
if st, _ := j.Look(t.Context()); st.Accounts[0].State != Unknown {
t.Fatalf("no package manager to ask: %+v", st.Accounts[0])
}
}
func TestANotJudgedSearchStillSaysAWayFound(t *testing.T) {
j := New(fakeReader{ways: []string{"in the group docker, which grants root"}, err: &SetuidNotJudged{Pending: true, Since: time.Now()}})
j.Set([]Account{{Module: "m", ID: "m.a", Name: "agent", Root: true}})
st, _ := j.Look(context.Background())
if v := st.Accounts[0]; v.State != Unhealthy || !strings.Contains(v.Reason, "docker") || !strings.Contains(v.Reason, ReasonPending) {
t.Fatalf("a way found is unhealthy, with the search's state beside it: %+v", v)
}
j = New(fakeReader{err: &SetuidNotJudged{Since: time.Now(), Err: errors.New("timeout")}})
j.Set([]Account{{Module: "m", ID: "m.a", Name: "agent", Root: true}})
st, _ = j.Look(context.Background())
if v := st.Accounts[0]; v.State != Unknown || !strings.HasPrefix(v.Reason, ReasonIncomplete) {
t.Fatalf("nothing found and the search incomplete: unknown, said so: %+v", v)
}
}
type fakeReader struct {
ways []string
err error
}
func (f fakeReader) InDatabase(context.Context, string) ([]string, error) { return nil, nil }
func (f fakeReader) Session(context.Context, string, []string) (Session, error) {
return Session{}, nil
}
func (f fakeReader) Escalation(context.Context, string, []string) ([]string, error) {
return f.ways, f.err
}
func TestParseACL(t *testing.T) {
entry := func(tag, perm uint16, id uint32) []byte {
b := make([]byte, 8)
binary.LittleEndian.PutUint16(b, tag)
binary.LittleEndian.PutUint16(b[2:], perm)
binary.LittleEndian.PutUint32(b[4:], id)
return b
}
raw := []byte{2, 0, 0, 0}
raw = append(raw, entry(0x01, 6, 0xffffffff)...)
raw = append(raw, entry(0x02, 6, 1600)...) // user agent rw
raw = append(raw, entry(0x04, 4, 0xffffffff)...)
raw = append(raw, entry(0x08, 6, 970)...) // group 970 rw
raw = append(raw, entry(0x10, 4, 0xffffffff)...) // mask r--
raw = append(raw, entry(0x20, 0, 0xffffffff)...)
acl, err := ParseACL(raw)
if err != nil || len(acl) != 2 {
t.Fatalf("%v %v", acl, err)
}
if !acl[0].User || acl[0].ID != 1600 || !acl[0].Read || acl[0].Write {
t.Fatalf("the mask takes write away: %+v", acl[0])
}
if grantsByACL(acl, 1600, nil, true) || !grantsByACL(acl, 1601, map[int]bool{970: true}, false) {
t.Fatal("grants")
}
if _, err := ParseACL([]byte{2, 0, 0}); err == nil {
t.Fatal("a short ACL")
}
}
func TestTheJudgesCommandsRunInTheCLocale(t *testing.T) {
t.Setenv("LC_ALL", "de_DE.UTF-8")
t.Setenv("LANG", "de_DE.UTF-8")
out, err := CLocale(t.Context(), "sh", "-c", `echo "$LC_ALL $LANG"`)
if err != nil || strings.TrimSpace(out) != "C C" {
t.Fatalf("%q %v", out, err)
}
if _, err := CLocale(t.Context(), "sh", "-c", "echo nope >&2; exit 3"); err == nil ||
!strings.Contains(err.Error(), "exited 3: nope") {
t.Fatalf("an exit is said with its words: %v", err)
}
}
func TestTheJudgedListIsWrittenDown(t *testing.T) {
if len(Judged) < 8 || len(NotJudged) == 0 {
t.Fatal("what is judged and what is not are both written down")
}
}
func TestDistributionRulesForASeatOrForSomebodyAreNotEveryone(t *testing.T) {
for _, rule := range []string{
`polkit.addRule(function(a, s) { if (s.local && s.active) return polkit.Result.YES; });`,
`polkit.addRule(function(a, s) { if (s.isInGroup("wheel")) return polkit.Result.YES; });`,
`polkit.addAdminRule(function(a, s) { return ["unix-group:wheel"]; });`,
} {
if PolkitGrantsEveryone("/x.rules", rule) {
t.Errorf("not every account: %s", rule)
}
}
if PolkitGrantsEveryone("/x.pkla", "[a]\nIdentity=unix-group:wheel\nResultAny=yes\n") {
t.Error("a pkla for wheel is not every account")
}
}
func TestAMachineMissingWhatTheJudgeReadsIsUnknown(t *testing.T) {
for _, file := range []string{"/proc/sys/fs/protected_hardlinks", "/proc/net/tcp", "/proc/mounts"} {
m := clean()
delete(m.proc, file)
if v := lookAgent(t, m); v.State != Unknown {
t.Errorf("%s unread: %+v", file, v)
}
}
}
func TestSuidMountsLeaveOutNosuidPseudoAndContainerLayers(t *testing.T) {
got := SuidMounts("/dev/sdb1 /srv xfs rw 0 0\n/dev/sda2 / ext4 rw 0 0\noverlay /var/lib/docker/overlay2/x/merged overlay rw 0 0\n" +
"/dev/sdc1 /data ext4 rw,nosuid 0 0\nproc /proc proc rw 0 0\nhost:/x /mnt/nfs nfs4 rw 0 0\n/dev/sdd1 /my\\040disk ext4 rw 0 0\n")
if strings.Join(got, "|") != "/|/my disk|/srv" {
t.Fatalf("got %q", got)
}
}
// Each polkit rule is judged on its own, comments taken out (the third review of 2026-10-08).
func TestAnUnconditionalPolkitYesIsNotMaskedByAnotherRuleOrAComment(t *testing.T) {
cases := []struct {
name, text string
every bool
}{
{"another rule names a user", `polkit.addRule(function(a, s) { if (s.user == "operator") return polkit.Result.YES; });
polkit.addRule(function(a, s) { return polkit.Result.YES; });`, true},
{"a user named only in a comment", `// subject.user == "x"
polkit.addRule(function(a, s) { /* isInGroup("wheel") */ return polkit.Result.YES; });`, true},
{"the yes only in a comment", `polkit.addRule(function(a, s) { // return polkit.Result.YES;
return polkit.Result.NO; });`, false},
{"a condition in the same rule", `polkit.addRule(function(a, s) { if (s.isInGroup("wheel")) { return polkit.Result.YES; } });`, false},
{"a comment marker inside a string", `polkit.addRule(function(a, s) { var u = "http://x"; return polkit.Result.YES; });`, true},
}
for _, c := range cases {
if got := PolkitGrantsEveryone("/etc/polkit-1/rules.d/x.rules", c.text); got != c.every {
t.Errorf("%s: grants every account %v, want %v", c.name, got, c.every)
}
}
if PolkitGrantsEveryone("/x.pkla", "# [a]\n# Identity=unix-user:*\n# ResultAny=yes\n") {
t.Error("a .pkla section only in comments grants nobody")
}
if !PolkitGrantsEveryone("/x.pkla", "[a]\nIdentity=unix-group:wheel\nResultAny=yes\n[b]\nIdentity=unix-user:*\nResultActive=yes\n") {
t.Error("each .pkla section on its own")
}
}
// The package manager is asked about every path found in one call, and answers each: owned, or owned by nothing.
func TestThePackageManagerIsAskedOnceForEveryPathFound(t *testing.T) {
m := clean()
m.setuid = "/usr/bin/sudo\n/usr/bin/passwd\n/usr/local/bin/x\n/opt/y\n"
m.packaged = map[string]bool{"/usr/bin/sudo": true, "/usr/bin/passwd": true}
search := func(e Exec) ([]string, error) {
return e.setuidSearch(t.Context(), &SetuidProgress{Mounts: []string{"/"}}, func() {}, nil)
}
found, err := search(Exec{Run: m.run, Walk: m.walk})
if err != nil {
t.Fatal(err)
}
if m.pacmanCalls != 1 {
t.Errorf("pacman was asked %d times for four paths", m.pacmanCalls)
}
if strings.Join(found, ",") != "/opt/y,/usr/local/bin/x" {
t.Errorf("unowned: %v", found)
}
// An error pacman says that is not "owned by nothing" is no answer.
odd := func(ctx context.Context, name string, args ...string) (string, error) {
if name == "pacman" {
return "", errors.New("pacman exited 1: error: could not open database")
}
return m.run(ctx, name, args...)
}
if _, err := search(Exec{Run: odd, Walk: m.walk}); err == nil {
t.Error("a package manager that could not answer was read as an answer")
}
// A walk that did not complete is no answer either.
m.findErr = errors.New("could not read /mnt/data/private: permission denied")
if _, err := search(Exec{Run: m.run, Walk: m.walk}); err == nil {
t.Error("a walk that did not complete was read as an answer")
}
}
+927
View File
@@ -0,0 +1,927 @@
// Package agent is the node agent (novox/hq ADR 0429 §4, ADR 0539): the node-engine's own process that follows the
// node's wants and applies one module at a time.
//
// **It watches, it never polls.** A want `want.<node>.<module>` names a fingerprint and the digest of the one composition
// the assigner allows the node to apply for it; the composition itself waits under `<node>.<module>.<fingerprint>` in
// the node's bucket `composed-<node>`. When both are there and the composition's digest is the want's, and what runs
// differs, the agent applies that module's composition and writes its have with the fingerprint and the digest. A want
// `removed` removes that module's own resources. Nothing is applied for a composition no want names, and a failed apply
// is retried by the next change of either key, never by a clock (research 063 §9.1) — except an apply that failed only
// because the bus did not answer for a moment (Result.Transient: no responders while the bus reloads its accounts, a
// reconnect): that one is retried with a bounded backoff (Env.Backoff), and at once when the engine is told the bus is
// back (RetryTransient), since no key will change for it.
package agent
import (
"bytes"
"context"
"crypto/ed25519"
"encoding/json"
"fmt"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Update is one change of a key in a watched bucket. Caught marks the end of the keys that were there when the watch
// began; every Update after it is a change.
type Update struct {
Key string
Value []byte
Revision uint64
Deleted bool
Caught bool
}
// Watcher follows one bucket: the keys matching the pattern now, a Caught update, then every change, until ctx ends.
type Watcher interface {
Watch(ctx context.Context, pattern string) (<-chan Update, error)
}
// NodeModule is the scope of the node's own composition, `<node>.node`.
const NodeModule = "node"
// Have is what a module's have key says, as the agent reads and writes it.
type Have struct {
Module string
FP string
Composed string
// Phase is "running", "failed", "held" or "removed"; empty reads as running.
Phase string
Detail string
}
// Haves is the node's have bucket as the agent uses it.
type Haves interface {
// Read is what the node-engine last wrote for a module, and whether it wrote anything.
Read(ctx context.Context, module string) (Have, bool)
// Write writes a module's have where it differs from the bucket's.
Write(ctx context.Context, h Have)
// Removed writes the module's have with the phase removed, once, if a key of it exists.
Removed(ctx context.Context, module string)
}
// Result is how one apply or removal came out.
type Result struct {
Failed bool
// Transient says the failure was the bus not answering for a moment (no responders, a timeout, a reconnect, a
// JetStream API error while the bus reloads), not the module's: the apply is tried again without a new revision.
Transient bool
Detail string
}
// Backup is the answer of the backup asked before a module moves.
type Backup struct {
Done bool
Words string
}
// Env is everything the agent needs of the node.
type Env struct {
Node string
Wants Watcher
Composed Watcher
Haves Haves
// Apply applies one module's composition (scope NodeModule for the node's own), given the compositions the node's
// other modules run so a path, unit or group they hold is not removed or taken. It is called with one module at a
// time.
Apply func(ctx context.Context, c delivery.Composed, others []delivery.Composed) Result
// Remove removes the module's own resources and nothing of another's.
Remove func(ctx context.Context, module string, others []delivery.Composed) Result
// Backup backs a module up before a new composition of it is applied; nil backs up nothing.
Backup func(ctx context.Context, module string) Backup
// Kept is the resources the node last applied, by module, from the declaration it kept before the switch (ADR 0539
// §9), as JSON; the node's own are under NodeModule. Nil when the node kept none. A module the agent does not hold yet
// still runs from it.
Kept func() map[string][]json.RawMessage
// KeptFilled is the paths the kept declaration names as filled by the provisioner, by the module whose file is at
// each (the node's own under NodeModule). A module that still runs from the kept declaration keeps its filled files
// filled: without them the filling refused each one as "not named as filled" (issue 1945). Nil names none.
KeptFilled func() map[string][]string
// Trusted is the public keys a composition may be signed by: the controller's and the operator's (ADR 0539 §3). A
// composition unsigned, signed by another key, or changed after it was signed is refused and nothing of it is
// applied. Nil trusts no key, so nothing is applied.
Trusted func() []ed25519.PublicKey
// Recorded says the engine holds a record of every resource of the composition, so it was applied and not removed
// since; nil says yes. A composition the kept declaration or a have says runs is applied again when it says no.
Recorded func(ctx context.Context, c delivery.Composed) bool
// Revisions are a composition key's revisions the bucket keeps, oldest first; nil reads none. The composition a want
// names is taken from them when a newer one replaced it under the same key (ADR 0539 clause 1).
Revisions func(ctx context.Context, key string) ([]Update, error)
// Settled is told a module's apply or removal that succeeded, once what runs holds it (Everything), outside the
// agent's lock: the declaration the node applies now names what that composition names, its filled files among
// them. The node-engine's filling tells the provisioner's client then to fill again what it refused (mesh-host
// internal/filling Knock; ace, 2026-10-25 21:17). Nil tells nothing.
Settled func(module string)
// Backoff is how long to wait before the try-th retry (1, 2, …) of an apply that failed transiently. Nil retries a
// transient failure only on RetryTransient or a change of either key.
Backoff func(try int) time.Duration
Say func(string)
}
// transientFor is how many transient retries say the phase applying before the have says failed: past them the bus has
// not answered for long enough that it is no longer a moment, and the failure is loud while the retries go on.
const transientFor = 4
// Agent follows the wants.
type Agent struct {
env Env
mu sync.Mutex
wants map[string]wantNow // by module
comps map[string]compNow // by composed key
running map[string]delivery.Have
// applied is the composition each module was last applied from: a revision a want named from the key's history is
// not the key's latest value, and what it holds is still held out of the others' applies.
applied map[string]compNow
// attempt is the last apply of a module that failed or is held, with the revisions it was made from.
attempt map[string]attempt
checked map[string]bool
// prior is a module the node ran before this agent started, so it has something to back up.
prior map[string]bool
said map[string]string
retry chan []string
// tries counts the transient failures in a row of each module; again carries a retry whose wait ran out, and
// transient asks for every transiently failed module at once. stopped is closed when Run returns.
tries map[string]int
gen uint64
again chan againAsk
transient chan struct{}
stopped chan struct{}
}
type againAsk struct {
module string
gen uint64
}
type wantNow struct {
Want delivery.Want
Node bool // the node's own want
Present bool
Rev uint64
}
type compNow struct {
Raw []byte
C delivery.Composed
Dig string
Rev uint64
Err error
}
type attempt struct {
FP, Composed string
WantRev, CompRev uint64
Held bool
AnsweredRev uint64
// Transient marks a failure of the bus, retried by the clock; Gen names the retry that is due for it.
Transient bool
Gen uint64
}
// New is an agent for a node.
func New(env Env) *Agent {
if env.Say == nil {
env.Say = func(string) {}
}
return &Agent{env: env, wants: map[string]wantNow{}, comps: map[string]compNow{}, running: map[string]delivery.Have{}, applied: map[string]compNow{},
attempt: map[string]attempt{}, checked: map[string]bool{}, prior: map[string]bool{}, said: map[string]string{},
retry: make(chan []string, 8), tries: map[string]int{}, again: make(chan againAsk, 64),
transient: make(chan struct{}, 1), stopped: make(chan struct{})}
}
// Run follows the wants until ctx ends. The keys there when it starts are taken first, the node's own composition
// before any module's, then each change as it comes.
func (a *Agent) Run(ctx context.Context) error {
defer close(a.stopped)
wants, err := a.env.Wants.Watch(ctx, "want."+a.env.Node+".>")
if err != nil {
return fmt.Errorf("the agent cannot watch its wants: %w", err)
}
comps, err := a.env.Composed.Watch(ctx, a.env.Node+".>")
if err != nil {
return fmt.Errorf("the agent cannot watch its compositions: %w", err)
}
var wantsCaught, compsCaught bool
dirty := map[string]bool{}
for {
select {
case <-ctx.Done():
return nil
case modules := <-a.retry:
// Applied again though the keys did not change: a container that waited because it said busy (ADR 0360
// §11) is replaced once it is idle.
a.mu.Lock()
for _, m := range modules {
delete(a.running, m)
delete(a.attempt, m)
dirty[m] = true
}
a.mu.Unlock()
case ask := <-a.again:
// The wait of a transiently failed apply ran out: tried again from the keys it has, if nothing tried it since.
a.mu.Lock()
if att, ok := a.attempt[ask.module]; ok && att.Transient && att.Gen == ask.gen {
delete(a.attempt, ask.module)
dirty[ask.module] = true
}
a.mu.Unlock()
case <-a.transient:
a.mu.Lock()
for m, att := range a.attempt {
if att.Transient {
delete(a.attempt, m)
dirty[m] = true
}
}
a.mu.Unlock()
case u, ok := <-wants:
if !ok {
return fmt.Errorf("the watch of the wants ended")
}
if u.Caught {
wantsCaught = true
} else if m := a.takeWant(u); m != "" {
dirty[m] = true
}
case u, ok := <-comps:
if !ok {
return fmt.Errorf("the watch of the compositions ended")
}
if u.Caught {
compsCaught = true
} else if m := a.takeComposition(u); m != "" {
dirty[m] = true
// The node's own may wait for this one to read (unreadableModules): looked at again with it.
dirty[NodeModule] = true
}
}
// Nothing is applied until what was there at the start is all in: a module's composition may arrive after its
// want, and the node's own composition comes before any module's.
if !(wantsCaught && compsCaught) || len(dirty) == 0 {
continue
}
a.settle(ctx, dirty)
dirty = map[string]bool{}
}
}
// Retry asks for the modules to be applied again from the keys they have, though nothing changed: what ADR 0360 §11
// needs for a container that waits because it said busy. It never blocks; a retry asked while eight wait is dropped, and
// asked again by its caller's next look.
func (a *Agent) Retry(modules ...string) {
select {
case a.retry <- modules:
default:
}
}
// RetryTransient asks for every module whose last apply failed transiently to be applied again now: the engine says it
// when it knows the bus answers again (after it wrote the bus's user list and the bus reloaded its accounts). It never
// blocks.
func (a *Agent) RetryTransient() {
select {
case a.transient <- struct{}{}:
default:
}
}
// Running is the compositions the node runs now: each module's that the agent applied or found already running, and
// the node's own. What the node judges its health by and says of itself is built from them.
func (a *Agent) Running() []delivery.Composed {
a.mu.Lock()
defer a.mu.Unlock()
mods := make([]string, 0, len(a.running))
for m := range a.running {
mods = append(mods, m)
}
sort.Strings(mods)
var out []delivery.Composed
for _, m := range mods {
h := a.running[m]
if h.Phase == delivery.PhaseRemoved {
continue
}
if c, ok := a.compositionFor(m, h.FP); ok && c.Err == nil && c.Dig == h.Composed {
out = append(out, c.C)
} else if c, ok := a.applied[m]; ok && c.Dig == h.Composed {
out = append(out, c.C)
}
}
return out
}
// takeWant takes a change of a want key and returns the module it concerns, or "" for a key that is not a want.
func (a *Agent) takeWant(u Update) string {
a.mu.Lock()
defer a.mu.Unlock()
if _, err := delivery.ParseWantNodeKey(u.Key); err == nil {
w := wantNow{Node: true, Rev: u.Revision}
if !u.Deleted {
nw, err := delivery.DecodeNodeWant(u.Value)
if err != nil {
a.sayOnce("want:node", fmt.Sprintf("the node's own want is not one: %v", err))
return ""
}
w.Present, w.Want = true, delivery.Want{FP: "", Composed: nw.Composed}
}
a.wants[NodeModule] = w
return NodeModule
}
node, module, err := delivery.ParseWantKey(u.Key)
if err != nil || node != a.env.Node {
return ""
}
w := wantNow{Rev: u.Revision}
if !u.Deleted {
want, err := delivery.DecodeWant(u.Value)
if err != nil {
a.sayOnce("want:"+module, fmt.Sprintf("the want of %s is not one: %v", module, err))
return ""
}
w.Present, w.Want = true, want
}
a.wants[module] = w
return module
}
// takeComposition takes a change of a composition key and returns the module it concerns.
func (a *Agent) takeComposition(u Update) string {
node, module, _, isNode, err := delivery.ParseComposedKey(u.Key)
if err != nil || node != a.env.Node {
return ""
}
a.mu.Lock()
defer a.mu.Unlock()
if isNode {
module = NodeModule
}
if u.Deleted {
delete(a.comps, u.Key)
return module
}
now := compNow{Raw: u.Value, Rev: u.Revision}
c, err := delivery.DecodeComposed(u.Value)
if err == nil {
var trusted []ed25519.PublicKey
if a.env.Trusted != nil {
trusted = a.env.Trusted()
}
// The signature before anything of the composition is used: its digest is what the want names.
now.C = c
now.Dig, err = delivery.VerifyComposed(c, trusted)
if err != nil {
now.C = delivery.Composed{}
a.sayOnce("verify:"+u.Key, fmt.Sprintf("the composition %s is refused and nothing of it is applied: %v", u.Key, err))
err = fmt.Errorf("refused: %w", err)
}
} else {
a.sayOnce("decode:"+u.Key, fmt.Sprintf("the composition %s cannot be read, and nothing of it is applied: %v", u.Key, err))
}
now.Err = err
a.comps[u.Key] = now
return module
}
func (a *Agent) sayOnce(key, line string) {
if a.said[key] == line {
return
}
a.said[key] = line
a.env.Say(line)
}
// settle applies what the dirty modules' wants allow: the node's own first, then each module in name order.
func (a *Agent) settle(ctx context.Context, dirty map[string]bool) {
order := make([]string, 0, len(dirty))
for m := range dirty {
order = append(order, m)
}
sort.Slice(order, func(i, j int) bool {
if (order[i] == NodeModule) != (order[j] == NodeModule) {
return order[i] == NodeModule
}
return order[i] < order[j]
})
for _, m := range order {
if ctx.Err() != nil {
return
}
a.settleOne(ctx, m)
}
}
func (a *Agent) compositionFor(module, fp string) (compNow, bool) {
var key string
var err error
if module == NodeModule {
key, err = delivery.ComposedNodeKey(a.env.Node)
} else {
key, err = delivery.ComposedKey(a.env.Node, module, fp)
}
if err != nil {
return compNow{}, false
}
c, ok := a.comps[key]
return c, ok
}
// revisionNamed is the revision of a module's composition key whose digest is the one given, signed by a trusted key,
// from the revisions the bucket keeps; false when none is.
func (a *Agent) revisionNamed(ctx context.Context, module, fp, digest string) (compNow, bool) {
if a.env.Revisions == nil {
return compNow{}, false
}
var key string
var err error
if module == NodeModule {
key, err = delivery.ComposedNodeKey(a.env.Node)
} else {
key, err = delivery.ComposedKey(a.env.Node, module, fp)
}
if err != nil {
return compNow{}, false
}
revs, err := a.env.Revisions(ctx, key)
if err != nil {
a.mu.Lock()
a.sayOnce("history:"+key, fmt.Sprintf("the revisions of %s cannot be read: %v", key, err))
a.mu.Unlock()
return compNow{}, false
}
var trusted []ed25519.PublicKey
if a.env.Trusted != nil {
trusted = a.env.Trusted()
}
for i := len(revs) - 1; i >= 0; i-- {
u := revs[i]
if u.Deleted || u.Key != key {
continue
}
c, err := delivery.DecodeComposed(u.Value)
if err != nil {
continue
}
dig, err := delivery.VerifyComposed(c, trusted)
if err != nil || dig != digest {
continue
}
return compNow{Raw: u.Value, C: c, Dig: dig, Rev: u.Revision}, true
}
return compNow{}, false
}
// others are what the node's other modules run now, so what they hold is not taken: the compositions the agent runs and,
// for every module the agent does not hold yet, the resources the kept declaration has for it (ADR 0429 §4).
func (a *Agent) others(except string) []delivery.Composed {
var out []delivery.Composed
mods := make([]string, 0, len(a.running))
for m := range a.running {
if m != except {
mods = append(mods, m)
}
}
sort.Strings(mods)
for _, m := range mods {
h := a.running[m]
if h.Phase == delivery.PhaseRemoved {
continue
}
if c, ok := a.compositionFor(m, h.FP); ok && c.Err == nil && c.Dig == h.Composed {
out = append(out, c.C)
} else if c, ok := a.applied[m]; ok && c.Dig == h.Composed {
out = append(out, c.C)
}
}
if a.env.Kept != nil {
kept := a.env.Kept()
names := make([]string, 0, len(kept))
for m := range kept {
if _, held := a.running[m]; !held && m != except {
names = append(names, m)
}
}
sort.Strings(names)
for _, m := range names {
c := delivery.Composed{Node: a.env.Node, Module: m, Resources: kept[m]}
if a.env.KeptFilled != nil {
c.Filled = a.env.KeptFilled()[m]
}
if m == NodeModule {
c.Module = ""
}
out = append(out, c)
}
}
return out
}
// unreadableModules are the keys of the module compositions in the node's bucket that cannot be read or are refused, of a
// module the assigner has not removed: for each module only the composition its want names, else its newest, since a
// stale one nobody applies (written before a rule the SDK now holds) must not hold the node's own back. Called holding
// a.mu.
func (a *Agent) unreadableModules() []string {
type pick struct {
key string
c compNow
}
current := map[string]pick{}
for key, c := range a.comps {
_, m, fp, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode {
continue
}
w := a.wants[m]
if w.Present && w.Want.Removed {
continue
}
if w.Present && w.Want.FP != "" {
if w.Want.FP == fp {
current[m] = pick{key, c}
}
continue
}
if prev, ok := current[m]; !ok || c.Rev > prev.c.Rev {
current[m] = pick{key, c}
}
}
var out []string
for _, p := range current {
if p.c.Err != nil {
out = append(out, p.key)
}
}
sort.Strings(out)
return out
}
// heldOut is what an apply or a removal holds out: others, and every module whose own composition the node's bucket holds
// and the agent does not run yet. A module that left the node's own composition for one of its own keeps its resources
// until that one is applied, so the node's own apply never removes them in between (issue 1945, live on 2026-10-25:
// shanks lost systemd-resolved's guard while its composition waited for a credential). The composition its want names,
// else its newest. Not part of Everything: what has not been applied is not what runs.
func (a *Agent) heldOut(except string) []delivery.Composed {
out := a.others(except)
held := map[string]bool{}
for _, c := range out {
held[c.Module] = true
}
waiting := map[string]compNow{}
for key, c := range a.comps {
_, m, fp, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode || c.Err != nil || m == except || held[m] {
continue
}
if h, ok := a.running[m]; ok && h.Phase != delivery.PhaseRemoved {
continue
}
if w := a.wants[m]; w.Present && w.Want.Removed {
continue // a module the assigner removed holds nothing back
}
if w := a.wants[m]; w.Present && !w.Want.Removed && w.Want.FP == fp {
waiting[m] = c
continue
}
if w := a.wants[m]; w.Present && !w.Want.Removed && waiting[m].Rev != 0 && waiting[m].C.FP == w.Want.FP {
continue
}
if prev, ok := waiting[m]; !ok || c.Rev > prev.Rev {
waiting[m] = c
}
}
names := make([]string, 0, len(waiting))
for m := range waiting {
names = append(names, m)
}
sort.Strings(names)
for _, m := range names {
out = append(out, waiting[m].C)
}
return out
}
// Everything is what the node runs: the compositions the agent runs, and the kept declaration's resources for each module
// the agent does not hold yet. What the node judges its health by, and what the filling, the take act and the judgement
// of directories read, is built from it.
func (a *Agent) Everything() []delivery.Composed {
a.mu.Lock()
defer a.mu.Unlock()
return a.others("\x00")
}
// HoldsNode says whether the agent runs the node's own composition, so the node's parts are its and not the kept
// declaration's.
func (a *Agent) HoldsNode() bool {
a.mu.Lock()
defer a.mu.Unlock()
h, ok := a.running[NodeModule]
return ok && h.Phase != delivery.PhaseRemoved
}
func (a *Agent) settleOne(ctx context.Context, module string) {
a.mu.Lock()
w := a.wants[module]
a.mu.Unlock()
if !w.Present {
return
}
if w.Want.Removed {
a.remove(ctx, module, w)
return
}
a.mu.Lock()
comp, have := a.compositionFor(module, w.Want.FP)
a.mu.Unlock()
if (!have || comp.Err != nil || comp.Dig != w.Want.Composed) && w.Want.Composed != "" {
// The composition the want names may be an older revision of the key, replaced by a newer one at the same
// fingerprint (a want held there keeps its digest): taken from the revisions the bucket keeps (ADR 0539 clause 1).
if old, found := a.revisionNamed(ctx, module, w.Want.FP, w.Want.Composed); found {
comp, have = old, true
}
}
if !have || comp.Err != nil {
return // waits for the composition; a change of its key settles the module again
}
if comp.Dig != w.Want.Composed {
// A composition whose digest no want names is not applied.
a.mu.Lock()
a.sayOnce("digest:"+module, fmt.Sprintf("the composition of %s at %s has digest %s, and its want names %s; nothing is applied until they agree",
module, w.Want.FP, comp.Dig, w.Want.Composed))
a.mu.Unlock()
return
}
a.mu.Lock()
if !a.checked[module] {
a.checked[module] = true
a.mu.Unlock()
if a.baseline(ctx, module, w, comp) {
return
}
} else {
a.mu.Unlock()
}
a.mu.Lock()
if r, ok := a.running[module]; ok && r.FP == w.Want.FP && r.Composed == w.Want.Composed && r.Phase != delivery.PhaseRemoved {
a.mu.Unlock()
return
}
att, tried := a.attempt[module]
same := tried && att.FP == w.Want.FP && att.Composed == w.Want.Composed
if same && !att.Held && att.WantRev == w.Rev && att.CompRev == comp.Rev {
a.mu.Unlock()
return // failed already for exactly these keys: retried by the next change of either
}
if same && att.Held && att.AnsweredRev >= w.Rev && att.CompRev == comp.Rev {
a.mu.Unlock()
return // held for a failed backup, and no answer since
}
if module == NodeModule {
// Not while a module's composition cannot be read or is refused: what it carries may be what left the node's own,
// and the node's own apply would remove it, held out by nothing (issue 1945, the review's gap 2).
if bad := a.unreadableModules(); len(bad) > 0 {
a.sayOnce("node-waits:"+strings.Join(bad, ","), fmt.Sprintf("the node's own composition is not applied while "+
"%s cannot be read or is refused: what it carries would be removed", strings.Join(bad, ", ")))
a.mu.Unlock()
return
}
}
others := a.heldOut(module)
a.mu.Unlock()
a.apply(ctx, module, w, comp, others)
}
// baseline decides whether what runs already is what the want names, once per module at the agent's start: the node's
// own have (what this node-engine last applied), else the declaration it kept before the switch (ADR 0539 §9). True when
// it is, with the have written; nothing is applied.
func (a *Agent) baseline(ctx context.Context, module string, w wantNow, comp compNow) bool {
// What a have or the kept declaration says runs is believed only when the engine still holds a record of each of its
// resources: a resource removed since (a node's own apply took it, issue 1945) is made again by an apply.
if a.env.Recorded != nil && !a.env.Recorded(ctx, comp.C) {
if module != NodeModule {
if _, ok := a.env.Haves.Read(ctx, module); ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
}
if a.env.Kept != nil {
if _, ok := a.env.Kept()[module]; ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
}
a.mu.Lock()
a.sayOnce("unrecorded:"+module+":"+comp.Dig, fmt.Sprintf("%s's composition %s is applied again: this engine holds "+
"no record of some of its resources", module, short(comp.Dig)))
a.mu.Unlock()
return false
}
if module != NodeModule {
if h, ok := a.env.Haves.Read(ctx, module); ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
if h.Composed != "" {
// This node-engine wrote it: what it says is what runs, and the kept declaration says nothing more. A
// failed apply or a different composition is applied again.
if h.FP == w.Want.FP && h.Composed == w.Want.Composed &&
(h.Phase == "" || h.Phase == string(delivery.PhaseRunning)) {
a.mu.Lock()
a.running[module] = delivery.Have{FP: h.FP, Composed: h.Composed, Phase: delivery.PhaseRunning}
a.mu.Unlock()
// Said again, so the session holds the module and its key carries the health this engine judges,
// not the one an older engine left there (xorg's unknown on g14 and shanks, 2026-10-25).
a.env.Haves.Write(ctx, Have{Module: module, FP: h.FP, Composed: h.Composed,
Phase: string(delivery.PhaseRunning), Detail: h.Detail})
return true
}
return false
}
}
}
if a.env.Kept == nil {
return false
}
kept, ok := a.env.Kept()[module]
if ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
if !ok || !SameResources(kept, comp.C.Resources) {
return false
}
a.mu.Lock()
a.running[module] = delivery.Have{FP: w.Want.FP, Composed: w.Want.Composed, Phase: delivery.PhaseRunning}
a.mu.Unlock()
a.env.Haves.Write(ctx, Have{Module: module, FP: w.Want.FP, Composed: w.Want.Composed, Phase: string(delivery.PhaseRunning)})
return true
}
func (a *Agent) apply(ctx context.Context, module string, w wantNow, comp compNow, others []delivery.Composed) {
fp, dig := w.Want.FP, w.Want.Composed
a.mu.Lock()
_, ranBefore := a.running[module]
ranBefore = ranBefore || a.prior[module]
a.mu.Unlock()
// A data module is backed up before a new composition of it moves (ADR 0429 §14), unless the want says to go on
// without one (ADR 0539 §16).
if comp.C.Data && ranBefore && a.env.Backup != nil && w.Want.Backup != delivery.BackupSkip {
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseBackingUp)})
if got := a.env.Backup(ctx, module); !got.Done {
a.env.Say(fmt.Sprintf("%s is held: its backup before %s failed: %s", module, fp, got.Words))
a.mu.Lock()
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev, Held: true, AnsweredRev: w.Rev}
prev := a.running[module]
a.mu.Unlock()
// It keeps running what it has: the have says what it holds and why it waits.
a.env.Haves.Write(ctx, Have{Module: module, FP: prev.FP, Composed: prev.Composed, Phase: string(delivery.PhaseHeld),
Detail: got.Words})
return
}
}
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseApplying)})
res := a.env.Apply(ctx, comp.C, others)
a.mu.Lock()
what := fmt.Sprintf("%s at %s", module, fp)
if module == NodeModule {
what = "the node's own composition"
}
if res.Failed && res.Transient {
a.tries[module]++
try := a.tries[module]
a.gen++
gen := a.gen
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev, Transient: true, Gen: gen}
a.mu.Unlock()
phase, when := delivery.PhaseApplying, "when the bus answers again"
if a.env.Backoff != nil {
wait := a.env.Backoff(try)
when = "in " + wait.String()
time.AfterFunc(wait, func() {
select {
case a.again <- againAsk{module, gen}:
case <-a.stopped:
case <-ctx.Done():
}
})
}
if try > transientFor {
phase = delivery.PhaseFailed // still retried, and said as failed: no longer a moment
}
a.env.Say(fmt.Sprintf("applying %s failed because the bus did not answer (try %d), retried %s: %s", what, try, when, res.Detail))
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(phase),
Detail: fmt.Sprintf("retrying %s (try %d): %s", when, try, res.Detail)})
return
}
delete(a.tries, module)
if res.Failed {
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev}
a.mu.Unlock()
a.env.Say(fmt.Sprintf("applying %s failed: %s", what, res.Detail))
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed), Detail: res.Detail})
return
}
delete(a.attempt, module)
a.running[module] = delivery.Have{FP: fp, Composed: dig, Phase: delivery.PhaseRunning}
a.applied[module] = comp
a.mu.Unlock()
if a.env.Settled != nil {
a.env.Settled(module)
}
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseRunning), Detail: res.Detail})
}
func (a *Agent) remove(ctx context.Context, module string, w wantNow) {
a.mu.Lock()
att := a.attempt[module]
r, ran := a.running[module]
if att.FP == "removed" && att.WantRev == w.Rev {
a.mu.Unlock()
return
}
if ran && r.Phase == delivery.PhaseRemoved {
a.mu.Unlock()
return
}
others := a.heldOut(module)
a.mu.Unlock()
res := a.env.Remove(ctx, module, others)
a.mu.Lock()
if res.Failed {
a.attempt[module] = attempt{FP: "removed", WantRev: w.Rev}
a.mu.Unlock()
a.env.Say(fmt.Sprintf("removing %s failed: %s", module, res.Detail))
return
}
delete(a.attempt, module)
a.running[module] = delivery.Have{Phase: delivery.PhaseRemoved}
a.mu.Unlock()
if a.env.Settled != nil {
a.env.Settled(module)
}
a.env.Haves.Removed(ctx, module)
}
// SameResources says whether two lists of resources hold the same resources, each by its id and its canonical JSON.
func SameResources(a, b []json.RawMessage) bool {
ma, ok := canonByID(a)
if !ok {
return false
}
mb, ok := canonByID(b)
if !ok || len(ma) != len(mb) {
return false
}
for id, x := range ma {
if y, there := mb[id]; !there || x != y {
return false
}
}
return true
}
func canonByID(rs []json.RawMessage) (map[string]string, bool) {
out := map[string]string{}
for _, raw := range rs {
var head struct {
ID string `json:"id"`
}
if json.Unmarshal(raw, &head) != nil || head.ID == "" {
return nil, false
}
canon, err := Canonical(raw)
if err != nil {
return nil, false
}
out[head.ID] = string(canon)
}
return out, true
}
// Canonical is raw with its object keys sorted and its spacing dropped, its numbers kept as written.
func Canonical(raw json.RawMessage) (json.RawMessage, error) {
var v any
d := json.NewDecoder(bytes.NewReader(raw))
d.UseNumber()
if err := d.Decode(&v); err != nil {
return nil, err
}
return json.Marshal(v)
}
func short(s string) string {
if i := strings.IndexByte(s, ':'); i >= 0 {
s = s[i+1:]
}
if len(s) > 12 {
s = s[:12]
}
return s
}
+954
View File
@@ -0,0 +1,954 @@
package agent
import (
"context"
"crypto/ed25519"
"encoding/json"
"slices"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const node = "n1"
// The keys of the tests: the controller's and the operator's are trusted, a stranger's is not.
var controllerPub, controllerKey = newKey()
var operatorPub, operatorKey = newKey()
var _, strangerKey = newKey()
func newKey() (ed25519.PublicKey, ed25519.PrivateKey) {
pub, key, err := ed25519.GenerateKey(nil)
if err != nil {
panic(err)
}
return pub, key
}
func fpOf(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
// A watcher the test writes to; it says Caught once the keys written before the watch began are out.
type feed struct {
mu sync.Mutex
ch chan Update
rev uint64
first []Update
}
func newFeed() *feed { return &feed{ch: make(chan Update, 256)} }
func (f *feed) Watch(_ context.Context, _ string) (<-chan Update, error) {
f.mu.Lock()
defer f.mu.Unlock()
for _, u := range f.first {
f.ch <- u
}
f.ch <- Update{Caught: true}
return f.ch, nil
}
// before writes a key as if it was there when the watch began.
func (f *feed) before(key string, value []byte) {
f.mu.Lock()
defer f.mu.Unlock()
f.rev++
f.first = append(f.first, Update{Key: key, Value: value, Revision: f.rev})
}
func (f *feed) put(key string, value []byte) {
f.mu.Lock()
f.rev++
r := f.rev
f.mu.Unlock()
f.ch <- Update{Key: key, Value: value, Revision: r}
}
type haves struct {
mu sync.Mutex
stored map[string]Have
log []Have
removed []string
}
func (h *haves) Read(_ context.Context, module string) (Have, bool) {
h.mu.Lock()
defer h.mu.Unlock()
x, ok := h.stored[module]
return x, ok
}
func (h *haves) Write(_ context.Context, x Have) {
h.mu.Lock()
defer h.mu.Unlock()
if h.stored == nil {
h.stored = map[string]Have{}
}
h.stored[x.Module] = x
h.log = append(h.log, x)
}
func (h *haves) Removed(_ context.Context, m string) {
h.mu.Lock()
defer h.mu.Unlock()
h.removed = append(h.removed, m)
}
func (h *haves) last(module string) Have {
h.mu.Lock()
defer h.mu.Unlock()
return h.stored[module]
}
type bed struct {
t *testing.T
wants *feed
comps *feed
haves *haves
mu sync.Mutex
applied []string
removed []string
backups []string
failWith string
// transient marks failWith as the bus not answering; failTimes > 0 fails only the first that many applies.
transient bool
failTimes int
backup func(module string) Backup
kept map[string][]json.RawMessage
others [][]delivery.Composed
said []string
// settled is each module Settled was told, and settledRan whether what runs held it then.
settled []string
settledRan []bool
ag *Agent // the agent run() runs
env Env
}
func newBed(t *testing.T) *bed {
b := &bed{t: t, wants: newFeed(), comps: newFeed(), haves: &haves{}}
b.env = Env{Node: node, Wants: b.wants, Composed: b.comps, Haves: b.haves,
Apply: func(_ context.Context, c delivery.Composed, others []delivery.Composed) Result {
b.mu.Lock()
defer b.mu.Unlock()
b.others = append(b.others, others)
name := c.Module
if name == "" {
name = NodeModule
}
b.applied = append(b.applied, name)
if b.failWith != "" && (b.failTimes == 0 || len(b.applied) <= b.failTimes) {
return Result{Failed: true, Transient: b.transient, Detail: b.failWith}
}
return Result{}
},
Remove: func(_ context.Context, m string, _ []delivery.Composed) Result {
b.mu.Lock()
defer b.mu.Unlock()
b.removed = append(b.removed, m)
return Result{}
},
Backup: func(_ context.Context, m string) Backup {
b.mu.Lock()
defer b.mu.Unlock()
b.backups = append(b.backups, m)
if b.backup != nil {
return b.backup(m)
}
return Backup{Done: true}
},
Settled: func(m string) {
ran := false
if b.ag != nil {
for _, c := range b.ag.Everything() {
ran = ran || c.Module == m
}
}
b.mu.Lock()
defer b.mu.Unlock()
b.settled = append(b.settled, m)
b.settledRan = append(b.settledRan, ran)
},
Kept: func() map[string][]json.RawMessage { return b.kept },
Trusted: func() []ed25519.PublicKey { return []ed25519.PublicKey{controllerPub, operatorPub} },
Say: func(s string) {
t.Log(s)
b.mu.Lock()
b.said = append(b.said, s)
b.mu.Unlock()
},
}
return b
}
func (b *bed) run() (stop func()) {
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
b.ag = New(b.env)
go func() { defer close(done); _ = b.ag.Run(ctx) }()
return func() { cancel(); <-done }
}
func file(module, id, content string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": module + "." + id, "type": "file", "path": "/tmp/r6/" + module + "/" + id,
"content": content, "module": module})
return raw
}
// composition is a module's composition at a fingerprint: its key, its bytes and its digest, from the writer's code.
func composition(t *testing.T, module, fp string, data bool, rs ...json.RawMessage) (key string, raw []byte, digest string) {
t.Helper()
c := delivery.Composed{Node: node, Module: module, FP: fp, Data: data, Resources: rs}
var err error
if module == NodeModule {
c.Module, c.FP = "", ""
key, err = delivery.ComposedNodeKey(node)
} else {
key, err = delivery.ComposedKey(node, module, fp)
}
if err != nil {
t.Fatal(err)
}
return signedBy(t, c, key, controllerKey)
}
// signedBy is c signed with signer, as the writer's code signs it: its key, its bytes and its digest.
func signedBy(t *testing.T, c delivery.Composed, key string, signer ed25519.PrivateKey) (string, []byte, string) {
t.Helper()
signed, err := delivery.SignComposed(c, signer)
if err != nil {
t.Fatal(err)
}
raw, err := delivery.EncodeComposed(signed)
if err != nil {
t.Fatal(err)
}
digest, err := delivery.ComposedDigestOf(signed)
if err != nil {
t.Fatal(err)
}
return key, raw, digest
}
func wantFor(t *testing.T, module, fp, digest, backup string) (string, []byte) {
t.Helper()
key, err := delivery.WantKey(node, module)
if err != nil {
t.Fatal(err)
}
raw, err := delivery.EncodeWant(delivery.Want{FP: fp, Composed: digest, Backup: backup})
if err != nil {
t.Fatal(err)
}
return key, raw
}
func eventually(t *testing.T, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for !ok() {
if time.Now().After(deadline) {
t.Fatal(what)
}
time.Sleep(5 * time.Millisecond)
}
}
func (b *bed) appliedList() []string {
b.mu.Lock()
defer b.mu.Unlock()
return append([]string(nil), b.applied...)
}
// R6-A AC1 (the agent, with a bus that is a fake): given a want naming a digest and that composition, the module is
// applied and its have names the fingerprint and the digest; a composition whose digest no want names is not applied.
func TestAWantAndItsCompositionApplyTheModuleAndTheHaveNamesBoth(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
// A composition no want names yet is not applied.
b.comps.put(ck, craw)
time.Sleep(100 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a composition no want names was applied: %v", got)
}
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "the module was not applied", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "the have does not name both", func() bool {
h := b.haves.last("web")
return h.FP == fp && h.Composed == dig && h.Phase == string(delivery.PhaseRunning)
})
// Settled is told once what runs holds the module, so a declaration read then names its files (the filling's Knock;
// ace, 2026-10-25 21:17).
eventually(t, "Settled was not told web with web in what runs", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return slices.Equal(b.settled, []string{"web"}) && b.settledRan[0]
})
// A want naming a digest the composition at that fingerprint does not have: nothing is applied.
fp2 := fpOf("b")
ck2, craw2, _ := composition(t, "web", fp2, false, file("web", "x", "2"))
b.comps.put(ck2, craw2)
wk2, wraw2 := wantFor(t, "web", fp2, "sha256:"+strings.Repeat("0", 64), "")
b.wants.put(wk2, wraw2)
time.Sleep(100 * time.Millisecond)
if got := b.appliedList(); len(got) != 1 {
t.Fatalf("a want naming another digest applied the composition: %v", got)
}
}
// R6-A AC2: given a kept declaration and wants and compositions equal to it, when the agent starts, nothing is applied
// and every have names its fingerprint and digest.
func TestStartingOnWhatTheKeptDeclarationHoldsAppliesNothing(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
rs := []json.RawMessage{file("web", "x", "1"), file("web", "y", "2")}
ck, craw, dig := composition(t, "web", fp, false, rs...)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
// The kept declaration holds the same resources, in another order and spacing.
b.kept = map[string][]json.RawMessage{"web": {json.RawMessage(strings.ReplaceAll(string(rs[1]), ",", " , ")), rs[0]}}
stop := b.run()
defer stop()
eventually(t, "the have was not written", func() bool { return b.haves.last("web").Composed == dig })
if h := b.haves.last("web"); h.FP != fp || h.Phase != string(delivery.PhaseRunning) {
t.Fatalf("the have: %+v", h)
}
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that already runs what the want names was applied: %v", got)
}
}
// What the node-engine itself last applied is what runs after a restart, not the kept declaration.
func TestStartingOnItsOwnHaveAppliesNothing(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: dig, Phase: "running"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
time.Sleep(150 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that runs what the want names was applied after a restart: %v", got)
}
}
// A module found running on its own have after a restart is said again, once, through the haves: the session then holds
// it and writes the health this engine judges. Not said, its key kept the health an older engine wrote: xorg's have on
// g14 and shanks said unknown from before the engine said a module with nothing to judge ok, and the assigner held every
// module requiring x11-display back for it (2026-10-25).
func TestStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "xorg", fp, false, file("xorg", "x", "1"))
wk, wraw := wantFor(t, "xorg", fp, dig, "")
b.haves.stored = map[string]Have{"xorg": {Module: "xorg", FP: fp, Composed: dig, Phase: "running", Detail: "kept"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
eventually(t, "the have found running was not said again", func() bool {
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
return len(b.haves.log) > 0
})
time.Sleep(50 * time.Millisecond)
b.haves.mu.Lock()
log := append([]Have(nil), b.haves.log...)
b.haves.mu.Unlock()
if len(log) != 1 {
t.Fatalf("the have was said %d times, want once: %+v", len(log), log)
}
if h := log[0]; h.Module != "xorg" || h.FP != fp || h.Composed != dig || h.Phase != string(delivery.PhaseRunning) || h.Detail != "kept" {
t.Fatalf("the have said again: %+v", h)
}
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that runs what the want names was applied: %v", got)
}
}
// A module whose kept resources differ from the composition is applied, and a failed apply is retried only by a change
// of either key.
func TestAFailedApplyIsRetriedOnlyByAChangeOfEitherKey(t *testing.T) {
// A real failure is not retried by the clock, however short the backoff: only by a change of either key.
t.Run("a real failure", func(t *testing.T) {
b := newBed(t)
b.failWith = "web.x: no space"
b.env.Backoff = func(int) time.Duration { return time.Millisecond }
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "the have does not say failed", func() bool {
h := b.haves.last("web")
return h.Phase == string(delivery.PhaseFailed) && strings.Contains(h.Detail, "no space") && h.FP == fp
})
// An unrelated key changes: not a retry.
ck9, craw9, _ := composition(t, "web", fpOf("9"), false, file("web", "x", "9"))
b.comps.put(ck9, craw9)
time.Sleep(150 * time.Millisecond)
if n := len(b.appliedList()); n != 1 {
t.Fatalf("a failed apply was retried by an unrelated key or the clock (%d applies)", n)
}
// The want is written again: the retry.
b.mu.Lock()
b.failWith = ""
b.mu.Unlock()
b.wants.put(wk, wraw)
eventually(t, "the change of the want did not retry", func() bool { return len(b.appliedList()) == 2 })
eventually(t, "not running", func() bool { return b.haves.last("web").Phase == string(delivery.PhaseRunning) })
})
// The bus did not answer for a moment (mesh-vault on the bus node, 2026-10-25 23:04:24): tried again with no new
// revision of either key, by the backoff or by RetryTransient once the engine knows the bus is back; the have says
// retrying meanwhile, never failed.
for name, backoff := range map[string]func(int) time.Duration{
"a transient failure, by the backoff": func(int) time.Duration { return time.Millisecond },
"a transient failure, by RetryTransient": nil,
} {
t.Run(name, func(t *testing.T) {
b := newBed(t)
b.failWith, b.transient, b.failTimes = "fetching the object: nats: no responders available for request", true, 1
b.env.Backoff = backoff
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "vault", fp, false, file("vault", "x", "1"))
wk, wraw := wantFor(t, "vault", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) >= 1 })
if backoff == nil {
eventually(t, "the have does not say retrying", func() bool {
return strings.HasPrefix(b.haves.last("vault").Detail, "retrying")
})
b.ag.RetryTransient()
}
eventually(t, "not applied again without a new revision", func() bool { return len(b.appliedList()) == 2 })
eventually(t, "not running", func() bool { return b.haves.last("vault").Phase == string(delivery.PhaseRunning) })
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
for _, h := range b.haves.log {
if h.Phase == string(delivery.PhaseFailed) {
t.Errorf("a transient failure was written failed: %+v", h)
}
}
})
}
}
// R6-A AC5: a data module that ran before asks its backup before any resource changes; a refusal holds it, nothing is
// applied and the have says held; when the want then says skip, the module is applied.
func TestADataModuleIsBackedUpFirstAndHeldWhenItFails(t *testing.T) {
b := newBed(t)
fp1, fp2 := fpOf("a"), fpOf("b")
ck1, craw1, dig1 := composition(t, "db", fp1, true, file("db", "x", "1"))
wk1, wraw1 := wantFor(t, "db", fp1, dig1, "")
b.comps.before(ck1, craw1)
b.wants.before(wk1, wraw1)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}}
refuse := true
b.backup = func(string) Backup {
if refuse {
return Backup{Words: "restic: the repository is locked"}
}
return Backup{Done: true}
}
stop := b.run()
defer stop()
eventually(t, "baseline", func() bool { return b.haves.last("db").Composed == dig1 })
ck2, craw2, dig2 := composition(t, "db", fp2, true, file("db", "x", "2"))
b.comps.put(ck2, craw2)
wk2, wraw2 := wantFor(t, "db", fp2, dig2, "")
b.wants.put(wk2, wraw2)
eventually(t, "the backup was not asked", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.backups) == 1
})
eventually(t, "the have does not say held", func() bool {
h := b.haves.last("db")
return h.Phase == string(delivery.PhaseHeld) && strings.Contains(h.Detail, "repository is locked") && h.FP == fp1
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("something was applied behind a failed backup: %v", got)
}
// The answer arrives in the module's own want: back up again (it fails again), then skip.
wk, again := wantFor(t, "db", fp2, dig2, delivery.BackupAgain)
b.wants.put(wk, again)
eventually(t, "back up again did not ask the backup", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.backups) == 2
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied behind a second failed backup: %v", got)
}
_, skip := wantFor(t, "db", fp2, dig2, delivery.BackupSkip)
b.wants.put(wk, skip)
eventually(t, "skip did not apply the module", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
n := len(b.backups)
b.mu.Unlock()
if n != 2 {
t.Fatalf("a backup was asked after the answer was to skip: %d asks", n)
}
eventually(t, "not running at the new fingerprint", func() bool {
h := b.haves.last("db")
return h.Phase == string(delivery.PhaseRunning) && h.FP == fp2 && h.Composed == dig2
})
}
// A data module with a good backup is applied after it; one that never ran has nothing to back up.
func TestABackupDoneIsFollowedByTheApplyAndANewModuleIsNotBackedUp(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("c")
ck, craw, dig := composition(t, "db", fp, true, file("db", "x", "1"))
wk, wraw := wantFor(t, "db", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
n := len(b.backups)
b.mu.Unlock()
if n != 0 {
t.Fatalf("a module that never ran was backed up (%d)", n)
}
fp2 := fpOf("d")
ck2, craw2, dig2 := composition(t, "db", fp2, true, file("db", "x", "2"))
wk2, wraw2 := wantFor(t, "db", fp2, dig2, "")
b.comps.put(ck2, craw2)
b.wants.put(wk2, wraw2)
eventually(t, "second not applied", func() bool { return len(b.appliedList()) == 2 })
b.mu.Lock()
n = len(b.backups)
b.mu.Unlock()
if n != 1 {
t.Fatalf("a module that ran before was backed up %d times, want once", n)
}
}
// R6-A AC4 at the agent: a removed want removes that module only, and its have is written removed.
func TestARemovedWantRemovesThatModuleOnly(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
gone, _ := delivery.EncodeWant(delivery.Want{Removed: true})
b.wants.put(wk, gone)
eventually(t, "not removed", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.removed) == 1 && b.removed[0] == "web"
})
eventually(t, "the have was not written removed", func() bool {
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
return len(b.haves.removed) == 1
})
}
// The node's own composition is applied before any module's, whichever key came first.
func TestTheNodesOwnCompositionComesBeforeAnyModules(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, err := delivery.WantNodeKey(node)
if err != nil {
t.Fatal(err)
}
nwraw, err := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
if err != nil {
t.Fatal(err)
}
b.comps.before(ck, craw)
b.comps.before(nk, nraw)
b.wants.before(wk, wraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "both not applied", func() bool { return len(b.appliedList()) == 2 })
if got := b.appliedList(); got[0] != NodeModule || got[1] != "web" {
t.Fatalf("the order was %v, the node's own first", got)
}
}
// R6-A AC9: a composition signed by either trusted key is applied; one signed by another key, one changed after it was
// signed and one unsigned are refused and nothing of them is applied (ADR 0539 §3).
func TestOnlyACompositionSignedByATrustedKeyIsApplied(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
plain := func(module, fp, content string) delivery.Composed {
return delivery.Composed{Node: node, Module: module, FP: fp, Resources: []json.RawMessage{file(module, "x", content)}}
}
put := func(c delivery.Composed, signer ed25519.PrivateKey, tamper func([]byte) []byte) {
key, _ := delivery.ComposedKey(node, c.Module, c.FP)
var raw []byte
var dig string
if signer == nil {
raw, _ = delivery.EncodeComposed(c)
dig, _ = delivery.ComposedDigestOf(c)
} else {
_, raw, dig = signedBy(t, c, key, signer)
}
if tamper != nil {
raw = tamper(raw)
}
b.comps.put(key, raw)
wk, wraw := wantFor(t, c.Module, c.FP, dig, "")
b.wants.put(wk, wraw)
}
// Refused: signed by a stranger, changed after signing, unsigned.
put(plain("stranger", fpOf("1"), "x"), strangerKey, nil)
put(plain("changed", fpOf("2"), "before"), controllerKey, func(raw []byte) []byte {
return []byte(strings.Replace(string(raw), "before", "after!", 1))
})
put(plain("unsigned", fpOf("3"), "x"), nil, nil)
time.Sleep(200 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a composition that is not signed by a trusted key was applied: %v", got)
}
// Applied: signed by the controller, signed by the operator.
put(plain("bycontroller", fpOf("4"), "x"), controllerKey, nil)
put(plain("byoperator", fpOf("5"), "x"), operatorKey, nil)
eventually(t, "the trusted signers' compositions were not applied", func() bool { return len(b.appliedList()) == 2 })
got := b.appliedList()
if !(got[0] == "bycontroller" && got[1] == "byoperator") && !(got[0] == "byoperator" && got[1] == "bycontroller") {
t.Fatalf("applied %v", got)
}
if h := b.haves.last("byoperator"); h.Phase != string(delivery.PhaseRunning) || h.FP != fpOf("5") {
t.Fatalf("the operator's module has no running have: %+v", h)
}
}
// With no key trusted, nothing is applied: a node that does not know its signers fails closed.
func TestWithNoTrustedKeyNothingIsApplied(t *testing.T) {
b := newBed(t)
b.env.Trusted = nil
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
time.Sleep(200 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied with no trusted key: %v", got)
}
}
// What a module's apply must not take is every other module that runs, those the agent holds and those still running from
// the kept declaration (ADR 0429 §4): the resources of the kept declaration reach Apply as the others.
func TestApplyingOneModuleHoldsOutTheModulesStillRunningFromTheKeptDeclaration(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}, NodeModule: {file("node", "n", "n")}}
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
defer b.mu.Unlock()
var names []string
for _, c := range b.others[0] {
names = append(names, c.Module)
}
if len(names) != 2 || !strings.Contains(strings.Join(names, ","), "db") {
t.Fatalf("the others of web were %v: db and the node's own run from the kept declaration", names)
}
}
// Everything is what runs: the agent's compositions and the kept resources of each module it does not hold.
func TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}, "web": {file("web", "x", "old")}}
a := New(b.env)
if got := a.Everything(); len(got) != 2 {
t.Fatalf("before the agent holds anything: %d compositions, want the 2 kept", len(got))
}
a.running["web"] = delivery.Have{FP: fpOf("a"), Composed: "sha256:" + strings.Repeat("1", 64), Phase: delivery.PhaseRunning}
ck, craw, dig := composition(t, "web", fpOf("a"), false, file("web", "x", "new"))
a.running["web"] = delivery.Have{FP: fpOf("a"), Composed: dig, Phase: delivery.PhaseRunning}
a.takeComposition(Update{Key: ck, Value: craw, Revision: 1})
got := a.Everything()
if len(got) != 2 {
t.Fatalf("%d compositions", len(got))
}
for _, c := range got {
if c.Module == "web" && !strings.Contains(string(c.Resources[0]), "new") {
t.Fatal("web is the kept declaration's, not the composition the agent runs")
}
}
if a.HoldsNode() {
t.Fatal("the agent holds no node composition")
}
}
// A failed apply is applied again after a restart even when the kept declaration holds the composition: once this
// node-engine wrote its haves, they are what it knows (the kept declaration is only for the first start after the switch).
func TestAFailedApplyIsAppliedAgainAfterARestartWhateverIsKept(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
b.kept = map[string][]json.RawMessage{"web": {file("web", "x", "1")}}
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed)}}
stop := b.run()
defer stop()
eventually(t, "a failed apply was marked running from the kept declaration", func() bool { return len(b.appliedList()) == 1 })
}
// A have written before the switch carries no composition: then the kept declaration says what runs.
func TestAHaveWithNoCompositionDefersToTheKeptDeclaration(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
b.kept = map[string][]json.RawMessage{"web": {file("web", "x", "1")}}
b.haves.stored = map[string]Have{"web": {Module: "web", Phase: "running"}}
stop := b.run()
defer stop()
eventually(t, "the have was not written", func() bool { return b.haves.last("web").Composed == dig })
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied %v", got)
}
}
// A module that leaves the node's own composition for a composition of its own keeps its resources until that one is
// applied: the node's own apply holds out every module whose composition the node's bucket holds and the agent does not
// run yet (issue 1945, live on 2026-10-25 at 02:32: shanks' node-engine applied a node's own composition that no longer
// carried systemd-resolved, whose own composition waited for its credential, and removed its guard, so the machine lost
// its resolver).
func TestTheNodesOwnApplyHoldsOutAModuleThatMovedToItsOwnComposition(t *testing.T) {
b := newBed(t)
fp := fpOf("b")
// systemd-resolved's own composition is written; no want names it yet.
rk, rraw, _ := composition(t, "systemd-resolved", fp, false, file("systemd-resolved", "guard", "g"))
b.comps.before(rk, rraw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, err := delivery.WantNodeKey(node)
if err != nil {
t.Fatal(err)
}
nwraw, err := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
if err != nil {
t.Fatal(err)
}
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
defer b.mu.Unlock()
var names []string
for _, c := range b.others[0] {
names = append(names, c.Module)
}
if !strings.Contains(strings.Join(names, ","), "systemd-resolved") {
t.Fatalf("the others of the node's own apply were %v: systemd-resolved's resources would be removed", names)
}
}
// A module the assigner removed holds nothing back, its compositions still in the bucket or not: its resources are its
// removal's to take (issue 1945).
func TestARemovedModulesCompositionHoldsNothingBack(t *testing.T) {
b := newBed(t)
fp := fpOf("b")
rk, rraw, _ := composition(t, "gone", fp, false, file("gone", "x", "1"))
b.comps.before(rk, rraw)
gk, err := delivery.WantKey(node, "gone")
if err != nil {
t.Fatal(err)
}
graw, err := delivery.EncodeWant(delivery.Want{Removed: true})
if err != nil {
t.Fatal(err)
}
b.wants.before(gk, graw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool {
for _, m := range b.appliedList() {
if m == NodeModule {
return true
}
}
return false
})
b.mu.Lock()
defer b.mu.Unlock()
for _, others := range b.others {
for _, c := range others {
if c.Module == "gone" {
t.Fatal("a removed module's composition was held out")
}
}
}
}
// The node's own composition is not applied while a module's composition in the node's bucket cannot be read or is not
// signed by a trusted key: what that one carries may be what left the node's own, and the node's own apply would remove
// it (issue 1945, the review's gap 2). It is said, naming the composition, and applied once that one reads.
func TestTheNodesOwnWaitsWhileAModulesCompositionCannotBeRead(t *testing.T) {
b := newBed(t)
bad, err := delivery.ComposedKey(node, "broken", fpOf("c"))
if err != nil {
t.Fatal(err)
}
b.comps.before(bad, []byte(`{"v":1,"not":"a composition"}`))
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the wait not said", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
for _, s := range b.said {
if strings.Contains(s, bad) {
return true
}
}
return false
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied %v while a composition could not be read", got)
}
// Read again, the node's own is applied.
gk, graw, _ := composition(t, "broken", fpOf("c"), false, file("broken", "x", "1"))
b.comps.put(gk, graw)
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
}
// A stale refused composition beside a newer readable one of the same module does not hold the node's own apply back:
// only the composition the module's want names, else its newest, counts (the review of issue 1945: the bucket keeps
// compositions written before the filled paths, which the SDK now refuses, and nobody applies them).
func TestAStaleRefusedCompositionDoesNotHoldTheNodesOwnBack(t *testing.T) {
b := newBed(t)
stale, err := delivery.ComposedKey(node, "power", fpOf("a"))
if err != nil {
t.Fatal(err)
}
b.comps.before(stale, []byte(`{"v":1,"not":"a composition"}`))
gk, graw, _ := composition(t, "power", fpOf("b"), false, file("power", "x", "1"))
b.comps.before(gk, graw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
}
// A composition the kept declaration or the node's own have says runs is applied all the same when the engine holds no
// record of its resources: they were removed since (issue 1945, live on 2026-10-25: ace's and g14's node-engines removed
// ca-trust's, power's and matrix's resources in a node's own apply, and the wants that named those modules' own
// compositions were judged already running from the kept declaration, so nothing made them again).
func TestACompositionWhoseResourcesWereRemovedIsAppliedAgain(t *testing.T) {
for _, from := range []string{"kept", "have"} {
t.Run(from, func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
rs := []json.RawMessage{file("ca-trust", "trust", "1")}
ck, craw, dig := composition(t, "ca-trust", fp, false, rs...)
wk, wraw := wantFor(t, "ca-trust", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
if from == "kept" {
b.kept = map[string][]json.RawMessage{"ca-trust": rs}
} else {
b.haves.stored = map[string]Have{"ca-trust": {Module: "ca-trust", FP: fp, Composed: dig, Phase: "running"}}
}
b.env.Recorded = func(context.Context, delivery.Composed) bool { return false }
stop := b.run()
defer stop()
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
})
}
}
// The composition a want names is applied from the key's history when a newer one has replaced it under the same key:
// a want held at a fingerprint keeps naming the digest it was written with, and the controller writes every new
// composition at that fingerprint over the same key (ADR 0539 clause 1, the sweep's kept revisions; issue 1945, live on
// 2026-10-25: shanks' node-tools composition had digest 575a5c96, its held want named a36e63c8, and nothing was applied).
func TestTheRevisionAWantNamesIsAppliedFromHistory(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, oldRaw, oldDig := composition(t, "node-tools", fp, false, file("node-tools", "x", "old"))
_, newRaw, _ := composition(t, "node-tools", fp, false, file("node-tools", "x", "new"))
b.comps.before(ck, newRaw) // the key's latest value
wk, wraw := wantFor(t, "node-tools", fp, oldDig, "")
b.wants.before(wk, wraw)
b.env.Revisions = func(_ context.Context, key string) ([]Update, error) {
if key != ck {
return nil, nil
}
return []Update{{Key: ck, Value: oldRaw, Revision: 3}, {Key: ck, Value: newRaw, Revision: 7}}, nil
}
stop := b.run()
defer stop()
eventually(t, "the revision the want names was not applied", func() bool { return len(b.appliedList()) == 1 })
if h := b.haves.last("node-tools"); h.Composed != oldDig {
t.Fatalf("the have names %s, want %s", h.Composed, oldDig)
}
}
// 1945/AC18 (unit): a module that still runs from the kept declaration carries the kept declaration's filled paths of
// its files, so the filling writes them ("not named as filled" for claude-code and screen-lock, 2026-10-25).
func TestAModuleRunningFromTheKeptDeclarationKeepsItsFilledPaths(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"claude-code": {file("claude-code", "settings", "")}}
b.env.KeptFilled = func() map[string][]string {
return map[string][]string{"claude-code": {"/tmp/r6/claude-code/settings"}}
}
a := New(b.env)
got := a.Everything()
if len(got) != 1 || got[0].Module != "claude-code" || strings.Join(got[0].Filled, ",") != "/tmp/r6/claude-code/settings" {
t.Fatalf("what runs from the kept declaration: %+v", got)
}
}
-23
View File
@@ -3,7 +3,6 @@ package apply
import (
"context"
"os"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
@@ -32,28 +31,6 @@ func TestAnAccessPresentIsConfirmedAndNothingIsChanged(t *testing.T) {
}
}
// Absent is refused, not created. A bind mount whose source does not exist is made by the
// container runtime as root, with whatever mode it picks — the silent wrong-ownership
// 04-ISSUES/026 records. So the host says plainly that the operator must provide the path, rather
// than conjuring a directory it does not own.
func TestAnAccessThatIsAbsentIsRefusedClearlyAndNotCreated(t *testing.T) {
missing := t.TempDir() + "/media/library" // named, never created
d := declare(t, `{"id":"lib","type":"access","path":"`+missing+`"}`)
_, _, err := Apply(context.Background(), archHost(t), d, store.State{},
store.OriginDeclared, noServices, nil, nil)
if err == nil {
t.Fatal("an absent operator-owned path was accepted, and would be created as root by the runtime")
}
if !strings.Contains(err.Error(), "the operator must provide") ||
!strings.Contains(err.Error(), "does not own") {
t.Fatalf("the refusal does not say whose the path is: %v", err)
}
if _, statErr := os.Stat(missing); statErr == nil {
t.Fatal("the host created the path it does not own")
}
}
// Undeclaring an access never removes the path. Unassigning the module that reached the media
// library must not delete the library — that is the data loss ADR 0030 exists to prevent, on a
// directory the mesh never made. The record is dropped; the operator's data is left exactly as it
+76
View File
@@ -0,0 +1,76 @@
package apply
// What is never placed below an agent's home (novox/hq ADR 0266 §7).
//
// An account that never becomes root owns every directory in its home, so it can swap a link into the path
// between the engine's check and the engine's write. The engine writes a whole file there through a
// descriptor that followed no link (home_links.go), but it unpacks an archive, and writes into an existing
// file, by path. So below the home of an account the declaration says never becomes root, neither is placed.
// The catalogue's own check (checks/agenthome) refuses either under `${machine:agent-home}`, so no module
// asks; this is the engine refusing it again where it applies. The machine's own trees, other accounts'
// homes and a directory's owner below a home are placement_guard.go's (novox/hq issue 339).
import (
"fmt"
"path/filepath"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// AgentHomeRefusedError is an archive or a written-into file the engine will not place below an agent's home.
type AgentHomeRefusedError struct {
Path, What, Home string
}
func (e *AgentHomeRefusedError) Error() string {
return fmt.Sprintf("%s is not placed: %s below %s, the home of an account that never becomes root, which "+
"owns every parent there and could swap a link in between the check and the write (novox/hq ADR 0266); "+
"nothing was touched", e.Path, e.What, e.Home)
}
// refuseUnderAgentHome says why a resource is not placed below one of agentHomes; nil when it may be.
func refuseUnderAgentHome(r declaration.Resource, agentHomes []string) error {
what := ""
switch res := r.(type) {
case *declaration.Archive:
what = "an archive unpacked"
case *declaration.File:
if res.Into != "" {
what = "a file written into (" + res.Into + ")"
}
}
if what == "" {
return nil
}
path := filepath.Clean(r.Target())
for _, p := range []string{path, resolved(path)} {
for _, home := range agentHomes {
if atOrBelow(p, home) {
return &AgentHomeRefusedError{Path: path, What: what, Home: home}
}
}
}
return nil
}
// rootNeverHomes is the home of every account the declaration says never becomes root, from the user
// database; an account not made yet has no home to protect.
func rootNeverHomes(d *declaration.Declaration) []string {
if d == nil {
return nil
}
homes := accountsOfHomes()
var out []string
for _, r := range d.Resources {
u, ok := r.(*declaration.User)
if !ok || u.Root != declaration.RootNever {
continue
}
for home, a := range homes {
if a.Name == u.Name {
out = append(out, home)
}
}
}
return out
}
+36
View File
@@ -0,0 +1,36 @@
package apply
// Defends novox/hq ADR 0266 §7: no archive is unpacked and no file is written into below the home of an account
// the declaration says never becomes root; a whole file there still passes.
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
func TestNothingIsUnpackedOrWrittenIntoBelowAnAgentsHome(t *testing.T) {
was := accountsOfHomes
accountsOfHomes = func() map[string]homeAccount {
return map[string]homeAccount{"/home/agent": {"agent", 1001}, "/home/operator": {"operator", 1000}}
}
t.Cleanup(func() { accountsOfHomes = was })
agent := []string{"/home/agent"}
if err := refuseUnderAgentHome(&declaration.Archive{ID: "a.x", Type: declaration.TypeArchive, Path: "/home/agent/.local/x", Owner: "agent"}, agent); err == nil {
t.Error("an archive below an agent's home is refused")
}
if err := refuseUnderAgentHome(&declaration.File{ID: "a.f", Type: declaration.TypeFile, Path: "/home/agent/.claude.json", Owner: "agent", Into: "json", Content: "{}"}, agent); err == nil {
t.Error("a file written into below an agent's home is refused")
}
if err := refuseUnderAgentHome(&declaration.File{ID: "a.f", Type: declaration.TypeFile, Path: "/home/agent/x", Owner: "agent", Content: "x"}, agent); err != nil {
t.Errorf("a whole file there passes: %v", err)
}
if err := refuseUnderAgentHome(&declaration.Archive{ID: "a.x", Type: declaration.TypeArchive, Path: "/home/operator/x", Owner: "operator"}, agent); err != nil {
t.Errorf("an archive in a home whose account may become root is placement_guard.go's to judge, not this: %v", err)
}
d := parse(t, `{"declaration":1,"resources":[{"id":"c.agent","type":"user","name":"agent","root":"never"},`+
`{"id":"c.op","type":"user","name":"operator"}]}`)
if got := rootNeverHomes(d); len(got) != 1 || got[0] != "/home/agent" {
t.Errorf("the agent's home is known by the user database: %v", got)
}
}
+67
View File
@@ -0,0 +1,67 @@
package apply
import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// WriteAlone writes one file a module's composition declares, through the apply's own file step, and records it as an
// apply would; nothing else of the module is applied, and nothing that reads the file is moved (novox/hq ADR 0547 §8).
//
// **For a file whose reader takes the change itself.** The bus's user list is the one it is for: the bus module's
// entrypoint watches the file and signals the server to reload in place, so the list is in force without a want of the
// bus, which nobody writes (ADR 0539 §15), and without the container being touched. A file that a service or a container
// must be moved to read is not written alone: it goes through the module's apply, which moves what reads it.
//
// The file is written only when its content differs from what is there (the file step's own rule), read back, and
// recorded under the module whose resource it is, so the module's next apply finds it as its own and writes nothing.
// The caller holds the apply's lock. log is said the path, the action and the digest, never the content.
func WriteAlone(ctx context.Context, sys system.System, file *declaration.File, known store.State, run Runner,
log func(string)) (Outcome, store.State, error) {
if log == nil {
log = func(string) {}
}
if file.Into != "" || file.CreateOnce || file.Sealed != "" || file.VaultSealed != nil || len(file.Secrets) > 0 {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is not a whole file of plain content, "+
"and only such a file is written alone; nothing was written", file.Path)}
}
if file.IsReference() {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is given by digest %s and was not "+
"resolved to its content; it is not written empty", file.Path, file.Digest)}
}
if held, isHeld := known.HeldAt(file.ID); isHeld {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is held as found for a module not "+
"yet taken (%s), so nothing was written", file.Path, held.Why)}
}
was, _ := known.Find(file.ID)
if was.ID == "" {
// Only over a file the mesh wrote: a file this node holds no record of is the module's apply's to take, with
// the original kept, never this write's. A record of the path under another id (the declaration before the
// switch named it so) is the one written to, so no second record of the file is left for an apply to remove.
rec, ok := known.At(string(declaration.TypeFile), file.Path)
if !ok {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("this node holds no record of %s; it "+
"is written by the module's apply first, never alone", file.Path)}
}
renamed := *file
renamed.ID = rec.ID
file, was = &renamed, rec
}
in := inputs{declares: map[string]string{file.Identity(): declaredDigest(file)}, known: &known,
groups: groupsWanted([]declaration.Resource{file}), handed: Handed{}, dirsBefore: map[string]bool{},
files: map[string]bool{file.Path: true}}
outcome, err := applyOne(ctx, sys, file, run, map[string]bool{}, in, was, nil, nil)
if err != nil {
log(fmt.Sprintf(" failed %s (%s), alone: %v", file.ID, file.Path, err))
return outcome, known, &Error{Resource: file.ID, Err: err}
}
recordKeeping(&known, file, outcome, was)
if moved(outcome) {
log(fmt.Sprintf(" %s %s (%s), alone: %s", outcome.Action, file.ID, file.Path, outcome.wrote))
}
return outcome, known, nil
}
+90
View File
@@ -0,0 +1,90 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0547 §8: the bus node writes the bus's user list alone, with no want, through the apply's own
// file step: only when it differs, mode 0600, read back and recorded as the module's, and nothing else moves.
func TestAFileWrittenAloneMovesNothingElse(t *testing.T) {
const was, now = "user a\n", "user a\nuser b\n"
for _, c := range []struct {
name, fields, before string
recorded bool
recordedAs string // the id the record has, when not the composition's
action, refused string
after string
}{
{name: "no record of it", fields: `"content":%q,"mode":"0600"`, before: was, refused: "holds no record", after: was},
{name: "changed", fields: `"content":%q,"mode":"0600"`, before: was, recorded: true, action: "updated", after: now},
{name: "the same", fields: `"content":%q,"mode":"0600"`, before: now, recorded: true, action: "unchanged", after: now},
{name: "recorded under the id from before the switch", fields: `"content":%q,"mode":"0600"`, before: was,
recorded: true, recordedAs: "bus-users", action: "updated", after: now},
{name: "absent with a record", fields: `"content":%q,"mode":"0600"`, recorded: true, action: "created", after: now},
{name: "written into", fields: `"content":%q,"into":"block","owner-at-runtime":"managed"`, before: was,
recorded: true, refused: "only such a file", after: was},
{name: "sealed", fields: `"sealed":"c2VhbGVk"`, before: was, recorded: true,
refused: "only such a file", after: was},
} {
t.Run(c.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "accounts.conf")
if c.before != "" {
if err := os.WriteFile(path, []byte(c.before), 0o600); err != nil {
t.Fatal(err)
}
}
fields := strings.Replace(c.fields, "%q", fmt.Sprintf("%q", now), 1)
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"nats.bus-users","module":"nats",
"type":"file","path":%q,`+fields+`}]}`, path))
file := d.Resources[0].(*declaration.File)
known := store.State{}
id := file.ID
if c.recordedAs != "" {
id = c.recordedAs
}
if c.recorded {
known.Record(store.Applied{ID: id, Type: string(declaration.TypeFile), Module: "nats", Target: path,
Origin: store.OriginDeclared, Wrote: digestOf(c.before)})
}
calls := 0
counting := func(context.Context, string, ...string) (string, error) { calls++; return "", nil }
out, after, err := WriteAlone(context.Background(), archHost(t), file, known, counting, nil)
if c.refused != "" {
if err == nil || !strings.Contains(err.Error(), c.refused) {
t.Fatalf("not refused with %q: %v", c.refused, err)
}
} else if err != nil {
t.Fatal(err)
} else if out.Action != c.action {
t.Errorf("action %q, want %q", out.Action, c.action)
}
if got := readText(t, path); got != c.after {
t.Errorf("file holds %q, want %q", got, c.after)
}
if calls != 0 {
t.Errorf("%d commands were run: a file written alone moves nothing", calls)
}
if c.refused != "" {
return
}
if m := modeAt(t, path); m != 0o600 {
t.Errorf("mode %o", m)
}
rec, ok := after.Find(id)
if !ok || rec.Wrote != digestOf(now) || rec.Target != path {
t.Errorf("record %+v (found %v), want it to say %s was written", rec, ok, digestOf(now))
}
if n := len(after.Resources); n != 1 {
t.Errorf("%d records of one file", n)
}
})
}
}
+1052 -200
View File
File diff suppressed because it is too large Load Diff
+27 -1
View File
@@ -9,7 +9,7 @@ import (
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
@@ -313,6 +313,7 @@ func TestAServiceIsReadBackNotAssumed(t *testing.T) {
// running. A unit that starts and immediately dies satisfies the command.
started := false
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if args[0] == "show" {
if started {
return "LoadState=loaded\nActiveState=failed\n", nil // started, then died
@@ -337,6 +338,7 @@ func TestAServiceIsReadBackNotAssumed(t *testing.T) {
func TestAnUnknownServiceStateIsRefusedNotGuessed(t *testing.T) {
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "LoadState=loaded\nActiveState=reticent\n", nil
}
d := parse(t, `{"declaration":1,"resources":[
@@ -371,6 +373,7 @@ func TestADroppedServiceIsGivenBackTheStateItWasFoundIn(t *testing.T) {
t.Run(c.name, func(t *testing.T) {
var commands []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
commands = append(commands, strings.Join(args, " "))
if args[0] == "show" {
return "LoadState=loaded\nActiveState=active\n", nil
@@ -413,6 +416,7 @@ func TestAServiceRecordsTheStateItWasFoundInOnceAndCarriesIt(t *testing.T) {
active := "inactive"
enabled := "disabled"
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "show":
return "LoadState=loaded\nActiveState=" + active + "\n", nil
@@ -464,6 +468,7 @@ func TestAUnitThatDoesNotExistIsNotStopped(t *testing.T) {
// Absence read as satisfaction — 04-ISSUES/007 wearing a different hat, and the mirror of
// the degraded-init bug the capability detector had.
absent := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "LoadState=not-found\nActiveState=inactive\n", nil
}
d := parse(t, `{"declaration":1,"resources":[
@@ -486,6 +491,7 @@ func TestAMaskedUnitIsRefused(t *testing.T) {
// Masked means someone deliberately made it unstartable. Applying over that would undo a
// decision the host did not make and cannot see the reason for.
masked := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "LoadState=masked\nActiveState=inactive\n", nil
}
d := parse(t, `{"declaration":1,"resources":[
@@ -506,6 +512,7 @@ func TestForgettingAUnitThatIsGoneDoesNotStrandTheNode(t *testing.T) {
// already true.
var stopped bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if args[0] == "show" {
return "LoadState=not-found\nActiveState=inactive\n", nil
}
@@ -557,6 +564,7 @@ func TestABrokenPackageDatabaseIsNotReadAsNotInstalled(t *testing.T) {
// answer would silently reinstall on a machine whose package manager is broken, or report
// "installed nothing" as success. The apply must fail instead.
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "", errors.New("pacman: error: could not lock database")
}
d := parseTrusted(t, `{"declaration":1,"resources":[
@@ -575,6 +583,7 @@ func TestABrokenPackageDatabaseIsNotReadAsNotInstalled(t *testing.T) {
func TestAnInstalledPackageIsNotReinstalled(t *testing.T) {
var installed bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if args[0] == "-S" {
installed = true
}
@@ -602,6 +611,7 @@ func TestAPackageIsNeverUninstalled(t *testing.T) {
// machine may have had it before the mesh ever saw it. Undeclaring is not "remove it".
var uninstalled bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if len(args) > 0 && (args[0] == "-R" || args[0] == "-Rs") {
uninstalled = true
}
@@ -635,6 +645,7 @@ func TestAnActionThatIsAlreadyTrueDoesNotRun(t *testing.T) {
// database is, so "is it already there" is a question only the declaration can ask.
var ran bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "create-db" {
ran = true
}
@@ -661,6 +672,7 @@ func TestAnActionThatSucceedsAndDoesNothingFails(t *testing.T) {
// this repository's most expensive failure shape. Here the command "succeeds" every time
// and verify never passes.
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "has-db" {
return "", errors.New("no such database")
}
@@ -687,6 +699,7 @@ func TestAnActionRunsInsideTheContainerItNames(t *testing.T) {
// there is any mesh to ask.
var sawExec bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "docker" && args[0] == "exec" && args[1] == "store" {
sawExec = true
return "", nil
@@ -710,6 +723,7 @@ func TestAContainerThatExitsImmediatelyFailsTheApply(t *testing.T) {
// still running. A container whose entrypoint dies satisfies the command exactly as one
// that came up does — which is the read-back rule, in the place it matters most.
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch {
case args[0] == "info":
return "27.0\n", nil
@@ -748,6 +762,7 @@ func TestAContainerWhoseDeclarationChangedIsReplaced(t *testing.T) {
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "info":
return "27.0\n", nil
@@ -786,6 +801,7 @@ func TestAContainerThatMatchesIsLeftAlone(t *testing.T) {
var touched bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "info":
return "27.0\n", nil
@@ -831,6 +847,7 @@ func TestAContainerIsRecreatedWhenARestartOnResourceChanged(t *testing.T) {
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "info":
return "27.0\n", nil
@@ -886,6 +903,7 @@ func TestRestartOnFiresOnlyForResourcesThatChanged(t *testing.T) {
func systemctlStub(t *testing.T, load, active, enabled string, verbs *[]string) Runner {
t.Helper()
return func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "show":
return "LoadState=" + load + "\nActiveState=" + active + "\n", nil
@@ -1043,6 +1061,7 @@ func TestTheRuntimeProbeIsPerRuntime(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
var probedWith string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != tc.present {
return "", errors.New("not installed")
}
@@ -1071,6 +1090,7 @@ func TestAContainerUsesTheRuntimeTheMachineHas(t *testing.T) {
// machine already has (novox/hq research 012), so hardcoding one contradicts it.
var calledWith []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "docker" {
return "", errors.New("not installed")
}
@@ -1107,6 +1127,7 @@ func TestNoRuntimeIsSaidPlainly(t *testing.T) {
// Naming what was tried, because "docker: command not found" on a machine that deliberately
// runs podman sends the reader looking for the wrong thing.
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
return "", errors.New("not installed")
}
d := parseTrusted(t, `{"declaration":1,"resources":[
@@ -1250,6 +1271,7 @@ func TestAServiceIsNotRestartedByAChangeItDoesNotName(t *testing.T) {
// before and after one.
func recordingServices(commands *[]string) Runner {
return func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
line := name + " " + strings.Join(args, " ")
*commands = append(*commands, line)
switch {
@@ -1398,6 +1420,7 @@ func TestAFailedActionStopsWhatFollows(t *testing.T) {
func TestAContainerIsGivenTheMeshsNames(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("not installed")
}
@@ -1433,6 +1456,7 @@ func TestAContainerIsGivenTheMeshsNames(t *testing.T) {
func TestAContainerGivenNoNamesIsRunAsBefore(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("not installed")
}
@@ -1477,6 +1501,7 @@ func TestResourcesAreAppliedInTheOrderTheyWereDeclared(t *testing.T) {
var order []string
done := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
order = append(order, name+" "+strings.Join(args, " "))
// `verify` is the idempotency check, so it has to fail before the step and pass after —
// a stub that always succeeds means every action is already done and nothing ever runs,
@@ -1627,6 +1652,7 @@ func TestAContainerStaleFromAnEarlierApplyIsReplaced(t *testing.T) {
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "info":
return "27.0\n", nil
+1 -1
View File
@@ -13,7 +13,7 @@ import (
"path/filepath"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"path/filepath"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
+153
View File
@@ -0,0 +1,153 @@
package apply
import (
"context"
"fmt"
"strings"
"time"
"unicode/utf8"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that says it is busy is replaced once it is idle, within a bound (novox/hq ADR 0335).
//
// **Asked, never waited on.** The apply asks the container once, and while it says busy leaves it running
// as it is and goes on with everything else of the declaration: the machine's lock is held for one ask,
// never for the wait. The wait is carried between passes in the container's record (store.Applied's
// BusySince), so the next pass — the node-engine asks again sooner than its reconcile while anything waits
// — asks again, and so does a node-engine started again in between: the bound runs from the first apply
// that would have replaced the container, whoever applied it.
// waitsBusy is the action of a container left running because it said busy (novox/hq ADR 0335 §2). Its own
// word rather than "unchanged": its declaration moved, and the machine is not yet what was declared.
const waitsBusy = "waits-busy"
// What a report says of a busy container (novox/hq ADR 0335 §2, §3).
const (
// BusyWaiting is a container left running because it said busy, waiting to be replaced.
BusyWaiting = "waiting"
// BusyReplacedIdle is a container replaced on the first pass it said idle, after it had waited.
BusyReplacedIdle = "replaced-when-idle"
// BusyReplacedAtBound is a container still busy at the bound, replaced anyway.
BusyReplacedAtBound = "replaced-at-bound"
)
// BusyWait is one container a busy command held, as a report says it.
type BusyWait struct {
// ID is the resource's id (`<module>.<resource>`), Container its runtime name.
ID string `json:"id"`
Container string `json:"container"`
// State is BusyWaiting, BusyReplacedIdle or BusyReplacedAtBound.
State string `json:"state"`
// Since is when the first apply that would have replaced it ran; Until when the bound replaces it.
Since time.Time `json:"since"`
Until time.Time `json:"until"`
// Said is the last output its busy command gave.
Said string `json:"said,omitempty"`
}
// Words is the wait in a sentence, for an outcome's detail and the console.
func (b BusyWait) Words() string {
since := b.Since.UTC().Format(time.RFC3339)
said := ""
if b.Said != "" {
said = "; its busy command last said: " + b.Said
}
switch b.State {
case BusyReplacedAtBound:
return fmt.Sprintf("replaced while still busy: it said busy since %s and the bound of %s ran out "+
"(novox/hq ADR 0335 §3)%s", since, declaration.BusyBound, said)
case BusyReplacedIdle:
return fmt.Sprintf("replaced once it said idle, after waiting since %s (novox/hq ADR 0335 §2)%s", since, said)
}
return fmt.Sprintf("busy: left running and replaced on the first pass it says idle, or at %s at the latest; "+
"waiting since %s (novox/hq ADR 0335 §2)%s", b.Until.UTC().Format(time.RFC3339), since, said)
}
// owedSinceBusy is the reason a container still owes the replacement it was left running for, or empty: its
// record keeps when its busy wait began, under the same name.
func owedSinceBusy(r *declaration.Container, previous store.Applied) string {
if previous.BusySince == nil || previous.Target != r.Name {
return ""
}
return "the replacement owed since its busy wait began at " + previous.BusySince.UTC().Format(time.RFC3339)
}
// busyClock is the clock the bound is read on. A test moves it.
var busyClock = time.Now
// busySaidMost is how much of a busy command's output a report carries: its last lines, enough to say what
// it held.
const busySaidMost = 400
// askBeforeReplacing asks a running container that declares busy whether it may be replaced now, and says
// in out what came of it. True when it waits: out is then the whole outcome of this pass. False when the
// caller replaces it — idle, or past the bound — with out carrying the wait it ended, if there was one.
func askBeforeReplacing(ctx context.Context, r *declaration.Container, run Runner, cri string,
previous store.Applied, out *Outcome) bool {
now := busyClock().UTC()
busy, said := askBusy(ctx, r, run, cri)
var since time.Time
waited := previous.BusySince != nil && previous.Target == r.Name
if waited {
since = previous.BusySince.UTC()
// A wait kept from a clock that has since gone back reads as starting now, never as later: the bound
// holds from no later than this pass.
if since.After(now) {
since = now
}
}
wait := BusyWait{ID: r.ID, Container: r.Name, Since: since, Said: said}
switch {
case !busy && !waited:
// Idle at the first ask: replaced at once, as a container that declares nothing is.
return false
case !busy:
wait.State, wait.Until = BusyReplacedIdle, since.Add(declaration.BusyBound)
out.busy = &wait
return false
}
if !waited {
since = now
wait.Since = now
}
wait.Until = since.Add(declaration.BusyBound)
if !now.Before(wait.Until) {
wait.State = BusyReplacedAtBound
out.busy = &wait
return false
}
wait.State = BusyWaiting
out.Action, out.Detail = waitsBusy, wait.Words()
out.busySince, out.busy = &since, &wait
return true
}
// askBusy runs a container's busy command inside it, by its shell, as an exec health check is run: busy is
// an exit of 0 and nothing else. An ask that could not be made, or did not answer within its timeout, is
// not busy: the bound is what keeps a container whose command never answers from being held.
func askBusy(ctx context.Context, r *declaration.Container, run Runner, cri string) (bool, string) {
ask, cancel := context.WithTimeout(ctx, r.Busy.Within())
defer cancel()
output, err := run(ask, cri, "exec", r.Name, "/bin/sh", "-c", r.Busy.Command)
said := strings.TrimSpace(output)
if err != nil && said == "" {
said = err.Error()
}
return err == nil, lastOf(said, busySaidMost)
}
// lastOf is the end of a text, at most n bytes, on one line.
func lastOf(s string, n int) string {
s = strings.Join(strings.Fields(s), " ")
if len(s) <= n {
return s
}
from := len(s) - n
for from < len(s) && !utf8.RuneStart(s[from]) {
from++
}
return "…" + s[from:]
}
+434
View File
@@ -0,0 +1,434 @@
package apply
import (
"context"
"errors"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that says it is busy is replaced once it is idle, within a bound (novox/hq ADR 0335). The
// machine of these tests runs two containers, a build agent that declares `busy` and a web server that
// does not; every call that replaces a container is counted.
type aBusyMachine struct {
mu sync.Mutex
// containers is each container's name and the spec it was created with; running whether it runs.
containers map[string]string
busy bool // what the agent's busy command answers
said string // what it prints
asked int
calls []string
}
func newBusyMachine() *aBusyMachine {
return &aBusyMachine{containers: map[string]string{}}
}
func (m *aBusyMachine) run(_ context.Context, _ string, args ...string) (string, error) {
args = asideShape(args)
m.mu.Lock()
defer m.mu.Unlock()
switch args[0] {
case "info", "image":
return "27.0\n", nil
case "container": // inspect
name := args[len(args)-1]
spec, there := m.containers[name]
if !there {
return "", errors.New("no such container")
}
return "true\t" + spec, nil
case "exec":
m.asked++
m.calls = append(m.calls, "ask "+args[1])
if strings.Join(args[2:], " ") != "/bin/sh -c mesh-builder busy" {
return "", errors.New("asked something else: " + strings.Join(args[2:], " "))
}
if m.busy {
return m.said + "\n", nil
}
return m.said + "\n", errors.New("exited 1")
case "rm":
name := args[len(args)-1]
m.calls = append(m.calls, "rm "+name)
delete(m.containers, name)
case "run":
name := args[slices.Index(args, "--name")+1]
m.calls = append(m.calls, "create "+name)
for i, a := range args {
if a == "--label" && strings.HasPrefix(args[i+1], specLabel+"=") {
m.containers[name] = strings.TrimPrefix(args[i+1], specLabel+"=")
}
}
}
return "", nil
}
func (m *aBusyMachine) mark() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.calls)
}
// since is every call since mark that replaced a container, or asked one.
func (m *aBusyMachine) since(mark int) []string {
m.mu.Lock()
defer m.mu.Unlock()
return append([]string(nil), m.calls[mark:]...)
}
func (m *aBusyMachine) answer(busy bool, said string) {
m.mu.Lock()
defer m.mu.Unlock()
m.busy, m.said = busy, said
}
// twoContainers is the declaration at a version: v moves the agent's image tag through its env, w the web's.
func twoContainers(t *testing.T, agent, web string) *declaration.Declaration {
t.Helper()
return parse(t, `{"declaration":1,"resources":[
{"id":"build-agent.server","type":"container","name":"mesh-build-agent","image":"`+pinned+`",
"env":{"V":"`+agent+`"},"busy":{"command":"mesh-builder busy","timeout":"5s"}},
{"id":"web.server","type":"container","name":"web","image":"`+pinned+`","env":{"V":"`+web+`"}}
]}`)
}
// atClock moves the bound's clock for one test.
func atClock(t *testing.T, at *time.Time) {
t.Helper()
was := busyClock
busyClock = func() time.Time { return *at }
t.Cleanup(func() { busyClock = was })
}
func busyOf(r Report, id string) (BusyWait, bool) {
for _, b := range r.Busy {
if b.ID == id {
return b, true
}
}
return BusyWait{}, false
}
func applyBusy(t *testing.T, m *aBusyMachine, d *declaration.Declaration, known store.State) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatalf("the apply failed: %v", err)
}
return report, state
}
// The decision's first check: an apply that would replace a container whose busy command exits 0 leaves it
// running, applies the rest, and replaces it on the first pass the command exits non-zero.
func TestABusyContainerIsLeftRunningAndReplacedOnTheFirstPassItSaysIdle(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
// A merge moves both: the agent holds a build, the web server holds nothing.
m.answer(true, "building novox/mesh-controller (check of #280) since 00:58")
mark := m.mark()
report, state := applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm web", "create web"}) {
t.Fatalf("a busy agent was not left running while the rest applied: %v", got)
}
o := outcomeOf(report, "build-agent.server")
if o.Action != waitsBusy {
t.Fatalf("a busy container was reported %q, want %q: %+v", o.Action, waitsBusy, o)
}
for _, says := range []string{"busy", "first pass it says idle", "check of #280", "ADR 0335"} {
if !strings.Contains(o.Detail, says) {
t.Errorf("the outcome does not say %q: %s", says, o.Detail)
}
}
if (Report{Outcomes: []Outcome{o}}).Changed() {
t.Errorf("a container left running reads as a change: %+v", o)
}
b, said := busyOf(report, "build-agent.server")
if !said || b.State != BusyWaiting || !b.Since.Equal(now) || !b.Until.Equal(now.Add(15*time.Minute)) ||
b.Container != "mesh-build-agent" || !strings.Contains(b.Said, "check of #280") {
t.Fatalf("the report does not say the agent waits, since when and until when: %+v", report.Busy)
}
if !report.Waiting() {
t.Error("a report with a container waiting does not say so")
}
if rec, _ := state.Find("build-agent.server"); rec.BusySince == nil || !rec.BusySince.Equal(now) {
t.Fatalf("the wait is not kept in the container's record: %+v", rec)
}
// Asked again a minute later, still busy: still left running, the wait counted from the first apply.
now = now.Add(time.Minute)
mark = m.mark()
report, state = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent"}) {
t.Fatalf("a second pass while busy did more than ask: %v", got)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyWaiting || !b.Since.Equal(now.Add(-time.Minute)) {
t.Fatalf("the second pass does not keep the wait from the first apply: %+v", report.Busy)
}
// The build ends: the next pass replaces it, and says it waited.
now = now.Add(time.Minute)
m.answer(false, "idle")
mark = m.mark()
report, state = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("the agent was not replaced on the first pass it said idle: %v", got)
}
if o := outcomeOf(report, "build-agent.server"); o.Action != "updated" || !strings.Contains(o.Detail, "once it said idle") {
t.Errorf("the replacement does not say it waited: %+v", o)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyReplacedIdle {
t.Errorf("the report does not say the agent was replaced when idle: %+v", report.Busy)
}
if report.Waiting() {
t.Error("a report with nothing waiting says something waits")
}
if rec, _ := state.Find("build-agent.server"); rec.BusySince != nil {
t.Errorf("the wait is still kept after the replacement: %+v", rec)
}
// And the pass after that has nothing to say about it.
report, _ = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if len(report.Busy) != 0 {
t.Errorf("a container replaced is still said: %+v", report.Busy)
}
}
// An idle container is replaced at once, as one that declares nothing is: asked once, nothing waits.
func TestAnIdleContainerIsReplacedAtOnce(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(false, "")
mark := m.mark()
report, _ := applyBusy(t, m, twoContainers(t, "2", "1"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("an idle agent was not replaced at once: %v", got)
}
if len(report.Busy) != 0 {
t.Errorf("an idle container replaced at once is said as a wait: %+v", report.Busy)
}
// And a container whose declaration did not move is never asked.
mark = m.mark()
applyBusy(t, m, twoContainers(t, "2", "1"), state)
if got := m.since(mark); slices.Contains(got, "ask mesh-build-agent") {
t.Errorf("a container nothing replaces was asked: %v", got)
}
}
// Past the bound it is replaced anyway, the report carrying the command's last output — and the bound holds
// across a restart of the node-engine: the wait is read back from the state on disk.
func TestABusyContainerIsReplacedAtTheBoundEvenAfterARestart(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building novox/mesh-host (a check that never ends)")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
// The node-engine stops and starts again: what it knows is what it saved.
path := filepath.Join(t.TempDir(), "state.json")
if err := store.Save(path, state); err != nil {
t.Fatal(err)
}
reloaded, err := store.Load(path)
if err != nil {
t.Fatal(err)
}
now = now.Add(14 * time.Minute)
mark := m.mark()
report, reloaded := applyBusy(t, m, twoContainers(t, "2", "1"), reloaded)
if got := m.since(mark); slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("a container busy within the bound was replaced after a restart: %v", got)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyWaiting || !b.Since.Equal(now.Add(-14*time.Minute)) {
t.Fatalf("a restart started the wait again: %+v", report.Busy)
}
now = now.Add(time.Minute)
mark = m.mark()
report, _ = applyBusy(t, m, twoContainers(t, "2", "1"), reloaded)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("a container still busy at the bound was not replaced: %v", got)
}
b, _ := busyOf(report, "build-agent.server")
if b.State != BusyReplacedAtBound || !strings.Contains(b.Said, "never ends") {
t.Fatalf("the report does not say it replaced a busy container, with its last output: %+v", report.Busy)
}
if o := outcomeOf(report, "build-agent.server"); !strings.Contains(o.Detail, "still busy") ||
!strings.Contains(o.Detail, "never ends") {
t.Errorf("the outcome does not say a busy container was replaced, with what it said: %s", o.Detail)
}
}
// While one container waits, a send that changes another module on the same machine is applied at once: the
// apply returns after one ask, and holds nothing for the wait.
func TestASendToAnotherModuleIsAppliedWhileOneWaits(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
// The next send moves only the web server.
mark := m.mark()
done := make(chan struct{})
var report Report
go func() {
defer close(done)
report, _ = applyBusy(t, m, twoContainers(t, "2", "2"), state)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("an apply while a container waits did not return")
}
got := m.since(mark)
if !slices.Contains(got, "create web") || slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("the other module's send was not applied at once beside the wait: %v", got)
}
if o := outcomeOf(report, "web.server"); o.Action != "updated" {
t.Errorf("the web server was reported %q: %+v", o.Action, o)
}
}
// A container's busy is held to its shape where it arrives.
func TestBusyIsReadAndHeldToItsShape(t *testing.T) {
d := twoContainers(t, "1", "1")
c := d.Resources[0].(*declaration.Container)
if c.Busy == nil || c.Busy.Command != "mesh-builder busy" || c.Busy.Within() != 5*time.Second {
t.Fatalf("busy was not read: %+v", c.Busy)
}
for name, raw := range map[string]string{
"no command": `"busy":{"command":" "}`,
"a long timeout": `"busy":{"command":"x","timeout":"5m"}`,
"not a duration": `"busy":{"command":"x","timeout":"soon"}`,
"an unknown key": `"busy":{"command":"x","every":"1m"}`,
"on a step": `"run-once":true,"busy":{"command":"x"}`,
} {
_, err := declaration.Parse([]byte(`{"declaration":1,"resources":[{"id":"a.b","type":"container",` +
`"name":"b","image":"` + pinned + `",` + raw + `}]}`))
if err == nil {
t.Errorf("%s: a busy of that shape was accepted", name)
}
}
}
// A replacement owed to a restart-on trigger is still owed after the wait: a trigger that is no file or
// directory (a service, an identity's or a library's change) is in no container's spec and counts as changed only
// in the pass it changed, so the container must not read unchanged at the next pass and drop from the report as if
// replaced (novox/hq ADR 0335).
func TestAReplacementOwedToARestartTriggerIsMadeOnceIdle(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
d := parse(t, `{"declaration":1,"resources":[
{"id":"build-agent.server","type":"container","name":"mesh-build-agent","image":"`+pinned+`",
"restart-on":["identity:build-agent"],"busy":{"command":"mesh-builder busy"}}
]}`)
r := d.Resources[0].(*declaration.Container)
in := inputs{declares: map[string]string{}}
m := newBusyMachine()
m.containers["mesh-build-agent"] = containerSpec(r, in)
ctx := context.Background()
// The trigger moved this pass, and the agent holds a build: left running.
m.answer(true, "building")
out, err := applyContainer(ctx, r, m.run, map[string]bool{"identity:build-agent": true}, in, store.Applied{})
if err != nil || out.Action != waitsBusy || out.busySince == nil {
t.Fatalf("a busy container was not left running for its trigger: %+v %v", out, err)
}
record := store.Applied{ID: r.ID, Target: r.Name, BusySince: out.busySince}
// The next pass: nothing triggers now, and the replacement is still owed.
now = now.Add(time.Minute)
m.answer(false, "idle")
mark := m.mark()
out, err = applyContainer(ctx, r, m.run, map[string]bool{}, in, record)
if err != nil {
t.Fatal(err)
}
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("the replacement owed to the trigger was dropped: %v (%+v)", got, out)
}
if out.busy == nil || out.busy.State != BusyReplacedIdle || out.busySince != nil {
t.Fatalf("the replacement does not say it was made once idle: %+v", out)
}
}
// A wait kept from a clock that later went back is never longer than the bound from now.
func TestAWaitFromAClockThatWentBackIsClampedToNow(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
now = now.Add(-time.Hour) // the clock went back an hour
report, _ := applyBusy(t, m, twoContainers(t, "2", "1"), state)
b, _ := busyOf(report, "build-agent.server")
if b.State != BusyWaiting || b.Since.After(now) || !b.Until.Equal(now.Add(15*time.Minute)) {
t.Fatalf("a wait from a later clock is not clamped to now: %+v (now %s)", b, now)
}
}
// An ask that fails — no shell in the image, a timeout, a paused container — is not busy: replaced at once.
func TestAnAskThatFailsIsIdle(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
failing := func(ctx context.Context, name string, args ...string) (string, error) {
if args[0] == "exec" {
m.mu.Lock()
m.calls = append(m.calls, "ask "+args[1])
m.mu.Unlock()
return "", errors.New("OCI runtime exec failed: exec: \"/bin/sh\": no such file or directory")
}
return m.run(ctx, name, args...)
}
mark := m.mark()
report, _, err := Apply(context.Background(), archHost(t), twoContainers(t, "2", "1"), state,
store.OriginDeclared, failing, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("an ask that failed held the container: %v", got)
}
if len(report.Busy) != 0 {
t.Errorf("an ask that failed is said as a wait: %+v", report.Busy)
}
}
// A stopped container holds no work: it is replaced without being asked.
func TestAStoppedContainerIsNotAsked(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "would say busy")
stopped := func(ctx context.Context, name string, args ...string) (string, error) {
if args[0] == "container" && args[len(args)-1] == "mesh-build-agent" {
out, err := m.run(ctx, name, args...)
return strings.Replace(out, "true\t", "false\t", 1), err
}
return m.run(ctx, name, args...)
}
mark := m.mark()
if _, _, err := Apply(context.Background(), archHost(t), twoContainers(t, "2", "1"), state,
store.OriginDeclared, stopped, nil, nil); err != nil && !strings.Contains(err.Error(), "not running") {
t.Fatal(err)
}
got := m.since(mark)
if slices.Contains(got, "ask mesh-build-agent") || !slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("a stopped container was asked, or not replaced: %v", got)
}
}
+54
View File
@@ -0,0 +1,54 @@
package apply
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// Issue 1637 (novox/hq ADR 0480 clause 7): a container's supplementary groups reach the runtime as --group-add.
func groupsContainer(groups ...int64) *declaration.Container {
return &declaration.Container{ID: "n8n.app", Name: "n8n", Image: pinned, Groups: groups}
}
// 1637/AC1: both run commands carry one --group-add per group, and none without groups.
func TestAContainerIsAddedItsSupplementaryGroups(t *testing.T) {
with := groupsContainer(2000, 2001)
for name, args := range map[string][]string{
"detached": detachedRunArgs(with, "x"),
"foreground": foregroundRunArgs(with, "x"),
} {
got := " " + strings.Join(args, " ") + " "
if !strings.Contains(got, " --group-add 2000 ") || !strings.Contains(got, " --group-add 2001 ") {
t.Errorf("1637/AC1: the %s run does not add both groups: %s", name, got)
}
}
without := groupsContainer()
for name, args := range map[string][]string{
"detached": detachedRunArgs(without, "x"),
"foreground": foregroundRunArgs(without, "x"),
} {
if strings.Contains(strings.Join(args, " "), "--group-add") {
t.Errorf("1637/AC1: the %s run adds a group none was declared", name)
}
}
}
// 1637/AC2: a changed set of groups recreates the container; no groups leaves the spec as it was.
func TestAChangeOfGroupsRecreatesTheContainer(t *testing.T) {
none, a, b := groupsContainer(), groupsContainer(2000), groupsContainer(2000, 2001)
if containerSpec(none, inputs{}) == containerSpec(a, inputs{}) || containerSpec(a, inputs{}) == containerSpec(b, inputs{}) {
t.Error("1637/AC2: a change of groups does not change the container's spec")
}
same := groupsContainer(2000)
if containerSpec(a, inputs{}) != containerSpec(same, inputs{}) {
t.Error("1637/AC2: the same groups changed the spec")
}
plain := *none
plain.Groups = nil
if containerSpec(none, inputs{}) != containerSpec(&plain, inputs{}) {
t.Error("1637/AC2: a container with no groups has its spec moved")
}
}
@@ -6,7 +6,7 @@ import (
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
@@ -16,9 +16,13 @@ func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
var ran []string
connectFails := false
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("not installed")
}
if len(args) > 0 && args[0] == "container" && strings.HasSuffix(args[len(args)-1], asideSuffix) {
return "", errors.New("no such container") // nothing was left set aside
}
ran = append(ran, strings.Join(args, " "))
switch args[0] {
case "info":
@@ -74,71 +78,10 @@ func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
var specOfLast string
// A module the mesh left out of a declaration is not a module the mesh removed (novox/hq ADR 0163,
// rule 6): what the host wrote for it stays, recorded and said; what it holds for it stays held.
// A module simply absent is removed as it always was.
func TestALeftOutModuleIsNeitherRemovedNorForgotten(t *testing.T) {
var removed []string
gone := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", nil
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "rm":
removed = append(removed, args[len(args)-1])
gone[args[len(args)-1]] = true
case "container":
if gone[args[len(args)-1]] {
return "", errors.New("no such container")
}
return "true\tspec", nil
}
return "", nil
}
known := store.State{
Resources: []store.Applied{
{ID: "web.server", Type: "container", Target: "web", Origin: store.OriginDeclared},
{ID: "old.server", Type: "container", Target: "old", Origin: store.OriginDeclared},
},
Held: []store.Held{{ID: "web.page", Module: "web", Kind: "file", Target: "/srv/web/index.html"}},
}
d := parse(t, `{"declaration":1,"left_out":["web"],"resources":[
{"id":"notes.conf","type":"file","path":"`+t.TempDir()+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if len(removed) != 1 || removed[0] != "old" {
t.Fatalf("removed %v; only the module that is absent goes", removed)
}
if _, kept := state.At("container", "web"); !kept {
t.Fatal("the left-out module's record was forgotten")
}
if _, held := state.HeldAt("web.page"); !held {
t.Fatal("the left-out module's hold was released")
}
said := false
for _, o := range report.Outcomes {
if o.ID == "web.server" && o.Action == "unchanged" && strings.Contains(o.Detail, "web was left out of this declaration by the mesh") {
said = true
}
if o.ID == "web.server" && o.Action != "unchanged" {
t.Errorf("the left-out module's container was %s", o.Action)
}
}
if !said {
t.Fatalf("keeping the left-out module's container was not said: %+v", report.Outcomes)
}
}
// A container's capabilities reach the runtime and are part of its spec (novox/hq ADR 0170).
func TestACapabilityReachesTheRuntimeAndTheSpec(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", errors.New("not installed")
}
@@ -180,6 +123,7 @@ func TestAPackageDeclaredAbsentIsRemovedWhenPresentAndLeftWhenNot(t *testing.T)
installed := true
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
ran = append(ran, name+" "+strings.Join(args, " "))
if name != "pacman" {
return "", nil
+158
View File
@@ -0,0 +1,158 @@
package apply
import (
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// A module that declares its own system account and a .ssh in that account's home is placed (novox/hq issue 1542):
// the account is not in the user database yet when the declaration is judged, and its home is under /var/lib.
func declaredHomeFixture(t *testing.T) (home string, user *declaration.User) {
t.Helper()
dir := t.TempDir()
passwd := filepath.Join(dir, "passwd")
if err := os.WriteFile(passwd, []byte("root:x:0:0::/root:/bin/sh\n"), 0o644); err != nil {
t.Fatal(err)
}
before := passwdFile
passwdFile = passwd
t.Cleanup(func() { passwdFile = before })
home = filepath.Join(dir, "lib", "mesh-build-go")
return home, &declaration.User{ID: "mesh-build-go.account", Type: declaration.TypeUser, Name: "mesh-build-go",
Shell: "/usr/bin/nologin", Home: home}
}
// 1542/AC1: the .ssh of a declared account, owned by it, is refused by nothing.
func TestIssue1542ADeclaredAccountsSSHIsPlaced(t *testing.T) {
home, user := declaredHomeFixture(t)
ssh := &declaration.Directory{ID: "mesh-build-go.ssh-dir", Type: declaration.TypeDirectory, Path: home + "/.ssh",
Mode: "0700", Owner: "mesh-build-go"}
refused := refusedPlaces([]declaration.Resource{user, ssh})
if len(refused) != 0 {
t.Fatalf("a declared account's own .ssh was refused: %v", refused)
}
if err := (inputs{refused: refused}).refused[filepath.Clean(ssh.Path)]; err != nil {
t.Fatalf("the apply-time check refuses it: %v", err)
}
if line := JudgeDirectories(&declaration.Declaration{Resources: []declaration.Resource{user, ssh}}); strings.Contains(line, "refused already by the placement guard: ") {
t.Fatalf("the read-only judgement refuses it: %s", line)
}
}
// 1542/AC2: another account's directory below the declared home is refused as below any home, and a .ssh below
// no home of the machine or the declaration is refused with today's words.
func TestIssue1542OnlyTheDeclaredAccountsOwnIsPlaced(t *testing.T) {
home, user := declaredHomeFixture(t)
other := &declaration.Directory{ID: "notes.ssh-dir", Type: declaration.TypeDirectory, Path: home + "/.ssh",
Mode: "0700", Owner: "intruder"}
refused := refusedPlaces([]declaration.Resource{user, other})
err := refused[filepath.Clean(other.Path)]
if err == nil || !strings.Contains(err.Error(), "below mesh-build-go's home and declared intruder's") {
t.Fatalf("another account's directory below a declared home: %v", err)
}
stray := &declaration.Directory{ID: "mesh-build-go.ssh-dir", Type: declaration.TypeDirectory,
Path: filepath.Join(filepath.Dir(home), "elsewhere", ".ssh"), Mode: "0700", Owner: "mesh-build-go"}
refused = refusedPlaces([]declaration.Resource{user, stray})
err = refused[filepath.Clean(stray.Path)]
if err == nil || !strings.Contains(err.Error(), "a .ssh directory is placed only below its account's home, as that account's") {
t.Fatalf("a .ssh below no home: %v", err)
}
// Not a .ssh: a root-owned directory below the declared home is placed exactly as on main.
data := &declaration.Directory{ID: "mesh-build-go.data", Type: declaration.TypeDirectory, Path: home + "/data",
Mode: "0755", Owner: "root"}
if refused := refusedPlaces([]declaration.Resource{user, data}); len(refused) != 0 {
t.Fatalf("a root-owned directory below a declared home, not a .ssh, was refused: %v", refused)
}
// And without the account declared, the same .ssh is refused as it was.
own := &declaration.Directory{ID: "mesh-build-go.ssh-dir", Type: declaration.TypeDirectory, Path: home + "/.ssh",
Mode: "0700", Owner: "mesh-build-go"}
if refusedPlaces([]declaration.Resource{own})[filepath.Clean(own.Path)] == nil {
t.Fatal("a .ssh below an account nobody makes was placed")
}
}
// 1542/AC3: a declared account is no home to anyone but the directory placement guard.
func TestIssue1542DeclaredAccountsAreNotHomesToTheOtherGuards(t *testing.T) {
home, user := declaredHomeFixture(t)
_ = refusedPlaces([]declaration.Resource{user})
if _, ok := accountsOfHomes()[home]; ok {
t.Fatal("a declared account became a home of the machine")
}
if homeAbove(home+"/.ssh") != "" {
t.Fatal("a declared account became a home for the links and the move-out")
}
}
// A declared account is added as a home and never takes one over (reviewer, issue 1542): the person's own .ssh
// stays refused to every other account, whatever a declaration says about a user.
func TestIssue1542ADeclaredAccountTakesNoExistingHome(t *testing.T) {
dir := t.TempDir()
passwd := filepath.Join(dir, "passwd")
body := "root:x:0:0::/root:/bin/sh\njochen:x:1000:1000::/home/jochen:/bin/sh\n" +
"svc:x:998:998::/var/lib/svc:/usr/bin/nologin\n"
if err := os.WriteFile(passwd, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
before := passwdFile
passwdFile = passwd
t.Cleanup(func() { passwdFile = before })
newHome := filepath.Join(dir, "lib", "mesh-build-go")
ssh := func(path, owner string) *declaration.Directory {
return &declaration.Directory{ID: "x.ssh", Type: declaration.TypeDirectory, Path: path, Mode: "0700", Owner: owner}
}
user := func(name, home string) *declaration.User {
return &declaration.User{ID: "x.user", Type: declaration.TypeUser, Name: name, Home: home}
}
// A declared user over a person's home, whoever it is named for.
for _, name := range []string{"evil", "postgres", "root"} {
d := ssh("/home/jochen/.ssh", name)
if refusedPlaces([]declaration.Resource{user(name, "/home/jochen"), d})[d.Path] == nil {
t.Errorf("%s, declared over jochen's home, was given jochen's .ssh", name)
}
}
// A declared home below a person's home.
d := ssh("/home/jochen/sub/.ssh", "evil")
if refusedPlaces([]declaration.Resource{user("evil", "/home/jochen/sub"), d})[d.Path] == nil {
t.Error("a home declared below jochen's home was counted")
}
// A name the machine has already, declared with another home: that is not its home.
d = ssh(newHome+"/e/.ssh", "svc")
if refusedPlaces([]declaration.Resource{user("svc", newHome+"/e"), d})[d.Path] == nil {
t.Error("an existing account's declared home was counted")
}
// A number is no name: chown -1 leaves the owner as it was.
for _, owner := range []string{"-1", "-1:0", "0", "1001"} {
d := ssh(newHome+"/.ssh", owner)
if refusedPlaces([]declaration.Resource{user("mesh-build-go", newHome), d})[d.Path] == nil {
t.Errorf("owner %q was placed below a declared home", owner)
}
}
// And the account that is new, by name, still is.
d = ssh(newHome+"/.ssh", "mesh-build-go")
if got := refusedPlaces([]declaration.Resource{user("mesh-build-go", newHome), d}); len(got) != 0 {
t.Errorf("a new account's own .ssh was refused: %v", got)
}
}
// The account the mesh made, as declared, is judged the same at the next apply (issue 1542): once it is in the
// user database with exactly the declared home, its .ssh is still placed.
func TestIssue1542TheAccountTheMeshMadeIsStillPlacedAtTheNextApply(t *testing.T) {
home, user := declaredHomeFixture(t)
ssh := &declaration.Directory{ID: "mesh-build-go.ssh-dir", Type: declaration.TypeDirectory, Path: home + "/.ssh",
Mode: "0700", Owner: "mesh-build-go"}
resources := []declaration.Resource{user, ssh}
if got := refusedPlaces(resources); len(got) != 0 {
t.Fatalf("first apply, the account not yet made: %v", got)
}
body := "root:x:0:0::/root:/bin/sh\nmesh-build-go:x:998:998::" + home + ":/usr/bin/nologin\n"
if err := os.WriteFile(passwdFile, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
if got := refusedPlaces(resources); len(got) != 0 {
t.Fatalf("second apply, the account made as declared: %v", got)
}
}
+19
View File
@@ -0,0 +1,19 @@
package apply
import (
"context"
"strings"
"testing"
"time"
)
// A command killed because its context ended says so, and why: "docker exited -1:" said nothing while novox's bus
// container was half replaced (issue 1945, live on 2026-10-25 at 11:23).
func TestACommandKilledByItsContextSaysWhy(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
go func() { time.Sleep(50 * time.Millisecond); cancel() }()
_, err := ExecRunner(ctx, "sleep", "5")
if err == nil || !strings.Contains(err.Error(), "killed") || !strings.Contains(err.Error(), "context canceled") {
t.Fatalf("a command its context killed said: %v", err)
}
}
+1
View File
@@ -43,6 +43,7 @@ func watching(t *testing.T, path string) Runner {
var commands []string
services := recordingServices(&commands)
return func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if _, err := os.Stat(path); err != nil {
t.Errorf("the file was not there while %s %s ran: %v", name, strings.Join(args, " "), err)
}
+235
View File
@@ -0,0 +1,235 @@
package apply
import (
"context"
"fmt"
"slices"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// WriteFilled writes what the provisioner's client filled into one file the declaration names as filled (novox/hq
// ADR 0349 §4), and makes what reads the file take the change, exactly as an apply that changed it would.
//
// **The declared file says everything but the content.** Its path, mode, owner and who owns it at runtime are the
// declaration's; the client's request carries the content alone. So the file is applied as the declared one with
// that content in place of the declaration's: an owned file whole through applyFile, a seed only when absent, a
// managed file's keys through applyInto or its block through applyBlock — the same guards on where nothing is
// placed, the same Keep of an original found there, the same record. What the declaration seals or fills with
// `${secret:}` the client has already filled, so the content is written as it came; a file that carried a secret
// keeps the mode a secret is written with when the declaration gives none.
//
// **Then what reads it, in the apply's order** (afterWhatTheyRead): a service naming it under restart-on is
// restarted and under reload-on reloaded, a process or container naming it under restart-on is restarted, and a
// running container that reads it at creation — an env-file, a file mounted into it — is recreated, because what
// it read is recorded by digest and this write changed the digest. Each of those is applied and recorded as an
// apply does, and what moves goes on moving what names it. Nothing else of the declaration is touched.
//
// The caller holds the apply's lock: a write and an apply never write one file at once. log is said the path, the
// action and the digest, never the content.
//
// **It does not refresh the runtimes' data roots.** A filled write is one file, not an apply; the placement guard
// judges it with the roots the last apply read (runtimeDataRoots), falling back to the runtimes' configuration
// files when no apply has read them yet — the runtimes themselves are asked once, at an apply's start, not once
// per file (novox/hq issue 1138).
func WriteFilled(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
known store.State,
path string,
content []byte,
run Runner,
log func(string),
keep Keep,
windows *Windows,
) (report Report, _ store.State, _ error) {
if log == nil {
log = func(string) {}
}
if !d.Fills(path) {
return report, known, &Error{Resource: path, Err: fmt.Errorf("this node's declaration does not name %s "+
"as a file the provisioner fills, so nothing was written", path)}
}
var declared []*declaration.File
for _, r := range d.Resources {
if f, ok := r.(*declaration.File); ok && f.Path == path {
declared = append(declared, f)
}
}
if len(declared) != 1 {
// The parser refuses a filled path no file declares; two at one path would need the request to say whose
// keys or block it carries, and it carries the content only.
return report, known, &Error{Resource: path, Err: fmt.Errorf("%d resources declare %s, and a filled file "+
"is written for exactly one, so nothing was written", len(declared), path)}
}
file := filledAs(declared[0], content)
if held, isHeld := known.HeldAt(file.ID); isHeld {
return report, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is held as found for a module not "+
"yet taken (%s), so nothing was written", path, held.Why)}
}
in := inputs{declares: map[string]string{}, known: &known, windows: windows, away: newStoreAway(windows, log),
groups: groupsWanted(d.Resources), handed: Handed{}}
in.refused = refusedPlaces(d.Resources)
in.agentHomes = rootNeverHomes(d)
in.dirsBefore = map[string]bool{}
in.files = declaredFiles(d.Resources)
// A container that reads the file and mounts a library whose disk or mount is not there is held, as a full apply
// holds it: not created over the missing disk (novox/hq ADR 0328 §9).
in.held = heldLibraries(ctx, d.Libraries)
for _, r := range d.Resources {
in.declares[r.Identity()] = declaredDigest(r)
}
changed := map[string]bool{}
// The file, as the apply loop applies it.
was, _ := known.Find(file.ID)
var keepFound Keep
if (was.ID == "" || was.Target != file.Path) && !known.Recorded(string(declaration.TypeFile), file.Path) {
keepFound = keep
}
outcome, err := applyOne(ctx, sys, file, run, changed, in, was, nil, keepFound)
if err != nil {
log(fmt.Sprintf(" failed %s (%s): %v", file.ID, file.Path, err))
return report, known, &Error{Resource: file.ID, Err: err, Done: report}
}
recordKeeping(&known, file, outcome, was)
report.Outcomes = append(report.Outcomes, outcome)
if !moved(outcome) {
return report, known, nil
}
changed[file.ID] = true
line := fmt.Sprintf(" %s %s (%s), filled: %s", outcome.Action, file.ID, file.Path, outcome.wrote)
if outcome.Detail != "" {
line += ": " + outcome.Detail
}
log(line)
// What reads it, in the apply's order, each moving what names it in turn.
var failures []*Error
for _, r := range afterWhatTheyRead(d.Resources) {
if r.Identity() == file.ID || !readsChange(r, path, changed) {
continue
}
if _, isHeld := known.HeldAt(r.Identity()); isHeld {
log(fmt.Sprintf(" held %s (%s): held as found for a module not yet taken, so it was not restarted "+
"for %s", r.Identity(), r.Target(), file.ID))
continue
}
was, _ := known.Find(r.Identity())
o, err := applyOne(ctx, sys, r, run, changed, in, was, nil, nil)
if err != nil {
if o.found != nil {
known.KeepFound(r.Identity(), store.OriginDeclared, *o.found)
}
failures = append(failures, &Error{Resource: r.Identity(), Err: err})
log(fmt.Sprintf(" failed %s (%s): %v", r.Identity(), o.Target, err))
continue
}
if o.Action == "waiting" {
if o.found != nil {
known.KeepFound(r.Identity(), store.OriginDeclared, *o.found)
}
report.Outcomes = append(report.Outcomes, o)
log(fmt.Sprintf(" waiting %s (%s): %s", o.ID, o.Target, o.Detail))
continue
}
recordKeeping(&known, r, o, was)
report.Outcomes = append(report.Outcomes, o)
if o.busy != nil {
report.Busy = append(report.Busy, *o.busy)
}
if moved(o) {
changed[r.Identity()] = true
line := fmt.Sprintf(" %s %s (%s)", o.Action, o.ID, o.Target)
if o.Detail != "" {
line += ": " + o.Detail
}
log(line)
} else if o.Action != "unchanged" {
log(fmt.Sprintf(" %s %s (%s): %s", o.Action, o.ID, o.Target, o.Detail))
}
}
if len(failures) > 0 {
first := failures[0]
first.Done = report
first.Others = len(failures) - 1
return report, known, first
}
return report, known, nil
}
// filledAs is the declared file with the filled content in place of the declaration's.
func filledAs(declared *declaration.File, content []byte) *declaration.File {
f := *declared
f.Content, f.Sealed, f.VaultSealed, f.Bytes, f.Secrets = string(content), "", nil, "", nil
if f.Mode == "" && (declared.Secret() || len(declared.Secrets) > 0) {
// Written as the declaration would have written it: a file carrying a credential is not world-readable.
f.Mode = "0600"
}
return &f
}
// readsChange reports whether applying r is what makes it take a change of what changed: a service's restart-on or
// reload-on, a process's or container's restart-on, or a running container that reads path at creation.
func readsChange(r declaration.Resource, path string, changed map[string]bool) bool {
switch res := r.(type) {
case *declaration.Service:
return len(restartedBy(res.RestartOn, changed)) > 0 || len(restartedBy(res.ReloadOn, changed)) > 0
case *declaration.Process:
return len(restartedBy(res.RestartOn, changed)) > 0
case *declaration.Container:
if len(restartedBy(res.RestartOn, changed)) > 0 {
return true
}
if res.RunOnce || res.Schedule != "" {
return false
}
if slices.Contains(res.EnvFile, path) {
return true
}
for _, v := range res.Volumes {
if mountSource(v) == path {
return true
}
}
}
return false
}
// recordKeeping records r as applied, carrying where the original it replaced was kept from its record when this
// write kept none — for as long as the resource is recorded at the same target, as the apply loop carries it.
func recordKeeping(known *store.State, r declaration.Resource, o Outcome, was store.Applied) {
kept, keptMode, keptOwner := o.kept, o.keptMode, o.keptOwner
if kept == "" && was.Target == o.Target {
kept, keptMode, keptOwner = was.Kept, was.KeptMode, was.KeptOwner
}
recordApplied(known, store.OriginDeclared, r, o, kept, keptMode, keptOwner)
}
// leftToTheClient is a filled file as an apply leaves it (novox/hq ADR 0349 §5, design 57 P7): untouched, and said
// so. A file the mesh wrote at the same path under another id, going in this apply, is handed to it as an apply hands
// a whole file (novox/hq ADR 0223): its record — what the mesh wrote there, and the original it kept — is the filled
// file's now, so the verb judges what is there against it, and the file is neither removed nor written.
func leftToTheClient(f *declaration.File, known *store.State, handedFiles map[string]store.Applied, report *Report,
log func(string)) Outcome {
if from, handed := handedFiles[f.ID]; handed {
if _, recorded := known.Find(f.ID); !recorded {
rec := from
rec.ID = f.ID
known.Record(rec)
}
delete(handedFiles, f.ID)
known.Forget(from.ID)
detail := "no longer declared; " + f.ID + " declares the file now, filled by the provisioner's client, so " +
"it was handed to it, not removed"
report.Outcomes = append(report.Outcomes, Outcome{ID: from.ID, Type: from.Type, Target: from.Target,
Action: "forgotten", Detail: detail})
log(fmt.Sprintf(" forgotten %s (%s): %s", from.ID, from.Target, detail))
}
return Outcome{ID: f.ID, Type: string(declaration.TypeFile), Target: f.Path, Action: "unchanged",
Detail: "filled by the provisioner's client, which alone has it written"}
}
+218
View File
@@ -0,0 +1,218 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §5, one writer per file (design 57 P7, issue 673): a file the declaration names as
// filled is written only through the write verb (WriteFilled), on the provisioner client's word. An apply — the
// controller's send and the five-minute re-apply alike — never writes it, never records it, and removes it as an
// orphan only when the verb placed it.
// An apply leaves a filled file alone, whatever who owns it at runtime says: absent stays absent and is not
// recorded, and what the verb wrote stays as it wrote it through every re-apply.
func TestAnApplyNeverWritesAFilledFile(t *testing.T) {
for _, c := range []struct{ name, fields, content string }{
{"owned", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n"},
{"seed", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n"},
{"managed block", `"into":"block","content":"","owner-at-runtime":"managed"`, "10.42.0.1 ace\n"},
} {
t.Run(c.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
d := filledDecl(t, path, c.fields, "")
report, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); err == nil {
t.Fatalf("an apply wrote a filled file the client has not filled: %q", readText(t, path))
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("an apply recorded a filled file it did not write")
}
if o := outcomeOf(report, "m.conf"); o.Action != "unchanged" || !strings.Contains(o.Detail, "provisioner") {
t.Errorf("an apply did not say it left the file to the provisioner's client: %+v", o)
}
_, state, err = WriteFilled(context.Background(), archHost(t), d, state, path, []byte(c.content),
noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
placed, _ := state.Find("m.conf")
want, wantMode := readText(t, path), modeAt(t, path)
// The re-apply, twice over: nothing written, the verb's record kept as it wrote it.
for i := 0; i < 2; i++ {
var said []string
report, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared,
noServices, func(line string) { said = append(said, line) }, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != want || modeAt(t, path) != wantMode {
t.Fatalf("re-apply %d wrote the filled file again: %q, want %q", i+1, got, want)
}
if o := outcomeOf(report, "m.conf"); o.Action != "unchanged" {
t.Errorf("re-apply %d: %+v", i+1, o)
}
if rec, _ := state.Find("m.conf"); rec.Wrote != placed.Wrote || rec.Target != path {
t.Errorf("re-apply %d changed the verb's record: %+v, was %+v", i+1, rec, placed)
}
if len(said) != 0 {
t.Errorf("re-apply %d said something of an untouched file: %q", i+1, said)
}
}
})
}
}
// What reads a filled file is moved by the verb's write, once, never again by an apply: a service under
// restart-on is not restarted and a container reading it is not recreated by the re-apply.
func TestAnApplyMovesNothingForAFilledFile(t *testing.T) {
env := filepath.Join(t.TempDir(), "forge.env")
d := filledDecl(t, env, `"content":"","mode":"0600","owner-at-runtime":"owned"`,
`{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}`)
m := &machine{containers: map[string]*fakeContainer{}}
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, env, []byte("PORT=5432\n"),
m.run, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
m.asked = nil
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, m.run, nil,
nil); err != nil {
t.Fatal(err)
}
if m.removed("forge") || m.did("docker run") {
t.Errorf("the re-apply recreated the container reading the filled file: %v", m.asked)
}
if got := readText(t, env); got != "PORT=5432\n" {
t.Errorf("the re-apply wrote the filled file: %q", got)
}
conf := filepath.Join(t.TempDir(), "daemon.conf")
d = filledDecl(t, conf, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","restart-on":["m.conf"]}`)
var commands []string
_, state, err = WriteFilled(context.Background(), archHost(t), d, store.State{}, conf, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil)
if err != nil {
t.Fatal(err)
}
commands = nil
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared,
recordingServices(&commands), nil, nil); err != nil {
t.Fatal(err)
}
if n := count(commands, "systemctl stop m.service"); n != 0 {
t.Errorf("the re-apply restarted what reads the filled file: %v", commands)
}
}
// A filled file the verb never placed is not the mesh's to remove: one found there, or put there by hand, stays
// when the file leaves the declaration.
func TestAFilledFileTheVerbNeverPlacedIsNotRemoved(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
if err := os.WriteFile(path, []byte("the machine's own\n"), 0o644); err != nil {
t.Fatal(err)
}
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
gone := elsewhere(t)
report, state, err := Apply(context.Background(), archHost(t), gone, state, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != "the machine's own\n" {
t.Errorf("a filled file the verb never placed was touched when it left: %q", got)
}
if o := outcomeOf(report, "m.conf"); o.Action == "removed" {
t.Errorf("a filled file the verb never placed was removed: %+v", o)
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("a record of the filled file was left")
}
}
// What the verb placed is the mesh's: when the file leaves the declaration it is removed as an orphan, as a file
// an apply wrote is.
func TestAFilledFileTheVerbPlacedIsRemovedWhenItLeaves(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
_, state, err = WriteFilled(context.Background(), archHost(t), d, state, path, []byte("port = 8080\n"),
noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
gone := elsewhere(t)
report, state, err := Apply(context.Background(), archHost(t), gone, state, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); err == nil {
t.Errorf("a filled file the verb placed stayed when it left the declaration: %q", readText(t, path))
}
if o := outcomeOf(report, "m.conf"); o.Action != "removed" {
t.Errorf("not removed: %+v", o)
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("its record stayed")
}
}
// A file the mesh wrote the old way and that is now filled keeps its record for the verb to judge against, and is
// not written over by the apply that hands it to the client; a file at the same path recorded under another id
// is handed to the filled one, not removed.
func TestAFileThatMovesToTheClientIsHandedNotRewritten(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
old := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"m.old","type":"file","path":%q,`+
`"content":"port = 1\n"}]}`, path))
_, state, err := Apply(context.Background(), archHost(t), old, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
was, _ := state.Find("m.old")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != "port = 1\n" {
t.Errorf("the file handed to the client was rewritten or removed: %q", got)
}
if _, ok := state.Find("m.old"); ok {
t.Errorf("the record going was kept beside the filled file's")
}
if rec, ok := state.Find("m.conf"); !ok || rec.Wrote != was.Wrote || rec.Target != path {
t.Errorf("the filled file did not take the record of what the mesh wrote there: %+v", rec)
}
}
// elsewhere is a declaration with nothing at the filled path: one unrelated file.
func elsewhere(t *testing.T) *declaration.Declaration {
t.Helper()
return parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"n.other","type":"file","path":%q,`+
`"content":"x\n"}]}`, filepath.Join(t.TempDir(), "other")))
}
+57
View File
@@ -0,0 +1,57 @@
//go:build linux
package apply
import (
"context"
"fmt"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that reads a filled file and mounts a library whose disk is not there is held by the write verb as a
// full apply holds it (novox/hq ADR 0328 §9, issue 668): never created or recreated over the missing disk.
func TestAWriteDoesNotRecreateAContainerHeldForItsLibrary(t *testing.T) {
// No folder is given the group: the library's root is absent, and nothing is made.
gid := 61000
libraries(t, nil, nil)
base := t.TempDir()
root := filepath.Join(base, "gone", "media")
env := filepath.Join(base, "radarr.env")
raw := fmt.Sprintf(`{"declaration":1,"provisioner_account":"mesh-node-provisioner","filled_files":[%q],
"groups":[{"name":"mesh-media","gid":%d}],"libraries":[%s],"resources":[
{"id":"radarr.env","type":"file","path":%q,"content":"","mode":"0600","owner-at-runtime":"owned"},
{"id":"radarr.app","type":"container","name":"radarr","image":%q,"env-file":[%q],"volumes":[%q]}]}`,
env, gid, libJSON("media", root, gid, true, false, "movies"), env, pinned, env,
filepath.Join(root, "movies")+":/movies")
d, err := declaration.Parse([]byte(raw))
if err != nil {
t.Fatal(err)
}
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
return "", nil
}
var state store.State
for _, content := range []string{"PORT=1\n", "PORT=2\n"} {
var report Report
report, state, err = WriteFilled(context.Background(), archHost(t), d, state, env, []byte(content), run,
nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "radarr.app"); o.Action != "waiting" {
t.Errorf("after writing %q the container is %+v, not held for its library", content, o)
}
}
for _, line := range ran {
if strings.HasPrefix(line, "docker run") || strings.HasPrefix(line, "docker rm") {
t.Errorf("the container was created or removed over a missing disk: %v", ran)
}
}
}
+168
View File
@@ -0,0 +1,168 @@
package apply
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A filled write is one file, not an apply: it asks no container runtime, and judges where nothing is placed
// (issue 339) with the roots the last apply read, falling back to the runtimes' configuration files when no apply
// has read them yet (novox/hq issue 1138). These tests set the package's runtimeRoots and AskRuntimes, so none
// runs in parallel with another that reads or sets them, and each restores what it changed.
// setRuntimeRoots makes runtimeDataRoots answer roots, as if an apply had just read them (or, with nil, as if none
// ever had), restoring what a test before it left there once this one is done.
func setRuntimeRoots(t *testing.T, roots []string) {
t.Helper()
runtimeRootsMu.Lock()
before := runtimeRoots
runtimeRoots = roots
runtimeRootsMu.Unlock()
t.Cleanup(func() {
runtimeRootsMu.Lock()
runtimeRoots = before
runtimeRootsMu.Unlock()
})
}
// askNoRuntime makes AskRuntimes record the name of every runtime it is asked for, answering as one that is not
// running does (nothing, no error): were it consulted, that is what would say so. Restored once the test is done.
func askNoRuntime(t *testing.T, asked *[]string) {
t.Helper()
before := AskRuntimes
AskRuntimes = func(_ context.Context, name string, _ ...string) (string, error) {
*asked = append(*asked, name)
return "", nil
}
t.Cleanup(func() { AskRuntimes = before })
}
// aRuntimeConfig points the runtimes' configuration at a fresh, unmounted root below a temp directory, restoring
// the engine's real configuration once the test is done.
func aRuntimeConfig(t *testing.T) (dir, root string) {
t.Helper()
dir = t.TempDir()
root = filepath.Join(dir, "docker")
daemon := filepath.Join(dir, "daemon.json")
if err := os.WriteFile(daemon, []byte(`{"data-root":"`+root+`"}`), 0o644); err != nil {
t.Fatal(err)
}
before := runtimeConfigs
runtimeConfigs = runtimeFiles{daemonJSON: daemon, storageConf: filepath.Join(dir, "none.conf")}
t.Cleanup(func() { runtimeConfigs = before })
return dir, root
}
// refusedAsPlacement fails the test unless err is a PlacementRefusedError.
func refusedAsPlacement(t *testing.T, err error) {
t.Helper()
var refusal *PlacementRefusedError
if err == nil || !errors.As(err, &refusal) {
t.Fatalf("a filled file below a refused runtime root was not refused: %v", err)
}
}
// 1138/AC1: a filled write asks no runtime, and judges with the roots the last apply read.
func TestAFilledWriteAsksNoRuntimeAndUsesTheLastApplysRoots(t *testing.T) {
root := filepath.Join(t.TempDir(), "docker-root")
setRuntimeRoots(t, []string{root}) // as if an apply had just read it
var asked []string
askNoRuntime(t, &asked)
path := filepath.Join(root, "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
if len(asked) != 0 {
t.Errorf("a filled write asked the container runtimes: %v", asked)
}
}
// 1138/AC2: given a real apply that read a runtime data root, a filled write below it is refused as today, with
// no apply run again in between.
func TestAFilledWriteBelowARootARealApplyReadIsRefused(t *testing.T) {
dir, root := aRuntimeConfig(t)
t.Cleanup(func() {
runtimeRootsMu.Lock()
runtimeRoots = nil
runtimeRootsMu.Unlock()
})
// A full apply of something unrelated reads the runtimes' configuration once, at its start.
elsewhere := filepath.Join(dir, "elsewhere.conf")
if _, _, err := Apply(context.Background(), archHost(t), declare(t,
fmt.Sprintf(`{"id":"m.other","type":"file","path":%q,"content":"x\n"}`, elsewhere)),
store.State{}, store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
// A filled write below the runtime's root, with no apply run since, is still refused.
path := filepath.Join(root, "sub", "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
}
// 1138/AC3: with no apply having read the roots yet, a filled write judges by the runtimes' configuration files,
// asking no runtime.
func TestAFilledWriteWithNoApplyYetUsesTheRuntimesConfigFiles(t *testing.T) {
setRuntimeRoots(t, nil) // no apply has read the roots yet
_, root := aRuntimeConfig(t)
var asked []string
askNoRuntime(t, &asked)
path := filepath.Join(root, "sub", "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
if len(asked) != 0 {
t.Errorf("a filled write with no apply yet asked the container runtimes: %v", asked)
}
}
// 1138/AC4: a full apply still asks each runtime once, at its start, whatever the number of files it writes.
func TestAFullApplyAsksEachRuntimeOnceRegardlessOfFileCount(t *testing.T) {
dir := t.TempDir()
var asked []string
before := AskRuntimes
AskRuntimes = func(_ context.Context, name string, _ ...string) (string, error) {
asked = append(asked, name)
return "", errors.New("not running in this test")
}
t.Cleanup(func() {
AskRuntimes = before
runtimeRootsMu.Lock()
runtimeRoots = nil
runtimeRootsMu.Unlock()
})
var resources []string
for i := 0; i < 3; i++ {
resources = append(resources, fmt.Sprintf(`{"id":"m.f%d","type":"file","path":%q,"content":"x\n"}`, i,
filepath.Join(dir, fmt.Sprintf("f%d.conf", i))))
}
if _, _, err := Apply(context.Background(), archHost(t), declare(t, strings.Join(resources, ",")),
store.State{}, store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
if got := count(asked, "docker"); got != 1 {
t.Errorf("docker was asked %d times for one apply of 3 files; want 1: %v", got, asked)
}
if got := count(asked, "podman"); got != 1 {
t.Errorf("podman was asked %d times for one apply of 3 files; want 1: %v", got, asked)
}
}
+271
View File
@@ -0,0 +1,271 @@
package apply
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §4 (issue 668): the provisioner's client has the node-engine write a file it filled,
// and the node-engine writes it as an apply of the declared file with that content would — through applyFile,
// applyInto or applyBlock by who owns it at runtime — and makes what reads it take the change as an apply does.
// filledDecl is a declaration naming path as filled, with the file's fields after id, type and path, and more
// resources after it.
func filledDecl(t *testing.T, path, fields, more string) *declaration.Declaration {
t.Helper()
if more != "" {
more = "," + more
}
return parseTrusted(t, fmt.Sprintf(`{"declaration":1,"provisioner_account":"mesh-node-provisioner",
"filled_files":[%q],"resources":[{"id":"m.conf","type":"file","path":%q,%s}%s]}`, path, path, fields, more))
}
func TestAWriteOfAPathNotFilledIsRefused(t *testing.T) {
dir := t.TempDir()
filled := filepath.Join(dir, "filled.conf")
other := filepath.Join(dir, "other.conf")
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"filled_files":[%q],"resources":[
{"id":"m.conf","type":"file","path":%q,"content":"x\n","owner-at-runtime":"owned"},
{"id":"m.other","type":"file","path":%q,"content":"y\n"}
]}`, filled, filled, other))
for _, path := range []string{other, filepath.Join(dir, "nothing.conf")} {
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("evil\n"),
noServices, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "does not name") {
t.Errorf("%s: a write of a path the declaration does not fill was not refused: %v", path, err)
}
if _, err := os.Stat(path); err == nil && path != other {
t.Errorf("%s was written", path)
}
if len(state.Resources) != 0 {
t.Errorf("a refused write recorded something: %+v", state.Resources)
}
}
if _, err := os.Stat(other); err == nil {
t.Errorf("%s was written", other)
}
t.Run("a filled path two resources declare", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
if err := os.WriteFile(path, []byte(workstationHosts), 0o644); err != nil {
t.Fatal(err)
}
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"filled_files":[%q],"resources":[
{"id":"a.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.1 ace\n","owner-at-runtime":"managed"},
{"id":"b.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.2 g14\n","owner-at-runtime":"managed"}
]}`, path, path, path))
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("6.6.6.6 evil\n"),
noServices, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "2 resources declare") {
t.Fatalf("a path two resources declare was not refused: %v", err)
}
if got := readText(t, path); got != workstationHosts || len(state.Resources) != 0 {
t.Errorf("something was written or recorded: %q %+v", got, state.Resources)
}
})
}
// Each ownership is written by the write verb exactly as an apply of the declared file with the same content: the
// same action, the same bytes on disk, the same mode.
func TestAFilledFileIsWrittenAsItsOwnershipSays(t *testing.T) {
for _, c := range []struct {
name, fields, content, before, action string
}{
{"owned, absent", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n", "", "created"},
{"owned, over what is there", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n",
"port = 1\n", "updated"},
{"seed, absent", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n", "", "created"},
{"seed, present", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n",
"user default on\nuser app-one on\n", "kept"},
{"managed json", `"into":"json","content":"{}","owner-at-runtime":"managed"`,
`{"insecure-registries":["10.42.0.1:5000"]}`, machinesOwn, "updated"},
{"managed block", `"into":"block","content":"x\n","owner-at-runtime":"managed"`, "10.42.0.1 ace\n",
workstationHosts, "updated"},
} {
t.Run(c.name, func(t *testing.T) {
at := func() string {
path := filepath.Join(t.TempDir(), "the-file")
if c.before != "" {
if err := os.WriteFile(path, []byte(c.before), 0o644); err != nil {
t.Fatal(err)
}
}
return path
}
written := at()
report, state, err := WriteFilled(context.Background(), archHost(t), filledDecl(t, written, c.fields, ""),
store.State{}, written, []byte(c.content), noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
applied := at()
withContent := strings.Replace(c.fields, fmt.Sprintf(`"content":%q`, contentOf(t, c.fields)),
fmt.Sprintf(`"content":%q`, c.content), 1)
// The same file declared the old way: an apply writes a filled file only on the client's word (P7).
plain := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"m.conf","type":"file",`+
`"path":%q,%s}]}`, applied, withContent))
want, _, err := Apply(context.Background(), archHost(t), plain, store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != c.action || got != want.Outcomes[0].Action {
t.Errorf("written %s; an apply of the same content %s; want %s", got, want.Outcomes[0].Action, c.action)
}
if got, apply := readText(t, written), readText(t, applied); got != apply {
t.Errorf("written, the file holds\n%q\nan apply of the same content leaves\n%q", got, apply)
}
if a, b := modeAt(t, written), modeAt(t, applied); a != b {
t.Errorf("written mode %o, an apply's %o", a, b)
}
if rec, ok := state.Find("m.conf"); !ok || rec.Target != written || rec.Origin != store.OriginDeclared {
t.Errorf("the write is not recorded as the file's: %+v", rec)
}
})
}
t.Run("a filled file that carried a secret is not world-readable", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "token.env")
d := filledDecl(t, path, `"content":"TOKEN=${secret:token}\n","secrets":{"token":"c2VhbGVk"},"owner-at-runtime":"owned"`, "")
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("TOKEN=abc\n"),
noServices, nil, nil, nil); err != nil {
t.Fatal(err)
}
if m := modeAt(t, path); m != 0o600 {
t.Errorf("a filled file that carried a secret is mode %o", m)
}
})
}
func contentOf(t *testing.T, fields string) string {
t.Helper()
var holder struct {
Content string `json:"content"`
}
if err := json.Unmarshal([]byte("{"+fields+"}"), &holder); err != nil {
t.Fatal(err)
}
return holder.Content
}
func modeAt(t *testing.T, path string) os.FileMode {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
return info.Mode().Perm()
}
// A file that carried a secret keeps the mode a secret is written with: the client filled the value, and the file
// is no more readable for it.
func TestAWriteUnderReloadOnReloadsItsProgramOnce(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","reload-on":["m.conf"]},
{"id":"m.other","type":"service","unit":"other.service","state":"running"}`)
var commands []string
var said []string
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("level = 1\n"),
recordingServices(&commands), func(line string) { said = append(said, line) }, nil, nil)
if err != nil {
t.Fatal(err)
}
if n := count(commands, "systemctl reload m.service"); n != 1 {
t.Errorf("the program was reloaded %d times; commands were %v", n, commands)
}
joined := strings.Join(commands, "\n")
if strings.Contains(joined, "restart m.service") || strings.Contains(joined, "stop m.service") {
t.Errorf("the program was restarted, not reloaded: %v", commands)
}
if strings.Contains(joined, "other.service") {
t.Errorf("a service that does not read the file was touched: %v", commands)
}
if _, ok := state.Find("m.daemon"); !ok {
t.Errorf("the reloaded service was not recorded")
}
// The log says the path and the digest, never the content.
for _, line := range said {
if strings.Contains(line, "level = 1") {
t.Errorf("the content was said: %q", line)
}
}
if !strings.Contains(strings.Join(said, "\n"), digestOf("level = 1\n")) {
t.Errorf("the digest was not said: %q", said)
}
// The same content again moves nothing, and reloads nothing.
commands = nil
report, _, err := WriteFilled(context.Background(), archHost(t), d, state, path, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "unchanged" || count(commands, "reload") != 0 {
t.Errorf("an unchanged write reloaded: %v %+v", commands, report.Outcomes)
}
}
func TestAWriteUnderRestartOnRestartsItsProgram(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","restart-on":["m.conf"]}`)
var commands []string
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil); err != nil {
t.Fatal(err)
}
// A restart is a stop and a start, once.
stops, starts := count(commands, "systemctl stop m.service"), count(commands, "systemctl start m.service")
if stops != 1 || starts != 1 {
t.Errorf("the program was stopped %d and started %d times; commands were %v", stops, starts, commands)
}
}
// A running container that reads the file at creation is recreated, as an apply recreates it for a changed file.
func TestAWriteOfAFileAContainerReadsRecreatesIt(t *testing.T) {
env := filepath.Join(t.TempDir(), "forge.env")
d := filledDecl(t, env, `"content":"","mode":"0600","owner-at-runtime":"owned"`,
`{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}`)
m := &machine{containers: map[string]*fakeContainer{}}
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, env, []byte("PORT=5432\n"),
m.run, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !m.did("docker run") {
t.Fatalf("the container was not created: %v", m.asked)
}
m.asked = nil
report, _, err := WriteFilled(context.Background(), archHost(t), d, state, env, []byte("PORT=5433\n"), m.run,
nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !m.removed("forge") || !m.did("docker run") {
t.Fatalf("the container kept what it read before the write: %v", m.asked)
}
if o := outcomeOf(report, "forge.server"); o.Action != "updated" || !strings.Contains(o.Detail, env) {
t.Errorf("the recreate did not say which file changed: %+v", o)
}
}
func count(lines []string, prefix string) int {
n := 0
for _, l := range lines {
if strings.Contains(l, prefix) {
n++
}
}
return n
}
// Two resources declaring one filled path — two modules' blocks in one file — are refused: the request carries the
// content only, so it cannot say whose block it is, and nothing is written.
+1
View File
@@ -15,6 +15,7 @@ import (
// starts (ADR 0141).
func TestTheHostsOwnFormerArchiveIsLeftInPlaceNotFatal(t *testing.T) {
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "docker" && args[0] == "info" {
return "29.0.0\n", nil
}
+119
View File
@@ -136,3 +136,122 @@ func TestAHandOverIsForTheResourceThatAskedForIt(t *testing.T) {
t.Fatalf("a hand-over for notes.data gave other.data the directory: mode %o", info.Mode().Perm())
}
}
// **A handed-over directory is changed through no link, at the moment of use** (review of issue 356). The
// hand-over refused a link when it was recorded; between then and the apply its parent is swapped for a link to
// a victim tree. The apply refuses, and the victim's directory keeps its mode — and so does a directory whose own
// name became a link, and one whose grandparent did.
func TestAHandedOverDirectoryIsChangedThroughNoLink(t *testing.T) {
for _, c := range []struct {
swap string
applyOnce bool
}{{"parent", false}, {"grandparent", false}, {"itself", false}, {"parent", true}, {"grandparent", true}} {
swap, applyOnce := c.swap, c.applyOnce
name := swap
if applyOnce {
name += " after an apply gave it its owner"
}
t.Run(name, func(t *testing.T) {
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
found := filepath.Join(dir, "top", "parent", "data")
if err := os.MkdirAll(found, 0o700); err != nil {
t.Fatal(err)
}
decl := `{"id":"notes.data","type":"directory","path":"` + found + `","mode":"0755"}`
_, state, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if a, _ := state.Find("notes.data"); !a.AsFound {
t.Fatalf("not used as found: %+v", a)
}
before := HandedOver
HandedOver = func() map[string]bool { return map[string]bool{handOverKey("notes.data", found): true} }
t.Cleanup(func() { HandedOver = before })
if applyOnce {
// The hand-over takes: the directory is given its mode, and the record is no longer as found.
if _, state, err = Apply(context.Background(), archHost(t), declare(t, decl), state, store.OriginDeclared,
noServices, nil, nil); err != nil {
t.Fatal(err)
}
if a, _ := state.Find("notes.data"); a.AsFound {
t.Fatalf("still as found after the hand-over took: %+v", a)
}
if info, _ := os.Stat(found); info.Mode().Perm() != 0o755 {
t.Fatalf("the hand-over did not take: %o", info.Mode().Perm())
}
}
// The victim: a tree of the same shape somewhere else, its data directory mode 0700.
victim := filepath.Join(dir, "victim", "top", "parent", "data")
if err := os.MkdirAll(victim, 0o700); err != nil {
t.Fatal(err)
}
var swapped, target string
switch swap {
case "parent":
swapped, target = filepath.Join(dir, "top", "parent"), filepath.Join(dir, "victim", "top", "parent")
case "grandparent":
swapped, target = filepath.Join(dir, "top"), filepath.Join(dir, "victim", "top")
case "itself":
swapped, target = found, victim
}
if err := os.Rename(swapped, swapped+".real"); err != nil {
t.Fatal(err)
}
if err := os.Symlink(target, swapped); err != nil {
t.Fatal(err)
}
_, _, err = Apply(context.Background(), archHost(t), declare(t, decl), state, store.OriginDeclared,
noServices, nil, nil)
if err == nil {
t.Fatal("a handed-over directory reached through a link was applied")
}
// Refused by the path's re-check before any descriptor is opened (a link at the name itself is refused
// earlier still, by the directory applier's own look at it).
want := "reached through a link"
if swap == "itself" {
want = "symbolic link"
}
if !strings.Contains(err.Error(), want) {
t.Fatalf("refused as %v; want %q", err, want)
}
if info, _ := os.Stat(victim); info.Mode().Perm() != 0o700 {
t.Fatalf("the victim's directory was given the declared mode through the link: %o", info.Mode().Perm())
}
})
}
}
// The check at the moment of use, without the path's re-check: a component swapped for a link is refused by the
// descriptor walk itself, whatever was resolved a moment before.
func TestGiveHandedOverFollowsNoLink(t *testing.T) {
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
victim := filepath.Join(dir, "victim", "data")
if err := os.MkdirAll(victim, 0o700); err != nil {
t.Fatal(err)
}
if err := os.Symlink(filepath.Join(dir, "victim"), filepath.Join(dir, "parent")); err != nil {
t.Fatal(err)
}
if _, _, _, err := giveHandedOver(filepath.Join(dir, "parent", "data"), 0o755, 0, 0, false); err == nil {
t.Fatal("a directory reached through a link was changed")
}
if info, _ := os.Stat(victim); info.Mode().Perm() != 0o700 {
t.Fatalf("the victim was changed: %o", info.Mode().Perm())
}
mode, _, _, err := giveHandedOver(victim, 0o750, 0, 0, false)
if err != nil || mode.Perm() != 0o700 {
t.Fatalf("the directory itself: before %o, %v", mode, err)
}
if info, _ := os.Stat(victim); info.Mode().Perm() != 0o750 {
t.Fatalf("the directory itself was not given its mode: %o", info.Mode().Perm())
}
}
+1
View File
@@ -59,6 +59,7 @@ func TestAUnitWhoseFileTheMeshWroteIsStoppedWhateverItsRecordSays(t *testing.T)
m := unitsMachine(map[string]*fakeUnit{"mesh-guard.service": {active: "active", enabled: "enabled"}})
fileThereAtStop := false
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "systemctl" && args[0] == "stop" {
_, err := os.Stat(unitFile)
fileThereAtStop = err == nil
+171
View File
@@ -0,0 +1,171 @@
package apply
import (
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A unit the mesh found running and enabled, and held stopped and disabled by its own declaration, is
// given back running and enabled when that declaration goes (the zfs module holds the distribution's
// scrub timer stopped while its own timer scrubs the pool; unassigning it must not leave the pool with no
// schedule at all). The report says so.
func TestAUnitTheMeshStoppedAndDisabledIsGivenBackRunningAndEnabled(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{
"zfs-scrub-weekly@storage.timer": {active: "active", enabled: "enabled"},
})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"zfs.distribution-weekly","type":"service","unit":"zfs-scrub-weekly@storage.timer","state":"stopped","boot":"disabled"}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
if u := m.units["zfs-scrub-weekly@storage.timer"]; u.active != "inactive" || u.enabled != "disabled" {
t.Fatalf("the fixture did not stop and disable it: %s and %s", u.active, u.enabled)
}
report, _, err := applyOn(t, nothingButA(t), copyOf(t, state), m)
if err != nil {
t.Fatal(err)
}
if u := m.units["zfs-scrub-weekly@storage.timer"]; u.active != "active" || u.enabled != "enabled" {
t.Errorf("given back as %s and %s; it was found running and enabled", u.active, u.enabled)
}
said := ""
for _, r := range report.Outcomes {
if r.ID == "zfs.distribution-weekly" {
said = r.Action + ": " + r.Detail
}
}
if !strings.HasPrefix(said, "restored: ") || !strings.Contains(said, "started") || !strings.Contains(said, "enabled at boot") {
t.Errorf("the report says %q", said)
}
}
// Only what the mesh's own declaration did is undone: a unit the mesh held stopped that was found stopped
// stays stopped, and one found running that the mesh never stopped is left running as before.
func TestOnlyWhatTheMeshsDeclarationStoppedIsStartedAgain(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{
"found-stopped.timer": {active: "inactive", enabled: "disabled"},
"left-running.timer": {active: "active", enabled: "enabled"},
})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"a","type":"service","unit":"found-stopped.timer","state":"stopped","boot":"disabled"},
{"id":"b","type":"service","unit":"left-running.timer","state":"running","boot":"enabled"}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
if _, _, err := applyOn(t, nothingButA(t), state, m); err != nil {
t.Fatal(err)
}
if u := m.units["found-stopped.timer"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("found stopped and disabled, given back as %s and %s", u.active, u.enabled)
}
if u := m.units["left-running.timer"]; u.active != "active" || u.enabled != "enabled" {
t.Errorf("found running, given back as %s and %s", u.active, u.enabled)
}
}
// A record from before the host kept what it declared says nothing about who stopped the unit: it is
// left as it is, never started.
func TestARecordWithoutWhatWasDeclaredStartsNothing(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{"old.timer": {active: "inactive", enabled: "disabled"}})
state := store.State{}
state.Record(store.Applied{ID: "old", Type: "service", Target: "old.timer", Origin: store.OriginDeclared,
Found: &store.FoundUnit{Unit: "old.timer", State: "running", Boot: "enabled"}})
if _, _, err := applyOn(t, nothingButA(t), state, m); err != nil {
t.Fatal(err)
}
if u := m.units["old.timer"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("an old record started %s and %s", u.active, u.enabled)
}
}
// The plan says it before the apply does it.
func TestThePlanSaysAStoppedUnitIsStartedAgain(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{"zfs-scrub-weekly@storage.timer": {active: "active", enabled: "enabled"}})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"w","type":"service","unit":"zfs-scrub-weekly@storage.timer","state":"stopped","boot":"disabled"}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
said := ""
for _, s := range Plan(parse(t, nothingButA(t)), state, store.OriginDeclared) {
if s.ID == "w" {
said = s.Verb + ": " + s.Why
}
}
if !strings.HasPrefix(said, "restore: ") || !strings.Contains(said, "restore if still as the mesh held it") ||
strings.Contains(said, "so it is started") {
t.Errorf("the plan says %q", said)
}
}
// Third review: on removal a unit is started again only when it is exactly as the mesh held it, stopped and
// disabled. A masked unit, or one somebody started or enabled since, is left alone, and the report says why.
func TestAMaskedOrChangedUnitIsLeftAloneOnRemoval(t *testing.T) {
for _, c := range []struct {
name, active, enabled, says string
}{
{"masked", "inactive", "masked", "masked"},
{"enabled since", "inactive", "enabled", "not the stopped and disabled"},
{"started since", "active", "disabled", "not the stopped and disabled"},
} {
m := unitsMachine(map[string]*fakeUnit{"w.timer": {active: "active", enabled: "enabled"}})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"w","type":"service","unit":"w.timer","state":"stopped","boot":"disabled"}]}`, store.State{}, m)
if err != nil {
t.Fatal(err)
}
m.units["w.timer"].active, m.units["w.timer"].enabled = c.active, c.enabled
report, _, err := applyOn(t, nothingButA(t), state, m)
if err != nil {
t.Fatalf("%s: %v", c.name, err)
}
if u := m.units["w.timer"]; u.active != c.active || u.enabled != c.enabled {
t.Errorf("%s: changed to %s and %s", c.name, u.active, u.enabled)
}
said := ""
for _, o := range report.Outcomes {
if o.ID == "w" {
said = o.Action + ": " + o.Detail
}
}
if !strings.HasPrefix(said, "forgotten: ") || !strings.Contains(said, c.says) {
t.Errorf("%s: the report says %q", c.name, said)
}
}
}
// A restore that fails — the start refused, or the unit not running after it — waits: the record stays,
// the report says so, and no apply is made fatal by it.
func TestAFailedRestoreWaitsAndIsNeverFatal(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{"w.timer": {active: "active", enabled: "enabled"}})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"w","type":"service","unit":"w.timer","state":"stopped","boot":"disabled"}]}`, store.State{}, m)
if err != nil {
t.Fatal(err)
}
m.units["w.timer"].wontStart = true
for i := 0; i < 2; i++ {
report, next, err := applyOn(t, nothingButA(t), state, m)
if err != nil {
t.Fatalf("apply %d failed: %v", i+1, err)
}
said := ""
for _, o := range report.Outcomes {
if o.ID == "w" {
said = o.Action + ": " + o.Detail
}
}
if !strings.Contains(said, "wait") || !strings.Contains(said, "not running") {
t.Errorf("apply %d: the report says %q", i+1, said)
}
if _, kept := next.Find("w"); !kept {
t.Errorf("apply %d: the record went, so nothing would try again", i+1)
}
state = next
}
}
+12 -3
View File
@@ -5,7 +5,7 @@ import (
"fmt"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
@@ -63,7 +63,7 @@ func (w Wanted) othersAsk(account, group, except string) []string {
// applyGroups makes the account be in every group the resource declares, and takes it out of every group
// this resource put it in and no longer asks for. What the mesh put the account in is recorded on out.
func applyGroups(ctx context.Context, sys system.System, r *declaration.User, run Runner,
previous store.Applied, wanted Wanted, out *Outcome) error {
previous store.Applied, wanted Wanted, handed Handed, out *Outcome) error {
// What this resource put the account in before, for this account only: a declaration that renamed its
// user says nothing about the new one's groups.
var added []string
@@ -83,6 +83,10 @@ func applyGroups(ctx context.Context, sys system.System, r *declaration.User, ru
var put []string
for _, want := range r.Groups {
if already[want] {
// A record that went handed it here: the mesh's, and this resource's to take back (novox/hq ADR 0312 §5).
if handed.has(r.Name, want) && !contains(added, want) {
added = append(added, want)
}
continue
}
exists, err := system.GroupExists(ctx, system.Runner(run), want)
@@ -139,7 +143,11 @@ func applyGroups(ctx context.Context, sys system.System, r *declaration.User, ru
// giveGroupsBack is removeUser's groups: every group the mesh put the account in, taken back unless
// another declared resource still asks for it. Never fatal, for the shell's reason: a removal that failed
// would stay recorded and fail the same way on every apply after. Empty when there was nothing to say.
func giveGroupsBack(ctx context.Context, sys system.System, a store.Applied, run Runner, wanted Wanted) (bool, string) {
//
// A group left because another declared resource still asks for it is handed to that resource (handed), which
// records it as the mesh's, so it is taken back once nobody asks (novox/hq ADR 0312 §5).
func giveGroupsBack(ctx context.Context, sys system.System, a store.Applied, run Runner, wanted Wanted,
handed Handed) (bool, string) {
if len(a.Groups) == 0 {
return false, ""
}
@@ -157,6 +165,7 @@ func giveGroupsBack(ctx context.Context, sys system.System, a store.Applied, run
}
if others := wanted.othersAsk(a.Target, g, a.ID); len(others) > 0 {
said = append(said, fmt.Sprintf("left in %s, which %s still asks for", g, strings.Join(others, ", ")))
handed.hand(a.Target, g)
continue
}
gone, why := leaveGroup(ctx, sys, a.Target, g, run)
+2
View File
@@ -35,6 +35,8 @@ func newGroupDB(in []string, exists ...string) *groupDB {
}
func (g *groupDB) run(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
g.asked = append(g.asked, name+" "+strings.Join(args, " "))
switch {
case name == "getent" && args[0] == "passwd":
+3
View File
@@ -30,6 +30,8 @@ type aMachine struct {
}
func (m *aMachine) run(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
line := name + " " + strings.Join(args, " ")
m.commands = append(m.commands, line)
switch {
@@ -245,6 +247,7 @@ func TestAStepThatFailsGatesItsModule(t *testing.T) {
body, digest := anArchive(t, map[string]string{"mesh-controller": "#!/bin/sh\n"})
src := serving(t, body)
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name == "systemctl" && len(args) > 1 && args[0] == "start" {
return "", errors.New("Job for mesh-controller-prepare.service failed")
}

Some files were not shown because too many files have changed in this diff Show More