Author SHA1 Message Date
jschoubben 1a190090ab The engine states its key again until it hears the vault's key, and at a slow interval after, so a vault delivered after it or a changed vault key is heard (review of the adoption handing, ADR 0547 clause 9) 2026-10-26 12:03:12 +01:00
jschoubben fd381567f4 The bus node hands the loaded list's hashes to the vault, sealed to its key, in the bus module's have (ADR 0547 clause 9, issue 1945)
The vault holds no value for the controller's own account, the node accounts and the enrolled devices, so it cannot
hash them, and take-over waits on their hashes. The bus node's engine is the one part that has them, in the list the
bus loaded. Its follower of the user list now puts every loaded user's bcrypt hash in the bus module's have, sealed to
the vault's own key (mesh-sdk delivery.AccountsHanded), so of every reader of the have bucket the vault alone opens
it; the clear part says only the list's digest, the key's fingerprint and how many users. The vault's key is heard from
its answer to node-key, which the engine already asks at start; a first or changed key looks at the list again. It is
sealed again only when the list on the machine or the vault's key changes; a refused list hands the list kept on the
machine. The vault takes from it only the users it holds no value for.

Also: the user list follower's test sent its later puts before the watch had handed the keys there before it, so two
of its cases failed on main whenever the goroutine started late; the feed now says when the watch began.
2026-10-26 11:32:36 +01:00
mesh-tools 1195cf4597 Merge pull request 'The node-engine retries an apply that met a passing bus error, so a reload does not leave a module failed (issue 1945)' (#144) from fix/apply-retries-transient into main 2026-10-25 22:42:41 +00:00
jschoubben 99285cb080 Retry an apply, a fetch by digest and a have write that met the bus reloading, and keep the cause in the have
mesh/delivery merged: source.main moved to novox/mesh-host@1195cf45971e
mesh/check pass 99285cb080db on 1195cf45971e | unit 303 run, 1080 kept, 5 skipped | integration 306 run, 1082 kept, 4 skipped | over bound: unit (slow…
The bus node's engine writes the bus user list (ADR 0547 bridge); each write makes the bus reload its accounts, and
for about three seconds every JetStream request meets no responders. An apply that failed then was kept failed until a
key changed (mesh-vault on novox, 2026-10-25 23:04:24), holding every module that needs the secret seat. A transient
bus failure is now retried with a backoff (2 s, 5 s, 15 s, 60 s, then every 5 min), the have saying retrying meanwhile;
after the engine writes the user list, the transiently failed modules are tried again once the reload is over. A have
write that met it is written again the same way. shorten keeps the head and the tail, the cause.
2026-10-25 23:37:02 +01:00
mesh-tools d75c66dc7e Merge pull request 'The bus node writes the newest signed bus user list by itself, without a want (ADR 0547 bridge, issue 1945)' (#143) from fix/1945-user-list-follows into main 2026-10-25 21:16:06 +00:00
mesh-tools e196058f6f Merge pull request 'The node-engine wakes the provisioner when an apply lets a file it refused be written (issue 924)' (#142) from fix/provisioner-retries-on-declaration into main 2026-10-25 21:08:52 +00:00
jschoubben 5e71ab6ae6 Count the bus's users from the list a digest names, take only a composition of this node and module, and refuse a list that would lock the mesh out (review of the bridge, ADR 0547 §8)
mesh/delivery merged: source.main moved to novox/mesh-host@d75c66dc7e27
mesh/delivery-group group fix/1945-user-list-follows merged: every member is merged
mesh/check pass 5e71ab6ae612 on d75c66dc7e27 | unit 303 run, 1077 kept, 5 skipped | integration 306 run, 1079 kept, 4 skipped | over bound: unit, inte…
B1: the controller sends the user list by digest (over 64 KiB), so its users are counted from the list as resolved, not
from the composition's inline content, which is empty then.
R1: a composition signed for another node or module than its key names is no source.
R2: a list without the controller's own user, or one that drops more than half the users of the file on the machine,
is not written; the have says why and the file on the machine is kept.
2026-10-25 22:00:05 +01:00
jschoubben 3f02bd7ff6 The bus node writes the bus's user list alone from the newest composition of the bus module, with no want, and says it in the bus module's have (ADR 0547 §8, issue 1945)
The list reached the bus only through a want of the bus module, which nobody writes (ADR 0539 §15): on 2026-10-25 it
stood from 11:23 to 20:50 and was put back by hand. The node agent now follows its composition bucket for the newest
signed composition carrying the user list, writes that file alone through the apply's file step when it differs
(mode 0600, read back, recorded as the module's), moves nothing that reads it (the bus module's entrypoint reloads the
server in place), never goes back to an older list, looks again after every apply of the bus module, and says source,
revision, wanted and written digests, users and any error in the bus module's have (mesh-sdk UserListWritten).

Read with mesh-delivery-view.node_haves (module nats); the bus's own nats_user_can says in_force once it reloaded.
2026-10-25 21:50:49 +01:00
jschoubben 338797a527 Knock under the apply's lock, and read what is owed first (review of this fix)
mesh/delivery merged: source.main moved to novox/mesh-host@e196058f6fb0
mesh/delivery-group group fix/provisioner-retries-on-declaration merged: every member is merged
mesh/check pass 338797a52703 on d75c66dc7e27 | unit 2301 run, 3410 kept, 10 skipped | integration 2326 run, 3431 kept, 13 skipped | over bound: unit, …
A write judged by the declaration before the apply settled could record its
refusal just after Knock found nothing owed: the original fault in a small
window. Knock now holds the apply's lock as a write does, reads no declaration
while nothing is owed, and its comment names Settled as its caller.
2026-10-25 21:49:23 +01:00
jschoubben 2c93f79429 Tell the provisioner's client to fill again once a settled apply may let through a write the filling refused (ace, 2026-10-25 21:17)
ace's filling refused mesh-build-go's files as not named as filled 32 s before
its want arrived; after the apply that named them nothing asked again, and the
runtime crash-looped on its missing broker credential. The filling now keeps
what it refused or failed and, when the node agent says a module's apply or
removal settled (agent.Env.Settled, called once what runs holds it), touches its
write socket if the declaration now names such a path, names the provisioner's
account, or a write had failed; the client watches that socket and fills every
module again. Said in the journal each time.
2026-10-25 21:42:41 +01:00
mesh-tools 5f16cf507a Merge pull request 'Say a module's health is not yet judged while it changes, and judge it as soon as its change ends (issue 1945)' (#141) from fix/r6-health-unknown into main 2026-10-25 15:04:54 +00:00
jschoubben b32a27c115 Owe a look when a module's change ends unjudged, so a module with nothing long-running is ok within a look of its apply
mesh/delivery merged: source.main moved to novox/mesh-host@5f16cf507a0f
mesh/check pass b32a27c115a2 on 0dfbe1a179f2 | unit 292 run, 1060 kept, 5 skipped | integration 295 run, 1062 kept, 4 skipped | over bound: unit (slow…
The facts look inside an apply runs while the module is still applying and judges nothing; the
running write then left xorg running and unknown until the five-minute statement, holding every
x11-display consumer behind it. The running write now nudges the health loop, which looks and says
at once. A declaration's report gets the same motion and composition check, and a removed module
leaves nothing marked unjudged (review of fix 15).
2026-10-25 15:46:05 +01:00
jschoubben c16e625a08 Say a module's health is not yet judged while its change is under way and until a look after it
A have said applying and failed for the controller on novox at 15:27:36, from a look at the
container being replaced, two seconds before its launch said healthy; the assigner held the
canary on it and set novox back. A look now judges only a module that is not in motion, and a
module whose composition moved carries unknown until a look after the move.
2026-10-25 15:32:59 +01:00
jschoubben 1cdb4d1a29 Say a module found running on its own have again at the agent's start, so its health is this engine's and not an older engine's unknown
xorg's have on g14 and shanks kept the unknown an engine before 958ccbd wrote: the
baseline believed the have and never handed the module to the session, so no look
rewrote its health, and the assigner held every module requiring x11-display back.
2026-10-25 15:27:23 +01:00
mesh-tools 0dfbe1a179 Merge pull request 'Drop the node's own copy of a resource a module composition declares, and give the kept declaration's filled paths to their modules (issue 1945)' (#140) from fix/r6-own-composition-dupes into main 2026-10-25 13:51:00 +00:00
jschoubben 1bdbf47ffa Leave out the node's own copy only by kind and id, never by a shared target (issue 1945, review)
mesh/delivery merged: source.main moved to novox/mesh-host@0dfbe1a179f2
mesh/check pass 1bdbf47ffa83 on 0dfbe1a179f2 | unit 302 run, 1044 kept, 5 skipped | integration 305 run, 1046 kept, 4 skipped | over bound: unit (slow…
A unit or container of the node's own that shares only its name with a
module's resource is not the module's: dropped, its record became an orphan
the apply removes, stopping the module's unit (1945/AC17).
2026-10-25 14:29:06 +01:00
jschoubben 16f84b53da Leave the node's own copy of a module's resource out, and carry the kept declaration's filled paths (issue 1945)
The node's own composition and a module's both declared the same files, and
the joined declaration was refused whole ("2 resources declare X"): the
node's own copy is now left out when joining and at the node's own apply, and
said once (1945/AC17). A module still running from the kept declaration now
carries its filled paths, so the filling no longer refuses them as not named
as filled (1945/AC18).
2026-10-25 14:10:52 +01:00
mesh-tools 57b553ac5a Merge pull request 'Apply the revision of a composition its want names, read from the key's history, when the latest has another digest (issue 1945)' (#139) from fix/r6-want-names-revision into main 2026-10-25 12:30:40 +00:00
jschoubben 11a635796d Bound a key's history read to 10 s and test it on a real bus; the witness test waits for its process to run from its image (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-host@57b553ac5a57
mesh/check pass 11a635796df9 on 57b553ac5a57 | unit 299 run, 1043 kept, 5 skipped | integration 302 run, 1045 kept, 4 skipped | over bound: unit (slow…
2026-10-25 12:51:00 +01:00
jschoubben 8343c4fb36 Apply the revision of a composition key whose digest the want names, from the revisions the bucket keeps (issue 1945) 2026-10-25 12:42:20 +01:00
mesh-tools 3e3e43de52 Merge pull request 'Never lose the bus to a replace of its own container (issue 1945)' (#138) from fix/r6-bus-replace into main 2026-10-25 11:25:11 +00:00
jschoubben 175fb45c19 Take a container left set aside as the next apply's, and bound an apply to 30 minutes (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-host@3e3e43de5214
mesh/check pass 175fb45c19e3 on 3e3e43de5214 | unit 299 run, 1042 kept, 5 skipped | integration 302 run, 1043 kept, 4 skipped | over bound: unit (slow…
2026-10-25 12:18:50 +01:00
jschoubben 219169dbd9 Set a running container aside until its replacement runs, and put it back when the replacement does not come up (issue 1945) 2026-10-25 12:14:41 +01:00
jschoubben b99d0b45bf Run an apply in a context the bus's loss cannot end, and say why a killed command was killed (issue 1945) 2026-10-25 12:09:28 +01:00
jschoubben f06058390c Merge pull request 'Apply again a module whose resources are gone, move a module with nothing to back up, and report ok for a running module with nothing to watch (issue 1944)' (#137) from fix/r6-reapply-missing into main
Reviewed-on: #137
2026-10-25 10:11:19 +00:00
jschoubben 958ccbde3e Say a running module ok once looked at when nothing of it is judged, so mesh-host's canary can be healthy and its other nodes get the new engine (issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 958ccbde3ed4 on f06058390c0f | unit 297 run, 1038 kept, 5 skipped | integration 300 run, 1039 kept, 4 skipped | over bound: unit (slow…
2026-10-25 11:08:10 +01:00
jschoubben 43d1c2456f Apply a module again only for a newly missing resource, leave filled files out of the record check, and say when the applied state cannot be read (issue 1945) 2026-10-25 11:04:08 +01:00
jschoubben c46937da04 Apply a module again when the engine holds no record of its resources or its health says one is missing, and move a module with nothing to back up without a backup (issue 1944, issue 1945) 2026-10-25 10:59:21 +01:00
jschoubben cf455bc4fe Merge pull request 'Never remove a module's resources while its own composition waits to be applied (issue 1944)' (#136) from fix/r6-node-own-keeps-moved into main
Reviewed-on: #136
2026-10-25 09:23:28 +00:00
jschoubben ecf142e451 Let only a module's wanted or newest composition hold the node's own apply back, never a stale refused one (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-host@cf455bc4fe3e
mesh/check pass ecf142e4513f on cf455bc4fe3e | unit 292 run, 1033 kept, 5 skipped | integration 295 run, 1034 kept, 4 skipped | over bound: unit (slow…
2026-10-25 02:01:24 +01:00
jschoubben fa9e4483c8 Hold back the node's own apply while a module's composition cannot be read or is refused, naming it (issue 1944, issue 1945) 2026-10-25 02:56:20 +02:00
jschoubben e62c32c8c3 Leave a record another module's composition now holds, whatever its type, instead of removing it in the node's own apply; a removed module holds nothing back (issue 1944, issue 1945) 2026-10-25 02:54:58 +02:00
jschoubben 2bfde7d659 Hold out a module whose own composition waits to be applied, so the node's own apply never removes the resources it moved there (issue 1944, issue 1945) 2026-10-25 02:52:10 +02:00
jschoubben db06feed2e Merge pull request 'Refuse an apply that would write a provisioner file empty, naming the file (issue 1944)' (#135) from fix/r6-node-composition into main
Reviewed-on: #135
2026-10-25 00:17:56 +00:00
jschoubben 4e6b756bd2 Refuse an apply that would write a provisioner's file empty, read the node's own filled paths, and name the node's own composition when its apply fails (issue 1944, issue 1945)
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check checking 4e6b756bd225 on b833d09d2872 | novox/mesh-host#135 in group fix/r6-node-composition
2026-10-25 01:49:00 +02:00
jschoubben b833d09d28 Merge pull request 'R6: the node-engine applies from its wants and signed compositions, one module at a time; operator-signers (issue 1944)' (#134) from feat/r6 into main
Reviewed-on: #134
2026-10-24 23:17:39 +00:00
jschoubben 81758b24a8 Pin mesh-sdk to feat/r6 748c564, which carries main's composer (issue 1943)
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/delivery delivering: 0 machine step(s) passed
mesh/check checking 81758b24a8e4 on 285395d523e1 | novox/mesh-host#134 in group feat/r6
2026-10-24 23:06:00 +02:00
jschoubben 25ad5ce42c Merge remote-tracking branch 'origin/main' into feat/r6 2026-10-24 23:00:09 +02:00
jschoubben 285395d523 Merge pull request 'R4g finish: merge-check.sh gone, reads-outside markers (issues 1295, 1935)' (#133) from feat/r4g-finish into main
Reviewed-on: #133
2026-10-24 20:57:23 +00:00
jschoubben b47df31834 Pin mesh-sdk to d7a9b96 2026-10-24 21:18:33 +02:00
jschoubben fd05c8235d Pin mesh-sdk to 7f9cc0e 2026-10-24 20:43:47 +02:00
jschoubben 98d1f165b7 Pin mesh-sdk to e45bdc0 2026-10-24 20:22:00 +02:00
jschoubben 332d8f161b The carried genesis template no longer lists the controller's plan-moved event 2026-10-24 18:32:58 +02:00
jschoubben 5088755e5e gofmt 2026-10-24 18:28:22 +02:00
jschoubben e72c682450 Pin mesh-sdk to a523a8c 2026-10-24 18:18:06 +02:00
jschoubben be5f31ce66 What runs is the agent's compositions plus the kept declaration's rest: health, filling, take act, directories and the others held out of an apply read it; a restart trusts the node-engine's own haves; the bed tests run off Arch (review round 1) 2026-10-24 18:13:40 +02:00
jschoubben 67db003813 Pin mesh-sdk to the head of feat/r6 (5da87a0) 2026-10-24 17:50:20 +02:00
jschoubben e5a06cc73d The node agent waits for a backup's end on the holder's event, not by polling; the manifest consumes it (R6-A A5) 2026-10-24 17:03:30 +02:00
jschoubben 20d0998cdd The operator's public keys reach the node as the file operator-keys, fed by the node-engine's setting operator-signers (ADR 0539 clause 3) 2026-10-24 16:57:15 +02:00
jschoubben 83f16cff8d apply FILE is no hand path: mesh-compose and a want are (ADR 0539 clause 10); genesis keeps the carried bundle 2026-10-24 16:55:23 +02:00
jschoubben 96d0322498 The node-engine reports reads-wants; a failed apply is said; the have's fingerprint and digest follow its health (R6-A A6) 2026-10-24 16:43:26 +02:00
jschoubben b2003e7a38 A composition is applied only when the controller or the operator signed it; the node agent's integration tests on an embedded bus (R6-A A9, A1, A5) 2026-10-24 16:42:11 +02:00
jschoubben 3303797bfb The node agent runs in the node-engine: watches the wants and compositions on the link, applies one module's composition, asks the backup, writes the haves (R6-A A1, A5, A6) 2026-10-24 16:39:51 +02:00
jschoubben d5dd7761c6 The node agent: follows the wants and the node's compositions, applies one module at a time, backs a data module up first, holds it on a failed backup (R6-A A1, A2, A5, A6) 2026-10-24 16:36:21 +02:00
jschoubben 5f5f37b0b2 Pin mesh-sdk to feat/r6 S1-S3 (d6cef58): the composition bucket, the want's composition and backup answer, the have's composition and held phase 2026-10-24 16:31:20 +02:00
jschoubben c30b86ae22 Proved: the evidence is a have under the running build (comment) 2026-10-24 16:26:39 +02:00
jschoubben 1598156752 The node-engine hears no declaration: the queue, orders, put-backs, unsaid, numbers and news go; a session says the facts in the node-engine's own order, and the first have proves a build (R6-A A7, A8) 2026-10-24 16:25:11 +02:00
jschoubben e2b6f37168 Apply one module at a time: a scope of the module and what the others hold, and an owner record, so a removal is the module's own (R6-A A3, A4) 2026-10-24 16:16:32 +02:00
jschoubben 13bdcb694b Mark the packages whose tests read files outside their directory, found by the review (issue 1935)
mesh/delivery delivered
mesh/delivery-group group feat/r4g-finish merged: every member is merged
mesh/check checking 13bdcb694be0 on 3c3628c0ce46 | novox/mesh-host#133 in group feat/r4g-finish
2026-10-24 04:39:01 +02:00
jschoubben 89ebd0eefe Merge remote-tracking branch 'origin/feat/r4g-finish-1935' into HEAD 2026-10-24 04:26:19 +02:00
jschoubben b4f6537f80 Mark the packages whose tests read files outside their directory, so mesh-build-go never keeps their verdict (issue 1935) 2026-10-24 04:01:02 +02:00
jschoubben 7618f661ea Fail, not skip, when the checker places no controller beside the node-engine (issue 1295) 2026-10-24 03:43:49 +02:00
jschoubben 3c3628c0ce Merge pull request 'R4g (mesh-host): merge-check.sh goes; the checker judges the repository (design 47)' (#132) from feat/r4g into main
Reviewed-on: #132
2026-10-23 17:50:55 +00:00
jschoubben 4b4bb09a05 Make the apply tests fit the checker's isolated step: accept any who in the refused line, skip aloud where no group can be set (issue 1919, R4g)
mesh/delivery delivered
mesh/delivery-group group feat/r4g merged: every member is merged
mesh/check checking 4b4bb09a05a0 on 48c3e3834c69 | novox/mesh-host#132 in group feat/r4g
2026-10-23 18:54:53 +02:00
jschoubben 11b0a1fb29 Skip aloud the controller's step-nine check where the running checker places no controller (issue 1295, R4g)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: github.com/novox/mesh-host/internal/apply.TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing: judge_directories_…
2026-10-23 18:35:32 +02:00
jschoubben 6ea77844d6 Judge the other genesis modules' toolchains; set aside only build-agent (issue 1152, R4g)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: github.com/novox/mesh-host/internal/bootstrap.TestTheControllersOwnManifestPassesStepNine: genesis_form_test.go:249: the cont…
2026-10-23 18:20:50 +02:00
jschoubben 2633573012 Skip aloud the tests whose subject R4g removes: genesis's build agent (issue 1152)
The build agent left mesh-controller with R4g (ADR 0524 section 5); genesis still installs it until issue 1152 decides where a new mesh's builder comes from. The tests that read its manifest say NOT JUDGED and are judged again with 1152.
2026-10-23 18:20:11 +02:00
jschoubben 976c9998d6 Delete merge-check.sh; the checker's levels run what it ran (issue 1295, S3; ADR 0529)
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/check failed: unit: github.com/novox/mesh-host/internal/bootstrap.TestGenesisInstallsTheCataloguesBuildAgent: builder_test.go:180: mesh-controlle…
2026-10-23 15:26:03 +02:00
mesh-tools 48c3e3834c Merge pull request 'Take a root-made directory in the mesh's own tree, and hold a container until the file it mounts exists (issue 1873)' (#131) from fix/1873-mesh-tree-dirs into main 2026-10-22 20:41:08 +00:00
jschoubben 0d12fb11a4 Make no container before a file it mounts is there, and clear the empty directory a runtime left at a file's path (issue 1873)
mesh/merge-gate head 0d12fb11a48b merged into trunk 59e6a6fa2a4a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 0d12fb11a48b merged into trunk 59e6a6fa2a4a — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: mesh-build-go: no go build node qualifies: builders disagree: go1.27.1-X:nodwarf5 recipe 74744709d277ded9 on ace, g14, shanks; go1.…
On ace the postgres container was replaced at 22:11:51 mounting superuser.secret, a
file the provisioner's client fills only after the apply. The runtime made the
missing source a directory, and every fill after it failed with "is a directory".

A container whose mount source is a file the declaration declares, not yet on the
machine, now waits and is neither created nor replaced. The write of that file
(WriteFilled) applies what reads it, as before, so the container is made once the
file is there.

What the runtime already left is cured by the mesh: a declared file whose path is
an empty directory in the mesh's own tree (by its path and by where it leads),
owned by the engine's account, has that directory removed before the file is
written, and the outcome says so. A directory with anything in it, or outside the
tree, is never removed.
2026-10-22 22:29:39 +02:00
jschoubben 03b9c47af4 Judge the mesh's tree by where a path leads, not only by its text (issue 1873, review)
A root-only link inside /var/lib/mesh pointing out of it would have made a
directory outside the tree, owned by the engine's account, the mesh's to give
away: the fault of issue 1873 again. The path and the tree's root are now both
resolved, links followed, and the directory is the tree's only when the resolved
path lies below the resolved root as well.
2026-10-22 22:27:45 +02:00
jschoubben cdbff1777e Take a directory in the mesh's own tree that the engine's account owns as the mesh's, not as found (issue 1873)
postgres's data directory moved to /var/lib/mesh/postgres. On ace and novox that
directory was there, root:root 0700, before the directory resource was applied, so
the engine used it as found (issue 339's rule) and left it to root: the postgres
provider, running as the operator's account, cannot reach its own secrets,
consumers and grants there.

/var/lib/mesh is the engine's tree (root, 0755): only the engine's account makes a
directory in it. A directory below it owned by that account is now the mesh's and
is given its declared owner and mode. Anywhere else, or owned by another account,
issue 339's rule stands.
2026-10-22 22:23:04 +02:00
111 changed files with 7943 additions and 4891 deletions
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"context"
"testing"
"time"
)
// An apply runs in a context the bus's loss cannot end: the link's context ends when the bus goes, and the bus may go
// because of the apply itself (it replaces the bus's own container), which then must finish (issue 1945, live on
// 2026-10-25 at 11:23: novox's node-engine removed the bus container, its link's context ended, and the docker run that
// would have made the new one was never made).
func TestAnApplyOutlivesTheLinksContext(t *testing.T) {
type key struct{}
link, cancel := context.WithCancel(context.WithValue(context.Background(), key{}, "v"))
ctx, done := applyContext(link)
defer done()
cancel()
if ctx.Err() != nil {
t.Fatalf("the apply's context ended with the link's: %v", ctx.Err())
}
if ctx.Value(key{}) != "v" {
t.Fatal("the apply's context lost the link's values")
}
}
// An apply is bounded all the same: a step that hangs (a docker call that never returns) cannot hold the node agent for
// ever (the review of issue 1945).
func TestAnApplyIsBounded(t *testing.T) {
ctx, done := applyContext(context.Background())
defer done()
at, ok := ctx.Deadline()
if !ok {
t.Fatal("an apply has no bound")
}
if left := time.Until(at); left > applyBound || left < applyBound-time.Minute {
t.Fatalf("the bound is %s away, want about %s", left, applyBound)
}
}
+41
View File
@@ -0,0 +1,41 @@
package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// applyAndKeep applies a whole declaration the way the carried bundle is applied, and says what came of it as the facts
// the node says: what the tests of the apply's steps read. The node-engine applies no whole declaration on its own now
// (novox/hq ADR 0539 §9); this is the test's way to put a machine in a state.
func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.Declared, sched *apply.Scheduler,
say link.Announce) link.Report {
declared, err := declaration.Parse(raw)
if err != nil {
return link.Report{Refused: err.Error()}
}
got := applyModuleHeld(ctx, opts, declared, nil, sched, say)
if got.Refused != "" {
return link.Report{Refused: got.Refused}
}
held := map[string]bool{}
for _, h := range got.Updated.Held {
held[h.ID] = true
}
report := factsOf(ctx, opts, declared, got.Updated, got.Outcome, held)
if signed != nil {
if err := store.SaveDeclared(store.DeclaredPath(opts.state), *signed); err != nil {
report.Failed = map[string]string{"keeping the declaration": err.Error()}
}
}
if got.Err != nil {
report.Failed = map[string]string{"apply": got.Err.Error()}
}
report.Failed = withUnanswered(report.Failed, got.Probe)
report.Unapplied = unapplied(declared, got.Outcome, got.Err)
return report
}
+42
View File
@@ -0,0 +1,42 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
)
func backupEvent(node, module string, ok bool, at time.Time, why string) []byte {
raw, _ := json.Marshal(map[string]any{"node": node, "module": module, "ok": ok, "at": at, "error": why})
return raw
}
// A backup is done when the holder says so for this node and module, held on its words when it says it failed, and held
// when it says nothing within the bound; another node's or module's event, or one from before the ask, is not the answer
// (R6-A A5, novox/hq ADR 0429 §14).
func TestABackupEndsWhenItsEventSaysSo(t *testing.T) {
since := time.Now()
run := func(within time.Duration, events ...[]byte) (done bool, words string) {
ch := make(chan []byte, len(events))
for _, e := range events {
ch <- e
}
got := awaitBackup(context.Background(), ch, "n1", "db", since, within)
return got.Done, got.Words
}
if done, _ := run(time.Second,
backupEvent("n2", "db", true, since, ""), backupEvent("n1", "web", true, since, ""),
backupEvent("n1", "db", false, since.Add(-time.Hour), "an old night"),
backupEvent("n1", "db", true, since.Add(time.Second), "")); !done {
t.Error("the event of this node and module was not the answer")
}
if done, words := run(time.Second, backupEvent("n1", "db", false, since, "the repository is locked")); done ||
!strings.Contains(words, "repository is locked") {
t.Errorf("a failed backup: done=%v words=%q", done, words)
}
if done, words := run(50 * time.Millisecond); done || !strings.Contains(words, "did not say it was done") {
t.Errorf("silence within the bound: done=%v words=%q", done, words)
}
}
+29
View File
@@ -0,0 +1,29 @@
package main
import (
"encoding/json"
"testing"
)
// A module that declares nothing to back up on this machine is not backed up before it moves, and so never held for a
// backup that cannot run (issue 1945, live on 2026-10-25: redis and searxng on ace were held: "redis declares nothing to
// back up on this machine"). What node-backup.backed-up answers decides it.
func TestAModuleWithNothingToBackUpIsNotBackedUp(t *testing.T) {
for _, c := range []struct {
said string
nothing bool
}{
{`[{"module":"postgres","runs":1,"paths":["/var/lib/postgres"]}]`, true},
{`[{"module":"redis","runs":0,"paths":null,"data":[{"item":"cache"}]}]`, true},
{`[]`, true},
{`[{"module":"redis","runs":1,"paths":["/var/lib/redis"]}]`, false},
} {
nothing, ok := declaresNothingToBackUp(json.RawMessage(c.said), "redis")
if !ok || nothing != c.nothing {
t.Errorf("%s: nothing %v (read %v), want %v", c.said, nothing, ok, c.nothing)
}
}
if _, ok := declaresNothingToBackUp(json.RawMessage(`{"error":"x"}`), "redis"); ok {
t.Error("an answer that is not the list was read")
}
}
+9 -41
View File
@@ -5,62 +5,30 @@ import (
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-host/internal/link"
)
// A container waiting because it said busy is news when it starts waiting and when it is replaced, never on
// its output alone (novox/hq ADR 0335): the walk waiting on this machine starts its gate at the report that
// says it replaced, and a reconcile is otherwise silent.
func TestABusyWaitIsSaidWhenItStartsAndWhenItEnds(t *testing.T) {
quiet := link.Report{Outward: []string{"eth0"}}
waiting := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "waiting", Said: "building a"}}}
still := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "waiting", Said: "building b, a minute on"}}}
replaced := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "replaced-when-idle"}}}
if adoptionFingerprint(waiting) == adoptionFingerprint(quiet) {
t.Error("a container starting to wait is not news")
}
if adoptionFingerprint(still) != adoptionFingerprint(waiting) {
t.Error("a wait whose output changed reads as news on every ask")
}
if adoptionFingerprint(replaced) == adoptionFingerprint(waiting) {
t.Error("a container replaced after it waited is not news")
}
if !worthSaying(link.Report{Busy: replaced.Busy}) {
t.Error("a replacement after a wait is not worth saying")
}
}
// While a container waits, a pass is asked for every so often, not at the reconcile's five minutes; with
// nothing waiting, none (novox/hq ADR 0335 §2).
func TestAWaitIsAskedAgainSoonerThanTheReconcile(t *testing.T) {
if BusyAskEvery >= ReconcileEvery || BusyAskEvery > time.Minute {
// While a container waits, a pass is asked for every so often, not at some long interval; with nothing waiting, none
// (novox/hq ADR 0335 §2, ADR 0360 §11).
func TestAWaitIsAskedAgainSoon(t *testing.T) {
if BusyAskEvery > time.Minute {
t.Fatalf("a waiting container is asked every %s", BusyAskEvery)
}
var reconciles atomic.Int32
queue := &link.Queue{Reconcile: func(context.Context) (link.Report, bool) {
reconciles.Add(1)
return link.Report{}, false
}}
var asked atomic.Int32
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go queue.Run(ctx)
busyWaiting.Store(false)
t.Cleanup(func() { busyWaiting.Store(false) })
go busyPasses(ctx, queue, 10*time.Millisecond)
go busyPasses(ctx, func() { asked.Add(1) }, 10*time.Millisecond)
time.Sleep(100 * time.Millisecond)
if n := reconciles.Load(); n != 0 {
if n := asked.Load(); n != 0 {
t.Fatalf("%d passes were asked with nothing waiting", n)
}
busyWaiting.Store(true)
deadline := time.Now().Add(2 * time.Second)
for reconciles.Load() == 0 && time.Now().Before(deadline) {
for asked.Load() == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if reconciles.Load() == 0 {
if asked.Load() == 0 {
t.Fatal("no pass was asked while a container waits")
}
}
+33
View File
@@ -0,0 +1,33 @@
package main
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A composition that names a file the provisioner fills without saying it is filled would have this node-engine write
// that file empty: its apply is refused, loudly, before anything is written (issue 1945, live on 2026-10-25 at 01:22,
// when ace, g14 and shanks wrote claude-code's, node-env's and power's files empty).
func TestAFileItCannotProduceIsNeverWrittenEmpty(t *testing.T) {
unit := json.RawMessage(`{"id":"power.after-boot-unit","type":"file","path":"/etc/systemd/system/mesh-power-after-boot.service","mode":"0644","owner-at-runtime":"owned"}`)
c := delivery.Composed{Node: "ace", Resources: []json.RawMessage{unit}}
d, err := declarationOf(c)
if err != nil {
t.Fatal(err)
}
if err := refuseEmptyWrites(d); err == nil || !strings.Contains(err.Error(), "power.after-boot-unit") {
t.Fatalf("a file the provisioner fills, not filled here: %v", err)
}
c.Filled = []string{"/etc/systemd/system/mesh-power-after-boot.service"}
if d, _ = declarationOf(c); refuseEmptyWrites(d) != nil {
t.Fatalf("a filled file was refused: %v", refuseEmptyWrites(d))
}
plain := delivery.Composed{Node: "ace", Resources: []json.RawMessage{json.RawMessage(
`{"id":"x.f","type":"file","path":"/etc/x","mode":"0644","content":"x"}`)}}
if d, _ = declarationOf(plain); refuseEmptyWrites(d) != nil {
t.Fatalf("a file with its content was refused: %v", refuseEmptyWrites(d))
}
}
-77
View File
@@ -1,77 +0,0 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
"path/filepath"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// **A declaration from an older lease epoch is refused before anything is applied** (novox/hq to-be
// 45 §6, ADR 0227 rule 2). The controller that sent it lost its lease and goes on sending; this node
// applied the holder's. Refused on the host's own apply path — the one the queue's worker runs — before
// the machine is read or touched, with a report naming what was refused and what is held, and the
// kept declaration left as it was.
func TestADeclarationFromAnOlderEpochIsRefusedBeforeAnythingIsApplied(t *testing.T) {
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
_, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "a.conf")
declared := func(epoch, sequence int64) store.Declared {
body, err := json.Marshal(map[string]any{"declaration": 1, "epoch": epoch, "sequence": sequence,
"resources": []any{map[string]any{"id": "a", "type": "file", "path": target, "content": "x\n"}}})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: ed25519.Sign(private, body)}
}
held := declared(57, 3)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), held); err != nil {
t.Fatal(err)
}
stale := declared(41, 12)
report := applyAndKeep(context.Background(), opts, stale.Declaration, &stale, nil, nil)
if report.OlderThan == nil || *report.OlderThan != (link.Order{Epoch: 57, Sequence: 3}) ||
report.Order != (link.Order{Epoch: 41, Sequence: 12}) || report.Declared != digestOf(stale.Declaration) ||
report.Refused == "" {
t.Fatalf("the refusal does not name what was refused and what is held: %+v", report)
}
if _, err := os.Stat(target); !errors.Is(err, os.ErrNotExist) {
t.Fatal("the refused declaration touched the machine")
}
kept, err := store.ReadDeclared(store.DeclaredPath(opts.state))
if err != nil || string(kept.Declaration) != string(held.Declaration) {
t.Fatal("the refused declaration replaced what this node kept")
}
}
// The queue's counts are kept beside the state and read back by the next host; unreadable is an
// error, never zero.
func TestTheNumbersSurviveTheHost(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
if sequence, refused, err := numbersBeside(state).Read(); err != nil || sequence != 0 || refused != 0 {
t.Fatalf("a node that never reported read %d, %d, %v", sequence, refused, err)
}
if err := numbersBeside(state).Save(41, 2); err != nil {
t.Fatal(err)
}
if sequence, refused, err := numbersBeside(state).Read(); err != nil || sequence != 41 || refused != 2 {
t.Fatalf("read back %d, %d, %v", sequence, refused, err)
}
if err := os.WriteFile(store.NumbersPath(state), []byte(`{"report_sequence":`), 0o600); err != nil {
t.Fatal(err)
}
if _, _, err := numbersBeside(state).Read(); err == nil {
t.Fatal("unreadable numbers were read as zero")
}
}
+28 -4
View File
@@ -2,6 +2,7 @@ package main
import (
"context"
"fmt"
"os"
"path/filepath"
"strconv"
@@ -9,13 +10,20 @@ import (
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/link"
)
// aFilesLink is a link that serves objects by name.
type aFilesLink struct{ objects map[string][]byte }
// aFilesLink is a link that serves objects by name, or fails every read with err.
type aFilesLink struct {
objects map[string][]byte
err error
}
func (a aFilesLink) FileObject(_ context.Context, name string, _ int64) ([]byte, error) {
if a.err != nil {
return nil, a.err
}
if b, ok := a.objects[name]; ok {
return b, nil
}
@@ -40,7 +48,7 @@ func TestADeclarationWithAFileByDigestAppliesFromTheObject(t *testing.T) {
bad := append([]byte(nil), body...)
bad[10] ^= 1
ctx := link.WithFiles(context.Background(), aFilesLink{map[string][]byte{name: bad}})
ctx := link.WithFiles(context.Background(), aFilesLink{objects: map[string][]byte{name: bad}})
report := applyAndKeep(ctx, opts, raw, nil, nil, nil)
if report.Refused == "" || !strings.Contains(report.Refused, "m.big") {
t.Errorf("1695/AC2: a differing object did not refuse the declaration: %+v", report)
@@ -48,8 +56,24 @@ func TestADeclarationWithAFileByDigestAppliesFromTheObject(t *testing.T) {
if _, err := os.Stat(target); err == nil {
t.Error("1695/AC2: a file was written from an object that differs")
}
if failedWith(report.Refused).Transient {
t.Errorf("an object that differs is a real failure, and was taken as the bus not answering: %s", report.Refused)
}
ctx = link.WithFiles(context.Background(), aFilesLink{map[string][]byte{name: body}})
// The bus answers no JetStream request while it reloads its accounts (2026-10-25 23:04:24, mesh-vault on the bus
// node): the failure is transient, so the agent tries it again, and the detail kept is its cause, the tail.
noResponders := fmt.Errorf("fetching the object 2bd98ebd of this node's own object bucket: reading the chunks of "+
"object %s: %w", strings.Repeat("x", 300), nats.ErrNoResponders)
ctx = link.WithFiles(context.Background(), aFilesLink{err: noResponders})
report = applyAndKeep(ctx, opts, raw, nil, nil, nil)
failed := failedWith("a file of this composition could not be had by its digest: file mesh-vault.kept: " + report.Refused)
if !failed.Transient || !strings.HasSuffix(failed.Detail, nats.ErrNoResponders.Error()) || len(failed.Detail) > 310 ||
!strings.HasPrefix(failed.Detail, "a file of this composition") {
t.Errorf("a fetch that met no responders: transient %v, detail %q (%d bytes); want transient, its head and its cause",
failed.Transient, failed.Detail, len(failed.Detail))
}
ctx = link.WithFiles(context.Background(), aFilesLink{objects: map[string][]byte{name: body}})
report = applyAndKeep(ctx, opts, raw, nil, nil, nil)
if report.Refused != "" {
t.Fatalf("1695/AC1: refused: %s", report.Refused)
+27 -6
View File
@@ -4,7 +4,9 @@ import (
"context"
"crypto/ed25519"
"errors"
"fmt"
"path/filepath"
"sync/atomic"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
@@ -16,22 +18,41 @@ import (
// serveFilling writes filled files on the provisioner client's request for as long as this engine serves (novox/hq
// ADR 0349 §4). A socket that cannot be served is said once, and this engine goes on applying.
func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, say func(string)) {
if err := filling.Serve(ctx, filling.Socket, fillingEngine(opts, signer, say), say); err != nil {
e := fillingEngine(opts, signer, say)
fillingNow.Store(e)
defer fillingNow.CompareAndSwap(e, nil)
if err := filling.Serve(ctx, filling.Socket, e, say); err != nil {
say("filled files are not written on request on this machine: " + err.Error())
}
}
// fillingNow is the filling this engine serves, for knockFilling; nil while none is served.
var fillingNow atomic.Pointer[filling.Engine]
// knockFilling tells the provisioner's client to fill again what the filling refused or failed and the declaration
// applied now may let through (filling.Engine.Knock), once a module's apply or removal settled (agent.Env.Settled).
// What it told, or why it could not, is said.
func knockFilling(module string, say func(string)) {
e := fillingNow.Load()
if e == nil {
return
}
told, err := e.Knock(filling.Socket)
switch {
case err != nil:
say(fmt.Sprintf("after %s settled: %v", module, err))
case told != "":
say(fmt.Sprintf("after %s settled, the provisioner's client is told to fill again: %s", module, told))
}
}
// fillingEngine is the write of a filled file as this engine does it: the declaration it keeps, verified, read under
// the apply's lock — the same two locks every apply takes, the process's and the one beside the state — and the file
// written through the apply (apply.WriteFilled), its outcome recorded in the node's state as an apply's is.
func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *filling.Engine {
return &filling.Engine{
Declared: func() (*declaration.Declaration, error) {
raw, err := store.LoadDeclared(store.DeclaredPath(opts.state), signer)
if err != nil {
return nil, err
}
return declaration.Parse(raw)
return declaredNow(opts.state, signer)
},
UIDOf: filling.UIDOf,
Lock: func() (func(), error) {
-40
View File
@@ -1,40 +0,0 @@
package main
import (
"context"
"testing"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// **The generation applied is kept, and a put-back never lowers it**: after a put-back of generation 5
// over 7, a send from generation 6 is still older than what this machine applied.
func TestThePutBackDoesNotLowerTheGenerationApplied(t *testing.T) {
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
f := machineThatApplied(t, 7)
newer := f.declared(t, map[string]any{"sequence": 9, "generation": 8}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, newer.Declaration, &newer, nil, nil); r.Refused != "" {
t.Fatalf("a newer generation was refused: %s", r.Refused)
}
back := f.declared(t, map[string]any{"sequence": 10, "generation": 5, "put_back": true}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, back.Declaration, &back, nil, nil); r.Refused != "" {
t.Fatalf("a put-back was refused: %s", r.Refused)
}
state, err := store.Load(f.opts.state)
if err != nil || state.Generation != 8 {
t.Fatalf("the generation applied is %d after a put-back of 5 over 8 (%v)", state.Generation, err)
}
stale := f.declared(t, map[string]any{"sequence": 11, "generation": 7}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, stale.Declaration, &stale, nil, nil); r.OlderGeneration == nil {
t.Fatalf("a send from generation 7 after 8 was applied because a put-back came between: %+v", r)
}
}
-170
View File
@@ -1,170 +0,0 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The assignment generation a declaration was composed from (novox/hq issue 234).
//
// On 2026-10-04 a declaration newer in sequence than every other, and composed from a view of the
// assignments older than the one this machine had applied, named four fewer modules — and the host,
// ordering by sequence alone, undeclared all four. The sequence orders arrival; it cannot tell a later
// send carrying earlier content. The generation can: the node-engine refuses a declaration composed from
// an older generation than the one it applied, before anything is read from the machine.
//
// These read what the report says through its JSON, as the controller does, so they run against a
// node-engine that has never heard of a generation and fail there for that reason.
// generationFixture is a machine that applied generation 7: a file the mesh declared, kept in its state,
// and the declaration it kept — signed — that named it.
type generationFixture struct {
opts options
private ed25519.PrivateKey
kept store.Declared
// stays is the file the older generation's declaration does not name.
stays string
}
func machineThatApplied(t *testing.T, generation int64) generationFixture {
t.Helper()
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
_, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
stays := filepath.Join(dir, "docker.conf")
if err := os.WriteFile(stays, []byte("kept\n"), 0o644); err != nil {
t.Fatal(err)
}
// The state as a node-engine that applied generation 7 keeps it: the file it owns, and the highest
// generation it applied. Written as JSON, so the fixture is the same bytes for any build under test.
state := map[string]any{"resources": []any{map[string]any{"id": "docker.conf", "type": "file",
"origin": store.OriginDeclared, "target": stays}}, "generation": generation}
raw, err := json.Marshal(state)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(opts.state, raw, 0o600); err != nil {
t.Fatal(err)
}
f := generationFixture{opts: opts, private: private, stays: stays}
f.kept = f.declared(t, map[string]any{"sequence": 3, "generation": generation}, "docker.conf", stays)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), f.kept); err != nil {
t.Fatal(err)
}
return f
}
// declared is a signed declaration with the envelope's keys given and one file.
func (f generationFixture) declared(t *testing.T, envelope map[string]any, id, path string) store.Declared {
t.Helper()
body := map[string]any{"declaration": 1,
"resources": []any{map[string]any{"id": id, "type": "file", "path": path, "content": "x\n"}}}
for k, v := range envelope {
body[k] = v
}
raw, err := json.Marshal(body)
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: raw, Signature: ed25519.Sign(f.private, raw)}
}
// reportJSON is a report as the controller reads it.
func reportJSON(t *testing.T, r link.Report) map[string]any {
t.Helper()
raw, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
var out map[string]any
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
return out
}
// **A newer sequence from an older generation is refused, and nothing is undeclared** (issue 234).
func TestANewerSequenceFromAnOlderGenerationIsRefusedAndNothingIsUndeclared(t *testing.T) {
f := machineThatApplied(t, 7)
other := filepath.Join(filepath.Dir(f.opts.state), "sudo.conf")
stale := f.declared(t, map[string]any{"sequence": 9, "generation": 6}, "sudo.conf", other)
report := applyAndKeep(context.Background(), f.opts, stale.Declaration, &stale, nil, nil)
said := reportJSON(t, report)
older, _ := said["older_generation"].(map[string]any)
if older == nil || older["generation"] != float64(6) || older["applied"] != float64(7) {
t.Fatalf("the refusal does not name the generation refused and the one applied: %v", said)
}
if report.Refused == "" || !strings.Contains(report.Refused, "generation 6") ||
!strings.Contains(report.Refused, "generation 7") {
t.Errorf("the refusal is not said in words naming both generations: %q", report.Refused)
}
if report.Declared != digestOf(stale.Declaration) || report.Sequence != 9 {
t.Errorf("the refusal does not name the declaration refused, so the mesh cannot name who sent it: %+v", report)
}
if _, err := os.Stat(f.stays); err != nil {
t.Fatalf("a refused declaration undeclared what the machine applied: %v", err)
}
if _, err := os.Stat(other); err == nil {
t.Error("the refused declaration touched the machine")
}
state, err := store.Load(f.opts.state)
if err != nil {
t.Fatal(err)
}
if _, owned := state.Find("docker.conf"); !owned {
t.Error("the refused declaration took a resource out of what this machine owns")
}
kept, err := store.ReadDeclared(store.DeclaredPath(f.opts.state))
if err != nil || string(kept.Declaration) != string(f.kept.Declaration) {
t.Error("the refused declaration replaced what this node kept, so the next reconcile would apply it")
}
}
// refusedForItsGeneration says a report refused its declaration for the generation it came from.
func refusedForItsGeneration(t *testing.T, r link.Report) bool {
t.Helper()
return reportJSON(t, r)["older_generation"] != nil || strings.Contains(r.Refused, "generation")
}
// refusedAsUnread says a report refused its declaration as one this node-engine cannot read.
func refusedAsUnread(r link.Report) bool { return strings.Contains(r.Refused, "not a declaration") }
// **A gate's put-back is applied, whatever generation it carries**: it carries the generation of what it
// puts back, and says it is a put-back (issue 234).
func TestAPutBackFromAnOlderGenerationIsNotRefused(t *testing.T) {
f := machineThatApplied(t, 7)
back := f.declared(t, map[string]any{"sequence": 9, "generation": 5, "put_back": true}, "docker.conf", f.stays)
report := applyAndKeep(context.Background(), f.opts, back.Declaration, &back, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a put-back was refused: %s", report.Refused)
}
}
// **A declaration that carries no generation is applied as it always was**: an older controller sends
// none, and a node-engine that refused it would strand every machine the moment the controller is older.
func TestADeclarationWithoutAGenerationIsNotRefusedForOne(t *testing.T) {
f := machineThatApplied(t, 7)
older := f.declared(t, map[string]any{"sequence": 9}, "docker.conf", f.stays)
report := applyAndKeep(context.Background(), f.opts, older.Declaration, &older, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a declaration without a generation was refused: %s", report.Refused)
}
// And the same generation again is no older.
same := f.declared(t, map[string]any{"sequence": 10, "generation": 7}, "docker.conf", f.stays)
report = applyAndKeep(context.Background(), f.opts, same.Declaration, &same, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a declaration of the generation applied was refused: %s", report.Refused)
}
}
+90
View File
@@ -0,0 +1,90 @@
package main
import (
"context"
"errors"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
)
// owedBus is a link that takes reports and statements and keeps a have bucket in memory.
type owedBus struct {
mu sync.Mutex
kv map[string][]byte
rv map[string]uint64
}
func (b *owedBus) Report(context.Context, string, []byte) error { return nil }
func (b *owedBus) Health(context.Context, string, []byte) error { return nil }
func (b *owedBus) HaveBind(context.Context) (link.HaveBucket, error) { return b, nil }
func (b *owedBus) Get(_ context.Context, key string) ([]byte, uint64, error) {
b.mu.Lock()
defer b.mu.Unlock()
v, ok := b.kv[key]
if !ok {
return nil, 0, link.ErrHaveAbsent
}
return v, b.rv[key], nil
}
func (b *owedBus) Set(_ context.Context, key string, value []byte, revision uint64) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.rv[key] != revision {
return errors.New("wrong last sequence")
}
b.kv[key], b.rv[key] = value, revision+1
return nil
}
func (b *owedBus) have(key string) (delivery.Have, bool) {
b.mu.Lock()
defer b.mu.Unlock()
h, err := delivery.DecodeHave(b.kv[key])
return h, err == nil
}
// A module with nothing long-running (xorg) is ok within one look of its apply, not at the five-minute statement: the
// facts look inside the apply runs while it is applying and judges nothing, and the running write then owes a look,
// which the health loop makes at once (review of fix 15, B1).
func TestAModuleWithNothingLongRunningIsOkWithinALookOfItsApply(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
bus := &owedBus{kv: map[string][]byte{}, rv: map[string]uint64{}}
queue := &link.Session{Membership: link.Membership{Node: "n1"}, Timeout: time.Second}
queue.Attach(ctx, bus)
j, err := liveness.Open(filepath.Join(t.TempDir(), "liveness.json"), nil)
if err != nil {
t.Fatal(err)
}
fact := link.ModuleFact{Module: "xorg", FP: "src2:" + strings.Repeat("a", 64), Composed: "sha256:" + strings.Repeat("b", 64)}
applying := fact
applying.Phase = string(delivery.PhaseApplying)
queue.NoteModule(ctx, applying)
go judgeWhatRuns(ctx, j, queue, func(string) {}, nil)
// The facts look inside the apply, while the module is still applying.
queue.SayFacts(ctx, link.Report{Health: &link.Health{Contract: link.LivenessContract, At: time.Now().UTC()}})
if h, _ := bus.have("n1.xorg"); h.Health != delivery.HealthUnknown {
t.Fatalf("judged while applying: %+v", h)
}
queue.NoteModule(ctx, fact)
deadline := time.Now().Add(2 * time.Second) // well within one look (liveness.LookEvery, 15 s)
for {
h, _ := bus.have("n1.xorg")
if h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
return
}
if time.Now().After(deadline) {
t.Fatalf("not ok within a look of its apply: %+v", h)
}
time.Sleep(10 * time.Millisecond)
}
}
+2 -7
View File
@@ -4,7 +4,6 @@ import (
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
@@ -13,16 +12,12 @@ import (
// judgeDirectories judges every directory of the declaration this node keeps, read-only (novox/hq issue 358): which
// the walk that follows no link another account could make would refuse. Nothing is made, chmodded or chowned.
func judgeDirectories(statePath string) (string, error) {
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
d, err := declaredNow(statePath, nil)
if errors.Is(err, store.ErrNothingDeclared) {
return "", errors.New("this node keeps no declaration, so it is declared no directory")
}
if err != nil {
return "", err
}
d, err := declaration.Parse(kept.Declaration)
if err != nil {
return "", fmt.Errorf("the declaration this node keeps does not read: %w", err)
return "", fmt.Errorf("what runs on this node does not read: %w", err)
}
return apply.JudgeDirectories(d), nil
}
+2 -6
View File
@@ -251,16 +251,12 @@ func startKeeping(ctx context.Context, say func(string)) {
// declaration this node keeps, and run beside the applies.
func accessActFrom(statePath string, timeout time.Duration) link.AccessActFunc {
return func(act link.AccessAct) (string, error) {
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
d, err := declaredNow(statePath, nil)
if errors.Is(err, store.ErrNothingDeclared) {
return "", errors.New("this node keeps no declaration, so it is declared no library")
}
if err != nil {
return "", err
}
d, err := declaration.Parse(kept.Declaration)
if err != nil {
return "", fmt.Errorf("the declaration this node keeps does not read: %w", err)
return "", fmt.Errorf("what runs on this node does not read: %w", err)
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
-29
View File
@@ -1,29 +0,0 @@
package main
import (
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/outward"
"github.com/novox/mesh-host/internal/profile"
)
func reportOn(networks ...outward.Network) link.Report {
return link.Report{Outward: []string{"eth0"},
Profile: profileAsReported(profile.Profile{Machine: &profile.Machine{Networks: networks}})}
}
// **A change of the networks a machine is attached to is said unasked** (novox/hq ADR 0340 §7): a port
// reached from the local network is admitted from them, so a laptop that joins another network is said at
// the next reconcile, not at the next push — and the same networks again are not news.
func TestAChangedNetworkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
home := reportOn(outward.Network{Link: "eth0", Network: "192.168.1.0/24"})
w.said(home)
if w.differs(reportOn(outward.Network{Link: "eth0", Network: "192.168.1.0/24"})) {
t.Fatal("the same networks read as a change, so the machine would speak on every reconcile")
}
if !w.differs(reportOn(outward.Network{Link: "eth0", Network: "10.20.0.0/16"})) {
t.Fatal("a machine that moved to another network does not say so until it is pushed to")
}
}
-107
View File
@@ -1,107 +0,0 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/link"
)
// Defends novox/hq issue 160: a machine nobody pushed to still says what it is made of once a day, so the
// newest of it the mesh holds is no older than a day. Before, the reconcile said it only with something
// else (a hold, a firewall, an outward link changed), and a converged machine's picture was as old as the
// last delivery to it.
func TestAMachineNobodyPushedToSaysWhatItIsOnceADay(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
quiet := link.Report{Outward: []string{"wlan0"},
Profile: map[string]any{"machine": map[string]any{"observed_at": at}}}
if !w.news(quiet) {
t.Fatal("the first reconcile after the engine started did not say what the machine is")
}
w.said(quiet)
for _, after := range []time.Duration{5 * time.Minute, 12 * time.Hour, 22*time.Hour + 55*time.Minute} {
at = time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC).Add(after)
if w.news(quiet) {
t.Errorf("%s after the mesh heard it, an unchanged reconcile was said again", after)
}
}
// The reconcile that falls past 23 hours says it, so it reaches the mesh inside the day.
at = time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC).Add(23*time.Hour + 5*time.Minute)
if !w.news(quiet) {
t.Fatal("a day on, the reconcile did not say what the machine is")
}
w.said(quiet)
at = at.Add(5 * time.Minute)
if w.news(quiet) {
t.Error("once said, it was said again at the next reconcile")
}
}
// A delivery's account carries the profile too: heard, it starts the day again.
func TestADeliveryHeardStartsTheDayAgain(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
report := link.Report{Outward: []string{"eth0"}, Profile: map[string]any{"capabilities": []any{}}}
w.said(report)
at = at.Add(20 * time.Hour)
w.said(report) // a delivery's apply
at = at.Add(20 * time.Hour)
if w.news(report) {
t.Error("said again unasked 20 hours after a delivery told the mesh")
}
}
// A refusal carries nothing about the machine, so it is never said for being a day old.
func TestARefusalIsNotSaidForTheMachinesSake(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
if w.machineDue(link.Report{Refused: "no"}) {
t.Error("a refusal was due for what the machine is made of")
}
}
// A laptop that slept still says what it is within a day of wall time (review of mesh-host#92). Go's monotonic
// clock stops while a machine is suspended, and Sub between two times that both carry a monotonic reading uses
// it, so a watch that kept one would count a day asleep as almost nothing. Simulated with a clock that first
// reads now (a monotonic reading), and after the sleep reads the wall time a day on: the watch must keep no
// monotonic reading, and judge the day as the wall clock says it.
func TestALaptopThatSleptStillSaysWhatItIsWithinADay(t *testing.T) {
before := time.Now() // carries a monotonic reading
asleep := before.Add(3 * time.Second)
if asleep.Sub(before) != 3*time.Second {
t.Fatal("the monotonic reading is not what this test simulates with")
}
calls := 0
w := &adoptionWatch{now: func() time.Time {
calls++
if calls == 1 {
return before
}
// The wall clock a day on, as a machine's clock reads after waking.
return asleep.Round(0).Add(24 * time.Hour)
}}
report := link.Report{Outward: []string{"wlan0"}, Profile: map[string]any{"capabilities": []any{}}}
w.said(report)
if strings.Contains(w.machineSaid.String(), "m=") {
t.Fatalf("the watch kept a monotonic reading (%s): a sleep would not count toward the day", w.machineSaid)
}
if !w.news(report) {
t.Error("a day of wall time on, a laptop that slept did not say what it is")
}
}
// And the watch's own clock never carries a monotonic reading.
func TestTheWatchClockIsWallTime(t *testing.T) {
w := &adoptionWatch{}
if s := w.clock().String(); strings.Contains(s, "m=") {
t.Errorf("the watch's clock carries a monotonic reading: %s", s)
}
w = &adoptionWatch{now: time.Now}
if s := w.clock().String(); strings.Contains(s, "m=") {
t.Errorf("a given clock's time kept its monotonic reading: %s", s)
}
}
+205 -703
View File
File diff suppressed because it is too large Load Diff
+4 -295
View File
@@ -3,7 +3,6 @@ package main
import (
"bytes"
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
@@ -14,8 +13,6 @@ import (
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/bundle"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
@@ -98,26 +95,6 @@ func TestNoCommandIsNotAnError(t *testing.T) {
}
}
func TestApplyNeedsExactlyOneDeclaration(t *testing.T) {
// `apply` takes a file where every other command takes nothing, so the leftover-argument
// rule has an exception — and an exception is where a parser stops refusing things it
// should. Both directions are checked.
if _, _, err := parseArgs([]string{"apply"}); err == nil {
t.Error("apply with no file was accepted")
}
if _, _, err := parseArgs([]string{"apply", "a.json", "b.json"}); err == nil {
t.Error("apply with two files was accepted")
}
command, opts, err := parseArgs([]string{"apply", "decl.json", "--dry-run"})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if command != "apply" || opts.file != "decl.json" || !opts.dryRun {
t.Errorf("parsed as command=%q file=%q dry-run=%v", command, opts.file, opts.dryRun)
}
}
func TestTheStateHasADocumentedDefault(t *testing.T) {
// A host that wrote its state somewhere unexpected would forget what it owns on the next
// run, and then leave everything it had applied behind forever.
@@ -135,93 +112,22 @@ func TestAFlagAfterAPositionalIsRead(t *testing.T) {
// off the front fixed the flag after the COMMAND and not the flag after its ARGUMENT: the
// standard library stops at the first non-flag argument wherever that argument is.
for _, args := range [][]string{
{"apply", "decl.json", "--dry-run", "--json"},
{"apply", "--dry-run", "decl.json", "--json"},
{"apply", "--dry-run", "--json", "decl.json"},
{"hand-over", "decl.json", "--dry-run", "--json"},
{"hand-over", "--dry-run", "decl.json", "--json"},
{"hand-over", "--dry-run", "--json", "decl.json"},
} {
command, opts, err := parseArgs(args)
if err != nil {
t.Errorf("%v: unexpected error: %v", args, err)
continue
}
if command != "apply" || opts.file != "decl.json" || !opts.dryRun || !opts.json {
if command != "hand-over" || opts.file != "decl.json" || !opts.dryRun || !opts.json {
t.Errorf("%v parsed as file=%q dry-run=%v json=%v",
args, opts.file, opts.dryRun, opts.json)
}
}
}
// Defends novox/hq ADR 0100: a reconcile on an adopted node speaks unasked only when what it holds
// or its firewall changed — which is how a predecessor still writing is caught, without a report
// every five minutes saying nothing new.
func TestAReconcileSpeaksOnlyWhenWhatIsHeldChanged(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page"}, {ID: "hello-web.server"}}}
if !w.changed(held) {
t.Fatal("the first report of a hold was not said")
}
again := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.server"}, {ID: "hello-web.page"}}}
if w.changed(again) {
t.Error("the same holds in another order were said again")
}
rewritten := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}, {ID: "hello-web.server"}}}
if !w.changed(rewritten) {
t.Error("a held file rewritten by something else was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a changed firewall was not said")
}
// What filters the machine is part of it (novox/hq ADR 0168): a predecessor's chain removed by
// hand, or the found firewall enabled again, is said without being asked.
filtered := link.Report{Firewall: "none", Held: rewritten.Held,
Filters: []link.Filter{{Where: "chain HAL-MESH-ONLY (iptables-legacy)", Owner: "other", Refuses: "-j DROP"}}}
if !w.changed(filtered) {
t.Error("a filter appearing was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a filter removed was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held, FoundFirewall: &link.FoundFirewall{Kind: "ufw", Active: true}}) {
t.Error("the found firewall coming back was not said")
}
if !worthSaying(link.Report{Filters: filtered.Filters}) {
t.Error("a report carrying only what filters the machine is not worth saying")
}
}
// What the mesh heard — a delivery's account as much as a reconcile's — counts as said: the queue
// tells the watch every report the broker took (link.Queue.Heard).
func TestWhatTheMeshHeardCountsAsSaid(t *testing.T) {
w := &adoptionWatch{}
report := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "a"}}}
heard := w.said
heard(report)
if w.changed(report) {
t.Error("a reconcile repeated what the mesh had just heard")
}
}
func TestAReconcileSpeaksWhenWhatIsReachableChanged(t *testing.T) {
// The controller previews a flip from what the node last said is reachable; a port that opened
// since must reach it without waiting for the next delivery (novox/hq ADR 0100).
w := &adoptionWatch{}
before := link.Report{Firewall: "ufw", Reachable: []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}}
if !w.changed(before) {
t.Fatal("the first report was not said")
}
reordered := link.Report{Firewall: "ufw", Reachable: []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}}
if w.changed(reordered) {
t.Error("the same reachable set was said again")
}
opened := link.Report{Firewall: "ufw", Reachable: append(before.Reachable,
link.Reach{Protocol: "tcp", Address: "0.0.0.0", Port: 8080, By: "hello-web", Published: true, ContainerPort: 80})}
if !w.changed(opened) {
t.Error("a newly published port was not said")
}
}
// Defends the node's own record: the link and the reconcile loop both apply, and each reads the
// state, acts, and writes it back — so they must not run at the same time, or the last save loses
// what the other recorded.
@@ -270,24 +176,6 @@ func TestOnlyOneApplyRunsAtATime(t *testing.T) {
}
}
// Defends novox/hq ADR 0100: a change is counted as said only once the mesh has been told. Queued
// and lost — the link down when the reconcile spoke — it must be said again.
func TestAChangeThatNeverReachedTheMeshIsSaidAgain(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}}}
if !w.differs(held) {
t.Fatal("the first report of a change was not new")
}
// The link was down: nothing published it, so nothing says it was said.
if !w.differs(held) {
t.Error("a change that never reached the mesh was counted as said")
}
w.said(held)
if w.differs(held) {
t.Error("a change the mesh was told was said again")
}
}
// Defends novox/hq issue 104: a declaration for the other mode than this node is in is refused at
// the point of application, whichever command delivered it, naming both — an adopted control-node
// once applied its converged genesis bundle and closed itself for forty-five minutes.
@@ -468,125 +356,6 @@ func TestADryRunChangesNothingAndListsTheActions(t *testing.T) {
}
}
// Defends novox/hq issue 104: once the mesh has told this node anything, `reconcile` holds it to
// that — never to the bundle the host carries, which genesis consumed.
func TestReconcileAfterAControllerDeclarationDoesNotReapplyTheBundle(t *testing.T) {
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
opts := stateWithMode(t, store.ModeAdopted)
controllerPublic, controllerPrivate, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
said := []byte(`{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), store.Declared{Declaration: said,
Signature: ed25519.Sign(controllerPrivate, said)}); err != nil {
t.Fatal(err)
}
// Told, and unable to prove by whom: refused, and the bundle is not applied in its place.
_, _, _, err = reconcileSource(opts)
if err == nil || !strings.Contains(err.Error(), "not applied in its place") {
t.Errorf("a node that cannot prove what it was told fell back to something: %v", err)
}
mine, err := identity.Generate("workstation")
if err != nil {
t.Fatal(err)
}
mine.Membership = identity.Membership{Broker: "198.51.100.10:5671", Fingerprint: "sha256:0",
Signer: controllerPublic, Password: "issued"}
if err := identity.Save(identity.Path(opts.state), mine); err != nil {
t.Fatal(err)
}
d, raw, from, err := reconcileSource(opts)
if err != nil {
t.Fatal(err)
}
if from != fromDeclared || !bytes.Equal(raw, said) || d.Adoption == nil {
t.Errorf("reconcile chose %d with %d bytes, not what the mesh last said", from, len(raw))
}
// And before the mesh has said anything: the bundle, as it always was. A test binary carries
// only the placeholder, and asking for it is what proves the path.
if err := os.Remove(store.DeclaredPath(opts.state)); err != nil {
t.Fatal(err)
}
_, _, _, err = reconcileSource(opts)
if !errors.Is(err, bundle.ErrEmpty) {
t.Errorf("with nothing said, reconcile did not reach for the carried bundle: %v", err)
}
}
// **A machine says which links face outside without being asked.**
//
// The mesh composes no filter for a machine that has not said (novox/hq ADR 0140), and a machine only
// speaks unasked when this fingerprint changes. Left out of it, a machine that has just learnt to say
// could speak only when a declaration arrived — and a declaration cannot be composed until it has
// spoken. A machine waiting for a push that is waiting for the machine.
func TestANewOutwardLinkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
first := link.Report{Firewall: "none"}
if !w.differs(first) {
t.Fatal("the first report should differ from nothing")
}
w.said(first)
// Only the links changed, and nothing about adoption.
learnt := link.Report{Firewall: "none", Outward: []string{"eth0"}}
if !w.differs(learnt) {
t.Fatal("a machine that has just learnt which links face outside would never say so, " +
"and could then never be sent a filter")
}
w.said(learnt)
if w.differs(link.Report{Firewall: "none", Outward: []string{"eth0"}}) {
t.Fatal("the same links are reported as a change, so the machine would speak on every reconcile")
}
// And a link that changes — a laptop moving from a cable to a radio — is said too, because the
// filter is written around the old one until it is.
if !w.differs(link.Report{Firewall: "none", Outward: []string{"wlan0"}}) {
t.Fatal("a changed outward link is not said, so the filter stays written around the old one")
}
}
// **A converged machine can say which links face outside, unasked.**
//
// A reconcile is otherwise silent, and the condition deciding when it speaks asked only what an
// adopted node reports — what it holds, and the firewall it found. A converged node has neither, so
// it could speak only in reply to a declaration, and the mesh composes no declaration for a node
// that has not said which links face outside (novox/hq ADR 0140). Measured: three converged machines
// sat silent while the control plane refused to send them a filter.
func TestAConvergedMachineSaysItsOutwardLinksUnasked(t *testing.T) {
// A converged node's reconcile: nothing held, no found firewall, and the links it can see.
converged := link.Report{Outward: []string{"eth0"}}
if !worthSaying(converged) {
t.Fatal("a converged machine cannot say which links face outside, so it can never be " +
"sent a filter — a machine waiting for a push that is waiting for the machine")
}
// An adopted node's reasons still hold, because that is how a predecessor still writing is caught.
if !worthSaying(link.Report{Firewall: "ufw"}) {
t.Fatal("an adopted machine no longer says which firewall it found")
}
if !worthSaying(link.Report{Held: []link.Held{{ID: "a-file"}}}) {
t.Fatal("an adopted machine no longer says what it holds")
}
// And a reconcile with nothing to say stays silent, or every machine speaks every five minutes
// about nothing.
if worthSaying(link.Report{}) {
t.Fatal("a reconcile with nothing to say speaks anyway")
}
// A refused report says nothing about the machine; the refusal is for the console.
if worthSaying(link.Report{Outward: []string{"eth0"}, Refused: "not for this node"}) {
t.Fatal("a refused report is offered as news about the machine")
}
}
// **A host running as a service says what its apply did.**
//
// The serving path passed nil where the apply writes its detail, and nil is silence. The one-shot path
@@ -611,63 +380,3 @@ func TestTheApplysLogIsNeverNil(t *testing.T) {
t.Fatalf("the apply's detail did not reach the caller's announce: %v", said)
}
}
// Defends novox/hq issues 257 and 261: a reconcile due while the link applies a newer declaration
// applies that newer one once its turn comes — never the one kept when its timer fired. Read before
// waiting, it gave back a module assigned a second earlier, and reported a declaration the mesh no
// longer recorded as sent.
func TestAReconcileAppliesWhatWasKeptWhenItsTurnComes(t *testing.T) {
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "a.conf")
kept := func(content string) store.Declared {
body := []byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"` + target +
`","content":"` + content + `\n"}]}`)
return store.Declared{Declaration: body, Signature: ed25519.Sign(private, body)}
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state), kept("older")); err != nil {
t.Fatal(err)
}
// The link is applying: the reconcile's timer fires and it waits its turn.
applying.Lock()
done := make(chan error, 1)
go func() {
_, err := reapplyKept(context.Background(), opts, public, nil, nil)
done <- err
}()
time.Sleep(50 * time.Millisecond)
// The link's apply ends by keeping the newer declaration, and lets go.
if err := store.SaveDeclared(store.DeclaredPath(opts.state), kept("newer")); err != nil {
applying.Unlock()
t.Fatal(err)
}
applying.Unlock()
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(10 * time.Second):
t.Fatal("the reconcile never ran once the node was free")
}
got, err := os.ReadFile(target)
if err != nil {
t.Fatal(err)
}
if string(got) != "newer\n" {
t.Errorf("the reconcile applied %q, what was kept before the newer declaration", got)
}
}
+2 -2
View File
@@ -11,14 +11,14 @@ import (
// serveMeshCLI answers mesh-cli on this machine for as long as this engine serves (novox/hq ADR 0272): the account
// from the kernel, the line asked of the controller over the link open at the time. A socket that cannot be served
// is said once, and this engine goes on applying; mesh-cli then says the engine does not answer.
func serveMeshCLI(ctx context.Context, queue *link.Queue, say func(string)) {
func serveMeshCLI(ctx context.Context, queue *link.Session, say func(string)) {
if err := meshcli.Serve(ctx, cli.Socket, forwardOver(queue), say); err != nil {
say("mesh-cli is not answered on this machine: " + err.Error())
}
}
// forwardOver asks the controller over the queue's link, and follows the line to what came of it.
func forwardOver(queue *link.Queue) meshcli.Forward {
func forwardOver(queue *link.Session) meshcli.Forward {
return func(ctx context.Context, asked meshcli.Asked) cli.Answer {
return meshcli.AskTheController(ctx, queue.AskController, asked)
}
+1 -1
View File
@@ -27,7 +27,7 @@ type vaultAsk func(verb string, args any) (json.RawMessage, error)
// moveSecretFrom is the engine's move: this machine's sealing key, read for each ask, and the vault asked over
// the link open now.
func moveSecretFrom(statePath string, queue *link.Queue) link.MoveSecretFunc {
func moveSecretFrom(statePath string, queue *link.Session) link.MoveSecretFunc {
return func(m link.MoveSecret) (link.MovedSecret, error) {
key, err := identity.LoadSealingKey(identity.SealingKeyPath(statePath))
if err != nil {
+15
View File
@@ -0,0 +1,15 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// The checker runs gofmt, go vet and both test levels by Go's convention, and the build module starts the bus the
// integration level needs (novox/hq ADR 0529): no script of the repository's own runs beside them.
func TestTheRepositoryHoldsNoCheckScript(t *testing.T) {
if _, err := os.Stat(filepath.Join("..", "..", "merge-check.sh")); err == nil {
t.Fatal("merge-check.sh is back: the checker's levels run the tests")
}
}
+750
View File
@@ -0,0 +1,750 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The node agent's wiring (novox/hq ADR 0429 §4, ADR 0539 §9): the buckets it watches, the haves it writes, the backup
// it asks and the apply of one module's composition.
// bucketWatch is a bucket of the bus as the agent watches it, through the session's link.
type bucketWatch struct {
session *link.Session
bucket string
}
func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Update, error) {
in, err := w.session.KVWatch(ctx, w.bucket, pattern)
if err != nil {
return nil, err
}
out := make(chan agent.Update, 64)
go func() {
defer close(out)
for u := range in {
select {
case out <- agent.Update{Key: u.Key, Value: u.Value, Revision: u.Revision, Deleted: u.Deleted, Caught: u.Caught}:
case <-ctx.Done():
return
}
}
}()
return out, nil
}
// sessionHaves writes the haves the agent says, through the session.
type sessionHaves struct{ session *link.Session }
func (h sessionHaves) Read(ctx context.Context, module string) (agent.Have, bool) {
got, ok := h.session.ReadHave(ctx, module)
if !ok {
return agent.Have{}, false
}
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail}, true
}
func (h sessionHaves) Write(ctx context.Context, x agent.Have) {
if x.Module == agent.NodeModule {
// The node's own key holds its facts and the digest of the composition it runs.
if x.Phase == "" || x.Phase == string(delivery.PhaseRunning) {
h.session.NoteNode(ctx, x.Composed)
}
return
}
phase := x.Phase
if phase == string(delivery.PhaseRunning) {
phase = ""
}
h.session.NoteModule(ctx, link.ModuleFact{Module: x.Module, FP: x.FP, Composed: x.Composed, Phase: phase,
Failed: phase == string(delivery.PhaseFailed), Detail: shorten(x.Detail)})
}
func (h sessionHaves) Removed(ctx context.Context, module string) {
h.session.NoteRemoved(ctx, module, "")
}
// shorten keeps a have's detail within what a have value holds: its head, which says what failed, and its tail, which is
// the cause (an error wrapped five times names the bus's "no responders" only at its end: 2026-10-25 23:04, mesh-vault on
// the bus node said only which object it read).
func shorten(s string) string {
const most, head = 300, 100
if len(s) <= most {
return s
}
const gap = " … "
tail := s[len(s)-(most-head-len(gap)):]
return strings.ToValidUTF8(s[:head], "") + gap + strings.ToValidUTF8(tail, "")
}
// failedWith is a failed apply that says why, transient when the words are the bus not answering for a moment
// (link.TransientWords): the agent then tries it again without a new revision.
func failedWith(why string) agent.Result {
return agent.Result{Failed: true, Transient: link.TransientWords(why), Detail: shorten(why)}
}
// backupWithin is how long a backup is given from its ask to say it is done (ADR 0429 §14).
var backupWithin = 10 * time.Minute
// backedUpEvent is where the holder of node-backup says a backup taken by hand ended, also one that could not start.
const backedUpEvent = "mesh.mod.restic.event.backed-up"
// backupNode asks the holder of node-backup on this node to back a module up now, and waits for its event to say how it
// went: the holder answers `now` when it has started, and says `backed-up` when the backup ends.
func backupNode(session *link.Session) func(ctx context.Context, module string) agent.Backup {
return func(ctx context.Context, module string) agent.Backup {
// Heard before the ask, so an event that comes at once is not lost.
events, stop, err := session.Hear(backedUpEvent)
if err != nil {
return agent.Backup{Words: "the end of the backup cannot be heard: " + err.Error()}
}
defer stop()
// A module this machine backs nothing of up moves without a backup: never held for one that cannot run (issue
// 1945). An answer that cannot be had or read asks for the backup as before.
if said, err := session.AskSeat(ctx, "node-backup", "backed-up", map[string]any{"module": module}, 30*time.Second); err == nil {
if nothing, read := declaresNothingToBackUp(said, module); read && nothing {
return agent.Backup{Done: true, Words: module + " declares nothing to back up on this machine"}
}
}
asked := time.Now()
if _, err := session.AskSeat(ctx, "node-backup", "now", map[string]any{"module": module}, 30*time.Second); err != nil {
return agent.Backup{Words: err.Error()}
}
return awaitBackup(ctx, events, session.Membership.Node, module, asked, backupWithin)
}
}
// awaitBackup waits for the event that says the backup of a module on this node ended, within the bound: done when it
// says ok, held on its words when it does not, and held when it says nothing within the bound. An event of another node
// or module, or from before the ask, is not the answer.
func awaitBackup(ctx context.Context, events <-chan []byte, node, module string, since time.Time, within time.Duration) agent.Backup {
bound := time.NewTimer(within)
defer bound.Stop()
for {
select {
case <-ctx.Done():
return agent.Backup{Words: "the node agent was stopped while the backup ran"}
case <-bound.C:
return agent.Backup{Words: fmt.Sprintf("the backup did not say it was done within %s", within)}
case raw := <-events:
var e struct {
Node string `json:"node"`
Module string `json:"module"`
OK bool `json:"ok"`
At time.Time `json:"at"`
Error string `json:"error"`
}
if json.Unmarshal(raw, &e) != nil || e.Node != node || e.Module != module || e.At.Before(since.Add(-time.Second)) {
continue
}
if e.OK {
return agent.Backup{Done: true}
}
words := "the backup failed"
if e.Error != "" {
words += ": " + e.Error
}
return agent.Backup{Words: words}
}
}
}
// declarationOf is the declaration the compositions make together: their resources, and the node's own parts when the
// node's composition is among them. It is what the node judges and says its health by, and what one module's apply is
// made of (with a single composition).
//
// **A resource the node's own composition shares with a module's is the module's** (issue 1945): from 2026-10-25 12:43
// the controller put modules it knew no fingerprint of the new form for into the node's own composition while their
// wants named their own, and the declaration joined of both declared each such file twice, which the filling refuses
// ("2 resources declare X") on every node. The node's own copy is left out, and which were left out is said once.
func declarationOf(cs ...delivery.Composed) (*declaration.Declaration, error) {
d := &declaration.Declaration{Version: declaration.Version}
var modules []delivery.Composed
for _, c := range cs {
if c.Module != "" {
modules = append(modules, c)
}
}
if len(modules) > 0 {
joined := make([]delivery.Composed, 0, len(cs))
for _, c := range cs {
if c.Module == "" {
var dropped []string
c, dropped = ownWithout(c, modules)
sayDroppedOnce(dropped)
}
joined = append(joined, c)
}
cs = joined
}
for _, c := range cs {
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
return nil, fmt.Errorf("a resource of %q: %w", c.Module, err)
}
d.Resources = append(d.Resources, r)
}
d.Filled = append(d.Filled, c.Filled...)
if c.Module == "" {
d.Adoption = c.Own.Adoption
d.Identities = append(d.Identities, c.Own.Identities...)
d.Groups = append(d.Groups, c.Own.Groups...)
d.Libraries = append(d.Libraries, c.Own.Libraries...)
d.Provisioner = c.Own.Provisioner
}
}
return d, nil
}
// ownWithout is the node's own composition without the resources a module's composition among modules also declares:
// the same kind and id. A resource that shares only its target (a unit or container of the same name, another id) is
// not the module's, and stays: dropped, its record would be an orphan the apply removes, stopping the module's unit. It
// answers what it left out, as "<id> (<module>)".
func ownWithout(own delivery.Composed, modules []delivery.Composed) (delivery.Composed, []string) {
byID := map[string]string{}
for _, c := range modules {
if c.Module == "" {
continue
}
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
continue
}
byID[string(r.Kind())+"\x00"+r.Identity()] = c.Module
}
}
if len(byID) == 0 {
return own, nil
}
var dropped []string
kept := make([]json.RawMessage, 0, len(own.Resources))
for _, raw := range own.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
kept = append(kept, raw) // refused where it is parsed, with its words
continue
}
module, dup := byID[string(r.Kind())+"\x00"+r.Identity()]
if dup {
dropped = append(dropped, fmt.Sprintf("%s (%s)", r.Identity(), module))
continue
}
kept = append(kept, raw)
}
if len(dropped) == 0 {
return own, nil
}
own.Resources = kept
return own, dropped
}
// droppedSaid is every set of the node's own resources left out that was said, so each is said once.
var droppedSaid sync.Map
// sayDroppedOnce says once which of the node's own resources were left out because a module's composition declares them.
func sayDroppedOnce(dropped []string) {
if len(dropped) == 0 {
return
}
sort.Strings(dropped)
key := strings.Join(dropped, ", ")
if _, said := droppedSaid.LoadOrStore(key, true); said {
return
}
fmt.Fprintf(os.Stderr, "mesh-host: the node's own composition also declares %d resource(s) a module's composition "+
"declares, and they are left out of it: %s (issue 1945)\n", len(dropped), key)
}
func scopeName(c delivery.Composed) string {
if c.Module == "" {
return agent.NodeModule
}
return c.Module
}
func resourcesOf(cs []delivery.Composed) []declaration.Resource {
var out []declaration.Resource
for _, c := range cs {
for _, raw := range c.Resources {
if r, err := declaration.ParseResource(raw); err == nil {
out = append(out, r)
}
}
}
return out
}
// operatorKeysFile is the file beside the node's state that holds the operator's public keys, one ed25519 key in base64 a
// line (a line that starts with # is a note). The switch's last declaration writes it as a file resource, beside the
// controller's key the node was enrolled with (novox/hq ADR 0539 §3, §11); a node adopted later has it at its adoption.
const operatorKeysFile = "operator-keys"
// trustedKeys are the keys a composition may be signed by: the controller's, which the node was enrolled with, and the
// operator's, read from operatorKeysFile each time it is asked, so a key that arrives while the agent runs is trusted at
// the next composition. A file that cannot be read trusts the controller's alone, and says so.
func trustedKeys(opts options, controller ed25519.PublicKey) func() []ed25519.PublicKey {
return func() []ed25519.PublicKey {
keys := []ed25519.PublicKey{}
if len(controller) == ed25519.PublicKeySize {
keys = append(keys, controller)
}
raw, err := os.ReadFile(filepath.Join(filepath.Dir(opts.state), operatorKeysFile))
if err != nil {
return keys
}
for _, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
key, err := base64.StdEncoding.DecodeString(line)
if err != nil || len(key) != ed25519.PublicKeySize {
fmt.Fprintf(os.Stderr, "mesh-host: a line of %s is not an ed25519 public key in base64, and is not trusted\n", operatorKeysFile)
continue
}
keys = append(keys, ed25519.PublicKey(key))
}
return keys
}
}
// keptOnce reads the kept declaration once and splits it by module: its resources (the node's own are those that say
// none, ADR 0539 §9) and its filled paths.
func keptOnce(opts options, signer ed25519.PublicKey) func() (map[string][]json.RawMessage, map[string][]string) {
var once sync.Once
var by map[string][]json.RawMessage
var filled map[string][]string
return func() (map[string][]json.RawMessage, map[string][]string) {
once.Do(func() {
raw, err := store.LoadDeclared(store.DeclaredPath(opts.state), signer)
if err != nil {
return
}
by, filled = splitKept(raw)
})
return by, filled
}
}
// splitKept is a kept declaration's resources by the module they say (the node's own under agent.NodeModule) and its
// filled paths by the module whose file is at each path; nil for a declaration that does not read.
func splitKept(raw []byte) (map[string][]json.RawMessage, map[string][]string) {
var kept struct {
Resources []json.RawMessage `json:"resources"`
Filled []string `json:"filled_files"`
}
if json.Unmarshal(raw, &kept) != nil {
return nil, nil
}
isFilled := make(map[string]bool, len(kept.Filled))
for _, p := range kept.Filled {
isFilled[p] = true
}
by, filled := map[string][]json.RawMessage{}, map[string][]string{}
named := map[string]bool{}
for _, r := range kept.Resources {
var head struct {
Module string `json:"module"`
Type string `json:"type"`
Path string `json:"path"`
}
_ = json.Unmarshal(r, &head)
name := head.Module
if name == "" {
name = agent.NodeModule
}
by[name] = append(by[name], r)
if head.Type == string(declaration.TypeFile) && isFilled[head.Path] && !named[name+"\x00"+head.Path] {
named[name+"\x00"+head.Path] = true
filled[name] = append(filled[name], head.Path)
}
}
return by, filled
}
// nodeAgent runs the node agent under the session's links for as long as ctx lasts: it waits for a link, follows the
// node's wants and compositions on it, and begins again with the next link. A watch it cannot make (a bucket this node's
// account may not read) is said, and tried again with a growing wait, as the link is.
func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, session *link.Session, sched *apply.Scheduler,
say link.Announce, afterApply func(), current *agentNow, sayFacts func(ctx context.Context, ag *agent.Agent)) {
wait := 2 * time.Second
for ctx.Err() == nil {
linked, cancel, err := session.WhenLinked(ctx)
if err != nil {
return
}
began := time.Now()
node := session.Membership.Node
composedBucket, err := delivery.ComposedBucketFor(node)
if err != nil {
cancel()
say("the node agent cannot name this node's composition bucket: " + err.Error())
return
}
var ag *agent.Agent
kept := keptOnce(opts, signer)
// The bus's user list, written alone from the newest composition of the bus module (ADR 0547 §8): on every
// node, and a node whose compositions carry no list writes nothing.
users := userListFollower(opts, session, composedBucket, trustedKeys(opts, signer), say, func() {
// This engine wrote the list, so the bus reloads its accounts and answers no JetStream request for a few
// seconds: what failed in them is tried again once that is over, not at its next wait.
time.AfterFunc(reloadSettles, func() {
if now := current.get(); now != nil {
now.RetryTransient()
}
})
})
// A vault key heard (or heard again changed) looks at the list again, so its hashes are handed (ADR 0547 §9).
vaultKey.following(users.Again)
go func() {
if err := users.Run(linked); err != nil && linked.Err() == nil {
say("the bus's user list is not followed on this link: " + err.Error())
}
}()
env := agent.Env{
Node: node,
Wants: bucketWatch{session, delivery.AssignerWantsBucket},
Composed: bucketWatch{session, composedBucket},
Haves: sessionHaves{session},
Kept: func() map[string][]json.RawMessage { by, _ := kept(); return by },
// The same kept declaration's filled paths, so a module still running from it keeps its files filled.
KeptFilled: func() map[string][]string { _, filled := kept(); return filled },
Recorded: recordedIn(opts.state, say),
Revisions: func(ctx context.Context, key string) ([]agent.Update, error) {
got, err := session.KVHistory(ctx, composedBucket, key)
if err != nil {
return nil, err
}
out := make([]agent.Update, 0, len(got))
for _, u := range got {
out = append(out, agent.Update{Key: u.Key, Value: u.Value, Revision: u.Revision, Deleted: u.Deleted})
}
return out, nil
},
Trusted: trustedKeys(opts, signer),
Backup: backupNode(session),
// What the filling refused before this apply may be named now: the provisioner's client is told to fill
// again (ace, 2026-10-25 21:17).
Settled: func(module string) { knockFilling(module, say) },
Say: say,
// A failure of the bus not answering for a moment is tried again: 2 s, 5 s, 15 s, 60 s, then every 5 min.
Backoff: link.Backoff,
Apply: func(ctx context.Context, c delivery.Composed, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := applyComposition(actx, opts, c, others, sched, say)
done()
// An apply of the bus module may write an older list than the newest composition holds: looked at again.
if module, _ := users.State(); module != "" && module == c.Module {
users.Again()
}
afterApply()
sayFacts(ctx, ag)
return res
},
Remove: func(ctx context.Context, module string, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := removeComposition(actx, opts, module, others, sched, say)
done()
afterApply()
sayFacts(ctx, ag)
return res
},
}
ag = agent.New(env)
current.set(ag)
err = ag.Run(linked)
cancel()
current.set(nil)
if ctx.Err() != nil {
return
}
if time.Since(began) > 30*time.Second {
wait = 2 * time.Second
}
if err != nil {
say(fmt.Sprintf("the node agent stopped: %v — beginning again in %s", err, wait))
}
select {
case <-ctx.Done():
return
case <-time.After(wait):
}
if wait *= 2; wait > 2*time.Minute {
wait = 2 * time.Minute
}
}
}
// reloadSettles is how long after this engine wrote the bus's user list the bus is taken to have reloaded its accounts:
// the bus module's entrypoint looks at the list every 5 s and then signals the server, whose reload of the accounts
// answers no JetStream request for about three seconds (on the bus node, 2026-10-25).
var reloadSettles = 10 * time.Second
// agentNow is the agent running now, which the node's facts are built from.
type agentNow struct {
mu sync.Mutex
ag *agent.Agent
}
func (c *agentNow) set(a *agent.Agent) {
c.mu.Lock()
c.ag = a
c.mu.Unlock()
}
func (c *agentNow) get() *agent.Agent {
c.mu.Lock()
defer c.mu.Unlock()
return c.ag
}
// applyComposition applies one composition: a module's, or the node's own.
func applyComposition(ctx context.Context, opts options, c delivery.Composed, others []delivery.Composed,
sched *apply.Scheduler, say link.Announce) agent.Result {
name := scopeName(c)
if c.Module == "" {
// What another module's composition declares is that module's, never the node's own apply's (issue 1945).
var dropped []string
c, dropped = ownWithout(c, others)
sayDroppedOnce(dropped)
}
d, err := declarationOf(c)
if err != nil {
return failedWith(err.Error())
}
if err := refuseEmptyWrites(d); err != nil {
return failedWith(err.Error())
}
scope := apply.Scope{Module: name, Others: resourcesOf(others)}
got := applyModuleHeld(ctx, opts, d, &scope, sched, say)
if got.Refused != "" {
return failedWith(got.Refused)
}
results := apply.ModuleResults(d, got.Outcome, got.Err)
if r, ok := results[name]; ok && r.Failed {
return failedWith(r.Detail)
}
if got.Err != nil {
return failedWith(got.Err.Error())
}
if got.Probe != nil {
return failedWith(got.Probe.partial().Error())
}
return agent.Result{}
}
// removeComposition removes one module's own resources and nothing of another's.
func removeComposition(ctx context.Context, opts options, module string, others []delivery.Composed,
sched *apply.Scheduler, say link.Announce) agent.Result {
scope := apply.Scope{Module: module, Others: resourcesOf(others)}
got := applyModuleHeld(ctx, opts, &declaration.Declaration{Version: declaration.Version}, &scope, sched, say)
switch {
case got.Refused != "":
return agent.Result{Failed: true, Detail: shorten(got.Refused)}
case got.Err != nil:
return agent.Result{Failed: true, Detail: shorten(got.Err.Error())}
}
return agent.Result{}
}
// busyModules are the modules whose last apply left a container running because it said busy.
func busyModules() []string {
busyMu.Lock()
defer busyMu.Unlock()
var out []string
for m, w := range busyBy {
if w && m != "" {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// keptDeclaration is the declaration the node kept before the switch, verified with the mesh's key when it is given.
func keptDeclaration(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
var raw []byte
if len(signer) > 0 {
var err error
if raw, err = store.LoadDeclared(store.DeclaredPath(statePath), signer); err != nil {
return nil, err
}
} else {
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
if err != nil {
return nil, err
}
raw = kept.Declaration
}
return declaration.Parse(raw)
}
// whatRuns is what runs on this node as the declaration of it (novox/hq ADR 0539 §9): the compositions the agent runs
// and, for every module the agent does not hold yet, what the kept declaration has for it, with the node's own parts
// (adoption, identities, groups, libraries) from the node's composition once the agent holds it and from the kept
// declaration until then. What the node judges its health by, and what the filling, the take act and the judgement of
// directories read.
func whatRuns(everything []delivery.Composed, holdsNode bool, kept func() (*declaration.Declaration, error)) (*declaration.Declaration, error) {
d, err := declarationOf(everything...)
if err != nil {
return nil, err
}
if holdsNode {
return d, nil
}
old, err := kept()
if err != nil {
if len(everything) > 0 {
return d, nil
}
return nil, err
}
d.Adoption, d.Identities, d.Groups, d.Libraries = old.Adoption, old.Identities, old.Groups, old.Libraries
d.Provisioner, d.Builds = old.Provisioner, old.Builds
d.Filled = append(d.Filled, old.Filled...)
return d, nil
}
// runsHere says what runs on this node; the running serve sets it to the agent's view, and a command run by hand reads
// the kept declaration.
var runsHere struct {
mu sync.Mutex
fn func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error)
}
func setRunsHere(fn func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error)) {
runsHere.mu.Lock()
runsHere.fn = fn
runsHere.mu.Unlock()
}
// declaredNow is the declaration of what runs on this node now.
func declaredNow(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
runsHere.mu.Lock()
fn := runsHere.fn
runsHere.mu.Unlock()
if fn != nil {
return fn(statePath, signer)
}
return keptDeclaration(statePath, signer)
}
// runsFromAgent is what runs now as the agent in hand sees it.
func runsFromAgent(current *agentNow) func(string, ed25519.PublicKey) (*declaration.Declaration, error) {
return func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
kept := func() (*declaration.Declaration, error) { return keptDeclaration(statePath, signer) }
ag := current.get()
if ag == nil {
return kept()
}
return whatRuns(ag.Everything(), ag.HoldsNode(), kept)
}
}
// refuseEmptyWrites refuses a declaration that names a file the provisioner fills (owned at runtime, no content, no
// bytes, no digest, no sealed value) without saying it is filled: applied, the file would be written empty over what the
// provisioner wrote (issue 1945, live on 2026-10-25 at 01:22 on ace, g14 and shanks). The apply fails, naming the file,
// and nothing is written.
func refuseEmptyWrites(d *declaration.Declaration) error {
filled := make(map[string]bool, len(d.Filled))
for _, p := range d.Filled {
filled[p] = true
}
var refused []string
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || f.OwnerAtRuntime != declaration.RuntimeOwned || filled[f.Path] {
continue
}
if f.Content == "" && f.Bytes == "" && f.Digest == "" && f.Sealed == "" && f.VaultSealed == nil {
refused = append(refused, fmt.Sprintf("%s (%s)", f.ID, f.Path))
}
}
if len(refused) > 0 {
return fmt.Errorf("the composition names %s with no content and does not say the provisioner fills it, so it "+
"would be written empty; nothing was applied", strings.Join(refused, ", "))
}
return nil
}
// recordedIn says the engine's applied state holds a record of every resource of a composition: what the agent believes
// of a have or the kept declaration only when it does, so a resource removed since is made again (issue 1945). A state
// that cannot be read says yes: the agent then believes what it was told, as before.
func recordedIn(statePath string, say func(string)) func(ctx context.Context, c delivery.Composed) bool {
return func(_ context.Context, c delivery.Composed) bool {
known, err := store.Load(statePath)
if err != nil {
if say != nil {
say(fmt.Sprintf("the applied state cannot be read, so what runs is believed as told: %v", err))
}
return true
}
filled := make(map[string]bool, len(c.Filled))
for _, p := range c.Filled {
filled[p] = true
}
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
return true
}
// A file the provisioner client fills is left to it, and has no record until it writes it.
if f, ok := r.(*declaration.File); ok && filled[f.Path] {
continue
}
if _, ok := known.Find(r.Identity()); !ok {
return false
}
}
return true
}
}
// declaresNothingToBackUp reads node-backup.backed-up's answer for a module: true when this machine backs nothing of it
// up (no runs, no paths), and whether the answer could be read at all.
func declaresNothingToBackUp(said json.RawMessage, module string) (nothing bool, read bool) {
var list []struct {
Module string `json:"module"`
Runs int `json:"runs"`
Paths []string `json:"paths"`
}
if err := json.Unmarshal(said, &list); err != nil {
return false, false
}
for _, m := range list {
if m.Module == module && (m.Runs > 0 || len(m.Paths) > 0) {
return false, true
}
}
return true, true
}
// applyBound is how long one apply or removal may take: long enough for the slowest image pull and backup step, short
// enough that a step that hangs cannot hold the node agent for ever.
const applyBound = 30 * time.Minute
// applyContext is the context an apply or a removal runs in: the link's values, never its end, and a bound of its own.
// The link's context ends when the bus goes, and the bus may go because of the apply itself (it replaces the bus's own
// container), which must then finish: a container removed and never made again took the mesh down (issue 1945, live on
// 2026-10-25 at 11:23). applyBound keeps a hung step from blocking the agent for ever.
func applyContext(ctx context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(context.WithoutCancel(ctx), applyBound)
}
+313
View File
@@ -0,0 +1,313 @@
//go:build integration
package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/link"
)
// The node agent whole, against a real bus with JetStream started in the test: the wants bucket, the node's composition
// bucket and the have bucket, a session on the bus, and the real apply of a module's composition into a directory of the
// test. R6-A AC1, AC5 and AC9 (novox/hq ADR 0539).
const itNode = "it1"
type itBed struct {
t *testing.T
conn *nats.Conn
js nats.JetStreamContext
wants nats.KeyValue
composed nats.KeyValue
haves nats.KeyValue
dir string
opts options
session *link.Session
controll ed25519.PrivateKey
operator ed25519.PrivateKey
said []string
mu sync.Mutex
}
func aServer(t *testing.T) *nats.Conn {
t.Helper()
srv, err := server.NewServer(&server.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true})
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the embedded server did not start")
}
t.Cleanup(srv.Shutdown)
conn, err := nats.Connect(srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
return conn
}
func newItBed(t *testing.T) *itBed {
t.Helper()
// Not skipped off Arch: a package manager that answers is put on PATH, so the apply of files runs on any build node.
bin := t.TempDir()
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
b := &itBed{t: t, conn: aServer(t), dir: t.TempDir()}
var err error
if b.js, err = b.conn.JetStream(); err != nil {
t.Fatal(err)
}
composedBucket, _ := delivery.ComposedBucketFor(itNode)
for _, c := range []struct {
name string
into *nats.KeyValue
history uint8
}{{delivery.AssignerWantsBucket, &b.wants, 8}, {composedBucket, &b.composed, delivery.ComposedHistory},
{delivery.HaveBucket, &b.haves, delivery.HaveHistory}} {
kv, err := b.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: c.name, History: c.history})
if err != nil {
t.Fatal(err)
}
*c.into = kv
}
b.opts = options{state: filepath.Join(b.dir, "state", "state.json"), timeout: 2 * time.Second}
if err := os.MkdirAll(filepath.Dir(b.opts.state), 0o755); err != nil {
t.Fatal(err)
}
_, b.controll, _ = ed25519.GenerateKey(nil)
_, b.operator, _ = ed25519.GenerateKey(nil)
return b
}
// start runs the node agent under a session on the bus, trusting the controller's key, and the operator's when asked.
func (b *itBed) start(trustOperator bool) (stop func()) {
b.t.Helper()
if trustOperator {
pub := b.operator.Public().(ed25519.PublicKey)
if err := os.WriteFile(filepath.Join(filepath.Dir(b.opts.state), operatorKeysFile),
[]byte("# the operator\n"+base64.StdEncoding.EncodeToString(pub)+"\n"), 0o600); err != nil {
b.t.Fatal(err)
}
}
b.session = &link.Session{Membership: link.Membership{Node: itNode}, Timeout: 2 * time.Second,
Say: b.say}
b.session.Attach(context.Background(), link.OverNATS{Conn: b.conn, JS: b.js, Node: itNode})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
defer close(done)
nodeAgent(ctx, b.opts, b.controll.Public().(ed25519.PublicKey), b.session, nil, b.say, func() {},
&agentNow{}, func(context.Context, *agent.Agent) {})
}()
return func() { cancel(); <-done }
}
func (b *itBed) say(line string) {
b.mu.Lock()
b.said = append(b.said, line)
b.mu.Unlock()
b.t.Log(line)
}
func fpIt(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
// compose writes a module's signed composition of one file into the node's bucket and returns its digest.
func (b *itBed) compose(module, fp, content string, data bool, signer ed25519.PrivateKey) string {
b.t.Helper()
path := filepath.Join(b.dir, module+".conf")
res, _ := json.Marshal(map[string]any{"id": module + ".conf", "type": "file", "path": path, "content": content, "module": module})
c := delivery.Composed{Node: itNode, Module: module, FP: fp, Data: data, Resources: []json.RawMessage{res}}
signed, err := delivery.SignComposed(c, signer)
if err != nil {
b.t.Fatal(err)
}
raw, err := delivery.EncodeComposed(signed)
if err != nil {
b.t.Fatal(err)
}
dig, _ := delivery.ComposedDigestOf(signed)
key, _ := delivery.ComposedKey(itNode, module, fp)
if _, err := b.composed.Put(key, raw); err != nil {
b.t.Fatal(err)
}
return dig
}
func (b *itBed) want(module, fp, digest, backup string) {
b.t.Helper()
raw, err := delivery.EncodeWant(delivery.Want{FP: fp, Composed: digest, Backup: backup})
if err != nil {
b.t.Fatal(err)
}
key, _ := delivery.WantKey(itNode, module)
if _, err := b.wants.Put(key, raw); err != nil {
b.t.Fatal(err)
}
}
func (b *itBed) file(module string) string {
got, _ := os.ReadFile(filepath.Join(b.dir, module+".conf"))
return string(got)
}
func (b *itBed) have(module string) delivery.Have {
key, _ := delivery.HaveKey(itNode, module)
e, err := b.haves.Get(key)
if err != nil {
return delivery.Have{}
}
h, _ := delivery.DecodeHave(e.Value())
return h
}
func within(t *testing.T, d time.Duration, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(d)
for !ok() {
if time.Now().After(deadline) {
t.Fatal(what)
}
time.Sleep(10 * time.Millisecond)
}
}
// R6-A AC1: given a running agent on a bus, when a want naming a digest and that composition are written, the module is
// applied within one second and its have names the fingerprint and the digest; a composition whose digest no want names
// is not applied.
func TestNatsAWantAndItsCompositionAreAppliedWithinASecond(t *testing.T) {
b := newItBed(t)
stop := b.start(false)
defer stop()
time.Sleep(300 * time.Millisecond) // the watches are up
fp := fpIt("a")
dig := b.compose("web", fp, "one\n", false, b.controll)
time.Sleep(300 * time.Millisecond)
if b.file("web") != "" {
t.Fatal("a composition no want names was applied")
}
began := time.Now()
b.want("web", fp, dig, "")
within(t, time.Second, "the module was not applied within one second of its want", func() bool { return b.file("web") == "one\n" })
t.Logf("applied %s after the want", time.Since(began))
within(t, 2*time.Second, "the have does not name the fingerprint and the digest", func() bool {
h := b.have("web")
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
})
// A want naming a digest the composition at its fingerprint does not have applies nothing.
fp2 := fpIt("b")
b.compose("web", fp2, "two\n", false, b.controll)
b.want("web", fp2, "sha256:"+strings.Repeat("0", 64), "")
time.Sleep(300 * time.Millisecond)
if b.file("web") != "one\n" {
t.Fatalf("a want naming another digest changed the module: %q", b.file("web"))
}
}
// R6-A AC5: a data module that ran before asks its backup before any resource changes; a backup that fails holds the
// module (nothing applied, the have says held with the backup's words); when its want then says skip, it is applied.
func TestNatsADataModuleAsksItsBackupFirstAndIsHeldWhenItFails(t *testing.T) {
b := newItBed(t)
var mu sync.Mutex
var asked []string
var onDiskWhenAsked string
if _, err := b.conn.Subscribe("mesh.seat.node-backup.tool.now."+itNode, func(m *nats.Msg) {
mu.Lock()
asked = append(asked, "now")
onDiskWhenAsked = b.file("db")
mu.Unlock()
_ = m.Respond([]byte(`{"result":{"started":"db"}}`))
// The holder says the backup ended on its event.
ev, _ := json.Marshal(map[string]any{"node": itNode, "module": "db", "ok": false, "at": time.Now().UTC(),
"error": "restic: the repository is locked"})
_ = b.conn.Publish(backedUpEvent, ev)
}); err != nil {
t.Fatal(err)
}
stop := b.start(false)
defer stop()
time.Sleep(300 * time.Millisecond)
fp1, fp2 := fpIt("a"), fpIt("b")
b.want("db", fp1, b.compose("db", fp1, "v1\n", true, b.controll), "")
within(t, 3*time.Second, "the first composition was not applied", func() bool { return b.file("db") == "v1\n" })
mu.Lock()
if len(asked) != 0 {
t.Fatalf("a module that never ran was backed up: %v", asked)
}
mu.Unlock()
dig2 := b.compose("db", fp2, "v2\n", true, b.controll)
b.want("db", fp2, dig2, "")
within(t, 3*time.Second, "the module was not held behind the failed backup", func() bool {
h := b.have("db")
return h.Phase == delivery.PhaseHeld && strings.Contains(h.Detail, "repository is locked")
})
mu.Lock()
if len(asked) != 1 || onDiskWhenAsked != "v1\n" {
t.Fatalf("the backup was asked %v, with %q on disk: it is asked before any resource changes", asked, onDiskWhenAsked)
}
mu.Unlock()
if b.file("db") != "v1\n" {
t.Fatalf("something was applied behind a failed backup: %q", b.file("db"))
}
if h := b.have("db"); h.FP != fp1 {
t.Fatalf("a held module should still name what it runs: %+v", h)
}
b.want("db", fp2, dig2, delivery.BackupSkip)
within(t, 3*time.Second, "the module was not applied once its want said skip", func() bool { return b.file("db") == "v2\n" })
within(t, 2*time.Second, "the have does not say running at the new fingerprint", func() bool {
h := b.have("db")
return h.Phase == delivery.PhaseRunning && h.FP == fp2 && h.Composed == dig2
})
}
// R6-A AC9, the hand path end to end: with no controller running, a composition signed by the operator's key and a want
// naming it are applied and the have written; the same signed by a key nobody trusts is not.
func TestNatsTheHandPathAppliesACompositionTheOperatorSigned(t *testing.T) {
b := newItBed(t)
stop := b.start(true)
defer stop()
time.Sleep(300 * time.Millisecond)
_, stranger, _ := ed25519.GenerateKey(nil)
fpS := fpIt("c")
b.want("evil", fpS, b.compose("evil", fpS, "no\n", false, stranger), "")
time.Sleep(300 * time.Millisecond)
if b.file("evil") != "" {
t.Fatal("a composition signed by a key nobody trusts was applied")
}
fp := fpIt("d")
dig := b.compose("rescue", fp, "back\n", false, b.operator)
b.want("rescue", fp, dig, "")
within(t, 2*time.Second, "the operator's composition was not applied", func() bool { return b.file("rescue") == "back\n" })
within(t, 2*time.Second, "the have was not written", func() bool {
h := b.have("rescue")
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
})
}
+42
View File
@@ -0,0 +1,42 @@
package main
import (
"io/fs"
"os"
"path/filepath"
"strings"
"testing"
)
// R6-A AC8: nothing in the node-engine receives, orders or reconciles a whole declaration (novox/hq ADR 0539 §9, ADR
// 0429 §8): no package subscribes to a node's declare subject or names the deleted symbols.
func TestNoPackageHearsOrOrdersAWholeDeclaration(t *testing.T) {
deleted := []string{
"DeclareSubject", "mesh.node.\" + node + \".declare", "ReconcileDue", "ReconcileEvery", "olderThanKept",
"refuseOlder", "olderGeneration", "appliedGeneration", "reconcileKept", "reapplyKept", "adoptionWatch",
"unsaidFile", "numbersFile", "func gather(", "func handleBody(", "Supersedes(", "ReportSequence",
"RefusedOlder", "ReadsGeneration", "ReadsModules", "GenerationRefusal", "LeftOutModuleOf",
}
root := filepath.Join("..", "..")
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "nodeclare_test.go") {
return nil
}
body, err := os.ReadFile(path)
if err != nil {
return err
}
for _, word := range deleted {
if strings.Contains(string(body), word) {
t.Errorf("%s still names %q", path, word)
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
}
+85 -5
View File
@@ -2,7 +2,10 @@ package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"time"
"github.com/novox/mesh-host/internal/identity"
@@ -22,12 +25,77 @@ const (
nodeKeyLook = 30 * time.Second
)
// heardKey is the vault's own public key as its answer to node-key gave it (ADR 0547 clause 9, issue 1945): the key the
// bus node hands the loaded list's hashes sealed to. One for the process, as the key stated is; kick is the follower of
// the user list on the link open now, told when the key is first heard or changes.
type heardKey struct {
mu sync.Mutex
pem string
kick func()
}
// vaultKey is the vault's key this engine heard.
var vaultKey = &heardKey{}
func (h *heardKey) get() string {
h.mu.Lock()
defer h.mu.Unlock()
return h.pem
}
// heard keeps the key a node-key answer gave, and tells the follower when it is a first or changed one. An answer
// with no key (a vault before this one) changes nothing.
func (h *heardKey) heard(answer json.RawMessage) {
var a struct {
VaultKey string `json:"vault_key"`
}
if json.Unmarshal(answer, &a) != nil || !strings.Contains(a.VaultKey, "PUBLIC KEY") {
return
}
h.mu.Lock()
changed := a.VaultKey != h.pem
h.pem = a.VaultKey
kick := h.kick
h.mu.Unlock()
if changed && kick != nil {
kick()
}
}
// following names the follower told when the key changes, and tells it at once when a key is known.
func (h *heardKey) following(kick func()) {
h.mu.Lock()
h.kick = kick
known := h.pem != ""
h.mu.Unlock()
if known && kick != nil {
kick()
}
}
// nodeKeyReask is how often the key is stated again though it did not change, so a changed vault key is heard: the
// vault takes the same key again and changes nothing (ADR 0547 clause 9).
const nodeKeyReask = 10 * time.Minute
// keyHearing is what the keeper does with the vault's own key in node-key's answers (ADR 0547 clause 9): heard takes
// it from an answer; held says whether one is held, and while none is the key is stated again at every look, so a vault
// delivered after this engine is heard within a look; every is how often it is stated again all the same, so a changed
// vault key is heard.
type keyHearing struct {
heard func(json.RawMessage)
held func() bool
// look is how often it is stated again while no vault key is held; every how often once one is.
look, every time.Duration
}
// keepNodeKeyStated states the sealing key load answers to the vault at start, retrying with a growing wait from
// retryFrom to retryUpTo while it does not answer, and states it again whenever the key changes. It says one line
// when a key is stated and one when a try fails for another reason than the last, and returns when ctx ends.
// when a key is stated and one when a try fails for another reason than the last, and returns when ctx ends. With
// hearing, each answer goes to it, and the key is stated again while no vault key is held and at its interval.
func keepNodeKeyStated(ctx context.Context, load func() (identity.SealingKey, error), ask vaultAsk, say func(string),
retryFrom, retryUpTo time.Duration) {
retryFrom, retryUpTo time.Duration, hearing *keyHearing) {
stated, lastFailure := "", ""
var askedAt time.Time
wait := retryFrom
fail := func(why string) {
if why != lastFailure {
@@ -40,17 +108,29 @@ func keepNodeKeyStated(ctx context.Context, load func() (identity.SealingKey, er
for {
delay := look
key, err := load()
again := false
if hearing != nil && stated != "" {
since := time.Since(askedAt)
held := hearing.held == nil || hearing.held()
again = (!held && since >= hearing.look) || (held && hearing.every > 0 && since >= hearing.every)
}
switch {
case err != nil:
fail(err.Error())
delay = wait
case key.Public != stated:
if _, err := ask("node-key", map[string]string{"key": key.Public}); err != nil {
case key.Public != stated || again:
if answer, err := ask("node-key", map[string]string{"key": key.Public}); err != nil {
fail(err.Error())
delay = wait
} else {
askedAt = time.Now()
if hearing != nil && hearing.heard != nil {
hearing.heard(answer)
}
if key.Public != stated {
say(fmt.Sprintf("mesh-host: this machine's sealing key %s is stated to the vault", key.Public))
}
stated, lastFailure, wait = key.Public, "", retryFrom
say(fmt.Sprintf("mesh-host: this machine's sealing key %s is stated to the vault", key.Public))
}
}
select {
+81 -3
View File
@@ -19,6 +19,10 @@ type keyVault struct {
stated []string
fail int
verbs []string
// keyless is how many answers carry no vault key (a vault before ADR 0547 clause 9's, or not yet started); key the
// vault's key the others carry.
keyless int
key string
}
func (v *keyVault) ask(verb string, args any) (json.RawMessage, error) {
@@ -35,7 +39,12 @@ func (v *keyVault) ask(verb string, args any) (json.RawMessage, error) {
}
_ = json.Unmarshal(raw, &a)
v.stated = append(v.stated, a.Key)
return json.RawMessage(`{"recorded":true}`), nil
if v.key == "" || v.keyless > 0 {
v.keyless--
return json.RawMessage(`{"recorded":true}`), nil
}
answer, _ := json.Marshal(map[string]any{"recorded": true, "vault_key": v.key})
return answer, nil
}
func (v *keyVault) said() []string {
@@ -71,9 +80,14 @@ func TestTheEngineStatesItsSealingKeyToTheVaultAtStartRetriesAndRestatesAChange(
say := func(l string) { lmu.Lock(); lines = append(lines, l); lmu.Unlock() }
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go keepNodeKeyStated(ctx, load, vault.ask, say, time.Millisecond, 5*time.Millisecond)
var hmu sync.Mutex
var answers []string
heard := func(a json.RawMessage) { hmu.Lock(); answers = append(answers, string(a)); hmu.Unlock() }
go keepNodeKeyStated(ctx, load, vault.ask, say, time.Millisecond, 5*time.Millisecond, &keyHearing{heard: heard})
waitFor(t, "the first key stated after two refusals", func() bool { return len(vault.said()) == 1 })
// Its answer is handed on, so the vault's own key in it is heard (ADR 0547 clause 9); refusals hand nothing.
waitFor(t, "the answer heard", func() bool { hmu.Lock(); defer hmu.Unlock(); return len(answers) == 1 })
if vault.said()[0] != first.Public {
t.Errorf("stated %q, want the engine's own key %q", vault.said()[0], first.Public)
}
@@ -102,11 +116,75 @@ func TestTheEngineStatesItsSealingKeyToTheVaultAtStartRetriesAndRestatesAChange(
if len(lines) == 0 {
t.Error("the engine said nothing")
}
asksAgainUntilItHearsTheVaultsKey(t)
}
// The statement a host makes names this contract (issue 786), one above the vault's box.
// ADR 0547 clause 9: the vault's key is heard though the engine's own key never changes. While none is held the key
// is stated again at each look, so a vault delivered after the engine (or answering no key yet) is heard; once one is
// held, again only at the slow interval, so a changed vault key is heard; never more.
func asksAgainUntilItHearsTheVaultsKey(t *testing.T) {
sealing, _ := identity.GenerateSealingKey()
load := func() (identity.SealingKey, error) { return sealing, nil }
pem := func(c string) string { return "-----BEGIN PUBLIC KEY-----\n" + c + "\n-----END PUBLIC KEY-----\n" }
for _, c := range []struct {
name string
keyless int
every time.Duration
change bool
wantAtLeast int
wantAtMost int
}{
{name: "a vault with no key yet is asked again until it answers one, then not", keyless: 3, wantAtLeast: 4, wantAtMost: 4},
{name: "a held key is asked again at the slow interval, and a changed one heard", every: 20 * time.Millisecond,
change: true, wantAtLeast: 3, wantAtMost: 1 << 20},
} {
t.Run(c.name, func(t *testing.T) {
vault := &keyVault{keyless: c.keyless, key: pem("AAAA")}
h := &heardKey{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go keepNodeKeyStated(ctx, load, vault.ask, func(string) {}, time.Millisecond, 5*time.Millisecond,
&keyHearing{heard: h.heard, held: func() bool { return h.get() != "" }, look: time.Millisecond, every: c.every})
waitFor(t, "the vault's key heard", func() bool { return h.get() == pem("AAAA") })
if c.change {
vault.mu.Lock()
vault.key = pem("BBBB")
vault.mu.Unlock()
waitFor(t, "the changed vault key heard", func() bool { return h.get() == pem("BBBB") })
}
time.Sleep(60 * time.Millisecond)
n := len(vault.said())
if n < c.wantAtLeast || n > c.wantAtMost {
t.Errorf("node-key asked %d times, want %d to %d", n, c.wantAtLeast, c.wantAtMost)
}
})
}
}
// The statement a host makes names this contract (issue 786), one above the vault's box. And the vault's key its
// answer gives is kept, told to the follower once when first heard and again only when changed; an answer without one
// (an older vault) or with no public key changes nothing (ADR 0547 clause 9).
func TestTheNodeKeyContractIsThirteen(t *testing.T) {
if link.NodeKeyContract != 13 {
t.Errorf("NodeKeyContract %d", link.NodeKeyContract)
}
h := &heardKey{}
kicks := 0
h.following(func() { kicks++ })
h.heard(json.RawMessage(`{"recorded":true}`))
h.heard(json.RawMessage(`{"recorded":true,"vault_key":"not a key"}`))
if h.get() != "" || kicks != 0 {
t.Fatalf("an answer with no key was kept: %q, %d kicks", h.get(), kicks)
}
pem := "-----BEGIN PUBLIC KEY-----\nAAAA\n-----END PUBLIC KEY-----\n"
answer, _ := json.Marshal(map[string]any{"recorded": true, "vault_key": pem})
h.heard(answer)
h.heard(answer)
if h.get() != pem || kicks != 1 {
t.Fatalf("key %q, %d kicks, want the key and one", h.get(), kicks)
}
h.following(func() { kicks += 10 })
if kicks != 11 {
t.Fatalf("a follower named after the key was heard was not told: %d", kicks)
}
}
-95
View File
@@ -1,95 +0,0 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// A declaration carries no order, so a host cannot tell an older one from a newer (novox/hq
// 04-ISSUES/107). Its only identity was the digest of its bytes: "not the last" could be said,
// "older" could not.
func keptWith(t *testing.T, sequence int64) store.Declared {
t.Helper()
return keptAt(t, link.Order{Sequence: sequence})
}
func keptAt(t *testing.T, order link.Order) store.Declared {
t.Helper()
body, err := json.Marshal(map[string]any{
"declaration": 1, "resources": []any{}, "owns_nothing": true, "sequence": order.Sequence,
"epoch": order.Epoch,
})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: []byte("x")}
}
func TestAnOlderDeclarationFromTheMeshIsRefused(t *testing.T) {
err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 5})
if err == nil {
t.Fatal("sequence 5 was accepted over a kept 7")
}
if !strings.Contains(err.Error(), "older") || !strings.Contains(err.Error(), "nothing was applied") {
t.Fatalf("the refusal does not say what it is: %v", err)
}
}
func TestANewerOrEqualDeclarationIsNot(t *testing.T) {
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 8}); err != nil {
t.Fatalf("sequence 8 was refused over a kept 7: %v", err)
}
// Equal is the same declaration again, which reconciling is for.
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 7}); err != nil {
t.Fatalf("the same sequence was refused: %v", err)
}
}
func TestNoOrderClaimedMeansNoOrderCompared(t *testing.T) {
// An older controller sends none; a host that received before it understood them kept none.
// Refusing on a guess would strand a node the moment the controller is older than the host.
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 0}); err != nil {
t.Fatalf("a declaration claiming no order was refused: %v", err)
}
if err := refuseOlder(keptWith(t, 0), nil, link.Order{Sequence: 3}); err != nil {
t.Fatalf("a declaration was refused against a kept one that claimed no order: %v", err)
}
if err := refuseOlder(store.Declared{}, store.ErrNothingDeclared, link.Order{Sequence: 3}); err != nil {
t.Fatalf("a first declaration was refused: %v", err)
}
}
// **The lease epoch decides first** (novox/hq to-be 45 §6): a declaration from a controller that lost
// its lease is refused whatever its sequence, and a new lease holder's is taken whatever its sequence.
func TestAnOlderEpochIsRefusedAndANewerTaken(t *testing.T) {
kept := keptAt(t, link.Order{Epoch: 57, Sequence: 3})
err := refuseOlder(kept, nil, link.Order{Epoch: 41, Sequence: 12})
if err == nil || !strings.Contains(err.Error(), "epoch 41") || !strings.Contains(err.Error(), "epoch 57") {
t.Fatalf("an older epoch was not refused by name: %v", err)
}
if err := refuseOlder(kept, nil, link.Order{Epoch: 58, Sequence: 1}); err != nil {
t.Fatalf("a new lease holder's first declaration was refused for its sequence: %v", err)
}
if err := refuseOlder(kept, nil, link.Order{Epoch: 57, Sequence: 2}); err == nil {
t.Fatal("a lower sequence under the same epoch was accepted")
}
// A controller with no lease — older, or rolled back to a build from before it — is today's
// behaviour: no epoch compared.
if err := refuseOlder(kept, nil, link.Order{Sequence: 4}); err != nil {
t.Fatalf("a declaration with no epoch was refused for one: %v", err)
}
if held, older := olderThanKept(kept, nil, link.Order{Epoch: 41, Sequence: 12}); !older ||
held != (link.Order{Epoch: 57, Sequence: 3}) {
t.Fatalf("olderThanKept said %v, holding %+v", older, held)
}
// What is kept cannot be read: no order to compare, so nothing is refused on a guess.
if _, older := olderThanKept(store.Declared{Declaration: []byte("garbled")}, nil,
link.Order{Epoch: 41, Sequence: 1}); older {
t.Fatal("refused against a kept declaration nobody can read")
}
}
-81
View File
@@ -9,7 +9,6 @@ import (
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/system"
)
@@ -127,83 +126,3 @@ func TestAPackageManagerThatIsNotThereRefusesWhole(t *testing.T) {
t.Error("the refused declaration touched the machine")
}
}
// Defends the review of novox/hq issue 233: **a re-apply whose result changed is said unasked**. A machine
// reconciles every five minutes and speaks only when its account is news; how the apply went was not part of
// it, so a package manager that stopped answering between deliveries was never said, and one that answered
// again never cleared. Now a failure is said once, the same failure again is not, and a clean re-apply after it
// is — which is what clears the mesh's condition.
func TestAReapplyWhoseResultChangedIsSaidUnasked(t *testing.T) {
dir := t.TempDir()
conf := filepath.Join(dir, "elsewhere", "pacman.conf") // not declared: the apply cannot make it answer
pacmanAnsweringOnce(t, conf)
opts := options{state: filepath.Join(dir, "state.json")}
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"motd","type":"file","path":"` + filepath.Join(dir, "motd") + `","content":"hello\n"},` +
`{"id":"pacman.base","type":"package","package":"base"}]}`)
w := &adoptionWatch{}
// Clean once, as it was before: base installed while pacman answered.
if err := os.MkdirAll(filepath.Dir(conf), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(conf, []byte("[options]\n"), 0o644); err != nil {
t.Fatal(err)
}
clean := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if faulty(clean) {
t.Fatalf("the first apply was not clean: %+v", clean)
}
w.said(clean)
if w.news(applyAndKeep(context.Background(), opts, raw, nil, nil, nil)) {
t.Error("a clean re-apply after a clean one was said, so every machine speaks every five minutes")
}
// The configuration goes: the re-apply fails, and that is news once.
if err := os.Remove(conf); err != nil {
t.Fatal(err)
}
failed := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if !faulty(failed) {
t.Fatalf("the re-apply did not fail: %+v", failed)
}
if !w.news(failed) {
t.Fatal("a re-apply that failed after a clean one was not said, so the mesh never hears the machine fails")
}
w.said(failed)
if w.news(applyAndKeep(context.Background(), opts, raw, nil, nil, nil)) {
t.Error("the same failure was said again on the next re-apply")
}
// It comes back: the clean re-apply is news, and is what clears the mesh's condition.
if err := os.WriteFile(conf, []byte("[options]\n"), 0o644); err != nil {
t.Fatal(err)
}
again := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if faulty(again) {
t.Fatalf("the re-apply did not come back clean: %+v", again)
}
if !w.news(again) {
t.Fatal("a clean re-apply after a failed one was not said, so the mesh's condition never clears")
}
}
// A refusal is news too, once, and a reconcile with nothing to say after a clean one stays silent.
func TestARefusalIsSaidOnceAndNothingIsNotSaid(t *testing.T) {
w := &adoptionWatch{}
w.said(link.Report{})
if w.news(link.Report{}) {
t.Error("nothing was said again")
}
refused := link.Report{Refused: "pacman does not answer here"}
if !w.news(refused) {
t.Fatal("a refusal was not said")
}
w.said(refused)
if w.news(link.Report{Refused: "pacman does not answer here, at another time"}) {
t.Error("the same refusal in other words was said again")
}
if !w.news(link.Report{}) {
t.Error("a clean apply after a refusal was not said")
}
}
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+94
View File
@@ -0,0 +1,94 @@
package main
import (
"context"
"encoding/json"
"path/filepath"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/store"
)
// The record check reads the engine's applied state: a composition one of whose resources has no record there is not
// recorded (issue 1945).
func TestACompositionIsRecordedOnlyWhenEveryResourceIs(t *testing.T) {
path := filepath.Join(t.TempDir(), "state.json")
var s store.State
s.Record(store.Applied{ID: "ca-trust.trust", Type: "file", Target: "/etc/ca"})
if err := store.Save(path, s); err != nil {
t.Fatal(err)
}
c := delivery.Composed{Node: "ace", Module: "ca-trust", Resources: []json.RawMessage{
json.RawMessage(`{"id":"ca-trust.trust","type":"file","module":"ca-trust","path":"/etc/ca","mode":"0644","content":"x"}`)}}
recorded := recordedIn(path, func(string) {})
if !recorded(context.Background(), c) {
t.Fatal("a composition whose every resource has a record was not recorded")
}
c.Resources = append(c.Resources, json.RawMessage(
`{"id":"ca-trust.more","type":"file","module":"ca-trust","path":"/etc/ca2","mode":"0644","content":"y"}`))
if recorded(context.Background(), c) {
t.Fatal("a composition with a resource the state holds no record of was recorded")
}
}
// The modules applied again are those with a missing resource, each once; one down but there is not among them.
func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
st := liveness.Statement{Resources: []liveness.State{
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.homeserver"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.db"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "ca-trust", ID: "ca-trust.trust"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "web", ID: "web.server"}, State: liveness.Unhealthy},
}}
if got := missingModules(st); len(got) != 2 || got[0] != "ca-trust" || got[1] != "matrix" {
t.Fatalf("applied again: %v", got)
}
}
// A module is applied again only for a resource newly missing: one still missing at the next change of the statement is
// not acted on again, so a resource that cannot be made is not applied (and backed up) at every unrelated change. One
// that comes back and goes again is newly missing again (the review of issue 1945).
func TestOnlyANewlyMissingResourceAppliesItsModuleAgain(t *testing.T) {
missing := func(ids ...string) liveness.Statement {
var st liveness.Statement
for _, id := range ids {
st.Resources = append(st.Resources, liveness.State{Resource: liveness.Resource{Module: "matrix", ID: id},
State: liveness.Unhealthy, Missing: true})
}
return st
}
var last map[string]bool
var got []string
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 1 {
t.Fatalf("first missing: %v", got)
}
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 0 {
t.Fatalf("still missing was acted on again: %v", got)
}
got, last = newlyMissing(last, missing())
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 1 {
t.Fatalf("missing again after it came back: %v", got)
}
}
// A file the provisioner client fills is left to it by the apply and has no record until the client writes it: it does
// not make its composition unrecorded (the review of issue 1945).
func TestAFilledFileDoesNotMakeItsCompositionUnrecorded(t *testing.T) {
path := filepath.Join(t.TempDir(), "state.json")
var s store.State
s.Record(store.Applied{ID: "ca-trust.trust", Type: "file", Target: "/etc/ca"})
if err := store.Save(path, s); err != nil {
t.Fatal(err)
}
c := delivery.Composed{Node: "ace", Module: "ca-trust", Filled: []string{"/etc/ca-filled"}, Resources: []json.RawMessage{
json.RawMessage(`{"id":"ca-trust.trust","type":"file","module":"ca-trust","path":"/etc/ca","mode":"0644","content":"x"}`),
json.RawMessage(`{"id":"ca-trust.filled","type":"file","module":"ca-trust","path":"/etc/ca-filled","mode":"0644","owner-at-runtime":"owned"}`)}}
if !recordedIn(path, func(string) {})(context.Background(), c) {
t.Fatal("a filled file the client has not written made its composition unrecorded")
}
}
-64
View File
@@ -1,64 +0,0 @@
package main
import (
"os"
"path/filepath"
"reflect"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The last apply's report, kept beside the state until the mesh takes it (novox/hq issue 264).
// What a host that died between applying and reporting kept is read back by the next one exactly —
// the digest and the outcome, never a new account of the machine.
func TestTheKeptReportSurvivesTheHost(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
made := link.Report{Declared: "sha256:d1", Applied: []string{"a"}, Failed: map[string]string{"b": "no room"},
Host: "v2"}
if err := unsaidBeside(state).Keep(made); err != nil {
t.Fatal(err)
}
kept, ok, err := unsaidBeside(state).Pending()
if err != nil || !ok {
t.Fatalf("nothing kept: %v", err)
}
if !reflect.DeepEqual(kept, made) {
t.Fatalf("kept %+v, made %+v", kept, made)
}
// Taken about another declaration, it stays; taken about this one, it goes.
if err := unsaidBeside(state).Said("sha256:other"); err != nil {
t.Fatal(err)
}
if _, ok, _ := unsaidBeside(state).Pending(); !ok {
t.Fatal("a report about another declaration cleared this one")
}
if err := unsaidBeside(state).Said("sha256:d1"); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(store.UnsaidPath(state)); !os.IsNotExist(err) {
t.Fatalf("the report was taken and is still kept: %v", err)
}
}
// A node that never applied anything has nothing to say again; and an apply refused before its
// declaration was read clears an older report, which would otherwise be re-said after it.
func TestNothingKeptWhenNothingWasApplied(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
if _, ok, err := unsaidBeside(state).Pending(); ok || err != nil {
t.Fatalf("a node that applied nothing has a report to say: %v %v", ok, err)
}
if err := unsaidBeside(state).Keep(link.Report{Declared: "sha256:d1"}); err != nil {
t.Fatal(err)
}
if err := unsaidBeside(state).Keep(link.Report{Refused: "forged"}); err != nil {
t.Fatal(err)
}
if _, ok, _ := unsaidBeside(state).Pending(); ok {
t.Fatal("an older report is kept after a later declaration was refused")
}
}
+127
View File
@@ -0,0 +1,127 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"fmt"
"os"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
"github.com/novox/mesh-host/internal/userlist"
)
// The bridge of novox/hq ADR 0547 §8, wired: the bus node's engine writes the bus's user list alone from the newest
// composition of the bus module, under the apply's two locks as a filled file is written (filling.go), and says what
// it wrote in the bus module's have.
// userListFollower is the follower of the bus's user list for one link: the node's composition bucket, the keys the
// agent trusts, the write alone, and the have.
func userListFollower(opts options, session *link.Session, composedBucket string, trusted func() []ed25519.PublicKey,
say link.Announce, wrote func()) *userlist.Follower {
return userlist.New(userlist.Env{
Node: session.Membership.Node,
Composed: bucketWatch{session, composedBucket},
Trusted: trusted,
Resolve: func(ctx context.Context, raw json.RawMessage) ([]byte, error) {
return resolveUserList(session.WithLink(ctx), raw)
},
OnDisk: userListOnDisk,
Write: func(ctx context.Context, module string, raw json.RawMessage, body []byte) (string, bool, error) {
digest, changed, err := writeUserListAlone(session.WithLink(ctx), opts, raw, body, say)
if changed && wrote != nil {
wrote()
}
return digest, changed, err
},
Report: func(ctx context.Context, module string, w userlist.Written) {
session.NoteUserList(ctx, module, w)
},
// The vault's key from its answer to node-key: the loaded list's hashes are handed sealed to it (ADR 0547
// clause 9).
VaultKey: vaultKey.get,
Say: announceOr(say),
})
}
// userListFile is a user-list resource as the file it declares.
func userListFile(raw []byte) (*declaration.File, error) {
r, err := declaration.ParseResource(raw)
if err != nil {
return nil, fmt.Errorf("the user list is not a resource: %w", err)
}
f, ok := r.(*declaration.File)
if !ok {
return nil, fmt.Errorf("the user list is a %s, not a file", r.Kind())
}
return f, nil
}
// resolveUserList is the list a user-list resource holds: its content, or the object its digest names in this node's
// file bucket, fetched and verified (apply.ResolveFileReferences, ADR 0486 §1).
func resolveUserList(ctx context.Context, raw []byte) ([]byte, error) {
f, err := userListFile(raw)
if err != nil {
return nil, err
}
d := &declaration.Declaration{Version: declaration.Version, Resources: []declaration.Resource{f}}
if err := apply.ResolveFileReferences(ctx, d, link.FilesFrom(ctx)); err != nil {
return nil, fmt.Errorf("the user list could not be had by its digest: %w", err)
}
return []byte(d.Resources[0].(*declaration.File).Content), nil
}
// userListOnDisk is what the user list's file holds on this machine now; nil when there is none.
func userListOnDisk(raw json.RawMessage) ([]byte, error) {
f, err := userListFile(raw)
if err != nil {
return nil, err
}
body, err := os.ReadFile(f.Path)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
return body, err
}
// writeUserListAlone writes one user-list resource alone (apply.WriteAlone) with body as its content, the node's state
// read and saved under the apply's locks, and answers the digest of the file on the machine after.
func writeUserListAlone(ctx context.Context, opts options, raw []byte, body []byte, say link.Announce) (string, bool, error) {
declared, err := userListFile(raw)
if err != nil {
return "", false, err
}
file := *declared
file.Content, file.Digest, file.Size = string(body), "", 0
applying.Lock()
defer applying.Unlock()
unlock, err := store.Lock(opts.state, nil)
if err != nil {
return "", false, err
}
defer unlock()
known, err := store.Load(opts.state)
if err != nil {
return "", false, err
}
built, err := system.For(builtFor)
if err != nil {
return "", false, err
}
out, updated, writeErr := apply.WriteAlone(ctx, built, &file, known, apply.ExecRunner, announceOr(say))
// Saved whichever way it went, as an apply's: what was written is on the machine either way.
if err := store.Save(opts.state, updated); err != nil {
writeErr = errors.Join(writeErr, errors.New("written, and the node's state could not be saved: "+err.Error()))
}
digest := ""
if body, err := os.ReadFile(file.Path); err == nil {
digest = userlist.Digest(body)
}
return digest, writeErr == nil && out.Action != "unchanged", writeErr
}
+6 -7
View File
@@ -3,7 +3,7 @@ package main
// Issue 755's proof on a real bus (novox/hq ADR 0349 §16): a core credential the vault sealed to this machine crosses
// the bus inside the declaration, which the controller carries and cannot open, and this engine opens it with its
// sealing key and writes the value. A listener on every subject keeps every byte that crossed; the value is never
// among them. Run against the throwaway bus merge-check.sh starts (MESH_TEST_NATS).
// among them. Run against the throwaway bus the build module starts for the integration level (MESH_TEST_NATS).
import (
"context"
@@ -22,7 +22,6 @@ import (
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
@@ -61,14 +60,14 @@ func TestNatsTheEngineFillsACoreCredentialFromTheVaultsBoxWithNoValueInTheClear(
}
_ = tap.Flush()
// The engine hears its declaration on its own subject.
const node = "vaultbox"
// The engine hears the composition on a subject of the test (the node agent reads it from its bucket).
const subject = "mesh.test.vaultbox.composition"
engine, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer engine.Close()
heard, err := engine.SubscribeSync(link.DeclareSubject(node))
heard, err := engine.SubscribeSync(subject)
if err != nil {
t.Fatal(err)
}
@@ -92,7 +91,7 @@ func TestNatsTheEngineFillsACoreCredentialFromTheVaultsBoxWithNoValueInTheClear(
t.Fatal(err)
}
defer controller.Close()
if err := controller.Publish(link.DeclareSubject(node), body); err != nil {
if err := controller.Publish(subject, body); err != nil {
t.Fatal(err)
}
_ = controller.Flush()
@@ -123,7 +122,7 @@ func TestNatsTheEngineFillsACoreCredentialFromTheVaultsBoxWithNoValueInTheClear(
mu.Lock()
all := strings.Join(crossed, "\n")
mu.Unlock()
if !strings.Contains(all, link.DeclareSubject(node)) || !strings.Contains(all, "vault-sealed") {
if !strings.Contains(all, subject) || !strings.Contains(all, "vault-sealed") {
t.Fatalf("the listener heard nothing of the declaration:\n%s", all)
}
for _, form := range []string{coreCredential, base64.StdEncoding.EncodeToString([]byte(coreCredential))} {
+139
View File
@@ -0,0 +1,139 @@
package main
import (
"crypto/ed25519"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/agent"
)
func aFileOf(module, id string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": module + "." + id, "type": "file", "path": "/tmp/r6/" + module + "/" + id,
"content": "x", "module": module})
return raw
}
// What runs is the agent's compositions and the kept declaration's resources for each module the agent does not hold;
// the node's own parts are the kept declaration's until the agent holds the node's composition (ADR 0539 §9). One module
// under the agent must not make the rest invisible to the liveness, units, accounts, libraries, filling and the take act.
func TestWhatRunsIsTheAgentsCompositionsAndTheKeptRest(t *testing.T) {
kept := &declaration.Declaration{Identities: []declaration.Identity{{Module: "db", Name: "mesh-db"}},
Libraries: []declaration.Library{{Library: "media"}}}
everything := []delivery.Composed{
{Node: "n1", Module: "web", Resources: []json.RawMessage{aFileOf("web", "a")}},
{Node: "n1", Module: "db", Resources: []json.RawMessage{aFileOf("db", "a")}}, // still the kept declaration's
}
d, err := whatRuns(everything, false, func() (*declaration.Declaration, error) { return kept, nil })
if err != nil {
t.Fatal(err)
}
if len(d.Resources) != 2 || len(d.Identities) != 1 || len(d.Libraries) != 1 {
t.Fatalf("%d resources, %d identities, %d libraries: the kept modules and the node's kept parts are missing",
len(d.Resources), len(d.Identities), len(d.Libraries))
}
// Once the agent holds the node's composition, its parts are the node's.
node := delivery.Composed{Node: "n1", Own: delivery.NodeOwn{Identities: []declaration.Identity{{Module: "web", Name: "mesh-web"}}}}
d, err = whatRuns(append(everything, node), true, func() (*declaration.Declaration, error) { return nil, errors.New("not read") })
if err != nil {
t.Fatal(err)
}
if len(d.Identities) != 1 || d.Identities[0].Name != "mesh-web" || len(d.Libraries) != 0 {
t.Fatalf("the node's parts: %+v %+v", d.Identities, d.Libraries)
}
}
// What the take act, the judgement of directories and the filling read is what runs, not the frozen kept declaration.
func TestDeclaredNowReadsWhatRunsWhenTheAgentIsRunning(t *testing.T) {
setRunsHere(func(string, ed25519.PublicKey) (*declaration.Declaration, error) {
return &declaration.Declaration{Version: 1, Resources: nil, Provisioner: "from-agent"}, nil
})
defer setRunsHere(nil)
d, err := declaredNow("/nowhere", nil)
if err != nil || d.Provisioner != "from-agent" {
t.Fatalf("declaredNow read %+v (%v), not what runs", d, err)
}
}
// 1945/AC17 (unit): a resource the node's own composition shares with a running module's composition is declared once,
// the module's: the declaration joined of both names each file once, so the filling writes it; the node's own resources
// no module declares stay. (Live from 2026-10-25 12:43: "2 resources declare X" on every node.)
func TestTheNodesOwnCopyOfAModulesResourceIsLeftOut(t *testing.T) {
own := delivery.Composed{Node: "n1", Resources: []json.RawMessage{aFileOf("zsh", "rc"), aFileOf("mesh-wireguard", "conf")},
Filled: []string{"/tmp/r6/zsh/rc"}}
zsh := delivery.Composed{Node: "n1", Module: "zsh", FP: "src2:aa", Resources: []json.RawMessage{aFileOf("zsh", "rc")},
Filled: []string{"/tmp/r6/zsh/rc"}}
d, err := declarationOf(own, zsh)
if err != nil {
t.Fatal(err)
}
at := map[string]int{}
for _, r := range d.Resources {
at[r.Target()]++
}
if at["/tmp/r6/zsh/rc"] != 1 || at["/tmp/r6/mesh-wireguard/conf"] != 1 || len(d.Resources) != 2 {
t.Errorf("the joined declaration declares %v, want each path once", at)
}
if !d.Fills("/tmp/r6/zsh/rc") {
t.Errorf("the filled path is no longer named")
}
// The same at the node's own apply: what the others declare is theirs.
trimmed, dropped := ownWithout(own, []delivery.Composed{zsh})
if len(trimmed.Resources) != 1 || len(dropped) != 1 || dropped[0] != "zsh.rc (zsh)" {
t.Errorf("the node's own without zsh's: %d resources, dropped %v", len(trimmed.Resources), dropped)
}
if len(own.Resources) != 2 {
t.Errorf("ownWithout changed its argument")
}
// A node's own composition alone is all of it.
if d, err := declarationOf(own); err != nil || len(d.Resources) != 2 {
t.Errorf("the node's own alone: %v (%v)", d, err)
}
}
// 1945/AC18 (unit): the kept declaration's filled paths go to the module whose file is at each path, the node's own to
// the node's; and what runs from it, joined with the node's composition, names them filled.
func TestTheKeptDeclarationsFilledPathsGoToTheirModules(t *testing.T) {
raw := []byte(`{"resources":[
{"id":"claude-code.settings","type":"file","module":"claude-code","path":"/etc/cc.json"},
{"id":"screen-lock.conf","type":"file","module":"screen-lock","path":"/etc/lock.conf"},
{"id":"host.motd","type":"file","path":"/etc/motd"},
{"id":"zsh.rc","type":"file","module":"zsh","path":"/etc/zshrc"}],
"filled_files":["/etc/cc.json","/etc/lock.conf","/etc/motd"]}`)
by, filled := splitKept(raw)
if len(by["claude-code"]) != 1 || len(by[agent.NodeModule]) != 1 {
t.Fatalf("resources by module: %v", by)
}
if strings.Join(filled["claude-code"], ",") != "/etc/cc.json" || strings.Join(filled["screen-lock"], ",") != "/etc/lock.conf" ||
strings.Join(filled[agent.NodeModule], ",") != "/etc/motd" || len(filled["zsh"]) != 0 {
t.Fatalf("filled by module: %v", filled)
}
cc := delivery.Composed{Node: "n1", Module: "claude-code", Resources: by["claude-code"], Filled: filled["claude-code"]}
node := delivery.Composed{Node: "n1"}
d, err := whatRuns([]delivery.Composed{cc, node}, true, func() (*declaration.Declaration, error) { return nil, errors.New("not read") })
if err != nil || !d.Fills("/etc/cc.json") {
t.Fatalf("what runs does not name claude-code's file filled: %v (%v)", d, err)
}
if splitBy, _ := splitKept([]byte("not json")); splitBy != nil {
t.Errorf("a kept declaration that does not read split into %v", splitBy)
}
}
// 1945/AC17 (unit, review): a resource of the node's own that shares only its target with a module's (another id: a
// unit or a container of the same name) is not the module's, and is neither dropped nor left to be removed.
func TestASharedTargetWithAnotherIDIsNotDropped(t *testing.T) {
ownFile, _ := json.Marshal(map[string]any{"id": "host.zshrc", "type": "file", "path": "/tmp/r6/zsh/rc", "content": "x"})
own := delivery.Composed{Node: "n1", Resources: []json.RawMessage{ownFile}}
zsh := delivery.Composed{Node: "n1", Module: "zsh", FP: "src2:aa", Resources: []json.RawMessage{aFileOf("zsh", "rc")}}
trimmed, dropped := ownWithout(own, []delivery.Composed{zsh})
if len(dropped) != 0 || len(trimmed.Resources) != 1 {
t.Errorf("a resource sharing only its target was dropped: %v", dropped)
}
}
+10 -11
View File
@@ -41,8 +41,9 @@ func rolledBackHosts(statePath string) []string {
return upgrade.RolledBackVersions(verdicts)
}
// proving is this host waiting for the mesh to take a report it made under its own build — the
// evidence its launcher's trial waits for — and acting on it once.
// proving is this host waiting to write a have key under its own build — the evidence its launcher's trial waits for
// (novox/hq ADR 0539 §9, ADR 0227 §8: the component replaced is never the only witness of its successor) — and acting
// on it once.
type proving struct {
statePath string
version string
@@ -56,16 +57,14 @@ func unproved(statePath, version string) bool {
return err != nil || known != version
}
// heard is told every account of the machine the mesh has taken. The first one about a declaration,
// made by this build, proves it.
func (p *proving) heard(r link.Report) {
if r.Declared == "" || r.Refused != "" || r.Host != p.version {
return
}
// haveWritten is told when this build has written a have key, or found the bucket already holding what it would write.
// The first proves it. **A declaration's report does not**: a node-engine that receives none is not on trial for it, and
// one that cannot write its haves is what the launcher keeps rolling back.
func (p *proving) haveWritten() {
p.once.Do(func() {
retired, err := upgrade.Proved(p.statePath, upgrade.VersionsDir(""), p.version)
if err != nil {
p.say(fmt.Sprintf("this host reported under its own build %s, and proving it was not complete: %v", p.version, err))
p.say(fmt.Sprintf("this host wrote its first have under its own build %s, and proving it was not complete: %v", p.version, err))
}
if len(retired) > 0 {
p.say(fmt.Sprintf("host %s is proved; retired the host version(s) %s", p.version, strings.Join(retired, ", ")))
@@ -146,7 +145,7 @@ func launcherReplaced(pid int, proc, path string) bool {
}
// watchWhatThisHostPlaced judges the core builds this host placed, until ctx ends (to-be 45 §8).
func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Queue, say func(string)) {
func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Session, say func(string)) {
host, _ := os.Hostname()
w := &witness.Watcher{
Root: apply.DaemonRoot(), Node: node, Host: host,
@@ -159,7 +158,7 @@ func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Queue
},
// Said now, not at the next report: the verdict is on every report from here, and the mesh
// is asked for one at once.
Concluded: func(link.Rollback) { queue.ReportAsked() },
Concluded: func(link.Rollback) { queue.FactsAsked(ctx) },
Say: say,
}
w.Watch(ctx)
+15 -23
View File
@@ -13,9 +13,9 @@ import (
"github.com/novox/mesh-host/internal/upgrade"
)
// The first account the mesh takes from this build, about a declaration, proves it to the launcher;
// nothing else does — a refusal, a report about no declaration, another build's report (to-be 45 §8).
func TestOnlyAnAccountUnderThisBuildProvesIt(t *testing.T) {
// The first have this build writes proves it to the launcher; nothing else does — not a report about a declaration, not
// a facts event, not the haves of another build (novox/hq ADR 0539 §9, ADR 0227 §8). R6-A AC7.
func TestOnlyAHaveUnderThisBuildProvesIt(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
t.Setenv("MESH_HOST_LIBEXEC", t.TempDir())
// Never the machine's own engine placed by hand: proving renews it (issue 356).
@@ -27,22 +27,20 @@ func TestOnlyAnAccountUnderThisBuildProvesIt(t *testing.T) {
var said []string
p := &proving{statePath: state, version: "2.0", say: func(s string) { said = append(said, s) }}
for _, r := range []link.Report{
{Host: "2.0", Refused: "no"},
{Host: "2.0"},
{Host: "1.0", Declared: "abc"},
} {
p.heard(r)
if !unproved(state, "2.0") {
t.Fatalf("%+v proved this build", r)
}
if got, _ := os.ReadFile(fallback); string(got) != "a stale engine placed by hand" {
t.Fatalf("%+v renewed the engine placed by hand", r)
}
// Facts said, a link opened, a declaration of the old kind reported: none of it is a have.
if !unproved(state, "2.0") {
t.Fatal("a build that wrote no have is proved")
}
p.heard(link.Report{Host: "2.0", Declared: "abc"})
if got, _ := os.ReadFile(fallback); string(got) != "a stale engine placed by hand" {
t.Fatal("the engine placed by hand was renewed before any have")
}
// The session tells the proof when it has a have key saying what it would write.
session := &link.Session{HaveWritten: p.haveWritten}
_ = session
p.haveWritten()
if unproved(state, "2.0") {
t.Fatalf("an account of a declaration under this build did not prove it (%v)", said)
t.Fatalf("the first have under this build did not prove it (%v)", said)
}
if got, _ := upgrade.ReadKnownGood(upgrade.KnownGoodPath(state)); got != "2.0" {
t.Fatalf("known-good is %q", got)
@@ -78,12 +76,6 @@ func TestAStandingRollbackIsSaidOnEveryReport(t *testing.T) {
got[0].Outcome != link.RolledBack {
t.Fatalf("what the report says is %+v", got)
}
if !worthSaying(link.Report{Rollbacks: got}) {
t.Fatal("a reconcile with a rollback standing does not say it")
}
if adoptionFingerprint(link.Report{Rollbacks: got}) == adoptionFingerprint(link.Report{}) {
t.Fatal("a rollback reached or ended is not news to the reconcile")
}
}
// The launcher running this host is known to have been replaced only when it certainly was: a shell
+1 -1
View File
@@ -221,7 +221,7 @@
"path": "/var/lib/nats-module/conf/accounts.conf",
"mode": "0600",
"create-once": true,
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_assignments.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_declared-config.>\", \"$KV.mesh-controller_lease.>\", \"$KV.mesh-controller_predefined.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.assignment-changed\", \"mesh.seat.mesh-controller.event.declared-config-changed\", \"mesh.seat.mesh-controller.event.predefined-changed\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.operator-channel.tool.tell\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_assignments.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_declared-config.>\", \"$KV.mesh-controller_lease.>\", \"$KV.mesh-controller_predefined.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.assignment-changed\", \"mesh.seat.mesh-controller.event.declared-config-changed\", \"mesh.seat.mesh-controller.event.predefined-changed\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.operator-channel.tool.tell\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
// The upstream server, until the broker step adopts it: the `nats` module's own image, built by the
// mesh, takes its place there once, on these same directories (ADR 0333 §3), as the bus's planned
+7 -1
View File
@@ -3,14 +3,20 @@ module github.com/novox/mesh-host
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021153908-725590e06e75
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
golang.org/x/sys v0.48.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/time v0.15.0 // indirect
)
+21 -2
View File
@@ -1,7 +1,23 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021153908-725590e06e75 h1:AcYfyJpYaUufhI93VG76+b001msXY4XX7kuRjnf33HY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021153908-725590e06e75/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f h1:lUEyNPPF3w4Q0LIgcUUD6ofkKVhCF0n7dVjMoixQmXo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456 h1:He2o9vSKjgKbWOCOgOJY0WlaDh1n8RcGhvGY/dwXsBQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e h1:k/1NjjW5HuuuHW2ldyWTvjp+1u3fN0RO18Q57Iup6Dk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936 h1:HGBoWimZXYZp+Qn9VCstYDitOP84t9EUQYd1yM9j2eg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
@@ -10,5 +26,8 @@ github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+927
View File
@@ -0,0 +1,927 @@
// Package agent is the node agent (novox/hq ADR 0429 §4, ADR 0539): the node-engine's own process that follows the
// node's wants and applies one module at a time.
//
// **It watches, it never polls.** A want `want.<node>.<module>` names a fingerprint and the digest of the one composition
// the assigner allows the node to apply for it; the composition itself waits under `<node>.<module>.<fingerprint>` in
// the node's bucket `composed-<node>`. When both are there and the composition's digest is the want's, and what runs
// differs, the agent applies that module's composition and writes its have with the fingerprint and the digest. A want
// `removed` removes that module's own resources. Nothing is applied for a composition no want names, and a failed apply
// is retried by the next change of either key, never by a clock (research 063 §9.1) — except an apply that failed only
// because the bus did not answer for a moment (Result.Transient: no responders while the bus reloads its accounts, a
// reconnect): that one is retried with a bounded backoff (Env.Backoff), and at once when the engine is told the bus is
// back (RetryTransient), since no key will change for it.
package agent
import (
"bytes"
"context"
"crypto/ed25519"
"encoding/json"
"fmt"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Update is one change of a key in a watched bucket. Caught marks the end of the keys that were there when the watch
// began; every Update after it is a change.
type Update struct {
Key string
Value []byte
Revision uint64
Deleted bool
Caught bool
}
// Watcher follows one bucket: the keys matching the pattern now, a Caught update, then every change, until ctx ends.
type Watcher interface {
Watch(ctx context.Context, pattern string) (<-chan Update, error)
}
// NodeModule is the scope of the node's own composition, `<node>.node`.
const NodeModule = "node"
// Have is what a module's have key says, as the agent reads and writes it.
type Have struct {
Module string
FP string
Composed string
// Phase is "running", "failed", "held" or "removed"; empty reads as running.
Phase string
Detail string
}
// Haves is the node's have bucket as the agent uses it.
type Haves interface {
// Read is what the node-engine last wrote for a module, and whether it wrote anything.
Read(ctx context.Context, module string) (Have, bool)
// Write writes a module's have where it differs from the bucket's.
Write(ctx context.Context, h Have)
// Removed writes the module's have with the phase removed, once, if a key of it exists.
Removed(ctx context.Context, module string)
}
// Result is how one apply or removal came out.
type Result struct {
Failed bool
// Transient says the failure was the bus not answering for a moment (no responders, a timeout, a reconnect, a
// JetStream API error while the bus reloads), not the module's: the apply is tried again without a new revision.
Transient bool
Detail string
}
// Backup is the answer of the backup asked before a module moves.
type Backup struct {
Done bool
Words string
}
// Env is everything the agent needs of the node.
type Env struct {
Node string
Wants Watcher
Composed Watcher
Haves Haves
// Apply applies one module's composition (scope NodeModule for the node's own), given the compositions the node's
// other modules run so a path, unit or group they hold is not removed or taken. It is called with one module at a
// time.
Apply func(ctx context.Context, c delivery.Composed, others []delivery.Composed) Result
// Remove removes the module's own resources and nothing of another's.
Remove func(ctx context.Context, module string, others []delivery.Composed) Result
// Backup backs a module up before a new composition of it is applied; nil backs up nothing.
Backup func(ctx context.Context, module string) Backup
// Kept is the resources the node last applied, by module, from the declaration it kept before the switch (ADR 0539
// §9), as JSON; the node's own are under NodeModule. Nil when the node kept none. A module the agent does not hold yet
// still runs from it.
Kept func() map[string][]json.RawMessage
// KeptFilled is the paths the kept declaration names as filled by the provisioner, by the module whose file is at
// each (the node's own under NodeModule). A module that still runs from the kept declaration keeps its filled files
// filled: without them the filling refused each one as "not named as filled" (issue 1945). Nil names none.
KeptFilled func() map[string][]string
// Trusted is the public keys a composition may be signed by: the controller's and the operator's (ADR 0539 §3). A
// composition unsigned, signed by another key, or changed after it was signed is refused and nothing of it is
// applied. Nil trusts no key, so nothing is applied.
Trusted func() []ed25519.PublicKey
// Recorded says the engine holds a record of every resource of the composition, so it was applied and not removed
// since; nil says yes. A composition the kept declaration or a have says runs is applied again when it says no.
Recorded func(ctx context.Context, c delivery.Composed) bool
// Revisions are a composition key's revisions the bucket keeps, oldest first; nil reads none. The composition a want
// names is taken from them when a newer one replaced it under the same key (ADR 0539 clause 1).
Revisions func(ctx context.Context, key string) ([]Update, error)
// Settled is told a module's apply or removal that succeeded, once what runs holds it (Everything), outside the
// agent's lock: the declaration the node applies now names what that composition names, its filled files among
// them. The node-engine's filling tells the provisioner's client then to fill again what it refused (mesh-host
// internal/filling Knock; ace, 2026-10-25 21:17). Nil tells nothing.
Settled func(module string)
// Backoff is how long to wait before the try-th retry (1, 2, …) of an apply that failed transiently. Nil retries a
// transient failure only on RetryTransient or a change of either key.
Backoff func(try int) time.Duration
Say func(string)
}
// transientFor is how many transient retries say the phase applying before the have says failed: past them the bus has
// not answered for long enough that it is no longer a moment, and the failure is loud while the retries go on.
const transientFor = 4
// Agent follows the wants.
type Agent struct {
env Env
mu sync.Mutex
wants map[string]wantNow // by module
comps map[string]compNow // by composed key
running map[string]delivery.Have
// applied is the composition each module was last applied from: a revision a want named from the key's history is
// not the key's latest value, and what it holds is still held out of the others' applies.
applied map[string]compNow
// attempt is the last apply of a module that failed or is held, with the revisions it was made from.
attempt map[string]attempt
checked map[string]bool
// prior is a module the node ran before this agent started, so it has something to back up.
prior map[string]bool
said map[string]string
retry chan []string
// tries counts the transient failures in a row of each module; again carries a retry whose wait ran out, and
// transient asks for every transiently failed module at once. stopped is closed when Run returns.
tries map[string]int
gen uint64
again chan againAsk
transient chan struct{}
stopped chan struct{}
}
type againAsk struct {
module string
gen uint64
}
type wantNow struct {
Want delivery.Want
Node bool // the node's own want
Present bool
Rev uint64
}
type compNow struct {
Raw []byte
C delivery.Composed
Dig string
Rev uint64
Err error
}
type attempt struct {
FP, Composed string
WantRev, CompRev uint64
Held bool
AnsweredRev uint64
// Transient marks a failure of the bus, retried by the clock; Gen names the retry that is due for it.
Transient bool
Gen uint64
}
// New is an agent for a node.
func New(env Env) *Agent {
if env.Say == nil {
env.Say = func(string) {}
}
return &Agent{env: env, wants: map[string]wantNow{}, comps: map[string]compNow{}, running: map[string]delivery.Have{}, applied: map[string]compNow{},
attempt: map[string]attempt{}, checked: map[string]bool{}, prior: map[string]bool{}, said: map[string]string{},
retry: make(chan []string, 8), tries: map[string]int{}, again: make(chan againAsk, 64),
transient: make(chan struct{}, 1), stopped: make(chan struct{})}
}
// Run follows the wants until ctx ends. The keys there when it starts are taken first, the node's own composition
// before any module's, then each change as it comes.
func (a *Agent) Run(ctx context.Context) error {
defer close(a.stopped)
wants, err := a.env.Wants.Watch(ctx, "want."+a.env.Node+".>")
if err != nil {
return fmt.Errorf("the agent cannot watch its wants: %w", err)
}
comps, err := a.env.Composed.Watch(ctx, a.env.Node+".>")
if err != nil {
return fmt.Errorf("the agent cannot watch its compositions: %w", err)
}
var wantsCaught, compsCaught bool
dirty := map[string]bool{}
for {
select {
case <-ctx.Done():
return nil
case modules := <-a.retry:
// Applied again though the keys did not change: a container that waited because it said busy (ADR 0360
// §11) is replaced once it is idle.
a.mu.Lock()
for _, m := range modules {
delete(a.running, m)
delete(a.attempt, m)
dirty[m] = true
}
a.mu.Unlock()
case ask := <-a.again:
// The wait of a transiently failed apply ran out: tried again from the keys it has, if nothing tried it since.
a.mu.Lock()
if att, ok := a.attempt[ask.module]; ok && att.Transient && att.Gen == ask.gen {
delete(a.attempt, ask.module)
dirty[ask.module] = true
}
a.mu.Unlock()
case <-a.transient:
a.mu.Lock()
for m, att := range a.attempt {
if att.Transient {
delete(a.attempt, m)
dirty[m] = true
}
}
a.mu.Unlock()
case u, ok := <-wants:
if !ok {
return fmt.Errorf("the watch of the wants ended")
}
if u.Caught {
wantsCaught = true
} else if m := a.takeWant(u); m != "" {
dirty[m] = true
}
case u, ok := <-comps:
if !ok {
return fmt.Errorf("the watch of the compositions ended")
}
if u.Caught {
compsCaught = true
} else if m := a.takeComposition(u); m != "" {
dirty[m] = true
// The node's own may wait for this one to read (unreadableModules): looked at again with it.
dirty[NodeModule] = true
}
}
// Nothing is applied until what was there at the start is all in: a module's composition may arrive after its
// want, and the node's own composition comes before any module's.
if !(wantsCaught && compsCaught) || len(dirty) == 0 {
continue
}
a.settle(ctx, dirty)
dirty = map[string]bool{}
}
}
// Retry asks for the modules to be applied again from the keys they have, though nothing changed: what ADR 0360 §11
// needs for a container that waits because it said busy. It never blocks; a retry asked while eight wait is dropped, and
// asked again by its caller's next look.
func (a *Agent) Retry(modules ...string) {
select {
case a.retry <- modules:
default:
}
}
// RetryTransient asks for every module whose last apply failed transiently to be applied again now: the engine says it
// when it knows the bus answers again (after it wrote the bus's user list and the bus reloaded its accounts). It never
// blocks.
func (a *Agent) RetryTransient() {
select {
case a.transient <- struct{}{}:
default:
}
}
// Running is the compositions the node runs now: each module's that the agent applied or found already running, and
// the node's own. What the node judges its health by and says of itself is built from them.
func (a *Agent) Running() []delivery.Composed {
a.mu.Lock()
defer a.mu.Unlock()
mods := make([]string, 0, len(a.running))
for m := range a.running {
mods = append(mods, m)
}
sort.Strings(mods)
var out []delivery.Composed
for _, m := range mods {
h := a.running[m]
if h.Phase == delivery.PhaseRemoved {
continue
}
if c, ok := a.compositionFor(m, h.FP); ok && c.Err == nil && c.Dig == h.Composed {
out = append(out, c.C)
} else if c, ok := a.applied[m]; ok && c.Dig == h.Composed {
out = append(out, c.C)
}
}
return out
}
// takeWant takes a change of a want key and returns the module it concerns, or "" for a key that is not a want.
func (a *Agent) takeWant(u Update) string {
a.mu.Lock()
defer a.mu.Unlock()
if _, err := delivery.ParseWantNodeKey(u.Key); err == nil {
w := wantNow{Node: true, Rev: u.Revision}
if !u.Deleted {
nw, err := delivery.DecodeNodeWant(u.Value)
if err != nil {
a.sayOnce("want:node", fmt.Sprintf("the node's own want is not one: %v", err))
return ""
}
w.Present, w.Want = true, delivery.Want{FP: "", Composed: nw.Composed}
}
a.wants[NodeModule] = w
return NodeModule
}
node, module, err := delivery.ParseWantKey(u.Key)
if err != nil || node != a.env.Node {
return ""
}
w := wantNow{Rev: u.Revision}
if !u.Deleted {
want, err := delivery.DecodeWant(u.Value)
if err != nil {
a.sayOnce("want:"+module, fmt.Sprintf("the want of %s is not one: %v", module, err))
return ""
}
w.Present, w.Want = true, want
}
a.wants[module] = w
return module
}
// takeComposition takes a change of a composition key and returns the module it concerns.
func (a *Agent) takeComposition(u Update) string {
node, module, _, isNode, err := delivery.ParseComposedKey(u.Key)
if err != nil || node != a.env.Node {
return ""
}
a.mu.Lock()
defer a.mu.Unlock()
if isNode {
module = NodeModule
}
if u.Deleted {
delete(a.comps, u.Key)
return module
}
now := compNow{Raw: u.Value, Rev: u.Revision}
c, err := delivery.DecodeComposed(u.Value)
if err == nil {
var trusted []ed25519.PublicKey
if a.env.Trusted != nil {
trusted = a.env.Trusted()
}
// The signature before anything of the composition is used: its digest is what the want names.
now.C = c
now.Dig, err = delivery.VerifyComposed(c, trusted)
if err != nil {
now.C = delivery.Composed{}
a.sayOnce("verify:"+u.Key, fmt.Sprintf("the composition %s is refused and nothing of it is applied: %v", u.Key, err))
err = fmt.Errorf("refused: %w", err)
}
} else {
a.sayOnce("decode:"+u.Key, fmt.Sprintf("the composition %s cannot be read, and nothing of it is applied: %v", u.Key, err))
}
now.Err = err
a.comps[u.Key] = now
return module
}
func (a *Agent) sayOnce(key, line string) {
if a.said[key] == line {
return
}
a.said[key] = line
a.env.Say(line)
}
// settle applies what the dirty modules' wants allow: the node's own first, then each module in name order.
func (a *Agent) settle(ctx context.Context, dirty map[string]bool) {
order := make([]string, 0, len(dirty))
for m := range dirty {
order = append(order, m)
}
sort.Slice(order, func(i, j int) bool {
if (order[i] == NodeModule) != (order[j] == NodeModule) {
return order[i] == NodeModule
}
return order[i] < order[j]
})
for _, m := range order {
if ctx.Err() != nil {
return
}
a.settleOne(ctx, m)
}
}
func (a *Agent) compositionFor(module, fp string) (compNow, bool) {
var key string
var err error
if module == NodeModule {
key, err = delivery.ComposedNodeKey(a.env.Node)
} else {
key, err = delivery.ComposedKey(a.env.Node, module, fp)
}
if err != nil {
return compNow{}, false
}
c, ok := a.comps[key]
return c, ok
}
// revisionNamed is the revision of a module's composition key whose digest is the one given, signed by a trusted key,
// from the revisions the bucket keeps; false when none is.
func (a *Agent) revisionNamed(ctx context.Context, module, fp, digest string) (compNow, bool) {
if a.env.Revisions == nil {
return compNow{}, false
}
var key string
var err error
if module == NodeModule {
key, err = delivery.ComposedNodeKey(a.env.Node)
} else {
key, err = delivery.ComposedKey(a.env.Node, module, fp)
}
if err != nil {
return compNow{}, false
}
revs, err := a.env.Revisions(ctx, key)
if err != nil {
a.mu.Lock()
a.sayOnce("history:"+key, fmt.Sprintf("the revisions of %s cannot be read: %v", key, err))
a.mu.Unlock()
return compNow{}, false
}
var trusted []ed25519.PublicKey
if a.env.Trusted != nil {
trusted = a.env.Trusted()
}
for i := len(revs) - 1; i >= 0; i-- {
u := revs[i]
if u.Deleted || u.Key != key {
continue
}
c, err := delivery.DecodeComposed(u.Value)
if err != nil {
continue
}
dig, err := delivery.VerifyComposed(c, trusted)
if err != nil || dig != digest {
continue
}
return compNow{Raw: u.Value, C: c, Dig: dig, Rev: u.Revision}, true
}
return compNow{}, false
}
// others are what the node's other modules run now, so what they hold is not taken: the compositions the agent runs and,
// for every module the agent does not hold yet, the resources the kept declaration has for it (ADR 0429 §4).
func (a *Agent) others(except string) []delivery.Composed {
var out []delivery.Composed
mods := make([]string, 0, len(a.running))
for m := range a.running {
if m != except {
mods = append(mods, m)
}
}
sort.Strings(mods)
for _, m := range mods {
h := a.running[m]
if h.Phase == delivery.PhaseRemoved {
continue
}
if c, ok := a.compositionFor(m, h.FP); ok && c.Err == nil && c.Dig == h.Composed {
out = append(out, c.C)
} else if c, ok := a.applied[m]; ok && c.Dig == h.Composed {
out = append(out, c.C)
}
}
if a.env.Kept != nil {
kept := a.env.Kept()
names := make([]string, 0, len(kept))
for m := range kept {
if _, held := a.running[m]; !held && m != except {
names = append(names, m)
}
}
sort.Strings(names)
for _, m := range names {
c := delivery.Composed{Node: a.env.Node, Module: m, Resources: kept[m]}
if a.env.KeptFilled != nil {
c.Filled = a.env.KeptFilled()[m]
}
if m == NodeModule {
c.Module = ""
}
out = append(out, c)
}
}
return out
}
// unreadableModules are the keys of the module compositions in the node's bucket that cannot be read or are refused, of a
// module the assigner has not removed: for each module only the composition its want names, else its newest, since a
// stale one nobody applies (written before a rule the SDK now holds) must not hold the node's own back. Called holding
// a.mu.
func (a *Agent) unreadableModules() []string {
type pick struct {
key string
c compNow
}
current := map[string]pick{}
for key, c := range a.comps {
_, m, fp, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode {
continue
}
w := a.wants[m]
if w.Present && w.Want.Removed {
continue
}
if w.Present && w.Want.FP != "" {
if w.Want.FP == fp {
current[m] = pick{key, c}
}
continue
}
if prev, ok := current[m]; !ok || c.Rev > prev.c.Rev {
current[m] = pick{key, c}
}
}
var out []string
for _, p := range current {
if p.c.Err != nil {
out = append(out, p.key)
}
}
sort.Strings(out)
return out
}
// heldOut is what an apply or a removal holds out: others, and every module whose own composition the node's bucket holds
// and the agent does not run yet. A module that left the node's own composition for one of its own keeps its resources
// until that one is applied, so the node's own apply never removes them in between (issue 1945, live on 2026-10-25:
// shanks lost systemd-resolved's guard while its composition waited for a credential). The composition its want names,
// else its newest. Not part of Everything: what has not been applied is not what runs.
func (a *Agent) heldOut(except string) []delivery.Composed {
out := a.others(except)
held := map[string]bool{}
for _, c := range out {
held[c.Module] = true
}
waiting := map[string]compNow{}
for key, c := range a.comps {
_, m, fp, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode || c.Err != nil || m == except || held[m] {
continue
}
if h, ok := a.running[m]; ok && h.Phase != delivery.PhaseRemoved {
continue
}
if w := a.wants[m]; w.Present && w.Want.Removed {
continue // a module the assigner removed holds nothing back
}
if w := a.wants[m]; w.Present && !w.Want.Removed && w.Want.FP == fp {
waiting[m] = c
continue
}
if w := a.wants[m]; w.Present && !w.Want.Removed && waiting[m].Rev != 0 && waiting[m].C.FP == w.Want.FP {
continue
}
if prev, ok := waiting[m]; !ok || c.Rev > prev.Rev {
waiting[m] = c
}
}
names := make([]string, 0, len(waiting))
for m := range waiting {
names = append(names, m)
}
sort.Strings(names)
for _, m := range names {
out = append(out, waiting[m].C)
}
return out
}
// Everything is what the node runs: the compositions the agent runs, and the kept declaration's resources for each module
// the agent does not hold yet. What the node judges its health by, and what the filling, the take act and the judgement
// of directories read, is built from it.
func (a *Agent) Everything() []delivery.Composed {
a.mu.Lock()
defer a.mu.Unlock()
return a.others("\x00")
}
// HoldsNode says whether the agent runs the node's own composition, so the node's parts are its and not the kept
// declaration's.
func (a *Agent) HoldsNode() bool {
a.mu.Lock()
defer a.mu.Unlock()
h, ok := a.running[NodeModule]
return ok && h.Phase != delivery.PhaseRemoved
}
func (a *Agent) settleOne(ctx context.Context, module string) {
a.mu.Lock()
w := a.wants[module]
a.mu.Unlock()
if !w.Present {
return
}
if w.Want.Removed {
a.remove(ctx, module, w)
return
}
a.mu.Lock()
comp, have := a.compositionFor(module, w.Want.FP)
a.mu.Unlock()
if (!have || comp.Err != nil || comp.Dig != w.Want.Composed) && w.Want.Composed != "" {
// The composition the want names may be an older revision of the key, replaced by a newer one at the same
// fingerprint (a want held there keeps its digest): taken from the revisions the bucket keeps (ADR 0539 clause 1).
if old, found := a.revisionNamed(ctx, module, w.Want.FP, w.Want.Composed); found {
comp, have = old, true
}
}
if !have || comp.Err != nil {
return // waits for the composition; a change of its key settles the module again
}
if comp.Dig != w.Want.Composed {
// A composition whose digest no want names is not applied.
a.mu.Lock()
a.sayOnce("digest:"+module, fmt.Sprintf("the composition of %s at %s has digest %s, and its want names %s; nothing is applied until they agree",
module, w.Want.FP, comp.Dig, w.Want.Composed))
a.mu.Unlock()
return
}
a.mu.Lock()
if !a.checked[module] {
a.checked[module] = true
a.mu.Unlock()
if a.baseline(ctx, module, w, comp) {
return
}
} else {
a.mu.Unlock()
}
a.mu.Lock()
if r, ok := a.running[module]; ok && r.FP == w.Want.FP && r.Composed == w.Want.Composed && r.Phase != delivery.PhaseRemoved {
a.mu.Unlock()
return
}
att, tried := a.attempt[module]
same := tried && att.FP == w.Want.FP && att.Composed == w.Want.Composed
if same && !att.Held && att.WantRev == w.Rev && att.CompRev == comp.Rev {
a.mu.Unlock()
return // failed already for exactly these keys: retried by the next change of either
}
if same && att.Held && att.AnsweredRev >= w.Rev && att.CompRev == comp.Rev {
a.mu.Unlock()
return // held for a failed backup, and no answer since
}
if module == NodeModule {
// Not while a module's composition cannot be read or is refused: what it carries may be what left the node's own,
// and the node's own apply would remove it, held out by nothing (issue 1945, the review's gap 2).
if bad := a.unreadableModules(); len(bad) > 0 {
a.sayOnce("node-waits:"+strings.Join(bad, ","), fmt.Sprintf("the node's own composition is not applied while "+
"%s cannot be read or is refused: what it carries would be removed", strings.Join(bad, ", ")))
a.mu.Unlock()
return
}
}
others := a.heldOut(module)
a.mu.Unlock()
a.apply(ctx, module, w, comp, others)
}
// baseline decides whether what runs already is what the want names, once per module at the agent's start: the node's
// own have (what this node-engine last applied), else the declaration it kept before the switch (ADR 0539 §9). True when
// it is, with the have written; nothing is applied.
func (a *Agent) baseline(ctx context.Context, module string, w wantNow, comp compNow) bool {
// What a have or the kept declaration says runs is believed only when the engine still holds a record of each of its
// resources: a resource removed since (a node's own apply took it, issue 1945) is made again by an apply.
if a.env.Recorded != nil && !a.env.Recorded(ctx, comp.C) {
if module != NodeModule {
if _, ok := a.env.Haves.Read(ctx, module); ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
}
if a.env.Kept != nil {
if _, ok := a.env.Kept()[module]; ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
}
a.mu.Lock()
a.sayOnce("unrecorded:"+module+":"+comp.Dig, fmt.Sprintf("%s's composition %s is applied again: this engine holds "+
"no record of some of its resources", module, short(comp.Dig)))
a.mu.Unlock()
return false
}
if module != NodeModule {
if h, ok := a.env.Haves.Read(ctx, module); ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
if h.Composed != "" {
// This node-engine wrote it: what it says is what runs, and the kept declaration says nothing more. A
// failed apply or a different composition is applied again.
if h.FP == w.Want.FP && h.Composed == w.Want.Composed &&
(h.Phase == "" || h.Phase == string(delivery.PhaseRunning)) {
a.mu.Lock()
a.running[module] = delivery.Have{FP: h.FP, Composed: h.Composed, Phase: delivery.PhaseRunning}
a.mu.Unlock()
// Said again, so the session holds the module and its key carries the health this engine judges,
// not the one an older engine left there (xorg's unknown on g14 and shanks, 2026-10-25).
a.env.Haves.Write(ctx, Have{Module: module, FP: h.FP, Composed: h.Composed,
Phase: string(delivery.PhaseRunning), Detail: h.Detail})
return true
}
return false
}
}
}
if a.env.Kept == nil {
return false
}
kept, ok := a.env.Kept()[module]
if ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
if !ok || !SameResources(kept, comp.C.Resources) {
return false
}
a.mu.Lock()
a.running[module] = delivery.Have{FP: w.Want.FP, Composed: w.Want.Composed, Phase: delivery.PhaseRunning}
a.mu.Unlock()
a.env.Haves.Write(ctx, Have{Module: module, FP: w.Want.FP, Composed: w.Want.Composed, Phase: string(delivery.PhaseRunning)})
return true
}
func (a *Agent) apply(ctx context.Context, module string, w wantNow, comp compNow, others []delivery.Composed) {
fp, dig := w.Want.FP, w.Want.Composed
a.mu.Lock()
_, ranBefore := a.running[module]
ranBefore = ranBefore || a.prior[module]
a.mu.Unlock()
// A data module is backed up before a new composition of it moves (ADR 0429 §14), unless the want says to go on
// without one (ADR 0539 §16).
if comp.C.Data && ranBefore && a.env.Backup != nil && w.Want.Backup != delivery.BackupSkip {
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseBackingUp)})
if got := a.env.Backup(ctx, module); !got.Done {
a.env.Say(fmt.Sprintf("%s is held: its backup before %s failed: %s", module, fp, got.Words))
a.mu.Lock()
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev, Held: true, AnsweredRev: w.Rev}
prev := a.running[module]
a.mu.Unlock()
// It keeps running what it has: the have says what it holds and why it waits.
a.env.Haves.Write(ctx, Have{Module: module, FP: prev.FP, Composed: prev.Composed, Phase: string(delivery.PhaseHeld),
Detail: got.Words})
return
}
}
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseApplying)})
res := a.env.Apply(ctx, comp.C, others)
a.mu.Lock()
what := fmt.Sprintf("%s at %s", module, fp)
if module == NodeModule {
what = "the node's own composition"
}
if res.Failed && res.Transient {
a.tries[module]++
try := a.tries[module]
a.gen++
gen := a.gen
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev, Transient: true, Gen: gen}
a.mu.Unlock()
phase, when := delivery.PhaseApplying, "when the bus answers again"
if a.env.Backoff != nil {
wait := a.env.Backoff(try)
when = "in " + wait.String()
time.AfterFunc(wait, func() {
select {
case a.again <- againAsk{module, gen}:
case <-a.stopped:
case <-ctx.Done():
}
})
}
if try > transientFor {
phase = delivery.PhaseFailed // still retried, and said as failed: no longer a moment
}
a.env.Say(fmt.Sprintf("applying %s failed because the bus did not answer (try %d), retried %s: %s", what, try, when, res.Detail))
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(phase),
Detail: fmt.Sprintf("retrying %s (try %d): %s", when, try, res.Detail)})
return
}
delete(a.tries, module)
if res.Failed {
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev}
a.mu.Unlock()
a.env.Say(fmt.Sprintf("applying %s failed: %s", what, res.Detail))
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed), Detail: res.Detail})
return
}
delete(a.attempt, module)
a.running[module] = delivery.Have{FP: fp, Composed: dig, Phase: delivery.PhaseRunning}
a.applied[module] = comp
a.mu.Unlock()
if a.env.Settled != nil {
a.env.Settled(module)
}
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseRunning), Detail: res.Detail})
}
func (a *Agent) remove(ctx context.Context, module string, w wantNow) {
a.mu.Lock()
att := a.attempt[module]
r, ran := a.running[module]
if att.FP == "removed" && att.WantRev == w.Rev {
a.mu.Unlock()
return
}
if ran && r.Phase == delivery.PhaseRemoved {
a.mu.Unlock()
return
}
others := a.heldOut(module)
a.mu.Unlock()
res := a.env.Remove(ctx, module, others)
a.mu.Lock()
if res.Failed {
a.attempt[module] = attempt{FP: "removed", WantRev: w.Rev}
a.mu.Unlock()
a.env.Say(fmt.Sprintf("removing %s failed: %s", module, res.Detail))
return
}
delete(a.attempt, module)
a.running[module] = delivery.Have{Phase: delivery.PhaseRemoved}
a.mu.Unlock()
if a.env.Settled != nil {
a.env.Settled(module)
}
a.env.Haves.Removed(ctx, module)
}
// SameResources says whether two lists of resources hold the same resources, each by its id and its canonical JSON.
func SameResources(a, b []json.RawMessage) bool {
ma, ok := canonByID(a)
if !ok {
return false
}
mb, ok := canonByID(b)
if !ok || len(ma) != len(mb) {
return false
}
for id, x := range ma {
if y, there := mb[id]; !there || x != y {
return false
}
}
return true
}
func canonByID(rs []json.RawMessage) (map[string]string, bool) {
out := map[string]string{}
for _, raw := range rs {
var head struct {
ID string `json:"id"`
}
if json.Unmarshal(raw, &head) != nil || head.ID == "" {
return nil, false
}
canon, err := Canonical(raw)
if err != nil {
return nil, false
}
out[head.ID] = string(canon)
}
return out, true
}
// Canonical is raw with its object keys sorted and its spacing dropped, its numbers kept as written.
func Canonical(raw json.RawMessage) (json.RawMessage, error) {
var v any
d := json.NewDecoder(bytes.NewReader(raw))
d.UseNumber()
if err := d.Decode(&v); err != nil {
return nil, err
}
return json.Marshal(v)
}
func short(s string) string {
if i := strings.IndexByte(s, ':'); i >= 0 {
s = s[i+1:]
}
if len(s) > 12 {
s = s[:12]
}
return s
}
+954
View File
@@ -0,0 +1,954 @@
package agent
import (
"context"
"crypto/ed25519"
"encoding/json"
"slices"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const node = "n1"
// The keys of the tests: the controller's and the operator's are trusted, a stranger's is not.
var controllerPub, controllerKey = newKey()
var operatorPub, operatorKey = newKey()
var _, strangerKey = newKey()
func newKey() (ed25519.PublicKey, ed25519.PrivateKey) {
pub, key, err := ed25519.GenerateKey(nil)
if err != nil {
panic(err)
}
return pub, key
}
func fpOf(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
// A watcher the test writes to; it says Caught once the keys written before the watch began are out.
type feed struct {
mu sync.Mutex
ch chan Update
rev uint64
first []Update
}
func newFeed() *feed { return &feed{ch: make(chan Update, 256)} }
func (f *feed) Watch(_ context.Context, _ string) (<-chan Update, error) {
f.mu.Lock()
defer f.mu.Unlock()
for _, u := range f.first {
f.ch <- u
}
f.ch <- Update{Caught: true}
return f.ch, nil
}
// before writes a key as if it was there when the watch began.
func (f *feed) before(key string, value []byte) {
f.mu.Lock()
defer f.mu.Unlock()
f.rev++
f.first = append(f.first, Update{Key: key, Value: value, Revision: f.rev})
}
func (f *feed) put(key string, value []byte) {
f.mu.Lock()
f.rev++
r := f.rev
f.mu.Unlock()
f.ch <- Update{Key: key, Value: value, Revision: r}
}
type haves struct {
mu sync.Mutex
stored map[string]Have
log []Have
removed []string
}
func (h *haves) Read(_ context.Context, module string) (Have, bool) {
h.mu.Lock()
defer h.mu.Unlock()
x, ok := h.stored[module]
return x, ok
}
func (h *haves) Write(_ context.Context, x Have) {
h.mu.Lock()
defer h.mu.Unlock()
if h.stored == nil {
h.stored = map[string]Have{}
}
h.stored[x.Module] = x
h.log = append(h.log, x)
}
func (h *haves) Removed(_ context.Context, m string) {
h.mu.Lock()
defer h.mu.Unlock()
h.removed = append(h.removed, m)
}
func (h *haves) last(module string) Have {
h.mu.Lock()
defer h.mu.Unlock()
return h.stored[module]
}
type bed struct {
t *testing.T
wants *feed
comps *feed
haves *haves
mu sync.Mutex
applied []string
removed []string
backups []string
failWith string
// transient marks failWith as the bus not answering; failTimes > 0 fails only the first that many applies.
transient bool
failTimes int
backup func(module string) Backup
kept map[string][]json.RawMessage
others [][]delivery.Composed
said []string
// settled is each module Settled was told, and settledRan whether what runs held it then.
settled []string
settledRan []bool
ag *Agent // the agent run() runs
env Env
}
func newBed(t *testing.T) *bed {
b := &bed{t: t, wants: newFeed(), comps: newFeed(), haves: &haves{}}
b.env = Env{Node: node, Wants: b.wants, Composed: b.comps, Haves: b.haves,
Apply: func(_ context.Context, c delivery.Composed, others []delivery.Composed) Result {
b.mu.Lock()
defer b.mu.Unlock()
b.others = append(b.others, others)
name := c.Module
if name == "" {
name = NodeModule
}
b.applied = append(b.applied, name)
if b.failWith != "" && (b.failTimes == 0 || len(b.applied) <= b.failTimes) {
return Result{Failed: true, Transient: b.transient, Detail: b.failWith}
}
return Result{}
},
Remove: func(_ context.Context, m string, _ []delivery.Composed) Result {
b.mu.Lock()
defer b.mu.Unlock()
b.removed = append(b.removed, m)
return Result{}
},
Backup: func(_ context.Context, m string) Backup {
b.mu.Lock()
defer b.mu.Unlock()
b.backups = append(b.backups, m)
if b.backup != nil {
return b.backup(m)
}
return Backup{Done: true}
},
Settled: func(m string) {
ran := false
if b.ag != nil {
for _, c := range b.ag.Everything() {
ran = ran || c.Module == m
}
}
b.mu.Lock()
defer b.mu.Unlock()
b.settled = append(b.settled, m)
b.settledRan = append(b.settledRan, ran)
},
Kept: func() map[string][]json.RawMessage { return b.kept },
Trusted: func() []ed25519.PublicKey { return []ed25519.PublicKey{controllerPub, operatorPub} },
Say: func(s string) {
t.Log(s)
b.mu.Lock()
b.said = append(b.said, s)
b.mu.Unlock()
},
}
return b
}
func (b *bed) run() (stop func()) {
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
b.ag = New(b.env)
go func() { defer close(done); _ = b.ag.Run(ctx) }()
return func() { cancel(); <-done }
}
func file(module, id, content string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": module + "." + id, "type": "file", "path": "/tmp/r6/" + module + "/" + id,
"content": content, "module": module})
return raw
}
// composition is a module's composition at a fingerprint: its key, its bytes and its digest, from the writer's code.
func composition(t *testing.T, module, fp string, data bool, rs ...json.RawMessage) (key string, raw []byte, digest string) {
t.Helper()
c := delivery.Composed{Node: node, Module: module, FP: fp, Data: data, Resources: rs}
var err error
if module == NodeModule {
c.Module, c.FP = "", ""
key, err = delivery.ComposedNodeKey(node)
} else {
key, err = delivery.ComposedKey(node, module, fp)
}
if err != nil {
t.Fatal(err)
}
return signedBy(t, c, key, controllerKey)
}
// signedBy is c signed with signer, as the writer's code signs it: its key, its bytes and its digest.
func signedBy(t *testing.T, c delivery.Composed, key string, signer ed25519.PrivateKey) (string, []byte, string) {
t.Helper()
signed, err := delivery.SignComposed(c, signer)
if err != nil {
t.Fatal(err)
}
raw, err := delivery.EncodeComposed(signed)
if err != nil {
t.Fatal(err)
}
digest, err := delivery.ComposedDigestOf(signed)
if err != nil {
t.Fatal(err)
}
return key, raw, digest
}
func wantFor(t *testing.T, module, fp, digest, backup string) (string, []byte) {
t.Helper()
key, err := delivery.WantKey(node, module)
if err != nil {
t.Fatal(err)
}
raw, err := delivery.EncodeWant(delivery.Want{FP: fp, Composed: digest, Backup: backup})
if err != nil {
t.Fatal(err)
}
return key, raw
}
func eventually(t *testing.T, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for !ok() {
if time.Now().After(deadline) {
t.Fatal(what)
}
time.Sleep(5 * time.Millisecond)
}
}
func (b *bed) appliedList() []string {
b.mu.Lock()
defer b.mu.Unlock()
return append([]string(nil), b.applied...)
}
// R6-A AC1 (the agent, with a bus that is a fake): given a want naming a digest and that composition, the module is
// applied and its have names the fingerprint and the digest; a composition whose digest no want names is not applied.
func TestAWantAndItsCompositionApplyTheModuleAndTheHaveNamesBoth(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
// A composition no want names yet is not applied.
b.comps.put(ck, craw)
time.Sleep(100 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a composition no want names was applied: %v", got)
}
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "the module was not applied", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "the have does not name both", func() bool {
h := b.haves.last("web")
return h.FP == fp && h.Composed == dig && h.Phase == string(delivery.PhaseRunning)
})
// Settled is told once what runs holds the module, so a declaration read then names its files (the filling's Knock;
// ace, 2026-10-25 21:17).
eventually(t, "Settled was not told web with web in what runs", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return slices.Equal(b.settled, []string{"web"}) && b.settledRan[0]
})
// A want naming a digest the composition at that fingerprint does not have: nothing is applied.
fp2 := fpOf("b")
ck2, craw2, _ := composition(t, "web", fp2, false, file("web", "x", "2"))
b.comps.put(ck2, craw2)
wk2, wraw2 := wantFor(t, "web", fp2, "sha256:"+strings.Repeat("0", 64), "")
b.wants.put(wk2, wraw2)
time.Sleep(100 * time.Millisecond)
if got := b.appliedList(); len(got) != 1 {
t.Fatalf("a want naming another digest applied the composition: %v", got)
}
}
// R6-A AC2: given a kept declaration and wants and compositions equal to it, when the agent starts, nothing is applied
// and every have names its fingerprint and digest.
func TestStartingOnWhatTheKeptDeclarationHoldsAppliesNothing(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
rs := []json.RawMessage{file("web", "x", "1"), file("web", "y", "2")}
ck, craw, dig := composition(t, "web", fp, false, rs...)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
// The kept declaration holds the same resources, in another order and spacing.
b.kept = map[string][]json.RawMessage{"web": {json.RawMessage(strings.ReplaceAll(string(rs[1]), ",", " , ")), rs[0]}}
stop := b.run()
defer stop()
eventually(t, "the have was not written", func() bool { return b.haves.last("web").Composed == dig })
if h := b.haves.last("web"); h.FP != fp || h.Phase != string(delivery.PhaseRunning) {
t.Fatalf("the have: %+v", h)
}
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that already runs what the want names was applied: %v", got)
}
}
// What the node-engine itself last applied is what runs after a restart, not the kept declaration.
func TestStartingOnItsOwnHaveAppliesNothing(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: dig, Phase: "running"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
time.Sleep(150 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that runs what the want names was applied after a restart: %v", got)
}
}
// A module found running on its own have after a restart is said again, once, through the haves: the session then holds
// it and writes the health this engine judges. Not said, its key kept the health an older engine wrote: xorg's have on
// g14 and shanks said unknown from before the engine said a module with nothing to judge ok, and the assigner held every
// module requiring x11-display back for it (2026-10-25).
func TestStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "xorg", fp, false, file("xorg", "x", "1"))
wk, wraw := wantFor(t, "xorg", fp, dig, "")
b.haves.stored = map[string]Have{"xorg": {Module: "xorg", FP: fp, Composed: dig, Phase: "running", Detail: "kept"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
eventually(t, "the have found running was not said again", func() bool {
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
return len(b.haves.log) > 0
})
time.Sleep(50 * time.Millisecond)
b.haves.mu.Lock()
log := append([]Have(nil), b.haves.log...)
b.haves.mu.Unlock()
if len(log) != 1 {
t.Fatalf("the have was said %d times, want once: %+v", len(log), log)
}
if h := log[0]; h.Module != "xorg" || h.FP != fp || h.Composed != dig || h.Phase != string(delivery.PhaseRunning) || h.Detail != "kept" {
t.Fatalf("the have said again: %+v", h)
}
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that runs what the want names was applied: %v", got)
}
}
// A module whose kept resources differ from the composition is applied, and a failed apply is retried only by a change
// of either key.
func TestAFailedApplyIsRetriedOnlyByAChangeOfEitherKey(t *testing.T) {
// A real failure is not retried by the clock, however short the backoff: only by a change of either key.
t.Run("a real failure", func(t *testing.T) {
b := newBed(t)
b.failWith = "web.x: no space"
b.env.Backoff = func(int) time.Duration { return time.Millisecond }
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "the have does not say failed", func() bool {
h := b.haves.last("web")
return h.Phase == string(delivery.PhaseFailed) && strings.Contains(h.Detail, "no space") && h.FP == fp
})
// An unrelated key changes: not a retry.
ck9, craw9, _ := composition(t, "web", fpOf("9"), false, file("web", "x", "9"))
b.comps.put(ck9, craw9)
time.Sleep(150 * time.Millisecond)
if n := len(b.appliedList()); n != 1 {
t.Fatalf("a failed apply was retried by an unrelated key or the clock (%d applies)", n)
}
// The want is written again: the retry.
b.mu.Lock()
b.failWith = ""
b.mu.Unlock()
b.wants.put(wk, wraw)
eventually(t, "the change of the want did not retry", func() bool { return len(b.appliedList()) == 2 })
eventually(t, "not running", func() bool { return b.haves.last("web").Phase == string(delivery.PhaseRunning) })
})
// The bus did not answer for a moment (mesh-vault on the bus node, 2026-10-25 23:04:24): tried again with no new
// revision of either key, by the backoff or by RetryTransient once the engine knows the bus is back; the have says
// retrying meanwhile, never failed.
for name, backoff := range map[string]func(int) time.Duration{
"a transient failure, by the backoff": func(int) time.Duration { return time.Millisecond },
"a transient failure, by RetryTransient": nil,
} {
t.Run(name, func(t *testing.T) {
b := newBed(t)
b.failWith, b.transient, b.failTimes = "fetching the object: nats: no responders available for request", true, 1
b.env.Backoff = backoff
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "vault", fp, false, file("vault", "x", "1"))
wk, wraw := wantFor(t, "vault", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) >= 1 })
if backoff == nil {
eventually(t, "the have does not say retrying", func() bool {
return strings.HasPrefix(b.haves.last("vault").Detail, "retrying")
})
b.ag.RetryTransient()
}
eventually(t, "not applied again without a new revision", func() bool { return len(b.appliedList()) == 2 })
eventually(t, "not running", func() bool { return b.haves.last("vault").Phase == string(delivery.PhaseRunning) })
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
for _, h := range b.haves.log {
if h.Phase == string(delivery.PhaseFailed) {
t.Errorf("a transient failure was written failed: %+v", h)
}
}
})
}
}
// R6-A AC5: a data module that ran before asks its backup before any resource changes; a refusal holds it, nothing is
// applied and the have says held; when the want then says skip, the module is applied.
func TestADataModuleIsBackedUpFirstAndHeldWhenItFails(t *testing.T) {
b := newBed(t)
fp1, fp2 := fpOf("a"), fpOf("b")
ck1, craw1, dig1 := composition(t, "db", fp1, true, file("db", "x", "1"))
wk1, wraw1 := wantFor(t, "db", fp1, dig1, "")
b.comps.before(ck1, craw1)
b.wants.before(wk1, wraw1)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}}
refuse := true
b.backup = func(string) Backup {
if refuse {
return Backup{Words: "restic: the repository is locked"}
}
return Backup{Done: true}
}
stop := b.run()
defer stop()
eventually(t, "baseline", func() bool { return b.haves.last("db").Composed == dig1 })
ck2, craw2, dig2 := composition(t, "db", fp2, true, file("db", "x", "2"))
b.comps.put(ck2, craw2)
wk2, wraw2 := wantFor(t, "db", fp2, dig2, "")
b.wants.put(wk2, wraw2)
eventually(t, "the backup was not asked", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.backups) == 1
})
eventually(t, "the have does not say held", func() bool {
h := b.haves.last("db")
return h.Phase == string(delivery.PhaseHeld) && strings.Contains(h.Detail, "repository is locked") && h.FP == fp1
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("something was applied behind a failed backup: %v", got)
}
// The answer arrives in the module's own want: back up again (it fails again), then skip.
wk, again := wantFor(t, "db", fp2, dig2, delivery.BackupAgain)
b.wants.put(wk, again)
eventually(t, "back up again did not ask the backup", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.backups) == 2
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied behind a second failed backup: %v", got)
}
_, skip := wantFor(t, "db", fp2, dig2, delivery.BackupSkip)
b.wants.put(wk, skip)
eventually(t, "skip did not apply the module", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
n := len(b.backups)
b.mu.Unlock()
if n != 2 {
t.Fatalf("a backup was asked after the answer was to skip: %d asks", n)
}
eventually(t, "not running at the new fingerprint", func() bool {
h := b.haves.last("db")
return h.Phase == string(delivery.PhaseRunning) && h.FP == fp2 && h.Composed == dig2
})
}
// A data module with a good backup is applied after it; one that never ran has nothing to back up.
func TestABackupDoneIsFollowedByTheApplyAndANewModuleIsNotBackedUp(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("c")
ck, craw, dig := composition(t, "db", fp, true, file("db", "x", "1"))
wk, wraw := wantFor(t, "db", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
n := len(b.backups)
b.mu.Unlock()
if n != 0 {
t.Fatalf("a module that never ran was backed up (%d)", n)
}
fp2 := fpOf("d")
ck2, craw2, dig2 := composition(t, "db", fp2, true, file("db", "x", "2"))
wk2, wraw2 := wantFor(t, "db", fp2, dig2, "")
b.comps.put(ck2, craw2)
b.wants.put(wk2, wraw2)
eventually(t, "second not applied", func() bool { return len(b.appliedList()) == 2 })
b.mu.Lock()
n = len(b.backups)
b.mu.Unlock()
if n != 1 {
t.Fatalf("a module that ran before was backed up %d times, want once", n)
}
}
// R6-A AC4 at the agent: a removed want removes that module only, and its have is written removed.
func TestARemovedWantRemovesThatModuleOnly(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
gone, _ := delivery.EncodeWant(delivery.Want{Removed: true})
b.wants.put(wk, gone)
eventually(t, "not removed", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.removed) == 1 && b.removed[0] == "web"
})
eventually(t, "the have was not written removed", func() bool {
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
return len(b.haves.removed) == 1
})
}
// The node's own composition is applied before any module's, whichever key came first.
func TestTheNodesOwnCompositionComesBeforeAnyModules(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, err := delivery.WantNodeKey(node)
if err != nil {
t.Fatal(err)
}
nwraw, err := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
if err != nil {
t.Fatal(err)
}
b.comps.before(ck, craw)
b.comps.before(nk, nraw)
b.wants.before(wk, wraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "both not applied", func() bool { return len(b.appliedList()) == 2 })
if got := b.appliedList(); got[0] != NodeModule || got[1] != "web" {
t.Fatalf("the order was %v, the node's own first", got)
}
}
// R6-A AC9: a composition signed by either trusted key is applied; one signed by another key, one changed after it was
// signed and one unsigned are refused and nothing of them is applied (ADR 0539 §3).
func TestOnlyACompositionSignedByATrustedKeyIsApplied(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
plain := func(module, fp, content string) delivery.Composed {
return delivery.Composed{Node: node, Module: module, FP: fp, Resources: []json.RawMessage{file(module, "x", content)}}
}
put := func(c delivery.Composed, signer ed25519.PrivateKey, tamper func([]byte) []byte) {
key, _ := delivery.ComposedKey(node, c.Module, c.FP)
var raw []byte
var dig string
if signer == nil {
raw, _ = delivery.EncodeComposed(c)
dig, _ = delivery.ComposedDigestOf(c)
} else {
_, raw, dig = signedBy(t, c, key, signer)
}
if tamper != nil {
raw = tamper(raw)
}
b.comps.put(key, raw)
wk, wraw := wantFor(t, c.Module, c.FP, dig, "")
b.wants.put(wk, wraw)
}
// Refused: signed by a stranger, changed after signing, unsigned.
put(plain("stranger", fpOf("1"), "x"), strangerKey, nil)
put(plain("changed", fpOf("2"), "before"), controllerKey, func(raw []byte) []byte {
return []byte(strings.Replace(string(raw), "before", "after!", 1))
})
put(plain("unsigned", fpOf("3"), "x"), nil, nil)
time.Sleep(200 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a composition that is not signed by a trusted key was applied: %v", got)
}
// Applied: signed by the controller, signed by the operator.
put(plain("bycontroller", fpOf("4"), "x"), controllerKey, nil)
put(plain("byoperator", fpOf("5"), "x"), operatorKey, nil)
eventually(t, "the trusted signers' compositions were not applied", func() bool { return len(b.appliedList()) == 2 })
got := b.appliedList()
if !(got[0] == "bycontroller" && got[1] == "byoperator") && !(got[0] == "byoperator" && got[1] == "bycontroller") {
t.Fatalf("applied %v", got)
}
if h := b.haves.last("byoperator"); h.Phase != string(delivery.PhaseRunning) || h.FP != fpOf("5") {
t.Fatalf("the operator's module has no running have: %+v", h)
}
}
// With no key trusted, nothing is applied: a node that does not know its signers fails closed.
func TestWithNoTrustedKeyNothingIsApplied(t *testing.T) {
b := newBed(t)
b.env.Trusted = nil
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
time.Sleep(200 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied with no trusted key: %v", got)
}
}
// What a module's apply must not take is every other module that runs, those the agent holds and those still running from
// the kept declaration (ADR 0429 §4): the resources of the kept declaration reach Apply as the others.
func TestApplyingOneModuleHoldsOutTheModulesStillRunningFromTheKeptDeclaration(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}, NodeModule: {file("node", "n", "n")}}
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
defer b.mu.Unlock()
var names []string
for _, c := range b.others[0] {
names = append(names, c.Module)
}
if len(names) != 2 || !strings.Contains(strings.Join(names, ","), "db") {
t.Fatalf("the others of web were %v: db and the node's own run from the kept declaration", names)
}
}
// Everything is what runs: the agent's compositions and the kept resources of each module it does not hold.
func TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}, "web": {file("web", "x", "old")}}
a := New(b.env)
if got := a.Everything(); len(got) != 2 {
t.Fatalf("before the agent holds anything: %d compositions, want the 2 kept", len(got))
}
a.running["web"] = delivery.Have{FP: fpOf("a"), Composed: "sha256:" + strings.Repeat("1", 64), Phase: delivery.PhaseRunning}
ck, craw, dig := composition(t, "web", fpOf("a"), false, file("web", "x", "new"))
a.running["web"] = delivery.Have{FP: fpOf("a"), Composed: dig, Phase: delivery.PhaseRunning}
a.takeComposition(Update{Key: ck, Value: craw, Revision: 1})
got := a.Everything()
if len(got) != 2 {
t.Fatalf("%d compositions", len(got))
}
for _, c := range got {
if c.Module == "web" && !strings.Contains(string(c.Resources[0]), "new") {
t.Fatal("web is the kept declaration's, not the composition the agent runs")
}
}
if a.HoldsNode() {
t.Fatal("the agent holds no node composition")
}
}
// A failed apply is applied again after a restart even when the kept declaration holds the composition: once this
// node-engine wrote its haves, they are what it knows (the kept declaration is only for the first start after the switch).
func TestAFailedApplyIsAppliedAgainAfterARestartWhateverIsKept(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
b.kept = map[string][]json.RawMessage{"web": {file("web", "x", "1")}}
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed)}}
stop := b.run()
defer stop()
eventually(t, "a failed apply was marked running from the kept declaration", func() bool { return len(b.appliedList()) == 1 })
}
// A have written before the switch carries no composition: then the kept declaration says what runs.
func TestAHaveWithNoCompositionDefersToTheKeptDeclaration(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
b.kept = map[string][]json.RawMessage{"web": {file("web", "x", "1")}}
b.haves.stored = map[string]Have{"web": {Module: "web", Phase: "running"}}
stop := b.run()
defer stop()
eventually(t, "the have was not written", func() bool { return b.haves.last("web").Composed == dig })
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied %v", got)
}
}
// A module that leaves the node's own composition for a composition of its own keeps its resources until that one is
// applied: the node's own apply holds out every module whose composition the node's bucket holds and the agent does not
// run yet (issue 1945, live on 2026-10-25 at 02:32: shanks' node-engine applied a node's own composition that no longer
// carried systemd-resolved, whose own composition waited for its credential, and removed its guard, so the machine lost
// its resolver).
func TestTheNodesOwnApplyHoldsOutAModuleThatMovedToItsOwnComposition(t *testing.T) {
b := newBed(t)
fp := fpOf("b")
// systemd-resolved's own composition is written; no want names it yet.
rk, rraw, _ := composition(t, "systemd-resolved", fp, false, file("systemd-resolved", "guard", "g"))
b.comps.before(rk, rraw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, err := delivery.WantNodeKey(node)
if err != nil {
t.Fatal(err)
}
nwraw, err := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
if err != nil {
t.Fatal(err)
}
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
defer b.mu.Unlock()
var names []string
for _, c := range b.others[0] {
names = append(names, c.Module)
}
if !strings.Contains(strings.Join(names, ","), "systemd-resolved") {
t.Fatalf("the others of the node's own apply were %v: systemd-resolved's resources would be removed", names)
}
}
// A module the assigner removed holds nothing back, its compositions still in the bucket or not: its resources are its
// removal's to take (issue 1945).
func TestARemovedModulesCompositionHoldsNothingBack(t *testing.T) {
b := newBed(t)
fp := fpOf("b")
rk, rraw, _ := composition(t, "gone", fp, false, file("gone", "x", "1"))
b.comps.before(rk, rraw)
gk, err := delivery.WantKey(node, "gone")
if err != nil {
t.Fatal(err)
}
graw, err := delivery.EncodeWant(delivery.Want{Removed: true})
if err != nil {
t.Fatal(err)
}
b.wants.before(gk, graw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool {
for _, m := range b.appliedList() {
if m == NodeModule {
return true
}
}
return false
})
b.mu.Lock()
defer b.mu.Unlock()
for _, others := range b.others {
for _, c := range others {
if c.Module == "gone" {
t.Fatal("a removed module's composition was held out")
}
}
}
}
// The node's own composition is not applied while a module's composition in the node's bucket cannot be read or is not
// signed by a trusted key: what that one carries may be what left the node's own, and the node's own apply would remove
// it (issue 1945, the review's gap 2). It is said, naming the composition, and applied once that one reads.
func TestTheNodesOwnWaitsWhileAModulesCompositionCannotBeRead(t *testing.T) {
b := newBed(t)
bad, err := delivery.ComposedKey(node, "broken", fpOf("c"))
if err != nil {
t.Fatal(err)
}
b.comps.before(bad, []byte(`{"v":1,"not":"a composition"}`))
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the wait not said", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
for _, s := range b.said {
if strings.Contains(s, bad) {
return true
}
}
return false
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied %v while a composition could not be read", got)
}
// Read again, the node's own is applied.
gk, graw, _ := composition(t, "broken", fpOf("c"), false, file("broken", "x", "1"))
b.comps.put(gk, graw)
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
}
// A stale refused composition beside a newer readable one of the same module does not hold the node's own apply back:
// only the composition the module's want names, else its newest, counts (the review of issue 1945: the bucket keeps
// compositions written before the filled paths, which the SDK now refuses, and nobody applies them).
func TestAStaleRefusedCompositionDoesNotHoldTheNodesOwnBack(t *testing.T) {
b := newBed(t)
stale, err := delivery.ComposedKey(node, "power", fpOf("a"))
if err != nil {
t.Fatal(err)
}
b.comps.before(stale, []byte(`{"v":1,"not":"a composition"}`))
gk, graw, _ := composition(t, "power", fpOf("b"), false, file("power", "x", "1"))
b.comps.before(gk, graw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
}
// A composition the kept declaration or the node's own have says runs is applied all the same when the engine holds no
// record of its resources: they were removed since (issue 1945, live on 2026-10-25: ace's and g14's node-engines removed
// ca-trust's, power's and matrix's resources in a node's own apply, and the wants that named those modules' own
// compositions were judged already running from the kept declaration, so nothing made them again).
func TestACompositionWhoseResourcesWereRemovedIsAppliedAgain(t *testing.T) {
for _, from := range []string{"kept", "have"} {
t.Run(from, func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
rs := []json.RawMessage{file("ca-trust", "trust", "1")}
ck, craw, dig := composition(t, "ca-trust", fp, false, rs...)
wk, wraw := wantFor(t, "ca-trust", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
if from == "kept" {
b.kept = map[string][]json.RawMessage{"ca-trust": rs}
} else {
b.haves.stored = map[string]Have{"ca-trust": {Module: "ca-trust", FP: fp, Composed: dig, Phase: "running"}}
}
b.env.Recorded = func(context.Context, delivery.Composed) bool { return false }
stop := b.run()
defer stop()
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
})
}
}
// The composition a want names is applied from the key's history when a newer one has replaced it under the same key:
// a want held at a fingerprint keeps naming the digest it was written with, and the controller writes every new
// composition at that fingerprint over the same key (ADR 0539 clause 1, the sweep's kept revisions; issue 1945, live on
// 2026-10-25: shanks' node-tools composition had digest 575a5c96, its held want named a36e63c8, and nothing was applied).
func TestTheRevisionAWantNamesIsAppliedFromHistory(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, oldRaw, oldDig := composition(t, "node-tools", fp, false, file("node-tools", "x", "old"))
_, newRaw, _ := composition(t, "node-tools", fp, false, file("node-tools", "x", "new"))
b.comps.before(ck, newRaw) // the key's latest value
wk, wraw := wantFor(t, "node-tools", fp, oldDig, "")
b.wants.before(wk, wraw)
b.env.Revisions = func(_ context.Context, key string) ([]Update, error) {
if key != ck {
return nil, nil
}
return []Update{{Key: ck, Value: oldRaw, Revision: 3}, {Key: ck, Value: newRaw, Revision: 7}}, nil
}
stop := b.run()
defer stop()
eventually(t, "the revision the want names was not applied", func() bool { return len(b.appliedList()) == 1 })
if h := b.haves.last("node-tools"); h.Composed != oldDig {
t.Fatalf("the have names %s, want %s", h.Composed, oldDig)
}
}
// 1945/AC18 (unit): a module that still runs from the kept declaration carries the kept declaration's filled paths of
// its files, so the filling writes them ("not named as filled" for claude-code and screen-lock, 2026-10-25).
func TestAModuleRunningFromTheKeptDeclarationKeepsItsFilledPaths(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"claude-code": {file("claude-code", "settings", "")}}
b.env.KeptFilled = func() map[string][]string {
return map[string][]string{"claude-code": {"/tmp/r6/claude-code/settings"}}
}
a := New(b.env)
got := a.Everything()
if len(got) != 1 || got[0].Module != "claude-code" || strings.Join(got[0].Filled, ",") != "/tmp/r6/claude-code/settings" {
t.Fatalf("what runs from the kept declaration: %+v", got)
}
}
+67
View File
@@ -0,0 +1,67 @@
package apply
import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// WriteAlone writes one file a module's composition declares, through the apply's own file step, and records it as an
// apply would; nothing else of the module is applied, and nothing that reads the file is moved (novox/hq ADR 0547 §8).
//
// **For a file whose reader takes the change itself.** The bus's user list is the one it is for: the bus module's
// entrypoint watches the file and signals the server to reload in place, so the list is in force without a want of the
// bus, which nobody writes (ADR 0539 §15), and without the container being touched. A file that a service or a container
// must be moved to read is not written alone: it goes through the module's apply, which moves what reads it.
//
// The file is written only when its content differs from what is there (the file step's own rule), read back, and
// recorded under the module whose resource it is, so the module's next apply finds it as its own and writes nothing.
// The caller holds the apply's lock. log is said the path, the action and the digest, never the content.
func WriteAlone(ctx context.Context, sys system.System, file *declaration.File, known store.State, run Runner,
log func(string)) (Outcome, store.State, error) {
if log == nil {
log = func(string) {}
}
if file.Into != "" || file.CreateOnce || file.Sealed != "" || file.VaultSealed != nil || len(file.Secrets) > 0 {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is not a whole file of plain content, "+
"and only such a file is written alone; nothing was written", file.Path)}
}
if file.IsReference() {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is given by digest %s and was not "+
"resolved to its content; it is not written empty", file.Path, file.Digest)}
}
if held, isHeld := known.HeldAt(file.ID); isHeld {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is held as found for a module not "+
"yet taken (%s), so nothing was written", file.Path, held.Why)}
}
was, _ := known.Find(file.ID)
if was.ID == "" {
// Only over a file the mesh wrote: a file this node holds no record of is the module's apply's to take, with
// the original kept, never this write's. A record of the path under another id (the declaration before the
// switch named it so) is the one written to, so no second record of the file is left for an apply to remove.
rec, ok := known.At(string(declaration.TypeFile), file.Path)
if !ok {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("this node holds no record of %s; it "+
"is written by the module's apply first, never alone", file.Path)}
}
renamed := *file
renamed.ID = rec.ID
file, was = &renamed, rec
}
in := inputs{declares: map[string]string{file.Identity(): declaredDigest(file)}, known: &known,
groups: groupsWanted([]declaration.Resource{file}), handed: Handed{}, dirsBefore: map[string]bool{},
files: map[string]bool{file.Path: true}}
outcome, err := applyOne(ctx, sys, file, run, map[string]bool{}, in, was, nil, nil)
if err != nil {
log(fmt.Sprintf(" failed %s (%s), alone: %v", file.ID, file.Path, err))
return outcome, known, &Error{Resource: file.ID, Err: err}
}
recordKeeping(&known, file, outcome, was)
if moved(outcome) {
log(fmt.Sprintf(" %s %s (%s), alone: %s", outcome.Action, file.ID, file.Path, outcome.wrote))
}
return outcome, known, nil
}
+90
View File
@@ -0,0 +1,90 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0547 §8: the bus node writes the bus's user list alone, with no want, through the apply's own
// file step: only when it differs, mode 0600, read back and recorded as the module's, and nothing else moves.
func TestAFileWrittenAloneMovesNothingElse(t *testing.T) {
const was, now = "user a\n", "user a\nuser b\n"
for _, c := range []struct {
name, fields, before string
recorded bool
recordedAs string // the id the record has, when not the composition's
action, refused string
after string
}{
{name: "no record of it", fields: `"content":%q,"mode":"0600"`, before: was, refused: "holds no record", after: was},
{name: "changed", fields: `"content":%q,"mode":"0600"`, before: was, recorded: true, action: "updated", after: now},
{name: "the same", fields: `"content":%q,"mode":"0600"`, before: now, recorded: true, action: "unchanged", after: now},
{name: "recorded under the id from before the switch", fields: `"content":%q,"mode":"0600"`, before: was,
recorded: true, recordedAs: "bus-users", action: "updated", after: now},
{name: "absent with a record", fields: `"content":%q,"mode":"0600"`, recorded: true, action: "created", after: now},
{name: "written into", fields: `"content":%q,"into":"block","owner-at-runtime":"managed"`, before: was,
recorded: true, refused: "only such a file", after: was},
{name: "sealed", fields: `"sealed":"c2VhbGVk"`, before: was, recorded: true,
refused: "only such a file", after: was},
} {
t.Run(c.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "accounts.conf")
if c.before != "" {
if err := os.WriteFile(path, []byte(c.before), 0o600); err != nil {
t.Fatal(err)
}
}
fields := strings.Replace(c.fields, "%q", fmt.Sprintf("%q", now), 1)
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"nats.bus-users","module":"nats",
"type":"file","path":%q,`+fields+`}]}`, path))
file := d.Resources[0].(*declaration.File)
known := store.State{}
id := file.ID
if c.recordedAs != "" {
id = c.recordedAs
}
if c.recorded {
known.Record(store.Applied{ID: id, Type: string(declaration.TypeFile), Module: "nats", Target: path,
Origin: store.OriginDeclared, Wrote: digestOf(c.before)})
}
calls := 0
counting := func(context.Context, string, ...string) (string, error) { calls++; return "", nil }
out, after, err := WriteAlone(context.Background(), archHost(t), file, known, counting, nil)
if c.refused != "" {
if err == nil || !strings.Contains(err.Error(), c.refused) {
t.Fatalf("not refused with %q: %v", c.refused, err)
}
} else if err != nil {
t.Fatal(err)
} else if out.Action != c.action {
t.Errorf("action %q, want %q", out.Action, c.action)
}
if got := readText(t, path); got != c.after {
t.Errorf("file holds %q, want %q", got, c.after)
}
if calls != 0 {
t.Errorf("%d commands were run: a file written alone moves nothing", calls)
}
if c.refused != "" {
return
}
if m := modeAt(t, path); m != 0o600 {
t.Errorf("mode %o", m)
}
rec, ok := after.Find(id)
if !ok || rec.Wrote != digestOf(now) || rec.Target != path {
t.Errorf("record %+v (found %v), want it to say %s was written", rec, ok, digestOf(now))
}
if n := len(after.Resources); n != 1 {
t.Errorf("%d records of one file", n)
}
})
}
}
+226 -39
View File
@@ -21,7 +21,6 @@ import (
"os"
"os/exec"
"path/filepath"
"slices"
"sort"
"strconv"
"strings"
@@ -300,6 +299,60 @@ func ApplyMindingWindows(
unseal Unseal,
keep Keep,
windows *Windows,
) (Report, store.State, error) {
return applyScoped(ctx, sys, d, nil, known, origin, run, log, unseal, keep, windows)
}
// Scope is what makes an apply one module's (novox/hq ADR 0429 §4, ADR 0539 §9): the module whose want changed, and
// the resources the node's other modules hold, so a path, a unit or a group another module still asks for is neither
// removed nor taken from it. Only that module's own records are orphans; a removed module is a Scope with an empty
// declaration.
type Scope struct {
// Module is the module applied. NodeModule (store.NodeModule) is the node's own composition.
Module string
// Others are the resources of every other module the node has a composition for.
Others []declaration.Resource
}
// owns reports whether a record belongs to the scope's module.
func (sc *Scope) owns(a store.Applied) bool {
return store.OwnerOf(a, declaration.ModuleGuess) == sc.Module
}
// isNode is whether the scope is the node's own composition, the one that carries identities, groups, libraries and
// the adoption.
func (sc *Scope) isNode() bool { return sc == nil || sc.Module == store.NodeModule }
// ApplyModule makes the machine match one module's declaration, and only that module's: everything else the node
// runs is left as it is. d holds the module's resources; scope names the module and what the others hold.
func ApplyModule(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
scope Scope,
known store.State,
origin string,
run Runner,
log func(string),
unseal Unseal,
keep Keep,
windows *Windows,
) (Report, store.State, error) {
return applyScoped(ctx, sys, d, &scope, known, origin, run, log, unseal, keep, windows)
}
func applyScoped(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
scope *Scope,
known store.State,
origin string,
run Runner,
log func(string),
unseal Unseal,
keep Keep,
windows *Windows,
) (report Report, _ store.State, _ error) {
// Said whichever way the apply ended, failure included: a window being open is a fact about the
// machine, not about this apply.
@@ -326,7 +379,9 @@ func ApplyMindingWindows(
}
// What an unfinished apply found a unit as is kept only while its service is declared: one
// declared again later is read afresh, as any resource with no record is (novox/hq ADR 0118).
known.DropFoundUndeclared(declared, origin)
if scope == nil {
known.DropFoundUndeclared(declared, origin)
}
// Which units the mesh made, read before any removal can take a unit file's record away — so
// whichever order a module declared a unit and its file in, the unit is known for the mesh's
@@ -351,7 +406,11 @@ func ApplyMindingWindows(
// it back would stop or disable a unit the declaration says is running, every container on it
// with it, only for the declared one to start it again in the same apply. The record is
// forgotten; the unit's found state is the remaining record's to give back.
heldUnits := unitsHeld(d.Resources)
everything := d.Resources
if scope != nil {
everything = append(append([]declaration.Resource{}, d.Resources...), scope.Others...)
}
heldUnits := unitsHeld(everything)
// **A whole file still declared at its path by another resource is handed to it, never removed
// first** (novox/hq ADR 0223). A file can change owners between modules — the machine's resolver
@@ -361,20 +420,55 @@ func ApplyMindingWindows(
// and stayed so until the new resource's turn came, with every resource in between applied on a
// machine without it. The record going is kept until the one declaring the path now is recorded
// in its place: then it is forgotten, and what it kept of the original goes with the file.
heldFiles := filesHeld(d.Resources)
heldFiles := filesHeld(everything)
handedFiles := map[string]store.Applied{}
// **A group still asked for by another resource is not given back when one record of it goes**
// (novox/hq ADR 0252), on the unit's rule above: two modules may each want the account in one group,
// and the one going takes nothing the other still needs.
wanted := groupsWanted(d.Resources)
wanted := groupsWanted(everything)
// And what a record going hands to a resource that still asks for its group (novox/hq ADR 0252 §3).
handed := Handed{}
// **A resource another module's composition now holds is that module's, and is left** (issue 1945): a module that
// leaves the node's own composition for one of its own takes its resources with it, by their identity, and the node's
// own apply leaves each as it is, record and all, until that module's apply takes it. Removed, a node lost its
// resolver guard, its sudoers file and a running container between two applies.
heldByOthers := map[string]string{}
if scope != nil {
for _, r := range scope.Others {
heldByOthers[r.Identity()] = r.DeclaredModule()
}
}
removeOrphan := func(orphan store.Applied) error {
var action, detail string
var err error
if module, held := heldByOthers[orphan.ID]; held && !declared[orphan.ID] {
if module == "" {
module = "another module"
}
detail := "no longer declared here; the composition of " + module + " holds it, so it was left as it is"
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "kept", Detail: detail,
})
log(fmt.Sprintf(" kept %s (%s): %s", orphan.ID, orphan.Target, detail))
return nil
}
if by, held := heldFiles[fileKey(orphan)]; held && by != orphan.ID {
if scope != nil && !declared[by] {
// Another module declares the path: it is that module's, so this record is let go and the file is
// neither removed nor taken from it (ADR 0539 §9).
known.Forget(orphan.ID)
detail := "no longer declared; " + by + " of another module still holds the path, so it was left as it is"
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "forgotten", Detail: detail,
})
log(fmt.Sprintf(" forgotten %s (%s): %s", orphan.ID, orphan.Target, detail))
return nil
}
handedFiles[by] = orphan
return nil
}
@@ -482,20 +576,11 @@ func ApplyMindingWindows(
}
handover := map[string][]store.Applied{}
for _, orphan := range known.Orphans(declared, origin) {
if d.Adoption == nil && strings.HasPrefix(orphan.ID, declaration.AdoptionPrefix) {
protecting = append(protecting, orphan)
if scope != nil && !scope.owns(orphan) {
continue
}
// **A module the mesh left out is not a module the mesh removed** (novox/hq ADR 0163, rule
// 6): its resources are absent because a setting stored for it cannot compose, and the
// mesh said so by name. What the host wrote for it stays as it is, recorded, until the
// module is declared again or unassigned.
if module, left := d.LeftOutModuleOf(orphan.ID); left {
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "unchanged", Detail: "kept: " + module + " was left out of this declaration by the mesh, not removed",
})
log(fmt.Sprintf(" kept %s (%s): %s was left out of this declaration by the mesh, not removed", orphan.ID, orphan.Target, module))
if d.Adoption == nil && strings.HasPrefix(orphan.ID, declaration.AdoptionPrefix) {
protecting = append(protecting, orphan)
continue
}
if by, replaced := replacedBy[orphan.ID]; replaced {
@@ -538,12 +623,7 @@ func ApplyMindingWindows(
// what is assigned: a carried bundle's silence is not an unassignment.
if origin == store.OriginDeclared {
for _, h := range append([]store.Held{}, known.Held...) {
if declared[h.ID] {
continue
}
if slices.Contains(d.LeftOut, h.Module) {
// Left out, not unassigned (ADR 0163, rule 6): still held for the module, as the
// mesh asked.
if declared[h.ID] || (scope != nil && h.Module != scope.Module) {
continue
}
known.Release(h.ID)
@@ -584,6 +664,7 @@ func ApplyMindingWindows(
in.refused = refusedPlaces(d.Resources)
in.agentHomes = rootNeverHomes(d)
in.dirsBefore = map[string]bool{}
in.files = declaredFiles(d.Resources)
for _, resource := range d.Resources {
in.declares[resource.Identity()] = declaredDigest(resource)
if dir, ok := resource.(*declaration.Directory); ok {
@@ -744,7 +825,7 @@ func ApplyMindingWindows(
in.held = heldRoots(verdicts, d.Libraries)
// A tool runner no longer put in a group leaves it before anything is applied, so its unit, which names the
// change under restart-on, is restarted in this same apply to lose it (ADR 0328 §11, ADR 0334 §2, ADR 0252 §3).
if origin == store.OriginDeclared {
if origin == store.OriginDeclared && scope.isNode() {
for _, o := range leaveRunners(ctx, sys, d.Libraries, d.Identities, &known, run, wanted) {
if o.Action == "restored" {
changed[o.changes] = true
@@ -1036,6 +1117,11 @@ func ApplyMindingWindows(
}
// Only now. The record follows the fact, never leads it.
recordApplied(&known, origin, resource, outcome, kept, keptMode, keptOwner)
if scope != nil {
known.SetModule(resource.Identity(), scope.Module)
} else if module, ours := d.ModuleOfID(resource.Identity()); ours {
known.SetModule(resource.Identity(), module)
}
appliedOK[resource.Identity()] = true
// Its module has been taken, and what was held for it is now the mesh's. A file written
// into, or a service whose lifecycle is the machine's, replaced nothing that was found, so
@@ -1138,7 +1224,7 @@ func ApplyMindingWindows(
}
// Only the mesh says what is assigned: a carried bundle's silence undeclares no identity and no folder.
if origin == store.OriginDeclared {
if origin == store.OriginDeclared && scope.isNode() {
// And an identity, then a library group, no longer declared goes, once what ran as it went above (ADR 0328
// §3, §4).
if tool, err := toolFor(sys); err == nil {
@@ -1157,7 +1243,7 @@ func ApplyMindingWindows(
// the mesh's derived filter among it — applied cleanly (novox/hq ADR 0100), and on every
// converged apply, not once (ADR 0168). Skipped, it is said: a step that does nothing is never
// silent (issue 143).
if len(failures) == 0 {
if len(failures) == 0 && scope.isNode() {
did, err := retireFirewall(ctx, d, origin, &known, run, log)
if err != nil {
return report, known, &Error{Resource: "the firewall found on this machine", Err: err, Done: report}
@@ -1168,7 +1254,7 @@ func ApplyMindingWindows(
return report, known, err
}
}
} else if rec := known.Firewall; origin == store.OriginDeclared && d.Adoption == nil && rec != nil &&
} else if rec := known.Firewall; len(failures) > 0 && scope.isNode() && origin == store.OriginDeclared && d.Adoption == nil && rec != nil &&
rec.Kind == string(firewall.UFW) && rec.WasActive && firewall.Active(ctx, run) {
report.Firewall = fmt.Sprintf("left in force: %d resource(s) failed, and the found firewall is retired only after a clean apply", len(failures))
log(" kept ufw in force: " + report.Firewall)
@@ -1333,6 +1419,16 @@ func applyOne(ctx context.Context, sys system.System, r declaration.Resource, ru
"created once it is (novox/hq ADR 0328 §9)"
return out, nil
}
// Not created or replaced before a file it mounts is there: the runtime would make the missing source a
// directory, and the file could never be written there (novox/hq issue 1873). The write of the file applies
// what reads it (WriteFilled, readsChange), this container among them.
if src := mountsAFileNotThere(res, in.files); src != "" {
out := begin(r)
out.Action = "waiting"
out.Detail = "not created: it mounts " + src + ", a file the declaration declares that is not there " +
"yet; created once it is (novox/hq issue 1873)"
return out, nil
}
}
// Nothing below a home is touched through a link an account put there (novox/hq ADR 0266).
// And nothing is unpacked or written into below an agent's home (agent_home_guard.go).
@@ -1482,6 +1578,9 @@ func applyDirectory(r *declaration.Directory, previous store.Applied, wasBefore
}
if existed && wasBefore {
ours := previous.Target != "" && filepath.Clean(previous.Target) == filepath.Clean(r.Path) && !previous.AsFound
// In the mesh's own tree, a directory the engine's own account owns was made by the mesh, on the way to
// something below it or by an earlier apply that did not record it: it is the mesh's (novox/hq issue 1873).
ours = ours || madeInTheMeshsTree(r.Path, before)
if !ours {
owned, err := ownedBy(r.Path, r.Owner)
if err != nil {
@@ -1557,6 +1656,30 @@ func applyDirectory(r *declaration.Directory, previous store.Applied, wasBefore
return out, nil
}
// madeInTheMeshsTree is whether a directory lies in the mesh's own tree (meshDataRoot, below it, never the root
// itself) and belongs to the engine's own account: only that account makes a directory there, so it is the mesh's
// and not found (novox/hq issue 1873). Anywhere else, or owned by any other account, issue 339's rule stands.
func madeInTheMeshsTree(path string, found os.FileInfo) bool {
// By its text and by where it leads: a link in the tree pointing out of it leads outside, and what is there is
// not the tree's (review of issue 1873).
if !inTheMeshsTree(path) {
return false
}
uid, _, ok := ownerOf(found)
return ok && uid == os.Geteuid()
}
// inTheMeshsTree is whether a path lies strictly below the mesh's own tree, both as written and with every link in
// it followed (resolved), against the tree's root resolved the same way (novox/hq issue 1873).
func inTheMeshsTree(path string) bool {
root := filepath.Clean(meshDataRoot)
sep := string(filepath.Separator)
if !strings.HasPrefix(filepath.Clean(path), root+sep) {
return false
}
return strings.HasPrefix(resolved(filepath.Clean(path)), resolved(root)+sep)
}
// applyHandedOver gives a directory the operator handed to the mesh its declared mode and owner (novox/hq issue
// 356). **No link, checked again at the moment of use**: the hand-over refused a link when it was recorded, and a
// component of the path may have become one since — a parent swapped for a link to /etc would carry root's chmod
@@ -1606,6 +1729,19 @@ func applyHandedOver(r *declaration.Directory, out Outcome, mode os.FileMode) (O
// keepFound, when not nil, is where the original of a file this host has no record of is kept
// before it is written over (novox/hq ADR 0100): once, never overwritten, and named in the outcome.
func applyFile(r *declaration.File, previous store.Applied, unseal Unseal, keepFound Keep) (Outcome, error) {
cleared, err := clearRuntimesEmptyDirectory(r.Path)
if err != nil {
return begin(r), err
}
out, err := applyFileAt(r, previous, unseal, keepFound)
if cleared != "" {
out.Detail = joinDetail(cleared, out.Detail)
}
return out, err
}
// applyFileAt writes a declared file: applyFile's work once its path is cleared of what a runtime left there.
func applyFileAt(r *declaration.File, previous store.Applied, unseal Unseal, keepFound Keep) (Outcome, error) {
if r.Into == declaration.IntoBlock {
return applyBlock(r, previous)
}
@@ -2328,12 +2464,18 @@ func ExecRunner(ctx context.Context, name string, args ...string) (string, error
if err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
// The words as they always were, and the exit underneath them, so a caller asks the
// code (system.ExitCode) rather than matching text that this line is free to reword.
return string(out), &exited{
words: fmt.Sprintf("%s exited %d: %s", name, exit.ExitCode(), strings.TrimSpace(string(exit.Stderr))),
exit: exit,
words := fmt.Sprintf("%s exited %d: %s", name, exit.ExitCode(), strings.TrimSpace(string(exit.Stderr)))
if exit.ExitCode() == -1 {
// Killed, not exited: say by what, and the context's end when that is what killed it (issue 1945:
// "docker exited -1:" said nothing while a bus container was half replaced).
words = fmt.Sprintf("%s was %s", name, exit.String())
if ctx.Err() != nil {
words += " because its context ended: " + ctx.Err().Error()
}
}
// The words, and the exit underneath them, so a caller asks the code (system.ExitCode) rather than
// matching text that this line is free to reword.
return string(out), &exited{words: words, exit: exit}
}
return string(out), fmt.Errorf("%s: %w", name, err)
}
@@ -2438,6 +2580,9 @@ type inputs struct {
// held is the root of every library whose disk or mount is not there, with its name: no container that mounts
// it is created (novox/hq ADR 0328 §9).
held map[string]string
// files is the path of every file the declaration declares: a container mounting one that is not there yet waits
// for it (novox/hq issue 1873).
files map[string]bool
// handed is every group a record that went this apply handed to a resource still asking for it.
handed Handed
}
@@ -2819,6 +2964,14 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
return out, nil
}
aside := ""
if !existed {
// A container an engine set aside and then stopped before it ran the replacement (it died between the two):
// the set-aside one of this apply, put back if the replacement fails and removed once it runs (issue 1945).
if _, err := containerState(ctx, r.Name+asideSuffix, run); err == nil {
aside = r.Name + asideSuffix
}
}
switch {
case existed && (before.Spec == want || legacy) && before.Running && len(reasons) == 0:
out.Action = "unchanged"
@@ -2841,15 +2994,40 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
if err := ensureImage(ctx, cri, r.Image, run); err != nil {
return out, fmt.Errorf("not replacing container %s, which keeps running: %w", r.Name, err)
}
if _, err := run(ctx, cri, "rm", "-f", r.Name); err != nil {
return out, fmt.Errorf("replacing container %s: %w", r.Name, err)
// **The running container is set aside, never removed, until its replacement runs** (issue 1945): stopped and
// renamed, so its name and ports are free, and started again under its own name when the replacement does not
// come up. Removed first, the bus's own container left a node without a bus when its replacement was never
// made (2026-10-25 at 11:23). A container set aside by an earlier apply that stopped midway goes first.
aside = r.Name + asideSuffix
_, _ = run(ctx, cri, "rm", "-f", aside)
if _, err := run(ctx, cri, "rename", r.Name, aside); err != nil {
return out, fmt.Errorf("replacing container %s: setting the running one aside: %w", r.Name, err)
}
if _, err := run(ctx, cri, "stop", aside); err != nil {
_, _ = run(ctx, cri, "rename", aside, r.Name)
return out, fmt.Errorf("replacing container %s: stopping the running one: %w", r.Name, err)
}
}
// putBack starts the container set aside again under its own name, after a replacement that did not come up.
putBack := func(cause error) error {
if aside == "" {
return cause
}
_, _ = run(ctx, cri, "rm", "-f", r.Name)
if _, err := run(ctx, cri, "rename", aside, r.Name); err != nil {
return fmt.Errorf("%w; and the container it replaced could not be put back: %v", cause, err)
}
if _, err := run(ctx, cri, "start", r.Name); err != nil {
return fmt.Errorf("%w; and the container it replaced could not be started again: %v", cause, err)
}
return fmt.Errorf("%w; the container it replaced was started again", cause)
}
args := detachedRunArgs(r, want)
if _, err := run(ctx, cri, args...); err != nil {
return out, fmt.Errorf("starting container %s: %w", r.Name, err)
return out, putBack(fmt.Errorf("starting container %s: %w", r.Name, err))
}
// Read back. `docker run --detach` returning an id says the container was created, not
@@ -2857,15 +3035,21 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
// command exactly as one that came up does.
after, err := containerState(ctx, r.Name, run)
if err != nil {
return out, fmt.Errorf("started container %s and it is not there: %w", r.Name, err)
return out, putBack(fmt.Errorf("started container %s and it is not there: %w", r.Name, err))
}
if !after.Running {
return out, fmt.Errorf(
return out, putBack(fmt.Errorf(
"container %s was started and is not running. It exited; ask the runtime for its "+
"logs", r.Name)
"logs", r.Name))
}
if after.Spec != want {
return out, fmt.Errorf("container %s is not the one that was declared after creating it", r.Name)
return out, putBack(fmt.Errorf("container %s is not the one that was declared after creating it", r.Name))
}
// Only now, with the replacement running, is the container set aside removed.
if aside != "" {
if _, err := run(ctx, cri, "rm", "-f", aside); err != nil {
out.Detail = fmt.Sprintf("the container it replaced, set aside as %s, could not be removed: %v", aside, err)
}
}
// The found networks a per-machine setting keeps for it (novox/hq ADR 0163, rule 4), joined
// once it runs: a runtime starts a container on one network, and the others are connected.
@@ -2901,6 +3085,9 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
return out, nil
}
// asideSuffix names a running container set aside while its replacement is started (issue 1945).
const asideSuffix = ".replaced"
// detachedRunArgs is the `docker run` that starts a service container and leaves it running.
func detachedRunArgs(r *declaration.Container, want string) []string {
args := []string{"run", "--detach", "--name", r.Name, "--restart", "unless-stopped"}
+1
View File
@@ -33,6 +33,7 @@ func newBusyMachine() *aBusyMachine {
}
func (m *aBusyMachine) run(_ context.Context, _ string, args ...string) (string, error) {
args = asideShape(args)
m.mu.Lock()
defer m.mu.Unlock()
switch args[0] {
@@ -20,6 +20,9 @@ func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
if name != "docker" {
return "", errors.New("not installed")
}
if len(args) > 0 && args[0] == "container" && strings.HasSuffix(args[len(args)-1], asideSuffix) {
return "", errors.New("no such container") // nothing was left set aside
}
ran = append(ran, strings.Join(args, " "))
switch args[0] {
case "info":
@@ -75,69 +78,6 @@ func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
var specOfLast string
// A module the mesh left out of a declaration is not a module the mesh removed (novox/hq ADR 0163,
// rule 6): what the host wrote for it stays, recorded and said; what it holds for it stays held.
// A module simply absent is removed as it always was.
func TestALeftOutModuleIsNeitherRemovedNorForgotten(t *testing.T) {
var removed []string
gone := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", nil
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "rm":
removed = append(removed, args[len(args)-1])
gone[args[len(args)-1]] = true
case "container":
if gone[args[len(args)-1]] {
return "", errors.New("no such container")
}
return "true\tspec", nil
}
return "", nil
}
known := store.State{
Resources: []store.Applied{
{ID: "web.server", Type: "container", Target: "web", Origin: store.OriginDeclared},
{ID: "old.server", Type: "container", Target: "old", Origin: store.OriginDeclared},
},
Held: []store.Held{{ID: "web.page", Module: "web", Kind: "file", Target: "/srv/web/index.html"}},
}
d := parse(t, `{"declaration":1,"left_out":["web"],"resources":[
{"id":"notes.conf","type":"file","path":"`+t.TempDir()+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if len(removed) != 1 || removed[0] != "old" {
t.Fatalf("removed %v; only the module that is absent goes", removed)
}
if _, kept := state.At("container", "web"); !kept {
t.Fatal("the left-out module's record was forgotten")
}
if _, held := state.HeldAt("web.page"); !held {
t.Fatal("the left-out module's hold was released")
}
said := false
for _, o := range report.Outcomes {
if o.ID == "web.server" && o.Action == "unchanged" && strings.Contains(o.Detail, "web was left out of this declaration by the mesh") {
said = true
}
if o.ID == "web.server" && o.Action != "unchanged" {
t.Errorf("the left-out module's container was %s", o.Action)
}
}
if !said {
t.Fatalf("keeping the left-out module's container was not said: %+v", report.Outcomes)
}
}
// A container's capabilities reach the runtime and are part of its spec (novox/hq ADR 0170).
func TestACapabilityReachesTheRuntimeAndTheSpec(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
+19
View File
@@ -0,0 +1,19 @@
package apply
import (
"context"
"strings"
"testing"
"time"
)
// A command killed because its context ended says so, and why: "docker exited -1:" said nothing while novox's bus
// container was half replaced (issue 1945, live on 2026-10-25 at 11:23).
func TestACommandKilledByItsContextSaysWhy(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
go func() { time.Sleep(50 * time.Millisecond); cancel() }()
_, err := ExecRunner(ctx, "sleep", "5")
if err == nil || !strings.Contains(err.Error(), "killed") || !strings.Contains(err.Error(), "context canceled") {
t.Fatalf("a command its context killed said: %v", err)
}
}
+1
View File
@@ -76,6 +76,7 @@ func WriteFilled(
in.refused = refusedPlaces(d.Resources)
in.agentHomes = rootNeverHomes(d)
in.dirsBefore = map[string]bool{}
in.files = declaredFiles(d.Resources)
// A container that reads the file and mounts a library whose disk or mount is not there is held, as a full apply
// holds it: not created over the missing disk (novox/hq ADR 0328 §9).
in.held = heldLibraries(ctx, d.Libraries)
+29 -29
View File
@@ -52,6 +52,24 @@ func TestAWriteOfAPathNotFilledIsRefused(t *testing.T) {
if _, err := os.Stat(other); err == nil {
t.Errorf("%s was written", other)
}
t.Run("a filled path two resources declare", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
if err := os.WriteFile(path, []byte(workstationHosts), 0o644); err != nil {
t.Fatal(err)
}
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"filled_files":[%q],"resources":[
{"id":"a.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.1 ace\n","owner-at-runtime":"managed"},
{"id":"b.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.2 g14\n","owner-at-runtime":"managed"}
]}`, path, path, path))
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("6.6.6.6 evil\n"),
noServices, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "2 resources declare") {
t.Fatalf("a path two resources declare was not refused: %v", err)
}
if got := readText(t, path); got != workstationHosts || len(state.Resources) != 0 {
t.Errorf("something was written or recorded: %q %+v", got, state.Resources)
}
})
}
// Each ownership is written by the write verb exactly as an apply of the declared file with the same content: the
@@ -112,6 +130,17 @@ func TestAFilledFileIsWrittenAsItsOwnershipSays(t *testing.T) {
}
})
}
t.Run("a filled file that carried a secret is not world-readable", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "token.env")
d := filledDecl(t, path, `"content":"TOKEN=${secret:token}\n","secrets":{"token":"c2VhbGVk"},"owner-at-runtime":"owned"`, "")
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("TOKEN=abc\n"),
noServices, nil, nil, nil); err != nil {
t.Fatal(err)
}
if m := modeAt(t, path); m != 0o600 {
t.Errorf("a filled file that carried a secret is mode %o", m)
}
})
}
func contentOf(t *testing.T, fields string) string {
@@ -136,17 +165,6 @@ func modeAt(t *testing.T, path string) os.FileMode {
// A file that carried a secret keeps the mode a secret is written with: the client filled the value, and the file
// is no more readable for it.
func TestAFilledFileThatCarriedASecretIsNotWorldReadable(t *testing.T) {
path := filepath.Join(t.TempDir(), "token.env")
d := filledDecl(t, path, `"content":"TOKEN=${secret:token}\n","secrets":{"token":"c2VhbGVk"},"owner-at-runtime":"owned"`, "")
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("TOKEN=abc\n"),
noServices, nil, nil, nil); err != nil {
t.Fatal(err)
}
if m := modeAt(t, path); m != 0o600 {
t.Errorf("a filled file that carried a secret is mode %o", m)
}
}
func TestAWriteUnderReloadOnReloadsItsProgramOnce(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.conf")
@@ -251,21 +269,3 @@ func count(lines []string, prefix string) int {
// Two resources declaring one filled path — two modules' blocks in one file — are refused: the request carries the
// content only, so it cannot say whose block it is, and nothing is written.
func TestAFilledPathTwoResourcesDeclareIsRefused(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
if err := os.WriteFile(path, []byte(workstationHosts), 0o644); err != nil {
t.Fatal(err)
}
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"filled_files":[%q],"resources":[
{"id":"a.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.1 ace\n","owner-at-runtime":"managed"},
{"id":"b.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.2 g14\n","owner-at-runtime":"managed"}
]}`, path, path, path))
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("6.6.6.6 evil\n"),
noServices, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "2 resources declare") {
t.Fatalf("a path two resources declare was not refused: %v", err)
}
if got := readText(t, path); got != workstationHosts || len(state.Resources) != 0 {
t.Errorf("something was written or recorded: %q %+v", got, state.Resources)
}
}
+17 -1
View File
@@ -111,7 +111,10 @@ type fakeContainer struct {
// options. The host passes `--` before every unit (third review of mesh-catalog #147), which
// internal/system's own test holds; the fakes read the unit where they always did.
func oldShape(name string, args []string) []string {
if name != "systemctl" || len(args) < 2 || args[len(args)-2] != "--" {
if name != "systemctl" {
return asideShape(args)
}
if len(args) < 2 || args[len(args)-2] != "--" {
return args
}
v := 0
@@ -1128,3 +1131,16 @@ func TestAPathTheMeshOnlySetAccessOnIsStillFound(t *testing.T) {
t.Errorf("it was re-moded to %o", info.Mode().Perm())
}
}
// asideShape reads the steps of a container set aside while its replacement starts (issue 1945) as the removal the
// tests were written against: setting it aside is its removal, and what is done to the set-aside container afterwards
// is no step of theirs.
func asideShape(args []string) []string {
if len(args) == 3 && args[0] == "rename" && args[2] == args[1]+asideSuffix {
return []string{"rm", "-f", args[1]}
}
if len(args) > 0 && strings.HasSuffix(args[len(args)-1], asideSuffix) && (args[0] == "rm" || args[0] == "stop") {
return []string{"aside"}
}
return args
}
+5 -2
View File
@@ -11,7 +11,9 @@ import (
// The read-only judgement of a node's directories (novox/hq issue 358): one line each — placed, would be refused
// naming the link and who could replace it, or not there yet (placed: an apply makes it) — and nothing made or
// changed.
// changed. Who could replace the link is "every account", "the account X" or "the group X", by the account and the
// directories the test runs as: the checker's isolated step runs as the account nobody, which owns / there (issue 1919,
// R4g), so the test demands a who in any of the forms couldReplace and linkReplacer produce, not "every account" alone.
func TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing(t *testing.T) {
base := t.TempDir()
plain := filepath.Join(base, "plain")
@@ -39,7 +41,8 @@ func TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing(t *testing.
t.Errorf("a directory the walk reaches is not said placed: %q", said)
}
if !strings.Contains(lines[2], "would be refused") || !strings.Contains(lines[2], filepath.Join(base, "linked")) ||
!strings.Contains(lines[2], "every account") {
!(strings.Contains(lines[2], "every account (") || strings.Contains(lines[2], "the account ") ||
strings.Contains(lines[2], "the group ")) {
t.Errorf("the linked directory is not said refused naming the link and who: %q", lines[2])
}
if _, err := os.Stat(filepath.Join(base, "missing")); !os.IsNotExist(err) {
+26
View File
@@ -11,6 +11,7 @@ import (
"path/filepath"
"strconv"
"strings"
"syscall"
"testing"
"time"
@@ -23,6 +24,23 @@ import (
// The engine's half of ADR 0328's checks (novox/hq ADR 0328, "How it is checked", issue 567). The engine runs as
// root; a test may not, and a non-root test gives its folders a group it is in itself.
// canSetGroups skips aloud a test that gives a folder a group when this step cannot. **A step that cannot set a folder's
// group judges nothing here** (issue 1919, R4g): the checker runs the tests in the build module's isolation, a user
// namespace that maps no group but the step's own, so a chown to any other group is refused (EINVAL or EPERM). A test that
// sets a group calls it first: the probe gives a file of the test's own that group, and when it is refused the test skips
// aloud; outside such an isolation it runs and is judged.
func canSetGroups(t *testing.T, gid int) {
t.Helper()
probe := filepath.Join(t.TempDir(), "probe")
if err := os.WriteFile(probe, nil, 0o600); err != nil {
t.Fatal(err)
}
if err := os.Chown(probe, -1, gid); errors.Is(err, syscall.EINVAL) || errors.Is(err, syscall.EPERM) {
t.Skipf("NOT JUDGED: this step cannot set a folder's group (a user namespace maps only its own account, as the "+
"checker's isolated step does): %v; issue 1919", err)
}
}
// testGID is a group the test may give its folders: any number as root, else one of its own added groups that is
// no root group's.
func testGID(t *testing.T) int {
@@ -121,6 +139,7 @@ func statOf(t *testing.T, path string) unix.Stat_t {
// those folders alone, never what is in them; owner, owner's and others' bits unchanged.
func TestALibraryIsKeptInItsGroup(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o755); err != nil {
@@ -170,6 +189,7 @@ func TestALibraryIsKeptInItsGroup(t *testing.T) {
// Row 9: a missing sub-path is made with the root's owner, the library's group, group rwx and setgid.
func TestAMissingSubPathIsMadeAsTheRootIs(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o751); err != nil {
@@ -226,6 +246,7 @@ func TestARootThatHoldsDataWaitsForTheOperatorsWord(t *testing.T) {
// (the act's ask, internal/link); an undeclared library refused; a walk over its bound goes on from its cursor.
func TestTheTakeActSetsTheGroupUnderTheRoot(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
base := t.TempDir()
root := filepath.Join(base, "media")
@@ -512,6 +533,7 @@ func TestNoLibraryGroupIsRoot(t *testing.T) {
// A library's filesystem that cannot be named this time is unknown, never a move; another filesystem at the root is.
func TestALibraryNotNamedIsUnknownAndAnotherFilesystemIsAMove(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
base := t.TempDir()
pool := filepath.Join(base, "pool")
root := filepath.Join(pool, "media")
@@ -650,6 +672,7 @@ func TestALibraryIsNeverTheMachinesOwnTree(t *testing.T) {
// and sample all in shape are kept; once taken, never asked again on that filesystem.
func TestARootInShapeWithDataBelowNotInTheGroupWaits(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
movies := filepath.Join(root, "movies")
@@ -699,6 +722,7 @@ func TestARootInShapeWithDataBelowNotInTheGroupWaits(t *testing.T) {
// file never keeps its setgid bit once regrouped.
func TestRootsLinkedFilesAreLeftAndSetgidIsNeverKept(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
f, err := os.CreateTemp(t.TempDir(), "file")
if err != nil {
t.Fatal(err)
@@ -729,6 +753,7 @@ func TestRootsLinkedFilesAreLeftAndSetgidIsNeverKept(t *testing.T) {
// waits for the operator's word as the root would, and nothing of it is changed; an empty one is kept.
func TestASubPathHoldingOldDataWaitsForTheOperatorsWord(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o755); err != nil {
@@ -795,6 +820,7 @@ func TestAnApplyDoesNotWaitForATakeWalk(t *testing.T) {
// the root alone, says whether the data is the library's already (§7).
func TestTheSampleBeneathARootInShapeIsJudged(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.MkdirAll(filepath.Join(root, "old"), 0o755); err != nil {
@@ -270,6 +270,7 @@ func newRegistryMachine(spec string) *aRegistryMachine {
}
func (m *aRegistryMachine) run(_ context.Context, _ string, args ...string) (string, error) {
args = asideShape(args)
m.mu.Lock()
switch args[0] {
case "info", "image":
+5
View File
@@ -124,3 +124,8 @@ func JudgeDirectories(d *declaration.Declaration) string {
placed+refused+guarded+homed, placed, refused, guarded, homed)
return strings.Join(append([]string{head}, lines...), "\n")
}
// meshDataRoot is the mesh's own tree, where a module placed "mesh" keeps its directory (<data root>/mesh/<module>):
// the engine's, 0755, so only the engine's account makes a directory in it (novox/hq issue 1873). A variable so a
// test can place it.
var meshDataRoot = "/var/lib/mesh"
+100
View File
@@ -0,0 +1,100 @@
//go:build linux
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// novox/hq issue 1873: a module's directory in the mesh's own tree (<data root>/mesh/<module>, place "mesh") that is
// there before its declaration, owned by the engine's own account, was made by the mesh on the way to something below
// it: it is the mesh's, given its declared mode and owner, never used as found. Only the engine's account makes a
// directory in a tree that is the engine's (0755, root); on ace and novox /var/lib/mesh/postgres was there as
// root:root 0700 before postgres's data directory was applied, and the engine left it to root, so the postgres
// provider, running as the operator's account, could not reach its own files.
func TestADirectoryInTheMeshsOwnTreeMadeByTheEngineIsTheMeshs(t *testing.T) {
root := t.TempDir()
before := meshDataRoot
meshDataRoot = filepath.Join(root, "mesh")
t.Cleanup(func() { meshDataRoot = before })
made := filepath.Join(meshDataRoot, "postgres")
if err := os.MkdirAll(made, 0o700); err != nil {
t.Fatal(err)
}
if err := os.Chmod(made, 0o700); err != nil {
t.Fatal(err)
}
decl := `{"id":"postgres.data","type":"directory","path":"` + made + `","mode":"0750"}`
var said []string
_, state, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, func(line string) { said = append(said, line) }, nil)
if err != nil {
t.Fatal(err)
}
if strings.Contains(strings.Join(said, "\n"), "WARNING postgres.data") {
t.Errorf("a directory the engine's account made in the mesh's tree was used as found: %q", said)
}
if a, _ := state.Find("postgres.data"); a.AsFound {
t.Error("it is recorded as found")
}
if info, _ := os.Stat(made); info.Mode().Perm() != 0o750 {
t.Errorf("it was not given its declared mode: %04o", info.Mode().Perm())
}
// Outside the mesh's tree the rule of issue 339 stands: found, used as found.
other := filepath.Join(root, "var-lib", "notes")
if err := os.MkdirAll(other, 0o700); err != nil {
t.Fatal(err)
}
said = nil
decl = `{"id":"notes.data","type":"directory","path":"` + other + `","mode":"0750"}`
if _, _, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, func(line string) { said = append(said, line) }, nil); err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(said, "\n"), "WARNING notes.data") {
t.Errorf("a directory found outside the mesh's tree was not used as found: %q", said)
}
}
// Review of issue 1873's fix: the tree is judged by where the path leads, not by its text. A root-only link inside the
// mesh's tree pointing out of it (to /var/lib/systemd, say) would otherwise make a directory outside the tree, owned by
// the engine's account, the mesh's to chown: issue 1873's fault again. It is used as found.
func TestALinkInTheMeshsTreeLeadingOutOfItIsNotTheTree(t *testing.T) {
rootAlone(t, "")
root := t.TempDir()
before := meshDataRoot
meshDataRoot = filepath.Join(root, "mesh")
t.Cleanup(func() { meshDataRoot = before })
outside := filepath.Join(root, "var-lib", "systemd")
if err := os.MkdirAll(filepath.Join(outside, "data"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Chmod(filepath.Join(outside, "data"), 0o700); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(meshDataRoot, 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink(outside, filepath.Join(meshDataRoot, "postgres")); err != nil {
t.Fatal(err)
}
r := &declaration.Directory{ID: "postgres.data", Path: filepath.Join(meshDataRoot, "postgres", "data"), Mode: "0750"}
out, err := applyDirectory(r, store.Applied{}, true)
if err != nil {
t.Fatal(err)
}
if !out.asFound {
t.Errorf("a directory reached through a link out of the mesh's tree was taken as the mesh's: %+v", out)
}
if got := permOf(t, filepath.Join(outside, "data")); got != 0o700 {
t.Errorf("the directory outside the tree was given mode %o (was 700)", got)
}
}
+148
View File
@@ -0,0 +1,148 @@
package apply
import (
"context"
"path/filepath"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// R6-A AC3: given two modules declaring one path, when one module's want changes, the shared path is neither removed
// nor taken from the other module (novox/hq ADR 0539 §9).
func TestASharedPathIsNeitherRemovedNorTakenFromTheOtherModule(t *testing.T) {
dir := t.TempDir()
shared := filepath.Join(dir, "shared.conf")
own := filepath.Join(dir, "a-own.conf")
a := declare(t, `{"id":"a.shared","type":"file","path":"`+shared+`","content":"x\n","module":"a"},
{"id":"a.own","type":"file","path":"`+own+`","content":"a\n","module":"a"}`)
b := declare(t, `{"id":"b.shared","type":"file","path":"`+shared+`","content":"x\n","module":"b"}`)
state := store.State{}
var err error
_, state, err = ApplyModule(context.Background(), archHost(t), a, Scope{Module: "a", Others: b.Resources},
state, store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
_, state, err = ApplyModule(context.Background(), archHost(t), b, Scope{Module: "b", Others: a.Resources},
state, store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
// A's want changes: it no longer declares the shared path, and b still does.
a2 := declare(t, `{"id":"a.own","type":"file","path":"`+own+`","content":"a\n","module":"a"}`)
report, state, err := ApplyModule(context.Background(), archHost(t), a2, Scope{Module: "a", Others: b.Resources},
state, store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !exists(shared) {
t.Fatal("the path another module still declares was removed")
}
if _, still := state.Find("b.shared"); !still {
t.Fatal("the other module's record was taken away")
}
if _, still := state.Find("a.shared"); still {
t.Fatal("the module's own record of the shared path was kept")
}
if o := outcomeOf(report, "a.shared"); o.Action != "forgotten" {
t.Fatalf("the shared path: %+v", o)
}
}
// R6-A AC4: given modules A and B on a node, when A's want is removed, only A's resources are removed and B is
// untouched.
func TestARemovedModuleTakesOnlyItsOwnResources(t *testing.T) {
dir := t.TempDir()
fa, fb := filepath.Join(dir, "a.conf"), filepath.Join(dir, "b.conf")
d := declare(t, `{"id":"a.conf","type":"file","path":"`+fa+`","content":"a\n","module":"a"},
{"id":"b.conf","type":"file","path":"`+fb+`","content":"b\n","module":"b"}`)
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if rec, _ := state.Find("a.conf"); rec.Module != "a" {
t.Fatalf("the owner was not recorded: %+v", rec)
}
gone := &declaration.Declaration{}
b := declare(t, `{"id":"b.conf","type":"file","path":"`+fb+`","content":"b\n","module":"b"}`)
report, state, err := ApplyModule(context.Background(), archHost(t), gone, Scope{Module: "a", Others: b.Resources},
state, store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if exists(fa) || !exists(fb) {
t.Fatalf("a's file exists %v, b's file exists %v", exists(fa), exists(fb))
}
if o := outcomeOf(report, "a.conf"); o.Action != "removed" {
t.Fatalf("a's file: %+v", o)
}
if _, still := state.Find("b.conf"); !still {
t.Fatal("b's record was removed")
}
if _, still := state.Find("a.conf"); still {
t.Fatal("a's record is still there")
}
}
// A record from before the owner was kept is read by its id, so the first removal after the switch is the module's too.
func TestARecordWithoutAnOwnerBelongsToTheModuleItsIdNames(t *testing.T) {
dir := t.TempDir()
fa, fb := filepath.Join(dir, "a.conf"), filepath.Join(dir, "b.conf")
d := declare(t, `{"id":"a.conf","type":"file","path":"`+fa+`","content":"a\n"},
{"id":"b.conf","type":"file","path":"`+fb+`","content":"b\n"}`)
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
for i := range state.Resources {
state.Resources[i].Module = ""
}
_, state, err = ApplyModule(context.Background(), archHost(t), &declaration.Declaration{}, Scope{Module: "a"},
state, store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if exists(fa) || !exists(fb) {
t.Fatalf("a's file exists %v, b's file exists %v", exists(fa), exists(fb))
}
}
// A resource that leaves the node's own composition for its module's own is that module's now: the node's own apply
// leaves it as it is, whatever its type, until the module's own apply takes it (issue 1945, live on 2026-10-25 at 02:32:
// shanks and g14 lost systemd-resolved's guard, its resolver files and sudo's operator file, ace qbittorrent's server,
// because the node's own apply removed every record no longer in it, the others holding the very same resources or not).
func TestAResourceThatMovedToItsModulesCompositionIsLeftByTheNodesOwnApply(t *testing.T) {
dir := t.TempDir()
fact := filepath.Join(dir, "suffix")
gone := filepath.Join(dir, "gone.conf")
own := declare(t, `{"id":"systemd-resolved.fact-suffix","type":"file","path":"`+fact+`","content":"mesh\n"},
{"id":"old.gone","type":"file","path":"`+gone+`","content":"g\n"}`)
_, state, err := ApplyModule(context.Background(), archHost(t), own, Scope{Module: store.NodeModule},
state0(), store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
// systemd-resolved now has a composition of its own, not applied yet; old.gone is no module's any more.
moved := declare(t, `{"id":"systemd-resolved.fact-suffix","type":"file","path":"`+fact+`","content":"mesh\n","module":"systemd-resolved"}`)
report, state, err := ApplyModule(context.Background(), archHost(t), &declaration.Declaration{},
Scope{Module: store.NodeModule, Others: moved.Resources}, state, store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !exists(fact) {
t.Fatalf("the moved resource was removed: %+v", outcomeOf(report, "systemd-resolved.fact-suffix"))
}
if exists(gone) {
t.Fatal("a resource no composition holds any more was kept")
}
if _, still := state.Find("systemd-resolved.fact-suffix"); !still {
t.Fatal("the moved resource's record was dropped, so its module's apply would not know it")
}
}
func state0() store.State { return store.State{} }
+58
View File
@@ -0,0 +1,58 @@
package apply
import (
"errors"
"os"
"path/filepath"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// declaredFiles is the path of every file a declaration declares.
func declaredFiles(resources []declaration.Resource) map[string]bool {
out := map[string]bool{}
for _, r := range resources {
if f, ok := r.(*declaration.File); ok {
out[filepath.Clean(f.Path)] = true
}
}
return out
}
// mountsAFileNotThere is the first mount source of a container that is a file the declaration declares and is not
// on the machine yet, or "" (novox/hq issue 1873): the runtime makes a missing source a directory.
func mountsAFileNotThere(c *declaration.Container, files map[string]bool) string {
for _, v := range c.Volumes {
src := mountSource(v)
if !filepath.IsAbs(src) || !files[filepath.Clean(src)] {
continue
}
if _, err := os.Lstat(src); errors.Is(err, os.ErrNotExist) {
return src
}
}
return ""
}
// clearRuntimesEmptyDirectory removes what a container runtime leaves at a file's path when it mounted the file before
// it was there (novox/hq issue 1873): an empty directory, in the mesh's own tree by its path and by where its path
// leads, owned by the engine's own account. It says what it removed, or "" when nothing was. Anything else at the path
// is left for the write to meet as before: a directory with anything in it, or one outside the tree, is never removed.
func clearRuntimesEmptyDirectory(path string) (string, error) {
info, err := os.Lstat(path)
if err != nil || !info.IsDir() || !inTheMeshsTree(path) {
return "", nil
}
if uid, _, ok := ownerOf(info); !ok || uid != os.Geteuid() {
return "", nil
}
entries, err := os.ReadDir(path)
if err != nil || len(entries) > 0 {
return "", nil
}
if err := os.Remove(path); err != nil {
return "", err
}
return "removed the empty directory a container runtime left at " + path + ", where this file belongs " +
"(novox/hq issue 1873)", nil
}
+121
View File
@@ -0,0 +1,121 @@
//go:build linux
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// novox/hq issue 1873, its second fault: on ace the postgres container was replaced at 22:11:51 mounting
// superuser.secret, a file the provisioner's client fills only after the apply; the runtime made the missing source
// a directory, and every fill after it failed with "is a directory". A container whose mount source is a file the
// declaration declares, not yet there, is not created or replaced: it waits, and the write of that file (WriteFilled)
// applies what reads it, the container among them (readsChange), once the file is there.
func TestAContainerWaitsForAFileItMountsAndIsMadeWhenTheFillLands(t *testing.T) {
secret := filepath.Join(t.TempDir(), "postgres", "superuser.secret")
d := filledDecl(t, secret, `"content":"","mode":"0600","owner-at-runtime":"owned"`,
`{"id":"postgres.server","type":"container","name":"postgres","image":"`+pinned+`","volumes":["`+
secret+`:/run/secrets/superuser:ro"]}`)
m := &machine{containers: map[string]*fakeContainer{}}
report, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "postgres.server"); o.Action != "waiting" || !strings.Contains(o.Detail, secret) {
t.Errorf("the container mounting a file not yet filled is %+v, not waiting for it", o)
}
if m.did("docker run") {
t.Fatalf("the container was made before the file it mounts was there: %v", m.asked)
}
if _, err := os.Lstat(secret); err == nil {
t.Fatal("something was made at the file's path before the fill")
}
report, _, err = WriteFilled(context.Background(), archHost(t), d, state, secret, []byte("s3cret"), m.run, nil,
nil, nil)
if err != nil {
t.Fatal(err)
}
if !m.did("docker run") {
t.Fatalf("the container was not made when the file it mounts was filled: %v", m.asked)
}
if info, err := os.Lstat(secret); err != nil || !info.Mode().IsRegular() {
t.Fatalf("the filled file is not a file: %v %v", info, err)
}
}
// novox/hq issue 1873: what the runtime left on ace is cured by the mesh. A declared file whose path is an empty
// directory in the mesh's own tree, the engine's account's, is the runtime's missing mount source: the engine
// removes that empty directory and writes the file, and says so. A directory with anything in it, or one outside the
// tree, is never removed: the write fails as before.
func TestAnEmptyDirectoryTheRuntimeLeftAtAFilesPathIsReplacedByTheFile(t *testing.T) {
rootAlone(t, "")
root := t.TempDir()
before := meshDataRoot
meshDataRoot = filepath.Join(root, "mesh")
t.Cleanup(func() { meshDataRoot = before })
secret := filepath.Join(meshDataRoot, "postgres", "superuser.secret")
if err := os.MkdirAll(secret, 0o755); err != nil {
t.Fatal(err)
}
d := filledDecl(t, secret, `"content":"","mode":"0600","owner-at-runtime":"owned"`, "")
m := &machine{containers: map[string]*fakeContainer{}}
report, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, secret, []byte("s3cret"), m.run,
nil, nil, nil)
if err != nil {
t.Fatalf("the file was not written over the empty directory the runtime left: %v", err)
}
if info, err := os.Lstat(secret); err != nil || !info.Mode().IsRegular() {
t.Fatalf("%s is not a file: %v %v", secret, info, err)
}
if o := outcomeOf(report, "m.conf"); !strings.Contains(o.Detail, "empty directory") {
t.Errorf("the write did not say it removed the empty directory: %+v", o)
}
// Not empty: kept, and the write fails.
full := filepath.Join(meshDataRoot, "postgres", "reader.secret")
if err := os.MkdirAll(filepath.Join(full, "something"), 0o755); err != nil {
t.Fatal(err)
}
d = filledDecl(t, full, `"content":"","mode":"0600","owner-at-runtime":"owned"`, "")
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, full, []byte("x"), m.run, nil,
nil, nil); err == nil {
t.Error("a directory with something in it was replaced by a file")
}
if _, err := os.Stat(filepath.Join(full, "something")); err != nil {
t.Errorf("what was in the directory is gone: %v", err)
}
// Outside the tree: kept, and the write fails.
outside := filepath.Join(root, "var-lib", "postgres", "superuser.secret")
if err := os.MkdirAll(outside, 0o755); err != nil {
t.Fatal(err)
}
d = filledDecl(t, outside, `"content":"","mode":"0600","owner-at-runtime":"owned"`, "")
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, outside, []byte("x"), m.run, nil,
nil, nil); err == nil {
t.Error("an empty directory outside the mesh's tree was replaced by a file")
}
// Reached through a link out of the tree: kept, and the write fails.
elsewhere := filepath.Join(root, "elsewhere")
if err := os.MkdirAll(filepath.Join(elsewhere, "x.secret"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink(elsewhere, filepath.Join(meshDataRoot, "linked")); err != nil {
t.Fatal(err)
}
through := filepath.Join(meshDataRoot, "linked", "x.secret")
d = filledDecl(t, through, `"content":"","mode":"0600","owner-at-runtime":"owned"`, "")
_, _, _ = WriteFilled(context.Background(), archHost(t), d, store.State{}, through, []byte("x"), m.run, nil, nil, nil)
if info, err := os.Lstat(filepath.Join(elsewhere, "x.secret")); err != nil || !info.IsDir() {
t.Errorf("a directory outside the tree, reached through a link in it, was replaced: %v %v", info, err)
}
}
+126
View File
@@ -91,3 +91,129 @@ func TestAReplacementImageIsPresentBeforeTheRunningContainerIsRemoved(t *testing
t.Errorf("order of runtime calls = %s, want the image pulled before rm", got)
}
}
// asideRun is a runtime whose container "store" runs from an older spec, recording the steps a replacement takes; run
// fails when failRun, and the new container reads back as declared otherwise.
func asideRun(t *testing.T, want string, failRun bool, order *[]string) Runner {
created := false
return func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "container":
if created {
return "true\t" + want, nil
}
return "true\tsome-older-spec", nil
case "image":
return "", nil
case "rm", "rename", "stop", "start":
*order = append(*order, strings.Join(args, " "))
return "", nil
case "run":
*order = append(*order, "run")
if failRun {
return "", errors.New("port is already allocated")
}
created = true
return "deadbeef\n", nil
}
return "", nil
}
}
// A container whose replacement does not come up is put back: the running one is set aside (renamed and stopped), never
// removed before the new one runs, and started again under its name when the new one fails, the apply failing loudly.
// Removed first, the bus's own container left novox without a bus when its replacement was never made (issue 1945,
// live on 2026-10-25 at 11:23).
func TestAContainerWhoseReplacementFailsIsPutBack(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"store","image":"`+pinned+`","env":{"PGDATA":"/data"}}
]}`)
want := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
var order []string
_, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, asideRun(t, want, true, &order), nil, nil)
if err == nil || !strings.Contains(err.Error(), "started again") {
t.Fatalf("a failed replacement: %v", err)
}
got := strings.Join(order, "; ")
if !strings.Contains(got, "rename store store.replaced; stop store.replaced; run") ||
!strings.Contains(got, "rename store.replaced store; start store") {
t.Fatalf("the steps were %q", got)
}
if strings.Contains(got, "rm -f store;") && strings.Index(got, "rm -f store;") < strings.Index(got, "run") {
t.Fatalf("the running container was removed before its replacement ran: %q", got)
}
}
// A replacement that runs removes the container set aside, and only after it runs.
func TestAReplacementThatRunsRemovesTheContainerSetAside(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"store","image":"`+pinned+`","env":{"PGDATA":"/data"}}
]}`)
want := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
var order []string
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, asideRun(t, want, false, &order), nil, nil); err != nil {
t.Fatal(err)
}
got := strings.Join(order, "; ")
run, removed := strings.Index(got, "run"), strings.LastIndex(got, "rm -f store.replaced")
if run < 0 || removed < run || strings.Contains(got, "start store") {
t.Fatalf("the steps were %q: the set-aside container is removed after the replacement runs", got)
}
}
// A container left set aside by an engine that stopped between setting it aside and running its replacement is the
// set-aside one of the next apply: put back when the replacement fails, removed once it runs (the review of issue
// 1945).
func TestAContainerLeftAsideIsPutBackOrRemovedByTheNextApply(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"store","image":"`+pinned+`","env":{"PGDATA":"/data"}}
]}`)
want := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
for _, failRun := range []bool{true, false} {
var order []string
created := false
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "container":
switch args[len(args)-1] {
case "store":
if created {
return "true\t" + want, nil
}
return "", errors.New("no such container")
case "store.replaced":
return "false\tsome-older-spec", nil
}
return "", errors.New("no such container")
case "image":
return "", nil
case "rm", "rename", "stop", "start":
order = append(order, strings.Join(args, " "))
return "", nil
case "run":
order = append(order, "run")
if failRun {
return "", errors.New("port is already allocated")
}
created = true
return "deadbeef\n", nil
}
return "", nil
}
_, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
got := strings.Join(order, "; ")
if failRun {
if err == nil || !strings.Contains(got, "rename store.replaced store; start store") {
t.Fatalf("a failed run beside a container left aside: %v; steps %q", err, got)
}
continue
}
if err != nil || !strings.Contains(got, "run; rm -f store.replaced") {
t.Fatalf("a run beside a container left aside: %v; steps %q", err, got)
}
}
}
+12 -1
View File
@@ -127,6 +127,17 @@ func (s *Scheduler) RecordWindowsIn(w *Windows) {
// work on the predecessor's data, under the predecessor's service, which is the one thing an
// adopted node must not do. Nil on a converged node, where nothing is held.
func (s *Scheduler) Sync(d *declaration.Declaration, held map[string]bool) {
s.sync(d, held, nil)
}
// SyncModule is Sync for one module's declaration (novox/hq ADR 0539 §9): the schedules of that module are armed,
// re-armed or forgotten, and the schedules of every other module are left as they are. owns says whether a scheduled
// step's id is the module's.
func (s *Scheduler) SyncModule(d *declaration.Declaration, held map[string]bool, owns func(id string) bool) {
s.sync(d, held, owns)
}
func (s *Scheduler) sync(d *declaration.Declaration, held map[string]bool, owns func(id string) bool) {
s.mu.Lock()
defer s.mu.Unlock()
@@ -177,7 +188,7 @@ func (s *Scheduler) Sync(d *declaration.Declaration, held map[string]bool) {
}
for id := range s.jobs {
if !seen[id] {
if !seen[id] && (owns == nil || owns(id)) {
delete(s.jobs, id)
}
}
+9 -3
View File
@@ -150,13 +150,19 @@ func TestAForgeManifestTheCatalogueDoesNotHaveIsNamed(t *testing.T) {
// theCataloguesModule is a module's own manifest, never a copy: the clone beside a merge check (MESH_CHECK_BESIDE), where it
// must be; else a checkout beside this one; else the test skips aloud. The catalogue holds it, but for the build agent,
// which moved whole into the controller's repository (ADR 0420, issue 1019, issue 1151) and is read from there.
// which moved whole into the controller's repository (ADR 0420, issue 1019, issue 1151) and was read from there.
//
// **The build agent is not there any more** (R4g, ADR 0524 section 5): the group deletes it from the controller with the seat
// node-build-agent. Genesis still installs it as a new mesh's builder, and where genesis gets a builder instead is issue
// 1152's decision, queued out of that milestone. Until then every test whose subject is the build agent's manifest skips
// aloud here, and is judged again with issue 1152.
func theCataloguesModule(t *testing.T, module string) []byte {
t.Helper()
repository := "mesh-catalog"
if module == BuilderModule {
repository = "mesh-controller"
t.Skipf("NOT JUDGED: genesis still installs build-agent, which left mesh-controller with R4g (ADR 0524 section 5); " +
"a new mesh cannot raise a builder until issue 1152 gives genesis one, and this test is judged again with it")
}
repository := "mesh-catalog"
inside := filepath.Join(repository, "modules", module, "module.json")
if root := os.Getenv("MESH_CHECK_BESIDE"); root != "" {
raw, err := os.ReadFile(filepath.Join(root, inside))
+7
View File
@@ -241,9 +241,16 @@ func TestAFileNothingOfTheControllerWritesIsStillRefused(t *testing.T) {
// The controller's own manifest, as the mesh runs it, passes step 9 (issue 597): in a merge check the
// node-engine's check has the controller cloned beside it (MESH_CHECK_BESIDE), where it must be; else a
// checkout beside this one; else it skips aloud.
//
// The checker places the controller beside this repository (mesh-delivery's besideRules, since R4g, issue 1295), so a
// MESH_CHECK_BESIDE without a mesh-controller directory, or without its module.json, fails. Only a run with no
// MESH_CHECK_BESIDE and no checkout beside this one (a developer's clone alone) skips aloud.
func TestTheControllersOwnManifestPassesStepNine(t *testing.T) {
var manifest []byte
if root := os.Getenv("MESH_CHECK_BESIDE"); root != "" {
if info, err := os.Stat(filepath.Join(root, "mesh-controller")); err != nil || !info.IsDir() {
t.Fatalf("the controller is not beside this check in MESH_CHECK_BESIDE=%s: the checker places mesh-controller beside the node-engine's check (mesh-delivery besideRules, issue 1295), and this test judges the controller's own manifest", root)
}
raw, err := os.ReadFile(filepath.Join(root, "mesh-controller", "module.json"))
if err != nil {
t.Fatalf("the controller is not beside this check in MESH_CHECK_BESIDE=%s: %v", root, err)
@@ -26,6 +26,11 @@ func TestTheBaseStepBuildsEveryToolchainAGenesisModuleIsCompiledIn(t *testing.T)
built[tc.Language] = true
}
for _, module := range genesisModules {
if module == BuilderModule {
t.Logf("NOT JUDGED: genesis still installs build-agent, which left mesh-controller with R4g (ADR 0524 section 5); " +
"only build-agent is set aside here until issue 1152 gives genesis a builder, the other modules are judged")
continue
}
var m struct {
Build struct {
Artifacts []struct {
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the repository's root files or other directories): mesh-build-go never keeps this package's verdict (issue 1935).
+107 -1
View File
@@ -29,6 +29,7 @@ import (
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
@@ -87,6 +88,28 @@ type Engine struct {
// change. Called holding the lock, with the declaration read under it.
Place func(ctx context.Context, d *declaration.Declaration, path string, content []byte) (action, digest string,
err error)
// owed is what this engine refused or failed to write since it last told the client to fill again (Knock).
owed owed
}
// owed is the writes a later declaration may let through: a path refused as not named as filled, a write refused
// while the declaration named no provisioner account, and a path whose write failed. The client is told to fill
// again only once a declaration may end one (Knock): it has no other way to learn that the declaration changed.
type owed struct {
mu sync.Mutex
unnamed map[string]bool // refused: the declaration did not name it as filled
failed map[string]bool // attempted, and the write or what reads it failed
noAccount bool // refused: the declaration named no provisioner account
}
func (o *owed) add(kind *map[string]bool, path string) {
o.mu.Lock()
defer o.mu.Unlock()
if *kind == nil {
*kind = map[string]bool{}
}
(*kind)[path] = true
}
// Accepts reads the declaration and judges the account.
@@ -95,7 +118,21 @@ func (e *Engine) Accepts(uid uint32) error {
if err != nil {
return fmt.Errorf("this node's declaration cannot be read, so no write is accepted: %w", err)
}
return JudgeAccount(d, uid, e.UIDOf)
if err := JudgeAccount(d, uid, e.UIDOf); err != nil {
e.oweAccount(d)
return err
}
return nil
}
// oweAccount remembers a refusal a declaration naming the provisioner's account may end.
func (e *Engine) oweAccount(d *declaration.Declaration) {
if d.Provisioner != "" {
return // another account than the client's: no declaration lets it through
}
e.owed.mu.Lock()
e.owed.noAccount = true
e.owed.mu.Unlock()
}
// Write takes the apply's lock, reads the declaration under it, judges the account and the path again — the
@@ -114,18 +151,87 @@ func (e *Engine) Write(ctx context.Context, uid uint32, req Request) Answer {
return Refusal(req.Path, "this node's declaration cannot be read, so nothing was written: "+err.Error())
}
if err := JudgeAccount(d, uid, e.UIDOf); err != nil {
e.oweAccount(d)
return Refusal(req.Path, err.Error())
}
if err := JudgePath(d, req.Path); err != nil {
e.owed.add(&e.owed.unnamed, req.Path)
return Refusal(req.Path, err.Error())
}
action, digest, err := e.Place(ctx, d, req.Path, req.Content)
if err != nil {
e.owed.add(&e.owed.failed, req.Path)
return Answer{Path: req.Path, Action: action, Digest: digest, Failed: err.Error()}
}
return Answer{Path: req.Path, Action: action, Digest: digest}
}
// Knock tells the provisioner's client to fill again when the declaration this node applies now may let through a
// write this engine refused or failed (issue 1873; ace, 2026-10-25 21:17: mesh-build-go's files were refused as not
// named as filled 32 s before its want arrived, and nothing filled them after the apply that named them): a path
// refused as not named that the declaration now names, a write refused while no provisioner account was named once
// one is, and any path whose write failed. Called once a module's apply or removal settled (the node agent's
// Settled, after what runs holds it): an event of the engine's own, never a timer. It holds the apply's lock, as a
// write does, so a write judged by the declaration before the apply settled has recorded its refusal before Knock
// reads what is owed, and one judged after is judged by the settled declaration. The client is told by touching the
// write socket, whose attributes it watches (mesh-provisioner-client socketwatch.go, IN_ATTRIB), and then fills every
// module it holds again. A refusal the declaration still does not end is kept for the next apply. It answers what it
// told the client of, for the journal; nothing when it told nothing. With nothing owed it reads no declaration.
func (e *Engine) Knock(socket string) (string, error) {
unlock, err := e.Lock()
if err != nil {
return "", fmt.Errorf("the apply's lock could not be taken, so the provisioner's client is not told to fill "+
"again: %w", err)
}
defer unlock()
e.owed.mu.Lock()
nothing := len(e.owed.unnamed) == 0 && len(e.owed.failed) == 0 && !e.owed.noAccount
e.owed.mu.Unlock()
if nothing {
return "", nil
}
d, err := e.Declared()
if err != nil {
return "", fmt.Errorf("this node's declaration cannot be read, so the provisioner's client is not told to "+
"fill again: %w", err)
}
e.owed.mu.Lock()
var why []string
for _, p := range sortedSet(e.owed.unnamed) {
if d.Fills(p) {
why = append(why, p+" (now named as filled)")
delete(e.owed.unnamed, p)
}
}
for _, p := range sortedSet(e.owed.failed) {
why = append(why, p+" (its write failed)")
delete(e.owed.failed, p)
}
if e.owed.noAccount && d.Provisioner != "" {
why = append(why, "every file (the provisioner's account is named now)")
e.owed.noAccount = false
}
e.owed.mu.Unlock()
if len(why) == 0 {
return "", nil
}
now := time.Now()
if err := os.Chtimes(socket, now, now); err != nil {
return "", fmt.Errorf("the provisioner's client could not be told to fill %s again: %w",
strings.Join(why, ", "), err)
}
return strings.Join(why, ", "), nil
}
func sortedSet(m map[string]bool) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// JudgeAccount refuses every account but the one the declaration names as the provisioner's client's, by its user
// id here — and every account while the declaration names none.
func JudgeAccount(d *declaration.Declaration, uid uint32, uidOf func(string) (uint32, error)) error {
+132
View File
@@ -88,6 +88,138 @@ func TestAnUndeclaredPathIsRefused(t *testing.T) {
if len(e.placed) != 0 {
t.Errorf("placed %v", e.placed)
}
// A refusal a later declaration ends is followed by a knock on the socket, which the provisioner's client watches,
// once a declaration applied after it ends it, and only then (ace, 2026-10-25 21:17: mesh-build-go's files refused
// 32 s before its want, and never filled after the apply that named them).
socket := filepath.Join(t.TempDir(), "fill.sock")
if err := os.WriteFile(socket, nil, 0o666); err != nil {
t.Fatal(err)
}
long := time.Now().Add(-time.Hour)
knocked := func() bool {
info, err := os.Stat(socket)
if err != nil {
t.Fatal(err)
}
return info.ModTime().After(long.Add(time.Minute))
}
namesOther := strings.NewReplacer(`"filled_files":["/etc/m/m.conf"]`,
`"filled_files":["/etc/m/m.conf","/etc/m/other.conf"]`, `"content":"y"}`,
`"content":"y","owner-at-runtime":"owned"}`).Replace(withAccount)
failing := errors.New("is a directory")
var mu2 sync.Mutex // guards the race row's count of declaration reads
for _, row := range []struct {
name string
before string // the declaration the write is judged by
account uint32
path string
place error
after string // the declaration applied afterwards
knock string // what the knock names; "" for none
twice bool // a second knock names nothing
}{
{name: "an undeclared path the next declaration does not name either", before: withAccount,
account: provisionerUID, path: "/etc/m/other.conf", after: withAccount},
{name: "an undeclared path the next declaration names", before: withAccount, account: provisionerUID,
path: "/etc/m/other.conf", after: namesOther, knock: "/etc/m/other.conf (now named as filled)", twice: true},
{name: "a write refused while no account was named, then named", before: withoutAccount,
account: provisionerUID, path: "/etc/m/m.conf", after: withAccount,
knock: "every file (the provisioner's account is named now)", twice: true},
{name: "a write that failed", before: withAccount, account: provisionerUID, path: "/etc/m/m.conf",
place: failing, after: withAccount, knock: "/etc/m/m.conf (its write failed)", twice: true},
{name: "another account, refused whatever is declared", before: withAccount, account: 1000,
path: "/etc/m/m.conf", after: withAccount},
} {
t.Run(row.name, func(t *testing.T) {
if err := os.Chtimes(socket, long, long); err != nil {
t.Fatal(err)
}
e := newEngine(t, row.before, &sync.Mutex{})
if row.place != nil {
e.Place = func(context.Context, *declaration.Declaration, string, []byte) (string, string, error) {
return "", "", row.place
}
}
_ = e.Accepts(row.account)
e.Write(context.Background(), row.account, Request{Path: row.path, Content: []byte("x")})
after := declared(t, row.after)
e.Declared = func() (*declaration.Declaration, error) { return after, nil }
told, err := e.Knock(socket)
if err != nil || told != row.knock || knocked() != (row.knock != "") {
t.Fatalf("knock told %q (%v), socket touched %v; want %q", told, err, knocked(), row.knock)
}
if row.twice {
if err := os.Chtimes(socket, long, long); err != nil {
t.Fatal(err)
}
if told, _ := e.Knock(socket); told != "" || knocked() {
t.Fatalf("a second knock told %q: what was told is no longer owed", told)
}
}
})
}
// The race (review of this fix): a write judged by the declaration before the apply settled, its refusal not yet
// recorded when Knock runs. Knock waits for the apply's lock the write holds, so it sees that refusal and knocks.
t.Run("a write still judging when the apply settles", func(t *testing.T) {
if err := os.Chtimes(socket, long, long); err != nil {
t.Fatal(err)
}
before, after := declared(t, withAccount), declared(t, namesOther)
judging, release := make(chan struct{}), make(chan struct{})
var mu sync.Mutex
knockLocking := make(chan struct{}, 1)
calls := 0
e := newEngine(t, withAccount, &mu)
e.Declared = func() (*declaration.Declaration, error) {
mu2.Lock()
calls++
first := calls == 1
mu2.Unlock()
if first {
close(judging)
<-release
return before, nil
}
return after, nil
}
e.Lock = func() (func(), error) {
mu2.Lock()
second := calls > 0
mu2.Unlock()
if second {
knockLocking <- struct{}{}
}
mu.Lock()
return mu.Unlock, nil
}
wrote := make(chan Answer)
go func() { wrote <- e.Write(context.Background(), provisionerUID, Request{Path: "/etc/m/other.conf"}) }()
<-judging
type result struct {
told string
err error
}
knocked2 := make(chan result, 1)
go func() { told, err := e.Knock(socket); knocked2 <- result{told, err} }()
// Released once Knock waits for the lock, or once it has answered without taking it.
var r result
select {
case <-knockLocking:
close(release)
r = <-knocked2
case r = <-knocked2:
close(release)
}
if a := <-wrote; a.Refused == "" {
t.Fatalf("the write was not refused: %+v", a)
}
if r.err != nil || r.told != "/etc/m/other.conf (now named as filled)" || !knocked() {
t.Fatalf("knock told %q (%v), socket touched %v: the refusal still being judged was missed", r.told, r.err,
knocked())
}
})
}
func TestAnotherAccountIsRefused(t *testing.T) {
-130
View File
@@ -1,130 +0,0 @@
package link
import (
"context"
"strings"
"testing"
"time"
)
// The `report` verb (novox/hq to-be 45 §6), what healer H1 asks a machine whose send went unreported:
// the node says again what it last applied, as an ordinary report through the one queue.
// **Asked, the reconcile's account is said even when it is not news** — the mesh asked because it does
// not know, and silence would be the very fault it is asking about.
func TestAskedToReportSaysWhatWasLastAppliedEvenWhenNotNews(t *testing.T) {
m, _ := aMember(t)
l := newQuietLink()
var lines []string
q := aQueue(m, (&applies{}).apply, &keptInMemory{})
q.Say = func(s string) { lines = append(lines, s) }
q.Reconcile = func(context.Context) (Report, bool) {
return Report{Declared: "d-kept", Order: Order{Epoch: 7, Sequence: 12}, Applied: []string{"a"}}, true
}
q.News = func(Report) bool { return false }
q.attach(context.Background(), l)
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
q.ReconcileDue() // an ordinary reconcile: not news, not said
time.Sleep(50 * time.Millisecond)
if n := len(l.said()); n != 0 {
t.Fatalf("an ordinary reconcile that is not news said %d report(s)", n)
}
q.ReportAsked()
eventually(t, "asked, the node said nothing", func() bool { return len(l.said()) == 1 })
stop()
<-worker
r := l.said()[0]
if r.Declared != "d-kept" || r.Sequence != 12 || r.Epoch != 7 || r.ReportSequence == 0 || r.Node != m.Node {
t.Fatalf("the account said is not the one kept, in order: %+v", r)
}
if !strings.Contains(strings.Join(lines, "\n"), "asked by the mesh") {
t.Errorf("being asked is not said in the journal: %q", lines)
}
}
// **A delivery waiting answers the question better**: it is applied, once, and its report is the answer.
func TestAskedWhileADeliveryWaitsIsThatDeliverysReport(t *testing.T) {
m, key := aMember(t)
l := newQuietLink()
a := &applies{}
q := aQueue(m, a.apply, &keptInMemory{})
reconciled := 0
q.Reconcile = func(context.Context) (Report, bool) { reconciled++; return Report{}, true }
q.attach(context.Background(), l)
q.Deliver(ordered(t, key, "new", 7, 13))
q.ReportAsked()
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
eventually(t, "the delivery was never reported", func() bool { return len(l.said()) == 1 })
time.Sleep(50 * time.Millisecond)
stop()
<-worker
if reconciled != 0 || len(l.said()) != 1 || l.said()[0].Sequence != 13 {
t.Fatalf("reconciled %d, said %+v", reconciled, l.said())
}
}
// askingLink is a quiet link the mesh can ask.
type askingLink struct {
*quietLink
asks chan struct{}
}
func (l askingLink) AskedToReport() <-chan struct{} { return l.asks }
// **The link hands the question to the queue**: asked on the link, the account is said on it.
func TestALinkThatIsAskedHandsItToTheQueue(t *testing.T) {
m, _ := aMember(t)
l := askingLink{quietLink: newQuietLink(), asks: make(chan struct{}, 1)}
q := aQueue(m, (&applies{}).apply, &keptInMemory{})
q.Reconcile = func(context.Context) (Report, bool) { return Report{Declared: "d-kept"}, true }
ctx, stop := context.WithCancel(context.Background())
defer stop()
worker := runWorker(ctx, q)
served := make(chan error, 1)
go func() { served <- serve(ctx, l, m, q, nil, time.Second) }()
l.asks <- struct{}{}
eventually(t, "asked on the link, nothing was said", func() bool {
for _, r := range l.said() {
if r.Declared == "d-kept" {
return true
}
}
return false
})
stop()
<-served
<-worker
}
// **Against a real server**: the question on the node's own subject reaches it, and another's does not.
func TestNatsTheMeshAsksANodeToReport(t *testing.T) {
conn, js := aBus(t)
l := &natsLink{conn: conn, js: js, node: "asked"}
l.hearAsks()
defer l.Close()
if err := conn.Publish(AskReportSubject("another"), []byte(`{}`)); err != nil {
t.Fatal(err)
}
if err := conn.Publish(AskReportSubject("asked"), []byte(`{"by":"the controller's healer H1"}`)); err != nil {
t.Fatal(err)
}
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
select {
case <-l.AskedToReport():
case <-time.After(5 * time.Second):
t.Fatal("the question on the node's own subject never reached it")
}
select {
case <-l.AskedToReport():
t.Fatal("asked twice: another node's question reached this one")
case <-time.After(200 * time.Millisecond):
}
if AskReportSubject("asked") != "mesh.node.asked.ask.report" {
t.Fatalf("the subject %q is not the one the controller asks on and grants", AskReportSubject("asked"))
}
}
+2 -2
View File
@@ -25,8 +25,8 @@ func TestNatsAToolCheckIsAskedOnTheLink(t *testing.T) {
t.Fatal(err)
}
q := &Queue{Membership: Membership{Node: node}}
q.attach(t.Context(), &natsLink{conn: conn, js: js, node: node})
q := &Session{Membership: Membership{Node: node}}
q.Attach(t.Context(), &natsLink{conn: conn, js: js, node: node})
ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second)
defer cancel()
a, err := q.AskTool(ctx, "sensors", "sensors_health")
+1 -1
View File
@@ -184,7 +184,7 @@ func (l handingLink) HearHandOvers(h HandOverFunc) { l.given <- h }
func TestALinkThatHearsHandOversIsGivenTheQueues(t *testing.T) {
m, _ := aMember(t)
l := handingLink{quietLink: newQuietLink(), given: make(chan HandOverFunc, 1)}
q := aQueue(m, (&applies{}).apply, &keptInMemory{})
q := &Session{Membership: m, Timeout: time.Second}
q.HandOver = func(path, by string) (string, error) { return "the queue's: " + path + " by " + by, nil }
ctx, stop := context.WithCancel(context.Background())
served := make(chan error, 1)
+318 -32
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"sort"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -36,12 +37,88 @@ type haveState struct {
mu sync.Mutex
modules map[string]ModuleFact
health map[string]delivery.Health
node *delivery.HaveNode
// looked says a look's health was taken: a running module it judges no resource of then has nothing that could be
// down, and is ok (issue 1945).
looked bool
node *delivery.HaveNode
// nodeComposed is the digest of the node's own composition it runs (ADR 0539 §9), kept beside the node's facts.
nodeComposed string
// written is the build of each module this node-engine wrote a have key for and that the declaration it applied
// still holds; removed is each such module the latest declaration no longer holds, with the build its key last
// said, until the key is written once with the phase removed (issue 1369, novox/hq ADR 0408 §4: never deleted).
written map[string]string
removed map[string]string
// userList is what the bus node's engine last wrote of the bus's user list, by the module whose have says it (novox/hq
// ADR 0547 §8); kept apart from the module's fact, which every apply replaces.
userList map[string]delivery.UserListWritten
// unjudged is each module whose composition moved (an apply began, or the fact names another build) and that no look
// has judged since it stopped moving: its health is not yet judged, never the last look's of what ran before.
unjudged map[string]bool
}
// inMotion says a module's phase is one of a change still under way: what a look finds of it then judges nothing of the
// composition being applied (a container stopped to be replaced reads unhealthy), so its health is not yet judged.
func inMotion(phase string) bool {
switch delivery.Phase(phase) {
case delivery.PhaseBackingUp, delivery.PhaseApplying, delivery.PhaseChecking:
return true
}
return false
}
// healthOf is the health a module's have carries: unknown (not yet judged) while its change is under way or until a look
// after it, the last look's otherwise. The caller holds the lock.
func (s *haveState) healthOf(module string, f ModuleFact) delivery.Health {
if inMotion(f.Phase) {
return delivery.HealthUnknown
}
if got, known := s.health[module]; known {
return got
}
if s.unjudged[module] {
return delivery.HealthUnknown
}
return nothingJudged(f, s.looked)
}
// moved marks a module's health not yet judged when its fact f is a change under way, or names another composition than
// the fact before it: the last look judged what ran before. The caller holds the lock.
func (s *haveState) moved(f ModuleFact) {
if prev, known := s.modules[f.Module]; inMotion(f.Phase) || known && (prev.FP != f.FP || prev.Composed != f.Composed) {
delete(s.health, f.Module)
if s.unjudged == nil {
s.unjudged = map[string]bool{}
}
s.unjudged[f.Module] = true
}
}
// owed says some module left motion and no look judged it since. The caller holds the lock.
func (s *haveState) owed() bool {
for module, f := range s.modules {
if s.unjudged[module] && !inMotion(f.Phase) {
return true
}
}
return false
}
// takeLook keeps a look's health: of a module still in motion nothing, since the look judged what is being replaced;
// of every other module what the look found, which is then judged. The caller holds the lock.
func (s *haveState) takeLook(h *Health) {
next := moduleHealths(h)
for module, f := range s.modules {
if inMotion(f.Phase) {
delete(next, module)
if s.unjudged == nil {
s.unjudged = map[string]bool{}
}
s.unjudged[module] = true
continue
}
delete(s.unjudged, module)
}
s.health, s.looked = next, true
}
// moduleHealths is the health of each module a look names, from the states of its resources: failed when any is
@@ -97,11 +174,23 @@ func nodeFacts(r Report) delivery.HaveNode {
}
// noteApply takes what an apply's report says of the modules and the node.
func (q *Queue) noteApply(r Report) bool {
if r.Modules == nil {
return false
}
func (q *Session) noteApply(r Report) bool {
s := &q.haves
if r.Modules == nil {
// Facts that say nothing of any module (the node agent writes those, ADR 0539 §9): the node's own key and the
// health the facts carry.
if r.Refused != "" {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
if r.Health != nil {
s.takeLook(r.Health)
}
n := nodeFacts(r)
s.node = &n
return true
}
s.mu.Lock()
defer s.mu.Unlock()
next := map[string]ModuleFact{}
@@ -136,9 +225,20 @@ func (q *Queue) noteApply(r Report) bool {
for module := range next {
delete(s.removed, module) // added back: its have is written with its new phase
}
for module := range s.modules {
if _, held := next[module]; !held {
delete(s.unjudged, module)
}
}
for _, f := range next {
s.moved(f)
}
s.modules = next
if r.Health != nil {
s.health = moduleHealths(r.Health)
s.takeLook(r.Health)
}
if s.owed() {
q.oweHealth()
}
n := nodeFacts(r)
s.node = &n
@@ -146,31 +246,26 @@ func (q *Queue) noteApply(r Report) bool {
}
// noteHealth takes a look's health, and says whether any module's changed.
func (q *Queue) noteHealth(h Health) bool {
func (q *Session) noteHealth(h Health) bool {
s := &q.haves
s.mu.Lock()
defer s.mu.Unlock()
next := moduleHealths(&h)
before := map[string]delivery.Health{}
for module, f := range s.modules {
before[module] = s.healthOf(module, f)
}
s.takeLook(&h)
changed := false
for module := range s.modules {
want, ok := next[module]
if !ok {
want = delivery.HealthUnknown
}
have, known := s.health[module]
if !known {
have = delivery.HealthUnknown
}
if have != want {
for module, f := range s.modules {
if s.healthOf(module, f) != before[module] {
changed = true
}
}
s.health = next
return changed
}
// sayHave is a line in the node-engine's log, if it has one.
func (q *Queue) sayHave(s string) {
func (q *Session) sayHave(s string) {
if q.Say != nil {
q.Say(s)
}
@@ -180,7 +275,7 @@ func (q *Queue) sayHave(s string) {
// §4, issue 1132): each module's have, and the node's own facts. It reads the key first and writes nothing when the value
// is the same, otherwise writes over the revision it read, and creates the key when there is none. A write the bucket
// refuses is said and left: the next apply or look writes again, and the report was sent before this ran.
func (q *Queue) writeHaves(ctx context.Context, bus Bus) {
func (q *Session) writeHaves(ctx context.Context, bus Bus) {
hb, ok := bus.(HaveBus)
if !ok || bus == nil {
return
@@ -199,6 +294,7 @@ func (q *Queue) writeHaves(ctx context.Context, bus Bus) {
bucket, err := hb.HaveBind(write)
if err != nil {
q.sayHave(fmt.Sprintf("could not bind the have bucket, and wrote nothing there; the next apply or look writes again: %v", err))
q.haveFailed(err)
return
}
modules := make([]string, 0, len(s.modules))
@@ -208,13 +304,17 @@ func (q *Queue) writeHaves(ctx context.Context, bus Bus) {
sort.Strings(modules)
for _, module := range modules {
f := s.modules[module]
h := delivery.Have{V: delivery.Version, Build: f.Build, Phase: delivery.PhaseRunning,
Health: delivery.HealthUnknown, Detail: f.Detail}
if f.Failed {
h := delivery.Have{V: delivery.Version, FP: f.FP, Composed: f.Composed, Build: f.Build,
Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown, Detail: f.Detail}
switch {
case f.Phase != "":
h.Phase = delivery.Phase(f.Phase)
case f.Failed:
h.Phase = delivery.PhaseFailed
}
if got, known := s.health[module]; known {
h.Health = got
h.Health = s.healthOf(module, f)
if w, ok := s.userList[module]; ok {
h.UserList = &w
}
key, err := delivery.HaveKey(node, module)
if err != nil {
@@ -244,24 +344,68 @@ func (q *Queue) writeHaves(ctx context.Context, bus Bus) {
key, err := delivery.HaveNodeKey(node)
if err == nil {
var raw []byte
if raw, err = delivery.EncodeHaveNode(*s.node); err == nil {
want := *s.node
q.putHave(write, bucket, key, raw, func(current []byte) bool {
facts := *s.node
facts.Composed = s.nodeComposed
if raw, err = delivery.EncodeHaveNode(facts); err == nil {
want := facts
if !q.putHave(write, bucket, key, raw, func(current []byte) bool {
c, err := delivery.DecodeHaveNode(current)
return err == nil && delivery.SameHaveNode(c, want)
})
}) {
return
}
}
}
if err != nil {
q.sayHave(fmt.Sprintf("cannot write this node's own facts: %v", err))
}
}
q.haveSucceeded()
}
// haveFailed takes a failure of a pass over the have bucket: a transient one (the bus reloading its accounts, a
// reconnect) writes the pass again after a backoff (Backoff), one pending at a time, since no apply or look may come
// for minutes and the haves would say what is no longer so. Another failure waits for the next apply or look, as before.
func (q *Session) haveFailed(err error) {
if !Transient(err) {
return
}
q.haveAgain.Lock()
defer q.haveAgain.Unlock()
if q.haveAgainPending {
return
}
q.haveAgainPending = true
q.haveAgainTries++
wait := haveBackoff(q.haveAgainTries)
q.sayHave(fmt.Sprintf("the have bucket did not answer (try %d); written again in %s", q.haveAgainTries, wait))
time.AfterFunc(wait, func() {
q.haveAgain.Lock()
q.haveAgainPending = false
q.haveAgain.Unlock()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus != nil {
q.writeHaves(context.Background(), bus) // a link opened later writes them anyway
}
})
}
// haveBackoff is the wait before a write of the haves again; a test shortens it.
var haveBackoff = Backoff
// haveSucceeded takes a pass that wrote every have: the next transient failure waits the first backoff again.
func (q *Session) haveSucceeded() {
q.haveAgain.Lock()
q.haveAgainTries = 0
q.haveAgain.Unlock()
}
// writeRemoved writes, once, each module the applied declaration no longer holds and this node-engine wrote a have key
// for, with the phase removed over the revision it reads (issue 1369). The key is never deleted, and a key that is not
// in the bucket is not created. False when the pass should stop. The caller holds the lock of the haves.
func (q *Queue) writeRemoved(ctx context.Context, b HaveBucket, node string) bool {
func (q *Session) writeRemoved(ctx context.Context, b HaveBucket, node string) bool {
s := &q.haves
gone := make([]string, 0, len(s.removed))
for module := range s.removed {
@@ -294,6 +438,7 @@ func (q *Queue) writeRemoved(ctx context.Context, b HaveBucket, node string) boo
continue
case err != nil:
q.sayHave(fmt.Sprintf("could not read %s in the have bucket, and wrote nothing more there: %v", key, err))
q.haveFailed(err)
return false
}
if c, err := delivery.DecodeHave(current); err == nil && c.Phase == delivery.PhaseRemoved {
@@ -302,6 +447,7 @@ func (q *Queue) writeRemoved(ctx context.Context, b HaveBucket, node string) boo
}
if err := b.Set(ctx, key, raw, revision); err != nil {
q.sayHave(fmt.Sprintf("could not write %s in the have bucket; the next apply or look writes it again: %v", key, err))
q.haveFailed(err)
return false
}
delete(s.removed, module)
@@ -311,7 +457,7 @@ func (q *Queue) writeRemoved(ctx context.Context, b HaveBucket, node string) boo
// putHave reads a key and writes it unless same says it already holds the value. False when the pass should stop: the
// bucket failed, the error is said once, and the rest wait for the next pass.
func (q *Queue) putHave(ctx context.Context, b HaveBucket, key string, value []byte, same func([]byte) bool) bool {
func (q *Session) putHave(ctx context.Context, b HaveBucket, key string, value []byte, same func([]byte) bool) bool {
if ctx.Err() != nil {
q.sayHave(fmt.Sprintf("the time for writing the have bucket ran out before %s; the next apply or look writes it", key))
return false
@@ -322,13 +468,153 @@ func (q *Queue) putHave(ctx context.Context, b HaveBucket, key string, value []b
revision = 0
case err != nil:
q.sayHave(fmt.Sprintf("could not read %s in the have bucket, and wrote nothing more there: %v", key, err))
q.haveFailed(err)
return false
case same(current):
q.haveWritten()
return true
}
if err := b.Set(ctx, key, value, revision); err != nil {
q.sayHave(fmt.Sprintf("could not write %s in the have bucket; the next apply or look writes it again: %v", key, err))
q.haveFailed(err)
return false
}
q.haveWritten()
return true
}
// haveWritten tells the caller this node-engine has a have key in the bucket saying what it says now: written by it, or
// already holding the same. It is what proves a node-engine on trial to its launcher (novox/hq ADR 0539 §9): the first
// have under its own build, not a declaration's report.
func (q *Session) haveWritten() {
if q.HaveWritten != nil {
q.HaveWritten()
}
}
// NoteModule takes how one module's apply came out and writes its have where it differs from the bucket's (novox/hq ADR
// 0539 §9): the node agent's word for a module, with the fingerprint and the digest it runs. It keeps the module's last
// health, which a look changes without an apply.
func (q *Session) NoteModule(ctx context.Context, f ModuleFact) {
q.init()
s := &q.haves
s.mu.Lock()
if s.modules == nil {
s.modules = map[string]ModuleFact{}
}
// A change under way, or a fact naming another composition than the one looked at, leaves the module's health not
// yet judged until a look after it: the last look judged what ran before (2026-10-25, the canary of the controller on
// novox held on a have saying applying and failed two seconds before its launch said healthy).
s.moved(f)
s.modules[f.Module] = f
delete(s.removed, f.Module)
// Out of motion and not judged: a look is owed now, not at the loop's next statement (review of fix 15, B1).
owe := !inMotion(f.Phase) && s.unjudged[f.Module]
s.mu.Unlock()
if owe {
q.oweHealth()
}
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
// NoteUserList takes what the bus node's engine wrote of the bus's user list, and writes it in the bus module's have
// where it differs from the bucket's (novox/hq ADR 0547 §8). A module this engine has said no have of yet carries it
// from its first have on.
func (q *Session) NoteUserList(ctx context.Context, module string, w delivery.UserListWritten) {
q.init()
s := &q.haves
s.mu.Lock()
if s.userList == nil {
s.userList = map[string]delivery.UserListWritten{}
}
s.userList[module] = w
s.mu.Unlock()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
// NoteRemoved takes that a module's own resources were removed, and writes its have with the phase removed once (issue
// 1369): the key is never deleted.
func (q *Session) NoteRemoved(ctx context.Context, module, build string) {
q.init()
s := &q.haves
s.mu.Lock()
if s.removed == nil {
s.removed = map[string]string{}
}
if f, ok := s.modules[module]; ok && build == "" {
build = f.Build
}
s.removed[module] = build
delete(s.modules, module)
delete(s.written, module)
delete(s.unjudged, module)
s.mu.Unlock()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
// NoteNode takes the digest of the node's own composition it runs, and writes the node's key where it differs.
func (q *Session) NoteNode(ctx context.Context, composed string) {
q.init()
s := &q.haves
s.mu.Lock()
s.nodeComposed = composed
if s.node == nil {
n := delivery.HaveNode{V: delivery.Version}
s.node = &n
}
s.mu.Unlock()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
// ReadHave is the have this node-engine last wrote for a module, and whether the bucket holds one. A bucket that cannot
// be read answers as none: the agent then compares with the declaration it kept, and applies only what differs.
func (q *Session) ReadHave(ctx context.Context, module string) (delivery.Have, bool) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
hb, ok := bus.(HaveBus)
if !ok || bus == nil {
return delivery.Have{}, false
}
key, err := delivery.HaveKey(q.Membership.Node, module)
if err != nil {
return delivery.Have{}, false
}
read, cancel := context.WithTimeout(ctx, q.timeout())
defer cancel()
bucket, err := hb.HaveBind(read)
if err != nil {
return delivery.Have{}, false
}
value, _, err := bucket.Get(read, key)
if err != nil {
return delivery.Have{}, false
}
h, err := delivery.DecodeHave(value)
if err != nil {
return delivery.Have{}, false
}
return h, true
}
// nothingJudged is the health of a module a look judged no resource of: ok when it was looked at and the module applied
// and runs (nothing of it could be down), unknown otherwise.
func nothingJudged(f ModuleFact, looked bool) delivery.Health {
if looked && !f.Failed && (f.Phase == "" || f.Phase == string(delivery.PhaseRunning)) {
return delivery.HealthOK
}
return delivery.HealthUnknown
}
+15 -22
View File
@@ -4,7 +4,6 @@ package link
import (
"context"
"encoding/json"
"sync/atomic"
"testing"
"time"
@@ -32,26 +31,20 @@ func TestNatsHavesAreWrittenOnceAndOnlyOnChange(t *testing.T) {
_ = kv.Purge(node + "." + name)
}
m, key := aMember(t)
m, _ := aMember(t)
m.Node = node
var failB atomic.Bool
var applied atomic.Int32
failB.Store(true)
apply := func(_ context.Context, raw, _ []byte) Report {
applied.Add(1)
var d struct{ Order }
_ = json.Unmarshal(raw, &d)
return Report{Declared: digest(raw), Order: d.Order, Applied: []string{"a.x", "b.x"},
facts := func() Report {
return Report{Applied: []string{"a.x", "b.x"},
Modules: []ModuleFact{{Module: "a", Build: "b-1"},
{Module: "b", Build: "b-2", Failed: failB.Load(), Detail: "b.x failed: no space"}},
Health: healthOf(map[string]string{"a": StateHealthy, "b": StateHealthy}),
Windows: []Window{{Step: "backup", Holds: []string{"db"}}}}
}
q := aQueue(m, apply, &keptInMemory{})
q.attach(context.Background(), OverNATS{Conn: conn, JS: js})
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
defer func() { stop(); <-worker }()
q := &Session{Membership: m, Timeout: 2 * time.Second}
q.Attach(context.Background(), OverNATS{Conn: conn, JS: js})
ctx := context.Background()
revisions := func() map[string]uint64 {
out := map[string]uint64{}
@@ -62,14 +55,14 @@ func TestNatsHavesAreWrittenOnceAndOnlyOnChange(t *testing.T) {
}
return out
}
arrives := func(what string, id string, seq int64) {
want := applied.Load() + 1
q.Deliver(ordered(t, key, id, 7, seq))
eventually(t, what, func() bool { return applied.Load() >= want })
arrives := func(what string) {
if !q.SayFacts(ctx, facts()) {
t.Fatal(what + ": the facts were not taken")
}
time.Sleep(300 * time.Millisecond)
}
arrives("the first apply", "one", 1)
eventually(t, "the three keys were never written", func() bool { return len(revisions()) == 3 })
arrives("the first facts")
eventuallyHeld(t, "the three keys were never written", func() bool { return len(revisions()) == 3 })
first := revisions()
a, _ := kv.Get(node + ".a")
@@ -87,7 +80,7 @@ func TestNatsHavesAreWrittenOnceAndOnlyOnChange(t *testing.T) {
t.Fatalf("the node key holds %+v (%v)", hn, err)
}
arrives("the second apply", "two", 2)
arrives("the second facts")
time.Sleep(200 * time.Millisecond)
for name, rev := range revisions() {
if rev != first[name] {
@@ -96,8 +89,8 @@ func TestNatsHavesAreWrittenOnceAndOnlyOnChange(t *testing.T) {
}
failB.Store(false)
arrives("the third apply", "three", 3)
eventually(t, "b was never updated", func() bool { return revisions()["b"] > first["b"] })
arrives("the third facts")
eventuallyHeld(t, "b was never updated", func() bool { return revisions()["b"] > first["b"] })
if now := revisions(); now["a"] != first["a"] || now["node"] != first["node"] {
t.Errorf("a key that did not change moved: %v from %v", now, first)
}
+266 -54
View File
@@ -4,12 +4,14 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
)
// What the node-engine writes beside its report: a module's have, and the node's own facts (novox/hq ADR 0429 §4,
@@ -26,6 +28,8 @@ type haveLink struct {
binds int
setTries int
bindFails, bindStuck bool
// bindNoResponders fails that many binds with the bus's no responders, as while it reloads its accounts.
bindNoResponders int
}
type haveEntry struct {
@@ -51,7 +55,14 @@ func (l *haveLink) HaveBind(ctx context.Context) (HaveBucket, error) {
l.hmu.Lock()
l.binds++
stuck, refuse := l.bindStuck, l.bindFails
reloading := l.bindNoResponders > 0
if reloading {
l.bindNoResponders--
}
l.hmu.Unlock()
if reloading {
return nil, fmt.Errorf("binding node-engine_have: %w", nats.ErrNoResponders)
}
if stuck {
<-ctx.Done()
return nil, ctx.Err()
@@ -126,20 +137,20 @@ func healthOf(states map[string]string) *Health {
}
func appliedModules() Report {
return Report{Declared: "d1", Applied: []string{"a.x", "b.x"},
return Report{Applied: []string{"a.x", "b.x"},
Modules: []ModuleFact{{Module: "a", Build: "b-1"}, {Module: "b", Build: "b-2", Failed: true, Detail: "b.x failed: no space"}},
Health: healthOf(map[string]string{"a": StateHealthy, "b": StateUnhealthy})}
}
func aHaveQueue(l *haveLink) (*Queue, *[]string) {
func aHaveQueue(l *haveLink) (*Session, *[]string) {
var said []string
var mu sync.Mutex
q := &Queue{Membership: Membership{Node: "n1"}, Timeout: time.Second, Say: func(s string) {
q := &Session{Membership: Membership{Node: "n1"}, Timeout: time.Second, Say: func(s string) {
mu.Lock()
said = append(said, s)
mu.Unlock()
}}
q.attach(context.Background(), l)
q.Attach(context.Background(), l)
return q, &said
}
@@ -148,7 +159,7 @@ func aHaveQueue(l *haveLink) (*Queue, *[]string) {
func TestAnAppliedModuleIsWrittenRunningAndAFailedOneFailed(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
q.SayFacts(context.Background(), appliedModules())
a, b := l.have(t, "n1.a"), l.have(t, "n1.b")
if a.Phase != delivery.PhaseRunning || a.Build != "b-1" || a.Health != delivery.HealthOK {
t.Errorf("a: %+v", a)
@@ -168,7 +179,7 @@ func TestAHaveIsWrittenOnlyWhenItDiffers(t *testing.T) {
l.put(t, "n1.a", delivery.Have{V: delivery.Version, Build: "b-1", Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
l.put(t, "n1.b", delivery.Have{V: delivery.Version, Build: "b-0", Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
q, _ := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
q.SayFacts(context.Background(), appliedModules())
if l.kv["n1.a"].rev != 1 {
t.Errorf("a equal to the bucket was written again, to revision %d", l.kv["n1.a"].rev)
}
@@ -176,7 +187,7 @@ func TestAHaveIsWrittenOnlyWhenItDiffers(t *testing.T) {
t.Errorf("b was not updated over its last revision: %+v at %d", l.have(t, "n1.b"), l.kv["n1.b"].rev)
}
before := l.setCount()
q.tell(context.Background(), appliedModules(), applied)
q.SayFacts(context.Background(), appliedModules())
if l.setCount() != before {
t.Errorf("an apply that changed nothing wrote %d keys", l.setCount()-before)
}
@@ -186,9 +197,9 @@ func TestAHaveIsWrittenOnlyWhenItDiffers(t *testing.T) {
func TestAModulesHealthFollowsTheLook(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
rep := Report{Declared: "d1", Modules: []ModuleFact{{Module: "a", Build: "b"}, {Module: "b", Build: "b"},
rep := Report{Modules: []ModuleFact{{Module: "a", Build: "b"}, {Module: "b", Build: "b"},
{Module: "c", Build: "b"}, {Module: "d", Build: "b"}}}
q.tell(context.Background(), rep, applied)
q.SayFacts(context.Background(), rep)
for _, m := range []string{"a", "b", "c", "d"} {
if h := l.have(t, "n1."+m).Health; h != delivery.HealthUnknown {
t.Errorf("%s with no look is %s, want unknown", m, h)
@@ -230,7 +241,7 @@ func TestTheReportIsSentFirstAndTheNodeKeyCarriesTheNodesFacts(t *testing.T) {
rep.Strays = []Stray{{Kind: "container", Name: "old"}}
rep.Filters = []Filter{{Where: "inet mesh", Owner: "mesh"}}
rep.Profile = map[string]any{"cores": 8}
q.tell(context.Background(), rep, applied)
q.SayFacts(context.Background(), rep)
if len(l.events) < 2 || l.events[0] != "report" {
t.Fatalf("the report was not first: %v", l.events)
}
@@ -255,7 +266,7 @@ func TestAFailedHaveWriteIsSaidAndNeverBlocksTheReport(t *testing.T) {
l := newHaveLink()
l.failing = true
q, said := aHaveQueue(l)
if !q.tell(context.Background(), appliedModules(), applied) {
if !q.SayFacts(context.Background(), appliedModules()) {
t.Fatal("the report did not go out")
}
if len(l.said()) != 1 {
@@ -267,7 +278,7 @@ func TestAFailedHaveWriteIsSaidAndNeverBlocksTheReport(t *testing.T) {
l.hmu.Lock()
l.failing = false
l.hmu.Unlock()
q.tell(context.Background(), appliedModules(), applied)
q.SayFacts(context.Background(), appliedModules())
if l.have(t, "n1.a").Phase != delivery.PhaseRunning {
t.Error("the next apply did not write again")
}
@@ -276,39 +287,19 @@ func TestAFailedHaveWriteIsSaidAndNeverBlocksTheReport(t *testing.T) {
// A link without a bucket, and a report with no module facts (a refusal), write nothing and say nothing wrong.
func TestNoBucketAndNoFactsWriteNothing(t *testing.T) {
plain := newQuietLink()
q := &Queue{Membership: Membership{Node: "n1"}, Timeout: time.Second}
q.attach(context.Background(), plain)
if !q.tell(context.Background(), appliedModules(), applied) {
q := &Session{Membership: Membership{Node: "n1"}, Timeout: time.Second}
q.Attach(context.Background(), plain)
if !q.SayFacts(context.Background(), appliedModules()) {
t.Fatal("the report did not go out")
}
l := newHaveLink()
q2, _ := aHaveQueue(l)
q2.tell(context.Background(), Report{Refused: "no"}, applied)
q2.SayFacts(context.Background(), Report{Refused: "no"})
if l.setCount() != 0 {
t.Errorf("a refusal wrote %d keys", l.setCount())
}
}
// 1132/AC5: a reconcile whose report is not news still writes the have keys that differ, so a node-engine that
// restarted writes them again; and writes nothing when they are the same.
func TestASilentReconcileStillWritesTheHaves(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
q.Reconcile = func(context.Context) (Report, bool) { return appliedModules(), true }
q.reconcile(context.Background(), false)
if len(l.said()) != 0 {
t.Fatalf("a reconcile that is not news said %d reports", len(l.said()))
}
if l.have(t, "n1.a").Phase != delivery.PhaseRunning {
t.Error("the reconcile wrote no have")
}
before := l.setCount()
q.reconcile(context.Background(), false)
if l.setCount() != before {
t.Errorf("an unchanged reconcile wrote %d keys", l.setCount()-before)
}
}
// 1132/AC7: a bind that fails, or never answers, writes nothing and returns within the queue's timeout; a write that
// fails ends the pass, and is said once.
func TestAFailedBindOrWriteEndsThePassAndReturnsInTime(t *testing.T) {
@@ -321,7 +312,7 @@ func TestAFailedBindOrWriteEndsThePassAndReturnsInTime(t *testing.T) {
q, said := aHaveQueue(l)
q.Timeout = 100 * time.Millisecond
start := time.Now()
if !q.tell(context.Background(), appliedModules(), applied) {
if !q.SayFacts(context.Background(), appliedModules()) {
t.Fatalf("%s: the report did not go out", name)
}
if took := time.Since(start); took > time.Second {
@@ -334,14 +325,32 @@ func TestAFailedBindOrWriteEndsThePassAndReturnsInTime(t *testing.T) {
l := newHaveLink()
l.failing = true
q, said := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
q.SayFacts(context.Background(), appliedModules())
if l.binds != 1 || l.setTries != 1 || len(*said) != 1 {
t.Errorf("a failed write: binds %d, writes tried %d, said %v; want one of each and one line", l.binds, l.setTries, *said)
}
// A bind that meets no responders while the bus reloads its accounts (2026-10-25 23:04:24, four in one second on the
// bus node) is written again after a backoff, with no apply or look to come.
defer func(was func(int) time.Duration) { haveBackoff = was }(haveBackoff)
haveBackoff = func(int) time.Duration { return time.Millisecond }
l = newHaveLink()
l.bindNoResponders = 1
q, _ = aHaveQueue(l)
q.SayFacts(context.Background(), appliedModules())
deadline := time.Now().Add(3 * time.Second)
for l.setCount() == 0 && time.Now().Before(deadline) {
time.Sleep(5 * time.Millisecond)
}
if l.setCount() == 0 {
t.Fatal("a bind that met no responders was not written again")
}
if l.have(t, "n1.a").Phase != delivery.PhaseRunning {
t.Error("the have written again does not say running")
}
}
func onlyA() Report {
return Report{Declared: "d2", Applied: []string{"a.x"}, Modules: []ModuleFact{{Module: "a", Build: "b-1"}},
return Report{Applied: []string{"a.x"}, Modules: []ModuleFact{{Module: "a", Build: "b-1"}},
Health: healthOf(map[string]string{"a": StateHealthy})}
}
@@ -350,9 +359,9 @@ func onlyA() Report {
func TestAModuleLeavingTheDeclarationIsWrittenRemoved(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
q.SayFacts(context.Background(), appliedModules())
revA, revB := l.kv["n1.a"].rev, l.kv["n1.b"].rev
q.tell(context.Background(), onlyA(), applied)
q.SayFacts(context.Background(), onlyA())
b := l.have(t, "n1.b")
if b.Phase != delivery.PhaseRemoved || b.Build != "b-2" {
t.Errorf("b: %+v", b)
@@ -364,7 +373,7 @@ func TestAModuleLeavingTheDeclarationIsWrittenRemoved(t *testing.T) {
t.Errorf("a changed: %+v at %d", l.have(t, "n1.a"), l.kv["n1.a"].rev)
}
before := l.setCount()
q.tell(context.Background(), onlyA(), applied)
q.SayFacts(context.Background(), onlyA())
if l.setCount() != before {
t.Errorf("a second apply without b wrote %d keys, removed is written once", l.setCount()-before)
}
@@ -374,13 +383,13 @@ func TestAModuleLeavingTheDeclarationIsWrittenRemoved(t *testing.T) {
func TestARemovedModuleAddedBackIsWrittenWithItsNewPhase(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
q.tell(context.Background(), onlyA(), applied)
q.tell(context.Background(), appliedModules(), applied)
q.SayFacts(context.Background(), appliedModules())
q.SayFacts(context.Background(), onlyA())
q.SayFacts(context.Background(), appliedModules())
if b := l.have(t, "n1.b"); b.Phase != delivery.PhaseFailed || b.Build != "b-2" {
t.Errorf("b after it came back: %+v", b)
}
q.tell(context.Background(), onlyA(), applied)
q.SayFacts(context.Background(), onlyA())
if b := l.have(t, "n1.b"); b.Phase != delivery.PhaseRemoved {
t.Errorf("b removed again: %+v", b)
}
@@ -392,19 +401,19 @@ func TestAModuleNeverWrittenIsNotWrittenRemoved(t *testing.T) {
l := newHaveLink()
l.put(t, "n1.c", delivery.Have{V: delivery.Version, Build: "b-9", Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
q, _ := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
q.tell(context.Background(), onlyA(), applied)
q.SayFacts(context.Background(), appliedModules())
q.SayFacts(context.Background(), onlyA())
if c := l.have(t, "n1.c"); c.Phase != delivery.PhaseRunning || l.kv["n1.c"].rev != 1 {
t.Errorf("c, which this node-engine never wrote, changed: %+v at %d", c, l.kv["n1.c"].rev)
}
// a key that vanished from the bucket is not created by the removal
l2 := newHaveLink()
q2, _ := aHaveQueue(l2)
q2.tell(context.Background(), appliedModules(), applied)
q2.SayFacts(context.Background(), appliedModules())
l2.hmu.Lock()
delete(l2.kv, "n1.b")
l2.hmu.Unlock()
q2.tell(context.Background(), onlyA(), applied)
q2.SayFacts(context.Background(), onlyA())
if _, there := l2.kv["n1.b"]; there {
t.Error("the removal created a key that was not there")
}
@@ -414,15 +423,15 @@ func TestAModuleNeverWrittenIsNotWrittenRemoved(t *testing.T) {
func TestARefusedRemovalIsWrittenByTheNextPass(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
q.SayFacts(context.Background(), appliedModules())
l.hmu.Lock()
l.failing = true
l.hmu.Unlock()
q.tell(context.Background(), onlyA(), applied)
q.SayFacts(context.Background(), onlyA())
l.hmu.Lock()
l.failing = false
l.hmu.Unlock()
q.tell(context.Background(), onlyA(), applied)
q.SayFacts(context.Background(), onlyA())
if b := l.have(t, "n1.b"); b.Phase != delivery.PhaseRemoved {
t.Errorf("b: %+v", b)
}
@@ -437,7 +446,7 @@ func TestARestartedNodeEngineWritesTheRemovalFromTheReportsBefore(t *testing.T)
q, _ := aHaveQueue(l) // fresh: nothing written by this process
r := onlyA()
r.Before = []string{"a", "b", "c"}
q.tell(context.Background(), r, applied)
q.SayFacts(context.Background(), r)
if b := l.have(t, "n1.b"); b.Phase != delivery.PhaseRemoved {
t.Errorf("b: %+v", b)
}
@@ -448,3 +457,206 @@ func TestARestartedNodeEngineWritesTheRemovalFromTheReportsBefore(t *testing.T)
t.Errorf("a was written, revision %d", l.kv["n1.a"].rev)
}
}
// R6-A AC7: the first have this node-engine writes — or finds already saying what it would write — is told to the
// proof, which proves a node-engine on trial to its launcher (novox/hq ADR 0539 §9); a write the bucket refuses is not.
func TestTheFirstHaveWrittenIsToldToTheProof(t *testing.T) {
l := newHaveLink()
l.failing = true
q, _ := aHaveQueue(l)
var told int
q.HaveWritten = func() { told++ }
q.SayFacts(context.Background(), appliedModules())
if told != 0 {
t.Fatalf("a refused write proved the build %d time(s)", told)
}
l.hmu.Lock()
l.failing = false
l.hmu.Unlock()
q.SayFacts(context.Background(), appliedModules())
if told == 0 {
t.Fatal("a have written under this build was not told")
}
// The bucket already holds what this node-engine would write: a restart with nothing changed still proves it.
same := newHaveLink()
q2, _ := aHaveQueue(same)
q2.SayFacts(context.Background(), appliedModules())
again, _ := aHaveQueue(same)
told = 0
again.HaveWritten = func() { told++ }
again.SayFacts(context.Background(), appliedModules())
if told == 0 {
t.Fatal("a have already saying the same was not told")
}
}
// R6-A AC6: an applied module's have names its fingerprint and the digest of the composition applied; a health check that
// fails after the module's start window gives health failed, and the have keeps both.
func TestAnAppliedModulesHaveNamesTheFingerprintAndTheDigestAndFollowsItsHealth(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
fp := "src2:" + strings.Repeat("a", 64)
dig := "sha256:" + strings.Repeat("b", 64)
q.NoteModule(context.Background(), ModuleFact{Module: "web", FP: fp, Composed: dig})
h := l.have(t, "n1.web")
if h.FP != fp || h.Composed != dig || h.Phase != delivery.PhaseRunning || h.Health != delivery.HealthUnknown {
t.Fatalf("the have after the apply: %+v", h)
}
// Starting: it waits. Past its start window the check fails: failed.
q.SayHealth(context.Background(), *healthOf(map[string]string{"web": StateStarting}))
if got := l.have(t, "n1.web"); got.Health != delivery.HealthWaits || got.FP != fp || got.Composed != dig {
t.Fatalf("while it starts: %+v", got)
}
q.SayHealth(context.Background(), *healthOf(map[string]string{"web": StateUnhealthy}))
if got := l.have(t, "n1.web"); got.Health != delivery.HealthFailed || got.FP != fp || got.Composed != dig {
t.Fatalf("a failing check after the start window: %+v", got)
}
// The bus's user list written alone rides on the module's have, and an apply after it keeps it (ADR 0547 §8).
w := delivery.UserListWritten{Source: "n1.web." + fp, Revision: 4, Wanted: dig, Digest: dig, Users: 3,
At: time.Date(2026, 10, 25, 21, 0, 0, 0, time.UTC)}
q.NoteUserList(context.Background(), "web", w)
if got := l.have(t, "n1.web"); got.UserList == nil || got.UserList.Revision != 4 || got.FP != fp {
t.Fatalf("the user list is not in the have: %+v", got)
}
q.NoteModule(context.Background(), ModuleFact{Module: "web", FP: fp, Composed: dig})
if got := l.have(t, "n1.web"); got.UserList == nil || got.UserList.Users != 3 {
t.Fatalf("an apply lost the user list: %+v", got)
}
// A module held for a failed backup says so and what it keeps running; removed is written once, over its key.
q.NoteModule(context.Background(), ModuleFact{Module: "web", FP: fp, Composed: dig, Phase: string(delivery.PhaseHeld), Detail: "the repository is locked"})
if got := l.have(t, "n1.web"); got.Phase != delivery.PhaseHeld || got.Detail == "" {
t.Fatalf("held: %+v", got)
}
q.NoteRemoved(context.Background(), "web", "")
if got := l.have(t, "n1.web"); got.Phase != delivery.PhaseRemoved {
t.Fatalf("removed: %+v", got)
}
q.NoteNode(context.Background(), dig)
if n, err := delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || n.Composed != dig {
t.Fatalf("the node key: %+v %v", n, err)
}
}
// A module applied and running that has nothing long-running to judge is ok once the engine has looked: there is nothing
// of it that could be down. Unknown kept it from ever being healthy, so the assigner's canary rule held every other node
// of mesh-host back for good (issue 1945, live on 2026-10-25: ace's have of mesh-host said unknown, and novox and shanks
// never got the new node-engine). A failed one stays failed, and before any look it is unknown.
func TestAModuleWithNothingToJudgeIsOkOnceLookedAt(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
q.SayFacts(context.Background(), Report{Modules: []ModuleFact{{Module: "mesh-host", Build: "b"}, {Module: "web", Build: "b"},
{Module: "broken", Build: "b", Failed: true, Detail: "x"}}})
if h := l.have(t, "n1.mesh-host").Health; h != delivery.HealthUnknown {
t.Fatalf("before any look: %s", h)
}
if !q.SayHealth(context.Background(), *healthOf(map[string]string{"web": StateHealthy})) {
t.Fatal("not said")
}
if h := l.have(t, "n1.mesh-host").Health; h != delivery.HealthOK {
t.Fatalf("a running module with nothing to judge after a look: %s", h)
}
if h := l.have(t, "n1.broken").Health; h == delivery.HealthOK {
t.Fatalf("a failed module was said ok")
}
}
// A have an older engine left unknown is written over with this engine's health once the module is said again, as the
// node agent says a module it finds running on its own have: a packages-only module (xorg, nothing long-running) is ok
// after a look, so the assigner no longer holds back what requires its provision (2026-10-25, g14 and shanks).
func TestAModuleSaidAgainOverAnOlderEnginesUnknownIsWrittenWithThisEnginesHealth(t *testing.T) {
l := newHaveLink()
fp := "src2:" + strings.Repeat("a", 64)
dig := "sha256:" + strings.Repeat("b", 64)
l.put(t, "n1.xorg", delivery.Have{V: delivery.Version, FP: fp, Composed: dig, Phase: delivery.PhaseRunning,
Health: delivery.HealthUnknown})
q, _ := aHaveQueue(l)
if !q.SayHealth(context.Background(), *healthOf(map[string]string{"web": StateHealthy})) {
t.Fatal("not said")
}
if h := l.have(t, "n1.xorg"); h.Health != delivery.HealthUnknown {
t.Fatalf("a module the session does not hold was written: %+v", h)
}
q.NoteModule(context.Background(), ModuleFact{Module: "xorg", FP: fp, Composed: dig})
if h := l.have(t, "n1.xorg"); h.Health != delivery.HealthOK || h.FP != fp || h.Composed != dig || h.Phase != delivery.PhaseRunning {
t.Fatalf("said again after a look: %+v", h)
}
}
// A module's health is not yet judged (unknown) while its change is under way and until a look after it, never failed
// from a look at what was being replaced: on 2026-10-25 15:27:36 the controller's canary on novox was held on a have
// saying applying and failed, two seconds before its launch said healthy.
func TestAHealthIsNotJudgedWhileAModuleIsApplied(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
ctx := context.Background()
old := ModuleFact{Module: "ctl", FP: "src2:" + strings.Repeat("a", 64), Composed: "sha256:" + strings.Repeat("1", 64)}
q.NoteModule(ctx, old)
q.SayHealth(ctx, *healthOf(map[string]string{"ctl": StateUnhealthy}))
if h := l.have(t, "n1.ctl"); h.Health != delivery.HealthFailed {
t.Fatalf("the running module's failed look was not said: %+v", h)
}
next := ModuleFact{Module: "ctl", FP: "src2:" + strings.Repeat("b", 64), Composed: "sha256:" + strings.Repeat("2", 64)}
applying := next
applying.Phase = string(delivery.PhaseApplying)
q.NoteModule(ctx, applying)
if h := l.have(t, "n1.ctl"); h.Phase != delivery.PhaseApplying || h.Health != delivery.HealthUnknown {
t.Fatalf("applying carried the last look's health: %+v", h)
}
q.SayHealth(ctx, *healthOf(map[string]string{"ctl": StateUnhealthy}))
if h := l.have(t, "n1.ctl"); h.Health != delivery.HealthUnknown {
t.Fatalf("a look during the apply judged it: %+v", h)
}
q.NoteModule(ctx, next)
if h := l.have(t, "n1.ctl"); h.Phase != delivery.PhaseRunning || h.Health != delivery.HealthUnknown {
t.Fatalf("running before a look after the apply: %+v", h)
}
q.SayHealth(ctx, *healthOf(map[string]string{"ctl": StateHealthy}))
if h := l.have(t, "n1.ctl"); h.Health != delivery.HealthOK {
t.Fatalf("the look after the apply was not taken: %+v", h)
}
}
// A fact naming another composition than the one looked at, with no apply phase between, is not judged by the old look.
func TestAHealthIsNotCarriedOverToAnotherComposition(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
ctx := context.Background()
q.NoteModule(ctx, ModuleFact{Module: "ctl", FP: "src2:" + strings.Repeat("a", 64)})
q.SayHealth(ctx, *healthOf(map[string]string{"ctl": StateUnhealthy}))
q.NoteModule(ctx, ModuleFact{Module: "ctl", FP: "src2:" + strings.Repeat("b", 64)})
if h := l.have(t, "n1.ctl"); h.Health != delivery.HealthUnknown {
t.Fatalf("the old composition's look judged the new one: %+v", h)
}
}
// A declaration's report naming another composition of a module, with no health of its own, does not carry the old
// look's health over; and a module removed leaves nothing marked unjudged behind.
func TestADeclarationReportMovesAHealthAndARemovalForgetsIt(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
ctx := context.Background()
q.SayFacts(ctx, Report{Applied: []string{"a.x"}, Modules: []ModuleFact{{Module: "a", Build: "b-1"}},
Health: healthOf(map[string]string{"a": StateUnhealthy})})
if h := l.have(t, "n1.a"); h.Health != delivery.HealthFailed {
t.Fatalf("the failed look was not said: %+v", h)
}
q.SayFacts(ctx, Report{Applied: []string{"a.x"}, Modules: []ModuleFact{{Module: "a", Build: "b-1", FP: "src2:" + strings.Repeat("c", 64)}}})
if h := l.have(t, "n1.a"); h.Health != delivery.HealthUnknown {
t.Fatalf("the old composition's look judged the new one: %+v", h)
}
select {
case <-q.HealthOwed():
default:
t.Fatal("a module out of motion and unjudged owed no look")
}
q.NoteRemoved(ctx, "a", "")
q.haves.mu.Lock()
left := q.haves.unjudged["a"]
q.haves.mu.Unlock()
if left {
t.Fatal("a removed module is still marked unjudged")
}
}
+8 -8
View File
@@ -23,14 +23,14 @@ func (l *sayingLink) Health(_ context.Context, node string, body []byte) error {
}
func TestAHealthStatementIsSaidOnTheLinkOpenNow(t *testing.T) {
q := &Queue{Membership: Membership{Node: "anchor"}}
q := &Session{Membership: Membership{Node: "anchor"}}
h := Health{Contract: LivenessContract, At: time.Now().UTC(),
Resources: []ResourceHealth{{Module: "letta", Resource: "letta.server", State: StateUnhealthy, Reason: ReasonRestarting}}}
if q.SayHealth(t.Context(), h) {
t.Fatal("said with no link open")
}
l := &sayingLink{quietLink: newQuietLink()}
q.attach(t.Context(), l)
q.Attach(t.Context(), l)
if !q.SayHealth(t.Context(), h) || len(l.said) != 1 {
t.Fatalf("not said on the open link: %d", len(l.said))
}
@@ -42,8 +42,8 @@ func TestAHealthStatementIsSaidOnTheLinkOpenNow(t *testing.T) {
t.Fatalf("said %+v", got)
}
// A link that cannot say it is not an error: the next report carries the statement.
q.detach(l)
q.attach(t.Context(), newQuietLink())
q.Detach(l)
q.Attach(t.Context(), newQuietLink())
if q.SayHealth(t.Context(), h) {
t.Fatal("said on a link that cannot")
}
@@ -77,22 +77,22 @@ func TestAToolsAnswerIsHealthyOnlyWhenItSaysSo(t *testing.T) {
// the link was up, because the NATS link could not Ask). With no link open, and with a link that cannot
// ask, the check fails and says which.
func TestAToolCheckIsAskedOnTheLinkTheEngineHolds(t *testing.T) {
q := &Queue{Membership: Membership{Node: "anchor"}}
q := &Session{Membership: Membership{Node: "anchor"}}
if _, err := q.AskTool(t.Context(), "sensors", "sensors_health"); err == nil || err.Error() != "no link to the bus is open" {
t.Fatalf("with no link open: %v", err)
}
quiet := newQuietLink()
q.attach(t.Context(), quiet)
q.Attach(t.Context(), quiet)
if _, err := q.AskTool(t.Context(), "sensors", "sensors_health"); err == nil || err.Error() != "the link to the bus open now cannot ask a tool" {
t.Fatalf("on a link that cannot ask: %v", err)
}
q.detach(quiet)
q.Detach(quiet)
// The link Open returns, with no connection under it: the question must reach its Ask (and fail
// there), never stop at the queue's assertion.
held := &natsLink{node: "anchor"}
q.attach(t.Context(), held)
q.Attach(t.Context(), held)
_, err := q.AskTool(t.Context(), "sensors", "sensors_health")
if err == nil {
t.Fatal("asked on no connection and was answered")
+5 -28
View File
@@ -8,9 +8,8 @@ import (
// What a host hears, as the host's own words for it.
//
// The outbound half is behind `Bus` (bus.go); this is the other half — dialling, and the
// declarations that arrive. The run loop reads its own words for a declaration rather than the
// client library's delivery type, so nothing past this file knows what carried it.
// The outbound half is behind `Bus` (bus.go); this is the other half — dialling, and what the controller asks of this
// node. A node hears no declaration: it follows its wants (novox/hq ADR 0539).
//
// **The host still imports nothing of the mesh's own** (novox/hq ADR 0005). This is its own
// interface over its own libraries, and it agrees with the controller only because a conformance
@@ -21,12 +20,9 @@ import (
// One interface rather than two, because dialling once is what keeps both halves of a node on the
// same connection.
type Link interface {
// Bus is what this node says: what it applied, and that it is here.
// Bus is what this node says: its facts, and that it is here.
Bus
// Declarations is what the mesh tells this node to be.
Declarations() <-chan Declaration
// Lost says the link ended, and why.
//
// **Read rather than discovered.** A node that finds out by noticing silence is a node that
@@ -38,31 +34,12 @@ type Link interface {
Close()
}
// Asked is a link that hears the mesh asking this node to say again what it last applied (novox/hq
// to-be 45 §6, the `report` verb). Optional: a link without it is never asked, and the mesh then sends
// the current declaration again instead.
// Asked is a link that hears the mesh asking this node to say its facts again. Optional: a link without it is never
// asked.
type Asked interface {
AskedToReport() <-chan struct{}
}
// Declaration is one thing the mesh told this node to be.
//
// **Handled, once — after the report is published.** A node that dies between applying and
// reporting leaves the declaration with the mesh and applies it again on return, which is safe
// because applying is reconciliation: it converges rather than repeating.
//
// There is one way of being done rather than two. A declaration set aside because a newer arrived
// with it is settled exactly as an applied one is, and the difference between them is a fact the
// *report* carries — a second method here would be a distinction the bus does not make.
type Declaration interface {
// Body is the signed declaration as it arrived, bytes unchanged: a node verifies what it
// received rather than what it re-encoded.
Body() []byte
// Handled settles it. Called after the report for it has been published, either way.
Handled() error
}
// Open opens this node's link to its mesh.
//
// Named Open rather than Dial because Dial is this package's raw TLS dial, which the enrolment path
+10 -86
View File
@@ -11,37 +11,16 @@ import (
"github.com/nats-io/nats.go"
)
// The host's link on the bus being built.
//
// Two things a host does here that it cannot do on the other bus, and one it must not try.
//
// **It declares nothing.** On the bus the mesh has, a host declares its own queue on connecting,
// because a queue that is not there means a node that hears nothing. Here the object it reads
// through is a durable consumer, and a host's account reaches no part of the JetStream API — by
// design, because the controller is the only writer of consumer definitions (design 25 §3). So the
// host **binds** to a consumer the controller made when this node enrolled, and a missing one is
// said as what it is rather than quietly created with whatever configuration this client happens to
// default to.
//
// **It gets order for free, and keeps the drain anyway.** The declaration subject is last-per-subject
// (design 29 §4), so a node that was away receives exactly the current declaration rather than a
// queue of superseded ones, and the stream's sequence orders them definitively — the wire-level
// answer to novox/hq issue 107. What the drain in run.go still answers is the live case: three
// pushes to a *connected* node are three deliveries whatever the stream later retains.
// The host's link on the bus: one connection, and what is asked of this node over it. The node hears no declaration
// (novox/hq ADR 0539): it watches its wants and its compositions, and says its facts.
// EnrolSubject is where a joining machine asks. One subject for every node, because a machine
// enrolling has no name the mesh has agreed to yet — which is why its authority to publish here is
// the whole of what its enrolment user may do.
const EnrolSubject = "mesh.control.enrol"
// DeclareSubject is where this node's declaration lands. Its own, and no other node's: a host's
// account subscribes exactly this and the subject is the authority on which node a declaration is
// for.
func DeclareSubject(node string) string { return "mesh.node." + node + ".declare" }
// AskReportSubject is where the mesh asks this node to say again what it last applied (novox/hq to-be 45
// §6): its own, on core NATS and off any stream. The answer is an ordinary report on its report subject,
// the one thing it already says — nobody's inbox is answered.
// AskReportSubject is where the mesh asks this node to say its facts again: its own, on core NATS and off any stream.
// The answer is the facts event on its report subject — nobody's inbox is answered.
func AskReportSubject(node string) string { return "mesh.node." + node + ".ask.report" }
// natsURL is a bus address as the client wants it. A membership records host and port, because that
@@ -58,7 +37,6 @@ func natsURL(address string) string {
type natsLink struct {
conn *nats.Conn
js nats.JetStreamContext
sub *nats.Subscription
asking *nats.Subscription
asked chan struct{}
handing *nats.Subscription
@@ -72,9 +50,8 @@ type natsLink struct {
moving *nats.Subscription
node string
// signer is the mesh's key, which a hand-over ask is verified with as a declaration is (issue 356).
signer ed25519.PublicKey
arrived chan Declaration
lost chan error
signer ed25519.PublicKey
lost chan error
}
func dialNats(ctx context.Context, m Membership, timeout time.Duration) (Link, error) {
@@ -123,27 +100,10 @@ func dialNats(ctx context.Context, m Membership, timeout time.Duration) (Link, e
l := &natsLink{
conn: conn, js: js, node: m.Node, signer: m.Signer,
arrived: make(chan Declaration, drainDepth),
lost: make(chan error, 1),
asked: make(chan struct{}, 1),
lost: make(chan error, 1),
asked: make(chan struct{}, 1),
}
// Bound to the consumer the controller made for this node, named after the node because that is
// what the node's own ack grant allows (`$JS.ACK.NODES.<node>.>`).
feed := make(chan *nats.Msg, drainDepth)
// The subject as well as the binding: the client checks what is asked for against the
// consumer's own filter, and an empty subject is refused rather than taken to mean "whatever
// that consumer delivers".
sub, err := js.ChanSubscribe(DeclareSubject(m.Node), feed, nats.Bind("NODES", m.Node))
if err != nil {
conn.Close()
return nil, fmt.Errorf(
"this node cannot read its declarations: %w. The mesh creates that when a node enrols, "+
"and a host may not create one itself — so this is the mesh's to answer, not this "+
"machine's", err)
}
l.sub = sub
l.hearAsks()
conn.SetDisconnectErrHandler(func(_ *nats.Conn, err error) {
@@ -159,29 +119,6 @@ func dialNats(ctx context.Context, m Membership, timeout time.Duration) (Link, e
}
})
go func() {
defer close(l.arrived)
for {
select {
case <-ctx.Done():
return
case msg, ok := <-feed:
if !ok {
select {
case l.lost <- errors.New("the bus stopped delivering"):
default:
}
return
}
select {
case l.arrived <- natsDeclaration{msg}:
case <-ctx.Done():
return
}
}
}
}()
return l, nil
}
@@ -299,14 +236,10 @@ func (l *natsLink) HearMoveSecrets(handle MoveSecretFunc) {
}
}
func (l *natsLink) Declarations() <-chan Declaration { return l.arrived }
func (l *natsLink) AskedToReport() <-chan struct{} { return l.asked }
func (l *natsLink) Lost() <-chan error { return l.lost }
func (l *natsLink) AskedToReport() <-chan struct{} { return l.asked }
func (l *natsLink) Lost() <-chan error { return l.lost }
func (l *natsLink) Close() {
if l.sub != nil {
_ = l.sub.Unsubscribe()
}
if l.asking != nil {
_ = l.asking.Unsubscribe()
}
@@ -375,12 +308,3 @@ var (
_ Asker = (*natsLink)(nil)
_ Asked = (*natsLink)(nil)
)
// natsDeclaration is one declaration off the NODES stream.
type natsDeclaration struct{ msg *nats.Msg }
func (d natsDeclaration) Body() []byte { return d.msg.Data }
// Handled acknowledges it. The ack goes to this node's own ack subject, which is the one thing
// besides its reports a node's account may publish.
func (d natsDeclaration) Handled() error { return d.msg.Ack() }
-337
View File
@@ -1,337 +0,0 @@
package link
import (
"context"
"crypto/ed25519"
"encoding/json"
"fmt"
"os"
"reflect"
"testing"
"time"
"github.com/nats-io/nats.go"
)
// The host's link against a real server, because every claim here is about one.
//
// Whether binding to a consumer the host did not create works, whether a declaration on the node's
// own subject arrives, whether acknowledging it removes it from the consumer's pending — none of
// that can be reasoned out, and the first two are the ones that would leave a node silently hearing
// nothing:
//
// docker run -d --rm --name t -p 14223:4222 nats:2.10-alpine -js
// MESH_TEST_NATS=nats://127.0.0.1:14223 go test ./internal/link/ -run TestNats
func aBus(t *testing.T) (*nats.Conn, nats.JetStreamContext) {
t.Helper()
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
conn, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
js, err := conn.JetStream()
if err != nil {
t.Fatal(err)
}
// **Ensured and purged, not deleted and recreated.** Delete-then-add looked like a reset and is
// not one: a test that did that inherited the previous test's messages, and the symptom was a
// declaration counted as delivered twice — which reads as a redelivery bug in the code under
// test rather than as a dirty stream. Purge is defined to empty a stream; recreating one is a
// race with the server's own teardown.
for _, want := range []*nats.StreamConfig{
{Name: "NODES", Subjects: []string{"mesh.node.*.declare"}, MaxMsgsPerSubject: 1},
{Name: "CONTROL", Subjects: []string{"mesh.control.*.report", "mesh.control.enrol"},
Retention: nats.WorkQueuePolicy},
} {
if _, err := js.StreamInfo(want.Name); err != nil {
if _, err := js.AddStream(want); err != nil {
t.Fatal(err)
}
}
if err := js.PurgeStream(want.Name); err != nil {
t.Fatal(err)
}
}
return conn, js
}
// theMeshMakes is the consumer the controller creates when a node enrols. Made here by the test
// because the host may not: its account reaches no part of the JetStream API, which is the whole
// reason this binds rather than subscribes.
//
// Removed afterwards, and each test names its own node: two tests sharing a consumer name share its
// delivery count and its pending list, and the first thing that goes wrong reads as a fault in the
// host rather than in the test beside it.
func theMeshMakes(t *testing.T, js nats.JetStreamContext, node string) {
t.Helper()
t.Cleanup(func() { _ = js.DeleteConsumer("NODES", node) })
if _, err := js.AddConsumer("NODES", &nats.ConsumerConfig{
Durable: node,
FilterSubject: DeclareSubject(node),
AckPolicy: nats.AckExplicitPolicy,
AckWait: 300 * time.Second,
DeliverSubject: "_DELIVER." + node,
}); err != nil {
t.Fatal(err)
}
}
func signedBy(t *testing.T, key ed25519.PrivateKey, declaration []byte) []byte {
t.Helper()
body, err := json.Marshal(Signed{
Declaration: declaration, Signature: ed25519.Sign(key, declaration),
})
if err != nil {
t.Fatal(err)
}
return body
}
// A declaration on this node's own subject reaches the host, is applied, and acknowledging it
// empties the consumer — which is what tells the mesh the node has it.
func TestNatsADeclarationReachesTheHostAndIsSettled(t *testing.T) {
conn, js := aBus(t)
const node = "settling"
theMeshMakes(t, js, node)
public, private, _ := ed25519.GenerateKey(nil)
m := Membership{Node: node, Signer: public}
// Dialled directly rather than through Open: the test server has no TLS, and what is being
// checked is the subscription and the settling, not the pin — which PinnedConfig owns and its
// own tests cover.
l := &natsLink{conn: conn, js: js, node: node,
arrived: make(chan Declaration, drainDepth), lost: make(chan error, 1)}
feed := make(chan *nats.Msg, drainDepth)
sub, err := js.ChanSubscribe(DeclareSubject(node), feed, nats.Bind("NODES", node))
if err != nil {
t.Fatalf("the host could not bind to the consumer the mesh made for it: %v", err)
}
defer func() { _ = sub.Unsubscribe() }()
go func() {
for msg := range feed {
l.arrived <- natsDeclaration{msg}
}
}()
if _, err := js.Publish(DeclareSubject(node),
signedBy(t, private, []byte(`{"declared":"d1"}`))); err != nil {
t.Fatal(err)
}
select {
case d := <-l.Declarations():
report := handleBody(context.Background(), m, d.Body(),
func(context.Context, []byte, []byte) Report {
return Report{Applied: []string{"store"}}
})
if report.Refused != "" {
t.Fatalf("a declaration the mesh signed was refused: %s", report.Refused)
}
if err := d.Handled(); err != nil {
t.Fatalf("the node could not acknowledge its own declaration: %v", err)
}
case <-time.After(8 * time.Second):
t.Fatal("no declaration reached the host")
}
// **Nothing pending is the property**; a delivery count is not. Delivery is at-least-once by
// design, so pinning "delivered exactly once" would be asserting something the mesh does not
// rely on. What matters is that the acknowledgement landed, so the mesh can tell the node has
// it — and that no redelivery was needed to get there, which is what would say the node was
// too slow to answer for its own ack wait.
deadline := time.Now().Add(5 * time.Second)
var last string
for time.Now().Before(deadline) {
info, err := js.ConsumerInfo("NODES", node)
switch {
case err != nil:
last = err.Error()
case info.NumAckPending == 0 && info.NumRedelivered == 0:
return
default:
last = fmt.Sprintf("pending %d, redelivered %d", info.NumAckPending, info.NumRedelivered)
}
time.Sleep(20 * time.Millisecond)
}
t.Fatalf("the declaration was not settled, so the mesh cannot tell the node has it: %s", last)
}
// **A node that was away gets exactly the current declaration and nothing older.** Three pushed
// while nothing is listening leave one on the stream, and it is the newest — the wire-level answer
// to novox/hq issue 107, and the half of the drain that stops being the host's problem.
func TestNatsANodeThatWasAwayGetsOnlyTheNewest(t *testing.T) {
_, js := aBus(t)
const node = "returning"
_, private, _ := ed25519.GenerateKey(nil)
for _, id := range []string{"d1", "d2", "d3"} {
if _, err := js.Publish(DeclareSubject(node),
signedBy(t, private, []byte(`{"declared":"`+id+`"}`))); err != nil {
t.Fatal(err)
}
}
info, err := js.StreamInfo("NODES")
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != 1 {
t.Fatalf("%d declarations survived for one node; a node that was away would apply a backlog "+
"of things nobody wants any more", info.State.Msgs)
}
theMeshMakes(t, js, node)
feed := make(chan *nats.Msg, drainDepth)
sub, err := js.ChanSubscribe(DeclareSubject(node), feed, nats.Bind("NODES", node))
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
select {
case msg := <-feed:
if want := declaredIn(signedBy(t, private, []byte(`{"declared":"d3"}`))); declaredIn(msg.Data) != want {
t.Fatalf("the node was given %q rather than the newest", msg.Data)
}
case <-time.After(8 * time.Second):
t.Fatal("the node that was away was given nothing")
}
}
// A report goes through the stream: it must survive the controller's store restarting.
func TestNatsAReportIsKept(t *testing.T) {
conn, js := aBus(t)
bus := OverNATS{Conn: conn, JS: js}
ctx := context.Background()
body, _ := json.Marshal(Report{Node: "anchor", Declared: "d1"})
if err := bus.Report(ctx, "anchor", body); err != nil {
t.Fatal(err)
}
info, err := js.StreamInfo("CONTROL")
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != 1 {
t.Fatalf("%d messages were kept; a report must be, once", info.State.Msgs)
}
}
// **One apply queue, against a real bus** (novox/hq to-be 45 §6, R2). A declaration is being applied
// when two more are pushed and the five-minute reconcile comes due: the worker applies the one in
// hand, then only the newest — the reconcile is that apply — and the mesh's stream holds one account
// per apply, each naming the order it applied, numbered in the order they were made. Every
// declaration delivered is settled.
func TestNatsTheQueueAppliesTheNewestOnceAndReportsInOrder(t *testing.T) {
conn, js := aBus(t)
const node = "queueing"
theMeshMakes(t, js, node)
public, private, _ := ed25519.GenerateKey(nil)
m := Membership{Node: node, Signer: public}
ctx, stop := context.WithCancel(context.Background())
defer stop()
l := &natsLink{conn: conn, js: js, node: node,
arrived: make(chan Declaration, drainDepth), lost: make(chan error, 1)}
feed := make(chan *nats.Msg, drainDepth)
sub, err := js.ChanSubscribe(DeclareSubject(node), feed, nats.Bind("NODES", node))
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
go func() {
for msg := range feed {
l.arrived <- natsDeclaration{msg}
}
}()
reports, err := js.SubscribeSync(ReportSubject(node), nats.BindStream("CONTROL"))
if err != nil {
t.Fatal(err)
}
defer func() { _ = reports.Unsubscribe() }()
a := &applies{}
inFirst, release := make(chan struct{}, 1), make(chan struct{})
reconciled := 0
q := &Queue{Membership: m, Timeout: 5 * time.Second, Unsaid: &keptInMemory{},
Apply: func(ctx context.Context, raw, sig []byte) Report {
r := a.apply(ctx, raw, sig)
if r.Sequence == 1 {
inFirst <- struct{}{}
<-release
}
return r
},
Reconcile: func(context.Context) (Report, bool) { reconciled++; return Report{}, false },
}
worker := runWorker(ctx, q)
go func() { _ = serve(ctx, l, m, q, nil, 5*time.Second) }()
push := func(id string, sequence int64) {
inner, _ := json.Marshal(map[string]any{"declaration": 1, "epoch": 7, "sequence": sequence,
"resources": []any{map[string]any{"id": id}}})
if _, err := js.Publish(DeclareSubject(node), signedBy(t, private, inner)); err != nil {
t.Fatal(err)
}
}
push("one", 1)
select {
case <-inFirst:
case <-time.After(8 * time.Second):
t.Fatal("the first declaration never reached the worker")
}
push("two", 2)
push("three", 3)
q.ReconcileDue()
time.Sleep(2 * drainWindow) // the link gathers both and hands them to the queue
close(release)
var heard []Report
for len(accounts(heard)) < 2 {
msg, err := reports.NextMsg(8 * time.Second)
if err != nil {
t.Fatalf("the mesh heard %+v and then nothing: %v", heard, err)
}
var r Report
if err := json.Unmarshal(msg.Data, &r); err != nil {
t.Fatal(err)
}
heard = append(heard, r)
_ = msg.Ack()
}
stop()
<-worker
if got := a.all(); !reflect.DeepEqual(got, []string{"one", "three"}) {
t.Fatalf("applied %v; the one in hand and then only the newest were wanted", got)
}
if reconciled != 0 {
t.Fatal("the reconcile applied beside the delivery it was due with")
}
accounted := accounts(heard)
if accounted[0].Sequence != 1 || accounted[1].Sequence != 3 || accounted[1].Epoch != 7 {
t.Fatalf("the accounts do not name what was applied: %+v", accounted)
}
for i := 1; i < len(heard); i++ {
if heard[i].ReportSequence <= heard[i-1].ReportSequence {
t.Fatalf("the reports left out of the order they were made: %+v", heard)
}
}
deadline := time.Now().Add(5 * time.Second)
for {
info, err := js.ConsumerInfo("NODES", node)
if err == nil && info.NumAckPending == 0 {
break
}
if time.Now().After(deadline) {
t.Fatalf("a delivered declaration was left unsettled: %+v %v", info, err)
}
time.Sleep(20 * time.Millisecond)
}
}
-46
View File
@@ -1,46 +0,0 @@
package link
import (
"strings"
"testing"
)
// The count that did not add up was the only symptom sixteen held resources had, and reading it meant
// opening the node's state file by hand (novox/hq 04-ISSUES/125). The line that says what an apply did
// says what it did not, too.
func TestTheApplyLineSaysWhatItHeldAndForWhichModule(t *testing.T) {
got := heldNote([]Held{
{ID: "ca", Module: "route-proxy", Kind: "directory"},
{ID: "certs", Module: "route-proxy", Kind: "directory"},
{ID: "server", Module: "route-proxy", Kind: "container"},
{ID: "mail", Module: "mailu", Kind: "container"},
})
if !strings.Contains(got, "4 held") {
t.Fatalf("the count of what was held is not in the line: %q", got)
}
// The module is the thing an operator can act on: `take` takes a module.
if !strings.Contains(got, "route-proxy: 3") || !strings.Contains(got, "mailu: 1") {
t.Fatalf("the line does not break the holds down by module: %q", got)
}
// Ordered, so two machines holding the same things read the same and a diff of two reports is
// about what changed.
if strings.Index(got, "mailu") > strings.Index(got, "route-proxy") {
t.Fatalf("modules are not in a stated order: %q", got)
}
// It says why, because "held" alone reads as a failure and this is correct behaviour.
if !strings.Contains(got, "taken") {
t.Fatalf("the line does not say a hold ends when the module is taken: %q", got)
}
}
func TestAnApplyThatHeldNothingSaysNothingExtra(t *testing.T) {
// A converged machine holds nothing, which is most applies. Reporting "0 held" on every one of
// them is how a line stops being read.
if got := heldNote(nil); got != "" {
t.Fatalf("an apply with no holds added %q to its line", got)
}
if got := heldNote([]Held{}); got != "" {
t.Fatalf("an apply with no holds added %q to its line", got)
}
}
+68
View File
@@ -0,0 +1,68 @@
//go:build integration
package link
import (
"context"
"testing"
"time"
natsserver "github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
)
// The revisions a bucket keeps of a key, read over a real bus: oldest first, a delete and a purge said as deleted, and
// none for a key that never was (issue 1945: the node agent takes the composition its want names from them).
func TestNatsAKeysRevisionsAreReadBack(t *testing.T) {
srv, err := natsserver.NewServer(&natsserver.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true})
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
conn, err := nats.Connect(srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
js, err := conn.JetStream()
if err != nil {
t.Fatal(err)
}
kv, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: "composed-it", History: 10})
if err != nil {
t.Fatal(err)
}
for _, v := range []string{"one", "two", "three"} {
if _, err := kv.PutString("it.node-tools.x", v); err != nil {
t.Fatal(err)
}
}
bus := OverNATS{Conn: conn, JS: js}
ctx := context.Background()
got, err := bus.KVHistory(ctx, "composed-it", "it.node-tools.x")
if err != nil || len(got) != 3 || string(got[0].Value) != "one" || string(got[2].Value) != "three" ||
got[0].Revision >= got[2].Revision {
t.Fatalf("three revisions read back as %+v (%v)", got, err)
}
if err := kv.Delete("it.node-tools.x"); err != nil {
t.Fatal(err)
}
got, err = bus.KVHistory(ctx, "composed-it", "it.node-tools.x")
if err != nil || len(got) != 4 || !got[3].Deleted {
t.Fatalf("after a delete: %+v (%v)", got, err)
}
if err := kv.Purge("it.node-tools.x"); err != nil {
t.Fatal(err)
}
got, err = bus.KVHistory(ctx, "composed-it", "it.node-tools.x")
if err != nil || len(got) != 1 || !got[0].Deleted {
t.Fatalf("after a purge: %+v (%v)", got, err)
}
if got, err := bus.KVHistory(ctx, "composed-it", "it.never.x"); err != nil || len(got) != 0 {
t.Fatalf("a key that never was: %+v (%v)", got, err)
}
}
+19 -132
View File
@@ -18,22 +18,6 @@ const (
KeyReport = "report"
)
// Signed is a declaration and the signature over it.
//
// novox/hq ADR 0004: the transport is verified once at connect, and **each declaration is
// verified by its signature, every time**. The two are different questions — a node connects to
// the broker and takes instruction from the control plane behind it, and pinning only the first
// would make the second transitive.
//
// The signature is over Declaration exactly as it arrived, bytes unchanged. Re-encoding before
// verifying would mean checking a signature over something other than what was sent, and any
// difference in key order or spacing would break it — so the raw message is what is signed and
// what is checked.
type Signed struct {
Declaration []byte `json:"declaration"`
Signature []byte `json:"signature"`
}
// Report is what a node says after applying, and it is a statement rather than a write.
//
// A node states; the context that owns the data writes (novox/hq ADR 0006). The difference is the
@@ -51,14 +35,6 @@ type Report struct {
// Refused is set when the declaration was rejected whole rather than applied in part.
Refused string `json:"refused,omitempty"`
// Superseded names the newer declaration this one was set aside for, unapplied.
//
// A machine asked to be five successive things becomes the last one (novox/hq issue 031):
// when several declarations are waiting, the host applies the newest and acknowledges the
// rest without applying them. Each of those is still reported, because silence reads as a
// machine that ignored an instruction and "applied" would be a lie — this is the third word.
Superseded string `json:"superseded,omitempty"`
// Carried are the machine's ports held by what this host raised from its own bundle.
//
// **So the mesh can assign around what it did not put here** (novox/hq ADR 0038). A node
@@ -70,62 +46,13 @@ type Report struct {
// machine saying what is true of it, not asking for anything.
Carried []int `json:"carried,omitempty"`
// Declared is the digest of the declaration this report is about — sha256 of the exact bytes
// the mesh sent, which the mesh recorded when it sent them.
//
// **Which declaration, not when.** The mesh compared its send time to this report's arrival
// to decide whether a machine had caught up, and lost the race it invited: an apply started
// under the previous declaration finishes after the next one is sent, its report lands newer
// than the send, and the machine reads as caught up with words it has not read yet. Clocks
// cannot answer "which"; the digest is the answer itself.
Declared string `json:"declared,omitempty"`
// Order is where the declaration this report is about stands — its `epoch` and `sequence`, as
// the declaration carried them — so the mesh orders reports by what they are about rather than
// by when they arrived (novox/hq to-be 45 §6). On the wire as two top-level keys beside
// `declared`; absent for a declaration that claimed no order, and for a report about none.
// Writer, and Order — the node-engine's own epoch and sequence — say who wrote these facts and where they stand
// among everything it wrote (novox/hq ADR 0227 §2, ADR 0539 §9): the epoch is one per start of the node-engine,
// the sequence one higher for every facts event, so the controller refuses older facts by this order. On the
// wire as three top-level keys.
Writer string `json:"writer,omitempty"`
Order
// ReportSequence is this node-engine's own order among everything it reports: one higher for
// every report it makes, kept on disk so it goes on increasing across restarts and self-updates
// (to-be 45 §6). With the declaration's order it lets the mesh refuse an older report about the
// same declaration — a reconcile's account overtaking the apply that followed it (novox/hq issue
// 267) is then refused by number rather than set aside by digest. A report said again because
// it never reached the mesh (issue 264) carries the number it was made with.
//
// Zero claims no order: every report an older host makes, and a one-shot report a command makes
// (a rekey). **Its presence also says this host reads a declaration's `epoch`**, which is how the
// mesh knows it may send one: an older host decodes a declaration strictly and refuses a key it
// does not know.
ReportSequence int64 `json:"report_sequence,omitempty"`
// OlderThan is set on a report refusing a declaration older than one this node has applied: the
// order of the newer declaration it holds (to-be 45 §6, rule 2). The refused declaration is the
// report's own Declared and Order, and Refused says it in words. Applying it would make the
// machine into something a controller that lost its lease said, after one that holds it.
OlderThan *Order `json:"older_than,omitempty"`
// RefusedOlder is how many declarations this node-engine has refused as older, ever — kept on
// disk with the report sequence. On every report, so a count missed with a lost refusal is still
// read from the next one: what the mesh's stale-writer watchdog (to-be 45 §3, S13) counts.
RefusedOlder int64 `json:"refused_older,omitempty"`
// ReadsGeneration says this node-engine reads a declaration's assignment generation and its
// put-back mark (novox/hq issue 234), on every report it makes: the controller sends them only to a
// machine that said so, because an older node-engine decodes strictly and refuses a key it does not
// know, whole. A node-engine rolled back says so no longer.
ReadsGeneration bool `json:"reads_generation,omitempty"`
// ReadsModules says this node-engine reads a resource's `module` and a declaration's `builds` (novox/hq ADR 0429
// §4, issue 1132), on every report it makes, beside ReadsGeneration: the controller sends them only to a machine
// that said so, for the same reason.
ReadsModules bool `json:"reads_modules,omitempty"`
// OlderGeneration is set on a report refusing a declaration composed from an older assignment
// generation than this machine applied (novox/hq issue 234). The refused declaration is the report's
// own Declared and Order, so the controller can name who sent it from its record of sends.
OlderGeneration *GenerationRefusal `json:"older_generation,omitempty"`
// Held is what this adopted node found and is keeping as it was until its module is taken
// (novox/hq ADR 0100). Without it an adopted node reads as converged.
Held []Held `json:"held,omitempty"`
@@ -647,52 +574,14 @@ type Rollback struct {
At time.Time `json:"at"`
}
// Order is where a declaration stands among everything the mesh has sent this node (novox/hq to-be
// 45 §6): the controller's lease **epoch** it was sent under, and its **sequence** — one higher for
// every send to this node (04-ISSUES/107). The declaration carries both inside what is signed, as
// top-level `epoch` and `sequence` beside `declaration`; a report carries back the pair of the
// declaration it is about.
//
// Zero in either is "no order claimed", never "first": every declaration an older controller sent,
// and the bundle genesis applies.
// Order is where a writer's saying stands among everything it said (novox/hq ADR 0227 §2): its **epoch**, and its
// **sequence** within it. A node-engine's epoch is the time it started, so a later start orders after an earlier one
// with nothing kept on disk.
type Order struct {
Epoch int64 `json:"epoch,omitempty"`
Sequence int64 `json:"sequence,omitempty"`
}
// Older says whether a declaration of this order is older than one of order `than`, which this node
// has applied — the one rule by which a node-engine refuses a declaration (to-be 45 §6: "older epoch;
// same epoch, lower sequence").
//
// **Only when both claim an epoch.** A declaration with none is an older controller's — or a
// controller rolled back to a build from before the lease — and one kept with none is what this host
// held before any controller had a lease. Neither has an order to compare, and refusing on a guess
// would strand the machine the moment the controller is older than the host: without an epoch, today's
// behaviour stands. A higher epoch is a new lease holder and is never older, whatever its sequence.
func (o Order) Older(than Order) bool {
if o.Epoch <= 0 || than.Epoch <= 0 {
return false
}
if o.Epoch != than.Epoch {
return o.Epoch < than.Epoch
}
return o.Sequence > 0 && than.Sequence > 0 && o.Sequence < than.Sequence
}
// Supersedes says whether a declaration of order `o`, arriving after one of order `before`, takes its
// place among what is waiting to be applied. By epoch when both claim one and they differ, then by
// sequence when both claim one, and by arrival when they do not — what the drain had to go on before
// declarations said where they stand (04-ISSUES/107).
func (o Order) Supersedes(before Order) bool {
if o.Epoch > 0 && before.Epoch > 0 && o.Epoch != before.Epoch {
return o.Epoch > before.Epoch
}
if o.Sequence > 0 && before.Sequence > 0 {
return o.Sequence >= before.Sequence
}
return true
}
// Words is an order as a person reads it in a log line. Not String: Report embeds Order, and a
// Stringer promoted onto every report would print each one as its order alone.
func (o Order) Words() string {
@@ -842,18 +731,16 @@ type FoundFirewall struct {
RetiredBy string `json:"retired_by,omitempty"`
}
// GenerationRefusal is a declaration refused for the assignment generation it was composed from: that
// generation, and the highest this machine applied (novox/hq issue 234).
type GenerationRefusal struct {
Generation int64 `json:"generation"`
Applied int64 `json:"applied"`
}
// ModuleFact is how one module's part of an apply came out: the build the node was sent for it, and whether every
// resource of it applied. Detail names the failing resource and its error (issue 1132).
// ModuleFact is how one module's part of an apply came out (novox/hq ADR 0539 §9): the fingerprint FP the module runs and
// the digest Composed of the composition applied for it, beside the build the node was sent, the Phase it is in (empty
// is running, or failed when Failed), and whether every resource of it applied. Detail names the failing resource and its
// error (issue 1132).
type ModuleFact struct {
Module string
Build string
Failed bool
Detail string
Module string
Build string
FP string
Composed string
Phase string
Failed bool
Detail string
}
+3 -106
View File
@@ -1,115 +1,10 @@
package link
import (
"context"
"crypto/ed25519"
"encoding/json"
"testing"
)
// verified runs what Run does to a delivery body, without a broker: unmarshal, check the
// signature, and only then apply. Isolating it keeps this test about the check rather than about
// the bus, which is tested against a real one in the lab.
func verified(t *testing.T, signer ed25519.PublicKey, body []byte) (Report, bool) {
t.Helper()
applied := false
report := handleBody(context.Background(), Membership{Node: "anchor", Signer: signer}, body,
func(context.Context, []byte, []byte) Report {
applied = true
return Report{Applied: []string{"something"}}
})
return report, applied
}
func signedBody(t *testing.T, private ed25519.PrivateKey, declaration string) []byte {
t.Helper()
raw, err := json.Marshal(Signed{
Declaration: []byte(declaration),
Signature: ed25519.Sign(private, []byte(declaration)),
})
if err != nil {
t.Fatal(err)
}
return raw
}
func TestTheMeshsOwnDeclarationIsApplied(t *testing.T) {
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
report, applied := verified(t, public, signedBody(t, private, `{"declaration":1}`))
if !applied {
t.Fatalf("a declaration the mesh signed was not applied: %s", report.Refused)
}
}
// Defends novox/hq ADR 0002: everything reaching a node arrives over the broker — and therefore
// ADR 0004's consequence, that the broker is not trusted to say who is speaking.
//
// A transport nobody authenticates per-message would let whatever holds the connection attribute
// a declaration to any node it liked.
func TestAForgedDeclarationIsNeverApplied(t *testing.T) {
// The check that stands between "the mesh changes this machine" and "anybody does". The host
// applies whatever the link delivers, so a forged declaration is the whole machine.
public, _, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
_, other, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
report, applied := verified(t, public, signedBody(t, other, `{"declaration":1}`))
if applied {
t.Fatal("a declaration signed by another key was applied")
}
if report.Refused != ErrForged.Error() {
t.Errorf("refused, but not as a forgery: %q", report.Refused)
}
}
func TestATamperedDeclarationIsNeverApplied(t *testing.T) {
// A broker that changed the declaration in flight, keeping the signature. This is what makes
// pinning the transport insufficient on its own.
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
raw, err := json.Marshal(Signed{
Declaration: []byte(`{"declaration":1,"resources":["something else entirely"]}`),
Signature: ed25519.Sign(private, []byte(`{"declaration":1}`)),
})
if err != nil {
t.Fatal(err)
}
report, applied := verified(t, public, raw)
if applied {
t.Fatal("a declaration altered after signing was applied")
}
if report.Refused != ErrForged.Error() {
t.Errorf("refused, but not as a forgery: %q", report.Refused)
}
}
func TestAMalformedMessageIsToldApartFromAForgery(t *testing.T) {
// novox/hq ADR 0004 requires these to be distinguishable: one means somebody is trying, the
// other means something is broken, and they need different responses from a person.
public, _, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
report, applied := verified(t, public, []byte("this is not a message"))
if applied {
t.Fatal("something unparseable was applied")
}
if report.Refused == ErrForged.Error() {
t.Error("a malformed message was reported as a forgery; those must be distinguishable")
}
}
func TestTheWireFormatIsExactlyTheseFieldNames(t *testing.T) {
// The contract with the control plane, which defines these separately. A matching test lives
// there; rename a field on either side and both fail.
@@ -117,7 +12,9 @@ func TestTheWireFormatIsExactlyTheseFieldNames(t *testing.T) {
value any
expect []string
}{
{Signed{Declaration: []byte("{}"), Signature: []byte("x")}, []string{"declaration", "signature"}},
// novox/hq ADR 0539 §9: the node's facts carry the node-engine's own writer, epoch and sequence.
{Report{Node: "n", Writer: Writer, Order: Order{Epoch: 1, Sequence: 2}},
[]string{"node", "writer", "epoch", "sequence"}},
{Report{Node: "n", Applied: []string{"a"}, Failed: map[string]string{"k": "v"}, Refused: "r"},
[]string{"node", "applied", "failed", "refused"}},
// novox/hq ADR 0100: what an adopted node holds, the firewall it was found with, and what
+1 -1
View File
@@ -173,7 +173,7 @@ const VaultWithin = 15 * time.Second
// AskVault asks one of the vault's verbs in this node's engine's own name over the link open now, and answers its
// result, or its refusal as an error.
func (q *Queue) AskVault(ctx context.Context, verb string, args any) (json.RawMessage, error) {
func (q *Session) AskVault(ctx context.Context, verb string, args any) (json.RawMessage, error) {
q.init()
q.mu.Lock()
bus := q.bus
+3 -3
View File
@@ -112,8 +112,8 @@ func TestNatsAMoveIsAskedOfTheNodeAndTheVaultIsAskedInItsName(t *testing.T) {
}
defer vault.Unsubscribe()
l := &natsLink{conn: conn, js: js, node: m.Node, signer: m.Signer}
q := &Queue{Membership: m}
q.attach(t.Context(), l)
q := &Session{Membership: m}
q.Attach(t.Context(), l)
l.HearMoveSecrets(func(s MoveSecret) (MovedSecret, error) {
raw, err := q.AskVault(t.Context(), "move-in", map[string]string{"module": s.Module})
if err != nil || !strings.Contains(string(raw), `"call":"c1"`) {
@@ -135,7 +135,7 @@ func TestNatsAMoveIsAskedOfTheNodeAndTheVaultIsAskedInItsName(t *testing.T) {
if err := json.Unmarshal(reply.Data, &said); err != nil || said.Secret != "own/"+m.Node+"/postgres/superuser-password" {
t.Fatalf("answered %s, %v", reply.Data, err)
}
if _, err := (&Queue{Membership: Membership{Node: "nowhere"}}).AskVault(t.Context(), "move-in", nil); !errors.Is(err, ErrNotAsked) {
if _, err := (&Session{Membership: Membership{Node: "nowhere"}}).AskVault(t.Context(), "move-in", nil); !errors.Is(err, ErrNotAsked) {
t.Fatalf("a queue with no link asked the vault: %v", err)
}
}
-79
View File
@@ -1,79 +0,0 @@
package link
import (
"sync"
"testing"
"time"
)
type said struct {
body []byte
mu sync.Mutex
handled bool
}
func (s *said) Body() []byte { return s.body }
func (s *said) Handled() error {
s.mu.Lock()
defer s.mu.Unlock()
s.handled = true
return nil
}
func (s *said) wasHandled() bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.handled
}
func arriving(bodies ...string) chan Declaration {
ch := make(chan Declaration, 8)
for _, b := range bodies {
ch <- &said{body: []byte(b)}
}
return ch
}
// newest is what the queue's worker would take from what the link gathered behind `first`.
func newest(arriving <-chan Declaration, first Declaration, window time.Duration) (Declaration, []Declaration) {
return pick(gather(arriving, first, window))
}
// A machine asked to be five things becomes the last one: what is already waiting supersedes what
// arrived first, and everything set aside is named so it can be reported.
func TestWhatIsAlreadyWaitingSupersedesWhatArrivedFirst(t *testing.T) {
waiting := arriving("two", "three", "four")
apply, superseded := newest(waiting, &said{body: []byte("one")}, 50*time.Millisecond)
if string(apply.Body()) != "four" {
t.Fatalf("applied %q, not the newest", apply.Body())
}
if len(superseded) != 3 || string(superseded[0].Body()) != "one" ||
string(superseded[2].Body()) != "three" {
t.Fatalf("set aside %d: %v", len(superseded), superseded)
}
}
// One declaration with nothing behind it is applied as it always was, after the window.
func TestALoneDeclarationIsAppliedAfterTheWindow(t *testing.T) {
began := time.Now()
apply, superseded := newest(arriving(), &said{body: []byte("only")}, 30*time.Millisecond)
if string(apply.Body()) != "only" || len(superseded) != 0 {
t.Fatalf("got %q with %d set aside", apply.Body(), len(superseded))
}
if time.Since(began) < 30*time.Millisecond {
t.Fatal("did not wait the window for a straggler")
}
}
// A straggler within the window is taken; one after it is the next push.
func TestAStragglerWithinTheWindowIsTaken(t *testing.T) {
waiting := arriving()
go func() {
time.Sleep(20 * time.Millisecond)
waiting <- &said{body: []byte("late")}
}()
apply, superseded := newest(waiting, &said{body: []byte("first")}, 100*time.Millisecond)
if string(apply.Body()) != "late" || len(superseded) != 1 {
t.Fatalf("got %q with %d set aside", apply.Body(), len(superseded))
}
}
+1 -1
View File
@@ -35,7 +35,7 @@ func (l *listeningForAlive) count() int {
func TestTheLinkLoopPublishesNoAlive(t *testing.T) {
m, _ := aMember(t)
l := &listeningForAlive{quietLink: newQuietLink()}
q := aQueue(m, nil, &keptInMemory{})
q := &Session{Membership: m, Timeout: time.Second}
ctx, stop := context.WithCancel(context.Background())
done := make(chan error, 1)
-50
View File
@@ -1,50 +0,0 @@
package link
import (
"encoding/json"
"testing"
"time"
)
// The drain picked the last to arrive. A backlog longer than the batch, or a broker that split a
// burst, delivered a superseded declaration last (novox/hq 04-ISSUES/107).
func sequenced(t *testing.T, n int64) *said {
t.Helper()
inner, err := json.Marshal(map[string]any{"declaration": 1, "resources": []any{}, "sequence": n})
if err != nil {
t.Fatal(err)
}
body, err := json.Marshal(Signed{Declaration: inner, Signature: []byte("s")})
if err != nil {
t.Fatal(err)
}
return &said{body: body}
}
func TestTheDrainKeepsTheHighestSequenceNotTheLastToArrive(t *testing.T) {
waiting := make(chan Declaration, 8)
waiting <- sequenced(t, 9)
waiting <- sequenced(t, 4) // arrived last, composed earlier
latest, aside := newest(waiting, sequenced(t, 8), 30*time.Millisecond)
if got := orderOf(latest.Body()).Sequence; got != 9 {
t.Fatalf("the drain kept sequence %d, and 9 was waiting", got)
}
if len(aside) != 2 {
t.Fatalf("%d set aside, wanted 2 (the 8 and the late 4)", len(aside))
}
}
func TestWithoutSequencesTheLastToArriveStillWins(t *testing.T) {
// The behaviour this had before, kept for a controller that sends no order.
apply, aside := newest(arriving("two", "three"), &said{body: []byte("one")}, 30*time.Millisecond)
if string(apply.Body()) != "three" || len(aside) != 2 {
t.Fatalf("applied %q with %d set aside", apply.Body(), len(aside))
}
}
func TestAnUnreadableBodyClaimsNoOrder(t *testing.T) {
if got := orderOf([]byte("not json")); got != (Order{}) {
t.Fatalf("garbage claimed %+v", got)
}
}
-647
View File
@@ -1,647 +0,0 @@
package link
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/health"
)
// Queue is the node-engine's one apply queue (novox/hq to-be 45 §6, ADR 0227 rule 1).
//
// **One worker applies; everything else asks it to.** A declaration delivered over the link and the
// five-minute reconcile are two reasons to *enqueue* the same act, never two paths that apply. Before
// this they were two, each taking an apply lock, and every order between them was a fault met on the
// live mesh: the reconcile read what was kept before a delivery and applied it over the newer one
// (novox/hq issues 257, 261); the reconcile went first and its report about the older declaration
// reached the mesh after the delivery's (267). Each fix closed one order and left the class open.
//
// The queue holds at most one pending request, coalesced: whatever has been delivered and not yet
// taken, and whether a reconcile is due. When the worker starts it takes the **newest declaration held
// at that moment**, sets the rest aside — each reported as set aside, then settled unapplied — applies
// it once, and makes **one report**, naming the declaration's order and its own report sequence. A
// reconcile due while a delivery waits is that delivery's apply: applying the newest thing the mesh
// said is what reconciling is. Only with nothing delivered does a reconcile apply what was kept, read
// once its turn has come.
//
// **Reports leave in the order they are made**, because the worker makes and says them, one at a time.
// A report made while the link is down waits: an apply's in the unsaid store (issue 264), a
// reconcile's in one slot replaced by anything newer, and both are said, in that order, when the next
// link opens — before anything newly delivered is applied.
type Queue struct {
// Membership is whose declarations these are: their signer, and the node the reports are from.
Membership Membership
// Apply applies a declaration the mesh signed.
Apply Applier
// Reconcile holds the machine to what it was last told, read once it is this act's turn. False
// when there is nothing to hold it to, or the reconcile could not run; it says why itself.
Reconcile Reconcile
// News is whether a reconcile's report is worth saying unasked (novox/hq ADR 0100, ADR 0140). Nil
// is never: a reconcile is otherwise silent.
News func(Report) bool
// Heard is told every account of the machine the mesh has taken — an apply's or a reconcile's,
// not a refusal or a declaration set aside, which say nothing about the machine. Nil is allowed.
Heard func(Report)
// Unsaid keeps an apply's report until the mesh has taken it (issue 264). Nil keeps nothing.
Unsaid Unsaid
// Numbers keeps the report sequence and the refusals counted, across restarts and self-updates.
// Nil counts in memory, which is only right where nothing outlives the process — a test.
Numbers Numbers
// HandOver judges and records a directory used as found handed to the mesh by the operator, asked by
// the controller over the link (novox/hq issue 356). Nil answers every such ask with a refusal.
HandOver HandOverFunc
// SetuidSearch throws away the last search for setuid programs and starts a full one, asked by the
// controller's terminal over the link (novox/hq issue 361). Nil answers every such ask with a refusal.
SetuidSearch SetuidSearchFunc
// JudgeDirectories judges this node's declared directories read-only, asked by the controller over the
// link (novox/hq issue 358). Nil answers every such ask with a refusal.
JudgeDirectories JudgeDirectoriesFunc
// AccessAct judges and runs the take act on an access's folder, asked by the controller on an approved
// warrant (novox/hq ADR 0328 §7). Nil answers every such ask with a refusal.
AccessAct AccessActFunc
// MoveSecret moves one value sealed to this machine into the vault, asked by the controller during the move of
// the secrets (design 57's V3, issue 676). Nil answers every such ask with a refusal.
MoveSecret MoveSecretFunc
Say Announce
Timeout time.Duration
once sync.Once
wake chan struct{}
mu sync.Mutex
idle *sync.Cond
// delivering is a delivered declaration in hand: its report is owed on the link it came over, so
// that link is not let go until it is said (detach). A reconcile in hand holds no link — its report
// waits for the next one if this one goes.
delivering bool
waiting []Declaration // delivered and not yet taken, in the order they arrived
due bool // a reconcile asked for
// asked is the mesh asking this node to say what it last applied (the `report` verb, to-be 45 §6):
// the reconcile it enqueues says its report whether or not it is news.
asked bool
bus Bus // the link open now; nil while there is none
unasked *Report // a reconcile's report not yet said, made while there was no link
// saying is held while a report is made and said, so the order they leave in is the order they
// are made in — whoever says them, the worker or a link opening.
saying sync.Mutex
counted bool
numbers numbered
// haves is what this node-engine writes beside its reports (novox/hq ADR 0429 §4, issue 1132).
haves haveState
}
// Reconcile is the reconcile's act, run by the queue's worker. See Queue.Reconcile.
type Reconcile func(ctx context.Context) (Report, bool)
// Numbers is where the queue keeps what it counts, so a successor goes on from where it stopped.
type Numbers interface {
// Read is what was kept: zero for a node that never reported, an error when it cannot be read —
// never zero for "could not tell".
Read() (reportSequence, refusedOlder int64, err error)
// Save keeps them, on disk when it returns.
Save(reportSequence, refusedOlder int64) error
}
type numbered struct{ sequence, refused int64 }
func (q *Queue) init() {
q.once.Do(func() {
q.wake = make(chan struct{}, 1)
q.idle = sync.NewCond(&q.mu)
if q.Say == nil {
q.Say = func(string) {}
}
})
}
// Deliver enqueues what the link delivered, in the order it arrived. It never waits for an apply.
func (q *Queue) Deliver(arrived ...Declaration) {
q.init()
q.mu.Lock()
q.waiting = append(q.waiting, arrived...)
q.mu.Unlock()
q.signal()
}
// ReconcileDue enqueues a reconcile. One asked for while another is waiting is the same one.
func (q *Queue) ReconcileDue() {
q.init()
q.mu.Lock()
q.due = true
q.mu.Unlock()
q.signal()
}
// ReportAsked is the mesh asking this node to say again what it last applied (novox/hq to-be 45 §6, the
// `report` verb its healer H1 asks when a send went unreported). **An enqueue, like everything else**: a
// reconcile, whose report — the account of the declaration this node keeps, the last it applied — is
// said whether or not it is news. A delivery waiting meanwhile is applied instead and reported, which
// answers the question better. Asked again before the worker takes it is the same question.
func (q *Queue) ReportAsked() {
q.init()
q.mu.Lock()
q.due, q.asked = true, true
q.mu.Unlock()
q.signal()
}
func (q *Queue) signal() {
select {
case q.wake <- struct{}{}:
default:
// One is already waiting to be read, and the worker takes everything pending when it does.
}
}
// Run is the worker. It returns once the context ends and the act in hand is finished and said:
// asked to stop — the host standing aside for its successor — nothing further is applied.
func (q *Queue) Run(ctx context.Context) {
q.init()
for {
for q.step(ctx) {
}
select {
case <-ctx.Done():
return
case <-q.wake:
}
}
}
// step takes what is pending and does it, once. False when there was nothing to do, or the queue
// was asked to stop.
func (q *Queue) step(ctx context.Context) bool {
q.mu.Lock()
if ctx.Err() != nil || (len(q.waiting) == 0 && !q.due) {
q.mu.Unlock()
return false
}
batch, due, asked := q.waiting, q.due, q.asked
q.waiting, q.due, q.asked = nil, false, false
q.delivering = len(batch) > 0
q.mu.Unlock()
defer func() {
q.mu.Lock()
q.delivering = false
q.idle.Broadcast()
q.mu.Unlock()
}()
if len(batch) == 0 {
q.reconcile(ctx, asked)
return true
}
latest, aside := pick(batch)
for _, old := range aside {
if d := declaredIn(old.Body()); d != "" && d == declaredIn(latest.Body()) {
// The same declaration again — delivered twice while the worker was busy. Nothing is set
// aside: it is about to be applied.
_ = old.Handled()
continue
}
q.Say(fmt.Sprintf("set aside declaration %s (%s): a newer one arrived with it",
short(declaredIn(old.Body())), orderOf(old.Body()).Words()))
q.tell(ctx, Report{Declared: declaredIn(old.Body()), Order: orderOf(old.Body()),
Superseded: declaredIn(latest.Body())}, setAside, old)
}
report := handleBody(q.withFiles(ctx), q.Membership, latest.Body(), q.Apply)
switch {
case report.OlderThan != nil:
// **Refused, counted and said** (rule 2): one line here, the count on this and every later
// report, and the report itself naming what was refused and what is held.
total := q.countRefusal()
q.Say(fmt.Sprintf("refused declaration %s (%s): this node applied %s, from a newer lease "+
"holder — %d refused as older so far", short(report.Declared), report.Order.Words(),
report.OlderThan.Words(), total))
case report.OlderGeneration != nil:
// Said here, and the report names it, so the controller raises it naming the sender (novox/hq
// issue 234). Not counted with the lease's refusals: those are S13's, by epoch.
q.Say(fmt.Sprintf("refused declaration %s (%s): it was composed from assignment generation %d, and "+
"this node applied generation %d", short(report.Declared), report.Order.Words(),
report.OlderGeneration.Generation, report.OlderGeneration.Applied))
case report.Refused != "":
q.Say("refused a declaration: " + report.Refused)
case len(report.Failed) > 0:
q.Say(fmt.Sprintf("applied %d and failed: %v%s",
len(report.Applied), report.Failed, heldNote(report.Held)))
default:
q.Say(fmt.Sprintf("applied %d resource(s)%s", len(report.Applied), heldNote(report.Held)))
}
if due && report.Refused != "" {
// Nothing was applied, so the reconcile this stood in for has not happened. It runs next.
q.mu.Lock()
q.due = true
q.mu.Unlock()
}
// A refusal as older is not kept to be said again: what the mesh waits for from this node is the
// account of the last apply, and that is still what is kept.
kind := applied
if report.OlderThan != nil || report.OlderGeneration != nil {
kind = refusedOlder
}
q.tell(ctx, report, kind, latest)
return true
}
func (q *Queue) reconcile(ctx context.Context, asked bool) {
if q.Reconcile == nil {
if asked {
q.Say("asked to say what this node last applied, and it holds nothing to reconcile with")
}
return
}
report, ok := q.Reconcile(q.withFiles(ctx))
if !ok {
if asked {
q.Say("asked to say what this node last applied, and the reconcile could not say it")
}
return
}
if !asked && (q.News == nil || !q.News(report)) {
// Not worth a report, but a reconcile found how each module is now: the have keys say it (issue 1132), so a
// node-engine that restarted writes them again without waiting for a change.
if q.noteApply(report) {
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
return
}
if asked {
q.Say(fmt.Sprintf("asked by the mesh, saying again what this node last applied: declaration %s (%s)",
short(report.Declared), report.Order.Words()))
}
q.tell(ctx, report, unasked)
}
// What a report is, for what tell does with it besides saying it.
type kindOf int
const (
// applied is an apply's account: kept until said (issue 264), and fresher than any reconcile's
// report still waiting.
applied kindOf = iota
// setAside is a declaration a newer one took the place of, unapplied.
setAside
// refusedOlder is a declaration refused as older than what was applied. Not kept: what the mesh
// waits for from this node is still the account of the last apply.
refusedOlder
// unasked is a reconcile's report. Not said — no link, or the bus refused it — the newest waits
// for the next link, and anything the worker says before then replaces it.
unasked
)
// tell makes a report this node's — its node, its report sequence, the refusals counted — and says it
// on the link open now. Kept first, when it is an apply's, so a report lost between here and the bus
// is said by whichever host links next. Each declaration it settles is settled after it is said,
// either way. True when the broker took it.
func (q *Queue) tell(ctx context.Context, r Report, kind kindOf, settle ...Declaration) bool {
q.saying.Lock()
defer q.saying.Unlock()
r = q.stamp(r)
keep := kind == applied
if keep {
// What this says supersedes any reconcile's report still waiting to be said.
q.unasked = nil
if q.Unsaid != nil {
if err := q.Unsaid.Keep(r); err != nil {
q.Say("cannot keep this apply's report until it is said: " + err.Error())
}
}
}
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
published := false
if bus != nil {
// **Said even when the link is being let go** (issue 264): not cancelled with it, still
// bounded by the timeout, and the link is not closed until this returns (detach).
published = publishReport(context.WithoutCancel(ctx), bus, q.Membership, r, q.Say, q.Timeout)
}
if !published && kind == unasked {
q.unasked = &r
}
if published {
// After the report, which is sent first and unchanged; a write the bucket refuses never undoes it (issue 1132).
if q.noteApply(r) {
q.writeHaves(ctx, bus)
}
if keep && q.Unsaid != nil {
if err := q.Unsaid.Said(r.Declared); err != nil {
q.Say("said this apply's report, and cannot forget it: " + err.Error())
}
}
if q.Heard != nil && (kind == applied || kind == unasked) && r.Refused == "" {
q.Heard(r)
}
}
// Settled after the report is published. A node that dies between applying and reporting
// leaves the declaration with the mesh and applies it again on return, which is safe because
// applying is reconciliation — it converges rather than repeating.
for _, d := range settle {
_ = d.Handled()
}
return published
}
// stamp gives a report its node, the next report sequence and the refusals counted. Called holding
// saying.
func (q *Queue) stamp(r Report) Report {
q.load()
q.numbers.sequence++
q.keepNumbers()
r.Node = q.Membership.Node
r.ReportSequence = q.numbers.sequence
r.RefusedOlder = q.numbers.refused
// Every report of a node-engine that orders its reports says it reads a generation (novox/hq issue
// 234): this build does, and the controller sends one only to a machine that said so.
r.ReadsGeneration = true
// And the module of each resource and the build of each module (novox/hq ADR 0429 §4, issue 1132), the same way.
r.ReadsModules = true
return r
}
func (q *Queue) countRefusal() int64 {
q.saying.Lock()
defer q.saying.Unlock()
q.load()
q.numbers.refused++
q.keepNumbers()
return q.numbers.refused
}
// load reads what was counted, once. **Unreadable is said, and the sequence goes on from above
// anything it can have reached** — the time in milliseconds — rather than from one, which would make
// every report this host says next older than what it already said, and refused.
func (q *Queue) load() {
if q.counted {
return
}
q.counted = true
if q.Numbers == nil {
return
}
sequence, refused, err := q.Numbers.Read()
if err != nil {
floor := time.Now().UnixMilli()
q.Say(fmt.Sprintf("cannot read this node's report sequence: %v — going on from %d, above any "+
"it can have reached, and counting refusals again from none", err, floor))
q.numbers = numbered{sequence: floor}
return
}
q.numbers = numbered{sequence: sequence, refused: refused}
}
func (q *Queue) keepNumbers() {
if q.Numbers == nil {
return
}
if err := q.Numbers.Save(q.numbers.sequence, q.numbers.refused); err != nil {
// Said, and not fatal: the report is still worth saying. What is lost is that a successor
// could number one again.
q.Say("cannot keep this node's report sequence: " + err.Error())
}
}
// attach is a link opening. **What the last apply did, if the mesh never heard it**, is said first,
// exactly as it was made; then a reconcile's report that waited; and only then may the worker say
// anything on it — so a report said again can never land after, and read as newer than, a later one.
func (q *Queue) attach(ctx context.Context, bus Bus) {
q.init()
q.saying.Lock()
defer q.saying.Unlock()
reporting := context.WithoutCancel(ctx)
q.load()
said := true
if q.Unsaid != nil {
switch kept, ok, err := q.Unsaid.Pending(); {
case err != nil:
q.Say("cannot read the report this node kept unsaid: " + err.Error())
case ok:
q.Say("saying again what the last apply did: its report never reached the mesh")
if said = publishReport(reporting, bus, q.Membership, kept, q.Say, q.Timeout); said {
if err := q.Unsaid.Said(kept.Declared); err != nil {
q.Say("said the kept report, and cannot forget it: " + err.Error())
}
if q.Heard != nil && kept.Refused == "" {
q.Heard(kept)
}
}
}
}
if said && q.unasked != nil {
if publishReport(reporting, bus, q.Membership, *q.unasked, q.Say, q.Timeout) {
if q.Heard != nil {
q.Heard(*q.unasked)
}
q.unasked = nil
}
}
q.mu.Lock()
q.bus = bus
q.mu.Unlock()
}
// withFiles is ctx carrying the link now open, so an apply can fetch a declaration's files given by digest (novox/hq ADR
// 0486, issue 1695). A queue with no link open carries none.
func (q *Queue) withFiles(ctx context.Context) context.Context {
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return ctx
}
return WithFiles(ctx, bus)
}
// detach is the link closing: it waits for a delivery in hand to be applied and said — an apply that
// stood aside for its successor is reported on this link before it is let go (issue 264) — and then
// nothing more is said on it. A reconcile in hand is not waited for: a link that was dropped or roused
// is opened again at once, and the reconcile's report, if it is news, waits for it.
func (q *Queue) detach(bus Bus) {
q.init()
q.mu.Lock()
for q.delivering {
q.idle.Wait()
}
q.mu.Unlock()
q.saying.Lock()
defer q.saying.Unlock()
q.mu.Lock()
if q.bus == bus {
q.bus = nil
}
q.mu.Unlock()
}
// pick is the newest of what was delivered and the rest, in arrival order, to set aside: by order
// when the declarations claim one, by arrival when they do not (Order.Supersedes).
func pick(batch []Declaration) (Declaration, []Declaration) {
latest := batch[0]
var aside []Declaration
for _, next := range batch[1:] {
if orderOf(next.Body()).Supersedes(orderOf(latest.Body())) {
aside = append(aside, latest)
latest = next
continue
}
aside = append(aside, next)
}
return latest, aside
}
// orderOf is the order a signed declaration claims, or none when it claims none or cannot be read.
// Read from the envelope alone; the signature is verified when the winner is applied, and a forged
// message that lied about its order would only set aside real ones — which are reported as set aside,
// and the next push sends the current one again.
func orderOf(body []byte) Order {
var signed Signed
if err := json.Unmarshal(body, &signed); err != nil {
return Order{}
}
var o Order
if err := json.Unmarshal(signed.Declaration, &o); err != nil || o.Epoch < 0 || o.Sequence < 0 {
return Order{}
}
return o
}
// declaredIn names a signed declaration as the mesh does — the digest of the declaration's bytes, the
// same the apply's report carries — for a report about one that was not applied. Empty if the message
// is not one: a forged or garbled message is refused when its turn comes; here it is only named.
func declaredIn(body []byte) string {
var signed Signed
if err := json.Unmarshal(body, &signed); err != nil || len(signed.Declaration) == 0 {
return ""
}
sum := sha256.Sum256(signed.Declaration)
return hex.EncodeToString(sum[:])
}
// Linked says whether a link to the bus is open now (novox/hq ADR 0241: the bus is one part of the
// machine's networking its engine judges).
func (q *Queue) Linked() bool {
q.init()
q.mu.Lock()
defer q.mu.Unlock()
return q.bus != nil
}
// SayHealth says a health statement on the link open now (novox/hq ADR 0240, to-be 48 §4), and whether
// the bus took it. **Not through the worker**: a statement is a word about the machine as it is, not an
// account of an apply, and it must not wait behind one — a container crash-looping while a long apply
// runs is said while it runs. False with no link open, or one that cannot say it; the caller says it
// again on its next look, and the next report carries it whatever happens.
func (q *Queue) SayHealth(ctx context.Context, h Health) bool {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
sayer, ok := bus.(HealthBus)
if bus == nil || !ok {
return false
}
body, err := json.Marshal(HealthSaid{Node: q.Membership.Node, Health: h})
if err != nil {
return false
}
saying, cancel := context.WithTimeout(ctx, q.timeout())
defer cancel()
if err := sayer.Health(saying, q.Membership.Node, body); err != nil {
q.Say("could not say this machine's health: " + err.Error())
return false
}
// A look that changed a module's health changes its have (issue 1132).
if q.noteHealth(h) {
q.writeHaves(ctx, bus)
}
return true
}
func (q *Queue) timeout() time.Duration {
if q.Timeout <= 0 {
return 10 * time.Second
}
return q.Timeout
}
// AskTool asks a module's tool on this machine's own node tools, as a declared health check (novox/hq ADR
// 0240, to-be 48 §3), and reads its answer: healthy or not, with why. An error when there is no link open
// or nothing answered — said as the check's failure, never as healthy.
func (q *Queue) AskTool(ctx context.Context, module, tool string) (health.Answer, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return health.Answer{}, errors.New("no link to the bus is open")
}
asker, ok := bus.(ToolBus)
if !ok {
return health.Answer{}, errors.New("the link to the bus open now cannot ask a tool")
}
reply, err := asker.Ask(ctx, ToolSubject(module, tool, q.Membership.Node), []byte("{}"))
if err != nil {
return health.Answer{}, err
}
return ReadToolAnswer(reply)
}
// AskController asks the controller a line mesh-cli was given on this machine, on this node's own subject, over
// the link open now (novox/hq ADR 0272 §3), and returns its answer as it came. No link open is said, not waited for.
func (q *Queue) AskController(ctx context.Context, body []byte) ([]byte, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return nil, fmt.Errorf("%w: this machine has no link to the bus open now", ErrNotAsked)
}
asker, ok := bus.(ToolBus)
if !ok {
return nil, fmt.Errorf("%w: the link to the bus open now cannot ask the controller", ErrNotAsked)
}
return asker.Ask(ctx, CLISubject(q.Membership.Node), body)
}
// ErrNotAsked is a request the link knows it never delivered: no link open, or nobody listening on its subject. Any
// other failure of a request may come after it was delivered, so what came of it is not known (novox/hq ADR 0272).
var ErrNotAsked = errors.New("not asked")
// ReadToolAnswer reads a tool's reply envelope — `{result, error}`, as the node tools answer every call —
// as a health answer, through the SDK's reader (novox/hq ADR 0283): healthy, unhealthy, or waits for the
// operator with what it waits for. An error, and a result that is not an answer, are unhealthy; a wait the
// SDK cannot check is unhealthy too, so a malformed one never excuses anything.
func ReadToolAnswer(reply []byte) (health.Answer, error) {
var env struct {
Result json.RawMessage `json:"result"`
Error string `json:"error"`
}
if err := json.Unmarshal(reply, &env); err != nil {
return health.Answer{}, fmt.Errorf("its answer is not one: %w", err)
}
if env.Error != "" {
return health.Unhealthy(env.Error), nil
}
a, err := health.Read(env.Result)
if err != nil {
return health.Unhealthy("it answered something other than healthy or not"), nil
}
return a, nil
}
-588
View File
@@ -1,588 +0,0 @@
package link
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"reflect"
"strings"
"sync"
"testing"
"time"
)
// The node-engine's one apply queue (novox/hq to-be 45 §6): a delivery and the reconcile are two
// reasons to enqueue the same act, the newest declaration is applied once, and the reports leave in
// the order they are made. These are the races of issues 257, 261, 264 and 267 replayed against it.
func aQueue(m Membership, apply Applier, kept Unsaid) *Queue {
return &Queue{Membership: m, Apply: apply, Unsaid: kept, Timeout: time.Second}
}
// runWorker runs the queue's worker until ctx ends; the channel closes when it has.
func runWorker(ctx context.Context, q *Queue) <-chan struct{} {
done := make(chan struct{})
go func() {
defer close(done)
q.Run(ctx)
}()
return done
}
// ordered is a signed declaration claiming an order, with a resource id so two are distinct bytes.
func ordered(t *testing.T, key ed25519.PrivateKey, id string, epoch, sequence int64) *said {
t.Helper()
inner, err := json.Marshal(map[string]any{"declaration": 1, "resources": []any{map[string]any{"id": id}},
"epoch": epoch, "sequence": sequence})
if err != nil {
t.Fatal(err)
}
return &said{body: signedBy(t, key, inner)}
}
// applies records what the worker applied, by the id of the declaration's first resource, and
// reports it as applied — with the order the declaration carried, as the host's own apply does.
type applies struct {
mu sync.Mutex
seen []string
}
func (a *applies) apply(_ context.Context, raw, _ []byte) Report {
var d struct {
Resources []struct {
ID string `json:"id"`
} `json:"resources"`
Order
}
_ = json.Unmarshal(raw, &d)
id := ""
if len(d.Resources) > 0 {
id = d.Resources[0].ID
}
a.mu.Lock()
a.seen = append(a.seen, id)
a.mu.Unlock()
return Report{Declared: digest(raw), Order: d.Order, Applied: []string{id}}
}
func (a *applies) all() []string {
a.mu.Lock()
defer a.mu.Unlock()
return append([]string(nil), a.seen...)
}
func digest(raw []byte) string {
b, _ := json.Marshal(Signed{Declaration: raw})
return declaredIn(b)
}
// eventually waits for a condition the worker reaches on its own time.
func eventually(t *testing.T, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
for !ok() {
if time.Now().After(deadline) {
t.Fatal(what)
}
time.Sleep(5 * time.Millisecond)
}
}
// accounts is the reports the mesh heard that are an apply's account, not a set-aside's.
func accounts(reports []Report) []Report {
var out []Report
for _, r := range reports {
if r.Superseded == "" {
out = append(out, r)
}
}
return out
}
// **R2: a reconcile due during a delivery** (issues 257, 261, 267). The reconcile is asked for while a
// delivery waits: the worker applies the delivery once — applying the newest thing the mesh said is
// what reconciling is — and makes one report, naming the newest sequence. The reconcile never applies
// what was kept before.
func TestAReconcileDueDuringADeliveryIsThatDeliverysApply(t *testing.T) {
m, key := aMember(t)
l := newQuietLink()
a := &applies{}
reconciled := 0
q := aQueue(m, a.apply, &keptInMemory{})
q.Reconcile = func(context.Context) (Report, bool) {
reconciled++
return Report{Declared: "the one kept before", Applied: []string{"old"}}, true
}
q.News = func(Report) bool { return true }
q.attach(context.Background(), l)
q.Deliver(ordered(t, key, "new", 7, 12))
q.ReconcileDue()
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
eventually(t, "the delivery was never reported", func() bool { return len(l.said()) == 1 })
stop()
<-worker
if got := a.all(); !reflect.DeepEqual(got, []string{"new"}) {
t.Fatalf("applied %v; one apply of the newest was wanted", got)
}
if reconciled != 0 {
t.Fatalf("the reconcile applied what was kept %d time(s) beside the delivery", reconciled)
}
r := l.said()[0]
if r.Sequence != 12 || r.Epoch != 7 || r.ReportSequence != 1 || r.Node != m.Node {
t.Fatalf("the report does not name the order it applied: %+v", r)
}
}
// **The other order** (issue 267): the reconcile is running when a delivery arrives. Its report, about
// the declaration kept then, leaves first; the delivery's leaves after it, with a higher report
// sequence. The mesh's latest word is the newest declaration, whichever arrived first.
func TestAReconcileRunningWhenADeliveryArrivesIsSaidBeforeIt(t *testing.T) {
m, key := aMember(t)
l := newQuietLink()
a := &applies{}
q := aQueue(m, a.apply, &keptInMemory{})
inReconcile, release := make(chan struct{}), make(chan struct{})
q.Reconcile = func(context.Context) (Report, bool) {
close(inReconcile)
<-release
return Report{Declared: "d1", Order: Order{Epoch: 7, Sequence: 11}, Applied: []string{"a"},
Outward: []string{"eth0"}}, true
}
q.News = func(Report) bool { return true }
q.attach(context.Background(), l)
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
q.ReconcileDue()
<-inReconcile
q.Deliver(ordered(t, key, "b", 7, 12))
close(release)
eventually(t, "the delivery was never reported", func() bool { return len(l.said()) == 2 })
stop()
<-worker
reports := l.said()
if reports[0].Declared != "d1" || reports[1].Sequence != 12 {
t.Fatalf("the reports left out of the order they were made: %+v", reports)
}
if reports[0].ReportSequence >= reports[1].ReportSequence {
t.Fatalf("the later report carries no higher report sequence: %d then %d",
reports[0].ReportSequence, reports[1].ReportSequence)
}
}
// **Two deliveries in quick succession**: only the newest is applied, and there is one account of an
// apply — the first is reported as set aside, naming the one that took its place.
func TestTwoDeliveriesInQuickSuccessionApplyOnlyTheNewest(t *testing.T) {
m, key := aMember(t)
first, second := ordered(t, key, "first", 7, 12), ordered(t, key, "second", 7, 13)
l := newQuietLink(first, second)
a := &applies{}
q := aQueue(m, a.apply, &keptInMemory{})
ctx, stop := context.WithCancel(context.Background())
defer stop()
worker := runWorker(ctx, q)
go func() { _ = serve(ctx, l, m, q, nil, time.Second) }()
eventually(t, "nothing was reported", func() bool { return len(accounts(l.said())) == 1 })
stop()
<-worker
if got := a.all(); !reflect.DeepEqual(got, []string{"second"}) {
t.Fatalf("applied %v; only the newest was wanted", got)
}
reports := l.said()
if len(reports) != 2 || reports[0].Superseded != declaredIn(second.body) ||
reports[0].Declared != declaredIn(first.body) || reports[0].Sequence != 12 {
t.Fatalf("the first was not reported as set aside for the second: %+v", reports)
}
if !first.wasHandled() || !second.wasHandled() {
t.Fatal("a declaration was left unsettled")
}
}
// Deliveries that arrive while the worker is applying are coalesced: when it is free it takes the
// newest held at that moment — by order, not by arrival — and applies it once.
func TestDeliveriesWhileTheWorkerIsBusyAreOneApplyOfTheNewest(t *testing.T) {
m, key := aMember(t)
l := newQuietLink()
a := &applies{}
inApply, release := make(chan struct{}, 1), make(chan struct{})
q := aQueue(m, func(ctx context.Context, raw, sig []byte) Report {
r := a.apply(ctx, raw, sig)
if r.Sequence == 1 {
inApply <- struct{}{}
<-release
}
return r
}, &keptInMemory{})
q.attach(context.Background(), l)
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
q.Deliver(ordered(t, key, "one", 7, 1))
<-inApply
q.Deliver(ordered(t, key, "three", 7, 3))
q.Deliver(ordered(t, key, "two", 7, 2)) // arrived last, composed earlier
q.ReconcileDue() // and the five-minute timer fired too
close(release)
eventually(t, "the newest was never reported", func() bool { return len(accounts(l.said())) == 2 })
stop()
<-worker
if got := a.all(); !reflect.DeepEqual(got, []string{"one", "three"}) {
t.Fatalf("applied %v; the one in hand and then only the newest were wanted", got)
}
last := accounts(l.said())[1]
if last.Sequence != 3 {
t.Fatalf("the last account names sequence %d, not the newest", last.Sequence)
}
}
// **An older epoch is refused, counted and said** (rule 2): the refusal's report names the refused
// declaration and what this node holds, carries the count, and does not replace the kept account of
// the last apply — which is what the mesh is waiting for from this node.
func TestADeclarationFromAnOlderEpochIsRefusedCountedAndSaid(t *testing.T) {
m, key := aMember(t)
l := newQuietLink()
kept := &keptInMemory{}
lastApply := Report{Declared: "d-applied", Order: Order{Epoch: 57, Sequence: 3}, Applied: []string{"a"}}
_ = kept.Keep(lastApply)
stale := ordered(t, key, "stale", 41, 12)
q := aQueue(m, func(_ context.Context, raw, _ []byte) Report {
// What the host's apply answers for a declaration older than the one it kept.
return Report{Declared: digest(raw), Order: Order{Epoch: 41, Sequence: 12},
OlderThan: &Order{Epoch: 57, Sequence: 3}, Refused: "older than what this node applied"}
}, kept)
var lines saidSoFar
q.Say = lines.say
l.refusing = true // nothing is said on linking: the kept account stays kept
q.attach(context.Background(), l)
l.mu.Lock()
l.refusing = false
l.mu.Unlock()
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
q.Deliver(stale)
eventually(t, "the refusal was never reported", func() bool { return len(l.said()) == 1 })
stop()
<-worker
r := l.said()[0]
if r.OlderThan == nil || *r.OlderThan != (Order{Epoch: 57, Sequence: 3}) || r.Epoch != 41 ||
r.Declared != declaredIn(stale.body) || r.RefusedOlder != 1 {
t.Fatalf("the refusal does not name what was refused, what is held, and the count: %+v", r)
}
if !strings.Contains(lines.all(), "1 refused as older so far") {
t.Fatalf("the refusal was not said in the log:\n%s", lines.all())
}
if still, ok, _ := kept.Pending(); !ok || still.Declared != "d-applied" {
t.Fatalf("the refusal replaced the kept account of the last apply: %+v", still)
}
if !stale.wasHandled() {
t.Fatal("the refused declaration was not settled: the mesh would deliver it again")
}
}
// **Restart with an unsaid report**: the next host says it first, exactly as it was made, and numbers
// what it says next above it — the report sequence goes on across the restart rather than from one.
func TestAfterARestartTheUnsaidReportIsSaidFirstAndTheNumbersGoOn(t *testing.T) {
m, key := aMember(t)
kept := &keptInMemory{}
numbers := &numbersInMemory{}
// The first host applies; its report never reaches the mesh, and it is gone.
first := newQuietLink()
first.refusing = true
q1 := aQueue(m, (&applies{}).apply, kept)
q1.Numbers = numbers
q1.attach(context.Background(), first)
ctx1, stop1 := context.WithCancel(context.Background())
w1 := runWorker(ctx1, q1)
q1.Deliver(ordered(t, key, "a", 7, 12))
eventually(t, "the first host never kept its report", func() bool { _, ok, _ := kept.Pending(); return ok })
stop1()
<-w1
lost, _, _ := kept.Pending()
// The next host links, and is then delivered something new.
next := newQuietLink()
q2 := aQueue(m, (&applies{}).apply, kept)
q2.Numbers = numbers
q2.attach(context.Background(), next)
ctx2, stop2 := context.WithCancel(context.Background())
w2 := runWorker(ctx2, q2)
q2.Deliver(ordered(t, key, "b", 7, 13))
eventually(t, "the next host's apply was never reported", func() bool { return len(next.said()) == 2 })
stop2()
<-w2
reports := next.said()
if !reflect.DeepEqual(reports[0], lost) {
t.Fatalf("the unsaid report was not said first, as it was made: %+v", reports[0])
}
if reports[1].Sequence != 13 || reports[1].ReportSequence <= lost.ReportSequence {
t.Fatalf("the next host numbered from one again: %d after %d",
reports[1].ReportSequence, lost.ReportSequence)
}
}
// A report sequence that cannot be read is said, and numbering goes on above anything it can have
// reached — never from one, which would make every report older than those already said.
func TestUnreadableNumbersAreSaidAndNumberingGoesOnAboveThem(t *testing.T) {
m, _ := aMember(t)
l := newQuietLink()
q := aQueue(m, nil, nil)
q.Numbers = &numbersInMemory{broken: errors.New("unreadable")}
var lines saidSoFar
q.Say = lines.say
q.News = func(Report) bool { return true }
q.Reconcile = func(context.Context) (Report, bool) { return Report{Declared: "d"}, true }
before := time.Now().UnixMilli()
q.attach(context.Background(), l)
q.ReconcileDue()
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
eventually(t, "nothing was reported", func() bool { return len(l.said()) == 1 })
stop()
<-worker
if got := l.said()[0].ReportSequence; got < before {
t.Fatalf("numbering went on from %d, not above anything it can have reached", got)
}
if !strings.Contains(lines.all(), "cannot read this node's report sequence") {
t.Fatalf("the unreadable sequence was not said:\n%s", lines.all())
}
}
// A reconcile's report made with no link waits for the next one; an apply made before then replaces
// it, because the apply's account is fresher.
func TestAReconcileReportMadeOfflineWaitsAndIsReplacedByAnApply(t *testing.T) {
m, key := aMember(t)
q := aQueue(m, (&applies{}).apply, &keptInMemory{})
q.News = func(Report) bool { return true }
q.Reconcile = func(context.Context) (Report, bool) {
return Report{Declared: "d1", Outward: []string{"eth0"}}, true
}
ctx, stop := context.WithCancel(context.Background())
defer stop()
runWorker(ctx, q)
q.ReconcileDue()
eventually(t, "the reconcile's report was not held", func() bool {
q.saying.Lock()
defer q.saying.Unlock()
return q.unasked != nil
})
l := newQuietLink()
q.attach(context.Background(), l)
if reports := l.said(); len(reports) != 1 || reports[0].Declared != "d1" {
t.Fatalf("the reconcile's report was not said when the link opened: %+v", reports)
}
// Offline again: a reconcile's report, then an apply of a delivery still in hand.
q.detach(l)
q.ReconcileDue()
eventually(t, "the second reconcile's report was not held", func() bool {
q.saying.Lock()
defer q.saying.Unlock()
return q.unasked != nil
})
q.Deliver(ordered(t, key, "x", 7, 2))
eventually(t, "the apply did not replace the reconcile's report", func() bool {
q.saying.Lock()
defer q.saying.Unlock()
return q.unasked == nil
})
}
// The contract with the controller, on the wire: the keys a declaration's order is read from, and the
// keys a report carries back. A rename here must break this test before it breaks a machine.
func TestTheOrderOnTheWire(t *testing.T) {
body, err := json.Marshal(Report{Node: "n", Declared: "d", Order: Order{Epoch: 57, Sequence: 3},
ReportSequence: 9, OlderThan: &Order{Epoch: 58, Sequence: 1}, RefusedOlder: 2})
if err != nil {
t.Fatal(err)
}
var wire map[string]any
_ = json.Unmarshal(body, &wire)
want := map[string]any{"node": "n", "declared": "d", "epoch": 57.0, "sequence": 3.0,
"report_sequence": 9.0, "older_than": map[string]any{"epoch": 58.0, "sequence": 1.0},
"refused_older": 2.0}
if !reflect.DeepEqual(wire, want) {
t.Fatalf("a report's order on the wire is\n%s\nnot the contract", body)
}
// An older host's report, and an older controller's declaration, claim no order.
if body, _ := json.Marshal(Report{Node: "n", Declared: "d"}); strings.Contains(string(body), "sequence") ||
strings.Contains(string(body), "epoch") {
t.Fatalf("a report about a declaration with no order claims one: %s", body)
}
_, key := aMember(t)
if got := orderOf(signedBy(t, key, []byte(`{"declaration":1,"epoch":57,"sequence":3}`))); got != (Order{Epoch: 57, Sequence: 3}) {
t.Fatalf("a declaration's order was read as %+v", got)
}
}
func TestOlder(t *testing.T) {
for _, c := range []struct {
in, held Order
older bool
}{
{Order{41, 12}, Order{57, 3}, true}, // an older lease holder, whatever its sequence
{Order{57, 2}, Order{57, 3}, true}, // same holder, lower sequence
{Order{57, 3}, Order{57, 3}, false}, // the same declaration again: reconciling
{Order{58, 1}, Order{57, 3}, false}, // a new lease holder
{Order{0, 1}, Order{57, 3}, false}, // an older controller, or one rolled back: today's behaviour
{Order{41, 1}, Order{0, 9}, false}, // nothing held claims an epoch
{Order{57, 0}, Order{57, 3}, false}, // no sequence to compare
{Order{0, 0}, Order{0, 0}, false}, // neither claims anything
{Order{-1, 0}, Order{57, 3}, false}, // never a claim
{Order{57, 4}, Order{57, 3}, false}, // newer
{Order{56, 99}, Order{57, 1}, true}, // a higher sequence is no excuse for an older epoch
{Order{57, 1}, Order{56, 99}, false}, // nor a lower one a reason to refuse a newer epoch
{Order{57, 3}, Order{57, 0}, false}, // held with no sequence
{Order{100, 0}, Order{57, 0}, false}, // newer epoch, no sequences
{Order{10, 0}, Order{57, 0}, true}, // older epoch, no sequences
{Order{57, 2}, Order{0, 0}, false}, // nothing held at all
{Order{0, 2}, Order{0, 3}, false}, // sequence alone does not refuse on the link (no epoch)
{Order{57, 2}, Order{57, -1}, false}, // a held sequence below zero is not one
{Order{57, -1}, Order{57, 2}, false}, // nor an arriving one
{Order{-5, -5}, Order{-1, -1}, false}, // nothing below zero is an order
} {
if got := c.in.Older(c.held); got != c.older {
t.Errorf("%+v older than %+v = %v, want %v", c.in, c.held, got, c.older)
}
}
}
func TestSupersedes(t *testing.T) {
for _, c := range []struct {
next, before Order
want bool
}{
{Order{58, 1}, Order{57, 9}, true}, // a new lease holder
{Order{57, 9}, Order{58, 1}, false}, // the stale one arriving late
{Order{57, 4}, Order{57, 3}, true},
{Order{57, 2}, Order{57, 3}, false}, // arrived last, composed earlier
{Order{0, 4}, Order{0, 3}, true},
{Order{0, 2}, Order{0, 3}, false},
{Order{}, Order{0, 3}, true}, // no order: by arrival
{Order{0, 3}, Order{}, true},
} {
if got := c.next.Supersedes(c.before); got != c.want {
t.Errorf("%+v supersedes %+v = %v, want %v", c.next, c.before, got, c.want)
}
}
}
// numbersInMemory is Numbers without a disk, surviving a "restart" by being shared.
type numbersInMemory struct {
mu sync.Mutex
sequence, refused int64
broken error
}
func (n *numbersInMemory) Read() (int64, int64, error) {
n.mu.Lock()
defer n.mu.Unlock()
return n.sequence, n.refused, n.broken
}
func (n *numbersInMemory) Save(sequence, refused int64) error {
n.mu.Lock()
defer n.mu.Unlock()
n.sequence, n.refused = sequence, refused
return nil
}
// A link dropped or roused while a reconcile is in hand is let go at once and opened again: the
// reconcile holds no link, and its report, if it is news, waits for the next one. Only a delivery in
// hand keeps its link until its report is said.
func TestALinkIsNotHeldForAReconcileInHand(t *testing.T) {
m, _ := aMember(t)
inReconcile, release := make(chan struct{}), make(chan struct{})
q := aQueue(m, nil, nil)
q.News = func(Report) bool { return true }
q.Reconcile = func(context.Context) (Report, bool) {
close(inReconcile)
<-release
return Report{Declared: "d1", Outward: []string{"eth0"}}, true
}
l := newQuietLink()
q.attach(context.Background(), l)
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
q.ReconcileDue()
<-inReconcile
detached := make(chan struct{})
go func() { q.detach(l); close(detached) }()
select {
case <-detached:
case <-time.After(2 * time.Second):
t.Fatal("the link was held while a reconcile ran")
}
close(release)
eventually(t, "the reconcile's report was not held for the next link", func() bool {
q.saying.Lock()
defer q.saying.Unlock()
return q.unasked != nil
})
stop()
<-worker
if len(l.said()) != 0 {
t.Fatalf("a report was said on a link let go: %+v", l.said())
}
}
// **Every report says this node-engine reads a declaration's generation** (novox/hq issue 234): the
// controller sends one only to a machine that said so, because an older node-engine refuses the key.
func TestEveryReportSaysItReadsAGeneration(t *testing.T) {
m, key := aMember(t)
l := newQuietLink()
a := &applies{}
q := aQueue(m, a.apply, &keptInMemory{})
q.attach(context.Background(), l)
q.Deliver(ordered(t, key, "new", 7, 12))
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
eventually(t, "the delivery was never reported", func() bool { return len(l.said()) == 1 })
stop()
<-worker
raw, err := json.Marshal(l.said()[0])
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(raw), `"reads_generation":true`) {
t.Fatalf("the report does not say it reads a generation: %s", raw)
}
}
// 1132/AC2: every report says this node-engine reads a resource's module and a declaration's builds, beside the
// generation: the controller sends them only to a machine that said so, because an older node-engine refuses the keys.
func TestEveryReportSaysItReadsModules(t *testing.T) {
m, key := aMember(t)
l := newQuietLink()
a := &applies{}
q := aQueue(m, a.apply, &keptInMemory{})
q.attach(context.Background(), l)
q.Deliver(ordered(t, key, "new", 7, 12))
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
eventually(t, "the delivery was never reported", func() bool { return len(l.said()) == 1 })
stop()
<-worker
raw, err := json.Marshal(l.said()[0])
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(raw), `"reads_modules":true`) || !strings.Contains(string(raw), `"reads_generation":true`) {
t.Fatalf("the report does not say it reads modules beside the generation: %s", raw)
}
}
+2 -2
View File
@@ -13,8 +13,8 @@ import (
// The NATS link is attached with no connection under it, so no bus is needed: the question must reach the
// link and fail there, not stop at the queue as if no link were open.
func TestReplay331(t *testing.T) {
q := &Queue{Membership: Membership{Node: "laptop"}}
q.attach(t.Context(), &natsLink{node: "laptop"})
q := &Session{Membership: Membership{Node: "laptop"}}
q.Attach(t.Context(), &natsLink{node: "laptop"})
_, err := q.AskTool(t.Context(), "sensors", "sensors_health")
if err == nil {
t.Fatal("asked on no connection and was answered")
+12 -167
View File
@@ -6,18 +6,9 @@ import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
)
// ErrForged is what a node returns for a declaration whose signature is not the mesh's.
//
// Its own error, and it must never be confused with a malformed message. novox/hq ADR 0004
// requires a host to tell *this is not from the mesh I joined* apart from *this is malformed*:
// the first means somebody is trying, the second means something is broken.
var ErrForged = errors.New("this declaration was not signed by the mesh this node joined")
// Membership is what a node needs to reach its mesh again, held by the caller.
type Membership struct {
Node string
@@ -30,13 +21,6 @@ type Membership struct {
Transport string
}
// Applier is what the host does with a declaration that has been proved to come from the mesh.
//
// It receives the signature as well as the declaration, so the host can keep both: what it was
// told is kept signed and verified again when it is read back, which means the file on disk is
// trusted for the same reason the message was rather than for being local.
type Applier func(ctx context.Context, declaration, signature []byte) Report
// Announce is how the link says what is happening, so a node running unattended leaves an
// account of it. Nil is allowed and means say nothing.
type Announce func(string)
@@ -64,35 +48,14 @@ type Announce func(string)
// Nil is allowed and means nothing ever rouses it, which is every machine that does not suspend.
type Roused <-chan struct{}
func Hold(ctx context.Context, m Membership, queue *Queue, say Announce, timeout time.Duration) error {
func Hold(ctx context.Context, m Membership, queue *Session, say Announce, timeout time.Duration) error {
return HoldRoused(ctx, m, queue, say, timeout, nil)
}
// Unsaid keeps the report of the last apply until the mesh has taken it, so a report lost between
// applying and publishing — the host standing aside for a successor, a crash, a power cut — is said
// again the next time this node is linked (novox/hq issue 264). Without it the declaration has been
// settled or applied, the machine is what the mesh said, and the mesh waits for a report that the
// node believes is gone.
//
// **Exactly the report the apply made, never a new one.** Re-applying the kept declaration would
// describe the machine as it is now; what the mesh waits for is what was done with what it sent.
// Nil is allowed and keeps nothing.
type Unsaid interface {
// Keep holds the report of an apply until it is said. A report naming no declaration — refused
// before it was read — replaces nothing worth saying, and clears what was kept.
Keep(Report) error
// Said forgets the kept report once the broker has taken the report about the same declaration.
Said(declared string) error
// Pending is the report kept and not yet said, if any.
Pending() (Report, bool, error)
}
// HoldRoused is Hold, told when the machine has reason to think its link is stale.
//
// **What arrives is enqueued, never applied here** (novox/hq to-be 45 §6). The queue's one worker
// applies, outlives every link, and says its reports on whichever link is open; the caller runs it
// (Queue.Run) for as long as this holds.
func HoldRoused(ctx context.Context, m Membership, queue *Queue, say Announce,
// The session outlives every link and says the node's facts on whichever is open (novox/hq ADR 0539 §9).
func HoldRoused(ctx context.Context, m Membership, queue *Session, say Announce,
timeout time.Duration, roused Roused) error {
return holdWith(ctx, func(ctx context.Context) error {
@@ -185,11 +148,11 @@ func holdWith(ctx context.Context, run attempt, say Announce, roused Roused) err
}
}
// Run holds the link open once, applying what arrives and reporting what happened.
// Run holds the link open once, hearing what the controller asks and saying the node's facts.
//
// Outbound only, and nothing listens on this machine. Returns when the link ends, for any reason;
// Hold is what decides whether to open it again.
func Run(ctx context.Context, m Membership, queue *Queue, say Announce, timeout time.Duration) error {
func Run(ctx context.Context, m Membership, queue *Session, say Announce, timeout time.Duration) error {
link, err := Open(ctx, m, timeout)
if err != nil {
return err
@@ -200,7 +163,7 @@ func Run(ctx context.Context, m Membership, queue *Queue, say Announce, timeout
// serve is Run on a link already open: separated so what the node says, and when, can be tested
// without a broker.
func serve(ctx context.Context, link Link, m Membership, queue *Queue, say Announce,
func serve(ctx context.Context, link Link, m Membership, queue *Session, say Announce,
timeout time.Duration) error {
if say == nil {
say = func(string) {}
@@ -209,18 +172,13 @@ func serve(ctx context.Context, link Link, m Membership, queue *Queue, say Annou
// Said, because it is the event anybody watching actually wants. Without it a node logs every
// failure and nothing on success, so a log full of "trying again" and then silence reads as
// still broken when it means the opposite.
say("in the mesh, hearing what this node should be")
say("in the mesh, hearing what the controller asks of this node")
// **What the last apply did, if the mesh never heard it**, is said before anything newly
// delivered can be: the queue says it as the link is handed to it (novox/hq issue 264). And the
// link is not let go while the queue's worker is mid-act: an apply that ends the link — the host
// standing aside for a successor it delivered (novox/hq ADR 0141) — is still reported on it,
// before the deferred close.
queue.attach(ctx, link)
defer queue.detach(link)
// The session says this node's facts as the link is handed to it, and is not let go while it is mid-act.
queue.Attach(ctx, link)
defer queue.Detach(link)
declarations := link.Declarations()
// And the mesh asking what this node last applied (to-be 45 §6), on a link that hears the question.
// And the mesh asking this node to say its facts again, on a link that hears the question.
var asks <-chan struct{}
if a, ok := link.(Asked); ok {
asks = a.AskedToReport()
@@ -248,7 +206,6 @@ func serve(ctx context.Context, link Link, m Membership, queue *Queue, say Annou
}
for {
// Asked to stop — by standing aside, say — nothing further is taken in.
if ctx.Err() != nil {
return nil
}
@@ -258,90 +215,11 @@ func serve(ctx context.Context, link Link, m Membership, queue *Queue, say Annou
case reason := <-link.Lost():
return reason
case <-asks:
queue.ReportAsked()
case declaration, ok := <-declarations:
if !ok {
// The link's own reason, when it has managed to say one: "stopped delivering" on
// its own says nothing about why, and why is the whole of what an operator wants.
select {
case reason := <-link.Lost():
return reason
default:
return errors.New("the mesh stopped sending this node declarations")
}
}
// Whatever else is already waiting goes with it. **Enqueued, not applied**: the queue
// applies the newest of everything delivered and not yet taken, and reports the rest as
// set aside (novox/hq to-be 45 §6).
queue.Deliver(gather(declarations, declaration, drainWindow)...)
queue.FactsAsked(ctx)
}
}
}
// short is a declaration's digest as a person reads it in a log line.
func short(digest string) string {
if len(digest) > 12 {
return digest[:12]
}
return digest
}
// drainDepth is how many declarations the link holds unread; drainWindow is how long it waits for
// another to follow the one it has. Both small: a push is rare and a backlog is the exception this
// exists for, not the shape of ordinary traffic.
const (
drainDepth = 16
drainWindow = 750 * time.Millisecond
)
// gather takes what is already waiting behind `first`, in the order it arrived. It waits `window`
// for a straggler after each arrival and no longer: a declaration in flight from the mesh arrives
// within that; one that does not is the next push. Which of them is applied is the queue's to decide
// (pick), across everything delivered and not yet taken.
//
// **Its job narrows once declarations are state rather than messages, and does not disappear.**
// On the bus a declaration is last-per-subject (novox/hq design 29 §4), so a node that was away
// receives exactly the current one instead of a queue of superseded ones — the catch-up half is the
// stream's. And a declaration's order decides which is newest, where this window only infers it from
// arrival time, which is the wire-level answer to novox/hq issue 107.
//
// What remains is the live case: three pushes in quick succession to a *connected*, idle node are
// three deliveries, whatever the stream later retains, and gathering them is what makes them one
// apply rather than two. So this is narrowed, not deleted — and saying which half goes is worth more
// than a note that it "can probably be removed", which is how a load-bearing window gets deleted by
// somebody in a hurry.
func gather(arriving <-chan Declaration, first Declaration, window time.Duration) []Declaration {
batch := []Declaration{first}
for {
select {
case next, ok := <-arriving:
if !ok {
return batch
}
batch = append(batch, next)
case <-time.After(window):
return batch
}
}
}
// handleBody is the whole of deciding whether to trust a message, separated from the broker so it
// can be tested as the security check it is rather than as message plumbing.
func handleBody(ctx context.Context, m Membership, body []byte, apply Applier) Report {
var signed Signed
if err := json.Unmarshal(body, &signed); err != nil {
return Report{Node: m.Node, Refused: "this message is not a declaration: " + err.Error()}
}
// Before anything is read out of it, let alone applied. The host applies whatever the link
// delivers, so this check is the difference between the mesh changing this machine and
// anybody changing it.
if !ed25519.Verify(m.Signer, signed.Declaration, signed.Signature) {
return Report{Node: m.Node, Refused: ErrForged.Error()}
}
return apply(ctx, signed.Declaration, signed.Signature)
}
// publishReport tells the mesh what this node did, and says whether the broker took it.
// Publish sends one report on this node's own connection and returns: the one-shot path for a
// report a command makes rather than the running host — a rekey (novox/hq ADR 0105). The same
@@ -379,36 +257,3 @@ func publishReport(ctx context.Context, bus Bus, m Membership, report Report,
}
return true
}
// heldNote is what this apply did NOT do, for the line that says what it did.
//
// **A count that does not add up is the only symptom a held resource had** (novox/hq 04-ISSUES/125).
// An adopted node keeps what it found until its module is taken (ADR 0100), and that is correct — but
// it was recorded only in the node's own state file. On the edge cut-over the mesh sent 346 resources,
// the journal said it applied 330, and nothing anywhere said which sixteen or why. Reading it took
// opening state.json by hand; not reading it took every public name on the machine down, because the
// operator had four green surfaces and a discrepancy nobody could interpret.
//
// So the line that reports the apply carries it. Grouped by module and ordered by name, because the
// sentence an operator needs is "route-proxy is assigned and not taken", and the module is the thing
// they can act on — `take` is the verb, and it takes a module.
func heldNote(held []Held) string {
if len(held) == 0 {
return ""
}
byModule := map[string]int{}
for _, h := range held {
byModule[h.Module]++
}
names := make([]string, 0, len(byModule))
for name := range byModule {
names = append(names, name)
}
sort.Strings(names)
parts := make([]string, 0, len(names))
for _, name := range names {
parts = append(parts, fmt.Sprintf("%s: %d", name, byModule[name]))
}
return fmt.Sprintf(", %d held until their module is taken (%s)",
len(held), strings.Join(parts, ", "))
}
+336
View File
@@ -0,0 +1,336 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/health"
)
// Writer is what the node-engine calls itself on the facts it says (novox/hq ADR 0227 §2, ADR 0539 §9): the program
// that wrote them, beside its epoch and sequence.
const Writer = "node-engine"
// Session is the node-engine's hold on the bus (novox/hq ADR 0539 §9): the link open now, what the controller asks
// of this machine over it, and what the machine says on it unasked: its health, its haves and its facts.
//
// **It applies nothing and orders nothing.** A module is applied by the node agent from its want, and the facts are an
// event of the node, ordered by the node-engine's own epoch and sequence. Nothing here receives, orders or reconciles
// a whole declaration.
type Session struct {
// Membership is the node and the mesh's key a controller's ask is verified with.
Membership Membership
// OnLink is run, off the link's own loop, each time a link opens and when the controller asks the node to say its
// facts again. It says them with SayFacts. Nil says nothing.
OnLink func(ctx context.Context)
// Heard is told every facts event the mesh has taken. Nil is allowed.
Heard func(Report)
// HaveWritten is told each time a have key is written, or found already saying what this node-engine would write:
// the first proves a node-engine on trial (novox/hq ADR 0539 §9). Nil is allowed.
HaveWritten func()
// HandOver judges and records a directory used as found handed to the mesh by the operator, asked by
// the controller over the link (novox/hq issue 356). Nil answers every such ask with a refusal.
HandOver HandOverFunc
// SetuidSearch throws away the last search for setuid programs and starts a full one, asked by the
// controller's terminal over the link (novox/hq issue 361). Nil answers every such ask with a refusal.
SetuidSearch SetuidSearchFunc
// JudgeDirectories judges this node's declared directories read-only, asked by the controller over the
// link (novox/hq issue 358). Nil answers every such ask with a refusal.
JudgeDirectories JudgeDirectoriesFunc
// AccessAct judges and runs the take act on an access's folder, asked by the controller on an approved
// warrant (novox/hq ADR 0328 §7). Nil answers every such ask with a refusal.
AccessAct AccessActFunc
// MoveSecret moves one value sealed to this machine into the vault, asked by the controller during the move of
// the secrets (design 57's V3, issue 676). Nil answers every such ask with a refusal.
MoveSecret MoveSecretFunc
Say Announce
Timeout time.Duration
once sync.Once
mu sync.Mutex
bus Bus // the link open now; nil while there is none
// gone is closed when the link open now closes, and changed whenever a link opens or closes (WhenLinked).
gone chan struct{}
changed chan struct{}
// owed is HealthOwed's: a module left motion unjudged.
owed chan struct{}
// haveAgain guards the one write of the haves pending after a transient failure (haveFailed), and how many came in a
// row.
haveAgain sync.Mutex
haveAgainPending bool
haveAgainTries int
// saying is held while a facts event is made and said, so they leave in the order they are made.
saying sync.Mutex
// epoch is this process's own: the time it started, in milliseconds, so a later process orders after an earlier one
// without anything kept on disk; sequence is one higher for every facts event it says (ADR 0227 §2).
epoch int64
sequence int64
// haves is what this node-engine writes beside its facts (novox/hq ADR 0429 §4, issue 1132).
haves haveState
}
func (q *Session) init() {
q.once.Do(func() {
if q.Say == nil {
q.Say = func(string) {}
}
q.epoch = time.Now().UnixMilli()
q.changed = make(chan struct{})
q.owed = make(chan struct{}, 1)
})
}
// HealthOwed is told, at most once until it is read, when a module's change ended and no look has judged it since: the
// health loop looks and says its statement at once rather than at its next tick or five minutes on, so a module with
// nothing long-running is ok within a look of its apply, not five minutes after it (2026-10-25, review of fix 15).
func (q *Session) HealthOwed() <-chan struct{} {
q.init()
return q.owed
}
// oweHealth nudges the health loop; a nudge already waiting is enough.
func (q *Session) oweHealth() {
q.init()
select {
case q.owed <- struct{}{}:
default:
}
}
// SayFacts says this node's facts on the link open now, in the node-engine's own order, and whether the bus took them.
// A facts event made with no link open is not kept: the next link says the facts as they are then.
func (q *Session) SayFacts(ctx context.Context, r Report) bool {
q.init()
q.saying.Lock()
defer q.saying.Unlock()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return false
}
q.sequence++
r.Node = q.Membership.Node
r.Writer = Writer
r.Order = Order{Epoch: q.epoch, Sequence: q.sequence}
if !publishReport(context.WithoutCancel(ctx), bus, q.Membership, r, q.Say, q.Timeout) {
return false
}
// After the facts, which are sent first and unchanged; a write the bucket refuses never undoes them (issue 1132).
if q.noteApply(r) {
q.writeHaves(ctx, bus)
}
if q.Heard != nil && r.Refused == "" {
q.Heard(r)
}
return true
}
// FactsAsked is the mesh asking this node to say its facts again: run on the link open now.
func (q *Session) FactsAsked(ctx context.Context) {
q.init()
if q.OnLink != nil {
go q.OnLink(ctx)
}
}
// Attach is a link opening: it is the link facts are said on from now.
func (q *Session) Attach(ctx context.Context, bus Bus) {
q.init()
q.mu.Lock()
q.bus = bus
q.gone = make(chan struct{})
close(q.changed)
q.changed = make(chan struct{})
q.mu.Unlock()
q.FactsAsked(ctx)
}
// WithLink is ctx carrying the link now open, so an apply can fetch a composition's files given by digest (novox/hq ADR
// 0486, issue 1695). A session with no link open carries none.
func (q *Session) WithLink(ctx context.Context) context.Context {
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return ctx
}
return WithFiles(ctx, bus)
}
// Detach is the link closing: nothing more is said on it.
func (q *Session) Detach(bus Bus) {
q.init()
q.saying.Lock()
defer q.saying.Unlock()
q.mu.Lock()
if q.bus == bus {
q.bus = nil
close(q.gone)
close(q.changed)
q.changed = make(chan struct{})
}
q.mu.Unlock()
}
// Linked says whether a link to the bus is open now (novox/hq ADR 0241: the bus is one part of the
// machine's networking its engine judges).
func (q *Session) Linked() bool {
q.init()
q.mu.Lock()
defer q.mu.Unlock()
return q.bus != nil
}
// SayHealth says a health statement on the link open now (novox/hq ADR 0240, to-be 48 §4), and whether
// the bus took it. A statement is a word about the machine as it is, and it must not wait behind an apply: a
// container crash-looping while a long apply runs is said while it runs. False with no link open, or one that cannot
// say it; the caller says it again on its next look, and the next facts event carries it whatever happens.
func (q *Session) SayHealth(ctx context.Context, h Health) bool {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
sayer, ok := bus.(HealthBus)
if bus == nil || !ok {
return false
}
body, err := json.Marshal(HealthSaid{Node: q.Membership.Node, Health: h})
if err != nil {
return false
}
saying, cancel := context.WithTimeout(ctx, q.timeout())
defer cancel()
if err := sayer.Health(saying, q.Membership.Node, body); err != nil {
q.Say("could not say this machine's health: " + err.Error())
return false
}
// A look that changed a module's health changes its have (issue 1132).
if q.noteHealth(h) {
q.writeHaves(ctx, bus)
}
return true
}
func (q *Session) timeout() time.Duration {
if q.Timeout <= 0 {
return 10 * time.Second
}
return q.Timeout
}
// AskTool asks a module's tool on this machine's own node tools, as a declared health check (novox/hq ADR
// 0240, to-be 48 §3), and reads its answer: healthy or not, with why. An error when there is no link open
// or nothing answered — said as the check's failure, never as healthy.
func (q *Session) AskTool(ctx context.Context, module, tool string) (health.Answer, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return health.Answer{}, errors.New("no link to the bus is open")
}
asker, ok := bus.(ToolBus)
if !ok {
return health.Answer{}, errors.New("the link to the bus open now cannot ask a tool")
}
reply, err := asker.Ask(ctx, ToolSubject(module, tool, q.Membership.Node), []byte("{}"))
if err != nil {
return health.Answer{}, err
}
return ReadToolAnswer(reply)
}
// AskController asks the controller a line mesh-cli was given on this machine, on this node's own subject, over
// the link open now (novox/hq ADR 0272 §3), and returns its answer as it came. No link open is said, not waited for.
func (q *Session) AskController(ctx context.Context, body []byte) ([]byte, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return nil, fmt.Errorf("%w: this machine has no link to the bus open now", ErrNotAsked)
}
asker, ok := bus.(ToolBus)
if !ok {
return nil, fmt.Errorf("%w: the link to the bus open now cannot ask the controller", ErrNotAsked)
}
return asker.Ask(ctx, CLISubject(q.Membership.Node), body)
}
// ErrNotAsked is a request the link knows it never delivered: no link open, or nobody listening on its subject. Any
// other failure of a request may come after it was delivered, so what came of it is not known (novox/hq ADR 0272).
var ErrNotAsked = errors.New("not asked")
// ReadToolAnswer reads a tool's reply envelope — `{result, error}`, as the node tools answer every call —
// as a health answer, through the SDK's reader (novox/hq ADR 0283): healthy, unhealthy, or waits for the
// operator with what it waits for. An error, and a result that is not an answer, are unhealthy; a wait the
// SDK cannot check is unhealthy too, so a malformed one never excuses anything.
func ReadToolAnswer(reply []byte) (health.Answer, error) {
var env struct {
Result json.RawMessage `json:"result"`
Error string `json:"error"`
}
if err := json.Unmarshal(reply, &env); err != nil {
return health.Answer{}, fmt.Errorf("its answer is not one: %w", err)
}
if env.Error != "" {
return health.Unhealthy(env.Error), nil
}
a, err := health.Read(env.Result)
if err != nil {
return health.Unhealthy("it answered something other than healthy or not"), nil
}
return a, nil
}
// SeatToolSubject is where a node seat's verb is asked of one machine's holder (design 33 §4): the seat's verb subject
// with the machine as its last token.
func SeatToolSubject(seat, verb, node string) string {
return "mesh.seat." + seat + ".tool." + verb + "." + node
}
// AskSeat asks the holder of a seat on this node a verb, over the link open now, and answers its result, or its refusal
// as an error. The one call the node agent makes of another part: the backup before a data module moves (ADR 0429 §14).
func (q *Session) AskSeat(ctx context.Context, seat, verb string, args any, within time.Duration) (json.RawMessage, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return nil, fmt.Errorf("%w: this machine has no link to the bus open now", ErrNotAsked)
}
asker, ok := bus.(ToolBus)
if !ok {
return nil, fmt.Errorf("%w: the link to the bus open now cannot ask a seat", ErrNotAsked)
}
body, err := json.Marshal(args)
if err != nil {
return nil, err
}
asking, cancel := context.WithTimeout(ctx, within)
defer cancel()
reply, err := asker.Ask(asking, SeatToolSubject(seat, verb, q.Membership.Node), body)
if err != nil {
return nil, fmt.Errorf("%s.%s could not be asked: %w", seat, verb, err)
}
var env struct {
Result json.RawMessage `json:"result"`
Error string `json:"error"`
}
if err := json.Unmarshal(reply, &env); err != nil {
return nil, fmt.Errorf("%s.%s answered with something that is not a tool's answer", seat, verb)
}
if env.Error != "" {
return nil, errors.New(env.Error)
}
return env.Result, nil
}
+164
View File
@@ -0,0 +1,164 @@
package link
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
)
// quietLink is a link with no broker behind it. A report is refused once the context it is published on is cancelled,
// as the bus's is ("reporting: context canceled"), or while refusing.
type quietLink struct {
mu sync.Mutex
reports []Report
refusing bool
lost chan error
}
func newQuietLink() *quietLink { return &quietLink{lost: make(chan error, 1)} }
func (l *quietLink) Report(ctx context.Context, _ string, body []byte) error {
if err := ctx.Err(); err != nil {
return err
}
l.mu.Lock()
defer l.mu.Unlock()
if l.refusing {
return errors.New("refused")
}
var r Report
if err := json.Unmarshal(body, &r); err != nil {
return err
}
l.reports = append(l.reports, r)
return nil
}
func (l *quietLink) Lost() <-chan error { return l.lost }
func (l *quietLink) Close() {}
func (l *quietLink) said() []Report {
l.mu.Lock()
defer l.mu.Unlock()
return append([]Report(nil), l.reports...)
}
func aMember(t *testing.T) (Membership, ed25519.PrivateKey) {
t.Helper()
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
return Membership{Node: "n1", Signer: public}, private
}
// R6-A AC7: the facts event carries the node-engine's own writer, epoch and sequence (novox/hq ADR 0227 §2, ADR 0539
// §9): one epoch per start, a sequence one higher for every event, so the controller can refuse older facts.
func TestTheFactsEventCarriesTheNodeEnginesWriterEpochAndSequence(t *testing.T) {
m, _ := aMember(t)
l := newQuietLink()
s := &Session{Membership: m, Timeout: time.Second}
s.Attach(context.Background(), l)
if !s.SayFacts(context.Background(), Report{Firewall: "none"}) || !s.SayFacts(context.Background(), Report{}) {
t.Fatal("the facts were not taken")
}
got := l.said()
if len(got) != 2 {
t.Fatalf("%d facts events were said", len(got))
}
for _, r := range got {
if r.Writer != Writer || r.Node != "n1" || r.Epoch <= 0 {
t.Fatalf("the facts do not say who wrote them: %+v", r)
}
}
if got[0].Epoch != got[1].Epoch || got[1].Sequence != got[0].Sequence+1 || got[0].Sequence <= 0 {
t.Fatalf("the order is not one epoch with a rising sequence: %+v then %+v", got[0].Order, got[1].Order)
}
// A later start orders after this one with nothing kept on disk.
time.Sleep(2 * time.Millisecond)
later := &Session{Membership: m, Timeout: time.Second}
later.Attach(context.Background(), l)
later.SayFacts(context.Background(), Report{})
if last := l.said()[2]; last.Epoch <= got[1].Epoch {
t.Fatalf("a later start has epoch %d, not above %d", last.Epoch, got[1].Epoch)
}
}
// Facts made with no link open are not kept, and nothing is said: the next link says them as they are then.
func TestFactsWithNoLinkOpenAreNotSaid(t *testing.T) {
m, _ := aMember(t)
l := newQuietLink()
s := &Session{Membership: m, Timeout: time.Second}
if s.SayFacts(context.Background(), Report{}) {
t.Fatal("facts were said with no link")
}
s.Attach(context.Background(), l)
s.Detach(l)
if s.Linked() || s.SayFacts(context.Background(), Report{}) || len(l.said()) != 0 {
t.Fatal("facts were said on a link that was let go")
}
}
// A link opening says the node's facts, off the link's own loop.
func TestALinkOpeningSaysTheFacts(t *testing.T) {
m, _ := aMember(t)
l := newQuietLink()
opened := make(chan struct{}, 1)
s := &Session{Membership: m, Timeout: time.Second, OnLink: func(context.Context) { opened <- struct{}{} }}
s.Attach(context.Background(), l)
select {
case <-opened:
case <-time.After(2 * time.Second):
t.Fatal("a link opening did not ask for the facts")
}
}
// aBus is a connection to a real server, because some claims here are about one: set MESH_TEST_NATS to a server with
// JetStream, or the test is skipped.
func aBus(t *testing.T) (*nats.Conn, nats.JetStreamContext) {
t.Helper()
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
conn, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
js, err := conn.JetStream()
if err != nil {
t.Fatal(err)
}
want := &nats.StreamConfig{Name: "CONTROL", Subjects: []string{"mesh.control.*.report", "mesh.control.enrol"},
Retention: nats.WorkQueuePolicy}
if _, err := js.StreamInfo(want.Name); err != nil {
if _, err := js.AddStream(want); err != nil {
t.Fatal(err)
}
}
if err := js.PurgeStream(want.Name); err != nil {
t.Fatal(err)
}
return conn, js
}
// eventuallyHeld waits for a condition the bus reaches on its own time.
func eventuallyHeld(t *testing.T, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
for !ok() {
if time.Now().After(deadline) {
t.Fatal(what)
}
time.Sleep(5 * time.Millisecond)
}
}
+68
View File
@@ -0,0 +1,68 @@
package link
import (
"errors"
"strings"
"time"
"github.com/nats-io/nats.go"
)
// A transient failure of the bus: it did not answer for a moment, and the same request a little later is answered. The
// bus answers so while it reloads its accounts after the bus node's engine wrote its user list (novox/hq ADR 0547 §8,
// about three seconds: every JetStream request then meets "no responders"), and while a link reconnects. A part that
// meets one is retried, never left failed until a key changes (2026-10-25: mesh-vault on the bus node stayed failed at
// 23:04:24, and with it every module that needs the secret seat).
// transientErrs are the bus client's errors of a bus that does not answer for a moment.
var transientErrs = []error{
nats.ErrNoResponders, nats.ErrTimeout, nats.ErrConnectionClosed, nats.ErrConnectionReconnecting,
nats.ErrDisconnected, nats.ErrStaleConnection, nats.ErrJetStreamNotEnabled, nats.ErrJetStreamNotEnabledForAccount,
nats.ErrNoStreamResponse,
}
// transientWords are the same as words, for a failure that reaches its reader only as text (an apply's refusal).
var transientWords = []string{
"no responders available for request", "nats: timeout", "nats: connection closed", "nats: connection reconnecting",
"nats: server is disconnected", "nats: stale connection", "jetstream not enabled", "no response from stream",
"jetstream system temporarily unavailable",
}
// Transient says err is the bus not answering for a moment, by the client's error or its words.
func Transient(err error) bool {
if err == nil {
return false
}
for _, e := range transientErrs {
if errors.Is(err, e) {
return true
}
}
return TransientWords(err.Error())
}
// TransientWords says a failure's words are those of the bus not answering for a moment. A refused permission, a digest
// that does not match or anything the module's own apply says is not.
func TransientWords(s string) bool {
s = strings.ToLower(s)
for _, w := range transientWords {
if strings.Contains(s, w) {
return true
}
}
return false
}
// backoffs are the waits before the first retries of a transient failure; every later one waits the last.
var backoffs = []time.Duration{2 * time.Second, 5 * time.Second, 15 * time.Second, time.Minute, 5 * time.Minute}
// Backoff is the wait before the try-th retry (1, 2, …) of a transient failure: 2 s, 5 s, 15 s, 60 s, then every 5 min.
func Backoff(try int) time.Duration {
if try < 1 {
try = 1
}
if try > len(backoffs) {
return backoffs[len(backoffs)-1]
}
return backoffs[try-1]
}
-217
View File
@@ -1,217 +0,0 @@
package link
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"reflect"
"sync"
"testing"
"time"
)
// A report of an apply reaches the mesh even when the apply ended the link, and one lost anyway is
// said again the next time the node is linked (novox/hq issue 264).
// quietLink is a link with no broker behind it. A report is refused once the context it is
// published on is cancelled, as the bus's is ("reporting: context canceled"), or while refusing.
type quietLink struct {
mu sync.Mutex
reports []Report
refusing bool
delivered chan Declaration
lost chan error
}
func newQuietLink(arriving ...Declaration) *quietLink {
l := &quietLink{delivered: make(chan Declaration, len(arriving)), lost: make(chan error, 1)}
for _, d := range arriving {
l.delivered <- d
}
return l
}
func (l *quietLink) Report(ctx context.Context, _ string, body []byte) error {
if err := ctx.Err(); err != nil {
return err
}
l.mu.Lock()
defer l.mu.Unlock()
if l.refusing {
return errors.New("refused")
}
var r Report
if err := json.Unmarshal(body, &r); err != nil {
return err
}
l.reports = append(l.reports, r)
return nil
}
func (l *quietLink) Declarations() <-chan Declaration { return l.delivered }
func (l *quietLink) Lost() <-chan error { return l.lost }
func (l *quietLink) Close() {}
func (l *quietLink) said() []Report {
l.mu.Lock()
defer l.mu.Unlock()
return append([]Report(nil), l.reports...)
}
// keptInMemory is Unsaid without a disk.
type keptInMemory struct {
mu sync.Mutex
report *Report
}
func (k *keptInMemory) Keep(r Report) error {
k.mu.Lock()
defer k.mu.Unlock()
if r.Declared == "" {
k.report = nil
return nil
}
k.report = &r
return nil
}
func (k *keptInMemory) Said(declared string) error {
k.mu.Lock()
defer k.mu.Unlock()
if k.report != nil && k.report.Declared == declared {
k.report = nil
}
return nil
}
func (k *keptInMemory) Pending() (Report, bool, error) {
k.mu.Lock()
defer k.mu.Unlock()
if k.report == nil {
return Report{}, false, nil
}
return *k.report, true, nil
}
func aMember(t *testing.T) (Membership, ed25519.PrivateKey) {
t.Helper()
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
return Membership{Node: "n1", Signer: public}, private
}
// Measured on the anchor: a declaration carried a new controller and a new host, the host applied
// it, stood aside for its successor — which cancels the link — and the report of that apply failed
// with "context canceled". The plan waited on it until somebody pushed by hand.
//
// R3 of novox/hq to-be 45 §9, as a unit: the node-engine self-updates during its report, and the
// report still arrives — on the link the apply came over, before it is let go.
func TestAnApplyThatEndsTheLinkIsStillReported(t *testing.T) {
m, key := aMember(t)
declaration := &said{body: signedBy(t, key, []byte(`{"declaration":1}`))}
behind := &said{body: signedBy(t, key, []byte(`{"declaration":2}`))}
l := newQuietLink(declaration)
kept := &keptInMemory{}
ctx, standAside := context.WithCancel(context.Background())
defer standAside()
applied := 0
q := aQueue(m, func(context.Context, []byte, []byte) Report {
applied++
standAside() // the host delivered its successor, and stands aside for it
return Report{Declared: "d1", Applied: []string{"a"}}
}, kept)
worker := runWorker(ctx, q)
done := make(chan error, 1)
go func() { done <- serve(ctx, l, m, q, nil, time.Second) }()
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(5 * time.Second):
t.Fatal("the link did not end after the apply stood aside")
}
// Something delivered after the host stood aside is not applied by it: the successor will be
// sent it again.
q.Deliver(behind)
<-worker
reports := l.said()
if len(reports) != 1 || reports[0].Declared != "d1" || !reflect.DeepEqual(reports[0].Applied, []string{"a"}) {
t.Fatalf("the apply's report did not reach the mesh: %+v", reports)
}
if !declaration.wasHandled() {
t.Fatal("the declaration was not settled after its report")
}
if applied != 1 || behind.wasHandled() {
t.Fatalf("%d applies, and the declaration behind settled %v: after standing aside nothing "+
"further is applied", applied, behind.wasHandled())
}
if _, ok, _ := kept.Pending(); ok {
t.Fatal("a report the mesh took is still kept to be said again")
}
}
// A report that did not reach the mesh — the host died between applying and publishing, or the
// broker refused it — is said on the next link, before anything else, and exactly as it was made.
func TestAReportLostAfterTheApplyIsSaidOnTheNextLink(t *testing.T) {
m, key := aMember(t)
made := Report{Declared: "d1", Applied: []string{"a"}, Failed: map[string]string{"b": "no room"}}
kept := &keptInMemory{}
// The apply happens and its report is lost.
first := newQuietLink(&said{body: signedBy(t, key, []byte(`{"declaration":1}`))})
first.refusing = true
ctx, stop := context.WithCancel(context.Background())
q := aQueue(m, func(context.Context, []byte, []byte) Report { stop(); return made }, kept)
worker := runWorker(ctx, q)
if err := serve(ctx, first, m, q, nil, time.Second); err != nil {
t.Fatal(err)
}
<-worker
if len(first.said()) != 0 {
t.Fatal("the broker refused the report and it counted as said")
}
lost, ok, _ := kept.Pending()
if !ok || lost.ReportSequence != 1 {
t.Fatalf("the lost report was not kept as it was made: %+v", lost)
}
// The next host links. It is stopped at once, so all it does is what it does on linking.
next := newQuietLink()
gone, cancel := context.WithCancel(context.Background())
cancel()
never := func(context.Context, []byte, []byte) Report {
t.Fatal("nothing was delivered, and something was applied")
return Report{}
}
if err := serve(gone, next, m, aQueue(m, never, kept), nil, time.Second); err != nil {
t.Fatal(err)
}
reports := next.said()
if len(reports) != 1 || !reflect.DeepEqual(reports[0], lost) {
t.Fatalf("the lost report was not said again as it was made: %+v", reports)
}
if _, ok, _ := kept.Pending(); ok {
t.Fatal("a report said again is still kept")
}
}
// A node that never applied anything, or whose last report was taken, says nothing on linking.
func TestNothingIsSaidAgainWhenNothingWasLost(t *testing.T) {
m, _ := aMember(t)
l := newQuietLink()
gone, cancel := context.WithCancel(context.Background())
cancel()
if err := serve(gone, l, m, aQueue(m, nil, &keptInMemory{}), nil, time.Second); err != nil {
t.Fatal(err)
}
if err := serve(gone, l, m, aQueue(m, nil, nil), nil, time.Second); err != nil {
t.Fatal(err)
}
if reports := l.said(); len(reports) != 0 {
t.Fatalf("a node with nothing lost reported %+v", reports)
}
}
+241
View File
@@ -0,0 +1,241 @@
package link
import (
"context"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
)
// KVUpdate is one change of a key in a bucket the node-engine watches (novox/hq ADR 0539 §9). Caught marks the end of
// the keys that were there when the watch began.
type KVUpdate struct {
Key string
Value []byte
Revision uint64
Deleted bool
Caught bool
}
// KVBus is a link that can watch a bucket's keys: the node agent follows its wants and its compositions with it, and
// never polls.
type KVBus interface {
// KVWatch yields the keys of the bucket matching the pattern now, a Caught update, then every change, until ctx ends
// or the link does (the channel is then closed). A bucket the node's account may not read is an error.
KVWatch(ctx context.Context, bucket, pattern string) (<-chan KVUpdate, error)
}
// KVWatch is the watch on the link open now.
func (b OverNATS) KVWatch(ctx context.Context, bucket, pattern string) (<-chan KVUpdate, error) {
js := b.JS
if b.Conn != nil {
bound, err := b.Conn.JetStream()
if err != nil {
return nil, err
}
js = bound
}
kv, err := js.KeyValue(bucket)
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", bucket, err)
}
w, err := kv.Watch(pattern, nats.Context(ctx))
if err != nil {
return nil, fmt.Errorf("watching %s in %s: %w", pattern, bucket, err)
}
out := make(chan KVUpdate, 64)
go func() {
defer close(out)
defer func() { _ = w.Stop() }()
for {
select {
case <-ctx.Done():
return
case e, ok := <-w.Updates():
if !ok {
return
}
var u KVUpdate
if e == nil {
u.Caught = true
} else {
u = KVUpdate{Key: e.Key(), Value: e.Value(), Revision: e.Revision(),
Deleted: e.Operation() == nats.KeyValueDelete || e.Operation() == nats.KeyValuePurge}
}
select {
case out <- u:
case <-ctx.Done():
return
}
}
}
}()
return out, nil
}
func (l *natsLink) KVWatch(ctx context.Context, bucket, pattern string) (<-chan KVUpdate, error) {
return OverNATS{Conn: l.conn, JS: l.js}.KVWatch(ctx, bucket, pattern)
}
var (
_ KVBus = OverNATS{}
_ KVBus = (*natsLink)(nil)
)
// KVHistoryBus is a link that can read the revisions a bucket keeps of one key: the node agent takes the composition its
// want names from them when a newer one replaced it (novox/hq ADR 0539 clause 1).
type KVHistoryBus interface {
// KVHistory is the key's revisions, oldest first; none when the key has none.
KVHistory(ctx context.Context, bucket, key string) ([]KVUpdate, error)
}
// historyWithin bounds one read of a key's revisions.
const historyWithin = 10 * time.Second
// KVHistory reads the key's revisions on the link open now.
func (b OverNATS) KVHistory(ctx context.Context, bucket, key string) ([]KVUpdate, error) {
js := b.JS
if b.Conn != nil {
bound, err := b.Conn.JetStream()
if err != nil {
return nil, err
}
js = bound
}
kv, err := js.KeyValue(bucket)
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", bucket, err)
}
// Bounded on its own: a read that does not answer must not hold the node agent (the review of issue 1945).
ctx, cancel := context.WithTimeout(ctx, historyWithin)
defer cancel()
entries, err := kv.History(key, nats.Context(ctx))
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("the revisions of %s in %s: %w", key, bucket, err)
}
out := make([]KVUpdate, 0, len(entries))
for _, e := range entries {
out = append(out, KVUpdate{Key: e.Key(), Value: e.Value(), Revision: e.Revision(),
Deleted: e.Operation() == nats.KeyValueDelete || e.Operation() == nats.KeyValuePurge})
}
return out, nil
}
func (l *natsLink) KVHistory(ctx context.Context, bucket, key string) ([]KVUpdate, error) {
return OverNATS{Conn: l.conn, JS: l.js}.KVHistory(ctx, bucket, key)
}
var (
_ KVHistoryBus = OverNATS{}
_ KVHistoryBus = (*natsLink)(nil)
)
// KVHistory reads a key's revisions on the link open now.
func (q *Session) KVHistory(ctx context.Context, bucket, key string) ([]KVUpdate, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
h, ok := bus.(KVHistoryBus)
if bus == nil || !ok {
return nil, ErrNoLink
}
return h.KVHistory(ctx, bucket, key)
}
// ErrNoLink is what a watch asks for with no link open.
var ErrNoLink = errors.New("no link to the bus is open")
// KVWatch watches a bucket on the link open now. It ends when ctx does or the link closes.
func (q *Session) KVWatch(ctx context.Context, bucket, pattern string) (<-chan KVUpdate, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
w, ok := bus.(KVBus)
if bus == nil || !ok {
return nil, ErrNoLink
}
return w.KVWatch(ctx, bucket, pattern)
}
// WhenLinked waits for a link to be open and returns a context that ends when that link does (or ctx does): what the
// node agent runs under, so its watches end with the link and begin again with the next.
func (q *Session) WhenLinked(ctx context.Context) (context.Context, context.CancelFunc, error) {
q.init()
for {
q.mu.Lock()
if q.bus != nil {
gone := q.gone
q.mu.Unlock()
linked, cancel := context.WithCancel(ctx)
go func() {
select {
case <-gone:
cancel()
case <-linked.Done():
}
}()
return linked, cancel, nil
}
changed := q.changed
q.mu.Unlock()
select {
case <-ctx.Done():
return nil, nil, ctx.Err()
case <-changed:
}
}
}
// EventBus is a link that can hear an event on a subject of core NATS: the node agent waits for a backup's end with it,
// and never polls.
type EventBus interface {
// Hear yields the payloads published on the subject until stop is called or the link ends.
Hear(subject string) (events <-chan []byte, stop func(), err error)
}
func (b OverNATS) Hear(subject string) (<-chan []byte, func(), error) {
out := make(chan []byte, 16)
sub, err := b.Conn.Subscribe(subject, func(m *nats.Msg) {
select {
case out <- m.Data:
default:
}
})
if err != nil {
return nil, nil, err
}
if err := b.Conn.Flush(); err != nil {
_ = sub.Unsubscribe()
return nil, nil, err
}
return out, func() { _ = sub.Unsubscribe() }, nil
}
func (l *natsLink) Hear(subject string) (<-chan []byte, func(), error) {
return OverNATS{Conn: l.conn, JS: l.js}.Hear(subject)
}
var (
_ EventBus = OverNATS{}
_ EventBus = (*natsLink)(nil)
)
// Hear listens for an event on the link open now, from before the caller asks for what it waits on.
func (q *Session) Hear(subject string) (<-chan []byte, func(), error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
h, ok := bus.(EventBus)
if bus == nil || !ok {
return nil, nil, ErrNoLink
}
return h.Hear(subject)
}
+1 -1
View File
@@ -48,7 +48,7 @@ func DirectGetSubject(bucket, key string) string {
func PingSubject(service, id string) string { return "$SRV.PING." + service + "." + id }
// Asker is the link open now, when there is one and it can ask.
func (q *Queue) Asker() Asker {
func (q *Session) Asker() Asker {
q.init()
q.mu.Lock()
defer q.mu.Unlock()
+5 -1
View File
@@ -172,6 +172,9 @@ type State struct {
Restarts int `json:"restarts,omitempty"`
// Waits is what a resource stated Waiting waits for (ADR 0283).
Waits []health.Wait `json:"waits,omitempty"`
// Missing is a resource the runtime or the service manager does not have at all after its grace: removed, not
// stopped. The node agent applies its module again (issue 1945).
Missing bool `json:"missing,omitempty"`
}
// CheckOf and NeedsOf are a stated resource's declared check, in a word, and the provision it exercises.
@@ -358,7 +361,8 @@ func (j *Judge) Look(ctx context.Context) (Statement, bool) {
j.judge(k, o, now, r.Kind == KindContainer && held[r.Target], blind, why)
seen[r.ID] = true
st.Resources = append(st.Resources, State{Resource: k.Resource, State: k.State, Reason: k.Reason, Since: k.Since,
Streak: k.Streak, Restarts: k.Counted, Waits: waitsOf(k)})
Streak: k.Streak, Restarts: k.Counted, Waits: waitsOf(k),
Missing: k.State == Unhealthy && k.Reason == ReasonDown && !blind && !o.Found})
// What it waits for is part of what is said: a change of waits alone is a change.
word := k.State + "/" + k.Reason + "/" + waitsWord(waitsOf(k))
if j.said[r.ID] != word {
+25
View File
@@ -383,3 +383,28 @@ func TestAUserUnitsManagerNotRunningIsNotReadAndNotDown(t *testing.T) {
t.Fatalf("the running manager is read in the account's own: %q", last)
}
}
// A resource the runtime does not have at all after its grace is said missing, beside down: what the node agent applies
// again (issue 1945). One that is there and stopped is down and not missing, and one that cannot be read is neither.
func TestAResourceNotThereIsMissingAndOneStoppedIsNot(t *testing.T) {
c := &clock{now: t0}
rt := &fakeRuntime{containers: map[string]Observed{}}
j := aJudge(t, rt, c, filepath.Join(t.TempDir(), FileName))
j.Set([]Resource{server})
j.Look(t.Context())
c.now = t0.Add(2 * time.Minute)
st, _ := j.Look(t.Context())
if s := stateOf(t, st, "letta.server"); !s.Missing {
t.Fatalf("a container not there after its grace is not missing: %+v", s)
}
rt.containers["letta-server"] = Observed{Found: true, Identity: "c1"}
st, _ = j.Look(t.Context())
if s := stateOf(t, st, "letta.server"); s.Missing {
t.Fatalf("a container there and stopped: %+v", s)
}
rt.err = errors.New("cannot connect")
st, _ = j.Look(t.Context())
if s := stateOf(t, st, "letta.server"); s.Missing {
t.Fatalf("a runtime that does not answer says nothing is missing: %+v", s)
}
}
+20
View File
@@ -40,6 +40,11 @@ const (
// controller sends the field only to a node that reports it, since an older node-engine refuses the key.
CapContainerGroups = "container-groups"
// CapReadsWants says this node-engine follows its wants (novox/hq ADR 0539 §9): it watches `want.<node>.>` and its
// compositions and applies one module at a time, and hears no declaration. A fact about the program: the controller
// stops sending a node declarations only once it reports it.
CapReadsWants = "reads-wants"
// The network manager this machine runs, one capability per dialect (novox/hq ADR 0161): the
// uplink seat's holder declares its own, so the holder for a manager the machine does not run
// is refused the way any missing capability is, naming it. Active, not installed — a machine
@@ -190,6 +195,20 @@ func (containerGroups) Detect(context.Context) Verdict {
}
}
// readsWants reports that this node-engine follows its wants. True of the program that is running, so it is always
// present: its absence on a node is what tells the controller the node-engine still waits for declarations.
type readsWants struct{}
func (readsWants) Name() string { return CapReadsWants }
func (readsWants) Detect(context.Context) Verdict {
return Verdict{
Name: CapReadsWants, Present: true,
Detail: "follows its wants and applies one module's composition at a time; hears no declaration",
How: "a fact about this program, not probed: the node-engine that answers follows its wants",
}
}
// graphicalSession reports whether anything could display a window here.
//
// Environment rather than a probe, because a display server is reachable through a socket a
@@ -230,6 +249,7 @@ func Default(runner Runner) []Detector {
privileged{},
fileReferences{},
containerGroups{},
readsWants{},
graphicalSession{},
seat{},
battery{},
+22
View File
@@ -0,0 +1,22 @@
package profile
import (
"context"
"testing"
"time"
)
// R6-A: the node's profile carries the capability that says the node-engine follows its wants, which the controller
// checks before it stops sending the node declarations (novox/hq ADR 0539 §9).
func TestTheProfileSaysTheNodeEngineReadsItsWants(t *testing.T) {
got := Detect(context.Background(), Default(func(context.Context, string, ...string) (string, error) {
return "", nil
}), time.Second)
v := verdictOf(t, got, CapReadsWants)
if !v.Present || v.Detail == "" || v.How == "" {
t.Errorf("reads-wants = %+v, want present with a detail and a method", v)
}
if CapReadsWants != "reads-wants" {
t.Errorf("the capability is named %q", CapReadsWants)
}
}
-94
View File
@@ -1,94 +0,0 @@
package store
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
)
// What the node-engine counts about what it says, kept so the counting outlives the process.
//
// **The report sequence must increase across restarts and self-updates** (novox/hq to-be 45 §6). The
// mesh keeps, per machine, the highest report it accepted and refuses an older one; a host that began
// again from one after every restart — and a self-update is a restart — would have every report it
// made refused until it had counted past where its predecessor stopped. So the number is kept beside
// the state, where the successor reads it.
//
// And the count of declarations refused as older than what this node applied (rule 2), which the
// mesh's stale-writer watchdog reads from every report.
// NumbersName is where they live, beside the state.
const NumbersName = "numbers.json"
// NumbersPath is where the numbers live, given where the state lives.
func NumbersPath(statePath string) string {
return filepath.Join(filepath.Dir(statePath), NumbersName)
}
// Numbers is what is kept.
type Numbers struct {
// ReportSequence is the number of the last report this node made.
ReportSequence int64 `json:"report_sequence"`
// RefusedOlder is how many declarations it has refused as older, ever.
RefusedOlder int64 `json:"refused_older"`
}
// ReadNumbers is what was kept, or zero when nothing was — a node that has never reported.
//
// **Unreadable is an error, never zero** (novox/hq ADR 0227, rule 4): read as zero, every report the
// node makes next would be older than the ones it already made, and refused.
func ReadNumbers(path string) (Numbers, error) {
raw, err := os.ReadFile(path)
if errors.Is(err, os.ErrNotExist) {
return Numbers{}, nil
}
if err != nil {
return Numbers{}, err
}
var n Numbers
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&n); err != nil {
return Numbers{}, fmt.Errorf("the numbers kept at %s are unreadable: %w", path, err)
}
if n.ReportSequence < 0 || n.RefusedOlder < 0 {
return Numbers{}, fmt.Errorf("the numbers kept at %s are below zero, which nothing counting writes", path)
}
return n, nil
}
// SaveNumbers keeps them, replacing what was kept, and is on disk when it returns: a number used and
// not kept is one the next host would use again.
func SaveNumbers(path string, n Numbers) error {
raw, err := json.Marshal(n)
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(path), ".numbers-*")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
if err := tmp.Chmod(0o600); err != nil {
tmp.Close()
return err
}
if _, err := tmp.Write(raw); err != nil {
tmp.Close()
return err
}
if err := tmp.Sync(); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
return os.Rename(tmp.Name(), path)
}

Some files were not shown because too many files have changed in this diff Show More