Author SHA1 Message Date
jschoubben ae4b86e333 route-proxy's proof skips aloud when mesh-networking is not beside, since the running checker places none for the node-engine until this release is delivered
mesh/delivery-group group feat/sdk1 checking: 7 of 7 member(s) ready
mesh/delivery ready: it is merged on the Approve
mesh/check checking ae4b86e33300 on 98fbc490a4c9 | novox/mesh-host#198 in group feat/sdk1
With mesh-networking beside, or with nothing beside and the captured copy read, it still judges the manifest.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:39:56 +01:00
jschoubben 8f1e5f2547 Stop judging the controller's manifest at genesis's step 9: the controller's repository is emptied
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
mesh/delivery-group group feat/sdk1 checking: 6 of 7 member(s) ready
mesh/delivery closed: a newer head of the same pull request
The test read mesh-controller/module.json beside the check, which no longer exists after the
controller's removal (issue 1126, ADR 0593 §11), so it would fail every check of the node-engine.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:23:02 +01:00
jschoubben 60518f532d Pin mesh-sdk at f02f142, the sdk1 head whose tests no longer leak package globals, so the group's check builds against it
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:14:16 +01:00
jschoubben cdd17eaae8 Pin mesh-sdk at the sdk1 group's head 1c7356c, so the group's check builds every module against the narrowed SDK it was written for
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
mesh/delivery-group group feat/sdk1 checking: 7 of 7 member(s) ready
mesh/delivery closed: a newer head of the same pull request
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:29:53 +01:00
jschoubben 3293f6ddcd Take the environment's rendering integration tests with the code they test
ADR 0593 §7: the environment's rendering left mesh-sdk for the node-engine,
so its two integration tests (the POSIX environment sourced twice, and
the service manager reading the systemd rendering) move with it.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:21:10 +01:00
jschoubben 1ada760a64 Hold the proofs that a manifest composes, which left the judge's and route-proxy's tests
ADR 0593 §7: the composition is the node-engine's, so its tests hold
what the indexer's judge tests and route-proxy's tests proved by
composing: a bundle's words per machine, a Go bundle's delivery, the
credential form in a handed plan, settings refusals, placeholder and
vault refusals, the session's files, a field set aside composed whole,
route-proxy's manifest composing for a machine, a route endpoint's
machine port in six cases and the plan's route entry end to end. Two
table tests, one subtest per item; route-proxy's manifest is captured
beside the catalogue.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:19:49 +01:00
jschoubben d68e241724 Take the catalogue's composition rules into the node-engine
ADR 0593 §7: the composer with its refusals leaves mesh-sdk, so mesh-sdk
carries no composition rule. The catalogue functions only the
node-engine's composition reaches (Resolve, Resolution's Compose,
Declaration, DataItems, LeftOut, Rules and the rest, the settings,
placeholder and template refusals, the rendering helpers) are copied
with their tests into internal/compose/resolution; a method of a
mesh-sdk shape becomes a function taking it first. The composition
calls its own copies, and the captured catalogue its tests read lives
here now, shared by mesh-compose's tests.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:15:55 +01:00
jschoubben be0226b624 Stop verifying composition signatures: the node applies only its own
ADR 0593 §6: a composition read from composed-<node> is a record never
applied, and the engine applies only what it composes itself, so no
composition carries a signature and nothing is verified. The agent's
trusted signers, its signed-composition test and the unused signing
helper of the integration bed go, so mesh-sdk can drop SignComposed,
VerifyComposed and Composed.Signature.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:01:12 +01:00
jschoubben acb720d2bf Merge five moved overlay tests into two, offsetting the tests this release adds
The test budget: mesh-host is over 500 unit tests, so the three tests
added for ADR 0593 §6 and §7 are paid for by folding the private
network's hub refusals, its MTU cases and its no-self-peer check into
table-driven tests, each case kept.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:39:51 +01:00
jschoubben f6bda12a32 The node whole publishes its compositions and answers a candidate (H6)
ADR 0593 §6, §7, §9: the integration test runs the node agent against
the real bus under the grant the grants part composes for a node: it
publishes web's composition in composed-<node> with the digest its have
names, and answers compose-candidate for the indexer's candidate,
changing nothing. A node that cannot compose now answers unready, so
the checker counts it as no answer by the verdict and never by words.
The golden grant reads the indexer's candidates and verdicts as it
reads the builder's builds.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:39:12 +01:00
jschoubben 4b8b5c7329 The composition verb reads the revisions the engine publishes (H5)
ADR 0593 §6: composed-<node> is written again, by the node's engine,
each composition with its plan. The verb composition finds a published
revision by the digest the have and want name, its resources', so a plan
on the record does not hide it. The golden node grant also reads the
indexer's verdicts, which compose-candidate takes for a candidate's
manifests.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:35:00 +01:00
jschoubben 3fa41dfe1a Serve node-engine.compose-candidate from the node's shadow (H4)
ADR 0593 §7: a check's composition is the checker's, made from each
placed node's answer, since only a node can compose with its own sealed
values. The node composes the indexer's candidate in shadow on a copy of
its shadow's inputs, the candidate's modules laid over the indexer's
with their manifest verdicts, and answers per module composed (changed
or not), refused or held, and its own composition's verdict, within the
answer bound; it applies, writes and hands nothing. With input it also
answers the composition input, for the sandbox path.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:33:58 +01:00
jschoubben 4f51393b72 Publish every composition the node makes in composed-<node> (H3)
ADR 0593 §6: the provisioner, the assigner and the view read compositions
instead of composing, so the node's engine writes each one it makes to
its composition state: the wanted builds, every placed module at its
newest done build (so the plan exists before the want moves), each with
its provisioning plan, and a module refused alone with its refusal. A
key is written only when it differs, and kept while among the module's
five newest or named by a want. A placed module's trunk build landing
now makes a pass.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:29:20 +01:00
jschoubben 9a12f2851f The node-engine owns the composition it alone runs (H1, H2)
ADR 0593 §1 and §7: compose, its bus reader and runtime reader, takein,
validate, the overlay generator, the composer's run and mesh-compose, and
the judgement of a move leave mesh-sdk for the node-engine's internal
packages, so no other module composes and a change to them rebuilds only
the node-engine. The shapes others read (the composer's request and
verdict, the overlay's names, compositions and answers) stay in mesh-sdk.
The node's grant is now a golden file of what the grants part composes,
since the composer of the bus's user list leaves mesh-sdk too.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:24:05 +01:00
mesh-tools 98fbc490a4 Merge pull request 'The node-engine without the controller's stream and lease (issue 1126 group B)' (#197) from feat/1126-removal into main 2026-11-02 13:41:28 +00:00
jschoubben 049c3b8c11 Pin mesh-sdk to the group head of issue 1126
mesh/delivery merged: source.main moved to novox/mesh-host@98fbc490a4c9
mesh/delivery-group group feat/1126-removal merged: every member is merged
mesh/check checking 049c3b8c11c9 on 84737f4cb26d | novox/mesh-host#197 in group feat/1126-removal
Every member builds against mesh-sdk abf5eac, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:27:34 +01:00
jschoubben 813f3cae87 Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk f569924, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:20:14 +01:00
jschoubben 77c1b50019 Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk 729a86b, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:56:55 +01:00
jschoubben 6bcc4eab24 Say in the comments that the enrolment seat and the packages key stand alone now
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:37:11 +01:00
jschoubben 692b0f0160 Prove a rolled-back build is not placed again on the node tools, the one process judged by default
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:35:33 +01:00
jschoubben 897a5efe7c Name no controller signer or composer the SDK no longer holds
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:34:53 +01:00
jschoubben 98f4ff6163 Read the bus's user-list file's place from what the node runs, not from the controller's compositions
The bus module on the bus node runs from what its agent holds (its own composition, or the declaration kept before the
switch: every novox have read 2026-11-02 names its own composition or none, and nats names a build alone), while
composed-<node> was written by the controller, which ended (issue 1126). The follower now looks at the agent's record
of what runs whenever it changed, writes again only when the file it names moved, and trusts no composition signer.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:34:15 +01:00
jschoubben ec4fe14457 Remove what only served the controller's compositions: the own-secret resync, the shadow's wants watch and the vault key's follower
Every node composes its own since group A (issue 1126), so the resync that handed the vault an own secret's value
while a node applied the controller's compositions never runs, the shadow's answers read no want, and nothing follows
the heard vault key any more.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:29:47 +01:00
jschoubben 75eb2bb534 Say in the usage that overlay take is refused
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:28:36 +01:00
jschoubben 18521ba7af Remove the mesh-cli forward to the controller
nox now calls the seats' verbs through the console itself (mesh-catalog #593), and the controller the engine forwarded
its lines to ended (issue 1126), so the socket, its forward and the subject it asked on go. The kernel's word on a
peer moves into the filled-file socket, its one remaining reader.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:28:35 +01:00
jschoubben a9a0f7b4e0 Say the node's facts in its haves alone, and enrol through the seat alone
The CONTROL stream the facts, the enrolment and the overlay rekey were published into was the controller's, which ended
by its unassignment (issue 1126) and whose stream is removed after group B's delivery: a publish there would fail every
facts event before its haves are written. The facts go into the haves as they did after the report, the enrolment goes
through mesh-enrolment alone, and overlay take is refused because no seat takes a rekey now.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:27:28 +01:00
jschoubben f5ccbb2700 Open every link with no signer instead of following the controller's key
The controller that signed the hand-over, search, judgement, access and move asks ended (issue 1126), so a link
believes none of them by having no signer, and the per-ask key it was told to follow goes with the lease.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:24:05 +01:00
jschoubben 0d1bbe967d Answer the node seat's packages from the engine's own key alone
The controller's declared-packages mirror ended with the controller (issue 1126) and is deleted after group B's
delivery, so the verb no longer reads or compares it.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:22:54 +01:00
jschoubben ff4dc71ce4 Witness the node tools alone: the controller's lease names no build any more
The controller ended by its unassignment (issue 1126, ADR 0590), so no build is judged by the lease: the lease
witness, its contract and the direct get the witness asked it with are removed, and its tests now prove each rollback
path on the node tools' PING.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:22:21 +01:00
mesh-tools 84737f4cb2 Merge pull request 'The node-engine trusts the vault alone, loads the grants part's user list alone and applies only its own compositions (issue 1126)' (#196) from fix/1126-gates-host into main 2026-11-02 09:51:15 +00:00
jschoubben 38d666f223 Apply only the node's own compositions, never the controller's
mesh/delivery merged: source.main moved to novox/mesh-host@84737f4cb26d
mesh/check being carried to 38d666f22390 on 84737f4cb26d | from 14a12cc7b6f9
The controller ended by its unassignment, so a node that falls back to the
compositions in its bucket follows what nobody writes (issue 1126, A7). The
node agent now always runs its own composer: without a placement key it
composes nothing and keeps what runs, a placement it cannot watch stops the
agent and is said, and a candidate taking its placement away is refused
instead of answered as a return to the controller. The start-time placement
read, the end-when-placed watchers and the returning answer go, with their
tests and the three integration tests of the controller-composed path.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:40:50 +01:00
jschoubben 5cfc4ff4a7 Trust the vault's key alone and write the grants part's user list alone
The controller ended by its unassignment and its lease is never handed, so
the engine's lease gate kept trusting the controller's key and the bus node
kept the controller's list as a fallback nobody writes (issue 1126, A6).
The engine now trusts no controller signature, its hand-over line names the
node seat's hand-over, and the user-list follower writes only the grants
part's list: a key unreadable or deleted keeps the list loaded, and with no
key heard nothing is written. Compositions in the node's bucket name the
file's place alone. The guard no longer asks for the controller's user.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:37:27 +01:00
mesh-tools 14a12cc7b6 Merge pull request 'The engine dials as the login node-logins names, rotates it after a test-dial, and enrols through the enrolment seat (issue 774)' (#195) from feat/l3-enrol into main 2026-11-02 07:56:45 +00:00
jschoubben 44db32c31b Pin mesh-sdk at dff0ce2, the enrolment group's rebased head, so every member builds the same contract (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@14a12cc7b6f9
mesh/delivery-group group feat/l3-enrol merged: every member is merged
mesh/check failed (2 tests): integration: mesh-mail-reader/cmd/mesh-mail-reader.TestAcceptanceOneMatteringMailSendsExactlyOneTellNamingIt: timed out w…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:49:39 +01:00
jschoubben 12168ba4da The engine carries its login fall back in its health report, and says the enrolment fallback is the pre-lease path
A fall back to the old login during a rotation is now delivery.NodeHealth's Fallback (the node, the login fallen back from,
why, when), written with the node's health report, from which the watchman raises node-account-fallback; it is said no more
once the vault's key of the node moves on (blocker 7, ADR 0583 §6). mesh-host enrol's fall back to the controller's
enrolment subject says it is the path before the lease write, which the bus refuses once the lease is written. Tests: rows
in the existing rotation test; none added.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:49:01 +01:00
jschoubben 72aca552e1 The engine's rotation against a real bus whose user list is reloaded step by step
A stand-in vault reloads a nats-server's users as the grants part's list is loaded, and the engine's own rotation code runs
against it with real test-dials and a real confirmation request: at every step at least one login authenticates; the engine
switches only once the list holds both logins and confirms in the new login's name; at rotation-done the old login is
retired and its value dropped; with the list never loaded the test-dial is refused and the engine never switches; and an
abandon after the switch brings the engine back to the old login before the new value goes (ADR 0583 §2-§6). One integration
test; mesh-host holds seven of fifty. The real vault and the real engine together are proven live on g14 first.

(issue 774, 774/S65)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:49:01 +01:00
jschoubben 9ae30a7bbb mesh-host enrol redeems through the seat mesh-enrolment and refuses an answer the token's key did not sign
The engine dials as enrol.<node> on the pinned certificate, calls redeem with the public halves of its keys and a proof over
them (the SDK's RedeemProof, which binds the node and the keys and never the token's secret), asks again while the holder
says to, and verifies the vault's answer against the key its token carries before it keeps anything: an answer signed by
another key, naming another signer or node, or altered is refused and nothing is kept. A signed one is kept (membership,
login, keys), then the node dials as the login it was given, reads its key of node-logins, and calls confirm; nothing is
applied until its placement names a module. When the bus does not serve the seat to the token (the controller still holds its
lease) the controller's enrolment subject is used as before. The token's shape is mesh-sdk's now: identity keeps its names as
aliases. Tests: one added, paid by merging three identity token tests into one (net two fewer).

(issue 774, 774/S64)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:49:01 +01:00
jschoubben e362c800a4 The engine test-dials, switches, confirms, drops and falls back across a rotation of its login
On node-logins' ready the engine fetches the new value through the vault's own hand-out (own/<node>/mesh-host/<login>, opened
with its sealing key), dials a second connection as the new login on the pinned certificate, and switches only if that
authenticates, then calls mesh-vault.node-account-confirm over the link as the new login (the vault is asked in the name of
the login the open link authenticated as, and only once that link is open). It keeps the old value until the key says the
rotation is done and drops it then, never before; an abandon, after the switch before or after the confirmation, switches
back before the new value is dropped; repeated refusals of the new login's credentials before rotation-done bring it back
to the old one, said as the finding node-account-fallback (in the journal: no route for an engine's finding to the watchman
is decided), and the same rotation is not tried again. Tests: two added, paid by merging two identity sealing tests into one;
more to be paid by deleting the identity token tests that move to mesh-sdk with the token (774/S64).

(issue 774, 774/S62, 774/S63)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:49:01 +01:00
jschoubben c7468656af The engine dials as the login the vault's node-logins names
A membership may name its Logins (ADR 0583 §1): the login in use and the password of each login held, kept beside the
identity, so a restart dials as the login in use. The engine follows its own key of node-logins as a watch on every link
and dials as the login it names when it holds its value, with the node's one inbox; with no key, or a key naming the first
login, it dials node.<n> with the membership's password as before. A change of the login in use drops the link so it is
opened again as the new one, and the vault is asked in the name of the login dialled (the subject names the caller).
Tests: one added, paid by merging the pinned-broker test with the malformed-pin test (two into one).

(issue 774, 774/S61)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:49:01 +01:00
mesh-tools 598b0d8193 Merge pull request 'The bus node writes the grants part's signed user list alone once the lease is handed (issue 774)' (#194) from feat/l3-removal into main 2026-11-02 00:35:43 +00:00
jschoubben c27fad5c6e Pin mesh-sdk at c21d8fd, the removal group's head, so every member builds the same contract (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@598b0d8193a5
mesh/delivery-group group feat/l3-removal merged: every member is merged
mesh/check failed: mesh-build-go: the check of novox/mesh-catalog#590: the ask mvyhumdm860c7f's step on shanks did not end within 30m0s of its start |…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:53:42 +01:00
jschoubben 85826c633d The bus node's follower never falls back to the controller's user list once the grants part's key says the lease is handed (ADR 0571 §8, ADR 0547 §11)
A key that later cannot be read or is deleted is said, the key last heard stands and the list loaded stays. A key without
the flag still falls back, as before.
Tests: three rows added to the existing table test of the follower; no new test function. Built against the group's
mesh-sdk through a go.work outside the tree; go.mod is not re-pinned here.

(issue 774, 774/S31)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:23:11 +01:00
mesh-tools 9f7e898ecf Merge pull request 'Each node's engine writes its own declared packages and answers packages (ADR 0565 §6, issue 1953)' (#193) from feat/node-packages into main 2026-11-01 17:02:53 +00:00
jschoubben 74ecda8ea2 Pin the group's mesh-sdk head b2a27ed5 (issue 1953)
mesh/delivery merged: source.main moved to novox/mesh-host@9f7e898ecfeb
mesh/delivery-group group feat/node-packages merged: every member is merged
mesh/check checking 74ecda8ea2cf on bc960c51dbf0 | novox/mesh-host#193 in group feat/node-packages
The delivery group feat/node-packages pins one mesh-sdk head in every member: it carries DeclaredPackagesOf and the engine's read of the mirror this change uses.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 16:24:04 +01:00
jschoubben e61141f596 Keep the packages this node's compositions declare in the engine's own key, and answer them on the node seat (ADR 0565 clauses 6 and 15, issue 1953)
pacman still reads the controller's mirror because no engine wrote its key: the engine
now keeps each handed composition's package resources and, after a pass that handed
something new, writes mesh-host_packages.<node> only when the value differs, owed after
the growing bound when refused. The manifest declares the state so the grants part
creates it. The verb packages answers the key beside the mirror's key for the node and
their difference, and with write writes the key again from what was composed. H8's
integration test proves the write under the grant the grants part composes.

Needs mesh-sdk feat/node-packages (0ef7c0c) for DeclaredPackagesOf and the engine's read of the
mirror (go.mod pin left for the lead: the SDK commit is not pushed).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 16:09:13 +01:00
mesh-tools bc960c51db Merge pull request 'The node-engine takes the indexer's verdict and never ends a read on a refused manifest (ADR 0578 §3)' (#192) from feat/judge-once into main 2026-11-01 07:55:03 +00:00
jschoubben 07066fe7a0 Pin the group's mesh-sdk head after the gofmt fix
mesh/delivery merged: source.main moved to novox/mesh-host@bc960c51dbf0
mesh/delivery-group group feat/judge-once merged: every member is merged
mesh/check checking 07066fe7a0a8 on 233bb23bbcb1 | novox/mesh-host#192 in group feat/judge-once
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:27:07 +01:00
jschoubben 974177a074 Pin mesh-sdk after the first review round of the judge-once work (ADR 0578)
mesh/delivery-group group feat/judge-once checking: 6 of 6 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 974177a07478 on 233bb23bbcb1 | novox/mesh-host#192 in group feat/judge-once
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:13:14 +01:00
jschoubben 0422893873 Pin mesh-sdk at the final head of the judge-once work (ADR 0578)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:46:15 +01:00
jschoubben 1d3029fa1e Decode manifests in the packaging test, since the SDK's judge is deleted (ADR 0578)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:39:51 +01:00
jschoubben af66f72931 Take the indexer's verdict on a manifest instead of judging it, and name a held module (ADR 0578)
Issue 1585, S17: the engine decodes manifests leniently; a module whose manifest the indexer refused is held and named,
its data key stands, and every other module's items are still published. Policy, HoldsData and the certificate check no
longer parse with the judge.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:04:34 +01:00
mesh-tools 233bb23bbc Merge pull request 'The node-engine publishes each placed module's data items, so disk-load stops recomposing the node (ADR 0577)' (#191) from feat/data-items into main 2026-11-01 00:03:19 +00:00
jschoubben e031758e4a Forget a kept build when its key is written again, and retry data items on the composer's bound (review round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@233bb23bbcb1
mesh/delivery-group group feat/data-items merged: every member is merged
mesh/check pass e031758e4a37 on 6c1f7cc7f50f | unit 4510 run, 4259 kept, 49 skipped | integration 4589 run, 4361 kept, 51 skipped | over bound: unit, …
The builder claims a done key again as a new revision, so a build kept as done is dropped on any
write of its key. Data items that could not be worked out or written stay due and owe a pass after
the growing bound, so a first write refused before the state or grant exists is tried again
without waiting for an unrelated composition.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 00:27:29 +01:00
jschoubben 122f4dd727 Read a done build once, and work out the data items in the scope the pass composed in (ADR 0577)
The data items after a pass re-read each placed module's build over the bus (about 55 per pass on
ace, g14 and shanks); a done build never changes, so the composer keeps it. The items are now
worked out with the pass's own kept builds and modules run inside the node's own, as composed.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 00:18:30 +01:00
jschoubben e0261cdfc1 Keep each placed module's data items in the engine's state data, so disk-load measures without composing (ADR 0577)
After a pass that handed a new composition, the node's composer works out its modules' data
items from the inputs it composed from and writes each module's key <node>.<module> of
mesh-host_data only when it differs, deleting the keys of modules that left. Pins mesh-sdk
feat/data-items for the contract and the engine's grant. Two tool-answer tests are merged so
the unit count stays where it was.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 00:09:25 +01:00
mesh-tools 6c1f7cc7f5 Merge pull request 'A held module whose backup the node agent interrupted, recorded before #189, is not in the node own composition (issue 1289)' (#190) from fix/held-backup-old-record into main 2026-10-31 19:27:48 +00:00
jschoubben 71afdf3ad2 Keep a have held by an interrupted backup and naming nothing out of the node's own
mesh/delivery merged: source.main moved to novox/mesh-host@6c1f7cc7f50f
mesh/check being carried to 71afdf3ad234 on 6c1f7cc7f50f | from ffa63d25a066
novox's postgres stayed held after mesh-host #189: its have, written before that fix,
names no build, so the node's own composition still took it in and was refused for its
secret reader before the start could back it up again. Only that exact record leaves
the node's own; every other have naming nothing keeps main's rule. Completes #189 for a
have written before it (issue 1289).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-10-31 20:19:21 +01:00
mesh-tools ffa63d25a0 Merge pull request 'A backup the node agent interrupted runs again, and a held module keeps the build it runs (issue 1289)' (#189) from fix/held-backup into main 2026-10-31 16:33:54 +00:00
jschoubben 76c7ddfc52 Keep main's rule for which haves run inside the node's own composition
mesh/delivery merged: source.main moved to novox/mesh-host@ffa63d25a066
mesh/check being carried to 76c7ddfc5227 on ffa63d25a066 | from a4713e2d28d7
Review round 1: a have naming no build in a phase but running is also written for a
module that runs inside the node's own (waiting for a yes), so taking it out would
drop its resources before the yes. The fp the held have now keeps already fixes
postgres (issue 1289); a test drives an interrupted backup end to end.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-10-31 16:38:05 +01:00
jschoubben 39365c62c9 Resume a backup the node agent's own stop interrupted, and keep what a held module runs
On novox (2026-10-31) postgres's repeat backup was interrupted by the node-engine's
restart: its held have named no build, so the node's own composition took postgres in
and was refused for its secret reader, and nothing ran the backup again. Issue 1289
(R4c, its delivery held).

- A held have names what runs: read from the module's have before backing-up when
  the agent has no running entry, as the not-composed mark does.
- A have in any phase but running that names nothing is not run inside the node's own.
- A backup interrupted by the agent's stop is no failure: the next start backs the
  module up again and applies while the want asks a backup. A real failure stays
  held for the operator's answer.

Quick fix: a new backup ask by hand. Thorough, chosen: the fault fixed where it lives
in the node agent, so every data module's interrupted backup resumes by itself.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-10-31 16:33:49 +01:00
mesh-tools a4713e2d28 Merge pull request 'A node reads its placement per node and module and hands a module as the vault judged (issue 774)' (#188) from feat/072-moves into main 2026-10-31 10:31:14 +00:00
jschoubben 8bdfddab4a Hand a module exactly when the vault's own answer judged it handed, so the node never composes a secret the vault leaves to the provisioner (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@a4713e2d28d7
mesh/delivery-group group feat/072-moves merged: every member is merged
mesh/check failed: shanks: the check of novox/mesh-controller#487 failed: go mod download in modules/route-proxy: go: git.novox.be/novox/mesh-controll…
2026-10-31 09:49:14 +01:00
jschoubben a4c0cee251 Read this node's placement from its node and module keys, or its placement.<node> before the holder converts it, so a write of the holder's is composed only once its node key lands (issue 774, research 072 row 37) 2026-10-31 09:37:23 +01:00
mesh-tools bb8383b065 Merge pull request 'The node composes again only when what it composes from changed (issue 774)' (#187) from fix/compose-only-on-change into main 2026-10-31 01:07:23 +00:00
jschoubben da0f8f1cfd The node composes again only when what it composes from changed, so a want, placement, module or value written again asks the vault nothing (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@bb8383b065a1
mesh/check being carried to da0f8f1cfdb4 on bb8383b065a1 | from 62b211bb6040
Every put under want.<node>.>, every module's take-in and every placement write
made a pass, and each pass asked the vault for the node's own secrets. The
shadow's change check moves onto the composer, gains the want (by its
fingerprint, the only part composed) and serves both loops.
2026-10-31 01:59:58 +01:00
mesh-tools 62b211bb60 Merge pull request 'A record a module once left in the node's composition goes with that module, so the node's own apply is no longer refused (issue 774)' (#186) from fix/folded-record-goes-with-its-module into main 2026-10-31 00:21:40 +00:00
mesh-tools 3d1465606b Merge pull request 'A node says when it composes again, says what a long pass waits on, and composes on request (issue 774)' (#185) from fix/compose-retries-transient into main 2026-10-31 00:21:33 +00:00
jschoubben 1d8d726758 Leave a record the node's own carried for a module that composes its own, and let that module's apply remove it, so the node's own is not refused for ever (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@62b211bb6040
mesh/check checking 1d8d7267584e on 62b211bb6040
On the bus node restic's manifest replaced its process measure by awake (mesh-catalog 35ff67bd). The record of restic.measure was the node's, made while restic was folded, so restic's own apply never saw it, and the guard against removing a placed module's resources refused the node's own apply whole every two minutes from 2026-10-30 19:42: nothing of the node's own applied.

Quick fix considered: forget the one record by hand (a hand act on the node; refused). Thorough fix (this): the node's own apply keeps such a record and says so, never refusing or removing it; a module's own apply owns a node record whose id names it, so it alone removes it once neither composition declares it.
2026-10-31 01:13:11 +01:00
jschoubben b0d6684d37 Retry only a failed apply that is not held, answer a repeated compose from the last pass, and say the try the refusal named (issue 774, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@3d1465606bed
mesh/delivery-group group fix/compose-retries-transient merged: every member is merged
mesh/check being carried to b0d6684d3731 on 91fb323ce0cf | from mesh-sdk dec3bfe7a8cb | novox/mesh-host#185 in group fix/compose-retries-transient
The verb compose has no warrant (ADR 0563 §2): Agent.Retry released a module held for a failed backup and re-applied
one already running its want. RetryFailed tries again only a failed apply not held, and refuses in words otherwise.
A hand ask no longer restarts the growing bound, and a pass less than askedWithin old is the answer, so a caller in a
loop drives no passes against the vault. A composer whose loop ended refuses at once. The retry line said try N+1.
The test's first shadow now ends before slowAfter is set, a race the race detector found under the new timing.
2026-10-31 01:08:57 +01:00
jschoubben 56eb0b792c Pin mesh-sdk dfd5d82, so the node-engine announces the verb compose (issue 774) 2026-10-31 00:57:53 +01:00
jschoubben 9c15610f70 Say when the node composes again after a seat that did not answer, say what a long pass waits on, and serve the verb compose (issue 774)
On 2026-10-31 ace's route-proxy said only that the vault could not be asked, and ace said nothing for five minutes
while a pass waited after the vault restarted; the retry that already existed was invisible. A refusal for a seat or
a read that did not answer now says when the node composes again ("the node composes again at HH:MM:SS (try N)"), a
retry pass says it starts, a pass that runs on says every minute what it waits on, and the node-engine's verb
compose makes a pass now and says how it went, so a person recovers without a restart.
2026-10-31 00:56:47 +01:00
mesh-tools 91fb323ce0 Merge pull request 'A held module keeps what it contributes, a have stuck at waiting-yes clears, and the node-engine shows a composition (issue 774)' (#184) from fix/held-keeps-contributions into main 2026-10-30 23:52:52 +00:00
jschoubben 8da56ead81 Pin mesh-sdk 8ed3b87, so a held module's ungranted contribution never puts a stand-in in its provider (issue 774, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@91fb323ce0cf
mesh/delivery-group group fix/held-keeps-contributions merged: every member is merged
mesh/check checking 8da56ead8199 on 91fb323ce0cf | novox/mesh-host#184 in group fix/held-keeps-contributions
2026-10-31 00:36:17 +01:00
jschoubben 7bbba2e39e Keep a have's sealed values only when they are of what runs, when a waiting-yes have says running again (issue 774, review round 1) 2026-10-31 00:35:22 +01:00
jschoubben 08e9d15817 Serve the node-engine verb composition: a module's composition beside what runs, its judgement and its diff (issue 774)
Four providers waited for a yes on 2026-10-31 and nobody could read what waited: the have said only
"where its data is kept". The read verb composition answers it from the engine: the latest revision of the
want's build or the one asked by digest, every revision the bucket keeps, what runs and where that is known from,
the want and the have, the move's judgement and whether it waits for a yes, and with diff the resources that
differ, a received file naming its consumers. revisionNamed now reads the same revisions.
2026-10-31 00:26:57 +01:00
jschoubben 1b624c3ec7 Pin mesh-sdk 6bce1fb, so a held module keeps its contributions and the seat's contract names the verb composition (issue 774) 2026-10-31 00:24:56 +01:00
jschoubben 7e2d947f5f Say running again when a provider's composition goes back to what runs while it waited for a yes (issue 774)
waitsForYes returned at once for a composition equal to what runs, and settleOne then found nothing to apply, so
the have kept phase waiting-yes for a move that no longer existed (ace.postgres, novox.postgres, novox.keycloak and
novox.mongodb on 2026-10-31). The have is now rewritten to running.

Quick fix: rewrite the have where the equal composition is met. Thorough fix (not here): one place that writes a
module's have from the settled state after every settle, so no early return can leave a stale phase.
2026-10-31 00:20:43 +01:00
mesh-tools 1e833feb39 Merge pull request 'Pin mesh-sdk eff81f1, so a held module never refuses the node (issue 774)' (#183) from fix/held-module-and-assigner-contract into main 2026-10-30 21:17:39 +00:00
jschoubben 62cfb45789 Pin mesh-sdk eff81f1, so a module held for a missing secret never refuses the node's whole composition (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@1e833feb3940
mesh/delivery-group group fix/held-module-and-assigner-contract merged: every member is merged
mesh/check checking 62cfb45789c1 on 1e833feb3940 | novox/mesh-host#183 in group fix/held-module-and-assigner-contract
2026-10-30 21:21:19 +01:00
mesh-tools 16b8f591ed Merge pull request 'Keep the kept declaration read while a module runs at an old-form build, so novox and shanks apply their own composition again (issue 774)' (#182) from fix/kept-read-and-foreseen-account into main 2026-10-30 19:49:17 +00:00
jschoubben 15361c661f Pin mesh-sdk 4e9621ed, so a proposal foresees a bus account the Approve will mint and a held module's first reason leads a refusal (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@16b8f591ed6d
mesh/delivery-group group fix/kept-read-and-foreseen-account merged: every member is merged
mesh/check pass 15361c661f51 on b133a8949dcb | unit 2719 run, 602 kept, 36 skipped | integration 2731 run, 645 kept, 29 skipped | over bound: unit, in…
2026-10-30 20:34:12 +01:00
jschoubben cc3f7e8e01 Retire the kept read though mesh-wireguard has no fingerprint, since the node's own composes it from the manifest the mesh provides (issue 774)
The review of d5e8458 found that mesh-wireguard, provided by the mesh and never wanted at a fingerprint, kept the kept
declaration read on every node for ever, against ADR 0550 §2 once the lease is written. A selfcompose row shows the
node's own composes it from delivery.ProvidedManifest with nothing running, so the kept read holds nothing of it.

Quick fix: name mesh-wireguard in allHave. Thorough fix: allHave counts every module delivery.ProvidedByMesh says the
mesh provides as covered. Chose thorough: one list of provided modules, mesh-sdk's.
2026-10-30 20:30:37 +01:00
jschoubben d5e8458340 Keep reading the kept declaration while a placed module runs at a build of the old form, so the node's own never drops it (issue 774)
At 19:43:01 on 2026-10-30 shanks' new engine said "the kept declaration is no longer read: each of the 77 modules
placed here has a have of its own", and from 19:43:57 every apply of the node's own was refused: it would remove
systemd-resolved's resources (novox the same for nats and restic). systemd-resolved's have names only a controller
build (build-1791404191702719760), no fingerprint, so the assigner wants it nowhere and it runs folded in the node's
own, of which the kept declaration is the one record; retired, the folded module was composed as nothing.

Quick fix: never retire the kept declaration. Thorough fix: allHave counts only a have naming a fingerprint of the new
form (or removed); a module at a build of the old form keeps the read, named with why, until it has one. Chose
thorough: the read retires by itself once every placed module is wanted.
2026-10-30 20:27:12 +01:00
jschoubben b133a8949d Merge pull request 'The node-engine composes again when a build it waits on is done (issue 774)' (#181) from fix/build-not-done-holds-one-module into main
Reviewed-on: #181
2026-10-30 18:53:08 +00:00
jschoubben 33131dcbfe Compose again when a wanted build lands, so a node that waited on a build is not left refused until another input moves (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@b133a8949dcb
mesh/delivery-group group fix/build-not-done-holds-one-module merged: every member is merged
mesh/check checking 33131dcbfe77 on 2ae7c53376e8 | novox/mesh-host#181 in group fix/build-not-done-holds-one-module
At 19:30 on 2026-10-30 node-tools' want moved to a build not yet done; once it landed, ace, g14 and shanks stayed
refused until some other input moved (g14 and shanks at 19:40, ace after a restart), since the composer watched no
build.

Quick fix: compose again on a timer while anything is held. Thorough fix: the composer watches the builder's builds
(within the node's compose reads, NodeComposeReads) and makes a pass when a done build of a module the node wants, at
its want's or the trunk's build, lands; nothing else moves it. With mesh-sdk 9685990a (pinned here) a running module
whose build is not done is held, keeping what runs, so the pass that waited composes it as soon as it lands. Chose
thorough: event-driven, no poll.
2026-10-30 19:49:08 +01:00
jschoubben 2ae7c53376 Merge pull request 'Pin mesh-sdk 754edfa, so one refused manifest holds one module on this node (issue 774)' (#180) from fix/one-bad-item-holds-one-module into main
Reviewed-on: #180
2026-10-30 18:30:57 +00:00
jschoubben 4055e0094b Pin mesh-sdk 754edfae, so a manifest the mesh refuses holds that module alone and the node composes the rest (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@2ae7c53376e8
mesh/delivery-group group fix/one-bad-item-holds-one-module merged: every member is merged
mesh/check checking 4055e0094b48 on ee71ba4dea81 | novox/mesh-host#180 in group fix/one-bad-item-holds-one-module
At 19:13 on 2026-10-30 power's bar item (mesh-catalog ca8d6f33) made frombus.Compose refuse the whole node, and the
composer marked every module not composed (32 of 81 failed on shanks). The composer already holds what Compose names
held (NotComposedEach, issue 1126); the SDK now names a refused manifest's module held instead of refusing the node.

Quick fix: none in mesh-host; thorough fix is the SDK's input (one rule for Compose, Answer and PlansHeld). The test
replays the captured power manifest and fails on the old pin with the live words.
2026-10-30 19:25:37 +01:00
jschoubben ee71ba4dea Merge pull request 'The node-engine asks again a subscription the bus refused, so its seat answers; backups and health follow what runs (issue 774)' (#179) from fix/sweep-2026-10-30 into main
Reviewed-on: #179
2026-10-30 18:13:06 +00:00
jschoubben b9c24a7b42 Read the vault's structured move-in refusal and take a value it holds another of as superseded, so the move is not asked again for ever (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@ee71ba4dea81
mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged
mesh/check checking b9c24a7b42cb on 737bce9c9448 | novox/mesh-host#179 in group fix/sweep-2026-10-30
mesh-vault (mesh-catalog f3b13de2) answers a move it refuses with `refused: {reason, held}` and `why`, and
no read-back. The engine read only the read-back, so every such row failed "answered with no read-back"
and the controller asked it again on every pass.

Quick fix: treat any answer without a read-back as done. Thorough fix (chosen): read `refused`. For
holds-another-value the vault's value is the live one: the move answers done, superseded, at the held
generation with the held record in its words, so the controller records the row and stops asking; the
journal says it once per secret. Any other reason fails in the vault's own words.
2026-10-30 18:36:38 +01:00
jschoubben 744bfec4a0 Hand the health loop's judges what runs once a module settles, so a resource its new composition drops is no longer watched (issue 774)
restic's composition dropped its process restic.measure on g14 and shanks at 16:14; the engine removed it
and said it down on every look after. The judges were set only when facts were said, and the facts said
inside an apply read the module's composition from before it (the agent records the new one only after
the apply returns), so the removed resource was put back. Judge.Set itself prunes what is not declared.

Quick fix: set the judges after the restic apply. Thorough fix (chosen): one function hands every judge
(liveness, resolver, units, accounts, libraries) what is declared; it runs from the facts as before and
now also whenever a module settles (applied or removed) or is taken as running, so every removed health
entry of every module is pruned at recompose.
2026-10-30 18:12:56 +01:00
jschoubben 8bff438e5e Ask a data module's backup again while nothing on the machine answers it, within three minutes, so a tool-runner restart no longer holds the module silently (issue 774)
On 2026-10-30 ace's redis, postgres and restic (14:03), step-ca (13:55) and mesh-vault on novox (15:22)
were held on a single backup ask that met no responder while the tool runner restarted.

Quick fix: ask node-backup.now twice. Thorough fix (chosen): every node-backup ask (backed-up and now)
is asked again with the engine's backoff while it is known never to have been delivered (ErrNotAsked),
bounded at three minutes; an ask that may have been delivered (a timeout, a refusal) is never asked
twice, so no backup starts twice. Past the bound the backup fails saying how often and how long it was
asked, the module is held and its have carries those words for the assigner's ask and condition. This
was the engine's only one-shot seat ask.
2026-10-30 18:09:16 +01:00
jschoubben 7ecf7ba356 Ask a refused subscription again on a new link and announce the node seat as discovery reads it, so node-engine.root-free is answered again (issue 774)
Every engine's node-seat subscriptions were refused at 12:10 by a user list composed without them. nats
asks a refused subscription again only on a new connection, and a user-list reload makes none, so the
seat answered nobody all afternoon and the messenger held every Telegram ask. The endpoints also lacked
the kind and scope discovery reads, so the console said nothing in the mesh is called node-engine.

Quick fix: restart the engines. Thorough fix (chosen, the second time a stale user list left an engine
deaf): a refused subscription of any kind is said and the link is opened again after a pause (1 min,
doubling to 30 min), so every subscription the engine makes is asked again; the seat's endpoints carry
kind, seat, scope, module, description and schema like every tool runner's.
2026-10-30 18:06:52 +01:00
jschoubben 737bce9c94 Merge pull request 'Leave the engine's acts out of its declared state for now, so the bus's user list is composed again (issue 774)' (#178) from fix/774-engine-acts-not-carried into main
Reviewed-on: #178
2026-10-30 11:51:56 +00:00
jschoubben ec191247a0 Leave the engine's acts out of its declared state for now, so no node's tool runner carries a grant to the engine's own keys and the grants part composes the bus's user list again (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@737bce9c9448
mesh/check checking ec191247a06d on 737bce9c9448
2026-10-30 12:47:40 +01:00
mesh-tools a9edd2951c Merge pull request 'Design 47's last milestones: L3's engine re-landed with held modules never removed, provision secrets, the node's own health report (issues 774, 1126, 1954, 1952)' (#177) from feat/774-l3-reland into main 2026-10-30 11:02:15 +00:00
jschoubben 015334020a Merge main (delivery 1, the parse-only sdk) into L3's re-land, and pin mesh-sdk 2da0de7, the release's final head (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@a9edd2951c46
mesh/delivery-group group feat/774-l3-reland merged: every member is merged
mesh/check checking 015334020ad2 on a3c098a5c618 | novox/mesh-host#177 in group feat/774-l3-reland
The conflicts were go.mod, go.sum and testdata/beside/CAPTURED alone, taken from the release. 2da0de7 is 6e4a301's code;
its conformance events are unchanged.
2026-10-30 01:13:38 +01:00
mesh-tools a3c098a5c6 Merge pull request 'Pin the parse-only mesh-sdk in the node-engine, so every node can compose design 47's manifests (issue 774)' (#176) from fix/774-new-words-first into main 2026-10-30 00:10:21 +00:00
jschoubben 2fd165f5af Pin the parse-only mesh-sdk 78c9ec8 in place of 6e4a301, so main's engine reads the release's new words and means nothing new by them (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@a3c098a5c618
mesh/delivery-group group fix/774-new-words-first merged: every member is merged
mesh/check pass 2fd165f5af8e on 8c5af3e4a3a3 | unit 9174 run, 546 kept, 51 skipped | integration 9952 run, 659 kept, 60 skipped | over bound: unit, in…
78c9ec8 is sdk main 307b623, whose tree is main's pin dbcaa29, with one commit that parses the new words (a credential's
form, an artifact's bus, a log's max-age-days) and names the new seats; an offer with no form still means pair. No code
of ours changes. Unit and integration suites pass against a local bus and main's catalogue and controller beside: 1260
tests, 7 skipped for a container or a throwaway machine.
2026-10-30 00:39:20 +01:00
jschoubben db1151c773 Pin mesh-sdk 6e4a301 on main's engine, so the controller and the engine read the release's new manifest words before the release uses them (issue 774)
No code of ours changes. Whole unit and integration suites pass at main's code against a local bus: 1260 tests, 7
skipped for a container or a throwaway machine.
2026-10-30 00:25:10 +01:00
jschoubben 02f78c9949 Pin mesh-sdk 6e4a301, the release's final sdk (issue 774)
The whole integration suite passes at this head against a local bus (JetStream, and a bus with the witness grants) and
the release's mesh-catalog and mesh-controller beside it: 1260 tests, 7 skipped for a container or a throwaway machine.
2026-10-30 00:17:00 +01:00
jschoubben 192633edb8 Pin mesh-sdk 007b6fd, the release's final sdk (issue 774) 2026-10-30 00:07:07 +01:00
jschoubben e5e8ff5783 Say each account's root verdict and the engine's contract in the node's health report, so the self-check's DA and D-root judge without the controller (issue 1952, ADR 0266)
Every account resource marked root never goes into NodeHealth.Root; the statement's contract into Contract. Relogin,
root and found are judged each on its own, so an account may be both. Pins mesh-sdk f3677eb. Test: the report's row in
TestAHealthStatementIsSaidOnTheLinkOpenNow carries the agent account's verdict and the contract.
2026-10-29 23:49:36 +01:00
jschoubben c3ff18de4e Name the machines a failing network part points at in the node's health report, so the watchman holds it where the controller does (issue 774)
delivery.NetworkPart.Targets (mesh-sdk 6eae076) is filled where the report is made, by the controller's own rule:
- the hub's node for "hub";
- each resolver's node by its address on the private network;
- none when the part points at nothing, at this machine, or at an address no other mesh machine has.

The names come from the placements the node's own composer holds (Decided.Address, Decided.Hub). A spoke's tunnel names
only the hub, so the tunnel cannot name a resolver on another spoke. A node that does not compose its own names no
target.

Test: rows in TestAHealthStatementIsSaidOnTheLinkOpenNow (hub, two resolvers, a VPN's resolver, its own, a healthy
part).
2026-10-29 23:05:02 +01:00
jschoubben 3ed9abf658 Format the health report's aliases as gofmt does (issue 774) 2026-10-29 22:59:38 +01:00
jschoubben 552f97c919 Take the node's health report layout from mesh-sdk 94d0de5, so the engine and the watchman read one form (issue 774)
delivery.NodeHealth and its parts replace this package's copies. The message's network and units parts are aliases of
the same types, so its JSON is unchanged. The engine keeps the derivation from its statement and the write on a change.
Test: the key's bytes, decoded and encoded again by the sdk, are the same bytes.
2026-10-29 22:59:22 +01:00
jschoubben 65fbc1e6f8 Write each node's health report into its own key of mesh-host_health on a change, so the watchman raises the machine's alerts without the controller (issue 774, the operator's alert table)
Until now the six machine alerts came from a health message passing to the controller. The report is now kept in the
engine's declared state `health`, key `<node>`, created by the grants part as `acts` is. Only this node's engine
writes it, after comparing it on every look and only when what it says changed; it is read first after a restart.

It carries:
- the network parts (machine-network; network-rewritten through the resolver file part's writer and owner);
- the tunnel's peers, each seen within five minutes or not (network-unreachable);
- the failed units no module owns (machine-units);
- the accounts marked relogin needed (relogin-needed);
- the directories used as found (directory-used-as-found).

The streaks and the time of the look are left out, so a look alone writes nothing. A peer is reported seen or not,
never its handshake time, so a renewed handshake writes nothing.

The layout is internal/link's (NodeHealth) until it moves into mesh-sdk. Pins mesh-sdk 58a8324.

Tests:
- rows in TestOnTheHubAnyFreshPeerIsAHealthyTunnel: the peers, and a change only when one is seen or lost;
- a case in TestAHealthStatementIsSaidOnTheLinkOpenNow: what the report takes, and one write per change across a
  restart;
- the manifest's states in TestTheManifestConsumesTheBackupEvent.
2026-10-29 22:56:14 +01:00
jschoubben cbe8528f91 Resolve ${secret:<name>} through this node's provision keys, and check the reference by the vault's own name (issue 1954 L6-E1, E2; ADR 0570 clauses 1, 6; ADR 0094)
- A placeholder is `${secret:<name>}`, the name the module calls the credential (catalogue.SecretLocal): the local
  name where it keeps several values of one provision, else the provision. The `<provision>.<local>` form goes.
- The engine finds the one key under `<node>.<module>.` of mesh-provisioner's `provisions` going by that name. That is
  what node.<node> may read, by a watch to its caught-up mark; plans are not read.
- The reference's ID must be a credential name of that provision (provisions.VaultName). The vault's answer must carry
  exactly that name and generation. Anything else refuses the write, naming both.

Pins mesh-sdk 66ff33d. Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise (a local name, a name no provision
goes by, a reference to no credential of the provision, a mismatched answer).
2026-10-29 22:41:44 +01:00
jschoubben 5497d362dd Refuse a provision's value unless the vault's credential name and generation are the ones its reference names (issue 1954 L6-E1; ADR 0570 clauses 1, 6)
The value written is the one the provision names: a1-47 makes Reference.ID the vault's credential name. A mismatch in
either refuses the write, naming both. Test: a row in TestAWriteAndAnApplyOfTheSameFileSerialise.
2026-10-29 22:37:44 +01:00
jschoubben 6b32e07164 Fill a provision's ${secret:} at the write from the vault's value sealed to this machine, so the provisioner's client never holds it (issue 1954 L6-E1, E2; ADR 0570 clause 6)
The provisioner's client fills a handed module's file but for each `${secret:<provision>}` (or a whole `secrets` file
that is the placeholder alone).
- The node-engine reads the provision's key `<node>.<consumer>.<provision>[.<local>]` in mesh-provisioner's
  `provisions`. The key must be `provided` and carry the vault's reference.
- It asks the vault's `own` for that pair credential sealed to this machine, at least at the reference's generation,
  and opens it with the node's sealing key.
- It writes the file as declared (its owner), 0600 when the declaration gives no mode.
- Anything short of that refuses the write, naming the provision and why.

A module the node composes itself already has its `secrets` files written so: its composition carries the vault's
value from `own`, written 0600 with the declared owner.

Pins mesh-sdk 4a96793 (go/provisions). Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise.
2026-10-29 22:36:53 +01:00
jschoubben f7814853f9 Pin mesh-sdk 2e5cc4f, the re-land group's sdk head (issue 774) 2026-10-29 22:18:45 +01:00
jschoubben 58aacaad2d Pin mesh-sdk 7000bd5, the re-land group's sdk head: a tools bundle declares its own bus account (issue 1952 S5, issue 774) 2026-10-29 21:57:09 +01:00
jschoubben 81672b183d Keep the node's own as it runs from an engine's first start, and refuse the node's own apply removing a module no want removed (issue 774, review round 1)
The reviewer showed how 76e214a still lets the 07:54 outage happen at the re-landed engine's first start on novox.
applied/ is empty there, so loadApplied loads nothing. The kept declaration retires, because every placed module has a
have. Nothing then carries the node's own composition. The guard misses nats, which has no want there, so its
nats.conf is guessed as nats's and removed.

- At the start, when the store has no node's own composition, the node's own runs as the kept declaration has it, and
  the store keeps it now. The kept declaration retires only once the store holds `node`.
- What this engine applied runs until loadHad reads the composition a differing have names. The bucket no longer keeps
  the one novox's have names, so the stored one stays.
- In the node's own apply, a resource of another module goes only when that module's want says removed
  (Scope.RemovedByWant). A module the node's own composition carried has no want, so Placed never named it.
  Cost: the node's own apply cannot drop one of its own dotted resources without such a want. It is refused whole and
  said, never removed silently.

Tests, with no new test functions:
- A first-start case in the agent's Everything test (empty store, then a start with the declaration retired). It fails
  on 76e214a.
- Two rows in the guard's table: no want refuses; a want saying removed removes.
- A case for the store in the authority test.
2026-10-29 21:25:51 +01:00
jschoubben 76e214a8b9 Keep every composition the engine applied on the machine, so a module whose composition the bus no longer keeps runs as applied and is never composed as nothing (issue 774)
On novox on 2026-10-29 at 07:54 UTC the kept declaration was retired and the node's own composition was in no bucket.
The composer's first pass reads what runs before the agent reads any have (the agent waits for the composer's caught,
sent after that pass), so the node's own was composed with its folded modules running nothing, and the bus server and
four held modules were removed.

- Quick fix: read the node's own have in loadHad too. It still waits for the composer's first pass, and still finds
  nothing when the bucket no longer keeps the composition the have names.
- Thorough fix, chosen: the engine keeps each composition it applied (the node's own too) in `applied/` beside its
  state, 0600. The agent reads it at Run's start, before any watch, as what runs ahead of the bucket and the kept
  declaration. A have naming another running composition wins. An apply keeps; a removal forgets.

Test: a case in TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest (no new test function). It fails on L3's
engine 4f9227a (0 compositions at the first own composition) and passes here.
2026-10-29 21:16:38 +01:00
jschoubben acf02d52ff Pin mesh-sdk 22c9ddd, so a carried or held module serves its tools from what runs and a data flag alone moves no data (issues 1126, 924) 2026-10-29 21:05:12 +01:00
jschoubben a7acee3eed Refuse an apply that would remove a resource of a module a want still places here, so a composition that lost what runs can never remove the bus server or a held module again (issue 774) 2026-10-29 21:04:17 +01:00
jschoubben d2645a27e9 Compose a file that runs as the node last wrote it, so a stale kept declaration's export no longer names a swept object (issue 924)
novox's mesh-vault has failed since 01:16 on 2026-10-29: its have says failed, so loadHad (fbea777) takes no
composition from it, and what runs falls back to the kept declaration, whose export (mesh-vault.kept) names
f099eaa1, an object the controller's sweep removed. The node wrote a newer export since. The composer carries that
file as it runs (mesh-sdk frombus keptOf), so what runs now names the digest and size of what is on disk, but only
where the node's record says it wrote exactly that content at that path (Applied.Wrote, Target). The apply then
finds the file in place and needs no object.

Quick fix weighed: have the controller put the object back. Thorough (chosen, in the node): what runs is what the node
wrote, never a guess from an older declaration. The rows join the existing whole-apply test; the test count stays.
2026-10-29 21:04:17 +01:00
jschoubben 7d39510450 Keep the content of every file given by digest on the node, so an object the controller's sweep removed never refuses a composition the node already had (issue 924)
On novox at 01:16 on 2026-10-29, after the node began composing its own modules, mesh-vault's apply failed while the
vault ran: its own composition carries the export as it runs (mesh-sdk frombus KeptRuns) by a digest whose object the
controller's sweep (mesh-controller cmd/mesh-controller/declaration_sweep.go, sweepNode and sweepDeclarationFiles)
had removed, since the sweep keeps only what the controller's compositions in composed-<node> and the wants name.
The node may only read its bucket (ADR 0486 §2), so the node keeps what it had: a file here that verifies, or an
object fetched, goes to <state dir>/objects under the object's name, and a kept one is used before the bus is asked.
An object unused for 14 days is let go. A fetch that still fails says the keep never held it, before its cause.

Quick fix weighed: re-put the object from the controller's store. Thorough (chosen): the node never depends on the
bucket for content it once had, for every module, since the sweep and a node composing its own can always disagree.
The whole-apply test gains three rows that fail on the old code; the test count stays.
2026-10-29 21:04:17 +01:00
jschoubben f38ac87ac9 What runs from the kept declaration says Data as the module's manifest does, and only a have that runs counts at the start (issue 924)
The kept-declaration fallback built a composition without Data, so after a restart a data module (mongodb, mssql
on novox) was judged to move its data against its first composition and the operator was asked a false yes. Now
every such composition takes Data from the manifest (agent.Env.HoldsData, the composer's DataItems), and a module
whose manifest cannot be read is said to be unjudgeable rather than taken to hold none. The composition a have
names comes before the kept declaration in the judgement too.

loadHad takes a have only in the phases baseline takes as running (running, held, a composer's mark): one backing
up, applying, checking, failed or waiting names a composition not applied (review of 031e106). Two rows added to
the same table test; the count stays.
2026-10-29 21:04:17 +01:00
jschoubben d8de0a8853 Judge a module the agent has not taken yet by the composition its have names, and knock the filling at the start and whenever a module is taken as running (issue 924)
A held module on a node composing its own never gets a composition, so the agent never took it and the filling
judged its files by the kept declaration, older than power's three new drop-ins: refused from every engine start
on ace, g14 and shanks until some module settled (2026-10-29 00:09 UTC). The second meeting of issue 924.

Quick fix: knock the filling once the haves are read at the start. Thorough fix (chosen, the kind recurs for every
file added since the switch): what runs of such a module is the composition its have names, from the bucket or
its revisions, ahead of the kept declaration; the filling is knocked on every change of what runs, not only on a
settle. One unit test replaced by a table of four cases; the count stays.
2026-10-29 21:04:17 +01:00
jschoubben 45d0348044 Revert "Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#174) from revert/774-l3 into main"
This reverts commit 1ec73915b8, reversing
changes made to 0a4c35f731.
2026-10-29 21:04:13 +01:00
mesh-tools 8c5af3e4a3 Merge pull request 'Pin mesh-sdk dbcaa29, so a held module's hold no longer spreads to the modules that read from it (issue 1960)' (#175) from fix/1960-no-hold-spread into main 2026-10-29 15:35:33 +00:00
jschoubben ece607e215 Pin mesh-sdk dbcaa29, so a held module's hold no longer spreads to the modules that read from it (issue 1960)
mesh/delivery merged: source.main moved to novox/mesh-host@8c5af3e4a3a3
mesh/delivery-group group fix/1960-no-hold-spread merged: every member is merged
mesh/check pass ece607e21522 on 1ec73915b807 | unit 3318 run, 1951 kept, 38 skipped | integration 3994 run, 2125 kept, 38 skipped | over bound: unit, …
2026-10-29 16:00:21 +01:00
jschoubben 86a05babed Expect proxy handed beside held de-spiegel: a hold no longer spreads (issue 1960)
mesh-sdk's composer no longer holds a module behind a held one it reads from; the selfcompose test now expects
proxy handed, composed without de-spiegel's routes, and de-spiegel told alone. Needs mesh-sdk pinned at the fix.
2026-10-29 15:58:19 +01:00
mesh-tools 1ec73915b8 Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#174) from revert/774-l3 into main 2026-10-29 08:59:58 +00:00
jschoubben f81e6147ae Revert L3 (mesh-host, merge 0a4c35f): its node-engine removed every held module on novox, the bus server among them, at 08:54 CET on 2026-10-29 (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@1ec73915b807
mesh/delivery-group group revert/774-l3 merged: every member is merged
mesh/check being carried to f81e6147aee1 on 0a4c35f731a7 | from mesh-catalog 1555241393fe | novox/mesh-host#174 in group revert/774-l3
L3 lands again with two fixes: a held module is never removed, and the grants cover L3's new subjects.
2026-10-29 09:42:29 +01:00
mesh-tools 0a4c35f731 Merge pull request 'L3: the node trusts the vault's key, serves the node seat node-engine and keeps its acts on the bus (issue 774, ADR 0561), rebased onto L2' (#168) from feat/774-vault-mints-l2 into main 2026-10-29 07:44:19 +00:00
jschoubben 4f9227a186 Merge main (#171) into L3: pin mesh-sdk c2999a4, mesh-sdk #128's head, which holds L3's sdk and both seed fixes (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@0a4c35f731a7
mesh/delivery-group group feat/774-vault-mints-l2 merged: every member is merged
mesh/check pass 4f9227a1860f on fb8ea0e6838c | unit 1601 run, 5122 kept, 47 skipped | integration 1637 run, 5215 kept, 49 skipped | over bound: unit, …
2026-10-29 01:08:25 +01:00
mesh-tools fb8ea0e683 Merge pull request 'Pin mesh-sdk c2999a4, so a node's seed answer holds a held module as it runs and its neighbours compose (issue 1126)' (#171) from fix/1126-held-as-runs into main 2026-10-29 00:07:06 +00:00
jschoubben a6c70803ab Pin mesh-sdk at c2999a4: a held module no longer spreads its hold to its neighbours (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@fb8ea0e6838c
mesh/delivery-group group fix/1126-held-as-runs merged: every member is merged
mesh/check pass a6c70803ab72 on bb6e47293909 | unit 3054 run, 234 kept, 36 skipped | integration 3075 run, 266 kept, 29 skipped | over bound: unit (sl…
novox's seed p-20261028t231141-ec84b4 refused 11 modules behind de-spiegel's hold and gitea's handed credential. The
node composes and answers through mesh-sdk's frombus, so the fix is the pin.

- The quick fix: follow one more set of words in the answer. It moves the error again.
- The thorough fix, chosen (mesh-sdk fix/1126-held-as-runs): a held module composes only as what runs of it; a consumer
  or collector composes against that and is never held for the held one's stub, and a stub's handed credentials and own
  secrets are left out as they run rather than refused.
2026-10-29 00:50:01 +01:00
jschoubben 9bd1c84e0f Pin mesh-sdk 76f7c4d, the seed fix's own head: the checker refuses a pin to main's merge commit d4c61ee (same tree) as on no pull request (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-delivery@64d364a7686c was judged on trunk fa075dc8…
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 9bd1c84e0f4c on fb8ea0e6838c | novox/mesh-host#168 in group feat/774-vault-mints-l2
2026-10-29 00:20:01 +01:00
jschoubben 1f3fb52470 Merge main into L3: pin mesh-sdk main d4c61ee, which holds L3's sdk and the seed's held-as-today fixes (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 rejected: composed together: the checker's verdict at the members' heads (2199dae4e7d8): ace
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: the check of novox/mesh-host#168 failed: the pin of git.novox.be/novox/mesh-sdk/go at commit d4c61ee8ffd0 is on no pull reques…
2026-10-29 00:11:11 +01:00
mesh-tools bb6e472939 Merge pull request 'A node answers its seed with an old-form build, a failed or held module, and those held behind it, as they run today (issue 1126)' (#170) from fix/1126-seed-oldbuild into main 2026-10-28 23:05:08 +00:00
jschoubben 93f33e2305 Pin mesh-sdk 76f7c4d: a held module keeps its items (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@bb6e47293909
mesh/delivery-group group fix/1126-seed-oldbuild merged: every member is merged
mesh/check pass 93f33e2305de on d2a868f99f05 | unit 3018 run, 270 kept, 36 skipped | integration 3039 run, 302 kept, 29 skipped | over bound: unit (sl…
2026-10-28 23:56:51 +01:00
jschoubben abdf85aa4f Node agent: say how it keeps each module today, held, failed or at a build of the old form named by its have's build; pin mesh-sdk f609be7 (issue 1126) 2026-10-28 23:50:19 +01:00
jschoubben 91686fdb81 Node agent: count a module running a build of the old form as held today, so the seed keeps it as the controller does (issue 1126) 2026-10-28 23:45:28 +01:00
mesh-tools d2a868f99f Merge pull request 'Node agent: say what it holds today in its answer, so a seed holding the same is the same (issue 1126)' (#169) from fix/1126-seed-held into main 2026-10-28 22:20:59 +00:00
jschoubben a143678fd8 Pin mesh-sdk 77b48a4, the group's head (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@d2a868f99f05
mesh/delivery-group group fix/1126-seed-held merged: every member is merged
mesh/check pass a143678fd898 on 4f6b389a37a1 | unit 1199 run, 1015 kept, 33 skipped | integration 1218 run, 1081 kept, 26 skipped | over bound: unit (…
2026-10-28 22:41:14 +01:00
jschoubben 9914459830 Pin mesh-sdk 06b13ec, the group's head (issue 1126)
mesh/delivery-group group fix/1126-seed-held checking: 2 of 3 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 9914459830d7 on 4f6b389a37a1 | novox/mesh-host#169 in group fix/1126-seed-held
2026-10-28 22:18:26 +01:00
jschoubben b7159d71a6 Node agent: say what it holds today in its answer, so a seed that holds the same is the same; pin mesh-sdk b03bf4c (issue 1126) 2026-10-28 22:15:24 +01:00
jschoubben e0c7f36ca9 Pin mesh-sdk 8eafc0c, the L3 group's head on mesh-sdk main with #120 and #121 (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-host@e0c7f36ca90c was judged on trunk d2a868f99f05…
mesh/delivery closed: a newer head of the same pull request
mesh/check checking e0c7f36ca90c on bb6e47293909 | novox/mesh-host#168 in group feat/774-vault-mints-l2
2026-10-28 21:56:06 +01:00
jschoubben 9120f94aa4 Name the seat mesh-placement where the kept declaration's retirement reads the placement, as L2 moved it (issue 774, ADR 0567) 2026-10-28 20:36:41 +01:00
jschoubben 1900b40192 Pin mesh-sdk 4e6e5cc, the L3 group's head rebased on L2 (issue 774) 2026-10-28 20:36:41 +01:00
jschoubben 6cbdb3712e Judgements and refused setuid searches, which any caller may ask, roll in one bounded key of the node's and add no key to the bucket every node shares; a key per act is kept for hand-overs, access acts and searches started (issue 774, ADR 0563 §2; rev-l3 round 2) 2026-10-28 20:35:33 +01:00
jschoubben 2a0c428ef9 The last setuid search's start and each library's own take ask are keys of their own, which no number of other acts moves (issue 774, ADR 0563 §2, §4; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben a5d1f7dde4 From the lease write a directory used as found names the node seat's hand-over as its way out, following the lease the engine hears (issue 774, ADR 0563 §5; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben 95dd64c774 Every signed ask is verified with the controller's key the engine trusts at that ask, so after the lease write none signed by it is taken (issue 774, ADR 0563 §7, ADR 0550 §2; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben 81fa009bf4 The node seat's acts and the engine's own asks are the node-engine's declared state acts on the bus: one key each under its own node's keys, written by compare-and-set, one ask open per path made by a create; acts reads one by its id (issue 774, ADR 0563 §2, §3) 2026-10-28 20:35:33 +01:00
jschoubben 05ff90e3c1 The engine judges its node seat's hand-over and take as it judges the controller's ask, answers root-free for its own machine, and asks a library's take itself once (issue 774, ADR 0563 §3, §4, §6) 2026-10-28 20:35:33 +01:00
jschoubben aaffc19232 The node seat node-engine served on the engine's link: its verbs with their proven caller, its acts kept in its own state, a hand-over or access act asked of the operator as node-engine-<node> and performed only on the warrant (issue 774, ADR 0563 §1-§3) 2026-10-28 20:35:33 +01:00
jschoubben 479ac71050 The node trusts the vault's key beside the controller's until the lease is handed away, says what it trusts in its record, asks the vault for its own certificate, composes the vault's authority beside the controller's, and reads the kept declaration no more once every placed module has a have (issue 774, ADR 0550 §2) 2026-10-28 20:35:33 +01:00
mesh-tools 4f6b389a37 Merge pull request 'Node agent: compose from the seat mesh-placement and answer proposals (issue 1126, delivery 2 of 2)' (#167) from feat/1126-placement-seat into main 2026-10-28 18:33:12 +00:00
jschoubben 45f5e27526 Pin mesh-sdk to the group's head 38dbe1d (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@4f6b389a37a1
mesh/delivery-group group feat/1126-placement-seat merged: every member is merged
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
2026-10-28 18:50:39 +01:00
jschoubben dd8ba97989 Node agent: wait for the bus's word that the seat's placement and candidates states were created, then watch them (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben ab22efd8b1 Node agent: follow the placement key at start only and say why when it cannot; test the reached candidate and the switched rule; word the log by the seat's state (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 54aee345fc Node agent: a node that never composed its own takes a failed placement read as not placed; one that did keeps waiting (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 8639b373d2 Prove a missing or refused bucket is a watch error, so a node waits on the controller's compositions and tries again (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 5fc69444f6 Node agent integration tests: candidates keyed by proposal, under the node's real grant on the seat's states (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 2fe9d678fc Node agent: answer a proposal's candidate keys and follow the seat's placement, not the indexer's groups (issue 1126)
The answer names its proposal and the revision of the node's own candidate key; a node with no key in a proposal is not asked. The watch for its own placement key is tried again until the seat's state exists, so the node switches when its key is written.
2026-10-28 18:43:49 +01:00
mesh-tools 58b4f8b6bc Merge pull request 'Reconcile a module from the machine's facts at start and on a down edge (issue 1957, ADR 0568)' (#166) from fix/1957-boot-reconcile into main 2026-10-28 17:41:54 +00:00
jschoubben bf667a602d Reconcile a module from the machine's facts at engine start and at each edge of a declared-running resource going down (issue 1957, ADR 0568)
mesh/delivery merged: source.main moved to novox/mesh-host@58b4f8b6bc07
mesh/check being carried to bf667a602d21 on 58b4f8b6bc07 | from 9a8b3870bad0
A reboot left the have running, so a unit that failed to start at boot was never started again and nothing said so. Baseline now asks the machine; the judge's look calls Agent.Reconcile once per down edge, leaving held and unwanted modules alone.
2026-10-28 18:12:35 +01:00
mesh-tools 9a8b3870ba Merge pull request 'The node-engine applies every module but the held ones and marks only those (group fix/1126-apply-culprit, issue 1126)' (#165) from fix/1126-apply-culprit into main 2026-10-28 16:34:48 +00:00
jschoubben 24c23a1d06 A module the node holds alone marks that module, keeps what runs of it, and never refuses the node (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@9a8b3870bad0
mesh/delivery-group group fix/1126-apply-culprit merged: every member is merged
mesh/check pass 24c23a1d066f on a548a61e86d7 | unit 2667 run, 590 kept, 36 skipped | integration 2676 run, 633 kept, 29 skipped | over bound: unit (sl…
frombus.Compose now holds a module refused alone (mesh-sdk 0de3006, the rule
its answer follows too): composed as what runs of it, with what depends on
how it is held held too. The composer hands every other module, keeps what it
handed of a held one, and tells the agent which modules it holds and why;
agent.NotComposedEach marks those alone and clears the rest. Only a refusal
of no single module still marks every module (NotComposed).

heldOut also holds out, of the node's own composition as it runs, the
resources of each module held alone, until a composition of its own is
handed: the node's own apply never removes a module leaving the fold with its
names unset (review of the apply, round 1; issue 1945 by a new path).

Pins mesh-sdk fix/1126-apply-culprit at 0de3006 (a group).
2026-10-28 17:23:26 +01:00
mesh-tools a548a61e86 Merge pull request 'The node-engine reloads the service manager when a unit file or drop-in it writes changes (group fix/shutdown-bound, issue 1126)' (#164) from fix/shutdown-bound into main 2026-10-28 15:05:10 +00:00
jschoubben bc28606d45 apply: reload units only for a unit file actually written or removed (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@a548a61e86d7
mesh/delivery-group group fix/shutdown-bound merged: every member is merged
mesh/check pass bc28606d4566 on 690aa662a837 | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
A skipped or held file comes back on every apply; counting it would reload the
service manager on every apply of a node with a gated or held module. The same
allowlist as MovesRoot: created, updated, corrected, removed.
2026-10-28 15:49:34 +01:00
jschoubben 7f0eb9c51d apply: reload the service manager once when a file in a unit directory moved (issue 1126)
A drop-in that no service reflects, like the power module's bound on
reboot.target, was written but never loaded: the node-engine reloaded units only
for a file named in a service's restart-on, so the next reboot still ran the
target as loaded. Any file written or removed under a system unit directory, or
a drop-in directory below one, now gets one daemon-reload per apply.
2026-10-28 15:43:35 +01:00
mesh-tools 690aa662a8 Merge pull request 'The node's shadow says which input woke each answer (issue 1126)' (#163) from fix/1126-vault-own into main 2026-10-28 14:20:39 +00:00
jschoubben 8bd0b5db15 Say which input woke each placement candidate's answer, so a probe's safety check tells a loop from moving inputs (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@690aa662a837
mesh/delivery-group group fix/1126-vault-own merged: every member is merged
mesh/check pass 8bd0b5db15a3 on b4fd2cc210bf | unit 1639 run, 4201 kept, 46 skipped | integration 1674 run, 4215 kept, 47 skipped | over bound: unit, …
Each answer's line now reads 'answering placement candidate G (revision N; woken by X)', X being its first answer,
a retry after a read the bus did not answer, the candidate, or the inputs that changed (placement, records, modules,
values, haves, wants). The single input counter becomes one per input, which also decides whether a group is answered
again, as before.

Quick fix weighed: a counter word in the line. Thorough fix chosen: one count per input, so the line names exactly what
moved since the group's last answer.
2026-10-28 14:44:27 +01:00
mesh-tools b4fd2cc210 Merge pull request 'A one-shot at rest satisfies a stopped declaration, so the kernel's boot cleanup applies (issue 1126)' (#162) from fix/ace-tun-cleanup into main 2026-10-28 13:34:20 +00:00
jschoubben 1a021052fe Drop the never-ran reading: a start timestamp of 0 does not mean a one-shot never ran (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@b4fd2cc210bf
mesh/check being carried to 1a021052fe07 on b4fd2cc210bf | from 7a2af2376405
Review round 1 measured it on systemd 262.1: a finished one-shot nothing references is
unloaded, and loaded again it reads inactive, ExecMainStatus=0 and a start timestamp of
0, as one that never ran does. The previous commit's rule read such a unit stopped, so
a one-shot declared running would be started, read stopped and fail on every apply, and
the hold, the take-over and the giving back would judge it wrongly.

ServiceState keeps main's reading. ServiceAtRest and declaredState stay: they alone
satisfy a stopped declaration for a one-shot at rest, which is the kernel cleanup's
case. A one-shot declared running that must be seen to have run lingers
(RemainAfterExit) or is enabled at boot. Unit tests: the arch table has 5 rows, the
apply test 1; 1262 in all.
2026-10-28 14:29:16 +01:00
jschoubben cf738cdb90 Read a one-shot that never ran as stopped, and one at rest as stopped to a stopped declaration (issue 1126)
The kernel module's boot-time cleanup, linux-modules-cleanup.service (Type=oneshot,
declared stopped and enabled at boot), failed its delivery on ace with 'was asked to
be stopped and is running': ServiceState read every inactive one-shot with
ExecMainStatus=0 as running, and systemd says 0 for one that never ran too. Stopping
an inactive unit changes nothing, so the read back said running again; after a boot
that ran it, the same declaration would fail every apply.

ServiceState now reads ExecMainStartTimestampMonotonic: 0 means never ran, read as
stopped (a package's one-shot declared running is then started, and the hold no
longer counts it as in use). The service apply reads a one-shot at rest as stopped
to a declaration that it be stopped (ServiceAtRest), whatever its last run did.

Quick fix weighed: declare the cleanup running in the catalogue. Thorough chosen:
the reading was wrong for every one-shot, and this is its second fault of the kind
(the firewall's was the first). Unit tests: three arch one-shot tests merged into one
table (7 rows), one apply test (2 rows); 1263 to 1262.
2026-10-28 14:29:16 +01:00
mesh-tools 7a2af23764 Merge pull request 'Pin mesh-sdk f387581: the node composes refusals per module and folded modules' owned paths as the controller does (issue 1126, group fix/1126-seed-8)' (#161) from fix/1126-seed-8 into main 2026-10-28 13:27:09 +00:00
jschoubben daaa1962bd Pin mesh-sdk f387581, the group's head: a module's own refusal refuses that module, contributions in config words, folded modules own their manifest's paths (issue 1126, seed-8)
mesh/delivery merged: source.main moved to novox/mesh-host@7a2af2376405
mesh/delivery-group group fix/1126-seed-8 merged: every member is merged
mesh/check pass daaa1962bdab on df878ba1f780 | unit 3018 run, 234 kept, 36 skipped | integration 3038 run, 266 kept, 29 skipped | over bound: unit (sl…
2026-10-28 14:08:58 +01:00
mesh-tools df878ba1f7 Merge pull request 'The node answers a placement candidate once, retries a read the bus did not answer, and keeps a file's secrets entry by entry (issue 1126; group fix/1126-seed-same-3)' (#160) from fix/1126-seed-same-3 into main 2026-10-28 11:40:38 +00:00
jschoubben 1e01868d92 An unplaced node owes no pass, so no timer is armed again; every wait a test sets is put back when it ends; pin mesh-sdk 8520a4c, whose masking splits only a file's secrets (issue 1126; review of seed-same-3, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@df878ba1f780
mesh/delivery-group group fix/1126-seed-same-3 merged: every member is merged
mesh/check pass 1e01868d92c3 on c3a2ffde6755 | carried from mesh-catalog 774fcdfa93ad | unit 2188 run, 1064 kept, 36 skipped | integration 2197 run, 1…
2026-10-28 12:25:32 +01:00
jschoubben 9d04ae7edd Hold the secrets row to the have read's error, as the loop fix's Env.Held returns it (issue 1126) 2026-10-28 12:13:09 +01:00
jschoubben 16df7d0b7a Merge branch 'fix/1126-answer-once' into fix/1126-seed-same-3 2026-10-28 12:11:50 +01:00
jschoubben cd1d2ff447 A read the bus did not answer is tried again after a bound, never kept as an answer: the shadow's answer, the node's composition and the own-secret step (issue 1126)
Met twice on 2026-10-28: shanks kept 'want.shanks.brother could not be read ... no
responders' as the own-secret step's last word, and ace kept 'the vault's own: not yet
... no responders' as its answer to seed-7, each until another input moved. Every bus
read of the composer (manifests, builds, archives, served values, the vault's own and
credentials, the have of what runs) is noted (BusFailed); an answer or a pass that met
one is made again after a growing bound (5 s to 5 min), and a pass is not handed when
the have of its sealed values did not answer. The own-secret step tells a read or an
ask that did not answer (Retry, taken again) from a key absent or the vault's word.
2026-10-28 12:11:08 +01:00
jschoubben faf6a23d91 Fill go.sum for mesh-sdk 70de942 (issue 1126) 2026-10-28 12:06:25 +01:00
jschoubben 2f4e81f3b0 A file's secrets are kept entry by entry: the box that runs at one secret is reused only there, where it opens to the same value; pin mesh-sdk 70de942 (issue 1126, C1; the seed-7 probe) 2026-10-28 12:05:48 +01:00
jschoubben 4182f8fbfb The node answers a placement candidate revision once: a module's have whose health words alone moved, or a key written again with its value, no longer composes every group's answer again (issue 1126)
On shanks probe/1126-seed-7 was answered at revision 84 every 13-17 s: each look rewrote
the haves of its unhealthy modules with new words, and the shadow took any write of its
own module haves, placement, modules or config values as a change. Each answer loaded
the bus, the health tools timed out, the words moved again. Now only what an answer is
composed from wakes it (what runs, folded modules, a changed value), and each group is
answered again only when its keys' revisions or the node's inputs moved.
2026-10-28 12:02:57 +01:00
mesh-tools c3a2ffde67 Merge pull request 'The placement test's resolved values carry their source, as mesh-config writes them (issue 1126)' (#159) from fix/1126-seed-same-2 into main 2026-10-28 10:08:46 +00:00
jschoubben 6502ba6ad6 The placement test's resolved values carry their source, as mesh-config writes them: a value with no source is laid by neither the controller nor a node composing its own (issue 1126, C2; the group check of 3cb7f74b347d)
mesh/delivery merged: source.main moved to novox/mesh-host@c3a2ffde6755
mesh/check being carried to 6502ba6ad6e6 on c3a2ffde6755 | from f6c0520f26b9
2026-10-28 10:47:48 +01:00
mesh-tools f6c0520f26 Merge pull request 'Node shadow: reuse a seal only at its slot, lay values as the controller does, newest candidate first, and hand the vault a drifted own secret (issue 1126, group fix/1126-seed-same)' (#158) from fix/1126-seed-same into main 2026-10-28 09:46:47 +00:00
jschoubben c912bcab6e The node agent finds a drifted own secret's running file by its id as the node runs it, its module's name first (issue 1126, cause C4; review round 2)
mesh/delivery merged: source.main moved to novox/mesh-host@f6c0520f26b9
mesh/delivery-group group fix/1126-seed-same merged: every member is merged
mesh/check checking c912bcab6e73 on f6c0520f26b9 | novox/mesh-host#158 in group fix/1126-seed-same
Pins mesh-sdk 3028c7e (delivery.OwnNeedName). The fixtures carry the id shape captured from novox's shadow answer.
2026-10-28 10:01:38 +01:00
jschoubben 63c0b445d7 Fill go.sum for mesh-sdk 1062b33, the group's head (issue 1126) 2026-10-28 09:53:34 +01:00
jschoubben 653ff61662 The node agent hands the vault what runs of an own secret its shadow answer says another value of, through move-in's replace_running, while it applies the controller's compositions (issue 1126, cause C4)
A new step, internal/ownresync, after each shadow answer: for each own secret masked as another value, while the node
applies the controller's compositions and runs the one its want names, it asks the vault's move-in with
replace_running: true and the running value sealed to the call's key, once per value, and says what it asked and what
the vault answered. The answer written to the node's have key carries values_differ. Pins the group's mesh-sdk head
1062b33 (go.sum is filled once that head is pushed).
2026-10-28 09:51:48 +01:00
jschoubben a7cd0a7807 Node shadow: reuse a running seal only at its own slot, lay values as the controller does, answer the newest candidate first and say how it goes (issue 1126, C1 C2, T1 T2 T3)
A box is reused by slot (resource, field) after the composition, from what runs or the
have's slots; a have written before slots is reused within its module alone. Config
values are laid by mesh-sdk's one rule (never a default). The shadow answers the
newest candidate first and hands each answer as it lands, says when an answer starts,
every minute what it waits on, and when it ends, and fits an answer to the node's key.
Needs mesh-sdk fix/1126-seed-same (16b8d8a1).
2026-10-28 09:41:29 +01:00
mesh-tools 7271cf151b Merge pull request 'Node shadow: what runs is the composition at the have's digest, not the stale kept copy (issue 1126, group fix/1126-shadow-bodies)' (#157) from fix/1126-shadow-bodies into main 2026-10-28 07:46:33 +00:00
jschoubben 82d0e125b2 What runs is the composition the have names, never the copy of the declaration kept before the switch (issue 1126, probe #532: shanks' xterm, zsh and its plugins answered differing at the very digest their haves say runs)
mesh/delivery merged: source.main moved to novox/mesh-host@7271cf151b3a
mesh/delivery-group group fix/1126-shadow-bodies merged: every member is merged
mesh/check checking 82d0e125b2c0 on 7271cf151b3a | novox/mesh-host#157 in group fix/1126-shadow-bodies
2026-10-28 01:42:17 +01:00
jschoubben a4c24a1979 Pin mesh-sdk 362a3f1: a refusal names what another node cannot run here, and is answered without a composition per module (issue 1126, probe #532) 2026-10-28 01:40:18 +01:00
mesh-tools cff69893a3 Merge pull request 'Node shadow: decides 'same value' by opening seals in memory, says when it is not ready or slow, carries the bus membership (issue 1126, group fix/1126-shadow-shows)' (#156) from fix/1126-shadow-shows into main 2026-10-28 00:21:51 +00:00
jschoubben 047237c911 The shadow shows its work: differing resources with masked bodies and whether a sealed value is the same, a minute not ready or a slow answer said in the journal and the answer; pin mesh-sdk 4e0c577 (issue 1126, seed #531)
mesh/delivery merged: source.main moved to novox/mesh-host@cff69893a361
mesh/delivery-group group fix/1126-shadow-shows merged: every member is merged
mesh/check pass 047237c911a4 on 449cda1b003e | unit 2980 run, 270 kept, 36 skipped | integration 2999 run, 302 kept, 29 skipped | over bound: unit (sl…
ace, g14 and shanks answered nothing to seed #531 and said nothing either. The
shadow now says, after a minute not ready, which inputs it has not read, and
answers each group it knows refused for that; an answer running over a minute
is said while it runs and noted in the answer. The answer's differing resources
carry both bodies, masked, the node saying whether two sealed values open to the
same value (Composer.sameValue: only the comparison leaves the node).
2026-10-28 01:11:55 +01:00
mesh-tools 449cda1b00 Merge pull request 'Node agent: a switch never marks the new run's modules, a failed read or a bus pause keeps what runs, stale composer marks clear (issue 1126, group fix/1126-switch-safe)' (#155) from fix/1126-switch-safe into main 2026-10-27 23:52:28 +00:00
jschoubben a6e53a4858 A composer's mark never covers another cause, and a run that ended marks and clears nothing (issue 1126, review of the switch, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@449cda1b003e
mesh/delivery-group group fix/1126-switch-safe merged: every member is merged
mesh/check pass a6e53a485813 on 199f6b48b1df | unit 2992 run, 1008 kept, 36 skipped | integration 3025 run, 1102 kept, 29 skipped | over bound: unit (…
NotComposed marked from a read have whatever its phase: an apply that failed,
naming the want's build and digest, became a composer's mark, which baseline
then took for what runs, and never applied again. It now marks only a have that
runs, says nothing, or is a composer's mark already. And its loop, whose reads
can take seconds, checks the run's context at each module, so a run that ended
writes no have.
2026-10-28 00:41:25 +01:00
jschoubben 4993f1b1d7 Make the switch between composers safe: a composer tells only its own run's agent, a pass whose run ended refuses nothing, a bus that pauses is tried again, a mark keeps what runs and clears at the next start; pin mesh-sdk 7359d18 (issue 1126)
Live 2026-10-28: the own run's last pass, failing on its run's end, marked
every module of the controller-mode agent that replaced it (ace 60, shanks 63),
and the marks dropped the build and digest because the have read failed in the
bus's reload pause. Now: the refusal hook is bound to the run's agent and ctx;
a pass whose context ended returns nothing; JetStream not enabled or no
responders are tried again after 1, 2 and 4 s before any mark; NotComposed reads
the have again while the bus does not answer and marks nothing it cannot read;
baseline takes a have a composer marked as what runs, so a mark it did not make
clears; each run's shadow lives with the run.
2026-10-28 00:34:10 +01:00
mesh-tools 199f6b48b1 Merge pull request 'Node agent: a failed placement read picks no owner, a deleted placement returns the node to the controller without a restart (issue 1126, group fix/1126-shadow-all)' (#154) from fix/1126-shadow-all into main 2026-10-27 22:44:12 +00:00
jschoubben 4636261435 A failing placement read chooses neither composer, a deleting head is compared with the controller's compositions at the node's wants, and a placement watch that ends is said; pin mesh-sdk e32d777 (issue 1126, review of the shadow, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@199f6b48b1df
mesh/delivery-group group fix/1126-shadow-all merged: every member is merged
mesh/check checking 463626143567 on bb15aab50eda | novox/mesh-host#154 in group fix/1126-shadow-all
selfComposer took a KVHistory error for 'not placed': at any restart a timeout
or a bus not ready flipped a self-composing node to the controller's
compositions for a whole run. selfcompose.Placed chooses only on a read that
succeeds (a missing placement state is one), and tries again otherwise, said
once. The shadow's answer to a head deleting the placement reads the
controller's composition at each want in the node's bucket and compares it with
what runs; a want with none refuses, without returning.
2026-10-27 23:37:59 +01:00
jschoubben 4e8fdfdf04 The way back: a node whose placement key is deleted returns to the controller's compositions without a restart, says so in its journal and its have, and answers such a candidate as returning; pin mesh-sdk 472d5f3 (issue 1126, the seed's revert)
The seed's revert (mesh-catalog #526) was refused on 'would stop running node',
while a self-composing node would have refused its placement's absence, kept
what runs and marked every have failed until a restart. endWhenUnplaced ends the
agent's run when the key goes, so it begins again on the controller's
compositions, as at a start without a key; the composer refuses nothing while
unplaced; the node's have says whose compositions it applies (Composes); the
shadow answers a deleted placement Returns, verdict same.
2026-10-27 23:25:59 +01:00
jschoubben d00926e089 Compose with what the node runs, so a folded module of an old-form build is carried as it runs, and bound an archive read; pin mesh-sdk 2fd5ffd (issue 1126, the seed's shadow)
novox refused its own composition and its answer on de-spiegel, a module it runs
folded into its own composition at a build only the controller holds. The
composer now gives frombus what the node runs (Scope.Running), so such a module
is carried as it runs. The shadow's answer names every refusal (mesh-sdk). A
templates archive read is bounded at 30 s: nothing else held an unanswered store.
2026-10-27 23:19:40 +01:00
mesh-tools bb15aab50e Merge pull request 'Node agent: compose its own modules from their builds and providers' served keys under its own grant (issue 1126, group fix/1126-node-creds)' (#153) from fix/1126-node-creds into main 2026-10-27 20:32:37 +00:00
jschoubben 6a4718b712 Hold in a unit table which haves count a module as run inside the node's own composition, so a failed have naming nothing is caught without H8's timing (issue 1126, review round 2)
mesh/delivery merged: source.main moved to novox/mesh-host@bb15aab50eda
mesh/delivery-group group fix/1126-node-creds merged: every member is merged
mesh/check checking 6a4718b712b2 on bb15aab50eda | novox/mesh-host#153 in group fix/1126-node-creds
2026-10-27 21:29:51 +01:00
jschoubben d5adf9b72b Drop two debugging lines from the build read and H8, and say why the refused read's wait stays above the history bound (issue 1126, review round 1) 2026-10-27 21:21:19 +01:00
jschoubben 7365890298 Prove in H8 that a node reads a placed module's build and a provider's served key through its own session, refused without the grant, and take a failed have naming nothing as no module run inside the node's own; pin mesh-sdk 7a8e9f9 (issue 1126, review round 1)
H8 found it: a module the node could not compose yet got a failed have naming
no fingerprint, which counted as run inside the node's own composition, so the
next composition took it from its manifest past its build.
2026-10-27 21:18:45 +01:00
jschoubben 0862f97972 Pin mesh-sdk 538ffba, the group's head (issue 1126) 2026-10-27 21:04:02 +01:00
jschoubben 45439fd742 Compose each module from its build and read what providers elsewhere serve, pinning mesh-sdk cebc651, so a self-composing node hands its moved modules and settles mailu's domain (issue 1126)
The node read a behind module's source manifest digest and composed every module
from the indexer's raw manifest; frombus now takes the build itself (Scope.KeptAt,
a fingerprint). The reader gives frombus the builder's key, a templates archive
fetched as every archive is (apply.Fetch) once per digest and checked against it,
and another node's served keys one by one, as the node's grant allows them.
2026-10-27 21:01:16 +01:00
jschoubben 0184e2a5b8 Watch the have bucket only where the node's grant allows, so a node composing itself is not refused by the bus (issue 1126)
The fold's watch of every have key ('>') was refused (permissions violation on
CONSUMER.CREATE KV_node-engine_have), and every self-composing agent stopped
after the seed. The composer now watches '*.node' (records) and '<node>.>' (its
own haves), as buscompose grants a node; the shadow watches '<node>.>' too. The
selfcompose tests check every watch and get against PermissionsFor(KindNode),
and the agent's integration tests run the agent as the node under that grant.
2026-10-27 09:27:54 +01:00
mesh-tools 50c229f905 Merge pull request 'The node agent folds what it runs at no new-form build into its own composition, judged from its haves, so the first composition after a restart keeps the bus (issue 1126)' (#152) from fix/1126-fold into main 2026-10-27 08:09:06 +00:00
jschoubben 69b845c0f7 Read what a node runs inside its own composition from its haves, not from what the agent applied, so a restarted agent's first composition keeps the bus (issue 1126, review of the fold round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@50c229f90568
mesh/delivery-group group fix/1126-fold merged: every member is merged
mesh/check checking 69b845c0f7b2 on 0d22776b20ef | novox/mesh-host#152 in group fix/1126-fold
A fresh agent composes before it has applied anything (Running empty), so
InOwn of Running folded only mesh-wireguard and the switch would drop nats.
The composer now watches the node's haves: a placed module whose have names no
new-form fingerprint is folded; one with a new-form have whose want is removed
is not folded. The shadow uses the same set, over the candidate's placement.
2026-10-27 09:03:59 +01:00
jschoubben caafd9306e Keep what the node runs inside its own composition there when it composes itself, so the seed's answer and the node's own composition keep the tunnel and the bus (issue 1126)
Pins mesh-sdk 05b67831, whose compositions fold a module no new-form build
names into the node's own. The shadow composed a placed module it runs inside
its own composition (nats, photos) at the trunk's build, as if the candidate
added it; shadowBuilds now leaves those to the fold, and both paths pass
frombus.InOwn of what runs.
2026-10-27 08:51:41 +01:00
mesh-tools 0d22776b20 Merge pull request 'The node agent answers a placement candidate in shadow, applies nothing, and keeps its own sealed values in its have (issue 1126, ADR 0559 §4)' (#151) from feat/1126-l2b into main 2026-10-27 07:21:19 +00:00
jschoubben 1803d25c3a Watch afresh and read from nothing at each shadow run, so a candidate deleted while it was down is not answered and no earlier watch stays open (issue 1126, review of L2b round 2)
mesh/delivery merged: source.main moved to novox/mesh-host@0d22776b20ef
mesh/delivery-group group feat/1126-l2b merged: every member is merged
mesh/check checking 1803d25c3a67 on 0d22776b20ef | novox/mesh-host#151 in group feat/1126-l2b
2026-10-27 02:48:54 +01:00
jschoubben d56c576e9a Begin the shadow composer again with a backoff when an input cannot be watched yet, and pin mesh-sdk's fixed head, so a node answers candidates for the whole link (issue 1126, review of L2b round 1) 2026-10-27 02:44:39 +01:00
jschoubben fbc6d6330f Extend H8 with a placement candidate the node answers same and applies nothing for, and pin mesh-sdk's group head (issue 1126) 2026-10-27 02:29:17 +01:00
jschoubben fb0cdabe25 Answer every placement candidate in shadow in the node's own key, applying nothing, so a placement change is checked by the nodes it places before it merges (issue 1126, ADR 0559 §4) 2026-10-27 02:26:07 +01:00
jschoubben 0474a2a3cb Keep the node's own composition's sealed values in its key and read a policy within the agent's context, so a restart applies nothing again (issue 1126) 2026-10-27 02:21:36 +01:00
mesh-tools 180a7e7b80 Merge pull request 'A directory used as found is not a module's health, and a failed have names its cause (issue 1496, ADR 0557 §4, §5)' (#150) from fix/1496-start-health into main 2026-10-26 21:38:02 +00:00
mesh-tools be94d2689e Merge pull request 'The node composes its own modules once the catalogue places it, and writes its record (issue 1126)' (#149) from feat/1126-nodes-compose into main 2026-10-26 21:11:36 +00:00
jschoubben b225c7bb4c A directory used as found is not a module's health, and a failed have says why (ADR 0557 §3, issue 1496)
mesh/delivery merged: source.main moved to novox/mesh-host@180a7e7b80e2
mesh/delivery-group group fix/1496-start-health merged: every member is merged
mesh/check checking b225c7bb4cf5 on be94d2689e2f | novox/mesh-host#150 in group fix/1496-start-health
On ace, 2026-10-26 21:19, mesh-build-node's cache directory was there before the
mesh and used as found. The health statement says such a directory unhealthy, and
the have folded it into the module's health as failed, with no words, while the
module's own check answered healthy; the assigner read it as its canary failing
and held the version on all four nodes.

The directory waits for the operator's hand-over and keeps its own condition,
raised from the same statement; it no longer counts in the module's have. A have
whose health is failed or waits and whose apply said nothing carries the words of
the resource that made it so, so the next one is readable.
2026-10-26 22:06:32 +01:00
jschoubben 1d44288514 Pin mesh-sdk 9796444 (l2-next: feat/1126-nodes-compose merged with sdk main #101, the standing-test race fixes) (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@be94d2689e2f
mesh/delivery-group group feat/1126-nodes-compose merged: every member is merged
mesh/check checking 1d442885148a on be94d2689e2f | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 22:05:41 +01:00
jschoubben af2d2a2c14 Pin mesh-sdk 5f45175 (l2-next: feat/1126-nodes-compose merged with sdk main) after merging main (issue 1126) 2026-10-26 21:59:21 +01:00
jschoubben 59d77dd457 Pin mesh-sdk feat/1126-nodes-compose 03dce12, the group's head: the checker refuses a pin that is neither trunk nor the group's head (issue 1126)
mesh/delivery-group group feat/1126-nodes-compose rejected: composed together: the checker's verdict at the members' heads (0d06ff386dce): ace
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 59d77dd457da on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 20:35:52 +01:00
jschoubben 23d66187a8 Review of L2, round 3: a have written with no composition keeps what the module's last have said runs (fingerprint, digest, sealed values), for every writer; a record module run nothing of since the start judges its move against its last have and waits for a yes (issue 1126, ADR 0547 §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 23d66187a881 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 20:29:07 +01:00
jschoubben 4476e58080 Review of L2, round 2: a refusal at the first compose after a start keeps what the module's have said runs (1); each have of a self-composing node carries the values its composition holds sealed, and the composer reuses them after a restart, so a rotated secret moves no composition at a restart (2); pin mesh-sdk dd9d9d5 (issue 1126, ADR 0547 §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 4476e5808047 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 20:23:16 +01:00
jschoubben 58de6aa10a Review of L2: a node that cannot compose says so in each wanted module's have, failed, while it keeps what it runs (H2); the composer keeps the box the node runs when it opens to the vault's value, so neither the switch nor a restart moves a composition (H1); pin mesh-sdk 3968849, whose node asks the vault only for a module declaring a secret (H8's failure) (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: ..test: ✔ a transient empty list for four passes retires nothing; five stable passes retire, data kept (290.408814ms) | 58de6…
2026-10-26 20:09:40 +01:00
jschoubben cdd3f3fd4f Pin mesh-sdk feat/1126-nodes-compose 9713c30: placement carries a settings-form module's values (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | cdd3f3fd4f63 on ff9386dcccc2 | unit 1245…
2026-10-26 19:40:41 +01:00
jschoubben 21d6583b5e The node agent begins again composing its own when the catalogue places its node, with no restart (issue 1126, ADR 0549 §4)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
2026-10-26 19:34:24 +01:00
jschoubben bb46a6ad0f The node composes again when the vault says a secret changed, so a rotated secret or credential reaches it at once (issue 1126, ADR 0547 §2)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestIntegrationLevelRunsAgainstARealStoreAndBusAndLeavesNothing: result = &{Level:integr…
2026-10-26 19:26:25 +01:00
jschoubben f8afaefba0 The node keeps the mesh's own certificate it applies, from its kept declaration at start, for its record; one agent test folded to hold the count (issue 1126, ADR 0555 §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the grou…
2026-10-26 19:24:36 +01:00
jschoubben 1e38c42646 The node's composer asks the vault for the pair credentials its composition reads, as consumer and provider, sealed to its key (issue 1126, ADR 0550 §5)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the g…
2026-10-26 19:21:22 +01:00
jschoubben 1241ee7041 Pin mesh-sdk 4d5f430: the node's wants say it composes them (Self) (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk f1d7c7377dc4, which is neither on mesh-sdk's trunk nor the g…
2026-10-26 19:14:38 +01:00
jschoubben 2dc384d641 The node's composer asks the vault's own for its modules' secrets sealed to its key, and keeps a value first had while its generation stands; two agent tests folded to hold the count (issue 1126, ADR 0550 §5)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 2dc384d641a2 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 19:05:12 +01:00
jschoubben e4dbb49d2f Integration: a node the placement names composes and applies its own module with no controller composition, and again on a changed config value (issue 1126, ADR 0547 §2)
mesh/delivery-group group feat/1126-nodes-compose checking: 4 of 5 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking e4dbb49d2fa5 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 18:51:36 +01:00
jschoubben 02d9d7b053 The node composes its own modules once the catalogue places it: one composer per node drains a mark set by any input's change, the agent applies what it composed when it differs, and a record move waits for a yes (issue 1126, ADR 0547 §2, §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 02d9d7b0538f on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 18:44:58 +01:00
jschoubben 7fc0ea5478 Pin mesh-sdk feat/1126-nodes-compose 9789c83 (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 7fc0ea5478a3 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 18:34:35 +01:00
jschoubben 8be1c2ab0d The node agent writes what the machine observes of itself as its record in its own have key, only when it changed (issue 1126, ADR 0555 §3) 2026-10-26 18:32:54 +01:00
mesh-tools ff9386dccc Merge pull request 'The node-engine runs yielding processes in one capped idle slice (issue 1806, ADR 0553/0554, change A)' (#148) from fix/build-cpu-yields into main 2026-10-26 14:22:54 +00:00
mesh-tools bba9081ca1 Merge pull request 'The node-engine moves a running daemon's build aside instead of deleting it (issue 734)' (#147) from fix/daemon-dir-kept into main 2026-10-26 14:16:00 +00:00
jschoubben 65cd7b4787 Retire a daemon's running build instead of deleting it before the restart (issue 734)
mesh/delivery merged: source.main moved to novox/mesh-host@bba9081ca127
mesh/delivery-group group fix/daemon-dir-kept merged: every member is merged
mesh/check pass 65cd7b47873a on bba9081ca127 | carried from 94f9aa298cb8, mesh-sdk 21252194d530 | unit 971 run, 423 kept, 32 skipped | integration 976…
A process no witness judges had its directory removed before the new bundle
was unpacked and the unit restarted, so the old process ran on in a deleted
working directory: mesh-build-go's `go version` probe failed with "getwd: no
such file or directory" and refused three checks on ace on 2026-10-26, each in
the second a new build of it arrived.

Every process is now replaced the way a judged one is: its build is renamed
under .witness/<name>/retired/ and deleted only once nothing runs from it.
"Runs from it" now counts a process's working directory as well as its image,
so the children a daemon starts keep the directory too. Forget keeps a retired
build that still drains, and the witness's sweep covers every process it keeps
a directory for, judged or not.

Two unit-file tests are folded into one, so the unit count goes down by one.
2026-10-26 15:01:55 +01:00
jschoubben a78239b2aa Pin mesh-sdk at fix/build-cpu-yields' head (ADR 0553, ADR 0554)
mesh/delivery merged: source.main moved to novox/mesh-host@ff9386dcccc2
mesh/delivery-group group fix/build-cpu-yields merged: every member is merged
mesh/check being carried to a78239b2aa05 on ff9386dcccc2 | from bba9081ca127, mesh-catalog 9a8e54d988fe
2026-10-26 14:47:02 +01:00
jschoubben 2ac0c4686a The node-engine holds the node's one build CPU share as its own config (ADR 0553 §3, round 2)
The share belongs to the part that enforces it: build-cpu-share, default 50,
set per node through the node's settings layer and read by the composer for
the build seats' runtimes. Pin mesh-sdk at fix/build-cpu-yields' head.
2026-10-26 14:40:24 +01:00
jschoubben 5233f7584d Pin mesh-sdk at fix/build-cpu-yields' head (ADR 0553) 2026-10-26 14:23:13 +01:00
jschoubben 8858dba25e Merge main into fix/build-cpu-yields 2026-10-26 14:22:43 +01:00
jschoubben 9254180cab A yielding process runs in the node's one yielding slice, which holds the shared quota (ADR 0553, issue 1806)
Review round 1: one quota per unit let three build modules take three shares.
The engine now writes meshyield.slice, top-level so its idle weight binds
against the operator's session, with the node's share as its CPUQuota; each
yielding unit joins it at nice 19. Comments say the IO controls bind only under
bfq or io.cost, not under the none scheduler NVMe disks use.
2026-10-26 14:21:24 +01:00
mesh-tools 94f9aa298c Merge pull request 'The node-engine writes the followed user list into the bus module's have even when it does not apply that module (issue 1945)' (#146) from fix/have-keeps-user-list into main 2026-10-26 12:52:15 +00:00
jschoubben ee9fcaba20 Pin mesh-sdk at fix/build-cpu-yields' head (issue 1806) 2026-10-26 13:49:41 +01:00
jschoubben 43f2d6020e Write the bus's user list into a have no apply here says, merged over the key's value
mesh/delivery merged: source.main moved to novox/mesh-host@94f9aa298cb8
mesh/check pass 43f2d6020e11 on 94f9aa298cb8 | carried from 7f04d1204765, mesh-catalog a4f5df797b68, mesh-sdk b4cd95e7c4a4 | unit 972 run, 423 kept, 3…
On the bus node the bus module has no want and has left the declaration, so no
pass of this engine held its fact and the follower's user list was kept in
memory and never written: the vault and the grants part read no hashes and no
key. A module's user list now reaches its key on every pass, merged into what
the key says with every other field kept; a missing key is said once, never
made up (ADR 0547 §8).
2026-10-26 13:38:40 +01:00
jschoubben 6250b1274b A process that yields is written at idle priority under its share of the machine (issue 1806)
A build seat's runtime took 1650% of shanks' 24 processors and slowed the
operator's desktop. The unit of a process declared with yields now carries
CPUWeight=idle, IOWeight=1, Nice=19, the lowest best-effort IO level, and
CPUQuota for its share of the processors this machine has.
2026-10-26 13:37:48 +01:00
mesh-tools 7f04d12047 Merge pull request 'Bus node: follow the grants part's sealed user list after the operator's take-over, never both sources; hand the vault the loaded hashes (issue 1945)' (#145) from feat/1945-user-list-switch into main 2026-10-26 12:21:55 +00:00
jschoubben e88ddab241 An apply of the bus module waits when the list on the machine cannot be read while the grants part's is followed, so the file is never removed as an orphan (review of the switch, issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-host@7f04d1204765
mesh/delivery-group group feat/1945-user-list-switch merged: every member is merged
mesh/check being carried to e88ddab24170 on 7f04d1204765 | novox/mesh-host#145 in group feat/1945-user-list-switch
2026-10-26 12:54:06 +01:00
jschoubben 06f70e11ee Bring in the adoption handing; the bus node writes nothing before it heard the switch, retries the key's watch and keeps the follow last heard, and an apply never writes the controller's list while the grants part's is followed (review of the switch, issue 1945)
Merges fix/1945-adopt-hashes 1a190090. A new follower (one per link) waits for the grants part's key to catch up before
its first write, so a restart after take-over never writes the controller's list first. A refused or ended watch is
tried again on link.Backoff, the have says why it is not watched, and the key last heard, kept beside the node's state,
holds meanwhile. While the grants part's list is followed, the bus module's composition is applied with the list on the
machine in place of the controller's. The grants list is never handed to the vault. Pins mesh-sdk c2a410bb.
2026-10-26 12:30:07 +01:00
jschoubben 61cec66ba8 gofmt the follower's test (issue 1945) 2026-10-26 12:08:37 +01:00
jschoubben 5e2fadce15 Pin mesh-sdk at the group's head 83ee7769 (issue 1945) 2026-10-26 12:07:10 +01:00
jschoubben 882e75c954 The bus node follows the grants part's sealed user list from take-over, the controller's otherwise, never both (ADR 0547 clauses 7, 8; issue 1945)
On the node whose compositions carry the bus's list, the engine also watches mesh-grants' key user-list.sealed. While it
says follow grants, the object it names is read by name, opened with this machine's sealing key, checked against the
key's digest and the guard, and written to the same file at 0600 only when its digest differs; the bus reloads it
itself. The have says which list is followed and this machine's sealing key, which the grants part seals to.
The follower's test waits for its watch to catch up before a later put, which raced under load. Pins mesh-sdk 175a5567.
2026-10-26 12:06:17 +01:00
879 changed files with 87058 additions and 4227 deletions
-20
View File
@@ -171,26 +171,6 @@ carries. The claim is not. The foundation is four container services, and:
So `foundation.lock` here is a real bundle with a placeholder's content. Saying that plainly
beats shipping a host that claims a foundation it has never raised.
## mesh-cli's socket (novox/hq ADR 0272)
The engine serves `/run/nox-mesh-host/cli.sock` for `mesh-cli`, the operator's command (`internal/meshcli`). It
reads the asking account from the kernel (`SO_PEERCRED`) and asks the controller on this node's own subject; the
controller decides what the line may do, and the engine runs nothing it receives. One line per connection, each
connection's deadline set the moment it is accepted, two lines at once per account and eight for the machine,
"busy" beyond that.
**Only a login session can be the terminal.** The engine reads the asking process's cgroup and sends its login
session's scope (`session-<id>.scope` under the account's own `user-<uid>.slice`, which logind's own record in
`/run/systemd/sessions/<id>` must say is this account's, of class `user`), or nothing, read the moment the
connection is accepted; the controller makes
nothing else the terminal. A service running as the operator — the tool runner, a unit of the account's user
manager — is at most an ordinary call.
**A process outside the machine's own namespaces is refused**, whatever its user id: the peer's mount and user
namespace must be the engine's. So `mesh-cli` run inside a flatpak, a toolbox or distrobox container, or any other
container — even one running as the operator's own id — is refused with the reason; run it from the machine's own
shell.
## A capability is detected, never assumed
The reason this is the first thing built rather than a detail of it.
+154
View File
@@ -0,0 +1,154 @@
// mesh-compose is the node-engine's composer command (novox/hq ADR 0482 clause 4, ADR 0593 §7, issue 1621): it reads a
// composer.Request on standard input, reads every manifest, composes the machines named in each case and writes one
// composer.Verdict on standard output. Built at a commit of the node-engine's repository, it answers what that
// commit's composition answers, so a group that changes the node-engine's composition is judged by its own next code.
//
// mesh-compose < request.json > verdict.json
// mesh-compose -request request.json > verdict.json
// mesh-compose -catalogue <dir> [-node <name>]... [-depth resolve|compose] [-declarations] [< request.json]
//
// -catalogue reads <dir>/modules/*/module.json and a module.json at <dir>'s root as the first case's manifests (a
// checkout or `git archive` of the catalogue repository at a ref: the command fetches nothing). -node adds a machine to
// compose; -declarations answers each machine's declared paths with their owners and modes.
//
// It opens no network connection, bus, store or file beyond its standard input, the -request file and the -catalogue
// directory, and reads
// no environment. Exit codes: 0 the verdict is written (a machine that does not compose is a verdict, not an error);
// 1 the request cannot be read or run, said on standard error; 2 the flags are wrong.
package main
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"strings"
"github.com/novox/mesh-host/internal/composer"
)
func main() {
os.Exit(run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr))
}
type nodes []string
func (n *nodes) String() string { return strings.Join(*n, ",") }
func (n *nodes) Set(v string) error { *n = append(*n, v); return nil }
// run is the command: its arguments, standard input, output and error; it answers the exit code.
func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("mesh-compose", flag.ContinueOnError)
fs.SetOutput(stderr)
catalogueDir := fs.String("catalogue", "", "a catalogue checkout: <dir>/modules/*/module.json are the first case's manifests")
var machines nodes
fs.Var(&machines, "node", "a machine to compose (repeatable)")
depth := fs.String("depth", "", "resolve or compose (default: the request's, else compose)")
declarations := fs.Bool("declarations", false, "answer each composed machine's declared paths, owners and modes")
requestFile := fs.String("request", "", "read the request from this file instead of standard input (a run step that hands no standard input)")
if err := fs.Parse(args); err != nil {
return 2
}
if fs.NArg() > 0 {
fmt.Fprintf(stderr, "mesh-compose: unexpected arguments %q\n", fs.Args())
return 2
}
var raw []byte
var err error
if *requestFile != "" {
raw, err = os.ReadFile(*requestFile)
} else {
raw, err = io.ReadAll(stdin)
}
if err != nil {
fmt.Fprintln(stderr, "mesh-compose: the request cannot be read:", err)
return 1
}
var req composer.Request
if strings.TrimSpace(string(raw)) == "" {
if *catalogueDir == "" {
fmt.Fprintln(stderr, "mesh-compose: no request on standard input and no -catalogue")
return 1
}
req = composer.Request{V: composer.Version}
} else if req, err = composer.Decode(raw); err != nil {
fmt.Fprintln(stderr, "mesh-compose:", err)
return 1
}
if *catalogueDir != "" {
read, err := readCatalogue(*catalogueDir)
if err != nil {
fmt.Fprintln(stderr, "mesh-compose:", err)
return 1
}
if len(req.Cases) == 0 {
req.Cases = []composer.Case{{Name: "catalogue", Manifests: map[string]composer.Manifest{}}}
}
if req.Cases[0].Manifests == nil {
req.Cases[0].Manifests = map[string]composer.Manifest{}
}
for name, m := range read {
if _, given := req.Cases[0].Manifests[name]; !given {
req.Cases[0].Manifests[name] = m
}
}
// No list of predefined keys is read: the composer judges no manifest (novox/hq ADR 0578 §5).
}
req.Machines = append(req.Machines, machines...)
if *depth != "" {
req.Depth = *depth
}
if *declarations {
req.Declarations = true
}
v, err := composer.Run(req)
if err != nil {
fmt.Fprintln(stderr, "mesh-compose:", err)
return 1
}
enc := json.NewEncoder(stdout)
if err := enc.Encode(v); err != nil {
fmt.Fprintln(stderr, "mesh-compose: the verdict cannot be written:", err)
return 1
}
return 0
}
// readCatalogue reads every module.json of a catalogue checkout by its module's name: the root's and each of
// modules/*/. A manifest whose name cannot be read is kept under its directory's name, so the reader's refusal is said.
func readCatalogue(dir string) (map[string]composer.Manifest, error) {
paths, err := filepath.Glob(filepath.Join(dir, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
if _, err := os.Stat(filepath.Join(dir, "module.json")); err == nil {
paths = append(paths, filepath.Join(dir, "module.json"))
}
if len(paths) == 0 {
return nil, fmt.Errorf("the catalogue %s holds no module.json", dir)
}
sort.Strings(paths)
out := map[string]composer.Manifest{}
for _, p := range paths {
raw, err := os.ReadFile(p)
if err != nil {
return nil, err
}
var probe struct {
Module string `json:"module"`
}
name := filepath.Base(filepath.Dir(p))
if json.Unmarshal(raw, &probe) == nil && probe.Module != "" {
name = probe.Module
}
if _, twice := out[name]; twice {
return nil, errors.New("the catalogue " + dir + " defines " + name + " twice")
}
out[name] = composer.Manifest{Raw: raw}
}
return out, nil
}
+339
View File
@@ -0,0 +1,339 @@
package main
import (
"bytes"
"encoding/json"
"go/build"
"os"
"os/exec"
"path/filepath"
"reflect"
"runtime"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/beside"
"github.com/novox/mesh-host/internal/compose"
"github.com/novox/mesh-host/internal/composer"
)
// captured is every manifest of the catalogue captured in testdata/beside (internal/beside), raw, by module name.
func captured(t *testing.T) map[string]composer.Manifest {
t.Helper()
read, err := readCatalogue(filepath.Dir(beside.Catalogue(t)))
if err != nil {
t.Fatal(err)
}
if len(read) < 100 {
t.Fatalf("the captured catalogue holds %d manifests, fewer than the catalogue's", len(read))
}
return read
}
// meshOf is a mesh of three machines over the captured catalogue, each assigned real modules, the input the assigner
// composes with (nodes and their assignments, holdings from the claims): anchor runs the bus and the store, laptop a
// desktop's modules, ace a module whose requirement is provided elsewhere.
func meshOf() compose.Input {
return compose.Input{Nodes: []compose.Node{
{Name: "anchor", Assigned: []string{"nats", "postgres"}},
{Name: "laptop", Assigned: []string{"sudo", "zsh"}},
{Name: "ace", Assigned: []string{"mesh-reads", "nats"}},
}}
}
func request(t *testing.T, depth string) composer.Request {
t.Helper()
r := composer.Request{V: composer.Version, Depth: depth, HoldingsFromClaims: true,
Machines: []string{"anchor", "laptop", "ace"},
Cases: []composer.Case{{Name: "trunk", Manifests: captured(t)}}}
return withInput(t, r, meshOf())
}
// withInput is r with in as its input, as a caller puts it on the wire: the request's shape carries the input as JSON.
func withInput(t *testing.T, r composer.Request, in compose.Input) composer.Request {
t.Helper()
raw, err := json.Marshal(in)
if err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(raw, &r.Input); err != nil {
t.Fatal(err)
}
return r
}
func runJSON(t *testing.T, args []string, in []byte) (int, composer.Verdict, string) {
t.Helper()
var out, errOut bytes.Buffer
code := run(args, bytes.NewReader(in), &out, &errOut)
var v composer.Verdict
if code == 0 {
if err := json.Unmarshal(out.Bytes(), &v); err != nil {
t.Fatalf("the verdict is not JSON: %v\n%s", err, out.String())
}
}
return code, v, errOut.String()
}
func encode(t *testing.T, r composer.Request) []byte {
t.Helper()
raw, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
return raw
}
// 1621/AC1: the command reads the plain input and the manifests on standard input, validates every manifest, composes
// the machines named and writes one verdict: per machine composes or the reason, per manifest accepted or the field
// refused.
func TestTheCommandWritesOneVerdictPerManifestAndMachine(t *testing.T) {
r := request(t, composer.DepthResolve)
r.Cases[0].Manifests["later"] = composer.Manifest{Raw: json.RawMessage(`{"module":"later","version":"1","shares":["x"]}`)}
r.Cases[0].Manifests["refused"] = composer.Manifest{Raw: json.RawMessage(`{"module":"refused","version":"1"}`)}
r.Cases[0].Verdicts = map[string]delivery.ManifestVerdict{"refused": {State: delivery.VerdictRefused, Problems: []string{"it is wrong"}}}
code, v, stderr := runJSON(t, nil, encode(t, r))
if code != 0 {
t.Fatalf("exit %d: %s", code, stderr)
}
if len(v.Cases) != 1 || v.Cases[0].Name != "trunk" || v.SDK == "" {
t.Fatalf("verdict = %+v", v)
}
c := v.Cases[0]
// ADR 0578 §5, §6: the composer judges nothing; it sets an unknown field aside and names it, and holds a manifest
// its verdict refuses, named in the indexer's words.
if got := c.Manifests["later"]; !got.Accepted || !strings.Contains(got.SetAside, `"shares"`) {
t.Errorf("the manifest with an unknown field reads %+v, want accepted naming the field set aside", got)
}
if got := c.Manifests["refused"]; got.Accepted || !strings.Contains(got.Refused, "refused the manifest of refused: it is wrong") {
t.Errorf("the refused manifest reads %+v, want refused in the verdict's words", got)
}
if got := c.Manifests["nats"]; !got.Accepted {
t.Errorf("nats reads %+v, want accepted", got)
}
if len(c.Machines) != 3 {
t.Fatalf("machines = %v, want the three named", c.Machines)
}
for name, m := range c.Machines {
if !m.Composes && m.Reason == "" {
t.Errorf("%s does not compose and says no reason", name)
}
}
}
// 1621/AC3: over the captured catalogue and a mesh of three machines, the command's verdict equals, machine for
// machine and manifest for manifest, what the linked reader, resolution and composition answer, at both depths.
func TestTheCommandAnswersWhatTheLinkedCompositionAnswers(t *testing.T) {
for _, depth := range []string{composer.DepthResolve, composer.DepthCompose} {
r := request(t, depth)
code, v, stderr := runJSON(t, nil, encode(t, r))
if code != 0 {
t.Fatalf("%s: exit %d: %s", depth, code, stderr)
}
in := meshOf()
in.Catalogue = map[string]catalogue.Manifest{}
for name, m := range r.Cases[0].Manifests {
// A request without a list of predefined keys reads a fallback as the indexer does (Request.Predefined).
parsed, err := catalogue.Decode(m.Raw)
got := v.Cases[0].Manifests[name]
if (err == nil) != got.Accepted || (err != nil && err.Error() != got.Refused) {
t.Errorf("%s: the manifest of %s: linked %v, command %+v", depth, name, err, got)
}
if err == nil {
in.Catalogue[name] = parsed
}
}
in.Holdings = composer.HoldingsOf(in.Catalogue, in.Nodes)
composed := 0
for _, node := range r.Machines {
got := v.Cases[0].Machines[node]
var want composer.Machine
if depth == composer.DepthResolve {
if _, err := compose.Resolve(in, node); err != nil {
want = composer.Machine{Reason: err.Error()}
} else {
want = composer.Machine{Composes: true}
}
} else {
d, problems, err := compose.Compose(in, node)
switch {
case err != nil:
want = composer.Machine{Reason: err.Error()}
default:
want = composer.Machine{Composes: len(problems) == 0, Problems: problems, Refusals: d.Refusals()}
if len(problems) > 0 {
want.Reason = "the node-engine's validator refuses the declaration"
}
}
}
if !reflect.DeepEqual(got, want) {
t.Errorf("%s: %s: command %+v, linked %+v", depth, node, got, want)
}
if got.Composes {
composed++
} else {
t.Logf("%s: %s does not compose: %s %v", depth, node, got.Reason, got.Problems)
}
}
t.Logf("%s: %d of %d machines compose", depth, composed, len(r.Machines))
}
}
// 1621 note 6: a person composes a catalogue checkout (repository@ref checked out or archived: the command fetches
// nothing) for a node and reads its declaration: each path with its kind, owner and mode.
func TestACatalogueCheckoutIsComposedForANode(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "modules", "demo"), 0o755); err != nil {
t.Fatal(err)
}
manifest := `{"module": "demo", "version": "1", "resources": [{"id": "state", "type": "directory", ` +
`"path": "/var/lib/demo", "mode": "0750", "owner": "root"}]}`
if err := os.WriteFile(filepath.Join(dir, "modules", "demo", "module.json"), []byte(manifest), 0o644); err != nil {
t.Fatal(err)
}
in := encode(t, withInput(t, composer.Request{V: composer.Version}, compose.Input{Nodes: []compose.Node{{Name: "laptop", Assigned: []string{"demo"}}}}))
code, v, stderr := runJSON(t, []string{"-catalogue", dir, "-node", "laptop", "-declarations"}, in)
if code != 0 {
t.Fatalf("exit %d: %s", code, stderr)
}
m := v.Cases[0].Machines["laptop"]
want := []composer.Path{{Path: "/var/lib/demo", Kind: "directory", Owner: "root", Mode: "0750", Module: ""}}
if !m.Composes || len(m.Declared) != 1 || m.Declared[0].Path != want[0].Path || m.Declared[0].Mode != "0750" ||
m.Declared[0].Owner != "root" || m.Declared[0].Kind != "directory" {
t.Fatalf("laptop = %+v, want it composed declaring %+v", m, want)
}
}
// A run step hands the command no standard input: the request is read from the file -request names, and a file that is
// not there is a request it cannot read.
func TestTheRequestIsReadFromAFile(t *testing.T) {
file := filepath.Join(t.TempDir(), "request.json")
r := composer.Request{V: composer.Version, Cases: []composer.Case{{Name: "only", Manifests: map[string]composer.Manifest{
"demo": {Raw: json.RawMessage(`{"module":"demo","version":"1"}`)}}}}}
if err := os.WriteFile(file, encode(t, r), 0o600); err != nil {
t.Fatal(err)
}
code, v, stderr := runJSON(t, []string{"-request", file}, nil)
if code != 0 || len(v.Cases) != 1 || !v.Cases[0].Manifests["demo"].Accepted {
t.Fatalf("exit %d (%s), verdict %+v", code, stderr, v)
}
if code, _, stderr := runJSON(t, []string{"-request", file + ".gone"}, nil); code != 1 || !strings.Contains(stderr, "cannot be read") {
t.Errorf("a request file that is not there: exit %d, %q", code, stderr)
}
}
// The exit codes: 1 for a request that cannot be read or run, 2 for wrong flags, each said on standard error.
func TestTheExitCodesSayWhatFailed(t *testing.T) {
unfilled := encode(t, composer.Request{V: composer.Version, Cases: []composer.Case{{Name: "c", Manifests: map[string]composer.Manifest{"x": {Digest: "abc"}}}}})
for _, c := range []struct {
name string
args []string
in string
code int
says string
}{
{"not JSON", nil, "{", 1, "not JSON"},
{"another version", nil, `{"v":2}`, 1, "version 2"},
{"a digest left unfilled", nil, string(unfilled), 1, "digest"},
{"an unknown depth", nil, `{"v":1,"depth":"deep"}`, 1, "depth"},
{"nothing to read", nil, "", 1, "no request"},
{"a wrong flag", []string{"-nope"}, "", 2, "nope"},
{"an argument", []string{"extra"}, "", 2, "unexpected"},
} {
var out, errOut bytes.Buffer
if got := run(c.args, strings.NewReader(c.in), &out, &errOut); got != c.code || !strings.Contains(errOut.String(), c.says) {
t.Errorf("%s: exit %d saying %q, want %d saying %q", c.name, got, errOut.String(), c.code, c.says)
}
}
}
// ownPackages is every package of this repository the command imports, directly or not, with the packages each
// imports: what this repository adds to the composition; mesh-sdk's packages are judged in mesh-sdk.
func ownPackages(t *testing.T) map[string][]string {
t.Helper()
_, file, _, _ := runtime.Caller(0)
root := filepath.Join(filepath.Dir(file), "..", "..")
const module = "github.com/novox/mesh-host"
out := map[string][]string{}
var walk func(path string)
walk = func(path string) {
if _, done := out[path]; done {
return
}
rel := strings.TrimPrefix(strings.TrimPrefix(path, module), "/")
pkg, err := build.Default.ImportDir(filepath.Join(root, rel), 0)
if err != nil {
t.Fatalf("%s: %v", path, err)
}
out[path] = pkg.Imports
for _, imp := range pkg.Imports {
if strings.HasPrefix(imp, module+"/") {
walk(imp)
}
}
}
walk(composer.Package)
return out
}
// 1621/AC2 and AC4: the command imports, through every package of this repository it reaches, only the standard
// library, mesh-sdk and this repository's own packages, and none that reaches a bus, a store, a network service or
// another process.
func TestTheCommandImportsNoBusStoreNetworkOrProcess(t *testing.T) {
forbidden := []string{"net/http", "net/rpc", "net/smtp", "os/exec", "database/sql", "plugin", "syscall"}
own := ownPackages(t)
if len(own) < 4 {
t.Fatalf("the command reaches %d packages of this repository; the walk found too few to judge", len(own))
}
for path, imports := range own {
for _, imp := range imports {
std := !strings.Contains(strings.SplitN(imp, "/", 2)[0], ".")
if !std && !strings.HasPrefix(imp, "git.novox.be/novox/mesh-sdk/go") && !strings.HasPrefix(imp, "github.com/novox/mesh-host/") {
t.Errorf("%s imports %s, which is neither the standard library, mesh-sdk nor this repository", path, imp)
}
for _, f := range forbidden {
if imp == f {
t.Errorf("%s imports %s", path, imp)
}
}
}
}
}
// 1621/AC2: run with an empty network namespace, the command still writes its verdict: it needs no network. The test
// runs its own binary again under `unshare --net` as an unprivileged user's namespace; where the kernel or the sandbox
// allows no new namespace (a build module's run step is already one with no network), it says so and skips.
func TestTheCommandRunsWithNoNetwork(t *testing.T) {
if os.Getenv("MESH_COMPOSE_HELPER") == "1" {
os.Exit(run(nil, os.Stdin, os.Stdout, os.Stderr))
}
unshare, err := exec.LookPath("unshare")
if err != nil {
t.Skip("no unshare on this machine, so no empty network namespace can be made here")
}
if out, err := exec.Command(unshare, "--net", "--map-root-user", "true").CombinedOutput(); err != nil {
t.Skipf("this machine makes no unprivileged network namespace (%v: %s); a run step's own has no network", err, out)
}
r := request(t, composer.DepthResolve)
cmd := exec.Command(unshare, "--net", "--map-root-user", os.Args[0], "-test.run=^TestTheCommandRunsWithNoNetwork$")
cmd.Env = []string{"MESH_COMPOSE_HELPER=1"}
cmd.Stdin = bytes.NewReader(encode(t, r))
var out, errOut bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errOut
if err := cmd.Run(); err != nil {
t.Fatalf("the command in an empty network namespace: %v: %s", err, errOut.String())
}
var v composer.Verdict
if err := json.Unmarshal(out.Bytes(), &v); err != nil || len(v.Cases) != 1 || len(v.Cases[0].Machines) != 3 {
t.Fatalf("no verdict from the command with no network: %v\n%s", err, out.String())
}
names := make([]string, 0, len(v.Cases[0].Machines))
for n := range v.Cases[0].Machines {
names = append(names, n)
}
sort.Strings(names)
t.Logf("composed with no network: %v", names)
}
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+107
View File
@@ -0,0 +1,107 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
)
// appliedStore keeps every composition the agent applied, one file per module (the node's own as `node.json`), in the
// directory `applied` beside the node's state (agent.Env.Applied, issue 774). It is the machine's own record of what runs:
// read at the agent's start before anything is watched, so the node never composes its own as if a module it runs ran
// nothing because the bucket no longer keeps that composition or the kept declaration is retired. Files are 0600 in a
// 0700 directory: a composition carries values sealed to this machine.
type appliedStore struct{ dir string }
func appliedStoreFor(statePath string) appliedStore {
return appliedStore{dir: filepath.Join(filepath.Dir(statePath), "applied")}
}
// appliedName is a module name the store takes as a file name: a catalogue module's or the node's own, never a path.
var appliedName = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]*$`)
func (s appliedStore) path(module string) (string, error) {
if !appliedName.MatchString(module) || strings.Contains(module, "..") {
return "", fmt.Errorf("%q is not a module name the applied store takes", module)
}
return filepath.Join(s.dir, module+".json"), nil
}
// holds says whether the store keeps the module's composition.
func (s appliedStore) holds(module string) bool {
path, err := s.path(module)
if err != nil {
return false
}
_, err = os.Stat(path)
return err == nil
}
func (s appliedStore) Load() (map[string][]byte, error) {
entries, err := os.ReadDir(s.dir)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
out := map[string][]byte{}
var errs []error
for _, e := range entries {
module, ok := strings.CutSuffix(e.Name(), ".json")
if !ok || e.IsDir() || !appliedName.MatchString(module) {
continue
}
raw, err := os.ReadFile(filepath.Join(s.dir, e.Name()))
if err != nil {
errs = append(errs, err)
continue
}
out[module] = raw
}
return out, errors.Join(errs...)
}
func (s appliedStore) Keep(module string, raw []byte) error {
path, err := s.path(module)
if err != nil {
return err
}
if was, err := os.ReadFile(path); err == nil && string(was) == string(raw) {
return nil
}
if err := os.MkdirAll(s.dir, 0o700); err != nil {
return err
}
tmp, err := os.CreateTemp(s.dir, "."+module+".*")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
if _, err := tmp.Write(raw); err != nil {
tmp.Close()
return err
}
if err := tmp.Sync(); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
return os.Rename(tmp.Name(), path)
}
func (s appliedStore) Drop(module string) error {
path, err := s.path(module)
if err != nil {
return err
}
if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
return nil
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// authorityPath is where this engine keeps the mesh's own certificate as it last applied it (public): beside the
// certificate issued for the machine's name, read into its record (novox/hq ADR 0555 §3).
func authorityPath(state string) string { return filepath.Join(filepath.Dir(state), "authority.crt") }
// keepAuthority keeps the mesh's own certificate a composition carries (the resource a module's certificate.authority
// asks for, catalogue.AuthorityID), so the record says the authority this machine holds. A composition carrying none
// changes nothing.
func keepAuthority(state string, c delivery.Composed) error {
for _, raw := range c.Resources {
var r struct {
ID string `json:"id"`
Content string `json:"content"`
}
if json.Unmarshal(raw, &r) != nil || r.Content == "" || !strings.HasSuffix(r.ID, "."+catalogue.AuthorityID()) {
continue
}
path := authorityPath(state)
if was, err := os.ReadFile(path); err == nil && string(was) == r.Content {
return nil
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
return os.WriteFile(path, []byte(r.Content), 0o644)
}
return nil
}
+167
View File
@@ -0,0 +1,167 @@
package main
import (
"context"
"crypto/ed25519"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"os"
"path/filepath"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-host/internal/identity"
)
// Issue 1126 (ADR 0555 §3): the mesh's own certificate a composition carries is kept for the machine's record, and the
// record's keys read it; a composition carrying none changes nothing. And (L3 s6, s7; ADR 0550 §2) the record says what
// the node trusts a mesh signature from, and the node keeps the certificate the vault signed for it.
func TestTheAppliedAuthorityIsKeptForTheRecord(t *testing.T) {
state := filepath.Join(t.TempDir(), "state", "state.json")
res := func(id, content string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": id, "type": "file", "path": "/etc/x", "content": content})
return raw
}
if err := keepAuthority(state, delivery.Composed{Resources: []json.RawMessage{res("web.conf", "x")}}); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(authorityPath(state)); !os.IsNotExist(err) {
t.Fatalf("a composition with no authority kept one: %v", err)
}
if err := keepAuthority(state, delivery.Composed{Resources: []json.RawMessage{res("ca-trust.certificate-authority", "CA-1")}}); err != nil {
t.Fatal(err)
}
if got := recordKeys(state, nil)(); got.Authority != "CA-1" {
t.Errorf("the record's keys read the authority %q", got.Authority)
}
// Issue 1126, A6 (ADR 0550 §2; the controller ends by its unassignment): a node enrolled with the controller's key
// trusts it never, whatever the lease says; before the vault's signers are read it trusts nothing, once read the
// vault's key alone, and the record names the vault's alone. Each change tells the record's writer.
controllerPub, controller, _ := ed25519.GenerateKey(nil)
vaultPub, vault, _ := ed25519.GenerateKey(nil)
trust := newMeshTrust()
told := 0
trust.onChange(func() { told++ })
keys := recordKeys(state, trust.Trusts)
ctrlFP, vaultFP := delivery.SignerFingerprint(controllerPub), delivery.SignerFingerprint(vaultPub)
enrol, _ := delivery.SignContext(delivery.KindEnrolment)
msg := []byte("a reply")
if got := keys().Trusts; len(got) != 0 {
t.Errorf("before the signers are read the record trusts %v", got)
}
if _, err := trust.Verify(delivery.KindEnrolment, msg, delivery.SignMessage(controller, enrol, msg)); err == nil {
t.Error("a controller-signed message is accepted")
}
if _, err := trust.Verify(delivery.KindEnrolment, msg, delivery.SignMessage(vault, enrol, msg)); err == nil {
t.Error("a vault-signed message is accepted before the vault's key is read")
}
vaultCA := testAuthority(t, vaultPub, vault)
keyValue, _ := json.Marshal(delivery.NewSignerKey(vaultPub, time.Now()))
fp, _ := delivery.AuthorityFingerprint(vaultCA)
authValue, _ := json.Marshal(delivery.SignerAuthority{Certificate: vaultCA, Fingerprint: fp, Made: time.Now()})
trust.heardSigner(delivery.SignersKeyKey, keyValue, false, func(string) {})
trust.heardSigner(delivery.SignersAuthorityKey, authValue, false, func(string) {})
if s, err := trust.Verify(delivery.KindEnrolment, msg, delivery.SignMessage(vault, enrol, msg)); err != nil || s.Name != delivery.SignerVault {
t.Errorf("a vault-signed message on a node that read signers: %+v, %v", s, err)
}
if _, err := trust.Verify(delivery.KindEnrolment, msg, delivery.SignMessage(controller, enrol, msg)); err == nil {
t.Error("a controller-signed message is accepted once the vault's key is read")
}
if got := keys().Trusts; strings.Contains(strings.Join(got, " "), ctrlFP) || !delivery.TrustsKey(got, delivery.SignerVault, vaultFP) ||
trust.VaultAuthority() != vaultCA || told != 2 {
t.Errorf("after the signers are read the record trusts %v (told %d)", got, told)
}
// A signers' key naming another fingerprint is not trusted.
bad, _ := json.Marshal(delivery.SignerKey{Key: base64.StdEncoding.EncodeToString(controllerPub), Fingerprint: vaultFP})
trust.heardSigner(delivery.SignersKeyKey, bad, false, func(string) {})
if !delivery.TrustsKey(keys().Trusts, delivery.SignerVault, vaultFP) {
t.Error("a signers' key that does not check replaced the vault's")
}
// L3 s7: the certificate the vault signs for this node's internal name and serving key is kept where the record reads
// it, and the record is written again; one signed by another authority is refused and nothing is kept.
serving, err := identity.GenerateServingKey()
if err != nil {
t.Fatal(err)
}
if err := identity.WriteServingKey(identity.ServingKeyPath(state), serving); err != nil {
t.Fatal(err)
}
servingPub, _ := base64.StdEncoding.DecodeString(serving.Public)
signedBy := func(ca string, key ed25519.PrivateKey) string {
parent, _ := delivery.ParseCertificate(ca)
der, err := x509.CreateCertificate(nil, delivery.NodeCertificateTemplate(overlay.InternalName("n1"), servingPub, time.Now().Add(-time.Hour)),
parent, ed25519.PublicKey(servingPub), key)
if err != nil {
t.Fatal(err)
}
return string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))
}
var asked []any
answer := signedBy(vaultCA, vault)
ask := func(_ context.Context, verb string, args any) (json.RawMessage, error) {
asked = append(asked, verb, args)
return json.Marshal(delivery.SignAnswer{Kind: delivery.KindNodeCertificate, Node: "n1", Certificate: answer})
}
written := 0
if err := certifyThisNode(context.Background(), state, "n1", trust, ask, func() { written++ }); err != nil {
t.Fatal(err)
}
if got := recordKeys(state, nil)(); got.Certificate != answer || written != 1 || len(asked) != 2 || asked[0] != delivery.SignVerb {
t.Errorf("the vault's certificate: kept %v, written %d, asked %v", got.Certificate == answer, written, asked)
}
answer = signedBy(testAuthority(t, controllerPub, controller), controller)
if err := certifyThisNode(context.Background(), state, "n1", trust, ask, func() { written++ }); err == nil || written != 1 {
t.Errorf("a certificate of another authority was kept: %v", err)
}
// Issue 774: what the engine applied is kept beside the state too, one file per module, 0600 in a 0700 directory,
// written by a rename so a crash leaves the one before; a name that is no module's is refused, never a path.
t.Run("the compositions the engine applied are kept beside the state", func(t *testing.T) {
applied := appliedStoreFor(filepath.Join(t.TempDir(), "state", "state.json"))
if m, err := applied.Load(); err != nil || len(m) != 0 || applied.holds("node") {
t.Fatalf("an empty store loads %v, %v", m, err)
}
for _, tc := range []struct {
module string
takes bool
}{{"node", true}, {"nats", true}, {"mesh-delivery-view", true}, {"../state", false}, {"a/b", false}, {"..", false}, {"", false}} {
if err := applied.Keep(tc.module, []byte(`{"v":1}`)); (err == nil) != tc.takes {
t.Errorf("keeping %q: %v, want taken %v", tc.module, err, tc.takes)
}
}
if err := applied.Keep("node", []byte(`{"v":2}`)); err != nil {
t.Fatal(err)
}
entries, _ := os.ReadDir(applied.dir)
dirInfo, _ := os.Stat(applied.dir)
fileInfo, _ := os.Stat(filepath.Join(applied.dir, "node.json"))
if len(entries) != 3 || dirInfo.Mode().Perm() != 0o700 || fileInfo.Mode().Perm() != 0o600 {
t.Fatalf("the store holds %d files (want 3, no temporary one left), directory %v, file %v", len(entries), dirInfo.Mode(), fileInfo.Mode())
}
if err := applied.Drop("nats"); err != nil || applied.Drop("nats") != nil || applied.holds("nats") {
t.Fatalf("a dropped module is still held, or dropping twice fails: %v", err)
}
if m, err := applied.Load(); err != nil || string(m["node"]) != `{"v":2}` || len(m) != 2 {
t.Fatalf("the store loads %v, %v", m, err)
}
})
}
// testAuthority is a self-signed authority certificate for key, as the vault makes its own.
func testAuthority(t *testing.T, pub ed25519.PublicKey, key ed25519.PrivateKey) string {
t.Helper()
tpl := delivery.AuthorityTemplate(pub, time.Now().Add(-time.Hour))
der, err := x509.CreateCertificate(nil, tpl, tpl, pub, key)
if err != nil {
t.Fatal(err)
}
return string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))
}
+46
View File
@@ -3,9 +3,14 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/link"
)
func backupEvent(node, module string, ok bool, at time.Time, why string) []byte {
@@ -39,4 +44,45 @@ func TestABackupEndsWhenItsEventSaysSo(t *testing.T) {
if done, words := run(50 * time.Millisecond); done || !strings.Contains(words, "did not say it was done") {
t.Errorf("silence within the bound: done=%v words=%q", done, words)
}
// The node agent stopping while it waits is no failure of the backup: said as interrupted, so its next start backs
// the module up again (postgres on novox, 2026-10-31).
stopped, cancel := context.WithCancel(context.Background())
cancel()
if got := awaitBackup(stopped, make(chan []byte), "n1", "db", since, time.Second); got.Done || !got.Interrupted ||
got.Words != agent.BackupInterrupted {
t.Errorf("a stopped node agent: %+v", got)
}
if got := awaitBackup(context.Background(), make(chan []byte), "n1", "db", since, 10*time.Millisecond); got.Interrupted {
t.Errorf("silence within the bound taken as interrupted: %+v", got)
}
// The ask that starts it (sweep 2026-10-30): one that never reached the holder (nothing on this machine answers it
// while the tool runner restarts) is asked again within its bound, and past it fails saying so; one that may have
// reached it is never asked twice.
notAsked := fmt.Errorf("node-backup.now could not be asked: %w: nothing on this machine answers "+
"mesh.seat.node-backup.tool.now.ace", link.ErrNotAsked)
for _, row := range []struct {
name string
answers []error
within time.Duration
asked int
words string
}{
{"heard after the tool runner is back", []error{notAsked, notAsked, nil}, time.Second, 3, ""},
// Asked again until the bound, however slow the machine: at least twice, and fewer than it would answer.
{"never heard within the bound", []error{notAsked, notAsked, notAsked, notAsked, notAsked, nil}, 25 * time.Millisecond,
0, "times over"},
{"a timeout may have started it", []error{errors.New("no answer from x: context deadline exceeded"), nil},
time.Second, 1, "deadline exceeded"},
{"a refusal in words", []error{errors.New("restic is not on this machine"), nil}, time.Second, 1, "not on this"},
} {
asked := 0
_, err := askUntilHeard(context.Background(), func(context.Context) (json.RawMessage, error) {
asked++
return json.RawMessage(`{}`), row.answers[asked-1]
}, row.within, func(int) time.Duration { return 10 * time.Millisecond })
if row.asked == 0 && (asked < 2 || asked >= len(row.answers)) || row.asked != 0 && asked != row.asked || (row.words == "") != (err == nil) || err != nil && !strings.Contains(err.Error(), row.words) {
t.Errorf("%s: asked %d times, %v", row.name, asked, err)
}
}
}
+391
View File
@@ -0,0 +1,391 @@
package main
import (
"bufio"
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"sync/atomic"
"syscall"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
meshasker "git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/accounts"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/selfcompose"
"github.com/novox/mesh-host/internal/store"
)
// The node seat node-engine's judgements on this machine (novox/hq ADR 0563): what a hand-over and a library's take
// would set and what the disk holds now, and whether this machine is root-free.
// diskOf is what is on disk at a directory, as an act compares it when the operator's yes arrives: its owner and
// group, its mode, and which file it is.
func diskOf(info os.FileInfo) string {
owner := "unknown"
if uid, gid, ok := apply.OwnerOf(info); ok {
owner = fmt.Sprintf("%d:%d", uid, gid)
}
which := ""
if st, ok := info.Sys().(*syscall.Stat_t); ok {
which = fmt.Sprintf(" %v:%v", st.Dev, st.Ino)
}
return fmt.Sprintf("%s %o%s", owner, info.Mode().Perm(), which)
}
// handOverJudgeFrom judges a hand-over through the node seat as the controller's ask is judged (judgeFound): the
// declared owner and mode it will set, and the disk as it is.
func handOverJudgeFrom(statePath string) func(string) (link.Judged, error) {
return func(path string) (link.Judged, error) {
found, info, err := judgeFound(statePath, path)
if err != nil {
return link.Judged{}, err
}
return link.Judged{Subject: path, Path: path, Sets: orUnsaid(found.Declared), Disk: diskOf(info),
Words: "A directory of " + moduleOf(found.ID) + " is used as found: the mesh left its owner and mode."}, nil
}
}
func moduleOf(id string) string {
if at := strings.LastIndex(id, "."); at > 0 {
return id[:at]
}
return id
}
// accessJudgeFrom judges the take act on a library through the node seat (ADR 0328 §7, ADR 0563 §4): declared on this
// machine, its home, and its root holding data the mesh has not taken; what the take sets, and the disk as it is.
func accessJudgeFrom(statePath string) func(string, string) (link.Judged, error) {
return func(act, access string) (link.Judged, error) {
d, err := declaredNow(statePath, nil)
if errors.Is(err, store.ErrNothingDeclared) {
return link.Judged{}, errors.New("this node keeps no declaration, so it is declared no library")
}
if err != nil {
return link.Judged{}, fmt.Errorf("what runs on this node does not read: %w", err)
}
for _, l := range d.Libraries {
if apply.LibraryAccess(l.Library) != access {
continue
}
if !l.Home {
return link.Judged{}, fmt.Errorf("this machine is not the home of the library %s, and a library is "+
"taken on its home", l.Library)
}
ctx, cancel := context.WithTimeout(context.Background(), lookBound)
defer cancel()
if v := apply.JudgeLibrary(ctx, l); v.Kind != apply.VerdictWaitKeep {
return link.Judged{}, fmt.Errorf("the library %s does not wait for the operator's word to be kept "+
"(it is judged %s), so there is nothing to take", l.Library, v.Kind)
}
info, err := os.Lstat(l.Root)
if err != nil {
return link.Judged{}, fmt.Errorf("the library %s's root cannot be read: %v", l.Library, err)
}
return link.Judged{Subject: access, Path: filepath.Clean(l.Root),
Sets: "the group " + l.Group + ", group write and setgid on everything under its root",
Disk: diskOf(info), Words: "The library " + l.Library + " holds data the mesh has not taken."}, nil
}
return link.Judged{}, fmt.Errorf("the access %s names no library declared on this machine", access)
}
}
// askTakes asks the take of every library the look found waiting for the operator's word, once per library (ADR 0563
// §4): asked by the engine itself, never a second time while one is open or after one was asked; access-act asks again
// by hand.
func askTakes(acts *link.HandActs, waiting []string, say func(string)) {
if acts == nil {
return
}
for _, access := range waiting {
if acts.AskedFor(link.ActAccess, access) {
continue
}
if _, err := acts.AskAccess(link.EngineSeat+"-"+acts.Node, link.ActTake, access,
"its root holds data outside its group", true); err != nil && say != nil {
say("the take of " + access + " could not be asked: " + err.Error())
}
}
}
// lastAccounts is the newest look at every account (the health loop's), which root-free judges from.
var lastAccounts atomic.Pointer[accounts.Statement]
// rootFreshFor is how old the verdict root-free judges from may be (ADR 0563 §6, the controller's verdictFreshFor).
const rootFreshFor = 15 * time.Minute
// judgeRootFree is this machine's root-free (ADR 0259 §8, ADR 0563 §6): free only on the engine's own verdict, at most
// rootFreshFor old, that every account declared never to become root without a person cannot, at least one such
// account declared, and the login shell's execute not served here. Anything else is no pass, and says why.
func judgeRootFree(node string, st *accounts.Statement, now time.Time, served bool, servedWhy string) link.RootVerdict {
v := link.RootVerdict{Machine: node, Judged: now.UTC()}
var not []string
confined := ""
switch {
case st == nil:
not = append(not, "this machine's engine has judged no account yet")
case now.Sub(st.At) > rootFreshFor:
not = append(not, fmt.Sprintf("the engine's newest judgement of its accounts was made at %s, more than %d "+
"minutes ago, and a verdict that old is not a verdict on now", st.At.UTC().Format(time.RFC3339),
int(rootFreshFor.Minutes())))
default:
var named []string
for _, a := range st.Accounts {
if !a.Root {
continue
}
named = append(named, a.Name)
if a.State != accounts.Healthy {
not = append(not, "the account "+a.Name+" "+orNoneKnownWord(a.Reason))
}
}
if len(named) == 0 {
not = append(not, "no account here is declared never to become root, so agents run as the operator's "+
"account, which may become root")
} else if len(not) == 0 {
confined = "the account " + strings.Join(named, ", ") + " cannot become root without a person (judged " +
st.At.UTC().Format(time.RFC3339) + ")"
}
}
if served {
not = append(not, "the login shell runs any command an agent gives it as the machine's runtime account, which "+
"can become root ("+servedWhy+")")
}
if len(not) > 0 {
v.Why = strings.Join(not, "; ")
return v
}
v.Free, v.Why = true, confined+"; the login shell's execute is not served here"
return v
}
func orNoneKnownWord(s string) string {
if s == "" {
return "is not judged"
}
return s
}
// settingFileSuffix is the resource the login shell's holder renders its setting execute into (ADR 0268: zsh's
// tools-conf, `execute=serve` or `execute=withhold`), read beside the bus's discovery.
const settingFileSuffix = ".tools-conf"
// executeServedHere is whether the login shell's execute is served on this machine (ADR 0268), failing closed: served
// while the bus's discovery hears it answered here or could not be asked, or while the holder's setting file applied
// here says anything but withhold or cannot be read. why says which.
func executeServedHere(statePath string, heard func() (bool, error)) (bool, string) {
var why []string
if heard == nil {
why = append(why, "the bus could not be asked whether execute is answered here")
} else if h, err := heard(); err != nil {
why = append(why, "the bus could not be asked whether execute is answered here: "+err.Error())
} else if h {
why = append(why, "the bus hears execute answered here")
}
known, err := store.Load(statePath)
if err != nil {
why = append(why, "what this machine applied could not be read: "+err.Error())
}
for _, r := range known.Resources {
if r.Type != "file" || !strings.HasSuffix(r.ID, settingFileSuffix) {
continue
}
if value, err := settingIn(r.Target); err != nil {
why = append(why, moduleOf(r.ID)+"'s setting execute cannot be read: "+err.Error())
} else if value != "withhold" {
why = append(why, moduleOf(r.ID)+"'s setting execute here is "+value)
}
}
return len(why) > 0, strings.Join(why, "; ")
}
// settingIn is the value of execute in a setting file, "" when it says none.
func settingIn(path string) (string, error) {
f, err := os.Open(path)
if err != nil {
return "", err
}
defer f.Close()
value := ""
lines := bufio.NewScanner(f)
for lines.Scan() {
t := strings.TrimSpace(lines.Text())
if key, v, ok := strings.Cut(t, "="); ok && !strings.HasPrefix(t, "#") && strings.TrimSpace(key) == "execute" {
value = strings.TrimSpace(v)
}
}
if value == "" {
value = "not said"
}
return value, lines.Err()
}
// keepWaiting is every library a look found waiting for the operator's word to keep it (ADR 0328 §8).
func keepWaiting(vs []link.ResourceHealth) []string {
var out []string
for _, v := range vs {
for _, w := range v.AccessWaits {
if w.Reason == link.WaitKeep {
out = append(out, w.Access)
}
}
}
return out
}
// engineActs is the node seat this engine holds (ADR 0563): judged and performed by the same code as the controller's
// signed asks, its asks to the operator over the session's link, its acts kept beside the state.
func engineActs(opts options, node string, queue *link.Session, search *accounts.SetuidCache) *link.HandActs {
state := link.BusState{Session: queue, Node: node}
h := &link.HandActs{Node: node, State: state,
JudgeHandOver: handOverJudgeFrom(opts.state), HandOver: handOverFrom(opts.state),
JudgeAccess: accessJudgeFrom(opts.state), AccessAct: accessActFrom(opts.state, opts.timeout),
JudgeDirectories: judgeDirectoriesFrom(opts.state), SetuidSearch: queue.SetuidSearch, Searching: search.Running,
RootFree: func() link.RootVerdict {
served, why := executeServedHere(opts.state, func() (bool, error) {
return queue.HeardServed("node-login-shell", "execute")
})
return judgeRootFree(node, lastAccounts.Load(), time.Now(), served, why)
},
Say: queue.SayEngine}
asker := asks.EngineAsker(node)
h.Asker = &meshasker.Client{Module: asker, State: state, Publish: queue.PublishAsk,
Record: func(id string) (*stdio.StateEntry, error) { return queue.RouterRecord(asker + "." + id) }}
return h
}
// reconcileEvery is how often the engine catches up its open asks from the router's records (a warrant heard while no
// link was open, an ask past its expiry): besides each link's opening.
const reconcileEvery = 5 * time.Minute
func reconcileActs(ctx context.Context, h *link.HandActs) {
t := time.NewTicker(reconcileEvery)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
h.Reconcile()
}
}
}
// compositionReadFor bounds one read of the verb composition: the revisions of one key from the bucket.
const compositionReadFor = 20 * time.Second
// compositionFrom answers the verb composition from the node agent that runs now (issue 774); refused in words while
// none does.
func compositionFrom(current *agentNow) func(module, digest string, diff bool) (any, error) {
return func(module, digest string, diff bool) (any, error) {
ag := current.get()
if ag == nil {
return nil, errors.New("this engine's node agent has not started, so it holds no composition yet")
}
ctx, cancel := context.WithTimeout(context.Background(), compositionReadFor)
defer cancel()
return ag.Composition(ctx, module, digest, diff)
}
}
// composeReadFor bounds the verb compose's wait for its pass: a pass that waits on a seat longer says what it waits on.
const composeReadFor = 2 * time.Minute
// composeFrom answers the verb compose (issue 774): a pass of the composer that runs now, and with module that module's
// apply tried again by the node agent; refused in words while the node composes nothing of its own.
func composeFrom(current *agentNow, composer *atomic.Pointer[selfcompose.Composer]) func(module string) (any, error) {
return func(module string) (any, error) {
c := composer.Load()
if c == nil {
return nil, errors.New("this node composes none of its own modules now (it applies the controller's " +
"compositions, or its node agent is starting), so there is no pass to make")
}
out := map[string]any{}
if module != "" {
// Only a failed apply that is not held is tried again; what runs is never taken away (review round 1: the
// verb has no warrant, ADR 0563 §2, and a hold for a failed backup is the operator's to release, ADR 0568 §4).
ag := current.get()
if ag == nil {
return nil, errors.New("this engine's node agent has not started, so no apply is tried again")
}
if err := ag.RetryFailed(module); err != nil {
return nil, err
}
out["retried"] = module
}
ctx, cancel := context.WithTimeout(context.Background(), composeReadFor)
defer cancel()
pass, err := c.ComposeNow(ctx)
if err != nil {
return nil, err
}
out["pass"] = pass
return out, nil
}
}
// packagesReadFor bounds the verb packages' reads and its write.
const packagesReadFor = 20 * time.Second
// packagesFrom answers the verb packages (novox/hq ADR 0565 §6, §15): this node's key of the engine's state packages;
// with write, the key written again from what the
// composer that runs now handed, refused in words while the node composes none of its own.
func packagesFrom(node string, session *link.Session, composer *atomic.Pointer[selfcompose.Composer]) func(write bool) (any, error) {
return func(write bool) (any, error) {
ctx, cancel := context.WithTimeout(context.Background(), packagesReadFor)
defer cancel()
var wrote *bool
if write {
c := composer.Load()
if c == nil {
return nil, errors.New("this node composes none of its own modules now (its node agent is starting), " +
"so it has no packages of its own to write")
}
w, err := c.WritePackages(ctx)
if err != nil {
return nil, fmt.Errorf("this node's key of %s is not written: %w", link.PackagesBucket, err)
}
wrote = &w
}
key, keyErr := link.EnginePackages{Session: session}.Get(ctx, node)
v := link.PackagesAnswer(node, key, keyErr)
v.Wrote = wrote
return v, nil
}
}
// composeCandidateFrom answers the verb compose-candidate (novox/hq ADR 0593 §7): the check's candidate composed on this
// node in shadow by the run's shadow, applying and writing nothing, within delivery.ComposeAnswerBound. The answer is
// the seat's shape (delivery.ComposeCandidateAnswer), checked as it is encoded; a node whose shadow does not run yet
// answers unready, which the checker takes as no answer.
func composeCandidateFrom(node string, shadow *atomic.Pointer[selfcompose.Shadow]) func(candidate string, input bool) (any, error) {
return func(candidate string, input bool) (any, error) {
ask := delivery.ComposeCandidateAsk{Candidate: candidate, Input: input}
if _, err := delivery.EncodeComposeCandidateAsk(ask); err != nil {
return nil, err
}
var a delivery.ComposeCandidateAnswer
if s := shadow.Load(); s == nil {
// Unready, never an error the checker reads words of: its agent is starting (ADR 0593 §9).
a = delivery.ComposeCandidateAnswer{Node: node, Candidate: candidate, Verdict: delivery.ComposeCandidateUnready,
Refused: "this node's shadow does not run now (its node agent is starting), so it composes no candidate"}
} else {
ctx, cancel := context.WithTimeout(context.Background(), delivery.ComposeAnswerBound)
defer cancel()
a = s.ComposeCandidate(ctx, ask)
}
raw, err := delivery.EncodeComposeCandidateAnswer(a)
if err != nil {
return nil, fmt.Errorf("this node's answer to %s does not fit the seat's shape: %w", candidate, err)
}
return json.RawMessage(raw), nil
}
}
+86
View File
@@ -0,0 +1,86 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// enrolThroughSeat joins this machine through the seat mesh-enrolment (novox/hq ADR 0587 §6, issue 774): it redeems the token
// with the keys it made, verifies the vault's signed answer against the key the token carries before it keeps anything, keeps
// its membership and its login, dials as the login node-logins names and confirms. A bus that does not serve the seat to this
// token (link.ErrNoSeat) is said as the error: nothing was sent or kept.
func enrolThroughSeat(ctx context.Context, opts options, token identity.Token, node string, mine identity.Identity,
sealing identity.SealingKey, serving identity.ServingKey, reported map[string]any, found *link.Tunnel, identityPath string) error {
req := delivery.Redeem{Node: node, PublicKey: base64.StdEncoding.EncodeToString(mine.Public), OverlayKey: mine.Overlay.Public,
SealingKey: sealing.Public, ServingKey: serving.Public, Profile: reported}
if found != nil {
var err error
if req.Tunnel, err = json.Marshal(found); err != nil {
return err
}
}
req.Proof = base64.StdEncoding.EncodeToString(mine.Sign(delivery.RedeemProof(node, req.PublicKey, req.OverlayKey, req.SealingKey, req.ServingKey)))
to := link.Approach{Address: token.Broker, Fingerprint: token.Fingerprint, Transport: link.OnNATS}
membership := link.Membership{Node: node, Broker: token.Broker, Fingerprint: token.Fingerprint, Transport: link.OnNATS}
j := link.Joining{Node: node, Signer: token.SignerKey(),
Redeem: func(ctx context.Context) (delivery.RedeemAnswer, error) {
return link.Redeem(ctx, to, node, token.Secret, req, opts.timeout)
},
Open: func(a delivery.RedeemAnswer) (string, error) {
value, err := sealing.Open(a.Account)
return string(value), err
},
Keep: func(a delivery.RedeemAnswer, password string) error {
// The mesh's name for this node wins over what the machine called itself, as on the controller's path.
mine.Node = a.Node
mine.Membership = identity.Membership{Broker: token.Broker, Fingerprint: token.Fingerprint, Signer: token.Signer,
Password: password, Transport: link.OnNATS}
if err := writeLogins(loginsFile(opts.state), link.LoginsState{Current: a.Login, Values: map[string]string{a.Login: password}}); err != nil {
return fmt.Errorf("cannot keep the login %s: %w", a.Login, err)
}
if err := identity.Save(identityPath, mine); err != nil {
return err
}
if !mine.Membership.Joined() {
return errors.New("the answer did not say how to reach the mesh again, so the identity could not be used after a restart")
}
return keepJoinedKeys(opts, mine, sealing, serving)
},
Dial: func(ctx context.Context, login, password string) (link.Confirming, error) {
return link.DialConfirming(ctx, membership, login, password, opts.timeout)
},
}
answer, err := j.Run(ctx)
if err != nil {
return err
}
fmt.Printf("\nenrolled as %s through the seat mesh-enrolment, dialled as %s and confirmed\n", answer.Node, answer.Login)
fmt.Printf(" identity %s\n", identityPath)
fmt.Println(" nothing is applied until the placement of this node names a module")
return nil
}
// keepJoinedKeys writes the private halves of the keys this machine made, after its identity: an identity naming a key that is
// not there looks joined and cannot come up, so the identity is written first and a failure here is said plainly.
func keepJoinedKeys(opts options, mine identity.Identity, sealing identity.SealingKey, serving identity.ServingKey) error {
if err := os.WriteFile(identity.OverlayKeyPath(opts.state), []byte(mine.Overlay.Private+"\n"), 0o600); err != nil {
return fmt.Errorf("cannot write this node's overlay key: %w", err)
}
if err := os.WriteFile(identity.SealingKeyPath(opts.state), []byte(sealing.Private+"\n"), 0o600); err != nil {
return fmt.Errorf("cannot write this node's sealing key: %w", err)
}
if err := identity.WriteServingKey(identity.ServingKeyPath(opts.state), serving); err != nil {
return fmt.Errorf("cannot write this node's serving key: %w", err)
}
return nil
}
+73
View File
@@ -2,6 +2,7 @@ package main
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
@@ -10,6 +11,7 @@ import (
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/link"
)
@@ -82,6 +84,77 @@ func TestADeclarationWithAFileByDigestAppliesFromTheObject(t *testing.T) {
if err != nil || string(got) != string(body) {
t.Errorf("1695/AC1: the file holds %d bytes (%v), want %d", len(got), err, len(body))
}
// Issue 924 (novox, 2026-10-29 01:16): the controller's sweep removed an object a composition the node ran still named,
// and the module's apply failed while it ran. Content the node once had is its own keep's, and the bus is not asked.
gone := aFilesLink{err: fmt.Errorf("the object %s was deleted from the bucket declaration-files-novox", name)}
never := []byte(strings.Repeat("never had here\n", 6000))
for _, row := range []struct {
name string
raw []byte
onDisk []byte // nil removes the file
refused string // "" applies
}{
{"the file here drifted, its object deleted", raw, []byte("the module wrote it since"), ""},
{"the file here is gone, its object deleted", raw, nil, ""},
{"never had here, its object deleted", referenceDeclaration(target, never), nil, "never held it"},
} {
if row.onDisk == nil {
_ = os.Remove(target)
} else if err := os.WriteFile(target, row.onDisk, 0o600); err != nil {
t.Fatal(err)
}
report = applyAndKeep(link.WithFiles(context.Background(), gone), opts, row.raw, nil, nil, nil)
if row.refused == "" {
got, _ := os.ReadFile(target)
if report.Refused != "" || string(got) != string(body) {
t.Errorf("924: %s: refused %q, the file holds %d bytes; want it applied from the node's own keep",
row.name, report.Refused, len(got))
}
} else if !strings.Contains(report.Refused, row.refused) || !strings.Contains(report.Refused, "was deleted") {
t.Errorf("924: %s: refused %q; want a refusal saying the bucket and the keep both lack it", row.name, report.Refused)
}
}
// Issue 924, the live case: what runs came from the kept declaration, whose export names a digest swept long ago,
// while the node wrote a newer export since. Composed as it runs, the file names what the node wrote, and applies with
// no object; a file someone else changed since is left as named.
newer := []byte(strings.Repeat("the export the node wrote since\n", 3000))
newerName, _ := declaration.ObjectName(declaration.DigestOf(newer))
ctx = link.WithFiles(context.Background(), aFilesLink{objects: map[string][]byte{newerName: newer}})
if report = applyAndKeep(ctx, opts, referenceDeclaration(target, newer), nil, nil, nil); report.Refused != "" {
t.Fatalf("924: the newer export: %s", report.Refused)
}
stale := json.RawMessage(`{"id":"m.big","type":"file","path":"` + target + `","digest":"` + declaration.DigestOf(body) +
`","size":` + strconv.Itoa(len(body)) + `}`)
for _, row := range []struct {
name string
edit []byte // what someone else wrote there since; nil for nobody
digest string
}{
{"the file is what the node wrote", nil, declaration.DigestOf(newer)},
{"someone changed the file since", []byte("a hand edit"), declaration.DigestOf(body)},
} {
if row.edit != nil {
if err := os.WriteFile(target, row.edit, 0o600); err != nil {
t.Fatal(err)
}
}
runs := runsAsWritten([]delivery.Composed{{Node: "n", Module: "m", Resources: []json.RawMessage{stale}}}, opts.state)
var got struct {
Digest string `json:"digest"`
}
_ = json.Unmarshal(runs[0].Resources[0], &got)
if got.Digest != row.digest {
t.Errorf("924: %s: what runs names %s, want %s", row.name, got.Digest, row.digest)
}
if row.edit == nil {
raw := []byte(`{"declaration":1,"resources":[` + string(runs[0].Resources[0]) + `]}`)
if report = applyAndKeep(link.WithFiles(context.Background(), gone), opts, raw, nil, nil, nil); report.Refused != "" {
t.Errorf("924: %s: composed as it runs, refused %q; want applied with no object", row.name, report.Refused)
}
}
}
}
// 1695/AC8: the hand-run `apply <file>` and `reconcile` have no bus: a reference whose file here verifies is applied
+76 -10
View File
@@ -3,22 +3,25 @@ package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"fmt"
"path/filepath"
"sync/atomic"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/provisions"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// serveFilling writes filled files on the provisioner client's request for as long as this engine serves (novox/hq
// ADR 0349 §4). A socket that cannot be served is said once, and this engine goes on applying.
func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, say func(string)) {
e := fillingEngine(opts, signer, say)
func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, queue *link.Session, node string, say func(string)) {
e := fillingEngine(opts, signer, provisionSecretsOn(queue, node, opts.state), say)
fillingNow.Store(e)
defer fillingNow.CompareAndSwap(e, nil)
if err := filling.Serve(ctx, filling.Socket, e, say); err != nil {
@@ -30,9 +33,10 @@ func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, s
var fillingNow atomic.Pointer[filling.Engine]
// knockFilling tells the provisioner's client to fill again what the filling refused or failed and the declaration
// applied now may let through (filling.Engine.Knock), once a module's apply or removal settled (agent.Env.Settled).
// What it told, or why it could not, is said.
func knockFilling(module string, say func(string)) {
// applied now may let through (filling.Engine.Knock), after what runs changed: a module's apply or removal settled
// (agent.Env.Settled, "<module> settled"), the haves were read at the start, or a module was taken as running
// (agent.Env.RunsChanged, issue 924). What it told, or why it could not, is said.
func knockFilling(after string, say func(string)) {
e := fillingNow.Load()
if e == nil {
return
@@ -40,16 +44,19 @@ func knockFilling(module string, say func(string)) {
told, err := e.Knock(filling.Socket)
switch {
case err != nil:
say(fmt.Sprintf("after %s settled: %v", module, err))
say(fmt.Sprintf("after %s: %v", after, err))
case told != "":
say(fmt.Sprintf("after %s settled, the provisioner's client is told to fill again: %s", module, told))
say(fmt.Sprintf("after %s, the provisioner's client is told to fill again: %s", after, told))
}
}
// fillingEngine is the write of a filled file as this engine does it: the declaration it keeps, verified, read under
// the apply's lock — the same two locks every apply takes, the process's and the one beside the state — and the file
// written through the apply (apply.WriteFilled), its outcome recorded in the node's state as an apply's is.
func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *filling.Engine {
//
// A provision's `${secret:}` the client left in the content is filled first (secrets, novox/hq ADR 0570 clause 6), and a
// file so filled is written 0600 when the declaration gives no mode.
func fillingEngine(opts options, signer ed25519.PublicKey, secrets provisionSecrets, say func(string)) *filling.Engine {
return &filling.Engine{
Declared: func() (*declaration.Declaration, error) {
return declaredNow(opts.state, signer)
@@ -73,8 +80,13 @@ func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *fi
if err != nil {
return "", "", err
}
report, updated, writeErr := apply.WriteFilled(ctx, built, d, known, path, content, apply.ExecRunner,
announceOr(say), apply.KeepIn(filepath.Dir(opts.state)), apply.WindowsIn(filepath.Dir(opts.state)))
content, secret, err := secrets.fill(ctx, filledModule(d, path), content)
if err != nil {
return "", "", fmt.Errorf("%s was not written: %w", path, err)
}
report, updated, writeErr := apply.WriteFilledWith(ctx, built, d, known, path, content, apply.ExecRunner,
announceOr(say), apply.KeepIn(filepath.Dir(opts.state)), apply.WindowsIn(filepath.Dir(opts.state)),
apply.Filled{Secret: secret})
// Saved whichever way it went, as an apply's: what was written is on the machine either way.
if err := store.Save(opts.state, updated); err != nil {
return "", "", errors.Join(writeErr, errors.New("written, and the node's state could not be saved: "+
@@ -93,3 +105,57 @@ func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *fi
},
}
}
// filledModule is the module whose file the declaration declares at path, or "" when none or several do.
func filledModule(d *declaration.Declaration, path string) string {
module := ""
for _, r := range d.Resources {
if f, ok := r.(*declaration.File); ok && f.Path == path {
if module != "" {
return ""
}
module, _ = d.ModuleOfID(f.ID)
}
}
return module
}
// provisionSecretsOn reads provisions and asks the vault over the link, and opens with this machine's sealing key.
func provisionSecretsOn(queue *link.Session, node, statePath string) provisionSecrets {
p := provisionSecrets{node: node, open: sealOpener(statePath)}
if queue == nil {
return p
}
// What the bus holds under prefix, read by a watch to its caught-up mark: no poll, no list of every key.
p.list = func(ctx context.Context, prefix string) (map[string][]byte, error) {
wctx, cancel := context.WithCancel(ctx)
defer cancel()
updates, err := queue.KVWatch(wctx, provisions.ProvisionsBucket, prefix+">")
if err != nil {
return nil, err
}
out := map[string][]byte{}
for {
select {
case <-ctx.Done():
return nil, ctx.Err()
case u, ok := <-updates:
if !ok {
return nil, fmt.Errorf("the watch of %s ended before it caught up", provisions.ProvisionsBucket)
}
switch {
case u.Caught:
return out, nil
case u.Deleted:
delete(out, u.Key)
default:
out[u.Key] = u.Value
}
}
}
}
p.own = func(ctx context.Context, args map[string]any) (json.RawMessage, error) {
return queue.AskVault(ctx, "own", args)
}
return p
}
+102 -1
View File
@@ -7,9 +7,12 @@ import (
"os"
"os/user"
"path/filepath"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/provisions"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/store"
)
@@ -45,7 +48,34 @@ func TestAWriteAndAnApplyOfTheSameFileSerialise(t *testing.T) {
store.Declared{Declaration: raw, Signature: ed25519.Sign(private, raw)}); err != nil {
t.Fatal(err)
}
e := fillingEngine(opts, public, func(string) {})
// The provision m reads, as mesh-provisioner keeps it, and the vault's value of it, sealed to this machine (faked:
// the opener reads the box's "v").
provision := map[string]string{}
vaultName := "provision/n2/db"
secrets := provisionSecrets{node: "n1",
list: func(_ context.Context, prefix string) (map[string][]byte, error) {
out := map[string][]byte{}
for k, v := range provision {
if strings.HasPrefix(k, prefix) {
out[k] = []byte(v)
}
}
return out, nil
},
own: func(_ context.Context, args map[string]any) (json.RawMessage, error) {
raw, _ := json.Marshal(args)
var a struct{ Credentials []map[string]any }
_ = json.Unmarshal(raw, &a)
c := a.Credentials[0]
c["name"], c["generation"], c["sealed"] = vaultName, 2, map[string]string{"v": "s3cret"}
return json.Marshal(map[string]any{"node": "n1", "credentials": []any{c}})
},
open: func(sealed string) ([]byte, error) {
var b struct{ V string }
err := json.Unmarshal([]byte(sealed), &b)
return []byte(b.V), err
}}
e := fillingEngine(opts, public, secrets, func(string) {})
uid := uint32(os.Getuid())
if err := e.Accepts(uid); err != nil {
t.Fatal(err)
@@ -102,4 +132,75 @@ func TestAWriteAndAnApplyOfTheSameFileSerialise(t *testing.T) {
if a := <-done; a.Action != "updated" {
t.Fatalf("the second write: %+v", a)
}
// ADR 0570 clause 6: the client leaves a provision's `${secret:}` to the node-engine, which fills it at the write
// from the vault's value of a provided provision, and writes the file 0600; a provision not provided refuses it.
keyed := func(state string, ref bool, id, local string) string {
p := provisions.Provision{Provision: "db", Local: local, Consumer: provisions.End{Module: "m", Node: "n1"},
Provider: provisions.End{Module: "postgres", Node: "n2"}, State: state, At: time.Now()}
if state != provisions.Provided {
p.Why = "the provider has not answered"
}
if ref {
p.Reference = &provisions.Reference{ID: id, Generation: 2}
}
raw, err := provisions.Encode(p)
if err != nil {
t.Fatal(err)
}
return string(raw)
}
for _, tc := range []struct {
name, state string
ref bool
content string
want string // the file after; "" keeps what was there
mode os.FileMode
vault string // the vault's credential name, when not the one the provision names
refID string // the reference's id, when not the provision's shared value
local string // the provision's local name (ADR 0094): the placeholder names it
}{
{name: "a provision not provided is refused", state: provisions.Asked, content: "password=${secret:db}\n"},
{name: "provided with no reference is refused", state: provisions.Provided, content: "password=${secret:db}\n"},
{name: "a reference to no credential of the provision is refused", state: provisions.Provided, ref: true,
refID: "provision/n9/db", content: "password=${secret:db}\n"},
{name: "a name no provision goes by is refused", state: provisions.Provided, ref: true, content: "password=${secret:other}\n"},
{name: "the vault's value is another than the provision names", state: provisions.Provided, ref: true,
vault: "pair/n2/db/n1/m", content: "password=${secret:db}\n"},
{name: "provided: filled at the write, 0600", state: provisions.Provided, ref: true,
content: "password=${secret:db}\n", want: "password=s3cret\n", mode: 0o600},
{name: "a local name is the provision's by its key", state: provisions.Provided, ref: true, local: "admin",
content: "admin=${secret:admin}\n", want: "admin=s3cret\n", mode: 0o600},
{name: "no placeholder is written as it came, as the declaration says", content: "plain\n", want: "plain\n", mode: 0o644},
} {
before, _ := os.ReadFile(path)
if tc.state != "" {
id := "provision/n2/db"
if tc.refID != "" {
id = tc.refID
}
clear(provision)
key := "n1.m.db"
if tc.local != "" {
key += "." + tc.local
}
provision[key] = keyed(tc.state, tc.ref, id, tc.local)
}
vaultName = "provision/n2/db"
if tc.vault != "" {
vaultName = tc.vault
}
a := e.Write(context.Background(), uid, filling.Request{Path: path, Content: []byte(tc.content)})
got, _ := os.ReadFile(path)
if tc.want == "" {
if a.Failed == "" && a.Refused == "" || string(got) != string(before) {
t.Errorf("%s: answered %+v, the file holds %q", tc.name, a, got)
}
continue
}
info, _ := os.Stat(path)
if a.Failed != "" || a.Refused != "" || string(got) != tc.want || info.Mode().Perm() != tc.mode {
t.Errorf("%s: answered %+v, the file holds %q, mode %v", tc.name, a, got, info.Mode())
}
}
}
+29 -19
View File
@@ -64,16 +64,16 @@ func foundDirectories(known store.State) []store.Applied {
return out
}
// handOverLine is the operator's line that hands a directory to the mesh, with this node's name when it is
// known (novox/hq ADR 0272): typed on the control-node, from the operator's own account. The directory's
// path is the resource's target beside the reason, never in it: the controller says a reason on the
// operator's channel, which withholds a path (ADR 0234 §6), and keeps the target as evidence.
// handOverLine is the way out of a directory used as found, with this node's name when it is known: the node seat's
// hand-over with the directory's path, which asks the operator on their channel (novox/hq ADR 0563 §5). The controller
// that answered `nox node hand-over` ended by its unassignment (issue 1126), so that line has no answerer. The path is
// the resource's target beside the reason, never in it: the operator's channel withholds a path (ADR 0234 §6).
func handOverLine() string {
node, _ := foundNode.Load().(string)
if node == "" {
node = "<node>"
}
return fmt.Sprintf("`nox node hand-over %s <directory>` on the control-node", node)
return fmt.Sprintf("`%s/node-engine.hand-over` with the directory's path, then your yes on your channel", node)
}
// withFoundDirectories adds to a statement every directory used as found, as its module's resource of kind
@@ -129,16 +129,30 @@ func foundNow() []store.Applied {
// never follows a path it did not state: a hand-over is root giving a directory to an account at the next
// apply, and a path that resolves elsewhere would give that elsewhere.
func handOverFound(statePath, path, by string) (string, error) {
found, _, err := judgeFound(statePath, path)
if err != nil {
return "", err
}
if err := store.AddHandedOver(statePath, store.HandOver{ID: found.ID, Path: path, At: time.Now().UTC(), By: by}); err != nil {
return "", err
}
return fmt.Sprintf("%s (%s) is handed to the mesh by %s: the next apply gives it %s, and it is the mesh's from "+
"then on — removed when empty and no longer declared", path, found.ID, by, orUnsaid(found.Declared)), nil
}
// judgeFound is the judgement of a hand-over's path, recording nothing: the directory this machine uses as found at
// that path, as the engine states it, and what is on disk there.
func judgeFound(statePath, path string) (store.Applied, os.FileInfo, error) {
if !filepath.IsAbs(path) {
return "", fmt.Errorf("hand-over takes the directory's absolute path, and %q is not one; nothing was handed over", path)
return store.Applied{}, nil, fmt.Errorf("hand-over takes the directory's absolute path, and %q is not one; nothing was handed over", path)
}
if filepath.Clean(path) != path {
return "", fmt.Errorf("%q is not the directory's path as the engine states it (no `..`, no doubled or trailing "+
return store.Applied{}, nil, fmt.Errorf("%q is not the directory's path as the engine states it (no `..`, no doubled or trailing "+
"separator); nothing was handed over", path)
}
known, err := store.Load(statePath)
if err != nil {
return "", err
return store.Applied{}, nil, err
}
var found *store.Applied
for _, r := range foundDirectories(known) {
@@ -149,31 +163,27 @@ func handOverFound(statePath, path, by string) (string, error) {
}
}
if found == nil {
return "", errors.New(path + " is not a directory this machine uses as found (the module's condition names " +
return store.Applied{}, nil, errors.New(path + " is not a directory this machine uses as found (the module's condition names " +
"the ones that are); nothing was handed over")
}
info, err := os.Lstat(path)
switch {
case err != nil:
return "", fmt.Errorf("%s cannot be read on this machine: %v; nothing was handed over", path, err)
return store.Applied{}, nil, fmt.Errorf("%s cannot be read on this machine: %v; nothing was handed over", path, err)
case info.Mode()&os.ModeSymlink != 0:
return "", fmt.Errorf("%s is a link, and the mesh hands over no link (novox/hq ADR 0012); nothing was handed over", path)
return store.Applied{}, nil, fmt.Errorf("%s is a link, and the mesh hands over no link (novox/hq ADR 0012); nothing was handed over", path)
case !info.IsDir():
return "", fmt.Errorf("%s is not a directory on this machine; nothing was handed over", path)
return store.Applied{}, nil, fmt.Errorf("%s is not a directory on this machine; nothing was handed over", path)
}
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return "", fmt.Errorf("%s cannot be resolved on this machine: %v; nothing was handed over", path, err)
return store.Applied{}, nil, fmt.Errorf("%s cannot be resolved on this machine: %v; nothing was handed over", path, err)
}
if resolved != path {
return "", fmt.Errorf("%s is reached through a link (it is %s), and the mesh hands over nothing through one "+
return store.Applied{}, nil, fmt.Errorf("%s is reached through a link (it is %s), and the mesh hands over nothing through one "+
"(novox/hq issue 339); nothing was handed over", path, resolved)
}
if err := store.AddHandedOver(statePath, store.HandOver{ID: found.ID, Path: path, At: time.Now().UTC(), By: by}); err != nil {
return "", err
}
return fmt.Sprintf("%s (%s) is handed to the mesh by %s: the next apply gives it %s, and it is the mesh's from "+
"then on — removed when empty and no longer declared", path, found.ID, by, orUnsaid(found.Declared)), nil
return *found, info, nil
}
// handOverFrom is the engine's hand-over for the controller's ask: the state beside statePath, the path as asked.
+6 -5
View File
@@ -15,7 +15,7 @@ import (
// Every directory used as found is its module's resource of kind directory, unhealthy, naming its owner, on
// every statement (novox/hq issue 339): the controller raises that as the module's condition. The way out it
// names is the operator's line, with this node's name (novox/hq ADR 0272, issue 356).
// names is the node seat's hand-over, with this node's name (novox/hq ADR 0563 §5).
func TestAFoundDirectoryIsStatedUnhealthy(t *testing.T) {
dir := t.TempDir()
known := store.State{Resources: []store.Applied{
@@ -35,14 +35,15 @@ func TestAFoundDirectoryIsStatedUnhealthy(t *testing.T) {
t.Fatalf("stated as: %+v", r)
}
// The path is the target, kept as evidence, and never in the reason: the operator's channel withholds a path
// (novox/hq ADR 0234 §6), and the controller says the reason there.
if !strings.Contains(r.Reason, "`nox node hand-over laptop <directory>` on the control-node") ||
// (novox/hq ADR 0234 §6). The way out is the node seat's hand-over (ADR 0563 §5), never the controller's terminal line,
// which nothing answers since the controller ended (issue 1126).
if !strings.Contains(r.Reason, "`laptop/node-engine.hand-over`") || strings.Contains(r.Reason, "nox node hand-over") ||
strings.Contains(r.Reason, "mesh-host") || strings.Contains(r.Reason, dir) {
t.Fatalf("the way out is not the operator's line, or carries the path: %q", r.Reason)
t.Fatalf("the way out is not the node seat's hand-over, or carries the path: %q", r.Reason)
}
foundNode.Store("")
if r := withFoundDirectories(&link.Health{}, foundDirectories(known)).Resources[0]; !strings.Contains(r.Reason,
"`nox node hand-over <node> <directory>`") {
"`<node>/node-engine.hand-over`") {
t.Fatalf("before enrolment the line names a placeholder: %q", r.Reason)
}
}
+1 -2
View File
@@ -21,7 +21,6 @@ type owedBus struct {
rv map[string]uint64
}
func (b *owedBus) Report(context.Context, string, []byte) error { return nil }
func (b *owedBus) Health(context.Context, string, []byte) error { return nil }
func (b *owedBus) HaveBind(context.Context) (link.HaveBucket, error) { return b, nil }
@@ -69,7 +68,7 @@ func TestAModuleWithNothingLongRunningIsOkWithinALookOfItsApply(t *testing.T) {
applying := fact
applying.Phase = string(delivery.PhaseApplying)
queue.NoteModule(ctx, applying)
go judgeWhatRuns(ctx, j, queue, func(string) {}, nil)
go judgeWhatRuns(ctx, j, queue, func(string) {}, nil, nil)
// The facts look inside the apply, while the module is still applying.
queue.SayFacts(ctx, link.Report{Health: &link.Health{Contract: link.LivenessContract, At: time.Now().UTC()}})
if h, _ := bus.have("n1.xorg"); h.Health != delivery.HealthUnknown {
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"context"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/liveness"
)
// heldBy is the containers an open maintenance window holds still, by runtime name (ADR 0189), at now.
func heldBy(windows *apply.Windows) func(time.Time) map[string]bool {
return func(now time.Time) map[string]bool {
held := map[string]bool{}
for _, w := range windows.Open(now) {
for _, name := range w.Holds {
held[name] = true
}
}
return held
}
}
// inactiveIn is what the agent asks at its start of a module it believes runs (ADR 0568 §1): whether the machine shows a
// long-running resource the composition declares running not active, read from the service manager and the container
// runtime and from no record. A resource held by an open maintenance window, or waiting for a library, is not counted;
// one that cannot be read is not counted either (unknown is not inactive), and neither is one the manager is restarting.
func inactiveIn(rt liveness.Runtime, held func(time.Time) map[string]bool) func(context.Context, delivery.Composed) bool {
return func(ctx context.Context, c delivery.Composed) bool {
d, err := declarationOf(c)
if err != nil {
return false
}
var holding map[string]bool
if held != nil {
holding = held(time.Now())
}
return anyInactive(ctx, rt, liveness.LongRunning(d, withWaiting(nil, apply.WaitingOnLibrary(d))), holding)
}
}
// anyInactive reads each group of resources once (the containers, each service manager's units) and says whether any is
// not active.
func anyInactive(ctx context.Context, rt liveness.Runtime, resources []liveness.Resource, held map[string]bool) bool {
type group struct {
kind, scope, user string
targets []string
}
var groups []*group
at := map[string]*group{}
for _, r := range resources {
if r.Kind == liveness.KindContainer && held[r.Target] {
continue
}
g := &group{kind: r.Kind, scope: r.Scope, user: r.User}
if r.Kind == liveness.KindContainer {
g.scope, g.user = "", ""
}
key := g.kind + "/" + g.scope + "/" + g.user
if r.Kind != liveness.KindContainer {
key = "units/" + g.scope + "/" + g.user
}
if at[key] == nil {
at[key] = g
groups = append(groups, g)
}
at[key].targets = append(at[key].targets, r.Target)
}
for _, g := range groups {
var got map[string]liveness.Observed
var err error
if g.kind == liveness.KindContainer {
got, err = rt.Containers(ctx, g.targets)
} else {
got, err = rt.Units(ctx, g.scope, g.user, g.targets)
}
if err != nil {
continue
}
for _, t := range g.targets {
if o := got[t]; !o.Found || !(o.Running || o.Restarting) {
return true
}
}
}
return false
}
+89
View File
@@ -0,0 +1,89 @@
package main
import (
"context"
"encoding/json"
"errors"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/liveness"
)
// 1957/AC2 and AC4 (unit; ADR 0568 §2, §4): the judge's look acts on a resource that goes down, once per edge. A repeated
// statement acts nothing, the edge clears when the resource is healthy again, and a resource held, restarting, starting or
// missing (the other path's) is no edge.
func testOnlyAResourceNewlyDownIsAnEdge(t *testing.T) {
state := func(id, s, reason string, missing bool) liveness.State {
return liveness.State{Resource: liveness.Resource{Module: "dnsmasq", ID: id}, State: s, Reason: reason, Missing: missing}
}
down := state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonDown, false)
steps := []struct {
name string
st []liveness.State
want []string
}{
{"down: an edge", []liveness.State{down}, []string{"dnsmasq"}},
{"said again while down: none", []liveness.State{down}, nil},
{"healthy again: clears", []liveness.State{state("dnsmasq.unit", liveness.Healthy, "", false)}, nil},
{"down again: an edge", []liveness.State{down}, []string{"dnsmasq"}},
{"held by a window: none", []liveness.State{state("dnsmasq.unit", liveness.Held, "", false)}, nil},
{"restarting: none", []liveness.State{state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonRestarting, false)}, nil},
{"starting: none", []liveness.State{state("dnsmasq.unit", liveness.Starting, "", false)}, nil},
{"missing: the other path's", []liveness.State{state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonDown, true)}, nil},
}
var last map[string]bool
for _, s := range steps {
var got []string
got, last = newlyInactive(last, liveness.Statement{Resources: s.st})
if len(got) != len(s.want) || (len(got) == 1 && got[0] != s.want[0]) {
t.Fatalf("%s: acted on %v, want %v", s.name, got, s.want)
}
}
}
type fakeRuntime struct {
units map[string]liveness.Observed
containers map[string]liveness.Observed
err error
}
func (f fakeRuntime) Containers(_ context.Context, _ []string) (map[string]liveness.Observed, error) {
return f.containers, f.err
}
func (f fakeRuntime) Units(_ context.Context, _, _ string, _ []string) (map[string]liveness.Observed, error) {
return f.units, f.err
}
// 1957/AC1 (unit; ADR 0568 §1): the machine's facts say a declared-running resource is not active when its unit is stopped
// or unknown to the manager; everything up, a restart in progress, a read that failed and a container held by a
// maintenance window are not.
func testTheMachineShowsAResourceNotActive(t *testing.T) {
svc := []byte(`{"id":"dnsmasq.unit","type":"service","unit":"dnsmasq.service","state":"running","module":"dnsmasq"}`)
ctr := []byte(`{"id":"dnsmasq.box","type":"container","name":"box","image":"x@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","module":"dnsmasq"}`)
up := liveness.Observed{Found: true, Running: true}
for _, c := range []struct {
name string
rt fakeRuntime
held map[string]bool
want bool
}{
{"all up", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": up}}, nil, false},
{"unit inactive", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": {Found: true}}, containers: map[string]liveness.Observed{"box": up}}, nil, true},
{"unit unknown to the manager", fakeRuntime{containers: map[string]liveness.Observed{"box": up}}, nil, true},
{"unit restarting", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": {Found: true, Restarting: true}}, containers: map[string]liveness.Observed{"box": up}}, nil, false},
{"container not running", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": {Found: true}}}, nil, true},
{"container held by a window", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": {Found: true}}}, map[string]bool{"box": true}, false},
{"unread", fakeRuntime{err: errors.New("the service manager could not be read")}, nil, false},
} {
t.Run(c.name, func(t *testing.T) {
d, err := declarationOf(delivery.Composed{Node: "n1", Module: "dnsmasq", Resources: []json.RawMessage{svc, ctr}})
if err != nil {
t.Fatal(err)
}
if got := anyInactive(context.Background(), c.rt, liveness.LongRunning(d, nil), c.held); got != c.want {
t.Fatalf("inactive = %v, want %v", got, c.want)
}
})
}
}
+40
View File
@@ -8,6 +8,8 @@ import (
"testing"
"time"
meshasker "git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
@@ -53,4 +55,42 @@ func TestALibrarySaysItsWaitWithoutWordsToParse(t *testing.T) {
if strings.Contains(media.Reason, dir) || len(books.AccessWaits) != 1 || books.AccessWaits[0].Reason != link.WaitDisk {
t.Fatalf("a missing root does not wait for its disk, or a reason names a path: %+v %+v", books, media)
}
// 774 (ADR 0563 §4): the library's home engine asks its take of the operator itself, once: no second ask while one is
// open, nor after it ended; and a library waiting for its disk is not asked.
t.Run("the home engine asks the take once", func(t *testing.T) {
var asked []asks.Ask
state := &link.MemState{}
h := &link.HandActs{Node: "n1", State: state,
JudgeAccess: func(act, access string) (link.Judged, error) {
return link.Judged{Subject: access, Path: root, Sets: "the group mesh-media", Disk: "0:0 755"}, nil
},
AccessAct: func(link.AccessAct) (string, error) { return "taken", nil }}
h.Asker = &meshasker.Client{Module: asks.EngineAsker("n1"), State: state,
Publish: func(subject string, body any, id string) error {
if a, ok := body.(asks.Ask); ok {
asked = append(asked, a)
}
return nil
}}
waiting := keepWaiting(vs)
askTakes(h, waiting, nil)
askTakes(h, waiting, nil)
if len(waiting) != 1 || waiting[0] != "library:media" || len(asked) != 1 {
t.Fatalf("waiting %v asked %d times", waiting, len(asked))
}
h.Heard(asks.Warrant{Ask: asked[0].ID, Asker: asks.EngineAsker("n1"), Outcome: asks.OutcomeExpired, At: time.Now()})
askTakes(h, waiting, nil)
if len(asked) != 1 {
t.Fatalf("the take was asked again after its ask ended: %d", len(asked))
}
// Kept in a key of its own (rev-l3 round 1): the acts gone from the record never make it ask again.
keys, _ := state.Keys("act.")
for _, k := range keys {
_ = state.Delete(k)
}
askTakes(h, waiting, nil)
if len(asked) != 1 {
t.Fatalf("the take was asked again with the acts gone: %d", len(asked))
}
})
}
+154
View File
@@ -0,0 +1,154 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// loginsFile is where this engine keeps the logins it holds between runs (novox/hq ADR 0583 §1): beside its identity, readable by
// its owner alone, as the identity's membership password is.
func loginsFile(state string) string {
return filepath.Join(filepath.Dir(identity.Path(state)), "logins.json")
}
// loadLogins is the engine's logins: what it kept, with the membership's password as the first login's.
func loadLogins(state, node, first string) *link.Logins {
path := loginsFile(state)
var saved link.LoginsState
if raw, err := os.ReadFile(path); err == nil {
_ = json.Unmarshal(raw, &saved)
}
return link.NewLogins(node, first, saved, func(s link.LoginsState) error { return writeLogins(path, s) })
}
func writeLogins(path string, s link.LoginsState) error {
raw, err := json.Marshal(s)
if err != nil {
return err
}
tmp := path + ".new"
if err := os.WriteFile(tmp, raw, 0o600); err != nil {
return err
}
return os.Rename(tmp, path)
}
// loginsFollowAgain bounds the wait before a watch of node-logins that could not be made is tried again on the same link:
// the vault may not serve rotation yet, and until it does this node is not granted the read.
var loginsFollowAgain = 2 * time.Minute
// followLogins follows this node's key of the vault's node-logins on every link, as a watch (never a poll), for as long as ctx
// lasts, and hands each value to hear. A watch that cannot be made is said once and tried again.
func followLogins(ctx context.Context, session *link.Session, node string, hear func(delivery.NodeLogin), say link.Announce) {
said := ""
for ctx.Err() == nil {
linked, cancel, err := session.WhenLinked(ctx)
if err != nil {
return
}
watching, stop := context.WithCancel(linked)
updates, werr := session.KVWatch(watching, delivery.NodeLoginsBucket, node)
if werr != nil {
stop()
if line := "this node's login is not followed yet (" + werr.Error() + "): it dials as it last did, and looks again in " + loginsFollowAgain.String(); line != said {
said = line
say(line)
}
} else {
said = ""
for u := range updates {
if u.Caught || u.Deleted || len(u.Value) == 0 {
continue
}
nl, err := delivery.DecodeNodeLogin(u.Value)
if err != nil {
say("the key " + node + " of " + delivery.NodeLoginsRead + " is not read: " + err.Error())
continue
}
hear(nl)
}
stop()
}
select {
case <-linked.Done():
case <-time.After(loginsFollowAgain):
}
cancel()
}
}
// loginSecretModule is the module the vault names this engine's login values under: own/<node>/mesh-host/<login> (ADR 0583 §3,
// insight).
const loginSecretModule = "mesh-host"
// fetchLogin asks the vault, through its existing hand-out of a machine's own secrets, for the value of login, sealed to this
// node's key, and opens it with the sealing key; no new verb is asked for it.
func fetchLogin(queue *link.Session, statePath string) func(context.Context, string) (string, error) {
unseal := sealOpener(statePath)
return func(ctx context.Context, login string) (string, error) {
raw, err := queue.AskVault(ctx, "own", map[string]any{})
if err != nil {
return "", err
}
var a struct {
Own []struct {
Module string `json:"module"`
Name string `json:"name"`
Sealed json.RawMessage `json:"sealed"`
} `json:"own"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return "", fmt.Errorf("the vault's own answer is not one: %w", err)
}
for _, o := range a.Own {
if o.Module != loginSecretModule || o.Name != login {
continue
}
sealed := string(o.Sealed)
var text string
if json.Unmarshal(o.Sealed, &text) == nil {
sealed = text
}
value, err := unseal(sealed)
if err != nil {
return "", fmt.Errorf("the value of %s cannot be opened with this node's sealing key: %w", login, err)
}
return string(value), nil
}
return "", fmt.Errorf("the vault hands this node no value for %s (own/%s/%s/%s)", login, queueNode(queue), loginSecretModule, login)
}
}
func queueNode(q *link.Session) string { return q.Membership.Node }
// newRotator is the engine's side of a rotation over this session (ADR 0583 §3-§6): the value fetched through the vault's own
// hand-out, the test-dial on a second connection, the confirmation over the new link, and a fall back carried in the health
// report.
func newRotator(queue *link.Session, statePath string, timeout time.Duration, say link.Announce) *link.Rotator {
r := &link.Rotator{Logins: queue.Membership.Logins, Say: say,
Fetch: fetchLogin(queue, statePath),
TestDial: func(ctx context.Context, login, password string) error {
return link.TestDial(ctx, queue.Membership, login, password, timeout)
},
Confirm: func(ctx context.Context) error {
login, _ := queue.Membership.Logins.Current()
if err := queue.WhenLinkedAs(ctx, login); err != nil {
return err
}
_, err := queue.AskVault(ctx, "node-account-confirm", map[string]any{})
return err
}}
r.Watch()
// The fall back is carried in this node's health report, from which the watchman raises node-account-fallback (ADR 0583 §6).
queue.Fallback = r.Fallback
return r
}
+202 -181
View File
@@ -30,6 +30,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/accounts"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/apply"
@@ -99,7 +100,7 @@ func versionAt(self, stamped string) string {
const usage = `nox-mesh-host — the node-engine
A daemon: its launcher runs it, and it applies what the mesh declares for this machine. The
operator's command is nox (mesh-cli), on any node, from their own account (novox/hq ADR 0272);
operator's command is nox (mesh-cli), on any node, from their own account, through the seats' verbs;
the subcommands here are for installing a machine, joining it, and rescuing one no mesh can
reach, run as root at the machine.
@@ -109,8 +110,7 @@ reach, run as root at the machine.
key make this machine's tunnel key, or read the one it made, and print the public
half: what its join token is issued for (novox/hq ADR 0169)
enrol --token T join the mesh — through the tunnel when the token was issued for a key
overlay take take over the tunnel found here (novox/hq ADR 0105): its key becomes this
node's overlay key and the mesh is told, signed; --tunnel <iface> when several are up
overlay take refused: the controller that took a rekey ended and no seat takes one now
owned what this host has applied and still owns
hand-over <dir> hand a directory used as found to the mesh: the next apply gives it its declared
owner and mode. The operator's line is nox node hand-over <node> <dir> on the
@@ -516,7 +516,7 @@ func runApply(ctx context.Context, opts options, d *declaration.Declaration, raw
// A file given by digest is had before the plan is drawn from it (novox/hq ADR 0486 §1, issue 1695). This command has
// no link to the bus, so a file already here that verifies is used, and any other is refused: never written empty.
if err := apply.ResolveFileReferences(ctx, d, link.FilesFrom(ctx)); err != nil {
if err := apply.ResolveFileReferencesKept(ctx, d, link.FilesFrom(ctx), objectKeep(opts)); err != nil {
return fmt.Errorf("a file of this declaration could not be had by its digest: %w", err)
}
@@ -808,162 +808,21 @@ func enrol(ctx context.Context, opts options) error {
detected := profile.Detect(ctx, profile.Default(nil), opts.timeout)
reported := profileAsReported(detected)
// Signed with the identity just generated, so the mesh can tell this machine from anyone else
// who knows its public key (novox/hq issue 083).
proof := mine.Sign(link.EnrolProof(token.Secret, mine.Public, mine.Overlay.Public,
sealing.Public, serving.Public))
// The token says where to go and which certificate that address must present. It says nothing
// about which bus is there, and does not need to: there is one, and this host knows which
// (novox/hq ADR 0131 — the mesh speaks to one seat and the old transport is gone).
//
// **It used to leave this empty** and mean "whatever the mesh runs today", which was true
// while two buses existed and became a refusal the moment one did: an empty transport is not
// the bus's name, so every enrolment ended at "this token is for the \"\" bus"
// (novox/hq 04-ISSUES/146). Nothing caught it because nothing had enrolled since the bus
// changed.
reply, err := link.Enrol(ctx,
link.Approach{Address: token.Broker, Fingerprint: token.Fingerprint, Transport: link.OnNATS},
*name, token.Secret,
mine.Public, mine.Overlay.Public, sealing.Public, serving.Public, reported, proof, found,
opts.timeout)
if err != nil {
return err
}
// The mesh's name for this node wins over what the machine called itself: the token was
// issued for a node record, and that record is what the identity binds to.
mine.Node = reply.Node
mine.Membership = identity.Membership{
Broker: firstNonEmpty(reply.Broker, token.Broker),
Fingerprint: firstNonEmpty(reply.Fingerprint, token.Fingerprint),
Signer: firstNonEmpty2(reply.Signer, token.Signer),
Password: reply.Password,
// **Which bus this membership is for, said rather than left empty** (novox/hq
// 04-ISSUES/146). The link refuses a membership that names another bus, and an empty name
// is not this one's — so a node enrolled without it came up and reconnected for ever
// against its own record: "this membership is for \"\", and the mesh's bus is nats". The
// reply does not carry it because there is one bus and the host knows which (ADR 0131);
// what was missing was writing that down where the link reads it.
Transport: link.OnNATS,
}
if mine.Membership.Password == "" {
// The mesh did not replace the token's secret, so it is still this node's broker
// password. Said rather than silently kept: a one-time secret living on as a credential
// is worth knowing about.
mine.Membership.Password = token.Secret
fmt.Println("\nnote: the mesh issued no separate broker password, so the token's secret " +
"remains this node's credential")
}
// Saved only now, and only once the mesh has said it knows this node. A node holding an
// identity the mesh has never recorded would believe it had joined and be believed by
// nobody — worse than not having joined, because nothing would look wrong.
if err := identity.Save(identityPath, mine); err != nil {
return fmt.Errorf(
"the mesh accepted this node as %q and its identity could not be saved: %w\n"+
"That token is spent, so getting back needs a new one", reply.Node, err)
}
if !mine.Membership.Joined() {
return fmt.Errorf(
"the mesh accepted this node as %q but did not say how to reach it again, so this "+
"identity could not be used after a restart. Nothing was saved", reply.Node)
}
// Written before the identity, so a node that dies between the two has a key file with no
// identity — which enrols again cleanly — rather than an identity naming a key that is not
// there, which looks joined and cannot come up.
if err := os.WriteFile(identity.OverlayKeyPath(opts.state),
[]byte(mine.Overlay.Private+"\n"), 0o600); err != nil {
return fmt.Errorf("cannot write this node's overlay key: %w", err)
}
if err := os.WriteFile(identity.SealingKeyPath(opts.state),
[]byte(sealing.Private+"\n"), 0o600); err != nil {
return fmt.Errorf("cannot write this node's sealing key: %w", err)
}
if err := identity.WriteServingKey(identity.ServingKeyPath(opts.state), serving); err != nil {
return fmt.Errorf("cannot write this node's serving key: %w", err)
}
fmt.Printf("\nenrolled as %s\n", reply.Node)
fmt.Printf(" identity %s\n", identityPath)
fmt.Printf(" queue %s\n", reply.Queue)
return nil
// Through the seat mesh-enrolment (novox/hq ADR 0587 §6), the one way in: the controller's enrolment subject ended
// with the controller (issue 1126).
return enrolThroughSeat(ctx, opts, token, *name, mine, sealing, serving, reported, found, identityPath)
}
// overlayCommand is `mesh-host overlay take`: this node takes the tunnel found on its machine over
// after it enrolled (novox/hq ADR 0105). For a node that enrolled before the mesh knew to take a
// tunnel over — re-enrolling would rotate its identity, sealing and serving keys, and with them
// every credential the mesh sealed to it.
func overlayCommand(ctx context.Context, opts options) error {
// overlayCommand is `mesh-host overlay take`: a node taking the tunnel found on its machine over after it enrolled
// (novox/hq ADR 0105). Refused, and nothing is changed: the rekey it sent was taken by the controller alone, which
// ended by its unassignment (issue 1126), and no seat takes one now; a key written here that no part of the mesh knows
// would raise the mesh's interface on a key its hub does not peer.
func overlayCommand(_ context.Context, opts options) error {
if opts.file != "take" {
return errors.New("overlay take [--tunnel <iface>] — the one thing `overlay` does here")
}
identityPath := identity.Path(opts.state)
mine, err := identity.Load(identityPath)
if err != nil {
return err
}
found, err := tunnel.Find(ctx, apply.ExecRunner, opts.tunnel)
if err != nil {
return fmt.Errorf("%w. Nothing was changed", err)
}
taken, rekey, err := rekeyOnto(mine, found)
if err != nil {
return err
}
fmt.Printf("taking over %s: this node's overlay key becomes the tunnel's, %s\n", found, taken.Overlay.Public)
fmt.Printf(" identity, sealing and serving keys are untouched\n")
// Told first, then written: a mesh told and a machine not yet written is put right by running
// this again (the mesh refuses the stale second rekey and changes nothing; the files are
// rewritten the same). A machine written and a mesh not told would raise the mesh's interface
// on a key the mesh does not know at the next restart.
if err := link.Publish(ctx, link.Membership{
Node: mine.Node, Broker: mine.Membership.Broker, Fingerprint: mine.Membership.Fingerprint,
Password: mine.Membership.Password, Signer: mine.Membership.Signer,
Transport: mine.Membership.Transport,
}, link.Report{Node: mine.Node, Rekey: &rekey}, opts.timeout); err != nil {
return fmt.Errorf("the mesh could not be told; nothing was written here: %w", err)
}
fmt.Printf(" told the mesh signed rekey sent; `node show %s` on the controller says whether it took\n", mine.Node)
if err := os.WriteFile(identity.OverlayKeyPath(opts.state),
[]byte(taken.Overlay.Private+"\n"), 0o600); err != nil {
return fmt.Errorf("the mesh was told and this node's overlay key could not be written: %w — run this again", err)
}
if err := identity.Save(identityPath, taken); err != nil {
return fmt.Errorf("the mesh was told and this node's identity could not be saved: %w — run this again", err)
}
fmt.Printf(" written %s and the identity; the mesh's interface reads the key when the next push restarts it\n",
identity.OverlayKeyPath(opts.state))
fmt.Printf(" next on the controller: `overlay place %s --hub --endpoint <host>:%d …`, `plan %s --json`, then push\n",
mine.Node, found.Port, mine.Node)
return nil
}
// rekeyOnto is the identity with the found tunnel's key as its overlay key, and the signed rekey
// that tells the mesh. Pure, so it can be held to: node, sealing and serving keys are the same
// bytes in and out; only the overlay key moves. Run again after a take, the previous key it names
// is the one before the take, so the mesh can tell a repeat from a replay.
func rekeyOnto(mine identity.Identity, found tunnel.Found) (identity.Identity, link.Rekey, error) {
overlay, err := identity.OverlayKeyFrom(found.PrivateKey())
if err != nil {
return identity.Identity{}, link.Rekey{}, err
}
previous := mine.Overlay.Public
if previous == overlay.Public && mine.OverlayBefore != "" {
previous = mine.OverlayBefore
}
taken := mine
taken.Overlay = overlay
if previous != overlay.Public {
taken.OverlayBefore = previous
}
presented := carried(found)
rekey := link.Rekey{Previous: previous, OverlayKey: overlay.Public, Tunnel: presented}
rekey.Proof = mine.Sign(link.RekeyProof(mine.Node, previous, overlay.Public, presented))
return taken, rekey, nil
return errors.New("overlay take is refused: the controller that took a rekey ended (issue 1126) and no seat " +
"takes one now. Nothing was changed")
}
// carried is a found tunnel as it is presented to the mesh: everything but its private key.
@@ -1046,15 +905,7 @@ func runLink(ctx context.Context, opts options) error {
say(err.Error())
}
windows := apply.WindowsIn(filepath.Dir(opts.state))
j.HeldNow = func(now time.Time) map[string]bool {
held := map[string]bool{}
for _, w := range windows.Open(now) {
for _, name := range w.Holds {
held[name] = true
}
}
return held
}
j.HeldNow = heldBy(windows)
judging = j
// And which units its service managers say failed, and whose each is (novox/hq issue 315).
unitJudge = units.New(units.Exec{Run: apply.ExecRunner, System: builtFor})
@@ -1082,6 +933,8 @@ func runLink(ctx context.Context, opts options) error {
Password: mine.Membership.Password,
Transport: mine.Membership.Transport,
Signer: mine.Membership.Signer,
// Which of its two logins it dials as (ADR 0583 §1): the one the vault's node-logins names.
Logins: loadLogins(opts.state, mine.Node, mine.Membership.Password),
}
// What follows an apply of a module (the node agent's, novox/hq ADR 0539): a membership it delivered for another bus
@@ -1113,8 +966,13 @@ func runLink(ctx context.Context, opts options) error {
// the bus, answer what the controller asks of this machine, and say its facts: when a link opens, and when the
// controller asks.
proof := &proving{statePath: opts.state, version: runningVersion(), say: say}
// What this engine trusts a mesh signature from: the vault's key alone, once read from its signers (novox/hq ADR 0550
// §2; issue 1126: the controller's key never, the controller ended by its unassignment).
trust := newMeshTrust()
queue := &link.Session{
Membership: membership,
// And the public halves and certificate its record says (novox/hq ADR 0555 §3), read at each facts report.
Keys: recordKeys(opts.state, trust.Trusts),
// And the first have this build writes is what proves it to the launcher (ADR 0539 §9, ADR 0227 §8).
HaveWritten: proof.haveWritten,
// And the operator's hand-over of a directory used as found, carried by the controller from the
@@ -1135,7 +993,15 @@ func runLink(ctx context.Context, opts options) error {
Say: say,
Timeout: opts.timeout,
}
// And the node seat node-engine this engine holds (novox/hq ADR 0563): its verbs on every link, its own asks to the
// operator under node-engine-<node>, and its acts kept beside its state.
queue.Acts = engineActs(opts, mine.Node, queue, setuidSearch)
go reconcileActs(ctx, queue.Acts)
current := &agentNow{}
queue.Acts.Composition = compositionFrom(current)
queue.Acts.Compose = composeFrom(current, &composerNow)
queue.Acts.Packages = packagesFrom(mine.Node, queue, &composerNow)
queue.Acts.ComposeCandidate = composeCandidateFrom(mine.Node, &shadowNow)
setRunsHere(runsFromAgent(current))
factsOfNow := func(ctx context.Context, ag *agent.Agent) {
sayFacts(ctx, opts, mine.Membership.Signer, queue, say, ag)
@@ -1171,11 +1037,26 @@ func runLink(ctx context.Context, opts options) error {
if ag := current.get(); ag != nil {
ag.Retry(modules...)
}
}, func(modules ...string) {
if ag := current.get(); ag != nil {
ag.Reconcile(modules...)
}
})
}
// The node follows its wants: one module at a time from the composition its want names (novox/hq ADR 0539), and the
// busy containers are asked for again soon (ADR 0360 §11).
go nodeAgent(aside, opts, mine.Membership.Signer, queue, sched, say, afterApply, current, factsOfNow)
// And what it trusts, followed on the bus, said in its record whenever it changes (ADR 0550 §2).
trust.onChange(func() { factsOfNow(aside, current.get()) })
go followTrust(aside, queue, trust, say)
// And the login it dials as, which the vault's node-logins names for this node (novox/hq ADR 0583 §1).
rotator := newRotator(queue, opts.state, opts.timeout, say)
go followLogins(aside, queue, mine.Node, func(nl delivery.NodeLogin) {
if line := membership.Logins.Heard(nl); line != "" {
say(line)
}
rotator.Heard(aside, nl)
}, say)
go nodeAgent(aside, opts, mine.Membership.Signer, trust, queue, sched, say, afterApply, current, factsOfNow)
go busyPasses(aside, func() {
if ag := current.get(); ag != nil {
ag.Retry(busyModules()...)
@@ -1183,12 +1064,9 @@ func runLink(ctx context.Context, opts options) error {
}, BusyAskEvery)
// And the core builds this host placed are judged, whichever host placed them (to-be 45 §8).
go watchWhatThisHostPlaced(aside, mine.Node, queue, say)
// And mesh-cli is answered on this machine: the account from the kernel, the line asked of the controller
// (novox/hq ADR 0272).
go serveMeshCLI(aside, queue, say)
// And a filled file is written on the provisioner client's request, one at a time with the apply (novox/hq ADR
// 0349 §4).
go serveFilling(aside, opts, mine.Membership.Signer, say)
go serveFilling(aside, opts, mine.Membership.Signer, queue, mine.Node, say)
held := link.HoldRoused(aside, membership, queue, say, opts.timeout, rousedBySignal(ctx))
// **Cleanly**, or the launcher counts standing aside as a crash and rolls the new host back
@@ -1206,8 +1084,6 @@ func runLink(ctx context.Context, opts options) error {
// be a control surface on every machine, reachable by anything that can reach the machine, in
// exchange for saving twenty seconds — and the whole security argument rests on there not being
// one. A signal is delivered by the service manager to a process it already supervises.
// (mesh-cli's socket is no such surface: it has no network address and nothing sent to it acts on
// the machine — it is asked of the controller, novox/hq ADR 0272 §3.)
//
// SIGHUP, because that is the signal a long-running program conventionally reads as *look again*,
// and nothing here is being reloaded from a file that a different signal would suit better.
@@ -1334,13 +1210,15 @@ const (
// the bus when a state changed, again every minute while one is not healthy, and every five minutes
// anyway. A statement the bus did not take is said at the next look. It reads; it never acts (ADR 0240
// rule 6) — but for going on with a take act stopped at its bound (ADR 0328 §7), which runs beside it.
func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session, say link.Announce, reapply func(modules ...string)) {
var missingBefore map[string]bool
func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session, say link.Announce, reapply, reconcile func(modules ...string)) {
var missingBefore, inactiveBefore map[string]bool
ticker := time.NewTicker(liveness.LookEvery)
defer ticker.Stop()
var lastSaid time.Time
owed := true
spaced := 1.0
// healthUnwritten is why the health report was last not written, said once each time it changes.
var healthUnwritten string
// A module whose change ended unjudged is looked at and said at once (review of fix 15): a module with nothing
// long-running is then ok within a look of its apply, not at the five-minute statement.
var nudged <-chan struct{}
@@ -1372,6 +1250,7 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
if a := accountJudge; a != nil {
as, accountsChanged := a.Look(ctx)
accountSt = &as
lastAccounts.Store(&as)
owed = owed || accountsChanged
if accountsChanged {
for _, v := range as.Accounts {
@@ -1386,6 +1265,9 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
accessSt, accessesChanged := accesses.look(time.Now())
keeping.poke(ctx)
owed = owed || accessesChanged
// And a library whose root holds data outside its group is asked of the operator by this engine, once (novox/hq
// ADR 0563 §4).
askTakes(queue.Acts, keepWaiting(accessSt), say)
if accessesChanged {
for _, v := range accessSt {
if v.State != link.StateHealthy {
@@ -1420,6 +1302,11 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
healthy := st.Healthy() && (netSt == nil || netSt.State != network.Unhealthy) &&
(unitSt == nil || len(unitSt.Failed) == 0) && (accountSt == nil || accountSt.Healthy()) && len(found) == 0 &&
accessesHealthy(accessSt)
h := *withAccesses(withFoundDirectories(withAccounts(withUnits(healthAsReported(st, netSt),
unitSt), accountSt), found), accessSt)
// The node's own health report in its key on the bus, compared on every look and written only when what it says
// changed (issue 774): what the watchman raises the machine's alerts from, without the controller.
writeHealthReport(ctx, queue, h, netSt, &healthUnwritten, say)
if !owed && !(!healthy && since >= sayUnhealthyAgain) && since < sayAnyway {
continue
}
@@ -1437,14 +1324,49 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
say(fmt.Sprintf("applying %s again: a resource it declares is not on this machine", strings.Join(m, ", ")))
reapply(m...)
}
// A resource its module declares running that is down but still known to the manager is started by the same
// apply, once each time it goes down, never on a clock (ADR 0568 §2): the judge reads, the agent acts.
var inactive []string
inactive, inactiveBefore = newlyInactive(inactiveBefore, st)
if len(inactive) > 0 && reconcile != nil {
say(fmt.Sprintf("applying %s again: a resource it declares running is not active", strings.Join(inactive, ", ")))
reconcile(inactive...)
}
}
if queue.SayHealth(ctx, *withAccesses(withFoundDirectories(withAccounts(withUnits(healthAsReported(st, netSt),
unitSt), accountSt), found), accessSt)) {
if queue.SayHealth(ctx, h) {
lastSaid, owed = time.Now(), false
}
}
}
// writeHealthReport writes this node's health report to its key (link.WriteNodeHealth) when it changed. Why it could
// not is said once per reason, and said again when it is written after all.
func writeHealthReport(ctx context.Context, queue *link.Session, h link.Health, ns *network.Statement, unwritten *string,
say link.Announce) {
if queue == nil {
return
}
var peers []link.PeerSeen
if ns != nil {
for _, p := range ns.Peers {
peers = append(peers, link.PeerSeen{Key: p.Key, Addresses: p.Addresses, Seen: p.Seen})
}
}
var machines link.Machines
if c := machinesNow.Load(); c != nil {
machines.ByAddress, machines.Hub = c.Machines()
}
_, err := queue.WriteNodeHealth(ctx, link.NodeHealthOf(queue.Membership.Node, h, peers, machines))
switch {
case err != nil && err.Error() != *unwritten:
*unwritten = err.Error()
say("this machine's health report is not written to the bus: " + err.Error())
case err == nil && *unwritten != "":
*unwritten = ""
say("this machine's health report is written to the bus again")
}
}
// healthAsReported is a statement as the report and the event carry it.
func healthAsReported(st liveness.Statement, ns *network.Statement) *link.Health {
// LibraryContract: this engine reads a resource's declared `health` and judges it (ADR 0240 Phase B), judges a
@@ -1640,6 +1562,15 @@ func announceOr(say link.Announce) func(string) {
// run by hand beside this service — the lock beside the state does the same (store.Lock).
var applying sync.Mutex
// objectKeep is the node's own keep of the objects its compositions name by digest, beside its state (apply.ObjectKeep,
// issue 924): a composition the node once applied never again needs the controller's bucket to hold its files.
func objectKeep(opts options) *apply.ObjectKeep {
if opts.state == "" {
return nil
}
return apply.KeepObjectsIn(filepath.Join(filepath.Dir(opts.state), "objects"))
}
// appliedModule is how the apply of one module's declaration came out.
type appliedModule struct {
// Refused is why nothing was applied, when nothing was.
@@ -1679,7 +1610,7 @@ func applyModuleHeld(ctx context.Context, opts options, declared *declaration.De
// **A file given by digest is fetched and verified before anything of the composition is applied** (novox/hq ADR
// 0486 §1, issue 1695): one object missing or differing refuses the module whole.
if err := apply.ResolveFileReferences(ctx, declared, link.FilesFrom(ctx)); err != nil {
if err := apply.ResolveFileReferencesKept(ctx, declared, link.FilesFrom(ctx), objectKeep(opts)); err != nil {
return appliedModule{Refused: "a file of this composition could not be had by its digest: " + err.Error()}
}
@@ -1773,9 +1704,8 @@ func factsOf(ctx context.Context, opts options, declared *declaration.Declaratio
if j := judging; j != nil {
// A container that waits for a library whose disk or mount is not there was not created: the library says
// why, and liveness judges it neither alive nor dead (novox/hq ADR 0328 §9).
j.Set(liveness.LongRunning(declared, withWaiting(held, apply.WaitingOnLibrary(declared))))
judgeWhatIsDeclared(declared, held)
st, _ := j.Look(ctx)
netJudge.declare(declaredResolvConf(declared))
var netSt *network.Statement
if n := netJudge.get(); n != nil {
ns := n.Last()
@@ -1783,7 +1713,6 @@ func factsOf(ctx context.Context, opts options, declared *declaration.Declaratio
}
var unitSt *units.Statement
if u := unitJudge; u != nil {
u.Set(units.OwnedBy(declared, held))
us := u.Last()
unitSt = &us
}
@@ -1791,11 +1720,9 @@ func factsOf(ctx context.Context, opts options, declared *declaration.Declaratio
// said with it, relogin needed included, in the report that says the apply.
var accountSt *accounts.Statement
if a := accountJudge; a != nil {
a.Set(accounts.Of(declared, held))
as, _ := a.Look(ctx)
accountSt = &as
}
accesses.set(declared, held)
// A take act stopped at its bound goes on, beside (ADR 0328 §7).
keeping.poke(ctx)
accessSt, _ := accesses.look(time.Now())
@@ -1890,6 +1817,46 @@ func factsOf(ctx context.Context, opts options, declared *declaration.Declaratio
return report
}
// judgeWhatIsDeclared hands every judge of the health loop what is declared here now: the long-running resources, the
// resolver file, the units, the accounts and the libraries. A resource, unit, account or library no longer declared is
// forgotten by its judge, so a module whose composition drops one is no longer watched for it.
func judgeWhatIsDeclared(declared *declaration.Declaration, held map[string]bool) {
if j := judging; j != nil {
// A container that waits for a library whose disk or mount is not there was not created: the library says
// why, and liveness judges it neither alive nor dead (novox/hq ADR 0328 §9).
j.Set(liveness.LongRunning(declared, withWaiting(held, apply.WaitingOnLibrary(declared))))
}
netJudge.declare(declaredResolvConf(declared))
if u := unitJudge; u != nil {
u.Set(units.OwnedBy(declared, held))
}
if a := accountJudge; a != nil {
a.Set(accounts.Of(declared, held))
}
accesses.set(declared, held)
}
// judgeWhatRunsNow hands the judges what runs now, as the node agent holds it once a module settled (applied or
// removed) or was taken as running: the facts said inside an apply read the module's composition from before it, so a
// resource its new composition dropped stayed watched until the next facts (g14 and shanks, 2026-10-30 16:14: restic's
// process restic.measure, removed, then said down every look).
func judgeWhatRunsNow(statePath string, signer ed25519.PublicKey) {
if judging == nil {
return
}
declared, err := declaredNow(statePath, signer)
if err != nil {
return
}
held := map[string]bool{}
if known, err := store.Load(statePath); err == nil {
for _, h := range known.Held {
held[h.ID] = true
}
}
judgeWhatIsDeclared(declared, held)
}
// sayFacts says this node's facts on the link open now (novox/hq ADR 0539 §9): what the node is made of, how what runs
// here is, and its filters and ports. What runs is the compositions the node agent holds running (ag), or, before it
// has run, the declaration the node kept before the switch. It applies nothing; the haves are the agent's.
@@ -2131,3 +2098,57 @@ func newlyMissing(last map[string]bool, st liveness.Statement) ([]string, map[st
}
return missingModules(fresh), now
}
// newlyInactive are the modules of the resources a statement says down (not running after their grace, but still known to
// their manager, and not restarting) that the last statement acted on did not, each once, sorted, and the resources down
// now: a resource still down is not acted on again, and one that came back and went down again is (ADR 0568 §2).
func newlyInactive(last map[string]bool, st liveness.Statement) ([]string, map[string]bool) {
now := map[string]bool{}
var fresh liveness.Statement
for _, r := range st.Resources {
if r.State != liveness.Unhealthy || r.Reason != liveness.ReasonDown || r.Missing {
continue
}
now[r.ID] = true
if !last[r.ID] {
fresh.Resources = append(fresh.Resources, r)
}
}
return missingModules(withMissing(fresh)), now
}
// withMissing is the statement with each of its resources marked missing, so missingModules lists their modules.
func withMissing(st liveness.Statement) liveness.Statement {
for i := range st.Resources {
st.Resources[i].Missing = true
}
return st
}
// recordKeys reads the public halves of the node's keys and the certificate the mesh issued for its internal name from
// its state directory (novox/hq ADR 0555 §3): what the node's record says of them. A key that cannot be read is left
// empty, never invented; the certificate is there only after the first declaration that carried it.
func recordKeys(state string, trusts func() []string) func() link.RecordKeys {
return func() link.RecordKeys {
var k link.RecordKeys
if trusts != nil {
k.Trusts = trusts()
}
if id, err := identity.Load(identity.Path(state)); err == nil {
k.TunnelKey = id.Overlay.Public
}
if sk, err := identity.LoadSealingKey(identity.SealingKeyPath(state)); err == nil {
k.SealingKey = sk.Public
}
if sv, err := identity.LoadServingKey(identity.ServingKeyPath(state)); err == nil {
k.ServingKey = sv.Public
}
if cert, err := os.ReadFile(identity.CertificatePath(state)); err == nil {
k.Certificate = string(cert)
}
if authority, err := os.ReadFile(authorityPath(state)); err == nil {
k.Authority = string(authority)
}
return k
}
}
-25
View File
@@ -1,25 +0,0 @@
package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/cli"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/meshcli"
)
// serveMeshCLI answers mesh-cli on this machine for as long as this engine serves (novox/hq ADR 0272): the account
// from the kernel, the line asked of the controller over the link open at the time. A socket that cannot be served
// is said once, and this engine goes on applying; mesh-cli then says the engine does not answer.
func serveMeshCLI(ctx context.Context, queue *link.Session, say func(string)) {
if err := meshcli.Serve(ctx, cli.Socket, forwardOver(queue), say); err != nil {
say("mesh-cli is not answered on this machine: " + err.Error())
}
}
// forwardOver asks the controller over the queue's link, and follows the line to what came of it.
func forwardOver(queue *link.Session) meshcli.Forward {
return func(ctx context.Context, asked meshcli.Asked) cli.Answer {
return meshcli.AskTheController(ctx, queue.AskController, asked)
}
}
+54 -1
View File
@@ -9,6 +9,7 @@ import (
"errors"
"fmt"
"os"
"sync"
"time"
"golang.org/x/crypto/nacl/box"
@@ -40,6 +41,11 @@ func moveSecretFrom(statePath string, queue *link.Session) link.MoveSecretFunc {
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: a value was not moved into the vault: %v\n", err)
}
if moved.Superseded != "" {
if _, said := supersededSaid.LoadOrStore(moved.Secret, true); !said {
fmt.Fprintf(os.Stderr, "mesh-host: %s\n", moved.Superseded)
}
}
return moved, err
}
}
@@ -104,8 +110,30 @@ func moveIntoTheVault(ask vaultAsk, key identity.SealingKey, m link.MoveSecret)
Generation int `json:"generation"`
Unchanged bool `json:"unchanged"`
ReadBack string `json:"read_back"`
// Refused is the vault's refusal as fields (mesh-vault's MoveRefused, mesh-catalog f3b13de2): nothing was taken.
Refused *struct {
Reason string `json:"reason"`
Held *heldValue `json:"held"`
} `json:"refused"`
Why string `json:"why"`
}
if err := json.Unmarshal(raw, &kept); err != nil || kept.Name == "" || kept.ReadBack == "" {
if err := json.Unmarshal(raw, &kept); err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault's answer for %s does not read: %v", opened.Secret, err)
}
if r := kept.Refused; r != nil {
// The vault holds another value for the name, and the move never replaces one: the vault's value is the live
// one, and this machine's older value is superseded, never asked again (the sweep of 2026-10-30: it failed
// "answered with no read-back" on every ask).
if r.Reason == vaultHoldsAnother && r.Held != nil && r.Held.Name != "" {
return link.MovedSecret{Secret: r.Held.Name, Generation: r.Held.Generation, Superseded: r.Held.words()}, nil
}
why := kept.Why
if why == "" {
why = r.Reason
}
return link.MovedSecret{}, fmt.Errorf("the vault refused %s: %s", opened.Secret, why)
}
if kept.Name == "" || kept.ReadBack == "" {
return link.MovedSecret{}, fmt.Errorf("the vault answered %s with no read-back", opened.Secret)
}
blob, err := base64.StdEncoding.DecodeString(kept.ReadBack)
@@ -124,6 +152,31 @@ func moveIntoTheVault(ask vaultAsk, key identity.SealingKey, m link.MoveSecret)
return link.MovedSecret{Secret: kept.Name, Generation: kept.Generation, Unchanged: kept.Unchanged}, nil
}
// vaultHoldsAnother is the vault's reason when it holds another value for the name (mesh-vault RefusedHeldAnother).
const vaultHoldsAnother = "holds-another-value"
// heldValue is the record of the value the vault holds instead, never the value (mesh-vault HeldAnother).
type heldValue struct {
Name string `json:"name"`
Generation int `json:"generation"`
ChangedBy string `json:"changed_by"`
ChangedAt time.Time `json:"changed_at"`
Origin string `json:"origin"`
}
// words say the held value's record, for the journal and the controller.
func (h heldValue) words() string {
w := fmt.Sprintf("the vault holds another value for %s, generation %d, changed by %s at %s", h.Name, h.Generation,
h.ChangedBy, h.ChangedAt.UTC().Format(time.RFC3339))
if h.Origin != "" {
w += ", origin " + h.Origin
}
return w + ": that value is the live one, and this machine's older value is superseded and not moved"
}
// supersededSaid holds the names a superseded move was said of, so the journal says each once.
var supersededSaid sync.Map
func wipe(b []byte) {
for i := range b {
b[i] = 0
+34 -7
View File
@@ -17,13 +17,15 @@ import (
// standInVault answers move-in's two steps as the vault does: a key made for the call, the value opened with it,
// and the value kept read back sealed to the key the second step sent. lie makes its read-back another value.
type standInVault struct {
t *testing.T
pub *[32]byte
priv *[32]byte
opened map[string]any
kept string
lie string
refuse string
t *testing.T
pub *[32]byte
priv *[32]byte
opened map[string]any
kept string
lie string
refuse string
// answer replaces the second step's answer: the vault's refusal as fields (mesh-catalog f3b13de2).
answer map[string]any
carried []string
}
@@ -51,6 +53,9 @@ func (v *standInVault) ask(verb string, args any) (json.RawMessage, error) {
v.t.Fatal("what the engine sealed does not open with the call's key")
}
v.kept = string(value)
if v.answer != nil {
return json.Marshal(v.answer)
}
back := v.kept
if v.lie != "" {
back = v.lie
@@ -119,6 +124,28 @@ func TestTheEngineSaysAMoveThatDidNotHold(t *testing.T) {
!strings.Contains(err.Error(), "never replaces") {
t.Fatalf("the vault's refusal: %v", err)
}
// The vault's refusal as fields (mesh-catalog f3b13de2, no read-back): for another value held, the vault's is the
// live one and the move is done, superseded, at the held generation; any other reason fails in the vault's words.
held := map[string]any{"name": "own/anchor/postgres/superuser-password", "generation": 3,
"changed_by": "mesh-vault@novox", "changed_at": "2026-10-30T15:50:00Z", "origin": "made"}
for _, row := range []struct {
refused map[string]any
gen int
fails string
}{
{map[string]any{"reason": "holds-another-value", "held": held}, 3, ""},
{map[string]any{"reason": "locked"}, 0, "the vault refused"},
} {
v := &standInVault{t: t, answer: map[string]any{"refused": row.refused, "why": "in words"}}
moved, err := moveIntoTheVault(v.ask, key, aMoveOf(t, key))
switch {
case row.fails != "" && (err == nil || !strings.Contains(err.Error(), row.fails)):
t.Errorf("refused %v: %v", row.refused["reason"], err)
case row.fails == "" && (err != nil || moved.Generation != row.gen || moved.Secret != held["name"] ||
!strings.Contains(moved.Superseded, "changed by mesh-vault@novox")):
t.Errorf("refused %v: moved %+v, %v", row.refused["reason"], moved, err)
}
}
// A pair credential names both its ends.
pair := aMoveOf(t, key)
pair.Module, pair.Name = "", ""
+327 -61
View File
@@ -3,21 +3,26 @@ package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/compose/frombus"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/selfcompose"
"github.com/novox/mesh-host/internal/store"
)
@@ -35,6 +40,20 @@ func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Up
if err != nil {
return nil, err
}
return w.updates(ctx, in), nil
}
// WatchAwait is Watch for a state that its holder makes when it starts (the placement seat's): a bucket not on the bus
// yet is waited for on the bus's word that it was created (link.Session.KVWatchAwait), and waiting is told why.
func (w bucketWatch) WatchAwait(ctx context.Context, pattern string, waiting func(error)) (<-chan agent.Update, error) {
in, err := w.session.KVWatchAwait(ctx, w.bucket, pattern, waiting)
if err != nil {
return nil, err
}
return w.updates(ctx, in), nil
}
func (w bucketWatch) updates(ctx context.Context, in <-chan link.KVUpdate) <-chan agent.Update {
out := make(chan agent.Update, 64)
go func() {
defer close(out)
@@ -46,25 +65,52 @@ func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Up
}
}
}()
return out, nil
return out
}
// sessionHaves writes the haves the agent says, through the session.
type sessionHaves struct{ session *link.Session }
func (h sessionHaves) Read(ctx context.Context, module string) (agent.Have, bool) {
if module == agent.NodeModule {
// The node's own key: the digest of its own composition it runs, and what that carries sealed.
got, ok := h.session.ReadNodeHave(ctx)
if !ok || got.Composed == "" {
return agent.Have{}, false
}
return agent.Have{Module: module, Composed: got.Composed, Sealed: got.SealedAt}, true
}
got, ok := h.session.ReadHave(ctx, module)
if !ok {
return agent.Have{}, false
}
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail}, true
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail,
Sealed: got.SealedAt}, true
}
// ReadErr is Read telling a have absent from one the bus could not read (agent.HavesReadErr).
func (h sessionHaves) ReadErr(ctx context.Context, module string) (agent.Have, bool, error) {
if module == agent.NodeModule {
// The node's own key, its read the bus did not answer told as such (issue 1126, 2026-10-28).
got, ok, err := h.session.ReadNodeHaveErr(ctx)
if !ok || err != nil || got.Composed == "" {
return agent.Have{}, false, err
}
return agent.Have{Module: module, Composed: got.Composed, Sealed: got.SealedAt}, true, nil
}
got, ok, err := h.session.ReadHaveErr(ctx, module)
if !ok || err != nil {
return agent.Have{}, false, err
}
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail,
Sealed: got.SealedAt}, true, nil
}
func (h sessionHaves) Write(ctx context.Context, x agent.Have) {
if x.Module == agent.NodeModule {
// The node's own key holds its facts and the digest of the composition it runs.
if x.Phase == "" || x.Phase == string(delivery.PhaseRunning) {
h.session.NoteNode(ctx, x.Composed)
h.session.NoteNode(ctx, x.Composed, x.Sealed)
}
return
}
@@ -73,7 +119,8 @@ func (h sessionHaves) Write(ctx context.Context, x agent.Have) {
phase = ""
}
h.session.NoteModule(ctx, link.ModuleFact{Module: x.Module, FP: x.FP, Composed: x.Composed, Phase: phase,
Failed: phase == string(delivery.PhaseFailed), Detail: shorten(x.Detail)})
Failed: phase == string(delivery.PhaseFailed), Detail: shorten(x.Detail), Judgement: x.Judgement, Waits: x.Waits,
Sealed: x.Sealed})
}
func (h sessionHaves) Removed(ctx context.Context, module string) {
@@ -117,19 +164,54 @@ func backupNode(session *link.Session) func(ctx context.Context, module string)
defer stop()
// A module this machine backs nothing of up moves without a backup: never held for one that cannot run (issue
// 1945). An answer that cannot be had or read asks for the backup as before.
if said, err := session.AskSeat(ctx, "node-backup", "backed-up", map[string]any{"module": module}, 30*time.Second); err == nil {
ask := func(verb string) func(context.Context) (json.RawMessage, error) {
return func(ctx context.Context) (json.RawMessage, error) {
return session.AskSeat(ctx, "node-backup", verb, map[string]any{"module": module}, 30*time.Second)
}
}
if said, err := askUntilHeard(ctx, ask("backed-up"), backupAskFor, link.Backoff); err == nil {
if nothing, read := declaresNothingToBackUp(said, module); read && nothing {
return agent.Backup{Done: true, Words: module + " declares nothing to back up on this machine"}
}
}
asked := time.Now()
if _, err := session.AskSeat(ctx, "node-backup", "now", map[string]any{"module": module}, 30*time.Second); err != nil {
if _, err := askUntilHeard(ctx, ask("now"), backupAskFor, link.Backoff); err != nil {
return agent.Backup{Words: err.Error()}
}
return awaitBackup(ctx, events, session.Membership.Node, module, asked, backupWithin)
}
}
// backupAskFor is how long the backup's ask is asked again while it is known never to have reached the holder of
// node-backup: nothing on this machine answers it (the tool runner restarting, its subscriptions not made yet) or no
// link is open. The sweep of 2026-10-30 met it on ace's redis, postgres and restic (14:03), step-ca (13:55) and
// mesh-vault on novox (15:22): each was held on one unanswered ask. Past the bound the backup failed, said with how
// often and how long it was asked: the module is held, its have says so, and the assigner asks the operator.
var backupAskFor = 3 * time.Minute
// askUntilHeard asks, and asks again after wait(try) while the answer says the ask was never delivered (link.ErrNotAsked),
// for at most within: an ask that may have reached its holder (a timeout, a refusal in words) is never asked twice, so a
// backup is never started twice.
func askUntilHeard(ctx context.Context, ask func(context.Context) (json.RawMessage, error), within time.Duration,
wait func(int) time.Duration) (json.RawMessage, error) {
began := time.Now()
for try := 1; ; try++ {
said, err := ask(ctx)
if err == nil || !errors.Is(err, link.ErrNotAsked) {
return said, err
}
pause := wait(try)
if time.Since(began)+pause > within {
return nil, fmt.Errorf("asked %d times over %s and never heard: %w", try, time.Since(began).Round(time.Second), err)
}
select {
case <-ctx.Done():
return nil, fmt.Errorf("the node agent was stopped while it asked again: %w", err)
case <-time.After(pause):
}
}
}
// awaitBackup waits for the event that says the backup of a module on this node ended, within the bound: done when it
// says ok, held on its words when it does not, and held when it says nothing within the bound. An event of another node
// or module, or from before the ask, is not the answer.
@@ -139,7 +221,7 @@ func awaitBackup(ctx context.Context, events <-chan []byte, node, module string,
for {
select {
case <-ctx.Done():
return agent.Backup{Words: "the node agent was stopped while the backup ran"}
return agent.Backup{Words: agent.BackupInterrupted, Interrupted: true}
case <-bound.C:
return agent.Backup{Words: fmt.Sprintf("the backup did not say it was done within %s", within)}
case raw := <-events:
@@ -292,48 +374,17 @@ func resourcesOf(cs []delivery.Composed) []declaration.Resource {
return out
}
// operatorKeysFile is the file beside the node's state that holds the operator's public keys, one ed25519 key in base64 a
// line (a line that starts with # is a note). The switch's last declaration writes it as a file resource, beside the
// controller's key the node was enrolled with (novox/hq ADR 0539 §3, §11); a node adopted later has it at its adoption.
const operatorKeysFile = "operator-keys"
// trustedKeys are the keys a composition may be signed by: the controller's, which the node was enrolled with, and the
// operator's, read from operatorKeysFile each time it is asked, so a key that arrives while the agent runs is trusted at
// the next composition. A file that cannot be read trusts the controller's alone, and says so.
func trustedKeys(opts options, controller ed25519.PublicKey) func() []ed25519.PublicKey {
return func() []ed25519.PublicKey {
keys := []ed25519.PublicKey{}
if len(controller) == ed25519.PublicKeySize {
keys = append(keys, controller)
}
raw, err := os.ReadFile(filepath.Join(filepath.Dir(opts.state), operatorKeysFile))
if err != nil {
return keys
}
for _, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
key, err := base64.StdEncoding.DecodeString(line)
if err != nil || len(key) != ed25519.PublicKeySize {
fmt.Fprintf(os.Stderr, "mesh-host: a line of %s is not an ed25519 public key in base64, and is not trusted\n", operatorKeysFile)
continue
}
keys = append(keys, ed25519.PublicKey(key))
}
return keys
}
}
// keptOnce reads the kept declaration once and splits it by module: its resources (the node's own are those that say
// none, ADR 0539 §9) and its filled paths.
// none, ADR 0539 §9) and its filled paths. Once the read is retired (keptRetired) nothing is read.
func keptOnce(opts options, signer ed25519.PublicKey) func() (map[string][]json.RawMessage, map[string][]string) {
var once sync.Once
var by map[string][]json.RawMessage
var filled map[string][]string
return func() (map[string][]json.RawMessage, map[string][]string) {
once.Do(func() {
if keptRetired.Load() {
return
}
raw, err := store.LoadDeclared(store.DeclaredPath(opts.state), signer)
if err != nil {
return
@@ -383,7 +434,7 @@ func splitKept(raw []byte) (map[string][]json.RawMessage, map[string][]string) {
// nodeAgent runs the node agent under the session's links for as long as ctx lasts: it waits for a link, follows the
// node's wants and compositions on it, and begins again with the next link. A watch it cannot make (a bucket this node's
// account may not read) is said, and tried again with a growing wait, as the link is.
func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, session *link.Session, sched *apply.Scheduler,
func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, trust *meshTrust, session *link.Session, sched *apply.Scheduler,
say link.Announce, afterApply func(), current *agentNow, sayFacts func(ctx context.Context, ag *agent.Agent)) {
wait := 2 * time.Second
for ctx.Err() == nil {
@@ -400,10 +451,38 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
return
}
var ag *agent.Agent
// The kept release-5 declaration is no longer read once every module the seat's placement puts here has a have of its
// own (ADR 0550 §2: its signature is the controller's, trusted no more after the lease write).
// Nor before this engine keeps the node's own as it runs (applied/node.json): until then the kept declaration is the one
// record of what the node's own composition carries, its folded modules among them (issue 774, review of the
// re-land, round 1: novox's first start would compose its own with nats running nothing).
if retired, why := placedAllHave(linked, session, node); retired && !keptRetired.Load() {
if !appliedStoreFor(opts.state).holds(agent.NodeModule) {
say("the kept declaration is still read: " + why + ", but this engine keeps no node's own composition yet")
} else if !keptRetired.Swap(true) {
say("the kept declaration is no longer read: " + why)
}
}
kept := keptOnce(opts, signer)
// The bus's user list, written alone from the newest composition of the bus module (ADR 0547 §8): on every
// node, and a node whose compositions carry no list writes nothing.
users := userListFollower(opts, session, composedBucket, trustedKeys(opts, signer), say, func() {
// The mesh's own certificate the declaration kept from before carries, kept for the record before this machine
// composes its own (ADR 0555 §3): without it a module that asks for a certificate would be stood in for.
if byModule, _ := kept(); byModule != nil {
for module, resources := range byModule {
if err := keepAuthority(opts.state, delivery.Composed{Module: module, Resources: resources}); err != nil {
say("the mesh's certificate the kept declaration carries could not be kept for this machine's record: " + err.Error())
}
}
}
// The bus's user list, the grants part's alone (ADR 0547 §8, clause 11; issue 1126), written into the file the
// bus module this node runs declares, as its agent holds what runs (its own composition, or the declaration kept
// before the switch): never read from the node's composition bucket, which the controller wrote. On every node;
// a node that runs no list writes nothing. Looked at again whenever what runs changed (RunsChanged, Settled).
users := userListFollower(opts, session, func() []delivery.Composed {
if now := current.get(); now != nil {
return now.Everything()
}
return nil
}, say, func() {
// This engine wrote the list, so the bus reloads its accounts and answers no JetStream request for a few
// seconds: what failed in them is tried again once that is over, not at its next wait.
time.AfterFunc(reloadSettles, func() {
@@ -412,8 +491,6 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
}
})
})
// A vault key heard (or heard again changed) looks at the list again, so its hashes are handed (ADR 0547 §9).
vaultKey.following(users.Again)
go func() {
if err := users.Run(linked); err != nil && linked.Err() == nil {
say("the bus's user list is not followed on this link: " + err.Error())
@@ -428,6 +505,10 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
// The same kept declaration's filled paths, so a module still running from it keeps its files filled.
KeptFilled: func() map[string][]string { _, filled := kept(); return filled },
Recorded: recordedIn(opts.state, say),
// What this engine applied, kept on the machine and read at the start, so what runs is known before the node
// composes its own (issue 774).
Applied: appliedStoreFor(opts.state),
Inactive: inactiveIn(&liveness.Exec{Run: apply.ExecRunner, System: builtFor}, heldBy(apply.WindowsIn(filepath.Dir(opts.state)))),
Revisions: func(ctx context.Context, key string) ([]agent.Update, error) {
got, err := session.KVHistory(ctx, composedBucket, key)
if err != nil {
@@ -439,18 +520,42 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
}
return out, nil
},
Trusted: trustedKeys(opts, signer),
Backup: backupNode(session),
Backup: backupNode(session),
// What the filling refused before this apply may be named now: the provisioner's client is told to fill
// again (ace, 2026-10-25 21:17).
Settled: func(module string) { knockFilling(module, say) },
Say: say,
Settled: func(module string) {
knockFilling(module+" settled", say)
judgeWhatRunsNow(opts.state, signer)
users.RunsChanged()
},
// And once the haves are read at the start, and whenever a module is taken as running without an apply, so
// a refusal from the engine's start does not wait for some module to settle (issue 924).
RunsChanged: func(why string) {
knockFilling(why, say)
judgeWhatRunsNow(opts.state, signer)
users.RunsChanged()
},
Say: say,
// A failure of the bus not answering for a moment is tried again: 2 s, 5 s, 15 s, 60 s, then every 5 min.
Backoff: link.Backoff,
Apply: func(ctx context.Context, c delivery.Composed, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := applyComposition(actx, opts, c, others, sched, say)
// While the grants part's list is followed, the controller's list in the bus module's composition is not
// written: the apply keeps the list on the machine (ADR 0547 clause 8, never both).
held, herr := users.Hold(c)
var res agent.Result
if herr != nil {
res = failedWith(herr.Error())
} else {
res = applyComposition(actx, opts, held, others, ag.Placed(scopeName(c)), ag.RemovedByWant(), sched, say)
}
done()
// The mesh's own certificate it carries is kept for this machine's record (ADR 0555 §3).
if !res.Failed {
if err := keepAuthority(opts.state, c); err != nil {
say("the mesh's certificate in " + c.Module + "'s composition could not be kept for this machine's record: " + err.Error())
}
}
// An apply of the bus module may write an older list than the newest composition holds: looked at again.
if module, _ := users.State(); module != "" && module == c.Module {
users.Again()
@@ -461,18 +566,82 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
},
Remove: func(ctx context.Context, module string, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := removeComposition(actx, opts, module, others, sched, say)
res := removeComposition(actx, opts, module, others, ag.Placed(module), sched, say)
done()
afterApply()
sayFacts(ctx, ag)
return res
},
}
// The node composes its own modules (novox/hq ADR 0547 §2, ADR 0549, issue 1126).
run, stopRun := context.WithCancel(linked)
// What the node runs and how it opens a sealed value, for the composer to keep the boxes it runs; and the agent
// told when the node cannot compose, so each wanted module's have says it.
// The agent of this run alone: its composer tells it, never the agent of a run begun since (issue 1126: a pass in
// flight at the switch marked the next agent's modules failed).
own := &agentNow{}
hooks := composerHooks{
running: func() []delivery.Composed {
if now := own.get(); now != nil {
return runsAsWritten(now.Everything(), opts.state)
}
return nil
},
unseal: sealOpener(opts.state),
// And what each module's have says it runs sealed: after a restart the agent has applied nothing yet.
// A have the bus did not answer for is said as such, never as one holding no seals (issue 1126, 2026-10-28).
held: func(module string) (delivery.Seals, []string, error) {
if module == agent.NodeModule {
got, _, err := session.ReadNodeHaveErr(linked)
return got.SealedAt, got.Sealed, err
}
got, ok, err := session.ReadHaveErr(linked, module)
if !ok {
return nil, nil, err
}
return got.SealedAt, got.Sealed, nil
},
refused: func(why string, held map[string]string) {
if now := own.get(); now != nil && run.Err() == nil {
now.NotComposedEach(run, why, held) // a module held alone marks that module, never the others (issue 1126)
}
},
// And the vault's authority beside the controller's, and this node's certificate asked of the vault when a
// module here asks for one (ADR 0550 §2).
vaultAuthority: trust.VaultAuthority,
certify: func(ctx context.Context) error {
return certifyThisNode(ctx, opts.state, node, trust, session.AskVault, func() { sayFacts(ctx, ag) })
},
}
// And its answer to every placement candidate, placed or not, applying nothing (novox/hq ADR 0559 §4): written into
// its own key, `<node>.node`, under candidates.
shadowHooks := hooks
shadowHooks.refused, shadowHooks.certify = nil, nil
shadowEnv := composerEnv(session, node, say, shadowHooks)
shadow := selfcompose.NewShadow(shadowEnv,
func(ctx context.Context, answers map[string]delivery.CandidateAnswer) {
session.NoteCandidates(ctx, delivery.WithValuesDiffer(answers))
})
go shadow.Keep(run, link.Backoff) // this run's alone: a switch begins a run, and its shadow, again
shadowNow.Store(shadow) // what the verb compose-candidate composes a check's candidate with
// Its own composer, always (issue 1126, A7): a composition made elsewhere (the controller's, in the node's
// composition bucket) is never applied. While the seat's placement holds no key for this node it composes
// nothing and keeps running what it runs; a placement it cannot watch stops the agent, said, and it begins again.
composer := selfComposer(session, node, hooks, say)
machinesNow.Store(composer) // the machines a failing network part points at, by the placements (issue 774)
composerNow.Store(composer) // what the verb compose asks a pass of, this run's alone
env.Composed, env.SelfComposed, env.Policy = composer, true, composer.Policy
env.HoldsData = composer.HoldsData // what runs from the kept declaration says Data as its manifest does
session.NoteComposes(linked, delivery.ComposesOwn, "it composes its own modules, and applies no composition made elsewhere")
ag = agent.New(env)
own.set(ag)
current.set(ag)
err = ag.Run(linked)
err = ag.Run(run)
stopRun()
cancel()
current.set(nil)
composerNow.Store(nil)
shadowNow.Store(nil)
if ctx.Err() != nil {
return
}
@@ -517,7 +686,13 @@ func (c *agentNow) get() *agent.Agent {
}
// applyComposition applies one composition: a module's, or the node's own.
func applyComposition(ctx context.Context, opts options, c delivery.Composed, others []delivery.Composed,
//
// placed is the modules a want still places here besides this one: none of their resources is removed by this apply,
// which is refused instead (issue 774).
//
// removed is the modules whose want says removed: in the node's own apply, a resource of any other module goes only when
// its module is among them (issue 774).
func applyComposition(ctx context.Context, opts options, c delivery.Composed, others []delivery.Composed, placed, removed map[string]bool,
sched *apply.Scheduler, say link.Announce) agent.Result {
name := scopeName(c)
if c.Module == "" {
@@ -533,7 +708,13 @@ func applyComposition(ctx context.Context, opts options, c delivery.Composed, ot
if err := refuseEmptyWrites(d); err != nil {
return failedWith(err.Error())
}
scope := apply.Scope{Module: name, Others: resourcesOf(others)}
scope := apply.Scope{Module: name, Others: resourcesOf(others), Placed: placed}
if name == agent.NodeModule {
scope.RemovedByWant = removed
if scope.RemovedByWant == nil {
scope.RemovedByWant = map[string]bool{}
}
}
got := applyModuleHeld(ctx, opts, d, &scope, sched, say)
if got.Refused != "" {
return failedWith(got.Refused)
@@ -552,9 +733,9 @@ func applyComposition(ctx context.Context, opts options, c delivery.Composed, ot
}
// removeComposition removes one module's own resources and nothing of another's.
func removeComposition(ctx context.Context, opts options, module string, others []delivery.Composed,
func removeComposition(ctx context.Context, opts options, module string, others []delivery.Composed, placed map[string]bool,
sched *apply.Scheduler, say link.Announce) agent.Result {
scope := apply.Scope{Module: module, Others: resourcesOf(others)}
scope := apply.Scope{Module: module, Others: resourcesOf(others), Placed: placed}
got := applyModuleHeld(ctx, opts, &declaration.Declaration{Version: declaration.Version}, &scope, sched, say)
switch {
case got.Refused != "":
@@ -579,8 +760,31 @@ func busyModules() []string {
return out
}
// keptDeclaration is the declaration the node kept before the switch, verified with the mesh's key when it is given.
// machinesNow is the composer whose placements name the mesh's machines by address, for the health report's targets
// (writeHealthReport); nil on a node that does not compose its own, whose parts then name no target.
var machinesNow atomic.Pointer[selfcompose.Composer]
// composerNow is the composer of the run that composes the node's own now, nil while the node applies the controller's
// compositions or no run is going: what the verb compose asks a pass of (issue 774).
var composerNow atomic.Pointer[selfcompose.Composer]
// shadowNow is the shadow of the run now, which answers placement candidates and the verb compose-candidate; nil
// between runs.
var shadowNow atomic.Pointer[selfcompose.Shadow]
// keptRetired says the kept declaration is read no more: every module the seat's placement puts on this node has a have of
// its own (placedAllHave), so what runs is the agent's alone (ADR 0550 §2, design 47's L3 s9).
var keptRetired atomic.Bool
// errKeptRetired is keptDeclaration's answer once the read is retired.
var errKeptRetired = errors.New("the kept declaration is read no more: every module placed on this node has a have of its own")
// keptDeclaration is the declaration the node kept before the switch, verified with the mesh's key when it is given;
// once the read is retired, nothing is read.
func keptDeclaration(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
if keptRetired.Load() {
return nil, errKeptRetired
}
var raw []byte
if len(signer) > 0 {
var err error
@@ -748,3 +952,65 @@ const applyBound = 30 * time.Minute
func applyContext(ctx context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(context.WithoutCancel(ctx), applyBound)
}
// composerHooks are what the composer reads of the agent and the machine (selfcompose.Env Running, Unseal, Refused).
type composerHooks struct {
running func() []delivery.Composed
unseal func(string) ([]byte, error)
refused func(why string, held map[string]string)
held func(module string) (delivery.Seals, []string, error)
// vaultAuthority and certify are selfcompose.Env's VaultAuthority and Certify.
vaultAuthority func() string
certify func(ctx context.Context) error
}
// selfComposer is the node's own composer: it composes what the seat's placement places on the node, and nothing while
// the placement holds no key for it (issue 1126).
func selfComposer(session *link.Session, node string, hooks composerHooks, say func(string)) *selfcompose.Composer {
env := composerEnv(session, node, say, hooks)
// And it keeps each placed module's data items in its own state data, which the node's disk-load watches (novox/hq
// ADR 0577 clause 1); the shadow composes for candidates and keeps none.
env.Data = link.EngineData{Session: session}
// And the packages its compositions declare in its own key of the state packages, which pacman reads (novox/hq ADR
// 0565 §6).
env.Packages = link.EnginePackages{Session: session}
// And every composition it makes in its composition state composed-<node>, a record the provisioner, the assigner
// and the view read and nothing applies (novox/hq ADR 0593 §6).
env.ComposedState = link.EngineComposed{Session: session, Node: node}
return selfcompose.New(env)
}
// composerEnv is what the node's composers read of the bus, the vault and the agent: its own composer and its answer
// to placement candidates (selfcompose.Shadow) alike.
func composerEnv(session *link.Session, node string, say func(string), hooks composerHooks) selfcompose.Env {
return selfcompose.Env{Node: node,
Watch: func(ctx context.Context, bucket, pattern string) (<-chan agent.Update, error) {
if bucket == delivery.PlacementBucket || bucket == delivery.PlacementCandidatesBucket {
// The seat's states are made when their holder starts: a node-engine that starts first waits for the
// bus's word that the bucket was created, and the shadow and the composer say they wait.
return bucketWatch{session, bucket}.WatchAwait(ctx, pattern, func(err error) {
say("the seat's state " + bucket + " is not on the bus yet (" + err.Error() + "): it is watched when the bus says it was created")
})
}
return bucketWatch{session, bucket}.Watch(ctx, pattern)
},
Get: func(ctx context.Context, bucket, key string) ([]byte, error) {
h, err := session.KVHistory(ctx, bucket, key)
if err != nil || len(h) == 0 || h[len(h)-1].Deleted {
return nil, err
}
return h[len(h)-1].Value, nil
},
Mesh: frombus.Mesh{InternalSuffix: overlay.Suffix()},
Say: say,
// Its modules' own secrets, sealed by the vault to this machine's key (ADR 0550 §5).
Own: func(ctx context.Context, args map[string]any) (json.RawMessage, error) {
return session.AskVault(ctx, "own", args)
},
// And the vault's word that a secret changed, on which it composes again (ADR 0547 §2).
Hear: session.Hear,
// And a moved module's templates archive, in the artifact store, as every archive is fetched (issue 1126).
Fetch: apply.Fetch,
Running: hooks.running, Unseal: hooks.unseal, Refused: hooks.refused, Held: hooks.held,
VaultAuthority: hooks.vaultAuthority, Certify: hooks.certify}
}
+454 -149
View File
@@ -5,7 +5,6 @@ package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
@@ -14,11 +13,14 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/nodegrant"
"github.com/novox/mesh-host/internal/selfcompose"
)
// The node agent whole, against a real bus with JetStream started in the test: the wants bucket, the node's composition
@@ -28,9 +30,14 @@ import (
const itNode = "it1"
type itBed struct {
t *testing.T
conn *nats.Conn
js nats.JetStreamContext
t *testing.T
conn *nats.Conn
js nats.JetStreamContext
// nodeConn and nodeJS are the node's own, under its real grant (the grants part's, nodegrant.For): the agent runs
// on them, so a watch or a read the grant does not allow fails here as it fails on the bus (issue 1126, 2026-10-27
// 09:21: a watch of every have key stopped every node agent). conn is the test's, unrestricted, for setting up.
nodeConn *nats.Conn
nodeJS nats.JetStreamContext
wants nats.KeyValue
composed nats.KeyValue
haves nats.KeyValue
@@ -38,14 +45,24 @@ type itBed struct {
opts options
session *link.Session
controll ed25519.PrivateKey
operator ed25519.PrivateKey
said []string
// refusals are the bus's asynchronous errors on the node's connection: each permissions violation it met.
refusals []string
mu sync.Mutex
}
func aServer(t *testing.T) *nats.Conn {
// aServer is an embedded bus with JetStream and two users: the test's, unrestricted, and the node's, granted exactly what
// the grants part gives a node (nodegrant.For); it answers a connection of each.
func aServer(t *testing.T, perms nodegrant.Permissions, refused func(error)) (admin, node *nats.Conn) {
t.Helper()
srv, err := server.NewServer(&server.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true})
granted := &server.Permissions{Publish: &server.SubjectPermission{Allow: perms.Publish, Deny: perms.PublishDeny},
Subscribe: &server.SubjectPermission{Allow: perms.Subscribe}}
if perms.AllowResponses {
granted.Response = &server.ResponsePermission{MaxMsgs: 1, Expires: buscompose.ResponseTTL}
}
nodeUser := buscompose.Principal{Kind: buscompose.KindNode, Node: itNode}.Username()
srv, err := server.NewServer(&server.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true,
Users: []*server.User{{Username: "test", Password: "test"}, {Username: nodeUser, Password: "node", Permissions: granted}}})
if err != nil {
t.Fatal(err)
}
@@ -54,15 +71,25 @@ func aServer(t *testing.T) *nats.Conn {
t.Fatal("the embedded server did not start")
}
t.Cleanup(srv.Shutdown)
conn, err := nats.Connect(srv.ClientURL())
if err != nil {
if admin, err = nats.Connect(srv.ClientURL(), nats.UserInfo("test", "test")); err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
return conn
t.Cleanup(admin.Close)
if node, err = nats.Connect(srv.ClientURL(), nats.UserInfo(nodeUser, "node"), nats.CustomInboxPrefix("_INBOX."+nodeUser),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refused(err) })); err != nil {
t.Fatal(err)
}
t.Cleanup(node.Close)
return admin, node
}
func newItBed(t *testing.T) *itBed {
t.Helper()
return newItBedAs(t, nodegrant.For(itNode))
}
// newItBedAs is newItBed with the node's connection under the grant perms, a node's.
func newItBedAs(t *testing.T, perms nodegrant.Permissions) *itBed {
t.Helper()
// Not skipped off Arch: a package manager that answers is put on PATH, so the apply of files runs on any build node.
bin := t.TempDir()
@@ -74,11 +101,19 @@ func newItBed(t *testing.T) *itBed {
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
b := &itBed{t: t, conn: aServer(t), dir: t.TempDir()}
b := &itBed{t: t, dir: t.TempDir()}
b.conn, b.nodeConn = aServer(t, perms, func(err error) {
b.mu.Lock()
b.refusals = append(b.refusals, err.Error())
b.mu.Unlock()
})
var err error
if b.js, err = b.conn.JetStream(); err != nil {
t.Fatal(err)
}
if b.nodeJS, err = b.nodeConn.JetStream(); err != nil {
t.Fatal(err)
}
composedBucket, _ := delivery.ComposedBucketFor(itNode)
for _, c := range []struct {
name string
@@ -97,28 +132,21 @@ func newItBed(t *testing.T) *itBed {
t.Fatal(err)
}
_, b.controll, _ = ed25519.GenerateKey(nil)
_, b.operator, _ = ed25519.GenerateKey(nil)
return b
}
// start runs the node agent under a session on the bus, trusting the controller's key, and the operator's when asked.
func (b *itBed) start(trustOperator bool) (stop func()) {
// start runs the node agent under a session on the bus.
func (b *itBed) start() (stop func()) {
b.t.Helper()
if trustOperator {
pub := b.operator.Public().(ed25519.PublicKey)
if err := os.WriteFile(filepath.Join(filepath.Dir(b.opts.state), operatorKeysFile),
[]byte("# the operator\n"+base64.StdEncoding.EncodeToString(pub)+"\n"), 0o600); err != nil {
b.t.Fatal(err)
}
}
b.session = &link.Session{Membership: link.Membership{Node: itNode}, Timeout: 2 * time.Second,
Say: b.say}
b.session.Attach(context.Background(), link.OverNATS{Conn: b.conn, JS: b.js, Node: itNode})
Say: b.say, Keys: func() link.RecordKeys { return link.RecordKeys{} }}
b.session.Attach(context.Background(), link.OverNATS{Conn: b.nodeConn, JS: b.nodeJS, Node: itNode})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
defer close(done)
nodeAgent(ctx, b.opts, b.controll.Public().(ed25519.PublicKey), b.session, nil, b.say, func() {},
controller := b.controll.Public().(ed25519.PublicKey)
nodeAgent(ctx, b.opts, controller, newMeshTrust(), b.session, nil, b.say, func() {},
&agentNow{}, func(context.Context, *agent.Agent) {})
}()
return func() { cancel(); <-done }
@@ -133,27 +161,7 @@ func (b *itBed) say(line string) {
func fpIt(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
// compose writes a module's signed composition of one file into the node's bucket and returns its digest.
func (b *itBed) compose(module, fp, content string, data bool, signer ed25519.PrivateKey) string {
b.t.Helper()
path := filepath.Join(b.dir, module+".conf")
res, _ := json.Marshal(map[string]any{"id": module + ".conf", "type": "file", "path": path, "content": content, "module": module})
c := delivery.Composed{Node: itNode, Module: module, FP: fp, Data: data, Resources: []json.RawMessage{res}}
signed, err := delivery.SignComposed(c, signer)
if err != nil {
b.t.Fatal(err)
}
raw, err := delivery.EncodeComposed(signed)
if err != nil {
b.t.Fatal(err)
}
dig, _ := delivery.ComposedDigestOf(signed)
key, _ := delivery.ComposedKey(itNode, module, fp)
if _, err := b.composed.Put(key, raw); err != nil {
b.t.Fatal(err)
}
return dig
}
func fpDigest(c string) string { return "sha256:" + strings.Repeat(c, 64) }
func (b *itBed) want(module, fp, digest, backup string) {
b.t.Helper()
@@ -193,121 +201,418 @@ func within(t *testing.T, d time.Duration, what string, ok func() bool) {
}
}
// R6-A AC1: given a running agent on a bus, when a want naming a digest and that composition are written, the module is
// applied within one second and its have names the fingerprint and the digest; a composition whose digest no want names
// is not applied.
func TestNatsAWantAndItsCompositionAreAppliedWithinASecond(t *testing.T) {
b := newItBed(t)
stop := b.start(false)
defer stop()
time.Sleep(300 * time.Millisecond) // the watches are up
fp := fpIt("a")
dig := b.compose("web", fp, "one\n", false, b.controll)
time.Sleep(300 * time.Millisecond)
if b.file("web") != "" {
t.Fatal("a composition no want names was applied")
}
began := time.Now()
b.want("web", fp, dig, "")
within(t, time.Second, "the module was not applied within one second of its want", func() bool { return b.file("web") == "one\n" })
t.Logf("applied %s after the want", time.Since(began))
within(t, 2*time.Second, "the have does not name the fingerprint and the digest", func() bool {
h := b.have("web")
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
// Issue 1126 (ADR 0547 §2, the milestone's H8): the node agent started whole against a real bus, once the catalogue
// places its node (the placement written after it started), composes a module from its want, the indexer's manifest, its record and its mesh-config value, and applies it,
// with no key in the controller's compositions; a changed config value is composed again on its event and applied as a
// revision of what runs. And a catalogue pull request's placement candidate equal to its placement is answered "same"
// in the node's own key, and nothing is applied for it (ADR 0559 §4, subtask 10 of the milestone).
//
// And (issue 1126, ADR 0560) it reads, through its own session under the grant the grants part composes for it
// (nodegrant.For with the served key, as the grants part grants it), the build of a placed module that declares one and the one config value
// a provider on another node serves from (a mail domain); under a grant without that served key the bus refuses the
// read and the node composes nothing.
func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
t.Run("without the served key the bus refuses the read", func(t *testing.T) {
b := newItBed(t)
stop, _ := h8(t, b, false)
defer stop()
// 15 s, above link.historyWithin (10 s): the bus's refusal of a publish reaches only the connection's error
// handler (nats.go fails no request on it), so the refused read ends at its own bound, then the node says it.
within(t, 15*time.Second, "the bus did not refuse the served key's read, or the node did not say it", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return strings.Contains(strings.Join(b.refusals, "\n"), "Permissions Violation for Publish to \"$JS.API.CONSUMER.CREATE.KV_"+
selfcompose.ResolvedBucket) && strings.Contains(strings.Join(b.refusals, "\n"), ".it2.mailer.domain") &&
strings.Contains(strings.Join(b.said, "\n"), "what mailer serves on it2 (domain)")
})
if b.file("web") != "" {
t.Errorf("composed and applied under a grant without the served key: %q", b.file("web"))
}
})
// A want naming a digest the composition at its fingerprint does not have applies nothing.
fp2 := fpIt("b")
b.compose("web", fp2, "two\n", false, b.controll)
b.want("web", fp2, "sha256:"+strings.Repeat("0", 64), "")
time.Sleep(300 * time.Millisecond)
if b.file("web") != "one\n" {
t.Fatalf("a want naming another digest changed the module: %q", b.file("web"))
b := newItBedAs(t, nodegrant.For(itNode, "it2.mailer.domain"))
stop, st := h8(t, b, true)
defer stop()
placement, resolved, candidates := st.placement, st.resolved, st.candidates
hkey, place, fp := st.hkey, st.place, st.fp
within(t, 5*time.Second, "the node did not compose and apply its own module once placed", func() bool { return b.file("web") == "greeting=bonjour\n" })
// ADR 0565 §6 (1953): the node's engine writes its own key of the state packages, under the grant the grants part
// composes for it, with each package its compositions declare and the module declaring it; once, while nothing changes.
wantPkgs := `{"node":"` + itNode + `","packages":[{"package":"web-tools","modules":["web"],"absent":false}]}`
within(t, 5*time.Second, "the node did not write its key of the state packages", func() bool {
e, err := st.packages.Get(itNode)
return err == nil && string(e.Value()) == wantPkgs
})
written, _ := st.packages.Get(itNode)
h8rest(t, b, resolved, candidates, hkey, place, fp)
// web composed again for its changed config value declares the same packages: the key is not written again.
if e, _ := st.packages.Get(itNode); e == nil || written == nil || e.Revision() != written.Revision() {
t.Errorf("the node's packages key was written again with no package changed: %v, was %v", e, written)
}
t.Run("one answer per candidate revision", func(t *testing.T) { h8once(t, b, resolved, candidates, place) })
h8back(t, b, placement, resolved, candidates, hkey)
}
// R6-A AC5: a data module that ran before asks its backup before any resource changes; a backup that fails holds the
// module (nothing applied, the have says held with the backup's words); when its want then says skip, it is applied.
func TestNatsADataModuleAsksItsBackupFirstAndIsHeldWhenItFails(t *testing.T) {
b := newItBed(t)
var mu sync.Mutex
var asked []string
var onDiskWhenAsked string
if _, err := b.conn.Subscribe("mesh.seat.node-backup.tool.now."+itNode, func(m *nats.Msg) {
mu.Lock()
asked = append(asked, "now")
onDiskWhenAsked = b.file("db")
mu.Unlock()
_ = m.Respond([]byte(`{"result":{"started":"db"}}`))
// The holder says the backup ended on its event.
ev, _ := json.Marshal(map[string]any{"node": itNode, "module": "db", "ok": false, "at": time.Now().UTC(),
"error": "restic: the repository is locked"})
_ = b.conn.Publish(backedUpEvent, ev)
}); err != nil {
// h8once is issue 1126, 2026-10-28 (shanks answered probe/1126-seed-7 at revision 84 every 13-17 s, its bus flooded):
// a candidate revision is answered once. The node's own answer written, a module's have whose health words alone move
// (each look rewrites them) and a config value written again with the value it held answer nothing again; the
// candidate's next revision is answered once.
func h8once(t *testing.T, b *itBed, resolved, candidates nats.KeyValue, place []byte) {
answering := func() int {
b.mu.Lock()
defer b.mu.Unlock()
n := 0
for _, l := range b.said {
if strings.HasPrefix(l, "answering placement candidate feat-x ") {
n++
}
}
return n
}
time.Sleep(time.Second)
settled := answering()
if settled == 0 {
t.Fatal("feat-x was never answered")
}
hkey, _ := delivery.HaveKey(itNode, "web")
for _, detail := range []string{"its tool web_health: no link to the bus is open", "its tool web_health: no answer within 10s"} {
h := b.have("web")
h.Health, h.Detail = delivery.HealthFailed, detail
raw, err := delivery.EncodeHave(h)
if err != nil {
t.Fatal(err)
}
if _, err := b.haves.Put(hkey, raw); err != nil {
t.Fatal(err)
}
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err)
}
}
time.Sleep(1500 * time.Millisecond)
if got := answering(); got != settled {
t.Errorf("feat-x answered %d more time(s) at the same revision, with nothing it is composed from changed", got-settled)
}
same, _ := delivery.DecodePlacement(place)
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "feat-x", Placement: &same})
if err != nil {
t.Fatal(err)
}
stop := b.start(false)
defer stop()
time.Sleep(300 * time.Millisecond)
fp1, fp2 := fpIt("a"), fpIt("b")
b.want("db", fp1, b.compose("db", fp1, "v1\n", true, b.controll), "")
within(t, 3*time.Second, "the first composition was not applied", func() bool { return b.file("db") == "v1\n" })
mu.Lock()
if len(asked) != 0 {
t.Fatalf("a module that never ran was backed up: %v", asked)
ckey, _ := delivery.PlacementCandidateKey("feat-x", itNode)
if _, err := candidates.Put(ckey, craw); err != nil {
t.Fatal(err)
}
mu.Unlock()
dig2 := b.compose("db", fp2, "v2\n", true, b.controll)
b.want("db", fp2, dig2, "")
within(t, 3*time.Second, "the module was not held behind the failed backup", func() bool {
h := b.have("db")
return h.Phase == delivery.PhaseHeld && strings.Contains(h.Detail, "repository is locked")
})
mu.Lock()
if len(asked) != 1 || onDiskWhenAsked != "v1\n" {
t.Fatalf("the backup was asked %v, with %q on disk: it is asked before any resource changes", asked, onDiskWhenAsked)
within(t, 10*time.Second, "the candidate's next revision was not answered", func() bool { return answering() > settled })
time.Sleep(1500 * time.Millisecond)
if got := answering(); got != settled+1 {
t.Errorf("the candidate's next revision was answered %d time(s), want 1", got-settled)
}
mu.Unlock()
if b.file("db") != "v1\n" {
t.Fatalf("something was applied behind a failed backup: %q", b.file("db"))
}
if h := b.have("db"); h.FP != fp1 {
t.Fatalf("a held module should still name what it runs: %+v", h)
}
b.want("db", fp2, dig2, delivery.BackupSkip)
within(t, 3*time.Second, "the module was not applied once its want said skip", func() bool { return b.file("db") == "v2\n" })
within(t, 2*time.Second, "the have does not say running at the new fingerprint", func() bool {
h := b.have("db")
return h.Phase == delivery.PhaseRunning && h.FP == fp2 && h.Composed == dig2
})
}
// R6-A AC9, the hand path end to end: with no controller running, a composition signed by the operator's key and a want
// naming it are applied and the have written; the same signed by a key nobody trusts is not.
func TestNatsTheHandPathAppliesACompositionTheOperatorSigned(t *testing.T) {
b := newItBed(t)
stop := b.start(true)
defer stop()
time.Sleep(300 * time.Millisecond)
// h8back is A7 of issue 1126: no composition made elsewhere is applied. A candidate deleting the node's placement is
// refused, never answered as a return to the controller's compositions; the placement key deleted leaves the agent on
// its own composer, composing nothing and keeping what runs, said in the journal, the node's have still saying it
// composes its own.
func h8back(t *testing.T, b *itBed, placement, resolved, candidates nats.KeyValue, hkey string) {
t.Helper()
node := func() delivery.HaveNode {
e, err := b.haves.Get(hkey)
if err != nil {
return delivery.HaveNode{}
}
n, _ := delivery.DecodeHaveNode(e.Value())
return n
}
if n := node(); n.Composes != delivery.ComposesOwn {
t.Errorf("a node composing its own says it composes %q", n.Composes)
}
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "revert-x"})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.PlacementCandidateKey("revert-x", itNode)
if _, err := candidates.Put(ckey, craw); err != nil {
t.Fatal(err)
}
within(t, 10*time.Second, "the node did not refuse a candidate taking its placement away", func() bool {
a := node().Candidates["revert-x"]
return a.Proposal == "revert-x" && a.Returns == "" && a.Verdict == delivery.CandidateRefused &&
strings.Contains(a.Refused, "composes nothing")
})
ran := b.have("web").Composed
pkey, _ := delivery.PlacementNodeKey(itNode)
// A pass in flight when the key goes (issue 1126, 2026-10-28): a config value changed starts one, and the key goes at once.
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err)
}
if err := placement.Delete(pkey); err != nil {
t.Fatal(err)
}
within(t, 15*time.Second, "the node did not say it composes nothing once its placement went", func() bool {
b.mu.Lock()
said := strings.Join(b.said, "\n")
b.mu.Unlock()
return strings.Contains(said, "key in the seat's placement state is gone: it composes nothing of its own")
})
time.Sleep(3 * time.Second) // what a pass in flight would mark lands within it
b.mu.Lock()
said := strings.Join(b.said, "\n")
b.mu.Unlock()
if n := node(); n.Composes != delivery.ComposesOwn || strings.Contains(said, "applying the controller's compositions") {
t.Errorf("without its placement the node turned to the controller's compositions: %q", n.Composes)
}
if h := b.have("web"); h.Composed != ran || h.Phase == delivery.PhaseFailed || b.file("web") != "greeting=salut\n" {
t.Errorf("without its placement what runs moved: %+v, %q", h, b.file("web"))
}
}
_, stranger, _ := ed25519.GenerateKey(nil)
fpS := fpIt("c")
b.want("evil", fpS, b.compose("evil", fpS, "no\n", false, stranger), "")
time.Sleep(300 * time.Millisecond)
if b.file("evil") != "" {
t.Fatal("a composition signed by a key nobody trusts was applied")
// h8state is what h8 set up that the rest of H8 writes to.
type h8state struct {
placement, resolved, candidates nats.KeyValue
// packages is the engine's state packages (ADR 0565 §6), as the grants part creates it from the engine's manifest.
packages nats.KeyValue
hkey string
place []byte
fp string
}
// h8 sets H8's bus up and starts the agent, then places the node. With checkBuild, the placed module's build key is
// first one that does not decode, which the node must read through its session (and say), then a done one.
func h8(t *testing.T, b *itBed, checkBuild bool) (func(), h8state) {
t.Helper()
kvOf := func(bucket string) nats.KeyValue {
kv, err := b.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 5})
if err != nil {
t.Fatal(err)
}
return kv
}
placement, modules, manifests := kvOf(delivery.PlacementBucket), kvOf(delivery.ModulesBucket), kvOf(delivery.ManifestsBucket)
resolved := kvOf(selfcompose.ResolvedBucket)
builds := kvOf(delivery.BuilderBuildsBucket)
candidates := kvOf(delivery.PlacementCandidatesBucket)
packages := kvOf(link.PackagesBucket)
path := filepath.Join(b.dir, "web.conf")
man := `{"module":"web","version":"1","config":{"greeting":{"type":"string","description":"what the file says","default":"hello"}},` +
`"resources":[{"id":"conf","type":"file","path":"` + path + `","mode":"0644","content":"greeting=${config:greeting}\n"},` +
`{"id":"tools","type":"package","package":"web-tools"}],` +
`"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]}}`
digest := delivery.ManifestDigest([]byte(man))
mraw, _ := delivery.EncodeManifest(delivery.Manifest{Module: "web", JSON: man})
mkey, _ := delivery.ManifestsKey(digest)
fp := fpIt("a")
modv, _ := delivery.EncodeModule(delivery.Module{FP2: fp, Manifest: digest, Repository: "novox/web", Commit: strings.Repeat("1", 40)})
place, _ := delivery.EncodePlacement(delivery.Placement{Node: itNode, Modules: []delivery.PlacedModule{{Module: "web"}}})
rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: itNode, Capabilities: []string{"service-manager", "package-manager"}}})
// it1 in one key per module and node (research 072 row 37), it2 in the form before: a node reads both.
pkey, _ := delivery.PlacementNodeKey(itNode)
places, _ := delivery.PlacementValues(delivery.Placement{Node: itNode, Modules: []delivery.PlacedModule{{Module: "web"}}})
place = places[pkey]
hkey, _ := delivery.HaveNodeKey(itNode)
// A provider on it2 serving from its own config value: a mail domain, which it1's composition settles.
mailer := `{"module":"mailer","version":"1","config":{"domain":{"type":"string","description":"the mail domain","required":true}},` +
`"provides":[{"name":"smtp","scope":"mesh"}],"serves":{"smtp":{"port":587,"domain":"${config:domain}"}}}`
mdigest := delivery.ManifestDigest([]byte(mailer))
mlraw, _ := delivery.EncodeManifest(delivery.Manifest{Module: "mailer", JSON: mailer})
mlkey, _ := delivery.ManifestsKey(mdigest)
mlmod, _ := delivery.EncodeModule(delivery.Module{FP2: fpIt("b"), Manifest: mdigest, Repository: "novox/mailer", Commit: strings.Repeat("2", 40)})
place2, _ := delivery.EncodePlacement(delivery.Placement{Node: "it2", Modules: []delivery.PlacedModule{{Module: "mailer"}}})
rec2, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: "it2", Capabilities: []string{"service-manager"}}})
pkey2, _ := delivery.LegacyPlacementKey("it2")
hkey2, _ := delivery.HaveNodeKey("it2")
buildKey, _ := delivery.BuilderBuildKey("web", fp)
done, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: digest,
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Reference: "artifact-store://web/files/blobs/" + fpDigest("f"), Digest: fpDigest("f")}}})
firstBuild := done
if checkBuild {
firstBuild = []byte(`{"v":1,"state":"no such state"}`)
}
for _, w := range []struct {
kv nats.KeyValue
key string
value []byte
}{{manifests, mkey, mraw}, {modules, "web", modv}, {b.haves, hkey, rec},
{resolved, itNode + ".web.greeting", []byte(`{"value":"bonjour","source":"module"}`)},
{manifests, mlkey, mlraw}, {modules, "mailer", mlmod}, {b.haves, hkey2, rec2}, {placement, pkey2, place2},
{resolved, "it2.mailer.domain", []byte(`{"value":"example.org","source":"module"}`)}, {builds, buildKey, firstBuild}} {
if _, err := w.kv.Put(w.key, w.value); err != nil {
t.Fatal(err)
}
}
want, _ := delivery.EncodeWant(delivery.Want{FP: fp, Self: true})
wkey, _ := delivery.WantKey(itNode, "web")
if _, err := b.wants.Put(wkey, want); err != nil {
t.Fatal(err)
}
fp := fpIt("d")
dig := b.compose("rescue", fp, "back\n", false, b.operator)
b.want("rescue", fp, dig, "")
within(t, 2*time.Second, "the operator's composition was not applied", func() bool { return b.file("rescue") == "back\n" })
within(t, 2*time.Second, "the have was not written", func() bool {
h := b.have("rescue")
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
stop := b.start()
// Not placed yet: the agent applies the controller's compositions, of which there are none. The catalogue's seed then
// places the node, and the agent begins again composing its own (ADR 0549 §4), with no restart of the engine.
time.Sleep(300 * time.Millisecond)
if b.file("web") != "" {
t.Fatal("a node the catalogue does not place composed its own")
}
if _, err := placement.Put(itNode+".web", places[itNode+".web"]); err != nil {
t.Fatal(err)
}
if _, err := placement.Put(pkey, place); err != nil {
t.Fatal(err)
}
if checkBuild {
// The build of the placed module is read through the node's session: one that does not decode is said, and nothing
// is applied; a done one, then a config event, and the node composes.
within(t, 5*time.Second, "the node did not read web's build key through its session", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return strings.Contains(strings.Join(b.said, "\n"), "the builder's "+buildKey)
})
if b.file("web") != "" {
t.Fatal("composed with a build key that does not decode")
}
if _, err := builds.Put(buildKey, done); err != nil {
t.Fatal(err)
}
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"bonjour","source":"module"}`)); err != nil {
t.Fatal(err)
}
}
return stop, h8state{placement: placement, resolved: resolved, candidates: candidates, packages: packages, hkey: hkey, place: place, fp: fp}
}
// h8rest is H8 once the node composes its own: the have, a config revision, and the candidate answered "same".
func h8rest(t *testing.T, b *itBed, resolved, candidates nats.KeyValue, hkey string, place []byte, fp string) {
t.Helper()
within(t, 2*time.Second, "the have does not name the build", func() bool {
h := b.have("web")
return h.FP == fp && h.Composed != "" && h.Phase == delivery.PhaseRunning
})
// ADR 0593 §6: the node publishes what it composed in its composition state, under its own grant: web at the build
// its want names, with the digest its have names (the plan left out of it), and the node's own.
first := b.have("web").Composed
webKey, _ := delivery.ComposedKey(itNode, "web", fp)
within(t, 5*time.Second, "the node did not publish web's composition in its composition state", func() bool {
e, err := b.composed.Get(webKey)
if err != nil {
return false
}
c, err := delivery.DecodeComposed(e.Value())
if err != nil {
return false
}
d, err := delivery.ComposedDigestOf(c)
return err == nil && d == first
})
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err)
}
within(t, 3*time.Second, "a changed config value was not composed and applied again", func() bool { return b.file("web") == "greeting=salut\n" })
within(t, 2*time.Second, "the have does not name the revision", func() bool {
h := b.have("web")
return h.FP == fp && h.Composed != first && h.Phase == delivery.PhaseRunning
})
// The node says its facts (its record, as the test wrote it), then a candidate equal to its placement is written: the
// node answers "same" at the candidate's head and revision in its own key, and applies nothing.
b.session.SayFacts(context.Background(), link.Report{Profile: map[string]any{"capabilities": []any{
map[string]any{"name": "service-manager", "present": true}}}})
second := b.have("web").Composed
same, _ := delivery.DecodePlacement(place)
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "feat-x", Placement: &same})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.PlacementCandidateKey("feat-x", itNode)
crev, err := candidates.Put(ckey, craw)
if err != nil {
t.Fatal(err)
}
var answer delivery.CandidateAnswer
within(t, 10*time.Second, "the node did not answer its candidate", func() bool {
e, err := b.haves.Get(hkey)
if err != nil {
return false
}
n, err := delivery.DecodeHaveNode(e.Value())
answer = n.Candidates["feat-x"]
return err == nil && answer.Proposal == "feat-x" && n.Record != nil
})
if answer.Verdict != delivery.CandidateSame || answer.Revision != crev || len(answer.Same) != 1 || answer.Same[0] != "web" {
t.Errorf("the answer to a candidate equal to the placement: %+v", answer)
}
if h := b.have("web"); h.Composed != second || b.file("web") != "greeting=salut\n" {
t.Errorf("a candidate moved what runs: %+v, %q", h, b.file("web"))
}
h8candidate(t, b, fp, second)
}
// h8candidate is ADR 0593 §7: a check's candidate is composed on the node through its seat's verb compose-candidate,
// under its own grant: the indexer's candidate changing web's manifest is answered composed, web changed at the
// candidate's build, within the answer bound; nothing is applied and the composition state does not move.
func h8candidate(t *testing.T, b *itBed, fp, running string) {
t.Helper()
kvOf := func(bucket string) nats.KeyValue {
kv, err := b.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 5})
if err != nil {
t.Fatal(err)
}
return kv
}
cands, _ := kvOf(delivery.CandidatesBucket), kvOf(delivery.VerdictsBucket)
manifests, err := b.js.KeyValue(delivery.ManifestsBucket)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(b.dir, "web.conf")
man := `{"module":"web","version":"2","config":{"greeting":{"type":"string","description":"what the file says","default":"hello"}},` +
`"resources":[{"id":"conf","type":"file","path":"` + path + `","mode":"0640","content":"greeting=${config:greeting}\n"},` +
`{"id":"tools","type":"package","package":"web-tools"}]}`
digest := delivery.ManifestDigest([]byte(man))
mraw, _ := delivery.EncodeManifest(delivery.Manifest{Module: "web", JSON: man})
mkey, _ := delivery.ManifestsKey(digest)
if _, err := manifests.Put(mkey, mraw); err != nil {
t.Fatal(err)
}
head := delivery.Head{Repository: "novox/web", Number: 3, Head: strings.Repeat("3", 40)}
heads, _ := delivery.HeadsHash([]delivery.Head{head})
pulls := []delivery.CandidatePull{{Head: head, Modules: []string{"web"},
Fps: map[string]delivery.CandidateModule{"web": {FP2: fpIt("c"), Manifest: digest}}}}
craw, err := delivery.EncodeCandidate(delivery.Candidate{Group: "novox/web#3", HeadsHash: heads, Pulls: pulls,
FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
key, _ := delivery.CompositionKey("novox/web#3", heads)
if _, err := cands.Put(key, craw); err != nil {
t.Fatal(err)
}
before, _ := b.composed.Status()
got, err := composeCandidateFrom(itNode, &shadowNow)(key, false)
if err != nil {
t.Fatal(err)
}
raw, _ := got.(json.RawMessage)
a, err := delivery.DecodeComposeCandidateAnswer(raw)
if err != nil {
t.Fatalf("the answer %s: %v", raw, err)
}
if a.Verdict != delivery.ComposeCandidateComposed || len(a.Modules) != 1 || a.Modules[0].Module != "web" ||
a.Modules[0].FP != fpIt("c") || !a.Modules[0].Changed || a.Took > delivery.ComposeAnswerBound.Milliseconds() {
t.Errorf("the node's answer to a candidate changing web: %+v", a)
}
after, _ := b.composed.Status()
if h := b.have("web"); h.Composed != running || b.file("web") != "greeting=salut\n" || before == nil || after == nil ||
after.Values() != before.Values() {
t.Errorf("composing a candidate moved what runs or the composition state: %+v, %q", h, b.file("web"))
}
if _, err := b.composed.Get(mustComposedKey(t, "web", fpIt("c"))); err == nil {
t.Error("the candidate's composition was written into the composition state")
}
}
func mustComposedKey(t *testing.T, module, fp string) string {
t.Helper()
key, err := delivery.ComposedKey(itNode, module, fp)
if err != nil {
t.Fatal(err)
}
return key
}
+5
View File
@@ -22,6 +22,11 @@ func TestNoPackageHearsOrOrdersAWholeDeclaration(t *testing.T) {
if err != nil {
return err
}
// The composition moved in from mesh-sdk (novox/hq ADR 0593 §7) has a node's input field of the same name,
// ReadsModules, which says the node reads a module per resource; it is no reconciling of a declaration.
if d.IsDir() && path == filepath.Join(root, "internal", "compose") {
return filepath.SkipDir
}
if d.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "nodeclare_test.go") {
return nil
}
+4 -23
View File
@@ -26,12 +26,10 @@ const (
)
// heardKey is the vault's own public key as its answer to node-key gave it (ADR 0547 clause 9, issue 1945): the key the
// bus node hands the loaded list's hashes sealed to. One for the process, as the key stated is; kick is the follower of
// the user list on the link open now, told when the key is first heard or changes.
// bus node hands the loaded list's hashes sealed to. One for the process, as the key stated is.
type heardKey struct {
mu sync.Mutex
pem string
kick func()
mu sync.Mutex
pem string
}
// vaultKey is the vault's key this engine heard.
@@ -43,8 +41,7 @@ func (h *heardKey) get() string {
return h.pem
}
// heard keeps the key a node-key answer gave, and tells the follower when it is a first or changed one. An answer
// with no key (a vault before this one) changes nothing.
// heard keeps the key a node-key answer gave. An answer with no key (a vault before this one) changes nothing.
func (h *heardKey) heard(answer json.RawMessage) {
var a struct {
VaultKey string `json:"vault_key"`
@@ -53,24 +50,8 @@ func (h *heardKey) heard(answer json.RawMessage) {
return
}
h.mu.Lock()
changed := a.VaultKey != h.pem
h.pem = a.VaultKey
kick := h.kick
h.mu.Unlock()
if changed && kick != nil {
kick()
}
}
// following names the follower told when the key changes, and tells it at once when a key is known.
func (h *heardKey) following(kick func()) {
h.mu.Lock()
h.kick = kick
known := h.pem != ""
h.mu.Unlock()
if known && kick != nil {
kick()
}
}
// nodeKeyReask is how often the key is stated again though it did not change, so a changed vault key is heard: the
+6 -13
View File
@@ -162,29 +162,22 @@ func asksAgainUntilItHearsTheVaultsKey(t *testing.T) {
}
// The statement a host makes names this contract (issue 786), one above the vault's box. And the vault's key its
// answer gives is kept, told to the follower once when first heard and again only when changed; an answer without one
// (an older vault) or with no public key changes nothing (ADR 0547 clause 9).
// answer gives is kept; an answer without one (an older vault) or with no public key changes nothing (ADR 0547 clause
// 9).
func TestTheNodeKeyContractIsThirteen(t *testing.T) {
if link.NodeKeyContract != 13 {
t.Errorf("NodeKeyContract %d", link.NodeKeyContract)
}
h := &heardKey{}
kicks := 0
h.following(func() { kicks++ })
h.heard(json.RawMessage(`{"recorded":true}`))
h.heard(json.RawMessage(`{"recorded":true,"vault_key":"not a key"}`))
if h.get() != "" || kicks != 0 {
t.Fatalf("an answer with no key was kept: %q, %d kicks", h.get(), kicks)
if h.get() != "" {
t.Fatalf("an answer with no key was kept: %q", h.get())
}
pem := "-----BEGIN PUBLIC KEY-----\nAAAA\n-----END PUBLIC KEY-----\n"
answer, _ := json.Marshal(map[string]any{"recorded": true, "vault_key": pem})
h.heard(answer)
h.heard(answer)
if h.get() != pem || kicks != 1 {
t.Fatalf("key %q, %d kicks, want the key and one", h.get(), kicks)
}
h.following(func() { kicks += 10 })
if kicks != 11 {
t.Fatalf("a follower named after the key was heard was not told: %d", kicks)
if h.get() != pem {
t.Fatalf("key %q, want the key", h.get())
}
}
+168
View File
@@ -0,0 +1,168 @@
package main
import (
"context"
"encoding/json"
"fmt"
"regexp"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
// provisionSecrets fills a provision's value into a file the provisioner's client hands the node-engine to write
// (novox/hq ADR 0570 clause 6): the client fills the file but for each `${secret:<provision>}`, and the node-engine
// fills those at the write, so the client never holds the value. A whole `secrets` file of a handed consumer is the
// placeholder alone.
//
// A placeholder names the credential as the module calls it, `${secret:<name>}` (catalogue.SecretLocal, ADR 0094): the
// local name where the module keeps several values of one provision, else the provision. For each, it finds the one
// provision key `<node>.<consumer>.<provision>[.<local>]` of the seat mesh-provisioner's state `provisions` (clause 1)
// going by that name, which must be `provided` and carry the vault's reference; asks the vault for that pair
// credential sealed to this machine's key (the verb `own`, ADR 0349 §11 with the node-engine standing for the client,
// ADR 0550 §5); and opens it with this machine's sealing key. Anything short of that refuses the write, naming the
// provision and why: a file written with a placeholder left in it, or with no value, is a module that starts and cannot
// connect.
type provisionSecrets struct {
node string
// list is every key of mesh-provisioner's `provisions` under prefix (`<node>.<consumer>.`), by key: what this
// engine may read, its own node's (`<node>.>`).
list func(ctx context.Context, prefix string) (map[string][]byte, error)
// own asks the vault's verb `own` in this engine's name.
own func(ctx context.Context, args map[string]any) (json.RawMessage, error)
// open opens a value sealed to this machine.
open func(sealed string) ([]byte, error)
}
// secretPlaceholder is `${secret:<name>}`.
var secretPlaceholder = regexp.MustCompile(`\$\{secret:([A-Za-z0-9_-]+)\}`)
// fill is content with each provision placeholder of module's filled, and whether any was.
func (p provisionSecrets) fill(ctx context.Context, module string, content []byte) ([]byte, bool, error) {
found := secretPlaceholder.FindAllSubmatch(content, -1)
if len(found) == 0 {
return content, false, nil
}
if module == "" {
return nil, false, fmt.Errorf("the file names %d provision secret(s) and belongs to no module, so whose they are is not known", len(found))
}
if p.list == nil || p.own == nil || p.open == nil {
return nil, false, fmt.Errorf("%s's provisions cannot be read on this machine: the node-engine is not linked to the mesh", module)
}
prefix := p.node + "." + module + "."
keys, err := p.list(ctx, prefix)
if err != nil {
return nil, false, fmt.Errorf("%s's provisions cannot be read: %w", module, err)
}
out := string(content)
done := map[string]bool{}
for _, f := range found {
whole, name := string(f[0]), string(f[1])
if done[whole] {
continue
}
done[whole] = true
pr, key, err := byName(keys, name)
if err != nil {
return nil, false, fmt.Errorf("%s's secret %q: %w", module, name, err)
}
value, err := p.value(ctx, key, pr)
if err != nil {
return nil, false, err
}
out = strings.ReplaceAll(out, whole, string(value))
}
return []byte(out), true, nil
}
// byName is the one provision of keys the module calls name (catalogue.SecretLocal), and its key.
func byName(keys map[string][]byte, name string) (provisions.Provision, string, error) {
var hit []string
var got provisions.Provision
for key, raw := range keys {
pr, err := provisions.Decode(raw)
if err != nil {
return provisions.Provision{}, "", fmt.Errorf("the provision %s: %w", key, err)
}
if catalogue.SecretLocal(pr.Provision, pr.Local) == name {
hit, got = append(hit, key), pr
}
}
switch len(hit) {
case 0:
return provisions.Provision{}, "", fmt.Errorf("no provision of %s goes by that name: nothing asks for it", provisions.ProvisionsRead)
case 1:
return got, hit[0], nil
}
sort.Strings(hit)
return provisions.Provision{}, "", fmt.Errorf("the provisions %s all go by that name", strings.Join(hit, ", "))
}
// value is one provision's value on this node, opened.
func (p provisionSecrets) value(ctx context.Context, key string, pr provisions.Provision) ([]byte, error) {
if pr.State != provisions.Provided || pr.Reference == nil {
why := pr.State
if pr.Why != "" {
why += ": " + pr.Why
}
if pr.State == provisions.Provided {
why = "provided with no reference to the vault's value"
}
return nil, fmt.Errorf("the provision %s is %s, so its value is not written yet", key, why)
}
// The reference names a credential of this provision, as the vault names it (provisions.VaultName).
ours := false
for _, form := range []string{provisions.FormPair, provisions.FormShared, provisions.FormIssued} {
ours = ours || provisions.VaultName(form, pr) == pr.Reference.ID
}
if !ours {
return nil, fmt.Errorf("the provision %s names the vault's %s, which is no credential of it", key, pr.Reference.ID)
}
module, provision, local := pr.Consumer.Module, pr.Provision, pr.Local
type asked struct {
Role string `json:"role"`
Provision string `json:"provision"`
Consumer string `json:"consumer"`
ConsumerModule string `json:"consumer_module"`
Provider string `json:"provider"`
Local string `json:"local,omitempty"`
}
ask := asked{Role: "consumer", Provision: provision, Consumer: p.node, ConsumerModule: module, Provider: pr.Provider.Node, Local: local}
answer, err := p.own(ctx, map[string]any{"credentials": []asked{ask}})
if err != nil {
return nil, fmt.Errorf("the vault's value of %s: %w", key, err)
}
var a struct {
Node string `json:"node"`
Credentials []struct {
asked
Name string `json:"name"`
Generation int `json:"generation"`
Sealed json.RawMessage `json:"sealed"`
} `json:"credentials"`
}
if err := json.Unmarshal(answer, &a); err != nil {
return nil, fmt.Errorf("the vault's answer for %s is not one: %w", key, err)
}
if a.Node != p.node {
return nil, fmt.Errorf("the vault answered %s's credentials for %s on %s", a.Node, key, p.node)
}
for _, c := range a.Credentials {
if c.asked != ask {
continue
}
// The value the provision names, by the vault's credential name and its generation (clause 6): another is refused.
if c.Name != pr.Reference.ID || c.Generation != pr.Reference.Generation {
return nil, fmt.Errorf("the vault answered %s generation %d for %s, and the provision names %s generation %d: "+
"nothing is written until they agree", c.Name, c.Generation, key, pr.Reference.ID, pr.Reference.Generation)
}
value, err := p.open(string(c.Sealed))
if err != nil {
return nil, fmt.Errorf("the vault's value of %s does not open on this machine: %w", key, err)
}
return value, nil
}
return nil, fmt.Errorf("the vault answered no value of %s for this machine", key)
}
+2
View File
@@ -36,6 +36,7 @@ func TestACompositionIsRecordedOnlyWhenEveryResourceIs(t *testing.T) {
// The modules applied again are those with a missing resource, each once; one down but there is not among them.
func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
t.Run("1957: the machine shows a resource not active", testTheMachineShowsAResourceNotActive)
st := liveness.Statement{Resources: []liveness.State{
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.homeserver"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.db"}, State: liveness.Unhealthy, Missing: true},
@@ -51,6 +52,7 @@ func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
// not acted on again, so a resource that cannot be made is not applied (and backed up) at every unrelated change. One
// that comes back and goes again is newly missing again (the review of issue 1945).
func TestOnlyANewlyMissingResourceAppliesItsModuleAgain(t *testing.T) {
t.Run("1957: a resource newly down is an edge", testOnlyAResourceNewlyDownIsAnEdge)
missing := func(ids ...string) liveness.Statement {
var st liveness.Statement
for _, id := range ids {
-94
View File
@@ -1,94 +0,0 @@
package main
import (
"crypto/ed25519"
"strings"
"testing"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/tunnel"
)
// novox/hq ADR 0105: `overlay take` moves this node's overlay key onto the found tunnel's and
// nothing else — identity, sealing and serving keys stay as they were — and tells the mesh with a
// proof signed by the identity key, over the previous key, the new one and the tunnel.
func anEnrolledNode(t *testing.T) identity.Identity {
t.Helper()
mine, err := identity.Generate("anchor")
if err != nil {
t.Fatal(err)
}
mine.Overlay, err = identity.GenerateOverlayKey()
if err != nil {
t.Fatal(err)
}
mine.Membership = identity.Membership{Broker: "198.51.100.1:5671", Fingerprint: "sha256:aa",
Signer: make([]byte, ed25519.PublicKeySize), Password: "p"}
return mine
}
func aFoundTunnel(t *testing.T) tunnel.Found {
t.Helper()
private, err := identity.GenerateOverlayKey()
if err != nil {
t.Fatal(err)
}
found, err := tunnel.Parse([]byte("[Interface]\nPrivateKey = " + private.Private + "\nListenPort = 51900\n" +
"Address = 192.0.2.1/24\n[Peer]\nPublicKey = PEER-A=\nAllowedIPs = 192.0.2.2/32\n"))
if err != nil {
t.Fatal(err)
}
found.Interface, found.Unit, found.Config = "wg0", "wg-quick@wg0", "/etc/wireguard/wg0.conf"
return found
}
func TestTakingATunnelMovesOnlyTheOverlayKeyAndSignsForIt(t *testing.T) {
mine := anEnrolledNode(t)
found := aFoundTunnel(t)
before := mine.Overlay.Public
taken, rekey, err := rekeyOnto(mine, found)
if err != nil {
t.Fatal(err)
}
if taken.Overlay.Public != found.PublicKey || taken.Overlay.Private != found.PrivateKey() {
t.Fatal("the overlay key is not the tunnel's")
}
if string(taken.Public) != string(mine.Public) || string(taken.Private) != string(mine.Private) ||
taken.Node != mine.Node || taken.Membership.Password != mine.Membership.Password ||
taken.Membership.Broker != mine.Membership.Broker {
t.Fatal("something other than the overlay key moved")
}
if taken.OverlayBefore != before {
t.Errorf("the key before the take was not kept: %q", taken.OverlayBefore)
}
if rekey.Previous != before || rekey.OverlayKey != found.PublicKey || rekey.Tunnel == nil ||
rekey.Tunnel.PublicKey != found.PublicKey || len(rekey.Tunnel.Peers) != 1 {
t.Fatalf("the rekey does not say what moved: %+v", rekey)
}
if !ed25519.Verify(ed25519.PublicKey(mine.Public),
link.RekeyProof("anchor", before, found.PublicKey, rekey.Tunnel), rekey.Proof) {
t.Fatal("the rekey is not signed by this node's identity key over what it says")
}
if ed25519.Verify(ed25519.PublicKey(mine.Public),
link.RekeyProof("laptop", before, found.PublicKey, rekey.Tunnel), rekey.Proof) {
t.Fatal("the proof is not bound to the node")
}
for _, said := range []string{rekey.Previous, rekey.OverlayKey, rekey.Tunnel.Interface} {
if strings.Contains(said, found.PrivateKey()) {
t.Fatal("the private key travels")
}
}
// Run again after the take — the mesh not yet told, or told and refused — the previous key it
// names is still the one before the take, so the mesh can tell a repeat from a replay.
again, second, err := rekeyOnto(taken, found)
if err != nil {
t.Fatal(err)
}
if second.Previous != before || again.OverlayBefore != before || again.Overlay.Public != found.PublicKey {
t.Fatalf("a take run again does not name the key before the first: %+v", second)
}
}
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/store"
)
// runsAsWritten is what the node runs as the composer reads it (selfcompose.Env.Running), with every file given by digest
// named as this node last wrote it, where the two differ (issue 924, novox 2026-10-29 01:16). What runs of a module the
// agent has not applied since its start can come from the kept declaration, older than the compositions applied since:
// its export (mesh-vault.kept) then named a digest the controller's sweep had removed, while the file on disk was the
// newer export the node wrote. A node composing its own carries such a file as it runs (mesh-sdk frombus keptOf), so it
// names what is on disk: **only when the node's record says it wrote exactly that content there** (store Applied.Wrote,
// Target), never a file someone else changed. A file that cannot be read, or that the record does not vouch for, is
// left as the composition names it.
func runsAsWritten(running []delivery.Composed, statePath string) []delivery.Composed {
if len(running) == 0 || statePath == "" {
return running
}
known, err := store.Load(statePath)
if err != nil {
return running
}
out := make([]delivery.Composed, len(running))
for i, c := range running {
out[i] = c
var changed []json.RawMessage
for j, raw := range c.Resources {
if fixed, ok := asWritten(raw, known); ok {
if changed == nil {
changed = append([]json.RawMessage(nil), c.Resources...)
}
changed[j] = fixed
}
}
if changed != nil {
out[i].Resources = changed
}
}
return out
}
// asWritten is the file resource raw named by the digest and size of what this node last wrote at its path, when raw is a
// file given by another digest and the file there is what the record says the node wrote.
func asWritten(raw json.RawMessage, known store.State) (json.RawMessage, bool) {
var fields map[string]json.RawMessage
if json.Unmarshal(raw, &fields) != nil {
return nil, false
}
var kind, id, path, digest string
_ = json.Unmarshal(fields["type"], &kind)
_ = json.Unmarshal(fields["id"], &id)
_ = json.Unmarshal(fields["path"], &path)
_ = json.Unmarshal(fields["digest"], &digest)
if kind != string(declaration.TypeFile) || digest == "" || path == "" {
return nil, false
}
rec, ok := known.Find(id)
if !ok || rec.Wrote == "" || rec.Target != path {
return nil, false
}
body, err := os.ReadFile(path)
if err != nil {
return nil, false
}
sum := sha256.Sum256(body)
if hex.EncodeToString(sum[:]) != rec.Wrote {
return nil, false // changed since the node wrote it: not the node's to vouch for
}
if now := declaration.DigestOf(body); now == digest {
return nil, false
} else {
fields["digest"], _ = json.Marshal(now)
}
fields["size"], _ = json.Marshal(len(body))
fixed, err := json.Marshal(fields)
if err != nil {
return nil, false
}
return fixed, true
}
+289
View File
@@ -0,0 +1,289 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// **What this engine trusts a mesh signature from** (novox/hq ADR 0550 §2; design 47's L3 item d; issue 1126): the
// vault's key alone, read from the vault's state `signers`, which only the vault may write: the bus grants are its
// anchor, as the wants' and the placement's. The controller's key, which the node was enrolled with (its membership's
// signer), is trusted never: the controller ended by its unassignment (the operator, 2026-11-02) and its lease is never
// handed, so no gate waits on it. What it trusts now the node states in its record (NodeRecord.Trusts).
type meshTrust struct {
mu sync.Mutex
vault ed25519.PublicKey
authority string
// changed is told when what it trusts changed, so the record is written again.
changed func()
}
// newMeshTrust trusts nothing until the vault's key is read.
func newMeshTrust() *meshTrust { return &meshTrust{} }
// onChange names who is told when what it trusts changed.
func (t *meshTrust) onChange(f func()) {
t.mu.Lock()
t.changed = f
t.mu.Unlock()
}
// Signers are the keys a mesh signature is verified against now: the vault's once read, and none before.
func (t *meshTrust) Signers() []delivery.Signer {
t.mu.Lock()
defer t.mu.Unlock()
var out []delivery.Signer
if t.vault != nil {
out = append(out, delivery.Signer{Name: delivery.SignerVault, Key: t.vault})
}
return out
}
// Verify is the signer that signed msg as kind, or why none it trusts did.
func (t *meshTrust) Verify(kind string, msg, sig []byte) (delivery.Signer, error) {
context, err := delivery.SignContext(kind)
if err != nil {
return delivery.Signer{}, err
}
return delivery.VerifyBy(t.Signers(), context, msg, sig)
}
// VaultAuthority is the vault's authority certificate as read from `signers`, or "".
func (t *meshTrust) VaultAuthority() string {
if t == nil {
return ""
}
t.mu.Lock()
defer t.mu.Unlock()
return t.authority
}
// Trusts is what the record says it trusts: each signer's key and the vault's authority, by fingerprint.
func (t *meshTrust) Trusts() []string {
if t == nil {
return nil
}
authorities := map[string]string{}
if a := t.VaultAuthority(); a != "" {
authorities[delivery.SignerVault] = a
}
return delivery.Trusts(t.Signers(), authorities)
}
// change applies f under the lock and tells who follows when Trusts differs after it.
func (t *meshTrust) change(f func()) {
before := strings.Join(t.Trusts(), "\n")
t.mu.Lock()
f()
changed := t.changed
t.mu.Unlock()
if strings.Join(t.Trusts(), "\n") != before && changed != nil {
changed()
}
}
// heardSigner takes one key of the vault's `signers`: its key or its authority, refused (and said) when it does not
// check; a deleted key is no longer trusted.
func (t *meshTrust) heardSigner(key string, value []byte, deleted bool, say link.Announce) {
switch key {
case delivery.SignersKeyKey:
var pub ed25519.PublicKey
if !deleted && len(value) > 0 {
k, err := delivery.DecodeSignerKey(value)
if err == nil {
pub, err = k.PublicKey()
}
if err != nil {
say("the vault's signing key in " + delivery.SignersRead + " is not trusted: " + err.Error())
return
}
}
t.change(func() { t.vault = pub })
case delivery.SignersAuthorityKey:
pem := ""
if !deleted && len(value) > 0 {
a, err := delivery.DecodeSignerAuthority(value)
if err != nil {
say("the vault's authority in " + delivery.SignersRead + " is not trusted: " + err.Error())
return
}
pem = a.Certificate
}
t.change(func() { t.authority = pem })
}
}
// trustFollowAgain bounds the wait before a watch that could not be made is tried again on the same link: the vault may
// not serve `sign` yet, and until it does this node is not granted the reads.
var trustFollowAgain = 2 * time.Minute
// followTrust follows the vault's `signers` on every link, as watches (never a poll), for as
// long as ctx lasts. A watch that cannot be made is said once per reason and tried again.
func followTrust(ctx context.Context, session *link.Session, trust *meshTrust, say link.Announce) {
said := ""
sayOnce := func(line string) {
if line != said {
said = line
if line != "" {
say(line)
}
}
}
for ctx.Err() == nil {
linked, cancel, err := session.WhenLinked(ctx)
if err != nil {
return
}
watching, stop := context.WithCancel(linked)
signers, serr := session.KVWatch(watching, delivery.SignersBucket, ">")
if serr != nil {
stop()
sayOnce(fmt.Sprintf("what this node trusts a mesh signature from is not followed yet (the vault's signers: %v): "+
"it trusts what it last read, and looks again in %s", serr, trustFollowAgain))
} else {
sayOnce("")
for u := range signers {
if !u.Caught {
trust.heardSigner(u.Key, u.Value, u.Deleted, say)
}
}
}
stop()
select {
case <-linked.Done():
case <-time.After(trustFollowAgain):
}
cancel()
}
}
// placedAllHave says whether every module the seat mesh-placement places on node has a have of its own, read from the
// bus: then the kept declaration is read no more (keptRetired). A node the seat does not place, or whose placement or haves
// cannot be read, keeps reading it.
func placedAllHave(ctx context.Context, session *link.Session, node string) (bool, string) {
key, err := delivery.PlacementNodeKey(node)
if err != nil {
return false, err.Error()
}
legacy, _ := delivery.LegacyPlacementKey(node)
// Its node key names every module placed (research 072 row 37); before the holder converted it, its legacy key does.
values := map[string][]byte{}
for _, k := range []string{key, legacy} {
if h, err := session.KVHistory(ctx, delivery.PlacementBucket, k); err == nil && len(h) > 0 && !h[len(h)-1].Deleted {
values[k] = h[len(h)-1].Value
}
}
p, err := placedModules(node, values)
if err != nil {
return false, err.Error()
}
return allHave(p, func(module string) (delivery.Have, bool, error) {
return session.ReadHaveErr(ctx, module)
})
}
// placedModules is the modules node's node key names (a placement of their names alone), or its legacy key's placement
// where it has no node key; an error when neither reads.
func placedModules(node string, values map[string][]byte) (delivery.Placement, error) {
key, _ := delivery.PlacementNodeKey(node)
if raw, ok := values[key]; ok {
n, err := delivery.DecodePlacementNode(raw)
if err != nil {
return delivery.Placement{}, err
}
p := delivery.Placement{Node: node}
for _, r := range n.Modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: r.Module})
}
return p, nil
}
legacy, _ := delivery.LegacyPlacementKey(node)
raw, ok := values[legacy]
if !ok {
return delivery.Placement{}, errors.New("its placement is not read")
}
return delivery.DecodePlacement(raw)
}
// allHave says whether has finds, for every module p places, a have that names a fingerprint of the new form, and why
// not. A module whose have names only a build of the old form runs inside the node's own composition, and the kept
// declaration is the one record of what runs of it: retired then, the node's own composed it as nothing and refused to
// remove what runs (issue 774, 2026-10-30 19:43: systemd-resolved on shanks, nats and restic on novox, each node's own
// apply refused from then on).
func allHave(p delivery.Placement, has func(module string) (delivery.Have, bool, error)) (bool, string) {
if len(p.Modules) == 0 {
return false, "the seat mesh-placement places no module here"
}
for _, m := range p.Modules {
h, ok, err := has(m.Module)
if err != nil {
return false, "the have of " + m.Module + " is not read: " + err.Error()
}
if !ok {
return false, m.Module + " has no have yet"
}
// A module the mesh provides (mesh-wireguard) never has a fingerprint: the node's own composes it from the manifest
// mesh-sdk provides (delivery.ProvidedManifest), never from the kept declaration, so it keeps no read.
if h.Phase != delivery.PhaseRemoved && !strings.HasPrefix(h.FP, delivery.SourcePrefix2) && !delivery.ProvidedByMesh(m.Module) {
return false, m.Module + " runs at a build of the old form (" + h.Build + "), which only the kept declaration records"
}
}
return true, fmt.Sprintf("each of the %d modules placed here has a have of its own", len(p.Modules))
}
// certifyThisNode asks the vault's `sign` for this node's own certificate (kind node-certificate, ADR 0550 §2), checks
// it is for this node's internal name and serving key and signed by the authority the answer names, which must be the
// vault's this node trusts, keeps it where the record reads it, and writes the record again (written).
func certifyThisNode(ctx context.Context, state, node string, trust *meshTrust,
ask func(ctx context.Context, verb string, args any) (json.RawMessage, error), written func()) error {
serving, err := identity.LoadServingKey(identity.ServingKeyPath(state))
if err != nil {
return err
}
key, err := base64.StdEncoding.DecodeString(serving.Public)
if err != nil || len(key) != ed25519.PublicKeySize {
return fmt.Errorf("this node's serving key is not an ed25519 key")
}
raw, err := ask(ctx, delivery.SignVerb, map[string]any{"kind": delivery.KindNodeCertificate, "node": node})
if err != nil {
return err
}
var a delivery.SignAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return fmt.Errorf("the vault's answer is not a signing: %w", err)
}
trusted := trust.VaultAuthority()
if trusted == "" {
return fmt.Errorf("this node has read no authority of the vault's to check the certificate against")
}
if err := delivery.CheckNodeCertificate(a.Certificate, []string{trusted}, overlay.InternalName(node),
ed25519.PublicKey(key), time.Now()); err != nil {
return err
}
path := identity.CertificatePath(state)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
if err := os.WriteFile(path, []byte(a.Certificate), 0o644); err != nil {
return err
}
if written != nil {
written()
}
return nil
}
+59
View File
@@ -1,6 +1,9 @@
package main
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -9,6 +12,7 @@ import (
"github.com/novox/mesh-host/internal/accounts"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/units"
)
@@ -103,6 +107,61 @@ func TestTheStatementSaysAnAccountJudgedForRoot(t *testing.T) {
if got["claude-code.agent"] != "never" || got["openrazer.account"] != "" {
t.Fatalf("root said: %v", got)
}
// 774 (ADR 0563 §6): the node seat's root-free passes only on the engine's own positive verdict at most 15 minutes old
// with execute not served here; a stale, missing or negative verdict, or execute served, is no pass and says why.
t.Run("this machine's root-free", func(t *testing.T) {
agent := accounts.Verdict{Account: accounts.Account{Module: "claude-code", ID: "claude-code.agent", Name: "agent", Root: true},
State: accounts.Healthy, Since: at}
free := &accounts.Statement{At: at, Accounts: []accounts.Verdict{agent}}
for _, row := range []struct {
name string
st *accounts.Statement
after time.Duration
served bool
free bool
why string
}{
{name: "a positive verdict", st: free, after: 14 * time.Minute, free: true, why: "the account agent cannot become root"},
{name: "a stale verdict", st: free, after: 16 * time.Minute, why: "more than 15 minutes ago"},
{name: "no verdict", st: nil, why: "judged no account yet"},
{name: "a way to root", st: as, why: "in the group docker, which grants root"},
{name: "no agent account", st: &accounts.Statement{At: at}, why: "agents run as the operator's account"},
{name: "execute served", st: free, served: true, why: "the login shell runs any command"},
} {
v := judgeRootFree("ace", row.st, at.Add(row.after), row.served, "the bus hears execute answered here")
if v.Free != row.free || !strings.Contains(v.Why, row.why) || v.Machine != "ace" {
t.Errorf("%s: %+v", row.name, v)
}
}
// execute is served while the holder's setting file says anything but withhold, or the bus hears it.
dir := t.TempDir()
statePath, conf := filepath.Join(dir, "state.json"), filepath.Join(dir, "tools.conf")
if err := store.Save(statePath, store.State{Resources: []store.Applied{{ID: "zsh.tools-conf", Type: "file", Target: conf}}}); err != nil {
t.Fatal(err)
}
quiet := func() (bool, error) { return false, nil }
for _, row := range []struct {
name, setting string
heard func() (bool, error)
served bool
}{
{name: "withheld and not heard", setting: "# the mesh's\nexecute=withhold\n", heard: quiet},
{name: "served", setting: "execute=serve\n", heard: quiet, served: true},
{name: "withheld and heard", setting: "execute=withhold\n", heard: func() (bool, error) { return true, nil }, served: true},
{name: "the bus not asked", setting: "execute=withhold\n", heard: func() (bool, error) { return false, errors.New("no link") }, served: true},
{name: "no setting file", heard: quiet, served: true},
} {
_ = os.Remove(conf)
if row.setting != "" {
if err := os.WriteFile(conf, []byte(row.setting), 0o644); err != nil {
t.Fatal(err)
}
}
if served, why := executeServedHere(statePath, row.heard); served != row.served {
t.Errorf("%s: served %v (%s)", row.name, served, why)
}
}
})
}
// A verdict waiting for the setuid search says how far it got, in the reason the root-free verb repeats and as
+84 -29
View File
@@ -2,36 +2,35 @@ package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/seal"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
"github.com/novox/mesh-host/internal/userlist"
)
// The bridge of novox/hq ADR 0547 §8, wired: the bus node's engine writes the bus's user list alone from the newest
// composition of the bus module, under the apply's two locks as a filled file is written (filling.go), and says what
// The bridge of novox/hq ADR 0547 §8, wired: the bus node's engine writes the grants part's user list alone into the file
// the bus module's composition declares, under the apply's two locks as a filled file is written (filling.go), and says what
// it wrote in the bus module's have.
// userListFollower is the follower of the bus's user list for one link: the node's composition bucket, the keys the
// agent trusts, the write alone, and the have.
func userListFollower(opts options, session *link.Session, composedBucket string, trusted func() []ed25519.PublicKey,
func userListFollower(opts options, session *link.Session, running func() []delivery.Composed,
say link.Announce, wrote func()) *userlist.Follower {
return userlist.New(userlist.Env{
Node: session.Membership.Node,
Composed: bucketWatch{session, composedBucket},
Trusted: trusted,
Resolve: func(ctx context.Context, raw json.RawMessage) ([]byte, error) {
return resolveUserList(session.WithLink(ctx), raw)
},
OnDisk: userListOnDisk,
Node: session.Membership.Node,
Running: running,
OnDisk: userListOnDisk,
Write: func(ctx context.Context, module string, raw json.RawMessage, body []byte) (string, bool, error) {
digest, changed, err := writeUserListAlone(session.WithLink(ctx), opts, raw, body, say)
if changed && wrote != nil {
@@ -42,13 +41,42 @@ func userListFollower(opts options, session *link.Session, composedBucket string
Report: func(ctx context.Context, module string, w userlist.Written) {
session.NoteUserList(ctx, module, w)
},
// The vault's key from its answer to node-key: the loaded list's hashes are handed sealed to it (ADR 0547
// clause 9).
VaultKey: vaultKey.get,
Say: announceOr(say),
// From take-over, the grants part's list (ADR 0547 clauses 7, 8): its one key watched, its object read by name and
// opened with this machine's sealing key, the key's public half said in the have so the grants part seals to it.
Grants: bucketWatch{session, delivery.GrantsUserListBucket},
Fetch: func(ctx context.Context, name string) ([]byte, error) {
return session.GrantsObject(ctx, name, grantsListLimit)
},
Open: func(b seal.Box) ([]byte, error) {
key, err := identity.LoadSealingKey(identity.SealingKeyPath(opts.state))
if err != nil {
return nil, err
}
return key.OpenVaultBox(b)
},
Key: func() string {
key, err := identity.LoadSealingKey(identity.SealingKeyPath(opts.state))
if err != nil {
return ""
}
pem, err := key.PublicPEM()
if err != nil {
return ""
}
return pem
},
// The grants part's key as last heard, kept beside the node's state, followed while it cannot be watched; its
// watch tried again as a passing failure of the bus is (ADR 0547 clause 8).
Kept: func() ([]byte, uint64) { return keptGrantsKey(opts) },
Keep: func(value []byte, rev uint64) { keepGrantsKey(opts, value, rev, say) },
Backoff: link.Backoff,
Say: announceOr(say),
})
}
// grantsListLimit is the largest sealed list the bus node reads: the live list is about a hundred kilobytes.
const grantsListLimit = 16 << 20
// userListFile is a user-list resource as the file it declares.
func userListFile(raw []byte) (*declaration.File, error) {
r, err := declaration.ParseResource(raw)
@@ -62,20 +90,6 @@ func userListFile(raw []byte) (*declaration.File, error) {
return f, nil
}
// resolveUserList is the list a user-list resource holds: its content, or the object its digest names in this node's
// file bucket, fetched and verified (apply.ResolveFileReferences, ADR 0486 §1).
func resolveUserList(ctx context.Context, raw []byte) ([]byte, error) {
f, err := userListFile(raw)
if err != nil {
return nil, err
}
d := &declaration.Declaration{Version: declaration.Version, Resources: []declaration.Resource{f}}
if err := apply.ResolveFileReferences(ctx, d, link.FilesFrom(ctx)); err != nil {
return nil, fmt.Errorf("the user list could not be had by its digest: %w", err)
}
return []byte(d.Resources[0].(*declaration.File).Content), nil
}
// userListOnDisk is what the user list's file holds on this machine now; nil when there is none.
func userListOnDisk(raw json.RawMessage) ([]byte, error) {
f, err := userListFile(raw)
@@ -125,3 +139,44 @@ func writeUserListAlone(ctx context.Context, opts options, raw []byte, body []by
}
return digest, writeErr == nil && out.Action != "unchanged", writeErr
}
// grantsKeyPath is where the grants part's user-list key last heard is kept, beside the node's state.
func grantsKeyPath(opts options) string {
return filepath.Join(filepath.Dir(opts.state), "grants-user-list.json")
}
type keptKey struct {
Revision uint64 `json:"revision"`
Value json.RawMessage `json:"value"`
}
func keptGrantsKey(opts options) ([]byte, uint64) {
raw, err := os.ReadFile(grantsKeyPath(opts))
if err != nil {
return nil, 0
}
var k keptKey
if json.Unmarshal(raw, &k) != nil || len(k.Value) == 0 {
return nil, 0
}
return k.Value, k.Revision
}
func keepGrantsKey(opts options, value []byte, rev uint64, say link.Announce) {
path := grantsKeyPath(opts)
if value == nil {
if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
announceOr(say)("the grants part's user-list key kept on this machine could not be taken away: " + err.Error())
}
return
}
raw, _ := json.Marshal(keptKey{Revision: rev, Value: value})
tmp := path + ".new"
err := os.WriteFile(tmp, raw, 0o600)
if err == nil {
err = os.Rename(tmp, path)
}
if err != nil {
announceOr(say)("the grants part's user-list key could not be kept on this machine: " + err.Error())
}
}
+96
View File
@@ -1,16 +1,22 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"path/filepath"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/store"
)
func aFileOf(module, id string) json.RawMessage {
@@ -59,6 +65,96 @@ func TestDeclaredNowReadsWhatRunsWhenTheAgentIsRunning(t *testing.T) {
if err != nil || d.Provisioner != "from-agent" {
t.Fatalf("declaredNow read %+v (%v), not what runs", d, err)
}
// And the health loop's judges are handed what runs once a module settles (g14 and shanks, 2026-10-30 16:14):
// restic's composition dropped its process restic.measure, which stayed watched and said down on every look.
j, err := liveness.Open(filepath.Join(t.TempDir(), "liveness.json"), fakeRuntime{})
if err != nil {
t.Fatal(err)
}
judging = j
defer func() { judging = nil }()
process := func(name string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": "restic." + name, "type": "process", "name": "restic-" + name,
"source": "https://store.invalid/restic", "digest": "sha256:" + strings.Repeat("a", 64),
"run": []string{"./restic"}, "module": "restic"})
return raw
}
for _, row := range []struct {
runs []string
watch string
}{{[]string{"night", "measure"}, "restic.measure restic.night"}, {[]string{"night"}, "restic.night"}} {
var raws []json.RawMessage
for _, name := range row.runs {
raws = append(raws, process(name))
}
runs, err := whatRuns([]delivery.Composed{{Node: "n1", Module: "restic", Resources: raws}}, false,
func() (*declaration.Declaration, error) { return nil, errors.New("none kept") })
if err != nil {
t.Fatal(err)
}
setRunsHere(func(string, ed25519.PublicKey) (*declaration.Declaration, error) { return runs, nil })
judgeWhatRunsNow(filepath.Join(t.TempDir(), "state.json"), nil)
st, _ := j.Look(context.Background())
var watched []string
for _, r := range st.Resources {
watched = append(watched, r.ID)
}
sort.Strings(watched)
if got := strings.Join(watched, " "); got != row.watch {
t.Errorf("running %v, the judge watches %q, want %q", row.runs, got, row.watch)
}
}
// L3 s9 (ADR 0550 §2): a node whose placed modules all have a have reads the kept declaration no more, though the
// file is there and readable and the controller's key is dropped; until then it reads it.
state := filepath.Join(t.TempDir(), "state", "state.json")
if err := store.SaveDeclared(store.DeclaredPath(state), store.Declared{Declaration: []byte(`{"declaration":1,` +
`"resources":[{"id":"zsh.rc","type":"file","module":"zsh","path":"/etc/zshrc","content":"x"}]}`)}); err != nil {
t.Fatal(err)
}
if d, err := keptDeclaration(state, nil); err != nil || len(d.Resources) != 1 {
t.Fatalf("before the read is retired: %+v, %v", d, err)
}
placed := delivery.Placement{Node: "n1", Modules: []delivery.PlacedModule{{Module: "zsh"}, {Module: "lemurs"}}}
// Issue 774, 2026-10-30 19:43: a have naming only a build of the old form (systemd-resolved on shanks, folded in the
// node's own) keeps the kept declaration read, since it is the one record of what runs of it.
newForm := delivery.Have{FP: delivery.SourcePrefix2 + strings.Repeat("a", 64), Phase: delivery.PhaseRunning}
oldForm := delivery.Have{Build: "build-1791404191702719760", Phase: delivery.PhaseRunning}
for _, row := range []struct {
name string
has map[string]delivery.Have
want bool
}{
{"every placed module has a have of the new form", map[string]delivery.Have{"zsh": newForm, "lemurs": newForm}, true},
{"one has none yet", map[string]delivery.Have{"zsh": newForm}, false},
{"one runs at a build of the old form", map[string]delivery.Have{"zsh": newForm, "lemurs": oldForm}, false},
} {
if got, why := allHave(placed, func(m string) (delivery.Have, bool, error) { h, ok := row.has[m]; return h, ok, nil }); got != row.want {
t.Errorf("%s: %v (%s)", row.name, got, why)
}
}
// The module the mesh provides never has a fingerprint, and keeps no read (review of issue 774).
withWireguard := delivery.Placement{Node: "n1", Modules: []delivery.PlacedModule{{Module: "zsh"}, {Module: overlay.Name}}}
if got, why := allHave(withWireguard, func(m string) (delivery.Have, bool, error) {
if m == overlay.Name {
return oldForm, true, nil
}
return newForm, true, nil
}); !got {
t.Errorf("%s without a fingerprint beside zsh at the new form keeps the read: %s", overlay.Name, why)
}
if got, _ := allHave(delivery.Placement{Node: "n1"}, nil); got {
t.Error("a node with nothing placed retires the read")
}
keptRetired.Store(true)
defer keptRetired.Store(false)
if _, err := keptDeclaration(state, nil); !errors.Is(err, errKeptRetired) {
t.Errorf("after the read is retired: %v", err)
}
if by, filled := keptOnce(options{state: state}, nil)(); by != nil || filled != nil {
t.Errorf("after the read is retired the kept declaration was split: %v %v", by, filled)
}
}
// 1945/AC17 (unit): a resource the node's own composition shares with a running module's composition is declared once,
+1 -2
View File
@@ -146,9 +146,8 @@ func launcherReplaced(pid int, proc, path string) bool {
// watchWhatThisHostPlaced judges the core builds this host placed, until ctx ends (to-be 45 §8).
func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Session, say func(string)) {
host, _ := os.Hostname()
w := &witness.Watcher{
Root: apply.DaemonRoot(), Node: node, Host: host,
Root: apply.DaemonRoot(), Node: node,
Asker: func() link.Asker { return queue.Asker() },
Run: witness.Runner(apply.ExecRunner),
Hold: func(f func()) {
+1 -1
View File
@@ -3,7 +3,7 @@ module github.com/novox/mesh-host
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261102191058-f02f142af128
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
+2 -8
View File
@@ -1,11 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f h1:lUEyNPPF3w4Q0LIgcUUD6ofkKVhCF0n7dVjMoixQmXo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456 h1:He2o9vSKjgKbWOCOgOJY0WlaDh1n8RcGhvGY/dwXsBQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e h1:k/1NjjW5HuuuHW2ldyWTvjp+1u3fN0RO18Q57Iup6Dk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936 h1:HGBoWimZXYZp+Qn9VCstYDitOP84t9EUQYd1yM9j2eg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261102191058-f02f142af128 h1:aI1DhZAuqVBXbybSqhLNOGX5rUOrn9T0cHNCDUQF7IQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261102191058-f02f142af128/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+11
View File
@@ -240,6 +240,17 @@ func (c *SetuidCache) SearchAgain() {
c.forgetProgress()
}
// Running says whether a search walks now (novox/hq ADR 0563 §2: the node seat starts no second while one does).
// Nil c runs none.
func (c *SetuidCache) Running() bool {
if c == nil {
return false
}
c.mu.Lock()
defer c.mu.Unlock()
return c.running
}
// SearchNow throws away the last complete search, kept or in memory, and any walk under way or kept, and has
// the next look start a full search at once, past AgainAfter and any back-off (novox/hq issue 361): the
// operator changed by hand what the last one found, which no apply says. Until the new one completes the
+855 -67
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+221
View File
@@ -0,0 +1,221 @@
package agent
import (
"context"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/judge"
)
// The node-engine seat's read verb composition (issue 774): one module's composition beside what runs of it, so why a
// record module waits for a yes, or what its next apply would change, is read from the engine and not guessed from its
// have's words. It changes nothing.
// CompositionView is what the verb composition answers.
type CompositionView struct {
Node string `json:"node"`
Module string `json:"module"`
// Want is the module's want as the agent holds it; nil when it holds none.
Want *WantView `json:"want,omitempty"`
// Have is what the module's have key says now; nil when it says nothing.
Have *HaveView `json:"have,omitempty"`
// Composition is the revision read: the latest of the want's build, or the one whose digest was asked.
Composition RevisionView `json:"composition"`
// Revisions are the revisions of its key the bucket keeps, oldest first, each with its digest.
Revisions []RevisionView `json:"revisions,omitempty"`
// Runs is what runs of the module, and where that is known from; nil when nothing of it runs.
Runs *RunsView `json:"runs,omitempty"`
// Judged is the move from what runs to the composition read, as the agent judges it before an apply.
Judged JudgedView `json:"judged"`
// Diff is every resource that differs from what runs, when asked.
Diff []judge.ResourceChange `json:"diff,omitempty"`
}
// WantView is a want as the verb shows it.
type WantView struct {
FP string `json:"fp"`
Composed string `json:"composed,omitempty"`
Yes string `json:"yes,omitempty"`
Removed bool `json:"removed,omitempty"`
}
// HaveView is a have as the verb shows it.
type HaveView struct {
FP string `json:"fp,omitempty"`
Composed string `json:"composed,omitempty"`
Phase string `json:"phase"`
Detail string `json:"detail,omitempty"`
Judgement string `json:"judgement,omitempty"`
Waits string `json:"waits,omitempty"`
}
// RevisionView is one revision of a composition key.
type RevisionView struct {
Key string `json:"key"`
Revision uint64 `json:"revision,omitempty"`
Digest string `json:"digest,omitempty"`
Resources int `json:"resources,omitempty"`
Latest bool `json:"latest,omitempty"`
// Refused is why the revision cannot be read or trusted.
Refused string `json:"refused,omitempty"`
}
// RunsView is what runs of a module.
type RunsView struct {
FP string `json:"fp,omitempty"`
Composed string `json:"composed,omitempty"`
Phase string `json:"phase,omitempty"`
// From is applied (what this engine applied), have (what its have named at the start) or kept (the declaration the
// node kept before the switch).
From string `json:"from"`
Resources int `json:"resources"`
}
// JudgedView is a move's judgement.
type JudgedView struct {
Judgement string `json:"judgement"`
Words string `json:"words"`
// WaitsForYes says the agent would wait for a person's yes before applying it: a module of policy record, a move that
// recreates a container, moves data or cannot be judged, and no yes in its want for this digest.
WaitsForYes bool `json:"waits_for_yes"`
}
// Composition reads one module's composition beside what runs of it: the want's build (else the have's), its latest
// revision or the one whose digest is given, and the move's judgement; with diff, the resources that differ.
func (a *Agent) Composition(ctx context.Context, module, digest string, diff bool) (CompositionView, error) {
if module == "" {
return CompositionView{}, errors.New("name the module, or node for the node's own composition")
}
view := CompositionView{Node: a.env.Node, Module: module}
var have *Have
if a.env.Haves != nil {
if h, ok := a.env.Haves.Read(ctx, module); ok {
have = &h
phase := h.Phase
if phase == "" {
phase = string(delivery.PhaseRunning)
}
view.Have = &HaveView{FP: h.FP, Composed: h.Composed, Phase: phase, Detail: h.Detail, Judgement: h.Judgement, Waits: h.Waits}
}
}
a.mu.Lock()
w := a.wants[module]
fp := ""
if w.Present {
fp = w.Want.FP
view.Want = &WantView{FP: w.Want.FP, Composed: w.Want.Composed, Yes: w.Want.Yes, Removed: w.Want.Removed}
} else if have != nil {
fp = have.FP
}
latest, haveLatest := a.compositionFor(module, fp)
r, ran := a.running[module]
from, known := a.runningComposition(module)
source := "applied"
if _, ok := a.applied[module]; !ok {
if _, ok := a.had[module]; ok {
source = "have"
} else {
source = "kept"
}
}
a.mu.Unlock()
if module != NodeModule && fp == "" {
return view, fmt.Errorf("%s has no want and no have on %s, so no build of it is composed here", module, a.env.Node)
}
key, err := a.compositionKey(module, fp)
if err != nil {
return view, err
}
revs, err := a.revisionsOf(ctx, key)
if err != nil {
return view, fmt.Errorf("the revisions of %s cannot be read: %w", key, err)
}
var read *compNow
for i := range revs {
c := revs[i]
rv := RevisionView{Key: key, Revision: c.Rev, Digest: c.Dig, Resources: len(c.C.Resources),
Latest: haveLatest && c.Rev == latest.Rev}
if c.Err != nil {
rv.Refused = c.Err.Error()
}
view.Revisions = append(view.Revisions, rv)
if digest != "" && c.Err == nil && c.Dig == digest {
read = &revs[i]
}
}
switch {
case digest != "" && read == nil && haveLatest && latest.Err == nil && latest.Dig == digest:
read = &latest
case digest != "" && read == nil:
return view, fmt.Errorf("no revision of %s the bucket keeps has the digest %s; revisions lists those it keeps", key, digest)
case read == nil && !haveLatest:
return view, fmt.Errorf("the bucket holds no composition of %s at %s", module, fp)
case read == nil:
read = &latest
}
view.Composition = RevisionView{Key: key, Revision: read.Rev, Digest: read.Dig, Resources: len(read.C.Resources),
Latest: haveLatest && read.Rev == latest.Rev}
if read.Err != nil {
view.Composition.Refused = read.Err.Error()
return view, nil
}
var running *delivery.Composed
if known {
running = &from
rv := &RunsView{From: source, Resources: len(from.Resources), FP: from.FP}
if ran {
rv.FP, rv.Composed, rv.Phase = r.FP, r.Composed, string(r.Phase)
}
view.Runs = rv
}
j := judge.JudgeMove(running, read.C)
view.Judged = JudgedView{Judgement: string(j.Judgement), Words: j.Words}
if a.env.SelfComposed && a.env.Policy != nil && module != NodeModule && (!w.Present || w.Want.Yes != read.Dig) {
switch j.Judgement {
case delivery.JudgedRecreates, delivery.JudgedMovesData, delivery.JudgedCannot:
view.Judged.WaitsForYes = a.env.Policy(ctx, module) == catalogue.PolicyRecord
}
}
if diff {
view.Diff = judge.ChangedResources(running, read.C)
}
return view, nil
}
// compositionKey is a module's composition key at a build, or the node's own.
func (a *Agent) compositionKey(module, fp string) (string, error) {
if module == NodeModule {
return delivery.ComposedNodeKey(a.env.Node)
}
return delivery.ComposedKey(a.env.Node, module, fp)
}
// revisionsOf is every revision of a composition key the state keeps, oldest first, each with its digest (unsigned, ADR
// 0593 §6), or why it is refused. Nil Revisions
// reads none.
func (a *Agent) revisionsOf(ctx context.Context, key string) ([]compNow, error) {
if a.env.Revisions == nil {
return nil, nil
}
revs, err := a.env.Revisions(ctx, key)
if err != nil {
return nil, err
}
var out []compNow
for _, u := range revs {
if u.Deleted || u.Key != key {
continue
}
c, err := delivery.DecodeComposed(u.Value)
if err != nil {
out = append(out, compNow{Raw: u.Value, Rev: u.Revision, Err: err})
continue
}
dig, err := delivery.ComposedDigestOf(c) // unsigned: a record, never verified (ADR 0593 §6)
out = append(out, compNow{Raw: u.Value, C: c, Dig: dig, Rev: u.Revision, Err: err})
}
return out, nil
}
+177 -3
View File
@@ -312,11 +312,54 @@ type Scope struct {
Module string
// Others are the resources of every other module the node has a composition for.
Others []declaration.Resource
// Placed are the modules a want still places on the node (present, not removed), held ones included: no apply but
// the module's own removes a resource of one of them (issue 774: on 2026-10-29 the node's own apply on the bus node,
// composed without what its folded modules ran, removed the bus server and four held modules). Nil guards nothing.
Placed map[string]bool
// RemovedByWant are the modules whose want says removed. In the node's own apply (Module NodeModule) with this set
// non-nil, a resource of any other module is removed only when that module is here: a module the node runs folded
// in its own composition has no want, so Placed does not name it (issue 774, review of the re-land, round 1: nats on
// the bus node has none, and its nats.conf was guessed as nats's and removed). Nil guards by Placed alone.
RemovedByWant map[string]bool
}
// placedOwner is the module a want still places on the node that the record belongs to, other than the scope's own:
// the module recorded with it, else the one its id names (a resource the node's own composition carried for a folded
// module is recorded as the node's, its id naming the module). Empty when there is none.
func (sc *Scope) placedOwner(a store.Applied) string {
if sc == nil || (len(sc.Placed) == 0 && sc.RemovedByWant == nil) {
return ""
}
m := a.Module
if m == "" || m == store.NodeModule {
m, _ = declaration.ModuleGuess(a.ID)
}
if m == "" || m == sc.Module || m == store.NodeModule {
return ""
}
if sc.Placed[m] {
return m
}
if sc.Module == store.NodeModule && sc.RemovedByWant != nil && !sc.RemovedByWant[m] {
return m // a module the node's own composition carried, which no want removed
}
return ""
}
// owns reports whether a record belongs to the scope's module.
//
// In a module's own apply, a record the node's own composition made for the module while it was folded (recorded as the
// node's, its id naming the module) is the module's too: it is the only apply that may remove it once neither
// composition declares it (issue 774). One the node's own still declares stays, as another's held resource does.
func (sc *Scope) owns(a store.Applied) bool {
return store.OwnerOf(a, declaration.ModuleGuess) == sc.Module
if store.OwnerOf(a, declaration.ModuleGuess) == sc.Module {
return true
}
if sc.Module == store.NodeModule || a.Module != store.NodeModule {
return false
}
m, ok := declaration.ModuleGuess(a.ID)
return ok && m == sc.Module
}
// isNode is whether the scope is the node's own composition, the one that carries identities, groups, libraries and
@@ -441,6 +484,57 @@ func applyScoped(
}
}
// **A resource the node's own carried for a module that now has a composition of its own is that module's to remove**
// (issue 774, 2026-10-30 on the bus node): recorded as the node's while the module was folded, it is no longer in the
// node's own composition and no longer in the module's own either (restic's measure, replaced by awake). The node's own
// apply leaves it, record and all, and says so; the module's own apply sees it as its own (Scope.owns) and removes it.
// Refused instead, the node's own applied nothing at all, every two minutes, for as long as the record stayed.
takenByItsModule := map[string]string{}
if scope != nil && scope.Module == store.NodeModule {
composes := map[string]bool{}
for _, r := range scope.Others {
if m := r.DeclaredModule(); m != "" && m != store.NodeModule {
composes[m] = true
}
}
for _, orphan := range known.Orphans(declared, origin) {
if orphan.Module != store.NodeModule {
continue
}
if m, ok := declaration.ModuleGuess(orphan.ID); ok && composes[m] {
takenByItsModule[orphan.ID] = m
}
}
}
// **A module a want still places here is never removed by another's apply** (issue 774): its resources go only with
// its own apply or its want's removal. One that would is refused whole, before anything is changed, naming them: a
// composition that leaves them out lost what runs, and removing what runs is the worst a node can do with that.
if scope != nil {
var refused []string
for _, orphan := range known.Orphans(declared, origin) {
if !scope.owns(orphan) {
continue
}
if _, kept := heldByOthers[orphan.ID]; kept {
continue
}
if _, kept := takenByItsModule[orphan.ID]; kept {
continue
}
if m := scope.placedOwner(orphan); m != "" {
refused = append(refused, orphan.ID+" of "+m)
}
}
if len(refused) > 0 {
sort.Strings(refused)
err := fmt.Errorf("the apply of %s would remove %s, of modules a want still places here; a module's resources "+
"go only with its own apply or its want's removal, so nothing was applied", scope.Module, strings.Join(refused, ", "))
log(" refused: " + err.Error())
return report, known, &Error{Resource: "the resources of modules placed here", Err: err, Done: report}
}
}
removeOrphan := func(orphan store.Applied) error {
var action, detail string
var err error
@@ -456,6 +550,16 @@ func applyScoped(
log(fmt.Sprintf(" kept %s (%s): %s", orphan.ID, orphan.Target, detail))
return nil
}
if module, taken := takenByItsModule[orphan.ID]; taken {
detail := "no longer declared here; " + module + " has a composition of its own here, whose apply removes it, " +
"so it was left as it is"
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "kept", Detail: detail,
})
log(fmt.Sprintf(" kept %s (%s): %s", orphan.ID, orphan.Target, detail))
return nil
}
if by, held := heldFiles[fileKey(orphan)]; held && by != orphan.ID {
if scope != nil && !declared[by] {
// Another module declares the path: it is that module's, so this record is let go and the file is
@@ -1223,6 +1327,19 @@ func applyScoped(
}
}
// A file written or removed in a unit directory is read by the service manager only after a reload, whether or not
// a service reflects it: a drop-in on reboot.target is reflected by no service, and without the reload the next
// reboot still runs the target as it was loaded (issue 1126). Once per apply.
if at := unitFileMoved(report.Outcomes); at != "" {
if u, ok := sys.(unitReloader); ok {
if err := u.ReloadUnits(ctx, run); err != nil {
failures = append(failures, &Error{Resource: at, Err: fmt.Errorf("reloading the service manager's units: %w", err)})
} else {
log(fmt.Sprintf(" reloaded the service manager's units: %s changed", at))
}
}
}
// Only the mesh says what is assigned: a carried bundle's silence undeclares no identity and no folder.
if origin == store.OriginDeclared && scope.isNode() {
// And an identity, then a library group, no longer declared goes, once what ran as it went above (ADR 0328
@@ -1997,11 +2114,68 @@ type serviceReloader interface {
ReloadService(ctx context.Context, run system.Runner, unit string) error
}
// UnitDirs are the system service manager's unit directories: a file the apply writes or removes in one of them, or in a
// drop-in directory below one, is read by the manager only after a reload (issue 1126). Replaceable in tests.
var UnitDirs = []string{"/etc/systemd/system", "/run/systemd/system", "/usr/local/lib/systemd/system"}
// unitFileMoved is the first file this apply wrote or removed in a unit directory or a drop-in directory below one,
// or "" when it moved none.
func unitFileMoved(outcomes []Outcome) string {
for _, o := range outcomes {
// Only what changed the file on disk, as MovesRoot reads it: skipped, held, kept, forgotten and waiting come
// back on every apply and change nothing.
if declaration.Type(o.Type) != declaration.TypeFile {
continue
}
switch o.Action {
case "created", "updated", "corrected", "removed":
default:
continue
}
at := filepath.Clean(o.Target)
for _, dir := range UnitDirs {
if rel, err := filepath.Rel(dir, at); err == nil && rel != "." && !strings.HasPrefix(rel, "..") {
return at
}
}
}
return ""
}
// unitReloader is a service manager that caches unit files and must be told to read them again.
type unitReloader interface {
ReloadUnits(ctx context.Context, run system.Runner) error
}
// restReader is a service manager that can say whether a one-shot has nothing running now
// (novox/hq issue 1126).
type restReader interface {
ServiceAtRest(ctx context.Context, run system.Runner, unit string) (bool, error)
}
// declaredState is a unit's state as a declaration of want reads it. **A one-shot at rest is
// stopped to a declaration that it be stopped** (novox/hq issue 1126): a one-shot that does not
// linger reads "running" once it ran and worked — that is what a declaration that it run asks — but
// it has no process between its runs, so it is also as stopped as it can be, and stopping it changes
// nothing. Read only as "running", a boot-time cleanup declared stopped and enabled failed every
// apply after the boot that ran it: "was asked to be stopped and is running".
func declaredState(ctx context.Context, sys system.System, run Runner, unit, want string) (string, error) {
state, err := sys.ServiceState(ctx, run, unit)
if err != nil || want != "stopped" || state != "running" {
return state, err
}
if r, ok := sys.(restReader); ok {
rest, err := r.ServiceAtRest(ctx, run, unit)
if err != nil {
return "", err
}
if rest {
return "stopped", nil
}
}
return state, nil
}
// serviceSettle is how long the host waits before looking at a unit a second time. A test sets it
// to nothing; on a machine it is the window in which a daemon that refuses its configuration dies.
var serviceSettle = 2 * time.Second
@@ -2154,7 +2328,7 @@ func applyServiceIn(ctx context.Context, sys system.System, r *declaration.Servi
// own stopping from anybody else's (novox/hq ADR 0269).
out.declared = &store.FoundUnit{Unit: r.Unit, State: r.State, Boot: r.Boot}
before, err := sys.ServiceState(ctx, run, r.Unit)
before, err := declaredState(ctx, sys, run, r.Unit, r.State)
if err != nil {
return out, err
}
@@ -2165,7 +2339,7 @@ func applyServiceIn(ctx context.Context, sys system.System, r *declaration.Servi
// Read back. A service manager accepting a command says the transaction was accepted,
// not that the unit is running — one that starts and immediately dies satisfies it.
after, err := sys.ServiceState(ctx, run, r.Unit)
after, err := declaredState(ctx, sys, run, r.Unit, r.State)
if err == nil && r.State == "running" {
after, err = stayedRunning(ctx, sys, run, r.Unit)
}
+82
View File
@@ -350,6 +350,42 @@ func TestAnUnknownServiceStateIsRefusedNotGuessed(t *testing.T) {
}
}
// A boot-time one-shot a package ships, declared stopped and enabled at boot, is satisfied while
// nothing of it runs, whatever its last run did (novox/hq issue 1126). The kernel module's
// linux-modules-cleanup.service failed on ace with "was asked to be stopped and is running": an
// inactive one-shot with ExecMainStatus=0 reads "running", and stopping it changes nothing.
func TestABootOneShotDeclaredStoppedIsSatisfiedAtRest(t *testing.T) {
for _, c := range []struct{ name, status string }{
{"exited 0: never ran, ran at boot, or unloaded", "0"},
} {
t.Run(c.name, func(t *testing.T) {
var asked []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "show":
return "LoadState=loaded\nActiveState=inactive\nType=oneshot\nRemainAfterExit=no\n" +
"ExecMainStatus=" + c.status + "\n", nil
case "is-enabled":
return "enabled\n", nil
}
asked = append(asked, strings.Join(args, " "))
return "", nil
}
d := parse(t, `{"declaration":1,"resources":[
{"id":"cleanup","type":"service","unit":"linux-modules-cleanup.service","state":"stopped","boot":"enabled"}
]}`)
_, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatalf("a one-shot at rest declared stopped failed: %v", err)
}
if len(asked) > 0 {
t.Errorf("the service manager was asked %q for a unit already as declared", asked)
}
})
}
}
func TestADroppedServiceIsGivenBackTheStateItWasFoundIn(t *testing.T) {
// novox/hq ADR 0118: undeclaring removes what the mesh made, gives back what it changed, and
// leaves what was the machine's. A service resource never installs a unit — so undeclaring it
@@ -1225,6 +1261,52 @@ func TestAServiceIsRestartedWhenWhatItReflectsChanges(t *testing.T) {
if reloaded < 0 || reloaded > stop {
t.Errorf("the service was restarted without the unit files being read again first; commands were %v", commands)
}
// A drop-in in a unit directory that no service reflects — the power module's bound on reboot.target
// (issue 1126) — is read by the service manager only after a reload: one per apply, and no restart.
// Unchanged, it reloads nothing.
units := filepath.Join(dir, "system")
saved := UnitDirs
UnitDirs = []string{units}
t.Cleanup(func() { UnitDirs = saved })
dropIn := filepath.Join(units, "reboot.target.d", "mesh-power.conf")
withDropIn := parse(t, fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"conf","type":"file","path":%q,"content":"second\n","mode":"0644"},
{"id":"svc","type":"service","unit":"thing.service","state":"running","restart-on":["conf"]},
{"id":"units","type":"directory","path":%q,"mode":"0755"},
{"id":"drop-ins","type":"directory","path":%q,"mode":"0755"},
{"id":"bound","type":"file","path":%q,"content":"[Unit]\nJobTimeoutSec=10min\n","mode":"0644"},
{"id":"bound-too","type":"file","path":%q,"content":"[Unit]\nJobTimeoutSec=10min\n","mode":"0644"}
]}`, path, units, filepath.Dir(dropIn), dropIn, filepath.Join(filepath.Dir(dropIn), "other.conf")))
for _, row := range []struct {
name string
reloads int
}{{"a drop-in written", 1}, {"the same drop-in again", 0}} {
commands = nil
if _, state, err = Apply(context.Background(), archHost(t), withDropIn, state,
store.OriginCarried, run, nil, nil); err != nil {
t.Fatal(err)
}
reloads := 0
for _, c := range commands {
if strings.Contains(c, "daemon-reload") {
reloads++
}
if strings.Contains(c, "stop thing.service") {
t.Errorf("%s: a drop-in no service reflects restarted one: %v", row.name, commands)
}
}
if reloads != row.reloads {
t.Errorf("%s: %d reloads, want %d; commands were %v", row.name, reloads, row.reloads, commands)
}
}
// A drop-in whose module is skipped behind its step, or held as found, comes back so on every apply: it moved
// nothing, and reloads nothing.
for _, action := range []string{"skipped", "held"} {
if at := unitFileMoved([]Outcome{{ID: "bound", Type: "file", Target: dropIn, Action: action}}); at != "" {
t.Errorf("a drop-in %s reloads the service manager (%s)", action, at)
}
}
}
}
+4
View File
@@ -290,3 +290,7 @@ func unpack(body []byte, into string) (int, error) {
}
}
}
// Fetch is an archive at source as the host fetches every archive it unpacks, at most maxArchive bytes: what the node's
// own composer reads a moved module's templates archive with (issue 1126). Its digest is the caller's to check.
func Fetch(ctx context.Context, source string) ([]byte, error) { return fetch(ctx, source) }
+27 -3
View File
@@ -45,6 +45,30 @@ func WriteFilled(
log func(string),
keep Keep,
windows *Windows,
) (report Report, _ store.State, _ error) {
return WriteFilledWith(ctx, sys, d, known, path, content, run, log, keep, windows, Filled{})
}
// Filled is what the node-engine did to a filled file's content before its write.
type Filled struct {
// Secret says the node-engine filled a value into the content, a provision's `${secret:}` (novox/hq ADR 0570
// clause 6): the file is then written 0600 when the declaration gives no mode, as a file carrying a secret is.
Secret bool
}
// WriteFilledWith is WriteFilled for content the node-engine filled before the write (Filled).
func WriteFilledWith(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
known store.State,
path string,
content []byte,
run Runner,
log func(string),
keep Keep,
windows *Windows,
filled Filled,
) (report Report, _ store.State, _ error) {
if log == nil {
log = func(string) {}
@@ -65,7 +89,7 @@ func WriteFilled(
return report, known, &Error{Resource: path, Err: fmt.Errorf("%d resources declare %s, and a filled file "+
"is written for exactly one, so nothing was written", len(declared), path)}
}
file := filledAs(declared[0], content)
file := filledAs(declared[0], content, filled.Secret)
if held, isHeld := known.HeldAt(file.ID); isHeld {
return report, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is held as found for a module not "+
"yet taken (%s), so nothing was written", path, held.Why)}
@@ -163,10 +187,10 @@ func WriteFilled(
}
// filledAs is the declared file with the filled content in place of the declaration's.
func filledAs(declared *declaration.File, content []byte) *declaration.File {
func filledAs(declared *declaration.File, content []byte, secret bool) *declaration.File {
f := *declared
f.Content, f.Sealed, f.VaultSealed, f.Bytes, f.Secrets = string(content), "", nil, "", nil
if f.Mode == "" && (declared.Secret() || len(declared.Secrets) > 0) {
if f.Mode == "" && (secret || declared.Secret() || len(declared.Secrets) > 0) {
// Written as the declaration would have written it: a file carrying a credential is not world-readable.
f.Mode = "0600"
}
+101 -25
View File
@@ -3,6 +3,7 @@ package apply
import (
"context"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
@@ -116,32 +117,107 @@ func TestARecordWithoutAnOwnerBelongsToTheModuleItsIdNames(t *testing.T) {
// leaves it as it is, whatever its type, until the module's own apply takes it (issue 1945, live on 2026-10-25 at 02:32:
// shanks and g14 lost systemd-resolved's guard, its resolver files and sudo's operator file, ace qbittorrent's server,
// because the node's own apply removed every record no longer in it, the others holding the very same resources or not).
//
// And a resource the node's own composition carried for a module a want still places here (a folded module of the old
// form, held) is never removed by the node's own apply when a composition leaves it out: the apply is refused whole and
// says why (issue 774, live on 2026-10-29 at 07:54 UTC on the bus node: the node's own composition, composed without what
// it ran once the kept declaration was read no more, removed the bus server, the proxy and three held modules). Only its
// want's removal lets it go.
func TestAResourceThatMovedToItsModulesCompositionIsLeftByTheNodesOwnApply(t *testing.T) {
dir := t.TempDir()
fact := filepath.Join(dir, "suffix")
gone := filepath.Join(dir, "gone.conf")
own := declare(t, `{"id":"systemd-resolved.fact-suffix","type":"file","path":"`+fact+`","content":"mesh\n"},
{"id":"old.gone","type":"file","path":"`+gone+`","content":"g\n"}`)
_, state, err := ApplyModule(context.Background(), archHost(t), own, Scope{Module: store.NodeModule},
state0(), store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
// systemd-resolved now has a composition of its own, not applied yet; old.gone is no module's any more.
moved := declare(t, `{"id":"systemd-resolved.fact-suffix","type":"file","path":"`+fact+`","content":"mesh\n","module":"systemd-resolved"}`)
report, state, err := ApplyModule(context.Background(), archHost(t), &declaration.Declaration{},
Scope{Module: store.NodeModule, Others: moved.Resources}, state, store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !exists(fact) {
t.Fatalf("the moved resource was removed: %+v", outcomeOf(report, "systemd-resolved.fact-suffix"))
}
if exists(gone) {
t.Fatal("a resource no composition holds any more was kept")
}
if _, still := state.Find("systemd-resolved.fact-suffix"); !still {
t.Fatal("the moved resource's record was dropped, so its module's apply would not know it")
for _, tc := range []struct {
name string
id string // the resource the node's own composition carried, and no longer does
moved bool // its module's own composition now declares it (Others)
// dropped: its module has a composition of its own here that no longer declares it (Others holds another of its
// resources); the node's own apply leaves it, and the module's own apply removes it.
dropped bool
placed bool // a want still places its module here
refused bool // the node's own apply is refused whole
stays bool // the resource is still on the machine and recorded
// wants names the modules whose want says removed (Scope.RemovedByWant); nil leaves the guard to Placed alone.
wants []string
}{
{name: "moved to its module's own composition", id: "systemd-resolved.fact-suffix", moved: true, placed: true, stays: true},
{name: "a held folded module, in no composition", id: "nats.server-conf", placed: true, refused: true, stays: true},
{name: "its module's want removed", id: "nats.server-conf"},
// Issue 774, review of the re-land, round 1: a module folded in the node's own has no want, so Placed does not name
// it; in the node's own apply only a module a want removed loses its resources.
{name: "a folded module with no want", id: "nats.server-conf", wants: []string{}, refused: true, stays: true},
{name: "a folded module a want removed", id: "nats.server-conf", wants: []string{"nats", "old"}},
// 2026-10-30 19:42 on the bus node: restic's manifest replaced its process measure by awake. Its record was the
// node's, carried when restic was folded, so restic's own apply never saw it and the node's own was refused
// whole every two minutes, applying nothing of the node's own.
{name: "its module's own composition dropped it", id: "restic.measure", dropped: true, placed: true,
wants: []string{"old"}, stays: true},
} {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
fact := filepath.Join(dir, "carried")
gone := filepath.Join(dir, "gone.conf")
own := declare(t, `{"id":"`+tc.id+`","type":"file","path":"`+fact+`","content":"mesh\n"},
{"id":"old.gone","type":"file","path":"`+gone+`","content":"g\n"}`)
_, state, err := ApplyModule(context.Background(), archHost(t), own, Scope{Module: store.NodeModule},
state0(), store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
module, _ := declaration.ModuleGuess(tc.id)
scope := Scope{Module: store.NodeModule}
if tc.moved {
moved := declare(t, `{"id":"`+tc.id+`","type":"file","path":"`+fact+`","content":"mesh\n","module":"`+module+`"}`)
scope.Others = moved.Resources
}
if tc.dropped {
later := declare(t, `{"id":"`+module+`.awake","type":"file","path":"`+filepath.Join(dir, "awake")+
`","content":"a\n","module":"`+module+`"}`)
scope.Others = later.Resources
}
if tc.placed {
scope.Placed = map[string]bool{module: true}
}
if tc.wants != nil {
scope.RemovedByWant = map[string]bool{}
for _, m := range tc.wants {
scope.RemovedByWant[m] = true
}
}
report, after, err := ApplyModule(context.Background(), archHost(t), &declaration.Declaration{}, scope, state,
store.OriginDeclared, noServices, nil, nil, nil, nil)
if tc.refused != (err != nil) {
t.Fatalf("refused %v, want %v: %v", err != nil, tc.refused, err)
}
if tc.refused {
if !strings.Contains(err.Error(), tc.id+" of "+module) {
t.Fatalf("the refusal does not name what it kept: %v", err)
}
after = state
}
if exists(fact) != tc.stays {
t.Fatalf("the carried resource exists %v, want %v: %+v", exists(fact), tc.stays, outcomeOf(report, tc.id))
}
if _, still := after.Find(tc.id); still != tc.stays {
t.Fatalf("the carried resource's record kept %v, want %v", still, tc.stays)
}
if exists(gone) == !tc.refused {
t.Fatalf("a resource no composition holds any more exists %v, with the apply refused %v", exists(gone), tc.refused)
}
if tc.dropped {
if o := outcomeOf(report, tc.id); o.Action != "kept" {
t.Fatalf("the node's own apply did not say it left it: %+v", o)
}
later := declare(t, `{"id":"`+module+`.awake","type":"file","path":"`+filepath.Join(dir, "awake")+
`","content":"a\n","module":"`+module+`"}`)
_, last, err := ApplyModule(context.Background(), archHost(t), later, Scope{Module: module}, after,
store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if _, still := last.Find(tc.id); still || exists(fact) {
t.Fatalf("the module's own apply left what it no longer declares: recorded %v, on the machine %v",
still, exists(fact))
}
}
})
}
}
+111
View File
@@ -0,0 +1,111 @@
package apply
import (
"fmt"
"os"
"path/filepath"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// ObjectKeep is the node-engine's own keep of the content of files given by digest (novox/hq ADR 0486 §1, issue 924): one
// file per object, named as the object is, beside the node's state. **What a node once had it never asks the bus for
// again**, so neither the controller's sweep of the bucket (an object is removed a day after no composition of the
// controller names it) nor a bus that does not answer can refuse a composition the node already ran. The node's account
// may only read its bucket (ADR 0486 §2), so the keep is the node's, not the bucket's: nothing is written back.
//
// An object unused for KeptObjectsFor is let go when another is kept, so the keep holds what the node applied lately and
// not every export it ever ran. Every failure to keep is silent and costs only a fetch: the keep is a copy.
type ObjectKeep struct {
Dir string
// Now is the clock; nil is time.Now.
Now func() time.Time
}
// KeptObjectsFor is how long an object of the keep is held after it was last used.
const KeptObjectsFor = 14 * 24 * time.Hour
// KeepObjectsIn is the keep at dir; an empty dir keeps nothing.
func KeepObjectsIn(dir string) *ObjectKeep {
if dir == "" {
return nil
}
return &ObjectKeep{Dir: dir}
}
func (k *ObjectKeep) now() time.Time {
if k.Now != nil {
return k.Now()
}
return time.Now()
}
// get is the kept content of the object name when it is what f names, and marks it used.
func (k *ObjectKeep) get(name string, f *declaration.File) ([]byte, bool) {
if k == nil {
return nil, false
}
path := filepath.Join(k.Dir, name)
body, err := os.ReadFile(path)
if err != nil || int64(len(body)) != f.Size || f.Verify(body) != nil {
return nil, false
}
now := k.now()
_ = os.Chtimes(path, now, now)
return body, true
}
// put keeps body under name, unless it is kept already (then it is marked used), and lets go of what was unused too long.
func (k *ObjectKeep) put(name string, body []byte) {
if k == nil {
return
}
path := filepath.Join(k.Dir, name)
now := k.now()
if info, err := os.Stat(path); err == nil && info.Size() == int64(len(body)) {
_ = os.Chtimes(path, now, now)
return
}
if err := os.MkdirAll(k.Dir, 0o700); err != nil {
return
}
tmp, err := os.CreateTemp(k.Dir, ".put-*")
if err != nil {
return
}
_, werr := tmp.Write(body)
cerr := tmp.Close()
if werr != nil || cerr != nil || os.Chmod(tmp.Name(), 0o600) != nil || os.Rename(tmp.Name(), path) != nil {
_ = os.Remove(tmp.Name())
return
}
_ = os.Chtimes(path, now, now)
k.prune(now)
}
// prune lets go of every object unused for KeptObjectsFor.
func (k *ObjectKeep) prune(now time.Time) {
entries, err := os.ReadDir(k.Dir)
if err != nil {
return
}
for _, e := range entries {
info, err := e.Info()
if err != nil || !info.Mode().IsRegular() {
continue
}
if now.Sub(info.ModTime()) >= KeptObjectsFor {
_ = os.Remove(filepath.Join(k.Dir, e.Name()))
}
}
}
// never says, in a failed fetch's error before its cause (a transient one's detail keeps the tail), that the node's own
// keep did not hold the object either.
func (k *ObjectKeep) never() string {
if k == nil {
return ""
}
return fmt.Sprintf(" (this node's own keep of objects, %s, never held it)", k.Dir)
}
+91 -1
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"os"
"path/filepath"
"runtime"
"strings"
"time"
@@ -94,6 +95,14 @@ func applyProcess(ctx context.Context, r *declaration.Process, run Runner,
if r.Schedule != "" {
unit = r.Name + ".timer"
}
// The yielding slice is checked even then: a slice file removed by hand would leave the build unbound.
if wrote, err := placeYieldSlice(r); err != nil {
return out, err
} else if wrote {
if _, err := run(ctx, "systemctl", "daemon-reload"); err != nil {
return out, err
}
}
if active, err := run(ctx, "systemctl", "is-active", "--quiet", unit); err == nil {
_ = active
return out, nil
@@ -121,7 +130,12 @@ func applyProcess(ctx context.Context, r *declaration.Process, run Runner,
}
// Replaced rather than merged: the bundle is the whole of what it runs, and files left from a
// previous version would be loaded by a runtime that walks a directory.
if err := os.RemoveAll(at); err != nil {
// **Retired, never deleted under its process** (novox/hq issue 734): the old process runs until
// the restart below, from this directory and in it. Deleted here, every command it started
// before the restart failed with "getwd: no such file or directory" — a builder's toolchain probe
// refused three checks on 2026-10-26. Renamed aside, its process keeps a directory that exists,
// and the build is deleted once nothing runs from it, as a judged build's is.
if err := witness.Retire(daemonRoot, r.Name, at, time.Now()); err != nil {
return out, err
}
} else {
@@ -189,6 +203,9 @@ func applyProcess(ctx context.Context, r *declaration.Process, run Runner,
if err := os.WriteFile(unit, []byte(unitFor(r)), 0o644); err != nil {
return out, err
}
if _, err := placeYieldSlice(r); err != nil {
return out, err
}
if _, err := run(ctx, "systemctl", "daemon-reload"); err != nil {
return out, err
}
@@ -277,6 +294,7 @@ func unitFor(r *declaration.Process) string {
if r.User != "" {
fmt.Fprintf(&b, "User=%s\n", r.User)
}
writeYields(&b, r.Yields)
fmt.Fprintf(&b, "ExecStart=%s\n", strings.Join(runFrom(r), " "))
if r.Schedule != "" || r.RunOnce {
// Started by its timer, or once by the host, and expected to finish. Restarting it would have it run
@@ -293,6 +311,78 @@ func unitFor(r *declaration.Process) string {
return b.String()
}
// machineCPUs is how many processors this machine has, for a share of them in a quota. A variable only so a test can
// fix it.
var machineCPUs = runtime.NumCPU
// writeYields writes how a process gives way to everything else on its machine (novox/hq ADR 0553, issue 1806): a
// build seat's holder, whose builds must never slow the operator's desktop.
//
// **Into the node's one yielding slice**, where the CPU weight and the quota are (yieldSlice): every yielding process
// of the machine shares them, so three build modules take one share together, never one each. **Per process**, what a
// slice cannot say: Nice=19 for every process it starts, and the lowest best-effort IO level. The IO level binds only
// where the disk's scheduler honours it (bfq); under `none`, the usual one for NVMe, it does nothing. Not the idle IO
// class: a disk that is never idle would starve a build for good.
//
// The share is written into the unit as a comment, so a changed share changes the unit, the daemon's identity, and the
// slice is written again.
func writeYields(b *strings.Builder, y *declaration.Yields) {
if y == nil {
return
}
fmt.Fprintf(b, "# Yields: %s, %d%% of the machine shared in %s\n", y.Priority, y.CPUShare, declaration.YieldSlice)
fmt.Fprintf(b, "Slice=%s\n", declaration.YieldSlice)
if y.Priority == declaration.YieldIdle {
b.WriteString("Nice=19\nIOSchedulingClass=best-effort\nIOSchedulingPriority=7\n")
}
}
// yieldSlice is the node's one yielding slice (declaration.YieldSlice) for a process that yields: idle CPU weight and
// IO weight 1 for an idle priority, and CPUQuota for its share of the processors this machine has (systemd counts a
// quota in percent of one processor, so half of 24 is 1200%), at least one processor. At the top of the hierarchy, so
// its weight and quota bind against the operator's user slice and the system's.
//
// The IO weight binds only where the disk's scheduler honours it (bfq, or the io.cost controller); under `none` it does
// nothing, and the CPU bound is what holds.
//
// Every yielding process of a machine carries the node's one share (the composer's, ADR 0553 §3), so whichever writes
// it writes the same file. Left in place when nothing yields any more: an empty slice binds nothing.
func yieldSlice(y *declaration.Yields) string {
var b strings.Builder
b.WriteString("# Generated by the mesh. Do not edit — this file is replaced whenever a yielding\n")
b.WriteString("# process is declared again (novox/hq ADR 0553).\n")
b.WriteString("[Unit]\nDescription=The mesh's yielding processes: builds, at idle priority under one share\n\n")
b.WriteString("[Slice]\n")
if y.Priority == declaration.YieldIdle {
b.WriteString("CPUWeight=idle\nIOWeight=1\n")
}
if y.CPUShare > 0 {
quota := y.CPUShare * machineCPUs()
if quota < 100 {
quota = 100
}
fmt.Fprintf(&b, "CPUQuota=%d%%\n", quota)
}
return b.String()
}
// placeYieldSlice writes the yielding slice for a process that yields, when what is there differs, and says whether it
// wrote: the caller reloads the service manager before it starts anything.
func placeYieldSlice(r *declaration.Process) (bool, error) {
if r.Yields == nil {
return false, nil
}
path := filepath.Join(unitDir, declaration.YieldSlice)
want := yieldSlice(r.Yields)
if have, err := os.ReadFile(path); err == nil && string(have) == want {
return false, nil
}
if err := os.WriteFile(path, []byte(want), 0o644); err != nil {
return false, fmt.Errorf("the yielding slice %s: %w", path, err)
}
return true, nil
}
// timerFor is the cadence a scheduled process runs on.
//
// **The mesh's cron expression, handed to the machine's own timer.** The host already parses and
+63 -20
View File
@@ -21,10 +21,16 @@ func aProcess() *declaration.Process {
}
}
// The unit the mesh writes says what it runs, where, and that it comes back.
// The unit the mesh writes says what it runs, where, and that it comes back — and that it is the
// mesh's: **generated whole and saying so**, because an edit that survives until the next declaration
// and then vanishes is worse than one that is refused. The long-running mode stays up and is not a step.
func TestTheUnitRunsWhatTheDaemonSaid(t *testing.T) {
unit := unitFor(aProcess())
if !strings.HasPrefix(unit, "#") {
t.Fatalf("the unit does not open saying it is generated:\n%s", unit)
}
for _, want := range []string{
"Do not edit",
"ExecStart=node index.js",
"WorkingDirectory=/var/lib/mesh/daemons/greeter",
"Restart=always",
@@ -34,14 +40,8 @@ func TestTheUnitRunsWhatTheDaemonSaid(t *testing.T) {
t.Fatalf("the unit does not say %q:\n%s", want, unit)
}
}
}
// **Generated whole and saying so.** Every managed file on a machine carries this, because an edit
// that survives until the next declaration and then vanishes is worse than one that is refused.
func TestTheUnitSaysItIsTheMeshs(t *testing.T) {
unit := unitFor(aProcess())
if !strings.HasPrefix(unit, "#") || !strings.Contains(unit, "Do not edit") {
t.Fatalf("the unit does not say it is generated:\n%s", unit)
if strings.Contains(unit, "Type=oneshot") {
t.Fatalf("a process that should stay up is declared a step:\n%s", unit)
}
}
@@ -83,6 +83,60 @@ func TestAProcessRunsAsWhoItSaid(t *testing.T) {
if strings.Contains(unitFor(aProcess()), "User=") {
t.Fatalf("a process that named no user had one written for it:\n%s", unitFor(aProcess()))
}
// A process that yields (ADR 0553) runs in the node's one yielding slice at nice 19; the slice holds the idle weight
// and the quota for its share of the machine's processors; one that does not yield has none of it.
defer func(n func() int) { machineCPUs = n }(machineCPUs)
for _, c := range []struct {
cpus int
y *declaration.Yields
unit, slice, not []string
}{
{24, &declaration.Yields{Priority: declaration.YieldIdle, CPUShare: 50},
[]string{"Slice=meshyield.slice\n", "Nice=19\n", "IOSchedulingPriority=7\n", "50%"},
[]string{"CPUWeight=idle\n", "IOWeight=1\n", "CPUQuota=1200%\n"}, []string{"CPUQuota=", "CPUWeight="}},
{1, &declaration.Yields{CPUShare: 25}, []string{"Slice=meshyield.slice\n"}, []string{"CPUQuota=100%\n"},
[]string{"Nice="}},
{24, nil, nil, nil, []string{"Slice=", "Nice=", "CPUQuota="}},
} {
machineCPUs = func() int { return c.cpus }
p := aProcess()
p.Yields = c.y
unit := unitFor(p)
for _, w := range c.unit {
if !strings.Contains(unit, w) {
t.Errorf("yields %+v: the unit does not say %q:\n%s", c.y, w, unit)
}
}
for _, n := range c.not {
if strings.Contains(unit, n) {
t.Errorf("yields %+v: the unit says %q:\n%s", c.y, n, unit)
}
}
if c.y == nil {
continue
}
slice := yieldSlice(c.y)
for _, w := range c.slice {
if !strings.Contains(slice, w) {
t.Errorf("yields %+v on %d processors: the slice does not say %q:\n%s", c.y, c.cpus, w, slice)
}
}
}
// The slice is placed beside the units, once, and again only when it differs.
was := unitDir
unitDir = t.TempDir()
defer func() { unitDir = was }()
p := aProcess()
p.Yields = &declaration.Yields{Priority: declaration.YieldIdle, CPUShare: 50}
for i, want := range []bool{true, false} {
if wrote, err := placeYieldSlice(p); err != nil || wrote != want {
t.Errorf("placing %d: wrote %v (%v), want %v", i, wrote, err, want)
}
}
if b, err := os.ReadFile(filepath.Join(unitDir, "meshyield.slice")); err != nil || !strings.Contains(string(b), "[Slice]") {
t.Errorf("the slice file: %q %v", b, err)
}
}
// **Three modes, one kind.** A scheduled process is a timer plus a unit that finishes, not a unit
@@ -124,17 +178,6 @@ func TestACronBecomesATimersCalendar(t *testing.T) {
}
}
// And the long-running mode is unchanged by any of it: it stays up and comes back.
func TestAProcessThatStaysUpIsStillRestartedWhenItExits(t *testing.T) {
unit := unitFor(aProcess())
if !strings.Contains(unit, "Restart=always") {
t.Fatalf("a process that should stay up is not restarted when it exits:\n%s", unit)
}
if strings.Contains(unit, "Type=oneshot") {
t.Fatalf("a process that should stay up is declared a step:\n%s", unit)
}
}
// **A unit file reinterprets a value in three ways nothing else does**, and a module's environment
// routinely contains all three — a generated password is arbitrary bytes.
func TestAnEnvironmentValueMeansWhatTheDeclarationSaid(t *testing.T) {
+89 -11
View File
@@ -3,7 +3,9 @@ package apply
import (
"context"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
@@ -70,36 +72,112 @@ func TestARolledBackBuildIsNotPlacedAgain(t *testing.T) {
t.Cleanup(func() { unitDir, daemonRoot = wasUnits, wasBundles })
run := func(ctx context.Context, name string, args ...string) (string, error) { return "", nil }
oldBody, oldDigest := anArchive(t, map[string]string{"mesh-controller": "old\n"})
newBody, newDigest := anArchive(t, map[string]string{"mesh-controller": "new\n"})
controller := func(body []byte, digest, env string) string {
return `{"id":"mesh-controller.controller","type":"process","name":"mesh-controller","source":"` + serving(t, body) +
`","digest":"` + digest + `","run":["./mesh-controller","serve"],"env":{"X":"` + env + `"}}`
oldBody, oldDigest := anArchive(t, map[string]string{"node-tools": "old\n"})
newBody, newDigest := anArchive(t, map[string]string{"node-tools": "new\n"})
runtime := func(body []byte, digest, env string) string {
return `{"id":"node-tools.runtime","type":"process","name":"node-tools","source":"` + serving(t, body) +
`","digest":"` + digest + `","run":["./node-tools"],"env":{"X":"` + env + `"}}`
}
_, state, err := Apply(context.Background(), archHost(t), declare(t, controller(oldBody, oldDigest, "1")), store.State{},
_, state, err := Apply(context.Background(), archHost(t), declare(t, runtime(oldBody, oldDigest, "1")), store.State{},
store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
_, state, err = Apply(context.Background(), archHost(t), declare(t, controller(newBody, newDigest, "1")), state,
_, state, err = Apply(context.Background(), archHost(t), declare(t, runtime(newBody, newDigest, "1")), state,
store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if _, err := witness.Restore(context.Background(), bundles, "mesh-controller", "never took the lease", witness.Runner(run),
if _, err := witness.Restore(context.Background(), bundles, "node-tools", "never answered", witness.Runner(run),
time.Now()); err != nil {
t.Fatal(err)
}
// The mesh still declares the new build, and its unit changes: the process is re-placed.
report, _, err := Apply(context.Background(), archHost(t), declare(t, controller(newBody, newDigest, "2")), state,
report, _, err := Apply(context.Background(), archHost(t), declare(t, runtime(newBody, newDigest, "2")), state,
store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if got, _ := os.ReadFile(filepath.Join(bundles, "mesh-controller", "mesh-controller")); string(got) != "old\n" {
if got, _ := os.ReadFile(filepath.Join(bundles, "node-tools", "node-tools")); string(got) != "old\n" {
t.Fatalf("the rolled-back build was placed again: %q", got)
}
if o := outcomeOf(report, "mesh-controller.controller"); !strings.Contains(o.Detail, "rolled back on this machine") {
if o := outcomeOf(report, "node-tools.runtime"); !strings.Contains(o.Detail, "rolled back on this machine") {
t.Fatalf("the apply does not say why it kept the build before: %+v", o)
}
}
// **A process nobody judges keeps the directory it works in until its restart** (novox/hq issue 734).
// On 2026-10-26 three checks on one builder were refused in the second a new build of it arrived: the
// applier deleted the running build's directory before the restart, and the `go version` the old
// process started in it failed with "getwd: no such file or directory". Here a process works in the
// build's directory without running from it, as that `go version` did: the apply of a new build leaves
// its directory in place, and the retired build goes once nothing works in it.
func TestAProcessNobodyJudgesKeepsItsDirectoryUntilItsRestart(t *testing.T) {
if _, err := os.Stat("/proc/self/cwd"); err != nil {
t.Skip("which process works in a build is read from /proc")
}
units, bundles := t.TempDir(), t.TempDir()
wasUnits, wasBundles := unitDir, daemonRoot
unitDir, daemonRoot = units, bundles
t.Cleanup(func() { unitDir, daemonRoot = wasUnits, wasBundles })
run := func(ctx context.Context, name string, args ...string) (string, error) { return "", nil }
oldBody, oldDigest := anArchive(t, map[string]string{"mesh-build-go": "old\n"})
newBody, newDigest := anArchive(t, map[string]string{"mesh-build-go": "new\n"})
builder := func(body []byte, digest string) string {
return `{"id":"mesh-build-go.builder","type":"process","name":"mesh-build-go","source":"` + serving(t, body) +
`","digest":"` + digest + `","run":["./mesh-build-go"]}`
}
_, state, err := Apply(context.Background(), archHost(t), declare(t, builder(oldBody, oldDigest)), store.State{},
store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
sleep, err := exec.LookPath("sleep")
if err != nil {
t.Skip("no sleep to run")
}
working := exec.Command(sleep, "60")
working.Dir = filepath.Join(bundles, "mesh-build-go")
if err := working.Start(); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = working.Process.Kill(); _ = working.Wait() })
cwd := filepath.Join("/proc", strconv.Itoa(working.Process.Pid), "cwd")
for deadline := time.Now().Add(5 * time.Second); ; {
if at, err := os.Readlink(cwd); err == nil && at == working.Dir {
break
}
if time.Now().After(deadline) {
t.Fatalf("PID %d does not work in %s within 5s", working.Process.Pid, working.Dir)
}
time.Sleep(10 * time.Millisecond)
}
if _, _, err := Apply(context.Background(), archHost(t), declare(t, builder(newBody, newDigest)), state,
store.OriginDeclared, run, nil, nil); err != nil {
t.Fatal(err)
}
if got, _ := os.ReadFile(filepath.Join(bundles, "mesh-build-go", "mesh-build-go")); string(got) != "new\n" {
t.Fatalf("the new build is not where the unit runs it from: %q", got)
}
at, err := os.Readlink(cwd)
if err != nil || strings.HasSuffix(at, " (deleted)") {
t.Fatalf("the directory a running process works in was deleted before its restart: %q (%v)", at, err)
}
if got, _ := os.ReadFile(filepath.Join(at, "mesh-build-go")); string(got) != "old\n" {
t.Fatalf("the process's directory no longer holds its own build: %q", got)
}
if len(witness.Trials(bundles)) != 0 {
t.Fatal("a process nobody judges went on trial")
}
// Its process gone, the next look deletes the retired build and keeps nothing about the process.
_ = working.Process.Kill()
_ = working.Wait()
witness.SweepAll(bundles)
if _, err := os.Stat(witness.Dir(bundles, "mesh-build-go")); !os.IsNotExist(err) {
t.Fatalf("a retired build nothing works in was kept: %v", err)
}
}
+24 -8
View File
@@ -27,13 +27,23 @@ type FetchObject func(ctx context.Context, name string, limit int64) ([]byte, er
// unchanged declaration, and a node that is asleep and has no bus, apply as they always did. One object is fetched
// once however many files name its digest.
func ResolveFileReferences(ctx context.Context, d *declaration.Declaration, fetch FetchObject) error {
return ResolveFileReferencesKept(ctx, d, fetch, nil)
}
// ResolveFileReferencesKept is ResolveFileReferences with the node's own keep of objects (ObjectKeep): every content had
// (a file here that verifies, or an object of the bus) is kept, and a kept one is used before the bus is asked. **A node
// depends on its bucket only for content it never had** (issue 924, 2026-10-29: novox's own composition named the
// export mesh-vault runs by a digest whose object the controller's sweep had removed, so the vault's apply failed while
// it ran; the bucket's objects are kept only while a composition of the controller names them, and a node composing its
// own names objects the controller no longer knows of). A nil keep keeps nothing.
func ResolveFileReferencesKept(ctx context.Context, d *declaration.Declaration, fetch FetchObject, keep *ObjectKeep) error {
fetched := map[string][]byte{}
for i, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || !f.IsReference() {
continue
}
body, err := referencedContent(ctx, f, fetch, fetched)
body, err := referencedContent(ctx, f, fetch, fetched, keep)
if err != nil {
return err
}
@@ -50,31 +60,37 @@ func ResolveFileReferences(ctx context.Context, d *declaration.Declaration, fetc
}
func referencedContent(ctx context.Context, f *declaration.File, fetch FetchObject,
fetched map[string][]byte) ([]byte, error) {
fetched map[string][]byte, keep *ObjectKeep) ([]byte, error) {
if body, ok := fetched[f.Digest]; ok {
return body, nil
}
if body, ok := alreadyThere(f); ok {
fetched[f.Digest] = body
return body, nil
}
name, err := declaration.ObjectName(f.Digest)
if err != nil {
return nil, fmt.Errorf("file %s (%s): %w", f.ID, f.Path, err)
}
if body, ok := alreadyThere(f); ok {
fetched[f.Digest] = body
keep.put(name, body)
return body, nil
}
if body, ok := keep.get(name, f); ok {
fetched[f.Digest] = body
return body, nil
}
if fetch == nil {
return nil, fmt.Errorf("file %s (%s) is given by digest %s and is not on this machine, and this node-engine "+
"has no link to the bus to fetch it from", f.ID, f.Path, f.Digest)
}
body, err := fetch(ctx, name, f.Size)
if err != nil {
return nil, fmt.Errorf("file %s (%s): fetching the object %s of this node's own object bucket: %w",
f.ID, f.Path, name, err)
return nil, fmt.Errorf("file %s (%s): fetching the object %s of this node's own object bucket%s: %w",
f.ID, f.Path, name, keep.never(), err)
}
if err := f.Verify(body); err != nil {
return nil, fmt.Errorf("%s (%s): %w; nothing was applied", f.ID, f.Path, err)
}
fetched[f.Digest] = body
keep.put(name, body)
return body, nil
}
+134
View File
@@ -0,0 +1,134 @@
// Package beside is where a test finds another repository of the mesh it reads: the catalogue's manifests,
// the node-engine's genesis template (novox/hq issue 432), the SDK's conformance fixtures (issue 449).
//
// Copied from mesh-controller internal/beside at commit 6c2a8d62 (R4b, issue 1379); the one change is that Dir
// lays out the captured copy of a repository once per process, not once per test (see laidOut). In mesh-sdk the
// catalogue's tests read mesh-catalog and mesh-host only; the captured copy holds just what they read.
//
// A test used to read the checkout beside this one, `../../../mesh-catalog`, so its verdict depended on
// whatever sat on the machine running it: a stale or dirty checkout failed it on a desktop, and where none
// was beside it skipped and said nothing. Now there are two inputs, both named, and no third:
//
// - In a merge check, the build seat clones each core repository beside the one checked (the catalogue
// at its main, the node-engine at the commit the mesh runs) and says where in MESH_CHECK_BESIDE. A
// test judges against those clones, so agreement with the other repository is checked where
// `mesh/repo-check` runs; a repository missing there fails the test, never skips it. Which clones a
// check gets is chosen from the inventory: mesh-catalog by the source of the `nats` module, mesh-host
// by the source of `mesh-host`, and mesh-sdk as the controller's sibling at the commit the controller's
// go.mod pins. Each repository's tests read these siblings, and the checker places exactly these
// (mesh-delivery's checker, besideRules, follows this list): mesh-controller's read mesh-catalog,
// mesh-host and mesh-sdk; mesh-host's read mesh-catalog, mesh-sdk and mesh-controller (its step-nine
// test reads the controller's own manifest, novox/hq issue 1295); mesh-sdk's read mesh-catalog and
// mesh-host; mesh-catalog's read mesh-sdk and mesh-controller; mesh-delivery's read mesh-sdk and
// mesh-catalog. In a mesh where either module has no source repository nothing is
// cloned, and these tests fail loudly with "not beside this check": a cause in the setup, not in the
// change. And in a delivery group that holds a mesh-catalog pull request, these tests read the
// catalogue's main, not the group's head.
// - Anywhere else, the test judges against the copy captured in this repository's testdata/beside, at the
// commit testdata/beside/CAPTURED names. Never against a developer's own checkout: to judge one, set
// MESH_CHECK_BESIDE to the directory holding it, as the check does.
//
// The captured manifests are named module.json.captured, and put back as module.json in a directory of
// the test's own: a module.json anywhere in this repository is a module of it to the forge's announcer and
// the planner, and a merge would build and register a hundred copies of the catalogue's.
package beside
import (
"io/fs"
"os"
"path/filepath"
"runtime"
"strings"
"sync"
"testing"
)
// Env is where a merge check says the repositories cloned beside the one checked are (internal/builder's
// EnvBeside, repeated here so a test does not import the builder).
const Env = "MESH_CHECK_BESIDE"
// CapturedSuffix is what a captured file's name carries that the repository's own does not.
const CapturedSuffix = ".captured"
// Dir is the named repository a test reads — the clone beside a merge check, or the captured copy — and
// says which in the test's log.
func Dir(t testing.TB, repository string) string {
t.Helper()
if root := os.Getenv(Env); root != "" {
dir := filepath.Join(root, repository)
if _, err := os.Stat(dir); err != nil {
t.Fatalf("%s is not beside this check in %s=%s, so its agreement with this repository cannot be "+
"judged here: %v", repository, Env, root, err)
}
t.Logf("judged against %s as cloned beside this check", dir)
return dir
}
from := filepath.Join(Captured(), repository)
if _, err := os.Stat(from); err != nil {
t.Fatalf("no captured copy of %s at %s: %v", repository, from, err)
}
dir, err := layOut(repository, from)
if err != nil {
t.Fatalf("the captured copy of %s could not be laid out: %v", repository, err)
}
t.Logf("judged against the copy of %s captured in testdata/beside (see CAPTURED); a merge check "+
"judges it against the repository's own clone", repository)
return dir
}
// laidOut is the captured copies already put back in place in this process, by repository. A copy is
// read-only to every test, so it is laid out once per process and not once per test: with a copy per test
// the catalogue's tests alone took 2.14 s, over mesh-sdk's unit bound of 2 s, and with one 0.81 s (measured
// 2026-10-20). The directories are in the temp directory, which the system clears; a process cannot remove
// them while another test may still read them.
var (
laidOutMu sync.Mutex
laidOut = map[string]string{}
)
func layOut(repository, from string) (string, error) {
laidOutMu.Lock()
defer laidOutMu.Unlock()
if dir, ok := laidOut[repository]; ok {
return dir, nil
}
root, err := os.MkdirTemp("", "beside-")
if err != nil {
return "", err
}
dir := filepath.Join(root, repository)
err = filepath.WalkDir(from, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, _ := filepath.Rel(from, path)
into := filepath.Join(dir, strings.TrimSuffix(rel, CapturedSuffix))
if d.IsDir() {
return os.MkdirAll(into, 0o755)
}
raw, err := os.ReadFile(path)
if err != nil {
return err
}
return os.WriteFile(into, raw, 0o644)
})
if err != nil {
os.RemoveAll(root)
return "", err
}
laidOut[repository] = dir
return dir, nil
}
// Catalogue is the catalogue's modules directory, as Dir finds the catalogue.
func Catalogue(t testing.TB) string {
t.Helper()
return filepath.Join(Dir(t, "mesh-catalog"), "modules")
}
// Captured is this repository's testdata/beside, found from this file rather than from the working
// directory, so a test in any package reaches it.
func Captured() string {
_, file, _, _ := runtime.Caller(0)
return filepath.Join(filepath.Dir(file), "..", "..", "testdata", "beside")
}
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
-32
View File
@@ -237,35 +237,3 @@ func TestAFileNothingOfTheControllerWritesIsStillRefused(t *testing.T) {
t.Fatalf("a file nothing writes was accepted, or refused without its name: %v", err)
}
}
// The controller's own manifest, as the mesh runs it, passes step 9 (issue 597): in a merge check the
// node-engine's check has the controller cloned beside it (MESH_CHECK_BESIDE), where it must be; else a
// checkout beside this one; else it skips aloud.
//
// The checker places the controller beside this repository (mesh-delivery's besideRules, since R4g, issue 1295), so a
// MESH_CHECK_BESIDE without a mesh-controller directory, or without its module.json, fails. Only a run with no
// MESH_CHECK_BESIDE and no checkout beside this one (a developer's clone alone) skips aloud.
func TestTheControllersOwnManifestPassesStepNine(t *testing.T) {
var manifest []byte
if root := os.Getenv("MESH_CHECK_BESIDE"); root != "" {
if info, err := os.Stat(filepath.Join(root, "mesh-controller")); err != nil || !info.IsDir() {
t.Fatalf("the controller is not beside this check in MESH_CHECK_BESIDE=%s: the checker places mesh-controller beside the node-engine's check (mesh-delivery besideRules, issue 1295), and this test judges the controller's own manifest", root)
}
raw, err := os.ReadFile(filepath.Join(root, "mesh-controller", "module.json"))
if err != nil {
t.Fatalf("the controller is not beside this check in MESH_CHECK_BESIDE=%s: %v", root, err)
}
manifest = raw
} else if raw, err := os.ReadFile(filepath.Join("..", "..", "..", "mesh-controller", "module.json")); err == nil {
manifest = raw
} else {
t.Skip("NOT JUDGED: no controller beside this checkout and no MESH_CHECK_BESIDE; the merge check judges it")
}
wanted, err := theStepNineSecretsOf(t, manifest)
if err != nil {
t.Fatalf("genesis's step 9 refuses the controller the mesh runs: %v", err)
}
if wanted["MESH_STORE_INVENTORY"] != "inventory" {
t.Errorf("step 9 delivers %v", wanted)
}
}
+560
View File
@@ -0,0 +1,560 @@
package compose
import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose/resolution"
"github.com/novox/mesh-host/internal/overlay"
)
// Compositions is one node's compositions (novox/hq ADR 0539 clauses 1 and 2): its declaration composed over in, as
// the controller composes it, split into each module's composition at the fingerprint fps gives it and the node's own
// (`<node>.node`), by key (delivery.ComposedKey). Each resource says its module, whatever the node reported it reads.
// A module fps gives no fingerprint for is not composed and is named in skipped, so a caller composing one module
// gets that one; but for a module the node runs inside its own composition (Node.InOwn, and the
// private network's always), which fps names at no fingerprint of the new form: its resources go into the node's own,
// their module taken off and their ids still naming it, and its filled paths with them, as the controller's composed
// writer folds them (issue 1126). Each module's Data says whether its manifest declares data. The compositions are unsigned
// (ADR 0593 §6) and not yet by digest (Composed.ByDigest).
func Compositions(in Input, node string, fps map[string]string, mode Mode) (out map[string]delivery.Composed, skipped []string, err error) {
out, skipped, _, err = compositions(in, node, fps, mode, false)
return out, skipped, err
}
// ErrStoodIn is a composition that holds a stand-in for a value the input does not carry (an own secret, a grant's
// sealed copy, a vault box, the certificate, the bus's user list) or a port it does not give: what a send would make,
// never what may be applied.
var ErrStoodIn = errors.New("compose: the composition stands in for what its input does not carry")
// BusUsersStandIn is what a composition says it stood in for when its input carries no bus user list.
const BusUsersStandIn = "the bus's user list"
// CompositionsApplied is Compositions for a node that applies what it composed itself (novox/hq ADR 0547 §2, issue
// 1126): a declaration that stands in for anything (Declared.StoodIn, Foreseen, ForeseenPorts) is refused with
// ErrStoodIn, naming each, so a stand-in is never applied as if it were the value; but for the bus's user list
// (BusUsersStandIn), which the bus node holds out of every apply of a composition it made itself.
func CompositionsApplied(in Input, node string, fps map[string]string) (map[string]delivery.Composed, []string, error) {
out, skipped, _, err := compositions(in, node, fps, Reading, true)
return out, skipped, err
}
// CompositionsShadow is CompositionsApplied for a composition that is compared and never applied (a node's answer to a
// placement candidate, novox/hq ADR 0559 §4): composed in the Foreseeing mode, so an own secret never issued (a module
// the candidate adds) is a named stand-in rather than an error, and what it stands in for is answered (stood, sorted,
// the bus's user list left out) rather than refused, so the caller judges each stand-in: a value not yet issued for a
// module the candidate adds, or one missing for a module the node runs. A module whose every value is held composes
// as CompositionsApplied composes it: the modes differ only in an own secret never issued.
func CompositionsShadow(in Input, node string, fps map[string]string) (out map[string]delivery.Composed, skipped, stood []string, err error) {
return compositions(in, node, fps, Foreseeing, false)
}
// CompositionsIn is the node's compositions in mode with what they stand in for answered (stood, sorted, the bus's user
// list left out) rather than refused: CompositionsShadow in Foreseeing, and in Reading what CompositionsApplied refuses
// before it is refused, so a caller that applies refuses each stand-in to the module it is for alone (issue 1126).
func CompositionsIn(in Input, node string, fps map[string]string, mode Mode) (out map[string]delivery.Composed, skipped, stood []string, err error) {
return compositions(in, node, fps, mode, false)
}
func compositions(in Input, node string, fps map[string]string, mode Mode, applied bool) (out map[string]delivery.Composed, skipped, stood []string, err error) {
reading := in
reading.Nodes = append([]Node(nil), in.Nodes...)
found := false
inOwn := map[string]bool{}
held := map[string]bool{}
for i := range reading.Nodes {
if reading.Nodes[i].Name == node {
reading.Nodes[i].ReadsModules = true
found = true
for m, ok := range reading.Nodes[i].InOwn {
inOwn[m] = ok
}
for m, ok := range reading.Nodes[i].Held {
held[m] = ok
}
}
}
// The private network's module is built by nobody: no fingerprint of the new form ever names it.
inOwn[overlay.Name] = true
folds := func(module string) bool {
return module != "" && inOwn[module] && !strings.HasPrefix(fps[module], delivery.SourcePrefix2)
}
if !found {
return nil, nil, nil, fmt.Errorf("compose: the mesh has no node %q", node)
}
// A folded module built by a build of the old form (Node.RunsOwn) is left to the node: composed from what runs of it
// (Carried), so what it provides and requires still shapes the rest and what the others read of it (its published
// ports, the paths it owns) is what runs, and its resources are then the ones the node runs.
carried := map[string]bool{}
var runsOwn *delivery.Composed
for i := range reading.Nodes {
n := &reading.Nodes[i]
if n.Name != node || n.RunsOwn == nil {
continue
}
runsOwn = n.RunsOwn
candidates := map[string]bool{}
for m := range inOwn {
candidates[m] = true
}
for m := range held {
candidates[m] = true
}
for m := range candidates {
// A held module is carried as it runs wherever the node's own runs it, folded or leaving the fold for a
// composition of its own (its want new, its names still unset): its stub is never what the node's own applies,
// and what the node's own applies never drops what runs of it (issue 1126, review of the apply; issue 1945).
heldHere := held[m] && RunsAnyOf(runsOwn, m)
if !folds(m) && !heldHere {
continue
}
man, ok := n.Kept[m]
if !ok {
man, ok = reading.Catalogue[m]
}
if !ok || (!NamesBuild(man) && !held[m]) {
continue
}
carried[m] = true
kept := make(map[string]catalogue.Manifest, len(n.Kept)+1)
for k, v := range n.Kept {
kept[k] = v
}
kept[m] = Carried(man, m, runsOwn)
n.Kept = kept
}
}
// A module composed as what runs of it contributes to the runtime what the runtime serves of it as it runs
// (Node.RunsTools); one of its own account is served by a runtime of its own, from its manifest as before.
for i := range reading.Nodes {
n := &reading.Nodes[i]
if n.Name != node || n.RunsTools == nil {
continue
}
kept := make(map[string]catalogue.Manifest, len(n.Kept))
for m, man := range n.Kept {
if man.Stubbed && man.RunsAs == "" {
man.Bundles = runningBundles(n.RunsTools[m], reading.Catalogue[m])
}
kept[m] = man
}
n.Kept = kept
}
isCarried := func(module, id string) bool {
if module == "" {
module, _ = declaration.ModuleGuess(id)
}
return carried[module]
}
d, problems, err := ComposeIn(reading, node, mode)
if err != nil {
return nil, nil, nil, err
}
if len(problems) > 0 {
return nil, nil, nil, fmt.Errorf("compose: %s's declaration is refused by the node-engine's validator: %v", node, problems)
}
for _, x := range d.StoodIn {
// The bus's user list is the bus node's own to follow, written alone and never from a composition the node
// made itself (ADR 0547 §7, §8): its stand-in is held out of the apply, not applied.
if x != BusUsersStandIn {
stood = append(stood, x)
}
}
stood = append(stood, d.Foreseen...)
for _, p := range d.ForeseenPorts {
stood = append(stood, "port "+p)
}
sort.Strings(stood)
stood = slices.Compact(stood) // an own secret is both stood in for and foreseen
if applied && len(stood) > 0 {
return nil, nil, nil, fmt.Errorf("%w on %s: %s", ErrStoodIn, node, strings.Join(stood, ", "))
}
body, err := d.Body()
if err != nil {
return nil, nil, nil, err
}
var envelope struct {
Resources []json.RawMessage `json:"resources"`
Adoption *declaration.Adoption `json:"adoption"`
Filled []string `json:"filled_files"`
Provisioner string `json:"provisioner_account"`
}
if err := json.Unmarshal(body, &envelope); err != nil {
return nil, nil, nil, err
}
var kept []json.RawMessage
keptPaths := map[string]bool{}
foldedPaths := map[string]bool{}
left := map[string]bool{}
for _, raw := range envelope.Resources {
var head struct {
ID string `json:"id"`
Module string `json:"module"`
Type string `json:"type"`
Path string `json:"path"`
}
if err := json.Unmarshal(raw, &head); err != nil {
return nil, nil, nil, err
}
if isCarried(head.Module, head.ID) {
continue // what the stub composed of it: the node's own carries what runs of it instead
}
if folds(head.Module) {
// Into the node's own, in its place among the node's resources, its module taken off (mesh-controller's
// compositionsOf): the node's own composition is then the controller's, byte for byte.
var fields map[string]json.RawMessage
if err := json.Unmarshal(raw, &fields); err != nil {
return nil, nil, nil, err
}
delete(fields, "module")
if raw, err = json.Marshal(fields); err != nil {
return nil, nil, nil, err
}
if head.Type == string(declaration.TypeFile) {
foldedPaths[head.Path] = true
}
kept = append(kept, raw)
continue
}
if _, ok := fps[head.Module]; head.Module != "" && !ok {
left[head.Module] = true
continue
}
if head.Type == string(declaration.TypeFile) {
keptPaths[head.Path] = true
}
kept = append(kept, raw)
}
// What runs of each carried module, in the node's own composition, each after the resource it runs after.
if len(carried) > 0 {
var err error
if kept, foldedPaths, err = carry(kept, foldedPaths, runsOwn, isCarried); err != nil {
return nil, nil, nil, err
}
for _, p := range runsOwn.Filled {
if foldedPaths[p] && !slices.Contains(envelope.Filled, p) {
envelope.Filled = append(envelope.Filled, p)
}
}
}
// A filled file of a module's own composition goes with that module (SplitComposed); one of a folded module goes with
// the node's own, after them, in the declaration's order; one of a module left out is dropped.
var filled, ownFilled []string
for _, p := range envelope.Filled {
switch {
case foldedPaths[p]:
ownFilled = append(ownFilled, p)
case keptPaths[p]:
filled = append(filled, p)
}
}
out, err = delivery.SplitComposed(node, kept, fps, filled,
delivery.NodeOwn{Adoption: envelope.Adoption, Provisioner: envelope.Provisioner})
if err != nil {
return nil, nil, nil, err
}
if len(ownFilled) > 0 {
own, err := delivery.ComposedNodeKey(node)
if err != nil {
return nil, nil, nil, err
}
c := out[own]
c.Filled = append(c.Filled, ownFilled...)
out[own] = c
}
for key, c := range out {
if held[c.Module] {
delete(out, key) // its stub's: the node keeps what it last applied of it (Node.Held)
continue
}
if m, ok := in.Catalogue[c.Module]; ok && c.Module != "" {
c.Data = len(m.DataItems()) > 0
out[key] = c
}
}
for m := range left {
skipped = append(skipped, m)
}
sort.Strings(skipped)
return out, skipped, stood, nil
}
// RunsAnyOf says runs (a composition as it runs) holds a resource of module, by its module or the module its id names.
func RunsAnyOf(runs *delivery.Composed, module string) bool {
if runs == nil {
return false
}
for _, raw := range runs.Resources {
var head struct {
ID string `json:"id"`
Module string `json:"module"`
}
if json.Unmarshal(raw, &head) != nil {
continue
}
owner := head.Module
if owner == "" {
owner, _ = declaration.ModuleGuess(head.ID)
}
if owner == module {
return true
}
}
return false
}
// ToolsRunning is, by module, the tools bundles that run on the node (Node.RunsTools): each bundle archive a module's
// resource unpacks at its BundlePath, with what the runtime's own process loads of it and the words it hands it
// (catalogue.RuntimeToolModules, catalogue.RuntimeToolEnv, the mesh's own words left out). A bundle the runtime loads
// nothing of has no Loads: the runtime dropped it, and the composition reads its loads again from the module's build
// (issue 1126, review: shanks had already applied the drop of systemd-resolved). Nil when running holds no runtime process.
func ToolsRunning(running []delivery.Composed) map[string][]catalogue.Bundle {
runtimeID := catalogue.RuntimeModule + "." + resolution.RuntimeProcessID()
type resource struct {
ID string `json:"id"`
Module string `json:"module"`
Type string `json:"type"`
Path string `json:"path"`
Source string `json:"source"`
Digest string `json:"digest"`
Env map[string]string `json:"env"`
}
var env map[string]string
out := map[string][]catalogue.Bundle{}
for _, c := range running {
for _, raw := range c.Resources {
var r resource
if json.Unmarshal(raw, &r) != nil {
continue
}
owner := r.Module
if owner == "" {
owner, _ = declaration.ModuleGuess(r.ID)
}
switch {
case r.Type == "process" && r.ID == runtimeID && owner == catalogue.RuntimeModule:
env = r.Env
case r.Type == "archive" && owner != "" && strings.HasPrefix(r.ID, owner+"."+resolution.BundleID("")):
name := strings.TrimPrefix(r.ID, owner+"."+resolution.BundleID(""))
if r.Path == resolution.BundlePath(owner, name) {
out[owner] = append(out[owner], catalogue.Bundle{Name: name, Source: catalogue.Recorded(r.Source), Digest: r.Digest})
}
}
}
}
if env == nil {
return nil
}
var words map[string]map[string]string
_ = json.Unmarshal([]byte(env[catalogue.RuntimeToolEnv]), &words)
for _, entry := range strings.Split(env[catalogue.RuntimeToolModules], ",") {
module, path, ok := strings.Cut(entry, "=")
if !ok || !strings.HasPrefix(path, catalogue.BundleRoot+"/"+module+"/") {
continue
}
name, load, ok := strings.Cut(strings.TrimPrefix(path, catalogue.BundleRoot+"/"+module+"/"), "/")
bundles := out[module]
i := slices.IndexFunc(bundles, func(b catalogue.Bundle) bool { return b.Name == name })
if !ok || i < 0 {
continue // no archive of it runs: nothing to load
}
var own map[string]string
for w, v := range words[module] {
if w == catalogue.FilesOwnedPaths || w == catalogue.ReadsSecretPlaces {
continue // the mesh's to say, composed again
}
if own == nil {
own = map[string]string{}
}
own[w] = v
}
bundles[i].Loads = append(bundles[i].Loads, load)
bundles[i].Env = own
}
for _, bundles := range out {
sort.Slice(bundles, func(i, j int) bool { return bundles[i].Name < bundles[j].Name })
}
return out
}
// runningBundles is the bundles of a module composed as what runs of it (Node.RunsTools): each that runs, with what the
// runtime loads of it and its words; one the runtime loads nothing of takes its loads and words from the module's build
// (man: its built bundles, else its build's artifacts, catalogue.BundleLoads), so a drop already applied is undone.
func runningBundles(runs []catalogue.Bundle, man catalogue.Manifest) []catalogue.Bundle {
var out []catalogue.Bundle
for _, b := range runs {
if len(b.Loads) == 0 {
for _, built := range man.Bundles {
if built.Name == b.Name {
b.Loads, b.Env, b.Launchers, b.Binary = built.Loads, built.Env, built.Launchers, built.Binary
}
}
if man.Build != nil {
run := resolution.RunByAResource(man)
for _, a := range man.Build.Artifacts {
if a.Kind == catalogue.ArtifactBundle && a.Name == b.Name {
b.Loads, b.Env, b.Binary = catalogue.BundleLoads(man, a, run), a.Env, catalogue.BinaryOf(a)
}
}
}
}
out = append(out, b)
}
return out
}
// NamesBuild says a manifest names what a build makes: its build, or a resource naming an artifact. A node cannot
// compose such a module folded into its own composition: the build of the old form it runs is held by the controller
// alone (issue 1126).
func NamesBuild(m catalogue.Manifest) bool {
if m.Build != nil {
return true
}
for _, r := range m.Resources {
if _, ok := r["artifact"]; ok {
return true
}
}
return false
}
// Stub is m with nothing to build and no resource but its directories: what it provides, requires, serves and grants
// still shape the rest of the composition, the places its own secrets, data and grants name (${dir:…}) still resolve,
// and nothing else of its own is composed (issue 1126: nats's ${dir:mesh-state} on novox). A module refused in a
// node's answer is composed so; a folded one the node runs is Carried.
func Stub(m catalogue.Manifest) catalogue.Manifest {
// The paths it owns stay its manifest's, as the controller tells the holder of node-files them (issue 1126, seed-8).
if m.OwnsPaths == nil {
m.OwnsPaths = resolution.DeclaredPaths(m)
}
m.Build = nil
// What it hands the provisioner is left out as it runs, not refused for want of a plan (catalogue.Manifest.Stubbed).
m.Stubbed = true
// Nor is a template its build kept for its own files handed: a stub carries none of them (issue 1126: novox's node
// refused "postgres: its build kept a template for consumers, which it does not declare" with postgres held). Its
// items stay, and the templates they render through: a held module's contributions to other modules' collections
// (seat contributions, shell blocks, seat data) still reach their collectors.
if m.Handed != nil {
h := *m.Handed
h.Templates = nil
m.Handed = &h
}
var dirs []map[string]any
for _, r := range m.Resources {
if r["type"] == "directory" {
dirs = append(dirs, r)
}
}
m.Resources = dirs
return m
}
// Carried is m, folded into its node's own composition at a build of the old form, as the node runs it (issue 1126,
// C5): nothing to build; its containers and directories those the node's own composition (runs) carries of it, their
// module's word taken off their ids, with the manifest's directories it does not run. So what other modules read of it
// is what runs: the machine ports its containers publish (the filter's rules, the routes reaching it: MachineSide). The
// paths it owns, which the holder of node-files is told, are its manifest's, as for its Stub: the controller composes
// them from the manifest, not from every path a resource of it runs at (issue 1126, seed-8: f1-circuits, snake and
// txt-game on ace, whose manifests leave their directories to the mesh). Its other resources are not taken, since the
// composition makes some of them again from its manifest (its secrets' files) and one id twice is refused; none of them
// is composed into what is applied either way (compositions leaves a carried module's out, and carries the ones that
// run). What it provides, requires, listens on and grants is its manifest's. A module whose resources runs holds none
// of is its Stub, as before.
func Carried(m catalogue.Manifest, module string, runs *delivery.Composed) catalogue.Manifest {
if runs == nil {
return Stub(m)
}
var taken []map[string]any
runsAny := false
ids := map[string]bool{}
for _, raw := range runs.Resources {
var r map[string]any
if json.Unmarshal(raw, &r) != nil {
continue
}
id, _ := r["id"].(string)
owner, _ := r["module"].(string)
if owner == "" {
owner, _ = declaration.ModuleGuess(id)
}
if owner != module {
continue
}
runsAny = true
if typ := r["type"]; typ != "container" && typ != "directory" {
continue
}
r["id"] = strings.TrimPrefix(id, module+".")
delete(r, "module")
ids[r["id"].(string)] = true
taken = append(taken, r)
}
if !runsAny {
return Stub(m)
}
stub := Stub(m)
for _, r := range stub.Resources {
if id, _ := r["id"].(string); !ids[id] {
taken = append(taken, r)
}
}
stub.Resources = taken
return stub
}
// carry puts into the node's resources (kept, in the declaration's order) the resources of the node's own composition as
// it runs (runs) that belong to a carried module, each after the resource it runs after, or first when that one is
// not among them; a file among them is a folded path.
func carry(kept []json.RawMessage, foldedPaths map[string]bool, runs *delivery.Composed, carried func(module, id string) bool) ([]json.RawMessage, map[string]bool, error) {
idOf := func(raw json.RawMessage) (string, string, string, string, error) {
var head struct {
ID string `json:"id"`
Module string `json:"module"`
Type string `json:"type"`
Path string `json:"path"`
}
err := json.Unmarshal(raw, &head)
return head.ID, head.Module, head.Type, head.Path, err
}
present := map[string]bool{}
for _, raw := range kept {
if id, module, _, _, err := idOf(raw); err == nil && module == "" {
present[id] = true
}
}
// By the id of the resource of the node's own it runs after, "" for first: the last one before it that is composed.
after := map[string][]json.RawMessage{}
anchor := ""
for _, raw := range runs.Resources {
id, module, typ, path, err := idOf(raw)
if err != nil {
return nil, nil, fmt.Errorf("compose: a resource the node runs in its own composition: %w", err)
}
if !carried(module, id) {
if module == "" && present[id] {
anchor = id
}
continue
}
after[anchor] = append(after[anchor], raw)
if typ == string(declaration.TypeFile) {
foldedPaths[path] = true
}
}
out := make([]json.RawMessage, 0, len(kept)+len(runs.Resources))
out = append(out, after[""]...)
for _, raw := range kept {
out = append(out, raw)
if id, module, _, _, err := idOf(raw); err == nil && module == "" {
out = append(out, after[id]...)
}
}
return out, foldedPaths, nil
}
+201
View File
@@ -0,0 +1,201 @@
package compose
import (
"encoding/json"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose/resolution"
"github.com/novox/mesh-host/internal/overlay"
)
func fpFor(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
// 1943/AC5: a node's compositions are its declaration as Compose gives it, split by the module each resource belongs
// to: every resource once, each in its module's composition at the fingerprint given, the module's ownership said on
// it; a module given no fingerprint is named as skipped and composed nowhere.
func TestCompositionsAreTheDeclarationSplitByModule(t *testing.T) {
in := aSmallMesh(t)
d, _, err := Compose(in, "laptop")
if err != nil {
t.Fatal(err)
}
var want []string
for _, r := range d.Resources {
want = append(want, r["type"].(string)+":"+r["id"].(string))
}
sort.Strings(want)
fps := map[string]string{"mesh-wireguard": fpFor('a'), "files": fpFor('b')}
out, skipped, err := Compositions(in, "laptop", fps, Foreseeing)
if err != nil || len(skipped) != 0 {
t.Fatalf("Compositions = %v, skipped %v", err, skipped)
}
var got []string
for key, c := range out {
node, module, fp, isNode, err := delivery.ParseComposedKey(key)
if err != nil || node != "laptop" || module != c.Module || fp != c.FP || (isNode != (module == "")) {
t.Errorf("key %s holds %s at %s (%v)", key, c.Module, c.FP, err)
}
if !isNode && fp != fps[module] {
t.Errorf("%s is composed at %s, given %s", module, fp, fps[module])
}
if _, err := delivery.EncodeComposed(c); err != nil {
t.Errorf("%s: %v", key, err)
}
for _, raw := range c.Resources {
var r struct{ ID, Type, Module string }
_ = json.Unmarshal(raw, &r)
if r.Module != module {
t.Errorf("%s holds %s of %q", key, r.ID, r.Module)
}
got = append(got, r.Type+":"+r.ID)
}
}
sort.Strings(got)
if strings.Join(got, " ") != strings.Join(want, " ") {
t.Errorf("the compositions hold %v, the declaration %v", got, want)
}
one, skipped, err := Compositions(in, "laptop", map[string]string{"mesh-wireguard": fpFor('a')}, Foreseeing)
if err != nil || strings.Join(skipped, ",") != "files" {
t.Fatalf("one module: %v, skipped %v", err, skipped)
}
for _, c := range one {
if c.Module == "files" {
t.Errorf("a module given no fingerprint was composed")
}
}
// Issue 1126: a module the node runs inside its own composition (Node.InOwn), and the private network's always,
// given no fingerprint of the new form, is folded into the node's own as the controller's composed writer folds it:
// every resource in its place, its module taken off, its id still naming it, nothing skipped.
folding := in
folding.Nodes = append([]Node(nil), in.Nodes...)
for i := range folding.Nodes {
if folding.Nodes[i].Name == "laptop" {
folding.Nodes[i].InOwn = map[string]bool{"files": true}
}
}
all, skipped, err := Compositions(folding, "laptop", nil, Foreseeing)
nodeKey, _ := delivery.ComposedNodeKey("laptop")
if err != nil || len(skipped) != 0 || len(all) != 1 || len(all[nodeKey].Resources) != len(want) {
t.Fatalf("folded: %v, skipped %v, %d compositions, %d of %d resources in the node's own", err, skipped, len(all), len(all[nodeKey].Resources), len(want))
}
for i, raw := range all[nodeKey].Resources {
var r map[string]any
_ = json.Unmarshal(raw, &r)
if _, says := r["module"]; says || r["id"] != d.Resources[i]["id"] {
t.Errorf("folded resource %d is %v, the declaration's %v", i, r["id"], d.Resources[i]["id"])
}
}
if _, err := delivery.EncodeComposed(all[nodeKey]); err != nil {
t.Errorf("the node's own with the folded modules: %v", err)
}
if _, _, err := Compositions(in, "nowhere", fps, Foreseeing); err == nil {
t.Errorf("a node the mesh does not have was composed")
}
// Issue 1126, 2026-10-29: a module composed as what runs of it (folded at a build of the old form, or held) still
// contributes to the runtime what the runtime serves of it as it runs (Node.RunsTools): its manifest at the
// indexer's commit has no bundles (shanks's runtime lost the folded systemd-resolved, novox's the folded nats), and a
// held module's new build's bundles are not delivered.
dg := func(c byte) string { return "sha256:" + strings.Repeat(string(c), 64) }
runtime := catalogue.Manifest{Module: catalogue.RuntimeModule, Version: "1",
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/var/lib/mesh/node-tools/broker"}},
Bundles: []catalogue.Bundle{{Name: "runtime", Source: catalogue.ArtifactStoreScheme + "node-tools/runtime/blobs/" + dg('a'),
Digest: dg('a'), Language: "go", Binary: "node-tools"}}}
store := catalogue.Manifest{Module: "artifacts", Version: "1",
Serves: map[string]map[string]any{catalogue.ArtifactStoreProvision: {"port": 5100}}}
lamp := catalogue.Manifest{Module: "lamp", Version: "1", Tools: []string{"on"},
Resources: []map[string]any{{"id": "state", "type": "directory", "path": "/var/lib/lamp", "mode": "0700"}},
Build: &catalogue.Build{Artifacts: []catalogue.Artifact{{Name: "tools", Kind: catalogue.ArtifactBundle,
Language: "go", System: "arch", From: "cmd/lamp-tools", Env: map[string]string{"LAMP_CONF": "/var/lib/lamp/conf"}}}}}
newLamp, err := lamp.Resolve([]catalogue.Built{{Name: "tools", Kind: catalogue.ArtifactBundle,
Reference: catalogue.ArtifactStoreScheme + "lamp/tools/blobs/" + dg('c'), Digest: dg('c')}})
if err != nil {
t.Fatal(err)
}
served := "lamp=" + resolution.BundlePath("lamp", "tools") + "/lamp-tools"
runtimeRuns := func(dropped bool) delivery.Composed {
env := `"MESH_TOOL_MODULES":"` + served + `","MESH_TOOL_ENV":"{\"lamp\":{\"LAMP_CONF\":\"/var/lib/lamp/conf\"}}"`
if dropped {
env = `"MESH_TOOL_MODULES":""` // the drop already applied (shanks, 2026-10-29)
}
return delivery.Composed{Node: "anchor", Module: catalogue.RuntimeModule, Resources: []json.RawMessage{json.RawMessage(
`{"id":"node-tools.runtime","module":"node-tools","type":"process","name":"node-tools","env":{` + env + `}}`)}}
}
lampRuns := `{"id":"lamp.bundle-tools","type":"archive","source":"` + catalogue.ArtifactStoreScheme + `lamp/tools/blobs/` + dg('b') +
`","digest":"` + dg('b') + `","path":"` + resolution.BundlePath("lamp", "tools") + `"}`
for _, row := range []struct {
name string
set func(n *Node) delivery.Composed // and what runs of lamp
fps map[string]string
dropped bool
}{
{"folded at a build of the old form", func(n *Node) delivery.Composed {
n.InOwn = map[string]bool{"lamp": true}
n.RunsOwn = &delivery.Composed{Node: "anchor", Resources: []json.RawMessage{json.RawMessage(lampRuns)}}
return *n.RunsOwn
}, map[string]string{}, false},
{"folded, the runtime already dropped it", func(n *Node) delivery.Composed {
n.InOwn = map[string]bool{"lamp": true}
n.RunsOwn = &delivery.Composed{Node: "anchor", Resources: []json.RawMessage{json.RawMessage(lampRuns)}}
return *n.RunsOwn
}, map[string]string{}, true},
{"held, its new build not delivered", func(n *Node) delivery.Composed {
n.Held = map[string]bool{"lamp": true}
lr := delivery.Composed{Node: "anchor", Module: "lamp", Resources: []json.RawMessage{json.RawMessage(
strings.Replace(lampRuns, `"type"`, `"module":"lamp","type"`, 1))}}
n.Kept = map[string]catalogue.Manifest{"lamp": Carried(newLamp, "lamp", &lr)}
return lr
}, map[string]string{"lamp": fpFor('1')}, false},
} {
tools := aMesh(t)
tools.Catalogue[catalogue.RuntimeModule], tools.Catalogue["artifacts"], tools.Catalogue["lamp"] = runtime, store, lamp
anchor := nodeOf(t, &tools, "anchor")
anchor.Assigned = append(anchor.Assigned, "artifacts", catalogue.RuntimeModule, "lamp")
tools.BusMinted = []string{"anchor." + catalogue.RuntimeModule}
anchor.SealingKey = keyOf(t, "anchor")
anchor.RunsTools = ToolsRunning([]delivery.Composed{runtimeRuns(row.dropped), row.set(anchor)})
row.fps[overlay.Name], row.fps["artifacts"], row.fps[catalogue.RuntimeModule] = fpFor('a'), fpFor('5'), fpFor('e')
out, _, _, err := CompositionsIn(tools, "anchor", row.fps, Foreseeing)
if err != nil {
t.Fatalf("%s: %v", row.name, err)
}
var env map[string]any
for _, c := range out {
for _, raw := range c.Resources {
var r struct {
ID string `json:"id"`
Env map[string]any `json:"env"`
}
if json.Unmarshal(raw, &r) == nil && r.ID == "node-tools.runtime" {
env = r.Env
}
}
}
if env[catalogue.RuntimeToolModules] != served || env[catalogue.RuntimeToolEnv] != `{"lamp":{"LAMP_CONF":"/var/lib/lamp/conf"}}` {
t.Errorf("%s: the runtime serves %v with %v, want what runs of lamp, %s", row.name,
env[catalogue.RuntimeToolModules], env[catalogue.RuntimeToolEnv], served)
}
}
}
// 1943/AC1: the upgrade policies and sources a composition's value may name are the catalogue's own words.
func TestTheComposedPoliciesAreTheCatalogues(t *testing.T) {
for _, p := range []string{catalogue.PolicyRoll, catalogue.PolicyTogether, catalogue.PolicyRecord} {
for _, from := range []string{catalogue.FromPerson, catalogue.FromModule, catalogue.FromBus, catalogue.FromData, catalogue.FromDefault} {
v := delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("a", 64), Judgement: delivery.JudgedNeither, Policy: p, PolicyFrom: from}
if _, err := delivery.EncodeComposedDigest(v); err != nil {
t.Errorf("%s from %s: %v", p, from, err)
}
}
}
if delivery.PolicyRecord != catalogue.PolicyRecord {
t.Errorf("delivery.PolicyRecord = %q, the catalogue's %q", delivery.PolicyRecord, catalogue.PolicyRecord)
}
}
+499
View File
@@ -0,0 +1,499 @@
package compose
import (
"crypto/ecdh"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/compose/resolution"
)
// Mode says what a composition does about what a send would make. Both modes read what the Input holds and make nothing; they
// differ as the controller's Reading and Foreseeing modes do.
type Mode int
const (
// Reading composes what is held and leaves out an own secret the Input holds none for: that machine cannot be running it.
Reading Mode = iota
// Foreseeing composes an own secret the next send would make with a stand-in and names it foreseen, while one the send would
// refuse (a bus credential nobody issued, a given secret under an old key) is refused here, in the same words.
Foreseeing
// Planning composes for a provisioning plan alone (novox/hq ADR 0570 clause 2): every own secret the Input holds none
// for stands in, named foreseen, and nothing is refused for a value, since a plan names secrets and never carries one.
// The seat mesh-provisioner's holder composes so, reading no secret; a rendering in this mode is never applied.
Planning
)
// ForeseenCertificate and ForeseenBusUsers stand for what the Input does not carry, in a composition asked ahead of the send.
// Never sent.
const (
ForeseenCertificate = "foreseen: issued by the next send"
ForeseenBusUsers = "foreseen: composed from the bus records by the next send"
)
// grantsFor is every credential this node must create, because something elsewhere uses it (plan.go grantsFor): the mirror of
// what a consumer is given. A consumer whose identity overflows the provision's bound is left out of the grants and returned
// beside them (novox/hq ADR 0225); one whose resolution binds it to another provider is withdrawn and returned beside them
// (issue 274). Every other is granted.
func (c *composer) grantsFor(node string) ([]catalogue.Grant, []catalogue.Overflow, []catalogue.Unbound, []string, error) {
issued, err := c.rowsFor(node)
if err != nil {
return nil, nil, nil, nil, err
}
onNetwork, err := c.whereEveryoneIs()
if err != nil {
return nil, nil, nil, nil, err
}
out := make([]catalogue.Grant, 0, len(issued))
var withheld []catalogue.Overflow
var unbound []catalogue.Unbound
var foreseen []string
for _, s := range issued {
resolved, err := c.planFor(s.Consumer)
switch {
case unresolvable(err):
// Their set does not resolve. Skipped rather than fatal: this node is not the place to report another machine's problem.
continue
case err != nil:
// The mesh could not be asked what they wanted, which is not the same as their wanting nothing.
return nil, nil, nil, nil, fmt.Errorf("what %s asked of %s cannot be read: %w", s.Consumer, s.Provision, err)
}
plan := resolved.Plan
values, asks, err := resolution.ContributionsFrom(plan, s.Provision, s.ConsumerModule, resolved.Settings)
if err != nil {
return nil, nil, nil, nil, err
}
// A port in there is the consumer's software port until this: the assignment that moved it is that machine's.
consumer, err := c.in.node(s.Consumer)
if err != nil {
return nil, nil, nil, nil, err
}
values = resolution.AtPublishedPort(values, s.ConsumerModule, consumer.Ports[s.ConsumerModule])
from := s.ConsumerModule
if !asks {
// That module no longer wants this: left empty, which is how a login is withdrawn rather than kept working for ever.
from = ""
}
if bound := resolution.BindsFrom(plan, s.Provision, s.ConsumerModule, s.Local); from != "" && !slices.Contains(bound, node) {
// It still asks, and not of this node: withdrawn exactly as a credential nobody asks for, and said.
from = ""
unbound = append(unbound, catalogue.Unbound{Provision: s.Provision, Provider: node,
Consumer: s.Consumer, Module: s.ConsumerModule, Local: s.Local, BoundTo: bound})
}
// The consumer's identity slug, from its own manifest, carried on the grant so the provider derives the same login the
// consumer does (novox/hq ADR 0049).
slug := ""
for _, mm := range plan.Modules {
if mm.Module == s.ConsumerModule {
slug = mm.Slug
break
}
}
if from != "" {
bound := boundOfGrant(plan, s, node)
source := catalogue.IdentitySource(slug, s.ConsumerModule)
if catalogue.CheckIdentityWithin(s.Consumer, source, bound) != nil {
withheld = append(withheld, catalogue.Overflow{Provision: s.Provision, Provider: node,
Consumer: s.Consumer, Module: s.ConsumerModule,
Identity: catalogue.ConsumerIdentity(s.Consumer, source), Bound: bound})
continue
}
}
sealed := s.ForProvider
switch {
case credentiallessRow(plan, s, node):
// Its offer presents no credential (form none): the grant carries the consumer and nothing sealed, and
// nothing stands in for it, so neither end waits for a value nobody makes (issue 1126).
sealed = ""
case sealed == "":
sealed = catalogue.ForeseenSealed
foreseen = append(foreseen, fmt.Sprintf("the grant of %s for %s on %s", s.Provision, s.ConsumerModule, s.Consumer))
}
out = append(out, catalogue.Grant{
Provision: s.Provision, Consumer: s.Consumer, At: onNetwork[s.Consumer],
From: from, Values: values, Slug: slug, Sealed: sealed, Local: s.Local})
}
return out, withheld, unbound, foreseen, nil
}
// credentialless is whether a need's provider offers it with no credential (catalogue.CredentialFormNone): the vault
// mints nothing for it, so nothing is asked of the vault, sealed into a grant or a file, or stood in for (issue 1126).
func credentialless(n catalogue.Needed) bool { return n.CredentialForm == catalogue.CredentialFormNone }
// credentiallessRow is whether the consumer's resolution answers row's need from provider with an offer of form none.
func credentiallessRow(consumer catalogue.Resolution, row Issued, provider string) bool {
return slices.ContainsFunc(consumer.Needs, func(n catalogue.Needed) bool {
return n.Name == row.Provision && n.For == row.ConsumerModule && n.Local == row.Local && n.From == provider &&
credentialless(n)
})
}
// rowsFor is every pair credential this node's declaration must carry a grant for, in the order the controller reads them
// (SecretsFrom: by provision, consumer, consumer module and local name): the ones the Input says were made, and the ones the same send
// would make before it composes this node -- every need of a node that resolves (this one too), whose provider is this node, that is not
// answered by a record and not a credential only a person gives and nobody gave. Those have no sealed copy (ForProvider empty), so the
// grant is composed with a stand-in and named foreseen (plan.go: planFor makes the secret in every mode before grantsFor reads it).
func (c *composer) rowsFor(node string) ([]Issued, error) {
var rows []Issued
for _, row := range c.in.Issued {
if row.Provider == node {
rows = append(rows, row)
}
}
for _, other := range c.in.sorted() {
resolved, err := c.planFor(other.Name)
switch {
case unresolvable(err):
continue
case err != nil:
return nil, fmt.Errorf("what %s asked of %s cannot be read: %w", other.Name, node, err)
}
shelf := c.shelfFor(other)
for _, n := range resolved.Plan.Needs {
if n.From != node || n.ByRecord {
continue
}
cred, held := c.credential(n, other.Name)
if _, outside := issuedOutsideBy(shelf, n); cred.State == CredentialNotGiven || (outside && !(held && cred.State == CredentialHeld)) {
continue
}
if slices.ContainsFunc(rows, func(r Issued) bool {
return r.Provision == n.Name && r.Consumer == other.Name && r.ConsumerModule == n.For && r.Local == n.Local
}) {
continue
}
rows = append(rows, Issued{Provision: n.Name, Consumer: other.Name, ConsumerModule: n.For, Local: n.Local, Provider: node})
}
}
sort.SliceStable(rows, func(i, j int) bool {
a, b := rows[i], rows[j]
switch {
case a.Provision != b.Provision:
return a.Provision < b.Provision
case a.Consumer != b.Consumer:
return a.Consumer < b.Consumer
case a.ConsumerModule != b.ConsumerModule:
return a.ConsumerModule < b.ConsumerModule
}
return a.Local < b.Local
})
return rows, nil
}
// boundOfGrant is the identity bound the consumer's own resolution states for the requirement this grant answers. A requirement
// not found there is held to the tightest bound the mesh knows rather than to none.
func boundOfGrant(consumer catalogue.Resolution, s Issued, provider string) catalogue.IdentityBound {
for _, n := range consumer.Needs {
if n.Name == s.Provision && n.For == s.ConsumerModule && n.From == provider {
return n.Identity
}
}
return catalogue.DefaultIdentityBound
}
// ownSecretRow is the row for one own secret of a module on a node.
func (c *composer) ownSecretRow(node, module, name string) (OwnSecretRow, bool) {
for _, r := range c.in.OwnSecrets {
if r.Node == node && r.Module == module && r.Name == name {
return r, true
}
}
return OwnSecretRow{}, false
}
// givenMember is one own secret a person gave on a machine and whether it is sealed to the key the machine holds now.
type givenMember struct {
Name string
Sealed string
Current bool
}
// givenMembers is every member of a module's secret family given on a machine, by name (inv.GivenMembers): the mesh never
// makes a member, so only a value a person gave is one; a machine with no sealing key holds none.
func (c *composer) givenMembers(node Node, module, family string) []givenMember {
if node.SealingKey == "" {
return nil
}
prefix := catalogue.FamilyPrefix(family)
var out []givenMember
for _, r := range c.in.OwnSecrets {
if r.Node == node.Name && r.Module == module && r.Origin == "accepted" && strings.HasPrefix(r.Name, prefix) {
out = append(out, givenMember{Name: r.Name, Sealed: r.Sealed, Current: r.SealedTo == node.SealingKey})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
return out
}
// givenOwnSecret is an own secret issued outside the mesh as a person gave it on a machine (inv.GivenOwnSecret): not given when
// the value held was made by the mesh or the machine has no sealing key.
func (c *composer) givenOwnSecret(node Node, module, name string) (givenMember, bool) {
if node.SealingKey == "" {
return givenMember{}, false
}
r, ok := c.ownSecretRow(node.Name, module, name)
if !ok || r.Origin != "accepted" {
return givenMember{}, false
}
return givenMember{Name: name, Sealed: r.Sealed, Current: r.SealedTo == node.SealingKey}, true
}
// moduleSecretIfIssued is the read half of the controller's SecretForModule (inv.ModuleSecretIfIssued): the secret when one is
// held and sealed to the key the machine holds now, and otherwise not held. A key it was sealed to that the machine no longer
// has is reported as absent rather than as an error.
func (c *composer) moduleSecretIfIssued(node Node, module, name string) (sealed string, held bool) {
if node.SealingKey == "" {
return "", false
}
r, ok := c.ownSecretRow(node.Name, module, name)
if !ok || r.SealedTo != node.SealingKey {
return "", false
}
return r.Sealed, true
}
// wouldMakeSecretForModule says what the send would do about an own secret without doing it: nil when it would make one (or one
// is held), and otherwise the very refusal it would meet (inv.WouldMakeSecretForModule).
func (c *composer) wouldMakeSecretForModule(node Node, module, name string) error {
if node.SealingKey == "" {
return fmt.Errorf("%s needs a secret and %s has no sealing key, so nothing can be sealed to it", module, node.Name)
}
r, ok := c.ownSecretRow(node.Name, module, name)
switch {
case !ok:
return nil
case r.SealedTo != node.SealingKey && r.Origin == "accepted":
return fmt.Errorf("%s on %s holds %q, which was given to the mesh rather than made by it, and %s has "+
"since generated a new sealing key. The mesh cannot make another; issue it again", module, node.Name, name, node.Name)
}
return nil
}
// givenUnderAnOldKey is the refusal for a value a person gave sealed to a key the machine no longer holds.
func givenUnderAnOldKey(module, node, name string) error {
return fmt.Errorf("%s on %s holds %q, which was given to the mesh rather than made by it, and %s has "+
"since generated a new sealing key. The mesh cannot make another; give it again", module, node, name, node)
}
// ownSecretsFor is each module's own secrets as this node composes them (plan.go renderingFor): the needed ones sealed to the
// node, and, in the Foreseeing mode, the ones the next send would make named foreseen. A module handed to the provisioner has its
// secrets from the vault through the client, so none of them is composed here.
func (c *composer) ownSecretsFor(node Node, plan catalogue.Resolution, handing catalogue.Rendering, mode Mode) (
needed map[string]map[string]string, foreseen map[string]map[string]bool, err error) {
needed = map[string]map[string]string{}
foreseen = map[string]map[string]bool{}
mark := func(module, name string) {
if foreseen[module] == nil {
foreseen[module] = map[string]bool{}
}
foreseen[module][name] = true
}
for _, m := range plan.Modules {
if resolution.Handing(handing, m) {
continue
}
// A secret family is never made (novox/hq ADR 0283): each member a person gave on this machine is placed, and one not
// given is nothing -- the module says it waits for it.
for _, family := range m.OwnSecrets.Families() {
for _, g := range c.givenMembers(node, m.Module, family) {
if _, fam, ok := m.OwnSecrets.Lookup(g.Name); !ok || fam != family {
continue // a longer family's member, or a name no longer of this family
}
if !g.Current {
continue
}
if needed[m.Module] == nil {
needed[m.Module] = map[string]string{}
}
needed[m.Module][g.Name] = g.Sealed
if g.Sealed == "" {
// Given, and its value is not in the Input: placed with a stand-in, named.
needed[m.Module][g.Name] = catalogue.ForeseenSealed
mark(m.Module, g.Name)
}
}
}
names := make([]string, 0)
for name := range m.OwnSecrets.Plain() {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
own := m.OwnSecrets.Plain()[name]
// An own secret issued outside the mesh is never made (novox/hq issue 401, ADR 0228; ADR 0283 for a family): only a
// value a person gave is placed. One not given is nothing, and a value the mesh made for it before is not one either.
if own.IssuedBy == catalogue.IssuedOutside {
g, given := c.givenOwnSecret(node, m.Module, name)
if !given || !g.Current {
continue
}
if g.Sealed == "" {
mark(m.Module, name)
continue
}
if needed[m.Module] == nil {
needed[m.Module] = map[string]string{}
}
needed[m.Module][name] = g.Sealed
continue
}
sealed, held := c.moduleSecretIfIssued(node, m.Module, name)
if !held && mode == Planning {
mark(m.Module, name)
continue
}
if !held && mode == Foreseeing {
// Asked ahead of the send: what the send would do about it, by the send's own rules. Made by it -- a stand-in,
// named; refused by it -- refused here, the same words (novox/hq issue 275). A bus account not yet issued is
// foreseen, never refused: the vault mints it from the grants part's wanted list once the placement is written
// on the operator's Approve (ADR 0561 §1, ADR 0567 §5), so a composition asked ahead of that cannot hold it
// (issue 774, 2026-10-30: every proposal placing mesh-conditions, mesh-self-check or mesh-healers refused).
if err := c.wouldMakeSecretForModule(node, m.Module, name); err != nil {
return nil, nil, err
}
mark(m.Module, name)
continue
}
if !held {
// Never issued, so this machine cannot be running it: left out rather than invented.
continue
}
if sealed == "" {
// Held, and its value is not in the Input: composed with a stand-in, named.
mark(m.Module, name)
continue
}
if needed[m.Module] == nil {
needed[m.Module] = map[string]string{}
}
needed[m.Module][name] = sealed
}
}
return needed, foreseen, nil
}
// CoreBoxWanted is a core credential whose box the Input does not hold for a machine: the next send asks the vault for it and
// keeps the answer. It is returned for the caller to ask and keep; this package asks and keeps nothing.
type CoreBoxWanted struct {
Node string
Module string
Name string
}
// coreModulesOf is every module of the plan on the core's list whose own secrets this machine's declaration carries, with their
// plain own secret names, sorted: one handed to the provisioner has its secrets from the vault through the client, and one issued
// outside the mesh is never the vault's to seal here.
func coreModulesOf(modules []catalogue.Manifest, handed func(catalogue.Manifest) bool) map[string][]string {
out := map[string][]string{}
for _, m := range modules {
if !resolution.CarriesVaultSealed(m.Module) || (handed != nil && handed(m)) {
continue
}
var names []string
for name, own := range m.OwnSecrets.Plain() {
if own.IssuedBy != catalogue.IssuedOutside {
names = append(names, name)
}
}
if len(names) > 0 {
sort.Strings(names)
out[m.Module] = names
}
}
return out
}
// sealingKeyValid says a machine's sealing key, recorded raw in base64, is a raw X25519 key.
func sealingKeyValid(raw string) error {
b, err := base64.StdEncoding.DecodeString(strings.TrimSpace(raw))
if err != nil || len(b) != 32 {
return errors.New("the machine's sealing key is not a raw X25519 key")
}
_, err = ecdh.X25519().NewPublicKey(b)
return err
}
// coreBoxesFor decides each core credential of a machine in a composition that sends nothing and asks no vault (core_box.go
// coreBoxesOf, in the modes that are not a send): the kept box while it is sealed to the key the machine holds now, and otherwise a
// stand-in that the next send replaces by asking the vault -- returned in wanted for the caller to ask and keep. A machine with no
// sealing key, or one whose key is not a key, leaves its module out, said.
func (c *composer) coreBoxesFor(node Node, modules []catalogue.Manifest, handing catalogue.Rendering) (
sealed map[string]map[string]json.RawMessage, notPlaced map[string]string, wanted []CoreBoxWanted) {
core := coreModulesOf(modules, func(m catalogue.Manifest) bool { return resolution.Handing(handing, m) })
sealed, notPlaced = map[string]map[string]json.RawMessage{}, map[string]string{}
mods := make([]string, 0, len(core))
for m := range core {
mods = append(mods, m)
}
sort.Strings(mods)
for _, module := range mods {
if node.SealingKey == "" {
notPlaced[module] = fmt.Sprintf("%s has no sealing key, so the vault can seal %s's credentials to "+
"nothing there (novox/hq ADR 0349 §16)", node.Name, module)
continue
}
if err := sealingKeyValid(node.SealingKey); err != nil {
notPlaced[module] = fmt.Sprintf("%s's credentials cannot be asked of the vault: %v", module, err)
continue
}
boxes := map[string]json.RawMessage{}
for _, name := range core[module] {
var kept *CoreBox
for i := range c.in.KeptCoreBoxes {
k := &c.in.KeptCoreBoxes[i]
if k.Node == node.Name && k.Module == module && k.Name == name {
kept = k
}
}
if kept != nil && kept.SealedTo == node.SealingKey && len(kept.Box) > 0 {
boxes[name] = kept.Box
continue
}
boxes[name] = catalogue.ForeseenVaultBox
wanted = append(wanted, CoreBoxWanted{Node: node.Name, Module: module, Name: name})
}
sealed[module] = boxes
}
return sealed, notPlaced, wanted
}
// certificateFor is what the mesh certifies about one machine's internal name (plan.go certificateFor): refused for a machine that
// never told the mesh what key it serves with, and for one not on the private network, which has no name to be certified for.
// The certificate and the authority are public values the Input may carry; without them stand-ins are composed, and named.
func (c *composer) certificateFor(node Node) (certificate, authority string, foreseen bool, err error) {
if node.ServingKey == "" {
return "", "", false, fmt.Errorf(
"%s wants a certificate and has never told the mesh what key it serves with; it "+
"joins again to report one", node.Name)
}
where, err := c.whereEveryoneIs()
if err != nil {
return "", "", false, err
}
if where[node.Name] == "" {
return "", "", false, fmt.Errorf(
"%s wants a certificate and is not on the private network, so it has no name inside "+
"the mesh to be certified for", node.Name)
}
certificate, authority = node.Certificate, c.in.Authority
if certificate == "" || authority == "" {
return ForeseenCertificate, ForeseenCertificate, true, nil
}
return certificate, authority, false, nil
}
// holdsTheBus says whether a node's set includes the module that asks for the bus's user list.
func holdsTheBus(modules []catalogue.Manifest) bool {
for _, m := range modules {
if m.BusUsers != "" && m.ClaimsSeat("mesh-broker") {
return true
}
}
return false
}
+69
View File
@@ -0,0 +1,69 @@
package compose
import (
"fmt"
"sort"
)
// CredentialKey names one pair credential by the ends it joins: the provision, the consumer's node and module, the
// provider's node, and the local name where a module keeps several (Credential's and Issued's key).
type CredentialKey struct {
Provision string `json:"provision"`
Consumer string `json:"consumer"`
ConsumerModule string `json:"consumer_module"`
Provider string `json:"provider"`
Local string `json:"local,omitempty"`
}
// CredentialsNeeded is every pair credential node's composition reads (novox/hq ADR 0547 §2, ADR 0550 §5, issue 1126):
// as a consumer, each need of its resolution a provider's credential answers (a need answered by a record is a licence's,
// not a pair's); as a provider, each credential it must carry a grant for (rowsFor). A node that composes its own asks the
// vault for exactly these, sealed to its key, and composes with them; one it does not get is stood in for, which the
// node's own apply refuses (CompositionsApplied). A need whose provider offers it with no credential (form none) is
// in neither list: the vault mints nothing for it (issue 1126).
func CredentialsNeeded(in Input, node string) (asConsumer, asProvider []CredentialKey, err error) {
c := newComposer(in)
resolved, err := c.resolution(node)
if err != nil {
return nil, nil, err
}
for _, n := range resolved.Needs {
if n.ByRecord || n.From == "" || credentialless(n) {
continue
}
asConsumer = append(asConsumer, CredentialKey{Provision: n.Name, Consumer: node, ConsumerModule: n.For,
Provider: n.From, Local: n.Local})
}
rows, err := c.rowsFor(node)
if err != nil {
return nil, nil, fmt.Errorf("compose: the credentials %s provides: %w", node, err)
}
for _, r := range rows {
// A grant whose offer presents no credential (form none) has none to ask for (issue 1126).
if consumer, err := c.planFor(r.Consumer); err == nil && credentiallessRow(consumer.Plan, r, node) {
continue
}
asProvider = append(asProvider, CredentialKey{Provision: r.Provision, Consumer: r.Consumer,
ConsumerModule: r.ConsumerModule, Provider: node, Local: r.Local})
}
sortKeys(asConsumer)
sortKeys(asProvider)
return asConsumer, asProvider, nil
}
func sortKeys(ks []CredentialKey) {
sort.Slice(ks, func(i, j int) bool {
a, b := ks[i], ks[j]
switch {
case a.Provision != b.Provision:
return a.Provision < b.Provision
case a.Provider != b.Provider:
return a.Provider < b.Provider
case a.Consumer != b.Consumer:
return a.Consumer < b.Consumer
case a.ConsumerModule != b.ConsumerModule:
return a.ConsumerModule < b.ConsumerModule
}
return a.Local < b.Local
})
}
+549
View File
@@ -0,0 +1,549 @@
package compose
import (
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/overlay"
)
// secretValue is a value the Input carries that must never appear in anything the composition says.
const secretValue = "SECRET-VALUE-THAT-MUST-NOT-LEAK"
func rendered(t *testing.T, in Input, node string, mode Mode) Rendered {
t.Helper()
got, err := Render(in, node, mode)
if err != nil {
t.Fatalf("%v", err)
}
return got
}
// anAssistant is mesh-controller's outside_not_given_test.go anAssistant (cmd/mesh-controller at 6c2a8d62): a module with a secret
// the mesh makes and one only an outside party issues.
func anAssistant() catalogue.Manifest {
return catalogue.Manifest{Module: "assistant", Version: "1",
OwnSecrets: catalogue.OwnSecrets{
"password": {Path: "/var/lib/mesh/assistant/password", Taken: catalogue.TakenAtStart},
"vendor-key": {Path: "/var/lib/mesh/assistant/vendor-key", Taken: catalogue.TakenAtStart, IssuedBy: catalogue.IssuedOutside},
},
Resources: []map[string]any{{"id": "state", "type": "directory", "path": "/var/lib/mesh/assistant", "mode": "0700"}}}
}
func withAssistant(t *testing.T) Input {
t.Helper()
in := aMesh(t)
in.Catalogue["assistant"] = anAssistant()
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "assistant")
laptop.SealingKey = keyOf(t, "laptop")
return in
}
// Own secrets (plan.go renderingFor, novox/hq issue 275 and 401): one held and sealed to the machine's key is carried; one not held is
// foreseen in the Foreseeing mode and left out in the Reading mode, and nothing is made.
func TestAnOwnSecretHeldIsCarriedAndOneNotHeldIsForeseenOrLeftOut(t *testing.T) {
mk := func() Input {
in := withAssistant(t)
in.OwnSecrets = []OwnSecretRow{{Node: "laptop", Module: "assistant", Name: "password", Origin: "made",
SealedTo: nodeOf(t, &in, "laptop").SealingKey, Sealed: secretValue}}
return in
}
in := mk()
got := rendered(t, in, "laptop", Foreseeing)
if got.Rendering.Needed["assistant"]["password"] != secretValue || len(got.Rendering.Foreseen) != 0 {
t.Errorf("a held secret: needed %v foreseen %v", got.Rendering.Needed, got.Rendering.Foreseen)
}
// Not held: foreseen with a stand-in (Foreseeing), left out (Reading).
in.OwnSecrets = nil
got = rendered(t, in, "laptop", Foreseeing)
if !got.Rendering.Foreseen["assistant"]["password"] || got.Rendering.Needed["assistant"]["password"] != "" ||
len(got.Foreseen) != 1 || got.Foreseen[0] != "assistant/password" {
t.Errorf("a secret the send makes: foreseen %v / %v needed %v", got.Rendering.Foreseen, got.Foreseen, got.Rendering.Needed)
}
got = rendered(t, in, "laptop", Reading)
if len(got.Rendering.Needed["assistant"]) != 0 || len(got.Foreseen) != 0 || got.Rendering.Foreseen != nil {
t.Errorf("Reading composed a secret that was never issued: %v %v", got.Rendering.Needed, got.Foreseen)
}
// Held but sealed to a key the machine no longer has is not held.
in.OwnSecrets = []OwnSecretRow{{Node: "laptop", Module: "assistant", Name: "password", Origin: "made", SealedTo: "an-old-key", Sealed: secretValue}}
if got := rendered(t, in, "laptop", Foreseeing); !got.Rendering.Foreseen["assistant"]["password"] {
t.Errorf("a secret sealed to an old key is held: %v", got.Rendering.Needed)
}
// Held with no value in the Input: a stand-in, named.
in.OwnSecrets = []OwnSecretRow{{Node: "laptop", Module: "assistant", Name: "password", Origin: "made", SealedTo: nodeOf(t, &in, "laptop").SealingKey}}
if got := rendered(t, in, "laptop", Reading); !got.Rendering.Foreseen["assistant"]["password"] {
t.Errorf("a held secret with no value is not named: %v", got.Rendering.Foreseen)
}
unchanged(t, mk, func(in Input) { rendered(t, in, "laptop", Foreseeing) })
// 1126 (ADR 0547 §2): a node applying what it composed itself never applies a stand-in: a held secret with no value in the
// Input is refused, naming it; held with its value, it composes.
fps := map[string]string{"assistant": "src2:" + strings.Repeat("a", 64), overlay.Name: "src2:" + strings.Repeat("b", 64)}
if _, _, err := CompositionsApplied(in, "laptop", fps); !errors.Is(err, ErrStoodIn) || !strings.Contains(err.Error(), "assistant/password") {
t.Errorf("a stand-in was not refused for a node's own apply: %v", err)
}
if out, _, err := CompositionsApplied(mk(), "laptop", fps); err != nil || len(out) == 0 {
t.Errorf("a held secret with its value is refused for a node's own apply: %v", err)
}
}
// An own secret issued outside the mesh that nobody gave is never made (novox/hq issue 401, ADR 0228): nothing is composed or
// foreseen for it; a value the mesh made for it is not one a person gave; a value given is placed once it is sealed to the key held now.
func TestAnOutsideSecretIsPlacedOnlyOnceAPersonGaveIt(t *testing.T) {
in := withAssistant(t)
key := nodeOf(t, &in, "laptop").SealingKey
for _, mode := range []Mode{Reading, Foreseeing} {
got := rendered(t, in, "laptop", mode)
if _, there := got.Rendering.Needed["assistant"]["vendor-key"]; there || got.Rendering.Foreseen["assistant"]["vendor-key"] {
t.Fatalf("%v: a secret nobody gave is composed: %v %v", mode, got.Rendering.Needed, got.Rendering.Foreseen)
}
}
in.OwnSecrets = []OwnSecretRow{{Node: "laptop", Module: "assistant", Name: "vendor-key", Origin: "made", SealedTo: key, Sealed: secretValue}}
if _, there := rendered(t, in, "laptop", Foreseeing).Rendering.Needed["assistant"]["vendor-key"]; there {
t.Error("a value the mesh made for a secret issued outside was placed")
}
in.OwnSecrets = []OwnSecretRow{{Node: "laptop", Module: "assistant", Name: "vendor-key", Origin: "accepted", SealedTo: key, Sealed: "the vendor's key, sealed"}}
if got := rendered(t, in, "laptop", Foreseeing).Rendering.Needed["assistant"]["vendor-key"]; got != "the vendor's key, sealed" {
t.Errorf("a value given is placed as %q", got)
}
in.OwnSecrets[0].SealedTo = "an-old-key"
if _, there := rendered(t, in, "laptop", Foreseeing).Rendering.Needed["assistant"]["vendor-key"]; there {
t.Error("a value given under an old key was placed")
}
}
// 1364/AC3 (the send's refusals, novox/hq issue 275): in the Foreseeing mode a given secret under an old key is refused in the
// controller's words. No secret value is in an error. A bus account not yet issued is foreseen, issued or not (issue 774: the vault
// mints it once the placement is written on the Approve, ADR 0561 §1, ADR 0567 §5; mesh-self-check's proposal on ace was refused
// "has no bus credential: nothing was issued for ace.mesh-self-check").
func TestTheSendsRefusalsAreRefusedInTheSendsWords(t *testing.T) {
broker := catalogue.Manifest{Module: "mesh-self-check", Version: "1", RunsAs: "mesh-self-check",
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/var/lib/mesh/mesh-self-check/broker", Taken: catalogue.TakenAtStart}}}
in := withAssistant(t)
in.Catalogue["mesh-self-check"] = broker
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "mesh-self-check")
for _, minted := range [][]string{nil, {"laptop.mesh-self-check"}} {
in.BusMinted = minted
got, err := Render(in, "laptop", Foreseeing)
if err != nil || !got.Rendering.Foreseen["mesh-self-check"]["broker"] {
t.Fatalf("minted %v: the bus account is not foreseen: %v", minted, err)
}
// Reading refuses nothing for it: the machine is not running it.
if _, err := Render(in, "laptop", Reading); err != nil {
t.Fatalf("Reading refused a secret the machine is not running: %v", err)
}
}
// A given secret under an old key.
in.OwnSecrets = []OwnSecretRow{{Node: "laptop", Module: "assistant", Name: "password", Origin: "accepted", SealedTo: "an-old-key", Sealed: secretValue},
{Node: "laptop", Module: "mesh-self-check", Name: "broker", Origin: "made", SealedTo: laptop.SealingKey, Sealed: secretValue}}
_, err := Render(in, "laptop", Foreseeing)
wantOld := "assistant on laptop holds \"password\", which was given to the mesh rather than made by it, and laptop has since generated a new " +
"sealing key. The mesh cannot make another; issue it again"
if err == nil || err.Error() != wantOld {
t.Fatalf("got %v, want %q", err, wantOld)
}
if strings.Contains(err.Error(), secretValue) {
t.Error("a secret value is in an error")
}
// A machine with no sealing key cannot be sealed to.
laptop.SealingKey = ""
in.OwnSecrets = nil
if _, err := Render(in, "laptop", Foreseeing); err == nil || !strings.Contains(err.Error(), "laptop has no sealing key, so nothing can be sealed to it") {
t.Fatalf("no sealing key: %v", err)
}
}
// aGrantMesh is the objects provider on anchor and its consumer files on laptop, with the pair credential made.
func aGrantMesh(t *testing.T) Input {
t.Helper()
in := theObjectsMesh(t, objects())
in.Issued = []Issued{{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "files", Provider: "anchor", ForProvider: "sealed-to-anchor"}}
return in
}
// A credential made for a consumer is a grant on its provider's machine, carrying the consumer's values and the sealed copy; without
// the sealed copy a stand-in is composed and named (novox/hq plan.go grantsFor).
func TestAGrantIsCreatedForEachCredentialMadeForAConsumer(t *testing.T) {
in := aGrantMesh(t)
// 1126 (ADR 0550 §5): the credentials each node's composition reads, by their ends: laptop's files consumes anchor's
// s3-bucket, and anchor grants it; the node composing its own asks the vault for exactly these.
pair := CredentialKey{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "files", Provider: "anchor"}
if consumer, provider, err := CredentialsNeeded(in, "laptop"); err != nil || len(provider) != 0 || len(consumer) != 1 || consumer[0] != pair {
t.Errorf("laptop needs %v as consumer and %v as provider (%v), want %v as consumer", consumer, provider, err, pair)
}
if consumer, provider, err := CredentialsNeeded(in, "anchor"); err != nil || len(consumer) != 0 || len(provider) != 1 || provider[0] != pair {
t.Errorf("anchor needs %v as consumer and %v as provider (%v), want %v as provider", consumer, provider, err, pair)
}
got := rendered(t, in, "anchor", Foreseeing)
if len(got.Rendering.Grants) != 1 {
t.Fatalf("grants are %+v", got.Rendering.Grants)
}
g := got.Rendering.Grants[0]
if g.Provision != "s3-bucket" || g.Consumer != "laptop" || g.From != "files" || g.At != "laptop.internal" || g.Sealed != "sealed-to-anchor" {
t.Errorf("the grant is %+v", g)
}
if len(got.Foreseen) != 0 || len(got.Withheld) != 0 || len(got.Unbound) != 0 {
t.Errorf("foreseen %v withheld %v unbound %v", got.Foreseen, got.Withheld, got.Unbound)
}
in.Issued[0].ForProvider = ""
got = rendered(t, in, "anchor", Foreseeing)
if got.Rendering.Grants[0].Sealed != catalogue.ForeseenSealed || len(got.Foreseen) != 1 {
t.Errorf("a grant with no sealed copy: %+v foreseen %v", got.Rendering.Grants, got.Foreseen)
}
// A credential made for a consumer on another provider is not this node's grant.
if len(rendered(t, in, "laptop", Foreseeing).Rendering.Grants) != 0 {
t.Error("the consumer is granted its own credential")
}
// A consumer whose set does not resolve is skipped, not fatal.
nodeOf(t, &in, "laptop").Assigned = append(nodeOf(t, &in, "laptop").Assigned, "files")
in.Catalogue["files"] = catalogue.Manifest{Module: "files", Version: "1", Requires: []string{"s3-bucket"},
Claims: []catalogue.Claim{{Name: "x", Scope: catalogue.ScopeNode}}}
in.Catalogue["twin"] = catalogue.Manifest{Module: "twin", Version: "1", Claims: []catalogue.Claim{{Name: "x", Scope: catalogue.ScopeNode}}}
nodeOf(t, &in, "laptop").Assigned = append(nodeOf(t, &in, "laptop").Assigned, "twin")
if got, err := Render(in, "anchor", Foreseeing); err != nil || len(got.Rendering.Grants) != 0 {
t.Errorf("a consumer that does not resolve: %v %+v", err, got.Rendering.Grants)
}
}
// 1364/AC2: a consumer the grants leave out because its identity overflows the provision's bound is returned beside them, naming the
// consumer and the provision (novox/hq ADR 0225), and is not granted.
func TestAConsumerWhoseIdentityOverflowsIsLeftOutOfTheGrantsAndNamed(t *testing.T) {
in := aGrantMesh(t)
in.Nodes[1].Name = "a-machine-with-a-very-long-name"
in.Issued[0].Consumer = "a-machine-with-a-very-long-name"
got := rendered(t, in, "anchor", Foreseeing)
if len(got.Rendering.Grants) != 0 || len(got.Withheld) != 1 {
t.Fatalf("grants %+v withheld %+v", got.Rendering.Grants, got.Withheld)
}
w := got.Withheld[0]
if w.Consumer != "a-machine-with-a-very-long-name" || w.Provision != "s3-bucket" || w.Provider != "anchor" || w.Module != "files" ||
!strings.Contains(w.String(), "a-machine-with-a-very-long-name") || !strings.Contains(w.String(), "s3-bucket") {
t.Errorf("the consumer left out is %+v / %s", w, w.String())
}
if len(got.Rendering.Withheld) != 1 {
t.Errorf("the Rendering does not carry it: %+v", got.Rendering.Withheld)
}
}
// 1364/AC3: a credential on record for a consumer that its own resolution binds to another provider is withdrawn, named with where it is
// bound (novox/hq issue 274): the grant stays, with no consumer module asking.
func TestACredentialBoundToAnotherProviderIsWithdrawnAndSaidWhereItIsBound(t *testing.T) {
in := aGrantMesh(t)
second := objects()
second.Module = "objects2"
in.Catalogue["objects2"] = second
in.Nodes = append(in.Nodes, Node{Name: "ace", Place: Place{Site: "there", Address: "10.77.0.3"},
Assigned: []string{overlay.Name, "objects2"}, Capabilities: map[string]bool{"container-runtime": true, "wireguard": true, "systemd": true}})
nodeOf(t, &in, "laptop").Pins = map[string]catalogue.Chosen{"s3-bucket": {Node: "ace", Module: "objects2"}}
got := rendered(t, in, "anchor", Foreseeing)
if len(got.Unbound) != 1 {
t.Fatalf("unbound %+v grants %+v", got.Unbound, got.Rendering.Grants)
}
u := got.Unbound[0]
if u.Provision != "s3-bucket" || u.Provider != "anchor" || u.Consumer != "laptop" || u.Module != "files" ||
len(u.BoundTo) != 1 || u.BoundTo[0] != "ace" || !strings.Contains(u.String(), "ace") {
t.Errorf("unbound is %+v / %s", u, u.String())
}
if len(got.Rendering.Grants) != 1 || got.Rendering.Grants[0].From != "" {
t.Errorf("the withdrawn credential is still asked for: %+v", got.Rendering.Grants)
}
}
func aCertificateMesh(t *testing.T) Input {
t.Helper()
in := aMesh(t)
in.Catalogue["secure"] = catalogue.Manifest{Module: "secure", Version: "1", Certificate: &catalogue.Certificate{Into: "/etc/secure/tls.pem", Authority: "/etc/secure/ca.pem"}}
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "secure")
return in
}
// 1364/AC4: with no certificate and authority in the Input a stand-in is composed and named; given them, the composition carries them as
// given. A machine that never reported a serving key, and one not on the private network, are refused in the controller's words.
func TestACertificateIsCarriedAsGivenOrStoodInForAndNamed(t *testing.T) {
in := aCertificateMesh(t)
laptop := nodeOf(t, &in, "laptop")
if _, err := Render(in, "laptop", Foreseeing); err == nil || err.Error() != "laptop wants a certificate and has never told the mesh what key it serves with; it joins again to report one" {
t.Fatalf("no serving key: %v", err)
}
laptop.ServingKey = "serving-key"
got := rendered(t, in, "laptop", Foreseeing)
if got.Rendering.Certificate != ForeseenCertificate || got.Rendering.Authority != ForeseenCertificate || len(got.Foreseen) != 1 {
t.Errorf("no certificate in the Input: %q %q foreseen %v", got.Rendering.Certificate, got.Rendering.Authority, got.Foreseen)
}
laptop.Certificate, in.Authority = "the-certificate", "the-authority"
got = rendered(t, in, "laptop", Foreseeing)
if got.Rendering.Certificate != "the-certificate" || got.Rendering.Authority != "the-authority" || len(got.Foreseen) != 0 {
t.Errorf("a certificate given: %q %q foreseen %v", got.Rendering.Certificate, got.Rendering.Authority, got.Foreseen)
}
laptop.Place.Address = "" // placed nowhere: no name inside the mesh
if _, err := Render(in, "laptop", Foreseeing); err == nil || err.Error() != "laptop wants a certificate and is not on the private network, so it has no name inside the mesh to be certified for" {
t.Fatalf("off the network: %v", err)
}
}
func aValidKey(t *testing.T, name string) string { return keyOf(t, name) }
// 1364/AC5 and AC6: a core credential the node needs and the Input holds no box for is composed with a stand-in and returned as one
// to keep; a kept box sealed to the key held now is carried; a node-engine below the vault-box contract is sent none (and nothing is
// asked of it); a machine with no sealing key has its core module left out, said.
func TestACoreBoxTheInputHoldsNoneOfIsReturnedToKeepAndNothingIsWritten(t *testing.T) {
mk := func() Input {
in := aMesh(t)
in.Catalogue["nats"] = catalogue.Manifest{Module: "nats", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/var/lib/mesh/nats/broker", Taken: catalogue.TakenAtStart}}}
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "nats")
laptop.SealingKey = aValidKey(t, "laptop")
laptop.Stated, laptop.Contract = true, VaultBoxContract
in.BusMinted = []string{"laptop.nats"}
return in
}
in := mk()
got := rendered(t, in, "laptop", Foreseeing)
box := got.Rendering.VaultSealed["nats"]["broker"]
if string(box) != string(catalogue.ForeseenVaultBox) || len(got.CoreBoxesWanted) != 1 ||
got.CoreBoxesWanted[0] != (CoreBoxWanted{Node: "laptop", Module: "nats", Name: "broker"}) {
t.Errorf("no box held: %s wanted %v", box, got.CoreBoxesWanted)
}
kept := json.RawMessage(`{"v":1,"eph":"e","iv":"i","tag":"t","ct":"c"}`)
in.KeptCoreBoxes = []CoreBox{{Node: "laptop", Module: "nats", Name: "broker", Generation: 3, SealedTo: nodeOf(t, &in, "laptop").SealingKey, Box: kept}}
got = rendered(t, in, "laptop", Foreseeing)
if string(got.Rendering.VaultSealed["nats"]["broker"]) != string(kept) || len(got.CoreBoxesWanted) != 0 {
t.Errorf("a kept box: %s wanted %v", got.Rendering.VaultSealed["nats"]["broker"], got.CoreBoxesWanted)
}
in.KeptCoreBoxes[0].SealedTo = "an-old-key"
if got = rendered(t, in, "laptop", Foreseeing); len(got.CoreBoxesWanted) != 1 {
t.Errorf("a box sealed to an old key is kept: %v", got.CoreBoxesWanted)
}
// Below the vault-box contract: none, and the controller's own seal as before.
nodeOf(t, &in, "laptop").Contract = VaultBoxContract - 1
if got = rendered(t, in, "laptop", Foreseeing); got.Rendering.VaultSealed != nil || len(got.CoreBoxesWanted) != 0 {
t.Errorf("an engine below the contract is sent boxes: %v %v", got.Rendering.VaultSealed, got.CoreBoxesWanted)
}
// No sealing key: the module is left out, said.
nodeOf(t, &in, "laptop").Contract, nodeOf(t, &in, "laptop").SealingKey = VaultBoxContract, ""
got = rendered(t, in, "laptop", Reading)
if why := got.Rendering.NotPlaced["nats"]; !strings.Contains(why, "laptop has no sealing key, so the vault can seal nats's credentials to nothing there") {
t.Errorf("not placed: %v", got.Rendering.NotPlaced)
}
unchanged(t, mk, func(in Input) { rendered(t, in, "laptop", Foreseeing) })
}
// 1364/AC1: the bus's user list is carried for the machine that runs the bus and for no other; without it a stand-in is composed and
// named; the users left out for want of a credential are said; each machine's membership is carried. (The list is composed by the
// controller from the bus records, which stay the controller's: see the commit message.)
func TestTheBusUserListIsCarriedForTheMachineThatRunsTheBus(t *testing.T) {
in := aMesh(t)
in.Catalogue["bus"] = catalogue.Manifest{Module: "bus", Version: "1", BusUsers: "/etc/bus/accounts.conf",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}}}
anchor := nodeOf(t, &in, "anchor")
anchor.Assigned = append(anchor.Assigned, "bus")
in.BusUsers = "accounts { anchor.bus: hash }"
in.BusMissing = []string{"laptop.files", "laptop.web"}
in.BusMemberships = map[string]string{"laptop": "sealed-membership"}
got := rendered(t, in, "anchor", Foreseeing)
if got.Rendering.BusUsers != in.BusUsers || len(got.Foreseen) != 0 {
t.Errorf("the list is %q foreseen %v", got.Rendering.BusUsers, got.Foreseen)
}
if len(got.Said) != 1 || got.Said[0] != "the bus's user list leaves out 2 user(s) the mesh has minted no credential for: laptop.files, laptop.web. Each is a user that cannot connect until one is issued" {
t.Errorf("said %v", got.Said)
}
if other := rendered(t, in, "laptop", Foreseeing); other.Rendering.BusUsers != "" || other.Rendering.BusMembership != "sealed-membership" {
t.Errorf("laptop: list %q membership %q", other.Rendering.BusUsers, other.Rendering.BusMembership)
}
in.BusUsers = ""
got = rendered(t, in, "anchor", Foreseeing)
if got.Rendering.BusUsers != ForeseenBusUsers || len(got.Foreseen) != 1 {
t.Errorf("no list in the Input: %q foreseen %v", got.Rendering.BusUsers, got.Foreseen)
}
}
// 1364/AC7: nothing the composition says carries a secret value the Input holds.
func TestNoSecretValueOfTheInputIsInWhatTheCompositionSays(t *testing.T) {
in := aGrantMesh(t)
in.Issued[0].ForProvider = secretValue
laptop := nodeOf(t, &in, "laptop")
in.Catalogue["assistant"] = anAssistant()
laptop.Assigned = append(laptop.Assigned, "assistant")
laptop.SealingKey = keyOf(t, "laptop")
in.OwnSecrets = []OwnSecretRow{{Node: "laptop", Module: "assistant", Name: "password", Origin: "made", SealedTo: laptop.SealingKey, Sealed: secretValue}}
in.Credentials = []Credential{{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "files", Provider: "anchor", State: CredentialHeld, Sealed: secretValue}}
for _, node := range []string{"anchor", "laptop"} {
got := rendered(t, in, node, Foreseeing)
for _, said := range append(append([]string{}, got.Foreseen...), got.Said...) {
if strings.Contains(said, secretValue) {
t.Errorf("%s: %q carries a secret value", node, said)
}
}
}
}
// A consumer no credential was made for yet is granted with a stand-in and named foreseen, for every form whose credential
// the vault mints or takes in; an offer of form none has no credential (issue 1126, 2026-11-02: mesh-cli on ace held for
// "the grant of mcp-endpoint", whose provider node-tools offers form none): its grant carries the consumer and no
// credential, nothing stands in on either side, and neither end asks the vault for one.
func TestAConsumerNoCredentialWasMadeForYetIsGrantedWithAStandInUnlessItsOfferHasNone(t *testing.T) {
for _, tc := range []struct {
form string
cred *catalogue.OfferCredential
}{
{"undeclared", nil},
{catalogue.CredentialFormPair, &catalogue.OfferCredential{Form: catalogue.CredentialFormPair}},
{catalogue.CredentialFormShared, &catalogue.OfferCredential{Own: "access-key"}},
{catalogue.CredentialFormNone, &catalogue.OfferCredential{Form: catalogue.CredentialFormNone}},
} {
in := aGrantMesh(t)
in.Issued = nil
provider := in.Catalogue["objects"]
provider.Provides[0].Credential = tc.cred
if tc.cred != nil && tc.cred.Own != "" {
provider.OwnSecrets = catalogue.OwnSecrets{"access-key": {Path: "/var/lib/mesh/objects/access-key"}}
}
in.Catalogue["objects"] = provider
none := tc.form == catalogue.CredentialFormNone
got := rendered(t, in, "anchor", Foreseeing)
if len(got.Rendering.Grants) != 1 {
t.Fatalf("%s: grants are %+v", tc.form, got.Rendering.Grants)
}
g, wantSealed := got.Rendering.Grants[0], catalogue.ForeseenSealed
if none {
wantSealed = ""
}
if g.Provision != "s3-bucket" || g.Consumer != "laptop" || g.From != "files" || g.Sealed != wantSealed {
t.Errorf("%s: the grant is %+v", tc.form, g)
}
if has(got.Foreseen, "the grant of s3-bucket for files on laptop") == none {
t.Errorf("%s: foreseen %v", tc.form, got.Foreseen)
}
consumer, asProvider, err := CredentialsNeeded(in, "laptop")
_, anchorProvides, err2 := CredentialsNeeded(in, "anchor")
if err != nil || err2 != nil || len(asProvider) != 0 || (len(consumer) == 0) != none || (len(anchorProvides) == 0) != none {
t.Errorf("%s: laptop asks %v, anchor provides %v (%v %v)", tc.form, consumer, anchorProvides, err, err2)
}
resolved, err := Resolve(in, "laptop")
if err != nil {
t.Fatalf("%s: %v", tc.form, err)
}
if n := theFilesNeed(t, resolved); (n.Sealed == "") != none || (len(resolved.Foreseen) == 0) != none {
t.Errorf("%s: the consumer's credential is %q, foreseen %v", tc.form, n.Sealed, resolved.Foreseen)
}
if lap := rendered(t, in, "laptop", Foreseeing); none && len(lap.Foreseen) != 0 {
t.Errorf("%s: the consumer stands in for %v", tc.form, lap.Foreseen)
}
// Only the provider's own: the consumer is granted nothing, and nothing in the Input changed.
if len(rendered(t, in, "laptop", Foreseeing).Rendering.Grants) != 0 {
t.Errorf("%s: the consumer is granted its own credential", tc.form)
}
if len(in.Issued) != 0 {
t.Errorf("%s: a row was added to the Input", tc.form)
}
// A row made long ago for an offer of form none (the controller's) carries no credential either.
in.Issued = []Issued{{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "files", Provider: "anchor", ForProvider: "sealed-to-anchor"}}
if g := rendered(t, in, "anchor", Foreseeing).Rendering.Grants; none && (len(g) != 1 || g[0].Sealed != "") {
t.Errorf("%s: an old row's grant is %+v", tc.form, g)
}
}
// A credential only a person gives and nobody has is no grant.
in := aGrantMesh(t)
in.Issued = nil
in.Credentials = []Credential{{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "files", Provider: "anchor", State: CredentialNotGiven}}
if got := rendered(t, in, "anchor", Foreseeing); len(got.Rendering.Grants) != 0 {
t.Errorf("a credential nobody gave is granted: %+v", got.Rendering.Grants)
}
}
// And a consumer whose identity overflows the provision's bound is judged before any credential is made for it (issue 263).
func TestAnOverflowingConsumerNoCredentialWasMadeForYetIsWithheld(t *testing.T) {
in := aGrantMesh(t)
in.Issued = nil
in.Catalogue["resolver"] = storeManifests()[1]
in.Catalogue["networkmanager"] = catalogue.Manifest{Module: "networkmanager", Version: "1", Requires: []string{"wildcard-resolution", "s3-bucket"}}
nodeOf(t, &in, "laptop").Assigned = append(nodeOf(t, &in, "laptop").Assigned, "networkmanager")
nodeOf(t, &in, "anchor").Assigned = append(nodeOf(t, &in, "anchor").Assigned, "resolver")
in.Holdings = []catalogue.Held{{Claim: "mesh-dns-resolver", Scope: catalogue.ScopeMesh, Node: "anchor", Module: "resolver", Site: "here"}}
got := rendered(t, in, "anchor", Foreseeing)
if len(got.Withheld) != 1 || got.Withheld[0].Module != "networkmanager" {
t.Fatalf("withheld %+v", got.Withheld)
}
}
// The grants come in the order the controller reads them, whatever order the Input gives its rows.
func TestGrantsComeInTheOrderOfProvisionConsumerModuleAndLocal(t *testing.T) {
in := aGrantMesh(t)
in.Issued = []Issued{
{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "files", Local: "b", Provider: "anchor", ForProvider: "x"},
{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "files", Local: "a", Provider: "anchor", ForProvider: "x"},
}
rows, err := newComposer(in).rowsFor("anchor")
// The credential the same send would make (no local name) comes first, then the two the Input gives, a before b.
if err != nil || len(rows) != 3 || rows[0].Local != "" || rows[1].Local != "a" || rows[2].Local != "b" {
t.Errorf("rows are %+v %v", rows, err)
}
}
// Round 1, finding 4: a member of a secret family a person gave, whose sealed value the Input does not carry, is placed with a
// stand-in and named foreseen, not dropped as if nobody gave it.
func TestAGivenFamilyMemberWithoutItsValueIsAStandInAndNamedForeseen(t *testing.T) {
in := aMesh(t)
in.Catalogue["mailer"] = catalogue.Manifest{Module: "mailer", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"credential-*": {Path: "/var/lib/mailer/credential-*.secret", Taken: catalogue.TakenAtStart, IssuedBy: catalogue.IssuedOutside}}}
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "mailer")
laptop.SealingKey = keyOf(t, "laptop")
in.OwnSecrets = []OwnSecretRow{
{Node: "laptop", Module: "mailer", Name: "credential-a", Origin: "accepted", SealedTo: laptop.SealingKey},
{Node: "laptop", Module: "mailer", Name: "credential-b", Origin: "accepted", SealedTo: laptop.SealingKey, Sealed: "b-sealed"},
}
got := rendered(t, in, "laptop", Foreseeing)
needed := got.Rendering.Needed["mailer"]
if needed["credential-a"] != catalogue.ForeseenSealed || needed["credential-b"] != "b-sealed" {
t.Errorf("needed %v", needed)
}
if !has(got.Foreseen, "mailer/credential-a") || has(got.Foreseen, "mailer/credential-b") {
t.Errorf("foreseen %v", got.Foreseen)
}
}
// Round 1, finding 5: the suffix internal names end in is a field of the Input, and the environment is never read.
func TestTheInternalSuffixIsTheInputsAndNeverTheEnvironments(t *testing.T) {
t.Setenv(overlay.SuffixVar, "from-the-environment")
in := aGrantMesh(t)
got := rendered(t, in, "anchor", Foreseeing)
if got.Rendering.Suffix != "internal" || got.Rendering.Grants[0].At != "laptop.internal" {
t.Errorf("by default: suffix %q, grant at %q", got.Rendering.Suffix, got.Rendering.Grants[0].At)
}
in.InternalSuffix = ".lan"
got = rendered(t, in, "anchor", Foreseeing)
if got.Rendering.Suffix != "lan" || got.Rendering.Grants[0].At != "laptop.lan" {
t.Errorf("given: suffix %q, grant at %q", got.Rendering.Suffix, got.Rendering.Grants[0].At)
}
where, err := newComposer(in).whereEveryoneIs()
if err != nil || where["anchor"] != "anchor.lan" {
t.Errorf("where everyone is: %v %v", where, err)
}
if names := newComposer(in).namesInTheMesh(); names["anchor.lan"] == "" || names["anchor.internal"] != "" {
t.Errorf("names: %v", names)
}
}
// Round 2: a need answered on the same machine is a grant too (catalogue resolve.go gives it From: the node's own name).
func TestAConsumerOnTheProvidersOwnNodeIsGrantedWithAStandInAndNamedForeseen(t *testing.T) {
in := aGrantMesh(t)
in.Issued = nil
laptop, anchor := nodeOf(t, &in, "laptop"), nodeOf(t, &in, "anchor")
laptop.Assigned = []string{overlay.Name}
anchor.Assigned = append(anchor.Assigned, "files")
got := rendered(t, in, "anchor", Foreseeing)
if len(got.Rendering.Grants) != 1 || got.Rendering.Grants[0].Consumer != "anchor" || got.Rendering.Grants[0].Sealed != catalogue.ForeseenSealed {
t.Fatalf("grants are %+v", got.Rendering.Grants)
}
if !has(got.Foreseen, "the grant of s3-bucket for files on anchor") {
t.Errorf("foreseen %v", got.Foreseen)
}
}
+90
View File
@@ -0,0 +1,90 @@
package compose
import (
"crypto/ecdh"
"crypto/rand"
"encoding/base64"
"encoding/json"
"reflect"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/overlay"
)
// aPublicKey is a key a machine could have reported, made as mesh-controller's mesh_for_test.go aPublicKey makes it.
func aPublicKey(t *testing.T) string {
t.Helper()
k, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
return base64.StdEncoding.EncodeToString(k.PublicKey().Bytes())
}
// keys are made once per run, so two Inputs made the same way are equal.
var keys = map[string]string{}
func keyOf(t *testing.T, name string) string {
t.Helper()
if keys[name] == "" {
keys[name] = aPublicKey(t)
}
return keys[name]
}
// theOverlayModule is the module the mesh provides for itself, as mesh-controller's providedModules makes it
// (cmd/mesh-controller/modules.go at 6c2a8d62).
func theOverlayModule(t *testing.T) catalogue.Manifest {
t.Helper()
var m catalogue.Manifest
b, err := json.Marshal(overlay.Manifest())
if err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
return m
}
// aMesh is the Input of mesh-controller's aMesh (cmd/mesh-controller/mesh_for_test.go at 6c2a8d62): two placed
// machines, `anchor` the hub, both reporting container-runtime, wireguard and systemd, both assigned what puts them on the
// private network, each at <name>.example:51820 on the site `here`, anchor at 10.77.0.1 and laptop at 10.77.0.2.
func aMesh(t *testing.T) Input {
t.Helper()
in := Input{Catalogue: map[string]catalogue.Manifest{overlay.Name: theOverlayModule(t)}}
for i, name := range []string{"anchor", "laptop"} {
in.Nodes = append(in.Nodes, Node{
Name: name,
Place: Place{Key: keyOf(t, name), Endpoint: name + ".example:51820", Site: "here",
Hub: name == "anchor", Address: []string{"10.77.0.1", "10.77.0.2"}[i]},
Assigned: []string{overlay.Name},
Capabilities: map[string]bool{"container-runtime": true, "wireguard": true, "systemd": true},
})
}
return in
}
// nodeOf is the named node of an Input, for a test to change.
func nodeOf(t *testing.T, in *Input, name string) *Node {
t.Helper()
for i := range in.Nodes {
if in.Nodes[i].Name == name {
return &in.Nodes[i]
}
}
t.Fatalf("no node %s", name)
return nil
}
// unchanged runs call on one Input and compares it with another made the same way: the composition changes nothing in what
// it is given (deep equality, nothing shared).
func unchanged(t *testing.T, mk func() Input, call func(Input)) {
t.Helper()
used, same := mk(), mk()
call(used)
if !reflect.DeepEqual(used, same) {
t.Fatalf("the call changed its Input")
}
}
+583
View File
@@ -0,0 +1,583 @@
package frombus
import (
"context"
"errors"
"fmt"
"maps"
"regexp"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose"
"github.com/novox/mesh-host/internal/compose/resolution"
)
// Answer is a node's answer to its proposal's placement candidates (novox/hq ADR 0567 §3, ADR 0559 §4): s.Node composed in shadow with
// s.Placements laid over the trunk's placement (compose.CompositionsShadow, its own sealed values and boxes through r as
// for its own composition), compared with what it runs (delivery.CompareComposed). It applies nothing and writes
// nothing; the proposal and the revision are the candidate's, said back in the answer.
//
// The node's own composition is compared only where the node runs one. fps gives the build each module is composed at: what the node runs, and for a module the candidate places that the
// node does not run, the build the trunk holds; a placed module with none is added, with no build of it known. What the
// composition stands in for is judged: a value of a module the proposal adds (on this node or another: its credential, own
// secrets and boxes are made only once it is placed, ADR 0549 §5) is not yet issued, which is no refusal; any other
// stand-in is a value missing for a module the node runs, a refusal naming it ("node" when it is the node's own).
func Answer(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]string, running []delivery.Composed, proposal string, revision uint64) delivery.CandidateAnswer {
a := delivery.CandidateAnswer{Proposal: proposal, Revision: revision}
refuse := func(err error) delivery.CandidateAnswer {
a.Refused = err.Error()
a.Verdict = delivery.CandidateRefused
return a
}
trunk, err := r.Placements(ctx)
if err != nil {
return refuse(fmt.Errorf("frombus: placement: %w", err))
}
if s.Running == nil {
s.Running = running
}
// Every module it cannot compose is named, not the first only (issue 1126): a build it cannot read or take in, and
// below, a module the composition refuses, each composed as a stub so the rest is composed and compared.
failed := map[string]string{}
in, err := input(ctx, r, mesh, s, func(module string, err error) { failed[module] = err.Error() }, nil)
if err != nil {
return refuse(err)
}
asConsumer, asProvider, err := compose.CredentialsNeeded(in, s.Node)
if err != nil {
return refuse(err)
}
// What the node runs, by module. A consumer's credential is asked only for a module it runs: the vault refuses the
// whole ask for a consuming module not assigned to the node (mesh-vault's own), and a module the candidate adds is
// assigned only after the merge (ADR 0549 §5), so its credential is left missing, a stand-in answered as not yet
// issued under Added (review of L2b, round 1, finding 1).
runs := map[string]bool{}
for _, c := range running {
if c.Module != "" {
runs[c.Module] = true
}
}
var consumed []compose.CredentialKey
for _, k := range asConsumer {
if runs[k.ConsumerModule] {
consumed = append(consumed, k)
}
}
asConsumer = consumed
if len(asConsumer)+len(asProvider) > 0 {
if in.Credentials, in.Issued, err = r.Credentials(ctx, s.Node, asConsumer, asProvider); err != nil {
return refuse(fmt.Errorf("frombus: the credentials of %s: %w", s.Node, err))
}
}
composed, skipped, stood, err := composeEach(in, s.Node, fps, false, s.Running, failed, ranOf(ctx, r, mesh, s.Running, fps))
if err != nil {
// Not one module's: the node's own (a value served elsewhere, the mesh's network), said once as the node's.
if len(failed) == 0 {
return refuse(err)
}
failed[delivery.HaveNodeWord] = err.Error()
composed, skipped, stood = nil, nil, nil
}
// The node's own composition is judged only where the node runs one: one that runs none applies it only once a want
// names it (the node agent's own composer hands it on that want alone), so a candidate does not add it.
runsOwn := false
for _, c := range running {
runsOwn = runsOwn || c.Module == ""
}
if !runsOwn {
for key, c := range composed {
if c.Module == "" {
delete(composed, key)
}
}
}
if _, whole := failed[delivery.HaveNodeWord]; whole {
// Nothing composed to compare: every refusal is said, and nothing else.
for _, m := range sortedKeys(failed) {
a.Refusals = append(a.Refusals, delivery.ModuleRefused{Module: m, Why: failed[m]})
}
heldAsToday(&a, s.HeldToday)
a.Verdict = delivery.VerdictOf(a)
return a
}
// A module the composition folds inside the node's own is judged there: the agent also says it runs by module, from
// the declaration it kept before the switch (agent.Everything), which is no removal (issue 1126, seed #531).
if len(s.InOwn) > 0 {
var judged []delivery.Composed
for _, c := range running {
if c.Module == "" || !slices.Contains(s.InOwn, c.Module) {
judged = append(judged, c)
}
}
running = judged
}
cmp := delivery.CompareComposedWith(composed, running, s.SameValue)
// The answer rides in the node's have key: its shown bodies are bounded, the rest named only.
budget := MaxShown
for i := range cmp.Differs {
kept := cmp.Differs[i].Bodies[:0]
for _, b := range cmp.Differs[i].Bodies {
if n := len(b.Composed) + len(b.Running); n <= budget {
budget -= n
kept = append(kept, b)
} else {
kept = append(kept, delivery.ResourceBodies{ID: b.ID, Why: b.Why})
a.Note = "some differing resources are named without their bodies: the answer holds at most " +
fmt.Sprint(MaxShown) + " bytes of them"
}
}
cmp.Differs[i].Bodies = kept
}
// A refused module was composed as a stub: what it composed is no difference; its refusal is said.
notFailed := func(m string) bool { _, ok := failed[m]; return !ok }
a.Same = slices.DeleteFunc(cmp.Same, func(m string) bool { return !notFailed(m) })
a.Removed = slices.DeleteFunc(cmp.Removed, func(m string) bool { return !notFailed(m) })
a.Differs = slices.DeleteFunc(cmp.Differs, func(d delivery.ModuleDiffers) bool { return !notFailed(d.Module) })
cmp.Added = slices.DeleteFunc(cmp.Added, func(d delivery.ModuleAdded) bool { return !notFailed(d.Module) })
skipped = slices.DeleteFunc(skipped, func(m string) bool { return !notFailed(m) })
// What the proposal adds, by node and module: placed by a candidate and not by the trunk on that node.
added := map[string]bool{}
for node, p := range s.Placements {
if p == nil {
continue
}
was := map[string]bool{}
for _, m := range trunk[node].Modules {
was[m.Module] = true
}
for _, m := range p.Modules {
if !was[m.Module] {
added[node+"/"+m.Module] = true
}
}
}
unissued := map[string][]string{}
refusals := map[string][]string{}
for _, x := range stood {
node, module := standsInFor(x, s.Node)
switch {
case node == s.Node && runs[module]:
// A value missing for a module the node runs is a refusal, whatever the trunk places (the seed's trunk is
// empty, and every module counts as the group's).
refusals[module] = append(refusals[module], x)
case module != "" && added[node+"/"+module]:
if node == s.Node {
unissued[module] = append(unissued[module], x)
} // another node's addition: the module here that carries it differs, which the assigner judges
case module == "" || node != s.Node:
// The node's own, or a value this node holds for a module elsewhere (a grant it provides): said as the node's.
refusals[delivery.HaveNodeWord] = append(refusals[delivery.HaveNodeWord], x)
default:
refusals[module] = append(refusals[module], x)
}
}
addedHere := map[string]delivery.ModuleAdded{}
for _, m := range cmp.Added {
addedHere[m.Module] = m
}
for _, m := range skipped { // placed, and no build given to compose it at
addedHere[m] = delivery.ModuleAdded{Module: m, Unissued: []string{"no build of " + m + " is known to compose it at"}}
}
for m, xs := range unissued {
got := addedHere[m]
got.Module = m
got.Unissued = append(got.Unissued, xs...)
addedHere[m] = got
}
for _, m := range sortedKeys(addedHere) {
a.Added = append(a.Added, addedHere[m])
}
for _, m := range sortedKeys(refusals) {
if _, isAdded := addedHere[m]; isAdded {
continue // a module the candidate adds here: what it lacks is said under Added
}
if _, isFailed := failed[m]; isFailed {
continue // said below, as why it could not be composed
}
a.Refusals = append(a.Refusals, delivery.ModuleRefused{Module: m, Why: "stands in for " + strings.Join(refusals[m], ", ")})
}
for _, m := range sortedKeys(failed) {
a.Refusals = append(a.Refusals, delivery.ModuleRefused{Module: m, Why: failed[m]})
}
sort.Slice(a.Refusals, func(i, j int) bool { return a.Refusals[i].Module < a.Refusals[j].Module })
heldAsToday(&a, s.HeldToday)
a.Verdict = delivery.VerdictOf(a)
return a
}
// heldAsToday moves into Held each refusal of a module the node keeps today as it runs (Scope.HeldToday, with how:
// held, failed, running a build of the old form): the candidate keeps it as the node does, which is no change (issue
// 1126: a seed reproduces today, and today keeps them so). Each says how it stands today (ModuleRefused.Today). No
// module is held behind another (issue 1960): a module that reads from a held one composes with what it last offered.
func heldAsToday(a *delivery.CandidateAnswer, today map[string]string) {
if len(today) == 0 {
return
}
kept := a.Refusals[:0]
for _, r := range a.Refusals {
if w, keep := today[r.Module]; keep {
r.Today = w
a.Held = append(a.Held, r)
} else {
kept = append(kept, r)
}
}
a.Refusals = kept
if len(a.Refusals) == 0 {
a.Refusals = nil
}
}
// maxCulprits bounds the modules one refusal is tried on, each a whole composition.
const maxCulprits = 8
// MaxShown bounds the bytes of shown resource bodies in one answer (delivery.ModuleDiffers.Bodies).
const MaxShown = 256 << 10
var (
grantStandIn = regexp.MustCompile(`^the grant of \S+ for (\S+) on (\S+)$`)
// grantOfStandIn is the provision a grant's stand-in is of.
grantOfStandIn = regexp.MustCompile(`^the grant of (\S+) for \S+ on \S+$`)
needsStandIn = regexp.MustCompile(`^(\S+) on (\S+) needs `)
)
// standsInFor is the node and module a stand-in of compose's words is for: an own secret `module/name`, a port
// `port module:port`, a grant `the grant of <provision> for <module> on <node>`, a need `<module> on <node> needs ...`;
// the module empty when the words name none (the certificate, the authority).
func standsInFor(x, node string) (string, string) {
if m := grantStandIn.FindStringSubmatch(x); m != nil {
return m[2], m[1]
}
if m := needsStandIn.FindStringSubmatch(x); m != nil {
return m[2], m[1]
}
if rest, ok := strings.CutPrefix(x, "port "); ok {
if module, _, ok := strings.Cut(rest, ":"); ok {
return node, module
}
}
if module, _, ok := strings.Cut(x, "/"); ok && !strings.Contains(x, " ") {
return node, module
}
return node, ""
}
// composeEach is the one rule, for the node that applies and for its answer to a candidate alike, by which a module's
// own refusal refuses that module and never the node (issue 1126: seed-8 fixed it in the answer alone, and the
// applying path still refused all of novox for de-spiegel's unset route names). The node is composed; a refusal is laid
// on the module whose stub (compose.Stub) changes it (culprit), that module is held (compose.Node.Held) and said in
// failed with the refusal's words, and the rest is composed again, until it composes or the refusal is no single
// module's, which is answered as the node's (err). Applied composes as the node applies (compose.Reading) and holds the
// module placed here each stand-in is for; one for no module placed here (the node's own, a value this node holds for a
// module elsewhere) refuses the node (compose.ErrStoodIn). Not applied, it composes in compose.Foreseeing and answers
// every stand-in for the caller to judge (Answer).
//
// **A held module is what runs of it** (review of the apply, round 1): composed from what running holds of it
// (compose.Carried of its own composition there; the node's own carries it wherever it runs there), so what the others
// read of it (the machine ports its containers publish, the routes reaching it) is what runs, and none of its own
// composition is in out: the node keeps what it last applied.
//
// **A hold never spreads** (issue 1126, issue 1960: de-spiegel's hold for an unset value held mailu, which held
// route-proxy, nftables and postgres on novox). No module is held because a module it reads from is held: a consumer,
// a collector, the provider of what the held one contributes to and the filter where it listens each compose with what
// the held module last offered. That offer (what it contributes, contributes many, listens on) is its manifest at the
// build it runs where one is known (ran: a composition of its own running at another build than it is composed at),
// else the manifest it is composed from. Only the part of the offer that cannot compose is left out, by provision
// (culprit): de-spiegel's routes, its names unset, leave route-proxy without them while mailu keeps its smtp account,
// and the hold says so.
func composeEach(in compose.Input, node string, fps map[string]string, applied bool, running []delivery.Composed, failed map[string]string,
ran func(module string) (catalogue.Manifest, bool)) (out map[string]delivery.Composed, skipped, stood []string, err error) {
mode := compose.Foreseeing
if applied {
mode = compose.Reading
}
runs := map[string]*delivery.Composed{}
for i := range running {
if m := running[i].Module; m != "" {
runs[m] = &running[i]
}
}
src := map[string]catalogue.Manifest{} // each held module's manifest before it was held, offering what it last offered
left := map[string]map[string]bool{} // by held module, what of its offer is left out (leaveOut)
hold := func(m, why, leave string) {
if _, done := failed[m]; !done {
failed[m] = why // a held module tried again keeps the words it was first held for
}
if _, ok := src[m]; !ok {
man, ok := manifestOn(in, node, m)
if !ok {
return
}
if ran != nil {
if was, ok := ran(m); ok {
man.Contributes, man.ContributesMany, man.Listens = was.Contributes, was.ContributesMany, was.Listens
}
}
src[m] = man
}
if leave != "" {
if left[m] == nil {
left[m] = map[string]bool{}
}
left[m][leave] = true
if leave != leaveAll && leave != leaveOwn {
failed[m] += "; composed without what it contributes to " + leave + ", which does not compose"
}
}
in = heldAs(in, node, m, src[m], left[m], runs[m])
}
for _, m := range sortedKeys(failed) {
hold(m, failed[m], "")
}
for {
out, skipped, stood, err = compose.CompositionsIn(in, node, fps, mode)
if err != nil {
module, leave := culprit(in, node, fps, mode, failed, err)
if module == "" {
return nil, nil, nil, firstReason(err, failed)
}
hold(module, err.Error(), leave)
continue
}
again := false
if applied && len(stood) > 0 {
placed := map[string]bool{}
for _, m := range assignedOn(in, node) {
placed[m] = true
}
of := map[string][]string{}
var nodes []string
for _, x := range stood {
if n, m := standsInFor(x, node); n == node && placed[m] {
of[m] = append(of[m], x)
} else {
nodes = append(nodes, x)
}
}
for _, m := range sortedKeys(of) {
if _, done := failed[m]; !done {
hold(m, "stands in for "+strings.Join(of[m], ", "), "")
again = true
continue
}
// Already held, and still a stand-in in another module's composition: a grant not yet issued for what
// it contributes (a module held for its own secret keeps its contributions, leaveOwn). Its contribution
// to that provision is left out, so the provider never applies a stand-in (issue 774, review round 1).
for _, x := range of[m] {
if g := grantOfStandIn.FindStringSubmatch(x); g != nil && !left[m][g[1]] {
hold(m, failed[m], g[1])
again = true
}
}
}
if !again && len(nodes) > 0 {
return nil, nil, nil, fmt.Errorf("%w on %s: %s", compose.ErrStoodIn, node, strings.Join(nodes, ", "))
}
}
if !again {
break
}
}
skipped = slices.DeleteFunc(skipped, func(m string) bool { _, held := failed[m]; return held })
return out, skipped, stood, nil
}
// firstReason is err, a refusal no culprit is found for, led by the first reason of each held module it names: a held
// module's stand-in refused later never replaces why it was held (issue 774, 2026-10-30: mesh-self-check on ace held for
// "has no bus credential", then refused as "runs as its own account and declares no own secret broker", its bare stub's
// words, which were all the proposal's refusal said).
func firstReason(err error, failed map[string]string) error {
var named []string
for _, m := range sortedKeys(failed) {
if strings.Contains(err.Error(), m) {
named = append(named, m+" is held: "+failed[m])
}
}
if len(named) == 0 {
return err
}
return fmt.Errorf("%s; its stand-in is then refused: %w", strings.Join(named, "; "), err)
}
// blames says the composition with m's stub (again) makes m the culprit of err: it composes, or fails otherwise. The
// runtime's stub is never a runtime (no build is taken in, so it carries no bundle) and always fails otherwise: it is
// the culprit only where its stub composes (issue 774, 2026-10-30 19:30: node-tools held for any error naming it).
func blames(m string, again, err error) bool {
if m == catalogue.RuntimeModule {
// Or where its stub fails only for the modules of their own account it would run (issue 774): they are held
// after it, each for its own words, and the node composes the rest.
var none *resolution.NoRuntimeToRunError
return again == nil || errors.As(again, &none) && none.Held
}
return again == nil || again.Error() != err.Error()
}
// leaveAll leaves out the whole of a held module's offer and its own secrets (a bare stub); leaveOwn its own secrets
// alone, its offer kept; any other word leaves out what it contributes to that provision alone.
//
// leaveOwn is tried before leaveAll: a module held for an own secret never made (nextcloud's admin, issue 774,
// 2026-10-30) keeps what it contributes, so its providers' received files keep it and their helpers never retire its
// login while it is held.
const (
leaveAll = "*"
leaveOwn = "own-secrets"
)
// assignedOn is the modules placed on node, in placement order.
func assignedOn(in compose.Input, node string) []string {
for _, n := range in.Nodes {
if n.Name == node {
return n.Assigned
}
}
return nil
}
// manifestOn is the manifest node composes module from: its Kept, else the catalogue's.
func manifestOn(in compose.Input, node, module string) (catalogue.Manifest, bool) {
for _, n := range in.Nodes {
if n.Name == node {
if man, ok := n.Kept[module]; ok {
return man, true
}
}
}
man, ok := in.Catalogue[module]
return man, ok
}
// culprit is the module placed on node whose composition the error is of, and what of its offer to leave out: the
// first, in placement order, whose stub (compose.Stub) makes the composition in mode succeed or fail otherwise (leave
// ""); else the first whose stub without what it contributes to one provision does (leave that provision: de-spiegel's
// routes, its names unset, and not its smtp account, issue 1960); else the first whose bare stub does (leaveAll: its
// stub contributing nothing, its own secret never issued, issue 1126 seed-8). Empty when none does (the error is not
// one module's). Modules already refused are passed over for their stub.
func culprit(in compose.Input, node string, fps map[string]string, mode compose.Mode, failed map[string]string, err error) (string, string) {
assigned := assignedOn(in, node)
// Only the modules the error names: the words of a module's refusal name it. Each try is a whole composition, so a
// refusal naming none is no module's, answered at once rather than after a composition per module (issue 1126: ace's
// answer took 7m46s).
var order []string
for _, m := range assigned {
if strings.Contains(err.Error(), m) && len(order) < maxCulprits {
order = append(order, m)
}
}
changes := func(m, leave string) bool {
_, _, _, again := compose.CompositionsIn(stubbed(in, node, m, leave), node, fps, mode)
return blames(m, again, err)
}
for _, m := range order {
if _, done := failed[m]; done {
continue
}
if changes(m, "") {
return m, ""
}
}
for _, m := range order {
man, ok := manifestOn(in, node, m)
if !ok {
continue
}
// One already held is tried again too: held for its build, its stub may still refuse (its own secret never
// issued, issue 1126). A provision named in the error is tried first, each at one more composition.
var to []string
for p := range man.Contributes {
to = append(to, p)
}
for p := range man.ContributesMany {
if _, both := man.Contributes[p]; !both {
to = append(to, p)
}
}
sort.SliceStable(to, func(i, j int) bool {
return strings.Contains(err.Error(), " "+to[i]) && !strings.Contains(err.Error(), " "+to[j])
})
for _, p := range to {
if changes(m, p) {
return m, p
}
}
if isBare(man) {
continue
}
if len(man.OwnSecrets) > 0 && changes(m, leaveOwn) {
return m, leaveOwn
}
if changes(m, leaveAll) {
return m, leaveAll
}
}
return "", ""
}
// isBare says a stub contributes nothing and declares no own secret (stubbed, leaveAll).
func isBare(m catalogue.Manifest) bool {
return len(m.Contributes) == 0 && len(m.ContributesMany) == 0 && len(m.OwnSecrets) == 0
}
// stubbed is in with module held on node as its stub alone (heldAs, nothing of it running), from the manifest node
// composes it from, leave left out of its offer: a culprit's try.
func stubbed(in compose.Input, node, module, leave string) compose.Input {
man, ok := manifestOn(in, node, module)
if !ok {
return in
}
var left map[string]bool
if leave != "" {
left = map[string]bool{leave: true}
}
return heldAs(in, node, module, man, left, nil)
}
// heldAs is in with module held on node (compose.Node.Held): composed from man as what runs of it (compose.Carried of
// runs; its stub, compose.Stub, when runs is nil or holds nothing of it), as node's Kept; with what left names out of
// its offer: a provision's contributions, with leaveOwn its own secrets alone, or with leaveAll everything it contributes
// and its own secrets, none of a held module's own composition being applied, so what it has not been issued (restic's
// repository on a node it never ran on) is no refusal of the rest.
func heldAs(in compose.Input, node, module string, man catalogue.Manifest, left map[string]bool, runs *delivery.Composed) compose.Input {
out := in
out.Nodes = append([]compose.Node(nil), in.Nodes...)
for i := range out.Nodes {
if out.Nodes[i].Name != node {
continue
}
if left[leaveAll] {
man.Contributes, man.ContributesMany, man.OwnSecrets = nil, nil, nil
} else if len(left) > 0 {
if left[leaveOwn] {
man.OwnSecrets = nil
}
contributes := map[string]map[string]any{}
for p, v := range man.Contributes {
if !left[p] {
contributes[p] = v
}
}
many := maps.Clone(man.ContributesMany)
for p := range left {
delete(many, p)
}
man.Contributes, man.ContributesMany = contributes, many
}
kept := make(map[string]catalogue.Manifest, len(out.Nodes[i].Kept)+1)
for k, v := range out.Nodes[i].Kept {
kept[k] = v
}
kept[module] = compose.Carried(man, module, runs)
out.Nodes[i].Kept = kept
held := make(map[string]bool, len(out.Nodes[i].Held)+1)
for k, v := range out.Nodes[i].Held {
held[k] = v
}
held[module] = true
out.Nodes[i].Held = held
}
return out
}
@@ -0,0 +1,596 @@
package frombus
import (
"context"
"encoding/json"
"fmt"
"os"
"reflect"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose"
"github.com/novox/mesh-host/internal/compose/resolution"
"github.com/novox/mesh-host/internal/overlay"
)
// Issue 1126 (the seed's shadow, round 2): a node's answer names every module it cannot compose, not the first only;
// and a module folded into its own composition that names a build of the old form (de-spiegel on novox, nats, photos,
// systemd-resolved: their haves name a build-... id and no fingerprint) is left to the node, carried as it runs.
func answerNamesEveryRefusalAndLeavesAnOldBuildToTheNode(t *testing.T) {
ctx := context.Background()
image := "alpine@sha256:" + strings.Repeat("a", 64)
manifest := func(module string) []byte {
// Its data names its directory, as nats's names ${dir:mesh-state} (issue 1126: novox's own composition refused
// "nats says ${dir:mesh-state}, and nats declares no directory mesh-state" when its stub held no directory).
return []byte(`{"module":"` + module + `","version":"1","capabilities":["container-runtime"],` +
`"own-secrets":{"token":"${dir:state}/token"},` +
`"resources":[{"id":"state","type":"directory","path":"/var/lib/` + module + `","mode":"0700"},` +
`{"id":"server","type":"container","name":"` + module + `","artifact":"site"}],` +
`"build":{"artifacts":[{"name":"site","kind":"upstream","from":"` + image + `"}]}}`)
}
lemurs := []byte(`{"module":"lemurs","version":"1","resources":[{"id":"service","type":"service","unit":"lemurs.service","state":"running","boot":"enabled"}]}`)
fp := func(c string) string { return "src2:" + strings.Repeat(c, 64) }
mk := func(placed ...string) *bus {
p := delivery.Placement{Node: "n1", Decided: delivery.NodeDecided{Hub: true, Address: "10.77.0.1"}}
for _, m := range placed {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
b := &bus{placements: map[string]delivery.Placement{"n1": p},
records: map[string]delivery.NodeRecord{"n1": {Node: "n1", TunnelKey: publicKey(t), SealingKey: "sk", Capabilities: []string{"systemd", "container-runtime"}}},
modules: map[string]delivery.Module{"lemurs": {Manifest: delivery.ManifestDigest(lemurs)}},
manifests: map[string][]byte{delivery.ManifestDigest(lemurs): lemurs},
builds: map[string]*delivery.BuilderBuild{}}
for _, m := range []string{"web", "api", "spiegel"} {
b.own = append(b.own, compose.OwnSecretRow{Node: "n1", Module: m, Name: "token", Origin: "made", SealedTo: "sk", Sealed: `"box"`})
}
for _, m := range []string{"web", "api", "spiegel"} {
raw := manifest(m)
b.manifests[delivery.ManifestDigest(raw)] = raw
b.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
// web and api are built, their keys done and waiting for the reference of their image; spiegel has no build of
// the new form at all.
for i, m := range []string{"web", "api"} {
mod := b.modules[m]
mod.FP2 = fp(string(rune('1' + i)))
b.modules[m] = mod
b.builds[m+" "+mod.FP2] = &delivery.BuilderBuild{State: delivery.BuildDone, Manifest: mod.Manifest,
Artifacts: []delivery.BuiltArtifact{{Name: "site", Kind: "image", Digest: "sha256:" + strings.Repeat("b", 64)}}}
}
return b
}
given := func(b *bus) map[string]*delivery.Placement {
p := b.placements["n1"]
return map[string]*delivery.Placement{"n1": &p}
}
// Every refusal: two builds taken in, and a module the composition refuses (its image never built), each named.
b := mk("lemurs", "web", "api", "spiegel")
a := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b)},
map[string]string{"lemurs": fp("e"), "web": fp("1"), "api": fp("2"), "spiegel": fp("3")}, nil, "p-h1", 3)
var refused []string
for _, r := range a.Refusals {
refused = append(refused, r.Module)
}
if a.Verdict != delivery.CandidateRefused || a.Refused != "" || !reflect.DeepEqual(refused, []string{"api", "spiegel", "web"}) {
t.Errorf("every refusal: %+v, want api, spiegel and web refused, each named", a)
}
for _, r := range a.Refusals {
want := map[string]string{"api": "waits for the reference of site", "web": "waits for the reference of site",
"spiegel": "spiegel has not been built for this mesh"}[r.Module]
if !strings.Contains(r.Why, want) {
t.Errorf("%s refused as %q, want %q", r.Module, r.Why, want)
}
}
if raw, err := delivery.EncodeHaveNode(delivery.HaveNode{Candidates: map[string]delivery.CandidateAnswer{"g": a}}); err != nil {
t.Errorf("the answer is not one a have holds: %v (%s)", err, raw)
}
if lemursOK := slices.ContainsFunc(a.Added, func(m delivery.ModuleAdded) bool { return m.Module == "lemurs" }); !lemursOK {
t.Errorf("lemurs, which composes, is not answered beside the refusals: %+v", a)
}
// Issue 1126: a module the node holds today that the candidate holds too is held as today, no refusal: with every
// refused module held today the verdict is no refusal, and each is named under Held.
today := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), HeldToday: map[string]string{"api": "held", "spiegel": "held", "web": "held"}},
map[string]string{"lemurs": fp("e"), "web": fp("1"), "api": fp("2"), "spiegel": fp("3")}, nil, "p-h1", 3)
if today.Verdict == delivery.CandidateRefused || len(today.Refusals) != 0 || len(today.Held) != 3 || today.Held[0].Why == "" {
t.Errorf("held as today: %+v", today)
}
// A refused module the node does not hold today stays a refusal: holding only api today leaves spiegel and web refused.
part := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), HeldToday: map[string]string{"api": "held"}},
map[string]string{"lemurs": fp("e"), "web": fp("1"), "api": fp("2"), "spiegel": fp("3")}, nil, "p-h1", 3)
if part.Verdict != delivery.CandidateRefused || len(part.Refusals) != 2 || len(part.Held) != 1 || part.Held[0].Module != "api" {
t.Errorf("held as today in part: %+v", part)
}
// No module is held behind one kept today (issue 1960): mailu's own refusal stays a refusal beside de-spiegel's hold.
if a := (delivery.CandidateAnswer{Refusals: []delivery.ModuleRefused{{Module: "de-spiegel", Why: "a value is unset"},
{Module: "mailu", Why: "no build"}}}); true {
heldAsToday(&a, map[string]string{"de-spiegel": "runs a build of the old form"})
if len(a.Refusals) != 1 || a.Refusals[0].Module != "mailu" || len(a.Held) != 1 {
t.Errorf("held as today spreads: %+v", a)
}
}
// The fold: spiegel runs inside the node's own composition at a build of the old form. Its resources are carried as
// they run, each after the one it runs after: the node's own is the one it runs, and the node composes itself.
b = mk("lemurs", overlay.Name, "spiegel")
p := b.placements["n1"]
p.Decided.Endpoint = "n1.example:51820"
b.placements["n1"] = p
fps := map[string]string{"lemurs": fp("e")}
stub := Scope{Node: "n1"}
in, err := Input(ctx, b, Mesh{}, stub)
if err != nil {
t.Fatal(err)
}
in.Nodes[0].Kept = map[string]catalogue.Manifest{"spiegel": compose.Stub(in.Catalogue["spiegel"])}
base, _, err := compose.Compositions(in, "n1", fps, compose.Reading)
if err != nil {
t.Fatal(err)
}
// A refusal naming no module placed here is no module's: no compose is spent stubbing each in turn (issue 1126: ace's
// answer took 7m46s, one compose per module), though here a stub would compose.
if m, _ := culprit(in, "n1", fps, compose.Foreseeing, map[string]string{}, fmt.Errorf("nothing in this mesh provides \"artifact-store\"")); m != "" {
t.Errorf("a refusal naming no placed module was laid on %s", m)
}
ownKey, _ := delivery.ComposedNodeKey("n1")
own := base[ownKey]
carriedRes := []json.RawMessage{
json.RawMessage(`{"id":"spiegel.state","type":"directory","path":"/var/lib/spiegel","mode":"0700"}`),
json.RawMessage(`{"id":"spiegel.server","type":"container","name":"spiegel","image":"registry.example/spiegel@sha256:` + strings.Repeat("c", 64) + `"}`)}
// They run after the node's first own resource.
own.Resources = append(append(append([]json.RawMessage{}, own.Resources[:1]...), carriedRes...), own.Resources[1:]...)
var running []delivery.Composed
for k, c := range base {
if k == ownKey {
c = own
}
running = append(running, c)
}
scope := Scope{Node: "n1", InOwn: []string{"spiegel"}, Running: running}
got, _, held, err := Compose(ctx, b, Mesh{}, scope, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatalf("the node does not compose itself with spiegel folded at a build of the old form: %v", err)
}
gotOwn, _ := json.Marshal(got[ownKey].Resources)
wantOwn, _ := json.Marshal(own.Resources)
if string(gotOwn) != string(wantOwn) {
t.Errorf("the node's own composition is\n%s\nwant what runs\n%s", gotOwn, wantOwn)
}
a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, running, "p-h1", 3)
if a.Verdict != delivery.CandidateSame {
t.Errorf("the fold's answer: %+v, want the same", a)
}
// (A) The agent also says spiegel runs by module, from the declaration kept before the switch (agent.Everything): a
// module the composition folds is judged inside the node's own, not as removed (issue 1126, seed #531's answer).
keptToo := append(append([]delivery.Composed{}, running...), delivery.Composed{Node: "n1", Module: "spiegel", Resources: carriedRes})
if a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, keptToo, "p-h1", 3); a.Verdict != delivery.CandidateSame || len(a.Removed) != 0 {
t.Errorf("a folded module running from the kept declaration is answered %+v, want the same", a)
}
// (B) The node's own composition carries its bus membership as it runs it (the controller's, from its store): composed
// the same, not 'in what runs, not composed' (seed #531, novox).
membership := json.RawMessage(`{"id":"bus-membership","type":"file","path":"` + catalogue.BusMembershipPath + `","sealed":"{\"box\":\"m\"}","mode":"0600"}`)
withMembership := make([]delivery.Composed, 0, len(running))
for _, c := range running {
if c.Module == "" {
c.Resources = append(append([]json.RawMessage{}, c.Resources...), membership)
}
withMembership = append(withMembership, c)
}
if a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, withMembership, "p-h1", 3); a.Verdict != delivery.CandidateSame {
t.Errorf("the node's own with its bus membership is answered %+v, want the same", a)
}
// Without what runs, the fold composes spiegel from its manifest, as before, and the composition refuses it.
if _, _, held, err := Compose(ctx, b, Mesh{}, Scope{Node: "n1", InOwn: []string{"spiegel"}}, fps); err != nil ||
!strings.Contains(held["spiegel"], "spiegel has not been built for this mesh") {
t.Errorf("composed spiegel without what runs: %v %v, want spiegel held alone", err, held)
}
// (C) Issue 1126, C5: what the others read of a module carried as it runs is what runs, never its stub's: the filter
// opens the machine port its running container publishes (de-spiegel's 20005 on novox, not its software's 8080; a
// stub had no container, so snake's 80 on ace was opened beside the route proxy's). And C6: the operator export's file
// a module that keeps it runs is carried as it runs (mesh-vault.kept on novox), since the node holds no export.
listening := []byte(`{"module":"spiegel","version":"1","capabilities":["container-runtime"],` +
`"listens":[{"port":8080,"from":"mesh","why":"the site"}],"own-secrets":{"token":"${dir:state}/token"},` +
`"resources":[{"id":"state","type":"directory","path":"/var/lib/spiegel","mode":"0700"},` +
`{"id":"server","type":"container","name":"spiegel","artifact":"site","ports":["8080"]}],` +
`"build":{"artifacts":[{"name":"site","kind":"upstream","from":"` + image + `"}]}}`)
nft := []byte(`{"module":"nft","version":"1","capabilities":["systemd"],"filtering":{"into":"/etc/nftables.conf"}}`)
vault := []byte(`{"module":"vault","version":"1","capabilities":["systemd"],"keeps":"/var/lib/vault/root",` +
`"resources":[{"id":"service","type":"service","unit":"vault.service","state":"running","boot":"enabled"}]}`)
b = mk("lemurs", overlay.Name, "spiegel", "nft", "vault")
for m, raw := range map[string][]byte{"spiegel": listening, "nft": nft, "vault": vault} {
b.manifests[delivery.ManifestDigest(raw)] = raw
b.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
p = b.placements["n1"]
p.Decided.Endpoint = "n1.example:51820"
for i := range p.Modules {
if p.Modules[i].Module == "spiegel" {
p.Modules[i].Ports = map[string]int{"8080": 20005}
}
}
b.placements["n1"] = p
rec := b.records["n1"]
rec.OutwardLinks = []string{"eth0"}
b.records["n1"] = rec
fps = map[string]string{"lemurs": fp("e"), "nft": fp("f"), "vault": fp("9")}
if in, err = Input(ctx, b, Mesh{}, stub); err != nil {
t.Fatal(err)
}
in.Nodes[0].Kept = map[string]catalogue.Manifest{"spiegel": compose.Stub(in.Catalogue["spiegel"])}
if base, _, err = compose.Compositions(in, "n1", fps, compose.Reading); err != nil {
t.Fatal(err)
}
keptFile := json.RawMessage(`{"digest":"sha256:` + strings.Repeat("d", 64) + `","id":"vault.kept","mode":"0600","module":"vault","path":"/var/lib/vault/root/export.json","size":143604,"type":"file"}`)
running = nil
for _, c := range base {
switch c.Module {
case "":
c.Resources = append(append([]json.RawMessage{}, c.Resources...),
json.RawMessage(`{"id":"spiegel.state","type":"directory","path":"/var/lib/spiegel","mode":"0700"}`),
json.RawMessage(`{"id":"spiegel.server","type":"container","name":"spiegel","image":"registry.example/spiegel@sha256:`+strings.Repeat("c", 64)+`","ports":["10.77.0.1:20005:8080"]}`),
json.RawMessage(`{"id":"spiegel.server-env","type":"file","path":"/var/lib/spiegel/server.env","mode":"0600","content":"A=1\n"}`))
case "vault":
c.Resources = append(append([]json.RawMessage{}, c.Resources...), keptFile)
}
running = append(running, c)
}
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", InOwn: []string{"spiegel"}, Running: running}, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatalf("the node does not compose itself with spiegel carried: %v", err)
}
var rules string
var keptComposed json.RawMessage
for _, c := range got {
for _, raw := range c.Resources {
var r struct {
ID string `json:"id"`
Content string `json:"content"`
}
_ = json.Unmarshal(raw, &r)
switch r.ID {
case "nft." + catalogue.FilteringID():
rules = r.Content
case "vault." + resolution.KeptID():
keptComposed = raw
}
}
}
if !strings.Contains(rules, "# spiegel — the site") || !strings.Contains(rules, "dport 20005 accept") || strings.Contains(rules, "dport 8080") {
t.Errorf("the filter does not open the machine port spiegel runs at:\n%s", rules)
}
// The paths a carried module owns are its manifest's, as the controller composes them, never what runs of it (issue
// 1126, seed-8 #542: on ace, the folded f1-circuits, snake and txt-game, whose manifests leave their directories and
// files to ${dir:…}, owned their /var/lib paths on the node and none on the controller, shifting node-tools'
// MESH_OWNED_PATHS). f1-circuits' manifest at the indexer's commit (c0ea2c94) is captured; what runs of it is
// composed from that manifest by the controller's rule (the directory at /var/lib/f1-circuits, its file, its container).
for _, c := range running {
if c.Module == "" && !reflect.DeepEqual(compose.Carried(in.Catalogue["spiegel"], "spiegel", &c).OwnsPaths, []string{"/var/lib/spiegel"}) {
t.Errorf("spiegel carried owns %v, want its manifest's /var/lib/spiegel alone", compose.Carried(in.Catalogue["spiegel"], "spiegel", &c).OwnsPaths)
}
}
f1raw, err := os.ReadFile("testdata/f1-circuits-c0ea2c94.module.json")
if err != nil {
t.Fatal(err)
}
f1, err := catalogue.Decode(f1raw)
if err != nil {
t.Fatal(err)
}
f1runs := delivery.Composed{Node: "ace", Resources: []json.RawMessage{
json.RawMessage(`{"id":"f1-circuits.state","module":"f1-circuits","type":"directory","path":"/var/lib/f1-circuits","mode":"0700"}`),
json.RawMessage(`{"id":"f1-circuits.cache","module":"f1-circuits","type":"directory","path":"/var/lib/f1-circuits/cache","mode":"0700"}`),
json.RawMessage(`{"id":"f1-circuits.site-env","module":"f1-circuits","type":"file","path":"/var/lib/f1-circuits/site.env","mode":"0644","content":"F1_CIRCUITS_BASE_URL=https://f1.example\n"}`),
json.RawMessage(`{"id":"f1-circuits.server","module":"f1-circuits","type":"container","name":"f1-circuits","image":"registry.example/f1@sha256:` + strings.Repeat("c", 64) + `"}`)}}
if got, want := resolution.DeclaredPaths(compose.Carried(f1, "f1-circuits", &f1runs)), resolution.DeclaredPaths(f1); !reflect.DeepEqual(got, want) {
t.Errorf("f1-circuits carried owns %v, want what the controller composes from its manifest, %v", got, want)
}
// Issue 1126: a held module hands no template its build kept for a file it no longer carries (novox's node:
// "postgres: its build kept a template for consumers"), but keeps its items and their templates, so what it
// contributes to other modules' collections still reaches them.
for _, row := range []struct {
name string
handed *catalogue.Handed
items []catalogue.Item
}{
{"templates and items", &catalogue.Handed{Templates: map[string]string{"consumers": "files/consumers.json"},
Items: map[string]string{"routes": "files/routes.tmpl"}}, []catalogue.Item{{Collection: "routes", Content: "x"}}},
{"items alone", nil, []catalogue.Item{{Collection: "routes", Content: "x"}}},
} {
kept := f1
kept.Handed, kept.Items = row.handed, row.items
st := compose.Stub(kept)
if st.Handed != nil && st.Handed.Templates != nil {
t.Errorf("%s: a stub hands templates %v", row.name, st.Handed.Templates)
}
if !reflect.DeepEqual(st.Items, row.items) {
t.Errorf("%s: a stub collects %+v, want its items %+v", row.name, st.Items, row.items)
}
if row.handed != nil && (st.Handed == nil || !reflect.DeepEqual(st.Handed.Items, row.handed.Items) || row.handed.Templates == nil) {
t.Errorf("%s: a stub hands item templates %+v, want %v and its manifest untouched", row.name, st.Handed, row.handed.Items)
}
}
var k1, k2 any
_ = json.Unmarshal(keptComposed, &k1)
_ = json.Unmarshal(keptFile, &k2)
if !reflect.DeepEqual(k1, k2) {
t.Errorf("the operator export is composed as %s, want the file that runs %s", keptComposed, keptFile)
}
// (D) Issue 1126, seed-8 (mesh-catalog #542): a module's own refusal refuses that module, never the node. novox runs
// de-spiegel folded, and its manifest at the indexer's commit (e25e1beb, captured) asks four required config values
// for its route names that nothing sets. novox answered the whole candidate refused ("de-spiegel contributing
// praktijk to route says ${setting:praktijk-name} ...": its stub still contributed the route). Only de-spiegel is
// refused, in the config form's words; the rest is judged. Issue 1960, novox's chain on 2026-10-29: the hold spread
// to mailu (its smtp provider) and proxy (its route provider), and from them on. Neither is held now: proxy composes
// without de-spiegel's routes, which do not compose, and mailu with its smtp account, which does.
spiegel, err := os.ReadFile("testdata/de-spiegel-e25e1beb.module.json")
if err != nil {
t.Fatal(err)
}
proxy := []byte(`{"module":"proxy","version":"1","capabilities":["systemd"],"provides":[{"name":"route","scope":"mesh"}],` +
`"resources":[{"id":"service","type":"service","unit":"proxy.service","state":"running","boot":"enabled"}]}`)
mailu := []byte(`{"module":"mailu","version":"1","capabilities":["systemd"],"provides":[{"name":"smtp","scope":"mesh"}],` +
`"resources":[{"id":"service","type":"service","unit":"mailu.service","state":"running","boot":"enabled"}]}`)
b = mk("lemurs", overlay.Name, "mailu", "proxy", "de-spiegel")
for m, raw := range map[string][]byte{"de-spiegel": spiegel, "proxy": proxy, "mailu": mailu} {
b.manifests[delivery.ManifestDigest(raw)] = raw
b.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
p = b.placements["n1"]
p.Decided.Endpoint = "n1.example:51820"
b.placements["n1"] = p
fps = map[string]string{"lemurs": fp("e"), "proxy": fp("f"), "mailu": fp("d")}
a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"de-spiegel"}}, fps, nil, "p-h1", 3)
refused = nil
for _, r := range a.Refusals {
refused = append(refused, r.Module)
if r.Module == "de-spiegel" && (!strings.Contains(r.Why, "${config:praktijk-name}") || strings.Contains(r.Why, "${setting:")) {
t.Errorf("de-spiegel is refused in other words than its config form's: %s", r.Why)
}
}
if a.Refused != "" || !reflect.DeepEqual(refused, []string{"de-spiegel"}) {
t.Errorf("a module's own refusal: %+v, want de-spiegel refused alone, the node, mailu and proxy not", a)
}
for _, m := range []string{"lemurs", "mailu", "proxy"} {
if !slices.ContainsFunc(a.Added, func(x delivery.ModuleAdded) bool { return x.Module == m }) {
t.Errorf("%s, which composes, is not judged beside de-spiegel's refusal: %+v", m, a)
}
}
// (E) Issue 1126, the applying path (review of the seed): the composition novox applies follows the same rule. It
// composed with no culprit, so de-spiegel's unset route names refused the whole node and every module it runs was
// marked not composed. Now de-spiegel is held, its refusal said in its config form's words, with proxy, which
// receives its routes; lemurs composes, and the node's own composition carries de-spiegel as it runs, so applying it
// removes nothing of it.
spiegelRuns := []json.RawMessage{
json.RawMessage(`{"id":"de-spiegel.state","type":"directory","path":"/var/lib/de-spiegel","mode":"0755"}`),
json.RawMessage(`{"id":"de-spiegel.server-env","type":"file","path":"/var/lib/de-spiegel/server.env","mode":"0600","content":"A=1\n"}`),
json.RawMessage(`{"id":"de-spiegel.server","type":"container","name":"de-spiegel","image":"registry.example/de-spiegel@sha256:` + strings.Repeat("c", 64) + `"}`)}
if in, err = Input(ctx, b, Mesh{}, stub); err != nil {
t.Fatal(err)
}
in.Nodes[0].Kept = map[string]catalogue.Manifest{"de-spiegel": compose.Stub(in.Catalogue["de-spiegel"])}
in.Nodes[0].Kept["de-spiegel"] = func(m catalogue.Manifest) catalogue.Manifest {
m.Contributes, m.ContributesMany = nil, nil
return m
}(in.Nodes[0].Kept["de-spiegel"])
if base, _, err = compose.Compositions(in, "n1", fps, compose.Reading); err != nil {
t.Fatal(err)
}
running = nil
for _, c := range base {
if c.Module == "" {
c.Resources = append(append([]json.RawMessage{}, c.Resources...), spiegelRuns...)
}
running = append(running, c)
}
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", InOwn: []string{"de-spiegel"}, Running: running}, fps)
if err != nil {
t.Fatalf("de-spiegel's own refusal refuses the node's whole composition: %v", err)
}
if len(held) != 1 || !strings.Contains(held["de-spiegel"], "${config:praktijk-name}") || !strings.Contains(held["de-spiegel"], "without what it contributes to route") {
t.Errorf("held %v, want de-spiegel alone, in its config form's words, composed without its routes", held)
}
for _, m := range []string{"lemurs", "mailu", "proxy"} {
if key, _ := delivery.ComposedKey("n1", m, fps[m]); got[key].Module != m {
t.Errorf("%s is not composed beside de-spiegel's hold: %v", m, sortedKeys(got))
}
}
carries := func(c delivery.Composed, what string) {
t.Helper()
ownRes, _ := json.Marshal(c.Resources)
for _, raw := range spiegelRuns {
var r struct{ ID string }
_ = json.Unmarshal(raw, &r)
if !strings.Contains(string(ownRes), `"id":"`+r.ID+`"`) {
t.Errorf("%s: the node's own composition drops %s, which de-spiegel runs: %s", what, r.ID, ownRes)
}
}
}
carries(got[ownKey], "folded")
// (F) Leaving the fold (review of the apply, round 1, finding 1): de-spiegel's want names a build of the new form,
// so it is no longer folded, while its names are still unset and the node's own still runs it. Held, it is carried
// in the node's own as it runs until a composition of its own is handed: else the node's own apply removes its
// container, files and units (issue 1945 by a new path).
fps["de-spiegel"] = fp("7")
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", Running: running}, fps)
if err != nil || !strings.Contains(held["de-spiegel"], "${config:praktijk-name}") {
t.Fatalf("de-spiegel leaving the fold: %v, held %v", err, held)
}
spiegelKey, _ := delivery.ComposedKey("n1", "de-spiegel", fps["de-spiegel"])
if _, ok := got[spiegelKey]; ok {
t.Errorf("held de-spiegel's own composition is handed")
}
carries(got[ownKey], "leaving the fold")
// (G) A held module that runs a composition of its own (review of the apply, round 1, finding 2): what the others
// compose of it is what runs, never its stub's. site listens on 8080, placed at the machine's 20005, and its own
// secret was never issued here: held, the filter must not open 8080 and drop the 20005 that runs. The filter, which
// reads what site listens on, composes with it (issue 1960: a hold never spreads).
site := []byte(`{"module":"site","version":"1","capabilities":["container-runtime"],` +
`"listens":[{"port":8080,"from":"mesh","why":"the site"}],"own-secrets":{"key":"${dir:state}/key"},` +
`"resources":[{"id":"state","type":"directory","path":"/var/lib/site","mode":"0700"},` +
`{"id":"server","type":"container","name":"site","image":"registry.example/site@sha256:` + strings.Repeat("c", 64) + `","ports":["8080"]}]}`)
b = mk("lemurs", overlay.Name, "site", "nft")
for m, raw := range map[string][]byte{"site": site, "nft": nft} {
b.manifests[delivery.ManifestDigest(raw)] = raw
b.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
p = b.placements["n1"]
p.Decided.Endpoint = "n1.example:51820"
for i := range p.Modules {
if p.Modules[i].Module == "site" {
p.Modules[i].Ports = map[string]int{"8080": 20005}
}
}
b.placements["n1"] = p
rec = b.records["n1"]
rec.OutwardLinks = []string{"eth0"}
b.records["n1"] = rec
fps = map[string]string{"lemurs": fp("e"), "nft": fp("f"), "site": fp("8")}
siteRuns := delivery.Composed{Node: "n1", Module: "site", FP: fps["site"], Resources: []json.RawMessage{
json.RawMessage(`{"id":"site.state","module":"site","type":"directory","path":"/var/lib/site","mode":"0700"}`),
json.RawMessage(`{"id":"site.server","module":"site","type":"container","name":"site","image":"registry.example/site@sha256:` + strings.Repeat("c", 64) + `","ports":["10.77.0.1:20005:8080"]}`)}}
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", Running: []delivery.Composed{siteRuns}}, fps)
nftKey, _ := delivery.ComposedKey("n1", "nft", fps["nft"])
if err != nil || held["site"] == "" || held["nft"] != "" || got[nftKey].Module != "nft" {
t.Fatalf("site, its own secret never issued: %v, held %v; want site held alone, and the filter composed", err, held)
}
if rules, _ := json.Marshal(got[nftKey].Resources); !strings.Contains(string(rules), "dport 20005") {
t.Errorf("the filter drops held site's opening where it runs (20005): %s", rules)
}
for key, c := range got {
for _, raw := range c.Resources {
if strings.Contains(string(raw), "dport 8080") || (c.Module == "nft" && !strings.Contains(string(raw), "dport 20005") && strings.Contains(string(raw), "# site")) {
t.Errorf("%s composes held site from its stub, not from what runs: %s", key, raw)
}
}
}
// (G2) Issue 1126, the spreading hold: a held module is no dependence of its neighbours. site, held, provides page, and
// app requires and binds it: app composes against what runs of site and is not held (novox's seed held every postgres
// consumer behind de-spiegel's hold, "what it composes of …, held, is not what runs of them").
provider := []byte(strings.Replace(string(site), `"listens":`, `"provides":[{"name":"page","scope":"node"}],"serves":{"page":{"port":8080}},"listens":`, 1))
app := []byte(`{"module":"app","version":"1","capabilities":["systemd"],"requires":["page"],"binds":{"page":"/etc/app/page.json"},` +
`"resources":[{"id":"service","type":"service","unit":"app.service","state":"running","boot":"enabled"}]}`)
b2 := mk("lemurs", overlay.Name, "site", "nft", "app")
for m, raw := range map[string][]byte{"site": provider, "nft": nft, "app": app} {
b2.manifests[delivery.ManifestDigest(raw)] = raw
b2.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
p2 := b2.placements["n1"]
p2.Decided.Endpoint = "n1.example:51820"
for i := range p2.Modules {
if p2.Modules[i].Module == "site" {
p2.Modules[i].Ports = map[string]int{"8080": 20005}
}
}
b2.placements["n1"] = p2
b2.records["n1"] = rec
b2.held = true
fps["app"] = fp("9")
got, _, held, err = Compose(ctx, b2, Mesh{}, Scope{Node: "n1", Running: []delivery.Composed{siteRuns}}, fps)
appKey, _ := delivery.ComposedKey("n1", "app", fps["app"])
if _, composed := got[appKey]; err != nil || held["site"] == "" || held["app"] != "" || !composed {
t.Errorf("a consumer of held site: %v, held %v; want site held, and app composed against what runs of it", err, held)
}
if bound, _ := json.Marshal(got[appKey].Resources); !strings.Contains(string(bound), "20005") {
t.Errorf("app's binding does not reach held site where it runs (20005): %s", bound)
}
delete(fps, "app")
// A held module's offer is left out by provision, never whole when one part composes (issue 1960): de-spiegel's
// routes, its names unset, leave the proxy without them, and its smtp account stays with mailu.
withProxy := compose.Input{Catalogue: map[string]catalogue.Manifest{},
Nodes: []compose.Node{{Name: "n1", Assigned: []string{"feeder"}}}}
feeder := catalogue.Manifest{Module: "feeder", Contributes: map[string]map[string]any{"route": {"name": "x"}, "smtp": {"account": "feeder"}},
OwnSecrets: catalogue.OwnSecrets{"key": {Path: "/k"}}}
for _, row := range []struct {
left map[string]bool
route, smtp, own bool
}{{nil, true, true, true}, {map[string]bool{"route": true}, false, true, true}, {map[string]bool{leaveOwn: true}, true, true, false}, {map[string]bool{leaveAll: true}, false, false, false}} {
k := heldAs(withProxy, "n1", "feeder", feeder, row.left, nil).Nodes[0].Kept["feeder"]
if _, r := k.Contributes["route"]; r != row.route || (k.Contributes["smtp"] != nil) != row.smtp || (len(k.OwnSecrets) > 0) != row.own {
t.Errorf("held leaving out %v offers %+v, own secrets %v", row.left, k.Contributes, k.OwnSecrets)
}
}
// (H) Review of the apply, round 2: a held module is composed from its new manifest, so what the others read of its
// manifest (its listens, what it contributes) may not be what runs. site's new manifest moves it to 9090 while the
// build that runs (7) listens on 8080. The filter composes with what site last offered, its manifest at the build
// that runs (issue 1960: a hold never spreads), so it opens 8080 and nothing applied opens 9090.
moved := []byte(strings.Replace(string(site), `"port":8080`, `"port":9090`, 1))
b.manifests[delivery.ManifestDigest(moved)] = moved
b.modules["site"] = delivery.Module{Manifest: delivery.ManifestDigest(moved)}
for i := range p.Modules {
p.Modules[i].Ports = nil
}
b.placements["n1"] = p
siteRuns.Resources[1] = json.RawMessage(`{"id":"site.server","module":"site","type":"container","name":"site","image":"registry.example/site@sha256:` + strings.Repeat("c", 64) + `","ports":["8080"]}`)
siteRuns.FP = fp("7")
b.manifests[delivery.ManifestDigest(site)] = site
b.builds["site "+fp("7")] = &delivery.BuilderBuild{State: delivery.BuildDone, Manifest: delivery.ManifestDigest(site)}
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", Running: []delivery.Composed{siteRuns}}, fps)
if err != nil || held["site"] == "" || held["nft"] != "" {
t.Errorf("site moved to 9090 in its new manifest: %v, held %v; want site held alone", err, held)
}
if rules, _ := json.Marshal(got[nftKey].Resources); !strings.Contains(string(rules), "dport 8080") {
t.Errorf("the filter drops what held site last offered (8080): %s", rules)
}
for key, c := range got {
for _, raw := range c.Resources {
if strings.Contains(string(raw), "dport 9090") {
t.Errorf("%s opens 9090, which held site's new manifest names and nothing runs: %s", key, raw)
}
}
}
}
// A module another node cannot run in this composition offers nothing in it: the refusal names it and why
// (issue 1126, probe #532: ace and g14 answered "nothing in this mesh provides artifact-store … assign distribution to a
// node" while novox runs it). And a refusal no module's stub changes is answered at once, not after a compose per module
// (ace took 7m46s).
func otherNodeNotResolvedIsNamed(t *testing.T) {
ctx := context.Background()
prov := []byte(`{"module":"prov","version":"1","capabilities":["container-runtime"],"provides":[{"name":"thing","scope":"mesh"}]}`)
cons := []byte(`{"module":"cons","version":"1","requires":["thing"]}`)
// n2 runs prov, which needs a container runtime n2's record does not say it has: n2's set does not resolve here.
b := &bus{placements: map[string]delivery.Placement{
"n1": {Node: "n1", Decided: delivery.NodeDecided{Hub: true, Address: "10.77.0.1", Endpoint: "n1.example:51820"}, Modules: []delivery.PlacedModule{{Module: "cons"}, {Module: overlay.Name}}},
"n2": {Node: "n2", Decided: delivery.NodeDecided{Address: "10.77.0.2"}, Modules: []delivery.PlacedModule{{Module: "prov"}, {Module: overlay.Name}}}},
records: map[string]delivery.NodeRecord{"n1": {Node: "n1", TunnelKey: publicKey(t)}, "n2": {Node: "n2", TunnelKey: publicKey(t)}},
modules: map[string]delivery.Module{"prov": {Manifest: delivery.ManifestDigest(prov)}, "cons": {Manifest: delivery.ManifestDigest(cons)}},
manifests: map[string][]byte{delivery.ManifestDigest(prov): prov, delivery.ManifestDigest(cons): cons}}
given := map[string]*delivery.Placement{}
for n, p := range b.placements {
p := p
given[n] = &p
}
a := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given}, map[string]string{"cons": "src2:" + strings.Repeat("1", 64)}, nil, "p-h1", 1)
if a.Verdict != delivery.CandidateRefused || !strings.Contains(a.Refused, "prov on n2 lacks container-runtime") {
t.Errorf("a refusal for want of n2's provider = %+v", a)
}
}
// A node that reaches no artifact store says what its composition holds of one (issue 1126: ace and g14 answered
// "reaches no artifact store", and nothing said why).
func noStoreReachedSaysWhy(t *testing.T) {
in := compose.Input{Catalogue: map[string]catalogue.Manifest{
"distribution": {Module: "distribution", Serves: map[string]map[string]any{catalogue.ArtifactStoreProvision: {"port": 5000}}}},
Nodes: []compose.Node{{Name: "g14", Assigned: []string{"adwaita"}}, {Name: "novox", Assigned: []string{"distribution"}}}}
if got := storeSought(in); got != "the composition holds g14, novox; artifact-store is served by distribution on novox, none of it on the private network with a port" {
t.Errorf("said %q", got)
}
in.Nodes = in.Nodes[:1]
if got := storeSought(in); got != "the composition holds g14; no module placed there serves artifact-store" {
t.Errorf("said %q", got)
}
}
+112
View File
@@ -0,0 +1,112 @@
package frombus
import (
"fmt"
"sort"
"strconv"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose"
)
// Export is one node's placement as the composition input the controller fills says it (novox/hq ADR 0555 §6, ADR 0567
// §7): the inverse of Input for what a person decides, so the controller's one read-only verb `placement-export`
// answers, from its own store by its own code, exactly what the seat's adopt proposes as the seed. Its answer must stay
// stable for the same store, since adopt checks its digest again at the warrant.
// What the machine observes (its record) is not in it.
func Export(in compose.Input, node string) (delivery.Placement, error) {
var n *compose.Node
for i := range in.Nodes {
if in.Nodes[i].Name == node {
n = &in.Nodes[i]
}
}
if n == nil {
return delivery.Placement{}, fmt.Errorf("frombus: the input has no node %q", node)
}
// At the version a placement key is read at, with a list of modules even when none is placed: the export is the
// seed's value, and the reader refuses a value of version 0 or without its modules.
p := delivery.Placement{V: delivery.Version, Node: node, Modules: []delivery.PlacedModule{}, Decided: delivery.NodeDecided{
Endpoint: n.Place.Endpoint, Site: n.Place.Site, Hub: n.Place.Hub, Address: n.Place.Address,
PublicDomain: n.PublicDomain, Account: n.Account, AccountHome: n.AccountHome, AgentAccount: n.AgentAccount,
AgentHome: n.AgentAccountHome, Adopted: n.Adopted, Taken: n.Taken}}
// The tunnel found at the node's adoption: the one its private network took over, or, on the hub, the mesh's adopted
// one whether or not the hub took it; and the names a person gave the hub's carried peers.
if t, ok := in.Tunnels[node]; ok {
rt := recordTunnelOf(t.Tunnel)
p.Decided.Tunnel = &rt
} else if in.Adopted != nil && in.Adopted.Hub == node {
rt := recordTunnelOf(in.Adopted.Tunnel)
p.Decided.Tunnel = &rt
}
if n.Place.Hub {
for _, c := range in.CarriedPeers {
if c.Named != "" {
if p.Decided.PeerNames == nil {
p.Decided.PeerNames = map[string]string{}
}
p.Decided.PeerNames[c.PublicKey] = c.Named
}
}
}
for _, module := range n.Assigned {
m := delivery.PlacedModule{Module: module}
for port, at := range n.Ports[module] {
if m.Ports == nil {
m.Ports = map[string]int{}
}
m.Ports[strconv.Itoa(port)] = at
}
for provision, to := range n.Bound[module] {
if m.Bindings == nil {
m.Bindings = map[string]delivery.Provider{}
}
m.Bindings[provision] = delivery.Provider{Node: to.Node, Module: to.Module}
}
// A module in the settings form keeps its config values in its layers, not in mesh-config: they are exported with
// the placement. A module in the config form has mesh-config's values laid into its layer, which are not.
settingsForm := !in.Catalogue[module].DeclaresConfig()
for _, layer := range n.Settings[module] {
if layer.From == catalogue.MeshWideLayer {
m.HasMesh, m.MeshWide = true, layer.Placement
if settingsForm && len(layer.Values) > 0 {
m.MeshValues = layer.Values
}
} else {
m.HasLayer, m.Layer = true, layer.Placement
if settingsForm && len(layer.Values) > 0 {
m.Values = layer.Values
}
}
}
p.Modules = append(p.Modules, m)
}
for provision, to := range n.Pins {
if p.Pins == nil {
p.Pins = map[string]delivery.Provider{}
}
p.Pins[provision] = delivery.Provider{Node: to.Node, Module: to.Module}
}
for _, h := range in.Holdings {
if h.Node == node {
p.Holds = append(p.Holds, delivery.Holding{Claim: h.Claim, Scope: h.Scope, Module: h.Module, Site: h.Site, Kind: h.Kind})
}
}
sort.SliceStable(p.Holds, func(i, j int) bool {
if p.Holds[i].Claim != p.Holds[j].Claim {
return p.Holds[i].Claim < p.Holds[j].Claim
}
return p.Holds[i].Module < p.Holds[j].Module
})
return p, nil
}
func recordTunnelOf(t compose.Tunnel) delivery.RecordTunnel {
out := delivery.RecordTunnel{Interface: t.Interface, Unit: t.Unit, Config: t.Config, Port: t.Port, MTU: t.MTU,
Address: t.Address, Range: t.Range, PublicKey: t.PublicKey}
for _, peer := range t.Peers {
out.Peers = append(out.Peers, delivery.RecordTunnelPeer{PublicKey: peer.PublicKey, Address: peer.Address})
}
return out
}
File diff suppressed because it is too large Load Diff
+869
View File
@@ -0,0 +1,869 @@
package frombus
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/ecdh"
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"reflect"
"slices"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose"
"github.com/novox/mesh-host/internal/overlay"
)
// bus is a Reader over values held in memory, counting whose config values were read.
type bus struct {
vaultAuthority string
handed map[string]bool
placements map[string]delivery.Placement
records map[string]delivery.NodeRecord
modules map[string]delivery.Module
manifests map[string][]byte
values map[string]map[string]any // by node.module
valuesRead []string
own []compose.OwnSecretRow
ownErr error
// asked is every consumer credential asked; the fake refuses the whole ask, as the vault's own does, for a consuming
// module its placement does not put on the node.
asked []compose.CredentialKey
// builds are the builder's keys, by "<module> <fp>"; archives the artifact store's blobs, by digest, and fetched
// every URL asked; served another node's values by "<node>.<module>", and servedRead every ask with its keys.
builds map[string]*delivery.BuilderBuild
archives map[string][]byte
fetched []string
served map[string]map[string]any
servedRead []string
buildsRead []string
// held answers every credential asked as held, sealed to the consumer.
held bool
}
func (b *bus) Build(_ context.Context, module, fp string) (*delivery.BuilderBuild, error) {
b.buildsRead = append(b.buildsRead, module)
return b.builds[module+" "+fp], nil
}
func (b *bus) Archive(_ context.Context, url, digest string) ([]byte, error) {
b.fetched = append(b.fetched, url)
body, ok := b.archives[digest]
if !ok {
return nil, errors.New("the store holds no such blob")
}
return body, nil
}
func (b *bus) Served(_ context.Context, node, module string, keys []string) (map[string]any, error) {
b.servedRead = append(b.servedRead, node+"."+module+":"+strings.Join(keys, ","))
out := map[string]any{}
for _, k := range keys {
if v, ok := b.served[node+"."+module][k]; ok {
out[k] = v
}
}
return out, nil
}
func (b *bus) Placements(context.Context) (map[string]delivery.Placement, error) {
return b.placements, nil
}
func (b *bus) VaultAuthority(context.Context) string { return b.vaultAuthority }
func (b *bus) Handed(context.Context, string) map[string]bool { return b.handed }
func (b *bus) Records(context.Context) (map[string]delivery.NodeRecord, error) { return b.records, nil }
func (b *bus) Modules(context.Context) (map[string]delivery.Module, error) { return b.modules, nil }
func (b *bus) Manifest(_ context.Context, digest string) ([]byte, error) {
raw, ok := b.manifests[digest]
if !ok {
return nil, errors.New("no such manifest")
}
return raw, nil
}
func (b *bus) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow, error) {
return b.own, b.ownErr
}
func (b *bus) Credentials(_ context.Context, node string, asConsumer, asProvider []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
b.asked = append(b.asked, asConsumer...)
for _, k := range asConsumer {
placed := false
for _, m := range b.placements[node].Modules {
placed = placed || m.Module == k.ConsumerModule
}
if !placed {
return nil, nil, fmt.Errorf("%s is not on %s: neither assigned there nor required by a module there", k.ConsumerModule, node)
}
}
var out []compose.Credential
var issued []compose.Issued
for _, k := range append(append([]compose.CredentialKey{}, asConsumer...), asProvider...) {
if b.held {
out = append(out, compose.Credential{Provision: k.Provision, Consumer: k.Consumer, ConsumerModule: k.ConsumerModule,
Provider: k.Provider, Local: k.Local, State: compose.CredentialHeld, Sealed: `"box"`})
issued = append(issued, compose.Issued{Provision: k.Provision, Consumer: k.Consumer, ConsumerModule: k.ConsumerModule,
Provider: k.Provider, Local: k.Local, ForProvider: `"box"`})
}
}
return out, issued, nil
}
func (b *bus) Values(_ context.Context, node, module string) (map[string]any, error) {
b.valuesRead = append(b.valuesRead, node+"."+module)
return b.values[node+"."+module], nil
}
func publicKey(t *testing.T) string {
t.Helper()
k, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
return base64.StdEncoding.EncodeToString(k.PublicKey().Bytes())
}
// Issue 1126 (S3, S4, S5): the bus's placement, node records, manifests and config values fill the same compose.Input the
// controller's store fills for compose's own two-machine mesh (compose's aMesh: anchor the hub, laptop), and that Input
// composes; a node reads only its own config values; what cannot be composed is refused, naming it.
func TestInput(t *testing.T) {
// Issue 1126, the seed's shadow: every refusal named, an old-form module left to the node, and no store reached
// saying why (answer_rows_test.go).
t.Run("every refusal, and an old build left to the node", answerNamesEveryRefusalAndLeavesAnOldBuildToTheNode)
t.Run("no store reached says why", noStoreReachedSaysWhy)
// Issue 774: a node refused whole says what it held on the way.
t.Run("a node refused whole says what it held", aNodeRefusedWholeSaysWhatItHeld)
t.Run("a refused manifest never ends the read", aRefusedManifestNeverEndsTheRead)
t.Run("the runtime is blamed only where its stub composes", theRuntimeIsBlamedOnlyWhereItsStubComposes)
t.Run("a running module waits for its build", aRunningModuleWaitsForItsBuild)
t.Run("a held module's first reason leads", aHeldModulesFirstReasonLeads)
t.Run("a held module of its own account never refuses the node", aHeldModuleOfItsOwnAccountNeverRefusesTheNode)
t.Run("another node not resolved is named", otherNodeNotResolvedIsNamed)
ctx := context.Background()
xterm, err := os.ReadFile("testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
digest := delivery.ManifestDigest(xterm)
// A module in the settings form (plex and the media modules on one node, 2026-10-26): its config values are in its
// layers, not mesh-config, so the placement carries them (MeshValues, Values) and mesh-config's are not read for it.
media := []byte(`{"module":"media","version":"1","settings":{"lines":{"kind":"preference","default":0,"why":"kept"},` +
`"name":{"kind":"preference","default":"x","why":"shown"}},"own-secrets":{"token":"/srv/media/token"},"resources":[{"id":"conf","type":"file",` +
`"path":"/srv/media/conf","mode":"0644","content":"uid=${setting:lines} name=${setting:name}\n"}]}`)
mediaDigest := delivery.ManifestDigest(media)
keys := map[string]string{"anchor": publicKey(t), "laptop": publicKey(t)}
caps := []string{"container-runtime", "systemd", "wireguard"}
tunnel := &delivery.RecordTunnel{Interface: "wg0", PublicKey: keys["anchor"], Address: "10.77.0.1/24",
Peers: []delivery.RecordTunnelPeer{{PublicKey: "peer", Address: "10.77.0.9/32"}, {PublicKey: keys["laptop"], Address: "10.77.0.2/32"}}}
mk := func() *bus {
return &bus{
placements: map[string]delivery.Placement{
"anchor": {Node: "anchor", Decided: delivery.NodeDecided{Endpoint: "anchor.example:51820", Site: "here", Hub: true, Address: "10.77.0.1",
Tunnel: tunnel, PeerNames: map[string]string{"peer": "phone"}},
Modules: []delivery.PlacedModule{{Module: overlay.Name}},
Holds: []delivery.Holding{{Claim: "mesh-private-network", Scope: "mesh", Module: overlay.Name}}},
"laptop": {Node: "laptop", Decided: delivery.NodeDecided{Endpoint: "laptop.example:51820", Site: "here", Address: "10.77.0.2"},
Modules: []delivery.PlacedModule{{Module: overlay.Name},
{Module: "xterm", HasLayer: true, Ports: map[string]int{"80": 8080}},
{Module: "media", HasMesh: true, MeshValues: map[string]any{"name": "mesh"}, HasLayer: true,
Layer: map[string]any{"places": map[string]any{"data": "/srv/data"}}, Values: map[string]any{"lines": float64(1000)}}}},
},
records: map[string]delivery.NodeRecord{
"anchor": {Node: "anchor", TunnelKey: keys["anchor"], Capabilities: caps, Contract: 2},
"laptop": {Node: "laptop", TunnelKey: keys["laptop"], Capabilities: caps, Authority: "the mesh's certificate"},
},
modules: map[string]delivery.Module{"xterm": {Manifest: digest}, "media": {Manifest: mediaDigest}, "unread": {}},
manifests: map[string][]byte{digest: xterm, mediaDigest: media},
values: map[string]map[string]any{"laptop.xterm": {"font": "mono"}, "anchor.xterm": {"font": "serif"},
"laptop.media": {"lines": "not mesh-config's"}},
own: []compose.OwnSecretRow{{Node: "laptop", Module: "media", Name: "token", Origin: "made", SealedTo: "sk", Sealed: "{}"}},
}
}
// What the controller's store gives for the same mesh, as compose's aMesh and SettingsFor build it.
var private catalogue.Manifest
raw, _ := json.Marshal(overlay.Manifest())
if err := json.Unmarshal(raw, &private); err != nil {
t.Fatal(err)
}
parsed, err := catalogue.Decode(xterm)
if err != nil {
t.Fatal(err)
}
parsedMedia, err := catalogue.Decode(media)
if err != nil {
t.Fatal(err)
}
capsSet := map[string]bool{"container-runtime": true, "systemd": true, "wireguard": true}
tun := compose.Tunnel{Interface: "wg0", PublicKey: keys["anchor"], Address: "10.77.0.1/24",
Peers: []compose.TunnelPeer{{PublicKey: "peer", Address: "10.77.0.9/32"}, {PublicKey: keys["laptop"], Address: "10.77.0.2/32"}}}
want := compose.Input{
Catalogue: map[string]catalogue.Manifest{overlay.Name: private, "xterm": parsed, "media": parsedMedia},
Nodes: []compose.Node{
{Name: "anchor", Place: compose.Place{Key: keys["anchor"], Endpoint: "anchor.example:51820", Site: "here", Hub: true,
Address: "10.77.0.1"}, Assigned: []string{overlay.Name}, Capabilities: capsSet, Contract: 2, Stated: true},
{Name: "laptop", Place: compose.Place{Key: keys["laptop"], Endpoint: "laptop.example:51820", Site: "here",
Address: "10.77.0.2"}, Assigned: []string{overlay.Name, "xterm", "media"}, Capabilities: capsSet,
Ports: map[string]map[int]int{"xterm": {80: 8080}},
Settings: map[string][]catalogue.Layer{"xterm": {{From: "laptop", Values: map[string]any{"font": "mono"}}},
"media": {{From: catalogue.MeshWideLayer, Values: map[string]any{"name": "mesh"}},
{From: "laptop", Values: map[string]any{"lines": float64(1000)}, Placement: map[string]any{"places": map[string]any{"data": "/srv/data"}}}}}},
},
Holdings: []catalogue.Held{{Claim: "mesh-private-network", Scope: "mesh", Node: "anchor", Module: overlay.Name}},
OwnSecrets: []compose.OwnSecretRow{{Node: "laptop", Module: "media", Name: "token", Origin: "made", SealedTo: "sk", Sealed: "{}"}},
// The same value as a core credential's box, for a core module (ADR 0349 §16).
KeptCoreBoxes: []compose.CoreBox{{Node: "laptop", Module: "media", Name: "token", SealedTo: "sk", Box: json.RawMessage("{}")}},
Tunnels: map[string]compose.FoundTunnel{"anchor": {Tunnel: tun}},
Adopted: &compose.AdoptedTunnel{Tunnel: tun, Hub: "anchor", Adopted: true},
// The controller's order is by address; laptop enrolled under its peer's key, and a person named the other.
Authority: "the mesh's certificate",
CarriedPeers: []compose.CarriedPeer{{PublicKey: keys["laptop"], Address: "10.77.0.2", EnrolledAs: "laptop"},
{PublicKey: "peer", Address: "10.77.0.9", Named: "phone"}},
}
b := mk()
got, err := Input(ctx, b, Mesh{}, Scope{Node: "laptop"})
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, want) {
gj, _ := json.MarshalIndent(got, "", " ")
wj, _ := json.MarshalIndent(want, "", " ")
t.Fatalf("Input from the bus differs from the store's:\n got %s\nwant %s", gj, wj)
}
// L3 s8 (ADR 0550 §2): with the vault's authority published, the mesh's authority is the bundle of the controller's
// and the vault's; with none, the controller's alone, as above.
signing := mk()
signing.vaultAuthority = "the vault's certificate"
if bundled, err := Input(ctx, signing, Mesh{}, Scope{Node: "laptop"}); err != nil ||
bundled.Authority != "the mesh's certificate\nthe vault's certificate\n" {
t.Errorf("the authority with the vault's published: %q, %v", bundled.Authority, err)
}
// Issue 774: the modules the vault judged handed when it answered own are the node's to hand, and no other node's.
judged := mk()
judged.handed = map[string]bool{"media": true, "xterm": false}
if in, err := Input(ctx, judged, Mesh{}, Scope{Node: "laptop"}); err != nil ||
!reflect.DeepEqual(in.Nodes[1].Handed, judged.handed) || in.Nodes[0].Handed != nil {
t.Errorf("the vault's handed verdicts: %+v, %v", in.Nodes, err)
}
if !reflect.DeepEqual(b.valuesRead, []string{"laptop." + overlay.Name, "laptop.xterm", "laptop.media"}) {
t.Errorf("laptop read config values %v; a node reads its own slice alone", b.valuesRead)
}
// The same composer over both: the compositions from the bus are the store's, byte for byte.
fps := map[string]string{overlay.Name: "src2:" + strings.Repeat("a", 64), "xterm": "src2:" + strings.Repeat("b", 64),
"media": "src2:" + strings.Repeat("c", 64)}
fromBus, _, held, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop"}, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatal(err)
}
fromStore, _, err := compose.Compositions(want, "laptop", fps, compose.Reading)
if err != nil {
t.Fatal(err)
}
if len(fromBus) == 0 || !reflect.DeepEqual(fromBus, fromStore) {
t.Fatalf("compositions differ or are empty: %d from the bus, %d from the store", len(fromBus), len(fromStore))
}
// Issue 1126: a candidate placement equal to the node's composes what it runs, and CompareComposed says so; one
// thing changed in the group's candidate of another node (the hub's endpoint) names exactly the module it moves
// here. Placed puts the same placement over the store's input unchanged.
running := make([]delivery.Composed, 0, len(fromBus))
for _, c := range fromBus {
running = append(running, c)
}
laptop := mk().placements["laptop"]
asIs, _, held, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop", Placements: map[string]*delivery.Placement{"laptop": &laptop}}, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatal(err)
}
if c := delivery.CompareComposed(asIs, running); len(c.Differs)+len(c.Added)+len(c.Removed) != 0 || len(c.Same) != len(running) {
t.Errorf("the candidate equal to the placement: %+v", c)
}
moved := mk().placements["anchor"]
moved.Decided.Endpoint = "anchor.example:51821"
changed, _, held, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop", Placements: map[string]*delivery.Placement{"laptop": &laptop, "anchor": &moved}}, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatal(err)
}
if c := delivery.CompareComposed(changed, running); len(c.Differs) != 1 || c.Differs[0].Module != overlay.Name || c.Differs[0].Resource != overlay.Name+".overlay-config" {
t.Errorf("the hub's endpoint changed: %+v, want %s alone, naming its config", c, overlay.Name)
}
if c := delivery.CompareComposed(changed, running[:0]); len(c.Added) != len(changed) {
t.Errorf("nothing runs: %+v", c)
}
// ADR 0559 §4, the node's answer: the same; one module differing; a module the group adds, whose own secret is not
// yet issued (no refusal); a value missing for a module the node runs (a refusal); a module the candidate removes.
keeper := []byte(`{"module":"keeper","version":"1","own-secrets":{"password":"/var/lib/mesh/keeper/password"},` +
`"resources":[{"id":"state","type":"directory","path":"/var/lib/mesh/keeper","mode":"0700"}]}`)
lemurs := []byte(`{"module":"lemurs","version":"1","capabilities":["systemd"],"resources":[{"id":"service","type":"service","unit":"lemurs.service","state":"running","boot":"enabled"}]}`)
// A provider and a consumer of a pair credential on the one node (compose's objects and files).
objects, _ := json.Marshal(catalogue.Manifest{Module: "objects", Version: "1",
Provides: []catalogue.Offer{{Name: "s3-bucket", Scope: catalogue.ScopeMesh, Identity: &catalogue.OfferIdentity{Max: 20, In: "an S3 access key"}}},
Receives: map[string]string{"s3-bucket": "/var/lib/mesh/objects/mesh.json"}})
files, _ := json.Marshal(catalogue.Manifest{Module: "files", Version: "1", Requires: []string{"s3-bucket"},
Resources: []map[string]any{{"id": "config", "type": "file", "path": "/etc/files/config.json", "mode": "0600", "content": "{}", "merge": "json"}}})
one := func(placed ...string) *bus {
p := delivery.Placement{Node: "n1", Decided: delivery.NodeDecided{Hub: true, Address: "10.77.0.1"}}
for _, m := range placed {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
return &bus{placements: map[string]delivery.Placement{"n1": p},
records: map[string]delivery.NodeRecord{"n1": {Node: "n1", TunnelKey: keys["anchor"], SealingKey: "sk", Capabilities: []string{"systemd"}}},
modules: map[string]delivery.Module{"keeper": {Manifest: delivery.ManifestDigest(keeper)}, "lemurs": {Manifest: delivery.ManifestDigest(lemurs)},
"objects": {Manifest: delivery.ManifestDigest(objects)}, "files": {Manifest: delivery.ManifestDigest(files)}},
manifests: map[string][]byte{delivery.ManifestDigest(keeper): keeper, delivery.ManifestDigest(lemurs): lemurs,
delivery.ManifestDigest(objects): objects, delivery.ManifestDigest(files): files},
own: []compose.OwnSecretRow{{Node: "n1", Module: "keeper", Name: "password", Origin: "made", SealedTo: "sk", Sealed: `"box"`}}}
}
oneFps := map[string]string{"keeper": "src2:" + strings.Repeat("d", 64), "lemurs": "src2:" + strings.Repeat("e", 64),
"objects": "src2:" + strings.Repeat("f", 64), "files": "src2:" + strings.Repeat("0", 64)}
runs, _, held, err := Compose(ctx, one("keeper", "lemurs"), Mesh{}, Scope{Node: "n1"}, oneFps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatal(err)
}
var runsNow []delivery.Composed
for _, c := range runs {
runsNow = append(runsNow, c)
}
objectsRun, _, held, err := Compose(ctx, one("objects"), Mesh{}, Scope{Node: "n1"}, oneFps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatal(err)
}
var consumerRuns []delivery.Composed
for _, c := range objectsRun {
consumerRuns = append(consumerRuns, c)
}
withoutOwn := func(cs []delivery.Composed) []delivery.Composed {
var out []delivery.Composed
for _, c := range cs {
if c.Module != "" {
out = append(out, c)
}
}
return out
}
// What the controller runs on n1 for a module no want names (lemurs, at a build of the old form) and the private
// network's (built by nobody): its composed writer (mesh-controller compositionsOf) folds their resources into the
// node's own, each in its place, its module taken off; the rest split by module.
tunnelled := func() *bus {
b := one("keeper", "lemurs", overlay.Name)
p := b.placements["n1"]
p.Decided.Endpoint = "n1.example:51820"
b.placements["n1"] = p
return b
}
tunnelledAs := func() map[string]*delivery.Placement {
p := tunnelled().placements["n1"]
return map[string]*delivery.Placement{"n1": &p}
}
folded := func() []delivery.Composed {
b := tunnelled()
in, err := Input(ctx, b, Mesh{}, Scope{Node: "n1"})
if err != nil {
t.Fatal(err)
}
reading := in
reading.Nodes = append([]compose.Node(nil), in.Nodes...)
reading.Nodes[0].ReadsModules = true
d, _, err := compose.ComposeIn(reading, "n1", compose.Reading)
if err != nil {
t.Fatal(err)
}
body, err := d.Body()
if err != nil {
t.Fatal(err)
}
var envelope struct {
Resources []map[string]json.RawMessage `json:"resources"`
}
if err := json.Unmarshal(body, &envelope); err != nil {
t.Fatal(err)
}
var resources []json.RawMessage
for _, r := range envelope.Resources {
var module string
_ = json.Unmarshal(r["module"], &module)
if module == "lemurs" || module == overlay.Name {
delete(r, "module")
}
raw, _ := json.Marshal(r)
resources = append(resources, raw)
}
split, err := delivery.SplitComposed("n1", resources, map[string]string{"keeper": oneFps["keeper"]}, nil, delivery.NodeOwn{})
if err != nil {
t.Fatal(err)
}
var out []delivery.Composed
for _, c := range split {
out = append(out, c)
}
return out
}()
if got := InOwn(folded, tunnelled().placements["n1"]); !reflect.DeepEqual(got, []string{"lemurs", overlay.Name}) {
t.Errorf("InOwn of the controller's composition = %v, want lemurs and %s", got, overlay.Name)
}
placedAs := func(modules ...string) map[string]*delivery.Placement {
p := one(modules...).placements["n1"]
return map[string]*delivery.Placement{"n1": &p}
}
for name, c := range map[string]struct {
b *bus
given map[string]*delivery.Placement
running []delivery.Composed
verdict string
check func(delivery.CandidateAnswer) bool
}{
"the same": {one("keeper", "lemurs"), placedAs("keeper", "lemurs"), runsNow, delivery.CandidateSame,
func(a delivery.CandidateAnswer) bool {
return len(a.Same) == 3 && a.Proposal == "p-h1" && a.Revision == 7
}},
"the node runs no own composition": {one("keeper", "lemurs"), placedAs("keeper", "lemurs"), withoutOwn(runsNow), delivery.CandidateSame,
func(a delivery.CandidateAnswer) bool { return len(a.Same) == 2 && len(a.Added) == 0 }},
"one removed": {one("keeper", "lemurs"), placedAs("keeper"), runsNow, delivery.CandidateDiffers,
func(a delivery.CandidateAnswer) bool { return reflect.DeepEqual(a.Removed, []string{"lemurs"}) }},
"one added, its secret not yet issued": {func() *bus { b := one("lemurs"); b.own = nil; return b }(), placedAs("keeper", "lemurs"),
nil, delivery.CandidateDiffers, func(a delivery.CandidateAnswer) bool {
for _, m := range a.Added {
if m.Module == "keeper" {
return reflect.DeepEqual(m.Unissued, []string{"keeper/password"}) && len(a.Refusals) == 0
}
}
return false
}},
"a value missing for a module that runs": {func() *bus { b := one("keeper", "lemurs"); b.own = nil; return b }(), placedAs("keeper", "lemurs"),
runsNow, delivery.CandidateRefused, func(a delivery.CandidateAnswer) bool {
return len(a.Refusals) == 1 && a.Refusals[0].Module == "keeper" && strings.Contains(a.Refusals[0].Why, "keeper/password")
}},
// Review of L2b, round 1, finding 1: a consumer the candidate adds has no credential yet, and the vault refuses an
// ask for a module not on the node; it is not asked, and the module is added, its credential not yet issued.
"a consumer added, the vault not asked for it": {one("objects"), placedAs("objects", "files"), consumerRuns, delivery.CandidateDiffers,
func(a delivery.CandidateAnswer) bool {
return len(a.Added) == 1 && a.Added[0].Module == "files" && len(a.Added[0].Unissued) > 0 && len(a.Refusals) == 0
}},
// And with no trunk at all (the seed), a stand-in of a module the node runs is a refusal, not an addition.
"the seed, a value missing for a module that runs": {func() *bus { b := one(); b.own = nil; return b }(), placedAs("keeper", "lemurs"),
runsNow, delivery.CandidateRefused, func(a delivery.CandidateAnswer) bool {
return len(a.Refusals) == 1 && a.Refusals[0].Module == "keeper" && len(a.Added) == 0
}},
"cannot compose": {one("keeper"), map[string]*delivery.Placement{"n1": {Node: "n2"}}, runsNow, delivery.CandidateRefused,
func(a delivery.CandidateAnswer) bool { return a.Refused != "" }},
// Issue 1126 (the seed's review): the private network's module and one at a build of the old form, which no want
// names and the controller folds into the node's own, are folded there again: the node's own is the same, and
// neither is added.
"mesh-wireguard and an old-form module, folded": {tunnelled(), tunnelledAs(),
folded, delivery.CandidateSame, func(a delivery.CandidateAnswer) bool { return len(a.Same) == 2 && len(a.Added) == 0 }},
} {
fps := oneFps
if name == "mesh-wireguard and an old-form module, folded" {
fps = map[string]string{"keeper": oneFps["keeper"]} // the node's wants: no build of lemurs or the private network
}
a := Answer(ctx, c.b, Mesh{}, Scope{Node: "n1", Placements: c.given, InOwn: InOwn(c.running, *c.given["n1"])}, fps, c.running, "p-h1", 7)
if a.Verdict != c.verdict || !c.check(a) {
t.Errorf("%s: %+v", name, a)
}
if raw, err := delivery.EncodeHaveNode(delivery.HaveNode{Candidates: map[string]delivery.CandidateAnswer{"g": a}}); err != nil {
t.Errorf("%s: the answer is not one a have holds: %v (%s)", name, err, raw)
}
}
if got, err := Placed(want, map[string]*delivery.Placement{"laptop": &laptop}); err != nil || !reflect.DeepEqual(got, want) {
t.Errorf("Placed with the node's own placement changes the input: %v", err)
}
if got, err := Placed(want, map[string]*delivery.Placement{"laptop": nil}); err != nil || len(got.Nodes[1].Assigned) != 0 || got.Nodes[1].Place.Key != keys["laptop"] {
t.Errorf("Placed with no placement: %+v, %v", got.Nodes[1], err)
}
if _, err := Placed(want, map[string]*delivery.Placement{"stranger": {Node: "stranger"}}); err == nil {
t.Error("Placed a node the input holds no node for")
}
if _, err := Input(ctx, mk(), Mesh{}, Scope{Node: "laptop", Placements: map[string]*delivery.Placement{"laptop": {Node: "anchor"}}}); err == nil {
t.Error("a given placement naming another node was taken")
}
// ADR 0555 §6: Export is Input's inverse for what a person decides: the store's input exported, node by node, is the
// placement the bus holds, and read back it is the store's input again.
for node, placed := range mk().placements {
got, err := Export(want, node)
if err != nil {
t.Fatal(err)
}
if gj, _ := delivery.EncodePlacement(got); !delivery.SamePlacement(got, placed) {
pj, _ := delivery.EncodePlacement(placed)
t.Errorf("%s exported as %s, want %s", node, gj, pj)
}
}
if _, err := Export(want, "stranger"); err == nil {
t.Error("a node the input lacks was exported")
}
b = mk()
if every, err := Input(ctx, b, Mesh{}, Scope{Every: true}); err != nil || every.OwnSecrets != nil {
t.Fatalf("composing every node: %v, own secrets %v (the vault answers a node about itself alone)", err, every.OwnSecrets)
}
if !reflect.DeepEqual(b.valuesRead, []string{"anchor." + overlay.Name, "laptop." + overlay.Name, "laptop.xterm", "laptop.media"}) {
// Every node's placed modules, and nothing placed nowhere: anchor places no xterm.
t.Errorf("the check read %v", b.valuesRead)
}
// A node none of whose modules declares an own secret composes without asking the vault (the review of L2: a node
// with no secret composed nothing while the vault did not answer); one whose module does, refuses without it.
quiet := mk()
quiet.ownErr = errors.New("no responders")
if _, err := Input(ctx, quiet, Mesh{}, Scope{Node: "anchor"}); err != nil {
t.Errorf("anchor, with no own secret, needed the vault: %v", err)
}
if _, err := Input(ctx, quiet, Mesh{}, Scope{Node: "laptop"}); err == nil {
t.Error("laptop's media, which declares an own secret, composed without the vault")
}
// A module the node's want keeps at another build than the trunk's composes from that build's manifest, resolved with
// what it made (issue 1126), on the composing node alone.
kept := mk()
oldXterm := []byte(`{"module":"xterm","version":"1","resources":[{"id":"tools","type":"process","name":"xterm-tools",` +
`"artifact":"tools","run":["./xterm-tools"]}],"build":{"artifacts":[{"name":"tools","kind":"bundle","language":"go","system":"arch","from":"cmd/xterm-tools","binary":"xterm-tools"}]}}`)
kept.manifests["old"] = oldXterm
oldFP := "src2:" + strings.Repeat("9", 64)
toolsRef := "artifact-store://xterm/tools/blobs/sha256:" + strings.Repeat("1", 64)
kept.builds = map[string]*delivery.BuilderBuild{"xterm " + oldFP: {State: delivery.BuildDone, Manifest: "old",
Artifacts: []delivery.BuiltArtifact{{Name: "tools", Kind: "bundle", Reference: toolsRef, Digest: "sha256:" + strings.Repeat("1", 64)}}}}
parsedOld, err := catalogue.Decode(oldXterm)
if err != nil {
t.Fatal(err)
}
keptBuilt, err := parsedOld.Resolve([]catalogue.Built{{Name: "tools", Kind: "bundle", Reference: toolsRef, Digest: "sha256:" + strings.Repeat("1", 64)}})
if err != nil {
t.Fatal(err)
}
in, err := Input(ctx, kept, Mesh{}, Scope{Node: "laptop", KeptAt: map[string]string{"xterm": oldFP}})
if err != nil || !reflect.DeepEqual(in.Nodes[1].Kept, map[string]catalogue.Manifest{"xterm": keptBuilt}) || in.Nodes[0].Kept != nil {
t.Errorf("the want's build is kept on the composing node alone: %v, %v", in.Nodes, err)
}
// Issue 1126, the seed's refusals. restic and the provisioner's client as the catalogue holds them (mesh-catalog
// c05aa5e6), restic moved with an own secret the vault leaves out (it hands it to the client, mesh-vault ownnode.go
// handed): composed from the indexer's manifest restic is not handed and stands in for that secret, as novox's,
// ace's, g14's and shanks's answers refused; composed from its build, it is handed, its templates archive read
// where the node reaches the artifact store, and the composition is the one the controller's take-in gives.
resticRaw, err := os.ReadFile("testdata/catalogue-c05aa5e6-restic.module.json")
if err != nil {
t.Fatal(err)
}
clientRaw, err := os.ReadFile("testdata/catalogue-c05aa5e6-mesh-provisioner-client.module.json")
if err != nil {
t.Fatal(err)
}
store := []byte(`{"module":"store","version":"1","provides":[{"name":"artifact-store","scope":"mesh"}],` +
`"serves":{"artifact-store":{"port":5000}}}`)
texts := map[string]string{}
var packed bytes.Buffer
zw := gzip.NewWriter(&packed)
tw := tar.NewWriter(zw)
for _, f := range []string{"backups.conf", "data.conf", "retention.conf"} {
text, err := os.ReadFile("testdata/catalogue-c05aa5e6-restic.files/" + f)
if err != nil {
t.Fatal(err)
}
texts[f] = string(text)
if err := tw.WriteHeader(&tar.Header{Name: "./" + f, Mode: 0o644, Size: int64(len(text)), Typeflag: tar.TypeReg}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write(text); err != nil {
t.Fatal(err)
}
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
if err := zw.Close(); err != nil {
t.Fatal(err)
}
archiveDigest := delivery.ManifestDigest(packed.Bytes())
digestOf := func(c string) string { return "sha256:" + strings.Repeat(c, 64) }
movedFps := map[string]string{"restic": "src2:" + strings.Repeat("3", 64), "mesh-provisioner-client": "src2:" + strings.Repeat("4", 64),
"store": "src2:" + strings.Repeat("5", 64)}
resticArtifacts := []delivery.BuiltArtifact{{Name: "tools", Kind: "bundle", Reference: "artifact-store://restic/tools/blobs/" + digestOf("6"), Digest: digestOf("6")},
{Name: "files", Kind: "archive", Reference: "artifact-store://restic/files/blobs/" + archiveDigest, Digest: archiveDigest}}
clientArtifacts := []delivery.BuiltArtifact{{Name: "client", Kind: "bundle", Reference: "artifact-store://mesh-provisioner-client/client/blobs/" + digestOf("7"), Digest: digestOf("7")}}
movedMesh := func(built bool) *bus {
b := one()
p := b.placements["n1"]
p.Modules = []delivery.PlacedModule{{Module: "store"}, {Module: "mesh-provisioner-client"}, {Module: "restic"}}
p.Decided.Account, p.Decided.AccountHome = "operator", "/home/operator"
b.placements["n1"] = p
rec := b.records["n1"]
rec.Contract, rec.Capabilities = compose.HandedContract, []string{"systemd", "package-manager"}
b.records["n1"] = rec
b.modules = map[string]delivery.Module{}
b.manifests = map[string][]byte{}
for module, raw := range map[string][]byte{"restic": resticRaw, "mesh-provisioner-client": clientRaw, "store": store} {
b.modules[module] = delivery.Module{FP2: movedFps[module], Manifest: delivery.ManifestDigest(raw)}
b.manifests[delivery.ManifestDigest(raw)] = raw
}
// The vault's own answer: the client's own secret; restic's is the client's to ask, so own leaves it out.
b.own = []compose.OwnSecretRow{{Node: "n1", Module: "mesh-provisioner-client", Name: "provisioner-broker", Origin: "made", SealedTo: "sk", Sealed: `"box"`}}
if built {
b.builds = map[string]*delivery.BuilderBuild{
"restic " + movedFps["restic"]: {State: delivery.BuildDone, Manifest: delivery.ManifestDigest(resticRaw), Artifacts: resticArtifacts},
"mesh-provisioner-client " + movedFps["mesh-provisioner-client"]: {State: delivery.BuildDone, Manifest: delivery.ManifestDigest(clientRaw), Artifacts: clientArtifacts},
"store " + movedFps["store"]: {State: delivery.BuildDone, Manifest: delivery.ManifestDigest(store)},
}
b.archives = map[string][]byte{archiveDigest: packed.Bytes()}
}
return b
}
if out, _, held, err := Compose(ctx, movedMesh(false), Mesh{}, Scope{Node: "n1"}, movedFps); err != nil || !strings.Contains(held["restic"], `restic needs a secret called "repository"`) {
t.Errorf("restic composed from the indexer's manifest: %v %v, want its own secret refusing restic alone", err, held)
} else {
for key, c := range out {
if c.Module == "restic" {
// Its stub's composition, applied, would remove what restic runs: a held module keeps its last one.
t.Errorf("held restic is composed as %s, want none of its own composition handed", key)
}
}
}
builtBus := movedMesh(true)
fromBuild, _, held, err := Compose(ctx, builtBus, Mesh{}, Scope{Node: "n1"}, movedFps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatalf("restic composed from its build: %v", err)
}
if !reflect.DeepEqual(builtBus.fetched, []string{"http://127.0.0.1:5000/v2/restic/files/blobs/" + archiveDigest}) {
t.Errorf("the templates archive was read at %v", builtBus.fetched)
}
resticKey, _ := delivery.ComposedKey("n1", "restic", movedFps["restic"])
if c := fromBuild[resticKey]; len(c.Filled) != 4 { // its three templated files and its own secret
t.Errorf("restic is not handed to the provisioner: its filled files are %v", c.Filled)
}
// A done key an artifact of which names no reference is waited for, in the controller's words (ADR 0498 §8).
unreferenced := movedMesh(true)
unreferenced.builds["restic "+movedFps["restic"]].Artifacts = []delivery.BuiltArtifact{{Name: "tools", Kind: "bundle", Digest: digestOf("6")}, resticArtifacts[1]}
if _, _, held, err := Compose(ctx, unreferenced, Mesh{}, Scope{Node: "n1"}, movedFps); err != nil || !strings.Contains(held["restic"], "the build of restic is done and waits for the reference of tools") {
t.Errorf("a done key with an unreferenced artifact: %v %v, want restic held alone", err, held)
}
// A module the node runs inside its own composition (the fold) composes from the indexer's manifest though a done
// build of it is at the trunk's FP2: its build is not read.
folding := movedMesh(true)
foldIn, err := Input(ctx, folding, Mesh{}, Scope{Node: "n1", InOwn: []string{"mesh-provisioner-client"}})
if err != nil {
t.Fatal(err)
}
clientParsed, err := catalogue.Decode(clientRaw)
if err != nil {
t.Fatal(err)
}
if slices.Contains(folding.buildsRead, "mesh-provisioner-client") || !reflect.DeepEqual(foldIn.Catalogue["mesh-provisioner-client"], clientParsed) {
t.Errorf("a folded module was taken from its build: builds read %v", folding.buildsRead)
}
// The controller's: its take-in resolves each manifest with the key's artifacts and the archive's texts, and composes
// from those over the same input.
storeIn, err := Input(ctx, movedMesh(false), Mesh{}, Scope{Node: "n1"})
if err != nil {
t.Fatal(err)
}
for module, built := range map[string][]catalogue.Built{
"restic": {{Name: "tools", Kind: "bundle", Reference: resticArtifacts[0].Reference, Digest: digestOf("6")},
{Name: "files", Kind: "archive", Reference: resticArtifacts[1].Reference, Digest: archiveDigest, Texts: texts}},
"mesh-provisioner-client": {{Name: "client", Kind: "bundle", Reference: clientArtifacts[0].Reference, Digest: digestOf("7")}},
} {
resolved, err := storeIn.Catalogue[module].Resolve(built)
if err != nil {
t.Fatal(err)
}
// Kept as its bytes and read again, as the controller's store holds a build's manifest.
raw, _ := json.Marshal(resolved)
if storeIn.Catalogue[module], err = catalogue.Decode(raw); err != nil {
t.Fatal(err)
}
}
storeIn.OwnSecrets = movedMesh(true).own
fromStoreMoved, _, err := compose.Compositions(storeIn, "n1", movedFps, compose.Reading)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(fromBuild, fromStoreMoved) {
gj, _ := json.MarshalIndent(fromBuild, "", " ")
wj, _ := json.MarshalIndent(fromStoreMoved, "", " ")
t.Errorf("composed from the build differs from the controller's take-in:\n got %s\nwant %s", gj, wj)
}
// ADR 0570 clause 2 (issue 1954 L6-P1): the seat mesh-provisioner's plans, composed from the bus alone and with no
// secret read (the Planning mode), are the controller's handedPlans over its own input, which holds its secrets
// (compose_sdk.go: Resolve, Render in Foreseeing, Generators, Plan.Compose), and restic, handed, has one.
noSecrets := movedMesh(true)
noSecrets.own = nil
seat, err := Plans(ctx, noSecrets, Mesh{}, Scope{Node: "n1"})
if err != nil {
t.Fatalf("the plans from the bus: %v", err)
}
controllers, err := plansIn(storeIn, "n1", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
if _, handed := seat["restic"]; !handed || !reflect.DeepEqual(seat, controllers) {
gj, _ := json.MarshalIndent(seat, "", " ")
wj, _ := json.MarshalIndent(controllers, "", " ")
t.Errorf("the seat's plans differ from the controller's:\n got %s\nwant %s", gj, wj)
}
// ADR 0571 clause 3: the data items of n1's modules, placed, from the bus alone: what disk-load measures.
items, unplaced, err := DataItems(ctx, noSecrets, Mesh{}, Scope{Node: "n1"})
if err != nil || len(unplaced) > 0 || len(items) == 0 {
t.Errorf("the data items from the bus are %v, unplaced %v: %v", items, unplaced, err)
}
for module, its := range items {
for _, it := range its {
if it.Module != module || !strings.HasPrefix(it.Path, "/") || it.Measure == "" {
t.Errorf("%s's item %+v is not placed", module, it)
}
}
}
// And the same through StatesReader, over the fake's states as a module's runtime gives them (one reader for
// mesh-provisioner and disk-load, the lead 2026-10-29).
viaStates, _, err := DataItems(ctx, StatesReader(statesOf(t, ctx, noSecrets), func(_ context.Context, url string) ([]byte, error) {
for _, body := range noSecrets.archives {
return body, nil
}
return nil, errors.New("no archive")
}), Mesh{}, Scope{Node: "n1"})
if err != nil || !reflect.DeepEqual(viaStates, items) {
t.Errorf("through the states the data items are %v, %v; through the reader %v", viaStates, err, items)
}
// And what a provider on another node serves from its settings (mailu's smtp domain, issue 1126: shanks refused
// '${setting:domain} ... nothing sets "domain"'): read for that provider's served keys alone, through Served.
mailer := []byte(`{"module":"mailer","version":"1","config":{"domain":{"type":"string","description":"the mail domain","required":true}},` +
`"provides":[{"name":"smtp","scope":"mesh"}],"serves":{"smtp":{"port":587,"domain":"${config:domain}"}}}`)
withMailer := func(set bool) *bus {
b := mk()
p := b.placements["anchor"]
p.Modules = append(p.Modules, delivery.PlacedModule{Module: "mailer"})
b.placements["anchor"] = p
b.modules["mailer"] = delivery.Module{Manifest: delivery.ManifestDigest(mailer)}
b.manifests[delivery.ManifestDigest(mailer)] = mailer
if set {
b.served = map[string]map[string]any{"anchor.mailer": {"domain": "example.org"}}
}
return b
}
if _, _, _, err := Compose(ctx, withMailer(false), Mesh{}, Scope{Node: "laptop"}, fps); err == nil || !strings.Contains(err.Error(), `nothing sets "domain"`) {
t.Errorf("a provider's served setting nobody sets: %v", err)
}
served := withMailer(true)
if _, _, held, err := Compose(ctx, served, Mesh{}, Scope{Node: "laptop"}, fps); err != nil || len(held) > 0 {
t.Errorf("laptop with what anchor's mailer serves: %v %v", err, held)
}
if !reflect.DeepEqual(served.servedRead, []string{"anchor.mailer:domain"}) || slices.Contains(served.valuesRead, "anchor.mailer") {
t.Errorf("laptop read %v of other nodes (and %v of config values): the served keys of anchor's mailer alone", served.servedRead, served.valuesRead)
}
for name, c := range map[string]struct {
change func(*bus)
scope Scope
says string
}{
"no record": {func(b *bus) { delete(b.records, "laptop") }, Scope{Node: "anchor"}, "laptop, whose agent has written no record"},
"no placement": {func(b *bus) {}, Scope{Node: "stranger"}, "stranger has no placement"},
"a manifest unread": {func(b *bus) { delete(b.manifests, digest) }, Scope{Node: "laptop"}, "the manifest of xterm"},
"a kept one unbuilt": {func(b *bus) {}, Scope{Node: "laptop", KeptAt: map[string]string{"xterm": "src2:gone"}}, "holds no done build"},
} {
b := mk()
c.change(b)
if _, err := Input(ctx, b, Mesh{}, c.scope); err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: %v, want it to say %q", name, err, c.says)
}
}
}
// mapStates are read states by name, key to value, as a module's runtime gives them.
type mapStates map[string]map[string][]byte
func (m mapStates) Get(state, key string) ([]byte, bool, error) {
v, ok := m[state][key]
return v, ok, nil
}
func (m mapStates) Keys(state string) ([]string, error) {
var out []string
for k := range m[state] {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
// statesOf is b laid out as the states StatesReader reads.
func statesOf(t *testing.T, ctx context.Context, b *bus) mapStates {
t.Helper()
out := mapStates{}
put := func(state, key string, raw []byte, err error) {
if err != nil {
t.Fatal(err)
}
if out[state] == nil {
out[state] = map[string][]byte{}
}
out[state][key] = raw
}
placements, _ := b.Placements(ctx)
for _, p := range placements {
values, err := delivery.PlacementValues(p)
for k, raw := range values {
put(PlacementRead, k, raw, err)
}
}
records, _ := b.Records(ctx)
for node, r := range records {
k, _ := delivery.HaveNodeKey(node)
rec := r
raw, err := delivery.EncodeHaveNode(delivery.HaveNode{Record: &rec})
put(HaveRead, k, raw, err)
}
modules, _ := b.Modules(ctx)
for name, m := range modules {
raw, err := delivery.EncodeModule(m)
put(ModulesRead, name, raw, err)
var mf struct{ Module string }
_ = json.Unmarshal(b.manifests[m.Manifest], &mf)
k, _ := delivery.ManifestsKey(m.Manifest)
raw, err = delivery.EncodeManifest(delivery.Manifest{Module: mf.Module, JSON: string(b.manifests[m.Manifest])})
put(ManifestsRead, k, raw, err)
}
for key, build := range b.builds {
module, fp, _ := strings.Cut(key, " ")
k, _ := delivery.BuilderBuildKey(module, fp)
raw, err := delivery.EncodeBuilderBuild(*build)
put(BuildsRead, k, raw, err)
}
for nm, values := range b.values {
for k, v := range values {
value, _ := json.Marshal(v)
raw, err := json.Marshal(map[string]any{"value": json.RawMessage(value), "source": catalogue.SourceModule})
put(ResolvedRead, nm+"."+k, raw, err)
}
}
return out
}
@@ -0,0 +1,183 @@
// Package frombustest is a bus in memory for frombus's Reader, and a mesh on it whose composition hands a module to the
// provisioner with a provision to make, so a module that composes plans from the bus (mesh-provisioner, novox/hq ADR
// 0570 clause 2) tests against a real plan rather than one written by hand. The manifests are the catalogue's (mesh-catalog
// c05aa5e6: restic and the provisioner's client) as frombus's own tests read them, restic given one requirement more.
package frombustest
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"embed"
"encoding/json"
"errors"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose"
"github.com/novox/mesh-host/internal/compose/frombus"
)
//go:embed testdata
var testdata embed.FS
// Bus is a frombus.Reader over values held in memory. A credential asked is answered held when Held is set.
type Bus struct {
Placement map[string]delivery.Placement
Record map[string]delivery.NodeRecord
Module map[string]delivery.Module
Manifests map[string][]byte
Value map[string]map[string]any // by node.module
Own []compose.OwnSecretRow
Builds map[string]*delivery.BuilderBuild // by "<module> <fp>"
Archives map[string][]byte // by digest
Serves map[string]map[string]any // another node's values, by node.module
Held bool
}
var _ frombus.Reader = (*Bus)(nil)
func (b *Bus) Placements(context.Context) (map[string]delivery.Placement, error) {
return b.Placement, nil
}
func (b *Bus) Records(context.Context) (map[string]delivery.NodeRecord, error) { return b.Record, nil }
func (b *Bus) Modules(context.Context) (map[string]delivery.Module, error) { return b.Module, nil }
func (b *Bus) VaultAuthority(context.Context) string { return "" }
func (b *Bus) Manifest(_ context.Context, digest string) ([]byte, error) {
raw, ok := b.Manifests[digest]
if !ok {
return nil, errors.New("no such manifest")
}
return raw, nil
}
func (b *Bus) Values(_ context.Context, node, module string) (map[string]any, error) {
return b.Value[node+"."+module], nil
}
func (b *Bus) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow, error) { return b.Own, nil }
func (b *Bus) Credentials(_ context.Context, _ string, asConsumer, asProvider []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
var out []compose.Credential
var issued []compose.Issued
for _, k := range append(append([]compose.CredentialKey{}, asConsumer...), asProvider...) {
if b.Held {
out = append(out, compose.Credential{Provision: k.Provision, Consumer: k.Consumer, ConsumerModule: k.ConsumerModule,
Provider: k.Provider, Local: k.Local, State: compose.CredentialHeld, Sealed: `"box"`})
issued = append(issued, compose.Issued{Provision: k.Provision, Consumer: k.Consumer, ConsumerModule: k.ConsumerModule,
Provider: k.Provider, Local: k.Local, ForProvider: `"box"`})
}
}
return out, issued, nil
}
func (b *Bus) Build(_ context.Context, module, fp string) (*delivery.BuilderBuild, error) {
return b.Builds[module+" "+fp], nil
}
func (b *Bus) Archive(_ context.Context, _, digest string) ([]byte, error) {
body, ok := b.Archives[digest]
if !ok {
return nil, errors.New("the store holds no such blob")
}
return body, nil
}
func (b *Bus) Served(_ context.Context, node, module string, keys []string) (map[string]any, error) {
out := map[string]any{}
for _, k := range keys {
if v, ok := b.Serves[node+"."+module][k]; ok {
out[k] = v
}
}
return out, nil
}
// Node is the one node of the mesh Provisioned composes.
const Node = "n1"
// Provisioned is one node, n1, that runs the artifact store, a postgres providing postgres-database, the provisioner's
// client and restic: restic built and handed to the provisioner, and requiring postgres-database. Its plans (frombus.Plans
// with Scope{Node: Node}) hand restic, whose plan names that provision with postgres on n1 as its provider.
func Provisioned() (*Bus, error) {
read := func(name string) ([]byte, error) { return testdata.ReadFile("testdata/" + name) }
resticRaw, err := read("catalogue-c05aa5e6-restic.module.json")
if err != nil {
return nil, err
}
var restic map[string]any
if err := json.Unmarshal(resticRaw, &restic); err != nil {
return nil, err
}
restic["requires"] = []string{"postgres-database"}
if resticRaw, err = json.Marshal(restic); err != nil {
return nil, err
}
clientRaw, err := read("catalogue-c05aa5e6-mesh-provisioner-client.module.json")
if err != nil {
return nil, err
}
store := []byte(`{"module":"store","version":"1","provides":[{"name":"artifact-store","scope":"mesh"}],` +
`"serves":{"artifact-store":{"port":5000}}}`)
postgres := []byte(`{"module":"postgres","version":"1","provides":[{"name":"postgres-database","scope":"mesh"}],` +
`"serves":{"postgres-database":{"port":5432}}}`)
var packed bytes.Buffer
zw := gzip.NewWriter(&packed)
tw := tar.NewWriter(zw)
for _, f := range []string{"backups.conf", "data.conf", "retention.conf"} {
text, err := read("catalogue-c05aa5e6-restic.files/" + f)
if err != nil {
return nil, err
}
if err := tw.WriteHeader(&tar.Header{Name: "./" + f, Mode: 0o644, Size: int64(len(text)), Typeflag: tar.TypeReg}); err != nil {
return nil, err
}
if _, err := tw.Write(text); err != nil {
return nil, err
}
}
if err := tw.Close(); err != nil {
return nil, err
}
if err := zw.Close(); err != nil {
return nil, err
}
archive := delivery.ManifestDigest(packed.Bytes())
digestOf := func(c string) string { return "sha256:" + strings.Repeat(c, 64) }
fps := FPs()
b := &Bus{
Placement: map[string]delivery.Placement{Node: {Node: Node,
Decided: delivery.NodeDecided{Hub: true, Address: "10.77.0.1", Account: "operator", AccountHome: "/home/operator"},
Modules: []delivery.PlacedModule{{Module: "store"}, {Module: "postgres"}, {Module: "mesh-provisioner-client"}, {Module: "restic"}}}},
Record: map[string]delivery.NodeRecord{Node: {Node: Node, TunnelKey: "dGVzdGtleXRlc3RrZXl0ZXN0a2V5dGVzdGtleXRlc3Q=",
SealingKey: "sk", Contract: compose.HandedContract, Capabilities: []string{"systemd", "package-manager"}}},
Module: map[string]delivery.Module{}, Manifests: map[string][]byte{},
Own: []compose.OwnSecretRow{{Node: Node, Module: "mesh-provisioner-client", Name: "provisioner-broker", Origin: "made", SealedTo: "sk", Sealed: `"box"`}},
Builds: map[string]*delivery.BuilderBuild{
"restic " + fps["restic"]: {State: delivery.BuildDone, Manifest: delivery.ManifestDigest(resticRaw), Artifacts: []delivery.BuiltArtifact{
{Name: "tools", Kind: "bundle", Reference: "artifact-store://restic/tools/blobs/" + digestOf("6"), Digest: digestOf("6")},
{Name: "files", Kind: "archive", Reference: "artifact-store://restic/files/blobs/" + archive, Digest: archive}}},
"mesh-provisioner-client " + fps["mesh-provisioner-client"]: {State: delivery.BuildDone, Manifest: delivery.ManifestDigest(clientRaw),
Artifacts: []delivery.BuiltArtifact{{Name: "client", Kind: "bundle",
Reference: "artifact-store://mesh-provisioner-client/client/blobs/" + digestOf("7"), Digest: digestOf("7")}}},
"store " + fps["store"]: {State: delivery.BuildDone, Manifest: delivery.ManifestDigest(store)},
"postgres " + fps["postgres"]: {State: delivery.BuildDone, Manifest: delivery.ManifestDigest(postgres)},
},
Archives: map[string][]byte{archive: packed.Bytes()},
Held: true,
}
for module, raw := range map[string][]byte{"restic": resticRaw, "mesh-provisioner-client": clientRaw, "store": store, "postgres": postgres} {
b.Module[module] = delivery.Module{FP2: fps[module], Manifest: delivery.ManifestDigest(raw)}
b.Manifests[delivery.ManifestDigest(raw)] = raw
}
return b, nil
}
// FPs are the fingerprints Provisioned's modules are built at.
func FPs() map[string]string {
return map[string]string{"restic": "src2:" + strings.Repeat("3", 64), "mesh-provisioner-client": "src2:" + strings.Repeat("4", 64),
"store": "src2:" + strings.Repeat("5", 64), "postgres": "src2:" + strings.Repeat("8", 64)}
}
@@ -0,0 +1,272 @@
package frombustest
import (
"context"
"reflect"
"strings"
"testing"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose/frombus"
"github.com/novox/mesh-host/internal/compose/fromruntime"
)
// ADR 0570 clause 2 (issue 1954; a3's H5): the in-memory mesh composes a plan from the bus that hands restic to the
// provisioner and names its postgres-database, provided by postgres on n1. And one bad module must not drop its node
// (issue 774): with a module placed beside it whose built manifest cannot be used, or whose plan refuses, PlansHeld
// still hands restic's plan and names the bad one with why, where Plans refuses the whole node. A manifest the indexer
// holds that the mesh refuses (power's bar item with attributes its bar places by none of, mesh-catalog ca8d6f33,
// 2026-10-30) holds that module alone on its node, in the plans and in the node's composition (Compose), and on a node
// it is not placed on holds nothing.
func TestTheProvisionedMeshHandsAPlanWithItsProvision(t *testing.T) {
fp := "src2:" + strings.Repeat("9", 64)
for _, row := range []struct {
name string
bad string // the indexer's manifest of a module "backup" placed on n1; empty places none
// built is the manifest backup's done build names; empty gives it no build
built string
why string // words backup's hold says
// elsewhere leaves backup in the indexer's modules and places it on no node: nothing is held.
elsewhere bool
// refused is the indexer's verdict on bad, a problem its judge found (novox/hq ADR 0578 §1): Plans then holds
// backup too, and never ends the read.
refused string
// dataHeld says the data items' read holds backup too: a refusal it meets (a refused manifest, an unusable build,
// an item composition refuses), not one only a plan meets.
dataHeld bool
// dataOnly checks the data items' read alone: the other reads blame restic, whose templates collect the item.
dataOnly bool
}{
{name: "every module composes"},
{name: "a built manifest that cannot be used",
bad: `{"module":"backup","version":"1","build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]}}`,
built: `{"module":"backup","version":"1","requires":"amqp"}`, why: "backup", dataHeld: true},
{name: "a plan that refuses",
bad: `{"module":"backup","version":"1","resources":[{"id":"conf","type":"file","path":"/etc/backup.conf",` +
`"mode":"0600","content":"key=${secret:nope}"}]}`, why: "${secret:nope}"},
{name: "an indexer's manifest the mesh refuses", bad: unusableItem, why: `"shows" is none of them`,
refused: `"shows" is none of them`, dataHeld: true},
// ADR 0578 §3: an accepted manifest whose data item its composition refuses is held by the culprit rule, in every read.
{name: "a data item its composition refuses",
bad: `{"module":"backup","version":"1","data":{"own":[{"id":"d","path":"${dir:nope}","class":"cache"}]}}`,
why: "nope", dataHeld: true, dataOnly: true},
{name: "an indexer's manifest the mesh refuses, placed elsewhere", bad: unusableItem, elsewhere: true,
refused: `"shows" is none of them`},
} {
b, err := Provisioned()
if err != nil {
t.Fatal(err)
}
if row.bad != "" {
if !row.elsewhere {
p := b.Placement[Node]
p.Modules = append(p.Modules, delivery.PlacedModule{Module: "backup"})
b.Placement[Node] = p
}
digest := delivery.ManifestDigest([]byte(row.bad))
b.Manifests[digest] = []byte(row.bad)
b.Module["backup"] = delivery.Module{FP2: fp, Manifest: digest}
if row.refused != "" {
b.Module["backup"] = delivery.Module{FP2: fp, Manifest: digest, Verdict: delivery.VerdictRefused,
Problems: []string{row.refused}}
}
if row.built != "" {
builtDigest := delivery.ManifestDigest([]byte(row.built))
b.Manifests[builtDigest] = []byte(row.built)
b.Builds["backup "+fp] = &delivery.BuilderBuild{State: delivery.BuildDone, Manifest: builtDigest,
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Reference: "artifact-store://backup/files/blobs/x", Digest: "sha256:x"}}}
}
if row.dataOnly {
items, _, iheld, err := frombus.DataItemsHeld(context.Background(), b, frombus.Mesh{}, frombus.Scope{Node: Node})
if why, ok := iheld["backup"]; err != nil || !ok || !strings.Contains(why, row.why) || len(items["restic"]) == 0 {
t.Errorf("%s: data items %v held %v: %v", row.name, items, iheld, err)
}
continue
}
plans, err := frombus.Plans(context.Background(), b, frombus.Mesh{}, frombus.Scope{Node: Node})
switch {
case row.refused != "" && (err != nil || !planned(plans, "restic") || planned(plans, "backup")):
t.Errorf("%s: Plans did not hold the refused manifest alone: %v %v", row.name, plans, err)
case row.refused == "" && err == nil && !row.elsewhere:
t.Errorf("%s: Plans composed the node; the row does not refuse it", row.name)
}
}
if row.why != "" || row.bad == "" {
// The node's own composition holds by the same rule: every other module composed, the bad one named.
fps := FPs()
if row.bad != "" {
fps["backup"] = fp
}
out, _, held, err := frombus.Compose(context.Background(), b, frombus.Mesh{}, frombus.Scope{Node: Node}, fps)
if err != nil {
t.Fatalf("%s: the node's composition: %v", row.name, err)
}
if key, _ := delivery.ComposedKey(Node, "restic", fps["restic"]); out[key].Module != "restic" {
t.Errorf("%s: restic is not composed (held %v)", row.name, held)
}
if why, ok := held["backup"]; (row.bad == "" && len(held) > 0) || (row.bad != "" && (!ok || !strings.Contains(why, row.why) || len(held) != 1)) {
t.Errorf("%s: the node's composition held %v, want backup alone, saying %q", row.name, held, row.why)
}
}
plans, held, err := frombus.PlansHeld(context.Background(), b, frombus.Mesh{}, frombus.Scope{Node: Node})
if err != nil {
t.Fatalf("%s: %v", row.name, err)
}
restic, handed := plans["restic"]
if !handed || len(restic.Provisions) != 1 || restic.Provisions[0].Provision != "postgres-database" ||
restic.Provisions[0].Provider != Node || restic.Provisions[0].ProviderModule != "postgres" {
t.Fatalf("%s: the plans are %+v (held %v)", row.name, plans, held)
}
// Issue 1954: the plans at the wanted builds and at the newest, over one read, are what PlansHeld answers for each,
// and where the time went is said by phase.
keptAt := FPs()
at, newest, took := frombus.PlansHeldTwice(context.Background(), b, frombus.Mesh{}, frombus.Scope{Node: Node, KeptAt: keptAt})
wantAt, heldAt, errAt := frombus.PlansHeld(context.Background(), b, frombus.Mesh{}, frombus.Scope{Node: Node, KeptAt: keptAt})
if !reflect.DeepEqual(at, frombus.HeldPlans{Plans: wantAt, Held: heldAt, Err: errAt}) ||
!reflect.DeepEqual(newest, frombus.HeldPlans{Plans: plans, Held: held}) {
t.Errorf("%s: twice composed %+v and %+v, want %v %v %v and %v %v", row.name, at, newest, wantAt, heldAt, errAt, plans, held)
}
if said := frombus.PhasesSaid(took); !strings.HasPrefix(said, "read ") || !strings.Contains(said, "plans at newest ") {
t.Errorf("%s: the phases are said as %q", row.name, said)
}
if row.bad == "" || row.elsewhere {
if len(held) > 0 {
t.Errorf("%s: held %v", row.name, held)
}
continue
}
if why, ok := held["backup"]; !ok || !strings.Contains(why, row.why) || len(held) != 1 {
t.Errorf("%s: held %v, want backup alone, saying %q", row.name, held, row.why)
}
if _, planned := plans["backup"]; planned {
t.Errorf("%s: the held module has a plan", row.name)
}
// The data items are read by the same rule (ADR 0578 §3: disk-load's read): the bad module held, the read whole.
items, _, iheld, err := frombus.DataItemsHeld(context.Background(), b, frombus.Mesh{}, frombus.Scope{Node: Node})
// A refusal only a plan meets (a file's secret) holds nothing here: data items render no such file.
mustHold := row.dataHeld
if why, ok := iheld["backup"]; err != nil || ok != mustHold || mustHold && !strings.Contains(why, row.why) || items["backup"] != nil {
t.Errorf("%s: data items %v held %v: %v", row.name, items, iheld, err)
}
}
// ADR 0580 clause 1: the watched reader reads a state it composes from out of the copy, a build by key each time it is
// asked, and a manifest once per digest.
byKey := map[string]int{}
ws := stdio.WatchedStatesOver(func(state string, handle func(stdio.StateChange) error) error {
byKey[state+" watched"]++
return handle(stdio.StateChange{Key: "placement." + Node, Op: stdio.OpPut, Value: []byte(`{}`), Current: true})
}, func(state, key string) ([]byte, bool, error) { byKey[state]++; return []byte(key), true, nil })
w := fromruntime.Watched{WatchedStates: ws}
for range 2 {
_, _, _ = w.Get(frombus.BuildsRead, "restic."+fp)
_, _, _ = w.Get(frombus.ManifestsRead, "sha256-d")
if _, found, _ := w.Get(frombus.PlacementRead, "placement."+Node); !found {
t.Error("the watched reader does not answer the placement from its copy")
}
}
if byKey[frombus.BuildsRead] != 2 || byKey[frombus.ManifestsRead] != 1 || byKey[frombus.PlacementRead] != 0 ||
byKey[frombus.PlacementRead+" watched"] != 1 {
t.Errorf("the watched reader read %v, want a build by key each time, a manifest once, the placement watched once", byKey)
}
fetched := 0
keep := fromruntime.KeptFetch(func(context.Context, string) ([]byte, error) { fetched++; return []byte("tar"), nil })
for range 2 {
if body, err := keep(context.Background(), "http://store/restic/files/blobs/sha256:a"); err != nil || string(body) != "tar" {
t.Fatalf("the kept fetch answered %q, %v", body, err)
}
}
if fetched != 1 {
t.Errorf("a templates archive was fetched %d times, want once", fetched)
}
// Issue 1954: the states reader takes a build in once per its content: composed again, its archive is not read again
// and the plans are the same.
b, err := Provisioned()
if err != nil {
t.Fatal(err)
}
reader, archives := frombus.StatesReader(statesOf(t, b), func(_ context.Context, url string) ([]byte, error) {
fetched++
_, digest, _ := strings.Cut(url, "/blobs/")
return b.Archives[digest], nil
}), fetched
var first map[string]catalogue.Plan
for i := range 2 {
plans, held, err := frombus.PlansHeld(context.Background(), reader, frombus.Mesh{}, frombus.Scope{Node: Node})
if err != nil || len(held) > 0 || (i == 1 && !reflect.DeepEqual(plans, first)) || !planned(plans, "restic") {
t.Fatalf("composed %v (held %v): %v", plans, held, err)
}
first = plans
}
if fetched-archives != 1 {
t.Errorf("restic's archive was read %d times over two compositions, want once", fetched-archives)
}
}
// statesOf is the bus b as a module's read states hold it.
func statesOf(t *testing.T, b *Bus) frombus.States {
t.Helper()
s := states{}
put := func(state, key string, raw []byte, err error) {
if err != nil {
t.Fatal(err)
}
if s[state] == nil {
s[state] = map[string][]byte{}
}
s[state][key] = raw
}
for node, p := range b.Placement {
raw, err := delivery.EncodePlacement(p)
put(frombus.PlacementRead, "placement."+node, raw, err)
}
for node, rec := range b.Record {
raw, err := delivery.EncodeHaveNode(delivery.HaveNode{Record: &rec})
put(frombus.HaveRead, node+".node", raw, err)
}
for module, m := range b.Module {
key, _ := delivery.ModulesKey(module)
raw, err := delivery.EncodeModule(m)
put(frombus.ModulesRead, key, raw, err)
}
for digest, man := range b.Manifests {
key, _ := delivery.ManifestsKey(digest)
raw, err := delivery.EncodeManifest(delivery.Manifest{Module: "m", JSON: string(man)})
put(frombus.ManifestsRead, key, raw, err)
}
for at, build := range b.Builds {
module, fp, _ := strings.Cut(at, " ")
key, _ := delivery.BuilderBuildKey(module, fp)
raw, err := delivery.EncodeBuilderBuild(*build)
put(frombus.BuildsRead, key, raw, err)
}
return s
}
// states are read states in memory.
type states map[string]map[string][]byte
func (s states) Get(state, key string) ([]byte, bool, error) {
v, ok := s[state][key]
return v, ok, nil
}
func (s states) Keys(state string) ([]string, error) {
var keys []string
for k := range s[state] {
keys = append(keys, k)
}
return keys, nil
}
func planned[P any](plans map[string]P, module string) bool { _, ok := plans[module]; return ok }
// unusableItem is a manifest the mesh refuses as the node-engine did power's at mesh-catalog ca8d6f33: a bar block item
// with attributes its bar places by none of.
const unusableItem = `{"module":"backup","version":"1","moved":true,` +
`"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]},` +
`"items":[{"collection":"contribution:node-bar:block",` +
`"attributes":{"bar":"bottom","place":"status","shows":"state","state":"power.draw"},"template":"files/bar"}]}`
@@ -0,0 +1,60 @@
{
"module": "mesh-provisioner-client",
"version": "1",
"claims": [
{
"name": "node-provisioner",
"scope": "node"
}
],
"own-secrets": {
"provisioner-broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-provisioner-client",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-provisioner-client",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-provisioner-client"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-provisioner-client",
"place": ".",
"unknown": "program"
},
{
"id": "client",
"type": "process",
"name": "mesh-provisioner-client",
"artifact": "client",
"run": [
"./mesh-provisioner-client"
],
"env": {
"MESH_PROVISIONER_CREDENTIAL": "${dir:state}/broker",
"MESH_PROVISIONER_STATE": "${dir:state}"
},
"user": "mesh-provisioner-client",
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "client",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-provisioner-client",
"binary": "mesh-provisioner-client"
}
]
}
}
@@ -0,0 +1,2 @@
# What the modules on this machine back up, composed by the mesh (novox/hq to-be 43). Do not edit.
${backup:lines}
@@ -0,0 +1,2 @@
# The data the modules on this machine declare, composed by the mesh (novox/hq ADR 0233). Do not edit.
${backup:data}
@@ -0,0 +1,6 @@
# How many restore points the night keeps, written by the mesh from this machine's settings keep-daily,
# keep-weekly, keep-monthly (novox/hq issue 416) and keep-hand-taken-days (novox/hq issue 453). Replaced on every push; change the settings, never this file.
daily=${config:keep-daily}
weekly=${config:keep-weekly}
monthly=${config:keep-monthly}
hand=${config:keep-hand-taken-days}
@@ -0,0 +1,199 @@
{
"module": "restic",
"version": "1",
"moved": true,
"claims": [
{
"name": "node-backup",
"scope": "node",
"serves": [
"backed-up",
"now",
"restore"
]
}
],
"emits": [
"backed-up"
],
"tools": [
"backup_health",
"backup_delete_retired",
"backup_deleted"
],
"own-secrets": {
"repository": "${dir:state}/repository.secret"
},
"data": {
"own": [
{
"id": "repository",
"path": "${dir:repository}",
"class": "rebuildable",
"backup": "none",
"why": "the restore points themselves: a copy of data kept elsewhere on this machine, never the only copy of anything; lost, the history goes and nothing live does"
},
{
"id": "state",
"path": "${dir:state}",
"class": "cache",
"why": "what the last nights and measurements were; the next night writes it again"
}
]
},
"config": {
"keep-daily": {
"type": "number",
"description": "How many nightly restore points are kept, a count from 1 to 9999: two weeks undo a mistake noticed within a fortnight. Read by the night, which thins nothing while it does not read (novox/hq issue 416); lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 14
},
"keep-weekly": {
"type": "number",
"description": "How many weekly restore points are kept behind the dailies, a count from 1 to 9999, refused by the night like keep-daily; lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 8
},
"keep-monthly": {
"type": "number",
"description": "How many monthly restore points are kept behind the weeklies, a count from 1 to 9999, refused by the night like keep-daily; lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 6
},
"keep-hand-taken-days": {
"type": "number",
"description": "How many days a restore point taken by hand (node-backup.now, or the mesh's before a module moves) is kept at least, a count from 1 to 9999; it is kept until a later good night of its module exists, and this many days in any case, so the next thinning never takes the one taken against a risky change (novox/hq issue 453). Lowering it lets go of the older ones taken by hand at the next thinning, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 7
}
},
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": ".",
"unknown": "program"
},
{
"id": "repository",
"type": "directory",
"mode": "0700",
"unknown": "program"
},
{
"id": "declared",
"type": "file",
"path": "${dir:state}/backups.conf",
"mode": "0600",
"template": "files/backups.conf"
},
{
"id": "data-declared",
"type": "file",
"path": "${dir:state}/data.conf",
"mode": "0600",
"template": "files/data.conf"
},
{
"id": "retention",
"type": "file",
"path": "${dir:state}/retention.conf",
"mode": "0644",
"trusted": true,
"template": "files/retention.conf"
},
{
"id": "tool",
"type": "package",
"package": "restic"
},
{
"id": "sqlite",
"type": "package",
"package": "sqlite"
},
{
"id": "night",
"type": "process",
"name": "restic-night",
"artifact": "tools",
"run": [
"./restic-backups",
"night"
],
"user": "${machine:account}",
"env": {
"MESH_BACKUP_DECLARED": "${dir:state}/backups.conf",
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
},
"health": {
"kind": "tool",
"tool": "backup_health",
"interval": "60s",
"timeout": "20s",
"looks": 2,
"grace": "60s"
}
},
{
"id": "measure",
"type": "process",
"name": "restic-measure",
"artifact": "tools",
"run": [
"./restic-backups",
"measure"
],
"user": "${machine:account}",
"env": {
"MESH_BACKUP_DECLARED": "${dir:state}/backups.conf",
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
},
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/restic-backups",
"binary": "restic-backups",
"loads": [
"restic-backups"
],
"env": {
"MESH_BACKUP_DECLARED": "${dir:state}/backups.conf",
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
}
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
+299
View File
@@ -0,0 +1,299 @@
package frombus
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose"
"github.com/novox/mesh-host/internal/compose/resolution"
)
// aNode is a bus with n1 the hub, each of manifests placed on it and held at the indexer, in the order given.
func aNode(t *testing.T, manifests ...string) (*bus, map[string]string) {
t.Helper()
p := delivery.Placement{Node: "n1", Decided: delivery.NodeDecided{Hub: true, Address: "10.77.0.1"}}
b := &bus{records: map[string]delivery.NodeRecord{"n1": {Node: "n1", TunnelKey: publicKey(t), SealingKey: "sk", Capabilities: []string{"systemd"}}},
modules: map[string]delivery.Module{}, manifests: map[string][]byte{}, builds: map[string]*delivery.BuilderBuild{}}
fps := map[string]string{}
for i, raw := range manifests {
module := strings.SplitN(strings.TrimPrefix(raw, `{"module":"`), `"`, 2)[0]
p.Modules = append(p.Modules, delivery.PlacedModule{Module: module})
b.modules[module] = delivery.Module{Manifest: delivery.ManifestDigest([]byte(raw))}
b.manifests[delivery.ManifestDigest([]byte(raw))] = []byte(raw)
fps[module] = "src2:" + strings.Repeat(string(rune('1'+i)), 64)
}
b.placements = map[string]delivery.Placement{"n1": p}
return b, fps
}
// Issue 774, 2026-10-30 19:30: a node refused whole after holding a module says what it held and why, since the last
// refusal can be of the held module's stub (node-tools held, then "its build produced 0 bundle(s)", its own why unsaid).
func aNodeRefusedWholeSaysWhatItHeld(t *testing.T) {
refused := `{"module":"power","version":"1","items":[{"collection":"contribution:node-bar:block","attributes":{"shows":"state"},"template":"files/a"}]}`
needs := `{"module":"lamp","version":"1","requires":["nothing-provides-this"]}`
b, fps := aNode(t, refused, needs)
refuse(b, "power", `"shows" is none of them`)
_, _, held, err := Compose(context.Background(), b, Mesh{}, Scope{Node: "n1"}, fps)
if err == nil || !strings.Contains(err.Error(), "nothing provides") || !strings.Contains(err.Error(), `held before: power: `) ||
!strings.Contains(err.Error(), `"shows" is none of them`) {
t.Errorf("the node refused whole with %v (held %v), want lamp's refusal and power held before, with its why", err, held)
}
}
// refuse writes the indexer's refusal of module's manifest into its modules value (novox/hq ADR 0578 §1).
func refuse(b *bus, module string, problems ...string) {
m := b.modules[module]
m.Verdict, m.Problems = delivery.VerdictRefused, problems
b.modules[module] = m
}
// ADR 0578 §3: the bus reader never ends a whole read on a refused manifest. Input, Plans and DataItems compose the rest
// and hold the refused module, named with the indexer's words in Refused; a field the decoder does not know is set aside
// and named, and its module composed.
func aRefusedManifestNeverEndsTheRead(t *testing.T) {
ok := `{"module":"lamp","version":"1","resources":[{"id":"d","type":"directory","path":"/srv/lamp","mode":"0700"}],` +
`"data":{"own":[{"id":"d","path":"${dir:d}","class":"valuable"}]}}`
bad := `{"module":"power","version":"1","resources":[{"id":"d","type":"directory","path":"/srv/power","mode":"0700"}],` +
`"data":{"own":[{"id":"d","path":"${dir:d}","class":"valuable"}]}}`
later := `{"module":"later","version":"1","a-field-from-later":1}`
b, _ := aNode(t, ok, bad, later)
refuse(b, "power", "removes a stream it may not")
ctx := context.Background()
in, err := Input(ctx, b, Mesh{}, Scope{Node: "n1"})
if err != nil {
t.Fatalf("one refused manifest ended the read: %v", err)
}
if why := in.Refused["power"]; !strings.Contains(why, "refused the manifest of power: removes a stream it may not") ||
len(in.Refused) != 1 {
t.Fatalf("refused: %v", in.Refused)
}
if aside := in.SetAside["later"]; !strings.Contains(aside, "a-field-from-later") {
t.Fatalf("set aside: %v", in.SetAside)
}
if !in.Catalogue["power"].Stubbed || in.Catalogue["lamp"].Stubbed || in.Catalogue["later"].Module != "later" {
t.Fatalf("the catalogue: %+v", in.Catalogue)
}
if _, err := Plans(ctx, b, Mesh{}, Scope{Node: "n1"}); err != nil {
t.Fatalf("plans: %v", err)
}
items, _, held, err := DataItemsHeld(ctx, b, Mesh{}, Scope{Node: "n1"})
if err != nil || len(items["lamp"]) != 1 || items["power"] != nil || !strings.Contains(held["power"], "removes a stream") {
t.Fatalf("data items %v held %v: %v", items, held, err)
}
}
// Issue 774: the culprit rule takes a module whose stub composes or fails otherwise; the runtime only where its stub
// composes, since its stub always fails otherwise.
func theRuntimeIsBlamedOnlyWhereItsStubComposes(t *testing.T) {
err := errors.New("lamp refuses")
for _, row := range []struct {
module string
again error
blamed bool
}{
{"lamp", nil, true},
{"lamp", errors.New("other"), true},
{"lamp", errors.New("lamp refuses"), false},
{catalogue.RuntimeModule, nil, true},
{catalogue.RuntimeModule, errors.New("node-tools is assigned to n1 and its build produced 0 bundle(s)"), false},
{catalogue.RuntimeModule, fmt.Errorf("compose: %w", &resolution.NoRuntimeToRunError{Held: true}), true},
{catalogue.RuntimeModule, &resolution.NoRuntimeToRunError{}, false},
} {
if got := blames(row.module, row.again, err); got != row.blamed {
t.Errorf("%s with %v: blamed %v, want %v", row.module, row.again, got, row.blamed)
}
}
}
// Issue 774, 2026-10-30 19:30: the trunk's node-tools moved to a build not yet done, and ace, g14 and shanks composed
// it from the indexer's manifest, which names no bundle: "its build produced 0 bundle(s)", every module refused. A
// module that runs is now held until its build lands, the runtime composed as what runs of it; and a module wanted at
// another build than the trunk's is composed from the build its want names.
func aRunningModuleWaitsForItsBuild(t *testing.T) {
dg := func(c string) string { return "sha256:" + strings.Repeat(c, 64) }
tools := `{"module":"node-tools","version":"1","own-secrets":{"broker":"/var/lib/mesh/node-tools/broker"},` +
`"build":{"artifacts":[{"name":"runtime","kind":"bundle","language":"go","system":"arch","from":"cmd/node-tools"}]}}`
lamp := `{"module":"lamp","version":"1","resources":[{"id":"service","type":"service","unit":"lamp.service","state":"running","boot":"enabled"}]}`
store := `{"module":"artifacts","version":"1","provides":[{"name":"artifact-store","scope":"mesh"}],"serves":{"artifact-store":{"port":5100}},` +
`"receives":{"artifact-store":"/var/lib/artifacts/received.json"}}`
old, next := "src2:"+strings.Repeat("a", 64), "src2:"+strings.Repeat("b", 64)
running := []delivery.Composed{{Node: "n1", Module: catalogue.RuntimeModule, FP: old, Resources: []json.RawMessage{
json.RawMessage(`{"id":"node-tools.runtime","module":"node-tools","type":"process","name":"node-tools","env":{"MESH_TOOL_MODULES":""}}`),
json.RawMessage(`{"id":"node-tools.bundle-runtime","module":"node-tools","type":"archive","source":"artifact-store://node-tools/runtime/blobs/` +
dg("1") + `","digest":"` + dg("1") + `","path":"` + resolution.BundlePath("node-tools", "runtime") + `"}`)}}}
for _, row := range []struct {
name string
want string // the build node-tools' want names
done bool // the trunk's build is done
held string // words node-tools' hold says; empty: composed
digest string // the runtime bundle's digest composed
}{
{name: "the trunk's build not done yet", want: next, held: "no done build yet"},
{name: "wanted at the build that runs, the trunk's not done", want: old, digest: dg("1")},
{name: "the trunk's build done", want: next, done: true, digest: dg("2")},
} {
b, fps := aNode(t, store, tools, lamp)
b.held = true // every grant held, so a contribution's credential stands in for nothing
b.own = []compose.OwnSecretRow{{Node: "n1", Module: "node-tools", Name: "broker", Origin: "made", SealedTo: "sk", Sealed: `"box"`}}
mod := b.modules["node-tools"]
mod.FP2 = next
b.modules["node-tools"] = mod
b.builds["node-tools "+old] = &delivery.BuilderBuild{State: delivery.BuildDone, Manifest: mod.Manifest,
Artifacts: []delivery.BuiltArtifact{{Name: "runtime", Kind: "bundle", Reference: "artifact-store://node-tools/runtime/blobs/" + dg("1"), Digest: dg("1")}}}
b.builds["node-tools "+next] = &delivery.BuilderBuild{State: "building", Manifest: mod.Manifest}
if row.done {
b.builds["node-tools "+next] = &delivery.BuilderBuild{State: delivery.BuildDone, Manifest: mod.Manifest,
Artifacts: []delivery.BuiltArtifact{{Name: "runtime", Kind: "bundle", Reference: "artifact-store://node-tools/runtime/blobs/" + dg("2"), Digest: dg("2")}}}
}
fps["node-tools"] = row.want
out, _, held, err := Compose(context.Background(), b, Mesh{}, Scope{Node: "n1", Running: running}, fps)
if err != nil {
t.Errorf("%s: the node is refused: %v", row.name, err)
continue
}
if lampKey, _ := delivery.ComposedKey("n1", "lamp", fps["lamp"]); out[lampKey].Module != "lamp" {
t.Errorf("%s: lamp is not composed (held %v)", row.name, held)
}
if why := held["node-tools"]; (row.held == "") != (why == "") || !strings.Contains(why, row.held) {
t.Errorf("%s: node-tools held %q, want %q", row.name, why, row.held)
}
if row.digest != "" {
key, _ := delivery.ComposedKey("n1", "node-tools", row.want)
if c := out[key]; !strings.Contains(fmt.Sprint(rawStrings(c.Resources)), row.digest) {
t.Errorf("%s: node-tools is composed as %v, want its bundle %s", row.name, rawStrings(c.Resources), row.digest)
}
}
}
}
func rawStrings(rs []json.RawMessage) []string {
var out []string
for _, r := range rs {
out = append(out, string(r))
}
return out
}
// Issue 774: a refusal of a held module's stand-in leads with why the module was held, its first reason, as mesh-self-check's
// on ace should have; a refusal naming no held module is as it was.
func aHeldModulesFirstReasonLeads(t *testing.T) {
held := map[string]string{"mesh-self-check": "mesh-self-check on ace has no bus credential: nothing was issued for ace.mesh-self-check"}
for _, row := range []struct {
err, want string
}{
{"mesh-self-check runs as its own account and declares no own secret broker",
"mesh-self-check is held: mesh-self-check on ace has no bus credential: nothing was issued for ace.mesh-self-check; " +
"its stand-in is then refused: mesh-self-check runs as its own account and declares no own secret broker"},
{"ace is the hub and has no endpoint", "ace is the hub and has no endpoint"},
} {
err := errors.New(row.err)
got := firstReason(err, held)
if got.Error() != row.want || !errors.Is(got, err) {
t.Errorf("refused %q, want %q", got, row.want)
}
}
}
// Issue 774, 2026-10-30 21:04: mesh-conditions and mesh-healers on novox and mesh-self-check on ace were placed before their
// bus accounts were issued. Held for "needs a secret called broker and none was made", each one's bare stub was then
// refused as "runs as its own account and declares no own secret broker", and that refusal failed every module of the
// node (79 on novox, 74 on ace). A held module's stand-in is never refused by a rule about its own runtime or its own
// secrets: the node composes every other module, and the held one keeps the words it was first held for.
func aHeldModuleOfItsOwnAccountNeverRefusesTheNode(t *testing.T) {
dg := "sha256:" + strings.Repeat("1", 64)
tools := `{"module":"node-tools","version":"1","own-secrets":{"broker":"/var/lib/mesh/node-tools/broker"},` +
`"build":{"artifacts":[{"name":"runtime","kind":"bundle","language":"go","system":"arch","from":"cmd/node-tools"}]}}`
own := func(name string) string {
return `{"module":"` + name + `","version":"1","runs-as":"` + name + `","secrets-owner":"` + name + `","tools":["status"],` +
`"own-secrets":{"broker":"/var/lib/` + name + `/broker"},"contributes":{"artifact-store":{"name":"` + name + `"}},` +
`"resources":[{"id":"account","type":"user","name":"` + name + `"}],` +
`"build":{"artifacts":[{"name":"tools","kind":"bundle","language":"typescript","entrypoints":["tools/index.js"]}]}}`
}
lamp := `{"module":"lamp","version":"1","resources":[{"id":"service","type":"service","unit":"lamp.service","state":"running","boot":"enabled"}]}`
store := `{"module":"artifacts","version":"1","provides":[{"name":"artifact-store","scope":"mesh"}],"serves":{"artifact-store":{"port":5100}},` +
`"receives":{"artifact-store":"/var/lib/artifacts/received.json"}}`
for _, row := range []struct {
name string
issued []string // the own-account modules whose broker is issued
building bool // node-tools' build is not done, and nothing of it runs
held map[string]string // the modules held, each with words of the reason it was first held for
// ungranted gives the provider a grants file and no consumer's grant is issued yet (a newly placed module's
// usual state): a held module's contribution then stands in for its grant, and is left out (review round 1).
ungranted bool
}{
{name: "none issued", held: map[string]string{"check": "broker", "heal": "broker"}},
{name: "none issued, no grant issued", ungranted: true, held: map[string]string{"check": "broker", "heal": "broker"}},
{name: "one issued", issued: []string{"heal"}, held: map[string]string{"check": "broker"}},
{name: "both issued", issued: []string{"check", "heal"}},
// The runtime held: its stand-in is not refused for its bundle; the modules it would run are held, not the node.
{name: "the runtime's build not done", issued: []string{"check", "heal"}, building: true,
held: map[string]string{"node-tools": "its build produced 0 bundle(s)", "check": "not here to run it from", "heal": "not here to run it from"}},
} {
provider := store
if row.ungranted {
provider = strings.Replace(store, `"receives":`, `"grants":{"artifact-store":"/var/lib/artifacts/grants"},"receives":`, 1)
}
b, fps := aNode(t, provider, tools, own("check"), lamp, own("heal"))
// Every grant held, so the held modules' contributions are composed whole; the row ungranted issues none.
b.held = !row.ungranted
b.own = []compose.OwnSecretRow{{Node: "n1", Module: "node-tools", Name: "broker", Origin: "made", SealedTo: "sk", Sealed: `"box"`}}
for _, m := range row.issued {
b.own = append(b.own, compose.OwnSecretRow{Node: "n1", Module: m, Name: "broker", Origin: "made", SealedTo: "sk", Sealed: `"box"`})
}
for _, m := range []string{"node-tools", "check", "heal"} {
name := "tools"
if m == "node-tools" {
name = "runtime"
}
mod := b.modules[m]
mod.FP2 = fps[m]
b.modules[m] = mod
b.builds[m+" "+fps[m]] = &delivery.BuilderBuild{State: delivery.BuildDone, Manifest: mod.Manifest,
Artifacts: []delivery.BuiltArtifact{{Name: name, Kind: "bundle", Reference: "artifact-store://" + m + "/" + name + "/blobs/" + dg, Digest: dg}}}
if m == "node-tools" && row.building {
b.builds[m+" "+fps[m]] = &delivery.BuilderBuild{State: "building", Manifest: mod.Manifest}
}
}
out, _, held, err := Compose(context.Background(), b, Mesh{}, Scope{Node: "n1"}, fps)
if err != nil {
t.Errorf("%s: the node is refused: %v", row.name, err)
continue
}
// A held module keeps what it contributes (issue 774, 2026-10-31: nextcloud held for its admin secret dropped out
// of postgres's received file, and postgres's helper would have locked its login): only its own secrets are left
// out of its stand-in, never its contributions, so the provider's received file names every consumer.
storeKey, _ := delivery.ComposedKey("n1", "artifacts", fps["artifacts"])
// And never a stand-in for a grant not yet issued: the provider is applied, so a held module whose grant is
// not issued is composed without that contribution, never with "foreseen: made by the next send".
for key, c := range out {
if got, _ := json.Marshal(c.Resources); strings.Contains(string(got), "foreseen") {
t.Errorf("%s: %s applies a stand-in: %s", row.name, key, got)
}
}
for _, m := range []string{"check", "heal"} {
if row.ungranted {
break
}
if got, _ := json.Marshal(out[storeKey].Resources); !strings.Contains(string(got), `\"from\": \"`+m+`\"`) {
t.Errorf("%s: the provider's received file drops %s's contribution: %s", row.name, m, got)
}
}
for _, m := range []string{"node-tools", "check", "lamp", "heal"} {
key, _ := delivery.ComposedKey("n1", m, fps[m])
_, composed := out[key]
why, isHeld := held[m]
want, wantHeld := row.held[m]
if isHeld != wantHeld || composed == wantHeld {
t.Errorf("%s: %s held %v (%q), composed %v; want held %v", row.name, m, isHeld, why, composed, wantHeld)
} else if isHeld && !strings.Contains(why, want) {
t.Errorf("%s: %s is held for %q, want its first reason, %q", row.name, m, why, want)
}
}
}
}
+204
View File
@@ -0,0 +1,204 @@
package frombus
// **The bus read through a module's declared `reads`** (the lead, 2026-10-29: one reader in place of the copies in
// mesh-provisioner and disk-load): a Reader over the states a module reads, by a runtime's Get and Keys (a bundle's
// stdio.State, a process's own), for a holder that composes plans or data items and reads no value: placement, each
// node's record in its own have key, the indexer's modules and manifests, the builder's builds and mesh-config's resolved
// values. No own secret and no credential is read (compose.Planning). Pure as the rest of this package: a bundle's
// runtime states and an HTTP fetch are compose/fromruntime's.
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/compose"
"github.com/novox/mesh-host/internal/compose/resolution"
)
// The states a module names in its manifest's `reads` to compose through StatesReader.
const (
PlacementRead = delivery.PlacementRead
ModulesRead = delivery.IndexerModule + ".modules"
ManifestsRead = delivery.IndexerModule + ".manifests"
BuildsRead = delivery.BuilderModule + ".builds"
HaveRead = "node-engine.have"
ResolvedRead = "mesh-config.resolved"
)
// StatesReads are those reads, in a manifest's order.
var StatesReads = []string{PlacementRead, ModulesRead, ManifestsRead, BuildsRead, HaveRead, ResolvedRead}
// States is a module's read states as its runtime gives them: a key's value, false when it has none.
type States interface {
Get(state, key string) (value []byte, found bool, err error)
Keys(state string) ([]string, error)
}
// Fetch reads an archive at a URL (fromruntime.HTTPFetch over HTTP).
type Fetch func(ctx context.Context, url string) ([]byte, error)
// StatesReader is the Reader over s; fetch reads a templates archive, nil for a holder that reads none.
// It keeps what each build's take-in made, by the build's content (issue 1954), so a holder composing again and again
// takes in only a build it has not seen.
func StatesReader(s States, fetch Fetch) Reader {
return statesReader{s: s, fetch: fetch, kept: &keptTakeIns{}}
}
type statesReader struct {
s States
fetch Fetch
kept *keptTakeIns
}
func (r statesReader) takenIn(key string) ([]byte, bool) { return r.kept.takenIn(key) }
func (r statesReader) keepTakenIn(key string, raw []byte) { r.kept.keepTakenIn(key, raw) }
func (r statesReader) each(read string, visit func(key string, value []byte)) error {
keys, err := r.s.Keys(read)
if err != nil {
return fmt.Errorf("%s: %w", read, err)
}
for _, k := range keys {
v, found, err := r.s.Get(read, k)
if err != nil {
return fmt.Errorf("%s %s: %w", read, k, err)
}
if found {
visit(k, v)
}
}
return nil
}
func (r statesReader) Placements(context.Context) (map[string]delivery.Placement, error) {
keys := delivery.NewPlacementKeys()
err := r.each(PlacementRead, func(k string, v []byte) {
_, _ = keys.Put(k, v, 0, false) // a key that does not read places nothing, as before
})
return keys.All(), err
}
func (r statesReader) Records(context.Context) (map[string]delivery.NodeRecord, error) {
out := map[string]delivery.NodeRecord{}
err := r.each(HaveRead, func(k string, v []byte) {
if node, _, isNode, err := delivery.ParseHaveKey(k); err == nil && isNode {
if h, err := delivery.DecodeHaveNode(v); err == nil && h.Record != nil {
out[node] = *h.Record
}
}
})
return out, err
}
func (r statesReader) Modules(context.Context) (map[string]delivery.Module, error) {
out := map[string]delivery.Module{}
err := r.each(ModulesRead, func(k string, v []byte) {
if module, err := delivery.ParseModulesKey(k); err == nil {
if m, err := delivery.DecodeModule(v); err == nil {
out[module] = m
}
}
})
return out, err
}
func (r statesReader) Manifest(_ context.Context, digest string) ([]byte, error) {
key, err := delivery.ManifestsKey(digest)
if err != nil {
return nil, err
}
v, found, err := r.s.Get(ManifestsRead, key)
if err != nil {
return nil, err
}
if !found {
return nil, fmt.Errorf("the indexer holds no manifest %s", digest)
}
m, err := delivery.DecodeManifest(v)
if err != nil {
return nil, err
}
return []byte(m.JSON), nil
}
func (r statesReader) Values(_ context.Context, node, module string) (map[string]any, error) {
var out map[string]any
prefix := node + "." + module + "."
err := r.each(ResolvedRead, func(k string, v []byte) {
name, ok := strings.CutPrefix(k, prefix)
if !ok || strings.Contains(name, ".") {
return
}
if value, laid := resolution.LaidValue(v); laid {
if out == nil {
out = map[string]any{}
}
out[name] = value
}
})
return out, err
}
func (statesReader) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow, error) {
return nil, nil
}
func (statesReader) Credentials(context.Context, string, []compose.CredentialKey, []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
return nil, nil, nil
}
func (r statesReader) Build(_ context.Context, module, fp string) (*delivery.BuilderBuild, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return nil, err
}
v, found, err := r.s.Get(BuildsRead, key)
if err != nil || !found {
return nil, err
}
b, err := delivery.DecodeBuilderBuild(v)
if err != nil {
return nil, err
}
return &b, nil
}
// archiveWithin bounds one read of a templates archive.
const archiveWithin = 30 * time.Second
func (r statesReader) Archive(ctx context.Context, url, digest string) ([]byte, error) {
if r.fetch == nil {
return nil, fmt.Errorf("this holder reads no archive (%s)", url)
}
fetching, cancel := context.WithTimeout(ctx, archiveWithin)
defer cancel()
body, err := r.fetch(fetching, url)
if err != nil {
return nil, err
}
sum := sha256.Sum256(body)
if got := hex.EncodeToString(sum[:]); got != strings.TrimPrefix(digest, "sha256:") {
return nil, fmt.Errorf("%s was recorded as %s and what arrived is %s", url, digest, got)
}
return body, nil
}
func (r statesReader) Served(ctx context.Context, node, module string, keys []string) (map[string]any, error) {
all, err := r.Values(ctx, node, module)
var out map[string]any
for _, k := range keys {
if v, ok := all[k]; ok {
if out == nil {
out = map[string]any{}
}
out[k] = v
}
}
return out, err
}
@@ -0,0 +1,236 @@
{
"module": "xterm",
"version": "1",
"moved": true,
"capabilities": [
"package-manager"
],
"requires": [
"x11-display"
],
"claims": [
{
"name": "node-terminal-emulator",
"scope": "node",
"serves": [
"open"
]
}
],
"tools": [
"xterm_font",
"xterm_colours"
],
"config": {
"face": {
"type": "string",
"description": "The font family xterm draws with, as fc-list names it: JetBrainsMono Nerd Font, because research 026/04 chose it for its glyphs the prompt uses. A family the machine does not have falls back to the nearest installed face (xterm_font says which).",
"pattern": "[A-Za-z0-9 ._-]{1,64}",
"default": "JetBrainsMono Nerd Font"
},
"size": {
"type": "number",
"description": "The size of xterm's text, in points: 12 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 12
},
"save-lines": {
"type": "number",
"description": "How many lines of scrollback xterm keeps per terminal: 10000 holds a long build log, and each line costs memory in every open terminal.",
"minimum": 0,
"maximum": 1000000,
"default": 10000
},
"term-name": {
"type": "string",
"description": "The TERM value xterm sets for the programs it runs: xterm-256color, because the palette has 256 colours and every program knows the name.",
"pattern": "[A-Za-z0-9._-]{1,64}",
"default": "xterm-256color"
},
"render-font": {
"type": "boolean",
"description": "Whether xterm draws with fontconfig faces (true) or the old X core fonts (false): true, because the face is a fontconfig family and core fonts cannot show it.",
"default": true
},
"bold-mode": {
"type": "boolean",
"description": "Whether xterm draws bold text by overstriking the normal face (true): false, because the face has a real bold style and overstriking blurs it.",
"default": false
},
"always-bold-mode": {
"type": "boolean",
"description": "Whether xterm overstrikes bold even where the face has a bold style (true): false, for the same reason as bold-mode.",
"default": false
},
"allow-bold-fonts": {
"type": "boolean",
"description": "Whether xterm may use the face's bold style for bold text (true): true, so bold text is drawn with the real bold weight.",
"default": true
},
"background": {
"type": "string",
"description": "The terminal's background colour, as #rrggbb: black, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"foreground": {
"type": "string",
"description": "The terminal's text colour, as #rrggbb: a light grey, easier on the eyes than white on black.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a8a8a8"
},
"color0": {
"type": "string",
"description": "Palette entry 0, the normal black, as #rrggbb: #000000, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"color1": {
"type": "string",
"description": "Palette entry 1, the normal red, as #rrggbb: #9e1828, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#9e1828"
},
"color2": {
"type": "string",
"description": "Palette entry 2, the normal green, as #rrggbb: #008800, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#008800"
},
"color3": {
"type": "string",
"description": "Palette entry 3, the normal yellow, as #rrggbb: #d2bb4b, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#d2bb4b"
},
"color4": {
"type": "string",
"description": "Palette entry 4, the normal blue, as #rrggbb: #414171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#414171"
},
"color5": {
"type": "string",
"description": "Palette entry 5, the normal magenta, as #rrggbb: #963c59, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#963c59"
},
"color6": {
"type": "string",
"description": "Palette entry 6, the normal cyan, as #rrggbb: #418179, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#418179"
},
"color7": {
"type": "string",
"description": "Palette entry 7, the normal white, as #rrggbb: #bebebe, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bebebe"
},
"color8": {
"type": "string",
"description": "Palette entry 8, the bright black, as #rrggbb: #666666, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#666666"
},
"color9": {
"type": "string",
"description": "Palette entry 9, the bright red, as #rrggbb: #bc5766, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bc5766"
},
"color10": {
"type": "string",
"description": "Palette entry 10, the bright green, as #rrggbb: #61a171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#61a171"
},
"color11": {
"type": "string",
"description": "Palette entry 11, the bright yellow, as #rrggbb: #e7db52, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#e7db52"
},
"color12": {
"type": "string",
"description": "Palette entry 12, the bright blue, as #rrggbb: #5085af, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#5085af"
},
"color13": {
"type": "string",
"description": "Palette entry 13, the bright magenta, as #rrggbb: #a97a99, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a97a99"
},
"color14": {
"type": "string",
"description": "Palette entry 14, the bright cyan, as #rrggbb: #6ba4a4, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#6ba4a4"
},
"color15": {
"type": "string",
"description": "Palette entry 15, the bright white, as #rrggbb: #ffffff, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#ffffff"
},
"copy-binding": {
"type": "string",
"description": "The key combination that copies the selection to the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> C, so a plain Ctrl+C still interrupts the program. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> C"
},
"paste-binding": {
"type": "string",
"description": "The key combination that pastes the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> V, the pair of copy-binding. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> V"
}
},
"items": [
{
"collection": "environment",
"name": "terminal",
"attributes": {
"kind": "variable",
"name": "TERMINAL"
},
"template": "files/env-terminal"
},
{
"collection": "shell:xresources:normal",
"name": "xresources-normal",
"template": "files/xresources-normal"
}
],
"resources": [
{
"id": "package",
"type": "package",
"package": "xterm"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/xterm-tools",
"binary": "xterm-tools",
"loads": [
"xterm-tools"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,60 @@
{
"module": "mesh-provisioner-client",
"version": "1",
"claims": [
{
"name": "node-provisioner",
"scope": "node"
}
],
"own-secrets": {
"provisioner-broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-provisioner-client",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-provisioner-client",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-provisioner-client"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-provisioner-client",
"place": ".",
"unknown": "program"
},
{
"id": "client",
"type": "process",
"name": "mesh-provisioner-client",
"artifact": "client",
"run": [
"./mesh-provisioner-client"
],
"env": {
"MESH_PROVISIONER_CREDENTIAL": "${dir:state}/broker",
"MESH_PROVISIONER_STATE": "${dir:state}"
},
"user": "mesh-provisioner-client",
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "client",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-provisioner-client",
"binary": "mesh-provisioner-client"
}
]
}
}
@@ -0,0 +1,2 @@
# What the modules on this machine back up, composed by the mesh (novox/hq to-be 43). Do not edit.
${backup:lines}
@@ -0,0 +1,2 @@
# The data the modules on this machine declare, composed by the mesh (novox/hq ADR 0233). Do not edit.
${backup:data}
@@ -0,0 +1,6 @@
# How many restore points the night keeps, written by the mesh from this machine's settings keep-daily,
# keep-weekly, keep-monthly (novox/hq issue 416) and keep-hand-taken-days (novox/hq issue 453). Replaced on every push; change the settings, never this file.
daily=${config:keep-daily}
weekly=${config:keep-weekly}
monthly=${config:keep-monthly}
hand=${config:keep-hand-taken-days}
@@ -0,0 +1,199 @@
{
"module": "restic",
"version": "1",
"moved": true,
"claims": [
{
"name": "node-backup",
"scope": "node",
"serves": [
"backed-up",
"now",
"restore"
]
}
],
"emits": [
"backed-up"
],
"tools": [
"backup_health",
"backup_delete_retired",
"backup_deleted"
],
"own-secrets": {
"repository": "${dir:state}/repository.secret"
},
"data": {
"own": [
{
"id": "repository",
"path": "${dir:repository}",
"class": "rebuildable",
"backup": "none",
"why": "the restore points themselves: a copy of data kept elsewhere on this machine, never the only copy of anything; lost, the history goes and nothing live does"
},
{
"id": "state",
"path": "${dir:state}",
"class": "cache",
"why": "what the last nights and measurements were; the next night writes it again"
}
]
},
"config": {
"keep-daily": {
"type": "number",
"description": "How many nightly restore points are kept, a count from 1 to 9999: two weeks undo a mistake noticed within a fortnight. Read by the night, which thins nothing while it does not read (novox/hq issue 416); lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 14
},
"keep-weekly": {
"type": "number",
"description": "How many weekly restore points are kept behind the dailies, a count from 1 to 9999, refused by the night like keep-daily; lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 8
},
"keep-monthly": {
"type": "number",
"description": "How many monthly restore points are kept behind the weeklies, a count from 1 to 9999, refused by the night like keep-daily; lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 6
},
"keep-hand-taken-days": {
"type": "number",
"description": "How many days a restore point taken by hand (node-backup.now, or the mesh's before a module moves) is kept at least, a count from 1 to 9999; it is kept until a later good night of its module exists, and this many days in any case, so the next thinning never takes the one taken against a risky change (novox/hq issue 453). Lowering it lets go of the older ones taken by hand at the next thinning, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 7
}
},
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": ".",
"unknown": "program"
},
{
"id": "repository",
"type": "directory",
"mode": "0700",
"unknown": "program"
},
{
"id": "declared",
"type": "file",
"path": "${dir:state}/backups.conf",
"mode": "0600",
"template": "files/backups.conf"
},
{
"id": "data-declared",
"type": "file",
"path": "${dir:state}/data.conf",
"mode": "0600",
"template": "files/data.conf"
},
{
"id": "retention",
"type": "file",
"path": "${dir:state}/retention.conf",
"mode": "0644",
"trusted": true,
"template": "files/retention.conf"
},
{
"id": "tool",
"type": "package",
"package": "restic"
},
{
"id": "sqlite",
"type": "package",
"package": "sqlite"
},
{
"id": "night",
"type": "process",
"name": "restic-night",
"artifact": "tools",
"run": [
"./restic-backups",
"night"
],
"user": "${machine:account}",
"env": {
"MESH_BACKUP_DECLARED": "${dir:state}/backups.conf",
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
},
"health": {
"kind": "tool",
"tool": "backup_health",
"interval": "60s",
"timeout": "20s",
"looks": 2,
"grace": "60s"
}
},
{
"id": "measure",
"type": "process",
"name": "restic-measure",
"artifact": "tools",
"run": [
"./restic-backups",
"measure"
],
"user": "${machine:account}",
"env": {
"MESH_BACKUP_DECLARED": "${dir:state}/backups.conf",
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
},
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/restic-backups",
"binary": "restic-backups",
"loads": [
"restic-backups"
],
"env": {
"MESH_BACKUP_DECLARED": "${dir:state}/backups.conf",
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
}
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,129 @@
{
"module": "de-spiegel",
"version": "1",
"slug": "spiegel",
"capabilities": [
"container-runtime"
],
"requires": [
"route",
"smtp"
],
"contributes": {
"route": {
"site": {
"label": "de-spiegel",
"port": 35621
},
"praktijk": {
"name": "${config:praktijk-name}",
"port": 35621
},
"praktijk-www": {
"name": "${config:praktijk-www-name}",
"redirect": "https://${config:praktijk-name}"
},
"psychologie": {
"name": "${config:psychologie-name}",
"port": 35621
},
"psychologie-www": {
"name": "${config:psychologie-www-name}",
"redirect": "https://${config:psychologie-name}"
}
},
"smtp": {
"account": "de-spiegel"
}
},
"binds": {
"route": "/var/lib/de-spiegel/route.json",
"smtp": "/var/lib/de-spiegel/smtp.json"
},
"config": {
"praktijk-name": {
"type": "string",
"description": "The public name the practice's site is served on; the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"praktijk-www-name": {
"type": "string",
"description": "The public name that redirects to the practice's site (its www form); the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"psychologie-name": {
"type": "string",
"description": "The public name the psychology practice's site is served on; the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"psychologie-www-name": {
"type": "string",
"description": "The public name that redirects to the psychology practice's site (its www form); the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
}
},
"listens": [
{
"port": 35621,
"protocol": "tcp",
"from": "mesh",
"why": "the de-spiegel site and its /contact endpoint over http; the practice's three public names are route grants, and the proxy reaches every one of them here"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"path": "/var/lib/de-spiegel",
"mode": "0700"
},
{
"id": "server-env",
"type": "file",
"path": "/var/lib/de-spiegel/server.env",
"mode": "0600",
"content": "SMTP_HOST=${bound:smtp:at}\nSMTP_PORT=${bound:smtp:port}\nSMTP_AUTH_USER=de-spiegel@${bound:smtp:domain}\nSMTP_AUTH_PASS=${secret:smtp}\n"
},
{
"id": "net",
"type": "network",
"name": "de-spiegel"
},
{
"id": "server",
"type": "container",
"name": "de-spiegel",
"artifact": "site",
"network": "de-spiegel",
"env-file": [
"/var/lib/de-spiegel/server.env"
],
"ports": [
"35621"
],
"secrets-in-environment": "the application's own code reads SMTP_AUTH_USER/PASS from the environment (server/index.js); converting is this repository's change"
}
],
"build": {
"on": [
{
"arg": "NODE_BASE",
"image": "node@sha256:48e4b67d85f87bd551df43704e24d252f56cc5f8e9718841aace50f19948f0f9"
}
],
"artifacts": [
{
"name": "site",
"kind": "image",
"from": "Dockerfile"
}
]
},
"secrets": {
"smtp": "/var/lib/de-spiegel/smtp.secret"
}
}
@@ -0,0 +1,82 @@
{
"module": "f1-circuits",
"slug": "f1",
"version": "1",
"capabilities": [
"container-runtime"
],
"requires": [
"route"
],
"contributes": {
"route": {
"label": "f1-circuits",
"endpoint": "web"
}
},
"binds": {
"route": "${dir:state}/route.json"
},
"listens": [
{
"name": "web",
"port": 8000,
"protocol": "tcp",
"from": "mesh",
"why": "the track-layout CDN, the race posters and the Kometa metadata over http; its public name is a route grant and the proxy reaches it here. Kometa fetches the metadata and every poster by that public name"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "cache",
"type": "directory",
"mode": "0700"
},
{
"id": "site-env",
"type": "file",
"path": "${dir:state}/site.env",
"mode": "0644",
"content": "F1_CIRCUITS_BASE_URL=https://${bound:route:name}\n"
},
{
"id": "server",
"type": "container",
"name": "f1-circuits",
"artifact": "cdn",
"ports": [
"8000"
],
"env-file": [
"${dir:state}/site.env"
],
"volumes": [
"${dir:cache}:/app/cache"
],
"restart-on": [
"site-env"
]
}
],
"build": {
"on": [
{
"arg": "PYTHON_BASE",
"image": "python@sha256:6d43704baacd1bfbe7c295d7f13079d5d8104ed33568873133f8fc69980419df"
}
],
"artifacts": [
{
"name": "cdn",
"kind": "image",
"from": "service/Dockerfile"
}
]
}
}
+147
View File
@@ -0,0 +1,147 @@
package frombus
// **A node's plans at two sets of builds, over one read** (issue 1954; novox/hq ADR 0570 clause 2, ADR 0580 clause 2):
// the seat mesh-provisioner's holder composes each node at the builds its wants name and at the newest build of each
// placed module. Both are composed from one read of the bus (placement, records, the indexer's modules and manifests,
// values: base); only the builds the node's modules are composed from differ, each build read once for both, and a
// build taken in once per content by a reader that keeps it (StatesReader). Where the time went is said by phase.
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HeldPlans is one scope's plans, as PlansHeld answers them: err a refusal that is no single module's.
type HeldPlans struct {
Plans map[string]catalogue.Plan
Held map[string]string
Err error
}
// Phase is one step of a composition and how long it took.
type Phase struct {
Name string `json:"phase"`
Took time.Duration `json:"took"`
}
// PlansHeldTwice is PlansHeld at s and at s with no kept builds (each placed module at its newest build), over one
// read of the bus: what each answers is what PlansHeld answers for that scope. A read that fails is both scopes' err.
// took names each phase in order: the read, then each scope's builds and its plans.
func PlansHeldTwice(ctx context.Context, r Reader, mesh Mesh, s Scope) (at, newest HeldPlans, took []Phase) {
start := time.Now()
mark := func(name string) {
now := time.Now()
took = append(took, Phase{Name: name, Took: now.Sub(start)})
start = now
}
b, err := readBase(ctx, r, mesh, s)
mark("read")
builds := onceBuilds(r)
newestScope := s
newestScope.KeptAt = nil
for _, sc := range []struct {
scope Scope
into *HeldPlans
name string
}{{s, &at, "wanted"}, {newestScope, &newest, "newest"}} {
held := map[string]string{}
sc.into.Held = held
refuse := func(module string, err error) { held[module] = err.Error() }
b.refuseOwn(refuse)
if err != nil {
sc.into.Err = err
continue
}
in, err := b.finish(ctx, r, builds, mesh, sc.scope, b.cloned(), refuse, nil)
mark("builds at " + sc.name)
if err != nil {
sc.into.Err = err
continue
}
sc.into.Plans, sc.into.Err = plansHeldOf(in, s.Node, held)
mark("plans at " + sc.name)
}
return at, newest, took
}
// onceBuilds is r's Build, each module's build at a fingerprint read once: both scopes of one composition read the
// builds of the same moment.
func onceBuilds(r Reader) func(ctx context.Context, module, fp string) (*delivery.BuilderBuild, error) {
type answer struct {
b *delivery.BuilderBuild
err error
}
read := map[string]answer{}
return func(ctx context.Context, module, fp string) (*delivery.BuilderBuild, error) {
key := module + " " + fp
if a, ok := read[key]; ok {
return a.b, a.err
}
b, err := r.Build(ctx, module, fp)
read[key] = answer{b, err}
return b, err
}
}
// takenInKeeper is a Reader that keeps what a build's take-in made (StatesReader does): the resolved manifest's bytes,
// by the content it was made from, so a composition made again on every change takes in only a build it has not seen.
type takenInKeeper interface {
takenIn(key string) ([]byte, bool)
keepTakenIn(key string, raw []byte)
}
// takenInKey is what a take-in is made from: the artifact store's address and the build's record (its manifest's digest, its artifacts by digest), read
// whole; an archive is checked against its digest on every read, so the same record takes in to the same bytes.
func takenInKey(module, fp, address string, b delivery.BuilderBuild) (string, bool) {
raw, err := json.Marshal(b)
if err != nil {
return "", false
}
sum := sha256.Sum256(append([]byte(module+" "+fp+" "+address+" "), raw...))
return hex.EncodeToString(sum[:]), true
}
// takenInAtMost bounds the take-ins a reader keeps; past it they are emptied, which costs only a take-in again.
const takenInAtMost = 512
// keptTakeIns is a reader's kept take-ins.
type keptTakeIns struct {
mu sync.Mutex
kept map[string][]byte
}
func (k *keptTakeIns) takenIn(key string) ([]byte, bool) {
k.mu.Lock()
defer k.mu.Unlock()
raw, ok := k.kept[key]
return raw, ok
}
func (k *keptTakeIns) keepTakenIn(key string, raw []byte) {
k.mu.Lock()
defer k.mu.Unlock()
if k.kept == nil || len(k.kept) >= takenInAtMost {
k.kept = map[string][]byte{}
}
k.kept[key] = raw
}
// PhasesSaid says the phases in one line, each in milliseconds.
func PhasesSaid(took []Phase) string {
out := ""
for i, p := range took {
if i > 0 {
out += ", "
}
out += fmt.Sprintf("%s %d ms", p.Name, p.Took.Milliseconds())
}
return out
}
+101
View File
@@ -0,0 +1,101 @@
// Package fromruntime is compose/frombus's StatesReader over a bundle's runtime (the lead, 2026-10-29): the states a
// module's manifest names in `reads` (frombus.StatesReads), read through stdio.State, and a templates archive fetched
// over HTTP. Apart from frombus so that frombus stays pure for the mesh-compose command.
package fromruntime
import (
"context"
"fmt"
"io"
"net/http"
"sync"
stdio "git.novox.be/novox/mesh-sdk/go"
"github.com/novox/mesh-host/internal/compose/frombus"
)
// States are a bundle's read states through its runtime.
type States struct{}
// Get is a key's value, false when it has none.
func (States) Get(state, key string) ([]byte, bool, error) {
e, err := stdio.State(state).Get(key)
if err != nil || e == nil {
return nil, false, err
}
return e.Value, true, nil
}
// Keys are a state's keys.
func (States) Keys(state string) ([]string, error) { return stdio.State(state).Keys() }
// Reader is frombus's Reader over this bundle's read states key by key, fetching archives over HTTP. A holder that
// composes again and again reads through Watched instead (novox/hq ADR 0580).
func Reader() frombus.Reader { return frombus.StatesReader(States{}, HTTPFetch) }
// Watched is frombus's States over a watched copy (novox/hq ADR 0580 clause 1): the states a composition reads whole
// (placement, haves, modules, resolved values) from the copy, a build by key when asked, a manifest once per digest.
type Watched struct{ *stdio.WatchedStates }
// Get is a key's value: a build read now, a manifest kept once read, any other state's from the copy.
func (w Watched) Get(state, key string) ([]byte, bool, error) {
switch state {
case frombus.BuildsRead:
return w.Direct(state, key)
case frombus.ManifestsRead:
return w.Once(state, key)
}
return w.WatchedStates.Get(state, key)
}
// WatchedReader is frombus's Reader over w, fetching each templates archive over HTTP once (KeptFetch): a composition
// made again on every change reads it again otherwise.
func WatchedReader(w *stdio.WatchedStates) frombus.Reader {
return frombus.StatesReader(Watched{w}, KeptFetch(HTTPFetch))
}
// archivesAtMost bounds the archives KeptFetch keeps; past it they are emptied, which costs only a fetch again.
const archivesAtMost = 64
// KeptFetch is fetch kept once per URL: an archive's URL names its blob by digest, so its body never changes under it
// (frombus checks the digest on every read). A failed fetch is not kept.
func KeptFetch(fetch frombus.Fetch) frombus.Fetch {
var mu sync.Mutex
kept := map[string][]byte{}
return func(ctx context.Context, url string) ([]byte, error) {
mu.Lock()
body, ok := kept[url]
mu.Unlock()
if ok {
return body, nil
}
body, err := fetch(ctx, url)
if err != nil {
return nil, err
}
mu.Lock()
if len(kept) >= archivesAtMost {
clear(kept)
}
kept[url] = body
mu.Unlock()
return body, nil
}
}
// HTTPFetch reads an archive over HTTP, at most 256 MiB.
func HTTPFetch(ctx context.Context, url string) ([]byte, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("%s answered %s", url, resp.Status)
}
return io.ReadAll(io.LimitReader(resp.Body, 256<<20))
}
+365
View File
@@ -0,0 +1,365 @@
// Package compose is the controller's composition of a machine, as a function over a plain input (novox/hq ADR 0453
// clause 3, ADR 0446 clause 4): what the rest of the mesh offers and holds, the private network's generator, one
// node's resolution and, in the issues after, its rendering and its declaration.
//
// **It never allocates, makes, writes, logs or raises.** The controller composes with a store behind it; this
// package is its Reading and Foreseeing modes over data. Everything the controller reads from its store is a field of
// Input, and everything a send would make or record (a port, a credential, a seat passed on, a node left out) is
// returned for the caller to act on, never done. Input is never changed by a call. No field holds a secret value
// the composition needs: sealed values are optional inputs, and without them the composition uses stand-ins named
// foreseen. The logic is the controller's, ported from mesh-controller cmd/mesh-controller at commit
// 6c2a8d62100cb7d491f8eb9a623e1eeea5ec0de1 (plan.go, network.go, holdings.go, seat_pass.go), in the same order, with the
// same refusals in the same words, so that a comparison node for node is a fair one.
//
// It imports the standard library and mesh-sdk's own packages only.
// It reads no environment variable: the range the mesh allocates from and the suffix of its internal names are fields of Input.
package compose
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Input is everything the composition reads. Each field is what one read of the controller's inventory answers, named
// beside it; the controller's adapter fills it from its stores and the assigner from the mesh's facts and buckets
// (ADR 0446 clause 2).
type Input struct {
// Catalogue is the shelf: every module's manifest by name (inv.Catalogue), as it is to be composed. A recorded
// module kept at the build a machine runs (keepRecorded) is that node's Kept, applied to its own resolution only (never to the rest of the mesh).
Catalogue map[string]catalogue.Manifest
// Nodes is every node the mesh has (inv.Nodes and inv.Overlays together, one row per node).
Nodes []Node
// Holdings is every seat whose holder is on record (inv.Holdings).
Holdings []catalogue.Held
// Licences is every licence the mesh holds, with the machines that use it (open.Licences); none for a mesh with no
// licence.
Licences []Licence
// OverlayCIDR is the range the mesh allocates from when no tunnel was adopted, as genesis was told it (the
// controller's OverlayCIDRVar); empty means DefaultOverlayCIDR. The package reads no environment.
OverlayCIDR string
// InternalSuffix is what the mesh's internal names end in (the controller's MESH_INTERNAL_SUFFIX), with or without a leading
// dot; empty means overlay.DefaultSuffix. A fact of the whole mesh, so it is in the Input and never read from the environment.
InternalSuffix string
// Tunnels is, by node name, the found tunnel whose key is the node's overlay key (inv.Tunnels).
Tunnels map[string]FoundTunnel
// Adopted is the hub's found tunnel and the hub's name, whether or not the hub took it (inv.AdoptedTunnel); nil
// when no hub has one.
Adopted *AdoptedTunnel
// CarriedPeers is every peer of the adopted tunnel (inv.CarriedPeers).
CarriedPeers []CarriedPeer
// Joining is the nodes that hold a live enrolment token (inv.NodesWithALiveToken).
Joining []string
// Credentials is how each pair credential a node takes from elsewhere stands, never its value (what inv.SecretFor,
// SharedSecretFor and GivenPairSecret would answer): held, to be made by the next send, only a person's to give, or
// refused. A need with no entry here is one the next send would make.
Credentials []Credential
// Built is every repository the mesh has built (the keys of inv.Held): what tells a reference recorded with an address
// from an image a module runs straight from a public registry.
Built []string
// OperatorExport is every operator-sealed secret in the mesh, for a module that keeps them (inv.OperatorExport): sealed
// values the composition carries without opening. Nil when the mesh has no operator key.
OperatorExport *catalogue.KeptExport
// KeptRuns is, by node and module, the file of the export as that node runs it (its resource of a module's running
// composition): what a node composing its own carries where it holds no OperatorExport (catalogue.Rendering.KeptRuns,
// issue 1126, C6). Used only while OperatorExport is nil.
KeptRuns map[string]map[string]map[string]any
// Issued is the credentials made for the consumers of each provider (inv.SecretsFrom): one row for each pair credential
// that exists, by the provider node it is for. The grants a provider must create are read from these, so a credential not
// made yet has no grant until the send that makes it. An assigner fills it from the facts snapshot's Grants of each
// machine (go/facts, from format 2), which carry who each credential is for and never the credential.
Issued []Issued
// OwnSecrets is the own secrets held on each machine (the module_secret rows): who made each, and the key it is sealed to.
// Their sealed values are optional.
OwnSecrets []OwnSecretRow
// KeptCoreBoxes is the vault's box last kept for each core credential on each machine (inv.KeptCoreBoxes).
KeptCoreBoxes []CoreBox
// BusUsers is the bus's user list as the controller composes it from the bus records and the hashes it holds (composeBusUsers),
// for the machine that runs the bus: names and hashes, never passwords. Empty when the Input does not carry it, and then
// the composition uses a stand-in, named foreseen. BusMissing are the users left out of it for want of a credential.
BusUsers string
BusMissing []string
// BusMinted are the bus users the mesh has minted a password hash for (inv.BusUserHash): `<node>.<module>` for a module and
// `provisioner.<node>` for a machine's provisioner client.
BusMinted []string
// BusMemberships is, by node, the membership sealed to it for the bus the mesh is moving to (inv.BusMemberships).
BusMemberships map[string]string
// Authority is the mesh's own certificate (identity.EstablishAuthority), public. Optional.
Authority string
// CurrentBuilds is, by module, the commit of the build the mesh holds for it (inv.CurrentBuilds, the commit it was built from);
// BuildIDs the build made from a commit of a module (inv.BuildOf), by module then commit. Read only to name the builds a
// declaration carries to a node-engine that reads them.
CurrentBuilds map[string]string
BuildIDs map[string]map[string]string
// Refused is each module whose manifest the indexer's judge refused, or whose bytes do not decode, by module, with why
// (novox/hq ADR 0578 §3, §5): its verdict, carried in the plain input. Such a module stands in Catalogue as its stub
// (compose.Stub), so it is held: nothing of its own is composed, while what it provides and requires still shapes the
// rest. A module not named is accepted, a missing verdict included.
Refused map[string]string `json:",omitempty"`
// SetAside is each module whose manifest has a field the reader's decoder does not know, by module, with what was set
// aside (catalogue.Manifest.SetAside): the rest of it is composed, and the reader names it (ADR 0578 §6).
SetAside map[string]string `json:",omitempty"`
}
// Issued is one pair credential made for a consumer of a provider (inv.Secret): ForProvider is its copy sealed to the
// provider, optional.
type Issued struct {
Provision string
Consumer string
ConsumerModule string
Local string
Provider string
ForProvider string
}
// OwnSecretRow is one own secret held on a machine: Origin is `made` (the mesh generated it) or `accepted` (a person gave it),
// SealedTo the sealing key it is sealed to, and Sealed its value sealed to that key, optional.
type OwnSecretRow struct {
Node string
Module string
Name string
Origin string
SealedTo string
Sealed string
}
// CoreBox is the vault's box for one core credential of one machine, as last kept (inv.CoreBox).
type CoreBox struct {
Node string
Module string
Name string
Generation int
SealedTo string
Box json.RawMessage
}
// Node is one node of the mesh and what the composition reads about it.
type Node struct {
Name string
// Place is where it is on the private network (inv.Overlays).
Place Place
// Assigned is the modules assigned to it, in the order the controller reads them (inv.Assigned).
Assigned []string
// Handed is, by module, whether the vault judged it handed to the provisioner when it answered this node's own
// secrets (catalogue.AssignmentHands of the plan in its assignment): the node's own composition hands exactly those
// (catalogue.Rendering.Assigned, issue 774). Nil where the vault said nothing of it, as for the controller.
Handed map[string]bool
// Capabilities are the ones present on it (inv.ProfileOf).
Capabilities map[string]bool
// PublicDomain is the domain it composes its routed names under (inv.PublicDomainOf).
PublicDomain string
// Account, AccountHome, AgentAccount and AgentAccountHome are its operator and agent logins (inv.NodeByName).
Account, AccountHome, AgentAccount, AgentAccountHome string
// Adopted is whether it is adopted rather than converged.
Adopted bool
// Pins is the provider each provision was pinned to for it (inv.PinsFor).
Pins map[string]catalogue.Chosen
// Bound is where each of its consumers of a provision that keeps data was last sent (inv.BindingsFor), by consumer
// module and provision. An assigner fills it from the facts snapshot's Bindings of the machine (go/facts, from format 2).
Bound map[string]map[string]catalogue.Chosen
// Settings is its settings layers by module (inv.SettingsFor); a module with none is absent.
Settings map[string][]catalogue.Layer
// Ports is the machine port each module's software port was given on it (inv.PortsFor), by module then port.
Ports map[string]map[int]int
// SealingKey is the public key its secrets are sealed to (inv.SealingKeyOf); empty when it has none. A public key, so
// nothing secret.
SealingKey string
// Taken is the modules taken on it, for an adopted node (inv.Taken).
Taken []string
// OutwardLinks are the links it reported as facing outside it (inv.OutwardLinksOf).
OutwardLinks []string
// Networks are the networks it reported being attached to, as link and network (inv.MachineOf); NetworksSaid is whether
// its node-engine said any at all, which is not the same as saying none.
Networks []catalogue.AttachedNetwork
NetworksSaid bool
// ServingKey is the key the machine serves with, which a certificate is issued for (identity.ServingKeyOf); empty when it has
// never reported one. Certificate is the certificate the mesh issued for its internal name, public and optional.
ServingKey string
Certificate string
// Sequence, Epoch and Generation are what the declaration is sent under (inv.Sequence, SentEpoch, SentGeneration): the order
// against every other send to the node, the lease epoch and the assignment generation it was last sent. Zero is not sent.
Sequence int64
Epoch uint64
Generation int64
// ReadsModules says its node-engine reads a module per resource and a build per module (inv.ReadsModules, issue 1145); only
// then are they sent. SentBuilds is the build of each module it was last sent, by module to commit, and SentKnown whether
// that is known at all (inv.SentBuilds); Runs is what each of its assignments runs (inv.AssignmentRuns), by module.
ReadsModules bool
SentBuilds map[string]string
SentKnown bool
Runs map[string]Run
// Kept is, by module, the manifest of a recorded or core module kept at the build this machine runs (plan.go keepRecorded:
// the caller reads it with recordedKept and ManifestAt), and KeptBuilds the commit of that build. They apply to this node's
// own resolution only, never to another node's, and a declaration carries the kept commit for the module.
Kept map[string]catalogue.Manifest
KeptBuilds map[string]string
// Contract is the contract of its node-engine's newest statement (inv.HealthOf); Stated is whether it ever stated one.
Contract int
Stated bool
// InOwn is the modules it runs inside its own composition (`<node>.node`), not by a composition of their own, since
// no fingerprint of the new form names them where it runs them (novox/hq issue 1126): Compositions folds their
// resources into the node's own, as the controller's composed writer does, unless fps gives one of the new form.
// The private network's module (overlay.Name), which nobody builds, is folded whether or not it is named here.
InOwn map[string]bool
// RunsOwn is the node's own composition (`<node>.node`) as it runs, when the node composes itself: a module folded
// there (InOwn) whose manifest names a build (NamesBuild) runs a build of the old form only the controller holds,
// so it is left to the node: Compositions composes it as a stub (Stub) and carries its resources as they run, each
// after the resource it runs after (issue 1126). Nil composes every folded module from its manifest, as before.
RunsOwn *delivery.Composed
// Held is the modules refused alone on this node, each composed as what runs of it (Carried, in Kept; its Stub when
// nothing of it runs) so the rest is composed (novox/hq issue 1126): none of a held module's own composition is in what
// Compositions answers, so the node keeps what it last applied of it; one whose resources the node's own runs
// (RunsOwn), folded or leaving the fold, is carried there as it runs, whatever its manifest names, so the node's own
// never drops it.
Held map[string]bool
// RunsTools is, by module, the tools bundles the node's runtime serves of it as it runs (ToolsRunning of what the
// node runs); nil when not known. A module composed as what runs of it (its Stub or Carried: held, or folded at a
// build of the old form) contributes these to the runtime, never its manifest's: a manifest no build resolved has no
// bundles, and a held module's new build's bundles are not delivered (issue 1126, 2026-10-29: shanks's runtime lost
// the folded systemd-resolved, novox's the folded nats).
RunsTools map[string][]catalogue.Bundle
}
// Place is where a node is, as its placement says (inv.Overlay).
type Place struct {
Key string
Endpoint string
Site string
Hub bool
Address string
}
// Reachable says whether the node can be dialled.
func (p Place) Reachable() bool { return strings.TrimSpace(p.Endpoint) != "" }
// Tunnel is a tunnel found on a node (inv.Tunnel).
type Tunnel struct {
Interface string
Unit string
Config string
Port int
MTU int
// Address is the interface's own address with its prefix length, Range the network that prefix names.
Address string
Range string
PublicKey string
Peers []TunnelPeer
}
// TunnelPeer is one peer of a found tunnel.
type TunnelPeer struct {
PublicKey string
Address string
}
// FoundTunnel is a node's tunnel and whether the node is adopted (inv.FoundTunnel).
type FoundTunnel struct {
Tunnel
NodeAdopted bool
}
// AdoptedTunnel is the hub's tunnel, the hub's name, and whether the hub's overlay key is the tunnel's, which is
// whether the private network took the tunnel over.
type AdoptedTunnel struct {
Tunnel Tunnel
Hub string
Adopted bool
}
// CarriedPeer is a peer of the adopted tunnel (inv.CarriedPeer).
type CarriedPeer struct {
PublicKey string
Address string
EnrolledAs string
Named string
}
// Licence is one licence the mesh holds: its name, what a consumer may know of it, and the machines and modules
// that use it.
type Licence struct {
Name string
Serves map[string]any
Holders []Holder
// Keys is, by holder, the licence's key sealed to that holder's machine (held.KeyFor), empty where the mesh holds none:
// the plaintext was discarded when it was supplied. Optional: nil means not given, and the composition uses a stand-in named
// foreseen; given (even empty), a holder with no entry is one the mesh holds no key for.
Keys map[Holder]string
// Manager is the holder of a refreshable-grant licence that manages it (held.ManagerOf); the zero Holder when none.
Manager Holder
}
// Holder is a module on a node that uses a licence.
type Holder struct {
Node string
Module string
}
// ErrNoSuchNode is what the controller's inventory answers for a node it does not have.
var ErrNoSuchNode = errors.New("no node of that name")
// node is the node of that name, or the refusal the controller's inventory gives for one it does not have.
func (in Input) node(name string) (Node, error) {
for _, n := range in.Nodes {
if n.Name == name {
return n, nil
}
}
return Node{}, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
// sorted is the nodes by name, the order the controller reads them in.
func (in Input) sorted() []Node {
out := append([]Node(nil), in.Nodes...)
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
return out
}
// CredentialState is how one pair credential stands.
type CredentialState string
const (
// CredentialHeld is a credential held for both ends as they stand now. Sealed, when given, is the consumer's copy; a held
// credential with no value is composed with a stand-in, named foreseen.
CredentialHeld CredentialState = "held"
// CredentialWouldMake is a credential none is held for: the next send makes it, and the composition uses a stand-in,
// named foreseen. It is also what a need with no entry is.
CredentialWouldMake CredentialState = "would-make"
// CredentialNotGiven is a credential only a person gives (novox/hq ADR 0337 clause 4) that nobody has given, or that was given
// under a key one end no longer has: the consumer still resolves, without it. Why is the controller's words for it.
CredentialNotGiven CredentialState = "not-given"
// CredentialRefused is a credential the send would refuse to make, for the reason Why gives in the controller's words: a
// value a person gave sealed to a key the machine no longer has, a machine with no sealing key.
CredentialRefused CredentialState = "refused"
)
// Credential is the state of one pair credential, by the ends it joins: the provision, the consumer's node and module, the
// provider's node, and the local name where a module keeps several (inv.Secret's key).
type Credential struct {
Provision string
Consumer string
ConsumerModule string
Provider string
Local string
State CredentialState
// Sealed is the consumer's copy as it is sealed to the consumer's key; optional, and only the controller's adapter passes it.
Sealed string
// Why is the words of a CredentialNotGiven or CredentialRefused.
Why string
}
// Run is the build an assignment names: the commit it is of and the build made from it (chain.Ref).
type Run struct {
Commit string
Build string
}
+308
View File
@@ -0,0 +1,308 @@
package compose
import (
"errors"
"fmt"
"slices"
"sort"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/compose/resolution"
"github.com/novox/mesh-host/internal/overlay"
)
// The private network: who is on it, where, and what they are called. Ported from mesh-controller
// cmd/mesh-controller/network.go at 6c2a8d62.
// DefaultOverlayCIDR is the range the mesh allocates from when nothing says another.
const DefaultOverlayCIDR = "10.42.0.0/16"
// composer is one composition over an Input: it reads Input, changes nothing in it, and keeps what the controller
// said on its standard error as lines to return.
type composer struct {
in Input
said []string
seen map[string]bool
// unhostable is, from the last look at the rest of the mesh, each module assigned to another node that this
// composition finds that node cannot run: what it would have offered is missing here (issue 1126, probe #532).
unhostable []string
// What one composition looks at again and again, each worked out once over its Input, which it never changes (issue
// 1954: a node's plans resolved every machine of the mesh some two hundred times, 11.7 s of CPU for novox): who is
// on the private network, and the rest of the mesh as each node sees it.
onNetwork []Node
onNetworkSet bool
rest map[string]restOf
plans map[string]planOf
}
// restOf is theRestOfTheMesh's answer for one excluded node, kept: what it found unhostable beside it.
type restOf struct {
world catalogue.World
passes Passes
left []LeftOut
unhostable []string
err error
}
func newComposer(in Input) *composer { return &composer{in: in, seen: map[string]bool{}} }
// say keeps a line the controller would have written to its log, once.
func (c *composer) say(format string, args ...any) {
line := fmt.Sprintf(format, args...)
if c.seen[line] {
return
}
c.seen[line] = true
c.said = append(c.said, line)
}
// suffix is what the mesh's internal names end in: the Input's, and overlay.DefaultSuffix when it gives none. The package never calls
// overlay.Suffix or overlay.InternalName, which read the environment, so the assigner and the controller compose the same names
// from the same Input.
func (c *composer) suffix() string {
if v := strings.TrimPrefix(strings.TrimSpace(c.in.InternalSuffix), "."); v != "" {
return v
}
return overlay.DefaultSuffix
}
// internalName is a node's name inside the mesh.
func (c *composer) internalName(node string) string { return node + "." + c.suffix() }
// saidBeside is the lines already said and then every line the composer kept, once each, in the order they were found.
func (c *composer) saidBeside(lines []string) []string {
for _, l := range c.said {
if !slices.Contains(lines, l) {
lines = append(lines, l)
}
}
return lines
}
// overlayRange is the range the mesh allocates node addresses from: the adopted tunnel's first (novox/hq ADR 0105),
// then the range genesis was told, then the default.
func (c *composer) overlayRange() string {
if a := c.in.Adopted; a != nil && a.Adopted {
return a.Tunnel.Range
}
if v := strings.TrimSpace(c.in.OverlayCIDR); v != "" {
return v
}
return DefaultOverlayCIDR
}
// network builds the private network over the machines that resolved the module for it. A machine is on it because it
// was given the module, or because it is joining with a key and an address (novox/hq ADR 0169), or because its own set
// could not be resolved and it holds a place (a key and an address): that place is its last good one on the network, and
// it does not change because its modules failed, so every other machine keeps its path to it -- the hub above all -- and
// only its own declaration is refused (issue 1833). A machine refused with no place is left off.
func (c *composer) network(on map[string]bool, refused map[string]string) (*overlay.Generator, error) {
tunnels := c.in.Tunnels
carried := c.in.CarriedPeers
isJoining := map[string]bool{}
for _, name := range c.in.Joining {
isJoining[name] = true
}
var nodes []overlay.Node
for _, node := range c.in.sorted() {
p := node.Place
placed := p.Key != "" && p.Address != ""
_, wasRefused := refused[node.Name]
if !on[node.Name] && !((isJoining[node.Name] || wasRefused) && placed) {
continue
}
if !on[node.Name] && wasRefused && !isJoining[node.Name] {
c.say("%s could not be resolved and keeps its place on the private network at %s, so the others keep their path to it",
node.Name, p.Address)
}
n := overlay.Node{
Name: node.Name, Key: p.Key, Endpoint: p.Endpoint,
Site: p.Site, Hub: p.Hub, Address: p.Address,
}
if t, takes := tunnels[node.Name]; takes && t.NodeAdopted {
// Only an adopted node is told to take the found unit over. A hub whose placement disagrees with the tunnel is
// refused, not composed.
if wrong := disagrees(p, t.Tunnel); wrong != "" {
return nil, fmt.Errorf("%s takes over the tunnel on %s and its placement disagrees with it: %s. "+
"Re-place it — `overlay place %s --hub --endpoint <host>:%d …` — and push again; "+
"nothing was composed", node.Name, t.Interface, wrong, node.Name, t.Port)
}
n.TakesOver = &overlay.TakeOver{Interface: t.Interface, Unit: t.Unit, Config: t.Config, Port: t.Port, MTU: t.MTU}
}
if p.Hub {
for _, cp := range carried {
n.Carried = append(n.Carried, overlay.Carried{Key: cp.PublicKey, Address: cp.Address})
}
}
nodes = append(nodes, n)
}
if len(nodes) == 0 {
// Nobody was given it: an empty network is a legitimate mesh, not a broken one.
return overlay.Empty(), nil
}
g, err := overlay.From(nodes, c.overlayRange(), "")
if err != nil && len(refused) > 0 {
// The network is missing something, and some machines could not be resolved at all: almost always the same fact.
var who []string
for name, why := range refused {
who = append(who, fmt.Sprintf(" %s: %s", name, why))
}
sort.Strings(who)
return nil, fmt.Errorf("%w\n\nand %d node(s) could not be resolved at all, which is "+
"probably why:\n%s", err, len(refused), strings.Join(who, "\n"))
}
return g, err
}
// Graph is the whole mesh's network, for showing it: the nodes on it and each one's peers.
func Graph(in Input) ([]overlay.Node, overlay.Graph, error) {
c := newComposer(in)
on, refused, err := c.whoResolves(overlay.Requirement)
if err != nil {
return nil, nil, err
}
g, err := c.network(on, refused)
if err != nil {
return nil, nil, err
}
return g.Nodes(), g.Graph(), nil
}
// whoResolves is the machines whose resolution answers a requirement, and why the others did not: by what a module
// provides, not by its name, resolved rather than read from the assignments.
func (c *composer) whoResolves(requirement string) (map[string]bool, map[string]string, error) {
on := map[string]bool{}
// Why a node could not be resolved, kept rather than raised: one broken node must not stop the rest being described.
refused := map[string]string{}
for _, n := range c.in.sorted() {
resolved, err := c.planFor(n.Name)
switch {
case unresolvable(err):
refused[n.Name] = err.Error()
continue
case err != nil:
// Not a node that does not resolve: a question that went unanswered.
return nil, nil, fmt.Errorf("whether %s answers %q cannot be read: %w", n.Name, requirement, err)
}
for _, m := range resolved.Plan.Modules {
for _, offered := range m.Offers() {
if offered == requirement {
on[n.Name] = true
}
}
}
}
return on, refused, nil
}
// Generators is what every node's declaration is composed with, over the whole mesh: the private network's generator
// (one generator now; the names and a resolver's zone file are facts a module asks for in its manifest).
func Generators(in Input) (map[string]catalogue.Generator, error) {
c := newComposer(in)
return c.generators()
}
func (c *composer) generators() (map[string]catalogue.Generator, error) {
on, refused, err := c.whoResolves(overlay.Addressing)
if err != nil {
return nil, err
}
net, err := c.network(on, refused)
if err != nil {
return nil, err
}
return map[string]catalogue.Generator{overlay.Name: net}, nil
}
// disagrees says how a node's placement differs from the tunnel it takes over — its address not the tunnel's, its
// endpoint not on the tunnel's port — or nothing when both agree.
func disagrees(p Place, t Tunnel) string {
var wrong []string
want := t.Address
if i := strings.Index(want, "/"); i >= 0 {
want = want[:i]
}
if p.Address != want {
wrong = append(wrong, fmt.Sprintf("its address is %s and the tunnel's is %s", orNothing(p.Address), want))
}
if p.Reachable() && portOfEndpoint(p.Endpoint) != strconv.Itoa(t.Port) {
wrong = append(wrong, fmt.Sprintf("its endpoint %s is not on the tunnel's port %d", p.Endpoint, t.Port))
}
return strings.Join(wrong, "; ")
}
func orNothing(s string) string {
if s == "" {
return "unset"
}
return s
}
// portOfEndpoint is the port in host:port, or empty.
func portOfEndpoint(endpoint string) string {
if i := strings.LastIndex(endpoint, ":"); i >= 0 {
return endpoint[i+1:]
}
return ""
}
// whereEveryoneIs is each machine's name on the private network, for the ones on it. Resolved without consulting the
// rest of the mesh (it is called from that path), and refused without the catalogue.
func (c *composer) whereEveryoneIs() (map[string]string, error) {
if c.in.Catalogue == nil {
return nil, errors.New("asked where everyone is without the catalogue, which cannot be answered")
}
out := map[string]string{}
for _, p := range c.onTheNetwork() {
out[p.Name] = c.internalName(p.Name)
}
return out, nil
}
// onTheNetwork is every placed machine that resolves the private network: has an address AND runs what puts it there
// (novox/hq issue 079). A machine that does not resolve keeps its place when it holds one (a key and an address), by the
// same rule as network(), so its name and its address stay where the tunnel to it does (issue 1833); one without is not
// counted, and said.
func (c *composer) onTheNetwork() []Node {
if !c.onNetworkSet {
c.onNetwork, c.onNetworkSet = c.onTheNetworkNow(), true
}
return slices.Clone(c.onNetwork)
}
// onTheNetworkNow is onTheNetwork worked out: every placed machine's set resolved.
func (c *composer) onTheNetworkNow() []Node {
var out []Node
for _, n := range c.in.sorted() {
if n.Place.Address == "" {
continue
}
if len(n.Assigned) == 0 {
continue
}
got, err := resolution.Resolve(c.in.Catalogue, n.Assigned,
catalogue.Node{Name: n.Name, Site: n.Place.Site, Capabilities: n.Capabilities},
catalogue.World{Unchecked: true, Holdings: c.in.Holdings})
if err != nil {
if n.Place.Key != "" {
c.say("%s could not be resolved and keeps its place on the private network at %s, so the others keep their path to it",
n.Name, n.Place.Address)
out = append(out, n)
continue
}
c.say("%s is not counted as on the network: it does not resolve: %v", n.Name, err)
continue
}
for _, m := range got.Modules {
for _, offered := range m.Offers() {
if offered == overlay.Requirement {
out = append(out, n)
}
}
}
}
return out
}
+306
View File
@@ -0,0 +1,306 @@
package compose
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/overlay"
)
// onTheTunnel makes the hub of an Input what mesh-controller's network_test.go makes of anchor in
// TestTheOverlaysRangeIsTheAdoptedTunnels: adopted, enrolled with the found tunnel's key, and presenting the tunnel
// (Tunnel{wg0, wg-quick@wg0, port 51900, 192.0.2.1/24, range 192.0.2.0/24, one peer PEER-TWO}). Placed on it when
// replaced is true: its address the tunnel's, its endpoint on the tunnel's port, as `overlay place anchor --endpoint
// 198.51.100.10:51900 --site hosting --hub` leaves it.
func onTheTunnel(t *testing.T, in *Input, replaced bool) {
t.Helper()
const key = "THE-TUNNELS-KEY========================="
tunnel := Tunnel{Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf", Port: 51900,
Address: "192.0.2.1/24", Range: "192.0.2.0/24", PublicKey: key,
Peers: []TunnelPeer{{PublicKey: "PEER-TWO", Address: "192.0.2.2"}}}
hub := nodeOf(t, in, "anchor")
hub.Adopted = true
hub.Place.Key = key
if replaced {
hub.Place.Address, hub.Place.Endpoint, hub.Place.Site = "192.0.2.1", "198.51.100.10:51900", "hosting"
}
in.Tunnels = map[string]FoundTunnel{"anchor": {Tunnel: tunnel, NodeAdopted: true}}
in.Adopted = &AdoptedTunnel{Tunnel: tunnel, Hub: "anchor", Adopted: true}
in.CarriedPeers = []CarriedPeer{{PublicKey: "PEER-TWO", Address: "192.0.2.2"}}
}
// novox/hq ADR 0105 (mesh-controller network_test.go TestTheOverlaysRangeIsTheAdoptedTunnels): the range every address is
// composed from is the adopted tunnel's, read from the tunnel the hub holds, never stored anywhere else. The environment
// value of the controller is Input.OverlayCIDR here.
func TestTheOverlaysRangeIsTheAdoptedTunnels(t *testing.T) {
in := aMesh(t)
in.OverlayCIDR = "10.99.0.0/16"
if got := newComposer(in).overlayRange(); got != "10.99.0.0/16" {
t.Fatalf("without an adopted tunnel the range is not what genesis said: %q", got)
}
in.OverlayCIDR = ""
if got := newComposer(in).overlayRange(); got != DefaultOverlayCIDR {
t.Fatalf("with nothing said the range is %q, not the default", got)
}
in.OverlayCIDR = "10.99.0.0/16"
onTheTunnel(t, &in, true)
if got := newComposer(in).overlayRange(); got != "192.0.2.0/24" {
t.Fatalf("with an adopted tunnel the range is %q, not the tunnel's", got)
}
// A tunnel the hub found and did not take (its key is not the tunnel's) does not give its range.
in.Adopted.Adopted = false
if got := newComposer(in).overlayRange(); got != "10.99.0.0/16" {
t.Fatalf("a tunnel the hub did not take gave its range: %q", got)
}
in.Adopted.Adopted = true
// And the hub's declaration carries the peer and the takeover.
nodes, computed, err := Graph(in)
if err != nil {
t.Fatal(err)
}
var hubNode overlay.Node
for _, n := range nodes {
if n.Name == "anchor" {
hubNode = n
}
}
if hubNode.TakesOver == nil || hubNode.TakesOver.Unit != "wg-quick@wg0" {
t.Errorf("the hub is not told to take over the found tunnel: %+v", hubNode)
}
carried := false
for _, p := range computed["anchor"] {
if p.Key == "PEER-TWO" && p.Allowed == "192.0.2.2/32" {
carried = true
}
}
if !carried {
t.Errorf("the hub's peer list does not carry the tunnel's peer: %+v", computed["anchor"])
}
}
// A takeover is composed only for a hub whose placement agrees with the tunnel (mesh-controller network_test.go
// TestATakeoverIsNotComposedForAHubPlacedOffItsTunnel): an address or an endpoint port that differs would have the host stop
// the found interface and raise the mesh's where no peer is listening.
func TestATakeoverIsNotComposedForAHubPlacedOffItsTunnel(t *testing.T) {
in := aMesh(t)
onTheTunnel(t, &in, false)
// aMesh placed anchor at 10.77.0.1 on :51820, the record of a hub placed before it took the tunnel over.
_, _, err := Graph(in)
if err == nil {
t.Fatal("a takeover was composed for a hub whose address and port are not the tunnel's")
}
for _, want := range []string{"10.77.0.1", "192.0.2.1", "51820", "51900", "overlay place anchor"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
// Re-placed on the tunnel, it composes.
onTheTunnel(t, &in, true)
if _, _, err := Graph(in); err != nil {
t.Fatalf("re-placed on the tunnel, the graph still refuses: %v", err)
}
}
// 1360/AC2: the generator gives the addresses, peers and names of aMesh: anchor the hub at 10.77.0.1, laptop at 10.77.0.2
// peering through it, both written for their machines.
func TestTheGeneratorGivesTheMeshItsAddressesAndPeers(t *testing.T) {
in := aMesh(t)
gens, err := Generators(in)
if err != nil {
t.Fatal(err)
}
g, ok := gens[overlay.Name].(*overlay.Generator)
if !ok {
t.Fatalf("the generator is %T", gens[overlay.Name])
}
addr := map[string]string{}
for _, n := range g.Nodes() {
addr[n.Name] = n.Address
}
if addr["anchor"] != "10.77.0.1" || addr["laptop"] != "10.77.0.2" || len(addr) != 2 {
t.Fatalf("the addresses are %v", addr)
}
peers := g.Graph()
if len(peers["laptop"]) != 1 || peers["laptop"][0].Name != "anchor" || peers["laptop"][0].Endpoint != "anchor.example:51820" {
t.Errorf("laptop's peers are %+v, want the hub", peers["laptop"])
}
if len(peers["anchor"]) != 1 || peers["anchor"][0].Name != "laptop" || peers["anchor"][0].Allowed != "10.77.0.2/32" {
t.Errorf("anchor's peers are %+v, want laptop", peers["anchor"])
}
for _, node := range []string{"anchor", "laptop"} {
res, on, err := g.Resources(node)
if err != nil || !on || len(res) == 0 {
t.Errorf("%s is given no network: %v %v %d", node, on, err, len(res))
}
}
}
// 1360/AC3: a node not on the private network (it runs nothing that puts it there) gets no address and no network, and the
// one that stayed keeps its own.
func TestANodeNotOnThePrivateNetworkGetsNoAddressAndNoNetwork(t *testing.T) {
in := aMesh(t)
nodeOf(t, &in, "laptop").Assigned = nil // it keeps its place and its address, and stops running the network
gens, err := Generators(in)
if err != nil {
t.Fatal(err)
}
g := gens[overlay.Name].(*overlay.Generator)
if len(g.Nodes()) != 1 || g.Nodes()[0].Name != "anchor" {
t.Fatalf("the network is over %+v, want anchor alone", g.Nodes())
}
if res, on, err := g.Resources("laptop"); on || err != nil || len(res) != 0 {
t.Errorf("a machine that left the network is given one: %v %v %v", res, on, err)
}
if _, in := g.Graph()["laptop"]; in {
t.Error("a machine that left the network has peers")
}
}
// 1360/AC3: a node whose resolution is refused is said with today's words: the network missing its hub names the
// machine that could not be resolved, which is almost always why. The hub here holds no complete place (no key): a refused
// machine with one keeps it on the network (issue 1833), so only one without is missing from it.
func TestARefusedNodeIsSaidWhenTheNetworkCannotBeComposed(t *testing.T) {
in := aMesh(t)
nodeOf(t, &in, "anchor").Place.Key = ""
one := catalogue.Manifest{Module: "rival-one", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
two := catalogue.Manifest{Module: "rival-two", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
in.Catalogue["rival-one"], in.Catalogue["rival-two"] = one, two
hub := nodeOf(t, &in, "anchor")
hub.Assigned = append(hub.Assigned, "rival-one", "rival-two") // its own set is incoherent: the hub does not resolve
_, err := Generators(in)
if err == nil {
t.Fatal("a network without its hub was composed")
}
for _, want := range []string{overlay.ErrNoHub.Error(), "and 1 node(s) could not be resolved at all, which is probably why:", " anchor: "} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q:\n%v", want, err)
}
}
}
// 1360/AC4: Generators changes nothing in its Input.
func TestGeneratorsChangesNothingInItsInput(t *testing.T) {
unchanged(t, func() Input {
in := aMesh(t)
onTheTunnel(t, &in, true)
in.Joining = []string{"laptop"}
return in
}, func(in Input) {
if _, err := Generators(in); err != nil {
t.Fatal(err)
}
})
}
// Each field of Input that the network reads reaches the generator: change it and the generator changes.
func TestEachFieldOfTheInputReachesTheGenerator(t *testing.T) {
addressesOf := func(in Input) map[string]string {
t.Helper()
gens, err := Generators(in)
if err != nil {
t.Fatalf("%v", err)
}
out := map[string]string{}
for _, n := range gens[overlay.Name].(*overlay.Generator).Nodes() {
out[n.Name] = n.Address
}
return out
}
base := addressesOf(aMesh(t))
if len(base) != 2 {
t.Fatalf("the base mesh has %v", base)
}
t.Run("Catalogue: without the module offering it nobody is on the network", func(t *testing.T) {
in := aMesh(t)
// The module stays known, so every machine still resolves: a machine that does not resolve keeps its place (issue 1833).
in.Catalogue = map[string]catalogue.Manifest{overlay.Name: {Module: overlay.Name, Version: "1"}}
if got := addressesOf(in); len(got) != 0 {
t.Errorf("got %v", got)
}
})
t.Run("Nodes.Place: an address", func(t *testing.T) {
in := aMesh(t)
nodeOf(t, &in, "laptop").Place.Address = "10.77.0.9"
if got := addressesOf(in); got["laptop"] != "10.77.0.9" {
t.Errorf("got %v", got)
}
})
t.Run("Nodes.Assigned", func(t *testing.T) {
in := aMesh(t)
nodeOf(t, &in, "laptop").Assigned = nil
if got := addressesOf(in); len(got) != 1 {
t.Errorf("got %v", got)
}
})
t.Run("Nodes.Capabilities: a capability the module needs", func(t *testing.T) {
in := aMesh(t)
m := in.Catalogue[overlay.Name]
m.Capabilities = []string{"wireguard"}
in.Catalogue[overlay.Name] = m
nodeOf(t, &in, "laptop").Capabilities = map[string]bool{}
got, err := Generators(in)
if err != nil {
t.Fatal(err)
}
if n := got[overlay.Name].(*overlay.Generator).Nodes(); len(n) != 1 || n[0].Name != "anchor" {
t.Errorf("a machine without the capability is on the network: %+v", n)
}
})
t.Run("OverlayCIDR", func(t *testing.T) {
in := aMesh(t)
in.OverlayCIDR = "10.99.0.0/16"
_, graph, err := Graph(in)
if err != nil {
t.Fatal(err)
}
found := false
for _, p := range graph["laptop"] {
found = found || p.Allowed == "10.99.0.0/16"
}
if !found {
t.Errorf("the hub's route is not the range given: %+v", graph["laptop"])
}
})
t.Run("Tunnels and Adopted and CarriedPeers", func(t *testing.T) {
in := aMesh(t)
onTheTunnel(t, &in, true)
nodes, graph, err := Graph(in)
if err != nil || len(nodes) != 2 {
t.Fatalf("%v %v", nodes, err)
}
if len(graph["anchor"]) < 2 {
t.Errorf("the hub's peers do not include the carried one: %+v", graph["anchor"])
}
})
t.Run("Joining: a machine with a key and an address is on the network before it is anything else", func(t *testing.T) {
in := aMesh(t)
in.Nodes = append(in.Nodes, Node{Name: "newcomer", Place: Place{Key: keyOf(t, "newcomer"), Address: "10.77.0.3"}})
if got := addressesOf(in); got["newcomer"] != "" {
t.Fatalf("a machine that runs nothing is on the network: %v", got)
}
in.Joining = []string{"newcomer"}
if got := addressesOf(in); got["newcomer"] != "10.77.0.3" {
t.Errorf("a joining machine is not on the network: %v", got)
}
})
t.Run("Holdings: a holder on record settles a seat two assignments could hold", func(t *testing.T) {
withSeat(t)
in := aMesh(t)
for _, name := range []string{"holder-one", "holder-two"} {
in.Catalogue[name] = aBrokerClaiming(name)
}
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "holder-one", "holder-two")
// Without a key: a machine that does not resolve and holds a complete place keeps it on the network (issue 1833).
laptop.Place.Key = ""
if got := addressesOf(in); got["laptop"] != "" {
t.Fatalf("a machine whose set claims one seat twice is on the network: %v", got)
}
in.Holdings = []catalogue.Held{{Claim: "mesh-broker", Scope: catalogue.ScopeMesh, Node: "laptop", Module: "holder-one"}}
if got := addressesOf(in); got["laptop"] != "10.77.0.2" {
t.Errorf("a holder on record did not settle the seat: %v", got)
}
})
}
+352
View File
@@ -0,0 +1,352 @@
package compose
import (
"encoding/json"
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/compose/resolution"
"github.com/novox/mesh-host/internal/validate"
)
// Declared is a declaration as a machine is sent it (the controller's sendable, sendable.go): its resources and, for an adopted
// node, its mode and which modules were taken on it (novox/hq ADR 0100), with what the composition says beside it and never puts on
// the wire.
//
// **Body is the only place the envelope is marshalled**, so the digest the mesh compares and the body validated are one.
type Declared struct {
Resources []map[string]any
// Sequence, Epoch and Generation are what the Input gave the node: zero is not sent at all.
Sequence int64
Epoch uint64
Generation int64
// ReadsModules says the machine's node-engine reads a module per resource and a build per module (novox/hq issue 1145): only
// then are they sent. ModuleOf is the composer's owner of each resource id and BuildIDs the build each module carries.
ReadsModules bool
ModuleOf map[string]string
BuildIDs map[string]string
// Adoption is nil for a converged node, and then the body is byte for byte what it was before adoption existed.
Adoption *Adoption
// Filled is the path of every file this declaration hands to the provisioner, and Provisioner the account its client runs as
// (novox/hq ADR 0349): on the wire as `filled_files` and `provisioner_account`, both absent when nothing is handed.
Filled []string
Provisioner string
// LeftOut is every module of the machine's set left out because a stored setting cannot compose with its definition (novox/hq
// ADR 0163, rule 6), sorted; LeftOutWhy is why each was.
LeftOut []string
LeftOutWhy map[string]string
// Not on the wire:
Received map[string]map[string][]catalogue.Contribution
Mesh []string
BusUsers string
// Modules are the modules this declaration names, sorted.
Modules []string
// LocalNetwork is every port admitted from the local network and what it resolved to (novox/hq ADR 0340).
LocalNetwork []catalogue.LocalNetworkPort
Withheld []catalogue.Overflow
Unbound []catalogue.Unbound
Unplaced []string
HandedOnly []string
// RequiredBy is, for each handed module only pulled in by another, the module that requires it (issue 765).
RequiredBy map[string]string
// Foreseen is every own secret composed with a stand-in, as `module/name`; StoodIn is everything the composition stood in for
// (an own secret, a grant's sealed copy, the certificate, the bus's user list); ForeseenPorts the ports the mesh may assign that
// the Input holds none for. A declaration that has any is never sent: what a send would make, named, never invented.
Foreseen []string
StoodIn []string
ForeseenPorts []string
// CoreBoxesWanted are the core credentials whose box the send must ask the vault for and keep.
CoreBoxesWanted []CoreBoxWanted
// Passes are the seats the mesh would pass on, for the caller to record, and Said the lines found while composing.
Passes Passes
Said []string
}
// Adoption is what an adopted node is told about its mode. Taken is every module taken on it that it runs; Untaken is, for every
// module it runs that is not taken, the ids of every one of that module's resources: what the host keeps as found until the module
// is taken (ADR 0103).
type Adoption struct {
Taken []string `json:"taken"`
Untaken map[string][]string `json:"untaken,omitempty"`
}
// Body is the declaration's bytes, as sent and as digested.
func (d Declared) Body() ([]byte, error) {
envelope := map[string]any{"declaration": 1, "resources": d.Resources}
if d.ReadsModules {
// The module of each resource the composer owns, and the build of each module (novox/hq issue 1145): to a machine that
// said it reads them, and no other.
named := make([]map[string]any, len(d.Resources))
for n, r := range d.Resources {
module, owned := d.ModuleOf[fmt.Sprint(r["id"])]
if !owned {
named[n] = r
continue
}
with := make(map[string]any, len(r)+1)
for k, v := range r {
with[k] = v
}
with["module"] = module
named[n] = with
}
envelope["resources"] = named
if len(d.BuildIDs) > 0 {
envelope["builds"] = d.BuildIDs
}
}
if d.Adoption != nil {
envelope["adoption"] = d.Adoption
}
if d.Sequence > 0 {
envelope["sequence"] = d.Sequence
}
if d.Epoch > 0 {
envelope["epoch"] = d.Epoch
}
if len(d.LeftOut) > 0 {
envelope["left_out"] = d.LeftOut
}
if d.Generation > 0 {
envelope["generation"] = d.Generation
}
if len(d.Filled) > 0 {
envelope["filled_files"] = d.Filled
envelope["provisioner_account"] = d.Provisioner
}
// An empty declaration is deliberate: the node owns nothing the mesh put there (novox/hq issue 127), and the host refuses an
// empty body unless it is told the emptiness is meant.
if len(d.Resources) == 0 {
envelope["owns_nothing"] = true
}
return json.Marshal(envelope)
}
// Refusals are the findings a check refuses a change for, beside the validator's problems: a consumer left out of the grants
// because its identity overflows the provision's bound, and a credential bound to another consumer (novox/hq ADR 0446 clause 1).
// Each names the consumer and the provision.
func (d Declared) Refusals() []string {
var out []string
for _, o := range d.Withheld {
out = append(out, o.String())
}
for _, u := range d.Unbound {
out = append(out, u.String())
}
sort.Strings(out)
return out
}
// Compose composes one node's declaration (the issue calls it Node; the Input already has a type of that name) over an Input -- as a push would, without the sending (the controller's
// composedAndValidated in the Foreseeing mode) -- and runs the node-engine's own validator over its body. An error is that the
// node did not compose (a set that does not compose satisfies IsUnresolvable); problems are what the validator refuses, in the
// node-engine's words. It allocates, makes, keeps and writes nothing, and names what a send would do as foreseen.
func Compose(in Input, node string) (Declared, []string, error) {
return ComposeIn(in, node, Foreseeing)
}
// ComposeIn is Compose in a chosen mode.
func ComposeIn(in Input, node string, mode Mode) (Declared, []string, error) {
c := newComposer(in)
d, err := c.declaration(node, mode)
if err != nil {
return Declared{}, nil, err
}
body, err := d.Body()
if err != nil {
return Declared{}, nil, err
}
return d, validate.Declaration(body), nil
}
// declaration is the controller's declarationWith over the plain Input.
func (c *composer) declaration(nodeName string, mode Mode) (Declared, error) {
resolved, err := c.planFor(nodeName)
if err != nil {
return Declared{}, err
}
gens, err := c.generators()
if err != nil {
return Declared{}, err
}
rendered, err := c.render(nodeName, mode)
if err != nil {
return Declared{}, err
}
with := rendered.Rendering
with.Generators = gens
plan := resolved.Plan
composed, err := resolution.Compose(plan, with)
if err != nil {
return Declared{}, err
}
// Which ports the filter admits from the local network, and from which networks (novox/hq ADR 0340 clause 8): the same rules the
// composition rendered, never sent.
rules, err := resolution.Rules(plan, with)
if err != nil {
return Declared{}, err
}
node, err := c.in.node(nodeName)
if err != nil {
return Declared{}, err
}
out := Declared{Resources: composed.Resources, Adoption: adoptionFor(plan, node, composed),
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), LeftOutWhy: composed.LeftOut,
Withheld: rendered.Withheld, Unbound: rendered.Unbound,
Foreseen: composed.Foreseen, StoodIn: rendered.Foreseen, ForeseenPorts: rendered.ForeseenPorts,
Unplaced: composed.Unplaced, HandedOnly: composed.HandedOnly, Modules: namedModules(plan, composed.LeftOut),
Filled: composed.Filled, LocalNetwork: resolution.LocalNetworkPorts(rules, with.LocalNetworks),
CoreBoxesWanted: rendered.CoreBoxesWanted, Said: rendered.Said,
Sequence: node.Sequence, Epoch: node.Epoch, Generation: node.Generation}
if len(composed.Filled) > 0 {
out.Provisioner = with.Provisioner
}
// A handed module only pulled in by another keeps what requires it, for its value of its own in the assignments mirror (issue 765).
for module := range composed.Plans {
if by := resolution.RequiredBy(plan, module); by != "" {
if out.RequiredBy == nil {
out.RequiredBy = map[string]string{}
}
out.RequiredBy[module] = by
}
}
// And the module of each resource and the build of each module, to a machine whose node-engine said it reads them (novox/hq issue
// 1145) and to no other.
if node.ReadsModules {
carried := c.carriedOn(node, plan, composed.LeftOut)
out.ReadsModules, out.ModuleOf, out.BuildIDs = true, composed.Owner, c.wireBuilds(carried, node.Runs)
}
_, passes, left, err := c.theRestOfTheMesh("")
if err != nil {
return Declared{}, err
}
out.Passes = passes
for _, l := range left {
out.Said = append(out.Said, l.String())
}
out.Said = c.saidBeside(out.Said)
return out, nil
}
// carriedOn is the build of each module a declaration for the node carries (plan.go carriedOn, in a question that sends nothing): the
// module's current build for each module in it, and for a module left out of it the build it was last sent, since the machine keeps
// that one -- or nothing, when that is not known.
func (c *composer) carriedOn(node Node, plan catalogue.Resolution, leftOut map[string]string) map[string]string {
before := node.SentBuilds
if !node.SentKnown {
before = nil
}
out := map[string]string{}
for _, m := range plan.Modules {
if _, left := leftOut[m.Module]; left {
if was, kept := before[m.Module]; kept {
out[m.Module] = was
}
continue
}
out[m.Module] = c.in.CurrentBuilds[m.Module]
// A module kept at the build the machine runs is carried at that build (plan.go carriedOn).
if kept, ok := node.KeptBuilds[m.Module]; ok && kept != "" {
out[m.Module] = kept
}
}
return out
}
// wireBuilds is the build of each carried module that goes on the wire (novox/hq issue 1145): the build its assignment names when
// that assignment is of the commit carried, else the build made from that commit; a module with neither is not named.
func (c *composer) wireBuilds(carried map[string]string, runs map[string]Run) map[string]string {
out := map[string]string{}
for module, commit := range carried {
if commit == "" {
continue
}
if ref, ok := runs[module]; ok && ref.Commit == commit && ref.Build != "" {
out[module] = ref.Build
continue
}
if build := c.in.BuildIDs[module][commit]; build != "" {
out[module] = build
}
}
return out
}
// adoptionFor is the envelope computed from what was taken and who owns each resource: nil for a converged node. Every module the
// node runs that is not taken is untaken -- including one pulled in by another rather than assigned.
func adoptionFor(plan catalogue.Resolution, node Node, composed catalogue.Composed) *Adoption {
if !node.Adopted {
return nil
}
isTaken := map[string]bool{}
for _, m := range node.Taken {
isTaken[m] = true
}
out := &Adoption{Taken: []string{}}
runs := map[string]bool{}
for _, m := range plan.Modules {
runs[m.Module] = true
if isTaken[m.Module] {
out.Taken = append(out.Taken, m.Module)
}
}
sort.Strings(out.Taken)
for _, r := range composed.Resources {
id, _ := r["id"].(string)
module, owned := composed.Owner[id]
// Every kind, not only files and containers (novox/hq ADR 0103). What the mesh declares of its own is never held.
if !owned || !runs[module] || isTaken[module] || strings.HasPrefix(id, catalogue.AdoptionPrefix) {
continue
}
if out.Untaken == nil {
out.Untaken = map[string][]string{}
}
out.Untaken[module] = append(out.Untaken[module], id)
}
return out
}
// sortedKeysOf is a map's keys, sorted: so what a declaration says it left out does not move for a reordering nobody made.
func sortedKeysOf(m map[string]string) []string {
if len(m) == 0 {
return nil
}
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// namedModules are the modules a declaration names: its machine's set, less what was left out of it.
func namedModules(plan catalogue.Resolution, leftOut map[string]string) []string {
out := make([]string, 0, len(plan.Modules))
for _, m := range plan.Modules {
if _, left := leftOut[m.Module]; !left {
out = append(out, m.Module)
}
}
sort.Strings(out)
return out
}
// DataItems is every data item the modules placed on a node declare, placed there (novox/hq ADR 0571 clause 3): what the
// node-backup holder is handed as `${backup:data}`, composed over the Input as Compose composes the node, so disk-load
// reads the list from the bus with no composed file. Unplaced names each item left out and why.
func DataItems(in Input, node string) (map[string][]catalogue.PlacedDataItem, []string, error) {
c := newComposer(in)
resolved, err := c.planFor(node)
if err != nil {
return nil, nil, err
}
rendered, err := c.render(node, Planning)
if err != nil {
return nil, nil, err
}
return resolution.DataItems(resolved.Plan, rendered.Rendering)
}
+424
View File
@@ -0,0 +1,424 @@
package compose
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/overlay"
)
func resourceIDs(d Declared) []string {
var ids []string
for _, r := range d.Resources {
ids = append(ids, r["type"].(string)+":"+r["id"].(string))
}
return ids
}
func has(ids []string, want string) bool {
for _, id := range ids {
if id == want {
return true
}
}
return false
}
// aSmallMesh is three nodes, a provider and its consumer, a seat and the private network: aMesh with `ace` running a database and
// a resolver holding the mesh's resolver seat on record (the providers and consumer of mesh-controller's facts_test.go
// aMeshWithSecrets, the seat of bindings_test.go storeManifests).
func aSmallMesh(t *testing.T) Input {
t.Helper()
in := threeNodes(t)
for _, m := range storeManifests() {
if m.Module == "resolver" {
in.Catalogue[m.Module] = m
}
}
ace := nodeOf(t, &in, "ace")
ace.Assigned = append(ace.Assigned, "resolver")
in.Holdings = []catalogue.Held{{Claim: "mesh-dns-resolver", Scope: catalogue.ScopeMesh, Node: "ace", Module: "resolver", Site: "there"}}
return in
}
// 1298/AC1: each node of a small mesh composes, its body is accepted by the node-engine's validator, and it names the resources the
// controller composes for the same case: the private network's files on the nodes that run it, the provider's received file, the
// consumer's file.
func TestEachNodeOfASmallMeshComposesAndItsBodyIsAcceptedByTheValidator(t *testing.T) {
in := aSmallMesh(t)
want := map[string][]string{
"anchor": {"package:mesh-wireguard.overlay-tools", "file:mesh-wireguard.overlay-config", "service:mesh-wireguard.overlay-up", "file:objects.received-s3-bucket"},
"laptop": {"package:mesh-wireguard.overlay-tools", "file:mesh-wireguard.overlay-config", "service:mesh-wireguard.overlay-up", "file:files.config"},
"ace": nil,
}
for node, ids := range want {
d, problems, err := Compose(in, node)
if err != nil || len(problems) != 0 {
t.Fatalf("%s: %v %v", node, err, problems)
}
for _, id := range ids {
if !has(resourceIDs(d), id) {
t.Errorf("%s does not name %s: %v", node, id, resourceIDs(d))
}
}
body, err := d.Body()
if err != nil {
t.Fatal(err)
}
var parsed struct {
Declaration int `json:"declaration"`
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &parsed); err != nil || parsed.Declaration != 1 || len(parsed.Resources) != len(d.Resources) {
t.Errorf("%s: body %s %v", node, body, err)
}
}
// ace runs a resolver and a database and nothing the mesh declares of its own: an empty declaration says it owns nothing.
d, _, _ := Compose(in, "ace")
if body, _ := d.Body(); !strings.Contains(string(body), `"owns_nothing":true`) {
t.Errorf("an empty declaration is not said to be deliberate: %s", body)
}
}
// 1298/AC2 (mesh-controller merge_gate_integration_test.go TestIssue236AManifestTheNodeEngineRefusesFailsThePullRequest, the manifest):
// a service that omits its state is accepted by the catalogue's check and refused by the node-engine; Compose returns the validator's
// problems in the node-engine's words and no error.
func TestAManifestTheNodeEngineRefusesComesBackAsProblemsAndNoError(t *testing.T) {
in := aMesh(t)
var lemurs catalogue.Manifest
if err := json.Unmarshal([]byte(`{"module":"lemurs","version":"1","capabilities":["systemd"],
"resources":[{"id":"service","type":"service","unit":"lemurs.service","boot":"enabled"}]}`), &lemurs); err != nil {
t.Fatal(err)
}
in.Catalogue["lemurs"] = lemurs
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "lemurs")
d, problems, err := Compose(in, "laptop")
if err != nil {
t.Fatalf("a refusal of the validator is an error: %v", err)
}
if len(problems) == 0 || !strings.Contains(strings.Join(problems, "\n"), "lemurs.service") {
t.Fatalf("problems are %v", problems)
}
if !has(resourceIDs(d), "service:lemurs.service") {
t.Errorf("the declaration does not carry the resource: %v", resourceIDs(d))
}
// And the same, as the validator says it of the body.
body, _ := d.Body()
if again := strings.Join(problems, "\n"); again == "" || !strings.Contains(string(body), "lemurs.service") {
t.Errorf("the body does not name the service: %s", body)
}
}
// 1298/AC3: a node whose own set does not compose returns an error and no declaration, told apart from an unreadable Input.
func TestANodeWhoseSetDoesNotComposeReturnsAnErrorAndNoDeclaration(t *testing.T) {
in := aMesh(t)
in.Catalogue["rival-one"] = catalogue.Manifest{Module: "rival-one", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
in.Catalogue["rival-two"] = catalogue.Manifest{Module: "rival-two", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "rival-one", "rival-two")
d, problems, err := Compose(in, "laptop")
if err == nil || !IsUnresolvable(err) || d.Resources != nil || problems != nil {
t.Fatalf("a set that does not compose: %v %v %v", d.Resources, problems, err)
}
if _, _, err := Compose(in, "nowhere"); err == nil || IsUnresolvable(err) {
t.Fatalf("a node the Input does not have: %v", err)
}
}
// 1298/AC4: composing the same Input twice gives byte-identical bodies, and the Input is unchanged.
func TestComposingTwiceGivesTheSameBytesAndChangesNothing(t *testing.T) {
mk := func() Input {
in := aSmallMesh(t)
nodeOf(t, &in, "anchor").Account = "jochens"
nodeOf(t, &in, "laptop").Sequence = 7
return in
}
in := mk()
for _, node := range []string{"anchor", "laptop", "ace"} {
a, _, err := Compose(in, node)
if err != nil {
t.Fatal(err)
}
b, _, err := Compose(in, node)
if err != nil {
t.Fatal(err)
}
ab, _ := a.Body()
bb, _ := b.Body()
if string(ab) != string(bb) {
t.Errorf("%s: two compositions differ:\n%s\n%s", node, ab, bb)
}
}
unchanged(t, mk, func(in Input) {
for _, node := range []string{"anchor", "laptop", "ace"} {
if _, _, err := Compose(in, node); err != nil {
t.Fatal(err)
}
}
})
}
// 1298/AC5: what a send would make is named in the result as foreseen and never invented: an own secret, a port, a core box, a grant's
// sealed copy.
func TestWhatASendWouldMakeIsNamedAsForeseenAndNeverInvented(t *testing.T) {
in := aGrantMesh(t)
in.Issued[0].ForProvider = ""
in.Catalogue["assistant"] = anAssistant()
in.Catalogue["web"] = aWebModule()
in.Catalogue["nats"] = catalogue.Manifest{Module: "nats", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/var/lib/mesh/nats/broker", Taken: catalogue.TakenAtStart}}}
anchor := nodeOf(t, &in, "anchor")
anchor.Assigned = append(anchor.Assigned, "assistant", "web", "nats")
anchor.SealingKey = keyOf(t, "anchor")
anchor.Stated, anchor.Contract = true, VaultBoxContract
in.BusMinted = []string{"anchor.nats"}
d, problems, err := Compose(in, "anchor")
if err != nil || len(problems) != 0 {
t.Fatalf("%v %v", err, problems)
}
if !has(d.Foreseen, "assistant/password") || !has(d.Foreseen, "nats/broker") {
t.Errorf("own secrets foreseen: %v", d.Foreseen)
}
if len(d.ForeseenPorts) != 1 || d.ForeseenPorts[0] != "web:8080" {
t.Errorf("foreseen ports: %v", d.ForeseenPorts)
}
if len(d.CoreBoxesWanted) != 1 || d.CoreBoxesWanted[0].Module != "nats" {
t.Errorf("core boxes wanted: %v", d.CoreBoxesWanted)
}
if !has(d.StoodIn, "the grant of s3-bucket for files on laptop") {
t.Errorf("stood in for: %v", d.StoodIn)
}
body, _ := d.Body()
if strings.Contains(string(body), secretValue) {
t.Error("a secret value is in the body")
}
}
// 1298/AC6: the body carries the sequence, epoch and generation the Input gives, and the module of each resource and the carried builds
// only to a node whose Input says it reads modules (novox/hq issue 1145).
func TestTheBodyCarriesTheOrderTheInputGivesAndTheBuildsOnlyToAnEngineThatReadsThem(t *testing.T) {
in := aSmallMesh(t)
laptop := nodeOf(t, &in, "laptop")
laptop.Sequence, laptop.Epoch, laptop.Generation = 42, 57, 9
in.CurrentBuilds = map[string]string{"files": "commit-files", "mesh-wireguard": "commit-wg"}
in.BuildIDs = map[string]map[string]string{"files": {"commit-files": "build-7"}}
// An assignment of another commit than the one carried does not name the build; the build made from the commit carried does.
laptop.Runs = map[string]Run{"mesh-wireguard": {Commit: "commit-wg", Build: "build-9"}, "files": {Commit: "an-older-commit", Build: "build-6"}}
d, _, err := Compose(in, "laptop")
if err != nil {
t.Fatal(err)
}
body, _ := d.Body()
var got map[string]any
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
if got["sequence"] != float64(42) || got["epoch"] != float64(57) || got["generation"] != float64(9) {
t.Errorf("the order is %v %v %v", got["sequence"], got["epoch"], got["generation"])
}
if _, there := got["builds"]; there {
t.Errorf("builds are sent to an engine that does not read them: %s", body)
}
for _, r := range got["resources"].([]any) {
if _, there := r.(map[string]any)["module"]; there {
t.Errorf("a module is named on a resource of an engine that does not read them: %v", r)
}
}
laptop.ReadsModules = true
d, _, err = Compose(in, "laptop")
if err != nil {
t.Fatal(err)
}
body, _ = d.Body()
got = nil
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
builds, _ := got["builds"].(map[string]any)
if builds["files"] != "build-7" || builds["mesh-wireguard"] != "build-9" {
t.Errorf("builds are %v", builds)
}
named := false
for _, r := range got["resources"].([]any) {
if r.(map[string]any)["id"] == "files.config" && r.(map[string]any)["module"] == "files" {
named = true
}
}
if !named {
t.Errorf("a resource is not named for its module: %s", body)
}
// Zero is not sent at all.
laptop.Sequence, laptop.Epoch, laptop.Generation = 0, 0, 0
d, _, _ = Compose(in, "laptop")
body, _ = d.Body()
for _, key := range []string{`"sequence"`, `"epoch"`, `"generation"`} {
if strings.Contains(string(body), key) {
t.Errorf("%s is sent as zero: %s", key, body)
}
}
}
// A module left out of a build set keeps the build it was last sent (novox/hq issue 259): the machine keeps that one.
func TestAModuleLeftOutKeepsTheBuildItWasLastSent(t *testing.T) {
c := newComposer(Input{CurrentBuilds: map[string]string{"a": "new-a", "b": "new-b"}})
plan := catalogue.Resolution{Modules: []catalogue.Manifest{{Module: "a"}, {Module: "b"}, {Module: "c"}}}
node := Node{SentBuilds: map[string]string{"b": "old-b", "c": "old-c"}, SentKnown: true}
got := c.carriedOn(node, plan, map[string]string{"b": "its settings cannot compose"})
if got["a"] != "new-a" || got["b"] != "old-b" || got["c"] != "" {
t.Errorf("carried builds are %v", got)
}
node.SentKnown = false
if got := c.carriedOn(node, plan, map[string]string{"b": "x"}); len(got) != 2 || got["b"] != "" {
t.Errorf("when what was sent is not known: %v", got)
}
}
// An adopted node is told which modules were taken and, for every module it runs that is not, the ids of its resources
// (novox/hq ADR 0100 and 0103); a converged node is told nothing, and its body is as it was before adoption.
func TestAnAdoptedNodeIsToldWhatWasTakenAndWhatIsKept(t *testing.T) {
in := aSmallMesh(t)
d, _, err := Compose(in, "laptop")
if err != nil || d.Adoption != nil {
t.Fatalf("a converged node: %v %v", d.Adoption, err)
}
laptop := nodeOf(t, &in, "laptop")
laptop.Adopted, laptop.Taken = true, []string{"mesh-wireguard"}
d, problems, err := Compose(in, "laptop")
if err != nil || len(problems) != 0 {
t.Fatalf("%v %v", err, problems)
}
if d.Adoption == nil || len(d.Adoption.Taken) != 1 || d.Adoption.Taken[0] != "mesh-wireguard" ||
len(d.Adoption.Untaken["files"]) == 0 || d.Adoption.Untaken["mesh-wireguard"] != nil {
t.Fatalf("adoption is %+v", d.Adoption)
}
if body, _ := d.Body(); !strings.Contains(string(body), `"adoption"`) {
t.Errorf("the adoption is not on the wire: %s", body)
}
}
// Issue 263 (merge_gate_integration_test.go TestIssue263AnIdentityARealMachineNameOverflows...): a change makes the network manager
// require the object store's provision; on a machine whose name is six characters its identity is 26 against a bound of 20. The
// composition of the provider returns the refusal, naming the module and the provision.
func TestIssue263AnIdentityARealMachineNameOverflowsIsARefusal(t *testing.T) {
in := aGrantMesh(t)
in.Catalogue["resolver"] = storeManifests()[1]
in.Catalogue["networkmanager"] = catalogue.Manifest{Module: "networkmanager", Version: "1", Requires: []string{"wildcard-resolution", "s3-bucket"}}
laptop, anchor := nodeOf(t, &in, "laptop"), nodeOf(t, &in, "anchor")
laptop.Assigned = append(laptop.Assigned, "networkmanager")
anchor.Assigned = append(anchor.Assigned, "resolver")
in.Holdings = []catalogue.Held{{Claim: "mesh-dns-resolver", Scope: catalogue.ScopeMesh, Node: "anchor", Module: "resolver", Site: "here"}}
in.Issued = append(in.Issued, Issued{Provision: "s3-bucket", Consumer: "laptop", ConsumerModule: "networkmanager", Provider: "anchor", ForProvider: "sealed"})
d, problems, err := Compose(in, "anchor")
if err != nil || len(problems) != 0 {
t.Fatalf("%v %v", err, problems)
}
refusals := strings.Join(d.Refusals(), "\n")
if !strings.Contains(refusals, "networkmanager") || !strings.Contains(refusals, "s3-bucket") || !strings.Contains(refusals, "laptop") {
t.Fatalf("the refusal does not name the module, the provision and the machine: %q", refusals)
}
// And the consumer that fits is granted.
if len(d.Withheld) != 1 || d.Withheld[0].Module != "networkmanager" {
t.Errorf("withheld %+v", d.Withheld)
}
}
// Round 1, finding 1 (plan.go keepRecorded): a recorded module kept at the build a machine runs is that machine's own resolution
// and no other's; the commit carried for it is the kept one.
func TestAKeptModuleChangesOnlyTheResolutionOfTheNodeThatKeepsIt(t *testing.T) {
in := aSmallMesh(t)
in.Catalogue["tool"] = catalogue.Manifest{Module: "tool", Version: "2"}
for _, name := range []string{"anchor", "laptop"} {
n := nodeOf(t, &in, name)
n.Assigned = append(n.Assigned, "tool")
}
anchor := nodeOf(t, &in, "anchor")
anchor.Kept = map[string]catalogue.Manifest{"tool": {Module: "tool", Version: "1"}}
versionOn := func(in Input, node string) string {
t.Helper()
got, err := Resolve(in, node)
if err != nil {
t.Fatal(err)
}
for _, m := range got.Plan.Modules {
if m.Module == "tool" {
return m.Version
}
}
t.Fatalf("no tool on %s", node)
return ""
}
if v := versionOn(in, "anchor"); v != "1" {
t.Errorf("the node that keeps it runs version %s", v)
}
if v := versionOn(in, "laptop"); v != "2" {
t.Errorf("another node runs version %s: a kept build leaked", v)
}
if got := in.Catalogue["tool"].Version; got != "2" {
t.Errorf("the Input's catalogue was changed: %s", got)
}
// And the commit carried for it is the kept one.
c := newComposer(in)
plan := catalogue.Resolution{Modules: []catalogue.Manifest{{Module: "tool"}, {Module: "other"}}}
in.CurrentBuilds = map[string]string{"tool": "current", "other": "current-other"}
c = newComposer(in)
node := Node{KeptBuilds: map[string]string{"tool": "kept-commit"}}
if got := c.carriedOn(node, plan, nil); got["tool"] != "kept-commit" || got["other"] != "current-other" {
t.Errorf("carried builds are %v", got)
}
}
// Another node's kept build changes neither the rest of the mesh nor who is on the network: the controller reads both with the
// current manifests, and only then keeps a recorded module for the node it composes (recorded_kept.go keepRecorded).
func TestAnotherNodesKeptBuildDoesNotChangeTheRestOfTheMeshOrWhoIsOnTheNetwork(t *testing.T) {
in := threeNodes(t)
nodeOf(t, &in, "anchor").Kept = map[string]catalogue.Manifest{"objects": {Module: "objects", Version: "0"}, overlayName(): {Module: overlayName(), Version: "0"}}
w, _, _, err := TheRestOfTheMesh(in, "laptop")
if err != nil || len(providers(w, "s3-bucket")) != 1 {
t.Errorf("a kept build changed what anchor offers: %v %v", providers(w, "s3-bucket"), err)
}
where, err := newComposer(in).whereEveryoneIs()
if err != nil || where["anchor"] == "" {
t.Errorf("a kept build took anchor off the network: %v %v", where, err)
}
}
func overlayName() string { return overlay.Name }
// Round 1, finding 6: the nodes the composition did not count as on the network are returned, said, not lost. laptop holds no
// complete place (no key): one that does keeps it on the network (issue 1833) and is counted.
func TestANodeNotCountedAsOnTheNetworkIsSaidInTheResult(t *testing.T) {
in := aMesh(t)
nodeOf(t, &in, "laptop").Place.Key = ""
in.Catalogue["rival-one"] = catalogue.Manifest{Module: "rival-one", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
in.Catalogue["rival-two"] = catalogue.Manifest{Module: "rival-two", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "rival-one", "rival-two")
said := func(lines []string) bool {
for _, l := range lines {
if strings.HasPrefix(l, "laptop is not counted as on the network") {
return true
}
}
return false
}
got, err := Render(in, "anchor", Foreseeing)
if err != nil || !said(got.Said) {
t.Errorf("rendered: %v %v", got.Said, err)
}
d, _, err := Compose(in, "anchor")
if err != nil || !said(d.Said) {
t.Errorf("declared: %v %v", d.Said, err)
}
n := 0
for _, l := range d.Said {
if strings.HasPrefix(l, "laptop is not counted") {
n++
}
}
if n != 1 {
t.Errorf("said %d times: %v", n, d.Said)
}
}
+167
View File
@@ -0,0 +1,167 @@
package compose
import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/overlay"
)
// fourOnTheNetwork is aMesh and two more machines on the private network at the site `there`: `ace`, dialled at
// ace.example:51820, at 10.77.0.3, and `box`, behind no endpoint, at 10.77.0.4. A module `office` requires
// `document-server`, which nothing in the catalogue provides: the shape of issue 1833 (nextcloud on the hub wanting a
// document server nobody offered).
func fourOnTheNetwork(t *testing.T) Input {
t.Helper()
in := aMesh(t)
in.Catalogue["office"] = catalogue.Manifest{Module: "office", Version: "1", Requires: []string{"document-server"}}
caps := map[string]bool{"container-runtime": true, "wireguard": true, "systemd": true}
in.Nodes = append(in.Nodes,
Node{Name: "ace", Place: Place{Key: keyOf(t, "ace"), Endpoint: "ace.example:51820", Site: "there", Address: "10.77.0.3"},
Assigned: []string{overlay.Name}, Capabilities: caps},
Node{Name: "box", Place: Place{Key: keyOf(t, "box"), Site: "there", Address: "10.77.0.4"},
Assigned: []string{overlay.Name}, Capabilities: caps})
return in
}
// unresolvableBy makes one machine's own set fail to resolve: it is also given `office`, whose requirement nothing provides.
func unresolvableBy(t *testing.T, in *Input, name string) {
t.Helper()
n := nodeOf(t, in, name)
n.Assigned = append(n.Assigned, "office")
}
// networkOf is the generator of an Input's private network, failing the test when it is not composed.
func networkOf(t *testing.T, in Input) *overlay.Generator {
t.Helper()
gens, err := Generators(in)
if err != nil {
t.Fatalf("the private network was not composed: %v", err)
}
return gens[overlay.Name].(*overlay.Generator)
}
func placesOf(g *overlay.Generator) map[string]overlay.Node {
out := map[string]overlay.Node{}
for _, n := range g.Nodes() {
out[n.Name] = n
}
return out
}
// 1833/AC1: four machines, the hub's own set does not resolve; the other three compose their declarations, and the hub alone is
// refused, with an error that is its own set failing and names what could not be resolved.
func TestOneMachineThatDoesNotResolveIsRefusedAlone(t *testing.T) {
in := fourOnTheNetwork(t)
unresolvableBy(t, &in, "anchor")
for _, node := range []string{"laptop", "ace", "box"} {
d, problems, err := Compose(in, node)
if err != nil || len(problems) != 0 {
t.Errorf("%s does not compose because another machine does not resolve: %v %v", node, err, problems)
continue
}
if !has(resourceIDs(d), "file:mesh-wireguard.overlay-config") {
t.Errorf("%s is given no private network: %v", node, resourceIDs(d))
}
}
_, _, err := Compose(in, "anchor")
if err == nil || !IsUnresolvable(err) {
t.Fatalf("the machine that does not resolve is not refused as its own set failing: %v", err)
}
if !strings.Contains(err.Error(), `nothing provides "document-server", wanted by office`) {
t.Errorf("the refusal does not name what could not be resolved: %v", err)
}
}
// 1833/AC2: the private network is composed with every machine on it: the hub that does not resolve keeps its place (key,
// address, endpoint, site, hub role), so every other machine keeps its path through it.
func TestAMachineThatDoesNotResolveKeepsItsPlaceOnTheNetwork(t *testing.T) {
in := fourOnTheNetwork(t)
unresolvableBy(t, &in, "anchor")
got := placesOf(networkOf(t, in))
if len(got) != 4 {
t.Fatalf("the network is over %v, want all four machines", got)
}
hub := got["anchor"]
if !hub.Hub || hub.Key != keyOf(t, "anchor") || hub.Address != "10.77.0.1" || hub.Endpoint != "anchor.example:51820" || hub.Site != "here" {
t.Errorf("the hub did not keep its place: %+v", hub)
}
nodes, graph, err := Graph(in)
if err != nil || len(nodes) != 4 {
t.Fatalf("the graph is not shown with every machine: %v %v", nodes, err)
}
for _, node := range []string{"ace", "box"} {
through := false
for _, p := range graph[node] {
through = through || (p.Name == "anchor" && p.Endpoint == "anchor.example:51820")
}
if !through {
t.Errorf("%s has no path through the hub: %+v", node, graph[node])
}
}
}
// 1833/AC3: a machine that does not resolve and was never placed (a new machine: no key, no address) is left off the network,
// which is composed from the rest -- here with the hub that does not resolve on its last place -- and that machine alone is refused.
func TestANewMachineThatDoesNotResolveIsLeftOffAndRefusedAlone(t *testing.T) {
in := fourOnTheNetwork(t)
unresolvableBy(t, &in, "anchor")
in.Nodes = append(in.Nodes, Node{Name: "fresh", Place: Place{Site: "there"},
Assigned: []string{overlay.Name, "office"}, Capabilities: map[string]bool{"wireguard": true, "systemd": true}})
got := placesOf(networkOf(t, in))
if _, on := got["fresh"]; on || len(got) != 4 {
t.Fatalf("the network is over %v, want the four placed machines without fresh", got)
}
if _, _, err := Compose(in, "fresh"); err == nil || !IsUnresolvable(err) {
t.Errorf("the new machine is not refused as its own set failing: %v", err)
}
if _, problems, err := Compose(in, "box"); err != nil || len(problems) != 0 {
t.Errorf("another machine does not compose: %v %v", err, problems)
}
}
// 1833/AC4: a network that truly cannot be composed from what remains still fails: the hub does not resolve and has no
// complete place (no key), so nothing is the hub; the refusal is today's, naming the machine that could not be resolved.
func TestANetworkThatCannotBeComposedFromWhatRemainsStillFails(t *testing.T) {
in := fourOnTheNetwork(t)
unresolvableBy(t, &in, "anchor")
nodeOf(t, &in, "anchor").Place.Key = ""
_, err := Generators(in)
if err == nil {
t.Fatal("a network without a hub was composed")
}
for _, want := range []string{overlay.ErrNoHub.Error(), "and 1 node(s) could not be resolved at all, which is probably why:", " anchor: "} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q:\n%v", want, err)
}
}
if _, _, err := Compose(in, "laptop"); err == nil {
t.Error("a declaration was composed over a network that cannot be")
}
}
// 1833/AC2, a failure the composition's unchecked pass refuses too (two modules claiming one seat on the hub, the 1360/AC3
// shape): the hub keeps its place in what every other machine is sent besides the tunnel -- its internal name and its address
// among the sources "from the mesh" -- and another machine still composes.
func TestAMachineThatDoesNotResolveKeepsItsNameAndItsAddressInTheMesh(t *testing.T) {
in := fourOnTheNetwork(t)
in.Catalogue["rival-one"] = catalogue.Manifest{Module: "rival-one", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
in.Catalogue["rival-two"] = catalogue.Manifest{Module: "rival-two", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
hub := nodeOf(t, &in, "anchor")
hub.Assigned = append(hub.Assigned, "rival-one", "rival-two")
if _, err := Resolve(in, "anchor"); !IsUnresolvable(err) {
t.Fatalf("the hub is meant not to resolve: %v", err)
}
c := newComposer(in)
if got := c.namesInTheMesh()["anchor."+c.suffix()]; got != "10.77.0.1" {
t.Errorf("the hub's name is not in the mesh's names: %v", c.namesInTheMesh())
}
if got := c.onThePrivateNetwork(); !slices.Contains(got, "10.77.0.1") || len(got) != 4 {
t.Errorf("the sources from the mesh are %v, want all four machines with the hub", got)
}
if _, problems, err := Compose(in, "ace"); err != nil || len(problems) != 0 {
t.Errorf("another machine does not compose: %v %v", err, problems)
}
}
+150
View File
@@ -0,0 +1,150 @@
package compose
import (
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Rendered is everything a node's declaration is composed with (plan.go renderingFor), over an Input: the place half (Places)
// and the credential half. Nothing was allocated, made, kept or written; what a send would do is returned for the caller to do.
type Rendered struct {
Rendering catalogue.Rendering
// ForeseenPorts are the ports the mesh may assign that the Input holds none for ("<module>:<port>").
ForeseenPorts []string
// Foreseen are the stand-ins the composition used for what the Input does not hold: an own secret the next send makes
// ("<module>/<name>"), a grant's sealed copy, the certificate and the bus's user list.
Foreseen []string
// CoreBoxesWanted are the core credentials whose box the send must ask the vault for and keep.
CoreBoxesWanted []CoreBoxWanted
// Withheld are the consumers the grants leave out because their identity overflows the provision's bound (novox/hq ADR 0225),
// and Unbound those whose credential from this node is on record and whose own resolution binds them elsewhere (issue
// 274). The controller says them; a check refuses a change that adds one (ADR 0446 clause 1).
Withheld []catalogue.Overflow
Unbound []catalogue.Unbound
// Said are the lines the controller says while composing (the bus users left out for want of a credential).
Said []string
}
// Render composes the rendering of a node: the grants it must create, where it puts its ports, its own secrets, the vault's boxes,
// its certificate, the names of the mesh, the bus's user list and its membership. The refusals are the controller's, in its words;
// where more than one fault stands, the first reported may differ from the controller's, which interleaves the halves.
func Render(in Input, node string, mode Mode) (Rendered, error) {
return newComposer(in).render(node, mode)
}
// Composing is Resolve, Render and Generators of one node over one composition, which resolves each node of the mesh
// once for all three (issue 1954: three compositions resolved the mesh again each, 11.7 s for one node's plans). Each
// answers as it does alone; the first refusal ends it, in that order, with what follows it empty.
func Composing(in Input, node string, mode Mode) (Resolved, Rendered, map[string]catalogue.Generator, error) {
c := newComposer(in)
resolved, err := c.planFor(node)
if err != nil {
return Resolved{}, Rendered{}, nil, err
}
rendered, err := c.render(node, mode)
if err != nil {
return resolved, Rendered{}, nil, err
}
gens, err := c.generators()
if err != nil {
return resolved, rendered, nil, err
}
return resolved, rendered, gens, nil
}
func (c *composer) render(nodeName string, mode Mode) (Rendered, error) {
resolved, err := c.planFor(nodeName)
if err != nil {
return Rendered{}, err
}
node, err := c.in.node(nodeName)
if err != nil {
return Rendered{}, err
}
plan := resolved.Plan
var out Rendered
foreseen := func(s string) { out.Foreseen = append(out.Foreseen, s) }
grants, withheld, unbound, grantsForeseen, err := c.grantsFor(nodeName)
if err != nil {
return Rendered{}, err
}
placed, err := c.places(nodeName, resolved)
if err != nil {
return Rendered{}, err
}
r := placed.Rendering
out.ForeseenPorts = placed.ForeseenPorts
// A plan carries its contributions' machine ports, refused where the placement assigned none (issue 774).
r.Planned = mode == Planning
r.Grants, r.Withheld, r.Unbound = grants, withheld, unbound
out.Withheld, out.Unbound = withheld, unbound
for _, g := range grantsForeseen {
foreseen(g)
}
// And each module's own secrets: a module handed to the provisioner has its secrets from the vault through the client.
handing := catalogue.Rendering{Hands: r.Hands, Provisioner: r.Provisioner, Assigned: r.Assigned}
needed, foreseenOwn, err := c.ownSecretsFor(node, plan, handing, mode)
if err != nil {
return Rendered{}, err
}
r.Needed = needed
if mode == Foreseeing || mode == Planning || len(foreseenOwn) > 0 {
r.Foreseen = foreseenOwn
}
var names []string
for module, secrets := range foreseenOwn {
for name := range secrets {
names = append(names, module+"/"+name)
}
}
sort.Strings(names)
for _, n := range names {
foreseen(n)
}
// The core's own credentials as the vault sealed them, for an engine that opens the vault's box.
if node.Stated && node.Contract >= VaultBoxContract {
sealed, notPlaced, wanted := c.coreBoxesFor(node, plan.Modules, handing)
r.VaultSealed, r.NotPlaced = sealed, notPlaced
out.CoreBoxesWanted = wanted
}
// And a certificate for this machine's name inside the mesh, when anything on it asks.
for _, m := range plan.Modules {
if m.Certificate == nil {
continue
}
certificate, authority, stood, err := c.certificateFor(node)
if err != nil {
return Rendered{}, err
}
r.Certificate, r.Authority = certificate, authority
if stood {
foreseen("the certificate and the authority of " + nodeName)
}
break
}
// The bus's user list, for the machine that runs the bus: names and hashes the Input carries, never passwords.
if holdsTheBus(plan.Modules) {
r.BusUsers = c.in.BusUsers
if r.BusUsers == "" {
r.BusUsers = ForeseenBusUsers
foreseen(BusUsersStandIn)
}
if len(c.in.BusMissing) > 0 {
out.Said = append(out.Said, fmt.Sprintf("the bus's user list leaves out %d user(s) the mesh has minted no credential "+
"for: %s. Each is a user that cannot connect until one is issued", len(c.in.BusMissing), strings.Join(c.in.BusMissing, ", ")))
}
}
r.BusMembership = c.in.BusMemberships[nodeName]
out.Rendering = r
// And what the composition found on the way, which the controller writes to its log: a machine not counted as on the network.
out.Said = c.saidBeside(out.Said)
return out, nil
}
+544
View File
@@ -0,0 +1,544 @@
package compose
import (
"errors"
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/compose/resolution"
"github.com/novox/mesh-host/internal/overlay"
)
// The contracts of the node-engine's statement that gate what a machine is sent, as the controller's link package names them
// (mesh-controller internal/link/protocol.go at 6c2a8d62). An older engine parses strictly and refuses a field it does not
// know, whole.
const (
ReadinessContract = 2 // the engine reads a resource's `health` (novox/hq ADR 0240)
RootContract = 3 // the engine judges a user's declared `root` (ADR 0266)
BusyContract = 8 // the engine reads a container's `busy` (ADR 0335)
HandedContract = 11 // the engine writes a filled file on the provisioner client's word alone
VaultBoxContract = 12 // the engine opens the vault's box
)
// The provisioner client's seat and own secret (the controller's broker.NodeProvisionerSeat and ProvisionerSecret).
const (
NodeProvisionerSeat = "node-provisioner"
ProvisionerSecret = "provisioner-broker"
)
// ErrPortTaken is the controller's refusal for a machine port held twice on one machine.
var ErrPortTaken = errors.New("that port is already held on this machine")
// Placed is the place half of a rendering (plan.go renderingFor): where a machine puts what its modules need reachable, the
// seats and who holds them, the names of the mesh, the networks, accounts and the operator's export. It is a catalogue.Rendering
// with those fields set, and the ports it could not read named foreseen. Nothing was allocated.
type Placed struct {
Rendering catalogue.Rendering
// ForeseenPorts are the ports the mesh may assign that the Input holds none for, as "<module>:<port>": composed with the
// manifest's own number as the Reading mode composes them, and never invented. The controller's adapter allocates and
// composes again.
ForeseenPorts []string
}
// Places composes the place half of the rendering for a node (the controller's renderingFor in the Reading and Foreseeing modes,
// without the credentials): over an Input, allocating nothing and changing nothing in it. The refusals are the controller's,
// in its words.
func Places(in Input, node string) (Placed, error) {
c := newComposer(in)
resolved, err := c.planFor(node)
if err != nil {
return Placed{}, err
}
return c.places(node, resolved)
}
func (c *composer) places(nodeName string, resolved Resolved) (Placed, error) {
var out Placed
node, err := c.in.node(nodeName)
if err != nil {
return out, err
}
plan, settings := resolved.Plan, resolved.Settings
r := &out.Rendering
// Where this machine puts what each module needs reachable (novox/hq ADR 0038): what it was already given, and the ports
// it was given over those (ADR 0100).
given, err := c.portsFor(node, plan, settings, r, &out.ForeseenPorts)
if err != nil {
return out, err
}
// Hands: whether the engine writes a filled file on the provisioner client's word alone; engines older than a contract are
// not sent the field they would refuse.
r.Hands = takesHanded(node)
r.Provisioner = provisionerAccount(plan)
r.Assigned = node.Handed
r.ReadsHealth = node.Stated && node.Contract >= ReadinessContract
r.JudgesRoot = node.Stated && node.Contract >= RootContract
r.ReadsBusy = node.Stated && node.Contract >= BusyContract
// And who else is on the private network, which is what a rule saying "from the mesh" resolves to.
r.Mesh = c.onThePrivateNetwork()
// The private network's range, offered to a module as ${machine:mesh-range}.
r.MeshRange = c.overlayRange()
// The artifact store as this node reaches it now, with what the mesh has built.
store, err := c.artifactStoreAddress(nodeName)
if err != nil {
return out, err
}
r.ArtifactStore = store
r.Built = make(map[string]bool, len(c.in.Built))
for _, repository := range c.in.Built {
r.Built[repository] = true
}
// And every machine's name, so a container can reach one.
r.Names = c.namesInTheMesh()
// Each machine's operator account, so an ssh Host block can name the login for every node (novox/hq to-be 29).
r.Accounts = map[string]string{}
for _, n := range c.in.sorted() {
if n.Account != "" {
r.Accounts[n.Name] = n.Account
}
}
// The roster is the machines and nothing else (novox/hq ADR 0191).
r.Machines = make(map[string]string, len(r.Names))
for name, at := range r.Names {
r.Machines[name] = at
}
// And every zone a module in the mesh answers itself (novox/hq ADR 0199).
zones, err := c.zonesInTheMesh()
if err != nil {
return out, err
}
r.Zones = zones
// And who holds each replicated seat, where (novox/hq ADR 0223).
r.Holders = c.replicatedHolders()
r.Suffix, r.TunnelInterface = c.suffix(), overlay.Interface
// And, for a module that keeps them, every operator-sealed secret in the mesh (novox/hq ADR 0085, amended).
for _, m := range plan.Modules {
if m.Keeps == "" {
continue
}
r.Kept = c.in.OperatorExport
if r.Kept == nil {
r.KeptRuns = c.in.KeptRuns[node.Name]
}
break
}
// Whether this node is adopted (novox/hq ADR 0100), and which modules were taken there.
r.Adopted = node.Adopted
var taken map[string]bool
if node.Adopted {
taken = map[string]bool{}
for _, m := range node.Taken {
taken[m] = true
}
}
r.Taken = taken
// Where this node put the foundation's servers (novox/hq 04-ISSUES/102).
seats, err := c.seatsOn(node, plan.Modules, node.Adopted, taken)
if err != nil {
return out, err
}
r.Seats = seats
// Which of this machine's links face outside (novox/hq ADR 0140), and which of its networks count as its local network
// (ADR 0340).
r.OutwardLinks = node.OutwardLinks
r.LocalNetworks = resolution.CountedLocalNetworks(localNetworksOf(node))
// Where this machine reaches each mesh seat's holder, for ${seat:<seat>:reach} (novox/hq ADR 0222).
if store != "" {
r.SeatReach = map[string]string{"mesh-artifact-store": store}
}
r.Settings = settings
r.Given = given
return out, nil
}
// takesHanded says whether a machine's newest statement is of an engine that writes a filled file on the provisioner client's
// word alone.
func takesHanded(n Node) bool { return n.Stated && n.Contract >= HandedContract }
// provisionerAccount is the account the provisioner's client runs as on this machine (novox/hq ADR 0349 clause 4): the user
// the process of the module running it runs as, empty where no such module is on the machine.
func provisionerAccount(plan catalogue.Resolution) string {
for _, m := range plan.Modules {
if !runsTheProvisionerClient(m) {
continue
}
for _, r := range m.Resources {
if r["type"] == "process" {
if user, _ := r["user"].(string); user != "" && user != "root" {
return user
}
}
}
}
return ""
}
// runsTheProvisionerClient says a module runs its machine's provisioner client: it claims the seat and declares the own
// secret that account is delivered in.
func runsTheProvisionerClient(m catalogue.Manifest) bool {
_, reads := m.OwnSecrets[ProvisionerSecret]
return reads && m.ClaimsSeat(NodeProvisionerSeat)
}
// portsFor is where a node puts what each module needs reachable, in the Reading mode: the ports it was given win, then the
// ones the mesh assigned, and a port the Input holds none for is composed as the manifest says and named foreseen. One holder
// per machine port across the node's modules: settings written before this was refused are refused here.
func (c *composer) portsFor(node Node, plan catalogue.Resolution, settings catalogue.SettingsBy, r *catalogue.Rendering,
foreseen *[]string) (map[string]map[int]int, error) {
already := node.Ports
given := map[string]map[int]int{}
for _, m := range plan.Modules {
g, err := resolution.GivenPorts(m, settings[m.Module])
if err != nil {
// A given port its definition no longer publishes: the module is left out of the declaration, by name, when it is
// composed (novox/hq ADR 0163, rule 6) -- never the machine refused here for it.
continue
}
if g != nil {
given[m.Module] = g
}
}
holders := map[int]string{}
for _, m := range plan.Modules {
for _, at := range given[m.Module] {
if other, twice := holders[at]; twice && other != m.Module {
return nil, fmt.Errorf("%w: %s and %s are both given machine port %d on %s", ErrPortTaken, other, m.Module,
at, node.Name)
}
holders[at] = m.Module
}
}
ports := map[string]map[int]int{}
for _, m := range plan.Modules {
for _, l := range m.Listens {
if at, isGiven := given[m.Module][l.Port]; isGiven {
if ports[m.Module] == nil {
ports[m.Module] = map[int]int{}
}
ports[m.Module][l.Port] = at
continue
}
// Only a port the module actually publishes is the mesh's to move. Recorded either way, because this map means where
// this module's port is on this machine.
where, mayAssign := m.MachineSide(l.Port)
if mayAssign {
// Whatever was chosen last time, and nothing if there was no last time.
if at, known := already[m.Module][l.Port]; known {
where = at
} else {
*foreseen = append(*foreseen, fmt.Sprintf("%s:%d", m.Module, l.Port))
}
}
if ports[m.Module] == nil {
ports[m.Module] = map[int]int{}
}
ports[m.Module][l.Port] = where
}
}
r.Ports = ports
return given, nil
}
// onThePrivateNetwork is every node's address on the private network, sorted: the same set the names mean by it, so a rule
// saying "from the mesh" admits exactly the machines the mesh names. A node with no address is left out, not rendered as an
// empty source.
func (c *composer) onThePrivateNetwork() []string {
var out []string
for _, p := range c.onTheNetwork() {
if strings.TrimSpace(p.Place.Address) != "" {
out = append(out, p.Place.Address)
}
}
sort.Strings(out)
return out
}
// namesInTheMesh is every machine's internal name and the address behind it, for every machine on the private network, and
// the carried peers the operator has named (novox/hq issue 112).
func (c *composer) namesInTheMesh() map[string]string {
out := map[string]string{}
for _, p := range c.onTheNetwork() {
out[c.internalName(p.Name)] = p.Place.Address
}
for _, p := range c.in.CarriedPeers {
if p.Named == "" || p.EnrolledAs != "" {
continue
}
if _, taken := out[c.internalName(p.Named)]; taken {
continue // a node of the mesh owns the name; the stale statement loses
}
out[c.internalName(p.Named)] = p.Address
}
return out
}
// artifactStoreOnNetwork is the machine on this network that offers the artifact store, and the port that machine put it on.
func (c *composer) artifactStoreOnNetwork(on map[string]bool) (node, port string, found bool, err error) {
providers := map[string]catalogue.Manifest{}
for name, m := range c.in.Catalogue {
if _, offers := m.Serves[catalogue.ArtifactStoreProvision]; offers {
providers[name] = m
}
}
if len(providers) == 0 {
return "", "", false, nil
}
// In a stated order, so two machines offering it would always answer the same one.
machines := make([]string, 0, len(on))
for machine := range on {
machines = append(machines, machine)
}
sort.Strings(machines)
for _, machine := range machines {
n, err := c.in.node(machine)
if err != nil {
return "", "", false, fmt.Errorf("reading what %s is assigned: %w", machine, err)
}
for _, a := range n.Assigned {
m, offers := providers[a]
if !offers {
continue
}
serves, _, err := c.servedOnNode(machine, m, catalogue.ArtifactStoreProvision)
if err != nil {
return "", "", false, fmt.Errorf("reading where %s puts the artifact store: %w", machine, err)
}
if p, ok := serves["port"]; ok {
return machine, fmt.Sprintf("%v", p), true, nil
}
}
}
return "", "", false, nil
}
// ArtifactStoreAddress is the artifact store as node reaches it, by the rule its composition routes every kept reference
// with (artifactStoreAddress): empty when node reaches none. A node composing its own modules reads a templates archive
// there (issue 1126).
func ArtifactStoreAddress(in Input, node string) (string, error) {
return newComposer(in).artifactStoreAddress(node)
}
// artifactStoreAddress is the artifact store as `forNode` reaches it: `<node>.internal:<port>` over the private network, or
// `127.0.0.1:<port>` for the node that holds the store itself when nothing is on the network yet, and empty for any other
// (novox/hq 04-ISSUES/102).
func (c *composer) artifactStoreAddress(forNode string) (string, error) {
onNetwork, err := c.whereEveryoneIs()
if err != nil {
return "", err
}
on := map[string]bool{}
for name := range onNetwork {
on[name] = true
}
node, port, found, err := c.artifactStoreOnNetwork(on)
if err != nil {
return "", err
}
if found {
return c.internalName(node) + ":" + port, nil
}
all := map[string]bool{}
for _, n := range c.in.Nodes {
all[n.Name] = true
}
holder, port, found, err := c.artifactStoreOnNetwork(all)
if err != nil || !found || holder != forNode {
return "", err
}
return "127.0.0.1:" + port, nil
}
// zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR 0199): the zone settled
// from that node's settings, the node's private address, the port the answering listen is published on there. A node whose set
// does not compose declares nothing and is passed over; what the mesh refuses about the zones together is refused, by name.
func (c *composer) zonesInTheMesh() ([]catalogue.ZoneAt, error) {
address := map[string]string{}
for _, n := range c.in.sorted() {
if strings.TrimSpace(n.Place.Address) != "" {
address[n.Name] = n.Place.Address
}
}
var zones []catalogue.ZoneAt
var public []string
for _, n := range c.in.sorted() {
resolved, err := c.planFor(n.Name)
switch {
case unresolvable(err):
continue
case err != nil:
return nil, fmt.Errorf("the zones %s answers cannot be read: %w", n.Name, err)
}
plan := resolved.Plan
if plan.PublicDomain != "" {
public = append(public, plan.PublicDomain)
}
for _, m := range plan.Modules {
if m.Zone == nil {
continue
}
at := address[n.Name]
if at == "" {
// Not on the private network yet: nothing could reach its answerer.
continue
}
published, layers, err := c.portsGivenOn(n.Name, m)
if err != nil {
return nil, fmt.Errorf("the zone %s declares on %s cannot be read: %w", m.Module, n.Name, err)
}
z, err := resolution.ZoneOn(m, layers, published, n.Name, at)
if err != nil {
return nil, err
}
zones = append(zones, *z)
}
}
if problems := resolution.ZonesProblems(zones, c.suffix(), public); len(problems) > 0 {
return nil, fmt.Errorf("the mesh's zones cannot be forwarded:\n - %s", strings.Join(problems, "\n - "))
}
return zones, nil
}
// replicatedHolders is, for each replicated mesh seat, every machine on the private network holding it, by internal name at its
// private address (novox/hq ADR 0223): the holders on record, and only the sole claimant when there are none. A holder off
// the private network is left out.
func (c *composer) replicatedHolders() map[string]map[string]string {
var replicated []catalogue.Seat
for _, s := range catalogue.Seats() {
if s.Replicated && s.Scope == catalogue.ScopeMesh {
replicated = append(replicated, s)
}
}
if len(replicated) == 0 {
return nil
}
address := map[string]string{}
for _, p := range c.onTheNetwork() {
address[p.Name] = p.Place.Address
}
// Every module of the catalogue with the nodes it is assigned on, sorted, as the controller's Catalogued answers.
on := map[string][]string{}
for _, n := range c.in.sorted() {
for _, module := range n.Assigned {
on[module] = append(on[module], n.Name)
}
}
names := make([]string, 0, len(c.in.Catalogue))
for name := range c.in.Catalogue {
names = append(names, name)
}
sort.Strings(names)
out := map[string]map[string]string{}
for _, seat := range replicated {
var nodes []string
for _, h := range c.in.Holdings {
if hs, ok := catalogue.SeatNamed(h.Claim); ok && hs.Name == seat.Name && h.Scope == seat.Scope {
nodes = append(nodes, h.Node)
}
}
if len(nodes) == 0 {
var derived []string
for _, name := range names {
for _, cl := range c.in.Catalogue[name].Claims {
if cs, ok := catalogue.SeatNamed(cl.Name); ok && cs.Name == seat.Name && cl.At() == seat.Scope {
derived = append(derived, on[name]...)
}
}
}
if len(derived) == 1 {
nodes = derived
}
}
at := map[string]string{}
for _, n := range nodes {
if address[n] != "" {
at[c.internalName(n)] = address[n]
}
}
out[seat.Name] = at
}
return out
}
// seatsOn is where a node put the holder of each mesh-scoped seat, by seat and by the port the holder's software uses: what
// ${seat:...} answers with (novox/hq 04-ISSUES/102). Read for every module in the catalogue that claims a seat, in this node's set
// or not; a holder in this node's set wins over one that is not.
func (c *composer) seatsOn(node Node, inSet []catalogue.Manifest, adopted bool, taken map[string]bool) (map[string]map[int]int, error) {
assigned := map[string]bool{}
for _, m := range inSet {
assigned[m.Module] = true
}
names := make([]string, 0, len(c.in.Catalogue))
for name := range c.in.Catalogue {
names = append(names, name)
}
sort.Strings(names)
seats := map[string]map[int]int{}
for _, name := range names {
m := c.in.Catalogue[name]
var claims []string
for _, cl := range m.Claims {
if cl.At() == catalogue.ScopeMesh {
claims = append(claims, cl.Name)
}
}
if len(claims) == 0 {
continue
}
// Only a port the node was given or the mesh assigned, never the manifest's own number. And on an adopted node a holder
// assigned but not yet taken is the found container, so its mesh-assigned ports do not count there; a given port does.
ports, _, err := c.portsGivenOn(node.Name, m)
if err != nil {
return nil, err
}
if adopted && !taken[name] {
layers := node.Settings[m.Module]
ports = map[int]int{}
if given, err := resolution.GivenPorts(m, layers); err == nil {
ports = given
}
}
if len(ports) == 0 {
continue
}
for _, seat := range claims {
if seats[seat] == nil {
seats[seat] = map[int]int{}
}
for wanted, at := range ports {
if _, said := seats[seat][wanted]; said && !assigned[name] {
continue
}
seats[seat][wanted] = at
}
}
}
return seats, nil
}
// localNetworksOf is a machine's attached networks judged (novox/hq ADR 0340): which count as its local network, from what it
// last said of itself and the links it said face outside it. A machine that has said none judges to nothing.
func localNetworksOf(n Node) []catalogue.LocalNetwork {
if !n.NetworksSaid {
return nil
}
return resolution.JudgeLocalNetworks(n.Networks, n.OutwardLinks, overlay.Interface)
}
+329
View File
@@ -0,0 +1,329 @@
package compose
import (
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-host/internal/overlay"
)
// aWebModule publishes one port in the short form, which the mesh is free to move (MachineSide).
func aWebModule() catalogue.Manifest {
return catalogue.Manifest{Module: "web", Version: "1",
Listens: []catalogue.Listening{{Name: "http", Port: 8080, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "web", "ports": []any{"8080"},
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}}}
}
// aStoreModule is mesh-controller's sendable_integration_test.go TestConsumersAreToldTheGivenPort store (cmd/mesh-controller
// at 6c2a8d62): publishing 5432:5432, offering a database to the mesh.
func aStoreModule() catalogue.Manifest {
return catalogue.Manifest{Module: "store", Version: "1",
Provides: []catalogue.Offer{{Name: "database", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Guards: []int{5432},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"},
"image": "registry.example/pg@sha256:" + strings.Repeat("b", 64)}}}
}
func placed(t *testing.T, in Input, node string) Placed {
t.Helper()
got, err := Places(in, node)
if err != nil {
t.Fatalf("%v", err)
}
return got
}
// 1363/AC1 and AC2: a port the Input holds as allocated on the node is the one composed; with none, no number is invented: the
// manifest's own is composed and the port is named foreseen, as the Foreseeing mode does.
func TestAnAllocatedPortIsComposedAndAnUnallocatedOneIsNamedForeseenNotInvented(t *testing.T) {
mk := func() Input {
in := aMesh(t)
in.Catalogue["web"] = aWebModule()
nodeOf(t, &in, "laptop").Assigned = append(nodeOf(t, &in, "laptop").Assigned, "web")
return in
}
in := mk()
got := placed(t, in, "laptop")
if p := got.Rendering.Ports["web"][8080]; p != 8080 {
t.Errorf("a port nobody allocated is composed as %d, not as the manifest's own", p)
}
if len(got.ForeseenPorts) != 1 || got.ForeseenPorts[0] != "web:8080" {
t.Errorf("foreseen ports are %v", got.ForeseenPorts)
}
nodeOf(t, &in, "laptop").Ports = map[string]map[int]int{"web": {8080: 20123}}
got = placed(t, in, "laptop")
if p := got.Rendering.Ports["web"][8080]; p != 20123 || len(got.ForeseenPorts) != 0 {
t.Errorf("the allocated port is composed as %d (foreseen %v)", p, got.ForeseenPorts)
}
// A port the module does not publish is not the mesh's to move, so it is never foreseen.
m := aWebModule()
m.Resources = nil
in.Catalogue["web"] = m
nodeOf(t, &in, "laptop").Ports = nil
got = placed(t, in, "laptop")
if got.Rendering.Ports["web"][8080] != 8080 || len(got.ForeseenPorts) != 0 {
t.Errorf("an unpublished port: %v foreseen %v", got.Rendering.Ports, got.ForeseenPorts)
}
unchanged(t, mk, func(in Input) { placed(t, in, "laptop") })
// Only a composition for plans refuses a contribution whose port the module does not listen on (issue 774).
for _, mode := range []Mode{Reading, Foreseeing, Planning} {
if got := rendered(t, mk(), "laptop", mode).Rendering; got.Planned != (mode == Planning) {
t.Errorf("mode %d composes with Planned %v", mode, got.Planned)
}
}
}
// 1363/AC3: a port the operator gave is used, over the allocated one, and a consumer elsewhere is told it
// (mesh-controller sendable_integration_test.go TestConsumersAreToldTheGivenPort).
func TestAPortTheOperatorGaveIsUsedAndNotReallocated(t *testing.T) {
in := aMesh(t)
in.Catalogue["store"] = aStoreModule()
in.Catalogue["app"] = catalogue.Manifest{Module: "app", Version: "1", Requires: []string{"database"}}
anchor := nodeOf(t, &in, "anchor")
anchor.Assigned = append(anchor.Assigned, "store")
nodeOf(t, &in, "laptop").Assigned = append(nodeOf(t, &in, "laptop").Assigned, "app")
anchor.Ports = map[string]map[int]int{"store": {5432: 20999}}
anchor.Settings = map[string][]catalogue.Layer{"store": {{From: "anchor",
Placement: map[string]any{catalogue.PortsSetting: map[string]any{"5432": 5433}}}}}
got := placed(t, in, "anchor")
if p := got.Rendering.Ports["store"][5432]; p != 5433 {
t.Errorf("the store is on %d, not the port it was given", p)
}
if g := got.Rendering.Given["store"][5432]; g != 5433 {
t.Errorf("the given port is %d", g)
}
if len(got.ForeseenPorts) != 0 {
t.Errorf("a given port is foreseen: %v", got.ForeseenPorts)
}
resolved, err := Resolve(in, "laptop")
if err != nil {
t.Fatal(err)
}
if told := need(t, resolved.Plan, "app", "database").Serves["port"]; told != 5433 {
t.Errorf("the consumer is told the store is on %v", told)
}
// One holder per machine port across the node's modules: two given the same refused, in the controller's words.
in.Catalogue["store2"] = func() catalogue.Manifest {
m := aStoreModule()
m.Module, m.Provides = "store2", nil
m.Resources = []map[string]any{{"id": "server", "type": "container", "name": "other", "ports": []any{"5432:5432"},
"image": "registry.example/pg@sha256:" + strings.Repeat("c", 64)}}
return m
}()
anchor.Assigned = append(anchor.Assigned, "store2")
anchor.Settings["store2"] = []catalogue.Layer{{From: "anchor",
Placement: map[string]any{catalogue.PortsSetting: map[string]any{"5432": 5433}}}}
_, err = Places(in, "anchor")
if err == nil || !errors.Is(err, ErrPortTaken) || !strings.Contains(err.Error(), "store and store2 are both given machine port 5433 on anchor") &&
!strings.Contains(err.Error(), "store2 and store are both given machine port 5433 on anchor") {
t.Fatalf("two modules given one machine port: %v", err)
}
}
// 1363/AC4: the seats held and taken reach the Rendering: where each mesh-scoped seat's holder put its ports, from the node's
// given and assigned ports, never the manifest's own; and on an adopted node a holder assigned but not taken counts only by its
// given ports (novox/hq ADR 0100).
func TestSeatsHeldAndTakenReachTheRendering(t *testing.T) {
in := aMesh(t)
holder := aStoreModule()
holder.Claims = []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}}
in.Catalogue["store"] = holder
anchor := nodeOf(t, &in, "anchor")
anchor.Assigned = append(anchor.Assigned, "store")
// The shelf is read whole, in this node's set or not: laptop, running none, is told where the anchor's store is only by the
// settings and ports of its own machine, which it has none of.
anchor.Ports = map[string]map[int]int{"store": {5432: 20999}}
got := placed(t, in, "anchor")
if at := got.Rendering.Seats["mesh-store"][5432]; at != 20999 {
t.Errorf("the seat's holder is told to be on %d", at)
}
if len(placed(t, in, "laptop").Rendering.Seats) != 0 {
t.Errorf("laptop is told of a holder it has no port for: %v", placed(t, in, "laptop").Rendering.Seats)
}
// Adopted, and the holder assigned but not taken: the found container's ports, not the mesh's assignment.
anchor.Adopted = true
if len(placed(t, in, "anchor").Rendering.Seats) != 0 {
t.Errorf("an untaken holder on an adopted node counts by the mesh's assignment: %v", placed(t, in, "anchor").Rendering.Seats)
}
anchor.Taken = []string{"store"}
got = placed(t, in, "anchor")
if at := got.Rendering.Seats["mesh-store"][5432]; at != 20999 || !got.Rendering.Adopted || !got.Rendering.Taken["store"] {
t.Errorf("a taken holder: seats %v adopted %v taken %v", got.Rendering.Seats, got.Rendering.Adopted, got.Rendering.Taken)
}
}
// 1363/AC4: local networks and outward links (mesh-controller local_network_test.go TestNodeShowsWhichAttachedNetworksCountAndWhy,
// the values): a private network on a link facing outside counts, a public subnet and the private network's own link never do; a
// machine that said none judges to nothing.
func TestLocalNetworksAndOutwardLinksReachTheRendering(t *testing.T) {
in := aMesh(t)
laptop := nodeOf(t, &in, "laptop")
laptop.OutwardLinks = []string{"eth0", "eth1"}
laptop.Networks = []catalogue.AttachedNetwork{
{Link: "eth0", Network: "192.168.1.10/24"}, {Link: "eth1", Network: "203.0.113.7/24"}, {Link: overlay.Interface, Network: "10.10.0.4/24"}}
laptop.NetworksSaid = true
got := placed(t, in, "laptop").Rendering
if len(got.LocalNetworks) != 1 || got.LocalNetworks[0] != "192.168.1.0/24" {
t.Errorf("the local networks are %v", got.LocalNetworks)
}
if len(got.OutwardLinks) != 2 || got.OutwardLinks[0] != "eth0" {
t.Errorf("the outward links are %v", got.OutwardLinks)
}
laptop.NetworksSaid = false
if got := placed(t, in, "laptop").Rendering; len(got.LocalNetworks) != 0 {
t.Errorf("a machine that never reported has local networks %v", got.LocalNetworks)
}
}
// 1363/AC4: accounts, the names of the mesh, the mesh range and the operator's export reach the Rendering.
func TestAccountsNamesAndTheOperatorsExportReachTheRendering(t *testing.T) {
in := aMesh(t)
nodeOf(t, &in, "anchor").Account = "jochens"
in.OperatorExport = &catalogue.KeptExport{}
in.OverlayCIDR = "10.99.0.0/16"
got := placed(t, in, "anchor").Rendering
if got.Accounts["anchor"] != "jochens" || len(got.Accounts) != 1 {
t.Errorf("accounts are %v", got.Accounts)
}
if got.Names["anchor.internal"] != "10.77.0.1" || got.Names["laptop.internal"] != "10.77.0.2" || len(got.Names) != 2 {
t.Errorf("names are %v", got.Names)
}
if len(got.Machines) != 2 || got.Machines["laptop.internal"] != "10.77.0.2" {
t.Errorf("machines are %v", got.Machines)
}
if len(got.Mesh) != 2 || got.Mesh[0] != "10.77.0.1" || got.Mesh[1] != "10.77.0.2" {
t.Errorf("the mesh's addresses are %v", got.Mesh)
}
if got.MeshRange != "10.99.0.0/16" || got.Suffix != "internal" || got.TunnelInterface != overlay.Interface {
t.Errorf("range %q suffix %q tunnel %q", got.MeshRange, got.Suffix, got.TunnelInterface)
}
if got.Kept != nil {
t.Error("the operator's export reaches a node where nothing keeps it")
}
keeper := catalogue.Manifest{Module: "keeper", Version: "1", Keeps: "/var/lib/keeper/export.json"}
in.Catalogue["keeper"] = keeper
nodeOf(t, &in, "anchor").Assigned = append(nodeOf(t, &in, "anchor").Assigned, "keeper")
if got := placed(t, in, "anchor").Rendering; got.Kept != in.OperatorExport {
t.Errorf("a module that keeps the export is not given it: %v", got.Kept)
}
// And the names of a machine that left the network leave with it.
nodeOf(t, &in, "laptop").Assigned = nil
if got := placed(t, in, "anchor").Rendering; len(got.Names) != 1 || got.Names["laptop.internal"] != "" {
t.Errorf("a machine off the network is named: %v", got.Names)
}
}
// The names the operator gave to peers of an adopted tunnel answer until they enrol (novox/hq issue 112); a node of the mesh owns
// its name.
func TestACarriedPeerIsNamedUntilItEnrols(t *testing.T) {
in := aMesh(t)
in.CarriedPeers = []CarriedPeer{{PublicKey: "K1", Address: "192.0.2.5", Named: "printer"},
{PublicKey: "K2", Address: "192.0.2.6", Named: "laptop"}, {PublicKey: "K3", Address: "192.0.2.7", Named: "late", EnrolledAs: "late"}}
got := placed(t, in, "anchor").Rendering
if got.Names["printer.internal"] != "192.0.2.5" || got.Names["laptop.internal"] != "10.77.0.2" || got.Names["late.internal"] != "" {
t.Errorf("names are %v", got.Names)
}
}
// A node-engine older than a contract is not sent the field it would refuse (novox/hq ADR 0240 and after).
func TestTheNodeEnginesContractGatesWhatItIsSent(t *testing.T) {
in := aMesh(t)
laptop := nodeOf(t, &in, "laptop")
if r := placed(t, in, "laptop").Rendering; r.ReadsHealth || r.JudgesRoot || r.ReadsBusy || r.Hands {
t.Errorf("an engine that never stated anything reads %+v", r)
}
laptop.Stated, laptop.Contract = true, ReadinessContract
if r := placed(t, in, "laptop").Rendering; !r.ReadsHealth || r.JudgesRoot || r.ReadsBusy || r.Hands {
t.Errorf("an engine at the readiness contract reads %+v", r)
}
laptop.Contract = HandedContract
if r := placed(t, in, "laptop").Rendering; !r.ReadsHealth || !r.JudgesRoot || !r.ReadsBusy || !r.Hands {
t.Errorf("an engine at the handed contract reads %+v", r)
}
}
// The artifact store as a node reaches it: over the private network by its holder's name and the port it put it on, or on loopback
// for the holder alone while nothing is on the network (novox/hq 04-ISSUES/102).
func TestTheArtifactStoreIsReachedOverTheNetworkOrOnLoopbackForItsHolder(t *testing.T) {
in := aMesh(t)
in.Catalogue["artifacts"] = catalogue.Manifest{Module: "artifacts", Version: "1",
Serves: map[string]map[string]any{catalogue.ArtifactStoreProvision: {"port": 5100}}}
anchor := nodeOf(t, &in, "anchor")
anchor.Assigned = append(anchor.Assigned, "artifacts")
if got := placed(t, in, "laptop").Rendering.ArtifactStore; got != "anchor.internal:5100" {
t.Errorf("laptop reaches the store at %q", got)
}
nodeOf(t, &in, "laptop").Assigned = nil
anchor.Assigned = []string{"artifacts"} // off the network: the holder alone reaches it
if got := placed(t, in, "anchor").Rendering.ArtifactStore; got != "127.0.0.1:5100" {
t.Errorf("the holder reaches the store at %q", got)
}
if got := placed(t, in, "laptop").Rendering.ArtifactStore; got != "" {
t.Errorf("a machine off the network reaches the store at %q", got)
}
in.Built = []string{"mesh-sdk"}
if got := placed(t, in, "anchor").Rendering.Built; !got["mesh-sdk"] {
t.Errorf("built is %v", got)
}
}
// A zone a module declares is placed where the mesh put it: settled from that node's settings, at the node's private address, on
// the port the answering listen is published on there (novox/hq ADR 0199); the same zone declared twice is refused, by name.
func TestAZoneIsPlacedWhereTheMeshPutItAndTheSameZoneTwiceIsRefused(t *testing.T) {
in := aMesh(t)
lab := catalogue.Manifest{Module: "lab", Version: "1", Zone: &catalogue.Zone{Name: "${setting:zone}", Listen: "dns"},
Listens: []catalogue.Listening{{Name: "dns", Port: 53, Protocol: "udp", From: catalogue.FromMesh}}}
in.Catalogue["lab"] = lab
anchor := nodeOf(t, &in, "anchor")
anchor.Assigned = append(anchor.Assigned, "lab")
anchor.Settings = map[string][]catalogue.Layer{"lab": {{From: "anchor", Values: map[string]any{"zone": "Lab.Example."}}}}
anchor.Ports = map[string]map[int]int{"lab": {53: 5353}}
zones := placed(t, in, "laptop").Rendering.Zones
if len(zones) != 1 || zones[0] != (catalogue.ZoneAt{Zone: "lab.example", Node: "anchor", Module: "lab", Address: "10.77.0.1", Port: 5353}) {
t.Fatalf("zones are %+v", zones)
}
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "lab")
laptop.Settings = map[string][]catalogue.Layer{"lab": {{From: "laptop", Values: map[string]any{"zone": "lab.example"}}}}
if _, err := Places(in, "laptop"); err == nil || !strings.Contains(err.Error(), "the mesh's zones cannot be forwarded") {
t.Fatalf("one zone declared twice: %v", err)
}
}
// Every holder of a replicated seat is listed by internal name at its private address (novox/hq ADR 0223): the holders on record,
// and only the sole claimant when there are none; a holder off the private network is left out.
func TestEveryHolderOfAReplicatedSeatIsListedOnTheNetwork(t *testing.T) {
in := aMesh(t)
resolver := catalogue.Manifest{Module: "resolver", Version: "1",
Provides: []catalogue.Offer{{Name: "wildcard-resolution", Scope: catalogue.ScopeMesh}},
Claims: []catalogue.Claim{{Name: "mesh-dns-resolver", Scope: catalogue.ScopeMesh}}}
in.Catalogue["resolver"] = resolver
anchor, laptop := nodeOf(t, &in, "anchor"), nodeOf(t, &in, "laptop")
anchor.Assigned = append(anchor.Assigned, "resolver")
// One claimant and nothing on record: the sole claimant is listed.
if got := placed(t, in, "laptop").Rendering.Holders["mesh-dns-resolver"]; len(got) != 1 || got["anchor.internal"] != "10.77.0.1" {
t.Errorf("the sole claimant: %v", got)
}
// Both on record: both are listed (two claimants with nothing on record would be refused at resolution).
laptop.Assigned = append(laptop.Assigned, "resolver")
in.Holdings = []catalogue.Held{
{Claim: "mesh-dns-resolver", Scope: catalogue.ScopeMesh, Node: "anchor", Module: "resolver"},
{Claim: "mesh-dns-resolver", Scope: catalogue.ScopeMesh, Node: "laptop", Module: "resolver"}}
got := placed(t, in, "laptop").Rendering.Holders["mesh-dns-resolver"]
if len(got) != 2 || got["anchor.internal"] != "10.77.0.1" || got["laptop.internal"] != "10.77.0.2" {
t.Errorf("both holders on record: %v", got)
}
// A holder off the private network is left out: a resolver named at an address nothing answers waits out its timeout.
in.Nodes = append(in.Nodes, Node{Name: "ace", Place: Place{Address: "10.77.0.3"}, Assigned: []string{"resolver"}})
in.Holdings = append(in.Holdings, catalogue.Held{Claim: "mesh-dns-resolver", Scope: catalogue.ScopeMesh, Node: "ace", Module: "resolver"})
if got := placed(t, in, "laptop").Rendering.Holders["mesh-dns-resolver"]; len(got) != 2 || got["ace.internal"] != "" {
t.Errorf("a holder off the network is listed: %v", got)
}
}
@@ -0,0 +1,10 @@
package resolution
func indexOf(out []map[string]any, id string) int {
for i, r := range out {
if r["id"] == id {
return i
}
}
return -1
}
@@ -0,0 +1,80 @@
package resolution
import (
"strings"
"testing"
mcat "git.novox.be/novox/mesh-sdk/go/catalogue"
)
// The fault 04-ISSUES/036 records: a media stack is several modules — a library server, the
// acquisition managers, a download client — that must share directories on one machine. Written
// as owned `directory` resources, two of them on one node are refused as rivalrous owners, which
// is right in general and wrong here: sharing those directories is the whole point of the stack.
//
// Declared as accesses to an operator-owned path (novox/hq ADR 0051), they co-resolve. An access
// is not a resource and never enters the duplicate-owner map, so this is the exact case the
// resolver refuses above when the same path is owned — and does not, when it is merely accessed.
func TestTwoModulesAccessingOnePathCoResolve(t *testing.T) {
a := mod("a", nil, nil, nil)
a.Accesses = []mcat.Access{{Path: "/services/media/downloads", Mode: mcat.AccessReadWrite}}
b := mod("b", nil, nil, nil)
b.Accesses = []mcat.Access{{Path: "/services/media/downloads", Mode: mcat.AccessRead}}
got, err := Resolve(shelf(a, b), []string{"a", "b"}, workstation(), mcat.World{})
if err != nil {
t.Fatalf("two modules accessing one operator-owned path were refused: %v", err)
}
// And each accessing module's grant reaches the machine as its own `access` resource, so the
// host can find the path present or refuse — both naming the one operator-owned path.
var accesses int
for _, r := range mustDeclare(t, got) {
if r["type"] == "access" && r["path"] == "/services/media/downloads" {
accesses++
}
}
if accesses != 2 {
t.Errorf("expected both modules' accesses in the declaration, got %d", accesses)
}
}
// The mirror of the case above, so the sharing vocabulary does not weaken the rule it sits beside:
// two modules OWNING one path is still refused, exactly as before accesses existed.
func TestTwoModulesOwningOnePathAreStillRefused(t *testing.T) {
a := mod("a", nil, nil, nil)
a.Resources = []map[string]any{
{"id": "lib", "type": "directory", "path": "/services/media/movies", "mode": "0755"}}
b := mod("b", nil, nil, nil)
b.Resources = []map[string]any{
{"id": "lib", "type": "directory", "path": "/services/media/movies", "mode": "0755"}}
_, err := Resolve(shelf(a, b), []string{"a", "b"}, workstation(), mcat.World{})
if err == nil {
t.Fatal("two modules owning the same directory were both assigned")
}
if !strings.Contains(err.Error(), "/services/media/movies") {
t.Errorf("the refusal does not name the path: %v", err)
}
}
// Shared data is the operator's, owned by no module (novox/hq ADR 0051). A module owning the path
// another module was told to expect the operator to provide would create and chown it — and the
// two intentions cannot both hold, so it is refused by name. This is what keeps the distinction
// machine-checkable rather than a convention nobody enforces.
func TestAModuleMayNotOwnWhatAnotherAccesses(t *testing.T) {
owns := mod("owns", nil, nil, nil)
owns.Resources = []map[string]any{
{"id": "lib", "type": "directory", "path": "/services/media/movies", "mode": "0755"}}
uses := mod("uses", nil, nil, nil)
uses.Accesses = []mcat.Access{{Path: "/services/media/movies", Mode: mcat.AccessRead}}
_, err := Resolve(shelf(owns, uses), []string{"owns", "uses"}, workstation(), mcat.World{})
if err == nil {
t.Fatal("a module owning a path another accesses was allowed")
}
if !strings.Contains(err.Error(), "/services/media/movies") ||
!strings.Contains(err.Error(), "the operator's") {
t.Errorf("the refusal does not explain the ownership conflict: %v", err)
}
}
+177
View File
@@ -0,0 +1,177 @@
package resolution
import (
"fmt"
"sort"
"strconv"
"strings"
mcat "git.novox.be/novox/mesh-sdk/go/catalogue"
)
// GuardID, GuardUnitID and GuardRunningID are the guard's resource identities. The installer
// raises the same three on an adopted genesis, so the first push finds them already there.
func GuardID() string { return mcat.AdoptionPrefix + "guard" }
func GuardUnitID() string { return mcat.AdoptionPrefix + "guard-unit" }
func GuardRunningID() string { return mcat.AdoptionPrefix + "guard-running" }
// GuardPackageID is the tool that loads the guard, declared first: a node joining adopted has
// no filter module and may have no nft at all, and a table nothing can load guards nothing.
func GuardPackageID() string { return mcat.AdoptionPrefix + "guard-package" }
// OpeningID is an opening's resource identity: its protocol, port and path say what it is.
func OpeningID(protocol string, port int, path string) string {
return fmt.Sprintf("%sopening-%s-%d-%s", mcat.AdoptionPrefix, protocol, port, path)
}
// Openings are what the mesh needs reachable on an adopted node, from the same inputs as the
// filter it would load were the node converged, each from where that filter would admit it.
//
// `rules` is Filtering's answer — every module's listens, the hub's port, the per-node exposure —
// and `foundation` is the ports the mesh itself needs, from everywhere. A rule for this machine
// only opens nothing. `published` maps a machine port a container publishes to the container's
// port: a published port is forwarded, not received, so its opening names the forwarded path and
// the port the packet is forwarded to.
func Openings(rules []mcat.Rule, foundation []int, published map[string]map[int]int) []map[string]any {
type key struct {
protocol string
port int
}
from := map[key]string{}
var order []key
widen := func(k key, f string) {
was, seen := from[k]
if !seen {
order = append(order, k)
}
if !seen || was != mcat.OpeningFromEverywhere {
from[k] = f
}
}
for _, rule := range rules {
switch rule.From {
case mcat.FromEverywhere:
widen(key{rule.Protocol, rule.Port}, mcat.OpeningFromEverywhere)
case mcat.FromMesh:
widen(key{rule.Protocol, rule.Port}, mcat.OpeningFromMesh)
case mcat.FromLocalNetwork:
// The private network's part only (novox/hq ADR 0340). An adopted node's found firewall
// stays, and it is what admits the machine's local network; an opening has no word for a
// local network, and widening it to everywhere would open the port to the internet.
widen(key{rule.Protocol, rule.Port}, mcat.OpeningFromMesh)
}
}
for _, port := range foundation {
widen(key{"tcp", port}, mcat.OpeningFromEverywhere)
}
sort.Slice(order, func(a, b int) bool {
if order[a].port != order[b].port {
return order[a].port < order[b].port
}
return order[a].protocol < order[b].protocol
})
out := make([]map[string]any, 0, len(order))
for _, k := range order {
opening := map[string]any{"type": "opening", "port": k.port, "protocol": k.protocol,
"from": from[k]}
if to, forwarded := published[k.protocol][k.port]; forwarded {
opening["id"] = OpeningID(k.protocol, k.port, mcat.PathForwarded)
opening["path"] = mcat.PathForwarded
opening["to"] = to
} else {
opening["id"] = OpeningID(k.protocol, k.port, mcat.PathIncoming)
opening["path"] = mcat.PathIncoming
}
out = append(out, opening)
}
return out
}
// AsGuard renders the mesh's refusal-only table for the given machine ports.
//
// It passes everything by default and holds nothing but a refusal, so it cannot close anything
// the machine serves; and it is the mesh's own table, so the found firewall reloading does not
// touch it. It refuses only packets addressed to this machine, and the ports except from the
// machine itself — its loopback and the container runtime's own networks — and from the private
// network, known by the interface a packet arrives on and never by its source address. At
// prerouting, ahead of the runtime's destination translation, so it matches the port the packet
// was sent to; in the inet family, so both address families.
//
// **The machine's own interfaces are named, where the derived filter names address ranges.** The
// filter accepts the container runtime's networks by CIDR; this excludes its bridges by name — lo,
// docker0, the br-* a compose network gets, and the mesh's own mesh0. A runtime whose bridge is
// named anything else (a podman or libvirt bridge, or a docker network created with a fixed name)
// would have its containers' traffic to a guarded port refused, which reads as the port being
// down. Names rather than addresses is deliberate: a source address can be claimed by whoever
// sends the packet, and this table exists to refuse what the found firewall never sees. Widening
// it means adding names here and in the installer's copy together, which the golden test holds to
// one text.
//
// The same text the installer raises on an adopted genesis; a test holds both to it.
func AsGuard(ports []int) string {
sorted := append([]int{}, ports...)
sort.Ints(sorted)
listed := make([]string, len(sorted))
for i, p := range sorted {
listed[i] = strconv.Itoa(p)
}
var b strings.Builder
b.WriteString("table inet mesh_guard {}\n")
b.WriteString("delete table inet mesh_guard\n")
b.WriteString("table inet mesh_guard {\n")
b.WriteString("\tchain prerouting {\n")
b.WriteString("\t\ttype filter hook prerouting priority raw; policy accept;\n")
// Only packets addressed to this machine: traffic it routes for others — a predecessor's hub,
// say — is never the guard's business (novox/hq ADR 0103).
fmt.Fprintf(&b, "\t\tfib daddr type local iifname != \"lo\" iifname != \"docker0\" "+
"iifname != \"br-*\" iifname != \"mesh0\" tcp dport { %s } drop\n",
strings.Join(listed, ", "))
b.WriteString("\t}\n")
b.WriteString("}\n")
return b.String()
}
// GuardUnitText is the unit that loads the guard. Stopping it deletes only its own table: never
// a flush, which would take the container runtime's rules and the found firewall with it.
func GuardUnitText() string {
return "[Unit]\n" +
"Description=The mesh's guard: refuses its own ports from outside (novox/hq ADR 0100)\n" +
// Early, before the network is up, and without the default dependencies that would
// order it after the network; stopped at shutdown like any unit.
"DefaultDependencies=no\n" +
"Wants=network-pre.target\n" +
"Before=network-pre.target shutdown.target\n" +
"Conflicts=shutdown.target\n" +
"\n" +
"[Service]\n" +
"Type=oneshot\n" +
"RemainAfterExit=yes\n" +
"ExecStart=nft -f " + mcat.GuardPath + "\n" +
"ExecReload=nft -f " + mcat.GuardPath + "\n" +
"ExecStop=nft delete table inet mesh_guard\n" +
"\n" +
"[Install]\n" +
"WantedBy=multi-user.target\n"
}
// GuardResources are the guard as four resources of the existing kinds: the tool that loads it,
// the table, the unit, and the unit running — reloaded when the table changes, so the new table
// replaces the old in one `nft -f` through the unit's ExecReload with no moment unguarded, and
// restarted only when the unit itself changes. Nothing when there is nothing to guard: an empty
// set is not a table nft loads.
func GuardResources(ports []int) []map[string]any {
if len(ports) == 0 {
return nil
}
return []map[string]any{
{"id": GuardPackageID(), "type": "package", "package": mcat.GuardPackage},
{"id": GuardID(), "type": "file", "path": mcat.GuardPath, "content": AsGuard(ports),
"mode": "0644"},
{"id": GuardUnitID(), "type": "file", "path": mcat.GuardUnitPath, "content": GuardUnitText(),
"mode": "0644"},
{"id": GuardRunningID(), "type": "service", "unit": mcat.GuardUnit, "state": "running",
"boot": "enabled", "restart-on": []any{GuardUnitID()}, "reload-on": []any{GuardID()}},
}
}
@@ -0,0 +1,629 @@
package resolution
import (
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
mcat "git.novox.be/novox/mesh-sdk/go/catalogue"
)
// hub is the private network's generator on the hub: it opens the hub's port from anywhere.
type hub struct{}
func (hub) Resources(string) ([]map[string]any, bool, error) {
return []map[string]any{{"id": "config", "type": "file", "path": "/etc/wireguard/mesh0.conf",
"content": "[Interface]\n"}}, true, nil
}
func (hub) Listens(string) ([]mcat.Listening, error) {
return []mcat.Listening{{Port: 51820, Protocol: "udp", From: mcat.FromEverywhere}}, nil
}
// anAdoptedAnchor is the control-node's set: the store, the bus, the registry, the private network,
// a served module and the filter module.
func anAdoptedAnchor() mcat.Resolution {
return mcat.Resolution{Node: "anchor", Modules: []mcat.Manifest{
{Module: "network", Computed: "overlay"},
{Module: "postgres", Guards: []int{5432},
Listens: []mcat.Listening{{Port: 5432, From: mcat.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}}}},
{Module: "lavinmq", Guards: []int{15672},
Listens: []mcat.Listening{{Port: 5671, From: mcat.FromMesh}, {Port: 5672, From: mcat.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}}}},
{Module: "distribution",
Listens: []mcat.Listening{{Port: 5000, From: mcat.FromMesh}},
Resources: []map[string]any{{"id": "store", "type": "container", "name": "registry",
"ports": []any{"5000"}}}},
{Module: "hello-web",
Listens: []mcat.Listening{{Port: 8080, From: mcat.FromEverywhere}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "hello-web",
"ports": []any{"8080"}}}},
{Module: "helper", Listens: []mcat.Listening{{Port: 9000, From: mcat.FromMachine}}},
{Module: "nftables", Filtering: &mcat.Filtering{Into: "/etc/nftables.conf"},
Resources: []map[string]any{{"id": "load", "type": "service", "unit": "mesh-filter.service",
"state": "running", "restart-on": []any{"filtering"}}}},
}}
}
func anchorRendering(adopted bool) mcat.Rendering {
return mcat.Rendering{
Generators: map[string]mcat.Generator{"overlay": hub{}},
Ports: map[string]map[int]int{"distribution": {5000: 5000}, "hello-web": {8080: 20001}},
Settings: mcat.SettingsBy{"distribution": {{From: "node anchor",
Placement: map[string]any{mcat.ExposeSetting: map[string]any{"5000": mcat.FromEverywhere}}}}},
Mesh: []string{"10.42.0.1"},
Foundation: []int{5671},
// What the machine reported faces outside, which every rule in the filter is written
// around (novox/hq ADR 0140).
OutwardLinks: []string{"eth0"},
TunnelInterface: "mesh0",
Adopted: adopted,
// Genesis takes the foundation's modules.
Taken: map[string]bool{"postgres": true, "lavinmq": true},
}
}
func byID(resources []map[string]any) map[string]map[string]any {
out := map[string]map[string]any{}
for _, r := range resources {
out[r["id"].(string)] = r
}
return out
}
func TestAnAdoptedNodeIsDeclaredOpeningsFromTheSameInputsAsTheFilter(t *testing.T) {
composed, err := Compose(anAdoptedAnchor(), anchorRendering(true))
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
want := map[string]map[string]any{
// The hub's port, from anywhere, received.
"adoption.opening-udp-51820-incoming": {"port": 51820, "protocol": "udp",
"from": "everywhere", "path": "incoming"},
// The store's port from the private network only, and forwarded: a container publishes it.
"adoption.opening-tcp-5432-forwarded": {"port": 5432, "protocol": "tcp", "from": "mesh",
"path": "forwarded", "to": 5432},
// The bus from anywhere: a node enrols over it before it has a private address.
"adoption.opening-tcp-5671-forwarded": {"port": 5671, "protocol": "tcp",
"from": "everywhere", "path": "forwarded", "to": 5671},
"adoption.opening-tcp-5672-forwarded": {"port": 5672, "protocol": "tcp", "from": "mesh",
"path": "forwarded", "to": 5672},
// The registry from anywhere, by its node's exposure setting.
"adoption.opening-tcp-5000-forwarded": {"port": 5000, "protocol": "tcp",
"from": "everywhere", "path": "forwarded", "to": 5000},
// A published port names the machine port and the container port it is forwarded to.
"adoption.opening-tcp-20001-forwarded": {"port": 20001, "protocol": "tcp",
"from": "everywhere", "path": "forwarded", "to": 8080},
}
for id, fields := range want {
opening, ok := got[id]
if !ok {
t.Errorf("no %s among %v", id, keys(got))
continue
}
if opening["type"] != "opening" {
t.Errorf("%s is a %v", id, opening["type"])
}
for k, v := range fields {
if opening[k] != v {
t.Errorf("%s: %s is %v, want %v", id, k, opening[k], v)
}
}
}
for id := range got {
if strings.HasPrefix(id, "adoption.opening-") && want[id] == nil {
t.Errorf("an opening nothing asked for: %s", id)
}
}
// A port for this machine only opens nothing, and the management port is not opened at all.
for id := range got {
if strings.Contains(id, "-9000-") || strings.Contains(id, "-15672-") {
t.Errorf("%s is opened", id)
}
}
// And openings come first, in the order the machine applies them.
if !strings.HasPrefix(composed.Resources[0]["id"].(string), "adoption.opening-") {
t.Errorf("openings are not first: %v", composed.Resources[0]["id"])
}
}
func TestAnAdoptedNodeLoadsNoFilterOfTheMeshs(t *testing.T) {
composed, err := Compose(anAdoptedAnchor(), anchorRendering(true))
if err != nil {
t.Fatal(err)
}
for _, r := range composed.Resources {
if r["path"] == "/etc/nftables.conf" || strings.HasPrefix(r["id"].(string), "nftables.") {
t.Fatalf("an adopted node is declared the filter module's %v", r["id"])
}
if content, _ := r["content"].(string); strings.Contains(content, "policy drop") {
t.Fatalf("an adopted node is declared a table that drops by default: %v", r["id"])
}
// Nothing but refusals: the only accept in the guard is its policy.
if content, _ := r["content"].(string); r["id"] == GuardID() &&
strings.Count(content, "accept") != 1 {
t.Fatalf("the guard holds an accept:\n%s", content)
}
}
got := byID(composed.Resources)
guard := got[GuardID()]
if guard == nil || got[GuardUnitID()] == nil || got[GuardRunningID()] == nil {
t.Fatalf("no guard: %v", keys(got))
}
if guard["content"] != AsGuard([]int{5432, 5672, 15672}) {
t.Fatalf("the guard does not guard the store, the broker and its management port:\n%s",
guard["content"])
}
// The tool that loads it comes first, and the table after it: a node joining adopted has no
// filter module and may have no nft.
pkg, table := -1, -1
for i, r := range composed.Resources {
switch r["id"] {
case GuardPackageID():
pkg = i
if r["type"] != "package" || r["package"] != "nftables" {
t.Fatalf("the guard's package is %v", r)
}
case GuardID():
table = i
}
}
if pkg < 0 || pkg > table {
t.Fatalf("nftables is not declared before the guard's table (%d, %d)", pkg, table)
}
// A changed table is reloaded — one `nft -f`, atomic — never restarted, which would delete the
// table and leave the ports unguarded until it is loaded again. Only a changed unit restarts.
if !reflect.DeepEqual(got[GuardRunningID()]["reload-on"], []any{GuardID()}) ||
!reflect.DeepEqual(got[GuardRunningID()]["restart-on"], []any{GuardUnitID()}) {
t.Fatalf("the guard is not reloaded on its table and restarted on its unit: %v",
got[GuardRunningID()])
}
// Nothing of the mesh's own is anybody's to hold.
for id, module := range composed.Owner {
if strings.HasPrefix(id, mcat.AdoptionPrefix) {
t.Fatalf("%s is owned by %s", id, module)
}
}
}
func TestAConvergedNodeIsDeclaredItsFilterAndNoOpenings(t *testing.T) {
composed, err := Compose(anAdoptedAnchor(), anchorRendering(false))
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
if got["nftables.filtering"] == nil || got["nftables.load"] == nil {
t.Fatalf("a converged node lost its filter: %v", keys(got))
}
for id := range got {
if strings.HasPrefix(id, mcat.AdoptionPrefix) {
t.Fatalf("a converged node is declared %s", id)
}
}
plain, err := Declaration(anAdoptedAnchor(), anchorRendering(false))
if err != nil {
t.Fatal(err)
}
a, _ := json.Marshal(plain)
b, _ := json.Marshal(composed.Resources)
if string(a) != string(b) {
t.Fatal("Compose and Declaration disagree on a converged node")
}
}
// The table the installer raises and the controller declares, character for character.
func TestTheGuardIsExactlyThisTable(t *testing.T) {
const golden = `table inet mesh_guard {}
delete table inet mesh_guard
table inet mesh_guard {
chain prerouting {
type filter hook prerouting priority raw; policy accept;
fib daddr type local iifname != "lo" iifname != "docker0" iifname != "br-*" iifname != "mesh0" tcp dport { 5432, 15672 } drop
}
}
`
if got := AsGuard([]int{15672, 5432}); got != golden {
t.Fatalf("the guard changed:\n%s", got)
}
const unit = `[Unit]
Description=The mesh's guard: refuses its own ports from outside (novox/hq ADR 0100)
DefaultDependencies=no
Wants=network-pre.target
Before=network-pre.target shutdown.target
Conflicts=shutdown.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=nft -f /etc/mesh/guard.nft
ExecReload=nft -f /etc/mesh/guard.nft
ExecStop=nft delete table inet mesh_guard
[Install]
WantedBy=multi-user.target
`
if got := GuardUnitText(); got != unit {
t.Fatalf("the guard's unit changed:\n%s", got)
}
if GuardResources(nil) != nil {
t.Fatal("a guard with nothing to guard is an empty set nft refuses to load")
}
}
func keys[V any](m map[string]V) []string {
return sortedKeys(m)
}
// novox/hq ADR 0100: the foundation's ports are the node's. Given 5433 for the store, every place
// that uses the port reads it from there: the container, the filter, the openings, the guard.
func TestAGivenPortIsUsedEverywhereThePortIs(t *testing.T) {
given := map[string]map[int]int{"postgres": {5432: 5433}, "lavinmq": {15672: 15673}}
for _, adopted := range []bool{true, false} {
with := anchorRendering(adopted)
with.Given = given
with.Ports["postgres"] = map[int]int{5432: 5433}
composed, err := Compose(anAdoptedAnchor(), with)
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
if ports := got["postgres.server"]["ports"]; !reflect.DeepEqual(ports, []any{"5433:5432"}) {
t.Fatalf("the store's container publishes %v", ports)
}
if ports := got["lavinmq.server"]["ports"]; !reflect.DeepEqual(ports,
[]any{"5671:5671", "5672:5672", "127.0.0.1:15673:15672"}) {
t.Fatalf("the broker's container publishes %v", ports)
}
if !adopted {
filter, _ := got["nftables.filtering"]["content"].(string)
if !strings.Contains(filter, "tcp dport 5433 accept") || strings.Contains(filter, "5432") {
t.Fatalf("the filter does not use the given port:\n%s", filter)
}
continue
}
if o := got["adoption.opening-tcp-5433-forwarded"]; o == nil || o["to"] != 5432 {
t.Fatalf("no opening for the given port: %v", keys(got))
}
if guard := got[GuardID()]["content"]; guard != AsGuard([]int{5433, 5672, 15673}) {
t.Fatalf("the guard does not guard the given ports:\n%s", guard)
}
}
}
// novox/hq ADR 0103: the guard is derived, and from taken modules only — every machine port a taken
// module publishes that the filter admits from the private network only, and the ports its
// manifest guards. A module assigned but not taken is not guarded: its port may still be the
// predecessor's.
func TestTheGuardIsDerivedFromTakenModulesOnly(t *testing.T) {
guardOf := func(with mcat.Rendering) string {
t.Helper()
composed, err := Compose(anAdoptedAnchor(), with)
if err != nil {
t.Fatal(err)
}
content, _ := byID(composed.Resources)[GuardID()]["content"].(string)
return content
}
// The broker taken, the store not: the broker's plain port follows from its listens (from
// the mesh, published), its management port from its manifest; the store is not guarded, and
// neither is the bus, which the mesh needs from everywhere.
with := anchorRendering(true)
with.Taken = map[string]bool{"lavinmq": true}
if got := guardOf(with); got != AsGuard([]int{5672, 15672}) {
t.Fatalf("the guard is not the taken broker's ports:\n%s", got)
}
// A taken module publishing a port admitted from everywhere is not guarded; one admitted from
// the mesh is. The registry is exposed everywhere on this node, and hello-web listens from
// everywhere.
with.Taken = map[string]bool{"distribution": true, "hello-web": true}
if got := guardOf(with); got != "" {
t.Fatalf("a port admitted from everywhere is guarded:\n%s", got)
}
with.Settings = nil
if got := guardOf(with); got != AsGuard([]int{5000}) {
t.Fatalf("the registry, from the mesh only, is not guarded:\n%s", got)
}
// Nothing taken, nothing guarded — and no guard at all rather than an empty set.
with = anchorRendering(true)
with.Taken = nil
if got := guardOf(with); got != "" {
t.Fatalf("an untaken store is guarded:\n%s", got)
}
// A given port is followed: where the machine put it is what is refused.
with = anchorRendering(true)
with.Given = map[string]map[int]int{"lavinmq": {5672: 5682, 15672: 15673}}
with.Ports["lavinmq"] = map[int]int{5671: 5671, 5672: 5682}
if got := guardOf(with); got != AsGuard([]int{5432, 5682, 15673}) {
t.Fatalf("the guard does not follow the given ports:\n%s", got)
}
}
// novox/hq ADR 0038: only a published port is the mesh's to move. A module that binds the machine
// itself listens where its software was told to, so giving it a machine port is refused.
func TestAGivenPortIsRefusedForAPortNoContainerPublishes(t *testing.T) {
onTheMachine := mcat.Manifest{Module: "daemon",
Listens: []mcat.Listening{{Port: 9000, From: mcat.FromMesh}}, Guards: []int{9000}}
layers := []mcat.Layer{{From: "node anchor",
Placement: map[string]any{mcat.PortsSetting: map[string]any{"9000": float64(9100)}}}}
_, err := GivenPorts(onTheMachine, layers)
if err == nil || !strings.Contains(err.Error(), "does not publish") {
t.Fatalf("a port no container publishes was given: %v", err)
}
}
// novox/hq ADR 0103: a guarded port this node is told to open to everyone is opened, not guarded.
// An opening from everywhere beside a guard dropping it is one statement refusing the other.
func TestAGuardedPortOpenedToEveryoneIsNotGuarded(t *testing.T) {
with := anchorRendering(true)
with.Settings["postgres"] = []mcat.Layer{{From: "node anchor",
Placement: map[string]any{mcat.ExposeSetting: map[string]any{"5432": mcat.FromEverywhere}}}}
composed, err := Compose(anAdoptedAnchor(), with)
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
if o := got["adoption.opening-tcp-5432-forwarded"]; o == nil || o["from"] != mcat.OpeningFromEverywhere {
t.Fatalf("the store's port is not opened to everyone: %v", o)
}
if guard, _ := got[GuardID()]["content"].(string); guard != AsGuard([]int{5672, 15672}) {
t.Fatalf("a port opened to everyone is still guarded:\n%s", guard)
}
}
// A module that publishes its ssh port the long way — `2222:22`, because the machine's own daemon
// holds 22 — and says it listens on the machine side of that mapping, which is what anything
// reaching it dials.
func aForge() mcat.Manifest {
return mcat.Manifest{Module: "forge",
Provides: []mcat.Offer{{Name: "git-over-ssh", Scope: mcat.ScopeMesh}},
Serves: map[string]map[string]any{"git-over-ssh": {"port": 2222}},
Listens: []mcat.Listening{{Port: 3000, From: mcat.FromMesh}, {Port: 2222, From: mcat.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "forge",
"ports": []any{"3000", "2222:22"}}}}
}
// portsAsThePlanWould is where this machine puts each of a module's ports, derived the way
// cmd/mesh-controller/plan.go derives it: a given port first, looked up by the port the module
// says it listens on, and otherwise wherever the manifest's own mapping already put it. Written
// here because everything below — the filter, the openings, the guard, what a consumer is told —
// reads that map, and a given port that the lookup does not find moves the container's mapping
// and nothing else.
//
// It stands in for the plan only where the plan does not allocate: a port the manifest already
// placed, or one a node was given. For a short form with no given port the real plan asks the
// inventory for a machine port and may come back with one from the pool, which needs a store and
// is what cmd/mesh-controller's own tests exercise. So an assertion here about such a port asserts
// this helper, not the mesh; keep the assertions to the ports under test.
func portsAsThePlanWould(m mcat.Manifest, given map[int]int) map[int]int {
out := map[int]int{}
for _, l := range m.Listens {
if at, is := given[l.Port]; is {
out[l.Port] = at
continue
}
at, _ := m.MachineSide(l.Port)
out[l.Port] = at
}
return out
}
// novox/hq ADR 0100 and 0038: the machine side of a long-form mapping is a port the module
// publishes, so a node may move it — and a module may name a mapping by either end.
func TestAGivenPortNamesEitherEndOfWhatTheModulePublishes(t *testing.T) {
forge := aForge()
node := func(v any) []mcat.Layer {
return []mcat.Layer{{From: "node anchor", Placement: map[string]any{mcat.PortsSetting: v}}}
}
// The machine side — the number this module says it listens on, and the one the predecessor
// had somewhere else. Answered under 2222, the end the module itself names, which is what
// every reader of this map asks for. One entry, not two: a second key for the same answer is
// an entry another mapping's reader could find instead.
given, err := GivenPorts(forge, node(map[string]any{"2222": float64(222)}))
if err != nil {
t.Fatalf("the machine side of a mapping cannot be given a port: %v", err)
}
if want := map[int]int{2222: 222}; !reflect.DeepEqual(given, want) {
t.Fatalf("the forge was given %v; the mapping it names is filed under %v", given, want)
}
// The container's own port names the same mapping and means the same thing — and is filed
// under the same name, because the module's name for it has not changed.
if inside, err := GivenPorts(forge, node(map[string]any{"22": float64(222)})); err != nil ||
!reflect.DeepEqual(inside, map[int]int{2222: 222}) {
t.Fatalf("the container's end of the mapping was given %v: %v", inside, err)
}
// A short form is published on the number it names, and is unchanged by any of this.
if short, err := GivenPorts(forge, node(map[string]any{"3000": float64(2999)})); err != nil ||
short[3000] != 2999 {
t.Fatalf("the short form was given %v: %v", short, err)
}
// A port no container publishes is still refused, in the same words.
if _, err := GivenPorts(forge, node(map[string]any{"9000": float64(9100)})); err == nil ||
!strings.Contains(err.Error(), "does not publish") {
t.Fatalf("a port the forge does not publish was given: %v", err)
}
// And the two ends of one mapping given two different numbers is one setting contradicting
// the other: the machine publishes it once.
if _, err := GivenPorts(forge, node(map[string]any{
"2222": float64(222), "22": float64(300)})); err == nil ||
!strings.Contains(err.Error(), "one mapping") {
t.Fatalf("the two ends of one mapping were given different ports: %v", err)
}
// Said at both ends with the same number, it is still said twice, and refused where every
// other repeated machine port is — as the inventory refuses it when the setting is stored,
// which is the layer that sees it first.
if _, err := GivenPorts(forge, node(map[string]any{
"2222": float64(222), "22": float64(222)})); err == nil ||
!strings.Contains(err.Error(), "to both its 22 and its 2222") {
t.Fatalf("one mapping given one machine port at both ends: %v", err)
}
// A number that names two different mappings names neither: which one to move is not said.
twice := aForge()
twice.Resources[0]["ports"] = []any{"22", "2222:22"}
if _, err := GivenPorts(twice, node(map[string]any{"22": float64(222)})); err == nil ||
!strings.Contains(err.Error(), "twice") {
t.Fatalf("a number naming two of the module's mappings was accepted: %v", err)
}
// And the same refusal when the number naming two mappings is not the one the setting used
// but the one the answer would be filed under. Here `80` is the module's own name for a
// mapping, and two mappings wear it; filing an answer there is one container's port standing
// where the other's is read, and both containers then publish it.
shared := mcat.Manifest{Module: "gallery",
Listens: []mcat.Listening{{Port: 80, From: mcat.FromMesh}},
Resources: []map[string]any{
{"id": "a", "type": "container", "name": "a", "ports": []any{"4001:80"}},
{"id": "b", "type": "container", "name": "b", "ports": []any{"4002:80"}}}}
if _, err := GivenPorts(shared, node(map[string]any{"4001": float64(1234)})); err == nil ||
!strings.Contains(err.Error(), "twice") {
t.Fatalf("two containers were put on one machine port: %v", err)
}
// A module whose mappings chain — one's machine side is another's container port — keeps them
// apart, because each is filed under the port the module names it by and neither name is
// shared. Given both, each moves on its own and neither overwrites the other.
chained := mcat.Manifest{Module: "chain",
Listens: []mcat.Listening{{Port: 80, From: mcat.FromMesh}, {Port: 9090, From: mcat.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "chain",
"ports": []any{"8080:80", "9090:8080"}}}}
both, err := GivenPorts(chained, node(map[string]any{"80": float64(1234), "9090": float64(5678)}))
if err != nil || !reflect.DeepEqual(both, map[int]int{80: 1234, 9090: 5678}) {
t.Fatalf("chained mappings were given %v: %v", both, err)
}
if moved := givenOuter("8080:80", both); moved != "1234:80" {
t.Fatalf("the first mapping moved to %q, and it was given 1234", moved)
}
if moved := givenOuter("9090:8080", both); moved != "5678:8080" {
t.Fatalf("the second mapping moved to %q, and it was given 5678", moved)
}
}
// And the number reaches everything derived from it. The fault this is written against moved the
// container's mapping alone: the filter opened the port the software had left, the adopted node's
// opening named it too, the guard refused it, and a consumer was sent to it.
func TestAGivenMachineSideReachesTheFilterTheOpeningAndTheConsumer(t *testing.T) {
forge := aForge()
given, err := GivenPorts(forge, []mcat.Layer{{From: "node anchor",
Placement: map[string]any{mcat.PortsSetting: map[string]any{"2222": float64(222)}}}})
if err != nil {
t.Fatalf("the machine side of a mapping cannot be given a port: %v", err)
}
r := mcat.Resolution{Node: "anchor", Modules: []mcat.Manifest{forge}}
with := mcat.Rendering{
Ports: map[string]map[int]int{"forge": portsAsThePlanWould(forge, given)},
Given: map[string]map[int]int{"forge": given},
Mesh: []string{"10.77.0.1"},
Adopted: true,
OutwardLinks: []string{"eth0"},
TunnelInterface: "mesh0",
Taken: map[string]bool{"forge": true},
}
// What the runtime is handed: the machine's own port on the outside, the container's within.
composed, err := Compose(r, with)
if err != nil {
t.Fatalf("the forge does not compose: %v", err)
}
got := byID(composed.Resources)
if ports := got["forge.server"]["ports"]; !reflect.DeepEqual(ports, []any{"3000:3000", "222:22"}) {
t.Fatalf("the forge's container publishes %v", ports)
}
// What the filter would open, were the node converged.
rules, err := Rules(r, with)
if err != nil {
t.Fatal(err)
}
var opened []int
for _, rule := range rules {
opened = append(opened, rule.Port)
}
// Only the moved port is asserted: the forge's other port is a short form the real plan would
// allocate rather than read off the manifest, so its number here is portsAsThePlanWould's and
// not the mesh's.
if !slices.Contains(opened, 222) || slices.Contains(opened, 2222) {
t.Fatalf("the filter opens %v, not where the machine puts the forge", opened)
}
// And what it is declared instead, adopted: an opening on the machine's port, naming the
// container's port on the forwarded path — a published port is forwarded, never received.
opening := got[OpeningID("tcp", 222, mcat.PathForwarded)]
if opening == nil || opening["to"] != 22 || opening["from"] != mcat.OpeningFromMesh {
t.Fatalf("no opening for the port this node gave the forge: %v", keys(got))
}
for id := range got {
if strings.HasPrefix(id, "adoption.opening-tcp-2222-") {
t.Errorf("an opening for the port the forge was moved off: %s", id)
}
}
// The guard refuses it where the machine put it, and nothing where it used to be.
guard, _ := got[GuardID()]["content"].(string)
if !strings.Contains(guard, "222") || strings.Contains(guard, "2222") {
t.Fatalf("the guard does not follow the given port:\n%s", guard)
}
// And a consumer is sent to the same number, which is read from what the module serves.
if told := ServedOn(forge, "git-over-ssh", with.Ports["forge"])["port"]; told != 222 {
t.Fatalf("a consumer is told the forge answers on %v", told)
}
}
// And the mapping itself moves under either name, because Rendering.Given is a map anybody
// composing a declaration hands in: keyed by the machine side, which is what a module declaring
// 2222 calls its port, only the outside moves and the container's own port stays as written.
func TestAMappingIsMovedUnderEitherOfItsNames(t *testing.T) {
for _, c := range []struct {
written string
given map[int]int
want string
}{
{"2222:22", map[int]int{2222: 222}, "222:22"},
{"2222:22", map[int]int{22: 222}, "222:22"},
{"127.0.0.1:15672:15672/tcp", map[int]int{15672: 15673}, "127.0.0.1:15673:15672/tcp"},
{"2222:22", map[int]int{3000: 2999}, "2222:22"},
{"2222:22", nil, "2222:22"},
} {
if got := givenOuter(c.written, c.given); got != c.want {
t.Errorf("%s given %v is published as %s, want %s", c.written, c.given, got, c.want)
}
}
}
// The same on a node that was given nothing, which is where the derivation was never at fault:
// a long-form mapping is published where the manifest put it, and an adopted node opens that port
// on the forwarded path like any other. What broke it was the number reaching only the mapping.
func TestALongFormPortIsOpenedWhereTheManifestPublishesIt(t *testing.T) {
forge := aForge()
r := mcat.Resolution{Node: "anchor", Modules: []mcat.Manifest{forge}}
composed, err := Compose(r, mcat.Rendering{
Ports: map[string]map[int]int{"forge": portsAsThePlanWould(forge, nil)},
Mesh: []string{"10.77.0.1"},
Adopted: true,
OutwardLinks: []string{"eth0"},
TunnelInterface: "mesh0",
})
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
opening := got[OpeningID("tcp", 2222, mcat.PathForwarded)]
if opening == nil || opening["to"] != 22 {
t.Fatalf("no opening for the forge's published ssh port: %v", keys(got))
}
}

Some files were not shown because too many files have changed in this diff Show More