Author SHA1 Message Date
jschoubben 015334020a Merge main (delivery 1, the parse-only sdk) into L3's re-land, and pin mesh-sdk 2da0de7, the release's final head (issue 774)
mesh/delivery-group group feat/774-l3-reland checking: 6 of 7 member(s) ready
mesh/check checking 015334020ad2 on a3c098a5c618 | novox/mesh-host#177 in group feat/774-l3-reland
mesh/delivery ready: it is merged on the Approve
The conflicts were go.mod, go.sum and testdata/beside/CAPTURED alone, taken from the release. 2da0de7 is 6e4a301's code;
its conformance events are unchanged.
2026-10-30 01:13:38 +01:00
mesh-tools a3c098a5c6 Merge pull request 'Pin the parse-only mesh-sdk in the node-engine, so every node can compose design 47's manifests (issue 774)' (#176) from fix/774-new-words-first into main 2026-10-30 00:10:21 +00:00
jschoubben 2fd165f5af Pin the parse-only mesh-sdk 78c9ec8 in place of 6e4a301, so main's engine reads the release's new words and means nothing new by them (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@a3c098a5c618
mesh/delivery-group group fix/774-new-words-first merged: every member is merged
mesh/check pass 2fd165f5af8e on 8c5af3e4a3a3 | unit 9174 run, 546 kept, 51 skipped | integration 9952 run, 659 kept, 60 skipped | over bound: unit, in…
78c9ec8 is sdk main 307b623, whose tree is main's pin dbcaa29, with one commit that parses the new words (a credential's
form, an artifact's bus, a log's max-age-days) and names the new seats; an offer with no form still means pair. No code
of ours changes. Unit and integration suites pass against a local bus and main's catalogue and controller beside: 1260
tests, 7 skipped for a container or a throwaway machine.
2026-10-30 00:39:20 +01:00
jschoubben db1151c773 Pin mesh-sdk 6e4a301 on main's engine, so the controller and the engine read the release's new manifest words before the release uses them (issue 774)
No code of ours changes. Whole unit and integration suites pass at main's code against a local bus: 1260 tests, 7
skipped for a container or a throwaway machine.
2026-10-30 00:25:10 +01:00
jschoubben 02f78c9949 Pin mesh-sdk 6e4a301, the release's final sdk (issue 774)
The whole integration suite passes at this head against a local bus (JetStream, and a bus with the witness grants) and
the release's mesh-catalog and mesh-controller beside it: 1260 tests, 7 skipped for a container or a throwaway machine.
2026-10-30 00:17:00 +01:00
jschoubben 192633edb8 Pin mesh-sdk 007b6fd, the release's final sdk (issue 774) 2026-10-30 00:07:07 +01:00
jschoubben e5e8ff5783 Say each account's root verdict and the engine's contract in the node's health report, so the self-check's DA and D-root judge without the controller (issue 1952, ADR 0266)
Every account resource marked root never goes into NodeHealth.Root; the statement's contract into Contract. Relogin,
root and found are judged each on its own, so an account may be both. Pins mesh-sdk f3677eb. Test: the report's row in
TestAHealthStatementIsSaidOnTheLinkOpenNow carries the agent account's verdict and the contract.
2026-10-29 23:49:36 +01:00
jschoubben c3ff18de4e Name the machines a failing network part points at in the node's health report, so the watchman holds it where the controller does (issue 774)
delivery.NetworkPart.Targets (mesh-sdk 6eae076) is filled where the report is made, by the controller's own rule:
- the hub's node for "hub";
- each resolver's node by its address on the private network;
- none when the part points at nothing, at this machine, or at an address no other mesh machine has.

The names come from the placements the node's own composer holds (Decided.Address, Decided.Hub). A spoke's tunnel names
only the hub, so the tunnel cannot name a resolver on another spoke. A node that does not compose its own names no
target.

Test: rows in TestAHealthStatementIsSaidOnTheLinkOpenNow (hub, two resolvers, a VPN's resolver, its own, a healthy
part).
2026-10-29 23:05:02 +01:00
jschoubben 3ed9abf658 Format the health report's aliases as gofmt does (issue 774) 2026-10-29 22:59:38 +01:00
jschoubben 552f97c919 Take the node's health report layout from mesh-sdk 94d0de5, so the engine and the watchman read one form (issue 774)
delivery.NodeHealth and its parts replace this package's copies. The message's network and units parts are aliases of
the same types, so its JSON is unchanged. The engine keeps the derivation from its statement and the write on a change.
Test: the key's bytes, decoded and encoded again by the sdk, are the same bytes.
2026-10-29 22:59:22 +01:00
jschoubben 65fbc1e6f8 Write each node's health report into its own key of mesh-host_health on a change, so the watchman raises the machine's alerts without the controller (issue 774, the operator's alert table)
Until now the six machine alerts came from a health message passing to the controller. The report is now kept in the
engine's declared state `health`, key `<node>`, created by the grants part as `acts` is. Only this node's engine
writes it, after comparing it on every look and only when what it says changed; it is read first after a restart.

It carries:
- the network parts (machine-network; network-rewritten through the resolver file part's writer and owner);
- the tunnel's peers, each seen within five minutes or not (network-unreachable);
- the failed units no module owns (machine-units);
- the accounts marked relogin needed (relogin-needed);
- the directories used as found (directory-used-as-found).

The streaks and the time of the look are left out, so a look alone writes nothing. A peer is reported seen or not,
never its handshake time, so a renewed handshake writes nothing.

The layout is internal/link's (NodeHealth) until it moves into mesh-sdk. Pins mesh-sdk 58a8324.

Tests:
- rows in TestOnTheHubAnyFreshPeerIsAHealthyTunnel: the peers, and a change only when one is seen or lost;
- a case in TestAHealthStatementIsSaidOnTheLinkOpenNow: what the report takes, and one write per change across a
  restart;
- the manifest's states in TestTheManifestConsumesTheBackupEvent.
2026-10-29 22:56:14 +01:00
jschoubben cbe8528f91 Resolve ${secret:<name>} through this node's provision keys, and check the reference by the vault's own name (issue 1954 L6-E1, E2; ADR 0570 clauses 1, 6; ADR 0094)
- A placeholder is `${secret:<name>}`, the name the module calls the credential (catalogue.SecretLocal): the local
  name where it keeps several values of one provision, else the provision. The `<provision>.<local>` form goes.
- The engine finds the one key under `<node>.<module>.` of mesh-provisioner's `provisions` going by that name. That is
  what node.<node> may read, by a watch to its caught-up mark; plans are not read.
- The reference's ID must be a credential name of that provision (provisions.VaultName). The vault's answer must carry
  exactly that name and generation. Anything else refuses the write, naming both.

Pins mesh-sdk 66ff33d. Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise (a local name, a name no provision
goes by, a reference to no credential of the provision, a mismatched answer).
2026-10-29 22:41:44 +01:00
jschoubben 5497d362dd Refuse a provision's value unless the vault's credential name and generation are the ones its reference names (issue 1954 L6-E1; ADR 0570 clauses 1, 6)
The value written is the one the provision names: a1-47 makes Reference.ID the vault's credential name. A mismatch in
either refuses the write, naming both. Test: a row in TestAWriteAndAnApplyOfTheSameFileSerialise.
2026-10-29 22:37:44 +01:00
jschoubben 6b32e07164 Fill a provision's ${secret:} at the write from the vault's value sealed to this machine, so the provisioner's client never holds it (issue 1954 L6-E1, E2; ADR 0570 clause 6)
The provisioner's client fills a handed module's file but for each `${secret:<provision>}` (or a whole `secrets` file
that is the placeholder alone).
- The node-engine reads the provision's key `<node>.<consumer>.<provision>[.<local>]` in mesh-provisioner's
  `provisions`. The key must be `provided` and carry the vault's reference.
- It asks the vault's `own` for that pair credential sealed to this machine, at least at the reference's generation,
  and opens it with the node's sealing key.
- It writes the file as declared (its owner), 0600 when the declaration gives no mode.
- Anything short of that refuses the write, naming the provision and why.

A module the node composes itself already has its `secrets` files written so: its composition carries the vault's
value from `own`, written 0600 with the declared owner.

Pins mesh-sdk 4a96793 (go/provisions). Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise.
2026-10-29 22:36:53 +01:00
jschoubben f7814853f9 Pin mesh-sdk 2e5cc4f, the re-land group's sdk head (issue 774) 2026-10-29 22:18:45 +01:00
jschoubben 58aacaad2d Pin mesh-sdk 7000bd5, the re-land group's sdk head: a tools bundle declares its own bus account (issue 1952 S5, issue 774) 2026-10-29 21:57:09 +01:00
jschoubben 81672b183d Keep the node's own as it runs from an engine's first start, and refuse the node's own apply removing a module no want removed (issue 774, review round 1)
The reviewer showed how 76e214a still lets the 07:54 outage happen at the re-landed engine's first start on novox.
applied/ is empty there, so loadApplied loads nothing. The kept declaration retires, because every placed module has a
have. Nothing then carries the node's own composition. The guard misses nats, which has no want there, so its
nats.conf is guessed as nats's and removed.

- At the start, when the store has no node's own composition, the node's own runs as the kept declaration has it, and
  the store keeps it now. The kept declaration retires only once the store holds `node`.
- What this engine applied runs until loadHad reads the composition a differing have names. The bucket no longer keeps
  the one novox's have names, so the stored one stays.
- In the node's own apply, a resource of another module goes only when that module's want says removed
  (Scope.RemovedByWant). A module the node's own composition carried has no want, so Placed never named it.
  Cost: the node's own apply cannot drop one of its own dotted resources without such a want. It is refused whole and
  said, never removed silently.

Tests, with no new test functions:
- A first-start case in the agent's Everything test (empty store, then a start with the declaration retired). It fails
  on 76e214a.
- Two rows in the guard's table: no want refuses; a want saying removed removes.
- A case for the store in the authority test.
2026-10-29 21:25:51 +01:00
jschoubben 76e214a8b9 Keep every composition the engine applied on the machine, so a module whose composition the bus no longer keeps runs as applied and is never composed as nothing (issue 774)
On novox on 2026-10-29 at 07:54 UTC the kept declaration was retired and the node's own composition was in no bucket.
The composer's first pass reads what runs before the agent reads any have (the agent waits for the composer's caught,
sent after that pass), so the node's own was composed with its folded modules running nothing, and the bus server and
four held modules were removed.

- Quick fix: read the node's own have in loadHad too. It still waits for the composer's first pass, and still finds
  nothing when the bucket no longer keeps the composition the have names.
- Thorough fix, chosen: the engine keeps each composition it applied (the node's own too) in `applied/` beside its
  state, 0600. The agent reads it at Run's start, before any watch, as what runs ahead of the bucket and the kept
  declaration. A have naming another running composition wins. An apply keeps; a removal forgets.

Test: a case in TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest (no new test function). It fails on L3's
engine 4f9227a (0 compositions at the first own composition) and passes here.
2026-10-29 21:16:38 +01:00
jschoubben acf02d52ff Pin mesh-sdk 22c9ddd, so a carried or held module serves its tools from what runs and a data flag alone moves no data (issues 1126, 924) 2026-10-29 21:05:12 +01:00
jschoubben a7acee3eed Refuse an apply that would remove a resource of a module a want still places here, so a composition that lost what runs can never remove the bus server or a held module again (issue 774) 2026-10-29 21:04:17 +01:00
jschoubben d2645a27e9 Compose a file that runs as the node last wrote it, so a stale kept declaration's export no longer names a swept object (issue 924)
novox's mesh-vault has failed since 01:16 on 2026-10-29: its have says failed, so loadHad (fbea777) takes no
composition from it, and what runs falls back to the kept declaration, whose export (mesh-vault.kept) names
f099eaa1, an object the controller's sweep removed. The node wrote a newer export since. The composer carries that
file as it runs (mesh-sdk frombus keptOf), so what runs now names the digest and size of what is on disk, but only
where the node's record says it wrote exactly that content at that path (Applied.Wrote, Target). The apply then
finds the file in place and needs no object.

Quick fix weighed: have the controller put the object back. Thorough (chosen, in the node): what runs is what the node
wrote, never a guess from an older declaration. The rows join the existing whole-apply test; the test count stays.
2026-10-29 21:04:17 +01:00
jschoubben 7d39510450 Keep the content of every file given by digest on the node, so an object the controller's sweep removed never refuses a composition the node already had (issue 924)
On novox at 01:16 on 2026-10-29, after the node began composing its own modules, mesh-vault's apply failed while the
vault ran: its own composition carries the export as it runs (mesh-sdk frombus KeptRuns) by a digest whose object the
controller's sweep (mesh-controller cmd/mesh-controller/declaration_sweep.go, sweepNode and sweepDeclarationFiles)
had removed, since the sweep keeps only what the controller's compositions in composed-<node> and the wants name.
The node may only read its bucket (ADR 0486 §2), so the node keeps what it had: a file here that verifies, or an
object fetched, goes to <state dir>/objects under the object's name, and a kept one is used before the bus is asked.
An object unused for 14 days is let go. A fetch that still fails says the keep never held it, before its cause.

Quick fix weighed: re-put the object from the controller's store. Thorough (chosen): the node never depends on the
bucket for content it once had, for every module, since the sweep and a node composing its own can always disagree.
The whole-apply test gains three rows that fail on the old code; the test count stays.
2026-10-29 21:04:17 +01:00
jschoubben f38ac87ac9 What runs from the kept declaration says Data as the module's manifest does, and only a have that runs counts at the start (issue 924)
The kept-declaration fallback built a composition without Data, so after a restart a data module (mongodb, mssql
on novox) was judged to move its data against its first composition and the operator was asked a false yes. Now
every such composition takes Data from the manifest (agent.Env.HoldsData, the composer's DataItems), and a module
whose manifest cannot be read is said to be unjudgeable rather than taken to hold none. The composition a have
names comes before the kept declaration in the judgement too.

loadHad takes a have only in the phases baseline takes as running (running, held, a composer's mark): one backing
up, applying, checking, failed or waiting names a composition not applied (review of 031e106). Two rows added to
the same table test; the count stays.
2026-10-29 21:04:17 +01:00
jschoubben d8de0a8853 Judge a module the agent has not taken yet by the composition its have names, and knock the filling at the start and whenever a module is taken as running (issue 924)
A held module on a node composing its own never gets a composition, so the agent never took it and the filling
judged its files by the kept declaration, older than power's three new drop-ins: refused from every engine start
on ace, g14 and shanks until some module settled (2026-10-29 00:09 UTC). The second meeting of issue 924.

Quick fix: knock the filling once the haves are read at the start. Thorough fix (chosen, the kind recurs for every
file added since the switch): what runs of such a module is the composition its have names, from the bucket or
its revisions, ahead of the kept declaration; the filling is knocked on every change of what runs, not only on a
settle. One unit test replaced by a table of four cases; the count stays.
2026-10-29 21:04:17 +01:00
jschoubben 45d0348044 Revert "Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#174) from revert/774-l3 into main"
This reverts commit 1ec73915b8, reversing
changes made to 0a4c35f731.
2026-10-29 21:04:13 +01:00
mesh-tools 8c5af3e4a3 Merge pull request 'Pin mesh-sdk dbcaa29, so a held module's hold no longer spreads to the modules that read from it (issue 1960)' (#175) from fix/1960-no-hold-spread into main 2026-10-29 15:35:33 +00:00
jschoubben ece607e215 Pin mesh-sdk dbcaa29, so a held module's hold no longer spreads to the modules that read from it (issue 1960)
mesh/delivery merged: source.main moved to novox/mesh-host@8c5af3e4a3a3
mesh/delivery-group group fix/1960-no-hold-spread merged: every member is merged
mesh/check pass ece607e21522 on 1ec73915b807 | unit 3318 run, 1951 kept, 38 skipped | integration 3994 run, 2125 kept, 38 skipped | over bound: unit, …
2026-10-29 16:00:21 +01:00
jschoubben 86a05babed Expect proxy handed beside held de-spiegel: a hold no longer spreads (issue 1960)
mesh-sdk's composer no longer holds a module behind a held one it reads from; the selfcompose test now expects
proxy handed, composed without de-spiegel's routes, and de-spiegel told alone. Needs mesh-sdk pinned at the fix.
2026-10-29 15:58:19 +01:00
mesh-tools 1ec73915b8 Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#174) from revert/774-l3 into main 2026-10-29 08:59:58 +00:00
jschoubben f81e6147ae Revert L3 (mesh-host, merge 0a4c35f): its node-engine removed every held module on novox, the bus server among them, at 08:54 CET on 2026-10-29 (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@1ec73915b807
mesh/delivery-group group revert/774-l3 merged: every member is merged
mesh/check being carried to f81e6147aee1 on 0a4c35f731a7 | from mesh-catalog 1555241393fe | novox/mesh-host#174 in group revert/774-l3
L3 lands again with two fixes: a held module is never removed, and the grants cover L3's new subjects.
2026-10-29 09:42:29 +01:00
mesh-tools 0a4c35f731 Merge pull request 'L3: the node trusts the vault's key, serves the node seat node-engine and keeps its acts on the bus (issue 774, ADR 0561), rebased onto L2' (#168) from feat/774-vault-mints-l2 into main 2026-10-29 07:44:19 +00:00
jschoubben 4f9227a186 Merge main (#171) into L3: pin mesh-sdk c2999a4, mesh-sdk #128's head, which holds L3's sdk and both seed fixes (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@0a4c35f731a7
mesh/delivery-group group feat/774-vault-mints-l2 merged: every member is merged
mesh/check pass 4f9227a1860f on fb8ea0e6838c | unit 1601 run, 5122 kept, 47 skipped | integration 1637 run, 5215 kept, 49 skipped | over bound: unit, …
2026-10-29 01:08:25 +01:00
mesh-tools fb8ea0e683 Merge pull request 'Pin mesh-sdk c2999a4, so a node's seed answer holds a held module as it runs and its neighbours compose (issue 1126)' (#171) from fix/1126-held-as-runs into main 2026-10-29 00:07:06 +00:00
jschoubben a6c70803ab Pin mesh-sdk at c2999a4: a held module no longer spreads its hold to its neighbours (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@fb8ea0e6838c
mesh/delivery-group group fix/1126-held-as-runs merged: every member is merged
mesh/check pass a6c70803ab72 on bb6e47293909 | unit 3054 run, 234 kept, 36 skipped | integration 3075 run, 266 kept, 29 skipped | over bound: unit (sl…
novox's seed p-20261028t231141-ec84b4 refused 11 modules behind de-spiegel's hold and gitea's handed credential. The
node composes and answers through mesh-sdk's frombus, so the fix is the pin.

- The quick fix: follow one more set of words in the answer. It moves the error again.
- The thorough fix, chosen (mesh-sdk fix/1126-held-as-runs): a held module composes only as what runs of it; a consumer
  or collector composes against that and is never held for the held one's stub, and a stub's handed credentials and own
  secrets are left out as they run rather than refused.
2026-10-29 00:50:01 +01:00
jschoubben 9bd1c84e0f Pin mesh-sdk 76f7c4d, the seed fix's own head: the checker refuses a pin to main's merge commit d4c61ee (same tree) as on no pull request (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-delivery@64d364a7686c was judged on trunk fa075dc8…
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 9bd1c84e0f4c on fb8ea0e6838c | novox/mesh-host#168 in group feat/774-vault-mints-l2
2026-10-29 00:20:01 +01:00
jschoubben 1f3fb52470 Merge main into L3: pin mesh-sdk main d4c61ee, which holds L3's sdk and the seed's held-as-today fixes (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 rejected: composed together: the checker's verdict at the members' heads (2199dae4e7d8): ace
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: the check of novox/mesh-host#168 failed: the pin of git.novox.be/novox/mesh-sdk/go at commit d4c61ee8ffd0 is on no pull reques…
2026-10-29 00:11:11 +01:00
mesh-tools bb6e472939 Merge pull request 'A node answers its seed with an old-form build, a failed or held module, and those held behind it, as they run today (issue 1126)' (#170) from fix/1126-seed-oldbuild into main 2026-10-28 23:05:08 +00:00
jschoubben 93f33e2305 Pin mesh-sdk 76f7c4d: a held module keeps its items (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@bb6e47293909
mesh/delivery-group group fix/1126-seed-oldbuild merged: every member is merged
mesh/check pass 93f33e2305de on d2a868f99f05 | unit 3018 run, 270 kept, 36 skipped | integration 3039 run, 302 kept, 29 skipped | over bound: unit (sl…
2026-10-28 23:56:51 +01:00
jschoubben abdf85aa4f Node agent: say how it keeps each module today, held, failed or at a build of the old form named by its have's build; pin mesh-sdk f609be7 (issue 1126) 2026-10-28 23:50:19 +01:00
jschoubben 91686fdb81 Node agent: count a module running a build of the old form as held today, so the seed keeps it as the controller does (issue 1126) 2026-10-28 23:45:28 +01:00
mesh-tools d2a868f99f Merge pull request 'Node agent: say what it holds today in its answer, so a seed holding the same is the same (issue 1126)' (#169) from fix/1126-seed-held into main 2026-10-28 22:20:59 +00:00
jschoubben a143678fd8 Pin mesh-sdk 77b48a4, the group's head (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@d2a868f99f05
mesh/delivery-group group fix/1126-seed-held merged: every member is merged
mesh/check pass a143678fd898 on 4f6b389a37a1 | unit 1199 run, 1015 kept, 33 skipped | integration 1218 run, 1081 kept, 26 skipped | over bound: unit (…
2026-10-28 22:41:14 +01:00
jschoubben 9914459830 Pin mesh-sdk 06b13ec, the group's head (issue 1126)
mesh/delivery-group group fix/1126-seed-held checking: 2 of 3 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 9914459830d7 on 4f6b389a37a1 | novox/mesh-host#169 in group fix/1126-seed-held
2026-10-28 22:18:26 +01:00
jschoubben b7159d71a6 Node agent: say what it holds today in its answer, so a seed that holds the same is the same; pin mesh-sdk b03bf4c (issue 1126) 2026-10-28 22:15:24 +01:00
jschoubben e0c7f36ca9 Pin mesh-sdk 8eafc0c, the L3 group's head on mesh-sdk main with #120 and #121 (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-host@e0c7f36ca90c was judged on trunk d2a868f99f05…
mesh/delivery closed: a newer head of the same pull request
mesh/check checking e0c7f36ca90c on bb6e47293909 | novox/mesh-host#168 in group feat/774-vault-mints-l2
2026-10-28 21:56:06 +01:00
jschoubben 9120f94aa4 Name the seat mesh-placement where the kept declaration's retirement reads the placement, as L2 moved it (issue 774, ADR 0567) 2026-10-28 20:36:41 +01:00
jschoubben 1900b40192 Pin mesh-sdk 4e6e5cc, the L3 group's head rebased on L2 (issue 774) 2026-10-28 20:36:41 +01:00
jschoubben 6cbdb3712e Judgements and refused setuid searches, which any caller may ask, roll in one bounded key of the node's and add no key to the bucket every node shares; a key per act is kept for hand-overs, access acts and searches started (issue 774, ADR 0563 §2; rev-l3 round 2) 2026-10-28 20:35:33 +01:00
jschoubben 2a0c428ef9 The last setuid search's start and each library's own take ask are keys of their own, which no number of other acts moves (issue 774, ADR 0563 §2, §4; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben a5d1f7dde4 From the lease write a directory used as found names the node seat's hand-over as its way out, following the lease the engine hears (issue 774, ADR 0563 §5; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben 95dd64c774 Every signed ask is verified with the controller's key the engine trusts at that ask, so after the lease write none signed by it is taken (issue 774, ADR 0563 §7, ADR 0550 §2; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben 81fa009bf4 The node seat's acts and the engine's own asks are the node-engine's declared state acts on the bus: one key each under its own node's keys, written by compare-and-set, one ask open per path made by a create; acts reads one by its id (issue 774, ADR 0563 §2, §3) 2026-10-28 20:35:33 +01:00
jschoubben 05ff90e3c1 The engine judges its node seat's hand-over and take as it judges the controller's ask, answers root-free for its own machine, and asks a library's take itself once (issue 774, ADR 0563 §3, §4, §6) 2026-10-28 20:35:33 +01:00
jschoubben aaffc19232 The node seat node-engine served on the engine's link: its verbs with their proven caller, its acts kept in its own state, a hand-over or access act asked of the operator as node-engine-<node> and performed only on the warrant (issue 774, ADR 0563 §1-§3) 2026-10-28 20:35:33 +01:00
jschoubben 479ac71050 The node trusts the vault's key beside the controller's until the lease is handed away, says what it trusts in its record, asks the vault for its own certificate, composes the vault's authority beside the controller's, and reads the kept declaration no more once every placed module has a have (issue 774, ADR 0550 §2) 2026-10-28 20:35:33 +01:00
mesh-tools 4f6b389a37 Merge pull request 'Node agent: compose from the seat mesh-placement and answer proposals (issue 1126, delivery 2 of 2)' (#167) from feat/1126-placement-seat into main 2026-10-28 18:33:12 +00:00
jschoubben 45f5e27526 Pin mesh-sdk to the group's head 38dbe1d (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@4f6b389a37a1
mesh/delivery-group group feat/1126-placement-seat merged: every member is merged
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
2026-10-28 18:50:39 +01:00
jschoubben dd8ba97989 Node agent: wait for the bus's word that the seat's placement and candidates states were created, then watch them (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben ab22efd8b1 Node agent: follow the placement key at start only and say why when it cannot; test the reached candidate and the switched rule; word the log by the seat's state (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 54aee345fc Node agent: a node that never composed its own takes a failed placement read as not placed; one that did keeps waiting (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 8639b373d2 Prove a missing or refused bucket is a watch error, so a node waits on the controller's compositions and tries again (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 5fc69444f6 Node agent integration tests: candidates keyed by proposal, under the node's real grant on the seat's states (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 2fe9d678fc Node agent: answer a proposal's candidate keys and follow the seat's placement, not the indexer's groups (issue 1126)
The answer names its proposal and the revision of the node's own candidate key; a node with no key in a proposal is not asked. The watch for its own placement key is tried again until the seat's state exists, so the node switches when its key is written.
2026-10-28 18:43:49 +01:00
mesh-tools 58b4f8b6bc Merge pull request 'Reconcile a module from the machine's facts at start and on a down edge (issue 1957, ADR 0568)' (#166) from fix/1957-boot-reconcile into main 2026-10-28 17:41:54 +00:00
jschoubben bf667a602d Reconcile a module from the machine's facts at engine start and at each edge of a declared-running resource going down (issue 1957, ADR 0568)
mesh/delivery merged: source.main moved to novox/mesh-host@58b4f8b6bc07
mesh/check being carried to bf667a602d21 on 58b4f8b6bc07 | from 9a8b3870bad0
A reboot left the have running, so a unit that failed to start at boot was never started again and nothing said so. Baseline now asks the machine; the judge's look calls Agent.Reconcile once per down edge, leaving held and unwanted modules alone.
2026-10-28 18:12:35 +01:00
mesh-tools 9a8b3870ba Merge pull request 'The node-engine applies every module but the held ones and marks only those (group fix/1126-apply-culprit, issue 1126)' (#165) from fix/1126-apply-culprit into main 2026-10-28 16:34:48 +00:00
jschoubben 24c23a1d06 A module the node holds alone marks that module, keeps what runs of it, and never refuses the node (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@9a8b3870bad0
mesh/delivery-group group fix/1126-apply-culprit merged: every member is merged
mesh/check pass 24c23a1d066f on a548a61e86d7 | unit 2667 run, 590 kept, 36 skipped | integration 2676 run, 633 kept, 29 skipped | over bound: unit (sl…
frombus.Compose now holds a module refused alone (mesh-sdk 0de3006, the rule
its answer follows too): composed as what runs of it, with what depends on
how it is held held too. The composer hands every other module, keeps what it
handed of a held one, and tells the agent which modules it holds and why;
agent.NotComposedEach marks those alone and clears the rest. Only a refusal
of no single module still marks every module (NotComposed).

heldOut also holds out, of the node's own composition as it runs, the
resources of each module held alone, until a composition of its own is
handed: the node's own apply never removes a module leaving the fold with its
names unset (review of the apply, round 1; issue 1945 by a new path).

Pins mesh-sdk fix/1126-apply-culprit at 0de3006 (a group).
2026-10-28 17:23:26 +01:00
mesh-tools a548a61e86 Merge pull request 'The node-engine reloads the service manager when a unit file or drop-in it writes changes (group fix/shutdown-bound, issue 1126)' (#164) from fix/shutdown-bound into main 2026-10-28 15:05:10 +00:00
jschoubben bc28606d45 apply: reload units only for a unit file actually written or removed (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@a548a61e86d7
mesh/delivery-group group fix/shutdown-bound merged: every member is merged
mesh/check pass bc28606d4566 on 690aa662a837 | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
A skipped or held file comes back on every apply; counting it would reload the
service manager on every apply of a node with a gated or held module. The same
allowlist as MovesRoot: created, updated, corrected, removed.
2026-10-28 15:49:34 +01:00
jschoubben 7f0eb9c51d apply: reload the service manager once when a file in a unit directory moved (issue 1126)
A drop-in that no service reflects, like the power module's bound on
reboot.target, was written but never loaded: the node-engine reloaded units only
for a file named in a service's restart-on, so the next reboot still ran the
target as loaded. Any file written or removed under a system unit directory, or
a drop-in directory below one, now gets one daemon-reload per apply.
2026-10-28 15:43:35 +01:00
mesh-tools 690aa662a8 Merge pull request 'The node's shadow says which input woke each answer (issue 1126)' (#163) from fix/1126-vault-own into main 2026-10-28 14:20:39 +00:00
jschoubben 8bd0b5db15 Say which input woke each placement candidate's answer, so a probe's safety check tells a loop from moving inputs (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@690aa662a837
mesh/delivery-group group fix/1126-vault-own merged: every member is merged
mesh/check pass 8bd0b5db15a3 on b4fd2cc210bf | unit 1639 run, 4201 kept, 46 skipped | integration 1674 run, 4215 kept, 47 skipped | over bound: unit, …
Each answer's line now reads 'answering placement candidate G (revision N; woken by X)', X being its first answer,
a retry after a read the bus did not answer, the candidate, or the inputs that changed (placement, records, modules,
values, haves, wants). The single input counter becomes one per input, which also decides whether a group is answered
again, as before.

Quick fix weighed: a counter word in the line. Thorough fix chosen: one count per input, so the line names exactly what
moved since the group's last answer.
2026-10-28 14:44:27 +01:00
mesh-tools b4fd2cc210 Merge pull request 'A one-shot at rest satisfies a stopped declaration, so the kernel's boot cleanup applies (issue 1126)' (#162) from fix/ace-tun-cleanup into main 2026-10-28 13:34:20 +00:00
jschoubben 1a021052fe Drop the never-ran reading: a start timestamp of 0 does not mean a one-shot never ran (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@b4fd2cc210bf
mesh/check being carried to 1a021052fe07 on b4fd2cc210bf | from 7a2af2376405
Review round 1 measured it on systemd 262.1: a finished one-shot nothing references is
unloaded, and loaded again it reads inactive, ExecMainStatus=0 and a start timestamp of
0, as one that never ran does. The previous commit's rule read such a unit stopped, so
a one-shot declared running would be started, read stopped and fail on every apply, and
the hold, the take-over and the giving back would judge it wrongly.

ServiceState keeps main's reading. ServiceAtRest and declaredState stay: they alone
satisfy a stopped declaration for a one-shot at rest, which is the kernel cleanup's
case. A one-shot declared running that must be seen to have run lingers
(RemainAfterExit) or is enabled at boot. Unit tests: the arch table has 5 rows, the
apply test 1; 1262 in all.
2026-10-28 14:29:16 +01:00
jschoubben cf738cdb90 Read a one-shot that never ran as stopped, and one at rest as stopped to a stopped declaration (issue 1126)
The kernel module's boot-time cleanup, linux-modules-cleanup.service (Type=oneshot,
declared stopped and enabled at boot), failed its delivery on ace with 'was asked to
be stopped and is running': ServiceState read every inactive one-shot with
ExecMainStatus=0 as running, and systemd says 0 for one that never ran too. Stopping
an inactive unit changes nothing, so the read back said running again; after a boot
that ran it, the same declaration would fail every apply.

ServiceState now reads ExecMainStartTimestampMonotonic: 0 means never ran, read as
stopped (a package's one-shot declared running is then started, and the hold no
longer counts it as in use). The service apply reads a one-shot at rest as stopped
to a declaration that it be stopped (ServiceAtRest), whatever its last run did.

Quick fix weighed: declare the cleanup running in the catalogue. Thorough chosen:
the reading was wrong for every one-shot, and this is its second fault of the kind
(the firewall's was the first). Unit tests: three arch one-shot tests merged into one
table (7 rows), one apply test (2 rows); 1263 to 1262.
2026-10-28 14:29:16 +01:00
mesh-tools 7a2af23764 Merge pull request 'Pin mesh-sdk f387581: the node composes refusals per module and folded modules' owned paths as the controller does (issue 1126, group fix/1126-seed-8)' (#161) from fix/1126-seed-8 into main 2026-10-28 13:27:09 +00:00
jschoubben daaa1962bd Pin mesh-sdk f387581, the group's head: a module's own refusal refuses that module, contributions in config words, folded modules own their manifest's paths (issue 1126, seed-8)
mesh/delivery merged: source.main moved to novox/mesh-host@7a2af2376405
mesh/delivery-group group fix/1126-seed-8 merged: every member is merged
mesh/check pass daaa1962bdab on df878ba1f780 | unit 3018 run, 234 kept, 36 skipped | integration 3038 run, 266 kept, 29 skipped | over bound: unit (sl…
2026-10-28 14:08:58 +01:00
mesh-tools df878ba1f7 Merge pull request 'The node answers a placement candidate once, retries a read the bus did not answer, and keeps a file's secrets entry by entry (issue 1126; group fix/1126-seed-same-3)' (#160) from fix/1126-seed-same-3 into main 2026-10-28 11:40:38 +00:00
jschoubben 1e01868d92 An unplaced node owes no pass, so no timer is armed again; every wait a test sets is put back when it ends; pin mesh-sdk 8520a4c, whose masking splits only a file's secrets (issue 1126; review of seed-same-3, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@df878ba1f780
mesh/delivery-group group fix/1126-seed-same-3 merged: every member is merged
mesh/check pass 1e01868d92c3 on c3a2ffde6755 | carried from mesh-catalog 774fcdfa93ad | unit 2188 run, 1064 kept, 36 skipped | integration 2197 run, 1…
2026-10-28 12:25:32 +01:00
jschoubben 9d04ae7edd Hold the secrets row to the have read's error, as the loop fix's Env.Held returns it (issue 1126) 2026-10-28 12:13:09 +01:00
jschoubben 16df7d0b7a Merge branch 'fix/1126-answer-once' into fix/1126-seed-same-3 2026-10-28 12:11:50 +01:00
jschoubben cd1d2ff447 A read the bus did not answer is tried again after a bound, never kept as an answer: the shadow's answer, the node's composition and the own-secret step (issue 1126)
Met twice on 2026-10-28: shanks kept 'want.shanks.brother could not be read ... no
responders' as the own-secret step's last word, and ace kept 'the vault's own: not yet
... no responders' as its answer to seed-7, each until another input moved. Every bus
read of the composer (manifests, builds, archives, served values, the vault's own and
credentials, the have of what runs) is noted (BusFailed); an answer or a pass that met
one is made again after a growing bound (5 s to 5 min), and a pass is not handed when
the have of its sealed values did not answer. The own-secret step tells a read or an
ask that did not answer (Retry, taken again) from a key absent or the vault's word.
2026-10-28 12:11:08 +01:00
jschoubben faf6a23d91 Fill go.sum for mesh-sdk 70de942 (issue 1126) 2026-10-28 12:06:25 +01:00
jschoubben 2f4e81f3b0 A file's secrets are kept entry by entry: the box that runs at one secret is reused only there, where it opens to the same value; pin mesh-sdk 70de942 (issue 1126, C1; the seed-7 probe) 2026-10-28 12:05:48 +01:00
jschoubben 4182f8fbfb The node answers a placement candidate revision once: a module's have whose health words alone moved, or a key written again with its value, no longer composes every group's answer again (issue 1126)
On shanks probe/1126-seed-7 was answered at revision 84 every 13-17 s: each look rewrote
the haves of its unhealthy modules with new words, and the shadow took any write of its
own module haves, placement, modules or config values as a change. Each answer loaded
the bus, the health tools timed out, the words moved again. Now only what an answer is
composed from wakes it (what runs, folded modules, a changed value), and each group is
answered again only when its keys' revisions or the node's inputs moved.
2026-10-28 12:02:57 +01:00
mesh-tools c3a2ffde67 Merge pull request 'The placement test's resolved values carry their source, as mesh-config writes them (issue 1126)' (#159) from fix/1126-seed-same-2 into main 2026-10-28 10:08:46 +00:00
jschoubben 6502ba6ad6 The placement test's resolved values carry their source, as mesh-config writes them: a value with no source is laid by neither the controller nor a node composing its own (issue 1126, C2; the group check of 3cb7f74b347d)
mesh/delivery merged: source.main moved to novox/mesh-host@c3a2ffde6755
mesh/check being carried to 6502ba6ad6e6 on c3a2ffde6755 | from f6c0520f26b9
2026-10-28 10:47:48 +01:00
mesh-tools f6c0520f26 Merge pull request 'Node shadow: reuse a seal only at its slot, lay values as the controller does, newest candidate first, and hand the vault a drifted own secret (issue 1126, group fix/1126-seed-same)' (#158) from fix/1126-seed-same into main 2026-10-28 09:46:47 +00:00
jschoubben c912bcab6e The node agent finds a drifted own secret's running file by its id as the node runs it, its module's name first (issue 1126, cause C4; review round 2)
mesh/delivery merged: source.main moved to novox/mesh-host@f6c0520f26b9
mesh/delivery-group group fix/1126-seed-same merged: every member is merged
mesh/check checking c912bcab6e73 on f6c0520f26b9 | novox/mesh-host#158 in group fix/1126-seed-same
Pins mesh-sdk 3028c7e (delivery.OwnNeedName). The fixtures carry the id shape captured from novox's shadow answer.
2026-10-28 10:01:38 +01:00
jschoubben 63c0b445d7 Fill go.sum for mesh-sdk 1062b33, the group's head (issue 1126) 2026-10-28 09:53:34 +01:00
jschoubben 653ff61662 The node agent hands the vault what runs of an own secret its shadow answer says another value of, through move-in's replace_running, while it applies the controller's compositions (issue 1126, cause C4)
A new step, internal/ownresync, after each shadow answer: for each own secret masked as another value, while the node
applies the controller's compositions and runs the one its want names, it asks the vault's move-in with
replace_running: true and the running value sealed to the call's key, once per value, and says what it asked and what
the vault answered. The answer written to the node's have key carries values_differ. Pins the group's mesh-sdk head
1062b33 (go.sum is filled once that head is pushed).
2026-10-28 09:51:48 +01:00
jschoubben a7cd0a7807 Node shadow: reuse a running seal only at its own slot, lay values as the controller does, answer the newest candidate first and say how it goes (issue 1126, C1 C2, T1 T2 T3)
A box is reused by slot (resource, field) after the composition, from what runs or the
have's slots; a have written before slots is reused within its module alone. Config
values are laid by mesh-sdk's one rule (never a default). The shadow answers the
newest candidate first and hands each answer as it lands, says when an answer starts,
every minute what it waits on, and when it ends, and fits an answer to the node's key.
Needs mesh-sdk fix/1126-seed-same (16b8d8a1).
2026-10-28 09:41:29 +01:00
mesh-tools 7271cf151b Merge pull request 'Node shadow: what runs is the composition at the have's digest, not the stale kept copy (issue 1126, group fix/1126-shadow-bodies)' (#157) from fix/1126-shadow-bodies into main 2026-10-28 07:46:33 +00:00
jschoubben 82d0e125b2 What runs is the composition the have names, never the copy of the declaration kept before the switch (issue 1126, probe #532: shanks' xterm, zsh and its plugins answered differing at the very digest their haves say runs)
mesh/delivery merged: source.main moved to novox/mesh-host@7271cf151b3a
mesh/delivery-group group fix/1126-shadow-bodies merged: every member is merged
mesh/check checking 82d0e125b2c0 on 7271cf151b3a | novox/mesh-host#157 in group fix/1126-shadow-bodies
2026-10-28 01:42:17 +01:00
jschoubben a4c24a1979 Pin mesh-sdk 362a3f1: a refusal names what another node cannot run here, and is answered without a composition per module (issue 1126, probe #532) 2026-10-28 01:40:18 +01:00
mesh-tools cff69893a3 Merge pull request 'Node shadow: decides 'same value' by opening seals in memory, says when it is not ready or slow, carries the bus membership (issue 1126, group fix/1126-shadow-shows)' (#156) from fix/1126-shadow-shows into main 2026-10-28 00:21:51 +00:00
jschoubben 047237c911 The shadow shows its work: differing resources with masked bodies and whether a sealed value is the same, a minute not ready or a slow answer said in the journal and the answer; pin mesh-sdk 4e0c577 (issue 1126, seed #531)
mesh/delivery merged: source.main moved to novox/mesh-host@cff69893a361
mesh/delivery-group group fix/1126-shadow-shows merged: every member is merged
mesh/check pass 047237c911a4 on 449cda1b003e | unit 2980 run, 270 kept, 36 skipped | integration 2999 run, 302 kept, 29 skipped | over bound: unit (sl…
ace, g14 and shanks answered nothing to seed #531 and said nothing either. The
shadow now says, after a minute not ready, which inputs it has not read, and
answers each group it knows refused for that; an answer running over a minute
is said while it runs and noted in the answer. The answer's differing resources
carry both bodies, masked, the node saying whether two sealed values open to the
same value (Composer.sameValue: only the comparison leaves the node).
2026-10-28 01:11:55 +01:00
mesh-tools 449cda1b00 Merge pull request 'Node agent: a switch never marks the new run's modules, a failed read or a bus pause keeps what runs, stale composer marks clear (issue 1126, group fix/1126-switch-safe)' (#155) from fix/1126-switch-safe into main 2026-10-27 23:52:28 +00:00
jschoubben a6e53a4858 A composer's mark never covers another cause, and a run that ended marks and clears nothing (issue 1126, review of the switch, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@449cda1b003e
mesh/delivery-group group fix/1126-switch-safe merged: every member is merged
mesh/check pass a6e53a485813 on 199f6b48b1df | unit 2992 run, 1008 kept, 36 skipped | integration 3025 run, 1102 kept, 29 skipped | over bound: unit (…
NotComposed marked from a read have whatever its phase: an apply that failed,
naming the want's build and digest, became a composer's mark, which baseline
then took for what runs, and never applied again. It now marks only a have that
runs, says nothing, or is a composer's mark already. And its loop, whose reads
can take seconds, checks the run's context at each module, so a run that ended
writes no have.
2026-10-28 00:41:25 +01:00
jschoubben 4993f1b1d7 Make the switch between composers safe: a composer tells only its own run's agent, a pass whose run ended refuses nothing, a bus that pauses is tried again, a mark keeps what runs and clears at the next start; pin mesh-sdk 7359d18 (issue 1126)
Live 2026-10-28: the own run's last pass, failing on its run's end, marked
every module of the controller-mode agent that replaced it (ace 60, shanks 63),
and the marks dropped the build and digest because the have read failed in the
bus's reload pause. Now: the refusal hook is bound to the run's agent and ctx;
a pass whose context ended returns nothing; JetStream not enabled or no
responders are tried again after 1, 2 and 4 s before any mark; NotComposed reads
the have again while the bus does not answer and marks nothing it cannot read;
baseline takes a have a composer marked as what runs, so a mark it did not make
clears; each run's shadow lives with the run.
2026-10-28 00:34:10 +01:00
mesh-tools 199f6b48b1 Merge pull request 'Node agent: a failed placement read picks no owner, a deleted placement returns the node to the controller without a restart (issue 1126, group fix/1126-shadow-all)' (#154) from fix/1126-shadow-all into main 2026-10-27 22:44:12 +00:00
jschoubben 4636261435 A failing placement read chooses neither composer, a deleting head is compared with the controller's compositions at the node's wants, and a placement watch that ends is said; pin mesh-sdk e32d777 (issue 1126, review of the shadow, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@199f6b48b1df
mesh/delivery-group group fix/1126-shadow-all merged: every member is merged
mesh/check checking 463626143567 on bb15aab50eda | novox/mesh-host#154 in group fix/1126-shadow-all
selfComposer took a KVHistory error for 'not placed': at any restart a timeout
or a bus not ready flipped a self-composing node to the controller's
compositions for a whole run. selfcompose.Placed chooses only on a read that
succeeds (a missing placement state is one), and tries again otherwise, said
once. The shadow's answer to a head deleting the placement reads the
controller's composition at each want in the node's bucket and compares it with
what runs; a want with none refuses, without returning.
2026-10-27 23:37:59 +01:00
jschoubben 4e8fdfdf04 The way back: a node whose placement key is deleted returns to the controller's compositions without a restart, says so in its journal and its have, and answers such a candidate as returning; pin mesh-sdk 472d5f3 (issue 1126, the seed's revert)
The seed's revert (mesh-catalog #526) was refused on 'would stop running node',
while a self-composing node would have refused its placement's absence, kept
what runs and marked every have failed until a restart. endWhenUnplaced ends the
agent's run when the key goes, so it begins again on the controller's
compositions, as at a start without a key; the composer refuses nothing while
unplaced; the node's have says whose compositions it applies (Composes); the
shadow answers a deleted placement Returns, verdict same.
2026-10-27 23:25:59 +01:00
jschoubben d00926e089 Compose with what the node runs, so a folded module of an old-form build is carried as it runs, and bound an archive read; pin mesh-sdk 2fd5ffd (issue 1126, the seed's shadow)
novox refused its own composition and its answer on de-spiegel, a module it runs
folded into its own composition at a build only the controller holds. The
composer now gives frombus what the node runs (Scope.Running), so such a module
is carried as it runs. The shadow's answer names every refusal (mesh-sdk). A
templates archive read is bounded at 30 s: nothing else held an unanswered store.
2026-10-27 23:19:40 +01:00
mesh-tools bb15aab50e Merge pull request 'Node agent: compose its own modules from their builds and providers' served keys under its own grant (issue 1126, group fix/1126-node-creds)' (#153) from fix/1126-node-creds into main 2026-10-27 20:32:37 +00:00
jschoubben 6a4718b712 Hold in a unit table which haves count a module as run inside the node's own composition, so a failed have naming nothing is caught without H8's timing (issue 1126, review round 2)
mesh/delivery merged: source.main moved to novox/mesh-host@bb15aab50eda
mesh/delivery-group group fix/1126-node-creds merged: every member is merged
mesh/check checking 6a4718b712b2 on bb15aab50eda | novox/mesh-host#153 in group fix/1126-node-creds
2026-10-27 21:29:51 +01:00
jschoubben d5adf9b72b Drop two debugging lines from the build read and H8, and say why the refused read's wait stays above the history bound (issue 1126, review round 1) 2026-10-27 21:21:19 +01:00
jschoubben 7365890298 Prove in H8 that a node reads a placed module's build and a provider's served key through its own session, refused without the grant, and take a failed have naming nothing as no module run inside the node's own; pin mesh-sdk 7a8e9f9 (issue 1126, review round 1)
H8 found it: a module the node could not compose yet got a failed have naming
no fingerprint, which counted as run inside the node's own composition, so the
next composition took it from its manifest past its build.
2026-10-27 21:18:45 +01:00
jschoubben 0862f97972 Pin mesh-sdk 538ffba, the group's head (issue 1126) 2026-10-27 21:04:02 +01:00
jschoubben 45439fd742 Compose each module from its build and read what providers elsewhere serve, pinning mesh-sdk cebc651, so a self-composing node hands its moved modules and settles mailu's domain (issue 1126)
The node read a behind module's source manifest digest and composed every module
from the indexer's raw manifest; frombus now takes the build itself (Scope.KeptAt,
a fingerprint). The reader gives frombus the builder's key, a templates archive
fetched as every archive is (apply.Fetch) once per digest and checked against it,
and another node's served keys one by one, as the node's grant allows them.
2026-10-27 21:01:16 +01:00
jschoubben 0184e2a5b8 Watch the have bucket only where the node's grant allows, so a node composing itself is not refused by the bus (issue 1126)
The fold's watch of every have key ('>') was refused (permissions violation on
CONSUMER.CREATE KV_node-engine_have), and every self-composing agent stopped
after the seed. The composer now watches '*.node' (records) and '<node>.>' (its
own haves), as buscompose grants a node; the shadow watches '<node>.>' too. The
selfcompose tests check every watch and get against PermissionsFor(KindNode),
and the agent's integration tests run the agent as the node under that grant.
2026-10-27 09:27:54 +01:00
mesh-tools 50c229f905 Merge pull request 'The node agent folds what it runs at no new-form build into its own composition, judged from its haves, so the first composition after a restart keeps the bus (issue 1126)' (#152) from fix/1126-fold into main 2026-10-27 08:09:06 +00:00
jschoubben 69b845c0f7 Read what a node runs inside its own composition from its haves, not from what the agent applied, so a restarted agent's first composition keeps the bus (issue 1126, review of the fold round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@50c229f90568
mesh/delivery-group group fix/1126-fold merged: every member is merged
mesh/check checking 69b845c0f7b2 on 0d22776b20ef | novox/mesh-host#152 in group fix/1126-fold
A fresh agent composes before it has applied anything (Running empty), so
InOwn of Running folded only mesh-wireguard and the switch would drop nats.
The composer now watches the node's haves: a placed module whose have names no
new-form fingerprint is folded; one with a new-form have whose want is removed
is not folded. The shadow uses the same set, over the candidate's placement.
2026-10-27 09:03:59 +01:00
jschoubben caafd9306e Keep what the node runs inside its own composition there when it composes itself, so the seed's answer and the node's own composition keep the tunnel and the bus (issue 1126)
Pins mesh-sdk 05b67831, whose compositions fold a module no new-form build
names into the node's own. The shadow composed a placed module it runs inside
its own composition (nats, photos) at the trunk's build, as if the candidate
added it; shadowBuilds now leaves those to the fold, and both paths pass
frombus.InOwn of what runs.
2026-10-27 08:51:41 +01:00
mesh-tools 0d22776b20 Merge pull request 'The node agent answers a placement candidate in shadow, applies nothing, and keeps its own sealed values in its have (issue 1126, ADR 0559 §4)' (#151) from feat/1126-l2b into main 2026-10-27 07:21:19 +00:00
jschoubben 1803d25c3a Watch afresh and read from nothing at each shadow run, so a candidate deleted while it was down is not answered and no earlier watch stays open (issue 1126, review of L2b round 2)
mesh/delivery merged: source.main moved to novox/mesh-host@0d22776b20ef
mesh/delivery-group group feat/1126-l2b merged: every member is merged
mesh/check checking 1803d25c3a67 on 0d22776b20ef | novox/mesh-host#151 in group feat/1126-l2b
2026-10-27 02:48:54 +01:00
jschoubben d56c576e9a Begin the shadow composer again with a backoff when an input cannot be watched yet, and pin mesh-sdk's fixed head, so a node answers candidates for the whole link (issue 1126, review of L2b round 1) 2026-10-27 02:44:39 +01:00
jschoubben fbc6d6330f Extend H8 with a placement candidate the node answers same and applies nothing for, and pin mesh-sdk's group head (issue 1126) 2026-10-27 02:29:17 +01:00
jschoubben fb0cdabe25 Answer every placement candidate in shadow in the node's own key, applying nothing, so a placement change is checked by the nodes it places before it merges (issue 1126, ADR 0559 §4) 2026-10-27 02:26:07 +01:00
jschoubben 0474a2a3cb Keep the node's own composition's sealed values in its key and read a policy within the agent's context, so a restart applies nothing again (issue 1126) 2026-10-27 02:21:36 +01:00
mesh-tools 180a7e7b80 Merge pull request 'A directory used as found is not a module's health, and a failed have names its cause (issue 1496, ADR 0557 §4, §5)' (#150) from fix/1496-start-health into main 2026-10-26 21:38:02 +00:00
mesh-tools be94d2689e Merge pull request 'The node composes its own modules once the catalogue places it, and writes its record (issue 1126)' (#149) from feat/1126-nodes-compose into main 2026-10-26 21:11:36 +00:00
jschoubben b225c7bb4c A directory used as found is not a module's health, and a failed have says why (ADR 0557 §3, issue 1496)
mesh/delivery merged: source.main moved to novox/mesh-host@180a7e7b80e2
mesh/delivery-group group fix/1496-start-health merged: every member is merged
mesh/check checking b225c7bb4cf5 on be94d2689e2f | novox/mesh-host#150 in group fix/1496-start-health
On ace, 2026-10-26 21:19, mesh-build-node's cache directory was there before the
mesh and used as found. The health statement says such a directory unhealthy, and
the have folded it into the module's health as failed, with no words, while the
module's own check answered healthy; the assigner read it as its canary failing
and held the version on all four nodes.

The directory waits for the operator's hand-over and keeps its own condition,
raised from the same statement; it no longer counts in the module's have. A have
whose health is failed or waits and whose apply said nothing carries the words of
the resource that made it so, so the next one is readable.
2026-10-26 22:06:32 +01:00
jschoubben 1d44288514 Pin mesh-sdk 9796444 (l2-next: feat/1126-nodes-compose merged with sdk main #101, the standing-test race fixes) (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@be94d2689e2f
mesh/delivery-group group feat/1126-nodes-compose merged: every member is merged
mesh/check checking 1d442885148a on be94d2689e2f | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 22:05:41 +01:00
jschoubben af2d2a2c14 Pin mesh-sdk 5f45175 (l2-next: feat/1126-nodes-compose merged with sdk main) after merging main (issue 1126) 2026-10-26 21:59:21 +01:00
jschoubben 59d77dd457 Pin mesh-sdk feat/1126-nodes-compose 03dce12, the group's head: the checker refuses a pin that is neither trunk nor the group's head (issue 1126)
mesh/delivery-group group feat/1126-nodes-compose rejected: composed together: the checker's verdict at the members' heads (0d06ff386dce): ace
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 59d77dd457da on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 20:35:52 +01:00
jschoubben 23d66187a8 Review of L2, round 3: a have written with no composition keeps what the module's last have said runs (fingerprint, digest, sealed values), for every writer; a record module run nothing of since the start judges its move against its last have and waits for a yes (issue 1126, ADR 0547 §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 23d66187a881 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 20:29:07 +01:00
jschoubben 4476e58080 Review of L2, round 2: a refusal at the first compose after a start keeps what the module's have said runs (1); each have of a self-composing node carries the values its composition holds sealed, and the composer reuses them after a restart, so a rotated secret moves no composition at a restart (2); pin mesh-sdk dd9d9d5 (issue 1126, ADR 0547 §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 4476e5808047 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 20:23:16 +01:00
jschoubben 58de6aa10a Review of L2: a node that cannot compose says so in each wanted module's have, failed, while it keeps what it runs (H2); the composer keeps the box the node runs when it opens to the vault's value, so neither the switch nor a restart moves a composition (H1); pin mesh-sdk 3968849, whose node asks the vault only for a module declaring a secret (H8's failure) (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: ..test: ✔ a transient empty list for four passes retires nothing; five stable passes retire, data kept (290.408814ms) | 58de6…
2026-10-26 20:09:40 +01:00
jschoubben cdd3f3fd4f Pin mesh-sdk feat/1126-nodes-compose 9713c30: placement carries a settings-form module's values (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | cdd3f3fd4f63 on ff9386dcccc2 | unit 1245…
2026-10-26 19:40:41 +01:00
jschoubben 21d6583b5e The node agent begins again composing its own when the catalogue places its node, with no restart (issue 1126, ADR 0549 §4)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
2026-10-26 19:34:24 +01:00
jschoubben bb46a6ad0f The node composes again when the vault says a secret changed, so a rotated secret or credential reaches it at once (issue 1126, ADR 0547 §2)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestIntegrationLevelRunsAgainstARealStoreAndBusAndLeavesNothing: result = &{Level:integr…
2026-10-26 19:26:25 +01:00
jschoubben f8afaefba0 The node keeps the mesh's own certificate it applies, from its kept declaration at start, for its record; one agent test folded to hold the count (issue 1126, ADR 0555 §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the grou…
2026-10-26 19:24:36 +01:00
jschoubben 1e38c42646 The node's composer asks the vault for the pair credentials its composition reads, as consumer and provider, sealed to its key (issue 1126, ADR 0550 §5)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the g…
2026-10-26 19:21:22 +01:00
jschoubben 1241ee7041 Pin mesh-sdk 4d5f430: the node's wants say it composes them (Self) (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk f1d7c7377dc4, which is neither on mesh-sdk's trunk nor the g…
2026-10-26 19:14:38 +01:00
jschoubben 2dc384d641 The node's composer asks the vault's own for its modules' secrets sealed to its key, and keeps a value first had while its generation stands; two agent tests folded to hold the count (issue 1126, ADR 0550 §5)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 2dc384d641a2 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 19:05:12 +01:00
jschoubben e4dbb49d2f Integration: a node the placement names composes and applies its own module with no controller composition, and again on a changed config value (issue 1126, ADR 0547 §2)
mesh/delivery-group group feat/1126-nodes-compose checking: 4 of 5 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking e4dbb49d2fa5 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 18:51:36 +01:00
jschoubben 02d9d7b053 The node composes its own modules once the catalogue places it: one composer per node drains a mark set by any input's change, the agent applies what it composed when it differs, and a record move waits for a yes (issue 1126, ADR 0547 §2, §3)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 02d9d7b0538f on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 18:44:58 +01:00
jschoubben 7fc0ea5478 Pin mesh-sdk feat/1126-nodes-compose 9789c83 (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 7fc0ea5478a3 on ff9386dcccc2 | novox/mesh-host#149 in group feat/1126-nodes-compose
2026-10-26 18:34:35 +01:00
jschoubben 8be1c2ab0d The node agent writes what the machine observes of itself as its record in its own have key, only when it changed (issue 1126, ADR 0555 §3) 2026-10-26 18:32:54 +01:00
mesh-tools ff9386dccc Merge pull request 'The node-engine runs yielding processes in one capped idle slice (issue 1806, ADR 0553/0554, change A)' (#148) from fix/build-cpu-yields into main 2026-10-26 14:22:54 +00:00
mesh-tools bba9081ca1 Merge pull request 'The node-engine moves a running daemon's build aside instead of deleting it (issue 734)' (#147) from fix/daemon-dir-kept into main 2026-10-26 14:16:00 +00:00
jschoubben 65cd7b4787 Retire a daemon's running build instead of deleting it before the restart (issue 734)
mesh/delivery merged: source.main moved to novox/mesh-host@bba9081ca127
mesh/delivery-group group fix/daemon-dir-kept merged: every member is merged
mesh/check pass 65cd7b47873a on bba9081ca127 | carried from 94f9aa298cb8, mesh-sdk 21252194d530 | unit 971 run, 423 kept, 32 skipped | integration 976…
A process no witness judges had its directory removed before the new bundle
was unpacked and the unit restarted, so the old process ran on in a deleted
working directory: mesh-build-go's `go version` probe failed with "getwd: no
such file or directory" and refused three checks on ace on 2026-10-26, each in
the second a new build of it arrived.

Every process is now replaced the way a judged one is: its build is renamed
under .witness/<name>/retired/ and deleted only once nothing runs from it.
"Runs from it" now counts a process's working directory as well as its image,
so the children a daemon starts keep the directory too. Forget keeps a retired
build that still drains, and the witness's sweep covers every process it keeps
a directory for, judged or not.

Two unit-file tests are folded into one, so the unit count goes down by one.
2026-10-26 15:01:55 +01:00
jschoubben a78239b2aa Pin mesh-sdk at fix/build-cpu-yields' head (ADR 0553, ADR 0554)
mesh/delivery merged: source.main moved to novox/mesh-host@ff9386dcccc2
mesh/delivery-group group fix/build-cpu-yields merged: every member is merged
mesh/check being carried to a78239b2aa05 on ff9386dcccc2 | from bba9081ca127, mesh-catalog 9a8e54d988fe
2026-10-26 14:47:02 +01:00
jschoubben 2ac0c4686a The node-engine holds the node's one build CPU share as its own config (ADR 0553 §3, round 2)
The share belongs to the part that enforces it: build-cpu-share, default 50,
set per node through the node's settings layer and read by the composer for
the build seats' runtimes. Pin mesh-sdk at fix/build-cpu-yields' head.
2026-10-26 14:40:24 +01:00
jschoubben 5233f7584d Pin mesh-sdk at fix/build-cpu-yields' head (ADR 0553) 2026-10-26 14:23:13 +01:00
jschoubben 8858dba25e Merge main into fix/build-cpu-yields 2026-10-26 14:22:43 +01:00
jschoubben 9254180cab A yielding process runs in the node's one yielding slice, which holds the shared quota (ADR 0553, issue 1806)
Review round 1: one quota per unit let three build modules take three shares.
The engine now writes meshyield.slice, top-level so its idle weight binds
against the operator's session, with the node's share as its CPUQuota; each
yielding unit joins it at nice 19. Comments say the IO controls bind only under
bfq or io.cost, not under the none scheduler NVMe disks use.
2026-10-26 14:21:24 +01:00
mesh-tools 94f9aa298c Merge pull request 'The node-engine writes the followed user list into the bus module's have even when it does not apply that module (issue 1945)' (#146) from fix/have-keeps-user-list into main 2026-10-26 12:52:15 +00:00
jschoubben ee9fcaba20 Pin mesh-sdk at fix/build-cpu-yields' head (issue 1806) 2026-10-26 13:49:41 +01:00
jschoubben 43f2d6020e Write the bus's user list into a have no apply here says, merged over the key's value
mesh/delivery merged: source.main moved to novox/mesh-host@94f9aa298cb8
mesh/check pass 43f2d6020e11 on 94f9aa298cb8 | carried from 7f04d1204765, mesh-catalog a4f5df797b68, mesh-sdk b4cd95e7c4a4 | unit 972 run, 423 kept, 3…
On the bus node the bus module has no want and has left the declaration, so no
pass of this engine held its fact and the follower's user list was kept in
memory and never written: the vault and the grants part read no hashes and no
key. A module's user list now reaches its key on every pass, merged into what
the key says with every other field kept; a missing key is said once, never
made up (ADR 0547 §8).
2026-10-26 13:38:40 +01:00
jschoubben 6250b1274b A process that yields is written at idle priority under its share of the machine (issue 1806)
A build seat's runtime took 1650% of shanks' 24 processors and slowed the
operator's desktop. The unit of a process declared with yields now carries
CPUWeight=idle, IOWeight=1, Nice=19, the lowest best-effort IO level, and
CPUQuota for its share of the processors this machine has.
2026-10-26 13:37:48 +01:00
mesh-tools 7f04d12047 Merge pull request 'Bus node: follow the grants part's sealed user list after the operator's take-over, never both sources; hand the vault the loaded hashes (issue 1945)' (#145) from feat/1945-user-list-switch into main 2026-10-26 12:21:55 +00:00
jschoubben e88ddab241 An apply of the bus module waits when the list on the machine cannot be read while the grants part's is followed, so the file is never removed as an orphan (review of the switch, issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-host@7f04d1204765
mesh/delivery-group group feat/1945-user-list-switch merged: every member is merged
mesh/check being carried to e88ddab24170 on 7f04d1204765 | novox/mesh-host#145 in group feat/1945-user-list-switch
2026-10-26 12:54:06 +01:00
jschoubben 06f70e11ee Bring in the adoption handing; the bus node writes nothing before it heard the switch, retries the key's watch and keeps the follow last heard, and an apply never writes the controller's list while the grants part's is followed (review of the switch, issue 1945)
Merges fix/1945-adopt-hashes 1a190090. A new follower (one per link) waits for the grants part's key to catch up before
its first write, so a restart after take-over never writes the controller's list first. A refused or ended watch is
tried again on link.Backoff, the have says why it is not watched, and the key last heard, kept beside the node's state,
holds meanwhile. While the grants part's list is followed, the bus module's composition is applied with the list on the
machine in place of the controller's. The grants list is never handed to the vault. Pins mesh-sdk c2a410bb.
2026-10-26 12:30:07 +01:00
jschoubben 61cec66ba8 gofmt the follower's test (issue 1945) 2026-10-26 12:08:37 +01:00
jschoubben 5e2fadce15 Pin mesh-sdk at the group's head 83ee7769 (issue 1945) 2026-10-26 12:07:10 +01:00
jschoubben 882e75c954 The bus node follows the grants part's sealed user list from take-over, the controller's otherwise, never both (ADR 0547 clauses 7, 8; issue 1945)
On the node whose compositions carry the bus's list, the engine also watches mesh-grants' key user-list.sealed. While it
says follow grants, the object it names is read by name, opened with this machine's sealing key, checked against the
key's digest and the guard, and written to the same file at 0600 only when its digest differs; the bus reloads it
itself. The have says which list is followed and this machine's sealing key, which the grants part seals to.
The follower's test waits for its watch to catch up before a later put, which raced under load. Pins mesh-sdk 175a5567.
2026-10-26 12:06:17 +01:00
73 changed files with 12536 additions and 417 deletions
+107
View File
@@ -0,0 +1,107 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
)
// appliedStore keeps every composition the agent applied, one file per module (the node's own as `node.json`), in the
// directory `applied` beside the node's state (agent.Env.Applied, issue 774). It is the machine's own record of what runs:
// read at the agent's start before anything is watched, so the node never composes its own as if a module it runs ran
// nothing because the bucket no longer keeps that composition or the kept declaration is retired. Files are 0600 in a
// 0700 directory: a composition carries values sealed to this machine.
type appliedStore struct{ dir string }
func appliedStoreFor(statePath string) appliedStore {
return appliedStore{dir: filepath.Join(filepath.Dir(statePath), "applied")}
}
// appliedName is a module name the store takes as a file name: a catalogue module's or the node's own, never a path.
var appliedName = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]*$`)
func (s appliedStore) path(module string) (string, error) {
if !appliedName.MatchString(module) || strings.Contains(module, "..") {
return "", fmt.Errorf("%q is not a module name the applied store takes", module)
}
return filepath.Join(s.dir, module+".json"), nil
}
// holds says whether the store keeps the module's composition.
func (s appliedStore) holds(module string) bool {
path, err := s.path(module)
if err != nil {
return false
}
_, err = os.Stat(path)
return err == nil
}
func (s appliedStore) Load() (map[string][]byte, error) {
entries, err := os.ReadDir(s.dir)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
out := map[string][]byte{}
var errs []error
for _, e := range entries {
module, ok := strings.CutSuffix(e.Name(), ".json")
if !ok || e.IsDir() || !appliedName.MatchString(module) {
continue
}
raw, err := os.ReadFile(filepath.Join(s.dir, e.Name()))
if err != nil {
errs = append(errs, err)
continue
}
out[module] = raw
}
return out, errors.Join(errs...)
}
func (s appliedStore) Keep(module string, raw []byte) error {
path, err := s.path(module)
if err != nil {
return err
}
if was, err := os.ReadFile(path); err == nil && string(was) == string(raw) {
return nil
}
if err := os.MkdirAll(s.dir, 0o700); err != nil {
return err
}
tmp, err := os.CreateTemp(s.dir, "."+module+".*")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
if _, err := tmp.Write(raw); err != nil {
tmp.Close()
return err
}
if err := tmp.Sync(); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
return os.Rename(tmp.Name(), path)
}
func (s appliedStore) Drop(module string) error {
path, err := s.path(module)
if err != nil {
return err
}
if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
return nil
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// authorityPath is where this engine keeps the mesh's own certificate as it last applied it (public): beside the
// certificate issued for the machine's name, read into its record (novox/hq ADR 0555 §3).
func authorityPath(state string) string { return filepath.Join(filepath.Dir(state), "authority.crt") }
// keepAuthority keeps the mesh's own certificate a composition carries (the resource a module's certificate.authority
// asks for, catalogue.AuthorityID), so the record says the authority this machine holds. A composition carrying none
// changes nothing.
func keepAuthority(state string, c delivery.Composed) error {
for _, raw := range c.Resources {
var r struct {
ID string `json:"id"`
Content string `json:"content"`
}
if json.Unmarshal(raw, &r) != nil || r.Content == "" || !strings.HasSuffix(r.ID, "."+catalogue.AuthorityID()) {
continue
}
path := authorityPath(state)
if was, err := os.ReadFile(path); err == nil && string(was) == r.Content {
return nil
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
return os.WriteFile(path, []byte(r.Content), 0o644)
}
return nil
}
+173
View File
@@ -0,0 +1,173 @@
package main
import (
"context"
"crypto/ed25519"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"os"
"path/filepath"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-host/internal/identity"
)
// Issue 1126 (ADR 0555 §3): the mesh's own certificate a composition carries is kept for the machine's record, and the
// record's keys read it; a composition carrying none changes nothing. And (L3 s6, s7; ADR 0550 §2) the record says what
// the node trusts a mesh signature from, and the node keeps the certificate the vault signed for it.
func TestTheAppliedAuthorityIsKeptForTheRecord(t *testing.T) {
state := filepath.Join(t.TempDir(), "state", "state.json")
res := func(id, content string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": id, "type": "file", "path": "/etc/x", "content": content})
return raw
}
if err := keepAuthority(state, delivery.Composed{Resources: []json.RawMessage{res("web.conf", "x")}}); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(authorityPath(state)); !os.IsNotExist(err) {
t.Fatalf("a composition with no authority kept one: %v", err)
}
if err := keepAuthority(state, delivery.Composed{Resources: []json.RawMessage{res("ca-trust.certificate-authority", "CA-1")}}); err != nil {
t.Fatal(err)
}
if got := recordKeys(state, nil)(); got.Authority != "CA-1" {
t.Errorf("the record's keys read the authority %q", got.Authority)
}
// L3 s6: before the vault's signers are read the node trusts the controller's key alone; once read, a message the
// vault signed is accepted and the record names both; once the lease is handed away the controller's key is refused
// and the record names the vault's alone. Each change tells the record's writer.
controllerPub, controller, _ := ed25519.GenerateKey(nil)
vaultPub, vault, _ := ed25519.GenerateKey(nil)
trust := newMeshTrust(controllerPub)
told := 0
trust.onChange(func() { told++ })
keys := recordKeys(state, trust.Trusts)
ctrlFP, vaultFP := delivery.SignerFingerprint(controllerPub), delivery.SignerFingerprint(vaultPub)
if got := keys().Trusts; !delivery.TrustsKey(got, delivery.SignerController, ctrlFP) || delivery.TrustsKey(got, delivery.SignerVault, vaultFP) {
t.Errorf("before the signers are read the record trusts %v", got)
}
enrol, _ := delivery.SignContext(delivery.KindEnrolment)
msg := []byte("a reply")
if _, err := trust.Verify(delivery.KindEnrolment, msg, delivery.SignMessage(vault, enrol, msg)); err == nil {
t.Error("a vault-signed message is accepted before the vault's key is read")
}
vaultCA := testAuthority(t, vaultPub, vault)
keyValue, _ := json.Marshal(delivery.NewSignerKey(vaultPub, time.Now()))
fp, _ := delivery.AuthorityFingerprint(vaultCA)
authValue, _ := json.Marshal(delivery.SignerAuthority{Certificate: vaultCA, Fingerprint: fp, Made: time.Now()})
trust.heardSigner(delivery.SignersKeyKey, keyValue, false, func(string) {})
trust.heardSigner(delivery.SignersAuthorityKey, authValue, false, func(string) {})
if s, err := trust.Verify(delivery.KindEnrolment, msg, delivery.SignMessage(vault, enrol, msg)); err != nil || s.Name != delivery.SignerVault {
t.Errorf("a vault-signed message on a node that read signers: %+v, %v", s, err)
}
if got := keys().Trusts; !delivery.TrustsKey(got, delivery.SignerController, ctrlFP) || !delivery.TrustsKey(got, delivery.SignerVault, vaultFP) ||
trust.VaultAuthority() != vaultCA || told != 2 {
t.Errorf("after the signers are read the record trusts %v (told %d)", got, told)
}
handed, _ := json.Marshal(buscompose.LeaseHolder{HandedTo: "mesh-vault", HandedBy: "person.operator"})
trust.heardLease(handed, true, nil)
if _, err := trust.Verify(delivery.KindEnrolment, msg, delivery.SignMessage(controller, enrol, msg)); err == nil || trust.Controller() != nil {
t.Error("after the lease is handed away the controller's key is still trusted")
}
if got := keys().Trusts; delivery.TrustsKey(got, delivery.SignerController, ctrlFP) || !delivery.TrustsKey(got, delivery.SignerVault, vaultFP) || told != 3 {
t.Errorf("after the lease is handed away the record trusts %v (told %d)", got, told)
}
trust.heardLease(handed, true, nil)
if told != 3 {
t.Error("a lease heard again unchanged wrote the record again")
}
// A signers' key naming another fingerprint is not trusted.
bad, _ := json.Marshal(delivery.SignerKey{Key: base64.StdEncoding.EncodeToString(controllerPub), Fingerprint: vaultFP})
trust.heardSigner(delivery.SignersKeyKey, bad, false, func(string) {})
if !delivery.TrustsKey(keys().Trusts, delivery.SignerVault, vaultFP) {
t.Error("a signers' key that does not check replaced the vault's")
}
// L3 s7: the certificate the vault signs for this node's internal name and serving key is kept where the record reads
// it, and the record is written again; one signed by another authority is refused and nothing is kept.
serving, err := identity.GenerateServingKey()
if err != nil {
t.Fatal(err)
}
if err := identity.WriteServingKey(identity.ServingKeyPath(state), serving); err != nil {
t.Fatal(err)
}
servingPub, _ := base64.StdEncoding.DecodeString(serving.Public)
signedBy := func(ca string, key ed25519.PrivateKey) string {
parent, _ := delivery.ParseCertificate(ca)
der, err := x509.CreateCertificate(nil, delivery.NodeCertificateTemplate(overlay.InternalName("n1"), servingPub, time.Now().Add(-time.Hour)),
parent, ed25519.PublicKey(servingPub), key)
if err != nil {
t.Fatal(err)
}
return string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))
}
var asked []any
answer := signedBy(vaultCA, vault)
ask := func(_ context.Context, verb string, args any) (json.RawMessage, error) {
asked = append(asked, verb, args)
return json.Marshal(delivery.SignAnswer{Kind: delivery.KindNodeCertificate, Node: "n1", Certificate: answer})
}
written := 0
if err := certifyThisNode(context.Background(), state, "n1", trust, ask, func() { written++ }); err != nil {
t.Fatal(err)
}
if got := recordKeys(state, nil)(); got.Certificate != answer || written != 1 || len(asked) != 2 || asked[0] != delivery.SignVerb {
t.Errorf("the vault's certificate: kept %v, written %d, asked %v", got.Certificate == answer, written, asked)
}
answer = signedBy(testAuthority(t, controllerPub, controller), controller)
if err := certifyThisNode(context.Background(), state, "n1", trust, ask, func() { written++ }); err == nil || written != 1 {
t.Errorf("a certificate of another authority was kept: %v", err)
}
// Issue 774: what the engine applied is kept beside the state too, one file per module, 0600 in a 0700 directory,
// written by a rename so a crash leaves the one before; a name that is no module's is refused, never a path.
t.Run("the compositions the engine applied are kept beside the state", func(t *testing.T) {
applied := appliedStoreFor(filepath.Join(t.TempDir(), "state", "state.json"))
if m, err := applied.Load(); err != nil || len(m) != 0 || applied.holds("node") {
t.Fatalf("an empty store loads %v, %v", m, err)
}
for _, tc := range []struct {
module string
takes bool
}{{"node", true}, {"nats", true}, {"mesh-delivery-view", true}, {"../state", false}, {"a/b", false}, {"..", false}, {"", false}} {
if err := applied.Keep(tc.module, []byte(`{"v":1}`)); (err == nil) != tc.takes {
t.Errorf("keeping %q: %v, want taken %v", tc.module, err, tc.takes)
}
}
if err := applied.Keep("node", []byte(`{"v":2}`)); err != nil {
t.Fatal(err)
}
entries, _ := os.ReadDir(applied.dir)
dirInfo, _ := os.Stat(applied.dir)
fileInfo, _ := os.Stat(filepath.Join(applied.dir, "node.json"))
if len(entries) != 3 || dirInfo.Mode().Perm() != 0o700 || fileInfo.Mode().Perm() != 0o600 {
t.Fatalf("the store holds %d files (want 3, no temporary one left), directory %v, file %v", len(entries), dirInfo.Mode(), fileInfo.Mode())
}
if err := applied.Drop("nats"); err != nil || applied.Drop("nats") != nil || applied.holds("nats") {
t.Fatalf("a dropped module is still held, or dropping twice fails: %v", err)
}
if m, err := applied.Load(); err != nil || string(m["node"]) != `{"v":2}` || len(m) != 2 {
t.Fatalf("the store loads %v, %v", m, err)
}
})
}
// testAuthority is a self-signed authority certificate for key, as the vault makes its own.
func testAuthority(t *testing.T, pub ed25519.PublicKey, key ed25519.PrivateKey) string {
t.Helper()
tpl := delivery.AuthorityTemplate(pub, time.Now().Add(-time.Hour))
der, err := x509.CreateCertificate(nil, tpl, tpl, pub, key)
if err != nil {
t.Fatal(err)
}
return string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))
}
+277
View File
@@ -0,0 +1,277 @@
package main
import (
"bufio"
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"sync/atomic"
"syscall"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
meshasker "git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-host/internal/accounts"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The node seat node-engine's judgements on this machine (novox/hq ADR 0563): what a hand-over and a library's take
// would set and what the disk holds now, and whether this machine is root-free.
// diskOf is what is on disk at a directory, as an act compares it when the operator's yes arrives: its owner and
// group, its mode, and which file it is.
func diskOf(info os.FileInfo) string {
owner := "unknown"
if uid, gid, ok := apply.OwnerOf(info); ok {
owner = fmt.Sprintf("%d:%d", uid, gid)
}
which := ""
if st, ok := info.Sys().(*syscall.Stat_t); ok {
which = fmt.Sprintf(" %v:%v", st.Dev, st.Ino)
}
return fmt.Sprintf("%s %o%s", owner, info.Mode().Perm(), which)
}
// handOverJudgeFrom judges a hand-over through the node seat as the controller's ask is judged (judgeFound): the
// declared owner and mode it will set, and the disk as it is.
func handOverJudgeFrom(statePath string) func(string) (link.Judged, error) {
return func(path string) (link.Judged, error) {
found, info, err := judgeFound(statePath, path)
if err != nil {
return link.Judged{}, err
}
return link.Judged{Subject: path, Path: path, Sets: orUnsaid(found.Declared), Disk: diskOf(info),
Words: "A directory of " + moduleOf(found.ID) + " is used as found: the mesh left its owner and mode."}, nil
}
}
func moduleOf(id string) string {
if at := strings.LastIndex(id, "."); at > 0 {
return id[:at]
}
return id
}
// accessJudgeFrom judges the take act on a library through the node seat (ADR 0328 §7, ADR 0563 §4): declared on this
// machine, its home, and its root holding data the mesh has not taken; what the take sets, and the disk as it is.
func accessJudgeFrom(statePath string) func(string, string) (link.Judged, error) {
return func(act, access string) (link.Judged, error) {
d, err := declaredNow(statePath, nil)
if errors.Is(err, store.ErrNothingDeclared) {
return link.Judged{}, errors.New("this node keeps no declaration, so it is declared no library")
}
if err != nil {
return link.Judged{}, fmt.Errorf("what runs on this node does not read: %w", err)
}
for _, l := range d.Libraries {
if apply.LibraryAccess(l.Library) != access {
continue
}
if !l.Home {
return link.Judged{}, fmt.Errorf("this machine is not the home of the library %s, and a library is "+
"taken on its home", l.Library)
}
ctx, cancel := context.WithTimeout(context.Background(), lookBound)
defer cancel()
if v := apply.JudgeLibrary(ctx, l); v.Kind != apply.VerdictWaitKeep {
return link.Judged{}, fmt.Errorf("the library %s does not wait for the operator's word to be kept "+
"(it is judged %s), so there is nothing to take", l.Library, v.Kind)
}
info, err := os.Lstat(l.Root)
if err != nil {
return link.Judged{}, fmt.Errorf("the library %s's root cannot be read: %v", l.Library, err)
}
return link.Judged{Subject: access, Path: filepath.Clean(l.Root),
Sets: "the group " + l.Group + ", group write and setgid on everything under its root",
Disk: diskOf(info), Words: "The library " + l.Library + " holds data the mesh has not taken."}, nil
}
return link.Judged{}, fmt.Errorf("the access %s names no library declared on this machine", access)
}
}
// askTakes asks the take of every library the look found waiting for the operator's word, once per library (ADR 0563
// §4): asked by the engine itself, never a second time while one is open or after one was asked; access-act asks again
// by hand.
func askTakes(acts *link.HandActs, waiting []string, say func(string)) {
if acts == nil {
return
}
for _, access := range waiting {
if acts.AskedFor(link.ActAccess, access) {
continue
}
if _, err := acts.AskAccess(link.EngineSeat+"-"+acts.Node, link.ActTake, access,
"its root holds data outside its group", true); err != nil && say != nil {
say("the take of " + access + " could not be asked: " + err.Error())
}
}
}
// lastAccounts is the newest look at every account (the health loop's), which root-free judges from.
var lastAccounts atomic.Pointer[accounts.Statement]
// rootFreshFor is how old the verdict root-free judges from may be (ADR 0563 §6, the controller's verdictFreshFor).
const rootFreshFor = 15 * time.Minute
// judgeRootFree is this machine's root-free (ADR 0259 §8, ADR 0563 §6): free only on the engine's own verdict, at most
// rootFreshFor old, that every account declared never to become root without a person cannot, at least one such
// account declared, and the login shell's execute not served here. Anything else is no pass, and says why.
func judgeRootFree(node string, st *accounts.Statement, now time.Time, served bool, servedWhy string) link.RootVerdict {
v := link.RootVerdict{Machine: node, Judged: now.UTC()}
var not []string
confined := ""
switch {
case st == nil:
not = append(not, "this machine's engine has judged no account yet")
case now.Sub(st.At) > rootFreshFor:
not = append(not, fmt.Sprintf("the engine's newest judgement of its accounts was made at %s, more than %d "+
"minutes ago, and a verdict that old is not a verdict on now", st.At.UTC().Format(time.RFC3339),
int(rootFreshFor.Minutes())))
default:
var named []string
for _, a := range st.Accounts {
if !a.Root {
continue
}
named = append(named, a.Name)
if a.State != accounts.Healthy {
not = append(not, "the account "+a.Name+" "+orNoneKnownWord(a.Reason))
}
}
if len(named) == 0 {
not = append(not, "no account here is declared never to become root, so agents run as the operator's "+
"account, which may become root")
} else if len(not) == 0 {
confined = "the account " + strings.Join(named, ", ") + " cannot become root without a person (judged " +
st.At.UTC().Format(time.RFC3339) + ")"
}
}
if served {
not = append(not, "the login shell runs any command an agent gives it as the machine's runtime account, which "+
"can become root ("+servedWhy+")")
}
if len(not) > 0 {
v.Why = strings.Join(not, "; ")
return v
}
v.Free, v.Why = true, confined+"; the login shell's execute is not served here"
return v
}
func orNoneKnownWord(s string) string {
if s == "" {
return "is not judged"
}
return s
}
// settingFileSuffix is the resource the login shell's holder renders its setting execute into (ADR 0268: zsh's
// tools-conf, `execute=serve` or `execute=withhold`), read beside the bus's discovery.
const settingFileSuffix = ".tools-conf"
// executeServedHere is whether the login shell's execute is served on this machine (ADR 0268), failing closed: served
// while the bus's discovery hears it answered here or could not be asked, or while the holder's setting file applied
// here says anything but withhold or cannot be read. why says which.
func executeServedHere(statePath string, heard func() (bool, error)) (bool, string) {
var why []string
if heard == nil {
why = append(why, "the bus could not be asked whether execute is answered here")
} else if h, err := heard(); err != nil {
why = append(why, "the bus could not be asked whether execute is answered here: "+err.Error())
} else if h {
why = append(why, "the bus hears execute answered here")
}
known, err := store.Load(statePath)
if err != nil {
why = append(why, "what this machine applied could not be read: "+err.Error())
}
for _, r := range known.Resources {
if r.Type != "file" || !strings.HasSuffix(r.ID, settingFileSuffix) {
continue
}
if value, err := settingIn(r.Target); err != nil {
why = append(why, moduleOf(r.ID)+"'s setting execute cannot be read: "+err.Error())
} else if value != "withhold" {
why = append(why, moduleOf(r.ID)+"'s setting execute here is "+value)
}
}
return len(why) > 0, strings.Join(why, "; ")
}
// settingIn is the value of execute in a setting file, "" when it says none.
func settingIn(path string) (string, error) {
f, err := os.Open(path)
if err != nil {
return "", err
}
defer f.Close()
value := ""
lines := bufio.NewScanner(f)
for lines.Scan() {
t := strings.TrimSpace(lines.Text())
if key, v, ok := strings.Cut(t, "="); ok && !strings.HasPrefix(t, "#") && strings.TrimSpace(key) == "execute" {
value = strings.TrimSpace(v)
}
}
if value == "" {
value = "not said"
}
return value, lines.Err()
}
// keepWaiting is every library a look found waiting for the operator's word to keep it (ADR 0328 §8).
func keepWaiting(vs []link.ResourceHealth) []string {
var out []string
for _, v := range vs {
for _, w := range v.AccessWaits {
if w.Reason == link.WaitKeep {
out = append(out, w.Access)
}
}
}
return out
}
// engineActs is the node seat this engine holds (ADR 0563): judged and performed by the same code as the controller's
// signed asks, its asks to the operator over the session's link, its acts kept beside the state.
func engineActs(opts options, node string, queue *link.Session, search *accounts.SetuidCache) *link.HandActs {
state := link.BusState{Session: queue, Node: node}
h := &link.HandActs{Node: node, State: state,
JudgeHandOver: handOverJudgeFrom(opts.state), HandOver: handOverFrom(opts.state),
JudgeAccess: accessJudgeFrom(opts.state), AccessAct: accessActFrom(opts.state, opts.timeout),
JudgeDirectories: judgeDirectoriesFrom(opts.state), SetuidSearch: queue.SetuidSearch, Searching: search.Running,
RootFree: func() link.RootVerdict {
served, why := executeServedHere(opts.state, func() (bool, error) {
return queue.HeardServed("node-login-shell", "execute")
})
return judgeRootFree(node, lastAccounts.Load(), time.Now(), served, why)
},
Say: queue.SayEngine}
asker := asks.EngineAsker(node)
h.Asker = &meshasker.Client{Module: asker, State: state, Publish: queue.PublishAsk,
Record: func(id string) (*stdio.StateEntry, error) { return queue.RouterRecord(asker + "." + id) }}
return h
}
// reconcileEvery is how often the engine catches up its open asks from the router's records (a warrant heard while no
// link was open, an ask past its expiry): besides each link's opening.
const reconcileEvery = 5 * time.Minute
func reconcileActs(ctx context.Context, h *link.HandActs) {
t := time.NewTicker(reconcileEvery)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
h.Reconcile()
}
}
}
+73
View File
@@ -2,6 +2,7 @@ package main
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
@@ -10,6 +11,7 @@ import (
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/link"
)
@@ -82,6 +84,77 @@ func TestADeclarationWithAFileByDigestAppliesFromTheObject(t *testing.T) {
if err != nil || string(got) != string(body) {
t.Errorf("1695/AC1: the file holds %d bytes (%v), want %d", len(got), err, len(body))
}
// Issue 924 (novox, 2026-10-29 01:16): the controller's sweep removed an object a composition the node ran still named,
// and the module's apply failed while it ran. Content the node once had is its own keep's, and the bus is not asked.
gone := aFilesLink{err: fmt.Errorf("the object %s was deleted from the bucket declaration-files-novox", name)}
never := []byte(strings.Repeat("never had here\n", 6000))
for _, row := range []struct {
name string
raw []byte
onDisk []byte // nil removes the file
refused string // "" applies
}{
{"the file here drifted, its object deleted", raw, []byte("the module wrote it since"), ""},
{"the file here is gone, its object deleted", raw, nil, ""},
{"never had here, its object deleted", referenceDeclaration(target, never), nil, "never held it"},
} {
if row.onDisk == nil {
_ = os.Remove(target)
} else if err := os.WriteFile(target, row.onDisk, 0o600); err != nil {
t.Fatal(err)
}
report = applyAndKeep(link.WithFiles(context.Background(), gone), opts, row.raw, nil, nil, nil)
if row.refused == "" {
got, _ := os.ReadFile(target)
if report.Refused != "" || string(got) != string(body) {
t.Errorf("924: %s: refused %q, the file holds %d bytes; want it applied from the node's own keep",
row.name, report.Refused, len(got))
}
} else if !strings.Contains(report.Refused, row.refused) || !strings.Contains(report.Refused, "was deleted") {
t.Errorf("924: %s: refused %q; want a refusal saying the bucket and the keep both lack it", row.name, report.Refused)
}
}
// Issue 924, the live case: what runs came from the kept declaration, whose export names a digest swept long ago,
// while the node wrote a newer export since. Composed as it runs, the file names what the node wrote, and applies with
// no object; a file someone else changed since is left as named.
newer := []byte(strings.Repeat("the export the node wrote since\n", 3000))
newerName, _ := declaration.ObjectName(declaration.DigestOf(newer))
ctx = link.WithFiles(context.Background(), aFilesLink{objects: map[string][]byte{newerName: newer}})
if report = applyAndKeep(ctx, opts, referenceDeclaration(target, newer), nil, nil, nil); report.Refused != "" {
t.Fatalf("924: the newer export: %s", report.Refused)
}
stale := json.RawMessage(`{"id":"m.big","type":"file","path":"` + target + `","digest":"` + declaration.DigestOf(body) +
`","size":` + strconv.Itoa(len(body)) + `}`)
for _, row := range []struct {
name string
edit []byte // what someone else wrote there since; nil for nobody
digest string
}{
{"the file is what the node wrote", nil, declaration.DigestOf(newer)},
{"someone changed the file since", []byte("a hand edit"), declaration.DigestOf(body)},
} {
if row.edit != nil {
if err := os.WriteFile(target, row.edit, 0o600); err != nil {
t.Fatal(err)
}
}
runs := runsAsWritten([]delivery.Composed{{Node: "n", Module: "m", Resources: []json.RawMessage{stale}}}, opts.state)
var got struct {
Digest string `json:"digest"`
}
_ = json.Unmarshal(runs[0].Resources[0], &got)
if got.Digest != row.digest {
t.Errorf("924: %s: what runs names %s, want %s", row.name, got.Digest, row.digest)
}
if row.edit == nil {
raw := []byte(`{"declaration":1,"resources":[` + string(runs[0].Resources[0]) + `]}`)
if report = applyAndKeep(link.WithFiles(context.Background(), gone), opts, raw, nil, nil, nil); report.Refused != "" {
t.Errorf("924: %s: composed as it runs, refused %q; want applied with no object", row.name, report.Refused)
}
}
}
}
// 1695/AC8: the hand-run `apply <file>` and `reconcile` have no bus: a reference whose file here verifies is applied
+76 -10
View File
@@ -3,22 +3,25 @@ package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"fmt"
"path/filepath"
"sync/atomic"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/provisions"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// serveFilling writes filled files on the provisioner client's request for as long as this engine serves (novox/hq
// ADR 0349 §4). A socket that cannot be served is said once, and this engine goes on applying.
func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, say func(string)) {
e := fillingEngine(opts, signer, say)
func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, queue *link.Session, node string, say func(string)) {
e := fillingEngine(opts, signer, provisionSecretsOn(queue, node, opts.state), say)
fillingNow.Store(e)
defer fillingNow.CompareAndSwap(e, nil)
if err := filling.Serve(ctx, filling.Socket, e, say); err != nil {
@@ -30,9 +33,10 @@ func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, s
var fillingNow atomic.Pointer[filling.Engine]
// knockFilling tells the provisioner's client to fill again what the filling refused or failed and the declaration
// applied now may let through (filling.Engine.Knock), once a module's apply or removal settled (agent.Env.Settled).
// What it told, or why it could not, is said.
func knockFilling(module string, say func(string)) {
// applied now may let through (filling.Engine.Knock), after what runs changed: a module's apply or removal settled
// (agent.Env.Settled, "<module> settled"), the haves were read at the start, or a module was taken as running
// (agent.Env.RunsChanged, issue 924). What it told, or why it could not, is said.
func knockFilling(after string, say func(string)) {
e := fillingNow.Load()
if e == nil {
return
@@ -40,16 +44,19 @@ func knockFilling(module string, say func(string)) {
told, err := e.Knock(filling.Socket)
switch {
case err != nil:
say(fmt.Sprintf("after %s settled: %v", module, err))
say(fmt.Sprintf("after %s: %v", after, err))
case told != "":
say(fmt.Sprintf("after %s settled, the provisioner's client is told to fill again: %s", module, told))
say(fmt.Sprintf("after %s, the provisioner's client is told to fill again: %s", after, told))
}
}
// fillingEngine is the write of a filled file as this engine does it: the declaration it keeps, verified, read under
// the apply's lock — the same two locks every apply takes, the process's and the one beside the state — and the file
// written through the apply (apply.WriteFilled), its outcome recorded in the node's state as an apply's is.
func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *filling.Engine {
//
// A provision's `${secret:}` the client left in the content is filled first (secrets, novox/hq ADR 0570 clause 6), and a
// file so filled is written 0600 when the declaration gives no mode.
func fillingEngine(opts options, signer ed25519.PublicKey, secrets provisionSecrets, say func(string)) *filling.Engine {
return &filling.Engine{
Declared: func() (*declaration.Declaration, error) {
return declaredNow(opts.state, signer)
@@ -73,8 +80,13 @@ func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *fi
if err != nil {
return "", "", err
}
report, updated, writeErr := apply.WriteFilled(ctx, built, d, known, path, content, apply.ExecRunner,
announceOr(say), apply.KeepIn(filepath.Dir(opts.state)), apply.WindowsIn(filepath.Dir(opts.state)))
content, secret, err := secrets.fill(ctx, filledModule(d, path), content)
if err != nil {
return "", "", fmt.Errorf("%s was not written: %w", path, err)
}
report, updated, writeErr := apply.WriteFilledWith(ctx, built, d, known, path, content, apply.ExecRunner,
announceOr(say), apply.KeepIn(filepath.Dir(opts.state)), apply.WindowsIn(filepath.Dir(opts.state)),
apply.Filled{Secret: secret})
// Saved whichever way it went, as an apply's: what was written is on the machine either way.
if err := store.Save(opts.state, updated); err != nil {
return "", "", errors.Join(writeErr, errors.New("written, and the node's state could not be saved: "+
@@ -93,3 +105,57 @@ func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *fi
},
}
}
// filledModule is the module whose file the declaration declares at path, or "" when none or several do.
func filledModule(d *declaration.Declaration, path string) string {
module := ""
for _, r := range d.Resources {
if f, ok := r.(*declaration.File); ok && f.Path == path {
if module != "" {
return ""
}
module, _ = d.ModuleOfID(f.ID)
}
}
return module
}
// provisionSecretsOn reads provisions and asks the vault over the link, and opens with this machine's sealing key.
func provisionSecretsOn(queue *link.Session, node, statePath string) provisionSecrets {
p := provisionSecrets{node: node, open: sealOpener(statePath)}
if queue == nil {
return p
}
// What the bus holds under prefix, read by a watch to its caught-up mark: no poll, no list of every key.
p.list = func(ctx context.Context, prefix string) (map[string][]byte, error) {
wctx, cancel := context.WithCancel(ctx)
defer cancel()
updates, err := queue.KVWatch(wctx, provisions.ProvisionsBucket, prefix+">")
if err != nil {
return nil, err
}
out := map[string][]byte{}
for {
select {
case <-ctx.Done():
return nil, ctx.Err()
case u, ok := <-updates:
if !ok {
return nil, fmt.Errorf("the watch of %s ended before it caught up", provisions.ProvisionsBucket)
}
switch {
case u.Caught:
return out, nil
case u.Deleted:
delete(out, u.Key)
default:
out[u.Key] = u.Value
}
}
}
}
p.own = func(ctx context.Context, args map[string]any) (json.RawMessage, error) {
return queue.AskVault(ctx, "own", args)
}
return p
}
+102 -1
View File
@@ -7,9 +7,12 @@ import (
"os"
"os/user"
"path/filepath"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/provisions"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/store"
)
@@ -45,7 +48,34 @@ func TestAWriteAndAnApplyOfTheSameFileSerialise(t *testing.T) {
store.Declared{Declaration: raw, Signature: ed25519.Sign(private, raw)}); err != nil {
t.Fatal(err)
}
e := fillingEngine(opts, public, func(string) {})
// The provision m reads, as mesh-provisioner keeps it, and the vault's value of it, sealed to this machine (faked:
// the opener reads the box's "v").
provision := map[string]string{}
vaultName := "provision/n2/db"
secrets := provisionSecrets{node: "n1",
list: func(_ context.Context, prefix string) (map[string][]byte, error) {
out := map[string][]byte{}
for k, v := range provision {
if strings.HasPrefix(k, prefix) {
out[k] = []byte(v)
}
}
return out, nil
},
own: func(_ context.Context, args map[string]any) (json.RawMessage, error) {
raw, _ := json.Marshal(args)
var a struct{ Credentials []map[string]any }
_ = json.Unmarshal(raw, &a)
c := a.Credentials[0]
c["name"], c["generation"], c["sealed"] = vaultName, 2, map[string]string{"v": "s3cret"}
return json.Marshal(map[string]any{"node": "n1", "credentials": []any{c}})
},
open: func(sealed string) ([]byte, error) {
var b struct{ V string }
err := json.Unmarshal([]byte(sealed), &b)
return []byte(b.V), err
}}
e := fillingEngine(opts, public, secrets, func(string) {})
uid := uint32(os.Getuid())
if err := e.Accepts(uid); err != nil {
t.Fatal(err)
@@ -102,4 +132,75 @@ func TestAWriteAndAnApplyOfTheSameFileSerialise(t *testing.T) {
if a := <-done; a.Action != "updated" {
t.Fatalf("the second write: %+v", a)
}
// ADR 0570 clause 6: the client leaves a provision's `${secret:}` to the node-engine, which fills it at the write
// from the vault's value of a provided provision, and writes the file 0600; a provision not provided refuses it.
keyed := func(state string, ref bool, id, local string) string {
p := provisions.Provision{Provision: "db", Local: local, Consumer: provisions.End{Module: "m", Node: "n1"},
Provider: provisions.End{Module: "postgres", Node: "n2"}, State: state, At: time.Now()}
if state != provisions.Provided {
p.Why = "the provider has not answered"
}
if ref {
p.Reference = &provisions.Reference{ID: id, Generation: 2}
}
raw, err := provisions.Encode(p)
if err != nil {
t.Fatal(err)
}
return string(raw)
}
for _, tc := range []struct {
name, state string
ref bool
content string
want string // the file after; "" keeps what was there
mode os.FileMode
vault string // the vault's credential name, when not the one the provision names
refID string // the reference's id, when not the provision's shared value
local string // the provision's local name (ADR 0094): the placeholder names it
}{
{name: "a provision not provided is refused", state: provisions.Asked, content: "password=${secret:db}\n"},
{name: "provided with no reference is refused", state: provisions.Provided, content: "password=${secret:db}\n"},
{name: "a reference to no credential of the provision is refused", state: provisions.Provided, ref: true,
refID: "provision/n9/db", content: "password=${secret:db}\n"},
{name: "a name no provision goes by is refused", state: provisions.Provided, ref: true, content: "password=${secret:other}\n"},
{name: "the vault's value is another than the provision names", state: provisions.Provided, ref: true,
vault: "pair/n2/db/n1/m", content: "password=${secret:db}\n"},
{name: "provided: filled at the write, 0600", state: provisions.Provided, ref: true,
content: "password=${secret:db}\n", want: "password=s3cret\n", mode: 0o600},
{name: "a local name is the provision's by its key", state: provisions.Provided, ref: true, local: "admin",
content: "admin=${secret:admin}\n", want: "admin=s3cret\n", mode: 0o600},
{name: "no placeholder is written as it came, as the declaration says", content: "plain\n", want: "plain\n", mode: 0o644},
} {
before, _ := os.ReadFile(path)
if tc.state != "" {
id := "provision/n2/db"
if tc.refID != "" {
id = tc.refID
}
clear(provision)
key := "n1.m.db"
if tc.local != "" {
key += "." + tc.local
}
provision[key] = keyed(tc.state, tc.ref, id, tc.local)
}
vaultName = "provision/n2/db"
if tc.vault != "" {
vaultName = tc.vault
}
a := e.Write(context.Background(), uid, filling.Request{Path: path, Content: []byte(tc.content)})
got, _ := os.ReadFile(path)
if tc.want == "" {
if a.Failed == "" && a.Refused == "" || string(got) != string(before) {
t.Errorf("%s: answered %+v, the file holds %q", tc.name, a, got)
}
continue
}
info, _ := os.Stat(path)
if a.Failed != "" || a.Refused != "" || string(got) != tc.want || info.Mode().Perm() != tc.mode {
t.Errorf("%s: answered %+v, the file holds %q, mode %v", tc.name, a, got, info.Mode())
}
}
}
+33 -14
View File
@@ -68,14 +68,23 @@ func foundDirectories(known store.State) []store.Applied {
// known (novox/hq ADR 0272): typed on the control-node, from the operator's own account. The directory's
// path is the resource's target beside the reason, never in it: the controller says a reason on the
// operator's channel, which withholds a path (ADR 0234 §6), and keeps the target as evidence.
//
// From the lease write, heard as the engine hears it (meshTrust), the line has no answerer: the way out is the node
// seat's hand-over with the directory's path, which asks the operator on their channel (novox/hq ADR 0563 §5).
func handOverLine() string {
node, _ := foundNode.Load().(string)
if node == "" {
node = "<node>"
}
if t := foundTrust.Load(); t != nil && t.HandedAway() {
return fmt.Sprintf("`%s/node-engine.hand-over` with the directory's path, then your yes on your channel", node)
}
return fmt.Sprintf("`nox node hand-over %s <directory>` on the control-node", node)
}
// foundTrust is what the engine trusts, whose lease decides the way out a directory used as found names.
var foundTrust atomic.Pointer[meshTrust]
// withFoundDirectories adds to a statement every directory used as found, as its module's resource of kind
// directory, unhealthy: the reason names its owner and mode as they are now, what was declared, and the line
// that hands it over. The path is the target, which the controller keeps as evidence.
@@ -129,16 +138,30 @@ func foundNow() []store.Applied {
// never follows a path it did not state: a hand-over is root giving a directory to an account at the next
// apply, and a path that resolves elsewhere would give that elsewhere.
func handOverFound(statePath, path, by string) (string, error) {
found, _, err := judgeFound(statePath, path)
if err != nil {
return "", err
}
if err := store.AddHandedOver(statePath, store.HandOver{ID: found.ID, Path: path, At: time.Now().UTC(), By: by}); err != nil {
return "", err
}
return fmt.Sprintf("%s (%s) is handed to the mesh by %s: the next apply gives it %s, and it is the mesh's from "+
"then on — removed when empty and no longer declared", path, found.ID, by, orUnsaid(found.Declared)), nil
}
// judgeFound is the judgement of a hand-over's path, recording nothing: the directory this machine uses as found at
// that path, as the engine states it, and what is on disk there.
func judgeFound(statePath, path string) (store.Applied, os.FileInfo, error) {
if !filepath.IsAbs(path) {
return "", fmt.Errorf("hand-over takes the directory's absolute path, and %q is not one; nothing was handed over", path)
return store.Applied{}, nil, fmt.Errorf("hand-over takes the directory's absolute path, and %q is not one; nothing was handed over", path)
}
if filepath.Clean(path) != path {
return "", fmt.Errorf("%q is not the directory's path as the engine states it (no `..`, no doubled or trailing "+
return store.Applied{}, nil, fmt.Errorf("%q is not the directory's path as the engine states it (no `..`, no doubled or trailing "+
"separator); nothing was handed over", path)
}
known, err := store.Load(statePath)
if err != nil {
return "", err
return store.Applied{}, nil, err
}
var found *store.Applied
for _, r := range foundDirectories(known) {
@@ -149,31 +172,27 @@ func handOverFound(statePath, path, by string) (string, error) {
}
}
if found == nil {
return "", errors.New(path + " is not a directory this machine uses as found (the module's condition names " +
return store.Applied{}, nil, errors.New(path + " is not a directory this machine uses as found (the module's condition names " +
"the ones that are); nothing was handed over")
}
info, err := os.Lstat(path)
switch {
case err != nil:
return "", fmt.Errorf("%s cannot be read on this machine: %v; nothing was handed over", path, err)
return store.Applied{}, nil, fmt.Errorf("%s cannot be read on this machine: %v; nothing was handed over", path, err)
case info.Mode()&os.ModeSymlink != 0:
return "", fmt.Errorf("%s is a link, and the mesh hands over no link (novox/hq ADR 0012); nothing was handed over", path)
return store.Applied{}, nil, fmt.Errorf("%s is a link, and the mesh hands over no link (novox/hq ADR 0012); nothing was handed over", path)
case !info.IsDir():
return "", fmt.Errorf("%s is not a directory on this machine; nothing was handed over", path)
return store.Applied{}, nil, fmt.Errorf("%s is not a directory on this machine; nothing was handed over", path)
}
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return "", fmt.Errorf("%s cannot be resolved on this machine: %v; nothing was handed over", path, err)
return store.Applied{}, nil, fmt.Errorf("%s cannot be resolved on this machine: %v; nothing was handed over", path, err)
}
if resolved != path {
return "", fmt.Errorf("%s is reached through a link (it is %s), and the mesh hands over nothing through one "+
return store.Applied{}, nil, fmt.Errorf("%s is reached through a link (it is %s), and the mesh hands over nothing through one "+
"(novox/hq issue 339); nothing was handed over", path, resolved)
}
if err := store.AddHandedOver(statePath, store.HandOver{ID: found.ID, Path: path, At: time.Now().UTC(), By: by}); err != nil {
return "", err
}
return fmt.Sprintf("%s (%s) is handed to the mesh by %s: the next apply gives it %s, and it is the mesh's from "+
"then on — removed when empty and no longer declared", path, found.ID, by, orUnsaid(found.Declared)), nil
return *found, info, nil
}
// handOverFrom is the engine's hand-over for the controller's ask: the state beside statePath, the path as asked.
+15
View File
@@ -40,6 +40,21 @@ func TestAFoundDirectoryIsStatedUnhealthy(t *testing.T) {
strings.Contains(r.Reason, "mesh-host") || strings.Contains(r.Reason, dir) {
t.Fatalf("the way out is not the operator's line, or carries the path: %q", r.Reason)
}
// ADR 0563 §5: from the lease write the remedy is the node seat's hand-over, answered on the operator's channel, and
// no longer the controller's terminal line; the lease is the one the engine hears.
handed := newMeshTrust(nil)
handed.handedAway = true
foundTrust.Store(handed)
if r := withFoundDirectories(&link.Health{}, foundDirectories(known)).Resources[0]; !strings.Contains(r.Reason,
"`laptop/node-engine.hand-over`") || strings.Contains(r.Reason, "nox node hand-over") || strings.Contains(r.Reason, dir) {
t.Fatalf("after the lease write the way out is %q", r.Reason)
}
foundTrust.Store(newMeshTrust(nil))
if r := withFoundDirectories(&link.Health{}, foundDirectories(known)).Resources[0]; !strings.Contains(r.Reason,
"`nox node hand-over laptop <directory>`") {
t.Fatalf("while the controller holds its lease the way out is %q", r.Reason)
}
foundTrust.Store((*meshTrust)(nil))
foundNode.Store("")
if r := withFoundDirectories(&link.Health{}, foundDirectories(known)).Resources[0]; !strings.Contains(r.Reason,
"`nox node hand-over <node> <directory>`") {
+1 -1
View File
@@ -69,7 +69,7 @@ func TestAModuleWithNothingLongRunningIsOkWithinALookOfItsApply(t *testing.T) {
applying := fact
applying.Phase = string(delivery.PhaseApplying)
queue.NoteModule(ctx, applying)
go judgeWhatRuns(ctx, j, queue, func(string) {}, nil)
go judgeWhatRuns(ctx, j, queue, func(string) {}, nil, nil)
// The facts look inside the apply, while the module is still applying.
queue.SayFacts(ctx, link.Report{Health: &link.Health{Contract: link.LivenessContract, At: time.Now().UTC()}})
if h, _ := bus.have("n1.xorg"); h.Health != delivery.HealthUnknown {
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"context"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/liveness"
)
// heldBy is the containers an open maintenance window holds still, by runtime name (ADR 0189), at now.
func heldBy(windows *apply.Windows) func(time.Time) map[string]bool {
return func(now time.Time) map[string]bool {
held := map[string]bool{}
for _, w := range windows.Open(now) {
for _, name := range w.Holds {
held[name] = true
}
}
return held
}
}
// inactiveIn is what the agent asks at its start of a module it believes runs (ADR 0568 §1): whether the machine shows a
// long-running resource the composition declares running not active, read from the service manager and the container
// runtime and from no record. A resource held by an open maintenance window, or waiting for a library, is not counted;
// one that cannot be read is not counted either (unknown is not inactive), and neither is one the manager is restarting.
func inactiveIn(rt liveness.Runtime, held func(time.Time) map[string]bool) func(context.Context, delivery.Composed) bool {
return func(ctx context.Context, c delivery.Composed) bool {
d, err := declarationOf(c)
if err != nil {
return false
}
var holding map[string]bool
if held != nil {
holding = held(time.Now())
}
return anyInactive(ctx, rt, liveness.LongRunning(d, withWaiting(nil, apply.WaitingOnLibrary(d))), holding)
}
}
// anyInactive reads each group of resources once (the containers, each service manager's units) and says whether any is
// not active.
func anyInactive(ctx context.Context, rt liveness.Runtime, resources []liveness.Resource, held map[string]bool) bool {
type group struct {
kind, scope, user string
targets []string
}
var groups []*group
at := map[string]*group{}
for _, r := range resources {
if r.Kind == liveness.KindContainer && held[r.Target] {
continue
}
g := &group{kind: r.Kind, scope: r.Scope, user: r.User}
if r.Kind == liveness.KindContainer {
g.scope, g.user = "", ""
}
key := g.kind + "/" + g.scope + "/" + g.user
if r.Kind != liveness.KindContainer {
key = "units/" + g.scope + "/" + g.user
}
if at[key] == nil {
at[key] = g
groups = append(groups, g)
}
at[key].targets = append(at[key].targets, r.Target)
}
for _, g := range groups {
var got map[string]liveness.Observed
var err error
if g.kind == liveness.KindContainer {
got, err = rt.Containers(ctx, g.targets)
} else {
got, err = rt.Units(ctx, g.scope, g.user, g.targets)
}
if err != nil {
continue
}
for _, t := range g.targets {
if o := got[t]; !o.Found || !(o.Running || o.Restarting) {
return true
}
}
}
return false
}
+89
View File
@@ -0,0 +1,89 @@
package main
import (
"context"
"encoding/json"
"errors"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/liveness"
)
// 1957/AC2 and AC4 (unit; ADR 0568 §2, §4): the judge's look acts on a resource that goes down, once per edge. A repeated
// statement acts nothing, the edge clears when the resource is healthy again, and a resource held, restarting, starting or
// missing (the other path's) is no edge.
func testOnlyAResourceNewlyDownIsAnEdge(t *testing.T) {
state := func(id, s, reason string, missing bool) liveness.State {
return liveness.State{Resource: liveness.Resource{Module: "dnsmasq", ID: id}, State: s, Reason: reason, Missing: missing}
}
down := state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonDown, false)
steps := []struct {
name string
st []liveness.State
want []string
}{
{"down: an edge", []liveness.State{down}, []string{"dnsmasq"}},
{"said again while down: none", []liveness.State{down}, nil},
{"healthy again: clears", []liveness.State{state("dnsmasq.unit", liveness.Healthy, "", false)}, nil},
{"down again: an edge", []liveness.State{down}, []string{"dnsmasq"}},
{"held by a window: none", []liveness.State{state("dnsmasq.unit", liveness.Held, "", false)}, nil},
{"restarting: none", []liveness.State{state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonRestarting, false)}, nil},
{"starting: none", []liveness.State{state("dnsmasq.unit", liveness.Starting, "", false)}, nil},
{"missing: the other path's", []liveness.State{state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonDown, true)}, nil},
}
var last map[string]bool
for _, s := range steps {
var got []string
got, last = newlyInactive(last, liveness.Statement{Resources: s.st})
if len(got) != len(s.want) || (len(got) == 1 && got[0] != s.want[0]) {
t.Fatalf("%s: acted on %v, want %v", s.name, got, s.want)
}
}
}
type fakeRuntime struct {
units map[string]liveness.Observed
containers map[string]liveness.Observed
err error
}
func (f fakeRuntime) Containers(_ context.Context, _ []string) (map[string]liveness.Observed, error) {
return f.containers, f.err
}
func (f fakeRuntime) Units(_ context.Context, _, _ string, _ []string) (map[string]liveness.Observed, error) {
return f.units, f.err
}
// 1957/AC1 (unit; ADR 0568 §1): the machine's facts say a declared-running resource is not active when its unit is stopped
// or unknown to the manager; everything up, a restart in progress, a read that failed and a container held by a
// maintenance window are not.
func testTheMachineShowsAResourceNotActive(t *testing.T) {
svc := []byte(`{"id":"dnsmasq.unit","type":"service","unit":"dnsmasq.service","state":"running","module":"dnsmasq"}`)
ctr := []byte(`{"id":"dnsmasq.box","type":"container","name":"box","image":"x@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","module":"dnsmasq"}`)
up := liveness.Observed{Found: true, Running: true}
for _, c := range []struct {
name string
rt fakeRuntime
held map[string]bool
want bool
}{
{"all up", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": up}}, nil, false},
{"unit inactive", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": {Found: true}}, containers: map[string]liveness.Observed{"box": up}}, nil, true},
{"unit unknown to the manager", fakeRuntime{containers: map[string]liveness.Observed{"box": up}}, nil, true},
{"unit restarting", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": {Found: true, Restarting: true}}, containers: map[string]liveness.Observed{"box": up}}, nil, false},
{"container not running", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": {Found: true}}}, nil, true},
{"container held by a window", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": {Found: true}}}, map[string]bool{"box": true}, false},
{"unread", fakeRuntime{err: errors.New("the service manager could not be read")}, nil, false},
} {
t.Run(c.name, func(t *testing.T) {
d, err := declarationOf(delivery.Composed{Node: "n1", Module: "dnsmasq", Resources: []json.RawMessage{svc, ctr}})
if err != nil {
t.Fatal(err)
}
if got := anyInactive(context.Background(), c.rt, liveness.LongRunning(d, nil), c.held); got != c.want {
t.Fatalf("inactive = %v, want %v", got, c.want)
}
})
}
}
+40
View File
@@ -8,6 +8,8 @@ import (
"testing"
"time"
meshasker "git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
@@ -53,4 +55,42 @@ func TestALibrarySaysItsWaitWithoutWordsToParse(t *testing.T) {
if strings.Contains(media.Reason, dir) || len(books.AccessWaits) != 1 || books.AccessWaits[0].Reason != link.WaitDisk {
t.Fatalf("a missing root does not wait for its disk, or a reason names a path: %+v %+v", books, media)
}
// 774 (ADR 0563 §4): the library's home engine asks its take of the operator itself, once: no second ask while one is
// open, nor after it ended; and a library waiting for its disk is not asked.
t.Run("the home engine asks the take once", func(t *testing.T) {
var asked []asks.Ask
state := &link.MemState{}
h := &link.HandActs{Node: "n1", State: state,
JudgeAccess: func(act, access string) (link.Judged, error) {
return link.Judged{Subject: access, Path: root, Sets: "the group mesh-media", Disk: "0:0 755"}, nil
},
AccessAct: func(link.AccessAct) (string, error) { return "taken", nil }}
h.Asker = &meshasker.Client{Module: asks.EngineAsker("n1"), State: state,
Publish: func(subject string, body any, id string) error {
if a, ok := body.(asks.Ask); ok {
asked = append(asked, a)
}
return nil
}}
waiting := keepWaiting(vs)
askTakes(h, waiting, nil)
askTakes(h, waiting, nil)
if len(waiting) != 1 || waiting[0] != "library:media" || len(asked) != 1 {
t.Fatalf("waiting %v asked %d times", waiting, len(asked))
}
h.Heard(asks.Warrant{Ask: asked[0].ID, Asker: asks.EngineAsker("n1"), Outcome: asks.OutcomeExpired, At: time.Now()})
askTakes(h, waiting, nil)
if len(asked) != 1 {
t.Fatalf("the take was asked again after its ask ended: %d", len(asked))
}
// Kept in a key of its own (rev-l3 round 1): the acts gone from the record never make it ask again.
keys, _ := state.Keys("act.")
for _, k := range keys {
_ = state.Delete(k)
}
askTakes(h, waiting, nil)
if len(asked) != 1 {
t.Fatalf("the take was asked again with the acts gone: %d", len(asked))
}
})
}
+138 -17
View File
@@ -516,7 +516,7 @@ func runApply(ctx context.Context, opts options, d *declaration.Declaration, raw
// A file given by digest is had before the plan is drawn from it (novox/hq ADR 0486 §1, issue 1695). This command has
// no link to the bus, so a file already here that verifies is used, and any other is refused: never written empty.
if err := apply.ResolveFileReferences(ctx, d, link.FilesFrom(ctx)); err != nil {
if err := apply.ResolveFileReferencesKept(ctx, d, link.FilesFrom(ctx), objectKeep(opts)); err != nil {
return fmt.Errorf("a file of this declaration could not be had by its digest: %w", err)
}
@@ -1046,15 +1046,7 @@ func runLink(ctx context.Context, opts options) error {
say(err.Error())
}
windows := apply.WindowsIn(filepath.Dir(opts.state))
j.HeldNow = func(now time.Time) map[string]bool {
held := map[string]bool{}
for _, w := range windows.Open(now) {
for _, name := range w.Holds {
held[name] = true
}
}
return held
}
j.HeldNow = heldBy(windows)
judging = j
// And which units its service managers say failed, and whose each is (novox/hq issue 315).
unitJudge = units.New(units.Exec{Run: apply.ExecRunner, System: builtFor})
@@ -1113,8 +1105,17 @@ func runLink(ctx context.Context, opts options) error {
// the bus, answer what the controller asks of this machine, and say its facts: when a link opens, and when the
// controller asks.
proof := &proving{statePath: opts.state, version: runningVersion(), say: say}
// What this engine trusts a mesh signature from: the controller's key until its lease is handed away, and the vault's
// once read from its signers (novox/hq ADR 0550 §2).
trust := newMeshTrust(mine.Membership.Signer)
// And the way out a directory used as found names follows the lease it hears (novox/hq ADR 0563 §5).
foundTrust.Store(trust)
queue := &link.Session{
Membership: membership,
// And the public halves and certificate its record says (novox/hq ADR 0555 §3), read at each facts report.
Keys: recordKeys(opts.state, trust.Trusts),
// And the controller's key, read per signed ask: none once its lease is handed away (novox/hq ADR 0563 §7).
ControllerKey: trust.Controller,
// And the first have this build writes is what proves it to the launcher (ADR 0539 §9, ADR 0227 §8).
HaveWritten: proof.haveWritten,
// And the operator's hand-over of a directory used as found, carried by the controller from the
@@ -1135,6 +1136,10 @@ func runLink(ctx context.Context, opts options) error {
Say: say,
Timeout: opts.timeout,
}
// And the node seat node-engine this engine holds (novox/hq ADR 0563): its verbs on every link, its own asks to the
// operator under node-engine-<node>, and its acts kept beside its state.
queue.Acts = engineActs(opts, mine.Node, queue, setuidSearch)
go reconcileActs(ctx, queue.Acts)
current := &agentNow{}
setRunsHere(runsFromAgent(current))
factsOfNow := func(ctx context.Context, ag *agent.Agent) {
@@ -1171,11 +1176,18 @@ func runLink(ctx context.Context, opts options) error {
if ag := current.get(); ag != nil {
ag.Retry(modules...)
}
}, func(modules ...string) {
if ag := current.get(); ag != nil {
ag.Reconcile(modules...)
}
})
}
// The node follows its wants: one module at a time from the composition its want names (novox/hq ADR 0539), and the
// busy containers are asked for again soon (ADR 0360 §11).
go nodeAgent(aside, opts, mine.Membership.Signer, queue, sched, say, afterApply, current, factsOfNow)
// And what it trusts, followed on the bus, said in its record whenever it changes (ADR 0550 §2).
trust.onChange(func() { factsOfNow(aside, current.get()) })
go followTrust(aside, queue, trust, say)
go nodeAgent(aside, opts, mine.Membership.Signer, trust, queue, sched, say, afterApply, current, factsOfNow)
go busyPasses(aside, func() {
if ag := current.get(); ag != nil {
ag.Retry(busyModules()...)
@@ -1188,7 +1200,7 @@ func runLink(ctx context.Context, opts options) error {
go serveMeshCLI(aside, queue, say)
// And a filled file is written on the provisioner client's request, one at a time with the apply (novox/hq ADR
// 0349 §4).
go serveFilling(aside, opts, mine.Membership.Signer, say)
go serveFilling(aside, opts, mine.Membership.Signer, queue, mine.Node, say)
held := link.HoldRoused(aside, membership, queue, say, opts.timeout, rousedBySignal(ctx))
// **Cleanly**, or the launcher counts standing aside as a crash and rolls the new host back
@@ -1334,13 +1346,15 @@ const (
// the bus when a state changed, again every minute while one is not healthy, and every five minutes
// anyway. A statement the bus did not take is said at the next look. It reads; it never acts (ADR 0240
// rule 6) — but for going on with a take act stopped at its bound (ADR 0328 §7), which runs beside it.
func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session, say link.Announce, reapply func(modules ...string)) {
var missingBefore map[string]bool
func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session, say link.Announce, reapply, reconcile func(modules ...string)) {
var missingBefore, inactiveBefore map[string]bool
ticker := time.NewTicker(liveness.LookEvery)
defer ticker.Stop()
var lastSaid time.Time
owed := true
spaced := 1.0
// healthUnwritten is why the health report was last not written, said once each time it changes.
var healthUnwritten string
// A module whose change ended unjudged is looked at and said at once (review of fix 15): a module with nothing
// long-running is then ok within a look of its apply, not at the five-minute statement.
var nudged <-chan struct{}
@@ -1372,6 +1386,7 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
if a := accountJudge; a != nil {
as, accountsChanged := a.Look(ctx)
accountSt = &as
lastAccounts.Store(&as)
owed = owed || accountsChanged
if accountsChanged {
for _, v := range as.Accounts {
@@ -1386,6 +1401,9 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
accessSt, accessesChanged := accesses.look(time.Now())
keeping.poke(ctx)
owed = owed || accessesChanged
// And a library whose root holds data outside its group is asked of the operator by this engine, once (novox/hq
// ADR 0563 §4).
askTakes(queue.Acts, keepWaiting(accessSt), say)
if accessesChanged {
for _, v := range accessSt {
if v.State != link.StateHealthy {
@@ -1420,6 +1438,11 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
healthy := st.Healthy() && (netSt == nil || netSt.State != network.Unhealthy) &&
(unitSt == nil || len(unitSt.Failed) == 0) && (accountSt == nil || accountSt.Healthy()) && len(found) == 0 &&
accessesHealthy(accessSt)
h := *withAccesses(withFoundDirectories(withAccounts(withUnits(healthAsReported(st, netSt),
unitSt), accountSt), found), accessSt)
// The node's own health report in its key on the bus, compared on every look and written only when what it says
// changed (issue 774): what the watchman raises the machine's alerts from, without the controller.
writeHealthReport(ctx, queue, h, netSt, &healthUnwritten, say)
if !owed && !(!healthy && since >= sayUnhealthyAgain) && since < sayAnyway {
continue
}
@@ -1437,14 +1460,49 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
say(fmt.Sprintf("applying %s again: a resource it declares is not on this machine", strings.Join(m, ", ")))
reapply(m...)
}
// A resource its module declares running that is down but still known to the manager is started by the same
// apply, once each time it goes down, never on a clock (ADR 0568 §2): the judge reads, the agent acts.
var inactive []string
inactive, inactiveBefore = newlyInactive(inactiveBefore, st)
if len(inactive) > 0 && reconcile != nil {
say(fmt.Sprintf("applying %s again: a resource it declares running is not active", strings.Join(inactive, ", ")))
reconcile(inactive...)
}
}
if queue.SayHealth(ctx, *withAccesses(withFoundDirectories(withAccounts(withUnits(healthAsReported(st, netSt),
unitSt), accountSt), found), accessSt)) {
if queue.SayHealth(ctx, h) {
lastSaid, owed = time.Now(), false
}
}
}
// writeHealthReport writes this node's health report to its key (link.WriteNodeHealth) when it changed. Why it could
// not is said once per reason, and said again when it is written after all.
func writeHealthReport(ctx context.Context, queue *link.Session, h link.Health, ns *network.Statement, unwritten *string,
say link.Announce) {
if queue == nil {
return
}
var peers []link.PeerSeen
if ns != nil {
for _, p := range ns.Peers {
peers = append(peers, link.PeerSeen{Key: p.Key, Addresses: p.Addresses, Seen: p.Seen})
}
}
var machines link.Machines
if c := machinesNow.Load(); c != nil {
machines.ByAddress, machines.Hub = c.Machines()
}
_, err := queue.WriteNodeHealth(ctx, link.NodeHealthOf(queue.Membership.Node, h, peers, machines))
switch {
case err != nil && err.Error() != *unwritten:
*unwritten = err.Error()
say("this machine's health report is not written to the bus: " + err.Error())
case err == nil && *unwritten != "":
*unwritten = ""
say("this machine's health report is written to the bus again")
}
}
// healthAsReported is a statement as the report and the event carry it.
func healthAsReported(st liveness.Statement, ns *network.Statement) *link.Health {
// LibraryContract: this engine reads a resource's declared `health` and judges it (ADR 0240 Phase B), judges a
@@ -1640,6 +1698,15 @@ func announceOr(say link.Announce) func(string) {
// run by hand beside this service — the lock beside the state does the same (store.Lock).
var applying sync.Mutex
// objectKeep is the node's own keep of the objects its compositions name by digest, beside its state (apply.ObjectKeep,
// issue 924): a composition the node once applied never again needs the controller's bucket to hold its files.
func objectKeep(opts options) *apply.ObjectKeep {
if opts.state == "" {
return nil
}
return apply.KeepObjectsIn(filepath.Join(filepath.Dir(opts.state), "objects"))
}
// appliedModule is how the apply of one module's declaration came out.
type appliedModule struct {
// Refused is why nothing was applied, when nothing was.
@@ -1679,7 +1746,7 @@ func applyModuleHeld(ctx context.Context, opts options, declared *declaration.De
// **A file given by digest is fetched and verified before anything of the composition is applied** (novox/hq ADR
// 0486 §1, issue 1695): one object missing or differing refuses the module whole.
if err := apply.ResolveFileReferences(ctx, declared, link.FilesFrom(ctx)); err != nil {
if err := apply.ResolveFileReferencesKept(ctx, declared, link.FilesFrom(ctx), objectKeep(opts)); err != nil {
return appliedModule{Refused: "a file of this composition could not be had by its digest: " + err.Error()}
}
@@ -2131,3 +2198,57 @@ func newlyMissing(last map[string]bool, st liveness.Statement) ([]string, map[st
}
return missingModules(fresh), now
}
// newlyInactive are the modules of the resources a statement says down (not running after their grace, but still known to
// their manager, and not restarting) that the last statement acted on did not, each once, sorted, and the resources down
// now: a resource still down is not acted on again, and one that came back and went down again is (ADR 0568 §2).
func newlyInactive(last map[string]bool, st liveness.Statement) ([]string, map[string]bool) {
now := map[string]bool{}
var fresh liveness.Statement
for _, r := range st.Resources {
if r.State != liveness.Unhealthy || r.Reason != liveness.ReasonDown || r.Missing {
continue
}
now[r.ID] = true
if !last[r.ID] {
fresh.Resources = append(fresh.Resources, r)
}
}
return missingModules(withMissing(fresh)), now
}
// withMissing is the statement with each of its resources marked missing, so missingModules lists their modules.
func withMissing(st liveness.Statement) liveness.Statement {
for i := range st.Resources {
st.Resources[i].Missing = true
}
return st
}
// recordKeys reads the public halves of the node's keys and the certificate the mesh issued for its internal name from
// its state directory (novox/hq ADR 0555 §3): what the node's record says of them. A key that cannot be read is left
// empty, never invented; the certificate is there only after the first declaration that carried it.
func recordKeys(state string, trusts func() []string) func() link.RecordKeys {
return func() link.RecordKeys {
var k link.RecordKeys
if trusts != nil {
k.Trusts = trusts()
}
if id, err := identity.Load(identity.Path(state)); err == nil {
k.TunnelKey = id.Overlay.Public
}
if sk, err := identity.LoadSealingKey(identity.SealingKeyPath(state)); err == nil {
k.SealingKey = sk.Public
}
if sv, err := identity.LoadServingKey(identity.ServingKeyPath(state)); err == nil {
k.ServingKey = sv.Public
}
if cert, err := os.ReadFile(identity.CertificatePath(state)); err == nil {
k.Certificate = string(cert)
}
if authority, err := os.ReadFile(authorityPath(state)); err == nil {
k.Authority = string(authority)
}
return k
}
}
+374 -23
View File
@@ -5,19 +5,25 @@ import (
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/selfcompose"
"github.com/novox/mesh-host/internal/store"
)
@@ -35,6 +41,20 @@ func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Up
if err != nil {
return nil, err
}
return w.updates(ctx, in), nil
}
// WatchAwait is Watch for a state that its holder makes when it starts (the placement seat's): a bucket not on the bus
// yet is waited for on the bus's word that it was created (link.Session.KVWatchAwait), and waiting is told why.
func (w bucketWatch) WatchAwait(ctx context.Context, pattern string, waiting func(error)) (<-chan agent.Update, error) {
in, err := w.session.KVWatchAwait(ctx, w.bucket, pattern, waiting)
if err != nil {
return nil, err
}
return w.updates(ctx, in), nil
}
func (w bucketWatch) updates(ctx context.Context, in <-chan link.KVUpdate) <-chan agent.Update {
out := make(chan agent.Update, 64)
go func() {
defer close(out)
@@ -46,25 +66,52 @@ func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Up
}
}
}()
return out, nil
return out
}
// sessionHaves writes the haves the agent says, through the session.
type sessionHaves struct{ session *link.Session }
func (h sessionHaves) Read(ctx context.Context, module string) (agent.Have, bool) {
if module == agent.NodeModule {
// The node's own key: the digest of its own composition it runs, and what that carries sealed.
got, ok := h.session.ReadNodeHave(ctx)
if !ok || got.Composed == "" {
return agent.Have{}, false
}
return agent.Have{Module: module, Composed: got.Composed, Sealed: got.SealedAt}, true
}
got, ok := h.session.ReadHave(ctx, module)
if !ok {
return agent.Have{}, false
}
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail}, true
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail,
Sealed: got.SealedAt}, true
}
// ReadErr is Read telling a have absent from one the bus could not read (agent.HavesReadErr).
func (h sessionHaves) ReadErr(ctx context.Context, module string) (agent.Have, bool, error) {
if module == agent.NodeModule {
// The node's own key, its read the bus did not answer told as such (issue 1126, 2026-10-28).
got, ok, err := h.session.ReadNodeHaveErr(ctx)
if !ok || err != nil || got.Composed == "" {
return agent.Have{}, false, err
}
return agent.Have{Module: module, Composed: got.Composed, Sealed: got.SealedAt}, true, nil
}
got, ok, err := h.session.ReadHaveErr(ctx, module)
if !ok || err != nil {
return agent.Have{}, false, err
}
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail,
Sealed: got.SealedAt}, true, nil
}
func (h sessionHaves) Write(ctx context.Context, x agent.Have) {
if x.Module == agent.NodeModule {
// The node's own key holds its facts and the digest of the composition it runs.
if x.Phase == "" || x.Phase == string(delivery.PhaseRunning) {
h.session.NoteNode(ctx, x.Composed)
h.session.NoteNode(ctx, x.Composed, x.Sealed)
}
return
}
@@ -73,7 +120,8 @@ func (h sessionHaves) Write(ctx context.Context, x agent.Have) {
phase = ""
}
h.session.NoteModule(ctx, link.ModuleFact{Module: x.Module, FP: x.FP, Composed: x.Composed, Phase: phase,
Failed: phase == string(delivery.PhaseFailed), Detail: shorten(x.Detail)})
Failed: phase == string(delivery.PhaseFailed), Detail: shorten(x.Detail), Judgement: x.Judgement, Waits: x.Waits,
Sealed: x.Sealed})
}
func (h sessionHaves) Removed(ctx context.Context, module string) {
@@ -297,13 +345,14 @@ func resourcesOf(cs []delivery.Composed) []declaration.Resource {
// controller's key the node was enrolled with (novox/hq ADR 0539 §3, §11); a node adopted later has it at its adoption.
const operatorKeysFile = "operator-keys"
// trustedKeys are the keys a composition may be signed by: the controller's, which the node was enrolled with, and the
// operator's, read from operatorKeysFile each time it is asked, so a key that arrives while the agent runs is trusted at
// the next composition. A file that cannot be read trusts the controller's alone, and says so.
func trustedKeys(opts options, controller ed25519.PublicKey) func() []ed25519.PublicKey {
// trustedKeys are the keys a composition may be signed by: the controller's, which the node was enrolled with, while the
// controller holds its lease (ADR 0550 §2: trusted no more once it is handed away), and the operator's, read from
// operatorKeysFile each time it is asked, so a key that arrives while the agent runs is trusted at the next composition.
// A file that cannot be read trusts the controller's alone, and says so.
func trustedKeys(opts options, controllerNow func() ed25519.PublicKey) func() []ed25519.PublicKey {
return func() []ed25519.PublicKey {
keys := []ed25519.PublicKey{}
if len(controller) == ed25519.PublicKeySize {
if controller := controllerNow(); len(controller) == ed25519.PublicKeySize {
keys = append(keys, controller)
}
raw, err := os.ReadFile(filepath.Join(filepath.Dir(opts.state), operatorKeysFile))
@@ -327,13 +376,16 @@ func trustedKeys(opts options, controller ed25519.PublicKey) func() []ed25519.Pu
}
// keptOnce reads the kept declaration once and splits it by module: its resources (the node's own are those that say
// none, ADR 0539 §9) and its filled paths.
// none, ADR 0539 §9) and its filled paths. Once the read is retired (keptRetired) nothing is read.
func keptOnce(opts options, signer ed25519.PublicKey) func() (map[string][]json.RawMessage, map[string][]string) {
var once sync.Once
var by map[string][]json.RawMessage
var filled map[string][]string
return func() (map[string][]json.RawMessage, map[string][]string) {
once.Do(func() {
if keptRetired.Load() {
return
}
raw, err := store.LoadDeclared(store.DeclaredPath(opts.state), signer)
if err != nil {
return
@@ -383,7 +435,7 @@ func splitKept(raw []byte) (map[string][]json.RawMessage, map[string][]string) {
// nodeAgent runs the node agent under the session's links for as long as ctx lasts: it waits for a link, follows the
// node's wants and compositions on it, and begins again with the next link. A watch it cannot make (a bucket this node's
// account may not read) is said, and tried again with a growing wait, as the link is.
func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, session *link.Session, sched *apply.Scheduler,
func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, trust *meshTrust, session *link.Session, sched *apply.Scheduler,
say link.Announce, afterApply func(), current *agentNow, sayFacts func(ctx context.Context, ag *agent.Agent)) {
wait := 2 * time.Second
for ctx.Err() == nil {
@@ -400,10 +452,31 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
return
}
var ag *agent.Agent
// The kept release-5 declaration is no longer read once every module the seat's placement puts here has a have of its
// own (ADR 0550 §2: its signature is the controller's, trusted no more after the lease write).
// Nor before this engine keeps the node's own as it runs (applied/node.json): until then the kept declaration is the one
// record of what the node's own composition carries, its folded modules among them (issue 774, review of the
// re-land, round 1: novox's first start would compose its own with nats running nothing).
if retired, why := placedAllHave(linked, session, node); retired && !keptRetired.Load() {
if !appliedStoreFor(opts.state).holds(agent.NodeModule) {
say("the kept declaration is still read: " + why + ", but this engine keeps no node's own composition yet")
} else if !keptRetired.Swap(true) {
say("the kept declaration is no longer read: " + why)
}
}
kept := keptOnce(opts, signer)
// The mesh's own certificate the declaration kept from before carries, kept for the record before this machine
// composes its own (ADR 0555 §3): without it a module that asks for a certificate would be stood in for.
if byModule, _ := kept(); byModule != nil {
for module, resources := range byModule {
if err := keepAuthority(opts.state, delivery.Composed{Module: module, Resources: resources}); err != nil {
say("the mesh's certificate the kept declaration carries could not be kept for this machine's record: " + err.Error())
}
}
}
// The bus's user list, written alone from the newest composition of the bus module (ADR 0547 §8): on every
// node, and a node whose compositions carry no list writes nothing.
users := userListFollower(opts, session, composedBucket, trustedKeys(opts, signer), say, func() {
users := userListFollower(opts, session, composedBucket, trustedKeys(opts, trust.Controller), say, func() {
// This engine wrote the list, so the bus reloads its accounts and answers no JetStream request for a few
// seconds: what failed in them is tried again once that is over, not at its next wait.
time.AfterFunc(reloadSettles, func() {
@@ -428,6 +501,10 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
// The same kept declaration's filled paths, so a module still running from it keeps its files filled.
KeptFilled: func() map[string][]string { _, filled := kept(); return filled },
Recorded: recordedIn(opts.state, say),
// What this engine applied, kept on the machine and read at the start, so what runs is known before the node
// composes its own (issue 774).
Applied: appliedStoreFor(opts.state),
Inactive: inactiveIn(&liveness.Exec{Run: apply.ExecRunner, System: builtFor}, heldBy(apply.WindowsIn(filepath.Dir(opts.state)))),
Revisions: func(ctx context.Context, key string) ([]agent.Update, error) {
got, err := session.KVHistory(ctx, composedBucket, key)
if err != nil {
@@ -439,18 +516,35 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
}
return out, nil
},
Trusted: trustedKeys(opts, signer),
Trusted: trustedKeys(opts, trust.Controller),
Backup: backupNode(session),
// What the filling refused before this apply may be named now: the provisioner's client is told to fill
// again (ace, 2026-10-25 21:17).
Settled: func(module string) { knockFilling(module, say) },
Say: say,
Settled: func(module string) { knockFilling(module+" settled", say) },
// And once the haves are read at the start, and whenever a module is taken as running without an apply, so
// a refusal from the engine's start does not wait for some module to settle (issue 924).
RunsChanged: func(why string) { knockFilling(why, say) },
Say: say,
// A failure of the bus not answering for a moment is tried again: 2 s, 5 s, 15 s, 60 s, then every 5 min.
Backoff: link.Backoff,
Apply: func(ctx context.Context, c delivery.Composed, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := applyComposition(actx, opts, c, others, sched, say)
// While the grants part's list is followed, the controller's list in the bus module's composition is not
// written: the apply keeps the list on the machine (ADR 0547 clause 8, never both).
held, herr := users.Hold(c)
var res agent.Result
if herr != nil {
res = failedWith(herr.Error())
} else {
res = applyComposition(actx, opts, held, others, ag.Placed(scopeName(c)), ag.RemovedByWant(), sched, say)
}
done()
// The mesh's own certificate it carries is kept for this machine's record (ADR 0555 §3).
if !res.Failed {
if err := keepAuthority(opts.state, c); err != nil {
say("the mesh's certificate in " + c.Module + "'s composition could not be kept for this machine's record: " + err.Error())
}
}
// An apply of the bus module may write an older list than the newest composition holds: looked at again.
if module, _ := users.State(); module != "" && module == c.Module {
users.Again()
@@ -461,16 +555,100 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
},
Remove: func(ctx context.Context, module string, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := removeComposition(actx, opts, module, others, sched, say)
res := removeComposition(actx, opts, module, others, ag.Placed(module), sched, say)
done()
afterApply()
sayFacts(ctx, ag)
return res
},
}
// The node composes its own modules once the catalogue places it (novox/hq ADR 0547 §2, ADR 0549, issue 1126): its
// placement key is read at the agent's start; until then it applies the controller's compositions as before.
run, stopRun := context.WithCancel(linked)
// What the node runs and how it opens a sealed value, for the composer to keep the boxes it runs; and the agent
// told when the node cannot compose, so each wanted module's have says it.
// The agent of this run alone: its composer tells it, never the agent of a run begun since (issue 1126: a pass in
// flight at the switch marked the next agent's modules failed).
own := &agentNow{}
hooks := composerHooks{
running: func() []delivery.Composed {
if now := own.get(); now != nil {
return runsAsWritten(now.Everything(), opts.state)
}
return nil
},
unseal: sealOpener(opts.state),
// And what each module's have says it runs sealed: after a restart the agent has applied nothing yet.
// A have the bus did not answer for is said as such, never as one holding no seals (issue 1126, 2026-10-28).
held: func(module string) (delivery.Seals, []string, error) {
if module == agent.NodeModule {
got, _, err := session.ReadNodeHaveErr(linked)
return got.SealedAt, got.Sealed, err
}
got, ok, err := session.ReadHaveErr(linked, module)
if !ok {
return nil, nil, err
}
return got.SealedAt, got.Sealed, nil
},
refused: func(why string, held map[string]string) {
if now := own.get(); now != nil && run.Err() == nil {
now.NotComposedEach(run, why, held) // a module held alone marks that module, never the others (issue 1126)
}
},
// And the vault's authority beside the controller's, and this node's certificate asked of the vault when a
// module here asks for one (ADR 0550 §2).
vaultAuthority: trust.VaultAuthority,
certify: func(ctx context.Context) error {
return certifyThisNode(ctx, opts.state, node, trust, session.AskVault, func() { sayFacts(ctx, ag) })
},
}
// And its answer to every placement candidate, placed or not, applying nothing (novox/hq ADR 0559 §4): written into
// its own key, `<node>.node`, under candidates.
shadowHooks := hooks
shadowHooks.refused, shadowHooks.certify = nil, nil
shadowEnv := composerEnv(session, node, say, shadowHooks)
// And, after each answer, what runs of an own secret the answer says the vault holds another value of, handed to
// the vault while the node applies the controller's compositions (issue 1126, cause C4; ownresync).
composes := &composesNow{}
resync := ownResync(session, node, say, shadowEnv, composes)
shadow := selfcompose.NewShadow(shadowEnv,
func(ctx context.Context, answers map[string]delivery.CandidateAnswer) {
session.NoteCandidates(ctx, delivery.WithValuesDiffer(answers))
go resync.take(ctx, answers)
})
go shadow.Keep(run, link.Backoff) // this run's alone: a switch begins a run, and its shadow, again
composer, why := selfComposer(linked, session, node, say, hooks)
if linked.Err() != nil {
// The link ended before the node knew whose compositions to apply: it chose neither, and waits for the next.
stopRun()
cancel()
continue
}
if composer != nil {
machinesNow.Store(composer) // the machines a failing network part points at, by the placements (issue 774)
composes.set(delivery.ComposesOwn)
env.Composed, env.SelfComposed, env.Trusted, env.Policy = composer, true, nil, composer.Policy
env.HoldsData = composer.HoldsData // what runs from the kept declaration says Data as its manifest does
session.NoteComposes(linked, delivery.ComposesOwn, why)
// And once the catalogue no longer places it, the agent begins again on the controller's compositions: the
// placement key deleted ends this run, the way back (issue 1126, the seed's revert).
go endWhenUnplaced(run, session, node, stopRun, say)
} else {
composes.set(delivery.ComposesController)
go resync.again(run)
session.NoteComposes(linked, delivery.ComposesController, why)
// And once the catalogue places it, the agent begins again composing its own: the placement key appearing ends
// this run, so a node does not wait for a restart to follow the seed of its placement (ADR 0549 §4).
go endWhenPlaced(run, session, node, stopRun, say, func(reason string) {
session.NoteComposes(linked, delivery.ComposesController, why+"; "+reason)
})
}
ag = agent.New(env)
own.set(ag)
current.set(ag)
err = ag.Run(linked)
err = ag.Run(run)
stopRun()
cancel()
current.set(nil)
if ctx.Err() != nil {
@@ -517,7 +695,13 @@ func (c *agentNow) get() *agent.Agent {
}
// applyComposition applies one composition: a module's, or the node's own.
func applyComposition(ctx context.Context, opts options, c delivery.Composed, others []delivery.Composed,
//
// placed is the modules a want still places here besides this one: none of their resources is removed by this apply,
// which is refused instead (issue 774).
//
// removed is the modules whose want says removed: in the node's own apply, a resource of any other module goes only when
// its module is among them (issue 774).
func applyComposition(ctx context.Context, opts options, c delivery.Composed, others []delivery.Composed, placed, removed map[string]bool,
sched *apply.Scheduler, say link.Announce) agent.Result {
name := scopeName(c)
if c.Module == "" {
@@ -533,7 +717,13 @@ func applyComposition(ctx context.Context, opts options, c delivery.Composed, ot
if err := refuseEmptyWrites(d); err != nil {
return failedWith(err.Error())
}
scope := apply.Scope{Module: name, Others: resourcesOf(others)}
scope := apply.Scope{Module: name, Others: resourcesOf(others), Placed: placed}
if name == agent.NodeModule {
scope.RemovedByWant = removed
if scope.RemovedByWant == nil {
scope.RemovedByWant = map[string]bool{}
}
}
got := applyModuleHeld(ctx, opts, d, &scope, sched, say)
if got.Refused != "" {
return failedWith(got.Refused)
@@ -552,9 +742,9 @@ func applyComposition(ctx context.Context, opts options, c delivery.Composed, ot
}
// removeComposition removes one module's own resources and nothing of another's.
func removeComposition(ctx context.Context, opts options, module string, others []delivery.Composed,
func removeComposition(ctx context.Context, opts options, module string, others []delivery.Composed, placed map[string]bool,
sched *apply.Scheduler, say link.Announce) agent.Result {
scope := apply.Scope{Module: module, Others: resourcesOf(others)}
scope := apply.Scope{Module: module, Others: resourcesOf(others), Placed: placed}
got := applyModuleHeld(ctx, opts, &declaration.Declaration{Version: declaration.Version}, &scope, sched, say)
switch {
case got.Refused != "":
@@ -579,8 +769,23 @@ func busyModules() []string {
return out
}
// keptDeclaration is the declaration the node kept before the switch, verified with the mesh's key when it is given.
// machinesNow is the composer whose placements name the mesh's machines by address, for the health report's targets
// (writeHealthReport); nil on a node that does not compose its own, whose parts then name no target.
var machinesNow atomic.Pointer[selfcompose.Composer]
// keptRetired says the kept declaration is read no more: every module the seat's placement puts on this node has a have of
// its own (placedAllHave), so what runs is the agent's alone (ADR 0550 §2, design 47's L3 s9).
var keptRetired atomic.Bool
// errKeptRetired is keptDeclaration's answer once the read is retired.
var errKeptRetired = errors.New("the kept declaration is read no more: every module placed on this node has a have of its own")
// keptDeclaration is the declaration the node kept before the switch, verified with the mesh's key when it is given;
// once the read is retired, nothing is read.
func keptDeclaration(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
if keptRetired.Load() {
return nil, errKeptRetired
}
var raw []byte
if len(signer) > 0 {
var err error
@@ -748,3 +953,149 @@ const applyBound = 30 * time.Minute
func applyContext(ctx context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(context.WithoutCancel(ctx), applyBound)
}
// selfComposer is the node's own composer when the catalogue places the node (its key `placement.<node>` in the
// indexer's state), and nil, said once, while it does not or the key cannot be read.
// composerHooks are what the composer reads of the agent and the machine (selfcompose.Env Running, Unseal, Refused).
type composerHooks struct {
running func() []delivery.Composed
unseal func(string) ([]byte, error)
refused func(why string, held map[string]string)
held func(module string) (delivery.Seals, []string, error)
// vaultAuthority and certify are selfcompose.Env's VaultAuthority and Certify.
vaultAuthority func() string
certify func(ctx context.Context) error
}
func selfComposer(ctx context.Context, session *link.Session, node string, say func(string), hooks composerHooks) (*selfcompose.Composer, string) {
key, err := delivery.PlacementKey(node)
if err != nil {
say("this node applies the controller's compositions: " + err.Error())
return nil, err.Error()
}
env := composerEnv(session, node, say, hooks)
unreadable := ""
placed, err := selfcompose.Placed(ctx, env.Get, node, link.Backoff, func(m string) {
if strings.Contains(m, "cannot be read") {
unreadable = m
}
say(m)
}, selfcompose.ComposedItsOwn(ctx, env.Get, node))
if err != nil {
return nil, "its placement could not be read before the link ended: " + err.Error()
}
if !placed {
why := "the seat's placement state holds no key for it"
if unreadable != "" {
why = "its placement cannot be read, and it never composed its own: " + unreadable
}
say("this node applies the controller's compositions: " + why)
return nil, why
}
why := "the seat's placement state holds its key (" + key + ")"
say("this node composes its own modules: " + why)
return selfcompose.New(env), why
}
// composerEnv is what the node's composers read of the bus, the vault and the agent: its own composer and its answer
// to placement candidates (selfcompose.Shadow) alike.
func composerEnv(session *link.Session, node string, say func(string), hooks composerHooks) selfcompose.Env {
return selfcompose.Env{Node: node,
Watch: func(ctx context.Context, bucket, pattern string) (<-chan agent.Update, error) {
if bucket == delivery.PlacementBucket || bucket == delivery.PlacementCandidatesBucket {
// The seat's states are made when their holder starts: a node-engine that starts first waits for the
// bus's word that the bucket was created, and the shadow and the composer say they wait.
return bucketWatch{session, bucket}.WatchAwait(ctx, pattern, func(err error) {
say("the seat's state " + bucket + " is not on the bus yet (" + err.Error() + "): it is watched when the bus says it was created")
})
}
return bucketWatch{session, bucket}.Watch(ctx, pattern)
},
Get: func(ctx context.Context, bucket, key string) ([]byte, error) {
h, err := session.KVHistory(ctx, bucket, key)
if err != nil || len(h) == 0 || h[len(h)-1].Deleted {
return nil, err
}
return h[len(h)-1].Value, nil
},
Mesh: frombus.Mesh{InternalSuffix: overlay.Suffix()},
Say: say,
// Its modules' own secrets, sealed by the vault to this machine's key (ADR 0550 §5).
Own: func(ctx context.Context, args map[string]any) (json.RawMessage, error) {
return session.AskVault(ctx, "own", args)
},
// And the vault's word that a secret changed, on which it composes again (ADR 0547 §2).
Hear: session.Hear,
// And a moved module's templates archive, in the artifact store, as every archive is fetched (issue 1126).
Fetch: apply.Fetch,
Running: hooks.running, Unseal: hooks.unseal, Refused: hooks.refused, Held: hooks.held,
VaultAuthority: hooks.vaultAuthority, Certify: hooks.certify}
}
// endWhenUnplaced calls stop once this node's placement key is deleted from the seat's placement (after the watch
// caught up, or gone already), so the agent begins again and applies the controller's compositions, as a node the
// catalogue does not place does from its start: the way back (issue 1126, the seed's revert). A watch that cannot be
// made is said; the node then returns at its next start.
func endWhenUnplaced(ctx context.Context, session *link.Session, node string, stop func(), say link.Announce) {
key, err := delivery.PlacementKey(node)
if err != nil {
return
}
updates, err := bucketWatch{session, delivery.PlacementBucket}.Watch(ctx, key)
if err != nil {
say("whether this node's key in the seat's placement state still exists is not followed (" + err.Error() + "): it returns to the controller's compositions at its next start")
return
}
placed := false
for u := range updates {
switch {
case u.Caught && placed:
continue
case u.Caught || u.Deleted || len(u.Value) == 0:
say("this node's key in the seat's placement state is gone (" + key + "): the node agent begins again, applying the controller's compositions")
stop()
return
default:
placed = true
}
}
if ctx.Err() == nil {
say("the watch of " + key + " ended while the link stays: whether this node's key in the seat's placement state still exists is followed again at the node agent's next start")
}
}
// endWhenPlaced calls stop once this node's placement key appears in the seat's placement (a value after the watch
// caught up, or one already there), so the agent begins again and composes its own. A watch that cannot be made is said;
// the node then switches at its next start.
func endWhenPlaced(ctx context.Context, session *link.Session, node string, stop func(), say link.Announce, note func(string)) {
key, err := delivery.PlacementKey(node)
if err != nil {
return
}
// The seat's placement state is made when its holder starts; the delivery brings the grants part and the holder first,
// so it is normally there. A node-engine that starts before it waits for the bus's word that the bucket was created
// (no timer), says so once in the journal and in the node's own have, where the watchman reads it, and stays on the
// controller's compositions until its key is written. A watch the account may not make is said the same way, and the
// node stays until its next start.
updates, err := bucketWatch{session, delivery.PlacementBucket}.WatchAwait(ctx, key, func(err error) {
why := "this node's key in the seat's placement state is waited for, the state is not on the bus yet (" + err.Error() + "): it stays on the controller's compositions and is watched when the bus says the state was created"
say(why)
note(why)
})
if err != nil {
why := "this node's key in the seat's placement state cannot be followed (" + err.Error() + "): it stays on the controller's compositions until its next start"
say(why)
note(why)
return
}
for u := range updates {
if !u.Caught && !u.Deleted && len(u.Value) > 0 {
say("this node's key in the seat's placement state is written now (" + key + "): the node agent begins again, composing its own modules")
stop()
return
}
}
if ctx.Err() == nil {
say("the watch of " + key + " ended while the link stays: whether this node's key in the seat's placement state is written is followed again at the node agent's next start")
}
}
+478 -13
View File
@@ -14,11 +14,14 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/selfcompose"
"github.com/novox/mesh-host/internal/store"
)
// The node agent whole, against a real bus with JetStream started in the test: the wants bucket, the node's composition
@@ -28,9 +31,14 @@ import (
const itNode = "it1"
type itBed struct {
t *testing.T
conn *nats.Conn
js nats.JetStreamContext
t *testing.T
conn *nats.Conn
js nats.JetStreamContext
// nodeConn and nodeJS are the node's own, under its real grant (buscompose.PermissionsFor, KindNode): the agent runs
// on them, so a watch or a read the grant does not allow fails here as it fails on the bus (issue 1126, 2026-10-27
// 09:21: a watch of every have key stopped every node agent). conn is the test's, unrestricted, for setting up.
nodeConn *nats.Conn
nodeJS nats.JetStreamContext
wants nats.KeyValue
composed nats.KeyValue
haves nats.KeyValue
@@ -40,12 +48,27 @@ type itBed struct {
controll ed25519.PrivateKey
operator ed25519.PrivateKey
said []string
// refusals are the bus's asynchronous errors on the node's connection: each permissions violation it met.
refusals []string
mu sync.Mutex
}
func aServer(t *testing.T) *nats.Conn {
// aServer is an embedded bus with JetStream and two users: the test's, unrestricted, and the node's, granted exactly what
// buscompose.PermissionsFor gives a node (KindNode); it answers a connection of each.
func aServer(t *testing.T, principal buscompose.Principal, refused func(error)) (admin, node *nats.Conn) {
t.Helper()
srv, err := server.NewServer(&server.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true})
perms, err := buscompose.PermissionsFor(principal)
if err != nil {
t.Fatal(err)
}
granted := &server.Permissions{Publish: &server.SubjectPermission{Allow: perms.Publish, Deny: perms.PublishDeny},
Subscribe: &server.SubjectPermission{Allow: perms.Subscribe}}
if perms.AllowResponses {
granted.Response = &server.ResponsePermission{MaxMsgs: 1, Expires: buscompose.ResponseTTL}
}
nodeUser := buscompose.Principal{Kind: buscompose.KindNode, Node: itNode}.Username()
srv, err := server.NewServer(&server.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true,
Users: []*server.User{{Username: "test", Password: "test"}, {Username: nodeUser, Password: "node", Permissions: granted}}})
if err != nil {
t.Fatal(err)
}
@@ -54,15 +77,25 @@ func aServer(t *testing.T) *nats.Conn {
t.Fatal("the embedded server did not start")
}
t.Cleanup(srv.Shutdown)
conn, err := nats.Connect(srv.ClientURL())
if err != nil {
if admin, err = nats.Connect(srv.ClientURL(), nats.UserInfo("test", "test")); err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
return conn
t.Cleanup(admin.Close)
if node, err = nats.Connect(srv.ClientURL(), nats.UserInfo(nodeUser, "node"), nats.CustomInboxPrefix("_INBOX."+nodeUser),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refused(err) })); err != nil {
t.Fatal(err)
}
t.Cleanup(node.Close)
return admin, node
}
func newItBed(t *testing.T) *itBed {
t.Helper()
return newItBedAs(t, buscompose.Principal{Kind: buscompose.KindNode, Node: itNode})
}
// newItBedAs is newItBed with the node's connection under the grant of principal, a node's (KindNode).
func newItBedAs(t *testing.T, principal buscompose.Principal) *itBed {
t.Helper()
// Not skipped off Arch: a package manager that answers is put on PATH, so the apply of files runs on any build node.
bin := t.TempDir()
@@ -74,11 +107,19 @@ func newItBed(t *testing.T) *itBed {
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
b := &itBed{t: t, conn: aServer(t), dir: t.TempDir()}
b := &itBed{t: t, dir: t.TempDir()}
b.conn, b.nodeConn = aServer(t, principal, func(err error) {
b.mu.Lock()
b.refusals = append(b.refusals, err.Error())
b.mu.Unlock()
})
var err error
if b.js, err = b.conn.JetStream(); err != nil {
t.Fatal(err)
}
if b.nodeJS, err = b.nodeConn.JetStream(); err != nil {
t.Fatal(err)
}
composedBucket, _ := delivery.ComposedBucketFor(itNode)
for _, c := range []struct {
name string
@@ -112,13 +153,14 @@ func (b *itBed) start(trustOperator bool) (stop func()) {
}
}
b.session = &link.Session{Membership: link.Membership{Node: itNode}, Timeout: 2 * time.Second,
Say: b.say}
b.session.Attach(context.Background(), link.OverNATS{Conn: b.conn, JS: b.js, Node: itNode})
Say: b.say, Keys: func() link.RecordKeys { return link.RecordKeys{} }}
b.session.Attach(context.Background(), link.OverNATS{Conn: b.nodeConn, JS: b.nodeJS, Node: itNode})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
defer close(done)
nodeAgent(ctx, b.opts, b.controll.Public().(ed25519.PublicKey), b.session, nil, b.say, func() {},
controller := b.controll.Public().(ed25519.PublicKey)
nodeAgent(ctx, b.opts, controller, newMeshTrust(controller), b.session, nil, b.say, func() {},
&agentNow{}, func(context.Context, *agent.Agent) {})
}()
return func() { cancel(); <-done }
@@ -133,6 +175,8 @@ func (b *itBed) say(line string) {
func fpIt(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
func fpDigest(c string) string { return "sha256:" + strings.Repeat(c, 64) }
// compose writes a module's signed composition of one file into the node's bucket and returns its digest.
func (b *itBed) compose(module, fp, content string, data bool, signer ed25519.PrivateKey) string {
b.t.Helper()
@@ -197,6 +241,7 @@ func within(t *testing.T, d time.Duration, what string, ok func() bool) {
// applied within one second and its have names the fingerprint and the digest; a composition whose digest no want names
// is not applied.
func TestNatsAWantAndItsCompositionAreAppliedWithinASecond(t *testing.T) {
t.Run("1957: an engine start starts a declared-running unit", testNatsAnEngineStartStartsADeclaredRunningUnitTheHaveSaysRuns)
b := newItBed(t)
stop := b.start(false)
defer stop()
@@ -311,3 +356,423 @@ func TestNatsTheHandPathAppliesACompositionTheOperatorSigned(t *testing.T) {
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
})
}
// Issue 1126 (ADR 0547 §2, the milestone's H8): the node agent started whole against a real bus, once the catalogue
// places its node (the placement written after it started), composes a module from its want, the indexer's manifest, its record and its mesh-config value, and applies it,
// with no key in the controller's compositions; a changed config value is composed again on its event and applied as a
// revision of what runs. And a catalogue pull request's placement candidate equal to its placement is answered "same"
// in the node's own key, and nothing is applied for it (ADR 0559 §4, subtask 10 of the milestone).
//
// And (issue 1126, ADR 0560) it reads, through its own session under the grant the grants part composes for it
// (buscompose.Users, so it carries ServedReads), the build of a placed module that declares one and the one config value
// a provider on another node serves from (a mail domain); under a grant without that served key the bus refuses the
// read and the node composes nothing.
func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
users, err := buscompose.Users(buscompose.Records{Nodes: []string{itNode, "it2"},
Assigned: map[string][]buscompose.Declared{"it2": {{Module: "mailer", Served: []string{"domain"}}}}})
if err != nil {
t.Fatal(err)
}
var principal buscompose.Principal
for _, u := range users {
if u.Kind == buscompose.KindNode && u.Node == itNode {
principal = u
}
}
if len(principal.ServedReads) != 1 {
t.Fatalf("the grants part's principal of %s reads %v of the served keys", itNode, principal.ServedReads)
}
t.Run("without the served key the bus refuses the read", func(t *testing.T) {
b := newItBed(t)
stop, _ := h8(t, b, false)
defer stop()
// 15 s, above link.historyWithin (10 s): the bus's refusal of a publish reaches only the connection's error
// handler (nats.go fails no request on it), so the refused read ends at its own bound, then the node says it.
within(t, 15*time.Second, "the bus did not refuse the served key's read, or the node did not say it", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return strings.Contains(strings.Join(b.refusals, "\n"), "Permissions Violation for Publish to \"$JS.API.CONSUMER.CREATE.KV_"+
selfcompose.ResolvedBucket) && strings.Contains(strings.Join(b.refusals, "\n"), ".it2.mailer.domain") &&
strings.Contains(strings.Join(b.said, "\n"), "what mailer serves on it2 (domain)")
})
if b.file("web") != "" {
t.Errorf("composed and applied under a grant without the served key: %q", b.file("web"))
}
})
b := newItBedAs(t, principal)
stop, st := h8(t, b, true)
defer stop()
placement, resolved, candidates := st.placement, st.resolved, st.candidates
hkey, place, fp := st.hkey, st.place, st.fp
within(t, 5*time.Second, "the node did not compose and apply its own module once placed", func() bool { return b.file("web") == "greeting=bonjour\n" })
h8rest(t, b, resolved, candidates, hkey, place, fp)
t.Run("one answer per candidate revision", func(t *testing.T) { h8once(t, b, resolved, candidates, place) })
h8back(t, b, placement, resolved, candidates, hkey)
}
// h8once is issue 1126, 2026-10-28 (shanks answered probe/1126-seed-7 at revision 84 every 13-17 s, its bus flooded):
// a candidate revision is answered once. The node's own answer written, a module's have whose health words alone move
// (each look rewrites them) and a config value written again with the value it held answer nothing again; the
// candidate's next revision is answered once.
func h8once(t *testing.T, b *itBed, resolved, candidates nats.KeyValue, place []byte) {
answering := func() int {
b.mu.Lock()
defer b.mu.Unlock()
n := 0
for _, l := range b.said {
if strings.HasPrefix(l, "answering placement candidate feat-x ") {
n++
}
}
return n
}
time.Sleep(time.Second)
settled := answering()
if settled == 0 {
t.Fatal("feat-x was never answered")
}
hkey, _ := delivery.HaveKey(itNode, "web")
for _, detail := range []string{"its tool web_health: no link to the bus is open", "its tool web_health: no answer within 10s"} {
h := b.have("web")
h.Health, h.Detail = delivery.HealthFailed, detail
raw, err := delivery.EncodeHave(h)
if err != nil {
t.Fatal(err)
}
if _, err := b.haves.Put(hkey, raw); err != nil {
t.Fatal(err)
}
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err)
}
}
time.Sleep(1500 * time.Millisecond)
if got := answering(); got != settled {
t.Errorf("feat-x answered %d more time(s) at the same revision, with nothing it is composed from changed", got-settled)
}
same, _ := delivery.DecodePlacement(place)
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "feat-x", Placement: &same})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.PlacementCandidateKey("feat-x", itNode)
if _, err := candidates.Put(ckey, craw); err != nil {
t.Fatal(err)
}
within(t, 10*time.Second, "the candidate's next revision was not answered", func() bool { return answering() > settled })
time.Sleep(1500 * time.Millisecond)
if got := answering(); got != settled+1 {
t.Errorf("the candidate's next revision was answered %d time(s), want 1", got-settled)
}
}
// h8back is the way back (issue 1126, the seed's revert): a candidate deleting the node's placement is compared with
// the controller's compositions at its wants (here none: refused, naming web), and the placement key deleted returns the agent to them without a
// restart, said in the journal and the node's have, keeping what runs.
func h8back(t *testing.T, b *itBed, placement, resolved, candidates nats.KeyValue, hkey string) {
t.Helper()
node := func() delivery.HaveNode {
e, err := b.haves.Get(hkey)
if err != nil {
return delivery.HaveNode{}
}
n, _ := delivery.DecodeHaveNode(e.Value())
return n
}
if n := node(); n.Composes != delivery.ComposesOwn {
t.Errorf("a node composing its own says it composes %q", n.Composes)
}
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "revert-x"})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.PlacementCandidateKey("revert-x", itNode)
if _, err := candidates.Put(ckey, craw); err != nil {
t.Fatal(err)
}
// The controller holds no composition at web's want (the node composed its own): returning would leave web with
// nothing to apply, and the answer says so rather than "same" (review of the shadow, round 1).
within(t, 10*time.Second, "the node did not refuse returning to compositions the controller does not hold", func() bool {
a := node().Candidates["revert-x"]
return a.Proposal == "revert-x" && a.Returns == "" && a.Verdict == delivery.CandidateRefused &&
strings.Contains(a.Refused, "web: the controller holds no composition")
})
ran := b.have("web").Composed
pkey, _ := delivery.PlacementKey(itNode)
// A pass in flight at the switch (issue 1126, 2026-10-28: the last pass of the own run, failing on its run's end,
// marked the next agent's modules failed): a config value changed starts one, and the key goes at once.
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err)
}
if err := placement.Delete(pkey); err != nil {
t.Fatal(err)
}
stopWatch := make(chan struct{})
marked := make(chan Have1126, 1)
go func() {
for {
select {
case <-stopWatch:
return
case <-time.After(20 * time.Millisecond):
}
if h := b.have("web"); h.Phase == delivery.PhaseFailed {
select {
case marked <- Have1126{h.Phase, h.Detail}:
default:
}
return
}
}
}()
defer close(stopWatch)
within(t, 15*time.Second, "the node did not return to the controller's compositions when its placement went", func() bool {
b.mu.Lock()
said := strings.Join(b.said, "\n")
b.mu.Unlock()
return strings.Contains(said, "key in the seat's placement state is gone") && node().Composes == delivery.ComposesController
})
time.Sleep(3 * time.Second) // what a pass in flight would mark lands within it
select {
case h := <-marked:
t.Errorf("a pass in flight at the switch marked web failed: %+v", h)
default:
}
if h := b.have("web"); h.Composed != ran || h.Phase == delivery.PhaseFailed || b.file("web") != "greeting=salut\n" {
t.Errorf("returning to the controller's compositions moved what runs: %+v, %q", h, b.file("web"))
}
}
// Have1126 is what H8's way back watches of web's have.
type Have1126 struct {
Phase delivery.Phase
Detail string
}
// h8state is what h8 set up that the rest of H8 writes to.
type h8state struct {
placement, resolved, candidates nats.KeyValue
hkey string
place []byte
fp string
}
// h8 sets H8's bus up and starts the agent, then places the node. With checkBuild, the placed module's build key is
// first one that does not decode, which the node must read through its session (and say), then a done one.
func h8(t *testing.T, b *itBed, checkBuild bool) (func(), h8state) {
t.Helper()
kvOf := func(bucket string) nats.KeyValue {
kv, err := b.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 5})
if err != nil {
t.Fatal(err)
}
return kv
}
placement, modules, manifests := kvOf(delivery.PlacementBucket), kvOf(delivery.ModulesBucket), kvOf(delivery.ManifestsBucket)
resolved := kvOf(selfcompose.ResolvedBucket)
builds := kvOf(delivery.BuilderBuildsBucket)
candidates := kvOf(delivery.PlacementCandidatesBucket)
// The controller's stream the node's facts go into, so the node's key carries the facts it says.
if _, err := b.js.AddStream(&nats.StreamConfig{Name: "CONTROL", Subjects: []string{link.ReportSubject(itNode)}}); err != nil {
t.Fatal(err)
}
path := filepath.Join(b.dir, "web.conf")
man := `{"module":"web","version":"1","config":{"greeting":{"type":"string","description":"what the file says","default":"hello"}},` +
`"resources":[{"id":"conf","type":"file","path":"` + path + `","mode":"0644","content":"greeting=${config:greeting}\n"}],` +
`"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]}}`
digest := delivery.ManifestDigest([]byte(man))
mraw, _ := delivery.EncodeManifest(delivery.Manifest{Module: "web", JSON: man})
mkey, _ := delivery.ManifestsKey(digest)
fp := fpIt("a")
modv, _ := delivery.EncodeModule(delivery.Module{FP2: fp, Manifest: digest, Repository: "novox/web", Commit: strings.Repeat("1", 40)})
place, _ := delivery.EncodePlacement(delivery.Placement{Node: itNode, Modules: []delivery.PlacedModule{{Module: "web"}}})
rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: itNode, Capabilities: []string{"service-manager"}}})
pkey, _ := delivery.PlacementKey(itNode)
hkey, _ := delivery.HaveNodeKey(itNode)
// A provider on it2 serving from its own config value: a mail domain, which it1's composition settles.
mailer := `{"module":"mailer","version":"1","config":{"domain":{"type":"string","description":"the mail domain","required":true}},` +
`"provides":[{"name":"smtp","scope":"mesh"}],"serves":{"smtp":{"port":587,"domain":"${config:domain}"}}}`
mdigest := delivery.ManifestDigest([]byte(mailer))
mlraw, _ := delivery.EncodeManifest(delivery.Manifest{Module: "mailer", JSON: mailer})
mlkey, _ := delivery.ManifestsKey(mdigest)
mlmod, _ := delivery.EncodeModule(delivery.Module{FP2: fpIt("b"), Manifest: mdigest, Repository: "novox/mailer", Commit: strings.Repeat("2", 40)})
place2, _ := delivery.EncodePlacement(delivery.Placement{Node: "it2", Modules: []delivery.PlacedModule{{Module: "mailer"}}})
rec2, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: "it2", Capabilities: []string{"service-manager"}}})
pkey2, _ := delivery.PlacementKey("it2")
hkey2, _ := delivery.HaveNodeKey("it2")
buildKey, _ := delivery.BuilderBuildKey("web", fp)
done, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: digest,
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Reference: "artifact-store://web/files/blobs/" + fpDigest("f"), Digest: fpDigest("f")}}})
firstBuild := done
if checkBuild {
firstBuild = []byte(`{"v":1,"state":"no such state"}`)
}
for _, w := range []struct {
kv nats.KeyValue
key string
value []byte
}{{manifests, mkey, mraw}, {modules, "web", modv}, {b.haves, hkey, rec},
{resolved, itNode + ".web.greeting", []byte(`{"value":"bonjour","source":"module"}`)},
{manifests, mlkey, mlraw}, {modules, "mailer", mlmod}, {b.haves, hkey2, rec2}, {placement, pkey2, place2},
{resolved, "it2.mailer.domain", []byte(`{"value":"example.org","source":"module"}`)}, {builds, buildKey, firstBuild}} {
if _, err := w.kv.Put(w.key, w.value); err != nil {
t.Fatal(err)
}
}
want, _ := delivery.EncodeWant(delivery.Want{FP: fp, Self: true})
wkey, _ := delivery.WantKey(itNode, "web")
if _, err := b.wants.Put(wkey, want); err != nil {
t.Fatal(err)
}
stop := b.start(false)
// Not placed yet: the agent applies the controller's compositions, of which there are none. The catalogue's seed then
// places the node, and the agent begins again composing its own (ADR 0549 §4), with no restart of the engine.
time.Sleep(300 * time.Millisecond)
if b.file("web") != "" {
t.Fatal("a node the catalogue does not place composed its own")
}
if _, err := placement.Put(pkey, place); err != nil {
t.Fatal(err)
}
if checkBuild {
// The build of the placed module is read through the node's session: one that does not decode is said, and nothing
// is applied; a done one, then a config event, and the node composes.
within(t, 5*time.Second, "the node did not read web's build key through its session", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return strings.Contains(strings.Join(b.said, "\n"), "the builder's "+buildKey)
})
if b.file("web") != "" {
t.Fatal("composed with a build key that does not decode")
}
if _, err := builds.Put(buildKey, done); err != nil {
t.Fatal(err)
}
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"bonjour","source":"module"}`)); err != nil {
t.Fatal(err)
}
}
return stop, h8state{placement: placement, resolved: resolved, candidates: candidates, hkey: hkey, place: place, fp: fp}
}
// h8rest is H8 once the node composes its own: the have, a config revision, and the candidate answered "same".
func h8rest(t *testing.T, b *itBed, resolved, candidates nats.KeyValue, hkey string, place []byte, fp string) {
t.Helper()
within(t, 2*time.Second, "the have does not name the build", func() bool {
h := b.have("web")
return h.FP == fp && h.Composed != "" && h.Phase == delivery.PhaseRunning
})
if keys, _ := b.composed.Keys(); len(keys) != 0 {
t.Errorf("the controller's compositions hold %v; the node composed its own", keys)
}
first := b.have("web").Composed
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err)
}
within(t, 3*time.Second, "a changed config value was not composed and applied again", func() bool { return b.file("web") == "greeting=salut\n" })
within(t, 2*time.Second, "the have does not name the revision", func() bool {
h := b.have("web")
return h.FP == fp && h.Composed != first && h.Phase == delivery.PhaseRunning
})
// The node says its facts (its record, as the test wrote it), then a candidate equal to its placement is written: the
// node answers "same" at the candidate's head and revision in its own key, and applies nothing.
b.session.SayFacts(context.Background(), link.Report{Profile: map[string]any{"capabilities": []any{
map[string]any{"name": "service-manager", "present": true}}}})
second := b.have("web").Composed
same, _ := delivery.DecodePlacement(place)
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "feat-x", Placement: &same})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.PlacementCandidateKey("feat-x", itNode)
crev, err := candidates.Put(ckey, craw)
if err != nil {
t.Fatal(err)
}
var answer delivery.CandidateAnswer
within(t, 10*time.Second, "the node did not answer its candidate", func() bool {
e, err := b.haves.Get(hkey)
if err != nil {
return false
}
n, err := delivery.DecodeHaveNode(e.Value())
answer = n.Candidates["feat-x"]
return err == nil && answer.Proposal == "feat-x" && n.Record != nil
})
if answer.Verdict != delivery.CandidateSame || answer.Revision != crev || len(answer.Same) != 1 || answer.Same[0] != "web" {
t.Errorf("the answer to a candidate equal to the placement: %+v", answer)
}
if h := b.have("web"); h.Composed != second || b.file("web") != "greeting=salut\n" {
t.Errorf("a candidate moved what runs: %+v, %q", h, b.file("web"))
}
}
// 1957/AC1 (integration; ADR 0568 §1): a node agent starting on a bus that holds a `running` have for a module at its want,
// with a service manager (a systemctl on the PATH) that shows the module's unit inactive, starts the unit once and writes
// the have running; the same start with the unit active applies nothing.
func testNatsAnEngineStartStartsADeclaredRunningUnitTheHaveSaysRuns(t *testing.T) {
for _, active := range []bool{false, true} {
b := newItBed(t)
bin := t.TempDir()
stateFile, calls := filepath.Join(bin, "state"), filepath.Join(bin, "calls")
if active {
os.WriteFile(stateFile, []byte("active"), 0o644)
} else {
os.WriteFile(stateFile, []byte("inactive"), 0o644)
}
script := "#!/bin/sh\necho \"$@\" >> " + calls + "\ncase \"$1\" in\n" +
"show) printf 'LoadState=loaded\\nActiveState=%s\\nType=simple\\n' \"$(cat " + stateFile + ")\" ;;\n" +
"start) printf active > " + stateFile + " ;;\nis-enabled) echo enabled ;;\nesac\nexit 0\n"
if err := os.WriteFile(filepath.Join(bin, "systemctl"), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
// The engine's own record of the unit, as a previous run left it, and the have it wrote then.
res, _ := json.Marshal(map[string]any{"id": "web.unit", "type": "service", "unit": "web-it.service", "state": "running", "module": "web"})
var known store.State
known.Record(store.Applied{ID: "web.unit", Type: "service", Target: "web-it.service"})
if err := store.Save(b.opts.state, known); err != nil {
t.Fatal(err)
}
fp := fpIt("a")
c := delivery.Composed{Node: itNode, Module: "web", FP: fp, Resources: []json.RawMessage{res}}
signed, _ := delivery.SignComposed(c, b.controll)
raw, _ := delivery.EncodeComposed(signed)
dig, _ := delivery.ComposedDigestOf(signed)
key, _ := delivery.ComposedKey(itNode, "web", fp)
if _, err := b.composed.Put(key, raw); err != nil {
t.Fatal(err)
}
b.want("web", fp, dig, "")
hkey, _ := delivery.HaveKey(itNode, "web")
have, err := delivery.EncodeHave(delivery.Have{FP: fp, Composed: dig, Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown})
if err != nil {
t.Fatal(err)
}
if _, err := b.haves.Put(hkey, have); err != nil {
t.Fatal(err)
}
stop := b.start(false)
within(t, 5*time.Second, "the unit was not started and the have not running", func() bool {
got, _ := os.ReadFile(stateFile)
h := b.have("web")
return string(got) == "active" && h.Phase == delivery.PhaseRunning && h.Composed == dig
})
time.Sleep(300 * time.Millisecond)
stop()
starts := 0
log, _ := os.ReadFile(calls)
for _, l := range strings.Split(string(log), "\n") {
if strings.HasPrefix(l, "start") {
starts++
}
}
if want := map[bool]int{false: 1, true: 0}[active]; starts != want {
t.Fatalf("unit active=%v: started %d times, want %d", active, starts, want)
}
}
}
+94
View File
@@ -0,0 +1,94 @@
package main
import (
"context"
"encoding/json"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/ownresync"
"github.com/novox/mesh-host/internal/selfcompose"
)
// composesNow is whose compositions this run of the agent applies, once decided ("" before).
type composesNow struct {
mu sync.Mutex
now string
}
func (c *composesNow) set(s string) { c.mu.Lock(); c.now = s; c.mu.Unlock() }
func (c *composesNow) get() string { c.mu.Lock(); defer c.mu.Unlock(); return c.now }
// resyncStep is the step that hands the vault what runs of an own secret a shadow answer says another value of (issue
// 1126, cause C4; internal/ownresync): one pass at a time, and the last answers taken again once the agent has decided
// whose compositions it applies.
type resyncStep struct {
r *ownresync.Resync
one sync.Mutex
mu sync.Mutex
last map[string]delivery.CandidateAnswer
// tries and pending are the step taken again after a bound when a read or an ask the bus did not answer stopped it
// (Outcome.Retry, issue 1126, 2026-10-28), one wait at a time.
tries int
pending bool
}
func ownResync(session *link.Session, node string, say func(string), env selfcompose.Env, composes *composesNow) *resyncStep {
return &resyncStep{r: ownresync.New(ownresync.Env{Node: node, Composes: composes.get, Running: env.Running,
Get: env.Get, Unseal: env.Unseal, Seal: identity.Seal, Say: say,
Ask: func(ctx context.Context, verb string, args map[string]any) (json.RawMessage, error) {
return session.AskVault(ctx, verb, args)
}})}
}
// take keeps answers as the last and takes them.
func (s *resyncStep) take(ctx context.Context, answers map[string]delivery.CandidateAnswer) {
s.mu.Lock()
s.last = answers
s.mu.Unlock()
s.again(ctx)
}
// again takes the last answers, and once more after a bound (selfcompose.LaterWait) while the bus did not answer for one.
func (s *resyncStep) again(ctx context.Context) {
s.one.Lock()
defer s.one.Unlock()
s.mu.Lock()
last := s.last
s.mu.Unlock()
if last == nil {
return
}
retry := false
for _, o := range s.r.Take(ctx, last) {
retry = retry || o.Retry
}
s.mu.Lock()
defer s.mu.Unlock()
if !retry {
s.tries = 0
return
}
if s.pending || ctx.Err() != nil {
return
}
wait := selfcompose.LaterWait(s.tries)
s.tries++
s.pending = true
go func() {
select {
case <-ctx.Done():
case <-time.After(wait):
}
s.mu.Lock()
s.pending = false
s.mu.Unlock()
if ctx.Err() == nil {
s.again(ctx)
}
}()
}
+168
View File
@@ -0,0 +1,168 @@
package main
import (
"context"
"encoding/json"
"fmt"
"regexp"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
// provisionSecrets fills a provision's value into a file the provisioner's client hands the node-engine to write
// (novox/hq ADR 0570 clause 6): the client fills the file but for each `${secret:<provision>}`, and the node-engine
// fills those at the write, so the client never holds the value. A whole `secrets` file of a handed consumer is the
// placeholder alone.
//
// A placeholder names the credential as the module calls it, `${secret:<name>}` (catalogue.SecretLocal, ADR 0094): the
// local name where the module keeps several values of one provision, else the provision. For each, it finds the one
// provision key `<node>.<consumer>.<provision>[.<local>]` of the seat mesh-provisioner's state `provisions` (clause 1)
// going by that name, which must be `provided` and carry the vault's reference; asks the vault for that pair
// credential sealed to this machine's key (the verb `own`, ADR 0349 §11 with the node-engine standing for the client,
// ADR 0550 §5); and opens it with this machine's sealing key. Anything short of that refuses the write, naming the
// provision and why: a file written with a placeholder left in it, or with no value, is a module that starts and cannot
// connect.
type provisionSecrets struct {
node string
// list is every key of mesh-provisioner's `provisions` under prefix (`<node>.<consumer>.`), by key: what this
// engine may read, its own node's (`<node>.>`).
list func(ctx context.Context, prefix string) (map[string][]byte, error)
// own asks the vault's verb `own` in this engine's name.
own func(ctx context.Context, args map[string]any) (json.RawMessage, error)
// open opens a value sealed to this machine.
open func(sealed string) ([]byte, error)
}
// secretPlaceholder is `${secret:<name>}`.
var secretPlaceholder = regexp.MustCompile(`\$\{secret:([A-Za-z0-9_-]+)\}`)
// fill is content with each provision placeholder of module's filled, and whether any was.
func (p provisionSecrets) fill(ctx context.Context, module string, content []byte) ([]byte, bool, error) {
found := secretPlaceholder.FindAllSubmatch(content, -1)
if len(found) == 0 {
return content, false, nil
}
if module == "" {
return nil, false, fmt.Errorf("the file names %d provision secret(s) and belongs to no module, so whose they are is not known", len(found))
}
if p.list == nil || p.own == nil || p.open == nil {
return nil, false, fmt.Errorf("%s's provisions cannot be read on this machine: the node-engine is not linked to the mesh", module)
}
prefix := p.node + "." + module + "."
keys, err := p.list(ctx, prefix)
if err != nil {
return nil, false, fmt.Errorf("%s's provisions cannot be read: %w", module, err)
}
out := string(content)
done := map[string]bool{}
for _, f := range found {
whole, name := string(f[0]), string(f[1])
if done[whole] {
continue
}
done[whole] = true
pr, key, err := byName(keys, name)
if err != nil {
return nil, false, fmt.Errorf("%s's secret %q: %w", module, name, err)
}
value, err := p.value(ctx, key, pr)
if err != nil {
return nil, false, err
}
out = strings.ReplaceAll(out, whole, string(value))
}
return []byte(out), true, nil
}
// byName is the one provision of keys the module calls name (catalogue.SecretLocal), and its key.
func byName(keys map[string][]byte, name string) (provisions.Provision, string, error) {
var hit []string
var got provisions.Provision
for key, raw := range keys {
pr, err := provisions.Decode(raw)
if err != nil {
return provisions.Provision{}, "", fmt.Errorf("the provision %s: %w", key, err)
}
if catalogue.SecretLocal(pr.Provision, pr.Local) == name {
hit, got = append(hit, key), pr
}
}
switch len(hit) {
case 0:
return provisions.Provision{}, "", fmt.Errorf("no provision of %s goes by that name: nothing asks for it", provisions.ProvisionsRead)
case 1:
return got, hit[0], nil
}
sort.Strings(hit)
return provisions.Provision{}, "", fmt.Errorf("the provisions %s all go by that name", strings.Join(hit, ", "))
}
// value is one provision's value on this node, opened.
func (p provisionSecrets) value(ctx context.Context, key string, pr provisions.Provision) ([]byte, error) {
if pr.State != provisions.Provided || pr.Reference == nil {
why := pr.State
if pr.Why != "" {
why += ": " + pr.Why
}
if pr.State == provisions.Provided {
why = "provided with no reference to the vault's value"
}
return nil, fmt.Errorf("the provision %s is %s, so its value is not written yet", key, why)
}
// The reference names a credential of this provision, as the vault names it (provisions.VaultName).
ours := false
for _, form := range []string{provisions.FormPair, provisions.FormShared, provisions.FormIssued} {
ours = ours || provisions.VaultName(form, pr) == pr.Reference.ID
}
if !ours {
return nil, fmt.Errorf("the provision %s names the vault's %s, which is no credential of it", key, pr.Reference.ID)
}
module, provision, local := pr.Consumer.Module, pr.Provision, pr.Local
type asked struct {
Role string `json:"role"`
Provision string `json:"provision"`
Consumer string `json:"consumer"`
ConsumerModule string `json:"consumer_module"`
Provider string `json:"provider"`
Local string `json:"local,omitempty"`
}
ask := asked{Role: "consumer", Provision: provision, Consumer: p.node, ConsumerModule: module, Provider: pr.Provider.Node, Local: local}
answer, err := p.own(ctx, map[string]any{"credentials": []asked{ask}})
if err != nil {
return nil, fmt.Errorf("the vault's value of %s: %w", key, err)
}
var a struct {
Node string `json:"node"`
Credentials []struct {
asked
Name string `json:"name"`
Generation int `json:"generation"`
Sealed json.RawMessage `json:"sealed"`
} `json:"credentials"`
}
if err := json.Unmarshal(answer, &a); err != nil {
return nil, fmt.Errorf("the vault's answer for %s is not one: %w", key, err)
}
if a.Node != p.node {
return nil, fmt.Errorf("the vault answered %s's credentials for %s on %s", a.Node, key, p.node)
}
for _, c := range a.Credentials {
if c.asked != ask {
continue
}
// The value the provision names, by the vault's credential name and its generation (clause 6): another is refused.
if c.Name != pr.Reference.ID || c.Generation != pr.Reference.Generation {
return nil, fmt.Errorf("the vault answered %s generation %d for %s, and the provision names %s generation %d: "+
"nothing is written until they agree", c.Name, c.Generation, key, pr.Reference.ID, pr.Reference.Generation)
}
value, err := p.open(string(c.Sealed))
if err != nil {
return nil, fmt.Errorf("the vault's value of %s does not open on this machine: %w", key, err)
}
return value, nil
}
return nil, fmt.Errorf("the vault answered no value of %s for this machine", key)
}
+2
View File
@@ -36,6 +36,7 @@ func TestACompositionIsRecordedOnlyWhenEveryResourceIs(t *testing.T) {
// The modules applied again are those with a missing resource, each once; one down but there is not among them.
func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
t.Run("1957: the machine shows a resource not active", testTheMachineShowsAResourceNotActive)
st := liveness.Statement{Resources: []liveness.State{
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.homeserver"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.db"}, State: liveness.Unhealthy, Missing: true},
@@ -51,6 +52,7 @@ func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
// not acted on again, so a resource that cannot be made is not applied (and backed up) at every unrelated change. One
// that comes back and goes again is newly missing again (the review of issue 1945).
func TestOnlyANewlyMissingResourceAppliesItsModuleAgain(t *testing.T) {
t.Run("1957: a resource newly down is an edge", testOnlyAResourceNewlyDownIsAnEdge)
missing := func(ids ...string) liveness.Statement {
var st liveness.Statement
for _, id := range ids {
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/store"
)
// runsAsWritten is what the node runs as the composer reads it (selfcompose.Env.Running), with every file given by digest
// named as this node last wrote it, where the two differ (issue 924, novox 2026-10-29 01:16). What runs of a module the
// agent has not applied since its start can come from the kept declaration, older than the compositions applied since:
// its export (mesh-vault.kept) then named a digest the controller's sweep had removed, while the file on disk was the
// newer export the node wrote. A node composing its own carries such a file as it runs (mesh-sdk frombus keptOf), so it
// names what is on disk: **only when the node's record says it wrote exactly that content there** (store Applied.Wrote,
// Target), never a file someone else changed. A file that cannot be read, or that the record does not vouch for, is
// left as the composition names it.
func runsAsWritten(running []delivery.Composed, statePath string) []delivery.Composed {
if len(running) == 0 || statePath == "" {
return running
}
known, err := store.Load(statePath)
if err != nil {
return running
}
out := make([]delivery.Composed, len(running))
for i, c := range running {
out[i] = c
var changed []json.RawMessage
for j, raw := range c.Resources {
if fixed, ok := asWritten(raw, known); ok {
if changed == nil {
changed = append([]json.RawMessage(nil), c.Resources...)
}
changed[j] = fixed
}
}
if changed != nil {
out[i].Resources = changed
}
}
return out
}
// asWritten is the file resource raw named by the digest and size of what this node last wrote at its path, when raw is a
// file given by another digest and the file there is what the record says the node wrote.
func asWritten(raw json.RawMessage, known store.State) (json.RawMessage, bool) {
var fields map[string]json.RawMessage
if json.Unmarshal(raw, &fields) != nil {
return nil, false
}
var kind, id, path, digest string
_ = json.Unmarshal(fields["type"], &kind)
_ = json.Unmarshal(fields["id"], &id)
_ = json.Unmarshal(fields["path"], &path)
_ = json.Unmarshal(fields["digest"], &digest)
if kind != string(declaration.TypeFile) || digest == "" || path == "" {
return nil, false
}
rec, ok := known.Find(id)
if !ok || rec.Wrote == "" || rec.Target != path {
return nil, false
}
body, err := os.ReadFile(path)
if err != nil {
return nil, false
}
sum := sha256.Sum256(body)
if hex.EncodeToString(sum[:]) != rec.Wrote {
return nil, false // changed since the node wrote it: not the node's to vouch for
}
if now := declaration.DigestOf(body); now == digest {
return nil, false
} else {
fields["digest"], _ = json.Marshal(now)
}
fields["size"], _ = json.Marshal(len(body))
fixed, err := json.Marshal(fields)
if err != nil {
return nil, false
}
return fixed, true
}
+317
View File
@@ -0,0 +1,317 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// **What this engine trusts a mesh signature from** (novox/hq ADR 0550 §2; design 47's L3 item d). Two signers: the
// controller's key, which the node was enrolled with (its membership's signer), and the vault's key, read from the vault's
// state `signers`, which only the vault may write: the bus grants are its anchor, as the wants' and the placement's (the
// lead's design limit, 2026-10-28). The controller's key is trusted until its lease is handed away, heard on the lease key
// (buscompose.LeaseHandedAway) and never on a clock; from then the vault's alone. What it trusts now the node states in its
// record (NodeRecord.Trusts), so the gate before the lease write reads every node's from the bus.
type meshTrust struct {
mu sync.Mutex
controller ed25519.PublicKey
handedAway bool
vault ed25519.PublicKey
authority string
// changed is told when what it trusts changed, so the record is written again.
changed func()
}
// newMeshTrust trusts the controller's key alone until the vault's is read.
func newMeshTrust(controller ed25519.PublicKey) *meshTrust {
t := &meshTrust{}
if len(controller) == ed25519.PublicKeySize {
t.controller = controller
}
return t
}
// onChange names who is told when what it trusts changed.
func (t *meshTrust) onChange(f func()) {
t.mu.Lock()
t.changed = f
t.mu.Unlock()
}
// Controller is the controller's key while the controller holds its lease, and nil once the lease is handed away.
func (t *meshTrust) Controller() ed25519.PublicKey {
if t == nil {
return nil
}
t.mu.Lock()
defer t.mu.Unlock()
if t.handedAway {
return nil
}
return t.controller
}
// HandedAway says the controller's lease was handed away, as heard on the lease key.
func (t *meshTrust) HandedAway() bool {
if t == nil {
return false
}
t.mu.Lock()
defer t.mu.Unlock()
return t.handedAway
}
// Signers are the keys a mesh signature is verified against now: the controller's while it holds the lease, and the
// vault's once read.
func (t *meshTrust) Signers() []delivery.Signer {
t.mu.Lock()
defer t.mu.Unlock()
var out []delivery.Signer
if t.controller != nil && !t.handedAway {
out = append(out, delivery.Signer{Name: delivery.SignerController, Key: t.controller})
}
if t.vault != nil {
out = append(out, delivery.Signer{Name: delivery.SignerVault, Key: t.vault})
}
return out
}
// Verify is the signer that signed msg as kind, or why none it trusts did.
func (t *meshTrust) Verify(kind string, msg, sig []byte) (delivery.Signer, error) {
context, err := delivery.SignContext(kind)
if err != nil {
return delivery.Signer{}, err
}
return delivery.VerifyBy(t.Signers(), context, msg, sig)
}
// VaultAuthority is the vault's authority certificate as read from `signers`, or "".
func (t *meshTrust) VaultAuthority() string {
if t == nil {
return ""
}
t.mu.Lock()
defer t.mu.Unlock()
return t.authority
}
// Trusts is what the record says it trusts: each signer's key and the vault's authority, by fingerprint.
func (t *meshTrust) Trusts() []string {
if t == nil {
return nil
}
authorities := map[string]string{}
if a := t.VaultAuthority(); a != "" {
authorities[delivery.SignerVault] = a
}
return delivery.Trusts(t.Signers(), authorities)
}
// change applies f under the lock and tells who follows when Trusts differs after it.
func (t *meshTrust) change(f func()) {
before := strings.Join(t.Trusts(), "\n")
t.mu.Lock()
f()
changed := t.changed
t.mu.Unlock()
if strings.Join(t.Trusts(), "\n") != before && changed != nil {
changed()
}
}
// heardSigner takes one key of the vault's `signers`: its key or its authority, refused (and said) when it does not
// check; a deleted key is no longer trusted.
func (t *meshTrust) heardSigner(key string, value []byte, deleted bool, say link.Announce) {
switch key {
case delivery.SignersKeyKey:
var pub ed25519.PublicKey
if !deleted && len(value) > 0 {
k, err := delivery.DecodeSignerKey(value)
if err == nil {
pub, err = k.PublicKey()
}
if err != nil {
say("the vault's signing key in " + delivery.SignersRead + " is not trusted: " + err.Error())
return
}
}
t.change(func() { t.vault = pub })
case delivery.SignersAuthorityKey:
pem := ""
if !deleted && len(value) > 0 {
a, err := delivery.DecodeSignerAuthority(value)
if err != nil {
say("the vault's authority in " + delivery.SignersRead + " is not trusted: " + err.Error())
return
}
pem = a.Certificate
}
t.change(func() { t.authority = pem })
}
}
// heardLease takes the controller's lease key as read: once handed away to a holder that is no controller, the
// controller's key is trusted no more. A lease that cannot be read changes nothing.
func (t *meshTrust) heardLease(raw []byte, found bool, readErr error) {
if readErr != nil {
return
}
handed, _ := buscompose.LeaseHandedAway(raw, found, nil)
t.change(func() { t.handedAway = handed })
}
// trustFollowAgain bounds the wait before a watch that could not be made is tried again on the same link: the vault may
// not serve `sign` yet, and until it does this node is not granted the reads.
var trustFollowAgain = 2 * time.Minute
// followTrust follows the vault's `signers` and the controller's lease on every link, as watches (never a poll), for as
// long as ctx lasts. A watch that cannot be made is said once per reason and tried again.
func followTrust(ctx context.Context, session *link.Session, trust *meshTrust, say link.Announce) {
said := ""
sayOnce := func(line string) {
if line != said {
said = line
if line != "" {
say(line)
}
}
}
for ctx.Err() == nil {
linked, cancel, err := session.WhenLinked(ctx)
if err != nil {
return
}
watching, stop := context.WithCancel(linked)
signers, serr := session.KVWatch(watching, delivery.SignersBucket, ">")
lease, lerr := session.KVWatch(watching, buscompose.LeaseBucket, buscompose.ControllerLeaseKey)
if serr != nil || lerr != nil {
stop()
sayOnce(fmt.Sprintf("what this node trusts a mesh signature from is not followed yet (the vault's signers: %v; "+
"the controller's lease: %v): it trusts what it last read, and looks again in %s", serr, lerr, trustFollowAgain))
signers, lease = nil, nil
} else {
sayOnce("")
}
for signers != nil || lease != nil {
select {
case u, ok := <-signers:
if !ok {
signers = nil
continue
}
if !u.Caught {
trust.heardSigner(u.Key, u.Value, u.Deleted, say)
}
case u, ok := <-lease:
if !ok {
lease = nil
continue
}
if !u.Caught {
trust.heardLease(u.Value, !u.Deleted && len(u.Value) > 0, nil)
}
}
}
stop()
select {
case <-linked.Done():
case <-time.After(trustFollowAgain):
}
cancel()
}
}
// placedAllHave says whether every module the seat mesh-placement places on node has a have of its own, read from the
// bus: then the kept declaration is read no more (keptRetired). A node the seat does not place, or whose placement or haves
// cannot be read, keeps reading it.
func placedAllHave(ctx context.Context, session *link.Session, node string) (bool, string) {
key, err := delivery.PlacementKey(node)
if err != nil {
return false, err.Error()
}
h, err := session.KVHistory(ctx, delivery.PlacementBucket, key)
if err != nil || len(h) == 0 || h[len(h)-1].Deleted {
return false, "its placement is not read"
}
p, err := delivery.DecodePlacement(h[len(h)-1].Value)
if err != nil {
return false, err.Error()
}
return allHave(p, func(module string) (bool, error) {
_, ok, err := session.ReadHaveErr(ctx, module)
return ok, err
})
}
// allHave says whether has finds a have for every module p places, and why not.
func allHave(p delivery.Placement, has func(module string) (bool, error)) (bool, string) {
if len(p.Modules) == 0 {
return false, "the seat mesh-placement places no module here"
}
for _, m := range p.Modules {
ok, err := has(m.Module)
if err != nil {
return false, "the have of " + m.Module + " is not read: " + err.Error()
}
if !ok {
return false, m.Module + " has no have yet"
}
}
return true, fmt.Sprintf("each of the %d modules placed here has a have of its own", len(p.Modules))
}
// certifyThisNode asks the vault's `sign` for this node's own certificate (kind node-certificate, ADR 0550 §2), checks
// it is for this node's internal name and serving key and signed by the authority the answer names, which must be the
// vault's this node trusts, keeps it where the record reads it, and writes the record again (written).
func certifyThisNode(ctx context.Context, state, node string, trust *meshTrust,
ask func(ctx context.Context, verb string, args any) (json.RawMessage, error), written func()) error {
serving, err := identity.LoadServingKey(identity.ServingKeyPath(state))
if err != nil {
return err
}
key, err := base64.StdEncoding.DecodeString(serving.Public)
if err != nil || len(key) != ed25519.PublicKeySize {
return fmt.Errorf("this node's serving key is not an ed25519 key")
}
raw, err := ask(ctx, delivery.SignVerb, map[string]any{"kind": delivery.KindNodeCertificate, "node": node})
if err != nil {
return err
}
var a delivery.SignAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return fmt.Errorf("the vault's answer is not a signing: %w", err)
}
trusted := trust.VaultAuthority()
if trusted == "" {
return fmt.Errorf("this node has read no authority of the vault's to check the certificate against")
}
if err := delivery.CheckNodeCertificate(a.Certificate, []string{trusted}, overlay.InternalName(node),
ed25519.PublicKey(key), time.Now()); err != nil {
return err
}
path := identity.CertificatePath(state)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
if err := os.WriteFile(path, []byte(a.Certificate), 0o644); err != nil {
return err
}
if written != nil {
written()
}
return nil
}
+59
View File
@@ -1,6 +1,9 @@
package main
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -9,6 +12,7 @@ import (
"github.com/novox/mesh-host/internal/accounts"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/units"
)
@@ -103,6 +107,61 @@ func TestTheStatementSaysAnAccountJudgedForRoot(t *testing.T) {
if got["claude-code.agent"] != "never" || got["openrazer.account"] != "" {
t.Fatalf("root said: %v", got)
}
// 774 (ADR 0563 §6): the node seat's root-free passes only on the engine's own positive verdict at most 15 minutes old
// with execute not served here; a stale, missing or negative verdict, or execute served, is no pass and says why.
t.Run("this machine's root-free", func(t *testing.T) {
agent := accounts.Verdict{Account: accounts.Account{Module: "claude-code", ID: "claude-code.agent", Name: "agent", Root: true},
State: accounts.Healthy, Since: at}
free := &accounts.Statement{At: at, Accounts: []accounts.Verdict{agent}}
for _, row := range []struct {
name string
st *accounts.Statement
after time.Duration
served bool
free bool
why string
}{
{name: "a positive verdict", st: free, after: 14 * time.Minute, free: true, why: "the account agent cannot become root"},
{name: "a stale verdict", st: free, after: 16 * time.Minute, why: "more than 15 minutes ago"},
{name: "no verdict", st: nil, why: "judged no account yet"},
{name: "a way to root", st: as, why: "in the group docker, which grants root"},
{name: "no agent account", st: &accounts.Statement{At: at}, why: "agents run as the operator's account"},
{name: "execute served", st: free, served: true, why: "the login shell runs any command"},
} {
v := judgeRootFree("ace", row.st, at.Add(row.after), row.served, "the bus hears execute answered here")
if v.Free != row.free || !strings.Contains(v.Why, row.why) || v.Machine != "ace" {
t.Errorf("%s: %+v", row.name, v)
}
}
// execute is served while the holder's setting file says anything but withhold, or the bus hears it.
dir := t.TempDir()
statePath, conf := filepath.Join(dir, "state.json"), filepath.Join(dir, "tools.conf")
if err := store.Save(statePath, store.State{Resources: []store.Applied{{ID: "zsh.tools-conf", Type: "file", Target: conf}}}); err != nil {
t.Fatal(err)
}
quiet := func() (bool, error) { return false, nil }
for _, row := range []struct {
name, setting string
heard func() (bool, error)
served bool
}{
{name: "withheld and not heard", setting: "# the mesh's\nexecute=withhold\n", heard: quiet},
{name: "served", setting: "execute=serve\n", heard: quiet, served: true},
{name: "withheld and heard", setting: "execute=withhold\n", heard: func() (bool, error) { return true, nil }, served: true},
{name: "the bus not asked", setting: "execute=withhold\n", heard: func() (bool, error) { return false, errors.New("no link") }, served: true},
{name: "no setting file", heard: quiet, served: true},
} {
_ = os.Remove(conf)
if row.setting != "" {
if err := os.WriteFile(conf, []byte(row.setting), 0o644); err != nil {
t.Fatal(err)
}
}
if served, why := executeServedHere(statePath, row.heard); served != row.served {
t.Errorf("%s: served %v (%s)", row.name, served, why)
}
}
})
}
// A verdict waiting for the setuid search says how far it got, in the reason the root-free verb repeats and as
+77
View File
@@ -7,9 +7,13 @@ import (
"errors"
"fmt"
"os"
"path/filepath"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/seal"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
@@ -42,6 +46,35 @@ func userListFollower(opts options, session *link.Session, composedBucket string
Report: func(ctx context.Context, module string, w userlist.Written) {
session.NoteUserList(ctx, module, w)
},
// From take-over, the grants part's list (ADR 0547 clauses 7, 8): its one key watched, its object read by name and
// opened with this machine's sealing key, the key's public half said in the have so the grants part seals to it.
Grants: bucketWatch{session, delivery.GrantsUserListBucket},
Fetch: func(ctx context.Context, name string) ([]byte, error) {
return session.GrantsObject(ctx, name, grantsListLimit)
},
Open: func(b seal.Box) ([]byte, error) {
key, err := identity.LoadSealingKey(identity.SealingKeyPath(opts.state))
if err != nil {
return nil, err
}
return key.OpenVaultBox(b)
},
Key: func() string {
key, err := identity.LoadSealingKey(identity.SealingKeyPath(opts.state))
if err != nil {
return ""
}
pem, err := key.PublicPEM()
if err != nil {
return ""
}
return pem
},
// The grants part's key as last heard, kept beside the node's state, followed while it cannot be watched; its
// watch tried again as a passing failure of the bus is (ADR 0547 clause 8).
Kept: func() ([]byte, uint64) { return keptGrantsKey(opts) },
Keep: func(value []byte, rev uint64) { keepGrantsKey(opts, value, rev, say) },
Backoff: link.Backoff,
// The vault's key from its answer to node-key: the loaded list's hashes are handed sealed to it (ADR 0547
// clause 9).
VaultKey: vaultKey.get,
@@ -49,6 +82,9 @@ func userListFollower(opts options, session *link.Session, composedBucket string
})
}
// grantsListLimit is the largest sealed list the bus node reads: the live list is about a hundred kilobytes.
const grantsListLimit = 16 << 20
// userListFile is a user-list resource as the file it declares.
func userListFile(raw []byte) (*declaration.File, error) {
r, err := declaration.ParseResource(raw)
@@ -125,3 +161,44 @@ func writeUserListAlone(ctx context.Context, opts options, raw []byte, body []by
}
return digest, writeErr == nil && out.Action != "unchanged", writeErr
}
// grantsKeyPath is where the grants part's user-list key last heard is kept, beside the node's state.
func grantsKeyPath(opts options) string {
return filepath.Join(filepath.Dir(opts.state), "grants-user-list.json")
}
type keptKey struct {
Revision uint64 `json:"revision"`
Value json.RawMessage `json:"value"`
}
func keptGrantsKey(opts options) ([]byte, uint64) {
raw, err := os.ReadFile(grantsKeyPath(opts))
if err != nil {
return nil, 0
}
var k keptKey
if json.Unmarshal(raw, &k) != nil || len(k.Value) == 0 {
return nil, 0
}
return k.Value, k.Revision
}
func keepGrantsKey(opts options, value []byte, rev uint64, say link.Announce) {
path := grantsKeyPath(opts)
if value == nil {
if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
announceOr(say)("the grants part's user-list key kept on this machine could not be taken away: " + err.Error())
}
return
}
raw, _ := json.Marshal(keptKey{Revision: rev, Value: value})
tmp := path + ".new"
err := os.WriteFile(tmp, raw, 0o600)
if err == nil {
err = os.Rename(tmp, path)
}
if err != nil {
announceOr(say)("the grants part's user-list key could not be kept on this machine: " + err.Error())
}
}
+37
View File
@@ -4,6 +4,7 @@ import (
"crypto/ed25519"
"encoding/json"
"errors"
"path/filepath"
"strings"
"testing"
@@ -11,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/store"
)
func aFileOf(module, id string) json.RawMessage {
@@ -59,6 +61,41 @@ func TestDeclaredNowReadsWhatRunsWhenTheAgentIsRunning(t *testing.T) {
if err != nil || d.Provisioner != "from-agent" {
t.Fatalf("declaredNow read %+v (%v), not what runs", d, err)
}
// L3 s9 (ADR 0550 §2): a node whose placed modules all have a have reads the kept declaration no more, though the
// file is there and readable and the controller's key is dropped; until then it reads it.
state := filepath.Join(t.TempDir(), "state", "state.json")
if err := store.SaveDeclared(store.DeclaredPath(state), store.Declared{Declaration: []byte(`{"declaration":1,` +
`"resources":[{"id":"zsh.rc","type":"file","module":"zsh","path":"/etc/zshrc","content":"x"}]}`)}); err != nil {
t.Fatal(err)
}
if d, err := keptDeclaration(state, nil); err != nil || len(d.Resources) != 1 {
t.Fatalf("before the read is retired: %+v, %v", d, err)
}
placed := delivery.Placement{Node: "n1", Modules: []delivery.PlacedModule{{Module: "zsh"}, {Module: "lemurs"}}}
for _, row := range []struct {
name string
has map[string]bool
want bool
}{
{"every placed module has a have", map[string]bool{"zsh": true, "lemurs": true}, true},
{"one has none yet", map[string]bool{"zsh": true}, false},
} {
if got, why := allHave(placed, func(m string) (bool, error) { return row.has[m], nil }); got != row.want {
t.Errorf("%s: %v (%s)", row.name, got, why)
}
}
if got, _ := allHave(delivery.Placement{Node: "n1"}, nil); got {
t.Error("a node with nothing placed retires the read")
}
keptRetired.Store(true)
defer keptRetired.Store(false)
if _, err := keptDeclaration(state, nil); !errors.Is(err, errKeptRetired) {
t.Errorf("after the read is retired: %v", err)
}
if by, filled := keptOnce(options{state: state}, nil)(); by != nil || filled != nil {
t.Errorf("after the read is retired the kept declaration was split: %v %v", by, filled)
}
}
// 1945/AC17 (unit): a resource the node's own composition shares with a running module's composition is declared once,
+1 -1
View File
@@ -3,7 +3,7 @@ module github.com/novox/mesh-host
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
+2 -8
View File
@@ -1,11 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f h1:lUEyNPPF3w4Q0LIgcUUD6ofkKVhCF0n7dVjMoixQmXo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456 h1:He2o9vSKjgKbWOCOgOJY0WlaDh1n8RcGhvGY/dwXsBQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e h1:k/1NjjW5HuuuHW2ldyWTvjp+1u3fN0RO18Q57Iup6Dk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936 h1:HGBoWimZXYZp+Qn9VCstYDitOP84t9EUQYd1yM9j2eg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+11
View File
@@ -240,6 +240,17 @@ func (c *SetuidCache) SearchAgain() {
c.forgetProgress()
}
// Running says whether a search walks now (novox/hq ADR 0563 §2: the node seat starts no second while one does).
// Nil c runs none.
func (c *SetuidCache) Running() bool {
if c == nil {
return false
}
c.mu.Lock()
defer c.mu.Unlock()
return c.running
}
// SearchNow throws away the last complete search, kept or in memory, and any walk under way or kept, and has
// the next look start a full search at once, past AgainAfter and any back-off (novox/hq issue 361): the
// operator changed by hand what the last one found, which no apply says. Until the new one completes the
+777 -26
View File
File diff suppressed because it is too large Load Diff
+656 -14
View File
@@ -4,12 +4,15 @@ import (
"context"
"crypto/ed25519"
"encoding/json"
"fmt"
"maps"
"slices"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -71,6 +74,19 @@ type haves struct {
stored map[string]Have
log []Have
removed []string
// unreadable is each module whose have the bus cannot read now (ReadErr).
unreadable map[string]bool
}
func (h *haves) ReadErr(ctx context.Context, module string) (Have, bool, error) {
h.mu.Lock()
bad := h.unreadable[module]
h.mu.Unlock()
if bad {
return Have{}, false, fmt.Errorf("nats: no responders available for request")
}
x, ok := h.Read(ctx, module)
return x, ok, nil
}
func (h *haves) Read(_ context.Context, module string) (Have, bool) {
@@ -326,6 +342,44 @@ func TestStartingOnWhatTheKeptDeclarationHoldsAppliesNothing(t *testing.T) {
// What the node-engine itself last applied is what runs after a restart, not the kept declaration.
func TestStartingOnItsOwnHaveAppliesNothing(t *testing.T) {
t.Run("1957: an engine start reads the machine's facts", testAnEngineStartReconcilesAModuleFromTheMachinesFacts)
t.Run("says the have again", testStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines)
// An apply that failed is not what runs, though its have names the want's build and digest: the agent starting on
// it applies again (review of the switch, round 1).
t.Run("applies again after an apply failure", func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "avahi", fp, false, file("avahi", "x", "1"))
wk, wraw := wantFor(t, "avahi", fp, dig, "")
b.haves.stored = map[string]Have{"avahi": {Module: "avahi", FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed),
Detail: "apply: unit failed to start"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
eventually(t, "a failed apply was not applied again", func() bool { return len(b.appliedList()) == 1 })
})
// Issue 1126, the stale marks after the seed's revert: a have a composer marked failed names what still runs; the
// agent starting on it applies nothing and says it running, so a mark it did not make clears.
t.Run("clears a composer's mark", func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "avahi", fp, false, file("avahi", "x", "1"))
wk, wraw := wantFor(t, "avahi", fp, dig, "")
b.haves.stored = map[string]Have{"avahi": {Module: "avahi", FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed),
Detail: NotComposedDetail + "frombus: the credentials of ace: context canceled"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
eventually(t, "the mark was not cleared", func() bool {
h := b.haves.last("avahi")
return h.Phase == string(delivery.PhaseRunning) && h.FP == fp && h.Composed == dig && h.Detail == ""
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module whose mark names what the want names was applied: %v", got)
}
})
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
@@ -345,7 +399,7 @@ func TestStartingOnItsOwnHaveAppliesNothing(t *testing.T) {
// it and writes the health this engine judges. Not said, its key kept the health an older engine wrote: xorg's have on
// g14 and shanks said unknown from before the engine said a module with nothing to judge ok, and the assigner held every
// module requiring x11-display back for it (2026-10-25).
func TestStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines(t *testing.T) {
func testStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "xorg", fp, false, file("xorg", "x", "1"))
@@ -378,6 +432,7 @@ func TestStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines(t *testing
// A module whose kept resources differ from the composition is applied, and a failed apply is retried only by a change
// of either key.
func TestAFailedApplyIsRetriedOnlyByAChangeOfEitherKey(t *testing.T) {
t.Run("1957: the judge's edge reconciles, and a failed re-apply is failed", testReconcileAppliesARunningModuleAgainAndLeavesTheHeldAndTheUnwantedAlone)
// A real failure is not retried by the clock, however short the backoff: only by a change of either key.
t.Run("a real failure", func(t *testing.T) {
b := newBed(t)
@@ -545,6 +600,7 @@ func TestABackupDoneIsFollowedByTheApplyAndANewModuleIsNotBackedUp(t *testing.T)
// R6-A AC4 at the agent: a removed want removes that module only, and its have is written removed.
func TestARemovedWantRemovesThatModuleOnly(t *testing.T) {
t.Run("its composition holds nothing back", testARemovedModulesCompositionHoldsNothingBack)
b := newBed(t)
stop := b.run()
defer stop()
@@ -598,6 +654,7 @@ func TestTheNodesOwnCompositionComesBeforeAnyModules(t *testing.T) {
// R6-A AC9: a composition signed by either trusted key is applied; one signed by another key, one changed after it was
// signed and one unsigned are refused and nothing of them is applied (ADR 0539 §3).
func TestOnlyACompositionSignedByATrustedKeyIsApplied(t *testing.T) {
t.Run("with no trusted key", testWithNoTrustedKeyNothingIsApplied)
b := newBed(t)
stop := b.run()
defer stop()
@@ -648,7 +705,7 @@ func TestOnlyACompositionSignedByATrustedKeyIsApplied(t *testing.T) {
}
// With no key trusted, nothing is applied: a node that does not know its signers fails closed.
func TestWithNoTrustedKeyNothingIsApplied(t *testing.T) {
func testWithNoTrustedKeyNothingIsApplied(t *testing.T) {
b := newBed(t)
b.env.Trusted = nil
stop := b.run()
@@ -712,6 +769,162 @@ func TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest(t *testing.T) {
if a.HoldsNode() {
t.Fatal("the agent holds no node composition")
}
// Issue 774 (novox, 2026-10-29 07:54 UTC): the kept declaration retired and the node's own composition in no bucket, a
// fresh engine's first own composition was composed from what runs with the node's own missing, so its folded modules
// (the bus server among them) were composed as nothing and removed. What this engine applied is kept on the machine
// and is what runs from before the first composition is watched; an apply keeps, a removal forgets.
t.Run("what this engine applied is what runs before the node composes its own", func(t *testing.T) {
b := newBed(t)
b.env.SelfComposed, b.env.Trusted, b.kept = true, nil, nil
store := &memApplied{kept: map[string][]byte{}}
b.env.Applied = store
// A folded module's resource in the node's own composition: its id names the module, as the controller folds it.
natsConf, _ := json.Marshal(map[string]any{"id": "nats.conf", "type": "file", "path": "/tmp/r6/nats/conf", "content": "listen 4222"})
own := delivery.Composed{Node: node, Resources: []json.RawMessage{natsConf}}
ownRaw, err := delivery.EncodeComposed(own)
if err != nil {
t.Fatal(err)
}
ownDig, _ := delivery.ComposedDigestOf(own)
web := delivery.Composed{Node: node, Module: "web", FP: fpOf("a"), Resources: []json.RawMessage{file("web", "x", "1")}}
webRaw, _ := delivery.EncodeComposed(web)
webDig, _ := delivery.ComposedDigestOf(web)
store.kept[NodeModule], store.kept["web"] = ownRaw, webRaw
b.haves.Write(context.Background(), Have{Module: NodeModule, Composed: ownDig})
b.haves.Write(context.Background(), Have{Module: "web", FP: fpOf("a"), Composed: webDig, Phase: string(delivery.PhaseRunning)})
// The composer's first pass reads what runs when its watch begins, before the agent reads any have.
firstCh := make(chan []delivery.Composed, 1)
comps := b.comps
var ag *Agent
b.env.Composed = watchFunc(func(ctx context.Context, p string) (<-chan Update, error) {
firstCh <- ag.Everything()
return comps.Watch(ctx, p)
})
ag = New(b.env)
b.ag = ag
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { defer close(done); _ = ag.Run(ctx) }()
defer func() { cancel(); <-done }()
var first []delivery.Composed
select {
case first = <-firstCh:
case <-time.After(5 * time.Second):
t.Fatal("the composer was not watched")
}
var ranOwn, ranWeb bool
for _, c := range first {
ranOwn = ranOwn || (c.Module == "" && len(c.Resources) == 1 && strings.Contains(string(c.Resources[0]), "4222"))
ranWeb = ranWeb || c.Module == "web"
}
if !ranOwn || !ranWeb {
t.Fatalf("what runs at the node's first own composition: own %v, web %v (%d compositions); want both as this engine applied them",
ranOwn, ranWeb, len(first))
}
// An apply keeps the composition it applied; a removal forgets it.
web2 := delivery.Composed{Node: node, Module: "web", FP: fpOf("b"), Resources: []json.RawMessage{file("web", "x", "2")}}
ck, _ := delivery.ComposedKey(node, "web", fpOf("b"))
raw2, _ := delivery.EncodeComposed(web2)
comps.put(ck, raw2)
wk, _ := delivery.WantKey(node, "web")
wraw, _ := delivery.EncodeWant(delivery.Want{FP: fpOf("b"), Self: true})
b.wants.put(wk, wraw)
eventually(t, "the applied composition was not kept", func() bool { return string(store.get("web")) == string(raw2) })
rraw, _ := delivery.EncodeWant(delivery.Want{Removed: true})
b.wants.put(wk, rraw)
eventually(t, "the removed module's composition was not forgotten", func() bool { return store.get("web") == nil })
})
// Issue 774, review of the re-land, round 1: at the re-landed engine's first start nothing is kept yet, and the node's
// have names a composition the node composed itself, in no bucket. The node's own runs as the kept declaration has it
// and is kept then, so the next start, the declaration retired, still runs nats.
t.Run("at the first start nothing is kept: the node's own runs as the kept declaration has it", func(t *testing.T) {
natsConf, _ := json.Marshal(map[string]any{"id": "nats.conf", "type": "file", "path": "/tmp/r6/nats/conf", "content": "listen 4222"})
store := &memApplied{kept: map[string][]byte{}}
h := &haves{}
h.Write(context.Background(), Have{Module: NodeModule, Composed: "sha256:" + strings.Repeat("c", 64)})
firstPass := func(kept map[string][]json.RawMessage) []delivery.Composed {
b := newBed(t)
b.env.SelfComposed, b.env.Trusted, b.kept, b.env.Applied, b.env.Haves = true, nil, kept, store, h
firstCh := make(chan []delivery.Composed, 1)
comps := b.comps
var ag *Agent
b.env.Composed = watchFunc(func(ctx context.Context, p string) (<-chan Update, error) {
firstCh <- ag.Everything()
return comps.Watch(ctx, p)
})
ag = New(b.env)
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { defer close(done); _ = ag.Run(ctx) }()
defer func() { cancel(); <-done }()
select {
case got := <-firstCh:
return got
case <-time.After(5 * time.Second):
t.Fatal("the composer was not watched")
}
return nil
}
runsNats := func(cs []delivery.Composed) bool {
for _, c := range cs {
if c.Module == "" && len(c.Resources) == 1 && strings.Contains(string(c.Resources[0]), "4222") {
return true
}
}
return false
}
if got := firstPass(map[string][]json.RawMessage{NodeModule: {natsConf}}); !runsNats(got) {
t.Fatalf("the first start does not run nats in the node's own (%d compositions)", len(got))
}
if store.get(NodeModule) == nil {
t.Fatal("the first start kept no node's own composition")
}
if got := firstPass(nil); !runsNats(got) {
t.Fatalf("the next start, the kept declaration retired, does not run nats in the node's own (%d compositions)", len(got))
}
})
}
// watchFunc is a Watcher from a function.
type watchFunc func(ctx context.Context, pattern string) (<-chan Update, error)
func (f watchFunc) Watch(ctx context.Context, pattern string) (<-chan Update, error) {
return f(ctx, pattern)
}
// memApplied is an AppliedStore in memory.
type memApplied struct {
mu sync.Mutex
kept map[string][]byte
}
func (m *memApplied) Load() (map[string][]byte, error) {
m.mu.Lock()
defer m.mu.Unlock()
out := map[string][]byte{}
for k, v := range m.kept {
out[k] = v
}
return out, nil
}
func (m *memApplied) Keep(module string, raw []byte) error {
m.mu.Lock()
defer m.mu.Unlock()
m.kept[module] = raw
return nil
}
func (m *memApplied) Drop(module string) error {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.kept, module)
return nil
}
func (m *memApplied) get(module string) []byte {
m.mu.Lock()
defer m.mu.Unlock()
return m.kept[module]
}
// A failed apply is applied again after a restart even when the kept declaration holds the composition: once this
@@ -783,11 +996,42 @@ func TestTheNodesOwnApplyHoldsOutAModuleThatMovedToItsOwnComposition(t *testing.
if !strings.Contains(strings.Join(names, ","), "systemd-resolved") {
t.Fatalf("the others of the node's own apply were %v: systemd-resolved's resources would be removed", names)
}
// Issue 1126, review of the apply, round 1: a module the composer holds alone (de-spiegel leaving the fold with its
// names unset) has no composition of its own in the bucket: what runs of it in the node's own is held out of the
// node's own apply until one is handed.
b2 := newBed(t)
spiegel := file("de-spiegel", "server-env", "A=1")
var bare map[string]any
_ = json.Unmarshal(spiegel, &bare)
delete(bare, "module")
spiegel, _ = json.Marshal(bare)
b2.env.Kept = func() map[string][]json.RawMessage { return map[string][]json.RawMessage{NodeModule: {spiegel}} }
nk2, nraw2, ndig2 := composition(t, NodeModule, "", false)
nwraw2, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig2})
b2.comps.before(nk2, nraw2)
b2.wants.before(nwk, nwraw2)
ctx2, cancel2 := context.WithCancel(context.Background())
b2.ag = New(b2.env)
b2.ag.NotComposedEach(ctx2, "", map[string]string{"de-spiegel": "de-spiegel says ${config:praktijk-name}, and nothing sets it"})
done2 := make(chan struct{})
go func() { defer close(done2); _ = b2.ag.Run(ctx2) }()
defer func() { cancel2(); <-done2 }()
eventually(t, "the node's own not applied beside a held module", func() bool { return len(b2.appliedList()) == 1 })
b2.mu.Lock()
defer b2.mu.Unlock()
heldOut := false
for _, c := range b2.others[0] {
heldOut = heldOut || (c.Module == "de-spiegel" && len(c.Resources) == 1)
}
if !heldOut {
t.Errorf("the others of the node's own apply were %+v: held de-spiegel's resources would be removed", b2.others[0])
}
}
// A module the assigner removed holds nothing back, its compositions still in the bucket or not: its resources are its
// removal's to take (issue 1945).
func TestARemovedModulesCompositionHoldsNothingBack(t *testing.T) {
func testARemovedModulesCompositionHoldsNothingBack(t *testing.T) {
b := newBed(t)
fp := fpOf("b")
rk, rraw, _ := composition(t, "gone", fp, false, file("gone", "x", "1"))
@@ -938,17 +1182,415 @@ func TestTheRevisionAWantNamesIsAppliedFromHistory(t *testing.T) {
}
}
// 1945/AC18 (unit): a module that still runs from the kept declaration carries the kept declaration's filled paths of
// its files, so the filling writes them ("not named as filled" for claude-code and screen-lock, 2026-10-25).
func TestAModuleRunningFromTheKeptDeclarationKeepsItsFilledPaths(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"claude-code": {file("claude-code", "settings", "")}}
b.env.KeptFilled = func() map[string][]string {
return map[string][]string{"claude-code": {"/tmp/r6/claude-code/settings"}}
// 1945/AC18 and issue 924 (unit): what runs of a module the agent has not taken yet carries the filled paths the
// filling judges by. From the kept declaration while the module has no have naming a composition ("not named as filled"
// for claude-code and screen-lock, 2026-10-25); from the composition its have names once it has one, the kept
// declaration being older, read at the start before any module settles and told to the filling then, so a file the
// newer composition fills is accepted on the first fill (power's three drop-ins on ace, g14 and shanks, 2026-10-29).
// Only a have in a phase baseline takes as running names what runs; and what runs carries Data, from the composition
// or, from the kept declaration, from the module's manifest, so a data module is not judged to move its data against
// its first composition after a restart (mongodb and mssql on novox).
func TestWhatRunsBeforeTheAgentTakesAModuleNamesItsFilledPaths(t *testing.T) {
const old, added = "/tmp/r6/power/settings", "/tmp/r6/power/bound"
filledFile := func(id string) json.RawMessage {
return json.RawMessage(`{"id":"power.` + id + `","type":"file","module":"power","path":"/tmp/r6/power/` + id +
`","mode":"0644","owner-at-runtime":"owned"}`)
}
a := New(b.env)
got := a.Everything()
if len(got) != 1 || got[0].Module != "claude-code" || strings.Join(got[0].Filled, ",") != "/tmp/r6/claude-code/settings" {
t.Fatalf("what runs from the kept declaration: %+v", got)
fp := fpOf("a")
ck, _ := delivery.ComposedKey(node, "power", fp)
applied := delivery.Composed{Node: node, Module: "power", FP: fp, Filled: []string{old, added}, Data: true,
Resources: []json.RawMessage{filledFile("settings"), filledFile("bound")}}
_, appliedRaw, appliedDig := signedBy(t, applied, ck, controllerKey)
newer := applied
newer.Resources = append([]json.RawMessage{}, applied.Resources...)
newer.Resources = append(newer.Resources, file("power", "later", "1"))
_, newerRaw, _ := signedBy(t, newer, ck, controllerKey)
for _, tc := range []struct {
name string
// have is the module's have; bucket the value its composition key holds now, history the revisions kept.
have *Have
bucket []byte
history []Update
filled, told string
// data is what runs' Data; holdsData the manifest's answer (nil: none read).
data bool
holdsData func(context.Context, string) (bool, bool)
}{
{name: "no have: the kept declaration's", filled: old},
{name: "a have naming the bucket's value", have: &Have{Module: "power", FP: fp, Composed: appliedDig},
bucket: appliedRaw, filled: old + "," + added, told: "the haves were read at the start", data: true},
{name: "a have applying names nothing that runs yet", have: &Have{Module: "power", FP: fp, Composed: appliedDig,
Phase: string(delivery.PhaseApplying)}, bucket: appliedRaw, filled: old, told: "the haves were read at the start"},
{name: "a data module from the kept declaration says Data as its manifest does", filled: old,
told: "the haves were read at the start", data: true,
holdsData: func(context.Context, string) (bool, bool) { return true, true }},
{name: "a have naming a revision replaced since", have: &Have{Module: "power", FP: fp, Composed: appliedDig},
bucket: newerRaw, history: []Update{{Key: ck, Value: appliedRaw, Revision: 3}, {Key: ck, Value: newerRaw, Revision: 7}},
filled: old + "," + added, told: "the haves were read at the start", data: true},
{name: "a have naming a composition the bucket no longer keeps", have: &Have{Module: "power", FP: fp, Composed: appliedDig},
bucket: newerRaw, filled: old, told: "the haves were read at the start"},
} {
t.Run(tc.name, func(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"power": {filledFile("settings")}}
b.env.KeptFilled = func() map[string][]string { return map[string][]string{"power": {old}} }
if tc.have != nil {
b.haves.stored = map[string]Have{"power": *tc.have}
}
if tc.bucket != nil {
b.comps.before(ck, tc.bucket)
}
b.env.Revisions = func(_ context.Context, key string) ([]Update, error) { return tc.history, nil }
b.env.HoldsData = tc.holdsData
var told []string
var mu sync.Mutex
b.env.RunsChanged = func(why string) { mu.Lock(); told = append(told, why); mu.Unlock() }
filledNow := func() string {
for _, c := range b.ag.Everything() {
if c.Module == "power" {
if c.Data != tc.data {
t.Errorf("what runs says Data %v, want %v", c.Data, tc.data)
}
return strings.Join(c.Filled, ",")
}
}
return ""
}
if tc.told == "" { // no want, no have: nothing to start, and the kept declaration is what runs
a := New(b.env)
b.ag = a
if got := filledNow(); got != tc.filled {
t.Fatalf("filled %q, want %q", got, tc.filled)
}
return
}
stop := b.run()
defer stop()
eventually(t, "the filling was not told what runs at the start", func() bool {
mu.Lock()
defer mu.Unlock()
return len(told) > 0 && told[0] == tc.told
})
if got := filledNow(); got != tc.filled {
t.Fatalf("filled %q, want %q", got, tc.filled)
}
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied %v: nothing has a want", got)
}
})
}
}
// Issue 1126 (ADR 0547 §2, §3): with the node's own compositions, unsigned, and wants naming the build alone, the
// composition composed for the want's build is applied, and only when it differs from what runs; for a module of
// policy record a move that recreates a container waits in phase waiting-yes with its judgement until the want names
// the yes for that composition, and is then applied.
func TestTheNodesOwnCompositionIsAppliedWhenItDiffersAndARecordMoveWaitsForAYes(t *testing.T) {
b := newBed(t)
b.env.SelfComposed, b.env.Trusted = true, nil
b.env.Policy = func(_ context.Context, m string) string {
if m == "db" || m == "rec" {
return catalogue.PolicyRecord
}
return catalogue.PolicyRoll
}
stop := b.run()
defer stop()
unsigned := func(module, fp string, rs ...json.RawMessage) (string, []byte, string) {
c := delivery.Composed{Node: node, Module: module, FP: fp, Resources: rs}
key, _ := delivery.ComposedKey(node, module, fp)
raw, err := delivery.EncodeComposed(c)
if err != nil {
t.Fatal(err)
}
dig, _ := delivery.ComposedDigestOf(c)
return key, raw, dig
}
want := func(module, fp, yes string) {
key, _ := delivery.WantKey(node, module)
raw, err := delivery.EncodeWant(delivery.Want{FP: fp, Yes: yes, Self: true})
if err != nil {
t.Fatal(err)
}
b.wants.put(key, raw)
}
fp := fpOf("a")
pw, _ := json.Marshal(map[string]any{"id": "web.pw", "type": "file", "path": "/tmp/r6/web/pw", "sealed": "1", "module": "web"})
ck, craw, dig := unsigned("web", fp, file("web", "x", "1"), pw)
b.comps.put(ck, craw)
want("web", fp, "")
eventually(t, "the node's own composition was not applied", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "the have does not name the composed digest", func() bool { return b.haves.last("web").Composed == dig })
// The same composition handed again is no move.
b.comps.put(ck, craw)
time.Sleep(50 * time.Millisecond)
// The node's own composition carries its sealed values in its key, as a module's have does, so a restart reuses its
// core boxes and applies nothing again (review of L2, round 3, finding 1).
box, _ := json.Marshal(map[string]any{"id": "core-box", "type": "file", "path": "/tmp/r6/core-box", "sealed": "core"})
own := delivery.Composed{Node: node, Resources: []json.RawMessage{box}}
nk, _ := delivery.ComposedNodeKey(node)
nraw, _ := delivery.EncodeComposed(own)
ndig, _ := delivery.ComposedDigestOf(own)
b.comps.put(nk, nraw)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Self: true})
b.wants.put(nwk, nwraw)
eventually(t, "the node's own key does not carry its sealed values", func() bool {
h := b.haves.last(NodeModule)
return h.Composed == ndig && len(h.Sealed) == 1 && h.Sealed["core-box#sealed"] == "core"
})
// The node cannot compose (a stand-in refused): each wanted module's have says so, failed, naming why, and still
// names what runs; composed again, the have is what it runs. web was applied by this agent; idle was not since the
// agent's start (the switch and every delivery of the node-engine restart it), so what runs is what its have said.
idleFP, idleDig := fpOf("c"), "sha256:"+strings.Repeat("9", 64)
b.haves.Write(context.Background(), Have{Module: "idle", FP: idleFP, Composed: idleDig, Phase: string(delivery.PhaseRunning),
Sealed: delivery.Seals{"idle.key#sealed": `{"box":"idle"}`}})
want("idle", idleFP, "")
eventually(t, "idle's want was not taken", func() bool {
b.ag.mu.Lock()
defer b.ag.mu.Unlock()
return b.ag.wants["idle"].Present
})
if h := b.haves.last("web"); len(h.Sealed) != 1 || h.Sealed["web.pw#sealed"] != "1" {
t.Errorf("web's running have does not carry its sealed value: %+v", h)
}
b.ag.NotComposed(context.Background(), "a stand-in for web's password")
for _, row := range []struct {
module, fp, composed, slot, sealed string
}{{"web", fp, dig, "web.pw#sealed", "1"}, {"idle", idleFP, idleDig, "idle.key#sealed", `{"box":"idle"}`}} {
h := b.haves.last(row.module)
if h.Phase != string(delivery.PhaseFailed) || h.FP != row.fp || h.Composed != row.composed ||
len(h.Sealed) != 1 || h.Sealed[row.slot] != row.sealed || !strings.Contains(h.Detail, "web's password") {
t.Errorf("%s's have while the node cannot compose: %+v", row.module, h)
}
}
// A wanted module whose have cannot be read (the bus paused, issue 1126) is not marked: a mark would drop the build
// and digest its have names.
readAgain = time.Millisecond
b.haves.mu.Lock()
b.haves.unreadable = map[string]bool{"unread": true}
b.haves.mu.Unlock()
want("unread", fpOf("d"), "")
want("fresh", fpOf("e"), "") // never ran, its have absent: marked, naming nothing it runs
// A have that already says another cause keeps it (review of the switch, round 1): an apply that failed.
brokenFP, brokenDig := fpOf("f"), "sha256:"+strings.Repeat("8", 64)
b.haves.Write(context.Background(), Have{Module: "broken", FP: brokenFP, Composed: brokenDig, Phase: string(delivery.PhaseFailed),
Detail: "apply: unit failed to start"})
want("broken", brokenFP, "")
eventually(t, "unread's want was not taken", func() bool {
b.ag.mu.Lock()
defer b.ag.mu.Unlock()
return b.ag.wants["unread"].Present
})
b.ag.NotComposed(context.Background(), "a stand-in for web's password, again")
if h := b.haves.last("unread"); h.Module != "" {
t.Errorf("a module whose have could not be read was marked: %+v", h)
}
if h := b.haves.last("fresh"); h.Phase != string(delivery.PhaseFailed) || h.FP != "" {
t.Errorf("a module that never ran, its have absent, is not marked: %+v", h)
}
if h := b.haves.last("broken"); h.Detail != "apply: unit failed to start" || h.Composed != brokenDig {
t.Errorf("an apply failure was overwritten by a composer's mark: %+v", h)
}
// A run that ended marks nothing: what it would say is the next run's (review of the switch, round 1).
ended, end := context.WithCancel(context.Background())
end()
before := len(b.haves.log)
b.ag.NotComposed(ended, "a refusal from a run that ended")
if len(b.haves.log) != before {
t.Errorf("a run that ended wrote %d haves", len(b.haves.log)-before)
}
b.ag.NotComposed(context.Background(), "")
for _, row := range []struct{ module, composed string }{{"web", dig}, {"idle", idleDig}} {
if h := b.haves.last(row.module); h.Phase != string(delivery.PhaseRunning) || h.Composed != row.composed || len(h.Sealed) != 1 {
t.Errorf("%s's have once the node composes again: %+v", row.module, h)
}
}
// Issue 1126, the applying path: a module the node holds alone (its own refusal: de-spiegel's unset route names on
// novox) is marked alone, naming its own words; web, which composes, keeps running, unmarked. Composed again, idle's
// have is what it runs.
b.ag.NotComposedEach(context.Background(), "", map[string]string{"idle": "idle says ${config:name}, and nothing sets it"})
if h := b.haves.last("idle"); h.Phase != string(delivery.PhaseFailed) || h.Composed != idleDig || !strings.Contains(h.Detail, "${config:name}") {
t.Errorf("idle's have while it alone is held: %+v", h)
}
if h := b.haves.last("web"); h.Phase != string(delivery.PhaseRunning) || h.Composed != dig {
t.Errorf("web, which composes, is marked beside idle: %+v", h)
}
b.ag.NotComposedEach(context.Background(), "", nil)
if h := b.haves.last("idle"); h.Phase != string(delivery.PhaseRunning) || h.Composed != idleDig {
t.Errorf("idle's have once it composes again: %+v", h)
}
// A revision at the same build (a config value changed) is applied.
_, craw2, dig2 := unsigned("web", fp, file("web", "x", "2"))
b.comps.put(ck, craw2)
eventually(t, "the revision was not applied", func() bool { return len(b.appliedList()) == 3 && b.haves.last("web").Composed == dig2 })
// A record module: its first composition is no move; a new image recreates its container and waits for a yes.
container := func(image string) json.RawMessage {
module := strings.SplitN(image, "@", 2)[0]
raw, _ := json.Marshal(map[string]any{"id": module + ".server", "type": "container", "module": module, "name": module, "image": image})
return raw
}
dk, draw, _ := unsigned("db", fp, container("db@sha256:"+strings.Repeat("1", 64)))
b.comps.put(dk, draw)
want("db", fp, "")
eventually(t, "the record module's first composition was not applied", func() bool { return len(b.appliedList()) == 4 })
fpB := fpOf("b")
dk2, draw2, ddig2 := unsigned("db", fpB, container("db@sha256:"+strings.Repeat("2", 64)))
// rec is a record module this agent has run nothing of since its start (the switch and every delivery of the
// node-engine restart it): its have alone says what runs. Its move waits for a yes as db's does, and its have keeps
// what runs (fingerprint, digest, sealed values) rather than naming nothing (review of L2, round 3).
recFP, recDig := fpOf("d"), "sha256:"+strings.Repeat("8", 64)
b.haves.Write(context.Background(), Have{Module: "rec", FP: recFP, Composed: recDig, Phase: string(delivery.PhaseRunning),
Sealed: delivery.Seals{"rec.key#sealed": `{"box":"rec"}`}})
rk, rraw, rdig := unsigned("rec", fpB, container("rec@sha256:"+strings.Repeat("3", 64)))
b.comps.put(dk2, draw2)
b.comps.put(rk, rraw)
want("db", fpB, "")
want("rec", fpB, "")
applied := len(b.appliedList())
for _, row := range []struct {
module, fp, composed, waits string
sealed delivery.Seals
}{{"db", fp, "", ddig2, nil}, {"rec", recFP, recDig, rdig, delivery.Seals{"rec.key#sealed": `{"box":"rec"}`}}} {
eventually(t, row.module+"'s move did not wait for a yes", func() bool {
h := b.haves.last(row.module)
return h.Phase == string(delivery.PhaseWaitingYes) && h.Waits == row.waits
})
h := b.haves.last(row.module)
if h.FP != row.fp || (row.composed != "" && h.Composed != row.composed) || h.Composed == "" ||
(row.sealed != nil && !maps.Equal(h.Sealed, row.sealed)) {
t.Errorf("%s's have waiting for a yes does not keep what runs: %+v", row.module, h)
}
}
if got := b.appliedList(); len(got) != applied {
t.Fatalf("a move waiting for a yes was applied: %v", got)
}
want("db", fpB, ddig2)
want("rec", fpB, rdig)
eventually(t, "the yes did not apply the move", func() bool {
h, r := b.haves.last("db"), b.haves.last("rec")
return len(b.appliedList()) == applied+2 && h.Phase == string(delivery.PhaseRunning) && h.FP == fpB &&
r.Phase == string(delivery.PhaseRunning) && r.FP == fpB && r.Composed == rdig
})
}
// 1957/AC1 and AC4 (unit; ADR 0568 §1, §4): an engine start takes a module from what the machine shows, not only from the
// have or the kept declaration it kept. Every resource up applies nothing; one declared running and not active applies the
// module once, from a have or from the kept declaration; a module with no want is not looked at.
func testAnEngineStartReconcilesAModuleFromTheMachinesFacts(t *testing.T) {
for _, c := range []struct {
name string
have bool // the engine's own have says running at the want
held bool // else its have says held, from a failed backup
kept bool // else the kept declaration says it runs
inactive bool
unwanted bool
applied int
asked bool // the machine was looked at
}{
{"all up: nothing is applied", true, false, false, false, false, 0, true},
{"a unit not active: applied once", true, false, false, true, false, 1, true},
{"kept declaration, a unit not active: applied once", false, false, true, true, false, 1, true},
{"kept declaration, all up: nothing is applied", false, false, true, false, false, 0, true},
{"a held module applies nothing at start", false, true, false, true, false, 0, false},
{"no want: nothing is looked at or applied", true, false, false, true, true, 0, false},
} {
t.Run(c.name, func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
rs := []json.RawMessage{file("web", "x", "1")}
ck, craw, dig := composition(t, "web", fp, false, rs...)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
if !c.unwanted {
b.wants.before(wk, wraw)
}
if c.have {
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: dig, Phase: "running"}}
}
if c.held {
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fpOf("0"), Composed: "sha256:" + strings.Repeat("0", 64),
Phase: string(delivery.PhaseHeld), Detail: "restic: the repository is locked"}}
}
if c.kept {
b.kept = map[string][]json.RawMessage{"web": rs}
}
var mu sync.Mutex
asked := false
b.env.Inactive = func(context.Context, delivery.Composed) bool {
mu.Lock()
defer mu.Unlock()
asked = true
return c.inactive
}
stop := b.run()
defer stop()
if !c.unwanted && !c.held {
eventually(t, "the module was not settled", func() bool { return b.haves.last("web").Composed == dig })
}
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != c.applied {
t.Fatalf("applied %v, want %d", got, c.applied)
}
mu.Lock()
defer mu.Unlock()
if asked != c.asked {
t.Fatalf("the machine was asked: %v, want %v", asked, c.asked)
}
})
}
}
// 1957/AC2, AC3 and AC4 (unit; ADR 0568 §2, §3, §4): the judge's edge, Reconcile, applies a running module again; one held
// for a failed backup and one with no want are left alone; and an apply that fails writes the have `failed` with the
// apply's own words.
func testReconcileAppliesARunningModuleAgainAndLeavesTheHeldAndTheUnwantedAlone(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ckW, crawW, digW := composition(t, "web", fp, false, file("web", "x", "1"))
wkW, wrawW := wantFor(t, "web", fp, digW, "")
ckD, crawD, digD := composition(t, "db", fp, true, file("db", "x", "1"))
wkD, wrawD := wantFor(t, "db", fp, digD, "")
b.comps.before(ckW, crawW)
b.wants.before(wkW, wrawW)
b.comps.before(ckD, crawD)
b.wants.before(wkD, wrawD)
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: digW, Phase: "running"},
"db": {Module: "db", FP: fpOf("0"), Composed: "sha256:" + strings.Repeat("0", 64), Phase: "running"}}
b.backup = func(string) Backup { return Backup{Words: "restic: the repository is locked"} }
stop := b.run()
defer stop()
eventually(t, "web was not settled", func() bool { return b.haves.last("web").Composed == digW })
eventually(t, "db was not held", func() bool { return b.haves.last("db").Phase == string(delivery.PhaseHeld) })
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied at the start: %v", got)
}
b.ag.Reconcile("db", "ghost")
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a held module and an unwanted one were applied: %v", got)
}
if h := b.haves.last("db"); h.Phase != string(delivery.PhaseHeld) {
t.Fatalf("the held module's have changed: %+v", h)
}
b.ag.Reconcile("web")
eventually(t, "web was not applied again", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "web is not running again", func() bool { return b.haves.last("web").Phase == string(delivery.PhaseRunning) })
b.mu.Lock()
b.failWith = "systemctl start web.service: exit status 1"
b.mu.Unlock()
b.ag.Reconcile("web")
eventually(t, "the failed re-apply did not write the have failed with its words", func() bool {
h := b.haves.last("web")
return h.Phase == string(delivery.PhaseFailed) && h.Detail == "systemctl start web.service: exit status 1"
})
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 2 {
t.Fatalf("a failed apply was tried again with no edge: %v", got)
}
}
+129 -2
View File
@@ -312,6 +312,38 @@ type Scope struct {
Module string
// Others are the resources of every other module the node has a composition for.
Others []declaration.Resource
// Placed are the modules a want still places on the node (present, not removed), held ones included: no apply but
// the module's own removes a resource of one of them (issue 774: on 2026-10-29 the node's own apply on the bus node,
// composed without what its folded modules ran, removed the bus server and four held modules). Nil guards nothing.
Placed map[string]bool
// RemovedByWant are the modules whose want says removed. In the node's own apply (Module NodeModule) with this set
// non-nil, a resource of any other module is removed only when that module is here: a module the node runs folded
// in its own composition has no want, so Placed does not name it (issue 774, review of the re-land, round 1: nats on
// the bus node has none, and its nats.conf was guessed as nats's and removed). Nil guards by Placed alone.
RemovedByWant map[string]bool
}
// placedOwner is the module a want still places on the node that the record belongs to, other than the scope's own:
// the module recorded with it, else the one its id names (a resource the node's own composition carried for a folded
// module is recorded as the node's, its id naming the module). Empty when there is none.
func (sc *Scope) placedOwner(a store.Applied) string {
if sc == nil || (len(sc.Placed) == 0 && sc.RemovedByWant == nil) {
return ""
}
m := a.Module
if m == "" || m == store.NodeModule {
m, _ = declaration.ModuleGuess(a.ID)
}
if m == "" || m == sc.Module || m == store.NodeModule {
return ""
}
if sc.Placed[m] {
return m
}
if sc.Module == store.NodeModule && sc.RemovedByWant != nil && !sc.RemovedByWant[m] {
return m // a module the node's own composition carried, which no want removed
}
return ""
}
// owns reports whether a record belongs to the scope's module.
@@ -441,6 +473,31 @@ func applyScoped(
}
}
// **A module a want still places here is never removed by another's apply** (issue 774): its resources go only with
// its own apply or its want's removal. One that would is refused whole, before anything is changed, naming them: a
// composition that leaves them out lost what runs, and removing what runs is the worst a node can do with that.
if scope != nil {
var refused []string
for _, orphan := range known.Orphans(declared, origin) {
if !scope.owns(orphan) {
continue
}
if _, kept := heldByOthers[orphan.ID]; kept {
continue
}
if m := scope.placedOwner(orphan); m != "" {
refused = append(refused, orphan.ID+" of "+m)
}
}
if len(refused) > 0 {
sort.Strings(refused)
err := fmt.Errorf("the apply of %s would remove %s, of modules a want still places here; a module's resources "+
"go only with its own apply or its want's removal, so nothing was applied", scope.Module, strings.Join(refused, ", "))
log(" refused: " + err.Error())
return report, known, &Error{Resource: "the resources of modules placed here", Err: err, Done: report}
}
}
removeOrphan := func(orphan store.Applied) error {
var action, detail string
var err error
@@ -1223,6 +1280,19 @@ func applyScoped(
}
}
// A file written or removed in a unit directory is read by the service manager only after a reload, whether or not
// a service reflects it: a drop-in on reboot.target is reflected by no service, and without the reload the next
// reboot still runs the target as it was loaded (issue 1126). Once per apply.
if at := unitFileMoved(report.Outcomes); at != "" {
if u, ok := sys.(unitReloader); ok {
if err := u.ReloadUnits(ctx, run); err != nil {
failures = append(failures, &Error{Resource: at, Err: fmt.Errorf("reloading the service manager's units: %w", err)})
} else {
log(fmt.Sprintf(" reloaded the service manager's units: %s changed", at))
}
}
}
// Only the mesh says what is assigned: a carried bundle's silence undeclares no identity and no folder.
if origin == store.OriginDeclared && scope.isNode() {
// And an identity, then a library group, no longer declared goes, once what ran as it went above (ADR 0328
@@ -1997,11 +2067,68 @@ type serviceReloader interface {
ReloadService(ctx context.Context, run system.Runner, unit string) error
}
// UnitDirs are the system service manager's unit directories: a file the apply writes or removes in one of them, or in a
// drop-in directory below one, is read by the manager only after a reload (issue 1126). Replaceable in tests.
var UnitDirs = []string{"/etc/systemd/system", "/run/systemd/system", "/usr/local/lib/systemd/system"}
// unitFileMoved is the first file this apply wrote or removed in a unit directory or a drop-in directory below one,
// or "" when it moved none.
func unitFileMoved(outcomes []Outcome) string {
for _, o := range outcomes {
// Only what changed the file on disk, as MovesRoot reads it: skipped, held, kept, forgotten and waiting come
// back on every apply and change nothing.
if declaration.Type(o.Type) != declaration.TypeFile {
continue
}
switch o.Action {
case "created", "updated", "corrected", "removed":
default:
continue
}
at := filepath.Clean(o.Target)
for _, dir := range UnitDirs {
if rel, err := filepath.Rel(dir, at); err == nil && rel != "." && !strings.HasPrefix(rel, "..") {
return at
}
}
}
return ""
}
// unitReloader is a service manager that caches unit files and must be told to read them again.
type unitReloader interface {
ReloadUnits(ctx context.Context, run system.Runner) error
}
// restReader is a service manager that can say whether a one-shot has nothing running now
// (novox/hq issue 1126).
type restReader interface {
ServiceAtRest(ctx context.Context, run system.Runner, unit string) (bool, error)
}
// declaredState is a unit's state as a declaration of want reads it. **A one-shot at rest is
// stopped to a declaration that it be stopped** (novox/hq issue 1126): a one-shot that does not
// linger reads "running" once it ran and worked — that is what a declaration that it run asks — but
// it has no process between its runs, so it is also as stopped as it can be, and stopping it changes
// nothing. Read only as "running", a boot-time cleanup declared stopped and enabled failed every
// apply after the boot that ran it: "was asked to be stopped and is running".
func declaredState(ctx context.Context, sys system.System, run Runner, unit, want string) (string, error) {
state, err := sys.ServiceState(ctx, run, unit)
if err != nil || want != "stopped" || state != "running" {
return state, err
}
if r, ok := sys.(restReader); ok {
rest, err := r.ServiceAtRest(ctx, run, unit)
if err != nil {
return "", err
}
if rest {
return "stopped", nil
}
}
return state, nil
}
// serviceSettle is how long the host waits before looking at a unit a second time. A test sets it
// to nothing; on a machine it is the window in which a daemon that refuses its configuration dies.
var serviceSettle = 2 * time.Second
@@ -2154,7 +2281,7 @@ func applyServiceIn(ctx context.Context, sys system.System, r *declaration.Servi
// own stopping from anybody else's (novox/hq ADR 0269).
out.declared = &store.FoundUnit{Unit: r.Unit, State: r.State, Boot: r.Boot}
before, err := sys.ServiceState(ctx, run, r.Unit)
before, err := declaredState(ctx, sys, run, r.Unit, r.State)
if err != nil {
return out, err
}
@@ -2165,7 +2292,7 @@ func applyServiceIn(ctx context.Context, sys system.System, r *declaration.Servi
// Read back. A service manager accepting a command says the transaction was accepted,
// not that the unit is running — one that starts and immediately dies satisfies it.
after, err := sys.ServiceState(ctx, run, r.Unit)
after, err := declaredState(ctx, sys, run, r.Unit, r.State)
if err == nil && r.State == "running" {
after, err = stayedRunning(ctx, sys, run, r.Unit)
}
+82
View File
@@ -350,6 +350,42 @@ func TestAnUnknownServiceStateIsRefusedNotGuessed(t *testing.T) {
}
}
// A boot-time one-shot a package ships, declared stopped and enabled at boot, is satisfied while
// nothing of it runs, whatever its last run did (novox/hq issue 1126). The kernel module's
// linux-modules-cleanup.service failed on ace with "was asked to be stopped and is running": an
// inactive one-shot with ExecMainStatus=0 reads "running", and stopping it changes nothing.
func TestABootOneShotDeclaredStoppedIsSatisfiedAtRest(t *testing.T) {
for _, c := range []struct{ name, status string }{
{"exited 0: never ran, ran at boot, or unloaded", "0"},
} {
t.Run(c.name, func(t *testing.T) {
var asked []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "show":
return "LoadState=loaded\nActiveState=inactive\nType=oneshot\nRemainAfterExit=no\n" +
"ExecMainStatus=" + c.status + "\n", nil
case "is-enabled":
return "enabled\n", nil
}
asked = append(asked, strings.Join(args, " "))
return "", nil
}
d := parse(t, `{"declaration":1,"resources":[
{"id":"cleanup","type":"service","unit":"linux-modules-cleanup.service","state":"stopped","boot":"enabled"}
]}`)
_, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatalf("a one-shot at rest declared stopped failed: %v", err)
}
if len(asked) > 0 {
t.Errorf("the service manager was asked %q for a unit already as declared", asked)
}
})
}
}
func TestADroppedServiceIsGivenBackTheStateItWasFoundIn(t *testing.T) {
// novox/hq ADR 0118: undeclaring removes what the mesh made, gives back what it changed, and
// leaves what was the machine's. A service resource never installs a unit — so undeclaring it
@@ -1225,6 +1261,52 @@ func TestAServiceIsRestartedWhenWhatItReflectsChanges(t *testing.T) {
if reloaded < 0 || reloaded > stop {
t.Errorf("the service was restarted without the unit files being read again first; commands were %v", commands)
}
// A drop-in in a unit directory that no service reflects — the power module's bound on reboot.target
// (issue 1126) — is read by the service manager only after a reload: one per apply, and no restart.
// Unchanged, it reloads nothing.
units := filepath.Join(dir, "system")
saved := UnitDirs
UnitDirs = []string{units}
t.Cleanup(func() { UnitDirs = saved })
dropIn := filepath.Join(units, "reboot.target.d", "mesh-power.conf")
withDropIn := parse(t, fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"conf","type":"file","path":%q,"content":"second\n","mode":"0644"},
{"id":"svc","type":"service","unit":"thing.service","state":"running","restart-on":["conf"]},
{"id":"units","type":"directory","path":%q,"mode":"0755"},
{"id":"drop-ins","type":"directory","path":%q,"mode":"0755"},
{"id":"bound","type":"file","path":%q,"content":"[Unit]\nJobTimeoutSec=10min\n","mode":"0644"},
{"id":"bound-too","type":"file","path":%q,"content":"[Unit]\nJobTimeoutSec=10min\n","mode":"0644"}
]}`, path, units, filepath.Dir(dropIn), dropIn, filepath.Join(filepath.Dir(dropIn), "other.conf")))
for _, row := range []struct {
name string
reloads int
}{{"a drop-in written", 1}, {"the same drop-in again", 0}} {
commands = nil
if _, state, err = Apply(context.Background(), archHost(t), withDropIn, state,
store.OriginCarried, run, nil, nil); err != nil {
t.Fatal(err)
}
reloads := 0
for _, c := range commands {
if strings.Contains(c, "daemon-reload") {
reloads++
}
if strings.Contains(c, "stop thing.service") {
t.Errorf("%s: a drop-in no service reflects restarted one: %v", row.name, commands)
}
}
if reloads != row.reloads {
t.Errorf("%s: %d reloads, want %d; commands were %v", row.name, reloads, row.reloads, commands)
}
}
// A drop-in whose module is skipped behind its step, or held as found, comes back so on every apply: it moved
// nothing, and reloads nothing.
for _, action := range []string{"skipped", "held"} {
if at := unitFileMoved([]Outcome{{ID: "bound", Type: "file", Target: dropIn, Action: action}}); at != "" {
t.Errorf("a drop-in %s reloads the service manager (%s)", action, at)
}
}
}
}
+4
View File
@@ -290,3 +290,7 @@ func unpack(body []byte, into string) (int, error) {
}
}
}
// Fetch is an archive at source as the host fetches every archive it unpacks, at most maxArchive bytes: what the node's
// own composer reads a moved module's templates archive with (issue 1126). Its digest is the caller's to check.
func Fetch(ctx context.Context, source string) ([]byte, error) { return fetch(ctx, source) }
+27 -3
View File
@@ -45,6 +45,30 @@ func WriteFilled(
log func(string),
keep Keep,
windows *Windows,
) (report Report, _ store.State, _ error) {
return WriteFilledWith(ctx, sys, d, known, path, content, run, log, keep, windows, Filled{})
}
// Filled is what the node-engine did to a filled file's content before its write.
type Filled struct {
// Secret says the node-engine filled a value into the content, a provision's `${secret:}` (novox/hq ADR 0570
// clause 6): the file is then written 0600 when the declaration gives no mode, as a file carrying a secret is.
Secret bool
}
// WriteFilledWith is WriteFilled for content the node-engine filled before the write (Filled).
func WriteFilledWith(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
known store.State,
path string,
content []byte,
run Runner,
log func(string),
keep Keep,
windows *Windows,
filled Filled,
) (report Report, _ store.State, _ error) {
if log == nil {
log = func(string) {}
@@ -65,7 +89,7 @@ func WriteFilled(
return report, known, &Error{Resource: path, Err: fmt.Errorf("%d resources declare %s, and a filled file "+
"is written for exactly one, so nothing was written", len(declared), path)}
}
file := filledAs(declared[0], content)
file := filledAs(declared[0], content, filled.Secret)
if held, isHeld := known.HeldAt(file.ID); isHeld {
return report, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is held as found for a module not "+
"yet taken (%s), so nothing was written", path, held.Why)}
@@ -163,10 +187,10 @@ func WriteFilled(
}
// filledAs is the declared file with the filled content in place of the declaration's.
func filledAs(declared *declaration.File, content []byte) *declaration.File {
func filledAs(declared *declaration.File, content []byte, secret bool) *declaration.File {
f := *declared
f.Content, f.Sealed, f.VaultSealed, f.Bytes, f.Secrets = string(content), "", nil, "", nil
if f.Mode == "" && (declared.Secret() || len(declared.Secrets) > 0) {
if f.Mode == "" && (secret || declared.Secret() || len(declared.Secrets) > 0) {
// Written as the declaration would have written it: a file carrying a credential is not world-readable.
f.Mode = "0600"
}
+72 -25
View File
@@ -3,6 +3,7 @@ package apply
import (
"context"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
@@ -116,32 +117,78 @@ func TestARecordWithoutAnOwnerBelongsToTheModuleItsIdNames(t *testing.T) {
// leaves it as it is, whatever its type, until the module's own apply takes it (issue 1945, live on 2026-10-25 at 02:32:
// shanks and g14 lost systemd-resolved's guard, its resolver files and sudo's operator file, ace qbittorrent's server,
// because the node's own apply removed every record no longer in it, the others holding the very same resources or not).
//
// And a resource the node's own composition carried for a module a want still places here (a folded module of the old
// form, held) is never removed by the node's own apply when a composition leaves it out: the apply is refused whole and
// says why (issue 774, live on 2026-10-29 at 07:54 UTC on the bus node: the node's own composition, composed without what
// it ran once the kept declaration was read no more, removed the bus server, the proxy and three held modules). Only its
// want's removal lets it go.
func TestAResourceThatMovedToItsModulesCompositionIsLeftByTheNodesOwnApply(t *testing.T) {
dir := t.TempDir()
fact := filepath.Join(dir, "suffix")
gone := filepath.Join(dir, "gone.conf")
own := declare(t, `{"id":"systemd-resolved.fact-suffix","type":"file","path":"`+fact+`","content":"mesh\n"},
{"id":"old.gone","type":"file","path":"`+gone+`","content":"g\n"}`)
_, state, err := ApplyModule(context.Background(), archHost(t), own, Scope{Module: store.NodeModule},
state0(), store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
// systemd-resolved now has a composition of its own, not applied yet; old.gone is no module's any more.
moved := declare(t, `{"id":"systemd-resolved.fact-suffix","type":"file","path":"`+fact+`","content":"mesh\n","module":"systemd-resolved"}`)
report, state, err := ApplyModule(context.Background(), archHost(t), &declaration.Declaration{},
Scope{Module: store.NodeModule, Others: moved.Resources}, state, store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !exists(fact) {
t.Fatalf("the moved resource was removed: %+v", outcomeOf(report, "systemd-resolved.fact-suffix"))
}
if exists(gone) {
t.Fatal("a resource no composition holds any more was kept")
}
if _, still := state.Find("systemd-resolved.fact-suffix"); !still {
t.Fatal("the moved resource's record was dropped, so its module's apply would not know it")
for _, tc := range []struct {
name string
id string // the resource the node's own composition carried, and no longer does
moved bool // its module's own composition now declares it (Others)
placed bool // a want still places its module here
refused bool // the node's own apply is refused whole
stays bool // the resource is still on the machine and recorded
// wants names the modules whose want says removed (Scope.RemovedByWant); nil leaves the guard to Placed alone.
wants []string
}{
{name: "moved to its module's own composition", id: "systemd-resolved.fact-suffix", moved: true, placed: true, stays: true},
{name: "a held folded module, in no composition", id: "nats.server-conf", placed: true, refused: true, stays: true},
{name: "its module's want removed", id: "nats.server-conf"},
// Issue 774, review of the re-land, round 1: a module folded in the node's own has no want, so Placed does not name
// it; in the node's own apply only a module a want removed loses its resources.
{name: "a folded module with no want", id: "nats.server-conf", wants: []string{}, refused: true, stays: true},
{name: "a folded module a want removed", id: "nats.server-conf", wants: []string{"nats", "old"}},
} {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
fact := filepath.Join(dir, "carried")
gone := filepath.Join(dir, "gone.conf")
own := declare(t, `{"id":"`+tc.id+`","type":"file","path":"`+fact+`","content":"mesh\n"},
{"id":"old.gone","type":"file","path":"`+gone+`","content":"g\n"}`)
_, state, err := ApplyModule(context.Background(), archHost(t), own, Scope{Module: store.NodeModule},
state0(), store.OriginDeclared, noServices, nil, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
module, _ := declaration.ModuleGuess(tc.id)
scope := Scope{Module: store.NodeModule}
if tc.moved {
moved := declare(t, `{"id":"`+tc.id+`","type":"file","path":"`+fact+`","content":"mesh\n","module":"`+module+`"}`)
scope.Others = moved.Resources
}
if tc.placed {
scope.Placed = map[string]bool{module: true}
}
if tc.wants != nil {
scope.RemovedByWant = map[string]bool{}
for _, m := range tc.wants {
scope.RemovedByWant[m] = true
}
}
report, after, err := ApplyModule(context.Background(), archHost(t), &declaration.Declaration{}, scope, state,
store.OriginDeclared, noServices, nil, nil, nil, nil)
if tc.refused != (err != nil) {
t.Fatalf("refused %v, want %v: %v", err != nil, tc.refused, err)
}
if tc.refused {
if !strings.Contains(err.Error(), tc.id+" of "+module) {
t.Fatalf("the refusal does not name what it kept: %v", err)
}
after = state
}
if exists(fact) != tc.stays {
t.Fatalf("the carried resource exists %v, want %v: %+v", exists(fact), tc.stays, outcomeOf(report, tc.id))
}
if _, still := after.Find(tc.id); still != tc.stays {
t.Fatalf("the carried resource's record kept %v, want %v", still, tc.stays)
}
if exists(gone) == !tc.refused {
t.Fatalf("a resource no composition holds any more exists %v, with the apply refused %v", exists(gone), tc.refused)
}
})
}
}
+111
View File
@@ -0,0 +1,111 @@
package apply
import (
"fmt"
"os"
"path/filepath"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// ObjectKeep is the node-engine's own keep of the content of files given by digest (novox/hq ADR 0486 §1, issue 924): one
// file per object, named as the object is, beside the node's state. **What a node once had it never asks the bus for
// again**, so neither the controller's sweep of the bucket (an object is removed a day after no composition of the
// controller names it) nor a bus that does not answer can refuse a composition the node already ran. The node's account
// may only read its bucket (ADR 0486 §2), so the keep is the node's, not the bucket's: nothing is written back.
//
// An object unused for KeptObjectsFor is let go when another is kept, so the keep holds what the node applied lately and
// not every export it ever ran. Every failure to keep is silent and costs only a fetch: the keep is a copy.
type ObjectKeep struct {
Dir string
// Now is the clock; nil is time.Now.
Now func() time.Time
}
// KeptObjectsFor is how long an object of the keep is held after it was last used.
const KeptObjectsFor = 14 * 24 * time.Hour
// KeepObjectsIn is the keep at dir; an empty dir keeps nothing.
func KeepObjectsIn(dir string) *ObjectKeep {
if dir == "" {
return nil
}
return &ObjectKeep{Dir: dir}
}
func (k *ObjectKeep) now() time.Time {
if k.Now != nil {
return k.Now()
}
return time.Now()
}
// get is the kept content of the object name when it is what f names, and marks it used.
func (k *ObjectKeep) get(name string, f *declaration.File) ([]byte, bool) {
if k == nil {
return nil, false
}
path := filepath.Join(k.Dir, name)
body, err := os.ReadFile(path)
if err != nil || int64(len(body)) != f.Size || f.Verify(body) != nil {
return nil, false
}
now := k.now()
_ = os.Chtimes(path, now, now)
return body, true
}
// put keeps body under name, unless it is kept already (then it is marked used), and lets go of what was unused too long.
func (k *ObjectKeep) put(name string, body []byte) {
if k == nil {
return
}
path := filepath.Join(k.Dir, name)
now := k.now()
if info, err := os.Stat(path); err == nil && info.Size() == int64(len(body)) {
_ = os.Chtimes(path, now, now)
return
}
if err := os.MkdirAll(k.Dir, 0o700); err != nil {
return
}
tmp, err := os.CreateTemp(k.Dir, ".put-*")
if err != nil {
return
}
_, werr := tmp.Write(body)
cerr := tmp.Close()
if werr != nil || cerr != nil || os.Chmod(tmp.Name(), 0o600) != nil || os.Rename(tmp.Name(), path) != nil {
_ = os.Remove(tmp.Name())
return
}
_ = os.Chtimes(path, now, now)
k.prune(now)
}
// prune lets go of every object unused for KeptObjectsFor.
func (k *ObjectKeep) prune(now time.Time) {
entries, err := os.ReadDir(k.Dir)
if err != nil {
return
}
for _, e := range entries {
info, err := e.Info()
if err != nil || !info.Mode().IsRegular() {
continue
}
if now.Sub(info.ModTime()) >= KeptObjectsFor {
_ = os.Remove(filepath.Join(k.Dir, e.Name()))
}
}
}
// never says, in a failed fetch's error before its cause (a transient one's detail keeps the tail), that the node's own
// keep did not hold the object either.
func (k *ObjectKeep) never() string {
if k == nil {
return ""
}
return fmt.Sprintf(" (this node's own keep of objects, %s, never held it)", k.Dir)
}
+91 -1
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"os"
"path/filepath"
"runtime"
"strings"
"time"
@@ -94,6 +95,14 @@ func applyProcess(ctx context.Context, r *declaration.Process, run Runner,
if r.Schedule != "" {
unit = r.Name + ".timer"
}
// The yielding slice is checked even then: a slice file removed by hand would leave the build unbound.
if wrote, err := placeYieldSlice(r); err != nil {
return out, err
} else if wrote {
if _, err := run(ctx, "systemctl", "daemon-reload"); err != nil {
return out, err
}
}
if active, err := run(ctx, "systemctl", "is-active", "--quiet", unit); err == nil {
_ = active
return out, nil
@@ -121,7 +130,12 @@ func applyProcess(ctx context.Context, r *declaration.Process, run Runner,
}
// Replaced rather than merged: the bundle is the whole of what it runs, and files left from a
// previous version would be loaded by a runtime that walks a directory.
if err := os.RemoveAll(at); err != nil {
// **Retired, never deleted under its process** (novox/hq issue 734): the old process runs until
// the restart below, from this directory and in it. Deleted here, every command it started
// before the restart failed with "getwd: no such file or directory" — a builder's toolchain probe
// refused three checks on 2026-10-26. Renamed aside, its process keeps a directory that exists,
// and the build is deleted once nothing runs from it, as a judged build's is.
if err := witness.Retire(daemonRoot, r.Name, at, time.Now()); err != nil {
return out, err
}
} else {
@@ -189,6 +203,9 @@ func applyProcess(ctx context.Context, r *declaration.Process, run Runner,
if err := os.WriteFile(unit, []byte(unitFor(r)), 0o644); err != nil {
return out, err
}
if _, err := placeYieldSlice(r); err != nil {
return out, err
}
if _, err := run(ctx, "systemctl", "daemon-reload"); err != nil {
return out, err
}
@@ -277,6 +294,7 @@ func unitFor(r *declaration.Process) string {
if r.User != "" {
fmt.Fprintf(&b, "User=%s\n", r.User)
}
writeYields(&b, r.Yields)
fmt.Fprintf(&b, "ExecStart=%s\n", strings.Join(runFrom(r), " "))
if r.Schedule != "" || r.RunOnce {
// Started by its timer, or once by the host, and expected to finish. Restarting it would have it run
@@ -293,6 +311,78 @@ func unitFor(r *declaration.Process) string {
return b.String()
}
// machineCPUs is how many processors this machine has, for a share of them in a quota. A variable only so a test can
// fix it.
var machineCPUs = runtime.NumCPU
// writeYields writes how a process gives way to everything else on its machine (novox/hq ADR 0553, issue 1806): a
// build seat's holder, whose builds must never slow the operator's desktop.
//
// **Into the node's one yielding slice**, where the CPU weight and the quota are (yieldSlice): every yielding process
// of the machine shares them, so three build modules take one share together, never one each. **Per process**, what a
// slice cannot say: Nice=19 for every process it starts, and the lowest best-effort IO level. The IO level binds only
// where the disk's scheduler honours it (bfq); under `none`, the usual one for NVMe, it does nothing. Not the idle IO
// class: a disk that is never idle would starve a build for good.
//
// The share is written into the unit as a comment, so a changed share changes the unit, the daemon's identity, and the
// slice is written again.
func writeYields(b *strings.Builder, y *declaration.Yields) {
if y == nil {
return
}
fmt.Fprintf(b, "# Yields: %s, %d%% of the machine shared in %s\n", y.Priority, y.CPUShare, declaration.YieldSlice)
fmt.Fprintf(b, "Slice=%s\n", declaration.YieldSlice)
if y.Priority == declaration.YieldIdle {
b.WriteString("Nice=19\nIOSchedulingClass=best-effort\nIOSchedulingPriority=7\n")
}
}
// yieldSlice is the node's one yielding slice (declaration.YieldSlice) for a process that yields: idle CPU weight and
// IO weight 1 for an idle priority, and CPUQuota for its share of the processors this machine has (systemd counts a
// quota in percent of one processor, so half of 24 is 1200%), at least one processor. At the top of the hierarchy, so
// its weight and quota bind against the operator's user slice and the system's.
//
// The IO weight binds only where the disk's scheduler honours it (bfq, or the io.cost controller); under `none` it does
// nothing, and the CPU bound is what holds.
//
// Every yielding process of a machine carries the node's one share (the composer's, ADR 0553 §3), so whichever writes
// it writes the same file. Left in place when nothing yields any more: an empty slice binds nothing.
func yieldSlice(y *declaration.Yields) string {
var b strings.Builder
b.WriteString("# Generated by the mesh. Do not edit — this file is replaced whenever a yielding\n")
b.WriteString("# process is declared again (novox/hq ADR 0553).\n")
b.WriteString("[Unit]\nDescription=The mesh's yielding processes: builds, at idle priority under one share\n\n")
b.WriteString("[Slice]\n")
if y.Priority == declaration.YieldIdle {
b.WriteString("CPUWeight=idle\nIOWeight=1\n")
}
if y.CPUShare > 0 {
quota := y.CPUShare * machineCPUs()
if quota < 100 {
quota = 100
}
fmt.Fprintf(&b, "CPUQuota=%d%%\n", quota)
}
return b.String()
}
// placeYieldSlice writes the yielding slice for a process that yields, when what is there differs, and says whether it
// wrote: the caller reloads the service manager before it starts anything.
func placeYieldSlice(r *declaration.Process) (bool, error) {
if r.Yields == nil {
return false, nil
}
path := filepath.Join(unitDir, declaration.YieldSlice)
want := yieldSlice(r.Yields)
if have, err := os.ReadFile(path); err == nil && string(have) == want {
return false, nil
}
if err := os.WriteFile(path, []byte(want), 0o644); err != nil {
return false, fmt.Errorf("the yielding slice %s: %w", path, err)
}
return true, nil
}
// timerFor is the cadence a scheduled process runs on.
//
// **The mesh's cron expression, handed to the machine's own timer.** The host already parses and
+63 -20
View File
@@ -21,10 +21,16 @@ func aProcess() *declaration.Process {
}
}
// The unit the mesh writes says what it runs, where, and that it comes back.
// The unit the mesh writes says what it runs, where, and that it comes back — and that it is the
// mesh's: **generated whole and saying so**, because an edit that survives until the next declaration
// and then vanishes is worse than one that is refused. The long-running mode stays up and is not a step.
func TestTheUnitRunsWhatTheDaemonSaid(t *testing.T) {
unit := unitFor(aProcess())
if !strings.HasPrefix(unit, "#") {
t.Fatalf("the unit does not open saying it is generated:\n%s", unit)
}
for _, want := range []string{
"Do not edit",
"ExecStart=node index.js",
"WorkingDirectory=/var/lib/mesh/daemons/greeter",
"Restart=always",
@@ -34,14 +40,8 @@ func TestTheUnitRunsWhatTheDaemonSaid(t *testing.T) {
t.Fatalf("the unit does not say %q:\n%s", want, unit)
}
}
}
// **Generated whole and saying so.** Every managed file on a machine carries this, because an edit
// that survives until the next declaration and then vanishes is worse than one that is refused.
func TestTheUnitSaysItIsTheMeshs(t *testing.T) {
unit := unitFor(aProcess())
if !strings.HasPrefix(unit, "#") || !strings.Contains(unit, "Do not edit") {
t.Fatalf("the unit does not say it is generated:\n%s", unit)
if strings.Contains(unit, "Type=oneshot") {
t.Fatalf("a process that should stay up is declared a step:\n%s", unit)
}
}
@@ -83,6 +83,60 @@ func TestAProcessRunsAsWhoItSaid(t *testing.T) {
if strings.Contains(unitFor(aProcess()), "User=") {
t.Fatalf("a process that named no user had one written for it:\n%s", unitFor(aProcess()))
}
// A process that yields (ADR 0553) runs in the node's one yielding slice at nice 19; the slice holds the idle weight
// and the quota for its share of the machine's processors; one that does not yield has none of it.
defer func(n func() int) { machineCPUs = n }(machineCPUs)
for _, c := range []struct {
cpus int
y *declaration.Yields
unit, slice, not []string
}{
{24, &declaration.Yields{Priority: declaration.YieldIdle, CPUShare: 50},
[]string{"Slice=meshyield.slice\n", "Nice=19\n", "IOSchedulingPriority=7\n", "50%"},
[]string{"CPUWeight=idle\n", "IOWeight=1\n", "CPUQuota=1200%\n"}, []string{"CPUQuota=", "CPUWeight="}},
{1, &declaration.Yields{CPUShare: 25}, []string{"Slice=meshyield.slice\n"}, []string{"CPUQuota=100%\n"},
[]string{"Nice="}},
{24, nil, nil, nil, []string{"Slice=", "Nice=", "CPUQuota="}},
} {
machineCPUs = func() int { return c.cpus }
p := aProcess()
p.Yields = c.y
unit := unitFor(p)
for _, w := range c.unit {
if !strings.Contains(unit, w) {
t.Errorf("yields %+v: the unit does not say %q:\n%s", c.y, w, unit)
}
}
for _, n := range c.not {
if strings.Contains(unit, n) {
t.Errorf("yields %+v: the unit says %q:\n%s", c.y, n, unit)
}
}
if c.y == nil {
continue
}
slice := yieldSlice(c.y)
for _, w := range c.slice {
if !strings.Contains(slice, w) {
t.Errorf("yields %+v on %d processors: the slice does not say %q:\n%s", c.y, c.cpus, w, slice)
}
}
}
// The slice is placed beside the units, once, and again only when it differs.
was := unitDir
unitDir = t.TempDir()
defer func() { unitDir = was }()
p := aProcess()
p.Yields = &declaration.Yields{Priority: declaration.YieldIdle, CPUShare: 50}
for i, want := range []bool{true, false} {
if wrote, err := placeYieldSlice(p); err != nil || wrote != want {
t.Errorf("placing %d: wrote %v (%v), want %v", i, wrote, err, want)
}
}
if b, err := os.ReadFile(filepath.Join(unitDir, "meshyield.slice")); err != nil || !strings.Contains(string(b), "[Slice]") {
t.Errorf("the slice file: %q %v", b, err)
}
}
// **Three modes, one kind.** A scheduled process is a timer plus a unit that finishes, not a unit
@@ -124,17 +178,6 @@ func TestACronBecomesATimersCalendar(t *testing.T) {
}
}
// And the long-running mode is unchanged by any of it: it stays up and comes back.
func TestAProcessThatStaysUpIsStillRestartedWhenItExits(t *testing.T) {
unit := unitFor(aProcess())
if !strings.Contains(unit, "Restart=always") {
t.Fatalf("a process that should stay up is not restarted when it exits:\n%s", unit)
}
if strings.Contains(unit, "Type=oneshot") {
t.Fatalf("a process that should stay up is declared a step:\n%s", unit)
}
}
// **A unit file reinterprets a value in three ways nothing else does**, and a module's environment
// routinely contains all three — a generated password is arbitrary bytes.
func TestAnEnvironmentValueMeansWhatTheDeclarationSaid(t *testing.T) {
+78
View File
@@ -3,7 +3,9 @@ package apply
import (
"context"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
@@ -103,3 +105,79 @@ func TestARolledBackBuildIsNotPlacedAgain(t *testing.T) {
t.Fatalf("the apply does not say why it kept the build before: %+v", o)
}
}
// **A process nobody judges keeps the directory it works in until its restart** (novox/hq issue 734).
// On 2026-10-26 three checks on one builder were refused in the second a new build of it arrived: the
// applier deleted the running build's directory before the restart, and the `go version` the old
// process started in it failed with "getwd: no such file or directory". Here a process works in the
// build's directory without running from it, as that `go version` did: the apply of a new build leaves
// its directory in place, and the retired build goes once nothing works in it.
func TestAProcessNobodyJudgesKeepsItsDirectoryUntilItsRestart(t *testing.T) {
if _, err := os.Stat("/proc/self/cwd"); err != nil {
t.Skip("which process works in a build is read from /proc")
}
units, bundles := t.TempDir(), t.TempDir()
wasUnits, wasBundles := unitDir, daemonRoot
unitDir, daemonRoot = units, bundles
t.Cleanup(func() { unitDir, daemonRoot = wasUnits, wasBundles })
run := func(ctx context.Context, name string, args ...string) (string, error) { return "", nil }
oldBody, oldDigest := anArchive(t, map[string]string{"mesh-build-go": "old\n"})
newBody, newDigest := anArchive(t, map[string]string{"mesh-build-go": "new\n"})
builder := func(body []byte, digest string) string {
return `{"id":"mesh-build-go.builder","type":"process","name":"mesh-build-go","source":"` + serving(t, body) +
`","digest":"` + digest + `","run":["./mesh-build-go"]}`
}
_, state, err := Apply(context.Background(), archHost(t), declare(t, builder(oldBody, oldDigest)), store.State{},
store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
sleep, err := exec.LookPath("sleep")
if err != nil {
t.Skip("no sleep to run")
}
working := exec.Command(sleep, "60")
working.Dir = filepath.Join(bundles, "mesh-build-go")
if err := working.Start(); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = working.Process.Kill(); _ = working.Wait() })
cwd := filepath.Join("/proc", strconv.Itoa(working.Process.Pid), "cwd")
for deadline := time.Now().Add(5 * time.Second); ; {
if at, err := os.Readlink(cwd); err == nil && at == working.Dir {
break
}
if time.Now().After(deadline) {
t.Fatalf("PID %d does not work in %s within 5s", working.Process.Pid, working.Dir)
}
time.Sleep(10 * time.Millisecond)
}
if _, _, err := Apply(context.Background(), archHost(t), declare(t, builder(newBody, newDigest)), state,
store.OriginDeclared, run, nil, nil); err != nil {
t.Fatal(err)
}
if got, _ := os.ReadFile(filepath.Join(bundles, "mesh-build-go", "mesh-build-go")); string(got) != "new\n" {
t.Fatalf("the new build is not where the unit runs it from: %q", got)
}
at, err := os.Readlink(cwd)
if err != nil || strings.HasSuffix(at, " (deleted)") {
t.Fatalf("the directory a running process works in was deleted before its restart: %q (%v)", at, err)
}
if got, _ := os.ReadFile(filepath.Join(at, "mesh-build-go")); string(got) != "old\n" {
t.Fatalf("the process's directory no longer holds its own build: %q", got)
}
if len(witness.Trials(bundles)) != 0 {
t.Fatal("a process nobody judges went on trial")
}
// Its process gone, the next look deletes the retired build and keeps nothing about the process.
_ = working.Process.Kill()
_ = working.Wait()
witness.SweepAll(bundles)
if _, err := os.Stat(witness.Dir(bundles, "mesh-build-go")); !os.IsNotExist(err) {
t.Fatalf("a retired build nothing works in was kept: %v", err)
}
}
+24 -8
View File
@@ -27,13 +27,23 @@ type FetchObject func(ctx context.Context, name string, limit int64) ([]byte, er
// unchanged declaration, and a node that is asleep and has no bus, apply as they always did. One object is fetched
// once however many files name its digest.
func ResolveFileReferences(ctx context.Context, d *declaration.Declaration, fetch FetchObject) error {
return ResolveFileReferencesKept(ctx, d, fetch, nil)
}
// ResolveFileReferencesKept is ResolveFileReferences with the node's own keep of objects (ObjectKeep): every content had
// (a file here that verifies, or an object of the bus) is kept, and a kept one is used before the bus is asked. **A node
// depends on its bucket only for content it never had** (issue 924, 2026-10-29: novox's own composition named the
// export mesh-vault runs by a digest whose object the controller's sweep had removed, so the vault's apply failed while
// it ran; the bucket's objects are kept only while a composition of the controller names them, and a node composing its
// own names objects the controller no longer knows of). A nil keep keeps nothing.
func ResolveFileReferencesKept(ctx context.Context, d *declaration.Declaration, fetch FetchObject, keep *ObjectKeep) error {
fetched := map[string][]byte{}
for i, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || !f.IsReference() {
continue
}
body, err := referencedContent(ctx, f, fetch, fetched)
body, err := referencedContent(ctx, f, fetch, fetched, keep)
if err != nil {
return err
}
@@ -50,31 +60,37 @@ func ResolveFileReferences(ctx context.Context, d *declaration.Declaration, fetc
}
func referencedContent(ctx context.Context, f *declaration.File, fetch FetchObject,
fetched map[string][]byte) ([]byte, error) {
fetched map[string][]byte, keep *ObjectKeep) ([]byte, error) {
if body, ok := fetched[f.Digest]; ok {
return body, nil
}
if body, ok := alreadyThere(f); ok {
fetched[f.Digest] = body
return body, nil
}
name, err := declaration.ObjectName(f.Digest)
if err != nil {
return nil, fmt.Errorf("file %s (%s): %w", f.ID, f.Path, err)
}
if body, ok := alreadyThere(f); ok {
fetched[f.Digest] = body
keep.put(name, body)
return body, nil
}
if body, ok := keep.get(name, f); ok {
fetched[f.Digest] = body
return body, nil
}
if fetch == nil {
return nil, fmt.Errorf("file %s (%s) is given by digest %s and is not on this machine, and this node-engine "+
"has no link to the bus to fetch it from", f.ID, f.Path, f.Digest)
}
body, err := fetch(ctx, name, f.Size)
if err != nil {
return nil, fmt.Errorf("file %s (%s): fetching the object %s of this node's own object bucket: %w",
f.ID, f.Path, name, err)
return nil, fmt.Errorf("file %s (%s): fetching the object %s of this node's own object bucket%s: %w",
f.ID, f.Path, name, keep.never(), err)
}
if err := f.Verify(body); err != nil {
return nil, fmt.Errorf("%s (%s): %w; nothing was applied", f.ID, f.Path, err)
}
fetched[f.Digest] = body
keep.put(name, body)
return body, nil
}
+110
View File
@@ -6,6 +6,10 @@ import (
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/asks"
)
// The take act's ask (novox/hq ADR 0325 §8): a hand-over's fields and the act's own in one signed object, under a
@@ -68,6 +72,112 @@ func TestTheTakeActRunsOnlyOnAWarrant(t *testing.T) {
if a.Said != "started" || len(given) != 1 || given[0].Warrant != "w-1" || given[0].Path != "/srv/media/movies" {
t.Fatalf("a warranted ask did not reach the act whole: %+v %+v", a, given)
}
// 774 (ADR 0563 §3): a hand-over asked through the node seat is the engine's own ask, performed only on the
// operator's warrant to it: never on the call, not on a warrant for another digest, once on a warrant heard twice,
// and not when the disk changed since it was asked.
t.Run("the node seat's hand-over acts only on the operator's warrant", func(t *testing.T) {
for _, row := range []struct {
name string
warrant func(asked asks.Ask, w asks.Warrant) asks.Warrant
diskAfter string
twice bool
done int
state string
}{
{name: "no warrant", done: 0, state: HandAsked},
{name: "the operator's yes", warrant: yes, done: 1, state: HandDone},
{name: "the yes heard twice", warrant: yes, twice: true, done: 1, state: HandDone},
{name: "a warrant for another digest", warrant: func(a asks.Ask, w asks.Warrant) asks.Warrant {
w = yes(a, w)
w.AskDigest = "sha256:another"
return w
}, done: 0, state: HandAsked},
{name: "a no", warrant: func(a asks.Ask, w asks.Warrant) asks.Warrant {
w = yes(a, w)
w.Option, w.Label, w.Level = "no", "No, leave it", asks.Acknowledge
return w
}, done: 0, state: HandRefused},
{name: "the disk changed", warrant: yes, diskAfter: "0:0 700 9:2", done: 0, state: HandRefused},
{name: "expired", warrant: func(a asks.Ask, w asks.Warrant) asks.Warrant {
return asks.Warrant{Ask: a.ID, Asker: w.Asker, Outcome: asks.OutcomeExpired, At: w.At}
}, done: 0, state: HandRefused},
} {
t.Run(row.name, func(t *testing.T) {
h, asked, done := anEngine(t)
disk := "1000:1000 755 9:1"
h.JudgeHandOver = func(path string) (Judged, error) {
return Judged{Subject: path, Path: path, Sets: "owner jo, group jo, mode 0750", Disk: disk,
Words: "A directory used as found."}, nil
}
if _, err := h.Call(EngineCall{Verb: ActHandOver, Caller: "shanks.node-tools",
Args: map[string]any{"path": "/srv/x", "why": "a test"}}); err != nil {
t.Fatal(err)
}
if len(*asked) != 1 || *done != 0 {
t.Fatalf("the call asked %d and performed %d", len(*asked), *done)
}
// The act is one key, and the open ask one key made by a create (ADR 0563 §3).
if acts, _ := h.State.Keys("act."); len(acts) != 1 || acts[0] != "act."+(*asked)[0].ID {
t.Fatalf("the act is kept as %v", acts)
}
if open, _ := h.State.Keys("open."); len(open) != 1 || !strings.HasPrefix(open[0], "open.hand-over.") {
t.Fatalf("the open ask is kept as %v", open)
}
if _, err := h.Call(EngineCall{Verb: ActHandOver, Caller: "x", Args: map[string]any{"path": "/srv/x"}}); err == nil {
t.Fatal("a second ask of the same path was asked while one is open")
}
if row.diskAfter != "" {
disk = row.diskAfter
}
if row.warrant != nil {
w := row.warrant((*asked)[0], asks.Warrant{Asker: asks.EngineAsker("n1"), At: time.Now()})
h.Heard(w)
if row.twice {
h.Heard(w)
}
}
got, _ := h.Acts(0)
acts := got.(map[string]any)["acts"].([]HandAct)
if *done != row.done || len(acts) != 1 || acts[0].State != row.state {
t.Fatalf("performed %d (want %d), acts %+v (want %s)", *done, row.done, acts, row.state)
}
if open, _ := h.State.Keys("open."); (len(open) == 1) != (row.state == HandAsked) {
t.Fatalf("an ask %s leaves its open key as %v", row.state, open)
}
if row.state == HandDone && (acts[0].By != "operator" || acts[0].Via == "" || acts[0].Ask == "") {
t.Fatalf("the act does not record the warrant: %+v", acts[0])
}
})
}
})
}
// yes is the operator's yes to an ask, as the router writes it.
func yes(a asks.Ask, w asks.Warrant) asks.Warrant {
w.Ask, w.Outcome, w.Option, w.Label, w.Level = a.ID, asks.OutcomeChosen, "yes", "Yes", asks.Approve
w.By = &asks.Person{Who: asks.Operator, Kind: "telegram", Verified: "user id verified"}
w.AskDigest, w.Proofs = a.Digest(), []string{"P1"}
return w
}
// anEngine is a node seat with its own state in a temporary folder, a router that keeps what it was asked, and a
// hand-over that counts what it performed.
func anEngine(t *testing.T) (*HandActs, *[]asks.Ask, *int) {
t.Helper()
var asked []asks.Ask
done := 0
state := &MemState{}
h := &HandActs{Node: "n1", State: state,
HandOver: func(path, by string) (string, error) { done++; return path + " handed over by " + by, nil }}
h.Asker = &asker.Client{Module: asks.EngineAsker("n1"), State: state,
Publish: func(subject string, body any, id string) error {
if a, ok := body.(asks.Ask); ok && subject == asks.AskSubject(asks.EngineAsker("n1")) {
asked = append(asked, a)
}
return nil
},
Record: func(string) (*stdio.StateEntry, error) { return nil, nil }}
return h, &asked, &done
}
// The controller names an access by its declared resource id exactly, which may hold a `/`: it reaches the act,
+382
View File
@@ -0,0 +1,382 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/micro"
)
// The node seat node-engine, served on the engine's own link (novox/hq ADR 0563 §1).
//
// Each verb is answered on this node's subject alone, `mesh.seat.node-engine.tool.<verb>.<node>`, and on the same
// address naming its caller, `mesh.seat.node-engine.call.<verb>.<node>.<caller>`, whose last token the bus lets each user
// publish with its own name alone (novox/hq issue 365): the caller is that token, and a call on the subject naming none is
// from an unproven caller, recorded so. The body is the call's arguments; the answer is {result, error, node}, as every
// tool's (mesh-tools node-tools). One service, `node-engine`, answers discovery with each verb's seat, tool and node.
// EngineSeat is the seat.
const EngineSeat = catalogue.EngineSeat
// EngineToolSubject is where a verb of this node's seat is asked naming no caller.
func EngineToolSubject(verb, node string) string {
return "mesh.seat." + EngineSeat + ".tool." + verb + "." + node
}
// EngineCallPattern is where a verb of this node's seat is asked naming its caller, any caller.
func EngineCallPattern(verb, node string) string {
return "mesh.seat." + EngineSeat + ".call." + verb + "." + node + ".*"
}
// EngineEventSubject is where the seat says one of its events about this node.
func EngineEventSubject(event, node string) string {
return "mesh.seat." + EngineSeat + ".event." + event + "." + node
}
// EngineCall is one call of the seat: its verb, its caller as the bus proved it, and its arguments.
type EngineCall struct {
Verb string
Caller string
Args map[string]any
}
// EngineFunc answers one call: its result, or why it is refused.
type EngineFunc func(EngineCall) (any, error)
// CallerOfSubject is the caller a call's subject names for this node's verb: the last token of a caller-named subject,
// each `~` a dot, or stdio.Unproven on the subject that names none or on any other.
func CallerOfSubject(subject, verb, node string) string {
token, ok := strings.CutPrefix(subject, "mesh.seat."+EngineSeat+".call."+verb+"."+node+".")
if !ok || token == "" || strings.ContainsAny(token, ".*> ") {
return stdio.Unproven
}
return strings.ReplaceAll(token, "~", ".")
}
// MaxEngineCall bounds a call's arguments: a path, an access and a few words.
const MaxEngineCall = 16 << 10
// AnswerEngineCall is the envelope one call is answered with: the verb's result, or its refusal in words.
func AnswerEngineCall(handle EngineFunc, verb, node, subject string, body []byte) []byte {
envelope := map[string]any{"node": node}
say := func() []byte {
raw, err := json.Marshal(envelope)
if err != nil {
raw, _ = json.Marshal(map[string]any{"node": node, "error": "the answer could not be written: " + err.Error()})
}
return raw
}
args := map[string]any{}
switch {
case handle == nil:
envelope["error"] = "this engine serves no verb of " + EngineSeat
return say()
case len(body) > MaxEngineCall:
envelope["error"] = fmt.Sprintf("the call is %d bytes, more than a call of %s takes", len(body), EngineSeat)
return say()
case len(strings.TrimSpace(string(body))) > 0:
if err := json.Unmarshal(body, &args); err != nil {
envelope["error"] = "the call's arguments are not an object: " + err.Error()
return say()
}
}
result, err := handle(EngineCall{Verb: verb, Caller: CallerOfSubject(subject, verb, node), Args: args})
if err != nil {
envelope["error"] = err.Error()
} else {
envelope["result"] = result
}
return say()
}
// EngineSeats is a link that serves this node's seat. Optional, as Asked is.
type EngineSeats interface {
ServeEngineSeat(EngineFunc)
}
// ServeEngineSeat answers every verb of the seat on this node's subjects, as the service node-engine, which says each
// verb's seat, tool and node to discovery (novox/hq ADR 0197). A bus whose user list is older than the grant refuses the
// subscriptions, said by the client; a caller then hears nobody answer.
func (l *natsLink) ServeEngineSeat(handle EngineFunc) {
if handle == nil || l.conn == nil {
return
}
svc, err := micro.AddService(l.conn, micro.Config{Name: EngineSeat, Version: "1.0.0",
Description: "the node seat node-engine of " + l.node + " (novox/hq ADR 0563)", QueueGroupDisabled: true})
if err != nil {
return
}
for _, v := range catalogue.EngineVerbs() {
verb := v.Name
answer := micro.HandlerFunc(func(r micro.Request) {
_ = r.Respond(AnswerEngineCall(handle, verb, l.node, r.Subject(), r.Data()))
})
meta := map[string]string{"seat": EngineSeat, "tool": verb, "node": l.node}
_ = svc.AddEndpoint(verb, answer, micro.WithEndpointSubject(EngineToolSubject(verb, l.node)),
micro.WithEndpointMetadata(meta))
_ = svc.AddEndpoint(verb+"-called", answer, micro.WithEndpointSubject(EngineCallPattern(verb, l.node)),
micro.WithEndpointMetadata(meta))
}
l.engine = svc
}
// SayEngine says one of the seat's events about this node, on the events stream.
func (l *natsLink) SayEngine(event string, body any) error {
raw, err := json.Marshal(body)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_, err = l.js.Publish(EngineEventSubject(event, l.node), raw, nats.Context(ctx))
return err
}
// The engine asks the operator under its own asker name (asks.EngineAsker) on the router's seat (ADR 0259 §3, as ADR
// 0563 §3 changes it): its ask and cancel on the seat's work stream, its warrants heard on its own decided subject, and
// its asks' state read from the router's records under its own name.
// RouterRecords is the router's bucket of asks, which each asker reads under its own name: the state `asks` of the
// operator-channel's holder, the messenger (mesh-catalog modules/messenger).
const RouterRecords = "messenger_asks"
// OperatorAsks is a link that asks the operator as this node's engine. Optional.
type OperatorAsks interface {
PublishAsk(subject string, body any, id string) error
RouterRecord(key string) (*stdio.StateEntry, error)
HearWarrants(asker string, heard func(asks.Warrant)) error
}
// PublishAsk publishes an ask or a cancel on the router's seat, once per id within the stream's duplicate window.
func (l *natsLink) PublishAsk(subject string, body any, id string) error {
raw, err := json.Marshal(body)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_, err = l.js.Publish(subject, raw, nats.MsgId(id), nats.Context(ctx))
return err
}
// RouterRecord reads one key of the router's records, `<asker>.<id>`; nil when it holds none.
func (l *natsLink) RouterRecord(key string) (*stdio.StateEntry, error) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
msg, err := l.conn.RequestWithContext(ctx, "$JS.API.DIRECT.GET.KV_"+RouterRecords+".$KV."+RouterRecords+"."+key, nil)
if err != nil {
return nil, err
}
if status := msg.Header.Get("Status"); status == "404" {
return nil, nil
} else if status != "" {
return nil, fmt.Errorf("the router's record %s could not be read: %s %s", key, status, msg.Header.Get("Description"))
}
return &stdio.StateEntry{Key: key, Value: msg.Data}, nil
}
// HearWarrants hears the warrants for asker on its decided subject, each handed to heard as it arrives.
func (l *natsLink) HearWarrants(asker string, heard func(asks.Warrant)) error {
sub, err := l.conn.Subscribe(asks.DecidedSubject(asker), func(msg *nats.Msg) {
var w asks.Warrant
if json.Unmarshal(msg.Data, &w) == nil {
heard(w)
}
})
if err != nil {
return err
}
l.warrants = sub
return nil
}
// errNoAsking is a link that cannot ask the operator.
var errNoAsking = errors.New("the link open now cannot ask the operator")
// discoveryQuiet is how long the engine waits after the last answer to the bus's discovery, and discoveryMost the
// longest it waits in all.
const (
discoveryQuiet = 300 * time.Millisecond
discoveryMost = 2 * time.Second
)
// HeardServed asks the bus's discovery (novox/hq ADR 0197) whether a seat's verb is answered on this machine: an
// endpoint naming the seat, the verb as its tool and this node.
func (l *natsLink) HeardServed(seat, verb string) (bool, error) {
inbox := l.conn.NewRespInbox()
sub, err := l.conn.SubscribeSync(inbox)
if err != nil {
return false, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := l.conn.PublishRequest("$SRV.INFO", inbox, nil); err != nil {
return false, fmt.Errorf("asking the bus who serves what: %w", err)
}
heard := false
deadline := time.Now().Add(discoveryMost)
for time.Now().Before(deadline) {
msg, err := sub.NextMsg(discoveryQuiet)
if err != nil {
break
}
var info micro.Info
if json.Unmarshal(msg.Data, &info) != nil {
continue
}
for _, e := range info.Endpoints {
node := e.Metadata["node"]
if node == "" {
node = info.ID
}
if e.Metadata["seat"] == seat && e.Metadata["tool"] == verb && node == l.node {
heard = true
}
}
}
return heard, nil
}
// HeardServed is the link open now asking the bus's discovery whether a seat's verb is answered on this machine.
func (q *Session) HeardServed(seat, verb string) (bool, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
asker, ok := bus.(interface {
HeardServed(string, string) (bool, error)
})
if bus == nil || !ok {
return false, fmt.Errorf("%w: this machine has no link to the bus open now", ErrNotAsked)
}
return asker.HeardServed(seat, verb)
}
// ---- the engine's acts in the bus's own storage (ADR 0563 §2, §3) -------------------------------------------------
// actsStore is this node's keys of ActsBucket over the link open now: bound, never created (the grants part creates
// it, as the node-engine's declared state).
func (l *natsLink) actsStore() (nats.KeyValue, error) {
kv, err := l.js.KeyValue(ActsBucket)
if err != nil {
return nil, fmt.Errorf("the node-engine's state %s cannot be bound (it is created from the node-engine's "+
"manifest; asked again once it is): %w", ActsBucket, err)
}
return kv, nil
}
// BusState is the engine's ActStore: its own node's keys of ActsBucket, `<node>.<key>`, over whichever link the session
// has open; with none open every read and write is refused, and nothing is kept anywhere else.
type BusState struct {
Session *Session
Node string
}
func (b BusState) kv() (nats.KeyValue, error) {
b.Session.init()
b.Session.mu.Lock()
bus := b.Session.bus
b.Session.mu.Unlock()
l, ok := bus.(interface{ actsStore() (nats.KeyValue, error) })
if bus == nil || !ok {
return nil, fmt.Errorf("%w: this machine has no link to the bus open now, where its acts are kept", ErrNotAsked)
}
return l.actsStore()
}
func (b BusState) key(k string) string { return b.Node + "." + k }
// changed is a write that lost its compare-and-set, as stdio.ErrChanged.
func changed(err error) error {
var api *nats.APIError
if errors.Is(err, nats.ErrKeyExists) || (errors.As(err, &api) && api.ErrorCode == nats.JSErrCodeStreamWrongLastSequence) {
return stdio.ErrChanged
}
return err
}
func (b BusState) Get(key string) (*stdio.StateEntry, error) {
kv, err := b.kv()
if err != nil {
return nil, err
}
e, err := kv.Get(b.key(key))
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
return &stdio.StateEntry{Key: key, Value: e.Value(), Revision: e.Revision()}, nil
}
func (b BusState) Create(key string, value any) (uint64, error) {
kv, err := b.kv()
if err != nil {
return 0, err
}
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
rev, err := kv.Create(b.key(key), raw)
return rev, changed(err)
}
func (b BusState) Update(key string, value any, revision uint64) (uint64, error) {
kv, err := b.kv()
if err != nil {
return 0, err
}
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
rev, err := kv.Update(b.key(key), raw, revision)
return rev, changed(err)
}
func (b BusState) Delete(key string) error {
kv, err := b.kv()
if err != nil {
return err
}
return kv.Delete(b.key(key))
}
// Keys are this node's keys under prefix, without the node: a watch of `<node>.>` alone, the one filter the node's
// grant names, its values left out.
func (b BusState) Keys(prefix string) ([]string, error) {
kv, err := b.kv()
if err != nil {
return nil, err
}
w, err := kv.Watch(b.Node+".>", nats.MetaOnly(), nats.IgnoreDeletes())
if err != nil {
return nil, err
}
defer func() { _ = w.Stop() }()
var out []string
for {
select {
case e := <-w.Updates():
if e == nil {
sort.Strings(out)
return out, nil
}
if k := strings.TrimPrefix(e.Key(), b.Node+"."); strings.HasPrefix(k, prefix) {
out = append(out, k)
}
case <-time.After(10 * time.Second):
return nil, errors.New("the node-engine's acts could not be listed within 10s")
}
}
}
+50 -2
View File
@@ -10,6 +10,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
)
@@ -92,7 +93,32 @@ func (b OverNATS) FileObject(ctx context.Context, name string, limit int64) ([]b
if err != nil {
return nil, err
}
stream := declaration.FileStreamPrefix + bucket
return objectByName(ctx, js, bucket, declaration.FileStreamPrefix+bucket, name, limit)
}
// GrantsObject reads one object of the grants part's bucket of sealed user lists by its name (novox/hq ADR 0547 clause 7),
// with the same subjects and nothing more as FileObject: the bus node is granted exactly those (mesh-sdk
// buscompose.GrantsUserListFollows).
func (b OverNATS) GrantsObject(ctx context.Context, name string, limit int64) ([]byte, error) {
js := b.JS
wait := 10 * time.Second
if at, ok := ctx.Deadline(); ok {
if wait = time.Until(at); wait <= 0 {
return nil, context.DeadlineExceeded
}
}
if b.Conn != nil {
bounded, err := b.Conn.JetStream(nats.MaxWait(wait))
if err != nil {
return nil, err
}
js = bounded
}
return objectByName(ctx, js, delivery.GrantsUserListObjects, delivery.GrantsUserListObjectStream, name, limit)
}
// objectByName is one object of bucket (its stream stream) by its name, no larger than limit, verified against its digest.
func objectByName(ctx context.Context, js nats.JetStreamContext, bucket, stream, name string, limit int64) ([]byte, error) {
meta, err := js.GetLastMsg(stream, "$O."+bucket+".M."+base64.URLEncoding.EncodeToString([]byte(name)),
nats.DirectGet(), nats.Context(ctx))
if err != nil {
@@ -109,7 +135,7 @@ func (b OverNATS) FileObject(ctx context.Context, name string, limit int64) ([]b
return nil, fmt.Errorf("the object %s was deleted from the bucket %s", name, bucket)
}
if info.Size > uint64(limit) {
return nil, fmt.Errorf("the object is larger than the %d bytes the declaration names", limit)
return nil, fmt.Errorf("the object is larger than the %d bytes allowed", limit)
}
if info.Size == 0 {
return []byte{}, nil
@@ -142,6 +168,28 @@ func (l *natsLink) FileObject(ctx context.Context, name string, limit int64) ([]
return OverNATS{Conn: l.conn, JS: l.js, Node: l.node}.FileObject(ctx, name, limit)
}
func (l *natsLink) GrantsObject(ctx context.Context, name string, limit int64) ([]byte, error) {
return OverNATS{Conn: l.conn, JS: l.js, Node: l.node}.GrantsObject(ctx, name, limit)
}
// GrantsObjects is a link that can read an object of the grants part's bucket of sealed user lists.
type GrantsObjects interface {
GrantsObject(ctx context.Context, name string, limit int64) ([]byte, error)
}
// GrantsObject reads an object of the grants part's bucket by its name on the link open now.
func (q *Session) GrantsObject(ctx context.Context, name string, limit int64) ([]byte, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
g, ok := bus.(GrantsObjects)
if bus == nil || !ok {
return nil, ErrNoLink
}
return g.GrantsObject(ctx, name, limit)
}
var (
_ FilesBus = OverNATS{}
_ FilesBus = (*natsLink)(nil)
+769
View File
@@ -0,0 +1,769 @@
package link
import (
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/asks"
)
// The node seat node-engine's acts (novox/hq ADR 0563).
//
// A hand-over and an access act are the engine's own asks, performed only on the warrant (§3): a call performs nothing.
// The engine judges every value as it does for the controller's signed ask, then refuses in words or asks the operator
// under its own asker name (asks.EngineAsker) at the level approve, binding the digest of the verb, the node, every
// argument and the owner, group and mode it will set. A warrant is claimed once by compare-and-set in the engine's own
// state (asker.Client.Take), held to the ask it answers, and to the act bound and the disk as judged, judged again now;
// only then is the act performed and recorded with the warrant. A decline, an expiry or a changed disk says refused and
// changes nothing. One ask is open per path or access.
//
// The read-only verbs need no warrant (§2): judge-directories changes nothing; setuid-search is bounded here, one at a
// time and at most one started in an hour. Every act, with who asked as the runtime proved it, is kept in the engine's own
// declared state acts (ActsBucket, created by the grants part), under its own node's keys alone, read by the verb acts.
// The acts of the seat, as its verbs name them.
const (
ActHandOver = "hand-over"
ActAccess = "access-act"
ActJudgeDirectories = "judge-directories"
ActSetuidSearch = "setuid-search"
)
// The states of an act.
const (
HandAsked = "asked"
HandDone = "done"
HandRefused = "refused"
HandAnswered = "answered"
)
// SearchEvery is how often a setuid search may be started on a node (ADR 0563 §2): one in any hour.
const SearchEvery = time.Hour
// askMargin is how long past an ask's expiry the router is given to say it ended, before the engine closes it itself.
const askMargin = 5 * time.Minute
// ActsBucket is the node-engine's declared state acts (its manifest's state, ADR 0563 §2, §3), the bucket the grants
// part creates (mesh-sdk buscompose EngineActsBucket): keys `<node>.act.<id>`, `<node>.open.<act>.<digest>` and
// `<node>.ask.<id>`, each written by this node's engine alone.
const ActsBucket = "mesh-host_acts"
// Judged is what the engine judged of one act before it asks: what it is about, what it will set (the owner, group and
// mode, in words), and what the disk holds now (Disk), compared again when the warrant arrives.
type Judged struct {
// Subject is the path of a hand-over, or the access of an access act.
Subject string
// Path is the directory the act changes, as the engine states it.
Path string
// Sets is the owner, group and mode the act will set, in words; bound in the ask's digest.
Sets string
// Disk is what the engine read on disk when it judged: owner, group, mode and the file it is.
Disk string
// Words is what the operator is shown of it, without a path (ADR 0234 §6): the path is in the whole words alone.
Words string
}
// HandAct is one act asked through the seat, as acts reads it.
type HandAct struct {
ID string `json:"id"`
Act string `json:"act"`
Subject string `json:"subject,omitempty"`
Caller string `json:"caller"`
Why string `json:"why,omitempty"`
At time.Time `json:"at"`
State string `json:"state"`
Said string `json:"said,omitempty"`
Ended time.Time `json:"ended,omitempty"`
// Ask is the operator's ask of a hand-over or an access act; Sets and Disk what was judged when it was asked.
Ask string `json:"ask,omitempty"`
Sets string `json:"sets,omitempty"`
Disk string `json:"disk,omitempty"`
Path string `json:"path,omitempty"`
// Expires is when the ask ends unanswered.
Expires time.Time `json:"expires,omitempty"`
// By, Via, Proofs are the operator's warrant, recorded with the act (ADR 0563 §3).
By string `json:"by,omitempty"`
Via string `json:"via,omitempty"`
Proofs []string `json:"proofs,omitempty"`
// Auto says the engine asked it itself, on finding a library's root holding data outside its group (§4).
Auto bool `json:"auto,omitempty"`
}
// RootVerdict is the engine's answer to root-free (ADR 0563 §6), in the controller's words.
type RootVerdict struct {
Machine string `json:"machine"`
Free bool `json:"free"`
Why string `json:"why"`
Judged time.Time `json:"judged"`
}
// HandActs is the engine's side of the seat. Every func is the engine's own judgement or act; nil refuses.
type HandActs struct {
Node string
// State is where the acts and the asks are kept: this node's keys of ActsBucket (BusState), each key one act.
State ActStore
// Asker asks the operator as asks.EngineAsker(Node), with State as its store.
Asker *asker.Client
// JudgeHandOver judges a hand-over's path, refusing in words; HandOver records it on the warrant.
JudgeHandOver func(path string) (Judged, error)
HandOver HandOverFunc
// JudgeAccess judges an access act on an access, refusing in words; AccessAct runs it on the warrant.
JudgeAccess func(act, access string) (Judged, error)
AccessAct AccessActFunc
// JudgeDirectories, SetuidSearch and Searching are the read-only verbs' and whether a search runs now.
JudgeDirectories JudgeDirectoriesFunc
SetuidSearch SetuidSearchFunc
Searching func() bool
// RootFree is this machine's root-free verdict now.
RootFree func() RootVerdict
// Say says one of the seat's events, asked, done or refused; nil says nothing.
Say func(event string, body any)
Now func() time.Time
// Nonce keeps two asks of one second apart; nil is random.
Nonce func() string
mu sync.Mutex
}
func (h *HandActs) now() time.Time {
if h.Now != nil {
return h.Now()
}
return time.Now()
}
func (h *HandActs) nonce() string {
if h.Nonce != nil {
return h.Nonce()
}
var b [2]byte
_, _ = rand.Read(b[:])
return hex.EncodeToString(b[:])
}
func (h *HandActs) say(event string, a HandAct) {
if h.Say != nil {
h.Say(event, map[string]any{"act": a.Act, "subject": a.Subject, "caller": a.Caller, "ask": a.Ask,
"state": a.State, "said": a.Said, "node": h.Node})
}
}
// Call is the seat's one entry: a verb, the caller as the runtime proved it, and its arguments.
func (h *HandActs) Call(c EngineCall) (any, error) {
arg := func(k string) string { s, _ := c.Args[k].(string); return strings.TrimSpace(s) }
switch c.Verb {
case ActHandOver:
return h.AskHandOver(c.Caller, arg("path"), arg("why"), false)
case ActAccess:
return h.AskAccess(c.Caller, arg("act"), arg("access"), arg("why"), false)
case ActJudgeDirectories:
return h.Judge(c.Caller)
case ActSetuidSearch:
return h.Search(c.Caller, arg("why"))
case "root-free":
if h.RootFree == nil {
return nil, errors.New("this engine judges no machine root-free")
}
return map[string]any{"machines": []RootVerdict{h.RootFree()}}, nil
case "acts":
if id := arg("id"); id != "" {
return h.Act(id)
}
limit, _ := strconv.Atoi(arg("limit"))
if f, ok := c.Args["limit"].(float64); ok {
limit = int(f)
}
return h.Acts(limit)
}
return nil, fmt.Errorf("the node seat %s serves no verb %q", EngineSeat, c.Verb)
}
// ---- the record --------------------------------------------------------------------------------------------------
// ActStore is the engine's own state, this node's keys alone (BusState prefixes them with the node): each act one key,
// written by compare-and-set on its revision, an open ask one key made by a create.
type ActStore interface {
Get(key string) (*stdio.StateEntry, error)
Create(key string, value any) (uint64, error)
Update(key string, value any, revision uint64) (uint64, error)
Delete(key string) error
// Keys are the keys under prefix.
Keys(prefix string) ([]string, error)
}
func actKey(id string) string { return "act." + id }
// openKey is the one key an open ask of act on subject holds: made by a create, so a second ask of it is refused.
func openKey(act, subject string) string {
sum := sha256.Sum256([]byte(subject))
return "open." + act + "." + hex.EncodeToString(sum[:12])
}
// read is every act kept, oldest first.
func (h *HandActs) read() ([]HandAct, error) {
keys, err := h.State.Keys("act.")
if err != nil {
return nil, fmt.Errorf("the engine's acts cannot be listed: %w", err)
}
var out []HandAct
for _, k := range keys {
a, _, err := h.readOne(k)
if err != nil {
return nil, err
}
if a != nil {
out = append(out, *a)
}
}
sort.SliceStable(out, func(i, j int) bool { return out[i].At.Before(out[j].At) })
return out, nil
}
func (h *HandActs) readOne(key string) (*HandAct, uint64, error) {
e, err := h.State.Get(key)
if err != nil || e == nil {
return nil, 0, err
}
var a HandAct
if err := json.Unmarshal(e.Value, &a); err != nil {
return nil, 0, fmt.Errorf("the engine's act %s cannot be read: %w", key, err)
}
return &a, e.Revision, nil
}
// judgedKey keeps the node's judgements and refused searches, the last keptJudged of them: any caller may ask them, so
// they roll in one key and never add one (ADR 0563 §2: a wrong caller gains nothing, and fills no bucket).
const (
judgedKey = "judged"
keptJudged = 50
)
// roll adds a to the rolling key by compare-and-set, the oldest falling out.
func (h *HandActs) roll(a HandAct) error {
for range 5 {
e, err := h.State.Get(judgedKey)
if err != nil {
return err
}
var kept []HandAct
if e != nil {
if err := json.Unmarshal(e.Value, &kept); err != nil {
kept = nil // an unreadable roll is started again, never a reason to refuse
}
}
kept = append(kept, a)
if len(kept) > keptJudged {
kept = kept[len(kept)-keptJudged:]
}
if e == nil {
_, err = h.State.Create(judgedKey, kept)
} else {
_, err = h.State.Update(judgedKey, kept, e.Revision)
}
if !errors.Is(err, stdio.ErrChanged) {
return err
}
}
return errors.New("the engine's judgements kept changing under its write")
}
func (h *HandActs) rolled() ([]HandAct, error) {
e, err := h.State.Get(judgedKey)
if err != nil || e == nil {
return nil, err
}
var kept []HandAct
if err := json.Unmarshal(e.Value, &kept); err != nil {
return nil, fmt.Errorf("the engine's judgements cannot be read: %w", err)
}
return kept, nil
}
func (h *HandActs) record(a HandAct) error {
_, err := h.State.Create(actKey(a.ID), a)
return err
}
// end marks the open act of an ask ended, once, by compare-and-set on its revision, and frees its open key: false when
// it was not open.
func (h *HandActs) end(ask string, set func(*HandAct)) (HandAct, bool, error) {
for range 5 {
a, rev, err := h.readOne(actKey(ask))
if err != nil || a == nil || a.State != HandAsked {
return HandAct{}, false, err
}
set(a)
a.Ended = h.now().UTC()
if _, err := h.State.Update(actKey(ask), *a, rev); errors.Is(err, stdio.ErrChanged) {
continue
} else if err != nil {
return HandAct{}, false, err
}
_ = h.State.Delete(openKey(a.Act, a.Subject))
return *a, true, nil
}
return HandAct{}, false, errors.New("the engine's act " + ask + " kept changing under its write")
}
// Act is the verb acts with an id: that one act.
func (h *HandActs) Act(id string) (any, error) {
a, _, err := h.readOne(actKey(id))
if err != nil {
return nil, err
}
if a == nil {
return nil, fmt.Errorf("this engine keeps no act %q", id)
}
return map[string]any{"node": h.Node, "acts": []HandAct{*a}}, nil
}
// Acts is the verb acts: every act kept, newest first, at most limit when given.
func (h *HandActs) Acts(limit int) (any, error) {
acts, err := h.read()
if err != nil {
return nil, err
}
judged, err := h.rolled()
if err != nil {
return nil, err
}
acts = append(acts, judged...)
sort.SliceStable(acts, func(i, j int) bool { return acts[i].At.Before(acts[j].At) })
out := make([]HandAct, 0, len(acts))
for i := len(acts) - 1; i >= 0; i-- {
out = append(out, acts[i])
}
if limit > 0 && len(out) > limit {
out = out[:limit]
}
return map[string]any{"node": h.Node, "acts": out}, nil
}
// ---- the read-only verbs (§2) -------------------------------------------------------------------------------------
// Judge is judge-directories: the engine's judgement, read-only, recorded with its caller.
func (h *HandActs) Judge(caller string) (any, error) {
if h.JudgeDirectories == nil {
return nil, errors.New("this engine judges no directory")
}
said, err := h.JudgeDirectories(caller)
a := HandAct{ID: h.id(ActJudgeDirectories), Act: ActJudgeDirectories, Caller: caller, At: h.now().UTC(), State: HandAnswered}
if err != nil {
a.State, a.Said = HandRefused, err.Error()
}
if rerr := h.roll(a); rerr != nil && err == nil {
return nil, rerr
}
if err != nil {
return nil, err
}
return map[string]any{"node": h.Node, "said": said}, nil
}
// Search is setuid-search: refused while one runs and when one was started within the hour, naming when the next may
// start; otherwise the last search is thrown away and a full one starts, recorded with its caller.
func (h *HandActs) Search(caller, why string) (any, error) {
if h.SetuidSearch == nil {
return nil, errors.New("this engine keeps no search for setuid programs")
}
h.mu.Lock()
defer h.mu.Unlock()
now := h.now().UTC()
if h.Searching != nil && h.Searching() {
return nil, errors.New("a search for setuid programs runs on this machine now; no second is started while it does")
}
// The last start is a key of its own, claimed by compare-and-set before the search starts: no number of other acts,
// nor the record of acts, moves it (ADR 0563 §2).
if err := h.claimSearch(now, caller); err != nil {
return nil, err
}
said, err := h.SetuidSearch(caller)
a := HandAct{ID: h.id(ActSetuidSearch), Act: ActSetuidSearch, Caller: caller, Why: why, At: now, State: HandDone, Said: said}
if err != nil {
a.State, a.Said = HandRefused, err.Error()
}
// A search started is bounded to one an hour and kept as an act of its own; a refused one, which any caller may
// cause, rolls with the judgements.
record := h.record
if err != nil {
record = h.roll
}
if rerr := record(a); rerr != nil {
return nil, rerr
}
if err != nil {
return nil, err
}
return map[string]any{"node": h.Node, "said": said}, nil
}
// searchKey keeps the last search's start on this node, and libKey each library's own take ask: keys of their own, which
// no other act's key moves, kept for the bucket's age (ninety days, the node-engine's manifest).
const searchKey = "search.last"
func libKey(access string) string {
sum := sha256.Sum256([]byte(access))
return "lib." + hex.EncodeToString(sum[:12]) + ".asked"
}
// started is the last search's start as kept, and who asked it.
type started struct {
At time.Time `json:"at"`
By string `json:"by"`
}
// claimSearch refuses a search within SearchEvery of the last start, naming when the next may start, and otherwise
// claims this one by compare-and-set: a second claim at the same moment loses.
func (h *HandActs) claimSearch(now time.Time, caller string) error {
e, err := h.State.Get(searchKey)
if err != nil {
return err
}
var rev uint64
if e != nil {
var last started
if err := json.Unmarshal(e.Value, &last); err != nil {
return fmt.Errorf("the last search's start cannot be read, so none is started: %w", err)
}
if now.Sub(last.At) < SearchEvery {
return fmt.Errorf("a search was started at %s by %s, and one is started in any hour: the next may start "+
"at %s", last.At.Format(time.RFC3339), last.By, last.At.Add(SearchEvery).Format(time.RFC3339))
}
rev = e.Revision
}
claim := started{At: now, By: caller}
if rev == 0 {
_, err = h.State.Create(searchKey, claim)
} else {
_, err = h.State.Update(searchKey, claim, rev)
}
if errors.Is(err, stdio.ErrChanged) {
return errors.New("another search was started at this moment; no second is started")
}
return err
}
// ---- the asks (§3, §4) --------------------------------------------------------------------------------------------
func (h *HandActs) id(act string) string {
return act + "-" + h.now().UTC().Format("20060102t150405") + "-" + h.nonce()
}
// bound is the act an ask's yes binds: the verb, the node, every argument and what it will set (ADR 0563 §3).
func (h *HandActs) bound(a HandAct) asks.Act {
act := asks.Act{"verb": a.Act, "node": h.Node, "arg.subject": a.Subject, "arg.why": a.Why, "sets": a.Sets}
if a.Path != "" {
act["path"] = a.Path
}
return act
}
// AskHandOver is hand-over: judged, refused in words or asked of the operator. auto is the engine asking by itself.
func (h *HandActs) AskHandOver(caller, path, why string, auto bool) (any, error) {
if h.JudgeHandOver == nil || h.HandOver == nil {
return nil, errors.New("this engine takes no hand-over")
}
j, err := h.JudgeHandOver(path)
if err != nil {
return nil, err
}
return h.ask(ActHandOver, caller, why, auto, j)
}
// AskAccess is access-act: the act take on an access, judged on the library's home, refused or asked of the operator.
func (h *HandActs) AskAccess(caller, act, access, why string, auto bool) (any, error) {
if h.JudgeAccess == nil || h.AccessAct == nil {
return nil, errors.New("this engine runs no act on an accessed folder")
}
if act != ActTake {
return nil, fmt.Errorf("the act asked is %q, and the one act on an access is %q", act, ActTake)
}
j, err := h.JudgeAccess(act, access)
if err != nil {
return nil, err
}
return h.ask(ActAccess, caller, why, auto, j)
}
// AskedFor says whether an act on subject was ever asked by the engine itself: a library's take is asked by itself once
// (ADR 0563 §4), and again only by hand.
func (h *HandActs) AskedFor(act, subject string) bool {
if act == ActAccess {
e, err := h.State.Get(libKey(subject))
return err != nil || e != nil // unread is never a reason to ask again
}
acts, err := h.read()
if err != nil {
return true
}
for _, a := range acts {
if a.Act == act && a.Subject == subject && (a.Auto || a.State == HandAsked) {
return true
}
}
return false
}
func (h *HandActs) ask(act, caller, why string, auto bool, j Judged) (any, error) {
if h.Asker == nil {
return nil, errors.New("this engine reaches no operator to ask: nothing was asked and nothing changed")
}
h.mu.Lock()
defer h.mu.Unlock()
now := h.now().UTC()
a := HandAct{ID: h.id(act), Act: act, Subject: j.Subject, Path: j.Path, Caller: caller, Why: why, At: now,
State: HandAsked, Sets: j.Sets, Disk: j.Disk, Expires: now.Add(asks.ApproveLasts - time.Minute), Auto: auto}
a.Ask = a.ID
// One ask open per path or access: its open key made by a create, which a second ask cannot make.
if _, err := h.State.Create(openKey(act, j.Subject), map[string]string{"ask": a.Ask}); errors.Is(err, stdio.ErrChanged) {
held := "another"
if e, _ := h.State.Get(openKey(act, j.Subject)); e != nil {
var o struct {
Ask string `json:"ask"`
}
if json.Unmarshal(e.Value, &o) == nil && o.Ask != "" {
held = o.Ask
}
}
return nil, fmt.Errorf("%s of %s is asked already: the ask %s waits for the operator's answer", act, j.Subject, held)
} else if err != nil {
return nil, err
}
if err := h.record(a); err != nil {
_ = h.State.Delete(openKey(act, j.Subject))
return nil, err
}
bind, err := asks.ActDigest(h.bound(a))
if err != nil {
return nil, err
}
headline, does, stays := "Hand a directory on "+h.Node+" to the mesh?", "The mesh gives it "+j.Sets+".",
"The directory stays as it was found."
if act == ActAccess {
headline, does, stays = "Keep the library "+strings.TrimPrefix(j.Subject, "library:")+" on "+h.Node+"?",
"The library's group, group write and setgid are set on everything under its root.",
"The library waits and nothing under its root changes."
}
if len([]rune(headline)) > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
asked := asks.Ask{ID: a.Ask, Headline: headline, Who: asks.Operator,
Explanation: j.Words + " Your yes sets " + j.Sets + "; nothing changes until you answer.",
Whole: j.Words + " (" + j.Path + ") Your yes sets " + j.Sets + "; nothing changes until you answer.",
Details: "Asked by " + caller + " on " + h.Node + orWhy(why),
Options: []asks.Option{{ID: "yes", Label: "Yes", Does: does, Level: asks.Approve, Binds: bind},
{ID: "no", Label: "No, leave it", Does: stays, Level: asks.Acknowledge}},
Expires: a.Expires, OnExpiry: stays, About: EngineSeat + "-" + h.Node + "-" + act}
if err := h.Asker.Ask(asked); err != nil {
_, _, _ = h.end(a.Ask, func(x *HandAct) { x.State, x.Said = HandRefused, "the operator could not be asked: "+err.Error() })
return nil, fmt.Errorf("the operator could not be asked, and nothing changed: %w", err)
}
// Once asked, a library's own take ask is kept's own take ask: once, in a key of its own (ADR 0563 §4).
if auto && act == ActAccess {
_, _ = h.State.Create(libKey(j.Subject), map[string]any{"ask": a.Ask, "at": now})
}
h.say(HandAsked, a)
return map[string]any{"asked": a.Ask, "node": h.Node, "says": "the operator is asked on their channel; only " +
"their yes changes anything. acts shows where it stands."}, nil
}
func orWhy(why string) string {
if why == "" {
return ""
}
return ": " + why
}
// Warranted is the act on the operator's answer, handed once by asker.Client.Take after it held the warrant to the
// ask: a no is recorded refused; a yes is held to the act the option bound and the disk as judged, judged again now,
// and only then performed and recorded done with the warrant.
func (h *HandActs) Warranted(o asks.Option, w asks.Warrant) error {
a, _, err := h.readOne(actKey(w.Ask))
if err != nil {
return err
}
if a == nil || a.State != HandAsked {
return fmt.Errorf("the warrant answers the ask %s, which is not open here: nothing is done", w.Ask)
}
refuse := func(why string) error {
ended, ok, err := h.end(w.Ask, func(x *HandAct) { x.State, x.Said = HandRefused, why; warrantOn(x, w) })
if err == nil && ok {
h.say(HandRefused, ended)
}
return err
}
if o.ID != "yes" {
return refuse("the operator said no")
}
var j Judged
switch a.Act {
case ActHandOver:
j, err = h.JudgeHandOver(a.Subject)
case ActAccess:
j, err = h.JudgeAccess(ActTake, a.Subject)
default:
err = fmt.Errorf("the act %s is not performed on a warrant", a.Act)
}
if err != nil {
return refuse("judged again on the operator's yes, it is refused: " + err.Error())
}
now := *a
now.Sets, now.Path = j.Sets, j.Path
if err := o.Performs(h.bound(now)); err != nil {
return refuse("what the yes bound is not what would be done now: " + err.Error())
}
if j.Disk != a.Disk {
return refuse("the disk changed since the operator was asked (" + a.Disk + ", now " + j.Disk + "): ask again")
}
// Recorded with the way it came (ADR 0563 §5): the operator's yes to this engine's ask, not a terminal's line.
by := operatorOf(w) + ", to " + EngineSeat + "'s ask " + w.Ask
var said string
switch a.Act {
case ActHandOver:
said, err = h.HandOver(a.Subject, by)
case ActAccess:
said, err = h.AccessAct(AccessAct{Act: ActTake, Access: a.Subject, Path: j.Path, Warrant: w.Ask, By: by})
}
if err != nil {
return refuse(err.Error())
}
ended, ok, err := h.end(w.Ask, func(x *HandAct) { x.State, x.Said = HandDone, said; warrantOn(x, w) })
if err == nil && ok {
h.say(HandDone, ended)
}
return err
}
func warrantOn(a *HandAct, w asks.Warrant) {
a.Proofs = w.Proofs
if w.By != nil {
a.By, a.Via = w.By.Who, w.By.Kind
if w.By.Verified != "" {
a.Via += " (" + w.By.Verified + ")"
}
}
}
func operatorOf(w asks.Warrant) string {
if w.By == nil {
return "the operator"
}
return "the " + w.By.Who + " via " + w.By.Kind
}
// Heard is a warrant heard on the engine's decided subject: taken once (asker.Client.Take), and an ask that ended with
// no choice recorded refused.
func (h *HandActs) Heard(w asks.Warrant) {
if h.Asker == nil {
return
}
// A warrant the ask kept here does not hold to (another digest, another option, too late) ends nothing: it is no
// answer to the ask, which stays open for the operator's.
if handled, _ := h.Asker.Take(w, h.Warranted); handled == asker.Recorded {
why := "the ask ended " + string(w.Outcome)
if w.Words != "" {
why += ": " + w.Words
}
if ended, ok, _ := h.end(w.Ask, func(x *HandAct) { x.State, x.Said = HandRefused, why; warrantOn(x, w) }); ok {
h.say(HandRefused, ended)
}
}
}
// Reconcile catches up what the engine missed while it was away: a warrant the router holds for an open ask is taken,
// and an ask past its expiry and a margin is closed refused, expired.
func (h *HandActs) Reconcile() {
if h.Asker == nil {
return
}
acts, err := h.read()
if err != nil {
return
}
for _, a := range acts {
if a.State != HandAsked || a.Ask == "" {
continue
}
if _, w, err := h.Asker.Where(a.Ask); err == nil && w != nil {
h.Heard(*w)
continue
}
if !a.Expires.IsZero() && h.now().After(a.Expires.Add(askMargin)) {
if ended, ok, _ := h.end(a.Ask, func(x *HandAct) { x.State, x.Said = HandRefused, "the ask expired unanswered" }); ok {
h.say(HandRefused, ended)
}
}
}
}
// MemState is an ActStore in memory: an engine's acts while no bus is reached in a test.
type MemState struct {
mu sync.Mutex
keys map[string]stdio.StateEntry
top uint64
}
func (m *MemState) Get(key string) (*stdio.StateEntry, error) {
m.mu.Lock()
defer m.mu.Unlock()
if e, ok := m.keys[key]; ok {
return &e, nil
}
return nil, nil
}
func (m *MemState) Create(key string, value any) (uint64, error) { return m.put(key, value, 0, true) }
func (m *MemState) Update(key string, value any, revision uint64) (uint64, error) {
return m.put(key, value, revision, false)
}
func (m *MemState) put(key string, value any, revision uint64, create bool) (uint64, error) {
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
m.mu.Lock()
defer m.mu.Unlock()
was, kept := m.keys[key]
if (create && kept) || (!create && (!kept || was.Revision != revision)) {
return 0, stdio.ErrChanged
}
if m.keys == nil {
m.keys = map[string]stdio.StateEntry{}
}
m.top++
m.keys[key] = stdio.StateEntry{Key: key, Value: raw, Revision: m.top}
return m.top, nil
}
func (m *MemState) Delete(key string) error {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.keys, key)
return nil
}
func (m *MemState) Keys(prefix string) ([]string, error) {
m.mu.Lock()
defer m.mu.Unlock()
var out []string
for k := range m.keys {
if strings.HasPrefix(k, prefix) {
out = append(out, k)
}
}
sort.Strings(out)
return out, nil
}
+15 -5
View File
@@ -76,9 +76,13 @@ const MaxHandOverBy = 200
// HandOverCheck is what an ask is held to: the mesh's key, this node's name, the clock, and the nonces taken.
type HandOverCheck struct {
Signer ed25519.PublicKey
Node string
Now func() time.Time
Nonces *Nonces
// SignerNow, when set, is the controller's key as the engine trusts it at this ask, read per ask in place of Signer:
// nil once the controller's lease is handed away, which refuses every ask signed by its key (novox/hq ADR 0563 §7,
// ADR 0550 §2).
SignerNow func() ed25519.PublicKey
Node string
Now func() time.Time
Nonces *Nonces
}
// Nonces are the nonces of asks taken, each kept until its ask expires: an ask is taken once. Kept in memory —
@@ -157,8 +161,14 @@ func believedBytesWithin(check HandOverCheck, context string, withPath bool, bod
if err := json.Unmarshal(body, &signed); err != nil {
return HandOverAsk{}, nil, "the ask is not a signed ask: " + err.Error()
}
if len(check.Signer) != ed25519.PublicKeySize ||
!ed25519.Verify(check.Signer, append([]byte(context), signed.Ask...), signed.Signature) {
signer := check.Signer
if check.SignerNow != nil {
if signer = check.SignerNow(); signer == nil {
return HandOverAsk{}, nil, "the controller's lease is handed away, and no ask signed by its key is taken"
}
}
if len(signer) != ed25519.PublicKeySize ||
!ed25519.Verify(signer, append([]byte(context), signed.Ask...), signed.Signature) {
return HandOverAsk{}, nil, "the ask is not signed by the mesh"
}
var ask HandOverAsk
+117 -2
View File
@@ -9,6 +9,9 @@ import (
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"github.com/nats-io/nats.go"
)
// The hand-over's ask and answer hold these field names; the controller's side holds the same list
@@ -145,6 +148,21 @@ func TestAHandOverAskIsBelievedOnlySignedForThisNodeUnexpiredAndOnce(t *testing.
if strings.Join(given, "; ") != "/srv/notes by jo through mesh-cli on anchor; /refused by jo through mesh-cli on anchor" {
t.Fatalf("the handler was given %v", given)
}
// ADR 0563 §7, ADR 0550 §2: after the lease write the engine answers no ask signed by the controller's key. The key
// is read per ask from what the engine trusts now; nil (the lease handed away) refuses, and nothing is handed over.
trusted := m.Signer
following := HandOverCheck{Signer: m.Signer, SignerNow: func() ed25519.PublicKey { return trusted }, Node: m.Node,
Now: func() time.Time { return now }, Nonces: &Nonces{}}
before := len(given)
trusted = nil
late := signedAsk(t, key, with(func(a *HandOverAsk) { a.Expires = now.Add(time.Minute); a.Nonce = strings.Repeat("cd", 16) }))
if a := AnswerHandOver(following, handle, late); !strings.Contains(a.Refused, "lease is handed away") || len(given) != before {
t.Fatalf("after the lease write an ask signed by the controller was answered %+v", a)
}
trusted = m.Signer
if a := AnswerHandOver(following, handle, late); a.Refused != "" || len(given) != before+1 {
t.Fatalf("while the controller holds the lease its ask was answered %+v", a)
}
// A nonce is forgotten once its ask has expired, so the set does not grow for ever.
now = now.Add(HandOverGood + time.Second)
check.Nonces.take("ff", now.Add(time.Second), now)
@@ -174,17 +192,23 @@ func TestAHandOverIsAnsweredOnlyWhenItCarriesAReply(t *testing.T) {
// handingLink is a quiet link that hears hand-overs, keeping the handler it was given.
type handingLink struct {
*quietLink
given chan HandOverFunc
given chan HandOverFunc
signer chan func() ed25519.PublicKey
}
func (l handingLink) HearHandOvers(h HandOverFunc) { l.given <- h }
// FollowSigner keeps what the link is told to verify the controller's asks with.
func (l handingLink) FollowSigner(f func() ed25519.PublicKey) { l.signer <- f }
// **The link is given the queue's hand-over** when it is served: what the controller asks reaches the engine's
// judgement, and nothing else.
func TestALinkThatHearsHandOversIsGivenTheQueues(t *testing.T) {
m, _ := aMember(t)
l := handingLink{quietLink: newQuietLink(), given: make(chan HandOverFunc, 1)}
l := handingLink{quietLink: newQuietLink(), given: make(chan HandOverFunc, 1), signer: make(chan func() ed25519.PublicKey, 1)}
q := &Session{Membership: m, Timeout: time.Second}
// The link verifies the controller's asks with the key the engine trusts now (ADR 0563 §7).
q.ControllerKey = func() ed25519.PublicKey { return nil }
q.HandOver = func(path, by string) (string, error) { return "the queue's: " + path + " by " + by, nil }
ctx, stop := context.WithCancel(context.Background())
served := make(chan error, 1)
@@ -197,8 +221,40 @@ func TestALinkThatHearsHandOversIsGivenTheQueues(t *testing.T) {
case <-time.After(5 * time.Second):
t.Fatal("the link was never given the queue's hand-over")
}
select {
case f := <-l.signer:
if f() != nil {
t.Fatal("the link was given another key than the queue's")
}
case <-time.After(5 * time.Second):
t.Fatal("the link was never given the key the engine trusts now")
}
stop()
<-served
// 774 (ADR 0563 §1): a call of the node seat reaches its verb with its caller as the subject proves it, unproven on
// the subject that names none; a verb the seat does not serve is refused, and the answer is a tool's envelope.
t.Run("a call of the node seat reaches its verb with its proven caller", func(t *testing.T) {
h := &HandActs{Node: "n1", State: &MemState{},
RootFree: func() RootVerdict { return RootVerdict{Machine: "n1", Free: true, Why: "judged"} },
JudgeDirectories: func(by string) (string, error) { return "judged for " + by, nil }}
for _, row := range []struct{ verb, subject, want string }{
{"judge-directories", "mesh.seat.node-engine.call.judge-directories.n1.shanks~node-tools", `"said":"judged for shanks.node-tools"`},
{"judge-directories", EngineToolSubject("judge-directories", "n1"), `"said":"judged for unproven"`},
{"judge-directories", "mesh.seat.node-engine.call.judge-directories.n2.shanks~node-tools", `"said":"judged for unproven"`},
{"root-free", "mesh.seat.node-engine.call.root-free.n1.ace~messenger", `"free":true`},
{"unserved", "mesh.seat.node-engine.call.unserved.n1.x", `"error":"the node seat node-engine serves no verb \"unserved\""`},
} {
if got := string(AnswerEngineCall(h.Call, row.verb, "n1", row.subject, []byte(`{}`))); !strings.Contains(got, row.want) ||
!strings.Contains(got, `"node":"n1"`) {
t.Errorf("%s on %s: %s", row.verb, row.subject, got)
}
}
acts, _ := h.Acts(0)
if got := acts.(map[string]any)["acts"].([]HandAct); len(got) != 3 || got[0].Caller != "unproven" ||
got[2].Caller != "shanks.node-tools" || got[2].Act != ActJudgeDirectories {
t.Errorf("the judgements are recorded as %+v", got)
}
})
}
// **Against a real server**: a signed request on the node's own subject is answered; one on another node's is
@@ -237,4 +293,63 @@ func TestNatsAHandOverIsAskedOfTheNode(t *testing.T) {
if len(handled) != 1 {
t.Fatalf("handled %d asks; want only the one with a reply", len(handled))
}
// 774 (ADR 0563 §2, §3): the engine's acts are its node's keys of its declared state on the bus, each written by
// compare-and-set: a second create and a stale revision lose, and another node's keys are not this node's.
t.Run("the acts are kept in the bus's own storage under this node's keys", func(t *testing.T) {
_ = js.DeleteKeyValue(ActsBucket)
if _, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: ActsBucket}); err != nil {
t.Fatal(err)
}
q := &Session{Membership: m, Timeout: time.Second}
q.bus = l
mine, theirs := BusState{Session: q, Node: m.Node}, BusState{Session: q, Node: "another"}
rev, err := mine.Create("act.a-1", map[string]string{"state": "asked"})
if err != nil {
t.Fatal(err)
}
if _, err := mine.Create("act.a-1", map[string]string{}); !errors.Is(err, stdio.ErrChanged) {
t.Errorf("a second create: %v", err)
}
if _, err := mine.Update("act.a-1", map[string]string{"state": "done"}, rev+7); !errors.Is(err, stdio.ErrChanged) {
t.Errorf("an update at a stale revision: %v", err)
}
if _, err := mine.Update("act.a-1", map[string]string{"state": "done"}, rev); err != nil {
t.Errorf("an update at its revision: %v", err)
}
_, _ = theirs.Create("act.b-1", map[string]string{})
if keys, err := mine.Keys("act."); err != nil || len(keys) != 1 || keys[0] != "act.a-1" {
t.Errorf("this node's acts are %v, %v", keys, err)
}
if e, _ := js.KeyValue(ActsBucket); e != nil {
if got, err := e.Get(m.Node + ".act.a-1"); err != nil || !strings.Contains(string(got.Value()), "done") {
t.Errorf("the bucket holds %v, %v", got, err)
}
}
if err := mine.Delete("act.a-1"); err != nil {
t.Fatal(err)
}
if e, err := mine.Get("act.a-1"); err != nil || e != nil {
t.Errorf("a deleted act reads %v, %v", e, err)
}
})
// 774 (ADR 0563 §1): the node seat is served on this node's subjects alone, naming its caller, and said to discovery.
t.Run("the node seat is served on this node and discovered", func(t *testing.T) {
l.ServeEngineSeat(func(c EngineCall) (any, error) { return c.Verb + " for " + c.Caller, nil })
reply, err := conn.Request("mesh.seat.node-engine.call.acts."+m.Node+".shanks~node-tools", []byte(`{}`), 5*time.Second)
if err != nil || !strings.Contains(string(reply.Data), `"result":"acts for shanks.node-tools"`) {
t.Fatalf("the call answered %v, %v", reply, err)
}
if _, err := conn.Request(EngineToolSubject("acts", "another"), []byte(`{}`), 300*time.Millisecond); err == nil {
t.Fatal("another node's verb was answered here")
}
for seat, want := range map[string]bool{EngineSeat: true, "node-login-shell": false} {
verb := "acts"
if !want {
verb = "execute"
}
if heard, err := l.HeardServed(seat, verb); err != nil || heard != want {
t.Errorf("discovery hears %s.%s here: %v, %v", seat, verb, heard, err)
}
}
})
}
+281 -28
View File
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
@@ -37,12 +38,23 @@ type haveState struct {
mu sync.Mutex
modules map[string]ModuleFact
health map[string]delivery.Health
// words is, by module, what the last look found of the resource that made its health failed or waits: the have says
// it, so a failed health is never without words (issue 1496: "the node gave no words" hid a directory used as found).
words map[string]string
// looked says a look's health was taken: a running module it judges no resource of then has nothing that could be
// down, and is ok (issue 1945).
looked bool
node *delivery.HaveNode
// nodeComposed is the digest of the node's own composition it runs (ADR 0539 §9), kept beside the node's facts.
nodeComposed string
// nodeSealed is, on a node composing its own, each value its own composition carries sealed to its key (ADR 0547 §3,
// issue 1126): kept in its key so a restart reuses them and applies nothing again.
nodeSealed delivery.Seals
// nodeCandidates is the node's answer to each placement candidate proposal (novox/hq ADR 0559 §4), written into its key.
nodeCandidates map[string]delivery.CandidateAnswer
// nodeComposes and nodeComposesWhy are whose compositions the node applies and why (HaveNode.Composes), written into
// its key.
nodeComposes, nodeComposesWhy string
// written is the build of each module this node-engine wrote a have key for and that the declaration it applied
// still holds; removed is each such module the latest declaration no longer holds, with the build its key last
// said, until the key is written once with the phase removed (issue 1369, novox/hq ADR 0408 §4: never deleted).
@@ -51,6 +63,8 @@ type haveState struct {
// userList is what the bus node's engine last wrote of the bus's user list, by the module whose have says it (novox/hq
// ADR 0547 §8); kept apart from the module's fact, which every apply replaces.
userList map[string]delivery.UserListWritten
// userListNoKey is each module whose user list was said to have no have key to ride on, so it is said once.
userListNoKey map[string]bool
// unjudged is each module whose composition moved (an apply began, or the fact names another build) and that no look
// has judged since it stopped moving: its health is not yet judged, never the last look's of what ran before.
unjudged map[string]bool
@@ -106,10 +120,11 @@ func (s *haveState) owed() bool {
// takeLook keeps a look's health: of a module still in motion nothing, since the look judged what is being replaced;
// of every other module what the look found, which is then judged. The caller holds the lock.
func (s *haveState) takeLook(h *Health) {
next := moduleHealths(h)
next, words := moduleHealths(h)
for module, f := range s.modules {
if inMotion(f.Phase) {
delete(next, module)
delete(words, module)
if s.unjudged == nil {
s.unjudged = map[string]bool{}
}
@@ -118,19 +133,32 @@ func (s *haveState) takeLook(h *Health) {
}
delete(s.unjudged, module)
}
s.health, s.looked = next, true
s.health, s.words, s.looked = next, words, true
}
// wordsOf is what a have says of its health when the apply said nothing: the words of the resource that made it failed
// or waits, while it is so. The caller holds the lock.
func (s *haveState) wordsOf(module string, health delivery.Health) string {
if health != delivery.HealthFailed && health != delivery.HealthWaits {
return ""
}
return s.words[module]
}
// moduleHealths is the health of each module a look names, from the states of its resources: failed when any is
// unhealthy, waits when any waits (waiting, starting, held), ok when all are healthy, unknown otherwise.
func moduleHealths(h *Health) map[string]delivery.Health {
out := map[string]delivery.Health{}
// unhealthy, waits when any waits (waiting, starting, held), ok when all are healthy, unknown otherwise; and, for a
// module failed or waiting, the words of the first resource that made it so. A directory used as found is not the
// module's health: it says nothing of how the module runs, waits for the operator's hand-over and keeps its own
// condition, raised from the same statement (issue 1496: on ace mesh-build-node's cache directory, there before the
// mesh, made its have failed while its check answered healthy, and the assigner held its version on every node).
func moduleHealths(h *Health) (map[string]delivery.Health, map[string]string) {
out, words := map[string]delivery.Health{}, map[string]string{}
if h == nil {
return out
return out, words
}
rank := map[string]int{}
for _, r := range h.Resources {
if r.Module == "" {
if r.Module == "" || usedAsFound(r) {
continue
}
var this int
@@ -146,12 +174,32 @@ func moduleHealths(h *Health) map[string]delivery.Health {
}
if was, seen := rank[r.Module]; !seen || this > was {
rank[r.Module] = this
if this >= 2 {
words[r.Module] = resourceWords(r)
}
}
}
for module, r := range rank {
out[module] = [...]delivery.Health{delivery.HealthOK, delivery.HealthUnknown, delivery.HealthWaits, delivery.HealthFailed}[r]
if r < 2 {
delete(words, module)
}
}
return out
return out, words
}
// usedAsFound says a resource of a statement is a directory used as found (issue 339).
func usedAsFound(r ResourceHealth) bool {
return r.Kind == KindDirectory && strings.HasPrefix(r.Reason, ReasonUsedAsFound)
}
// resourceWords is one resource's state in the words a have carries: its id, its state and why.
func resourceWords(r ResourceHealth) string {
w := r.Resource + " " + r.State
if why := strings.TrimSpace(r.Reason); why != "" {
w += ": " + why
}
return w
}
// nodeFacts is the node's own facts of a report, each as the JSON the report carries it in.
@@ -187,8 +235,7 @@ func (q *Session) noteApply(r Report) bool {
if r.Health != nil {
s.takeLook(r.Health)
}
n := nodeFacts(r)
s.node = &n
s.node = q.nodeFactsWithRecord(r, s.node)
return true
}
s.mu.Lock()
@@ -240,24 +287,45 @@ func (q *Session) noteApply(r Report) bool {
if s.owed() {
q.oweHealth()
}
n := nodeFacts(r)
s.node = &n
s.node = q.nodeFactsWithRecord(r, s.node)
return true
}
// nodeFactsWithRecord is the node's own facts of a report with its record (ADR 0555 §3): a report that says nothing of
// the machine leaves the record last made standing, and one without a health statement keeps the contract last stated.
func (q *Session) nodeFactsWithRecord(r Report, before *delivery.HaveNode) *delivery.HaveNode {
n := nodeFacts(r)
if q.Keys != nil {
n.Record = recordOf(q.Membership.Node, r, q.Keys())
}
if before != nil && before.Record != nil {
switch {
case n.Record == nil:
n.Record = before.Record
case n.Record.Contract == 0:
n.Record.Contract = before.Record.Contract
}
}
return &n
}
// noteHealth takes a look's health, and says whether any module's changed.
func (q *Session) noteHealth(h Health) bool {
s := &q.haves
s.mu.Lock()
defer s.mu.Unlock()
before := map[string]delivery.Health{}
said := func(module string, f ModuleFact) string {
health := s.healthOf(module, f)
return string(health) + "\x00" + s.wordsOf(module, health)
}
before := map[string]string{}
for module, f := range s.modules {
before[module] = s.healthOf(module, f)
before[module] = said(module, f)
}
s.takeLook(&h)
changed := false
for module, f := range s.modules {
if s.healthOf(module, f) != before[module] {
if said(module, f) != before[module] {
changed = true
}
}
@@ -283,7 +351,7 @@ func (q *Session) writeHaves(ctx context.Context, bus Bus) {
s := &q.haves
s.mu.Lock()
defer s.mu.Unlock()
if s.modules == nil && s.node == nil && len(s.removed) == 0 {
if s.modules == nil && s.node == nil && len(s.removed) == 0 && len(s.userList) == 0 {
return
}
node := q.Membership.Node
@@ -306,13 +374,20 @@ func (q *Session) writeHaves(ctx context.Context, bus Bus) {
f := s.modules[module]
h := delivery.Have{V: delivery.Version, FP: f.FP, Composed: f.Composed, Build: f.Build,
Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown, Detail: f.Detail}
if delivery.Phase(f.Phase) == delivery.PhaseWaitingYes {
h.Judgement, h.Waits = delivery.Judgement(f.Judgement), f.Waits
}
switch {
case f.Phase != "":
h.Phase = delivery.Phase(f.Phase)
case f.Failed:
h.Phase = delivery.PhaseFailed
}
h.SealedAt = f.Sealed
h.Health = s.healthOf(module, f)
if h.Detail == "" {
h.Detail = s.wordsOf(module, h.Health)
}
if w, ok := s.userList[module]; ok {
h.UserList = &w
}
@@ -340,12 +415,18 @@ func (q *Session) writeHaves(ctx context.Context, bus Bus) {
if !q.writeRemoved(write, bucket, node) {
return
}
if !q.writeUserListsAlone(write, bucket, node) {
return
}
if s.node != nil {
key, err := delivery.HaveNodeKey(node)
if err == nil {
var raw []byte
facts := *s.node
facts.Composed = s.nodeComposed
facts.SealedAt = s.nodeSealed
facts.Candidates = s.nodeCandidates
facts.Composes, facts.ComposesWhy = s.nodeComposes, s.nodeComposesWhy
if raw, err = delivery.EncodeHaveNode(facts); err == nil {
want := facts
if !q.putHave(write, bucket, key, raw, func(current []byte) bool {
@@ -455,6 +536,78 @@ func (q *Session) writeRemoved(ctx context.Context, b HaveBucket, node string) b
return true
}
// writeUserListsAlone writes the user list of each module this node-engine holds no fact of (no apply here said its have)
// into that module's key as the bucket holds it, keeping every other field the key says (novox/hq ADR 0547 §8). The have
// of a module is one value with parts from more than one source: the apply's (fingerprint, digest, build, phase, health)
// and the user-list follower's. Written only on the pass over the modules this engine applied, the follower's part was
// lost whenever the bus module's apply was not this engine's to say (2026-10-26 on the bus node: the bus module had no
// want and had left the declaration, so its key kept an older engine's value and the vault read no hashes in it). A key
// that does not exist is not created: a have says a phase, and nothing here knows it; that is said once. False when the
// pass should stop. The caller holds the lock of the haves.
func (q *Session) writeUserListsAlone(ctx context.Context, b HaveBucket, node string) bool {
s := &q.haves
alone := make([]string, 0, len(s.userList))
for module := range s.userList {
_, held := s.modules[module]
_, gone := s.removed[module]
if !held && !gone {
alone = append(alone, module)
}
}
sort.Strings(alone)
for _, module := range alone {
key, err := delivery.HaveKey(node, module)
if err != nil {
q.sayHave(fmt.Sprintf("module %q has no key in the have bucket: %v", module, err))
continue
}
if ctx.Err() != nil {
q.sayHave(fmt.Sprintf("the time for writing the have bucket ran out before %s; the next apply or look writes it", key))
return false
}
current, revision, err := b.Get(ctx, key)
switch {
case errors.Is(err, ErrHaveAbsent):
if !s.userListNoKey[module] {
if s.userListNoKey == nil {
s.userListNoKey = map[string]bool{}
}
s.userListNoKey[module] = true
q.sayHave(fmt.Sprintf("the bus's user list written for %s is in no have: %s does not exist and no apply on "+
"this node says %s's phase, so it is written once one does", module, key, module))
}
continue
case err != nil:
q.sayHave(fmt.Sprintf("could not read %s in the have bucket, and wrote nothing more there: %v", key, err))
q.haveFailed(err)
return false
}
delete(s.userListNoKey, module)
h, err := delivery.DecodeHave(current)
if err != nil {
q.sayHave(fmt.Sprintf("the bus's user list is not written in %s: its value cannot be read: %v", key, err))
continue
}
w := s.userList[module]
next := h
next.UserList = &w
if delivery.SameHave(h, next) {
continue
}
raw, err := delivery.EncodeHave(next)
if err != nil {
q.sayHave(fmt.Sprintf("cannot encode %s: %v", key, err))
continue
}
if err := b.Set(ctx, key, raw, revision); err != nil {
q.sayHave(fmt.Sprintf("could not write %s in the have bucket; the next apply or look writes it again: %v", key, err))
q.haveFailed(err)
return false
}
}
return true
}
// putHave reads a key and writes it unless same says it already holds the value. False when the pass should stop: the
// bucket failed, the error is said once, and the rest wait for the next pass.
func (q *Session) putHave(ctx context.Context, b HaveBucket, key string, value []byte, same func([]byte) bool) bool {
@@ -561,12 +714,14 @@ func (q *Session) NoteRemoved(ctx context.Context, module, build string) {
q.writeHaves(ctx, bus)
}
// NoteNode takes the digest of the node's own composition it runs, and writes the node's key where it differs.
func (q *Session) NoteNode(ctx context.Context, composed string) {
// NoteNode takes the digest of the node's own composition it runs and the values it carries sealed to the node's key
// (nil for a composition the node did not make itself), and writes the node's key where it differs.
func (q *Session) NoteNode(ctx context.Context, composed string, sealed delivery.Seals) {
q.init()
s := &q.haves
s.mu.Lock()
s.nodeComposed = composed
s.nodeSealed = sealed
if s.node == nil {
n := delivery.HaveNode{V: delivery.Version}
s.node = &n
@@ -578,36 +733,134 @@ func (q *Session) NoteNode(ctx context.Context, composed string) {
q.writeHaves(ctx, bus)
}
// ReadHave is the have this node-engine last wrote for a module, and whether the bucket holds one. A bucket that cannot
// be read answers as none: the agent then compares with the declaration it kept, and applies only what differs.
func (q *Session) ReadHave(ctx context.Context, module string) (delivery.Have, bool) {
// NoteCandidates takes the node's answers to the placement candidates, by proposal (its shadow composer's, novox/hq ADR
// 0559 §4), and writes the node's key where it differs; a proposal left out is no longer answered.
func (q *Session) NoteCandidates(ctx context.Context, answers map[string]delivery.CandidateAnswer) {
q.init()
s := &q.haves
s.mu.Lock()
s.nodeCandidates = answers
if len(answers) == 0 {
s.nodeCandidates = nil
}
// Before the node's facts are first said, its key is not written for an answer alone: it would carry no record. The
// answers ride with the first facts.
known := s.node != nil
s.mu.Unlock()
if !known {
return
}
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
// NoteComposes says, in the node's own key, whose compositions the node applies (delivery.ComposesOwn or
// delivery.ComposesController) and why: written at each start of its agent, so a node returning to the controller's
// compositions says so where every have is read (issue 1126).
func (q *Session) NoteComposes(ctx context.Context, composes, why string) {
q.init()
s := &q.haves
s.mu.Lock()
changed := s.nodeComposes != composes || s.nodeComposesWhy != why
s.nodeComposes, s.nodeComposesWhy = composes, why
known := s.node != nil
s.mu.Unlock()
if !known || !changed {
return // said with the node's first facts
}
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
// ReadNodeHave is the node's own key as the bucket holds it, and whether it holds one: what the node's own composition
// it last ran is and carries sealed, read after a restart (ADR 0547 §3).
func (q *Session) ReadNodeHave(ctx context.Context) (delivery.HaveNode, bool) {
n, ok, _ := q.ReadNodeHaveErr(ctx)
return n, ok
}
// ReadNodeHaveErr is ReadNodeHave telling the key absent (false, nil) from one that could not be read (false, the
// error): a bus that did not answer is never kept as a node that ran nothing (issue 1126, 2026-10-28).
func (q *Session) ReadNodeHaveErr(ctx context.Context) (delivery.HaveNode, bool, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
hb, ok := bus.(HaveBus)
if !ok || bus == nil {
return delivery.Have{}, false
return delivery.HaveNode{}, false, ErrNotLinkedHaves
}
key, err := delivery.HaveKey(q.Membership.Node, module)
key, err := delivery.HaveNodeKey(q.Membership.Node)
if err != nil {
return delivery.Have{}, false
return delivery.HaveNode{}, false, err
}
read, cancel := context.WithTimeout(ctx, q.timeout())
defer cancel()
bucket, err := hb.HaveBind(read)
if err != nil {
return delivery.Have{}, false
return delivery.HaveNode{}, false, err
}
value, _, err := bucket.Get(read, key)
if errors.Is(err, ErrHaveAbsent) {
return delivery.HaveNode{}, false, nil
}
if err != nil {
return delivery.Have{}, false
return delivery.HaveNode{}, false, err
}
n, err := delivery.DecodeHaveNode(value)
if err != nil {
return delivery.HaveNode{}, false, nil // a key no engine can read names nothing it runs
}
return n, true, nil
}
// ReadHave is the have this node-engine last wrote for a module, and whether the bucket holds one. A bucket that cannot
// be read answers as none: the agent then compares with the declaration it kept, and applies only what differs.
func (q *Session) ReadHave(ctx context.Context, module string) (delivery.Have, bool) {
h, ok, _ := q.ReadHaveErr(ctx, module)
return h, ok
}
// ErrNotLinkedHaves is a have read with no link to the bus open that can bind the have bucket.
var ErrNotLinkedHaves = errors.New("no link to the bus that binds the have bucket is open")
// ReadHaveErr is ReadHave telling a key absent (false, nil) from one that could not be read (false, the error): a bus
// that does not answer for a moment is not a module that never ran (issue 1126).
func (q *Session) ReadHaveErr(ctx context.Context, module string) (delivery.Have, bool, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
hb, ok := bus.(HaveBus)
if !ok || bus == nil {
return delivery.Have{}, false, ErrNotLinkedHaves
}
key, err := delivery.HaveKey(q.Membership.Node, module)
if err != nil {
return delivery.Have{}, false, err
}
read, cancel := context.WithTimeout(ctx, q.timeout())
defer cancel()
bucket, err := hb.HaveBind(read)
if err != nil {
return delivery.Have{}, false, err
}
value, _, err := bucket.Get(read, key)
if errors.Is(err, ErrHaveAbsent) {
return delivery.Have{}, false, nil
}
if err != nil {
return delivery.Have{}, false, err
}
h, err := delivery.DecodeHave(value)
if err != nil {
return delivery.Have{}, false
return delivery.Have{}, false, nil // a have no engine can read names nothing it runs
}
return h, true
return h, true, nil
}
// nothingJudged is the health of a module a look judged no resource of: ok when it was looked at and the module applied
+135 -9
View File
@@ -10,6 +10,7 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
)
@@ -198,25 +199,35 @@ func TestAModulesHealthFollowsTheLook(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
rep := Report{Modules: []ModuleFact{{Module: "a", Build: "b"}, {Module: "b", Build: "b"},
{Module: "c", Build: "b"}, {Module: "d", Build: "b"}}}
{Module: "c", Build: "b"}, {Module: "d", Build: "b"}, {Module: "e", Build: "b"}}}
q.SayFacts(context.Background(), rep)
for _, m := range []string{"a", "b", "c", "d"} {
for _, m := range []string{"a", "b", "c", "d", "e"} {
if h := l.have(t, "n1."+m).Health; h != delivery.HealthUnknown {
t.Errorf("%s with no look is %s, want unknown", m, h)
}
}
h := healthOf(map[string]string{"a": StateHealthy, "b": StateUnhealthy, "c": StateStarting})
h.Resources[slicesIndex(h.Resources, "b.server")].Reason = ReasonRestarting
// 1496: a directory used as found is not the module's health (e runs healthy beside one); a failed or waiting module's
// have says the words of the resource that made it so.
h.Resources = append(h.Resources, ResourceHealth{Module: "a", Resource: "a.two", State: StateWaiting},
ResourceHealth{Module: "b", Resource: "b.two", State: StateHealthy}, ResourceHealth{Module: "d", Resource: "d.x", State: StateUnknown})
ResourceHealth{Module: "b", Resource: "b.two", State: StateHealthy}, ResourceHealth{Module: "d", Resource: "d.x", State: StateUnknown},
ResourceHealth{Module: "e", Resource: "e.server", State: StateHealthy},
ResourceHealth{Module: "e", Resource: "e.cache", Kind: KindDirectory, State: StateUnhealthy, Reason: ReasonUsedAsFound + " owned by 1:1"})
if !q.SayHealth(context.Background(), *h) {
t.Fatal("not said")
}
for m, want := range map[string]delivery.Health{"a": delivery.HealthWaits, "b": delivery.HealthFailed,
"c": delivery.HealthWaits, "d": delivery.HealthUnknown} {
"c": delivery.HealthWaits, "d": delivery.HealthUnknown, "e": delivery.HealthOK} {
if got := l.have(t, "n1."+m).Health; got != want {
t.Errorf("%s is %s, want %s", m, got, want)
}
}
for m, want := range map[string]string{"a": "a.two waiting", "b": "b.server unhealthy: restarting", "d": "", "e": ""} {
if got := l.have(t, "n1."+m).Detail; got != want {
t.Errorf("%s says %q, want %q", m, got, want)
}
}
before := l.setCount()
h2 := *h
q.SayHealth(context.Background(), h2)
@@ -225,8 +236,8 @@ func TestAModulesHealthFollowsTheLook(t *testing.T) {
}
h2.Resources = []ResourceHealth{{Module: "a", Resource: "a.server", State: StateHealthy}}
q.SayHealth(context.Background(), h2)
if got := l.have(t, "n1.a").Health; got != delivery.HealthOK {
t.Errorf("a after a healthy look is %s", got)
if got := l.have(t, "n1.a"); got.Health != delivery.HealthOK || got.Detail != "" {
t.Errorf("a after a healthy look is %s, saying %q", got.Health, got.Detail)
}
}
@@ -259,6 +270,44 @@ func TestTheReportIsSentFirstAndTheNodeKeyCarriesTheNodesFacts(t *testing.T) {
if !delivery.SameHaveNode(got, want) {
t.Errorf("the node key holds %+v, want %+v", got, want)
}
// 1126 (ADR 0555 §3): with its keys, the node key carries the node's record: the profile's present capabilities and
// networks, the outward links, the health's contract and the public keys; a report that changes none of it writes
// nothing, a changed fact writes the key once, and a report without a profile or health leaves them standing.
keys := RecordKeys{TunnelKey: "tk", SealingKey: "sk", ServingKey: "pk", Certificate: "cert",
Trusts: []string{"controller key sha256:aa", "vault key sha256:bb"}} // and what it trusts (L3 s6, ADR 0550 §2)
q.Keys = func() RecordKeys { return keys }
rep.Profile = map[string]any{"capabilities": []any{map[string]any{"name": "systemd", "present": true},
map[string]any{"name": "docker", "present": false}, map[string]any{"name": "apt", "present": true}},
"machine": map[string]any{"networks": []any{map[string]any{"link": "eth0", "network": "192.0.2.0/24"}}}}
rep.Outward = []string{"eth0"}
rep.Health = &Health{Contract: NodeKeyContract}
q.SayFacts(context.Background(), rep)
if got, err = delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || got.Record == nil {
t.Fatalf("the node key holds no record: %+v, %v", got, err)
}
wantRecord := delivery.NodeRecord{V: delivery.Version, Node: "n1", TunnelKey: "tk", SealingKey: "sk", ServingKey: "pk",
Certificate: "cert", Trusts: keys.Trusts, Capabilities: []string{"apt", "systemd"}, OutwardLinks: []string{"eth0"}, NetworksSaid: true,
Networks: []catalogue.AttachedNetwork{{Link: "eth0", Network: "192.0.2.0/24"}}, Contract: NodeKeyContract}
if !delivery.SameNodeRecord(*got.Record, wantRecord) {
t.Errorf("record = %+v, want %+v", *got.Record, wantRecord)
}
at := l.kv["n1.node"].rev
q.SayFacts(context.Background(), rep)
if l.kv["n1.node"].rev != at {
t.Errorf("a report changing nothing wrote the node key again")
}
keys.Certificate = "cert-2"
q.SayFacts(context.Background(), rep)
if got, _ = delivery.DecodeHaveNode(l.kv["n1.node"].value); l.kv["n1.node"].rev != at+1 || got.Record.Certificate != "cert-2" {
t.Errorf("a new certificate: revision %d (was %d), record %+v", l.kv["n1.node"].rev, at, got.Record)
}
bare := rep
bare.Health = nil
q.SayFacts(context.Background(), bare)
if got, _ = delivery.DecodeHaveNode(l.kv["n1.node"].value); got.Record.Contract != NodeKeyContract {
t.Errorf("a report without a health statement dropped the contract: %+v", got.Record)
}
}
// 1132/AC7: a write the bucket refuses is said in the log; the report went out, and the next apply writes again.
@@ -534,9 +583,22 @@ func TestAnAppliedModulesHaveNamesTheFingerprintAndTheDigestAndFollowsItsHealth(
if got := l.have(t, "n1.web"); got.Phase != delivery.PhaseRemoved {
t.Fatalf("removed: %+v", got)
}
q.NoteNode(context.Background(), dig)
if n, err := delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || n.Composed != dig {
t.Fatalf("the node key: %+v %v", n, err)
q.NoteNode(context.Background(), dig, delivery.Seals{"core-box#sealed": "core"})
if n, err := delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || n.Composed != dig || len(n.SealedAt) != 1 || n.SealedAt["core-box#sealed"] != "core" {
t.Fatalf("the node key, with its own composition's sealed values by slot: %+v %v", n, err)
}
if n, ok := q.ReadNodeHave(context.Background()); !ok || n.Composed != dig || len(n.SealedAt) != 1 {
t.Fatalf("the node key read back: %+v %v", n, ok)
}
// The node's answers to placement candidates ride in the same key (ADR 0559 §4), and go when no group is answered.
answers := map[string]delivery.CandidateAnswer{"p-1": {Proposal: "p-1", Revision: 1, Verdict: delivery.CandidateSame, Same: []string{"web"}}}
q.NoteCandidates(context.Background(), answers)
if n, err := delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || n.Candidates["p-1"].Proposal != "p-1" || n.Composed != dig {
t.Fatalf("the node key with its answers: %+v %v", n, err)
}
q.NoteCandidates(context.Background(), nil)
if n, _ := delivery.DecodeHaveNode(l.kv["n1.node"].value); n.Candidates != nil {
t.Fatalf("an answer left after its group went: %+v", n.Candidates)
}
}
@@ -660,3 +722,67 @@ func TestADeclarationReportMovesAHealthAndARemovalForgetsIt(t *testing.T) {
t.Fatal("a removed module is still marked unjudged")
}
}
// ADR 0547 §8, 2026-10-26 on the bus node: the bus's user list of a module this engine says no have of (no want, and no
// longer in the declaration) still reaches the module's key, merged into what the key says, every other field kept; a
// key that does not exist is not made up and is said once; a writer that drops the list again is written over.
func TestAUserListReachesTheHaveOfAModuleNoApplyHereSays(t *testing.T) {
l := newHaveLink()
older := delivery.Have{V: delivery.Version, Build: "build-1", Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
l.put(t, "n1.nats", older)
q, said := aHaveQueue(l)
dig := "sha256:" + strings.Repeat("c", 64)
w := delivery.UserListWritten{Source: "n1.nats", Revision: 7, Wanted: dig, Digest: dig, Users: 58,
At: time.Date(2026, 10, 26, 12, 26, 51, 0, time.UTC)}
q.NoteUserList(context.Background(), "nats", w)
got := l.have(t, "n1.nats")
if got.UserList == nil || got.UserList.Users != 58 || got.UserList.Digest != dig {
t.Fatalf("the user list is not in the have: %+v", got)
}
if got.Build != "build-1" || got.Phase != delivery.PhaseRunning || got.Health != delivery.HealthOK {
t.Fatalf("the merge lost the key's own fields: %+v", got)
}
rev := l.kv["n1.nats"].rev
q.NoteUserList(context.Background(), "nats", w)
if l.kv["n1.nats"].rev != rev {
t.Errorf("the same list was written again, to revision %d", l.kv["n1.nats"].rev)
}
// Another writer replaces the value without the list: the next pass merges it back.
l.put(t, "n1.nats", older)
q.SayFacts(context.Background(), Report{})
if got := l.have(t, "n1.nats"); got.UserList == nil || got.Build != "build-1" {
t.Fatalf("a value written without the list kept it out: %+v", got)
}
// No key at all: nothing is created, said once; the module's first have carries the list.
q.NoteUserList(context.Background(), "other", w)
q.NoteUserList(context.Background(), "other", w)
if _, ok := l.kv["n1.other"]; ok {
t.Fatalf("a have was made up for a module no apply says")
}
n := 0
for _, s := range *said {
if strings.Contains(s, "is in no have") {
n++
}
}
if n != 1 {
t.Errorf("the missing key was said %d times: %q", n, *said)
}
fp := "src2:" + strings.Repeat("a", 64)
q.NoteModule(context.Background(), ModuleFact{Module: "other", FP: fp, Composed: dig})
if got := l.have(t, "n1.other"); got.UserList == nil || got.FP != fp {
t.Fatalf("the first have of the module lost the list: %+v", got)
}
}
// slicesIndex is the index of the resource with id in rs, or -1.
func slicesIndex(rs []ResourceHealth, id string) int {
for i, r := range rs {
if r.Resource == id {
return i
}
}
return -1
}
+98
View File
@@ -3,9 +3,12 @@ package link
import (
"context"
"encoding/json"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A health statement is said on the link open now, as the machine's own word on its own subject, and
@@ -50,6 +53,101 @@ func TestAHealthStatementIsSaidOnTheLinkOpenNow(t *testing.T) {
if HealthSubject("anchor") != "mesh.control.anchor.health" {
t.Fatalf("the subject %s is not inside the host's own grant", HealthSubject("anchor"))
}
// Issue 774: the node's own health report in its key of mesh-host_health, for the watchman's six alerts: written on
// a change only, the streaks and the time of the look not being one, and read first after a restart.
t.Run("the node's health report is written on a change only", func(t *testing.T) {
since := time.Date(2026, 10, 29, 20, 0, 0, 0, time.UTC)
look := func(streak int, writer string) Health {
return Health{Contract: 7, At: time.Now(), Network: &NetworkHealth{State: StateUnhealthy, Since: since, Parts: []NetworkPart{
{Part: "resolv-conf", State: StateUnhealthy, Writer: writer, Owner: "dnsmasq", Since: since, Streak: streak}}},
Units: &UnitsHealth{State: "degraded", Failed: []FailedUnit{{Unit: "cups.service", Scope: "system", Since: since}}},
Resources: []ResourceHealth{
{Module: "brother", Resource: "brother.scanner", Kind: KindAccount, Account: "jochen", State: StateUnhealthy,
Reason: "relogin needed: the session began before scanner", Since: since},
{Module: "photos", Resource: "photos.library", Kind: KindDirectory, Target: "/srv/photos", State: StateUnhealthy,
Reason: ReasonUsedAsFound + " owned by root", Since: since},
{Module: "letta", Resource: "letta.server", Kind: "container", State: StateHealthy, Since: since},
{Module: "claude-code", Resource: "claude-code.agent", Kind: KindAccount, Account: "agent", Root: "never",
State: StateHealthy, Since: since}}}
}
peers := []PeerSeen{{Key: "B=", Seen: true}, {Key: "A=", Addresses: []string{"10.10.0.2/32"}}}
r := NodeHealthOf("anchor", look(1, "NetworkManager"), peers, Machines{})
if r.Network.Parts[0].Writer != "NetworkManager" || r.Network.Parts[0].Streak != 0 || r.Peers[0].Key != "A=" ||
len(r.Units.Failed) != 1 || len(r.Relogin) != 1 || r.Relogin[0].Account != "jochen" ||
len(r.Found) != 1 || r.Found[0].Path != "/srv/photos" ||
len(r.Root) != 1 || r.Root[0].Account != "agent" || r.Root[0].State != StateHealthy || r.Contract != 7 {
t.Fatalf("the report is %+v", r)
}
// The machines an unhealthy part points at, as the controller holds it there; none for the machine's own.
machines := Machines{ByAddress: map[string]string{"10.10.0.1": "novox", "10.10.0.4": "ace", "10.10.0.9": "anchor"}, Hub: "novox"}
for _, tc := range []struct {
name string
toward []string
state string
want []string
}{
{"the tunnel to the hub", []string{"hub"}, StateUnhealthy, []string{"novox"}},
{"two mesh resolvers", []string{"10.10.0.1", "10.10.0.4"}, StateUnhealthy, []string{"novox", "ace"}},
{"a resolver a VPN client wrote in is the machine's own", []string{"10.10.0.1", "192.0.2.53"}, StateUnhealthy, nil},
{"its own resolver is its own", []string{"10.10.0.9"}, StateUnhealthy, nil},
{"a healthy part points at nothing", []string{"hub"}, StateHealthy, nil},
} {
h := Health{Network: &NetworkHealth{State: tc.state, Parts: []NetworkPart{{Part: "names", State: tc.state, Toward: tc.toward}}}}
got := NodeHealthOf("anchor", h, nil, machines).Network.Parts[0].Targets
if !slices.Equal(got, tc.want) {
t.Errorf("%s: targets %v, want %v", tc.name, got, tc.want)
}
}
store := &memHealth{}
var last *NodeHealth
clock := func() time.Time { return since }
for i, step := range []struct {
h NodeHealth
fresh bool // a restart: nothing written in this run yet
wrote bool
}{
{h: r, wrote: true},
{h: NodeHealthOf("anchor", look(2, "NetworkManager"), peers, Machines{})},
{h: NodeHealthOf("anchor", look(3, "NetworkManager"), peers, Machines{}), fresh: true},
{h: NodeHealthOf("anchor", look(3, "openvpn"), peers, Machines{}), wrote: true},
{h: NodeHealthOf("anchor", look(3, "openvpn"), []PeerSeen{{Key: "A=", Seen: true}}, Machines{}), wrote: true},
} {
if step.fresh {
last = nil
}
wrote, err := writeNodeHealth(t.Context(), store, &last, step.h, clock)
if err != nil || wrote != step.wrote {
t.Fatalf("step %d: wrote %v (want %v), %v", i, wrote, step.wrote, err)
}
}
got, err := delivery.DecodeNodeHealth(store.values["anchor"])
if err != nil || got.Node != "anchor" || got.Network.Parts[0].Writer != "openvpn" || !got.Peers[0].Seen || got.At.IsZero() {
t.Fatalf("the key holds %+v, %v", got, err)
}
// The key's bytes are mesh-sdk's layout: decoded and encoded again by it, the same bytes.
if again, err := delivery.EncodeNodeHealth(got); err != nil || string(again) != string(store.values["anchor"]) {
t.Fatalf("the key's bytes are not the sdk's layout: %s\n%s (%v)", store.values["anchor"], again, err)
}
if store.puts != 3 {
t.Fatalf("%d writes, want 3: one per change", store.puts)
}
})
}
// memHealth is a HealthStore in memory.
type memHealth struct {
values map[string][]byte
puts int
}
func (m *memHealth) Get(_ context.Context, key string) ([]byte, error) { return m.values[key], nil }
func (m *memHealth) Put(_ context.Context, key string, value []byte) error {
if m.values == nil {
m.values = map[string][]byte{}
}
m.values[key], m.puts = value, m.puts+1
return nil
}
// A declared tool check reads the node tools' reply envelope: healthy only when the tool says so (novox/hq
+48 -17
View File
@@ -6,9 +6,11 @@ import (
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/micro"
)
// The host's link on the bus: one connection, and what is asked of this node over it. The node hears no declaration
@@ -35,11 +37,14 @@ func natsURL(address string) string {
// natsLink is this node's connection as a JetStream subscription.
type natsLink struct {
conn *nats.Conn
js nats.JetStreamContext
asking *nats.Subscription
asked chan struct{}
handing *nats.Subscription
// mu guards signerNow (FollowSigner).
mu sync.Mutex
signerNow func() ed25519.PublicKey
conn *nats.Conn
js nats.JetStreamContext
asking *nats.Subscription
asked chan struct{}
handing *nats.Subscription
// searching hears the controller's terminal asking for a fresh setuid search (novox/hq issue 361).
searching *nats.Subscription
// judging hears the controller asking for a read-only judgement of this node's directories (issue 358).
@@ -48,7 +53,10 @@ type natsLink struct {
acting *nats.Subscription
// moving hears the controller asking to move a value sealed to this machine into the vault (issue 676).
moving *nats.Subscription
node string
// engine serves the node seat node-engine, and warrants hears the operator's answers to its asks (ADR 0563).
engine micro.Service
warrants *nats.Subscription
node string
// signer is the mesh's key, which a hand-over ask is verified with as a declaration is (issue 356).
signer ed25519.PublicKey
lost chan error
@@ -141,6 +149,21 @@ func (l *natsLink) hearAsks() {
}
}
// FollowSigner has every signed ask heard on this link verified with the key the engine trusts at that ask, the
// controller's while it holds its lease and none from its hand-over (novox/hq ADR 0563 §7, ADR 0550 §2).
func (l *natsLink) FollowSigner(f func() ed25519.PublicKey) {
l.mu.Lock()
l.signerNow = f
l.mu.Unlock()
}
// check is how a signed ask is believed on this link: the static membership signer until FollowSigner names the trust.
func (l *natsLink) check() HandOverCheck {
l.mu.Lock()
defer l.mu.Unlock()
return HandOverCheck{Signer: l.signer, SignerNow: l.signerNow, Node: l.node}
}
// HearHandOvers listens for the controller asking this node to hand a directory used as found to the mesh
// (novox/hq issue 356): a request on this machine's own subject, answered once on the reply it carried. The
// ask is verified with the mesh's key before anything is read out of it (AnswerHandOver), because a responder
@@ -151,7 +174,7 @@ func (l *natsLink) HearHandOvers(handle HandOverFunc) {
if handle == nil {
return
}
check := HandOverCheck{Signer: l.signer, Node: l.node}
check := l.check()
handing, err := l.conn.Subscribe(AskHandOverSubject(l.node), func(msg *nats.Msg) {
if body, answer := HandOverReply(check, handle, msg.Reply, msg.Data); answer {
_ = msg.Respond(body)
@@ -170,7 +193,7 @@ func (l *natsLink) HearSetuidSearches(handle SetuidSearchFunc) {
if handle == nil {
return
}
check := HandOverCheck{Signer: l.signer, Node: l.node}
check := l.check()
searching, err := l.conn.Subscribe(AskSetuidSearchSubject(l.node), func(msg *nats.Msg) {
if body, answer := SetuidSearchReply(check, handle, msg.Reply, msg.Data); answer {
_ = msg.Respond(body)
@@ -189,7 +212,7 @@ func (l *natsLink) HearDirectoryJudgements(handle JudgeDirectoriesFunc) {
if handle == nil {
return
}
check := HandOverCheck{Signer: l.signer, Node: l.node}
check := l.check()
judging, err := l.conn.Subscribe(AskJudgeDirectoriesSubject(l.node), func(msg *nats.Msg) {
if body, answer := JudgeDirectoriesReply(check, handle, msg.Reply, msg.Data); answer {
_ = msg.Respond(body)
@@ -207,7 +230,7 @@ func (l *natsLink) HearAccessActs(handle AccessActFunc) {
if handle == nil {
return
}
check := HandOverCheck{Signer: l.signer, Node: l.node}
check := l.check()
acting, err := l.conn.Subscribe(AskAccessActSubject(l.node), func(msg *nats.Msg) {
if body, answer := AccessActReply(check, handle, msg.Reply, msg.Data); answer {
_ = msg.Respond(body)
@@ -225,7 +248,7 @@ func (l *natsLink) HearMoveSecrets(handle MoveSecretFunc) {
if handle == nil {
return
}
check := HandOverCheck{Signer: l.signer, Node: l.node}
check := l.check()
moving, err := l.conn.Subscribe(AskMoveSecretSubject(l.node), func(msg *nats.Msg) {
if body, answer := MoveSecretReply(check, handle, msg.Reply, msg.Data); answer {
_ = msg.Respond(body)
@@ -258,6 +281,12 @@ func (l *natsLink) Close() {
if l.judging != nil {
_ = l.judging.Unsubscribe()
}
if l.engine != nil {
_ = l.engine.Stop()
}
if l.warrants != nil {
_ = l.warrants.Unsubscribe()
}
if l.conn != nil {
l.conn.Close()
}
@@ -301,10 +330,12 @@ func (l *natsLink) Ask(ctx context.Context, subject string, body []byte) ([]byte
// What the queue asks of a live link beside Bus, each by a type assertion that fails quietly: said here
// so a link that drops one does not build.
var (
_ Bus = (*natsLink)(nil)
_ HealthBus = (*natsLink)(nil)
_ HaveBus = (*natsLink)(nil)
_ ToolBus = (*natsLink)(nil)
_ Asker = (*natsLink)(nil)
_ Asked = (*natsLink)(nil)
_ Bus = (*natsLink)(nil)
_ HealthBus = (*natsLink)(nil)
_ HaveBus = (*natsLink)(nil)
_ ToolBus = (*natsLink)(nil)
_ Asker = (*natsLink)(nil)
_ Asked = (*natsLink)(nil)
_ EngineSeats = (*natsLink)(nil)
_ OperatorAsks = (*natsLink)(nil)
)
+60 -2
View File
@@ -4,6 +4,7 @@ package link
import (
"context"
"sync/atomic"
"testing"
"time"
@@ -14,7 +15,10 @@ import (
// The revisions a bucket keeps of a key, read over a real bus: oldest first, a delete and a purge said as deleted, and
// none for a key that never was (issue 1945: the node agent takes the composition its want names from them).
func TestNatsAKeysRevisionsAreReadBack(t *testing.T) {
srv, err := natsserver.NewServer(&natsserver.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true})
// A second account reads as a node's does: refused the one bucket "denied-it" (the seat's state before its grant arrives).
srv, err := natsserver.NewServer(&natsserver.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true,
Users: []*natsserver.User{{Username: "admin", Password: "a"}, {Username: "node", Password: "n", Permissions: &natsserver.Permissions{
Publish: &natsserver.SubjectPermission{Allow: []string{">"}, Deny: []string{"$JS.API.*.*.KV_denied-it", "$JS.API.*.*.*.KV_denied-it", "$JS.API.*.*.*.KV_denied-it.>", "$KV.denied-it.>"}}}}}})
if err != nil {
t.Fatal(err)
}
@@ -23,7 +27,7 @@ func TestNatsAKeysRevisionsAreReadBack(t *testing.T) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
conn, err := nats.Connect(srv.ClientURL())
conn, err := nats.Connect(srv.ClientURL(), nats.UserInfo("admin", "a"))
if err != nil {
t.Fatal(err)
}
@@ -65,4 +69,58 @@ func TestNatsAKeysRevisionsAreReadBack(t *testing.T) {
if got, err := bus.KVHistory(ctx, "composed-it", "it.never.x"); err != nil || len(got) != 0 {
t.Fatalf("a key that never was: %+v (%v)", got, err)
}
// A bucket that is not there, and one the account may not read, are errors at once, never a wait: the node agent
// follows its placement key by one watch at its start (endWhenPlaced) and, refused, stays on the controller's compositions and says why.
if _, err := bus.KVWatch(ctx, "never-it", ">"); err == nil {
t.Error("a watch of a bucket that is not there gave no error")
}
if _, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: "denied-it", History: 1}); err != nil {
t.Fatal(err)
}
nodeConn, err := nats.Connect(srv.ClientURL(), nats.UserInfo("node", "n"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nodeConn.Close)
started := time.Now()
wctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if _, err := (OverNATS{Conn: nodeConn}).KVWatch(wctx, "denied-it", ">"); err == nil || wctx.Err() != nil {
t.Errorf("a watch the account may not make: error %v after %s, context %v", err, time.Since(started), wctx.Err())
} else {
t.Logf("a refused watch is an error after %s: %v", time.Since(started).Round(time.Millisecond), err)
}
// A bucket made after the watch was asked is then watched, on the bus's word that it was created and with no timer;
// a bucket the account may not read is no waiting but the error (endWhenPlaced, the shadow's candidates).
q := &Session{Membership: Membership{Node: "n1"}}
q.Attach(ctx, OverNATS{Conn: conn, JS: js})
var waited atomic.Int32
actx, acancel := context.WithTimeout(ctx, 20*time.Second)
defer acancel()
ch, err := q.KVWatchAwait(actx, "later-it", ">", func(error) { waited.Add(1) })
if err != nil || waited.Load() != 1 {
t.Fatalf("a bucket not there yet: watch %v, waiting told %d time(s)", err, waited.Load())
}
select {
case u := <-ch:
t.Fatalf("an update before the bucket exists: %+v", u)
case <-time.After(200 * time.Millisecond):
}
later, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: "later-it", History: 1})
if err != nil {
t.Fatal(err)
}
if _, err := later.PutString("placement.n1", "v"); err != nil {
t.Fatal(err)
}
for seen := false; !seen; {
select {
case u := <-ch:
seen = u.Key == "placement.n1" && string(u.Value) == "v"
case <-actx.Done():
t.Fatal("the bucket made after the watch was never watched")
}
}
}
+14 -52
View File
@@ -5,6 +5,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
@@ -312,58 +313,14 @@ const KindRunner = "runner"
// ReasonUsedAsFound starts the reason of a directory used as found.
const ReasonUsedAsFound = "used as found:"
// UnitsHealth is the machine's service managers in one statement (issue 315).
type UnitsHealth struct {
// State is running, degraded — a unit failed, the modules' or the machine's — or unknown when no
// manager could be read.
State string `json:"state"`
// Failed is every unit failed on two looks in a row that no module places: the machine's own.
Failed []FailedUnit `json:"failed"`
// Unread names a manager that could not be read, with why.
Unread []string `json:"unread,omitempty"`
// Away names an account's own manager not running, so not read: nothing in it is failed (issue 764).
Away []string `json:"away,omitempty"`
}
// FailedUnit is one failed unit no module places.
type FailedUnit struct {
Unit string `json:"unit"`
// Scope is system, or user: an account's own manager.
Scope string `json:"scope"`
// Load is loaded, not-found — a unit whose file is gone and whose failure its manager still holds…
Load string `json:"load,omitempty"`
// Result is how it failed: exit-code, timeout, start-limit-hit…
Result string `json:"result,omitempty"`
// Resource is the mesh's own resource that names it, when the mesh placed it in its own right.
Resource string `json:"resource,omitempty"`
Since time.Time `json:"since"`
}
// NetworkHealth is the machine's networking in one statement (ADR 0241): the worst of its parts, since
// when, and each part.
type NetworkHealth struct {
State string `json:"state"`
Since time.Time `json:"since"`
Parts []NetworkPart `json:"parts"`
}
// NetworkPart is one part of a machine's networking, as its engine judged it on its second look.
type NetworkPart struct {
// Part is resolv-conf, names, tunnel, bus or route.
Part string `json:"part"`
State string `json:"state"`
// Reason is why it is not healthy, in words with no address, path or domain; Said the detail.
Reason string `json:"reason,omitempty"`
Said string `json:"said,omitempty"`
// Writer is the program that rewrote the resolver file, when it can be named; Owner the module whose
// file it is — the uplink's holder.
Writer string `json:"writer,omitempty"`
Owner string `json:"owner,omitempty"`
// Toward is what the failure points at: "hub", or each resolver's address that failed.
Toward []string `json:"toward,omitempty"`
Since time.Time `json:"since"`
Streak int `json:"streak,omitempty"`
}
// The machine's networking and its service managers in a statement are the node's health report's parts (mesh-sdk
// delivery, issue 774): one layout for the message, the report and the watchman that reads it.
type (
NetworkHealth = delivery.NetworkHealth
NetworkPart = delivery.NetworkPart
UnitsHealth = delivery.UnitsHealth
FailedUnit = delivery.FailedUnit
)
// The states a long-running resource is said in (ADR 0240 §4).
const (
@@ -743,4 +700,9 @@ type ModuleFact struct {
Phase string
Failed bool
Detail string
// Judgement and Waits are, in phase waiting-yes, the judgement of the move and the composition waiting (ADR 0547 §3).
Judgement string
Waits string
// Sealed is the values the composition it names carries sealed to this node's key, by slot (delivery.Have.SealedAt).
Sealed delivery.Seals
}
+208
View File
@@ -0,0 +1,208 @@
package link
import (
"context"
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
)
// The node-engine's own health report in the bus's own storage (issue 774, the operator's alert table, 2026-10-29): one
// key per node, `<node>`, of the engine's declared state `health` (HealthBucket, created by the grants part from the
// engine's manifest as its state `acts` is), written by this node's engine alone and only when what it says changed.
// The watchman (seat mesh-watchdogs) reads it for six alerts the controller raised from the passing health message:
//
// - machine-network: Network, the machine's route, tunnel, name servers, resolver file and bus, each a part;
// - network-rewritten: the resolver file's part, unhealthy with Writer, the program that rewrote it, and Owner, the
// module whose file it is;
// - network-unreachable: Peers, each peer of the mesh's tunnel and whether this machine saw it within five minutes,
// read across every node's report;
// - machine-units: Units, the failed system units no module owns;
// - relogin-needed: Relogin, each account whose running session lacks a group it was given;
// - directory-used-as-found: Found, each directory a module declares that the engine uses as found.
//
// The report's layout is mesh-sdk's (delivery.NodeHealth): the bucket, its version and its parts.
const (
HealthBucket = delivery.NodeHealthBucket
NodeHealthVersion = delivery.NodeHealthVersion
)
type (
NodeHealth = delivery.NodeHealth
PeerSeen = delivery.PeerSeen
NodeAccount = delivery.NodeAccount
NodeDirectory = delivery.NodeDirectory
)
// ReasonReloginNeeded starts an account's reason when its session needs a new login (accounts.ReasonRelogin).
const ReasonReloginNeeded = "relogin needed"
// Machines names the mesh's machines for a report's targets: each node by its address on the private network, and the
// hub's node. The zero value names none, and every part is then the machine's own.
type Machines struct {
ByAddress map[string]string
Hub string
}
// targets is the machines a part points at, as the controller holds it (mesh-controller machine_network.go): the hub's
// node for TowardHub, each resolver's node by its address; none when it points at nothing, or at this machine or an
// address no other mesh machine has, as one of them makes the part the machine's own.
func (m Machines) targets(node string, toward []string) []string {
if len(toward) == 0 {
return nil
}
var out []string
for _, t := range toward {
x := m.ByAddress[t]
if t == TowardHub {
x = m.Hub
}
if x == "" || x == node {
return nil
}
if !slices.Contains(out, x) {
out = append(out, x)
}
}
return out
}
// TowardHub is what a part failing toward the hub names in Toward (network.TowardHub).
const TowardHub = "hub"
// NodeHealthOf is the report a health statement and the tunnel's peers make: the statement's network without its
// streaks, each unhealthy part naming the machines it points at (machines), its machine units, and each account and
// directory of its resources that the six alerts read.
func NodeHealthOf(node string, h Health, peers []PeerSeen, machines Machines) NodeHealth {
out := NodeHealth{V: NodeHealthVersion, Node: node, Units: h.Units, Contract: h.Contract}
if len(peers) > 0 {
out.Peers = append([]PeerSeen(nil), peers...)
sort.Slice(out.Peers, func(i, k int) bool { return out.Peers[i].Key < out.Peers[k].Key })
}
if h.Network != nil {
n := NetworkHealth{State: h.Network.State, Since: h.Network.Since, Parts: make([]NetworkPart, 0, len(h.Network.Parts))}
for _, p := range h.Network.Parts {
p.Streak = 0 // a look's count moves every look; the report says a part's state, not how often it was seen
p.Targets = nil
if p.State == StateUnhealthy {
p.Targets = machines.targets(node, p.Toward)
}
n.Parts = append(n.Parts, p)
}
out.Network = &n
}
for _, r := range h.Resources {
// Each alert reads its own: an account may need a new login and be declared never to become root at once.
if r.Kind == KindAccount && r.State == StateUnhealthy && strings.HasPrefix(r.Reason, ReasonReloginNeeded) {
out.Relogin = append(out.Relogin, NodeAccount{Account: r.Account, Module: r.Module, Resource: r.Resource,
Reason: r.Reason, Since: r.Since})
}
if r.Kind == KindAccount && r.Root == declaration.RootNever {
// The account declared never to become root without a person, judged (ADR 0266): what DA and D-root read.
out.Root = append(out.Root, delivery.RootVerdict{Account: r.Account, Module: r.Module, Resource: r.Resource,
State: r.State, Reason: r.Reason, Since: r.Since})
}
if r.Kind == KindDirectory && strings.HasPrefix(r.Reason, ReasonUsedAsFound) {
out.Found = append(out.Found, NodeDirectory{Module: r.Module, Resource: r.Resource, Path: r.Target,
Reason: r.Reason, Since: r.Since})
}
}
sort.Slice(out.Relogin, func(i, k int) bool {
return out.Relogin[i].Account+out.Relogin[i].Module < out.Relogin[k].Account+out.Relogin[k].Module
})
sort.Slice(out.Found, func(i, k int) bool { return out.Found[i].Path < out.Found[k].Path })
sort.Slice(out.Root, func(i, k int) bool {
return out.Root[i].Module+out.Root[i].Resource < out.Root[k].Module+out.Root[k].Resource
})
return out
}
// HealthStore is where a node's report is kept: its own key of HealthBucket.
type HealthStore interface {
// Get is the key's value, nil when it has none.
Get(ctx context.Context, key string) ([]byte, error)
Put(ctx context.Context, key string, value []byte) error
}
// healthStore binds HealthBucket over the link open now: bound, never created.
func (l *natsLink) healthStore() (HealthStore, error) {
kv, err := l.js.KeyValue(HealthBucket)
if err != nil {
return nil, fmt.Errorf("the node-engine's state %s cannot be bound (it is created from the node-engine's "+
"manifest; written again once it is): %w", HealthBucket, err)
}
return natsHealth{kv}, nil
}
type natsHealth struct{ kv nats.KeyValue }
func (n natsHealth) Get(_ context.Context, key string) ([]byte, error) {
e, err := n.kv.Get(key)
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
return e.Value(), nil
}
func (n natsHealth) Put(_ context.Context, key string, value []byte) error {
_, err := n.kv.Put(key, value)
return err
}
// WriteNodeHealth writes this node's report to its key when it says something other than the key holds, and says
// whether it wrote. The key is read before the first write of a run, so a restart that finds the same report writes
// nothing; a store that cannot be bound or written is an error, and the next look tries again.
func (q *Session) WriteNodeHealth(ctx context.Context, h NodeHealth) (bool, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
l, ok := bus.(interface{ healthStore() (HealthStore, error) })
if bus == nil || !ok {
return false, ErrNoLink
}
store, err := l.healthStore()
if err != nil {
return false, err
}
return writeNodeHealth(ctx, store, &q.healthWritten, h, time.Now)
}
// writeNodeHealth is WriteNodeHealth over a store: last is the report last written or read, nil before the first.
func writeNodeHealth(ctx context.Context, store HealthStore, last **NodeHealth, h NodeHealth, now func() time.Time) (bool, error) {
if *last == nil {
raw, err := store.Get(ctx, h.Node)
if err != nil {
return false, fmt.Errorf("the health report of %s cannot be read: %w", h.Node, err)
}
if raw != nil {
if was, err := delivery.DecodeNodeHealth(raw); err == nil {
*last = &was
}
}
}
if *last != nil && delivery.SameNodeHealth(**last, h) {
return false, nil
}
h.At = now().UTC()
raw, err := delivery.EncodeNodeHealth(h)
if err != nil {
return false, err
}
if err := store.Put(ctx, h.Node, raw); err != nil {
return false, fmt.Errorf("the health report of %s cannot be written: %w", h.Node, err)
}
*last = &h
return true, nil
}
+68
View File
@@ -0,0 +1,68 @@
package link
import (
"encoding/json"
"sort"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The node's record (novox/hq ADR 0555 §3, issue 1126): what the machine observes of itself, written in its own have key
// `<node>.node` (delivery.HaveNode.Record) with its facts, so every node and the parts that compose read it from the bus
// and hear its change as the key's. Each value is the one the controller took from this node's enrolment and reports
// until now (its profile's present capabilities and networks, its outward links, its health statement's contract, and
// the public halves of the keys it enrolled with), read here from the same sources, never from the controller.
// RecordKeys are the public halves the node holds and the certificate the mesh issued for its internal name, as read
// from its state directory at the time of a pass: the certificate arrives after enrolment, a rekey changes the tunnel key.
type RecordKeys struct {
TunnelKey string
SealingKey string
ServingKey string
Certificate string
// Authority is the mesh's own certificate as this machine last applied it (public).
Authority string
// Trusts is what this machine trusts a mesh signature from now (delivery.Trusts, ADR 0550 §2).
Trusts []string
}
// recordOf is the node's record from one facts report and its keys; nil for a report that carries no profile (one that
// says nothing of the machine, such as a rekey), so the record last written stands.
func recordOf(node string, r Report, keys RecordKeys) *delivery.NodeRecord {
if r.Profile == nil {
return nil
}
var p struct {
Capabilities []struct {
Name string `json:"name"`
Present bool `json:"present"`
} `json:"capabilities"`
Machine *struct {
Networks []catalogue.AttachedNetwork `json:"networks"`
} `json:"machine"`
}
raw, err := json.Marshal(r.Profile)
if err != nil || json.Unmarshal(raw, &p) != nil {
return nil
}
rec := &delivery.NodeRecord{V: delivery.Version, Node: node, TunnelKey: keys.TunnelKey, SealingKey: keys.SealingKey,
ServingKey: keys.ServingKey, Certificate: keys.Certificate, Authority: keys.Authority, Trusts: keys.Trusts}
for _, c := range p.Capabilities {
if c.Present && c.Name != "" {
rec.Capabilities = append(rec.Capabilities, c.Name)
}
}
sort.Strings(rec.Capabilities)
if p.Machine != nil {
rec.NetworksSaid = true
rec.Networks = p.Machine.Networks
}
if len(r.Outward) > 0 {
rec.OutwardLinks = append([]string(nil), r.Outward...)
}
if r.Health != nil {
rec.Contract = r.Health.Contract
}
return rec
}
+21
View File
@@ -7,6 +7,8 @@ import (
"errors"
"fmt"
"time"
meshasks "git.novox.be/novox/mesh-sdk/go/asks"
)
// Membership is what a node needs to reach its mesh again, held by the caller.
@@ -185,6 +187,12 @@ func serve(ctx context.Context, link Link, m Membership, queue *Session, say Ann
}
// And the controller asking this node to hand a directory used as found to the mesh (novox/hq issue
// 356), answered on the link the question came over.
// Every signed ask is verified with the key trusted at that ask, none after the lease write (ADR 0563 §7).
if f, ok := link.(interface {
FollowSigner(func() ed25519.PublicKey)
}); ok && queue.ControllerKey != nil {
f.FollowSigner(queue.ControllerKey)
}
if h, ok := link.(HandOvers); ok {
h.HearHandOvers(queue.HandOver)
}
@@ -200,6 +208,19 @@ func serve(ctx context.Context, link Link, m Membership, queue *Session, say Ann
if s, ok := link.(AccessActs); ok {
s.HearAccessActs(queue.AccessAct)
}
// And its node seat, node-engine, with the operator's answers to its own asks (novox/hq ADR 0563): served on this
// link, and what was answered while no link was open caught up from the router's records.
if queue.Acts != nil {
if s, ok := link.(EngineSeats); ok {
s.ServeEngineSeat(queue.Acts.Call)
}
if a, ok := link.(OperatorAsks); ok {
if err := a.HearWarrants(meshasks.EngineAsker(m.Node), queue.Acts.Heard); err != nil {
say("cannot hear the operator's answers to this engine's asks: " + err.Error())
}
go queue.Acts.Reconcile()
}
}
// And asking to move a value sealed to this machine into the vault (design 57's V3, issue 676).
if s, ok := link.(MoveSecrets); ok {
s.HearMoveSecrets(queue.MoveSecret)
+62 -2
View File
@@ -2,12 +2,14 @@ package link
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"fmt"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/health"
)
@@ -47,8 +49,17 @@ type Session struct {
// MoveSecret moves one value sealed to this machine into the vault, asked by the controller during the move of
// the secrets (design 57's V3, issue 676). Nil answers every such ask with a refusal.
MoveSecret MoveSecretFunc
Say Announce
Timeout time.Duration
// ControllerKey is the controller's key as the engine trusts it now: nil once its lease is handed away (novox/hq
// ADR 0563 §7). Each link verifies every signed ask with it, read per ask; nil ControllerKey keeps the membership's.
ControllerKey func() ed25519.PublicKey
// Acts is the node seat node-engine this engine holds (novox/hq ADR 0563): served on every link, and its asks to the
// operator asked and heard over it. Nil serves no seat.
Acts *HandActs
// Keys reads the node's public keys and its certificate for its record (ADR 0555 §3), at each facts report; nil writes
// no record.
Keys func() RecordKeys
Say Announce
Timeout time.Duration
once sync.Once
@@ -75,6 +86,9 @@ type Session struct {
// haves is what this node-engine writes beside its facts (novox/hq ADR 0429 §4, issue 1132).
haves haveState
// healthWritten is this node's health report as last written or read (WriteNodeHealth): the health loop alone
// writes it, one look at a time.
healthWritten *NodeHealth
}
func (q *Session) init() {
@@ -334,3 +348,49 @@ func (q *Session) AskSeat(ctx context.Context, seat, verb string, args any, with
}
return env.Result, nil
}
// PublishAsk publishes an ask or a cancel to the operator's router over the link open now (novox/hq ADR 0563 §3).
func (q *Session) PublishAsk(subject string, body any, id string) error {
asking, err := q.operatorAsks()
if err != nil {
return err
}
return asking.PublishAsk(subject, body, id)
}
// RouterRecord reads one key of the router's records over the link open now.
func (q *Session) RouterRecord(key string) (*stdio.StateEntry, error) {
asking, err := q.operatorAsks()
if err != nil {
return nil, err
}
return asking.RouterRecord(key)
}
// SayEngine says one of the node seat's events over the link open now; with none open it is not said.
func (q *Session) SayEngine(event string, body any) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if sayer, ok := bus.(interface{ SayEngine(string, any) error }); ok {
if err := sayer.SayEngine(event, body); err != nil {
q.Say("could not say the node seat's " + event + ": " + err.Error())
}
}
}
func (q *Session) operatorAsks() (OperatorAsks, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return nil, fmt.Errorf("%w: this machine has no link to the bus open now", ErrNotAsked)
}
asking, ok := bus.(OperatorAsks)
if !ok {
return nil, errNoAsking
}
return asking, nil
}
+78
View File
@@ -57,4 +57,82 @@ func TestASetuidSearchAskIsBelievedOnlySignedForThisNodeAndOnce(t *testing.T) {
if a := AnswerSetuidSearch(check, nil, body); a.Refused == "" {
t.Fatalf("no handler: %+v", a)
}
// 774 (ADR 0563 §2): the node seat's setuid-search starts none while one runs, at most one in any hour, naming when the
// next may start, and records who asked.
t.Run("the node seat's search is bounded", func(t *testing.T) {
at, running, started := now, false, 0
h := &HandActs{Node: "n1", State: &MemState{}, Now: func() time.Time { return at },
Searching: func() bool { return running },
SetuidSearch: func(by string) (string, error) { started++; return "a new search starts", nil }}
for _, row := range []struct {
name string
after time.Duration
running bool
refused string
started int
}{
{name: "one runs", running: true, refused: "runs on this machine now", started: 0},
{name: "the first", started: 1},
{name: "within the hour", after: 59 * time.Minute, refused: "the next may start at 2026-10-10T03:00:00Z", started: 1},
{name: "after the hour", after: time.Hour + time.Second, started: 2},
} {
at, running = now.Add(row.after), row.running
_, err := h.Call(EngineCall{Verb: ActSetuidSearch, Caller: "ace.node-tools"})
if (row.refused == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), row.refused)) || started != row.started {
t.Errorf("%s: %v, %d started", row.name, err, started)
}
}
acts, _ := h.Acts(1)
if got := acts.(map[string]any)["acts"].([]HandAct); len(got) != 1 || got[0].Caller != "ace.node-tools" || got[0].State != HandDone {
t.Errorf("the last search is recorded as %+v", got)
}
// The bound is kept in a key of its own (rev-l3 round 1): two hundred judgements before a search, or the acts
// gone from the record, never let a second search start within the hour.
h.JudgeDirectories = func(string) (string, error) { return "judged", nil }
for range 200 {
if _, err := h.Call(EngineCall{Verb: ActJudgeDirectories, Caller: "ace.node-tools"}); err != nil {
t.Fatal(err)
}
}
at = now.Add(time.Hour + 2*time.Minute)
if _, err := h.Call(EngineCall{Verb: ActSetuidSearch, Caller: "ace.node-tools"}); err == nil || !strings.Contains(err.Error(), "the next may start") {
t.Errorf("a search after two hundred judgements: %v", err)
}
keys, _ := h.State.Keys("act.")
for _, k := range keys {
_ = h.State.Delete(k)
}
if _, err := h.Call(EngineCall{Verb: ActSetuidSearch, Caller: "ace.node-tools"}); err == nil || !strings.Contains(err.Error(), "the next may start") || started != 2 {
t.Errorf("a search with the acts gone: %v, %d started", err, started)
}
// rev-l3 round 2 (ADR 0563 §2, a wrong caller gains nothing): judgements and refused searches, which any caller
// may ask, keep one rolling key of the node's, so ten thousand of them write a bounded number of bytes and add no
// key to the bucket every node shares.
keysNow := func() (int, int) {
all, _ := h.State.Keys("")
size := 0
for _, k := range all {
if e, _ := h.State.Get(k); e != nil {
size += len(e.Value)
}
}
return len(all), size
}
_, _ = h.Call(EngineCall{Verb: ActJudgeDirectories, Caller: "unproven"})
keysBefore, _ := keysNow()
for i := range 10000 {
verb := ActJudgeDirectories
if i%10 == 0 {
verb = ActSetuidSearch
}
_, _ = h.Call(EngineCall{Verb: verb, Caller: "unproven"})
}
if n, size := keysNow(); n != keysBefore || size > 64<<10 {
t.Errorf("ten thousand judgements and refused searches left %d keys (was %d) of %d bytes", n, keysBefore, size)
}
listed, _ := h.Acts(0)
if got := listed.(map[string]any)["acts"].([]HandAct); len(got) == 0 || got[0].Caller != "unproven" {
t.Errorf("acts does not show the latest judgement first: %d", len(got))
}
})
}
+57
View File
@@ -239,3 +239,60 @@ func (q *Session) Hear(subject string) (<-chan []byte, func(), error) {
}
return h.Hear(subject)
}
// CreatedAdvisory is the subject the bus says a bucket's stream was created on (the server's own advisory; the same
// subject mesh-sdk's process.AwaitBucket binds on, and the grants part gives a principal for each bucket it may read).
func CreatedAdvisory(bucket string) string {
return "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_" + bucket
}
// KVWatchAwait is KVWatch for a bucket that may not be on the bus yet (the placement seat's states, made when their
// holder starts): the channel is returned at once, and while the bucket is missing it yields nothing and waiting is told
// why; when the bus says the bucket's stream was created the watch is made and its updates follow. No timer: the
// advisory is heard before the first try, so a creation between the try and the wait is not missed. Any other error of
// the first try is the caller's, as for KVWatch: a bucket the account may not read is not waited for.
func (q *Session) KVWatchAwait(ctx context.Context, bucket, pattern string, waiting func(error)) (<-chan KVUpdate, error) {
created, stop, err := q.Hear(CreatedAdvisory(bucket))
if err != nil {
return nil, err
}
in, err := q.KVWatch(ctx, bucket, pattern)
if err == nil {
stop()
return in, nil
}
if !errors.Is(err, nats.ErrBucketNotFound) {
stop()
return nil, err
}
waiting(err)
out := make(chan KVUpdate, 64)
go func() {
defer close(out)
defer stop()
for in == nil {
select {
case <-ctx.Done():
return
case _, ok := <-created:
if !ok {
return
}
w, err := q.KVWatch(ctx, bucket, pattern)
if err != nil {
waiting(err)
continue
}
in = w
}
}
for u := range in {
select {
case out <- u:
case <-ctx.Done():
return
}
}
}()
return out, nil
}
+46 -7
View File
@@ -32,6 +32,7 @@ import (
"fmt"
"os"
"path/filepath"
"slices"
"sort"
"strconv"
"strings"
@@ -94,6 +95,18 @@ type Finding struct {
Writer string
// Toward is what the failure points at: TowardHub, or each resolver's address that failed.
Toward []string
// Peers is, for the tunnel, each peer of the mesh's interface and whether it was seen (Peer).
Peers []Peer
}
// Peer is one peer of the mesh's tunnel as this machine sees it: its public key, the addresses it is allowed, and
// whether it handshook within StaleHandshake. What the watchman reads to tell a machine nobody reaches from one whose
// tunnel alone is down (network-unreachable). Seen, not the handshake's time: the time moves every two minutes, the
// fact only when a peer comes or goes.
type Peer struct {
Key string `json:"key"`
Addresses []string `json:"addresses,omitempty"`
Seen bool `json:"seen"`
}
// Part is one part's state, as the statement says it.
@@ -115,6 +128,8 @@ type Statement struct {
Since time.Time `json:"since"`
At time.Time `json:"at"`
Parts []Part `json:"parts"`
// Peers are the tunnel's peers on the last look that read them, sorted by key; nil where the tunnel is not read.
Peers []Peer `json:"peers,omitempty"`
}
// Machine is what a look reads, replaced in tests.
@@ -266,6 +281,9 @@ func (j *Judge) Look(ctx context.Context) (Statement, bool) {
changed = true
}
st.State, st.Since = worst, j.overall.since
// The tunnel's peers as this look read them; a peer seen or lost is a change of its own (network-unreachable).
st.Peers = findings[PartTunnel].Peers
changed = changed || !samePeers(j.said.Peers, st.Peers)
j.said = st
return st, changed
}
@@ -435,14 +453,22 @@ func (j *Judge) lookTunnel(ctx context.Context, now time.Time) Finding {
}
}
hub := ""
allowed := map[string][]string{}
for _, line := range strings.Split(ips, "\n") {
f := strings.Fields(line)
for _, prefix := range f[min(1, len(f)):] {
allowed[f[0]] = append(allowed[f[0]], prefix)
if _, bits, ok := strings.Cut(prefix, "/"); ok && bits != "32" && bits != "128" {
hub = f[0]
}
}
}
peers := make([]Peer, 0, len(handshakes))
for key, at := range handshakes {
peers = append(peers, Peer{Key: key, Addresses: allowed[key], Seen: at > 0 && now.Sub(time.Unix(at, 0)) <= StaleHandshake})
}
sort.Slice(peers, func(i, k int) bool { return peers[i].Key < peers[k].Key })
found := func(f Finding) Finding { f.Peers = peers; return f }
age := func(at int64) string {
if at == 0 {
return "never"
@@ -452,23 +478,36 @@ func (j *Judge) lookTunnel(ctx context.Context, now time.Time) Finding {
if hub != "" {
at := handshakes[hub]
if at > 0 && now.Sub(time.Unix(at, 0)) <= StaleHandshake {
return Finding{OK: true}
return found(Finding{OK: true})
}
return Finding{Reason: "the tunnel to the hub has not handshaken for over five minutes",
Said: fmt.Sprintf("%s's newest handshake with the hub was %s", Interface, age(at)), Toward: []string{TowardHub}}
return found(Finding{Reason: "the tunnel to the hub has not handshaken for over five minutes",
Said: fmt.Sprintf("%s's newest handshake with the hub was %s", Interface, age(at)), Toward: []string{TowardHub}})
}
if len(handshakes) == 0 {
return Finding{OK: true}
return found(Finding{OK: true})
}
var newest int64
for _, at := range handshakes {
newest = max(newest, at)
}
if newest > 0 && now.Sub(time.Unix(newest, 0)) <= StaleHandshake {
return Finding{OK: true}
return found(Finding{OK: true})
}
return Finding{Reason: "no machine has handshaken with the hub's tunnel for over five minutes",
Said: fmt.Sprintf("%s's newest handshake with any of its %d peers was %s", Interface, len(handshakes), age(newest))}
return found(Finding{Reason: "no machine has handshaken with the hub's tunnel for over five minutes",
Said: fmt.Sprintf("%s's newest handshake with any of its %d peers was %s", Interface, len(handshakes), age(newest))})
}
// samePeers says two looks saw the same peers, each seen or not alike.
func samePeers(a, b []Peer) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i].Key != b[i].Key || a[i].Seen != b[i].Seen || !slices.Equal(a[i].Addresses, b[i].Addresses) {
return false
}
}
return true
}
func (j *Judge) lookBus() Finding {
+19 -1
View File
@@ -302,9 +302,27 @@ func TestOnTheHubAnyFreshPeerIsAHealthyTunnel(t *testing.T) {
f.hs = "A=\t" + itoa(f.now.Add(-time.Hour).Unix()) + "\nB=\t" + itoa(f.now.Unix()) + "\nC=\t0\n"
f.ips = "A=\t10.10.0.2/32\nB=\t10.10.0.3/32\nC=\t10.10.0.4/32\n"
j := f.judge(t)
if st := f.look(t, j); st.State != Healthy {
st := f.look(t, j)
if st.State != Healthy {
t.Fatalf("the hub with one fresh peer is %+v", st.Parts)
}
// Each peer and whether it was seen within five minutes, for the watchman's network-unreachable (issue 774); a
// peer seen again is a change, a handshake renewed alone is not.
want := []Peer{{Key: "A=", Addresses: []string{"10.10.0.2/32"}}, {Key: "B=", Addresses: []string{"10.10.0.3/32"}, Seen: true},
{Key: "C=", Addresses: []string{"10.10.0.4/32"}}}
if !samePeers(st.Peers, want) {
t.Fatalf("the peers are %+v", st.Peers)
}
f.hs = "A=\t" + itoa(f.now.Add(LookEvery).Unix()) + "\nB=\t" + itoa(f.now.Add(LookEvery).Unix()) + "\nC=\t0\n"
f.now = f.now.Add(LookEvery)
if st, changed := j.Look(t.Context()); !changed || !st.Peers[0].Seen {
t.Fatalf("a peer seen again is no change: changed %v, %+v", changed, st.Peers)
}
f.hs = "A=\t" + itoa(f.now.Add(LookEvery).Unix()) + "\nB=\t" + itoa(f.now.Add(LookEvery).Unix()) + "\nC=\t0\n"
f.now = f.now.Add(LookEvery)
if _, changed := j.Look(t.Context()); changed {
t.Fatal("handshakes renewed alone are a change")
}
}
func TestAnUnreadableTunnelIsSaidAndAMissingToolSkipsIt(t *testing.T) {
+319
View File
@@ -0,0 +1,319 @@
// Package ownresync hands the vault the value of this machine's own secret that runs, where the vault holds another
// (novox/hq issue 1126, cause C4).
//
// The vault's `move-in` never replaces a value it holds, so its first import of an own secret stays fixed; where the
// controller later made that secret anew (a rotation at start, a re-issue), the new value runs and the vault keeps the
// old one. The node's shadow composer sees it: composing from the vault's value, it opens both seals with this
// machine's key and its answer's shown body says delivery.MaskedAnotherValue. After each shadow answer this step asks the
// vault's `move-in` with replace_running: true and what runs, for each own secret so masked, once per running value, and
// says what it asked and what the vault answered. It is move-in's, the verb this node may ask already, so no seat and no
// bus grant changes for it.
//
// **Only while this node applies the controller's compositions**, and only for a module whose composition that runs is
// the one its want names: then the controller's value is what runs, and what runs is the truth the vault lacks. Once the
// node composes its own, the vault's value is the source and nothing is handed. **The switch** is the vault's
// (`own_take_switch`, a person's act): while it is off the vault refuses every ask, and this step says so once per value.
// The vault also refuses any value it changed in the last 15 minutes, so a re-issue the node has not applied yet is
// never put back.
package ownresync
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"sort"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/selfcompose"
)
// Verb is the vault's verb this step asks, with ReplaceRunning true in both steps.
const Verb = "move-in"
// ReplaceRunning is move-in's argument that makes it take the running value over the one the vault holds.
const ReplaceRunning = "replace_running"
// Env is what the step reads and asks.
type Env struct {
Node string
// Composes is whose compositions the node applies now: delivery.ComposesController, delivery.ComposesOwn, or ""
// while its agent has not decided. The step hands nothing but for ComposesController.
Composes func() string
// Running is what the node runs now (agent.Everything).
Running func() []delivery.Composed
// Get reads one key's latest value; nil and no error when there is none.
Get func(ctx context.Context, bucket, key string) ([]byte, error)
// Unseal opens a value sealed to this machine; Seal seals a value to a raw X25519 key in base64 (identity.Seal).
Unseal func(sealed string) ([]byte, error)
Seal func(to string, value []byte) (string, error)
// Ask asks one of the vault's verbs in this engine's own name (link.Session.AskVault).
Ask func(ctx context.Context, verb string, args map[string]any) (json.RawMessage, error)
Say func(string)
}
// Outcome is what the step did for one own secret: asked or not, and what was said.
type Outcome struct {
Module string
Name string
Asked bool
Said string
// Retry says a read or an ask the bus did not answer stopped it: no answer, kept as none, and the step is to be
// taken again after a bound (issue 1126, 2026-10-28: "want.shanks.brother could not be read ... no responders").
Retry bool
}
// Resync is the step of one node agent's run.
type Resync struct {
env Env
mu sync.Mutex
// done is, by module/name, the running value's fingerprint the vault was asked with or why it was not: each is
// asked, and each reason said, once.
done map[string]string
}
// New is the step for one node.
func New(env Env) *Resync {
if env.Say == nil {
env.Say = func(string) {}
}
return &Resync{env: env, done: map[string]string{}}
}
// Take reads the node's shadow answers and hands the vault what runs of each own secret they say another value of.
func (r *Resync) Take(ctx context.Context, answers map[string]delivery.CandidateAnswer) []Outcome {
seen := map[delivery.OwnValueDiffers]bool{}
var drift []delivery.OwnValueDiffers
proposals := make([]string, 0, len(answers))
for g := range answers {
proposals = append(proposals, g)
}
sort.Strings(proposals)
for _, g := range proposals {
for _, o := range delivery.OwnValuesDiffer(answers[g]) {
if !seen[o] {
seen[o] = true
drift = append(drift, o)
}
}
}
if len(drift) == 0 {
return nil
}
composes := ""
if r.env.Composes != nil {
composes = r.env.Composes()
}
var out []Outcome
for _, o := range drift {
res := r.one(ctx, composes, o)
if res.Said != "" || res.Retry {
out = append(out, res)
}
}
return out
}
// one is the step for one own secret: Said is empty when it was done before for the same value and reason.
func (r *Resync) one(ctx context.Context, composes string, o delivery.OwnValueDiffers) Outcome {
res := Outcome{Module: o.Module, Name: o.Name}
key := o.Module + "/" + o.Name
if composes == "" {
return res // the agent has not decided whose compositions it applies: taken again once it has
}
if composes != delivery.ComposesController {
return r.said(key, "not:composes", res, fmt.Sprintf("%s holds another value in the vault than runs here, and "+
"nothing is handed: this node composes its own (%q), so the vault's value is the source", key, composes))
}
sealed, why, failed := r.running(ctx, o)
if failed != nil {
// The bus did not answer: no reason to keep, and taken again (Retry); said once per reason.
res.Retry = true
return r.said(key, "retry:"+why, res, fmt.Sprintf("%s holds another value in the vault than runs here, and "+
"nothing is handed yet: %s; asked again", key, why))
}
if why != "" {
return r.said(key, "not:"+why, res, fmt.Sprintf("%s holds another value in the vault than runs here, and "+
"nothing is handed: %s", key, why))
}
sum := sha256.Sum256([]byte(sealed))
fp := hex.EncodeToString(sum[:])
r.mu.Lock()
asked := r.done[key] == fp
r.mu.Unlock()
if asked {
return res
}
value, err := r.env.Unseal(sealed)
if err != nil {
return r.said(key, "not:unseal:"+fp, res, fmt.Sprintf("%s: what runs does not open with this machine's key, so "+
"nothing is handed: %v", key, err))
}
defer wipe(value)
said, err := r.hand(ctx, o, value)
if err != nil {
if ctx.Err() != nil {
return res // the run ended: asked again in the next
}
if selfcompose.BusFailed(err) {
// The vault did not answer on the bus: no word of the vault's, so not kept as asked (Retry).
res.Retry = true
return r.said(key, "retry:ask", res, fmt.Sprintf("%s: the vault's %s with replace_running did not answer, "+
"and is asked again: %v", key, Verb, err))
}
said = fmt.Sprintf("%s: the vault's %s with replace_running refused: %v", key, Verb, err)
}
// Once per running value, whatever the vault answered: a refusal is the vault's word, said once, and a new value
// that runs asks again.
r.mu.Lock()
r.done[key] = fp
r.mu.Unlock()
res.Asked, res.Said = true, said
r.env.Say(said)
return res
}
// said says why once per reason, and keeps the reason as done.
func (r *Resync) said(key, reason string, res Outcome, why string) Outcome {
r.mu.Lock()
again := r.done[key] == reason
r.done[key] = reason
r.mu.Unlock()
if again {
return res
}
res.Said = why
r.env.Say(why)
return res
}
// running is the sealed value of o in the composition that runs, when that composition is the one the module's want
// names; or why it is not handed, with the error of a read the bus did not answer, which is no answer.
func (r *Resync) running(ctx context.Context, o delivery.OwnValueDiffers) (sealed, why string, failed error) {
var comp *delivery.Composed
if r.env.Running != nil {
for _, c := range r.env.Running() {
if c.Module == o.Module {
c := c
comp = &c
break
}
}
}
if comp == nil {
return "", o.Module + " runs no composition here", nil
}
wantKey, err := delivery.WantKey(r.env.Node, o.Module)
if err != nil {
return "", err.Error(), nil
}
raw, err := r.env.Get(ctx, delivery.AssignerWantsBucket, wantKey)
if err != nil {
return "", fmt.Sprintf("its want %s could not be read (%v)", wantKey, err), err
}
if raw == nil {
return "", fmt.Sprintf("it has no want %s", wantKey), nil
}
want, err := delivery.DecodeWant(raw)
if err != nil {
return "", fmt.Sprintf("its want %s is not one: %v", wantKey, err), nil
}
if want.Self || want.Removed || want.Composed == "" {
return "", fmt.Sprintf("its want %s names no composition of the controller's", wantKey), nil
}
// What runs is the composition the want names, as the module's have says: applied and running.
haveKey, err := delivery.HaveKey(r.env.Node, o.Module)
if err != nil {
return "", err.Error(), nil
}
raw, err = r.env.Get(ctx, delivery.HaveBucket, haveKey)
if err != nil {
return "", fmt.Sprintf("its have %s could not be read (%v)", haveKey, err), err
}
if raw == nil {
return "", fmt.Sprintf("it has no have %s", haveKey), nil
}
have, err := delivery.DecodeHave(raw)
if err != nil {
return "", fmt.Sprintf("its have %s is not one: %v", haveKey, err), nil
}
if have.Composed != want.Composed || have.Phase != delivery.PhaseRunning {
return "", fmt.Sprintf("what runs (%s, %s) is not yet the composition its want names (%s)", have.Composed,
have.Phase, want.Composed), nil
}
// On a node the resource's id carries its module's name (`only-office.needs-jwt`); delivery.OwnNeedName reads it.
id := o.Module + "." + catalogue.NeedID(o.Name)
for _, res := range comp.Resources {
var body struct {
ID string `json:"id"`
Sealed any `json:"sealed"`
}
if json.Unmarshal(res, &body) != nil {
continue
}
if name, own := delivery.OwnNeedName(o.Module, body.ID); !own || name != o.Name {
continue
}
if s, ok := body.Sealed.(string); ok && s != "" {
return s, "", nil
}
return "", fmt.Sprintf("its resource %s carries no value sealed to this machine", id), nil
}
return "", fmt.Sprintf("what runs holds no resource %s", id), nil
}
// hand asks the vault's two steps with value, and checks what the vault holds after by its fingerprint.
func (r *Resync) hand(ctx context.Context, o delivery.OwnValueDiffers, value []byte) (string, error) {
raw, err := r.env.Ask(ctx, Verb, map[string]any{ReplaceRunning: true, "module": o.Module, "name": o.Name,
"why": "this node's shadow answer says the vault holds another value than runs here"})
if err != nil {
return "", err
}
var opened struct {
Call string `json:"call"`
SealTo string `json:"seal_to"`
Secret string `json:"secret"`
}
if err := json.Unmarshal(raw, &opened); err != nil || opened.Call == "" || opened.SealTo == "" {
return "", fmt.Errorf("the vault answered no call and no key to seal to")
}
sealed, err := r.env.Seal(opened.SealTo, value)
if err != nil {
return "", fmt.Errorf("the vault's key for the call is not a sealing key: %w", err)
}
raw, err = r.env.Ask(ctx, Verb, map[string]any{ReplaceRunning: true, "call": opened.Call, "sealed": sealed})
if err != nil {
return "", err
}
var kept struct {
Secret string `json:"secret"`
From int `json:"from_generation"`
To int `json:"to_generation"`
Unchanged bool `json:"unchanged"`
Fingerprint string `json:"fingerprint"`
Said string `json:"said"`
}
if err := json.Unmarshal(raw, &kept); err != nil || kept.Secret == "" {
return "", fmt.Errorf("the vault answered %s with no secret", opened.Secret)
}
sum := sha256.Sum256(value)
if kept.Fingerprint != "sha256:"+hex.EncodeToString(sum[:]) {
return "", fmt.Errorf("the vault says it holds %s at generation %d, but not the value handed in", kept.Secret, kept.To)
}
if kept.Unchanged {
return fmt.Sprintf("handed the vault what runs of %s: it held that value already (generation %d)", kept.Secret, kept.To), nil
}
return fmt.Sprintf("handed the vault what runs of %s: generation %d replaced by %d, the value that runs", kept.Secret,
kept.From, kept.To), nil
}
func wipe(b []byte) {
for i := range b {
b[i] = 0
}
}
+117
View File
@@ -0,0 +1,117 @@
package ownresync
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Issue 1126, cause C4: after a shadow answer saying the vault holds another value of an own secret than runs, the node
// hands the vault what runs, in the verb's two steps, once per running value; and only while it applies the
// controller's compositions and runs the one its want names.
func TestTheNodeHandsTheVaultWhatRunsOfADriftedOwnSecret(t *testing.T) {
const running = "the-value-that-runs"
digest := "sha256:" + strings.Repeat("a", 64)
fp := "src2:" + strings.Repeat("b", 64)
sum := sha256.Sum256([]byte(running))
answers := map[string]delivery.CandidateAnswer{"g": {Proposal: "p-1", Revision: 1, Verdict: delivery.CandidateDiffers,
Differs: []delivery.ModuleDiffers{{Module: "only-office", Why: "the resource differs", Bodies: []delivery.ResourceBodies{
{ID: "only-office.needs-jwt", Composed: `{"sealed":"` + delivery.MaskedAnotherValue + `"}`, Running: `{"sealed":"` + delivery.MaskedAnotherValue + `"}`},
{ID: "only-office.needs-salt", Composed: `{"sealed":"<masked: another seal of the same value>"}`}}}}}}
for name, row := range map[string]struct {
composes string
runs string // the have's composition digest
refuse string // the vault's refusal
asks int // the vault's verb asked, both passes
says string
// Issue 1126, 2026-10-28: a read or an ask the bus did not answer is no answer: taken again (Retry), never kept.
wantRead string // the want's read: "" answers it, "fails" the bus does not answer, "absent" no key
retry bool
}{
"applies the controller's, runs its want": {delivery.ComposesController, digest, "", 2, "generation 1 replaced by 2", "", false},
"composes its own": {delivery.ComposesOwn, digest, "", 0, "composes its own", "", false},
"not decided yet": {"", digest, "", 0, "", "", false},
"runs not yet what its want names": {delivery.ComposesController, "sha256:" + strings.Repeat("c", 64), "", 0, "not yet the composition", "", false},
"the vault refuses": {delivery.ComposesController, digest, "switched off", 1, "switched off", "", false},
"the vault does not answer": {delivery.ComposesController, digest, "nats: no responders available for request", 2, "asked again", "", true},
"the want's read does not answer": {delivery.ComposesController, digest, "", 0, "asked again", "fails", true},
"it has no want": {delivery.ComposesController, digest, "", 0, "it has no want", "absent", false},
} {
var asked []map[string]any
var said []string
want, _ := delivery.EncodeWant(delivery.Want{FP: fp, Composed: digest})
have, _ := delivery.EncodeHave(delivery.Have{FP: fp, Composed: row.runs, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
r := New(Env{Node: "novox", Composes: func() string { return row.composes },
Running: func() []delivery.Composed {
return []delivery.Composed{{Node: "novox", Module: "only-office", FP: fp, Resources: []json.RawMessage{
// As novox runs it (its shadow answer, 2026-10-28): the id carries the module's name.
json.RawMessage(`{"id":"only-office.needs-jwt","module":"only-office","type":"file",` +
`"path":"/var/lib/only-office/jwt.secret","sealed":"box-of-the-running-value"}`)}}}
},
Get: func(_ context.Context, bucket, key string) ([]byte, error) {
switch {
case bucket == delivery.AssignerWantsBucket && key == "want.novox.only-office" && row.wantRead == "fails":
return nil, errors.New("the bucket mesh-delivery-assigner_wants: nats: no responders available for request")
case bucket == delivery.AssignerWantsBucket && key == "want.novox.only-office" && row.wantRead == "absent":
return nil, nil
case bucket == delivery.AssignerWantsBucket && key == "want.novox.only-office":
return want, nil
case bucket == delivery.HaveBucket && key == "novox.only-office":
return have, nil
}
return nil, nil
},
Unseal: func(sealed string) ([]byte, error) {
if sealed != "box-of-the-running-value" {
return nil, errors.New("not this machine's")
}
return []byte(running), nil
},
Seal: func(to string, value []byte) (string, error) { return to + "(" + string(value) + ")", nil },
Ask: func(_ context.Context, verb string, args map[string]any) (json.RawMessage, error) {
asked = append(asked, args)
if verb != "move-in" || args["replace_running"] != true {
t.Errorf("%s: asked %s with %v, not move-in with replace_running", name, verb, args)
}
if row.refuse != "" {
return nil, errors.New(row.refuse)
}
if args["call"] == nil {
if args["module"] != "only-office" || args["name"] != "jwt" || args["node"] != nil {
t.Errorf("%s: the first step names %v", name, args)
}
return json.RawMessage(`{"call":"c1","seal_to":"key","secret":"own/novox/only-office/jwt"}`), nil
}
if args["sealed"] != "key("+running+")" {
t.Errorf("%s: handed %v, not what runs sealed to the call's key", name, args["sealed"])
}
return json.RawMessage(`{"secret":"own/novox/only-office/jwt","from_generation":1,"to_generation":2,` +
`"fingerprint":"sha256:` + hex.EncodeToString(sum[:]) + `"}`), nil
},
Say: func(s string) { said = append(said, s) }})
first := r.Take(context.Background(), answers)
again := r.Take(context.Background(), answers)
if len(asked) != row.asks {
t.Errorf("%s: the vault asked %d times, want %d: %v", name, len(asked), row.asks, asked)
}
if retry := len(first) == 1 && first[0].Retry; retry != row.retry {
t.Errorf("%s: taken again %v, want %v: %+v", name, retry, row.retry, first)
}
if row.retry && (len(again) != 1 || !again[0].Retry || again[0].Said != "") || !row.retry && len(again) != 0 {
t.Errorf("%s: the same answer did or said something again: %+v", name, again)
}
all := strings.Join(said, "\n")
if row.says == "" && (len(first) != 0 || all != "") || row.says != "" && (len(first) != 1 || !strings.Contains(all, row.says)) {
t.Errorf("%s: did %+v and said %q, want %q", name, first, all, row.says)
}
if strings.Contains(all, running) {
t.Errorf("%s: the value is in what the node said", name)
}
}
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+839
View File
@@ -0,0 +1,839 @@
package selfcompose
import (
"bytes"
"context"
"fmt"
"reflect"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/agent"
)
// Shadow is the node's answer to placement candidates (novox/hq ADR 0559 §4, issue 1126): for every proposal with a key in
// the placement seat's state `candidates`, it composes this node in shadow from the trunk's placement with the proposal's
// keys laid over it and its own inputs (the sealed values and boxes it holds among them), compares the result with what
// the node runs, and hands the answers to be written into the node's own have key, `<node>.node`, under `candidates`.
// It applies nothing and hands no composition: what runs is still composed from `placement` alone, by the Composer.
//
// **Event-driven, like the Composer.** It watches every candidate key, the trunk's placement, every node's record, the
// indexer's modules, this node's config values, and this node's module haves (what it runs moved); a burst of changes
// answers once. It runs whether or not the node is placed yet: a node that applies the controller's compositions answers
// against what it runs from them. An answer is handed only when it changes, and a proposal's answer is dropped when its
// keys go.
type Shadow struct {
c *Composer
answer func(ctx context.Context, answers map[string]delivery.CandidateAnswer)
mu sync.Mutex
candidates map[string]map[string]candidateKey // by proposal, by node
answers map[string]delivery.CandidateAnswer
// wants are the node's wants by module (agent.NodeModule for its own): what a proposal deleting its placement returns
// it to, the controller's compositions at them.
wants map[string]delivery.Want
// ran is, by module, the digest the module's have says runs: what runs, where the agent's list holds only the copy
// from the declaration kept before the switch (issue 1126, probe #532 on shanks).
ran map[string]string
// heldNow is, by module, how its have says the node keeps it today as it runs: "held", "failed", or "runs a build of
// the old form" (a `build` and no `fp`, which no registered build of the trunk names); "" when it is composed as any
// other. A candidate that cannot compose it, or holds it only behind one of these, keeps it as today
// (frombus.Scope.HeldToday, issue 1126).
heldNow map[string]string
// answered is, by proposal, the candidate revisions and the input changes (moved) each answer was composed at: a proposal
// is answered again only when either moved (issue 1126, 2026-10-28: shanks answered probe/1126-seed-7 at revision 84
// every 13-17 s, woken by its own modules' haves whose health words changed at every look).
answered map[string]answeredAt
// moved counts, by input (inputWords), the changes of each of the node's own inputs, and retry names the proposals
// answered again after a read the bus did not answer: what woke each answer is said with it (issue 1126,
// 2026-10-28, seed-9's probe: a safety check could not tell a loop from inputs moving).
moved [len(inputWords)]uint64
retry map[string]bool
// owed says an answer met a read the bus did not answer: it is no answer to keep, and its proposal is answered again
// after a bound (laterAgain), whether or not an input moves (issue 1126, 2026-10-28: ace kept "the vault's own: not
// yet ... no responders" in its answer to seed-7).
owed bool
}
// answeredAt is what one proposal's answer was composed at: its keys' revisions, by node, and the changes of each input.
type answeredAt struct {
revs string
moved [len(inputWords)]uint64
}
// inputWords name the shadow's inputs, in Run's order, as an answer says which of them woke it.
var inputWords = [...]string{"the candidate", "placement", "records", "modules", "values", "haves", "wants"}
// wokenBy says what moved between the answer a proposal stands at (had, when done) and the one about to be composed (at):
// its first answer, a retry, its candidate's revisions, or the inputs that changed, by their words.
func wokenBy(had answeredAt, done, retry bool, at answeredAt) string {
switch {
case retry:
return "a retry after a read the bus did not answer"
case !done:
return "its first answer since the shadow began"
}
var words []string
if had.revs != at.revs {
words = append(words, inputWords[0])
}
for i := 1; i < len(inputWords); i++ {
if had.moved[i] != at.moved[i] {
words = append(words, inputWords[i])
}
}
if len(words) == 0 {
return "nothing that moved"
}
return strings.Join(words, ", ")
}
// candidateKey is one candidate key as last read: its value and revision.
type candidateKey struct {
c delivery.PlacementCandidate
rev uint64
}
// NewShadow is the shadow composer of one node; answer writes the node's answers into its own have key.
func NewShadow(env Env, answer func(ctx context.Context, answers map[string]delivery.CandidateAnswer)) *Shadow {
return &Shadow{c: New(env), answer: answer, candidates: map[string]map[string]candidateKey{},
answers: map[string]delivery.CandidateAnswer{}, wants: map[string]delivery.Want{}, ran: map[string]string{}}
}
// Keep runs the shadow until ctx ends, beginning again after backoff(try) whenever it stops: an input it cannot watch
// yet (the candidates' state not made, or the node's grant of it not loaded at a delivery) or a watch that ended is
// tried again rather than leaving the node answering nothing for the rest of the link (review of L2b, round 1, finding
// 2). Each distinct reason is said once.
func (s *Shadow) Keep(ctx context.Context, backoff func(try int) time.Duration) {
said := ""
for try := 1; ctx.Err() == nil; try++ {
began := time.Now()
err := s.Run(ctx)
if ctx.Err() != nil {
return
}
if time.Since(began) > time.Minute {
try = 1 // it ran: a later stop begins the waits again
}
why := "it stopped"
if err != nil {
why = err.Error()
}
if why != said {
s.c.env.Say("this node answers no placement candidate for now, and tries again: " + why)
said = why
}
select {
case <-ctx.Done():
return
case <-time.After(backoff(try)):
}
}
}
// Run answers until ctx ends or an input's watch ends.
func (s *Shadow) Run(ctx context.Context) error {
// Each run watches afresh and starts from nothing read: what changed while it was down (a candidate key deleted)
// is then not missed, and the watches of a run before are not left open (review of L2b, round 2).
ctx, cancel := context.WithCancel(ctx)
defer cancel()
s.mu.Lock()
s.candidates = map[string]map[string]candidateKey{}
s.wants = map[string]delivery.Want{}
s.ran = map[string]string{}
s.answered = map[string]answeredAt{}
s.moved = [len(inputWords)]uint64{}
s.retry = map[string]bool{}
s.c = New(s.c.env)
s.mu.Unlock()
c := s.c
node := c.env.Node
// In inputWords' order.
ins := []input{
{delivery.PlacementCandidatesBucket, ">", nil, nil},
{delivery.PlacementBucket, ">", (*Composer).takePlacement, nil},
{delivery.HaveBucket, "*." + delivery.HaveNodeWord, nil, nil},
{delivery.ModulesBucket, ">", (*Composer).takeModule, nil},
{ResolvedBucket, node + ".>", (*Composer).takeValue, nil},
// What this node runs moved: an apply ends in its module's have.
{delivery.HaveBucket, node + ".>", nil, nil},
// Its wants: a proposal deleting its placement returns it to the controller's compositions at them.
{delivery.AssignerWantsBucket, "want." + node + ".>", nil, nil},
}
type tagged struct {
in int
u agent.Update
ok bool
}
merged := make(chan tagged, 64)
for i, in := range ins {
ch, err := c.env.Watch(ctx, in.bucket, in.pattern)
if err != nil {
return fmt.Errorf("the node answers placement candidates from %s, which cannot be watched: %w", in.bucket, err)
}
go func(i int, ch <-chan agent.Update) {
for u := range ch {
select {
case merged <- tagged{in: i, u: u, ok: true}:
case <-ctx.Done():
return
}
}
select {
case merged <- tagged{in: i}:
case <-ctx.Done():
}
}(i, ch)
}
caught := make([]bool, len(ins))
ready, dirty := false, false
// Not ready after a minute is said, naming each input not read yet, in the journal and in an answer to each proposal it
// knows: a shadow that waits is never silent (issue 1126: ace, g14 and shanks answered nothing to seed #531).
waiting := time.NewTimer(notReadyAfter)
defer waiting.Stop()
var again <-chan time.Time
tries := 0
for {
select {
case <-ctx.Done():
return nil
case <-again:
again = nil
dirty = true
case <-waiting.C:
if !ready {
s.notReady(ctx, ins, caught)
}
continue
case t := <-merged:
if !t.ok {
return fmt.Errorf("the watch of %s ended; the node answers no placement candidate until its agent starts again", ins[t.in].bucket)
}
// Only a change of what an answer is composed from answers again: a key written again with the value it held,
// this node's own key (its answers), or a module's have whose health or words moved but not what it runs, is
// none (issue 1126, 2026-10-28: each look rewrote shanks' unhealthy modules' haves, and each woke a full answer).
changed := false
switch {
case t.u.Caught:
caught[t.in] = true
case t.in == 0:
changed = s.takeCandidate(t.u)
case t.in == 2:
changed = s.takeRecord(t.u)
case t.in == 5:
folded := c.takeModuleHave(t.u)
changed = s.takeRan(t.u) || folded
case t.in == 6:
changed = s.takeWant(t.u)
default:
changed = s.takeChanged(ins[t.in], t.u)
}
if changed {
if t.in != 0 {
s.moved[t.in]++ // the candidates' own changes are their revisions
}
dirty = true
}
if len(merged) > 0 {
continue
}
}
if !ready {
if ready = all(caught); !ready {
continue
}
dirty = true
}
if dirty {
dirty = false
s.answerAll(ctx)
if s.owed {
wait := LaterWait(tries)
tries++
c.env.Say(fmt.Sprintf("this node answers again in %s the placement candidates whose answer met a bus that did not answer", wait))
again = time.After(wait)
} else {
tries, again = 0, nil
}
}
}
}
// takeChanged takes u through in.take and says whether the entry it names changed: a placement, a module or a config
// value written again with the value it held changes nothing composed.
func (s *Shadow) takeChanged(in input, u agent.Update) bool {
before, had, known := s.entry(in.bucket, u.Key)
in.take(s.c, u)
if !known {
return true
}
after, has, _ := s.entry(in.bucket, u.Key)
return had != has || !reflect.DeepEqual(before, after)
}
// entry is what the composer holds of one key of bucket, whether it holds it, and whether the bucket is one it knows.
func (s *Shadow) entry(bucket, key string) (v any, has, known bool) {
c := s.c
c.mu.Lock()
defer c.mu.Unlock()
switch bucket {
case delivery.PlacementBucket:
node, err := delivery.ParsePlacementKey(key)
if err != nil {
return nil, false, true
}
p, ok := c.placements[node]
return p, ok, true
case delivery.ModulesBucket:
name, err := delivery.ParseModulesKey(key)
if err != nil {
return nil, false, true
}
m, ok := c.modules[name]
return m, ok, true
case ResolvedBucket:
rest, ok := strings.CutPrefix(key, c.env.Node+".")
if !ok {
return nil, false, true
}
module, k, ok := strings.Cut(rest, ".")
if !ok {
return nil, false, true
}
val, ok := c.values[module][k]
return val, ok, true
}
return nil, false, false
}
// revsOf is a proposal's keys' revisions, by node, as one word.
func revsOf(keys map[string]candidateKey) string {
nodes := make([]string, 0, len(keys))
for n := range keys {
nodes = append(nodes, n)
}
sort.Strings(nodes)
var b strings.Builder
for _, n := range nodes {
fmt.Fprintf(&b, "%s=%d;", n, keys[n].rev)
}
return b.String()
}
// takeCandidate takes one candidate key, and says whether it changed anything.
func (s *Shadow) takeCandidate(u agent.Update) bool {
proposal, node, err := delivery.ParsePlacementCandidateKey(u.Key)
if err != nil {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
if u.Deleted {
if _, ok := s.candidates[proposal][node]; !ok {
return false
}
delete(s.candidates[proposal], node)
if len(s.candidates[proposal]) == 0 {
delete(s.candidates, proposal)
}
return true
}
c, err := delivery.DecodePlacementCandidate(u.Value)
if err != nil || c.Proposal != proposal || (c.Placement != nil && c.Placement.Node != node) {
// Not one: read as gone.
if _, ok := s.candidates[proposal][node]; ok {
delete(s.candidates[proposal], node)
return true
}
return false
}
if s.candidates[proposal] == nil {
s.candidates[proposal] = map[string]candidateKey{}
}
s.candidates[proposal][node] = candidateKey{c: c, rev: u.Revision}
return true
}
// takeRecord takes a node's record, and says whether it changed: the node's own key changes with each answer written,
// which moves nothing composed.
func (s *Shadow) takeRecord(u agent.Update) bool {
node, _, isNode, err := delivery.ParseHaveKey(u.Key)
if err != nil || !isNode {
return false
}
c := s.c
c.mu.Lock()
before, had := c.records[node]
c.mu.Unlock()
c.takeRecord(u)
c.mu.Lock()
after, has := c.records[node]
c.mu.Unlock()
return had != has || (has && !delivery.SameNodeRecord(before, after))
}
// How long the shadow waits before it says it is not ready, and how long a proposal's answer runs before it is said slow.
var notReadyAfter, slowAfter = time.Minute, time.Minute
// notReady says which inputs the shadow has not read yet, and answers each proposal it knows refused for it.
func (s *Shadow) notReady(ctx context.Context, ins []input, caught []bool) {
var waits []string
for i, in := range ins {
if !caught[i] {
waits = append(waits, in.bucket+" ("+in.pattern+")")
}
}
why := fmt.Sprintf("the node has not read %s after %s; it answers once it has", strings.Join(waits, ", "), notReadyAfter)
s.c.env.Say("this node answers no placement candidate yet: " + why)
s.mu.Lock()
next := map[string]delivery.CandidateAnswer{}
for proposal, keys := range s.candidates {
own, ok := keys[s.c.env.Node]
if !ok {
continue // no key of this node's: nothing asks it to answer
}
next[proposal] = delivery.CandidateAnswer{Proposal: proposal, Revision: own.rev, Refused: why, Verdict: delivery.CandidateRefused}
}
s.answers = next
s.mu.Unlock()
if len(next) > 0 && s.answer != nil {
s.answer(ctx, next)
}
}
// answerAll answers every proposal, the newest candidate first, and hands the answers each time one changes (issue 1126,
// T3: ace answered a closed pull request's group for 22 minutes before the newer one, all handed at the end). Newest
// is the highest revision of the proposal's keys: a withdrawn proposal's candidate is the older and waits; the seat
// drops a closed one's keys, which ends its answer here (a node does not read the forge). Each answer says when it
// starts, every progressEvery while it runs what it waits on, and when it ends (T2: the node said nothing for 22
// minutes).
func (s *Shadow) answerAll(ctx context.Context) {
c := s.c
node := c.env.Node
s.mu.Lock()
type listed struct {
proposal string
rev uint64
at answeredAt
}
var proposals []listed
for g, keys := range s.candidates {
l := listed{proposal: g, at: answeredAt{revs: revsOf(keys), moved: s.moved}}
for _, k := range keys {
l.rev = max(l.rev, k.rev)
}
proposals = append(proposals, l)
}
// A proposal whose keys' revisions and the node's inputs are those its answer stands at is not answered again.
for g := range s.answered {
if _, ok := s.candidates[g]; !ok {
delete(s.answered, g)
}
}
// Each proposal's last answer stands until it is answered again; a proposal whose keys went is dropped.
next := map[string]delivery.CandidateAnswer{}
for _, l := range proposals {
if a, ok := s.answers[l.proposal]; ok {
next[l.proposal] = a
}
}
s.owed = false
s.mu.Unlock()
sort.Slice(proposals, func(i, j int) bool {
if proposals[i].rev != proposals[j].rev {
return proposals[i].rev > proposals[j].rev
}
return proposals[i].proposal < proposals[j].proposal
})
hand := func() {
s.mu.Lock()
same := len(next) == len(s.answers)
for g, a := range next {
if old, ok := s.answers[g]; !ok || !sameAnswer(old, a) {
same = false
}
}
out := make(map[string]delivery.CandidateAnswer, len(next))
for g, a := range next {
out[g] = a
}
if !same {
s.answers = out
}
s.mu.Unlock()
if !same && s.answer != nil {
given := make(map[string]delivery.CandidateAnswer, len(out))
for g, a := range out {
given[g] = a
}
s.answer(ctx, given)
c.env.Say(fmt.Sprintf("%s answered %d placement candidate proposal(s)", node, len(given)))
}
}
for _, l := range proposals {
if ctx.Err() != nil {
return
}
proposal := l.proposal
s.mu.Lock()
at, done := s.answered[proposal]
retry := s.retry[proposal]
s.mu.Unlock()
if done && at == l.at {
continue
}
began := time.Now()
c.env.Say(fmt.Sprintf("answering placement candidate %s (revision %d; woken by %s)", proposal, l.rev, wokenBy(at, done, retry, l.at)))
stop := make(chan struct{})
go func() {
tick := time.NewTicker(progressEvery)
defer tick.Stop()
for {
select {
case <-stop:
return
case <-tick.C:
c.env.Say(fmt.Sprintf("the answer to %s has run %s and waits on %s", proposal,
time.Since(began).Round(time.Second), c.waitingOn()))
}
}
}()
missed := c.missed.Load()
a, ok := s.answerProposal(ctx, proposal)
close(stop)
provisional := c.missed.Load() > missed && ctx.Err() == nil
took := time.Since(began)
if took > slowAfter {
c.env.Say(fmt.Sprintf("the answer to %s took %s", proposal, took.Round(time.Second)))
if ok {
// Words that do not move with the time it took, so an answer the same as before is not written again.
a.Note = strings.TrimPrefix(a.Note+"; ", "; ") + "this answer took over " + slowAfter.String() + " to compose"
}
}
if ok {
a = delivery.FitAnswer(a, delivery.MaxAnswerBytes)
next[proposal] = a
c.env.Say(fmt.Sprintf("answered placement candidate %s in %s: %s", proposal, took.Round(time.Millisecond), a.Verdict))
} else {
delete(next, proposal)
c.env.Say(fmt.Sprintf("placement candidate %s does not place this node: nothing to answer", proposal))
}
s.mu.Lock()
if provisional {
// What the bus did not answer is no answer: handed as it stands, and composed again after a bound.
s.owed = true
s.retry[proposal] = true
delete(s.answered, proposal)
c.env.Say(fmt.Sprintf("the answer to %s met a read on the bus that did not answer: it is answered again", proposal))
} else {
s.answered[proposal] = l.at
delete(s.retry, proposal)
}
s.mu.Unlock()
hand()
}
hand() // a proposal whose keys went, when no proposal is left to answer
}
// How often a running answer says what it waits on.
var progressEvery = time.Minute
// answerProposal is the node's answer to one proposal, and false when the node has nothing to answer: neither the trunk nor
// the proposal places it.
func (s *Shadow) answerProposal(ctx context.Context, proposal string) (delivery.CandidateAnswer, bool) {
c := s.c
node := c.env.Node
defer c.waiting("composing")()
s.mu.Lock()
keys := s.candidates[proposal]
given := make(map[string]*delivery.Placement, len(keys))
for n, k := range keys {
if k.c.Reached && k.c.Placement == nil {
continue // reached, with no key of its own: nothing laid over it
}
given[n] = k.c.Placement
}
// The revision answered is the node's own candidate key's; a node with none is not asked.
own, mine := keys[node]
rev := own.rev
s.mu.Unlock()
c.mu.Lock()
modules := make(map[string]delivery.Module, len(c.modules))
for k, v := range c.modules {
modules[k] = v
}
c.mu.Unlock()
if !mine {
return delivery.CandidateAnswer{}, false
}
var running []delivery.Composed
if c.env.Running != nil {
running = c.env.Running()
}
if p, ok := given[node]; ok && p == nil && !own.c.Reached {
// The proposal deletes this node's placement key: the node then returns to the controller's compositions, as one the
// catalogue does not place (endWhenUnplaced). Those at its wants are compared with what runs (issue 1126, the
// seed's revert, whose check read "would stop running node").
s.mu.Lock()
wants := make(map[string]delivery.Want, len(s.wants))
for m, w := range s.wants {
wants[m] = w
}
s.mu.Unlock()
return returning(ctx, c.get, node, wants, running, proposal, rev), true
}
placement := given[node]
if placement == nil {
c.mu.Lock()
p := c.placements[node]
c.mu.Unlock()
placement = &p
}
// What it runs inside its own composition, of what the candidate places (the seed's trunk places nothing).
c.mu.Lock()
inOwn := c.inOwnOf(*placement)
c.mu.Unlock()
fps := shadowBuilds(running, *placement, modules, inOwn)
kept := map[string]string{}
for module, fp := range fps {
if mod, ok := modules[module]; ok && mod.FP2 == fp {
continue
}
kept[module] = fp // composed from that build (frombus.Scope.KeptAt, issue 1126)
}
// The boxes it runs, by what each module's have and its own key say after a restart (Held), are reused for the
// modules it runs: a value sealed again is no difference.
held := map[string]delivery.Want{agent.NodeModule: {}}
for module := range fps {
held[module] = delivery.Want{FP: fps[module]}
}
c.mu.Lock()
c.wants = held
c.mu.Unlock()
a := frombus.Answer(ctx, reader{c: c}, c.env.Mesh, frombus.Scope{Node: node, KeptAt: kept, Placements: given, InOwn: inOwn,
SameValue: c.sameValue, HeldToday: s.heldToday()}, fps, running, proposal, rev)
s.mu.Lock()
ran := make(map[string]string, len(s.ran))
for m, d := range s.ran {
ran[m] = d
}
s.mu.Unlock()
return asHavesSay(a, ran), true
}
// heldToday is what the node holds today: each module whose have's phase is held, and what its own composer holds, the
// node's own composition ("node") among them when it composes none of its own.
func (s *Shadow) heldToday() map[string]string {
out := map[string]string{}
s.mu.Lock()
for m, how := range s.heldNow {
if how != "" {
out[m] = how
}
}
s.mu.Unlock()
for _, m := range s.c.heldNow() {
if _, said := out[m]; !said {
out[m] = "held"
}
}
return out
}
// takeRan keeps the digest one of this node's module haves says runs, and says whether it changed: a have whose health
// or words alone moved runs the same.
func (s *Shadow) takeRan(u agent.Update) bool {
node, module, isNode, err := delivery.ParseHaveKey(u.Key)
if err != nil || isNode || node != s.c.env.Node {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
before, had := s.ran[module]
h, err := delivery.DecodeHave(u.Value)
if s.heldNow == nil {
s.heldNow = map[string]string{}
}
wasHeld := s.heldNow[module]
s.heldNow[module] = ""
if !u.Deleted && err == nil {
switch {
case h.Phase == delivery.PhaseHeld:
s.heldNow[module] = "held"
case h.Phase == delivery.PhaseFailed:
s.heldNow[module] = "failed"
case h.Phase != delivery.PhaseRemoved && h.FP == "" && h.Build != "":
// A build of the old form: its have names a `build` and no `fp`, which no registered build of the trunk
// names, so the controller keeps it as it runs (issue 1126: novox's de-spiegel at build-…); a seed keeps it so.
s.heldNow[module] = "runs a build of the old form"
}
}
holdMoved := wasHeld != s.heldNow[module] // a hold's start or end changes the answer too
if u.Deleted || err != nil || h.Composed == "" || h.Phase == delivery.PhaseRemoved {
delete(s.ran, module)
return had || holdMoved
}
s.ran[module] = h.Composed
return !had || before != h.Composed || holdMoved
}
// asHavesSay is a with each module that differs only from the copy the agent keeps of the declaration before the switch
// counted the same where the composed digest is the one its have says runs: what runs is the composition at that
// digest, never the kept copy (issue 1126, probe #532: shanks' xterm, zsh, zsh-autosuggestions and
// zsh-syntax-highlighting).
func asHavesSay(a delivery.CandidateAnswer, ran map[string]string) delivery.CandidateAnswer {
if a.Refused != "" || len(a.Differs) == 0 {
return a
}
kept := a.Differs[:0:0]
for _, d := range a.Differs {
if d.Composed != "" && ran[d.Module] == d.Composed {
a.Same = append(a.Same, d.Module)
continue
}
kept = append(kept, d)
}
a.Differs = kept
sort.Strings(a.Same)
a.Verdict = delivery.VerdictOf(a)
return a
}
// shadowBuilds is the build each module placed is composed at in shadow: the one it runs, and for one the candidate
// adds, the trunk's. A module it runs inside its own composition (inOwn, from its haves: mesh-wireguard, one at a build
// of the old form) is given none, so it is folded there again, as the node's own composer folds it, rather than
// composed at the trunk's build as if the candidate added it (issue 1126: nats and photos have a trunk build that no
// want names).
func shadowBuilds(running []delivery.Composed, placement delivery.Placement, modules map[string]delivery.Module, inOwn []string) map[string]string {
fps := map[string]string{}
for _, r := range running {
if r.Module != "" && r.FP != "" {
fps[r.Module] = r.FP
}
}
folded := map[string]bool{}
for _, m := range inOwn {
folded[m] = true
}
for _, m := range placement.Modules {
if _, runs := fps[m.Module]; !runs && !folded[m.Module] {
if mod, ok := modules[m.Module]; ok && mod.FP2 != "" {
fps[m.Module] = mod.FP2
}
}
}
return fps
}
func sameAnswer(a, b delivery.CandidateAnswer) bool {
x, errA := delivery.EncodeHaveNode(delivery.HaveNode{Candidates: map[string]delivery.CandidateAnswer{"g": a}})
y, errB := delivery.EncodeHaveNode(delivery.HaveNode{Candidates: map[string]delivery.CandidateAnswer{"g": b}})
return errA == nil && errB == nil && string(x) == string(y)
}
// takeWant takes one of the node's wants, and says whether it changed.
func (s *Shadow) takeWant(u agent.Update) bool {
module := ""
var w delivery.Want
if _, err := delivery.ParseWantNodeKey(u.Key); err == nil {
module = agent.NodeModule
} else if node, m, err := delivery.ParseWantKey(u.Key); err == nil && node == s.c.env.Node {
module = m
if !u.Deleted {
if w, err = delivery.DecodeWant(u.Value); err != nil || w.Removed || w.FP == "" {
u.Deleted = true
}
}
} else {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
old, had := s.wants[module]
if u.Deleted {
delete(s.wants, module)
return had
}
s.wants[module] = w
return !had || old.FP != w.FP
}
// returning is the node's answer to a proposal deleting its placement: it returns to the controller's compositions, each
// read at the fingerprint its want names in the node's composition bucket and compared with what runs
// (delivery.CompareComposed). A want the controller holds no composition for, or one that cannot be read, refuses the
// answer, naming each: returning then would leave that module with nothing to apply.
func returning(ctx context.Context, get func(ctx context.Context, bucket, key string) ([]byte, error), node string,
wants map[string]delivery.Want, running []delivery.Composed, proposal string, rev uint64) delivery.CandidateAnswer {
a := delivery.CandidateAnswer{Proposal: proposal, Revision: rev}
bucket, err := delivery.ComposedBucketFor(node)
if err != nil {
a.Refused, a.Verdict = err.Error(), delivery.CandidateRefused
return a
}
modules := make([]string, 0, len(wants))
for m := range wants {
modules = append(modules, m)
}
sort.Strings(modules)
comps := map[string]delivery.Composed{}
var missing []string
for _, m := range modules {
var key string
if m == agent.NodeModule {
key, err = delivery.ComposedNodeKey(node)
} else {
key, err = delivery.ComposedKey(node, m, wants[m].FP)
}
if err != nil {
missing = append(missing, m+": "+err.Error())
continue
}
raw, err := get(ctx, bucket, key)
switch {
case err != nil:
missing = append(missing, fmt.Sprintf("%s: the controller's composition %s cannot be read: %v", m, key, err))
continue
case raw == nil:
missing = append(missing, fmt.Sprintf("%s: the controller holds no composition %s, which its want names", m, key))
continue
}
comp, err := delivery.DecodeComposed(raw)
if err != nil {
missing = append(missing, fmt.Sprintf("%s: the controller's composition %s: %v", m, key, err))
continue
}
comps[key] = comp
}
if len(missing) > 0 {
a.Refused = "returning to the controller's compositions, " + strings.Join(missing, "; ")
a.Verdict = delivery.CandidateRefused
return a
}
cmp := delivery.CompareComposed(comps, running)
a.Returns = delivery.ReturnsToController
a.Same, a.Differs, a.Added, a.Removed = cmp.Same, cmp.Differs, cmp.Added, cmp.Removed
a.Verdict = delivery.VerdictOf(a)
return a
}
// sameValue says whether two sealed values open, with this node's key, to the same value; unknown when either does not
// open (or the node holds no opener). Only the comparison leaves: never a value (issue 1126, the shadow shows its work).
func (c *Composer) sameValue(a, b string) (same, known bool) {
if c.env.Unseal == nil {
return false, false
}
x, errA := c.env.Unseal(a)
y, errB := c.env.Unseal(b)
if errA != nil || errB != nil {
return false, false
}
return bytes.Equal(x, y), true
}
@@ -0,0 +1,129 @@
{
"module": "de-spiegel",
"version": "1",
"slug": "spiegel",
"capabilities": [
"container-runtime"
],
"requires": [
"route",
"smtp"
],
"contributes": {
"route": {
"site": {
"label": "de-spiegel",
"port": 35621
},
"praktijk": {
"name": "${config:praktijk-name}",
"port": 35621
},
"praktijk-www": {
"name": "${config:praktijk-www-name}",
"redirect": "https://${config:praktijk-name}"
},
"psychologie": {
"name": "${config:psychologie-name}",
"port": 35621
},
"psychologie-www": {
"name": "${config:psychologie-www-name}",
"redirect": "https://${config:psychologie-name}"
}
},
"smtp": {
"account": "de-spiegel"
}
},
"binds": {
"route": "/var/lib/de-spiegel/route.json",
"smtp": "/var/lib/de-spiegel/smtp.json"
},
"config": {
"praktijk-name": {
"type": "string",
"description": "The public name the practice's site is served on; the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"praktijk-www-name": {
"type": "string",
"description": "The public name that redirects to the practice's site (its www form); the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"psychologie-name": {
"type": "string",
"description": "The public name the psychology practice's site is served on; the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"psychologie-www-name": {
"type": "string",
"description": "The public name that redirects to the psychology practice's site (its www form); the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
}
},
"listens": [
{
"port": 35621,
"protocol": "tcp",
"from": "mesh",
"why": "the de-spiegel site and its /contact endpoint over http; the practice's three public names are route grants, and the proxy reaches every one of them here"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"path": "/var/lib/de-spiegel",
"mode": "0700"
},
{
"id": "server-env",
"type": "file",
"path": "/var/lib/de-spiegel/server.env",
"mode": "0600",
"content": "SMTP_HOST=${bound:smtp:at}\nSMTP_PORT=${bound:smtp:port}\nSMTP_AUTH_USER=de-spiegel@${bound:smtp:domain}\nSMTP_AUTH_PASS=${secret:smtp}\n"
},
{
"id": "net",
"type": "network",
"name": "de-spiegel"
},
{
"id": "server",
"type": "container",
"name": "de-spiegel",
"artifact": "site",
"network": "de-spiegel",
"env-file": [
"/var/lib/de-spiegel/server.env"
],
"ports": [
"35621"
],
"secrets-in-environment": "the application's own code reads SMTP_AUTH_USER/PASS from the environment (server/index.js); converting is this repository's change"
}
],
"build": {
"on": [
{
"arg": "NODE_BASE",
"image": "node@sha256:48e4b67d85f87bd551df43704e24d252f56cc5f8e9718841aace50f19948f0f9"
}
],
"artifacts": [
{
"name": "site",
"kind": "image",
"from": "Dockerfile"
}
]
},
"secrets": {
"smtp": "/var/lib/de-spiegel/smtp.secret"
}
}
+73 -36
View File
@@ -187,38 +187,11 @@ func describeAge(when, now time.Time) string {
// so LoadState is what is read — and it is the thing an interface spanning systemd and OpenRC
// would have had to drop.
func (arch) ServiceState(ctx context.Context, run Runner, unit string) (string, error) {
out, asked := run(ctx, "systemctl", "show",
"--property=LoadState", "--property=ActiveState", "--property=Type",
"--property=RemainAfterExit", "--property=ExecMainStatus", "--", unit)
var load, active, kind, remains, exited string
for _, line := range strings.Split(out, "\n") {
key, value, found := strings.Cut(strings.TrimSpace(line), "=")
if !found {
continue
}
switch key {
case "LoadState":
load = value
case "ActiveState":
active = value
case "Type":
kind = value
case "RemainAfterExit":
remains = value
case "ExecMainStatus":
exited = value
}
u, err := showUnit(ctx, run, unit)
if err != nil {
return "", err
}
switch load {
case "":
// With why, where it said why: an account's manager that could not be reached says so
// here, and nowhere else (novox/hq ADR 0177).
if asked != nil {
return "", fmt.Errorf("the service manager said nothing about %s: %w", unit, asked)
}
return "", fmt.Errorf("the service manager said nothing about %s", unit)
switch u.load {
case "not-found":
return "", fmt.Errorf(
"%s does not exist on this machine. A declaration naming a unit that is not "+
@@ -227,7 +200,7 @@ func (arch) ServiceState(ctx context.Context, run Runner, unit string) (string,
case "masked":
return "", fmt.Errorf("%s is masked, so its state cannot be declared", unit)
case "error", "bad-setting":
return "", fmt.Errorf("%s is installed but its unit file cannot be loaded (%s)", unit, load)
return "", fmt.Errorf("%s is installed but its unit file cannot be loaded (%s)", unit, u.load)
}
// **A one-shot that finished is not stopped.** A unit whose whole job is to apply something
@@ -243,24 +216,88 @@ func (arch) ServiceState(ctx context.Context, run Runner, unit string) (string,
// wrong with.
//
// So for that shape, what "running" means is "it ran, and it worked".
if kind == "oneshot" && remains != "yes" && active == "inactive" {
if exited == "0" || exited == "" {
//
// It cannot be told from one that never ran: systemd says ExecMainStatus=0 and a start timestamp
// of 0 for a one-shot that never ran, and also for one that ran and was unloaded because nothing
// referenced it, then loaded again. So a one-shot declared running that must be seen to have run
// lingers (RemainAfterExit) or is enabled at boot; one declared stopped is judged at rest by the
// apply (ServiceAtRest, novox/hq issue 1126).
if u.oneShotAtRest() {
if u.exited == "0" || u.exited == "" {
return "running", nil
}
return "stopped", nil
}
switch active {
switch u.active {
case "active", "activating", "reloading":
return "running", nil
case "inactive", "failed", "deactivating":
return "stopped", nil
default:
return "", fmt.Errorf(
"the service manager reports %s as %q, which is neither running nor stopped", unit, active)
"the service manager reports %s as %q, which is neither running nor stopped", unit, u.active)
}
}
// ServiceAtRest is whether unit is a one-shot that does not linger and has nothing running now
// (novox/hq issue 1126). Such a unit has no state in which a process is up between its runs, so a
// declaration that it be stopped is satisfied whatever its last run did: there is nothing to stop,
// and asking the manager to stop it changes nothing. A boot-time cleanup a package ships is declared
// this way — enabled at boot, never started by an apply.
func (arch) ServiceAtRest(ctx context.Context, run Runner, unit string) (bool, error) {
u, err := showUnit(ctx, run, unit)
if err != nil {
return false, err
}
return u.oneShotAtRest(), nil
}
// shownUnit is what `systemctl show` says of a unit, as far as its state is read.
type shownUnit struct {
load, active, kind, remains, exited string
}
// oneShotAtRest is a one-shot that does not linger and is not running now.
func (u shownUnit) oneShotAtRest() bool {
return u.kind == "oneshot" && u.remains != "yes" && u.active == "inactive"
}
// showUnit reads a unit's state properties. A manager that said nothing is an error, with why
// where it said why: an account's manager that could not be reached says so here, and nowhere
// else (novox/hq ADR 0177).
func showUnit(ctx context.Context, run Runner, unit string) (shownUnit, error) {
out, asked := run(ctx, "systemctl", "show",
"--property=LoadState", "--property=ActiveState", "--property=Type",
"--property=RemainAfterExit", "--property=ExecMainStatus", "--", unit)
var u shownUnit
for _, line := range strings.Split(out, "\n") {
key, value, found := strings.Cut(strings.TrimSpace(line), "=")
if !found {
continue
}
switch key {
case "LoadState":
u.load = value
case "ActiveState":
u.active = value
case "Type":
u.kind = value
case "RemainAfterExit":
u.remains = value
case "ExecMainStatus":
u.exited = value
}
}
if u.load == "" {
if asked != nil {
return u, fmt.Errorf("the service manager said nothing about %s: %w", unit, asked)
}
return u, fmt.Errorf("the service manager said nothing about %s", unit)
}
return u, nil
}
func (arch) SetServiceState(ctx context.Context, run Runner, unit, state string) error {
verb := "start"
if state == "stopped" {
+45 -41
View File
@@ -5,47 +5,51 @@ import (
"testing"
)
// A one-shot unit that did its work and exited is satisfied, not stopped.
// How a one-shot unit's state is read: what it did, not whether a process is up.
//
// **This made the firewall permanently unsatisfiable.** Its unit loads a rule set and exits, so it
// is inactive the instant it succeeds — the host started it, it worked, the host read back
// "stopped" and reported the machine as not doing what it was told. On every apply, for ever, with
// the rules correctly in place the whole time.
func TestAOneShotThatFinishedIsRunning(t *testing.T) {
run := func(_ context.Context, _ string, _ ...string) (string, error) {
return "LoadState=loaded\nActiveState=inactive\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=0\n", nil
}
state, err := arch{}.ServiceState(context.Background(), run, "nftables.service")
if err != nil {
t.Fatalf("a one-shot that succeeded was an error: %v", err)
}
if state != "running" {
t.Fatalf("a one-shot that did its work reads as %q, so it can never be satisfied", state)
}
}
// And one that failed is still stopped, or the host would report success for work that did not
// happen — which is the opposite mistake and the worse one.
func TestAOneShotThatFailedIsStopped(t *testing.T) {
run := func(_ context.Context, _ string, _ ...string) (string, error) {
return "LoadState=loaded\nActiveState=inactive\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=1\n", nil
}
state, err := arch{}.ServiceState(context.Background(), run, "nftables.service")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if state != "stopped" {
t.Fatalf("a one-shot that failed reads as %q, so a broken firewall reports success", state)
}
}
// A unit that lingers deliberately is unaffected: it says so, and its active state is the answer.
func TestAOneShotThatRemainsIsJudgedByItsActiveState(t *testing.T) {
run := func(_ context.Context, _ string, _ ...string) (string, error) {
return "LoadState=loaded\nActiveState=active\nType=oneshot\nRemainAfterExit=yes\nExecMainStatus=0\n", nil
}
state, _ := arch{}.ServiceState(context.Background(), run, "thing.service")
if state != "running" {
t.Fatalf("a lingering one-shot reads as %q", state)
// **A one-shot that did its work and exited is satisfied, not stopped.** Its unit loads a rule set
// and exits, so it is inactive the instant it succeeds; reading that as stopped made the firewall
// permanently unsatisfiable, on every apply, with the rules correctly in place the whole time.
//
// **One that failed is still stopped**, or the host would report success for work that did not
// happen — the opposite mistake and the worse one.
//
// **At rest** (novox/hq issue 1126, the kernel's boot-time cleanup on ace) is whether a one-shot that does not linger has nothing running now: a declaration
// that it be stopped is then satisfied, whatever it last did, because there is nothing to stop.
func TestAOneShotIsReadByWhatItDid(t *testing.T) {
for _, c := range []struct {
name, show, want string
atRest bool
}{
{"finished", "ActiveState=inactive\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=0\n",
"running", true},
{"failed", "ActiveState=inactive\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=1\n",
"stopped", true},
{"running now", "ActiveState=activating\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=0\n",
"running", false},
{"lingers", "ActiveState=active\nType=oneshot\nRemainAfterExit=yes\nExecMainStatus=0\n",
"running", false},
{"a daemon, stopped", "ActiveState=inactive\nType=simple\nRemainAfterExit=no\nExecMainStatus=0\n",
"stopped", false},
} {
t.Run(c.name, func(t *testing.T) {
run := func(_ context.Context, _ string, _ ...string) (string, error) {
return "LoadState=loaded\n" + c.show, nil
}
state, err := arch{}.ServiceState(context.Background(), run, "thing.service")
if err != nil {
t.Fatalf("read as an error: %v", err)
}
if state != c.want {
t.Errorf("reads as %q, want %q", state, c.want)
}
rest, err := arch{}.ServiceAtRest(context.Background(), run, "thing.service")
if err != nil {
t.Fatalf("at rest read as an error: %v", err)
}
if rest != c.atRest {
t.Errorf("at rest is %v, want %v", rest, c.atRest)
}
})
}
}
+310 -6
View File
@@ -20,6 +20,13 @@
// (VaultKey), the have of the bus module carries the bcrypt hash of every user of the list on the machine, sealed to
// that key (delivery.AccountsHanded), so the vault adopts the accounts it holds no value for. It is sealed again only
// when the list on the machine or the vault's key changes.
//
// **From take-over it follows the grants part's list instead** (ADR 0547 clauses 7 and 8, issue 1945). On the node whose
// compositions carry the list, it also watches the grants part's one key `user-list.sealed`. While that key says
// FollowGrants, the list is the object the key names, read by its name, opened with this node's sealing key and checked
// against the key's digest, and written to the same file (mode 0600) only when the file's digest differs; otherwise the
// controller's compositions are the source, as before. Never both. The have says which it follows and this node's key,
// which the grants part seals to.
package userlist
import (
@@ -36,6 +43,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/seal"
"github.com/novox/mesh-host/internal/agent"
)
@@ -64,6 +72,19 @@ type Env struct {
Write func(ctx context.Context, module string, resource json.RawMessage, body []byte) (digest string, changed bool, err error)
// Report says what was written, in the module's have.
Report func(ctx context.Context, module string, w Written)
// Grants watches the grants part's state (its key delivery.GrantsUserListKey); nil follows the controller's alone.
Grants agent.Watcher
// Fetch reads an object of the grants part's bucket by its name; Open opens a box sealed to this node's key; Key is
// this node's sealing key's public half (SPKI PEM), said in the have so the grants part seals to it ("" unknown).
Fetch func(ctx context.Context, name string) ([]byte, error)
Open func(seal.Box) ([]byte, error)
Key func() string
// Kept is the grants part's key as last heard on this machine and its revision (nil when none), and Keep keeps it:
// what the follower follows while the key cannot be watched (ADR 0547 clause 8). Backoff is the wait before the
// try-th watch again.
Kept func() ([]byte, uint64)
Keep func(value []byte, rev uint64)
Backoff func(try int) time.Duration
// VaultKey is the vault's own public key (SPKI PEM) as its answer to node-key gave it, "" while none is known: the
// key the loaded list's hashes are handed sealed to. Nil hands nothing.
VaultKey func() string
@@ -91,6 +112,13 @@ type Follower struct {
state Written
mod string
said map[string]bool
// grants is the grants part's key as last heard (nil when none), and following the source last taken: FollowGrants
// or FollowController.
grants *delivery.GrantsUserList
grantsRev uint64
following string
// unwatched is why the grants part's key is not watched now; empty while it is, or where it need not be.
unwatched string
}
// New is a follower for a node.
@@ -101,7 +129,18 @@ func New(env Env) *Follower {
if env.Say == nil {
env.Say = func(string) {}
}
return &Follower{env: env, kick: make(chan struct{}, 1), srcs: map[string]source{}, said: map[string]bool{}}
if env.Backoff == nil {
env.Backoff = func(int) time.Duration { return time.Minute }
}
f := &Follower{env: env, kick: make(chan struct{}, 1), srcs: map[string]source{}, said: map[string]bool{}}
if env.Grants != nil && env.Kept != nil {
if raw, rev := env.Kept(); raw != nil {
if g, err := delivery.DecodeGrantsUserList(raw); err == nil {
f.grants, f.grantsRev = &g, rev
}
}
}
return f
}
// Again asks the follower to look at the file again, though no composition changed: after an apply of the bus module,
@@ -128,6 +167,33 @@ func (f *Follower) Run(ctx context.Context) error {
return fmt.Errorf("the user list cannot be followed: %w", err)
}
caught := false
// The grants part's key, on the node whose compositions carry the list: the bus node alone is granted it (mesh-sdk
// buscompose.GrantsUserListFollows). Until its watch caught up nothing is written, so a new follower (one per link)
// never writes the controller's list after take-over before it heard the switch; a watch refused or ended is tried
// again on the backoff, and meanwhile the follow last heard (kept on the machine) holds.
var grants <-chan agent.Update
var retry <-chan time.Time
opened, grantsCaught, tries := false, false, 0
failed := func(why string) {
grants, tries = nil, tries+1
retry = time.After(f.env.Backoff(tries))
f.mu.Lock()
f.unwatched = why
f.mu.Unlock()
f.sayOnce("the grants part's user-list key is not watched, tried again: " + why)
}
open := func() {
if grants != nil || retry != nil || f.env.Grants == nil || !f.busNode() {
return
}
opened = true
ch, err := f.env.Grants.Watch(ctx, delivery.GrantsUserListKey)
if err != nil {
failed(err.Error())
return
}
grants = ch
}
for {
again := false
select {
@@ -135,6 +201,8 @@ func (f *Follower) Run(ctx context.Context) error {
return nil
case <-f.kick:
again = true
case <-retry:
retry = nil
case u, ok := <-updates:
if !ok {
return fmt.Errorf("the watch of the compositions ended")
@@ -144,13 +212,99 @@ func (f *Follower) Run(ctx context.Context) error {
} else {
f.take(u)
}
case u, ok := <-grants:
if !ok {
failed("its watch ended")
break
}
if u.Caught {
grantsCaught, tries = true, 0
f.mu.Lock()
f.unwatched = ""
f.mu.Unlock()
break
}
f.takeGrants(u)
}
if caught {
f.settle(ctx, again)
if !caught {
continue
}
open()
if opened && !grantsCaught && grants != nil {
continue // the switch is not heard yet: nothing is written before it is
}
f.mu.Lock()
if f.unwatched != f.state.Unwatched {
again = true // the have says whether the key is watched
}
f.mu.Unlock()
f.settle(ctx, again)
}
}
// busNode says some composition of this node carries the bus's user list.
func (f *Follower) busNode() bool {
f.mu.Lock()
defer f.mu.Unlock()
return f.cur.key != "" || len(f.srcs) > 0
}
// takeGrants keeps the grants part's key as heard; a value it cannot read is said and leaves the controller's list
// followed (no list is followed that the node cannot check).
func (f *Follower) takeGrants(u agent.Update) {
f.mu.Lock()
defer f.mu.Unlock()
f.grantsRev = u.Revision
if u.Deleted {
f.grants = nil
f.keep(nil, 0)
return
}
g, err := delivery.DecodeGrantsUserList(u.Value)
if err != nil {
f.grants = nil
f.keep(nil, 0)
f.sayOnceLocked("the grants part's user-list key cannot be read, so the controller's list is followed: " + err.Error())
return
}
f.grants = &g
f.keep(u.Value, u.Revision)
}
// keep keeps the key last heard on the machine (Env.Keep), so a follower that cannot watch it follows what it said.
func (f *Follower) keep(value []byte, rev uint64) {
if f.env.Keep != nil {
f.env.Keep(value, rev)
}
}
// Following is which list the follower takes now: the grants part's while its key says so, the controller's otherwise.
func (f *Follower) Following() string {
f.mu.Lock()
defer f.mu.Unlock()
return f.followingLocked()
}
func (f *Follower) followingLocked() string {
if f.grants != nil && f.grants.Follow == delivery.FollowGrants {
return delivery.FollowGrants
}
return delivery.FollowController
}
func (f *Follower) unwatchedNow() string {
f.mu.Lock()
defer f.mu.Unlock()
return f.unwatched
}
func (f *Follower) key() string {
if f.env.Key == nil {
return ""
}
return f.env.Key()
}
// take keeps a composition key's change when it carries the bus's user list, and forgets a key deleted.
func (f *Follower) take(u agent.Update) {
node, module, _, isNode, err := delivery.ParseComposedKey(u.Key)
@@ -200,6 +354,17 @@ func (f *Follower) take(u agent.Update) {
// A source never goes back: a key deleted, or a newer one refused, leaves the list written as it is, since an older
// list would take away accounts a newer one gave.
func (f *Follower) settle(ctx context.Context, again bool) {
f.mu.Lock()
follow := f.followingLocked()
if follow != f.following {
// The switch turned: the other source is written, though neither changed.
again = true
}
f.mu.Unlock()
if follow == delivery.FollowGrants {
f.settleGrants(ctx, again)
return
}
f.mu.Lock()
newest := f.cur
for _, s := range f.srcs {
@@ -215,7 +380,8 @@ func (f *Follower) settle(ctx context.Context, again bool) {
at := fmt.Sprintf("%s@%d", newest.key, newest.rev)
f.mu.Unlock()
w := Written{Source: newest.key, Revision: newest.rev, Wanted: newest.wanted, At: f.env.Now().UTC()}
w := Written{Source: newest.key, Revision: newest.rev, Wanted: newest.wanted, At: f.env.Now().UTC(),
Follows: delivery.FollowController, Key: f.key(), Unwatched: f.unwatchedNow()}
var changed bool
body, err := f.env.Resolve(ctx, newest.resource)
if err == nil {
@@ -248,6 +414,7 @@ func (f *Follower) settle(ctx context.Context, again bool) {
f.mu.Lock()
f.mod, f.state = newest.module, w
f.written = err == nil
f.following = delivery.FollowController
f.mu.Unlock()
if err != nil {
f.sayOnce(fmt.Sprintf("the bus's user list of %s could not be written: %v", at, err))
@@ -256,12 +423,149 @@ func (f *Follower) settle(ctx context.Context, again bool) {
at, w.Users, short(w.Digest)))
}
// Reported when what it says changed: a look again that found the file as it was says nothing new.
if f.env.Report != nil && (changed || prev.Source != w.Source || prev.Revision != w.Revision ||
prev.Digest != w.Digest || prev.Error != w.Error || handedKey(prev.Handed) != handedKey(w.Handed)) {
if f.env.Report != nil && (changed || reportable(prev, w)) {
f.env.Report(ctx, newest.module, w)
}
}
// reportable says a have's user list says something new: a look again that found the file as it was says nothing.
func reportable(prev, w Written) bool {
return prev.Source != w.Source || prev.Revision != w.Revision || prev.Digest != w.Digest || prev.Error != w.Error ||
prev.Follows != w.Follows || prev.Key != w.Key || handedKey(prev.Handed) != handedKey(w.Handed) ||
prev.Unwatched != w.Unwatched
}
// settleGrants writes the grants part's list, the object its key names, into the file the controller's composition of
// the bus module declares, made 0600 (ADR 0547 clause 7): when the key moved, when the switch turned, or when asked.
func (f *Follower) settleGrants(ctx context.Context, again bool) {
f.mu.Lock()
g, rev := *f.grants, f.grantsRev
file := f.cur
for _, s := range f.srcs {
if s.rev > file.rev {
file = s
}
}
if !again && f.following == delivery.FollowGrants && f.state.Revision == rev && f.written {
f.mu.Unlock()
return
}
f.mu.Unlock()
w := Written{Source: delivery.GrantsUserListSource, Revision: rev, Wanted: g.Digest, At: f.env.Now().UTC(),
Follows: delivery.FollowGrants, Key: f.key(), Unwatched: f.unwatchedNow()}
var err error
var changed bool
resource, rerr := mode0600(file.resource)
switch {
case file.key == "" || rerr != nil:
err = fmt.Errorf("no composition of this node declares the bus's user-list file, so the grants part's list has nowhere to be written")
case g.Digest == "":
err = fmt.Errorf("the grants part's key names no list yet; the list on the machine is kept")
case f.env.Fetch == nil || f.env.Open == nil:
err = fmt.Errorf("this engine cannot read or open the grants part's list")
case g.SealedTo != "" && w.Key != "" && g.SealedTo != delivery.SealingKeyID(w.Key):
err = fmt.Errorf("the grants part's list is sealed to another key than this node's (%s); the list on the machine is kept until it seals to this one", g.SealedTo)
}
var body []byte
if err == nil {
var object []byte
if object, err = f.env.Fetch(ctx, g.Object); err == nil {
body, err = delivery.OpenUserList(object, g.Digest, f.env.Open)
}
if err != nil {
err = fmt.Errorf("the grants part's list %s could not be had: %w", short(g.Digest), err)
}
}
if err == nil {
w.Users = Users(body)
if err = f.guard(resource, body); err != nil {
if disk, derr := f.env.OnDisk(resource); derr == nil && disk != nil {
w.Digest = Digest(disk)
}
}
}
if err == nil {
w.Digest, changed, err = f.env.Write(ctx, file.module, resource, body)
} else if w.Digest == "" && file.key != "" && f.env.OnDisk != nil {
if disk, derr := f.env.OnDisk(resource); derr == nil && disk != nil {
w.Digest = Digest(disk)
}
}
if err != nil {
w.Error = err.Error()
}
f.mu.Lock()
prev := f.state
// The grants part's list is not handed to the vault: the handing is of what the controller's list loaded (clause 9).
w.Handed = prev.Handed
f.mod, f.state = file.module, w
f.written = err == nil
f.following = delivery.FollowGrants
f.mu.Unlock()
at := fmt.Sprintf("%s@%d", delivery.GrantsUserListSource, rev)
if err != nil {
f.sayOnce(fmt.Sprintf("the bus's user list of %s could not be written: %v", at, err))
} else if changed {
f.env.Say(fmt.Sprintf("the bus's user list from the grants part (%s, revision %d) is written alone (%d users, %s); the bus reloads it itself",
at, g.Revision, w.Users, short(w.Digest)))
}
if f.env.Report != nil && file.module != "" && (changed || reportable(prev, w)) {
f.env.Report(ctx, file.module, w)
}
}
// Hold is c as the apply may write it (C3 of the switch's review): while the grants part's list is followed, the bus
// module's composition from the controller carries the controller's list, so its user-list resource holds the list on the
// machine instead (made 0600), or is left out when the machine holds none. A list on the machine that cannot be read
// is an error: left out, the apply would remove the file as an orphan and the bus would refuse every connection at its
// next reload, so the bus module's apply waits instead. Any other composition, or the controller's list followed, is c
// unchanged.
func (f *Follower) Hold(c delivery.Composed) (delivery.Composed, error) {
if f.Following() != delivery.FollowGrants {
return c, nil
}
out := c
out.Resources = nil
for _, raw := range c.Resources {
if r, _, _, ok := userList([]json.RawMessage{raw}); ok {
var held json.RawMessage
body, err := f.env.OnDisk(r)
if err != nil {
return c, fmt.Errorf("the user list on the machine could not be read, so the apply waits and the file stays: %w", err)
}
if body != nil {
var m map[string]any
if json.Unmarshal(r, &m) == nil {
delete(m, "digest")
delete(m, "size")
m["content"], m["mode"] = string(body), "0600"
held, _ = json.Marshal(m)
}
}
if held != nil {
out.Resources = append(out.Resources, held)
}
continue
}
out.Resources = append(out.Resources, raw)
}
return out, nil
}
// mode0600 is a user-list resource made mode 0600, whatever its composition said (ADR 0547 clause 7).
func mode0600(raw json.RawMessage) (json.RawMessage, error) {
if len(raw) == 0 {
return nil, fmt.Errorf("no resource")
}
var r map[string]any
if err := json.Unmarshal(raw, &r); err != nil {
return nil, err
}
r["mode"] = "0600"
return json.Marshal(r)
}
// hand is the handing of the list on the machine to the vault: the one before when it is of the same list and key, a
// new one sealed now otherwise, and none while the vault's key is not known or the list holds no hash.
func (f *Follower) hand(loaded []byte, before *delivery.AccountsHanded) *delivery.AccountsHanded {
+177 -5
View File
@@ -33,18 +33,29 @@ func fpOf(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 6
// feed is a watch the test writes to: the keys there before the watch, Caught, then each put. watching is closed once
// the watch began, so a put after it is never handed before the keys there before it.
type feed struct {
ch chan agent.Update
rev uint64
first []agent.Update
ch chan agent.Update
rev uint64
first []agent.Update
// watching is closed once the watch caught up, so a later put is not read before the keys there before it.
watching chan struct{}
once sync.Once
// err refuses the watch, as the bus refuses a watch it grants nothing of.
err error
}
func (f *feed) Watch(context.Context, string) (<-chan agent.Update, error) {
if f.err != nil {
return nil, f.err
}
for _, u := range f.first {
f.ch <- u
}
f.ch <- agent.Update{Caught: true}
close(f.watching)
f.once.Do(func() {
if f.watching != nil {
close(f.watching)
}
})
return f.ch, nil
}
@@ -114,6 +125,15 @@ func comp(t *testing.T, module, fp, content string, signer ed25519.PrivateKey, p
return k, raw
}
// gput is the grants part's key as the bus node hears it at the watch's start (ADR 0547 clauses 7, 8): which list to
// follow, and the list sealed to the node's key; one sealed to another key, or one that does not open to the key's digest.
type gput struct {
follow, content string
otherKey, badDigest bool
// after puts it once the grants part's list was written, as a hand-back comes after a take-over.
after bool
}
type put struct {
module, fp, content string
stranger, deleted bool
@@ -135,6 +155,14 @@ func TestTheBusNodeWritesTheNewestSignedUserListAlone(t *testing.T) {
wantUsers int
wantErr string
wantRefused bool
grants []gput
// grantsRefused refuses the watch of the grants part's key; kept is the key last heard, kept on the machine;
// wantUnwatched says the have then says why it is not watched.
grantsRefused bool
kept *gput
wantUnwatched bool
// wantSource and wantFollows are what the last have says it took the list from and follows.
wantSource, wantFollows string
// vault gives the follower the vault's key; wantHanded is how many users it then hands from the list on the
// machine (ADR 0547 clause 9).
vault bool
@@ -180,6 +208,30 @@ func TestTheBusNodeWritesTheNewestSignedUserListAlone(t *testing.T) {
onDisk: hashed("controller", "a", "b"), wantUsers: 2, wantErr: "holds no user", wantRefused: true, wantHanded: 3},
{name: "a failed write is said and kept", before: []put{{module: "nats", fp: fpOf("a"), content: two}}, failWrite: true,
wantWrites: []string{two}, wantUsers: 3, wantErr: "disk full"},
// ADR 0547 clauses 7, 8: never both lists; the grants part's from take-over, the controller's otherwise.
{name: "from take-over the grants part's list is written into the same file",
before: []put{{module: "nats", fp: fpOf("a"), content: two}}, grants: []gput{{follow: delivery.FollowGrants, content: three}},
wantWrites: []string{three}, wantUsers: 4, wantSource: delivery.GrantsUserListSource, wantFollows: delivery.FollowGrants},
{name: "a hand-back writes the newest controller composition once",
before: []put{{module: "nats", fp: fpOf("a"), content: two}},
grants: []gput{{follow: delivery.FollowGrants, content: three}, {follow: delivery.FollowController, content: three, after: true}},
wantWrites: []string{three, two}, wantUsers: 3, wantSource: "novox.nats." + fpOf("a")[len(delivery.SourcePrefix2):], wantFollows: delivery.FollowController},
{name: "a refused watch with no key kept follows the controller's list and says it is not watched",
before: []put{{module: "nats", fp: fpOf("a"), content: two}}, grantsRefused: true, wantUnwatched: true,
wantWrites: []string{two}, wantUsers: 3, wantSource: "novox.nats." + fpOf("a")[len(delivery.SourcePrefix2):], wantFollows: delivery.FollowController},
{name: "a refused watch keeps the follow last heard, never the controller's list",
before: []put{{module: "nats", fp: fpOf("a"), content: two}}, grantsRefused: true, wantUnwatched: true,
kept: &gput{follow: delivery.FollowGrants, content: three},
wantWrites: []string{three}, wantUsers: 4, wantSource: delivery.GrantsUserListSource, wantFollows: delivery.FollowGrants},
{name: "in shadow the grants part's list is not taken",
before: []put{{module: "nats", fp: fpOf("a"), content: two}}, grants: []gput{{follow: delivery.FollowController, content: three}},
wantWrites: []string{two}, wantUsers: 3, wantSource: "novox.nats." + fpOf("a")[len(delivery.SourcePrefix2):], wantFollows: delivery.FollowController},
{name: "a grants list that does not open to its digest is refused and the file kept",
before: []put{{module: "nats", fp: fpOf("a"), content: two}}, grants: []gput{{follow: delivery.FollowGrants, content: three, badDigest: true}},
onDisk: two, wantErr: "could not be had", wantRefused: true, wantSource: delivery.GrantsUserListSource},
{name: "a grants list sealed to another key is refused and the file kept",
before: []put{{module: "nats", fp: fpOf("a"), content: two}}, grants: []gput{{follow: delivery.FollowGrants, content: three, otherKey: true}},
onDisk: two, wantErr: "sealed to another key", wantRefused: true, wantSource: delivery.GrantsUserListSource},
} {
t.Run(c.name, func(t *testing.T) {
f := &feed{ch: make(chan agent.Update, 64), watching: make(chan struct{})}
@@ -188,7 +240,66 @@ func TestTheBusNodeWritesTheNewestSignedUserListAlone(t *testing.T) {
var writes []string
var reports []Written
reported := make(chan struct{}, 64)
fl := New(Env{Node: node, Composed: f,
gf := &feed{ch: make(chan agent.Update, 64)}
if c.grantsRefused {
gf.err = errors.New("Permissions Violation for Publish to $JS.API.STREAM.INFO.KV_mesh-grants_user-list")
}
var grantsAfter []agent.Update
var keptRaw []byte
sealed := map[string][]byte{}
nodeKey, err := seal.GenerateKeyPair()
if err != nil {
t.Fatal(err)
}
gs := c.grants
if c.kept != nil {
gs = append(append([]gput(nil), gs...), *c.kept)
}
for i, g := range gs {
to := nodeKey.PublicKey
if g.otherKey {
other, _ := seal.GenerateKeyPair()
to = other.PublicKey
}
digest := delivery.UserListDigest([]byte(g.content))
if g.badDigest {
digest = delivery.UserListDigest([]byte("another list"))
}
object, err := delivery.SealUserList([]byte(g.content), to)
if err != nil {
t.Fatal(err)
}
sealed[delivery.UserListObjectName(digest)] = object
raw, err := delivery.EncodeGrantsUserList(delivery.GrantsUserList{Follow: g.follow, Digest: digest,
Object: delivery.UserListObjectName(digest), SealedTo: delivery.SealingKeyID(to), Node: node, Revision: uint64(i + 1)})
if err != nil {
t.Fatal(err)
}
u := agent.Update{Key: delivery.GrantsUserListKey, Value: raw, Revision: uint64(i + 1)}
switch {
case c.kept != nil && i == len(gs)-1:
keptRaw = raw
case g.after:
grantsAfter = append(grantsAfter, u)
default:
gf.first = append(gf.first, u)
}
}
fl := New(Env{Node: node, Composed: f, Grants: gf,
Kept: func() ([]byte, uint64) { return keptRaw, 9 },
Keep: func([]byte, uint64) {},
Backoff: func(int) time.Duration { return time.Hour },
Fetch: func(_ context.Context, name string) ([]byte, error) {
if o, ok := sealed[name]; ok {
return o, nil
}
return nil, errors.New("no object " + name)
},
Open: func(b seal.Box) ([]byte, error) {
out, err := seal.Open(b, nodeKey.PrivateKey)
return []byte(out), err
},
Key: func() string { return nodeKey.PublicKey },
Trusted: func() []ed25519.PublicKey { return []ed25519.PublicKey{controllerPub} },
Resolve: func(_ context.Context, r json.RawMessage) ([]byte, error) {
var res struct{ Content, Digest string }
@@ -254,9 +365,30 @@ func TestTheBusNodeWritesTheNewestSignedUserListAlone(t *testing.T) {
done := make(chan error, 1)
go func() { done <- fl.Run(ctx) }()
<-f.watching
// A later put after the first list is written: the first write waits for the grants part's key to catch up.
if len(c.after) > 0 {
for until := time.Now().Add(5 * time.Second); time.Now().Before(until); time.Sleep(5 * time.Millisecond) {
mu.Lock()
n := len(writes)
mu.Unlock()
if n > 0 {
break
}
}
}
for _, p := range c.after {
f.ch <- update(p)
}
if len(grantsAfter) > 0 {
for until := time.Now().Add(5 * time.Second); ; time.Sleep(5 * time.Millisecond) {
if _, st := fl.State(); st.Source == delivery.GrantsUserListSource || time.Now().After(until) {
break
}
}
for _, u := range grantsAfter {
gf.ch <- u
}
}
if c.again {
// After the first write is reported, so the look again is not folded into it.
select {
@@ -286,6 +418,14 @@ func TestTheBusNodeWritesTheNewestSignedUserListAlone(t *testing.T) {
t.Fatal("a refused list was not reported")
}
}
if c.wantSource != "" {
for until := time.Now().Add(5 * time.Second); ; {
if _, st := fl.State(); st.Source == c.wantSource && (c.wantErr == "" || st.Error != "") || time.Now().After(until) {
break
}
time.Sleep(5 * time.Millisecond)
}
}
time.Sleep(50 * time.Millisecond) // nothing more is written
cancel()
<-done
@@ -295,6 +435,38 @@ func TestTheBusNodeWritesTheNewestSignedUserListAlone(t *testing.T) {
t.Fatalf("written %q, want %q", writes, c.wantWrites)
}
module, st := fl.State()
if c.wantUnwatched != (st.Unwatched != "") {
t.Fatalf("the have says unwatched %q", st.Unwatched)
}
// While the grants part's list is followed, an apply of the controller's composition writes not its list.
if _, raw := comp(t, "nats", fpOf("a"), two, controllerKey, put{}); c.wantFollows != "" {
cmp, err := delivery.DecodeComposed(raw)
if err != nil {
t.Fatal(err)
}
held, err := fl.Hold(cmp)
if err != nil {
t.Fatal(err)
}
_, _, _, hasList := userList(held.Resources)
if want := c.wantFollows == delivery.FollowController; hasList != want || len(held.Resources) < 1 {
t.Fatalf("an apply while following %s holds the controller's list: %v", c.wantFollows, hasList)
}
// A list on the machine that cannot be read stops the apply; it is never left out to be removed.
if c.wantFollows == delivery.FollowGrants {
readErr := New(Env{Node: node, Composed: f, OnDisk: func(json.RawMessage) ([]byte, error) {
return nil, errors.New("permission denied")
}})
readErr.grants = &delivery.GrantsUserList{Follow: delivery.FollowGrants}
if _, err := readErr.Hold(cmp); err == nil || !strings.Contains(err.Error(), "could not be read") {
t.Fatalf("an unreadable list on the machine was held as %v", err)
}
}
}
if c.wantSource != "" && (st.Source != c.wantSource || st.Key != nodeKey.PublicKey ||
(c.wantFollows != "" && st.Follows != c.wantFollows)) {
t.Fatalf("the have says source %q follows %q key %t; want %q %q", st.Source, st.Follows, st.Key == nodeKey.PublicKey, c.wantSource, c.wantFollows)
}
switch {
case c.wantHanded == 0 && st.Handed != nil:
t.Fatalf("handed with no vault key or no hash: %+v", st.Handed)
+42 -13
View File
@@ -49,6 +49,13 @@ func keep(root, name string) error {
return nil
}
// Retire takes a process's build out of the way before another is unpacked where it was: renamed under
// retired/, and deleted at once if nothing runs from it, else at a later sweep. A process nobody judges
// is replaced through it as a judged one is (novox/hq issue 734): deleting the running build before the
// restart deleted the working directory of the process still running from it, and every command it
// started until the restart failed with "getwd: no such file or directory".
func Retire(root, name, dir string, now time.Time) error { return retire(root, name, dir, now) }
// retire takes a build that has no reader any more out of the way: renamed under retired/, and deleted
// at once if nothing runs from it, else at a later sweep. A path that is not there is nothing to do.
func retire(root, name, dir string, now time.Time) error {
@@ -98,8 +105,11 @@ func Sweep(root, name string) ([]string, error) {
return draining, nil
}
// RunningFrom is every process whose executable is a file under dir — renamed there, or deleted from
// there since it started. A variable so a test can name the machine's processes.
// RunningFrom is every process whose executable is a file under dir, or whose working directory is in
// it — renamed there, or deleted from there since it started. A working directory counts as much as an
// image: a process started by the one that runs from a build (a `go version` a builder asks) works in
// the build's directory and runs nothing from it (novox/hq issue 734). A variable so a test can name the
// machine's processes.
var RunningFrom = func(dir string) []int {
return runningFromProc("/proc", dir)
}
@@ -116,14 +126,17 @@ func runningFromProc(proc, dir string) []int {
if err != nil {
continue
}
exe, err := os.Readlink(filepath.Join(proc, e.Name(), "exe"))
if err != nil {
// Gone between the listing and the read, a kernel thread, or another user's process this
// engine may not read — it runs as root, so in practice the first two.
continue
}
if strings.HasPrefix(strings.TrimSuffix(exe, " (deleted)"), dir) {
pids = append(pids, pid)
for _, link := range []string{"exe", "cwd"} {
at, err := os.Readlink(filepath.Join(proc, e.Name(), link))
if err != nil {
// Gone between the listing and the read, a kernel thread, or another user's process this
// engine may not read — it runs as root, so in practice the first two.
continue
}
if strings.HasPrefix(strings.TrimSuffix(at, " (deleted)")+string(filepath.Separator), dir) {
pids = append(pids, pid)
break
}
}
}
sort.Ints(pids)
@@ -140,14 +153,30 @@ func joinPIDs(pids []int) string {
// SweepAll is Sweep for every process the engine keeps anything about: what the witness does on every
// look, so a build retired while its process drained goes once the process has.
//
// Every process the engine keeps a directory for, not only those it judges: a process nobody judges keeps
// nothing but the builds it retired, and those drain the same way (novox/hq issue 734).
func SweepAll(root string) []string {
entries, err := os.ReadDir(filepath.Join(root, ".witness"))
if err != nil {
return nil
}
var draining []string
for _, s := range all(root) {
kept, err := Sweep(root, s.Process)
for _, e := range entries {
if !e.IsDir() {
continue
}
name := e.Name()
kept, err := Sweep(root, name)
if err != nil {
draining = append(draining, fmt.Sprintf("%s: %v", s.Process, err))
draining = append(draining, fmt.Sprintf("%s: %v", name, err))
}
draining = append(draining, kept...)
// A process nobody judges whose last retired build has drained leaves nothing to keep.
if s, err := Load(root, name); err == nil && s.Process == "" && len(kept) == 0 {
_ = os.Remove(retiredDir(root, name))
_ = os.Remove(Dir(root, name))
}
}
return draining
}
+20 -2
View File
@@ -126,8 +126,26 @@ func Save(root string, s State) error {
return os.Rename(tmp.Name(), statePath(root, s.Process))
}
// Forget is a witnessed process no longer declared: everything kept about it goes with it.
func Forget(root, name string) error { return os.RemoveAll(Dir(root, name)) }
// Forget is a process no longer witnessed, or no longer declared: everything kept about it goes with it,
// but for a build a process still runs from. That one is retired and deleted once nothing runs from it
// (draining.go): a process whose witness was taken away, or which never had one, is still running from
// whatever was retired for it until its unit restarts (novox/hq issue 734).
func Forget(root, name string) error {
dir := Dir(root, name)
if _, err := os.Lstat(dir); os.IsNotExist(err) {
return nil
}
if err := retire(root, name, previousDir(root, name), time.Now()); err != nil {
return err
}
if err := os.Remove(statePath(root, name)); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
if draining, err := Sweep(root, name); err != nil || len(draining) > 0 {
return err
}
return os.RemoveAll(dir)
}
// Placing is what the applier is to do with a declared build of a witnessed process.
type Placing struct {
+17
View File
@@ -17,11 +17,28 @@
"consumes": [
"restic.backed-up"
],
"state": [
{
"name": "acts",
"ttl-seconds": 7776000
},
{
"name": "health"
}
],
"config": {
"operator-signers": {
"type": "string",
"description": "The operator's public keys, ed25519 in base64, one a line: the node applies a composition made by hand only when one of these signed it (ADR 0539 clause 3). Public keys, no secret. Empty trusts no operator key.",
"default": ""
},
"build-cpu-share": {
"type": "number",
"description": "The most of this node's processors its builds may take together, in percent (1 to 100): the node-engine holds every build module's runtime of the node in one slice at idle priority under this quota, and each build step sizes its threads from it (novox/hq ADR 0553). Set per node through the node's settings layer; the composer reads it for the build seats' runtimes, the engine's own code does not.",
"minimum": 1,
"maximum": 100,
"default": 50,
"runtime": true
}
},
"resources": [
+12
View File
@@ -68,4 +68,16 @@ func TestTheManifestConsumesTheBackupEvent(t *testing.T) {
if len(m.Consumes) != 1 || m.Consumes[0] != "restic.backed-up" {
t.Fatalf("consumes %v", m.Consumes)
}
// 774 (ADR 0563 §2, §3): the acts of the node seat and the engine's own asks are its declared state acts, the bucket
// mesh-host_acts the grants part creates and each node's engine writes under its own keys.
parsed, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
// 774 (the alert table): beside it the state health, one key per node, the node's own health report, kept until
// written again (no TTL: a node that stops writing keeps its last word, which the watchman reads with its time).
if len(parsed.State) != 2 || parsed.State[0].Name != "acts" || parsed.State[0].TTLSeconds != 90*24*60*60 ||
parsed.State[1].Name != "health" || parsed.State[1].TTLSeconds != 0 {
t.Fatalf("the manifest declares the state %+v", parsed.State)
}
}