Author SHA1 Message Date
mesh-tools a3c098a5c6 Merge pull request 'Pin the parse-only mesh-sdk in the node-engine, so every node can compose design 47's manifests (issue 774)' (#176) from fix/774-new-words-first into main 2026-10-30 00:10:21 +00:00
jschoubben 2fd165f5af Pin the parse-only mesh-sdk 78c9ec8 in place of 6e4a301, so main's engine reads the release's new words and means nothing new by them (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@a3c098a5c618
mesh/delivery-group group fix/774-new-words-first merged: every member is merged
mesh/check pass 2fd165f5af8e on 8c5af3e4a3a3 | unit 9174 run, 546 kept, 51 skipped | integration 9952 run, 659 kept, 60 skipped | over bound: unit, in…
78c9ec8 is sdk main 307b623, whose tree is main's pin dbcaa29, with one commit that parses the new words (a credential's
form, an artifact's bus, a log's max-age-days) and names the new seats; an offer with no form still means pair. No code
of ours changes. Unit and integration suites pass against a local bus and main's catalogue and controller beside: 1260
tests, 7 skipped for a container or a throwaway machine.
2026-10-30 00:39:20 +01:00
jschoubben db1151c773 Pin mesh-sdk 6e4a301 on main's engine, so the controller and the engine read the release's new manifest words before the release uses them (issue 774)
No code of ours changes. Whole unit and integration suites pass at main's code against a local bus: 1260 tests, 7
skipped for a container or a throwaway machine.
2026-10-30 00:25:10 +01:00
mesh-tools 8c5af3e4a3 Merge pull request 'Pin mesh-sdk dbcaa29, so a held module's hold no longer spreads to the modules that read from it (issue 1960)' (#175) from fix/1960-no-hold-spread into main 2026-10-29 15:35:33 +00:00
jschoubben ece607e215 Pin mesh-sdk dbcaa29, so a held module's hold no longer spreads to the modules that read from it (issue 1960)
mesh/delivery merged: source.main moved to novox/mesh-host@8c5af3e4a3a3
mesh/delivery-group group fix/1960-no-hold-spread merged: every member is merged
mesh/check pass ece607e21522 on 1ec73915b807 | unit 3318 run, 1951 kept, 38 skipped | integration 3994 run, 2125 kept, 38 skipped | over bound: unit, …
2026-10-29 16:00:21 +01:00
jschoubben 86a05babed Expect proxy handed beside held de-spiegel: a hold no longer spreads (issue 1960)
mesh-sdk's composer no longer holds a module behind a held one it reads from; the selfcompose test now expects
proxy handed, composed without de-spiegel's routes, and de-spiegel told alone. Needs mesh-sdk pinned at the fix.
2026-10-29 15:58:19 +01:00
mesh-tools 1ec73915b8 Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#174) from revert/774-l3 into main 2026-10-29 08:59:58 +00:00
jschoubben f81e6147ae Revert L3 (mesh-host, merge 0a4c35f): its node-engine removed every held module on novox, the bus server among them, at 08:54 CET on 2026-10-29 (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@1ec73915b807
mesh/delivery-group group revert/774-l3 merged: every member is merged
mesh/check being carried to f81e6147aee1 on 0a4c35f731a7 | from mesh-catalog 1555241393fe | novox/mesh-host#174 in group revert/774-l3
L3 lands again with two fixes: a held module is never removed, and the grants cover L3's new subjects.
2026-10-29 09:42:29 +01:00
mesh-tools 0a4c35f731 Merge pull request 'L3: the node trusts the vault's key, serves the node seat node-engine and keeps its acts on the bus (issue 774, ADR 0561), rebased onto L2' (#168) from feat/774-vault-mints-l2 into main 2026-10-29 07:44:19 +00:00
jschoubben 4f9227a186 Merge main (#171) into L3: pin mesh-sdk c2999a4, mesh-sdk #128's head, which holds L3's sdk and both seed fixes (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@0a4c35f731a7
mesh/delivery-group group feat/774-vault-mints-l2 merged: every member is merged
mesh/check pass 4f9227a1860f on fb8ea0e6838c | unit 1601 run, 5122 kept, 47 skipped | integration 1637 run, 5215 kept, 49 skipped | over bound: unit, …
2026-10-29 01:08:25 +01:00
mesh-tools fb8ea0e683 Merge pull request 'Pin mesh-sdk c2999a4, so a node's seed answer holds a held module as it runs and its neighbours compose (issue 1126)' (#171) from fix/1126-held-as-runs into main 2026-10-29 00:07:06 +00:00
jschoubben a6c70803ab Pin mesh-sdk at c2999a4: a held module no longer spreads its hold to its neighbours (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@fb8ea0e6838c
mesh/delivery-group group fix/1126-held-as-runs merged: every member is merged
mesh/check pass a6c70803ab72 on bb6e47293909 | unit 3054 run, 234 kept, 36 skipped | integration 3075 run, 266 kept, 29 skipped | over bound: unit (sl…
novox's seed p-20261028t231141-ec84b4 refused 11 modules behind de-spiegel's hold and gitea's handed credential. The
node composes and answers through mesh-sdk's frombus, so the fix is the pin.

- The quick fix: follow one more set of words in the answer. It moves the error again.
- The thorough fix, chosen (mesh-sdk fix/1126-held-as-runs): a held module composes only as what runs of it; a consumer
  or collector composes against that and is never held for the held one's stub, and a stub's handed credentials and own
  secrets are left out as they run rather than refused.
2026-10-29 00:50:01 +01:00
jschoubben 9bd1c84e0f Pin mesh-sdk 76f7c4d, the seed fix's own head: the checker refuses a pin to main's merge commit d4c61ee (same tree) as on no pull request (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-delivery@64d364a7686c was judged on trunk fa075dc8…
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 9bd1c84e0f4c on fb8ea0e6838c | novox/mesh-host#168 in group feat/774-vault-mints-l2
2026-10-29 00:20:01 +01:00
jschoubben 1f3fb52470 Merge main into L3: pin mesh-sdk main d4c61ee, which holds L3's sdk and the seed's held-as-today fixes (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 rejected: composed together: the checker's verdict at the members' heads (2199dae4e7d8): ace
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: the check of novox/mesh-host#168 failed: the pin of git.novox.be/novox/mesh-sdk/go at commit d4c61ee8ffd0 is on no pull reques…
2026-10-29 00:11:11 +01:00
mesh-tools bb6e472939 Merge pull request 'A node answers its seed with an old-form build, a failed or held module, and those held behind it, as they run today (issue 1126)' (#170) from fix/1126-seed-oldbuild into main 2026-10-28 23:05:08 +00:00
jschoubben 93f33e2305 Pin mesh-sdk 76f7c4d: a held module keeps its items (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@bb6e47293909
mesh/delivery-group group fix/1126-seed-oldbuild merged: every member is merged
mesh/check pass 93f33e2305de on d2a868f99f05 | unit 3018 run, 270 kept, 36 skipped | integration 3039 run, 302 kept, 29 skipped | over bound: unit (sl…
2026-10-28 23:56:51 +01:00
jschoubben abdf85aa4f Node agent: say how it keeps each module today, held, failed or at a build of the old form named by its have's build; pin mesh-sdk f609be7 (issue 1126) 2026-10-28 23:50:19 +01:00
jschoubben 91686fdb81 Node agent: count a module running a build of the old form as held today, so the seed keeps it as the controller does (issue 1126) 2026-10-28 23:45:28 +01:00
mesh-tools d2a868f99f Merge pull request 'Node agent: say what it holds today in its answer, so a seed holding the same is the same (issue 1126)' (#169) from fix/1126-seed-held into main 2026-10-28 22:20:59 +00:00
jschoubben a143678fd8 Pin mesh-sdk 77b48a4, the group's head (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@d2a868f99f05
mesh/delivery-group group fix/1126-seed-held merged: every member is merged
mesh/check pass a143678fd898 on 4f6b389a37a1 | unit 1199 run, 1015 kept, 33 skipped | integration 1218 run, 1081 kept, 26 skipped | over bound: unit (…
2026-10-28 22:41:14 +01:00
jschoubben 9914459830 Pin mesh-sdk 06b13ec, the group's head (issue 1126)
mesh/delivery-group group fix/1126-seed-held checking: 2 of 3 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 9914459830d7 on 4f6b389a37a1 | novox/mesh-host#169 in group fix/1126-seed-held
2026-10-28 22:18:26 +01:00
jschoubben b7159d71a6 Node agent: say what it holds today in its answer, so a seed that holds the same is the same; pin mesh-sdk b03bf4c (issue 1126) 2026-10-28 22:15:24 +01:00
jschoubben e0c7f36ca9 Pin mesh-sdk 8eafc0c, the L3 group's head on mesh-sdk main with #120 and #121 (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-host@e0c7f36ca90c was judged on trunk d2a868f99f05…
mesh/delivery closed: a newer head of the same pull request
mesh/check checking e0c7f36ca90c on bb6e47293909 | novox/mesh-host#168 in group feat/774-vault-mints-l2
2026-10-28 21:56:06 +01:00
jschoubben 9120f94aa4 Name the seat mesh-placement where the kept declaration's retirement reads the placement, as L2 moved it (issue 774, ADR 0567) 2026-10-28 20:36:41 +01:00
jschoubben 1900b40192 Pin mesh-sdk 4e6e5cc, the L3 group's head rebased on L2 (issue 774) 2026-10-28 20:36:41 +01:00
jschoubben 6cbdb3712e Judgements and refused setuid searches, which any caller may ask, roll in one bounded key of the node's and add no key to the bucket every node shares; a key per act is kept for hand-overs, access acts and searches started (issue 774, ADR 0563 §2; rev-l3 round 2) 2026-10-28 20:35:33 +01:00
jschoubben 2a0c428ef9 The last setuid search's start and each library's own take ask are keys of their own, which no number of other acts moves (issue 774, ADR 0563 §2, §4; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben a5d1f7dde4 From the lease write a directory used as found names the node seat's hand-over as its way out, following the lease the engine hears (issue 774, ADR 0563 §5; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben 95dd64c774 Every signed ask is verified with the controller's key the engine trusts at that ask, so after the lease write none signed by it is taken (issue 774, ADR 0563 §7, ADR 0550 §2; rev-l3 round 1) 2026-10-28 20:35:33 +01:00
jschoubben 81fa009bf4 The node seat's acts and the engine's own asks are the node-engine's declared state acts on the bus: one key each under its own node's keys, written by compare-and-set, one ask open per path made by a create; acts reads one by its id (issue 774, ADR 0563 §2, §3) 2026-10-28 20:35:33 +01:00
jschoubben 05ff90e3c1 The engine judges its node seat's hand-over and take as it judges the controller's ask, answers root-free for its own machine, and asks a library's take itself once (issue 774, ADR 0563 §3, §4, §6) 2026-10-28 20:35:33 +01:00
jschoubben aaffc19232 The node seat node-engine served on the engine's link: its verbs with their proven caller, its acts kept in its own state, a hand-over or access act asked of the operator as node-engine-<node> and performed only on the warrant (issue 774, ADR 0563 §1-§3) 2026-10-28 20:35:33 +01:00
jschoubben 479ac71050 The node trusts the vault's key beside the controller's until the lease is handed away, says what it trusts in its record, asks the vault for its own certificate, composes the vault's authority beside the controller's, and reads the kept declaration no more once every placed module has a have (issue 774, ADR 0550 §2) 2026-10-28 20:35:33 +01:00
mesh-tools 4f6b389a37 Merge pull request 'Node agent: compose from the seat mesh-placement and answer proposals (issue 1126, delivery 2 of 2)' (#167) from feat/1126-placement-seat into main 2026-10-28 18:33:12 +00:00
jschoubben 45f5e27526 Pin mesh-sdk to the group's head 38dbe1d (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@4f6b389a37a1
mesh/delivery-group group feat/1126-placement-seat merged: every member is merged
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
2026-10-28 18:50:39 +01:00
jschoubben dd8ba97989 Node agent: wait for the bus's word that the seat's placement and candidates states were created, then watch them (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben ab22efd8b1 Node agent: follow the placement key at start only and say why when it cannot; test the reached candidate and the switched rule; word the log by the seat's state (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 54aee345fc Node agent: a node that never composed its own takes a failed placement read as not placed; one that did keeps waiting (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 8639b373d2 Prove a missing or refused bucket is a watch error, so a node waits on the controller's compositions and tries again (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 5fc69444f6 Node agent integration tests: candidates keyed by proposal, under the node's real grant on the seat's states (issue 1126) 2026-10-28 18:43:49 +01:00
jschoubben 2fe9d678fc Node agent: answer a proposal's candidate keys and follow the seat's placement, not the indexer's groups (issue 1126)
The answer names its proposal and the revision of the node's own candidate key; a node with no key in a proposal is not asked. The watch for its own placement key is tried again until the seat's state exists, so the node switches when its key is written.
2026-10-28 18:43:49 +01:00
mesh-tools 58b4f8b6bc Merge pull request 'Reconcile a module from the machine's facts at start and on a down edge (issue 1957, ADR 0568)' (#166) from fix/1957-boot-reconcile into main 2026-10-28 17:41:54 +00:00
jschoubben bf667a602d Reconcile a module from the machine's facts at engine start and at each edge of a declared-running resource going down (issue 1957, ADR 0568)
mesh/delivery merged: source.main moved to novox/mesh-host@58b4f8b6bc07
mesh/check being carried to bf667a602d21 on 58b4f8b6bc07 | from 9a8b3870bad0
A reboot left the have running, so a unit that failed to start at boot was never started again and nothing said so. Baseline now asks the machine; the judge's look calls Agent.Reconcile once per down edge, leaving held and unwanted modules alone.
2026-10-28 18:12:35 +01:00
mesh-tools 9a8b3870ba Merge pull request 'The node-engine applies every module but the held ones and marks only those (group fix/1126-apply-culprit, issue 1126)' (#165) from fix/1126-apply-culprit into main 2026-10-28 16:34:48 +00:00
jschoubben 24c23a1d06 A module the node holds alone marks that module, keeps what runs of it, and never refuses the node (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@9a8b3870bad0
mesh/delivery-group group fix/1126-apply-culprit merged: every member is merged
mesh/check pass 24c23a1d066f on a548a61e86d7 | unit 2667 run, 590 kept, 36 skipped | integration 2676 run, 633 kept, 29 skipped | over bound: unit (sl…
frombus.Compose now holds a module refused alone (mesh-sdk 0de3006, the rule
its answer follows too): composed as what runs of it, with what depends on
how it is held held too. The composer hands every other module, keeps what it
handed of a held one, and tells the agent which modules it holds and why;
agent.NotComposedEach marks those alone and clears the rest. Only a refusal
of no single module still marks every module (NotComposed).

heldOut also holds out, of the node's own composition as it runs, the
resources of each module held alone, until a composition of its own is
handed: the node's own apply never removes a module leaving the fold with its
names unset (review of the apply, round 1; issue 1945 by a new path).

Pins mesh-sdk fix/1126-apply-culprit at 0de3006 (a group).
2026-10-28 17:23:26 +01:00
mesh-tools a548a61e86 Merge pull request 'The node-engine reloads the service manager when a unit file or drop-in it writes changes (group fix/shutdown-bound, issue 1126)' (#164) from fix/shutdown-bound into main 2026-10-28 15:05:10 +00:00
jschoubben bc28606d45 apply: reload units only for a unit file actually written or removed (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@a548a61e86d7
mesh/delivery-group group fix/shutdown-bound merged: every member is merged
mesh/check pass bc28606d4566 on 690aa662a837 | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
A skipped or held file comes back on every apply; counting it would reload the
service manager on every apply of a node with a gated or held module. The same
allowlist as MovesRoot: created, updated, corrected, removed.
2026-10-28 15:49:34 +01:00
jschoubben 7f0eb9c51d apply: reload the service manager once when a file in a unit directory moved (issue 1126)
A drop-in that no service reflects, like the power module's bound on
reboot.target, was written but never loaded: the node-engine reloaded units only
for a file named in a service's restart-on, so the next reboot still ran the
target as loaded. Any file written or removed under a system unit directory, or
a drop-in directory below one, now gets one daemon-reload per apply.
2026-10-28 15:43:35 +01:00
mesh-tools 690aa662a8 Merge pull request 'The node's shadow says which input woke each answer (issue 1126)' (#163) from fix/1126-vault-own into main 2026-10-28 14:20:39 +00:00
jschoubben 8bd0b5db15 Say which input woke each placement candidate's answer, so a probe's safety check tells a loop from moving inputs (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@690aa662a837
mesh/delivery-group group fix/1126-vault-own merged: every member is merged
mesh/check pass 8bd0b5db15a3 on b4fd2cc210bf | unit 1639 run, 4201 kept, 46 skipped | integration 1674 run, 4215 kept, 47 skipped | over bound: unit, …
Each answer's line now reads 'answering placement candidate G (revision N; woken by X)', X being its first answer,
a retry after a read the bus did not answer, the candidate, or the inputs that changed (placement, records, modules,
values, haves, wants). The single input counter becomes one per input, which also decides whether a group is answered
again, as before.

Quick fix weighed: a counter word in the line. Thorough fix chosen: one count per input, so the line names exactly what
moved since the group's last answer.
2026-10-28 14:44:27 +01:00
mesh-tools b4fd2cc210 Merge pull request 'A one-shot at rest satisfies a stopped declaration, so the kernel's boot cleanup applies (issue 1126)' (#162) from fix/ace-tun-cleanup into main 2026-10-28 13:34:20 +00:00
jschoubben 1a021052fe Drop the never-ran reading: a start timestamp of 0 does not mean a one-shot never ran (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-host@b4fd2cc210bf
mesh/check being carried to 1a021052fe07 on b4fd2cc210bf | from 7a2af2376405
Review round 1 measured it on systemd 262.1: a finished one-shot nothing references is
unloaded, and loaded again it reads inactive, ExecMainStatus=0 and a start timestamp of
0, as one that never ran does. The previous commit's rule read such a unit stopped, so
a one-shot declared running would be started, read stopped and fail on every apply, and
the hold, the take-over and the giving back would judge it wrongly.

ServiceState keeps main's reading. ServiceAtRest and declaredState stay: they alone
satisfy a stopped declaration for a one-shot at rest, which is the kernel cleanup's
case. A one-shot declared running that must be seen to have run lingers
(RemainAfterExit) or is enabled at boot. Unit tests: the arch table has 5 rows, the
apply test 1; 1262 in all.
2026-10-28 14:29:16 +01:00
jschoubben cf738cdb90 Read a one-shot that never ran as stopped, and one at rest as stopped to a stopped declaration (issue 1126)
The kernel module's boot-time cleanup, linux-modules-cleanup.service (Type=oneshot,
declared stopped and enabled at boot), failed its delivery on ace with 'was asked to
be stopped and is running': ServiceState read every inactive one-shot with
ExecMainStatus=0 as running, and systemd says 0 for one that never ran too. Stopping
an inactive unit changes nothing, so the read back said running again; after a boot
that ran it, the same declaration would fail every apply.

ServiceState now reads ExecMainStartTimestampMonotonic: 0 means never ran, read as
stopped (a package's one-shot declared running is then started, and the hold no
longer counts it as in use). The service apply reads a one-shot at rest as stopped
to a declaration that it be stopped (ServiceAtRest), whatever its last run did.

Quick fix weighed: declare the cleanup running in the catalogue. Thorough chosen:
the reading was wrong for every one-shot, and this is its second fault of the kind
(the firewall's was the first). Unit tests: three arch one-shot tests merged into one
table (7 rows), one apply test (2 rows); 1263 to 1262.
2026-10-28 14:29:16 +01:00
mesh-tools 7a2af23764 Merge pull request 'Pin mesh-sdk f387581: the node composes refusals per module and folded modules' owned paths as the controller does (issue 1126, group fix/1126-seed-8)' (#161) from fix/1126-seed-8 into main 2026-10-28 13:27:09 +00:00
jschoubben daaa1962bd Pin mesh-sdk f387581, the group's head: a module's own refusal refuses that module, contributions in config words, folded modules own their manifest's paths (issue 1126, seed-8)
mesh/delivery merged: source.main moved to novox/mesh-host@7a2af2376405
mesh/delivery-group group fix/1126-seed-8 merged: every member is merged
mesh/check pass daaa1962bdab on df878ba1f780 | unit 3018 run, 234 kept, 36 skipped | integration 3038 run, 266 kept, 29 skipped | over bound: unit (sl…
2026-10-28 14:08:58 +01:00
mesh-tools df878ba1f7 Merge pull request 'The node answers a placement candidate once, retries a read the bus did not answer, and keeps a file's secrets entry by entry (issue 1126; group fix/1126-seed-same-3)' (#160) from fix/1126-seed-same-3 into main 2026-10-28 11:40:38 +00:00
jschoubben 1e01868d92 An unplaced node owes no pass, so no timer is armed again; every wait a test sets is put back when it ends; pin mesh-sdk 8520a4c, whose masking splits only a file's secrets (issue 1126; review of seed-same-3, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@df878ba1f780
mesh/delivery-group group fix/1126-seed-same-3 merged: every member is merged
mesh/check pass 1e01868d92c3 on c3a2ffde6755 | carried from mesh-catalog 774fcdfa93ad | unit 2188 run, 1064 kept, 36 skipped | integration 2197 run, 1…
2026-10-28 12:25:32 +01:00
jschoubben 9d04ae7edd Hold the secrets row to the have read's error, as the loop fix's Env.Held returns it (issue 1126) 2026-10-28 12:13:09 +01:00
jschoubben 16df7d0b7a Merge branch 'fix/1126-answer-once' into fix/1126-seed-same-3 2026-10-28 12:11:50 +01:00
jschoubben cd1d2ff447 A read the bus did not answer is tried again after a bound, never kept as an answer: the shadow's answer, the node's composition and the own-secret step (issue 1126)
Met twice on 2026-10-28: shanks kept 'want.shanks.brother could not be read ... no
responders' as the own-secret step's last word, and ace kept 'the vault's own: not yet
... no responders' as its answer to seed-7, each until another input moved. Every bus
read of the composer (manifests, builds, archives, served values, the vault's own and
credentials, the have of what runs) is noted (BusFailed); an answer or a pass that met
one is made again after a growing bound (5 s to 5 min), and a pass is not handed when
the have of its sealed values did not answer. The own-secret step tells a read or an
ask that did not answer (Retry, taken again) from a key absent or the vault's word.
2026-10-28 12:11:08 +01:00
jschoubben faf6a23d91 Fill go.sum for mesh-sdk 70de942 (issue 1126) 2026-10-28 12:06:25 +01:00
jschoubben 2f4e81f3b0 A file's secrets are kept entry by entry: the box that runs at one secret is reused only there, where it opens to the same value; pin mesh-sdk 70de942 (issue 1126, C1; the seed-7 probe) 2026-10-28 12:05:48 +01:00
jschoubben 4182f8fbfb The node answers a placement candidate revision once: a module's have whose health words alone moved, or a key written again with its value, no longer composes every group's answer again (issue 1126)
On shanks probe/1126-seed-7 was answered at revision 84 every 13-17 s: each look rewrote
the haves of its unhealthy modules with new words, and the shadow took any write of its
own module haves, placement, modules or config values as a change. Each answer loaded
the bus, the health tools timed out, the words moved again. Now only what an answer is
composed from wakes it (what runs, folded modules, a changed value), and each group is
answered again only when its keys' revisions or the node's inputs moved.
2026-10-28 12:02:57 +01:00
mesh-tools c3a2ffde67 Merge pull request 'The placement test's resolved values carry their source, as mesh-config writes them (issue 1126)' (#159) from fix/1126-seed-same-2 into main 2026-10-28 10:08:46 +00:00
jschoubben 6502ba6ad6 The placement test's resolved values carry their source, as mesh-config writes them: a value with no source is laid by neither the controller nor a node composing its own (issue 1126, C2; the group check of 3cb7f74b347d)
mesh/delivery merged: source.main moved to novox/mesh-host@c3a2ffde6755
mesh/check being carried to 6502ba6ad6e6 on c3a2ffde6755 | from f6c0520f26b9
2026-10-28 10:47:48 +01:00
mesh-tools f6c0520f26 Merge pull request 'Node shadow: reuse a seal only at its slot, lay values as the controller does, newest candidate first, and hand the vault a drifted own secret (issue 1126, group fix/1126-seed-same)' (#158) from fix/1126-seed-same into main 2026-10-28 09:46:47 +00:00
jschoubben c912bcab6e The node agent finds a drifted own secret's running file by its id as the node runs it, its module's name first (issue 1126, cause C4; review round 2)
mesh/delivery merged: source.main moved to novox/mesh-host@f6c0520f26b9
mesh/delivery-group group fix/1126-seed-same merged: every member is merged
mesh/check checking c912bcab6e73 on f6c0520f26b9 | novox/mesh-host#158 in group fix/1126-seed-same
Pins mesh-sdk 3028c7e (delivery.OwnNeedName). The fixtures carry the id shape captured from novox's shadow answer.
2026-10-28 10:01:38 +01:00
jschoubben 63c0b445d7 Fill go.sum for mesh-sdk 1062b33, the group's head (issue 1126) 2026-10-28 09:53:34 +01:00
jschoubben 653ff61662 The node agent hands the vault what runs of an own secret its shadow answer says another value of, through move-in's replace_running, while it applies the controller's compositions (issue 1126, cause C4)
A new step, internal/ownresync, after each shadow answer: for each own secret masked as another value, while the node
applies the controller's compositions and runs the one its want names, it asks the vault's move-in with
replace_running: true and the running value sealed to the call's key, once per value, and says what it asked and what
the vault answered. The answer written to the node's have key carries values_differ. Pins the group's mesh-sdk head
1062b33 (go.sum is filled once that head is pushed).
2026-10-28 09:51:48 +01:00
jschoubben a7cd0a7807 Node shadow: reuse a running seal only at its own slot, lay values as the controller does, answer the newest candidate first and say how it goes (issue 1126, C1 C2, T1 T2 T3)
A box is reused by slot (resource, field) after the composition, from what runs or the
have's slots; a have written before slots is reused within its module alone. Config
values are laid by mesh-sdk's one rule (never a default). The shadow answers the
newest candidate first and hands each answer as it lands, says when an answer starts,
every minute what it waits on, and when it ends, and fits an answer to the node's key.
Needs mesh-sdk fix/1126-seed-same (16b8d8a1).
2026-10-28 09:41:29 +01:00
mesh-tools 7271cf151b Merge pull request 'Node shadow: what runs is the composition at the have's digest, not the stale kept copy (issue 1126, group fix/1126-shadow-bodies)' (#157) from fix/1126-shadow-bodies into main 2026-10-28 07:46:33 +00:00
jschoubben 82d0e125b2 What runs is the composition the have names, never the copy of the declaration kept before the switch (issue 1126, probe #532: shanks' xterm, zsh and its plugins answered differing at the very digest their haves say runs)
mesh/delivery merged: source.main moved to novox/mesh-host@7271cf151b3a
mesh/delivery-group group fix/1126-shadow-bodies merged: every member is merged
mesh/check checking 82d0e125b2c0 on 7271cf151b3a | novox/mesh-host#157 in group fix/1126-shadow-bodies
2026-10-28 01:42:17 +01:00
jschoubben a4c24a1979 Pin mesh-sdk 362a3f1: a refusal names what another node cannot run here, and is answered without a composition per module (issue 1126, probe #532) 2026-10-28 01:40:18 +01:00
mesh-tools cff69893a3 Merge pull request 'Node shadow: decides 'same value' by opening seals in memory, says when it is not ready or slow, carries the bus membership (issue 1126, group fix/1126-shadow-shows)' (#156) from fix/1126-shadow-shows into main 2026-10-28 00:21:51 +00:00
jschoubben 047237c911 The shadow shows its work: differing resources with masked bodies and whether a sealed value is the same, a minute not ready or a slow answer said in the journal and the answer; pin mesh-sdk 4e0c577 (issue 1126, seed #531)
mesh/delivery merged: source.main moved to novox/mesh-host@cff69893a361
mesh/delivery-group group fix/1126-shadow-shows merged: every member is merged
mesh/check pass 047237c911a4 on 449cda1b003e | unit 2980 run, 270 kept, 36 skipped | integration 2999 run, 302 kept, 29 skipped | over bound: unit (sl…
ace, g14 and shanks answered nothing to seed #531 and said nothing either. The
shadow now says, after a minute not ready, which inputs it has not read, and
answers each group it knows refused for that; an answer running over a minute
is said while it runs and noted in the answer. The answer's differing resources
carry both bodies, masked, the node saying whether two sealed values open to the
same value (Composer.sameValue: only the comparison leaves the node).
2026-10-28 01:11:55 +01:00
mesh-tools 449cda1b00 Merge pull request 'Node agent: a switch never marks the new run's modules, a failed read or a bus pause keeps what runs, stale composer marks clear (issue 1126, group fix/1126-switch-safe)' (#155) from fix/1126-switch-safe into main 2026-10-27 23:52:28 +00:00
jschoubben a6e53a4858 A composer's mark never covers another cause, and a run that ended marks and clears nothing (issue 1126, review of the switch, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@449cda1b003e
mesh/delivery-group group fix/1126-switch-safe merged: every member is merged
mesh/check pass a6e53a485813 on 199f6b48b1df | unit 2992 run, 1008 kept, 36 skipped | integration 3025 run, 1102 kept, 29 skipped | over bound: unit (…
NotComposed marked from a read have whatever its phase: an apply that failed,
naming the want's build and digest, became a composer's mark, which baseline
then took for what runs, and never applied again. It now marks only a have that
runs, says nothing, or is a composer's mark already. And its loop, whose reads
can take seconds, checks the run's context at each module, so a run that ended
writes no have.
2026-10-28 00:41:25 +01:00
jschoubben 4993f1b1d7 Make the switch between composers safe: a composer tells only its own run's agent, a pass whose run ended refuses nothing, a bus that pauses is tried again, a mark keeps what runs and clears at the next start; pin mesh-sdk 7359d18 (issue 1126)
Live 2026-10-28: the own run's last pass, failing on its run's end, marked
every module of the controller-mode agent that replaced it (ace 60, shanks 63),
and the marks dropped the build and digest because the have read failed in the
bus's reload pause. Now: the refusal hook is bound to the run's agent and ctx;
a pass whose context ended returns nothing; JetStream not enabled or no
responders are tried again after 1, 2 and 4 s before any mark; NotComposed reads
the have again while the bus does not answer and marks nothing it cannot read;
baseline takes a have a composer marked as what runs, so a mark it did not make
clears; each run's shadow lives with the run.
2026-10-28 00:34:10 +01:00
mesh-tools 199f6b48b1 Merge pull request 'Node agent: a failed placement read picks no owner, a deleted placement returns the node to the controller without a restart (issue 1126, group fix/1126-shadow-all)' (#154) from fix/1126-shadow-all into main 2026-10-27 22:44:12 +00:00
jschoubben 4636261435 A failing placement read chooses neither composer, a deleting head is compared with the controller's compositions at the node's wants, and a placement watch that ends is said; pin mesh-sdk e32d777 (issue 1126, review of the shadow, round 1)
mesh/delivery merged: source.main moved to novox/mesh-host@199f6b48b1df
mesh/delivery-group group fix/1126-shadow-all merged: every member is merged
mesh/check checking 463626143567 on bb15aab50eda | novox/mesh-host#154 in group fix/1126-shadow-all
selfComposer took a KVHistory error for 'not placed': at any restart a timeout
or a bus not ready flipped a self-composing node to the controller's
compositions for a whole run. selfcompose.Placed chooses only on a read that
succeeds (a missing placement state is one), and tries again otherwise, said
once. The shadow's answer to a head deleting the placement reads the
controller's composition at each want in the node's bucket and compares it with
what runs; a want with none refuses, without returning.
2026-10-27 23:37:59 +01:00
jschoubben 4e8fdfdf04 The way back: a node whose placement key is deleted returns to the controller's compositions without a restart, says so in its journal and its have, and answers such a candidate as returning; pin mesh-sdk 472d5f3 (issue 1126, the seed's revert)
The seed's revert (mesh-catalog #526) was refused on 'would stop running node',
while a self-composing node would have refused its placement's absence, kept
what runs and marked every have failed until a restart. endWhenUnplaced ends the
agent's run when the key goes, so it begins again on the controller's
compositions, as at a start without a key; the composer refuses nothing while
unplaced; the node's have says whose compositions it applies (Composes); the
shadow answers a deleted placement Returns, verdict same.
2026-10-27 23:25:59 +01:00
jschoubben d00926e089 Compose with what the node runs, so a folded module of an old-form build is carried as it runs, and bound an archive read; pin mesh-sdk 2fd5ffd (issue 1126, the seed's shadow)
novox refused its own composition and its answer on de-spiegel, a module it runs
folded into its own composition at a build only the controller holds. The
composer now gives frombus what the node runs (Scope.Running), so such a module
is carried as it runs. The shadow's answer names every refusal (mesh-sdk). A
templates archive read is bounded at 30 s: nothing else held an unanswered store.
2026-10-27 23:19:40 +01:00
mesh-tools bb15aab50e Merge pull request 'Node agent: compose its own modules from their builds and providers' served keys under its own grant (issue 1126, group fix/1126-node-creds)' (#153) from fix/1126-node-creds into main 2026-10-27 20:32:37 +00:00
jschoubben 6a4718b712 Hold in a unit table which haves count a module as run inside the node's own composition, so a failed have naming nothing is caught without H8's timing (issue 1126, review round 2)
mesh/delivery merged: source.main moved to novox/mesh-host@bb15aab50eda
mesh/delivery-group group fix/1126-node-creds merged: every member is merged
mesh/check checking 6a4718b712b2 on bb15aab50eda | novox/mesh-host#153 in group fix/1126-node-creds
2026-10-27 21:29:51 +01:00
jschoubben d5adf9b72b Drop two debugging lines from the build read and H8, and say why the refused read's wait stays above the history bound (issue 1126, review round 1) 2026-10-27 21:21:19 +01:00
jschoubben 7365890298 Prove in H8 that a node reads a placed module's build and a provider's served key through its own session, refused without the grant, and take a failed have naming nothing as no module run inside the node's own; pin mesh-sdk 7a8e9f9 (issue 1126, review round 1)
H8 found it: a module the node could not compose yet got a failed have naming
no fingerprint, which counted as run inside the node's own composition, so the
next composition took it from its manifest past its build.
2026-10-27 21:18:45 +01:00
jschoubben 0862f97972 Pin mesh-sdk 538ffba, the group's head (issue 1126) 2026-10-27 21:04:02 +01:00
jschoubben 45439fd742 Compose each module from its build and read what providers elsewhere serve, pinning mesh-sdk cebc651, so a self-composing node hands its moved modules and settles mailu's domain (issue 1126)
The node read a behind module's source manifest digest and composed every module
from the indexer's raw manifest; frombus now takes the build itself (Scope.KeptAt,
a fingerprint). The reader gives frombus the builder's key, a templates archive
fetched as every archive is (apply.Fetch) once per digest and checked against it,
and another node's served keys one by one, as the node's grant allows them.
2026-10-27 21:01:16 +01:00
28 changed files with 4107 additions and 487 deletions
+1 -1
View File
@@ -69,7 +69,7 @@ func TestAModuleWithNothingLongRunningIsOkWithinALookOfItsApply(t *testing.T) {
applying := fact applying := fact
applying.Phase = string(delivery.PhaseApplying) applying.Phase = string(delivery.PhaseApplying)
queue.NoteModule(ctx, applying) queue.NoteModule(ctx, applying)
go judgeWhatRuns(ctx, j, queue, func(string) {}, nil) go judgeWhatRuns(ctx, j, queue, func(string) {}, nil, nil)
// The facts look inside the apply, while the module is still applying. // The facts look inside the apply, while the module is still applying.
queue.SayFacts(ctx, link.Report{Health: &link.Health{Contract: link.LivenessContract, At: time.Now().UTC()}}) queue.SayFacts(ctx, link.Report{Health: &link.Health{Contract: link.LivenessContract, At: time.Now().UTC()}})
if h, _ := bus.have("n1.xorg"); h.Health != delivery.HealthUnknown { if h, _ := bus.have("n1.xorg"); h.Health != delivery.HealthUnknown {
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"context"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/liveness"
)
// heldBy is the containers an open maintenance window holds still, by runtime name (ADR 0189), at now.
func heldBy(windows *apply.Windows) func(time.Time) map[string]bool {
return func(now time.Time) map[string]bool {
held := map[string]bool{}
for _, w := range windows.Open(now) {
for _, name := range w.Holds {
held[name] = true
}
}
return held
}
}
// inactiveIn is what the agent asks at its start of a module it believes runs (ADR 0568 §1): whether the machine shows a
// long-running resource the composition declares running not active, read from the service manager and the container
// runtime and from no record. A resource held by an open maintenance window, or waiting for a library, is not counted;
// one that cannot be read is not counted either (unknown is not inactive), and neither is one the manager is restarting.
func inactiveIn(rt liveness.Runtime, held func(time.Time) map[string]bool) func(context.Context, delivery.Composed) bool {
return func(ctx context.Context, c delivery.Composed) bool {
d, err := declarationOf(c)
if err != nil {
return false
}
var holding map[string]bool
if held != nil {
holding = held(time.Now())
}
return anyInactive(ctx, rt, liveness.LongRunning(d, withWaiting(nil, apply.WaitingOnLibrary(d))), holding)
}
}
// anyInactive reads each group of resources once (the containers, each service manager's units) and says whether any is
// not active.
func anyInactive(ctx context.Context, rt liveness.Runtime, resources []liveness.Resource, held map[string]bool) bool {
type group struct {
kind, scope, user string
targets []string
}
var groups []*group
at := map[string]*group{}
for _, r := range resources {
if r.Kind == liveness.KindContainer && held[r.Target] {
continue
}
g := &group{kind: r.Kind, scope: r.Scope, user: r.User}
if r.Kind == liveness.KindContainer {
g.scope, g.user = "", ""
}
key := g.kind + "/" + g.scope + "/" + g.user
if r.Kind != liveness.KindContainer {
key = "units/" + g.scope + "/" + g.user
}
if at[key] == nil {
at[key] = g
groups = append(groups, g)
}
at[key].targets = append(at[key].targets, r.Target)
}
for _, g := range groups {
var got map[string]liveness.Observed
var err error
if g.kind == liveness.KindContainer {
got, err = rt.Containers(ctx, g.targets)
} else {
got, err = rt.Units(ctx, g.scope, g.user, g.targets)
}
if err != nil {
continue
}
for _, t := range g.targets {
if o := got[t]; !o.Found || !(o.Running || o.Restarting) {
return true
}
}
}
return false
}
+89
View File
@@ -0,0 +1,89 @@
package main
import (
"context"
"encoding/json"
"errors"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/liveness"
)
// 1957/AC2 and AC4 (unit; ADR 0568 §2, §4): the judge's look acts on a resource that goes down, once per edge. A repeated
// statement acts nothing, the edge clears when the resource is healthy again, and a resource held, restarting, starting or
// missing (the other path's) is no edge.
func testOnlyAResourceNewlyDownIsAnEdge(t *testing.T) {
state := func(id, s, reason string, missing bool) liveness.State {
return liveness.State{Resource: liveness.Resource{Module: "dnsmasq", ID: id}, State: s, Reason: reason, Missing: missing}
}
down := state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonDown, false)
steps := []struct {
name string
st []liveness.State
want []string
}{
{"down: an edge", []liveness.State{down}, []string{"dnsmasq"}},
{"said again while down: none", []liveness.State{down}, nil},
{"healthy again: clears", []liveness.State{state("dnsmasq.unit", liveness.Healthy, "", false)}, nil},
{"down again: an edge", []liveness.State{down}, []string{"dnsmasq"}},
{"held by a window: none", []liveness.State{state("dnsmasq.unit", liveness.Held, "", false)}, nil},
{"restarting: none", []liveness.State{state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonRestarting, false)}, nil},
{"starting: none", []liveness.State{state("dnsmasq.unit", liveness.Starting, "", false)}, nil},
{"missing: the other path's", []liveness.State{state("dnsmasq.unit", liveness.Unhealthy, liveness.ReasonDown, true)}, nil},
}
var last map[string]bool
for _, s := range steps {
var got []string
got, last = newlyInactive(last, liveness.Statement{Resources: s.st})
if len(got) != len(s.want) || (len(got) == 1 && got[0] != s.want[0]) {
t.Fatalf("%s: acted on %v, want %v", s.name, got, s.want)
}
}
}
type fakeRuntime struct {
units map[string]liveness.Observed
containers map[string]liveness.Observed
err error
}
func (f fakeRuntime) Containers(_ context.Context, _ []string) (map[string]liveness.Observed, error) {
return f.containers, f.err
}
func (f fakeRuntime) Units(_ context.Context, _, _ string, _ []string) (map[string]liveness.Observed, error) {
return f.units, f.err
}
// 1957/AC1 (unit; ADR 0568 §1): the machine's facts say a declared-running resource is not active when its unit is stopped
// or unknown to the manager; everything up, a restart in progress, a read that failed and a container held by a
// maintenance window are not.
func testTheMachineShowsAResourceNotActive(t *testing.T) {
svc := []byte(`{"id":"dnsmasq.unit","type":"service","unit":"dnsmasq.service","state":"running","module":"dnsmasq"}`)
ctr := []byte(`{"id":"dnsmasq.box","type":"container","name":"box","image":"x@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","module":"dnsmasq"}`)
up := liveness.Observed{Found: true, Running: true}
for _, c := range []struct {
name string
rt fakeRuntime
held map[string]bool
want bool
}{
{"all up", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": up}}, nil, false},
{"unit inactive", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": {Found: true}}, containers: map[string]liveness.Observed{"box": up}}, nil, true},
{"unit unknown to the manager", fakeRuntime{containers: map[string]liveness.Observed{"box": up}}, nil, true},
{"unit restarting", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": {Found: true, Restarting: true}}, containers: map[string]liveness.Observed{"box": up}}, nil, false},
{"container not running", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": {Found: true}}}, nil, true},
{"container held by a window", fakeRuntime{units: map[string]liveness.Observed{"dnsmasq.service": up}, containers: map[string]liveness.Observed{"box": {Found: true}}}, map[string]bool{"box": true}, false},
{"unread", fakeRuntime{err: errors.New("the service manager could not be read")}, nil, false},
} {
t.Run(c.name, func(t *testing.T) {
d, err := declarationOf(delivery.Composed{Node: "n1", Module: "dnsmasq", Resources: []json.RawMessage{svc, ctr}})
if err != nil {
t.Fatal(err)
}
if got := anyInactive(context.Background(), c.rt, liveness.LongRunning(d, nil), c.held); got != c.want {
t.Fatalf("inactive = %v, want %v", got, c.want)
}
})
}
}
+41 -11
View File
@@ -1046,15 +1046,7 @@ func runLink(ctx context.Context, opts options) error {
say(err.Error()) say(err.Error())
} }
windows := apply.WindowsIn(filepath.Dir(opts.state)) windows := apply.WindowsIn(filepath.Dir(opts.state))
j.HeldNow = func(now time.Time) map[string]bool { j.HeldNow = heldBy(windows)
held := map[string]bool{}
for _, w := range windows.Open(now) {
for _, name := range w.Holds {
held[name] = true
}
}
return held
}
judging = j judging = j
// And which units its service managers say failed, and whose each is (novox/hq issue 315). // And which units its service managers say failed, and whose each is (novox/hq issue 315).
unitJudge = units.New(units.Exec{Run: apply.ExecRunner, System: builtFor}) unitJudge = units.New(units.Exec{Run: apply.ExecRunner, System: builtFor})
@@ -1173,6 +1165,10 @@ func runLink(ctx context.Context, opts options) error {
if ag := current.get(); ag != nil { if ag := current.get(); ag != nil {
ag.Retry(modules...) ag.Retry(modules...)
} }
}, func(modules ...string) {
if ag := current.get(); ag != nil {
ag.Reconcile(modules...)
}
}) })
} }
// The node follows its wants: one module at a time from the composition its want names (novox/hq ADR 0539), and the // The node follows its wants: one module at a time from the composition its want names (novox/hq ADR 0539), and the
@@ -1336,8 +1332,8 @@ const (
// the bus when a state changed, again every minute while one is not healthy, and every five minutes // the bus when a state changed, again every minute while one is not healthy, and every five minutes
// anyway. A statement the bus did not take is said at the next look. It reads; it never acts (ADR 0240 // anyway. A statement the bus did not take is said at the next look. It reads; it never acts (ADR 0240
// rule 6) — but for going on with a take act stopped at its bound (ADR 0328 §7), which runs beside it. // rule 6) — but for going on with a take act stopped at its bound (ADR 0328 §7), which runs beside it.
func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session, say link.Announce, reapply func(modules ...string)) { func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session, say link.Announce, reapply, reconcile func(modules ...string)) {
var missingBefore map[string]bool var missingBefore, inactiveBefore map[string]bool
ticker := time.NewTicker(liveness.LookEvery) ticker := time.NewTicker(liveness.LookEvery)
defer ticker.Stop() defer ticker.Stop()
var lastSaid time.Time var lastSaid time.Time
@@ -1439,6 +1435,14 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Session,
say(fmt.Sprintf("applying %s again: a resource it declares is not on this machine", strings.Join(m, ", "))) say(fmt.Sprintf("applying %s again: a resource it declares is not on this machine", strings.Join(m, ", ")))
reapply(m...) reapply(m...)
} }
// A resource its module declares running that is down but still known to the manager is started by the same
// apply, once each time it goes down, never on a clock (ADR 0568 §2): the judge reads, the agent acts.
var inactive []string
inactive, inactiveBefore = newlyInactive(inactiveBefore, st)
if len(inactive) > 0 && reconcile != nil {
say(fmt.Sprintf("applying %s again: a resource it declares running is not active", strings.Join(inactive, ", ")))
reconcile(inactive...)
}
} }
if queue.SayHealth(ctx, *withAccesses(withFoundDirectories(withAccounts(withUnits(healthAsReported(st, netSt), if queue.SayHealth(ctx, *withAccesses(withFoundDirectories(withAccounts(withUnits(healthAsReported(st, netSt),
unitSt), accountSt), found), accessSt)) { unitSt), accountSt), found), accessSt)) {
@@ -2134,6 +2138,32 @@ func newlyMissing(last map[string]bool, st liveness.Statement) ([]string, map[st
return missingModules(fresh), now return missingModules(fresh), now
} }
// newlyInactive are the modules of the resources a statement says down (not running after their grace, but still known to
// their manager, and not restarting) that the last statement acted on did not, each once, sorted, and the resources down
// now: a resource still down is not acted on again, and one that came back and went down again is (ADR 0568 §2).
func newlyInactive(last map[string]bool, st liveness.Statement) ([]string, map[string]bool) {
now := map[string]bool{}
var fresh liveness.Statement
for _, r := range st.Resources {
if r.State != liveness.Unhealthy || r.Reason != liveness.ReasonDown || r.Missing {
continue
}
now[r.ID] = true
if !last[r.ID] {
fresh.Resources = append(fresh.Resources, r)
}
}
return missingModules(withMissing(fresh)), now
}
// withMissing is the statement with each of its resources marked missing, so missingModules lists their modules.
func withMissing(st liveness.Statement) liveness.Statement {
for i := range st.Resources {
st.Resources[i].Missing = true
}
return st
}
// recordKeys reads the public halves of the node's keys and the certificate the mesh issued for its internal name from // recordKeys reads the public halves of the node's keys and the certificate the mesh issued for its internal name from
// its state directory (novox/hq ADR 0555 §3): what the node's record says of them. A key that cannot be read is left // its state directory (novox/hq ADR 0555 §3): what the node's record says of them. A key that cannot be read is left
// empty, never invented; the certificate is there only after the first declaration that carried it. // empty, never invented; the certificate is there only after the first declaration that carried it.
+165 -37
View File
@@ -20,6 +20,7 @@ import (
"github.com/novox/mesh-host/internal/agent" "github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/apply" "github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link" "github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/selfcompose" "github.com/novox/mesh-host/internal/selfcompose"
"github.com/novox/mesh-host/internal/store" "github.com/novox/mesh-host/internal/store"
) )
@@ -38,6 +39,20 @@ func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Up
if err != nil { if err != nil {
return nil, err return nil, err
} }
return w.updates(ctx, in), nil
}
// WatchAwait is Watch for a state that its holder makes when it starts (the placement seat's): a bucket not on the bus
// yet is waited for on the bus's word that it was created (link.Session.KVWatchAwait), and waiting is told why.
func (w bucketWatch) WatchAwait(ctx context.Context, pattern string, waiting func(error)) (<-chan agent.Update, error) {
in, err := w.session.KVWatchAwait(ctx, w.bucket, pattern, waiting)
if err != nil {
return nil, err
}
return w.updates(ctx, in), nil
}
func (w bucketWatch) updates(ctx context.Context, in <-chan link.KVUpdate) <-chan agent.Update {
out := make(chan agent.Update, 64) out := make(chan agent.Update, 64)
go func() { go func() {
defer close(out) defer close(out)
@@ -49,7 +64,7 @@ func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Up
} }
} }
}() }()
return out, nil return out
} }
// sessionHaves writes the haves the agent says, through the session. // sessionHaves writes the haves the agent says, through the session.
@@ -62,14 +77,32 @@ func (h sessionHaves) Read(ctx context.Context, module string) (agent.Have, bool
if !ok || got.Composed == "" { if !ok || got.Composed == "" {
return agent.Have{}, false return agent.Have{}, false
} }
return agent.Have{Module: module, Composed: got.Composed, Sealed: got.Sealed}, true return agent.Have{Module: module, Composed: got.Composed, Sealed: got.SealedAt}, true
} }
got, ok := h.session.ReadHave(ctx, module) got, ok := h.session.ReadHave(ctx, module)
if !ok { if !ok {
return agent.Have{}, false return agent.Have{}, false
} }
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail, return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail,
Sealed: got.Sealed}, true Sealed: got.SealedAt}, true
}
// ReadErr is Read telling a have absent from one the bus could not read (agent.HavesReadErr).
func (h sessionHaves) ReadErr(ctx context.Context, module string) (agent.Have, bool, error) {
if module == agent.NodeModule {
// The node's own key, its read the bus did not answer told as such (issue 1126, 2026-10-28).
got, ok, err := h.session.ReadNodeHaveErr(ctx)
if !ok || err != nil || got.Composed == "" {
return agent.Have{}, false, err
}
return agent.Have{Module: module, Composed: got.Composed, Sealed: got.SealedAt}, true, nil
}
got, ok, err := h.session.ReadHaveErr(ctx, module)
if !ok || err != nil {
return agent.Have{}, false, err
}
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail,
Sealed: got.SealedAt}, true, nil
} }
func (h sessionHaves) Write(ctx context.Context, x agent.Have) { func (h sessionHaves) Write(ctx context.Context, x agent.Have) {
@@ -450,6 +483,7 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
// The same kept declaration's filled paths, so a module still running from it keeps its files filled. // The same kept declaration's filled paths, so a module still running from it keeps its files filled.
KeptFilled: func() map[string][]string { _, filled := kept(); return filled }, KeptFilled: func() map[string][]string { _, filled := kept(); return filled },
Recorded: recordedIn(opts.state, say), Recorded: recordedIn(opts.state, say),
Inactive: inactiveIn(&liveness.Exec{Run: apply.ExecRunner, System: builtFor}, heldBy(apply.WindowsIn(filepath.Dir(opts.state)))),
Revisions: func(ctx context.Context, key string) ([]agent.Update, error) { Revisions: func(ctx context.Context, key string) ([]agent.Update, error) {
got, err := session.KVHistory(ctx, composedBucket, key) got, err := session.KVHistory(ctx, composedBucket, key)
if err != nil { if err != nil {
@@ -509,29 +543,33 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
run, stopRun := context.WithCancel(linked) run, stopRun := context.WithCancel(linked)
// What the node runs and how it opens a sealed value, for the composer to keep the boxes it runs; and the agent // What the node runs and how it opens a sealed value, for the composer to keep the boxes it runs; and the agent
// told when the node cannot compose, so each wanted module's have says it. // told when the node cannot compose, so each wanted module's have says it.
// The agent of this run alone: its composer tells it, never the agent of a run begun since (issue 1126: a pass in
// flight at the switch marked the next agent's modules failed).
own := &agentNow{}
hooks := composerHooks{ hooks := composerHooks{
running: func() []delivery.Composed { running: func() []delivery.Composed {
if now := current.get(); now != nil { if now := own.get(); now != nil {
return now.Everything() return now.Everything()
} }
return nil return nil
}, },
unseal: sealOpener(opts.state), unseal: sealOpener(opts.state),
// And what each module's have says it runs sealed: after a restart the agent has applied nothing yet. // And what each module's have says it runs sealed: after a restart the agent has applied nothing yet.
held: func(module string) []string { // A have the bus did not answer for is said as such, never as one holding no seals (issue 1126, 2026-10-28).
held: func(module string) (delivery.Seals, []string, error) {
if module == agent.NodeModule { if module == agent.NodeModule {
got, _ := session.ReadNodeHave(linked) got, _, err := session.ReadNodeHaveErr(linked)
return got.Sealed return got.SealedAt, got.Sealed, err
} }
got, ok := session.ReadHave(linked, module) got, ok, err := session.ReadHaveErr(linked, module)
if !ok { if !ok {
return nil return nil, nil, err
} }
return got.Sealed return got.SealedAt, got.Sealed, nil
}, },
refused: func(why string) { refused: func(why string, held map[string]string) {
if now := current.get(); now != nil { if now := own.get(); now != nil && run.Err() == nil {
now.NotComposed(linked, why) now.NotComposedEach(run, why, held) // a module held alone marks that module, never the others (issue 1126)
} }
}, },
} }
@@ -539,19 +577,43 @@ func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, sess
// its own key, `<node>.node`, under candidates. // its own key, `<node>.node`, under candidates.
shadowHooks := hooks shadowHooks := hooks
shadowHooks.refused = nil shadowHooks.refused = nil
shadow := selfcompose.NewShadow(composerEnv(session, node, say, shadowHooks), shadowEnv := composerEnv(session, node, say, shadowHooks)
// And, after each answer, what runs of an own secret the answer says the vault holds another value of, handed to
// the vault while the node applies the controller's compositions (issue 1126, cause C4; ownresync).
composes := &composesNow{}
resync := ownResync(session, node, say, shadowEnv, composes)
shadow := selfcompose.NewShadow(shadowEnv,
func(ctx context.Context, answers map[string]delivery.CandidateAnswer) { func(ctx context.Context, answers map[string]delivery.CandidateAnswer) {
session.NoteCandidates(ctx, answers) session.NoteCandidates(ctx, delivery.WithValuesDiffer(answers))
go resync.take(ctx, answers)
}) })
go shadow.Keep(linked, link.Backoff) go shadow.Keep(run, link.Backoff) // this run's alone: a switch begins a run, and its shadow, again
if composer := selfComposer(linked, session, node, say, hooks); composer != nil { composer, why := selfComposer(linked, session, node, say, hooks)
if linked.Err() != nil {
// The link ended before the node knew whose compositions to apply: it chose neither, and waits for the next.
stopRun()
cancel()
continue
}
if composer != nil {
composes.set(delivery.ComposesOwn)
env.Composed, env.SelfComposed, env.Trusted, env.Policy = composer, true, nil, composer.Policy env.Composed, env.SelfComposed, env.Trusted, env.Policy = composer, true, nil, composer.Policy
session.NoteComposes(linked, delivery.ComposesOwn, why)
// And once the catalogue no longer places it, the agent begins again on the controller's compositions: the
// placement key deleted ends this run, the way back (issue 1126, the seed's revert).
go endWhenUnplaced(run, session, node, stopRun, say)
} else { } else {
composes.set(delivery.ComposesController)
go resync.again(run)
session.NoteComposes(linked, delivery.ComposesController, why)
// And once the catalogue places it, the agent begins again composing its own: the placement key appearing ends // And once the catalogue places it, the agent begins again composing its own: the placement key appearing ends
// this run, so a node does not wait for a restart to follow the seed of its placement (ADR 0549 §4). // this run, so a node does not wait for a restart to follow the seed of its placement (ADR 0549 §4).
go endWhenPlaced(run, session, node, stopRun, say) go endWhenPlaced(run, session, node, stopRun, say, func(reason string) {
session.NoteComposes(linked, delivery.ComposesController, why+"; "+reason)
})
} }
ag = agent.New(env) ag = agent.New(env)
own.set(ag)
current.set(ag) current.set(ag)
err = ag.Run(run) err = ag.Run(run)
stopRun() stopRun()
@@ -839,27 +901,38 @@ func applyContext(ctx context.Context) (context.Context, context.CancelFunc) {
type composerHooks struct { type composerHooks struct {
running func() []delivery.Composed running func() []delivery.Composed
unseal func(string) ([]byte, error) unseal func(string) ([]byte, error)
refused func(string) refused func(why string, held map[string]string)
held func(module string) []string held func(module string) (delivery.Seals, []string, error)
} }
func selfComposer(ctx context.Context, session *link.Session, node string, say func(string), hooks composerHooks) *selfcompose.Composer { func selfComposer(ctx context.Context, session *link.Session, node string, say func(string), hooks composerHooks) (*selfcompose.Composer, string) {
key, err := delivery.PlacementKey(node) key, err := delivery.PlacementKey(node)
if err != nil { if err != nil {
say("this node applies the controller's compositions: " + err.Error()) say("this node applies the controller's compositions: " + err.Error())
return nil return nil, err.Error()
} }
got, err := session.KVHistory(ctx, delivery.PlacementBucket, key) env := composerEnv(session, node, say, hooks)
if err != nil || len(got) == 0 || got[len(got)-1].Deleted { unreadable := ""
why := "the catalogue does not place it yet" placed, err := selfcompose.Placed(ctx, env.Get, node, link.Backoff, func(m string) {
if err != nil { if strings.Contains(m, "cannot be read") {
why = "its placement cannot be read: " + err.Error() unreadable = m
}
say(m)
}, selfcompose.ComposedItsOwn(ctx, env.Get, node))
if err != nil {
return nil, "its placement could not be read before the link ended: " + err.Error()
}
if !placed {
why := "the seat's placement state holds no key for it"
if unreadable != "" {
why = "its placement cannot be read, and it never composed its own: " + unreadable
} }
say("this node applies the controller's compositions: " + why) say("this node applies the controller's compositions: " + why)
return nil return nil, why
} }
say("this node composes its own modules: the catalogue places it (" + key + ")") why := "the seat's placement state holds its key (" + key + ")"
return selfcompose.New(composerEnv(session, node, say, hooks)) say("this node composes its own modules: " + why)
return selfcompose.New(env), why
} }
// composerEnv is what the node's composers read of the bus, the vault and the agent: its own composer and its answer // composerEnv is what the node's composers read of the bus, the vault and the agent: its own composer and its answer
@@ -867,6 +940,13 @@ func selfComposer(ctx context.Context, session *link.Session, node string, say f
func composerEnv(session *link.Session, node string, say func(string), hooks composerHooks) selfcompose.Env { func composerEnv(session *link.Session, node string, say func(string), hooks composerHooks) selfcompose.Env {
return selfcompose.Env{Node: node, return selfcompose.Env{Node: node,
Watch: func(ctx context.Context, bucket, pattern string) (<-chan agent.Update, error) { Watch: func(ctx context.Context, bucket, pattern string) (<-chan agent.Update, error) {
if bucket == delivery.PlacementBucket || bucket == delivery.PlacementCandidatesBucket {
// The seat's states are made when their holder starts: a node-engine that starts first waits for the
// bus's word that the bucket was created, and the shadow and the composer say they wait.
return bucketWatch{session, bucket}.WatchAwait(ctx, pattern, func(err error) {
say("the seat's state " + bucket + " is not on the bus yet (" + err.Error() + "): it is watched when the bus says it was created")
})
}
return bucketWatch{session, bucket}.Watch(ctx, pattern) return bucketWatch{session, bucket}.Watch(ctx, pattern)
}, },
Get: func(ctx context.Context, bucket, key string) ([]byte, error) { Get: func(ctx context.Context, bucket, key string) ([]byte, error) {
@@ -883,28 +963,76 @@ func composerEnv(session *link.Session, node string, say func(string), hooks com
return session.AskVault(ctx, "own", args) return session.AskVault(ctx, "own", args)
}, },
// And the vault's word that a secret changed, on which it composes again (ADR 0547 §2). // And the vault's word that a secret changed, on which it composes again (ADR 0547 §2).
Hear: session.Hear, Hear: session.Hear,
// And a moved module's templates archive, in the artifact store, as every archive is fetched (issue 1126).
Fetch: apply.Fetch,
Running: hooks.running, Unseal: hooks.unseal, Refused: hooks.refused, Held: hooks.held} Running: hooks.running, Unseal: hooks.unseal, Refused: hooks.refused, Held: hooks.held}
} }
// endWhenPlaced calls stop once this node's placement key appears in the indexer's placement (a value after the watch // endWhenUnplaced calls stop once this node's placement key is deleted from the seat's placement (after the watch
// caught up, or one already there), so the agent begins again and composes its own. A watch that cannot be made is said; // caught up, or gone already), so the agent begins again and applies the controller's compositions, as a node the
// the node then switches at its next start. // catalogue does not place does from its start: the way back (issue 1126, the seed's revert). A watch that cannot be
func endWhenPlaced(ctx context.Context, session *link.Session, node string, stop func(), say link.Announce) { // made is said; the node then returns at its next start.
func endWhenUnplaced(ctx context.Context, session *link.Session, node string, stop func(), say link.Announce) {
key, err := delivery.PlacementKey(node) key, err := delivery.PlacementKey(node)
if err != nil { if err != nil {
return return
} }
updates, err := bucketWatch{session, delivery.PlacementBucket}.Watch(ctx, key) updates, err := bucketWatch{session, delivery.PlacementBucket}.Watch(ctx, key)
if err != nil { if err != nil {
say("whether the catalogue places this node is not followed (" + err.Error() + "): it composes its own from its next start") say("whether this node's key in the seat's placement state still exists is not followed (" + err.Error() + "): it returns to the controller's compositions at its next start")
return
}
placed := false
for u := range updates {
switch {
case u.Caught && placed:
continue
case u.Caught || u.Deleted || len(u.Value) == 0:
say("this node's key in the seat's placement state is gone (" + key + "): the node agent begins again, applying the controller's compositions")
stop()
return
default:
placed = true
}
}
if ctx.Err() == nil {
say("the watch of " + key + " ended while the link stays: whether this node's key in the seat's placement state still exists is followed again at the node agent's next start")
}
}
// endWhenPlaced calls stop once this node's placement key appears in the seat's placement (a value after the watch
// caught up, or one already there), so the agent begins again and composes its own. A watch that cannot be made is said;
// the node then switches at its next start.
func endWhenPlaced(ctx context.Context, session *link.Session, node string, stop func(), say link.Announce, note func(string)) {
key, err := delivery.PlacementKey(node)
if err != nil {
return
}
// The seat's placement state is made when its holder starts; the delivery brings the grants part and the holder first,
// so it is normally there. A node-engine that starts before it waits for the bus's word that the bucket was created
// (no timer), says so once in the journal and in the node's own have, where the watchman reads it, and stays on the
// controller's compositions until its key is written. A watch the account may not make is said the same way, and the
// node stays until its next start.
updates, err := bucketWatch{session, delivery.PlacementBucket}.WatchAwait(ctx, key, func(err error) {
why := "this node's key in the seat's placement state is waited for, the state is not on the bus yet (" + err.Error() + "): it stays on the controller's compositions and is watched when the bus says the state was created"
say(why)
note(why)
})
if err != nil {
why := "this node's key in the seat's placement state cannot be followed (" + err.Error() + "): it stays on the controller's compositions until its next start"
say(why)
note(why)
return return
} }
for u := range updates { for u := range updates {
if !u.Caught && !u.Deleted && len(u.Value) > 0 { if !u.Caught && !u.Deleted && len(u.Value) > 0 {
say("the catalogue places this node now (" + key + "): the node agent begins again, composing its own modules") say("this node's key in the seat's placement state is written now (" + key + "): the node agent begins again, composing its own modules")
stop() stop()
return return
} }
} }
if ctx.Err() == nil {
say("the watch of " + key + " ended while the link stays: whether this node's key in the seat's placement state is written is followed again at the node agent's next start")
}
} }
+340 -16
View File
@@ -21,6 +21,7 @@ import (
"github.com/novox/mesh-host/internal/agent" "github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/link" "github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/selfcompose" "github.com/novox/mesh-host/internal/selfcompose"
"github.com/novox/mesh-host/internal/store"
) )
// The node agent whole, against a real bus with JetStream started in the test: the wants bucket, the node's composition // The node agent whole, against a real bus with JetStream started in the test: the wants bucket, the node's composition
@@ -47,14 +48,16 @@ type itBed struct {
controll ed25519.PrivateKey controll ed25519.PrivateKey
operator ed25519.PrivateKey operator ed25519.PrivateKey
said []string said []string
// refusals are the bus's asynchronous errors on the node's connection: each permissions violation it met.
refusals []string
mu sync.Mutex mu sync.Mutex
} }
// aServer is an embedded bus with JetStream and two users: the test's, unrestricted, and the node's, granted exactly what // aServer is an embedded bus with JetStream and two users: the test's, unrestricted, and the node's, granted exactly what
// buscompose.PermissionsFor gives a node (KindNode); it answers a connection of each. // buscompose.PermissionsFor gives a node (KindNode); it answers a connection of each.
func aServer(t *testing.T) (admin, node *nats.Conn) { func aServer(t *testing.T, principal buscompose.Principal, refused func(error)) (admin, node *nats.Conn) {
t.Helper() t.Helper()
perms, err := buscompose.PermissionsFor(buscompose.Principal{Kind: buscompose.KindNode, Node: itNode}) perms, err := buscompose.PermissionsFor(principal)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -78,7 +81,8 @@ func aServer(t *testing.T) (admin, node *nats.Conn) {
t.Fatal(err) t.Fatal(err)
} }
t.Cleanup(admin.Close) t.Cleanup(admin.Close)
if node, err = nats.Connect(srv.ClientURL(), nats.UserInfo(nodeUser, "node"), nats.CustomInboxPrefix("_INBOX."+nodeUser)); err != nil { if node, err = nats.Connect(srv.ClientURL(), nats.UserInfo(nodeUser, "node"), nats.CustomInboxPrefix("_INBOX."+nodeUser),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refused(err) })); err != nil {
t.Fatal(err) t.Fatal(err)
} }
t.Cleanup(node.Close) t.Cleanup(node.Close)
@@ -86,6 +90,12 @@ func aServer(t *testing.T) (admin, node *nats.Conn) {
} }
func newItBed(t *testing.T) *itBed { func newItBed(t *testing.T) *itBed {
t.Helper()
return newItBedAs(t, buscompose.Principal{Kind: buscompose.KindNode, Node: itNode})
}
// newItBedAs is newItBed with the node's connection under the grant of principal, a node's (KindNode).
func newItBedAs(t *testing.T, principal buscompose.Principal) *itBed {
t.Helper() t.Helper()
// Not skipped off Arch: a package manager that answers is put on PATH, so the apply of files runs on any build node. // Not skipped off Arch: a package manager that answers is put on PATH, so the apply of files runs on any build node.
bin := t.TempDir() bin := t.TempDir()
@@ -98,7 +108,11 @@ func newItBed(t *testing.T) *itBed {
t.Cleanup(func() { builtFor = was }) t.Cleanup(func() { builtFor = was })
b := &itBed{t: t, dir: t.TempDir()} b := &itBed{t: t, dir: t.TempDir()}
b.conn, b.nodeConn = aServer(t) b.conn, b.nodeConn = aServer(t, principal, func(err error) {
b.mu.Lock()
b.refusals = append(b.refusals, err.Error())
b.mu.Unlock()
})
var err error var err error
if b.js, err = b.conn.JetStream(); err != nil { if b.js, err = b.conn.JetStream(); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -160,6 +174,8 @@ func (b *itBed) say(line string) {
func fpIt(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) } func fpIt(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
func fpDigest(c string) string { return "sha256:" + strings.Repeat(c, 64) }
// compose writes a module's signed composition of one file into the node's bucket and returns its digest. // compose writes a module's signed composition of one file into the node's bucket and returns its digest.
func (b *itBed) compose(module, fp, content string, data bool, signer ed25519.PrivateKey) string { func (b *itBed) compose(module, fp, content string, data bool, signer ed25519.PrivateKey) string {
b.t.Helper() b.t.Helper()
@@ -224,6 +240,7 @@ func within(t *testing.T, d time.Duration, what string, ok func() bool) {
// applied within one second and its have names the fingerprint and the digest; a composition whose digest no want names // applied within one second and its have names the fingerprint and the digest; a composition whose digest no want names
// is not applied. // is not applied.
func TestNatsAWantAndItsCompositionAreAppliedWithinASecond(t *testing.T) { func TestNatsAWantAndItsCompositionAreAppliedWithinASecond(t *testing.T) {
t.Run("1957: an engine start starts a declared-running unit", testNatsAnEngineStartStartsADeclaredRunningUnitTheHaveSaysRuns)
b := newItBed(t) b := newItBed(t)
stop := b.start(false) stop := b.start(false)
defer stop() defer stop()
@@ -344,8 +361,205 @@ func TestNatsTheHandPathAppliesACompositionTheOperatorSigned(t *testing.T) {
// with no key in the controller's compositions; a changed config value is composed again on its event and applied as a // with no key in the controller's compositions; a changed config value is composed again on its event and applied as a
// revision of what runs. And a catalogue pull request's placement candidate equal to its placement is answered "same" // revision of what runs. And a catalogue pull request's placement candidate equal to its placement is answered "same"
// in the node's own key, and nothing is applied for it (ADR 0559 §4, subtask 10 of the milestone). // in the node's own key, and nothing is applied for it (ADR 0559 §4, subtask 10 of the milestone).
//
// And (issue 1126, ADR 0560) it reads, through its own session under the grant the grants part composes for it
// (buscompose.Users, so it carries ServedReads), the build of a placed module that declares one and the one config value
// a provider on another node serves from (a mail domain); under a grant without that served key the bus refuses the
// read and the node composes nothing.
func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) { func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
b := newItBed(t) users, err := buscompose.Users(buscompose.Records{Nodes: []string{itNode, "it2"},
Assigned: map[string][]buscompose.Declared{"it2": {{Module: "mailer", Served: []string{"domain"}}}}})
if err != nil {
t.Fatal(err)
}
var principal buscompose.Principal
for _, u := range users {
if u.Kind == buscompose.KindNode && u.Node == itNode {
principal = u
}
}
if len(principal.ServedReads) != 1 {
t.Fatalf("the grants part's principal of %s reads %v of the served keys", itNode, principal.ServedReads)
}
t.Run("without the served key the bus refuses the read", func(t *testing.T) {
b := newItBed(t)
stop, _ := h8(t, b, false)
defer stop()
// 15 s, above link.historyWithin (10 s): the bus's refusal of a publish reaches only the connection's error
// handler (nats.go fails no request on it), so the refused read ends at its own bound, then the node says it.
within(t, 15*time.Second, "the bus did not refuse the served key's read, or the node did not say it", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return strings.Contains(strings.Join(b.refusals, "\n"), "Permissions Violation for Publish to \"$JS.API.CONSUMER.CREATE.KV_"+
selfcompose.ResolvedBucket) && strings.Contains(strings.Join(b.refusals, "\n"), ".it2.mailer.domain") &&
strings.Contains(strings.Join(b.said, "\n"), "what mailer serves on it2 (domain)")
})
if b.file("web") != "" {
t.Errorf("composed and applied under a grant without the served key: %q", b.file("web"))
}
})
b := newItBedAs(t, principal)
stop, st := h8(t, b, true)
defer stop()
placement, resolved, candidates := st.placement, st.resolved, st.candidates
hkey, place, fp := st.hkey, st.place, st.fp
within(t, 5*time.Second, "the node did not compose and apply its own module once placed", func() bool { return b.file("web") == "greeting=bonjour\n" })
h8rest(t, b, resolved, candidates, hkey, place, fp)
t.Run("one answer per candidate revision", func(t *testing.T) { h8once(t, b, resolved, candidates, place) })
h8back(t, b, placement, resolved, candidates, hkey)
}
// h8once is issue 1126, 2026-10-28 (shanks answered probe/1126-seed-7 at revision 84 every 13-17 s, its bus flooded):
// a candidate revision is answered once. The node's own answer written, a module's have whose health words alone move
// (each look rewrites them) and a config value written again with the value it held answer nothing again; the
// candidate's next revision is answered once.
func h8once(t *testing.T, b *itBed, resolved, candidates nats.KeyValue, place []byte) {
answering := func() int {
b.mu.Lock()
defer b.mu.Unlock()
n := 0
for _, l := range b.said {
if strings.HasPrefix(l, "answering placement candidate feat-x ") {
n++
}
}
return n
}
time.Sleep(time.Second)
settled := answering()
if settled == 0 {
t.Fatal("feat-x was never answered")
}
hkey, _ := delivery.HaveKey(itNode, "web")
for _, detail := range []string{"its tool web_health: no link to the bus is open", "its tool web_health: no answer within 10s"} {
h := b.have("web")
h.Health, h.Detail = delivery.HealthFailed, detail
raw, err := delivery.EncodeHave(h)
if err != nil {
t.Fatal(err)
}
if _, err := b.haves.Put(hkey, raw); err != nil {
t.Fatal(err)
}
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err)
}
}
time.Sleep(1500 * time.Millisecond)
if got := answering(); got != settled {
t.Errorf("feat-x answered %d more time(s) at the same revision, with nothing it is composed from changed", got-settled)
}
same, _ := delivery.DecodePlacement(place)
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "feat-x", Placement: &same})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.PlacementCandidateKey("feat-x", itNode)
if _, err := candidates.Put(ckey, craw); err != nil {
t.Fatal(err)
}
within(t, 10*time.Second, "the candidate's next revision was not answered", func() bool { return answering() > settled })
time.Sleep(1500 * time.Millisecond)
if got := answering(); got != settled+1 {
t.Errorf("the candidate's next revision was answered %d time(s), want 1", got-settled)
}
}
// h8back is the way back (issue 1126, the seed's revert): a candidate deleting the node's placement is compared with
// the controller's compositions at its wants (here none: refused, naming web), and the placement key deleted returns the agent to them without a
// restart, said in the journal and the node's have, keeping what runs.
func h8back(t *testing.T, b *itBed, placement, resolved, candidates nats.KeyValue, hkey string) {
t.Helper()
node := func() delivery.HaveNode {
e, err := b.haves.Get(hkey)
if err != nil {
return delivery.HaveNode{}
}
n, _ := delivery.DecodeHaveNode(e.Value())
return n
}
if n := node(); n.Composes != delivery.ComposesOwn {
t.Errorf("a node composing its own says it composes %q", n.Composes)
}
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "revert-x"})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.PlacementCandidateKey("revert-x", itNode)
if _, err := candidates.Put(ckey, craw); err != nil {
t.Fatal(err)
}
// The controller holds no composition at web's want (the node composed its own): returning would leave web with
// nothing to apply, and the answer says so rather than "same" (review of the shadow, round 1).
within(t, 10*time.Second, "the node did not refuse returning to compositions the controller does not hold", func() bool {
a := node().Candidates["revert-x"]
return a.Proposal == "revert-x" && a.Returns == "" && a.Verdict == delivery.CandidateRefused &&
strings.Contains(a.Refused, "web: the controller holds no composition")
})
ran := b.have("web").Composed
pkey, _ := delivery.PlacementKey(itNode)
// A pass in flight at the switch (issue 1126, 2026-10-28: the last pass of the own run, failing on its run's end,
// marked the next agent's modules failed): a config value changed starts one, and the key goes at once.
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err)
}
if err := placement.Delete(pkey); err != nil {
t.Fatal(err)
}
stopWatch := make(chan struct{})
marked := make(chan Have1126, 1)
go func() {
for {
select {
case <-stopWatch:
return
case <-time.After(20 * time.Millisecond):
}
if h := b.have("web"); h.Phase == delivery.PhaseFailed {
select {
case marked <- Have1126{h.Phase, h.Detail}:
default:
}
return
}
}
}()
defer close(stopWatch)
within(t, 15*time.Second, "the node did not return to the controller's compositions when its placement went", func() bool {
b.mu.Lock()
said := strings.Join(b.said, "\n")
b.mu.Unlock()
return strings.Contains(said, "key in the seat's placement state is gone") && node().Composes == delivery.ComposesController
})
time.Sleep(3 * time.Second) // what a pass in flight would mark lands within it
select {
case h := <-marked:
t.Errorf("a pass in flight at the switch marked web failed: %+v", h)
default:
}
if h := b.have("web"); h.Composed != ran || h.Phase == delivery.PhaseFailed || b.file("web") != "greeting=salut\n" {
t.Errorf("returning to the controller's compositions moved what runs: %+v, %q", h, b.file("web"))
}
}
// Have1126 is what H8's way back watches of web's have.
type Have1126 struct {
Phase delivery.Phase
Detail string
}
// h8state is what h8 set up that the rest of H8 writes to.
type h8state struct {
placement, resolved, candidates nats.KeyValue
hkey string
place []byte
fp string
}
// h8 sets H8's bus up and starts the agent, then places the node. With checkBuild, the placed module's build key is
// first one that does not decode, which the node must read through its session (and say), then a done one.
func h8(t *testing.T, b *itBed, checkBuild bool) (func(), h8state) {
t.Helper()
kvOf := func(bucket string) nats.KeyValue { kvOf := func(bucket string) nats.KeyValue {
kv, err := b.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 5}) kv, err := b.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 5})
if err != nil { if err != nil {
@@ -355,7 +569,7 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
} }
placement, modules, manifests := kvOf(delivery.PlacementBucket), kvOf(delivery.ModulesBucket), kvOf(delivery.ManifestsBucket) placement, modules, manifests := kvOf(delivery.PlacementBucket), kvOf(delivery.ModulesBucket), kvOf(delivery.ManifestsBucket)
resolved := kvOf(selfcompose.ResolvedBucket) resolved := kvOf(selfcompose.ResolvedBucket)
kvOf(delivery.BuilderBuildsBucket) builds := kvOf(delivery.BuilderBuildsBucket)
candidates := kvOf(delivery.PlacementCandidatesBucket) candidates := kvOf(delivery.PlacementCandidatesBucket)
// The controller's stream the node's facts go into, so the node's key carries the facts it says. // The controller's stream the node's facts go into, so the node's key carries the facts it says.
if _, err := b.js.AddStream(&nats.StreamConfig{Name: "CONTROL", Subjects: []string{link.ReportSubject(itNode)}}); err != nil { if _, err := b.js.AddStream(&nats.StreamConfig{Name: "CONTROL", Subjects: []string{link.ReportSubject(itNode)}}); err != nil {
@@ -363,7 +577,8 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
} }
path := filepath.Join(b.dir, "web.conf") path := filepath.Join(b.dir, "web.conf")
man := `{"module":"web","version":"1","config":{"greeting":{"type":"string","description":"what the file says","default":"hello"}},` + man := `{"module":"web","version":"1","config":{"greeting":{"type":"string","description":"what the file says","default":"hello"}},` +
`"resources":[{"id":"conf","type":"file","path":"` + path + `","mode":"0644","content":"greeting=${config:greeting}\n"}]}` `"resources":[{"id":"conf","type":"file","path":"` + path + `","mode":"0644","content":"greeting=${config:greeting}\n"}],` +
`"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]}}`
digest := delivery.ManifestDigest([]byte(man)) digest := delivery.ManifestDigest([]byte(man))
mraw, _ := delivery.EncodeManifest(delivery.Manifest{Module: "web", JSON: man}) mraw, _ := delivery.EncodeManifest(delivery.Manifest{Module: "web", JSON: man})
mkey, _ := delivery.ManifestsKey(digest) mkey, _ := delivery.ManifestsKey(digest)
@@ -373,12 +588,32 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: itNode, Capabilities: []string{"service-manager"}}}) rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: itNode, Capabilities: []string{"service-manager"}}})
pkey, _ := delivery.PlacementKey(itNode) pkey, _ := delivery.PlacementKey(itNode)
hkey, _ := delivery.HaveNodeKey(itNode) hkey, _ := delivery.HaveNodeKey(itNode)
// A provider on it2 serving from its own config value: a mail domain, which it1's composition settles.
mailer := `{"module":"mailer","version":"1","config":{"domain":{"type":"string","description":"the mail domain","required":true}},` +
`"provides":[{"name":"smtp","scope":"mesh"}],"serves":{"smtp":{"port":587,"domain":"${config:domain}"}}}`
mdigest := delivery.ManifestDigest([]byte(mailer))
mlraw, _ := delivery.EncodeManifest(delivery.Manifest{Module: "mailer", JSON: mailer})
mlkey, _ := delivery.ManifestsKey(mdigest)
mlmod, _ := delivery.EncodeModule(delivery.Module{FP2: fpIt("b"), Manifest: mdigest, Repository: "novox/mailer", Commit: strings.Repeat("2", 40)})
place2, _ := delivery.EncodePlacement(delivery.Placement{Node: "it2", Modules: []delivery.PlacedModule{{Module: "mailer"}}})
rec2, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: "it2", Capabilities: []string{"service-manager"}}})
pkey2, _ := delivery.PlacementKey("it2")
hkey2, _ := delivery.HaveNodeKey("it2")
buildKey, _ := delivery.BuilderBuildKey("web", fp)
done, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: digest,
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Reference: "artifact-store://web/files/blobs/" + fpDigest("f"), Digest: fpDigest("f")}}})
firstBuild := done
if checkBuild {
firstBuild = []byte(`{"v":1,"state":"no such state"}`)
}
for _, w := range []struct { for _, w := range []struct {
kv nats.KeyValue kv nats.KeyValue
key string key string
value []byte value []byte
}{{manifests, mkey, mraw}, {modules, "web", modv}, {b.haves, hkey, rec}, }{{manifests, mkey, mraw}, {modules, "web", modv}, {b.haves, hkey, rec},
{resolved, itNode + ".web.greeting", []byte(`{"value":"bonjour"}`)}} { {resolved, itNode + ".web.greeting", []byte(`{"value":"bonjour","source":"module"}`)},
{manifests, mlkey, mlraw}, {modules, "mailer", mlmod}, {b.haves, hkey2, rec2}, {placement, pkey2, place2},
{resolved, "it2.mailer.domain", []byte(`{"value":"example.org","source":"module"}`)}, {builds, buildKey, firstBuild}} {
if _, err := w.kv.Put(w.key, w.value); err != nil { if _, err := w.kv.Put(w.key, w.value); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -390,7 +625,6 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
} }
stop := b.start(false) stop := b.start(false)
defer stop()
// Not placed yet: the agent applies the controller's compositions, of which there are none. The catalogue's seed then // Not placed yet: the agent applies the controller's compositions, of which there are none. The catalogue's seed then
// places the node, and the agent begins again composing its own (ADR 0549 §4), with no restart of the engine. // places the node, and the agent begins again composing its own (ADR 0549 §4), with no restart of the engine.
time.Sleep(300 * time.Millisecond) time.Sleep(300 * time.Millisecond)
@@ -400,7 +634,30 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
if _, err := placement.Put(pkey, place); err != nil { if _, err := placement.Put(pkey, place); err != nil {
t.Fatal(err) t.Fatal(err)
} }
within(t, 5*time.Second, "the node did not compose and apply its own module once placed", func() bool { return b.file("web") == "greeting=bonjour\n" }) if checkBuild {
// The build of the placed module is read through the node's session: one that does not decode is said, and nothing
// is applied; a done one, then a config event, and the node composes.
within(t, 5*time.Second, "the node did not read web's build key through its session", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return strings.Contains(strings.Join(b.said, "\n"), "the builder's "+buildKey)
})
if b.file("web") != "" {
t.Fatal("composed with a build key that does not decode")
}
if _, err := builds.Put(buildKey, done); err != nil {
t.Fatal(err)
}
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"bonjour","source":"module"}`)); err != nil {
t.Fatal(err)
}
}
return stop, h8state{placement: placement, resolved: resolved, candidates: candidates, hkey: hkey, place: place, fp: fp}
}
// h8rest is H8 once the node composes its own: the have, a config revision, and the candidate answered "same".
func h8rest(t *testing.T, b *itBed, resolved, candidates nats.KeyValue, hkey string, place []byte, fp string) {
t.Helper()
within(t, 2*time.Second, "the have does not name the build", func() bool { within(t, 2*time.Second, "the have does not name the build", func() bool {
h := b.have("web") h := b.have("web")
return h.FP == fp && h.Composed != "" && h.Phase == delivery.PhaseRunning return h.FP == fp && h.Composed != "" && h.Phase == delivery.PhaseRunning
@@ -409,7 +666,7 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
t.Errorf("the controller's compositions hold %v; the node composed its own", keys) t.Errorf("the controller's compositions hold %v; the node composed its own", keys)
} }
first := b.have("web").Composed first := b.have("web").Composed
if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut"}`)); err != nil { if _, err := resolved.Put(itNode+".web.greeting", []byte(`{"value":"salut","source":"module"}`)); err != nil {
t.Fatal(err) t.Fatal(err)
} }
within(t, 3*time.Second, "a changed config value was not composed and applied again", func() bool { return b.file("web") == "greeting=salut\n" }) within(t, 3*time.Second, "a changed config value was not composed and applied again", func() bool { return b.file("web") == "greeting=salut\n" })
@@ -424,12 +681,11 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
map[string]any{"name": "service-manager", "present": true}}}}) map[string]any{"name": "service-manager", "present": true}}}})
second := b.have("web").Composed second := b.have("web").Composed
same, _ := delivery.DecodePlacement(place) same, _ := delivery.DecodePlacement(place)
craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Group: "feat/x", craw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "feat-x", Placement: &same})
Pull: delivery.Head{Repository: delivery.CatalogueRepository, Number: 1, Head: "h1"}, Placement: &same})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
ckey, _ := delivery.PlacementCandidateKey("feat/x", itNode) ckey, _ := delivery.PlacementCandidateKey("feat-x", itNode)
crev, err := candidates.Put(ckey, craw) crev, err := candidates.Put(ckey, craw)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -441,8 +697,8 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
return false return false
} }
n, err := delivery.DecodeHaveNode(e.Value()) n, err := delivery.DecodeHaveNode(e.Value())
answer = n.Candidates["feat/x"] answer = n.Candidates["feat-x"]
return err == nil && answer.Head == "h1" && n.Record != nil return err == nil && answer.Proposal == "feat-x" && n.Record != nil
}) })
if answer.Verdict != delivery.CandidateSame || answer.Revision != crev || len(answer.Same) != 1 || answer.Same[0] != "web" { if answer.Verdict != delivery.CandidateSame || answer.Revision != crev || len(answer.Same) != 1 || answer.Same[0] != "web" {
t.Errorf("the answer to a candidate equal to the placement: %+v", answer) t.Errorf("the answer to a candidate equal to the placement: %+v", answer)
@@ -451,3 +707,71 @@ func TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn(t *testing.T) {
t.Errorf("a candidate moved what runs: %+v, %q", h, b.file("web")) t.Errorf("a candidate moved what runs: %+v, %q", h, b.file("web"))
} }
} }
// 1957/AC1 (integration; ADR 0568 §1): a node agent starting on a bus that holds a `running` have for a module at its want,
// with a service manager (a systemctl on the PATH) that shows the module's unit inactive, starts the unit once and writes
// the have running; the same start with the unit active applies nothing.
func testNatsAnEngineStartStartsADeclaredRunningUnitTheHaveSaysRuns(t *testing.T) {
for _, active := range []bool{false, true} {
b := newItBed(t)
bin := t.TempDir()
stateFile, calls := filepath.Join(bin, "state"), filepath.Join(bin, "calls")
if active {
os.WriteFile(stateFile, []byte("active"), 0o644)
} else {
os.WriteFile(stateFile, []byte("inactive"), 0o644)
}
script := "#!/bin/sh\necho \"$@\" >> " + calls + "\ncase \"$1\" in\n" +
"show) printf 'LoadState=loaded\\nActiveState=%s\\nType=simple\\n' \"$(cat " + stateFile + ")\" ;;\n" +
"start) printf active > " + stateFile + " ;;\nis-enabled) echo enabled ;;\nesac\nexit 0\n"
if err := os.WriteFile(filepath.Join(bin, "systemctl"), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
// The engine's own record of the unit, as a previous run left it, and the have it wrote then.
res, _ := json.Marshal(map[string]any{"id": "web.unit", "type": "service", "unit": "web-it.service", "state": "running", "module": "web"})
var known store.State
known.Record(store.Applied{ID: "web.unit", Type: "service", Target: "web-it.service"})
if err := store.Save(b.opts.state, known); err != nil {
t.Fatal(err)
}
fp := fpIt("a")
c := delivery.Composed{Node: itNode, Module: "web", FP: fp, Resources: []json.RawMessage{res}}
signed, _ := delivery.SignComposed(c, b.controll)
raw, _ := delivery.EncodeComposed(signed)
dig, _ := delivery.ComposedDigestOf(signed)
key, _ := delivery.ComposedKey(itNode, "web", fp)
if _, err := b.composed.Put(key, raw); err != nil {
t.Fatal(err)
}
b.want("web", fp, dig, "")
hkey, _ := delivery.HaveKey(itNode, "web")
have, err := delivery.EncodeHave(delivery.Have{FP: fp, Composed: dig, Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown})
if err != nil {
t.Fatal(err)
}
if _, err := b.haves.Put(hkey, have); err != nil {
t.Fatal(err)
}
stop := b.start(false)
within(t, 5*time.Second, "the unit was not started and the have not running", func() bool {
got, _ := os.ReadFile(stateFile)
h := b.have("web")
return string(got) == "active" && h.Phase == delivery.PhaseRunning && h.Composed == dig
})
time.Sleep(300 * time.Millisecond)
stop()
starts := 0
log, _ := os.ReadFile(calls)
for _, l := range strings.Split(string(log), "\n") {
if strings.HasPrefix(l, "start") {
starts++
}
}
if want := map[bool]int{false: 1, true: 0}[active]; starts != want {
t.Fatalf("unit active=%v: started %d times, want %d", active, starts, want)
}
}
}
+94
View File
@@ -0,0 +1,94 @@
package main
import (
"context"
"encoding/json"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/ownresync"
"github.com/novox/mesh-host/internal/selfcompose"
)
// composesNow is whose compositions this run of the agent applies, once decided ("" before).
type composesNow struct {
mu sync.Mutex
now string
}
func (c *composesNow) set(s string) { c.mu.Lock(); c.now = s; c.mu.Unlock() }
func (c *composesNow) get() string { c.mu.Lock(); defer c.mu.Unlock(); return c.now }
// resyncStep is the step that hands the vault what runs of an own secret a shadow answer says another value of (issue
// 1126, cause C4; internal/ownresync): one pass at a time, and the last answers taken again once the agent has decided
// whose compositions it applies.
type resyncStep struct {
r *ownresync.Resync
one sync.Mutex
mu sync.Mutex
last map[string]delivery.CandidateAnswer
// tries and pending are the step taken again after a bound when a read or an ask the bus did not answer stopped it
// (Outcome.Retry, issue 1126, 2026-10-28), one wait at a time.
tries int
pending bool
}
func ownResync(session *link.Session, node string, say func(string), env selfcompose.Env, composes *composesNow) *resyncStep {
return &resyncStep{r: ownresync.New(ownresync.Env{Node: node, Composes: composes.get, Running: env.Running,
Get: env.Get, Unseal: env.Unseal, Seal: identity.Seal, Say: say,
Ask: func(ctx context.Context, verb string, args map[string]any) (json.RawMessage, error) {
return session.AskVault(ctx, verb, args)
}})}
}
// take keeps answers as the last and takes them.
func (s *resyncStep) take(ctx context.Context, answers map[string]delivery.CandidateAnswer) {
s.mu.Lock()
s.last = answers
s.mu.Unlock()
s.again(ctx)
}
// again takes the last answers, and once more after a bound (selfcompose.LaterWait) while the bus did not answer for one.
func (s *resyncStep) again(ctx context.Context) {
s.one.Lock()
defer s.one.Unlock()
s.mu.Lock()
last := s.last
s.mu.Unlock()
if last == nil {
return
}
retry := false
for _, o := range s.r.Take(ctx, last) {
retry = retry || o.Retry
}
s.mu.Lock()
defer s.mu.Unlock()
if !retry {
s.tries = 0
return
}
if s.pending || ctx.Err() != nil {
return
}
wait := selfcompose.LaterWait(s.tries)
s.tries++
s.pending = true
go func() {
select {
case <-ctx.Done():
case <-time.After(wait):
}
s.mu.Lock()
s.pending = false
s.mu.Unlock()
if ctx.Err() == nil {
s.again(ctx)
}
}()
}
+2
View File
@@ -36,6 +36,7 @@ func TestACompositionIsRecordedOnlyWhenEveryResourceIs(t *testing.T) {
// The modules applied again are those with a missing resource, each once; one down but there is not among them. // The modules applied again are those with a missing resource, each once; one down but there is not among them.
func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) { func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
t.Run("1957: the machine shows a resource not active", testTheMachineShowsAResourceNotActive)
st := liveness.Statement{Resources: []liveness.State{ st := liveness.Statement{Resources: []liveness.State{
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.homeserver"}, State: liveness.Unhealthy, Missing: true}, {Resource: liveness.Resource{Module: "matrix", ID: "matrix.homeserver"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.db"}, State: liveness.Unhealthy, Missing: true}, {Resource: liveness.Resource{Module: "matrix", ID: "matrix.db"}, State: liveness.Unhealthy, Missing: true},
@@ -51,6 +52,7 @@ func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
// not acted on again, so a resource that cannot be made is not applied (and backed up) at every unrelated change. One // not acted on again, so a resource that cannot be made is not applied (and backed up) at every unrelated change. One
// that comes back and goes again is newly missing again (the review of issue 1945). // that comes back and goes again is newly missing again (the review of issue 1945).
func TestOnlyANewlyMissingResourceAppliesItsModuleAgain(t *testing.T) { func TestOnlyANewlyMissingResourceAppliesItsModuleAgain(t *testing.T) {
t.Run("1957: a resource newly down is an edge", testOnlyAResourceNewlyDownIsAnEdge)
missing := func(ids ...string) liveness.Statement { missing := func(ids ...string) liveness.Statement {
var st liveness.Statement var st liveness.Statement
for _, id := range ids { for _, id := range ids {
+1 -1
View File
@@ -3,7 +3,7 @@ module github.com/novox/mesh-host
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027075026-05b678310366 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0 golang.org/x/crypto v0.57.0
+2 -14
View File
@@ -1,17 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117 h1:88CNZOoQgu7YPYKW16qkWT8Gw9Ucg07aTrIImA2nYUk= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027011211-82f10c3bc9fa h1:tT6sl3vY0iVAXw+mqy3iGKUBUjVBRWvQvKUOTmTyWwk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027011211-82f10c3bc9fa/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027011927-ba398e2460ef h1:uMqybf/sTAjapyllVjy8mjCi8evpcOmgnjmqRdJRsFo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027011927-ba398e2460ef/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027012259-a37d2622ed56 h1:PUc5s3oaWEg0uyVmXhJSSK3x0IXZBXWGqq30U/CSx30=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027012259-a37d2622ed56/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027012704-062c29c3a22f h1:AKzV973UTWe0OYFoIjgGu0f+B26xh8QsAHzCpsTIjM8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027012704-062c29c3a22f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027014310-cadd7b1b2f8f h1:mimAGERguQmCGYVNdRTrFcZ1dhpU6GWo+Q3EDhHOrVk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027014310-cadd7b1b2f8f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027075026-05b678310366 h1:B7oQ82ar2McqZ9OrbJfyvG+50Ol6fBUoK3eYE2uTPvo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261027075026-05b678310366/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+226 -46
View File
@@ -24,6 +24,7 @@ import (
"time" "time"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
@@ -57,9 +58,10 @@ type Have struct {
// waits for a person's yes (ADR 0547 §3). // waits for a person's yes (ADR 0547 §3).
Judgement string Judgement string
Waits string Waits string
// Sealed is, on a node composing its own, each value the composition it names carries sealed to the node's key // Sealed is, on a node composing its own, each value the composition it names carries sealed to the node's key, by
// (delivery.Have.Sealed): kept in the have so the composer reuses them after a restart (ADR 0547 §3). // slot (delivery.Have.SealedAt): kept in the have so the composer reuses them after a restart (ADR 0547 §3), each at
Sealed []string // its own slot alone (issue 1126, C1).
Sealed delivery.Seals
} }
// Haves is the node's have bucket as the agent uses it. // Haves is the node's have bucket as the agent uses it.
@@ -116,6 +118,11 @@ type Env struct {
// Recorded says the engine holds a record of every resource of the composition, so it was applied and not removed // Recorded says the engine holds a record of every resource of the composition, so it was applied and not removed
// since; nil says yes. A composition the kept declaration or a have says runs is applied again when it says no. // since; nil says yes. A composition the kept declaration or a have says runs is applied again when it says no.
Recorded func(ctx context.Context, c delivery.Composed) bool Recorded func(ctx context.Context, c delivery.Composed) bool
// Inactive says a resource the composition declares running is not active on the machine now (a unit stopped, a
// container not running), read from the machine and not from any record (ADR 0568 §1). A have or a kept declaration
// that says the module runs is believed at the agent's start only when this says no; nil says no, and an unreadable
// machine says no (unknown is not inactive).
Inactive func(ctx context.Context, c delivery.Composed) bool
// Revisions are a composition key's revisions the bucket keeps, oldest first; nil reads none. The composition a want // Revisions are a composition key's revisions the bucket keeps, oldest first; nil reads none. The composition a want
// names is taken from them when a newer one replaced it under the same key (ADR 0539 clause 1). // names is taken from them when a newer one replaced it under the same key (ADR 0539 clause 1).
Revisions func(ctx context.Context, key string) ([]Update, error) Revisions func(ctx context.Context, key string) ([]Update, error)
@@ -161,7 +168,12 @@ type Agent struct {
said map[string]string said map[string]string
// notComposed is the modules whose have says the node could not compose them: why, and the have before (NotComposed). // notComposed is the modules whose have says the node could not compose them: why, and the have before (NotComposed).
notComposed map[string]notComposedMark notComposed map[string]notComposedMark
retry chan []string // heldAlone is the modules the composer holds alone (NotComposedEach): what runs of each in the node's own
// composition is held out of the node's own apply until a composition of its own is handed (heldOut).
heldAlone map[string]bool
retry chan []string
// reconcile carries the judge's edges: a resource found not active (ADR 0568 §2).
reconcile chan []string
// tries counts the transient failures in a row of each module; again carries a retry whose wait ran out, and // tries counts the transient failures in a row of each module; again carries a retry whose wait ran out, and
// transient asks for every transiently failed module at once. stopped is closed when Run returns. // transient asks for every transiently failed module at once. stopped is closed when Run returns.
tries map[string]int tries map[string]int
@@ -208,7 +220,7 @@ func New(env Env) *Agent {
} }
return &Agent{env: env, wants: map[string]wantNow{}, comps: map[string]compNow{}, running: map[string]delivery.Have{}, applied: map[string]compNow{}, return &Agent{env: env, wants: map[string]wantNow{}, comps: map[string]compNow{}, running: map[string]delivery.Have{}, applied: map[string]compNow{},
attempt: map[string]attempt{}, checked: map[string]bool{}, prior: map[string]bool{}, said: map[string]string{}, notComposed: map[string]notComposedMark{}, attempt: map[string]attempt{}, checked: map[string]bool{}, prior: map[string]bool{}, said: map[string]string{}, notComposed: map[string]notComposedMark{},
retry: make(chan []string, 8), tries: map[string]int{}, again: make(chan againAsk, 64), retry: make(chan []string, 8), reconcile: make(chan []string, 64), tries: map[string]int{}, again: make(chan againAsk, 64),
transient: make(chan struct{}, 1), stopped: make(chan struct{})} transient: make(chan struct{}, 1), stopped: make(chan struct{})}
} }
@@ -240,6 +252,17 @@ func (a *Agent) Run(ctx context.Context) error {
dirty[m] = true dirty[m] = true
} }
a.mu.Unlock() a.mu.Unlock()
case modules := <-a.reconcile:
a.mu.Lock()
for _, m := range modules {
if w := a.wants[m]; !w.Present || w.Want.Removed || a.attempt[m].Held {
continue
}
delete(a.running, m)
delete(a.attempt, m)
dirty[m] = true
}
a.mu.Unlock()
case ask := <-a.again: case ask := <-a.again:
// The wait of a transiently failed apply ran out: tried again from the keys it has, if nothing tried it since. // The wait of a transiently failed apply ran out: tried again from the keys it has, if nothing tried it since.
a.mu.Lock() a.mu.Lock()
@@ -298,6 +321,16 @@ func (a *Agent) Retry(modules ...string) {
} }
} }
// Reconcile asks for the modules to be applied again because a resource each declares running was found not active, once
// per edge (the judge's look, ADR 0568 §2). It is Retry, but a module whose apply is held (a failed backup) is left
// alone, as is one the node wants no more (ADR 0568 §4). It never blocks.
func (a *Agent) Reconcile(modules ...string) {
select {
case a.reconcile <- modules:
default:
}
}
// RetryTransient asks for every module whose last apply failed transiently to be applied again now: the engine says it // RetryTransient asks for every module whose last apply failed transiently to be applied again now: the engine says it
// when it knows the bus answers again (after it wrote the bus's user list and the bus reloaded its accounts). It never // when it knows the bus answers again (after it wrote the bus's user list and the bus reloaded its accounts). It never
// blocks. // blocks.
@@ -314,29 +347,97 @@ type notComposedMark struct {
prior Have prior Have
} }
// NotComposedDetail opens the detail of a have NotComposed marks: the module keeps running what its build and digest
// name, and only the node's composition of it failed.
const NotComposedDetail = "the node cannot compose it, and keeps running what it ran: "
// composerMark says h is a mark NotComposed wrote: failed, its detail opening with NotComposedDetail.
func composerMark(h Have) bool {
return h.Phase == string(delivery.PhaseFailed) && strings.HasPrefix(h.Detail, NotComposedDetail)
}
// HavesReadErr is what a Haves may answer besides Read: a have absent (false, nil) told from one that could not be read
// (false, the error).
type HavesReadErr interface {
ReadErr(ctx context.Context, module string) (Have, bool, error)
}
// readHave reads a module's have: found, absent, or unread. A read that fails is tried again after a short wait while
// the bus does not answer (a reload pauses it for a few seconds, issue 1126), and is unread when it still fails. A
// Haves that cannot tell an absent key from a failed read says absent.
func (a *Agent) readHave(ctx context.Context, module string) (h Have, found, unread bool) {
re, tells := a.env.Haves.(HavesReadErr)
if !tells {
h, found = a.env.Haves.Read(ctx, module)
return h, found, false
}
for try := 0; ; try++ {
h, found, err := re.ReadErr(ctx, module)
if err == nil {
return h, found, false
}
if try == 2 || ctx.Err() != nil {
return Have{}, false, true
}
select {
case <-ctx.Done():
return Have{}, false, true
case <-time.After(time.Duration(1<<try) * readAgain):
}
}
}
// readAgain is the first wait before a have is read again.
var readAgain = time.Second
// NotComposed says the node composing its own could not compose its modules (ADR 0547 §2: a stand-in refused, an // NotComposed says the node composing its own could not compose its modules (ADR 0547 §2: a stand-in refused, an
// input that cannot be read): each module with a want gets its have failed, naming why, while it keeps running what it // input that cannot be read): each module with a want gets its have failed, naming why, while it keeps running what it
// ran, so the failure is seen where every have is read and not only in the journal. The have keeps what runs: the // ran, so the failure is seen where every have is read and not only in the journal. The have keeps what runs: the
// fingerprint, the digest and the sealed values the agent runs, or, before it ran anything since its start, those its // fingerprint, the digest and the sealed values the agent runs, or, before it ran anything since its start, those its
// last have said (a refusal at the first compose after a start). Called with "" once the node composes again: each // last have said (a refusal at the first compose after a start). Called with "" once the node composes again: each
// module so marked has its have back as it was, unless an apply wrote it since. // module so marked has its have back as it was, unless an apply wrote it since.
func (a *Agent) NotComposed(ctx context.Context, why string) { func (a *Agent) NotComposed(ctx context.Context, why string) { a.NotComposedEach(ctx, why, nil) }
// NotComposedEach is NotComposed for the node refused whole (why), or, with why empty, for the modules the node holds
// alone (held, each with its own why: refused alone, the rest composed and applied, issue 1126): only those are marked,
// each naming its own why, and every other module marked before has its have back. Both empty, every mark is cleared.
func (a *Agent) NotComposedEach(ctx context.Context, why string, held map[string]string) {
a.mu.Lock() a.mu.Lock()
var modules []string if why == "" {
if why != "" { a.heldAlone = make(map[string]bool, len(held))
for module, w := range a.wants { for m := range held {
if w.Present && !w.Want.Removed && module != NodeModule && a.notComposed[module].why != why { a.heldAlone[m] = true
modules = append(modules, module)
}
} }
} else { }
for module := range a.notComposed { whyOf := map[string]string{} // the mark each module is to carry; a marked module not in it is cleared
for module, w := range a.wants {
if !w.Present || w.Want.Removed || module == NodeModule {
continue
}
if why != "" {
whyOf[module] = why
} else if h, ok := held[module]; ok && h != "" {
whyOf[module] = h
}
}
var modules []string
for module, w := range whyOf {
if a.notComposed[module].why != w {
modules = append(modules, module)
}
}
for module := range a.notComposed {
if _, still := whyOf[module]; !still && why == "" { // the node refused whole marks; it clears none
modules = append(modules, module) modules = append(modules, module)
} }
} }
a.mu.Unlock() a.mu.Unlock()
sort.Strings(modules) sort.Strings(modules)
for _, module := range modules { for _, module := range modules {
why := whyOf[module]
if ctx.Err() != nil {
return // the run ended: what it would mark or clear is the next run's to say (issue 1126)
}
a.mu.Lock() a.mu.Lock()
mark, marked := a.notComposed[module] mark, marked := a.notComposed[module]
r, ran := a.running[module] r, ran := a.running[module]
@@ -364,16 +465,39 @@ func (a *Agent) NotComposed(ctx context.Context, why string) {
if mark.prior.Phase == "" { if mark.prior.Phase == "" {
mark.prior.Phase = string(delivery.PhaseRunning) mark.prior.Phase = string(delivery.PhaseRunning)
} }
} else if h, ok := a.env.Haves.Read(ctx, module); ok { } else if h, found, unread := a.readHave(ctx, module); found {
mark.prior = h // what its last have said runs: this agent has run nothing of it since its start mark.prior = h // what its last have said runs: this agent has run nothing of it since its start
} else if unread {
if ctx.Err() != nil {
return
}
// What its have names cannot be read (the bus paused, issue 1126): a mark written now would drop the build
// and digest it names. Nothing is written; the next refusal marks it once the have reads.
a.mu.Lock()
a.sayOnce("unmarked:"+module, fmt.Sprintf("%s's have cannot be read, so it is not marked: the node cannot compose it: %s", module, why))
a.mu.Unlock()
continue
} }
} }
// A have that already says another cause (an apply that failed, a backup held, an apply under way) keeps it: a
// composer's mark over it would hide that cause, and baseline would take the mark for what runs (review of the
// switch, round 1).
if p := mark.prior.Phase; !marked && p != "" && p != string(delivery.PhaseRunning) && !composerMark(mark.prior) {
a.mu.Lock()
a.sayOnce("unmarked-cause:"+module, fmt.Sprintf("%s is not marked: its have says %s (%s), and the node cannot compose it: %s",
module, p, mark.prior.Detail, why))
a.mu.Unlock()
continue
}
if ctx.Err() != nil {
return
}
mark.why = why mark.why = why
a.mu.Lock() a.mu.Lock()
a.notComposed[module] = mark a.notComposed[module] = mark
a.mu.Unlock() a.mu.Unlock()
a.writeHave(ctx, Have{Module: module, FP: mark.prior.FP, Composed: mark.prior.Composed, Sealed: mark.prior.Sealed, a.writeHave(ctx, Have{Module: module, FP: mark.prior.FP, Composed: mark.prior.Composed, Sealed: mark.prior.Sealed,
Phase: string(delivery.PhaseFailed), Detail: "the node cannot compose it, and keeps running what it ran: " + why}) Phase: string(delivery.PhaseFailed), Detail: NotComposedDetail + why})
} }
} }
@@ -402,15 +526,15 @@ func (a *Agent) writeHave(ctx context.Context, h Have) {
a.env.Haves.Write(ctx, h) a.env.Haves.Write(ctx, h)
} }
func (a *Agent) sealedOf(ctx context.Context, module, fp, composed string) []string { func (a *Agent) sealedOf(ctx context.Context, module, fp, composed string) delivery.Seals {
a.mu.Lock() a.mu.Lock()
if c, ok := a.applied[module]; ok && c.Err == nil && c.Dig == composed { if c, ok := a.applied[module]; ok && c.Err == nil && c.Dig == composed {
a.mu.Unlock() a.mu.Unlock()
return SealedIn(c.C) return delivery.SealsIn(c.C)
} }
if c, ok := a.compositionFor(module, fp); ok && c.Err == nil && c.Dig == composed { if c, ok := a.compositionFor(module, fp); ok && c.Err == nil && c.Dig == composed {
a.mu.Unlock() a.mu.Unlock()
return SealedIn(c.C) return delivery.SealsIn(c.C)
} }
a.mu.Unlock() a.mu.Unlock()
if h, ok := a.env.Haves.Read(ctx, module); ok && h.Composed == composed { if h, ok := a.env.Haves.Read(ctx, module); ok && h.Composed == composed {
@@ -419,32 +543,6 @@ func (a *Agent) sealedOf(ctx context.Context, module, fp, composed string) []str
return nil return nil
} }
// SealedIn is each value a composition carries sealed to its node's key: a file's `sealed` and a vault box
// (catalogue.VaultSealedField), as their JSON text, in the order of its resources.
func SealedIn(c delivery.Composed) []string {
var out []string
for _, raw := range c.Resources {
var r map[string]json.RawMessage
if json.Unmarshal(raw, &r) != nil {
continue
}
for _, field := range []string{"sealed", catalogue.VaultSealedField} {
f, ok := r[field]
if !ok {
continue
}
var text string
if json.Unmarshal(f, &text) != nil {
text = string(f) // a vault box is carried as its JSON
}
if text != "" {
out = append(out, text)
}
}
}
return out
}
// Running is the compositions the node runs now: each module's that the agent applied or found already running, and // Running is the compositions the node runs now: each module's that the agent applied or found already running, and
// the node's own. What the node judges its health by and says of itself is built from them. // the node's own. What the node judges its health by and says of itself is built from them.
func (a *Agent) Running() []delivery.Composed { func (a *Agent) Running() []delivery.Composed {
@@ -762,6 +860,49 @@ func (a *Agent) heldOut(except string) []delivery.Composed {
sort.Strings(names) sort.Strings(names)
for _, m := range names { for _, m := range names {
out = append(out, waiting[m].C) out = append(out, waiting[m].C)
held[m] = true
}
// A module the composer holds alone keeps what runs of it in the node's own composition, until a composition of its
// own is handed: held out as one of its own, so the node's own apply never removes it (issue 1126, review of the
// apply, round 1: de-spiegel leaving the fold with its names unset; issue 1945).
if len(a.heldAlone) == 0 {
return out
}
var own *delivery.Composed
for _, c := range a.others("\x00") {
if c.Module == "" {
own = &c
break
}
}
alone := make([]string, 0, len(a.heldAlone))
for m := range a.heldAlone {
if !held[m] && m != except && own != nil {
alone = append(alone, m)
}
}
sort.Strings(alone)
for _, m := range alone {
var rs []json.RawMessage
for _, raw := range own.Resources {
var head struct {
ID string `json:"id"`
Module string `json:"module"`
}
if json.Unmarshal(raw, &head) != nil {
continue
}
owner := head.Module
if owner == "" {
owner, _ = declaration.ModuleGuess(head.ID)
}
if owner == m {
rs = append(rs, raw)
}
}
if len(rs) > 0 {
out = append(out, delivery.Composed{Node: a.env.Node, Module: m, Resources: rs})
}
} }
return out return out
} }
@@ -897,11 +1038,33 @@ func (a *Agent) baseline(ctx context.Context, module string, w wantNow, comp com
a.mu.Lock() a.mu.Lock()
a.prior[module] = true a.prior[module] = true
a.mu.Unlock() a.mu.Unlock()
if h.Phase == string(delivery.PhaseHeld) {
// Held for a failed backup, and the operator's answer is still to come: the hold is kept, nothing is
// backed up again or applied, and the next revision of the want is the answer (ADR 0568 §4).
a.mu.Lock()
a.attempt[module] = attempt{FP: w.Want.FP, Composed: w.Want.Composed, WantRev: w.Rev, CompRev: comp.Rev, Held: true,
AnsweredRev: w.Rev}
if h.Composed != "" {
a.running[module] = delivery.Have{FP: h.FP, Composed: h.Composed, Phase: delivery.PhaseRunning}
}
a.mu.Unlock()
return true
}
if h.Composed != "" { if h.Composed != "" {
// This node-engine wrote it: what it says is what runs, and the kept declaration says nothing more. A // This node-engine wrote it: what it says is what runs, and the kept declaration says nothing more. A
// failed apply or a different composition is applied again. // failed apply or a different composition is applied again.
// A have a composer marked failed (NotComposed) names what still runs: taken as running, so a mark this
// agent did not make clears once what it names is what the want names (issue 1126, the stale marks after
// the seed's revert).
marked := composerMark(h)
if marked {
h.Detail = ""
}
if h.FP == w.Want.FP && h.Composed == w.Want.Composed && if h.FP == w.Want.FP && h.Composed == w.Want.Composed &&
(h.Phase == "" || h.Phase == string(delivery.PhaseRunning)) { (h.Phase == "" || h.Phase == string(delivery.PhaseRunning) || marked) {
if a.inactiveAtStart(ctx, module, comp) {
return false
}
a.mu.Lock() a.mu.Lock()
a.running[module] = delivery.Have{FP: h.FP, Composed: h.Composed, Phase: delivery.PhaseRunning} a.running[module] = delivery.Have{FP: h.FP, Composed: h.Composed, Phase: delivery.PhaseRunning}
a.mu.Unlock() a.mu.Unlock()
@@ -927,6 +1090,9 @@ func (a *Agent) baseline(ctx context.Context, module string, w wantNow, comp com
if !ok || !SameResources(kept, comp.C.Resources) { if !ok || !SameResources(kept, comp.C.Resources) {
return false return false
} }
if a.inactiveAtStart(ctx, module, comp) {
return false
}
a.mu.Lock() a.mu.Lock()
a.running[module] = delivery.Have{FP: w.Want.FP, Composed: w.Want.Composed, Phase: delivery.PhaseRunning} a.running[module] = delivery.Have{FP: w.Want.FP, Composed: w.Want.Composed, Phase: delivery.PhaseRunning}
a.mu.Unlock() a.mu.Unlock()
@@ -934,6 +1100,20 @@ func (a *Agent) baseline(ctx context.Context, module string, w wantNow, comp com
return true return true
} }
// inactiveAtStart says the machine shows a resource the module declares running not active, so what a have or the kept
// declaration says runs is not believed and the module is applied once, as for a changed want (ADR 0568 §1).
func (a *Agent) inactiveAtStart(ctx context.Context, module string, comp compNow) bool {
if a.env.Inactive == nil || !a.env.Inactive(ctx, comp.C) {
return false
}
a.mu.Lock()
a.prior[module] = true
a.sayOnce("inactive:"+module+":"+comp.Dig, fmt.Sprintf("%s is applied again: a resource it declares running is not "+
"active on this machine", module))
a.mu.Unlock()
return true
}
func (a *Agent) apply(ctx context.Context, module string, w wantNow, comp compNow, others []delivery.Composed) { func (a *Agent) apply(ctx context.Context, module string, w wantNow, comp compNow, others []delivery.Composed) {
fp, dig := w.Want.FP, w.Want.Composed fp, dig := w.Want.FP, w.Want.Composed
a.mu.Lock() a.mu.Lock()
+263 -10
View File
@@ -4,6 +4,8 @@ import (
"context" "context"
"crypto/ed25519" "crypto/ed25519"
"encoding/json" "encoding/json"
"fmt"
"maps"
"slices" "slices"
"strings" "strings"
"sync" "sync"
@@ -72,6 +74,19 @@ type haves struct {
stored map[string]Have stored map[string]Have
log []Have log []Have
removed []string removed []string
// unreadable is each module whose have the bus cannot read now (ReadErr).
unreadable map[string]bool
}
func (h *haves) ReadErr(ctx context.Context, module string) (Have, bool, error) {
h.mu.Lock()
bad := h.unreadable[module]
h.mu.Unlock()
if bad {
return Have{}, false, fmt.Errorf("nats: no responders available for request")
}
x, ok := h.Read(ctx, module)
return x, ok, nil
} }
func (h *haves) Read(_ context.Context, module string) (Have, bool) { func (h *haves) Read(_ context.Context, module string) (Have, bool) {
@@ -327,7 +342,44 @@ func TestStartingOnWhatTheKeptDeclarationHoldsAppliesNothing(t *testing.T) {
// What the node-engine itself last applied is what runs after a restart, not the kept declaration. // What the node-engine itself last applied is what runs after a restart, not the kept declaration.
func TestStartingOnItsOwnHaveAppliesNothing(t *testing.T) { func TestStartingOnItsOwnHaveAppliesNothing(t *testing.T) {
t.Run("1957: an engine start reads the machine's facts", testAnEngineStartReconcilesAModuleFromTheMachinesFacts)
t.Run("says the have again", testStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines) t.Run("says the have again", testStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines)
// An apply that failed is not what runs, though its have names the want's build and digest: the agent starting on
// it applies again (review of the switch, round 1).
t.Run("applies again after an apply failure", func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "avahi", fp, false, file("avahi", "x", "1"))
wk, wraw := wantFor(t, "avahi", fp, dig, "")
b.haves.stored = map[string]Have{"avahi": {Module: "avahi", FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed),
Detail: "apply: unit failed to start"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
eventually(t, "a failed apply was not applied again", func() bool { return len(b.appliedList()) == 1 })
})
// Issue 1126, the stale marks after the seed's revert: a have a composer marked failed names what still runs; the
// agent starting on it applies nothing and says it running, so a mark it did not make clears.
t.Run("clears a composer's mark", func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "avahi", fp, false, file("avahi", "x", "1"))
wk, wraw := wantFor(t, "avahi", fp, dig, "")
b.haves.stored = map[string]Have{"avahi": {Module: "avahi", FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed),
Detail: NotComposedDetail + "frombus: the credentials of ace: context canceled"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
eventually(t, "the mark was not cleared", func() bool {
h := b.haves.last("avahi")
return h.Phase == string(delivery.PhaseRunning) && h.FP == fp && h.Composed == dig && h.Detail == ""
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module whose mark names what the want names was applied: %v", got)
}
})
b := newBed(t) b := newBed(t)
fp := fpOf("a") fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1")) ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
@@ -380,6 +432,7 @@ func testStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines(t *testing
// A module whose kept resources differ from the composition is applied, and a failed apply is retried only by a change // A module whose kept resources differ from the composition is applied, and a failed apply is retried only by a change
// of either key. // of either key.
func TestAFailedApplyIsRetriedOnlyByAChangeOfEitherKey(t *testing.T) { func TestAFailedApplyIsRetriedOnlyByAChangeOfEitherKey(t *testing.T) {
t.Run("1957: the judge's edge reconciles, and a failed re-apply is failed", testReconcileAppliesARunningModuleAgainAndLeavesTheHeldAndTheUnwantedAlone)
// A real failure is not retried by the clock, however short the backoff: only by a change of either key. // A real failure is not retried by the clock, however short the backoff: only by a change of either key.
t.Run("a real failure", func(t *testing.T) { t.Run("a real failure", func(t *testing.T) {
b := newBed(t) b := newBed(t)
@@ -787,6 +840,37 @@ func TestTheNodesOwnApplyHoldsOutAModuleThatMovedToItsOwnComposition(t *testing.
if !strings.Contains(strings.Join(names, ","), "systemd-resolved") { if !strings.Contains(strings.Join(names, ","), "systemd-resolved") {
t.Fatalf("the others of the node's own apply were %v: systemd-resolved's resources would be removed", names) t.Fatalf("the others of the node's own apply were %v: systemd-resolved's resources would be removed", names)
} }
// Issue 1126, review of the apply, round 1: a module the composer holds alone (de-spiegel leaving the fold with its
// names unset) has no composition of its own in the bucket: what runs of it in the node's own is held out of the
// node's own apply until one is handed.
b2 := newBed(t)
spiegel := file("de-spiegel", "server-env", "A=1")
var bare map[string]any
_ = json.Unmarshal(spiegel, &bare)
delete(bare, "module")
spiegel, _ = json.Marshal(bare)
b2.env.Kept = func() map[string][]json.RawMessage { return map[string][]json.RawMessage{NodeModule: {spiegel}} }
nk2, nraw2, ndig2 := composition(t, NodeModule, "", false)
nwraw2, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig2})
b2.comps.before(nk2, nraw2)
b2.wants.before(nwk, nwraw2)
ctx2, cancel2 := context.WithCancel(context.Background())
b2.ag = New(b2.env)
b2.ag.NotComposedEach(ctx2, "", map[string]string{"de-spiegel": "de-spiegel says ${config:praktijk-name}, and nothing sets it"})
done2 := make(chan struct{})
go func() { defer close(done2); _ = b2.ag.Run(ctx2) }()
defer func() { cancel2(); <-done2 }()
eventually(t, "the node's own not applied beside a held module", func() bool { return len(b2.appliedList()) == 1 })
b2.mu.Lock()
defer b2.mu.Unlock()
heldOut := false
for _, c := range b2.others[0] {
heldOut = heldOut || (c.Module == "de-spiegel" && len(c.Resources) == 1)
}
if !heldOut {
t.Errorf("the others of the node's own apply were %+v: held de-spiegel's resources would be removed", b2.others[0])
}
} }
// A module the assigner removed holds nothing back, its compositions still in the bucket or not: its resources are its // A module the assigner removed holds nothing back, its compositions still in the bucket or not: its resources are its
@@ -1014,39 +1098,89 @@ func TestTheNodesOwnCompositionIsAppliedWhenItDiffersAndARecordMoveWaitsForAYes(
b.wants.put(nwk, nwraw) b.wants.put(nwk, nwraw)
eventually(t, "the node's own key does not carry its sealed values", func() bool { eventually(t, "the node's own key does not carry its sealed values", func() bool {
h := b.haves.last(NodeModule) h := b.haves.last(NodeModule)
return h.Composed == ndig && len(h.Sealed) == 1 && h.Sealed[0] == "core" return h.Composed == ndig && len(h.Sealed) == 1 && h.Sealed["core-box#sealed"] == "core"
}) })
// The node cannot compose (a stand-in refused): each wanted module's have says so, failed, naming why, and still // The node cannot compose (a stand-in refused): each wanted module's have says so, failed, naming why, and still
// names what runs; composed again, the have is what it runs. web was applied by this agent; idle was not since the // names what runs; composed again, the have is what it runs. web was applied by this agent; idle was not since the
// agent's start (the switch and every delivery of the node-engine restart it), so what runs is what its have said. // agent's start (the switch and every delivery of the node-engine restart it), so what runs is what its have said.
idleFP, idleDig := fpOf("c"), "sha256:"+strings.Repeat("9", 64) idleFP, idleDig := fpOf("c"), "sha256:"+strings.Repeat("9", 64)
b.haves.Write(context.Background(), Have{Module: "idle", FP: idleFP, Composed: idleDig, Phase: string(delivery.PhaseRunning), b.haves.Write(context.Background(), Have{Module: "idle", FP: idleFP, Composed: idleDig, Phase: string(delivery.PhaseRunning),
Sealed: []string{`{"box":"idle"}`}}) Sealed: delivery.Seals{"idle.key#sealed": `{"box":"idle"}`}})
want("idle", idleFP, "") want("idle", idleFP, "")
eventually(t, "idle's want was not taken", func() bool { eventually(t, "idle's want was not taken", func() bool {
b.ag.mu.Lock() b.ag.mu.Lock()
defer b.ag.mu.Unlock() defer b.ag.mu.Unlock()
return b.ag.wants["idle"].Present return b.ag.wants["idle"].Present
}) })
if h := b.haves.last("web"); len(h.Sealed) != 1 || h.Sealed[0] != "1" { if h := b.haves.last("web"); len(h.Sealed) != 1 || h.Sealed["web.pw#sealed"] != "1" {
t.Errorf("web's running have does not carry its sealed value: %+v", h) t.Errorf("web's running have does not carry its sealed value: %+v", h)
} }
b.ag.NotComposed(context.Background(), "a stand-in for web's password") b.ag.NotComposed(context.Background(), "a stand-in for web's password")
for _, row := range []struct { for _, row := range []struct {
module, fp, composed, sealed string module, fp, composed, slot, sealed string
}{{"web", fp, dig, "1"}, {"idle", idleFP, idleDig, `{"box":"idle"}`}} { }{{"web", fp, dig, "web.pw#sealed", "1"}, {"idle", idleFP, idleDig, "idle.key#sealed", `{"box":"idle"}`}} {
h := b.haves.last(row.module) h := b.haves.last(row.module)
if h.Phase != string(delivery.PhaseFailed) || h.FP != row.fp || h.Composed != row.composed || if h.Phase != string(delivery.PhaseFailed) || h.FP != row.fp || h.Composed != row.composed ||
len(h.Sealed) != 1 || h.Sealed[0] != row.sealed || !strings.Contains(h.Detail, "web's password") { len(h.Sealed) != 1 || h.Sealed[row.slot] != row.sealed || !strings.Contains(h.Detail, "web's password") {
t.Errorf("%s's have while the node cannot compose: %+v", row.module, h) t.Errorf("%s's have while the node cannot compose: %+v", row.module, h)
} }
} }
// A wanted module whose have cannot be read (the bus paused, issue 1126) is not marked: a mark would drop the build
// and digest its have names.
readAgain = time.Millisecond
b.haves.mu.Lock()
b.haves.unreadable = map[string]bool{"unread": true}
b.haves.mu.Unlock()
want("unread", fpOf("d"), "")
want("fresh", fpOf("e"), "") // never ran, its have absent: marked, naming nothing it runs
// A have that already says another cause keeps it (review of the switch, round 1): an apply that failed.
brokenFP, brokenDig := fpOf("f"), "sha256:"+strings.Repeat("8", 64)
b.haves.Write(context.Background(), Have{Module: "broken", FP: brokenFP, Composed: brokenDig, Phase: string(delivery.PhaseFailed),
Detail: "apply: unit failed to start"})
want("broken", brokenFP, "")
eventually(t, "unread's want was not taken", func() bool {
b.ag.mu.Lock()
defer b.ag.mu.Unlock()
return b.ag.wants["unread"].Present
})
b.ag.NotComposed(context.Background(), "a stand-in for web's password, again")
if h := b.haves.last("unread"); h.Module != "" {
t.Errorf("a module whose have could not be read was marked: %+v", h)
}
if h := b.haves.last("fresh"); h.Phase != string(delivery.PhaseFailed) || h.FP != "" {
t.Errorf("a module that never ran, its have absent, is not marked: %+v", h)
}
if h := b.haves.last("broken"); h.Detail != "apply: unit failed to start" || h.Composed != brokenDig {
t.Errorf("an apply failure was overwritten by a composer's mark: %+v", h)
}
// A run that ended marks nothing: what it would say is the next run's (review of the switch, round 1).
ended, end := context.WithCancel(context.Background())
end()
before := len(b.haves.log)
b.ag.NotComposed(ended, "a refusal from a run that ended")
if len(b.haves.log) != before {
t.Errorf("a run that ended wrote %d haves", len(b.haves.log)-before)
}
b.ag.NotComposed(context.Background(), "") b.ag.NotComposed(context.Background(), "")
for _, row := range []struct{ module, composed string }{{"web", dig}, {"idle", idleDig}} { for _, row := range []struct{ module, composed string }{{"web", dig}, {"idle", idleDig}} {
if h := b.haves.last(row.module); h.Phase != string(delivery.PhaseRunning) || h.Composed != row.composed || len(h.Sealed) != 1 { if h := b.haves.last(row.module); h.Phase != string(delivery.PhaseRunning) || h.Composed != row.composed || len(h.Sealed) != 1 {
t.Errorf("%s's have once the node composes again: %+v", row.module, h) t.Errorf("%s's have once the node composes again: %+v", row.module, h)
} }
} }
// Issue 1126, the applying path: a module the node holds alone (its own refusal: de-spiegel's unset route names on
// novox) is marked alone, naming its own words; web, which composes, keeps running, unmarked. Composed again, idle's
// have is what it runs.
b.ag.NotComposedEach(context.Background(), "", map[string]string{"idle": "idle says ${config:name}, and nothing sets it"})
if h := b.haves.last("idle"); h.Phase != string(delivery.PhaseFailed) || h.Composed != idleDig || !strings.Contains(h.Detail, "${config:name}") {
t.Errorf("idle's have while it alone is held: %+v", h)
}
if h := b.haves.last("web"); h.Phase != string(delivery.PhaseRunning) || h.Composed != dig {
t.Errorf("web, which composes, is marked beside idle: %+v", h)
}
b.ag.NotComposedEach(context.Background(), "", nil)
if h := b.haves.last("idle"); h.Phase != string(delivery.PhaseRunning) || h.Composed != idleDig {
t.Errorf("idle's have once it composes again: %+v", h)
}
// A revision at the same build (a config value changed) is applied. // A revision at the same build (a config value changed) is applied.
_, craw2, dig2 := unsigned("web", fp, file("web", "x", "2")) _, craw2, dig2 := unsigned("web", fp, file("web", "x", "2"))
b.comps.put(ck, craw2) b.comps.put(ck, craw2)
@@ -1069,7 +1203,7 @@ func TestTheNodesOwnCompositionIsAppliedWhenItDiffersAndARecordMoveWaitsForAYes(
// what runs (fingerprint, digest, sealed values) rather than naming nothing (review of L2, round 3). // what runs (fingerprint, digest, sealed values) rather than naming nothing (review of L2, round 3).
recFP, recDig := fpOf("d"), "sha256:"+strings.Repeat("8", 64) recFP, recDig := fpOf("d"), "sha256:"+strings.Repeat("8", 64)
b.haves.Write(context.Background(), Have{Module: "rec", FP: recFP, Composed: recDig, Phase: string(delivery.PhaseRunning), b.haves.Write(context.Background(), Have{Module: "rec", FP: recFP, Composed: recDig, Phase: string(delivery.PhaseRunning),
Sealed: []string{`{"box":"rec"}`}}) Sealed: delivery.Seals{"rec.key#sealed": `{"box":"rec"}`}})
rk, rraw, rdig := unsigned("rec", fpB, container("rec@sha256:"+strings.Repeat("3", 64))) rk, rraw, rdig := unsigned("rec", fpB, container("rec@sha256:"+strings.Repeat("3", 64)))
b.comps.put(dk2, draw2) b.comps.put(dk2, draw2)
b.comps.put(rk, rraw) b.comps.put(rk, rraw)
@@ -1078,15 +1212,15 @@ func TestTheNodesOwnCompositionIsAppliedWhenItDiffersAndARecordMoveWaitsForAYes(
applied := len(b.appliedList()) applied := len(b.appliedList())
for _, row := range []struct { for _, row := range []struct {
module, fp, composed, waits string module, fp, composed, waits string
sealed []string sealed delivery.Seals
}{{"db", fp, "", ddig2, nil}, {"rec", recFP, recDig, rdig, []string{`{"box":"rec"}`}}} { }{{"db", fp, "", ddig2, nil}, {"rec", recFP, recDig, rdig, delivery.Seals{"rec.key#sealed": `{"box":"rec"}`}}} {
eventually(t, row.module+"'s move did not wait for a yes", func() bool { eventually(t, row.module+"'s move did not wait for a yes", func() bool {
h := b.haves.last(row.module) h := b.haves.last(row.module)
return h.Phase == string(delivery.PhaseWaitingYes) && h.Waits == row.waits return h.Phase == string(delivery.PhaseWaitingYes) && h.Waits == row.waits
}) })
h := b.haves.last(row.module) h := b.haves.last(row.module)
if h.FP != row.fp || (row.composed != "" && h.Composed != row.composed) || h.Composed == "" || if h.FP != row.fp || (row.composed != "" && h.Composed != row.composed) || h.Composed == "" ||
(row.sealed != nil && !slices.Equal(h.Sealed, row.sealed)) { (row.sealed != nil && !maps.Equal(h.Sealed, row.sealed)) {
t.Errorf("%s's have waiting for a yes does not keep what runs: %+v", row.module, h) t.Errorf("%s's have waiting for a yes does not keep what runs: %+v", row.module, h)
} }
} }
@@ -1101,3 +1235,122 @@ func TestTheNodesOwnCompositionIsAppliedWhenItDiffersAndARecordMoveWaitsForAYes(
r.Phase == string(delivery.PhaseRunning) && r.FP == fpB && r.Composed == rdig r.Phase == string(delivery.PhaseRunning) && r.FP == fpB && r.Composed == rdig
}) })
} }
// 1957/AC1 and AC4 (unit; ADR 0568 §1, §4): an engine start takes a module from what the machine shows, not only from the
// have or the kept declaration it kept. Every resource up applies nothing; one declared running and not active applies the
// module once, from a have or from the kept declaration; a module with no want is not looked at.
func testAnEngineStartReconcilesAModuleFromTheMachinesFacts(t *testing.T) {
for _, c := range []struct {
name string
have bool // the engine's own have says running at the want
held bool // else its have says held, from a failed backup
kept bool // else the kept declaration says it runs
inactive bool
unwanted bool
applied int
asked bool // the machine was looked at
}{
{"all up: nothing is applied", true, false, false, false, false, 0, true},
{"a unit not active: applied once", true, false, false, true, false, 1, true},
{"kept declaration, a unit not active: applied once", false, false, true, true, false, 1, true},
{"kept declaration, all up: nothing is applied", false, false, true, false, false, 0, true},
{"a held module applies nothing at start", false, true, false, true, false, 0, false},
{"no want: nothing is looked at or applied", true, false, false, true, true, 0, false},
} {
t.Run(c.name, func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
rs := []json.RawMessage{file("web", "x", "1")}
ck, craw, dig := composition(t, "web", fp, false, rs...)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
if !c.unwanted {
b.wants.before(wk, wraw)
}
if c.have {
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: dig, Phase: "running"}}
}
if c.held {
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fpOf("0"), Composed: "sha256:" + strings.Repeat("0", 64),
Phase: string(delivery.PhaseHeld), Detail: "restic: the repository is locked"}}
}
if c.kept {
b.kept = map[string][]json.RawMessage{"web": rs}
}
var mu sync.Mutex
asked := false
b.env.Inactive = func(context.Context, delivery.Composed) bool {
mu.Lock()
defer mu.Unlock()
asked = true
return c.inactive
}
stop := b.run()
defer stop()
if !c.unwanted && !c.held {
eventually(t, "the module was not settled", func() bool { return b.haves.last("web").Composed == dig })
}
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != c.applied {
t.Fatalf("applied %v, want %d", got, c.applied)
}
mu.Lock()
defer mu.Unlock()
if asked != c.asked {
t.Fatalf("the machine was asked: %v, want %v", asked, c.asked)
}
})
}
}
// 1957/AC2, AC3 and AC4 (unit; ADR 0568 §2, §3, §4): the judge's edge, Reconcile, applies a running module again; one held
// for a failed backup and one with no want are left alone; and an apply that fails writes the have `failed` with the
// apply's own words.
func testReconcileAppliesARunningModuleAgainAndLeavesTheHeldAndTheUnwantedAlone(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ckW, crawW, digW := composition(t, "web", fp, false, file("web", "x", "1"))
wkW, wrawW := wantFor(t, "web", fp, digW, "")
ckD, crawD, digD := composition(t, "db", fp, true, file("db", "x", "1"))
wkD, wrawD := wantFor(t, "db", fp, digD, "")
b.comps.before(ckW, crawW)
b.wants.before(wkW, wrawW)
b.comps.before(ckD, crawD)
b.wants.before(wkD, wrawD)
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: digW, Phase: "running"},
"db": {Module: "db", FP: fpOf("0"), Composed: "sha256:" + strings.Repeat("0", 64), Phase: "running"}}
b.backup = func(string) Backup { return Backup{Words: "restic: the repository is locked"} }
stop := b.run()
defer stop()
eventually(t, "web was not settled", func() bool { return b.haves.last("web").Composed == digW })
eventually(t, "db was not held", func() bool { return b.haves.last("db").Phase == string(delivery.PhaseHeld) })
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied at the start: %v", got)
}
b.ag.Reconcile("db", "ghost")
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a held module and an unwanted one were applied: %v", got)
}
if h := b.haves.last("db"); h.Phase != string(delivery.PhaseHeld) {
t.Fatalf("the held module's have changed: %+v", h)
}
b.ag.Reconcile("web")
eventually(t, "web was not applied again", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "web is not running again", func() bool { return b.haves.last("web").Phase == string(delivery.PhaseRunning) })
b.mu.Lock()
b.failWith = "systemctl start web.service: exit status 1"
b.mu.Unlock()
b.ag.Reconcile("web")
eventually(t, "the failed re-apply did not write the have failed with its words", func() bool {
h := b.haves.last("web")
return h.Phase == string(delivery.PhaseFailed) && h.Detail == "systemctl start web.service: exit status 1"
})
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 2 {
t.Fatalf("a failed apply was tried again with no edge: %v", got)
}
}
+72 -2
View File
@@ -1223,6 +1223,19 @@ func applyScoped(
} }
} }
// A file written or removed in a unit directory is read by the service manager only after a reload, whether or not
// a service reflects it: a drop-in on reboot.target is reflected by no service, and without the reload the next
// reboot still runs the target as it was loaded (issue 1126). Once per apply.
if at := unitFileMoved(report.Outcomes); at != "" {
if u, ok := sys.(unitReloader); ok {
if err := u.ReloadUnits(ctx, run); err != nil {
failures = append(failures, &Error{Resource: at, Err: fmt.Errorf("reloading the service manager's units: %w", err)})
} else {
log(fmt.Sprintf(" reloaded the service manager's units: %s changed", at))
}
}
}
// Only the mesh says what is assigned: a carried bundle's silence undeclares no identity and no folder. // Only the mesh says what is assigned: a carried bundle's silence undeclares no identity and no folder.
if origin == store.OriginDeclared && scope.isNode() { if origin == store.OriginDeclared && scope.isNode() {
// And an identity, then a library group, no longer declared goes, once what ran as it went above (ADR 0328 // And an identity, then a library group, no longer declared goes, once what ran as it went above (ADR 0328
@@ -1997,11 +2010,68 @@ type serviceReloader interface {
ReloadService(ctx context.Context, run system.Runner, unit string) error ReloadService(ctx context.Context, run system.Runner, unit string) error
} }
// UnitDirs are the system service manager's unit directories: a file the apply writes or removes in one of them, or in a
// drop-in directory below one, is read by the manager only after a reload (issue 1126). Replaceable in tests.
var UnitDirs = []string{"/etc/systemd/system", "/run/systemd/system", "/usr/local/lib/systemd/system"}
// unitFileMoved is the first file this apply wrote or removed in a unit directory or a drop-in directory below one,
// or "" when it moved none.
func unitFileMoved(outcomes []Outcome) string {
for _, o := range outcomes {
// Only what changed the file on disk, as MovesRoot reads it: skipped, held, kept, forgotten and waiting come
// back on every apply and change nothing.
if declaration.Type(o.Type) != declaration.TypeFile {
continue
}
switch o.Action {
case "created", "updated", "corrected", "removed":
default:
continue
}
at := filepath.Clean(o.Target)
for _, dir := range UnitDirs {
if rel, err := filepath.Rel(dir, at); err == nil && rel != "." && !strings.HasPrefix(rel, "..") {
return at
}
}
}
return ""
}
// unitReloader is a service manager that caches unit files and must be told to read them again. // unitReloader is a service manager that caches unit files and must be told to read them again.
type unitReloader interface { type unitReloader interface {
ReloadUnits(ctx context.Context, run system.Runner) error ReloadUnits(ctx context.Context, run system.Runner) error
} }
// restReader is a service manager that can say whether a one-shot has nothing running now
// (novox/hq issue 1126).
type restReader interface {
ServiceAtRest(ctx context.Context, run system.Runner, unit string) (bool, error)
}
// declaredState is a unit's state as a declaration of want reads it. **A one-shot at rest is
// stopped to a declaration that it be stopped** (novox/hq issue 1126): a one-shot that does not
// linger reads "running" once it ran and worked — that is what a declaration that it run asks — but
// it has no process between its runs, so it is also as stopped as it can be, and stopping it changes
// nothing. Read only as "running", a boot-time cleanup declared stopped and enabled failed every
// apply after the boot that ran it: "was asked to be stopped and is running".
func declaredState(ctx context.Context, sys system.System, run Runner, unit, want string) (string, error) {
state, err := sys.ServiceState(ctx, run, unit)
if err != nil || want != "stopped" || state != "running" {
return state, err
}
if r, ok := sys.(restReader); ok {
rest, err := r.ServiceAtRest(ctx, run, unit)
if err != nil {
return "", err
}
if rest {
return "stopped", nil
}
}
return state, nil
}
// serviceSettle is how long the host waits before looking at a unit a second time. A test sets it // serviceSettle is how long the host waits before looking at a unit a second time. A test sets it
// to nothing; on a machine it is the window in which a daemon that refuses its configuration dies. // to nothing; on a machine it is the window in which a daemon that refuses its configuration dies.
var serviceSettle = 2 * time.Second var serviceSettle = 2 * time.Second
@@ -2154,7 +2224,7 @@ func applyServiceIn(ctx context.Context, sys system.System, r *declaration.Servi
// own stopping from anybody else's (novox/hq ADR 0269). // own stopping from anybody else's (novox/hq ADR 0269).
out.declared = &store.FoundUnit{Unit: r.Unit, State: r.State, Boot: r.Boot} out.declared = &store.FoundUnit{Unit: r.Unit, State: r.State, Boot: r.Boot}
before, err := sys.ServiceState(ctx, run, r.Unit) before, err := declaredState(ctx, sys, run, r.Unit, r.State)
if err != nil { if err != nil {
return out, err return out, err
} }
@@ -2165,7 +2235,7 @@ func applyServiceIn(ctx context.Context, sys system.System, r *declaration.Servi
// Read back. A service manager accepting a command says the transaction was accepted, // Read back. A service manager accepting a command says the transaction was accepted,
// not that the unit is running — one that starts and immediately dies satisfies it. // not that the unit is running — one that starts and immediately dies satisfies it.
after, err := sys.ServiceState(ctx, run, r.Unit) after, err := declaredState(ctx, sys, run, r.Unit, r.State)
if err == nil && r.State == "running" { if err == nil && r.State == "running" {
after, err = stayedRunning(ctx, sys, run, r.Unit) after, err = stayedRunning(ctx, sys, run, r.Unit)
} }
+82
View File
@@ -350,6 +350,42 @@ func TestAnUnknownServiceStateIsRefusedNotGuessed(t *testing.T) {
} }
} }
// A boot-time one-shot a package ships, declared stopped and enabled at boot, is satisfied while
// nothing of it runs, whatever its last run did (novox/hq issue 1126). The kernel module's
// linux-modules-cleanup.service failed on ace with "was asked to be stopped and is running": an
// inactive one-shot with ExecMainStatus=0 reads "running", and stopping it changes nothing.
func TestABootOneShotDeclaredStoppedIsSatisfiedAtRest(t *testing.T) {
for _, c := range []struct{ name, status string }{
{"exited 0: never ran, ran at boot, or unloaded", "0"},
} {
t.Run(c.name, func(t *testing.T) {
var asked []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
switch args[0] {
case "show":
return "LoadState=loaded\nActiveState=inactive\nType=oneshot\nRemainAfterExit=no\n" +
"ExecMainStatus=" + c.status + "\n", nil
case "is-enabled":
return "enabled\n", nil
}
asked = append(asked, strings.Join(args, " "))
return "", nil
}
d := parse(t, `{"declaration":1,"resources":[
{"id":"cleanup","type":"service","unit":"linux-modules-cleanup.service","state":"stopped","boot":"enabled"}
]}`)
_, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatalf("a one-shot at rest declared stopped failed: %v", err)
}
if len(asked) > 0 {
t.Errorf("the service manager was asked %q for a unit already as declared", asked)
}
})
}
}
func TestADroppedServiceIsGivenBackTheStateItWasFoundIn(t *testing.T) { func TestADroppedServiceIsGivenBackTheStateItWasFoundIn(t *testing.T) {
// novox/hq ADR 0118: undeclaring removes what the mesh made, gives back what it changed, and // novox/hq ADR 0118: undeclaring removes what the mesh made, gives back what it changed, and
// leaves what was the machine's. A service resource never installs a unit — so undeclaring it // leaves what was the machine's. A service resource never installs a unit — so undeclaring it
@@ -1225,6 +1261,52 @@ func TestAServiceIsRestartedWhenWhatItReflectsChanges(t *testing.T) {
if reloaded < 0 || reloaded > stop { if reloaded < 0 || reloaded > stop {
t.Errorf("the service was restarted without the unit files being read again first; commands were %v", commands) t.Errorf("the service was restarted without the unit files being read again first; commands were %v", commands)
} }
// A drop-in in a unit directory that no service reflects — the power module's bound on reboot.target
// (issue 1126) — is read by the service manager only after a reload: one per apply, and no restart.
// Unchanged, it reloads nothing.
units := filepath.Join(dir, "system")
saved := UnitDirs
UnitDirs = []string{units}
t.Cleanup(func() { UnitDirs = saved })
dropIn := filepath.Join(units, "reboot.target.d", "mesh-power.conf")
withDropIn := parse(t, fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"conf","type":"file","path":%q,"content":"second\n","mode":"0644"},
{"id":"svc","type":"service","unit":"thing.service","state":"running","restart-on":["conf"]},
{"id":"units","type":"directory","path":%q,"mode":"0755"},
{"id":"drop-ins","type":"directory","path":%q,"mode":"0755"},
{"id":"bound","type":"file","path":%q,"content":"[Unit]\nJobTimeoutSec=10min\n","mode":"0644"},
{"id":"bound-too","type":"file","path":%q,"content":"[Unit]\nJobTimeoutSec=10min\n","mode":"0644"}
]}`, path, units, filepath.Dir(dropIn), dropIn, filepath.Join(filepath.Dir(dropIn), "other.conf")))
for _, row := range []struct {
name string
reloads int
}{{"a drop-in written", 1}, {"the same drop-in again", 0}} {
commands = nil
if _, state, err = Apply(context.Background(), archHost(t), withDropIn, state,
store.OriginCarried, run, nil, nil); err != nil {
t.Fatal(err)
}
reloads := 0
for _, c := range commands {
if strings.Contains(c, "daemon-reload") {
reloads++
}
if strings.Contains(c, "stop thing.service") {
t.Errorf("%s: a drop-in no service reflects restarted one: %v", row.name, commands)
}
}
if reloads != row.reloads {
t.Errorf("%s: %d reloads, want %d; commands were %v", row.name, reloads, row.reloads, commands)
}
}
// A drop-in whose module is skipped behind its step, or held as found, comes back so on every apply: it moved
// nothing, and reloads nothing.
for _, action := range []string{"skipped", "held"} {
if at := unitFileMoved([]Outcome{{ID: "bound", Type: "file", Target: dropIn, Action: action}}); at != "" {
t.Errorf("a drop-in %s reloads the service manager (%s)", action, at)
}
}
} }
} }
+4
View File
@@ -290,3 +290,7 @@ func unpack(body []byte, into string) (int, error) {
} }
} }
} }
// Fetch is an archive at source as the host fetches every archive it unpacks, at most maxArchive bytes: what the node's
// own composer reads a moved module's templates archive with (issue 1126). Its digest is the caller's to check.
func Fetch(ctx context.Context, source string) ([]byte, error) { return fetch(ctx, source) }
+66 -19
View File
@@ -49,9 +49,12 @@ type haveState struct {
nodeComposed string nodeComposed string
// nodeSealed is, on a node composing its own, each value its own composition carries sealed to its key (ADR 0547 §3, // nodeSealed is, on a node composing its own, each value its own composition carries sealed to its key (ADR 0547 §3,
// issue 1126): kept in its key so a restart reuses them and applies nothing again. // issue 1126): kept in its key so a restart reuses them and applies nothing again.
nodeSealed []string nodeSealed delivery.Seals
// nodeCandidates is the node's answer to each placement candidate group (novox/hq ADR 0559 §4), written into its key. // nodeCandidates is the node's answer to each placement candidate proposal (novox/hq ADR 0559 §4), written into its key.
nodeCandidates map[string]delivery.CandidateAnswer nodeCandidates map[string]delivery.CandidateAnswer
// nodeComposes and nodeComposesWhy are whose compositions the node applies and why (HaveNode.Composes), written into
// its key.
nodeComposes, nodeComposesWhy string
// written is the build of each module this node-engine wrote a have key for and that the declaration it applied // written is the build of each module this node-engine wrote a have key for and that the declaration it applied
// still holds; removed is each such module the latest declaration no longer holds, with the build its key last // still holds; removed is each such module the latest declaration no longer holds, with the build its key last
// said, until the key is written once with the phase removed (issue 1369, novox/hq ADR 0408 §4: never deleted). // said, until the key is written once with the phase removed (issue 1369, novox/hq ADR 0408 §4: never deleted).
@@ -380,7 +383,7 @@ func (q *Session) writeHaves(ctx context.Context, bus Bus) {
case f.Failed: case f.Failed:
h.Phase = delivery.PhaseFailed h.Phase = delivery.PhaseFailed
} }
h.Sealed = f.Sealed h.SealedAt = f.Sealed
h.Health = s.healthOf(module, f) h.Health = s.healthOf(module, f)
if h.Detail == "" { if h.Detail == "" {
h.Detail = s.wordsOf(module, h.Health) h.Detail = s.wordsOf(module, h.Health)
@@ -421,8 +424,9 @@ func (q *Session) writeHaves(ctx context.Context, bus Bus) {
var raw []byte var raw []byte
facts := *s.node facts := *s.node
facts.Composed = s.nodeComposed facts.Composed = s.nodeComposed
facts.Sealed = s.nodeSealed facts.SealedAt = s.nodeSealed
facts.Candidates = s.nodeCandidates facts.Candidates = s.nodeCandidates
facts.Composes, facts.ComposesWhy = s.nodeComposes, s.nodeComposesWhy
if raw, err = delivery.EncodeHaveNode(facts); err == nil { if raw, err = delivery.EncodeHaveNode(facts); err == nil {
want := facts want := facts
if !q.putHave(write, bucket, key, raw, func(current []byte) bool { if !q.putHave(write, bucket, key, raw, func(current []byte) bool {
@@ -712,7 +716,7 @@ func (q *Session) NoteRemoved(ctx context.Context, module, build string) {
// NoteNode takes the digest of the node's own composition it runs and the values it carries sealed to the node's key // NoteNode takes the digest of the node's own composition it runs and the values it carries sealed to the node's key
// (nil for a composition the node did not make itself), and writes the node's key where it differs. // (nil for a composition the node did not make itself), and writes the node's key where it differs.
func (q *Session) NoteNode(ctx context.Context, composed string, sealed []string) { func (q *Session) NoteNode(ctx context.Context, composed string, sealed delivery.Seals) {
q.init() q.init()
s := &q.haves s := &q.haves
s.mu.Lock() s.mu.Lock()
@@ -729,8 +733,8 @@ func (q *Session) NoteNode(ctx context.Context, composed string, sealed []string
q.writeHaves(ctx, bus) q.writeHaves(ctx, bus)
} }
// NoteCandidates takes the node's answers to the placement candidates, by group (its shadow composer's, novox/hq ADR // NoteCandidates takes the node's answers to the placement candidates, by proposal (its shadow composer's, novox/hq ADR
// 0559 §4), and writes the node's key where it differs; a group left out is no longer answered. // 0559 §4), and writes the node's key where it differs; a proposal left out is no longer answered.
func (q *Session) NoteCandidates(ctx context.Context, answers map[string]delivery.CandidateAnswer) { func (q *Session) NoteCandidates(ctx context.Context, answers map[string]delivery.CandidateAnswer) {
q.init() q.init()
s := &q.haves s := &q.haves
@@ -752,68 +756,111 @@ func (q *Session) NoteCandidates(ctx context.Context, answers map[string]deliver
q.writeHaves(ctx, bus) q.writeHaves(ctx, bus)
} }
// NoteComposes says, in the node's own key, whose compositions the node applies (delivery.ComposesOwn or
// delivery.ComposesController) and why: written at each start of its agent, so a node returning to the controller's
// compositions says so where every have is read (issue 1126).
func (q *Session) NoteComposes(ctx context.Context, composes, why string) {
q.init()
s := &q.haves
s.mu.Lock()
changed := s.nodeComposes != composes || s.nodeComposesWhy != why
s.nodeComposes, s.nodeComposesWhy = composes, why
known := s.node != nil
s.mu.Unlock()
if !known || !changed {
return // said with the node's first facts
}
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
// ReadNodeHave is the node's own key as the bucket holds it, and whether it holds one: what the node's own composition // ReadNodeHave is the node's own key as the bucket holds it, and whether it holds one: what the node's own composition
// it last ran is and carries sealed, read after a restart (ADR 0547 §3). // it last ran is and carries sealed, read after a restart (ADR 0547 §3).
func (q *Session) ReadNodeHave(ctx context.Context) (delivery.HaveNode, bool) { func (q *Session) ReadNodeHave(ctx context.Context) (delivery.HaveNode, bool) {
n, ok, _ := q.ReadNodeHaveErr(ctx)
return n, ok
}
// ReadNodeHaveErr is ReadNodeHave telling the key absent (false, nil) from one that could not be read (false, the
// error): a bus that did not answer is never kept as a node that ran nothing (issue 1126, 2026-10-28).
func (q *Session) ReadNodeHaveErr(ctx context.Context) (delivery.HaveNode, bool, error) {
q.init() q.init()
q.mu.Lock() q.mu.Lock()
bus := q.bus bus := q.bus
q.mu.Unlock() q.mu.Unlock()
hb, ok := bus.(HaveBus) hb, ok := bus.(HaveBus)
if !ok || bus == nil { if !ok || bus == nil {
return delivery.HaveNode{}, false return delivery.HaveNode{}, false, ErrNotLinkedHaves
} }
key, err := delivery.HaveNodeKey(q.Membership.Node) key, err := delivery.HaveNodeKey(q.Membership.Node)
if err != nil { if err != nil {
return delivery.HaveNode{}, false return delivery.HaveNode{}, false, err
} }
read, cancel := context.WithTimeout(ctx, q.timeout()) read, cancel := context.WithTimeout(ctx, q.timeout())
defer cancel() defer cancel()
bucket, err := hb.HaveBind(read) bucket, err := hb.HaveBind(read)
if err != nil { if err != nil {
return delivery.HaveNode{}, false return delivery.HaveNode{}, false, err
} }
value, _, err := bucket.Get(read, key) value, _, err := bucket.Get(read, key)
if errors.Is(err, ErrHaveAbsent) {
return delivery.HaveNode{}, false, nil
}
if err != nil { if err != nil {
return delivery.HaveNode{}, false return delivery.HaveNode{}, false, err
} }
n, err := delivery.DecodeHaveNode(value) n, err := delivery.DecodeHaveNode(value)
if err != nil { if err != nil {
return delivery.HaveNode{}, false return delivery.HaveNode{}, false, nil // a key no engine can read names nothing it runs
} }
return n, true return n, true, nil
} }
// ReadHave is the have this node-engine last wrote for a module, and whether the bucket holds one. A bucket that cannot // ReadHave is the have this node-engine last wrote for a module, and whether the bucket holds one. A bucket that cannot
// be read answers as none: the agent then compares with the declaration it kept, and applies only what differs. // be read answers as none: the agent then compares with the declaration it kept, and applies only what differs.
func (q *Session) ReadHave(ctx context.Context, module string) (delivery.Have, bool) { func (q *Session) ReadHave(ctx context.Context, module string) (delivery.Have, bool) {
h, ok, _ := q.ReadHaveErr(ctx, module)
return h, ok
}
// ErrNotLinkedHaves is a have read with no link to the bus open that can bind the have bucket.
var ErrNotLinkedHaves = errors.New("no link to the bus that binds the have bucket is open")
// ReadHaveErr is ReadHave telling a key absent (false, nil) from one that could not be read (false, the error): a bus
// that does not answer for a moment is not a module that never ran (issue 1126).
func (q *Session) ReadHaveErr(ctx context.Context, module string) (delivery.Have, bool, error) {
q.init() q.init()
q.mu.Lock() q.mu.Lock()
bus := q.bus bus := q.bus
q.mu.Unlock() q.mu.Unlock()
hb, ok := bus.(HaveBus) hb, ok := bus.(HaveBus)
if !ok || bus == nil { if !ok || bus == nil {
return delivery.Have{}, false return delivery.Have{}, false, ErrNotLinkedHaves
} }
key, err := delivery.HaveKey(q.Membership.Node, module) key, err := delivery.HaveKey(q.Membership.Node, module)
if err != nil { if err != nil {
return delivery.Have{}, false return delivery.Have{}, false, err
} }
read, cancel := context.WithTimeout(ctx, q.timeout()) read, cancel := context.WithTimeout(ctx, q.timeout())
defer cancel() defer cancel()
bucket, err := hb.HaveBind(read) bucket, err := hb.HaveBind(read)
if err != nil { if err != nil {
return delivery.Have{}, false return delivery.Have{}, false, err
} }
value, _, err := bucket.Get(read, key) value, _, err := bucket.Get(read, key)
if errors.Is(err, ErrHaveAbsent) {
return delivery.Have{}, false, nil
}
if err != nil { if err != nil {
return delivery.Have{}, false return delivery.Have{}, false, err
} }
h, err := delivery.DecodeHave(value) h, err := delivery.DecodeHave(value)
if err != nil { if err != nil {
return delivery.Have{}, false return delivery.Have{}, false, nil // a have no engine can read names nothing it runs
} }
return h, true return h, true, nil
} }
// nothingJudged is the health of a module a look judged no resource of: ok when it was looked at and the module applied // nothingJudged is the health of a module a look judged no resource of: ok when it was looked at and the module applied
+6 -6
View File
@@ -582,17 +582,17 @@ func TestAnAppliedModulesHaveNamesTheFingerprintAndTheDigestAndFollowsItsHealth(
if got := l.have(t, "n1.web"); got.Phase != delivery.PhaseRemoved { if got := l.have(t, "n1.web"); got.Phase != delivery.PhaseRemoved {
t.Fatalf("removed: %+v", got) t.Fatalf("removed: %+v", got)
} }
q.NoteNode(context.Background(), dig, []string{"core"}) q.NoteNode(context.Background(), dig, delivery.Seals{"core-box#sealed": "core"})
if n, err := delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || n.Composed != dig || len(n.Sealed) != 1 || n.Sealed[0] != "core" { if n, err := delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || n.Composed != dig || len(n.SealedAt) != 1 || n.SealedAt["core-box#sealed"] != "core" {
t.Fatalf("the node key, with its own composition's sealed values: %+v %v", n, err) t.Fatalf("the node key, with its own composition's sealed values by slot: %+v %v", n, err)
} }
if n, ok := q.ReadNodeHave(context.Background()); !ok || n.Composed != dig || len(n.Sealed) != 1 { if n, ok := q.ReadNodeHave(context.Background()); !ok || n.Composed != dig || len(n.SealedAt) != 1 {
t.Fatalf("the node key read back: %+v %v", n, ok) t.Fatalf("the node key read back: %+v %v", n, ok)
} }
// The node's answers to placement candidates ride in the same key (ADR 0559 §4), and go when no group is answered. // The node's answers to placement candidates ride in the same key (ADR 0559 §4), and go when no group is answered.
answers := map[string]delivery.CandidateAnswer{"feat/x": {Head: "h", Verdict: delivery.CandidateSame, Same: []string{"web"}}} answers := map[string]delivery.CandidateAnswer{"p-1": {Proposal: "p-1", Revision: 1, Verdict: delivery.CandidateSame, Same: []string{"web"}}}
q.NoteCandidates(context.Background(), answers) q.NoteCandidates(context.Background(), answers)
if n, err := delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || n.Candidates["feat/x"].Head != "h" || n.Composed != dig { if n, err := delivery.DecodeHaveNode(l.kv["n1.node"].value); err != nil || n.Candidates["p-1"].Proposal != "p-1" || n.Composed != dig {
t.Fatalf("the node key with its answers: %+v %v", n, err) t.Fatalf("the node key with its answers: %+v %v", n, err)
} }
q.NoteCandidates(context.Background(), nil) q.NoteCandidates(context.Background(), nil)
+60 -2
View File
@@ -4,6 +4,7 @@ package link
import ( import (
"context" "context"
"sync/atomic"
"testing" "testing"
"time" "time"
@@ -14,7 +15,10 @@ import (
// The revisions a bucket keeps of a key, read over a real bus: oldest first, a delete and a purge said as deleted, and // The revisions a bucket keeps of a key, read over a real bus: oldest first, a delete and a purge said as deleted, and
// none for a key that never was (issue 1945: the node agent takes the composition its want names from them). // none for a key that never was (issue 1945: the node agent takes the composition its want names from them).
func TestNatsAKeysRevisionsAreReadBack(t *testing.T) { func TestNatsAKeysRevisionsAreReadBack(t *testing.T) {
srv, err := natsserver.NewServer(&natsserver.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true}) // A second account reads as a node's does: refused the one bucket "denied-it" (the seat's state before its grant arrives).
srv, err := natsserver.NewServer(&natsserver.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true,
Users: []*natsserver.User{{Username: "admin", Password: "a"}, {Username: "node", Password: "n", Permissions: &natsserver.Permissions{
Publish: &natsserver.SubjectPermission{Allow: []string{">"}, Deny: []string{"$JS.API.*.*.KV_denied-it", "$JS.API.*.*.*.KV_denied-it", "$JS.API.*.*.*.KV_denied-it.>", "$KV.denied-it.>"}}}}}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -23,7 +27,7 @@ func TestNatsAKeysRevisionsAreReadBack(t *testing.T) {
t.Fatal("the bus did not come up") t.Fatal("the bus did not come up")
} }
t.Cleanup(srv.Shutdown) t.Cleanup(srv.Shutdown)
conn, err := nats.Connect(srv.ClientURL()) conn, err := nats.Connect(srv.ClientURL(), nats.UserInfo("admin", "a"))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -65,4 +69,58 @@ func TestNatsAKeysRevisionsAreReadBack(t *testing.T) {
if got, err := bus.KVHistory(ctx, "composed-it", "it.never.x"); err != nil || len(got) != 0 { if got, err := bus.KVHistory(ctx, "composed-it", "it.never.x"); err != nil || len(got) != 0 {
t.Fatalf("a key that never was: %+v (%v)", got, err) t.Fatalf("a key that never was: %+v (%v)", got, err)
} }
// A bucket that is not there, and one the account may not read, are errors at once, never a wait: the node agent
// follows its placement key by one watch at its start (endWhenPlaced) and, refused, stays on the controller's compositions and says why.
if _, err := bus.KVWatch(ctx, "never-it", ">"); err == nil {
t.Error("a watch of a bucket that is not there gave no error")
}
if _, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: "denied-it", History: 1}); err != nil {
t.Fatal(err)
}
nodeConn, err := nats.Connect(srv.ClientURL(), nats.UserInfo("node", "n"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nodeConn.Close)
started := time.Now()
wctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if _, err := (OverNATS{Conn: nodeConn}).KVWatch(wctx, "denied-it", ">"); err == nil || wctx.Err() != nil {
t.Errorf("a watch the account may not make: error %v after %s, context %v", err, time.Since(started), wctx.Err())
} else {
t.Logf("a refused watch is an error after %s: %v", time.Since(started).Round(time.Millisecond), err)
}
// A bucket made after the watch was asked is then watched, on the bus's word that it was created and with no timer;
// a bucket the account may not read is no waiting but the error (endWhenPlaced, the shadow's candidates).
q := &Session{Membership: Membership{Node: "n1"}}
q.Attach(ctx, OverNATS{Conn: conn, JS: js})
var waited atomic.Int32
actx, acancel := context.WithTimeout(ctx, 20*time.Second)
defer acancel()
ch, err := q.KVWatchAwait(actx, "later-it", ">", func(error) { waited.Add(1) })
if err != nil || waited.Load() != 1 {
t.Fatalf("a bucket not there yet: watch %v, waiting told %d time(s)", err, waited.Load())
}
select {
case u := <-ch:
t.Fatalf("an update before the bucket exists: %+v", u)
case <-time.After(200 * time.Millisecond):
}
later, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: "later-it", History: 1})
if err != nil {
t.Fatal(err)
}
if _, err := later.PutString("placement.n1", "v"); err != nil {
t.Fatal(err)
}
for seen := false; !seen; {
select {
case u := <-ch:
seen = u.Key == "placement.n1" && string(u.Value) == "v"
case <-actx.Done():
t.Fatal("the bucket made after the watch was never watched")
}
}
} }
+3 -2
View File
@@ -5,6 +5,7 @@ import (
"strings" "strings"
"time" "time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health" "git.novox.be/novox/mesh-sdk/go/health"
) )
@@ -746,6 +747,6 @@ type ModuleFact struct {
// Judgement and Waits are, in phase waiting-yes, the judgement of the move and the composition waiting (ADR 0547 §3). // Judgement and Waits are, in phase waiting-yes, the judgement of the move and the composition waiting (ADR 0547 §3).
Judgement string Judgement string
Waits string Waits string
// Sealed is the values the composition it names carries sealed to this node's key (delivery.Have.Sealed). // Sealed is the values the composition it names carries sealed to this node's key, by slot (delivery.Have.SealedAt).
Sealed []string Sealed delivery.Seals
} }
+57
View File
@@ -239,3 +239,60 @@ func (q *Session) Hear(subject string) (<-chan []byte, func(), error) {
} }
return h.Hear(subject) return h.Hear(subject)
} }
// CreatedAdvisory is the subject the bus says a bucket's stream was created on (the server's own advisory; the same
// subject mesh-sdk's process.AwaitBucket binds on, and the grants part gives a principal for each bucket it may read).
func CreatedAdvisory(bucket string) string {
return "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_" + bucket
}
// KVWatchAwait is KVWatch for a bucket that may not be on the bus yet (the placement seat's states, made when their
// holder starts): the channel is returned at once, and while the bucket is missing it yields nothing and waiting is told
// why; when the bus says the bucket's stream was created the watch is made and its updates follow. No timer: the
// advisory is heard before the first try, so a creation between the try and the wait is not missed. Any other error of
// the first try is the caller's, as for KVWatch: a bucket the account may not read is not waited for.
func (q *Session) KVWatchAwait(ctx context.Context, bucket, pattern string, waiting func(error)) (<-chan KVUpdate, error) {
created, stop, err := q.Hear(CreatedAdvisory(bucket))
if err != nil {
return nil, err
}
in, err := q.KVWatch(ctx, bucket, pattern)
if err == nil {
stop()
return in, nil
}
if !errors.Is(err, nats.ErrBucketNotFound) {
stop()
return nil, err
}
waiting(err)
out := make(chan KVUpdate, 64)
go func() {
defer close(out)
defer stop()
for in == nil {
select {
case <-ctx.Done():
return
case _, ok := <-created:
if !ok {
return
}
w, err := q.KVWatch(ctx, bucket, pattern)
if err != nil {
waiting(err)
continue
}
in = w
}
}
for u := range in {
select {
case out <- u:
case <-ctx.Done():
return
}
}
}()
return out, nil
}
+319
View File
@@ -0,0 +1,319 @@
// Package ownresync hands the vault the value of this machine's own secret that runs, where the vault holds another
// (novox/hq issue 1126, cause C4).
//
// The vault's `move-in` never replaces a value it holds, so its first import of an own secret stays fixed; where the
// controller later made that secret anew (a rotation at start, a re-issue), the new value runs and the vault keeps the
// old one. The node's shadow composer sees it: composing from the vault's value, it opens both seals with this
// machine's key and its answer's shown body says delivery.MaskedAnotherValue. After each shadow answer this step asks the
// vault's `move-in` with replace_running: true and what runs, for each own secret so masked, once per running value, and
// says what it asked and what the vault answered. It is move-in's, the verb this node may ask already, so no seat and no
// bus grant changes for it.
//
// **Only while this node applies the controller's compositions**, and only for a module whose composition that runs is
// the one its want names: then the controller's value is what runs, and what runs is the truth the vault lacks. Once the
// node composes its own, the vault's value is the source and nothing is handed. **The switch** is the vault's
// (`own_take_switch`, a person's act): while it is off the vault refuses every ask, and this step says so once per value.
// The vault also refuses any value it changed in the last 15 minutes, so a re-issue the node has not applied yet is
// never put back.
package ownresync
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"sort"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/selfcompose"
)
// Verb is the vault's verb this step asks, with ReplaceRunning true in both steps.
const Verb = "move-in"
// ReplaceRunning is move-in's argument that makes it take the running value over the one the vault holds.
const ReplaceRunning = "replace_running"
// Env is what the step reads and asks.
type Env struct {
Node string
// Composes is whose compositions the node applies now: delivery.ComposesController, delivery.ComposesOwn, or ""
// while its agent has not decided. The step hands nothing but for ComposesController.
Composes func() string
// Running is what the node runs now (agent.Everything).
Running func() []delivery.Composed
// Get reads one key's latest value; nil and no error when there is none.
Get func(ctx context.Context, bucket, key string) ([]byte, error)
// Unseal opens a value sealed to this machine; Seal seals a value to a raw X25519 key in base64 (identity.Seal).
Unseal func(sealed string) ([]byte, error)
Seal func(to string, value []byte) (string, error)
// Ask asks one of the vault's verbs in this engine's own name (link.Session.AskVault).
Ask func(ctx context.Context, verb string, args map[string]any) (json.RawMessage, error)
Say func(string)
}
// Outcome is what the step did for one own secret: asked or not, and what was said.
type Outcome struct {
Module string
Name string
Asked bool
Said string
// Retry says a read or an ask the bus did not answer stopped it: no answer, kept as none, and the step is to be
// taken again after a bound (issue 1126, 2026-10-28: "want.shanks.brother could not be read ... no responders").
Retry bool
}
// Resync is the step of one node agent's run.
type Resync struct {
env Env
mu sync.Mutex
// done is, by module/name, the running value's fingerprint the vault was asked with or why it was not: each is
// asked, and each reason said, once.
done map[string]string
}
// New is the step for one node.
func New(env Env) *Resync {
if env.Say == nil {
env.Say = func(string) {}
}
return &Resync{env: env, done: map[string]string{}}
}
// Take reads the node's shadow answers and hands the vault what runs of each own secret they say another value of.
func (r *Resync) Take(ctx context.Context, answers map[string]delivery.CandidateAnswer) []Outcome {
seen := map[delivery.OwnValueDiffers]bool{}
var drift []delivery.OwnValueDiffers
proposals := make([]string, 0, len(answers))
for g := range answers {
proposals = append(proposals, g)
}
sort.Strings(proposals)
for _, g := range proposals {
for _, o := range delivery.OwnValuesDiffer(answers[g]) {
if !seen[o] {
seen[o] = true
drift = append(drift, o)
}
}
}
if len(drift) == 0 {
return nil
}
composes := ""
if r.env.Composes != nil {
composes = r.env.Composes()
}
var out []Outcome
for _, o := range drift {
res := r.one(ctx, composes, o)
if res.Said != "" || res.Retry {
out = append(out, res)
}
}
return out
}
// one is the step for one own secret: Said is empty when it was done before for the same value and reason.
func (r *Resync) one(ctx context.Context, composes string, o delivery.OwnValueDiffers) Outcome {
res := Outcome{Module: o.Module, Name: o.Name}
key := o.Module + "/" + o.Name
if composes == "" {
return res // the agent has not decided whose compositions it applies: taken again once it has
}
if composes != delivery.ComposesController {
return r.said(key, "not:composes", res, fmt.Sprintf("%s holds another value in the vault than runs here, and "+
"nothing is handed: this node composes its own (%q), so the vault's value is the source", key, composes))
}
sealed, why, failed := r.running(ctx, o)
if failed != nil {
// The bus did not answer: no reason to keep, and taken again (Retry); said once per reason.
res.Retry = true
return r.said(key, "retry:"+why, res, fmt.Sprintf("%s holds another value in the vault than runs here, and "+
"nothing is handed yet: %s; asked again", key, why))
}
if why != "" {
return r.said(key, "not:"+why, res, fmt.Sprintf("%s holds another value in the vault than runs here, and "+
"nothing is handed: %s", key, why))
}
sum := sha256.Sum256([]byte(sealed))
fp := hex.EncodeToString(sum[:])
r.mu.Lock()
asked := r.done[key] == fp
r.mu.Unlock()
if asked {
return res
}
value, err := r.env.Unseal(sealed)
if err != nil {
return r.said(key, "not:unseal:"+fp, res, fmt.Sprintf("%s: what runs does not open with this machine's key, so "+
"nothing is handed: %v", key, err))
}
defer wipe(value)
said, err := r.hand(ctx, o, value)
if err != nil {
if ctx.Err() != nil {
return res // the run ended: asked again in the next
}
if selfcompose.BusFailed(err) {
// The vault did not answer on the bus: no word of the vault's, so not kept as asked (Retry).
res.Retry = true
return r.said(key, "retry:ask", res, fmt.Sprintf("%s: the vault's %s with replace_running did not answer, "+
"and is asked again: %v", key, Verb, err))
}
said = fmt.Sprintf("%s: the vault's %s with replace_running refused: %v", key, Verb, err)
}
// Once per running value, whatever the vault answered: a refusal is the vault's word, said once, and a new value
// that runs asks again.
r.mu.Lock()
r.done[key] = fp
r.mu.Unlock()
res.Asked, res.Said = true, said
r.env.Say(said)
return res
}
// said says why once per reason, and keeps the reason as done.
func (r *Resync) said(key, reason string, res Outcome, why string) Outcome {
r.mu.Lock()
again := r.done[key] == reason
r.done[key] = reason
r.mu.Unlock()
if again {
return res
}
res.Said = why
r.env.Say(why)
return res
}
// running is the sealed value of o in the composition that runs, when that composition is the one the module's want
// names; or why it is not handed, with the error of a read the bus did not answer, which is no answer.
func (r *Resync) running(ctx context.Context, o delivery.OwnValueDiffers) (sealed, why string, failed error) {
var comp *delivery.Composed
if r.env.Running != nil {
for _, c := range r.env.Running() {
if c.Module == o.Module {
c := c
comp = &c
break
}
}
}
if comp == nil {
return "", o.Module + " runs no composition here", nil
}
wantKey, err := delivery.WantKey(r.env.Node, o.Module)
if err != nil {
return "", err.Error(), nil
}
raw, err := r.env.Get(ctx, delivery.AssignerWantsBucket, wantKey)
if err != nil {
return "", fmt.Sprintf("its want %s could not be read (%v)", wantKey, err), err
}
if raw == nil {
return "", fmt.Sprintf("it has no want %s", wantKey), nil
}
want, err := delivery.DecodeWant(raw)
if err != nil {
return "", fmt.Sprintf("its want %s is not one: %v", wantKey, err), nil
}
if want.Self || want.Removed || want.Composed == "" {
return "", fmt.Sprintf("its want %s names no composition of the controller's", wantKey), nil
}
// What runs is the composition the want names, as the module's have says: applied and running.
haveKey, err := delivery.HaveKey(r.env.Node, o.Module)
if err != nil {
return "", err.Error(), nil
}
raw, err = r.env.Get(ctx, delivery.HaveBucket, haveKey)
if err != nil {
return "", fmt.Sprintf("its have %s could not be read (%v)", haveKey, err), err
}
if raw == nil {
return "", fmt.Sprintf("it has no have %s", haveKey), nil
}
have, err := delivery.DecodeHave(raw)
if err != nil {
return "", fmt.Sprintf("its have %s is not one: %v", haveKey, err), nil
}
if have.Composed != want.Composed || have.Phase != delivery.PhaseRunning {
return "", fmt.Sprintf("what runs (%s, %s) is not yet the composition its want names (%s)", have.Composed,
have.Phase, want.Composed), nil
}
// On a node the resource's id carries its module's name (`only-office.needs-jwt`); delivery.OwnNeedName reads it.
id := o.Module + "." + catalogue.NeedID(o.Name)
for _, res := range comp.Resources {
var body struct {
ID string `json:"id"`
Sealed any `json:"sealed"`
}
if json.Unmarshal(res, &body) != nil {
continue
}
if name, own := delivery.OwnNeedName(o.Module, body.ID); !own || name != o.Name {
continue
}
if s, ok := body.Sealed.(string); ok && s != "" {
return s, "", nil
}
return "", fmt.Sprintf("its resource %s carries no value sealed to this machine", id), nil
}
return "", fmt.Sprintf("what runs holds no resource %s", id), nil
}
// hand asks the vault's two steps with value, and checks what the vault holds after by its fingerprint.
func (r *Resync) hand(ctx context.Context, o delivery.OwnValueDiffers, value []byte) (string, error) {
raw, err := r.env.Ask(ctx, Verb, map[string]any{ReplaceRunning: true, "module": o.Module, "name": o.Name,
"why": "this node's shadow answer says the vault holds another value than runs here"})
if err != nil {
return "", err
}
var opened struct {
Call string `json:"call"`
SealTo string `json:"seal_to"`
Secret string `json:"secret"`
}
if err := json.Unmarshal(raw, &opened); err != nil || opened.Call == "" || opened.SealTo == "" {
return "", fmt.Errorf("the vault answered no call and no key to seal to")
}
sealed, err := r.env.Seal(opened.SealTo, value)
if err != nil {
return "", fmt.Errorf("the vault's key for the call is not a sealing key: %w", err)
}
raw, err = r.env.Ask(ctx, Verb, map[string]any{ReplaceRunning: true, "call": opened.Call, "sealed": sealed})
if err != nil {
return "", err
}
var kept struct {
Secret string `json:"secret"`
From int `json:"from_generation"`
To int `json:"to_generation"`
Unchanged bool `json:"unchanged"`
Fingerprint string `json:"fingerprint"`
Said string `json:"said"`
}
if err := json.Unmarshal(raw, &kept); err != nil || kept.Secret == "" {
return "", fmt.Errorf("the vault answered %s with no secret", opened.Secret)
}
sum := sha256.Sum256(value)
if kept.Fingerprint != "sha256:"+hex.EncodeToString(sum[:]) {
return "", fmt.Errorf("the vault says it holds %s at generation %d, but not the value handed in", kept.Secret, kept.To)
}
if kept.Unchanged {
return fmt.Sprintf("handed the vault what runs of %s: it held that value already (generation %d)", kept.Secret, kept.To), nil
}
return fmt.Sprintf("handed the vault what runs of %s: generation %d replaced by %d, the value that runs", kept.Secret,
kept.From, kept.To), nil
}
func wipe(b []byte) {
for i := range b {
b[i] = 0
}
}
+117
View File
@@ -0,0 +1,117 @@
package ownresync
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Issue 1126, cause C4: after a shadow answer saying the vault holds another value of an own secret than runs, the node
// hands the vault what runs, in the verb's two steps, once per running value; and only while it applies the
// controller's compositions and runs the one its want names.
func TestTheNodeHandsTheVaultWhatRunsOfADriftedOwnSecret(t *testing.T) {
const running = "the-value-that-runs"
digest := "sha256:" + strings.Repeat("a", 64)
fp := "src2:" + strings.Repeat("b", 64)
sum := sha256.Sum256([]byte(running))
answers := map[string]delivery.CandidateAnswer{"g": {Proposal: "p-1", Revision: 1, Verdict: delivery.CandidateDiffers,
Differs: []delivery.ModuleDiffers{{Module: "only-office", Why: "the resource differs", Bodies: []delivery.ResourceBodies{
{ID: "only-office.needs-jwt", Composed: `{"sealed":"` + delivery.MaskedAnotherValue + `"}`, Running: `{"sealed":"` + delivery.MaskedAnotherValue + `"}`},
{ID: "only-office.needs-salt", Composed: `{"sealed":"<masked: another seal of the same value>"}`}}}}}}
for name, row := range map[string]struct {
composes string
runs string // the have's composition digest
refuse string // the vault's refusal
asks int // the vault's verb asked, both passes
says string
// Issue 1126, 2026-10-28: a read or an ask the bus did not answer is no answer: taken again (Retry), never kept.
wantRead string // the want's read: "" answers it, "fails" the bus does not answer, "absent" no key
retry bool
}{
"applies the controller's, runs its want": {delivery.ComposesController, digest, "", 2, "generation 1 replaced by 2", "", false},
"composes its own": {delivery.ComposesOwn, digest, "", 0, "composes its own", "", false},
"not decided yet": {"", digest, "", 0, "", "", false},
"runs not yet what its want names": {delivery.ComposesController, "sha256:" + strings.Repeat("c", 64), "", 0, "not yet the composition", "", false},
"the vault refuses": {delivery.ComposesController, digest, "switched off", 1, "switched off", "", false},
"the vault does not answer": {delivery.ComposesController, digest, "nats: no responders available for request", 2, "asked again", "", true},
"the want's read does not answer": {delivery.ComposesController, digest, "", 0, "asked again", "fails", true},
"it has no want": {delivery.ComposesController, digest, "", 0, "it has no want", "absent", false},
} {
var asked []map[string]any
var said []string
want, _ := delivery.EncodeWant(delivery.Want{FP: fp, Composed: digest})
have, _ := delivery.EncodeHave(delivery.Have{FP: fp, Composed: row.runs, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
r := New(Env{Node: "novox", Composes: func() string { return row.composes },
Running: func() []delivery.Composed {
return []delivery.Composed{{Node: "novox", Module: "only-office", FP: fp, Resources: []json.RawMessage{
// As novox runs it (its shadow answer, 2026-10-28): the id carries the module's name.
json.RawMessage(`{"id":"only-office.needs-jwt","module":"only-office","type":"file",` +
`"path":"/var/lib/only-office/jwt.secret","sealed":"box-of-the-running-value"}`)}}}
},
Get: func(_ context.Context, bucket, key string) ([]byte, error) {
switch {
case bucket == delivery.AssignerWantsBucket && key == "want.novox.only-office" && row.wantRead == "fails":
return nil, errors.New("the bucket mesh-delivery-assigner_wants: nats: no responders available for request")
case bucket == delivery.AssignerWantsBucket && key == "want.novox.only-office" && row.wantRead == "absent":
return nil, nil
case bucket == delivery.AssignerWantsBucket && key == "want.novox.only-office":
return want, nil
case bucket == delivery.HaveBucket && key == "novox.only-office":
return have, nil
}
return nil, nil
},
Unseal: func(sealed string) ([]byte, error) {
if sealed != "box-of-the-running-value" {
return nil, errors.New("not this machine's")
}
return []byte(running), nil
},
Seal: func(to string, value []byte) (string, error) { return to + "(" + string(value) + ")", nil },
Ask: func(_ context.Context, verb string, args map[string]any) (json.RawMessage, error) {
asked = append(asked, args)
if verb != "move-in" || args["replace_running"] != true {
t.Errorf("%s: asked %s with %v, not move-in with replace_running", name, verb, args)
}
if row.refuse != "" {
return nil, errors.New(row.refuse)
}
if args["call"] == nil {
if args["module"] != "only-office" || args["name"] != "jwt" || args["node"] != nil {
t.Errorf("%s: the first step names %v", name, args)
}
return json.RawMessage(`{"call":"c1","seal_to":"key","secret":"own/novox/only-office/jwt"}`), nil
}
if args["sealed"] != "key("+running+")" {
t.Errorf("%s: handed %v, not what runs sealed to the call's key", name, args["sealed"])
}
return json.RawMessage(`{"secret":"own/novox/only-office/jwt","from_generation":1,"to_generation":2,` +
`"fingerprint":"sha256:` + hex.EncodeToString(sum[:]) + `"}`), nil
},
Say: func(s string) { said = append(said, s) }})
first := r.Take(context.Background(), answers)
again := r.Take(context.Background(), answers)
if len(asked) != row.asks {
t.Errorf("%s: the vault asked %d times, want %d: %v", name, len(asked), row.asks, asked)
}
if retry := len(first) == 1 && first[0].Retry; retry != row.retry {
t.Errorf("%s: taken again %v, want %v: %+v", name, retry, row.retry, first)
}
if row.retry && (len(again) != 1 || !again[0].Retry || again[0].Said != "") || !row.retry && len(again) != 0 {
t.Errorf("%s: the same answer did or said something again: %+v", name, again)
}
all := strings.Join(said, "\n")
if row.says == "" && (len(first) != 0 || all != "") || row.says != "" && (len(first) != 1 || !strings.Contains(all, row.says)) {
t.Errorf("%s: did %+v and said %q, want %q", name, first, all, row.says)
}
if strings.Contains(all, running) {
t.Errorf("%s: the value is in what the node said", name)
}
}
}
+460 -134
View File
@@ -11,19 +11,26 @@
package selfcompose package selfcompose
import ( import (
"bytes"
"context" "context"
"crypto/sha256"
"encoding/hex"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"slices"
"sort" "sort"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose" "git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus" "git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/agent" "github.com/novox/mesh-host/internal/agent"
) )
@@ -46,18 +53,28 @@ type Env struct {
// Hear subscribes a subject: here the vault's word that a secret changed (VaultSecretChanged), on which the node // Hear subscribes a subject: here the vault's word that a secret changed (VaultSecretChanged), on which the node
// composes again, so a rotated secret or credential reaches it at once (ADR 0547 §2). Nil hears nothing. // composes again, so a rotated secret or credential reaches it at once (ADR 0547 §2). Nil hears nothing.
Hear func(subject string) (<-chan []byte, func(), error) Hear func(subject string) (<-chan []byte, func(), error)
// Refused is told why the node cannot compose its modules, and "" once it composes again, so each wanted module's // Refused is told why the node cannot compose its modules (why, the node refused whole), or, of a node that composes,
// have says it (agent.NotComposed): a refusal is loud where haves are read, not one journal line. Nil tells nobody. // each module it holds and why (held: refused alone, the rest composed and applied, frombus.Compose), and both empty
Refused func(why string) // once it composes every module again, so each module's have says it (agent.NotComposedEach): a refusal is loud where
// haves are read, not one journal line, and one module's never marks the others (issue 1126). Nil tells nobody.
Refused func(why string, held map[string]string)
// Running is what the node runs now (agent.Everything), and Unseal opens a value sealed to this machine: a secret // Running is what the node runs now (agent.Everything), and Unseal opens a value sealed to this machine: a secret
// the vault seals again for each ask is kept as the box the node already runs when that box opens to the same // the vault seals again for each ask is kept as the box the node already runs at the same slot (the same field of
// value, so neither the switch to composing its own nor a restart changes a composition (ADR 0547 §3). Either nil // the same resource of the same module, delivery.KeepSeals) when that box opens to the same value, so neither the
// keeps the vault's box. // switch to composing its own nor a restart changes a composition (ADR 0547 §3); a box held at another slot is
// never taken, though it opens to the same value (issue 1126, C1: a provider's grant and its consumer's secret).
// Either nil keeps the vault's box.
Running func() []delivery.Composed Running func() []delivery.Composed
Unseal func(sealed string) ([]byte, error) Unseal func(sealed string) ([]byte, error)
// Held is the sealed values a module's have says it runs (delivery.Have.Sealed), kept by the agent on the bus: what // Held is the sealed values a module's have says it runs, kept by the agent on the bus: what runs after a restart,
// runs after a restart, before the agent applied anything, and after the switch once the kept declaration is old. // before the agent applied anything, and after the switch once the kept declaration is old. By slot
Held func(module string) []string // (delivery.Have.SealedAt); legacy is the list a have written before slots holds (delivery.Have.Sealed), reused
// only within its own module.
// An error is a have the bus did not answer for, never one absent: the pass is then tried again (BusFailed).
Held func(module string) (slots delivery.Seals, legacy []string, err error)
// Fetch reads an archive at a URL in the artifact store: a moved module's templates archive, whose texts its built
// manifest carries (issue 1126). Read once per digest. Nil reads none, and a moved module does not compose.
Fetch func(ctx context.Context, url string) ([]byte, error)
} }
// VaultSecretChanged is the vault's event that a secret changed: a name and its generation, never a value. // VaultSecretChanged is the vault's event that a secret changed: a name and its generation, never a value.
@@ -77,14 +94,41 @@ type Composer struct {
modules map[string]delivery.Module modules map[string]delivery.Module
values map[string]map[string]any // by module, this node's values map[string]map[string]any // by module, this node's
manifests map[string][]byte // by digest, read once manifests map[string][]byte // by digest, read once
archives map[string][]byte // templates archives by digest, read once
handed map[string]string // composed key to the digest handed handed map[string]string // composed key to the digest handed
// sealedOwn is each own secret's sealed value as first handed, by module/name and generation: a value sealed again // sealedOwn is each own secret's sealed value as first handed, by module/name and generation: a value sealed again
// is other bytes, so the one first had is kept while the generation stands, and the composition stays the same. // is other bytes, so the one first had is kept while the generation stands, and the composition stays the same.
sealedOwn map[string]ownSealed sealedOwn map[string]ownSealed
said string said string
refusal string refusal string
caught bool // held is what the composer's last pass held: each module refused alone, and the node's own ("node") when it could
rev uint64 // compose none (heldNow).
held []string
// doing is what the composer waits on now (waiting), for a running answer to say (issue 1126, T2); its own lock, so
// it is read while c.mu is held.
doingMu sync.Mutex
doing string
caught bool
rev uint64
// missed counts the reads the bus did not answer (BusFailed), for a pass or an answer to tell whether it met one:
// a read that failed on the bus is tried again after a bound, never kept as an answer (issue 1126, 2026-10-28).
missed atomic.Uint64
// owed is set when a pass refused for a bus that did not answer: the composer composes again after a bound.
owed atomic.Bool
}
// note counts err when it is the bus not answering, and returns it.
func (c *Composer) note(err error) error {
if BusFailed(err) {
c.missed.Add(1)
}
return err
}
// get is env.Get, its failures noted.
func (c *Composer) get(ctx context.Context, bucket, key string) ([]byte, error) {
raw, err := c.env.Get(ctx, bucket, key)
return raw, c.note(err)
} }
// New is a composer for one node. // New is a composer for one node.
@@ -94,7 +138,7 @@ func New(env Env) *Composer {
} }
return &Composer{env: env, wants: map[string]delivery.Want{}, placements: map[string]delivery.Placement{}, return &Composer{env: env, wants: map[string]delivery.Want{}, placements: map[string]delivery.Placement{},
records: map[string]delivery.NodeRecord{}, inOwn: map[string]bool{}, modules: map[string]delivery.Module{}, values: map[string]map[string]any{}, records: map[string]delivery.NodeRecord{}, inOwn: map[string]bool{}, modules: map[string]delivery.Module{}, values: map[string]map[string]any{},
manifests: map[string][]byte{}, handed: map[string]string{}, sealedOwn: map[string]ownSealed{}} manifests: map[string][]byte{}, archives: map[string][]byte{}, handed: map[string]string{}, sealedOwn: map[string]ownSealed{}}
} }
// ownSealed is one own secret's sealed value at a generation and the key it is sealed to. // ownSealed is one own secret's sealed value at a generation and the key it is sealed to.
@@ -187,11 +231,18 @@ func (c *Composer) Watch(ctx context.Context, _ string) (<-chan agent.Update, er
caught := make([]bool, len(ins)) caught := make([]bool, len(ins))
ready := false ready := false
dirty := false dirty := false
// A pass refused for a bus that did not answer is made again after a bound (laterAgain), not kept until another
// input moves (issue 1126, 2026-10-28).
var again <-chan time.Time
tries := 0
for { for {
// Drain whatever came, then compose once for all of it. // Drain whatever came, then compose once for all of it.
select { select {
case <-ctx.Done(): case <-ctx.Done():
return return
case <-again:
again = nil
dirty = true
case t := <-merged: case t := <-merged:
if !t.ok { if !t.ok {
c.env.Say(fmt.Sprintf("the watch of %s ended; the node composes no more until its agent starts again", ins[t.in].bucket)) c.env.Say(fmt.Sprintf("the watch of %s ended; the node composes no more until its agent starts again", ins[t.in].bucket))
@@ -230,6 +281,14 @@ func (c *Composer) Watch(ctx context.Context, _ string) (<-chan agent.Update, er
return return
} }
} }
if c.owed.Load() {
wait := LaterWait(tries)
tries++
c.env.Say(fmt.Sprintf("the node composes again in %s: a read on the bus did not answer", wait))
again = time.After(wait)
} else {
tries, again = 0, nil
}
if !c.caughtSent() { if !c.caughtSent() {
select { select {
case out <- agent.Update{Caught: true}: case out <- agent.Update{Caught: true}:
@@ -320,7 +379,7 @@ func (c *Composer) takeRecordChanged(u agent.Update) bool {
} }
// takeModuleHave takes one of this node's modules' haves, and says whether it changed which it runs inside its own // takeModuleHave takes one of this node's modules' haves, and says whether it changed which it runs inside its own
// composition: one whose have names no fingerprint of the new form and is not removed (nats on novox: a build-... // composition: one whose have names no fingerprint of the new form, is not removed, and runs something (nats on novox: a build-...
// id; mesh-wireguard: none). Read from the haves, which outlast a restart, never from what the agent has applied // id; mesh-wireguard: none). Read from the haves, which outlast a restart, never from what the agent has applied
// this run: a fresh agent composes before it applies anything (review of the fold, round 1). // this run: a fresh agent composes before it applies anything (review of the fold, round 1).
func (c *Composer) takeModuleHave(u agent.Update) bool { func (c *Composer) takeModuleHave(u agent.Update) bool {
@@ -331,7 +390,11 @@ func (c *Composer) takeModuleHave(u agent.Update) bool {
in := false in := false
if !u.Deleted { if !u.Deleted {
if h, err := delivery.DecodeHave(u.Value); err == nil { if h, err := delivery.DecodeHave(u.Value); err == nil {
in = h.Phase != delivery.PhaseRemoved && !strings.HasPrefix(h.FP, delivery.SourcePrefix2) // A have that names nothing it runs, failed (the node could not compose a module it never ran: NotComposed), is
// no module run inside the node's own composition: taken as one, it was composed from its manifest past its
// build (issue 1126, review round 1 of the builds).
runsNothing := h.Phase == delivery.PhaseFailed && h.FP == "" && h.Composed == ""
in = h.Phase != delivery.PhaseRemoved && !runsNothing && !strings.HasPrefix(h.FP, delivery.SourcePrefix2)
} }
} }
c.mu.Lock() c.mu.Lock()
@@ -385,11 +448,10 @@ func (c *Composer) takeValue(u agent.Update) {
} }
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
var r struct { // Only a value its source lays: a default is the composition's already, as the controller reads them
Value json.RawMessage `json:"value"` // (catalogue.LaidValue, issue 1126, C2: Element's links blanked by a laid default).
} v, laid := catalogue.LaidValue(u.Value)
var v any if u.Deleted || !laid {
if u.Deleted || json.Unmarshal(u.Value, &r) != nil || len(r.Value) == 0 || json.Unmarshal(r.Value, &v) != nil {
delete(c.values[module], key) delete(c.values[module], key)
return return
} }
@@ -418,29 +480,84 @@ func (c *Composer) compose(ctx context.Context) []agent.Update {
_, nodeWanted := c.wants[agent.NodeModule] _, nodeWanted := c.wants[agent.NodeModule]
// What it runs inside its own composition and no want names at a build of the new form stays there, as the // What it runs inside its own composition and no want names at a build of the new form stays there, as the
// controller's composed writer folds it (issue 1126); compose folds none a want names. // controller's composed writer folds it (issue 1126); compose folds none a want names.
inOwn := c.inOwnOf(c.placements[c.env.Node]) own, placed := c.placements[c.env.Node]
inOwn := c.inOwnOf(own)
r := reader{c: c} r := reader{c: c}
c.mu.Unlock() c.mu.Unlock()
sort.Strings(behind) if !placed {
for _, module := range behind { // The catalogue no longer places it: the agent returns to the controller's compositions (endWhenUnplaced), and
digest, err := c.builtManifest(ctx, module, fps[module]) // this is no refusal of what it runs (issue 1126, the seed's revert).
if err != nil { c.sayOnce("this node's key in the seat's placement state is gone: it composes nothing of its own, and keeps running what it runs")
why := fmt.Sprintf("%s is wanted at %s, whose manifest cannot be read, so the node does not compose: %v", module, fps[module], err) c.owed.Store(false) // nothing of its own to compose, so no pass is owed and no timer armed again
c.refused(why)
c.sayOnce(why)
return nil
}
kept[module] = digest
}
compositions, _, err := frombus.Compose(ctx, r, c.env.Mesh, frombus.Scope{Node: c.env.Node, Kept: kept, InOwn: inOwn}, fps)
if err != nil {
c.refused(fmt.Sprintf("%v", err))
c.sayOnce(fmt.Sprintf("the node cannot compose its modules, and keeps running what it runs: %v", err))
return nil return nil
} }
c.refused("") // Composed from the build its want names (frombus.Scope.KeptAt, issue 1126).
c.sayOnce("") for _, module := range behind {
kept[module] = fps[module]
}
// What it runs: a folded module of an old-form build is carried in its own composition as it runs (issue 1126).
var running []delivery.Composed
if c.env.Running != nil {
running = c.env.Running()
}
scope := frombus.Scope{Node: c.env.Node, KeptAt: kept, InOwn: inOwn, Running: running}
missed := c.missed.Load()
compositions, _, held, err := frombus.Compose(ctx, r, c.env.Mesh, scope, fps)
err = c.missedIn(missed, err)
// A bus that does not answer for a moment (a reload of its accounts pauses it a few seconds) is no refusal: the pass is
// tried again after a growing wait before anything is marked (issue 1126).
for try := 0; err != nil && Transient(err) && ctx.Err() == nil && try < len(composeAgain); try++ {
c.env.Say(fmt.Sprintf("the node's composition met a bus that did not answer, and is tried again in %s: %v", composeAgain[try], err))
select {
case <-ctx.Done():
case <-time.After(composeAgain[try]):
}
if ctx.Err() == nil {
missed = c.missed.Load()
compositions, _, held, err = frombus.Compose(ctx, r, c.env.Mesh, scope, fps)
err = c.missedIn(missed, err)
}
}
if ctx.Err() != nil {
// The run ended (the agent begins again, on its own compositions or the controller's): what failed with it is
// no refusal, and nothing is marked (issue 1126: a pass in flight at the switch marked the next agent's modules).
return nil
}
if err != nil {
// Said before the agent marks the modules: the marks read each have first, which a paused bus delays.
c.sayOnce(fmt.Sprintf("the node cannot compose its modules, and keeps running what it runs: %v", err))
c.refused(fmt.Sprintf("%v", err), nil)
// A bus that did not answer past every try is no answer: composed again after a bound (issue 1126, 2026-10-28).
c.owed.Store(BusFailed(err))
return nil
}
// The boxes it runs, slot by slot: read before c.mu is taken, since the agent's lock is taken to read what runs.
missed = c.missed.Load()
for key, comp := range compositions {
compositions[key] = c.keepRunning(comp, running)
}
if c.missed.Load() > missed {
// A have of what runs the bus did not answer for: its boxes would be the vault's, sealed again, and the
// composition another; nothing is handed, and the pass is made again after a bound (issue 1126, 2026-10-28).
c.sayOnce("the node composes nothing new for now, and keeps running what it runs: the have of what it runs did not answer on the bus")
c.owed.Store(true)
return nil
}
c.owed.Store(false)
// A module refused alone is held: it keeps what it runs, its have says why, and every other module is applied (issue
// 1126: one module's refusal froze the whole node).
c.refused("", held)
if len(held) == 0 {
c.sayOnce("")
} else {
var lines []string
for _, m := range heldModules(held) {
lines = append(lines, m+": "+held[m])
}
c.sayOnce("the node composes its modules but holds " + strings.Join(heldModules(held), ", ") +
", which keep running what they run: " + strings.Join(lines, "; "))
}
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
var out []agent.Update var out []agent.Update
@@ -474,6 +591,11 @@ func (c *Composer) compose(ctx context.Context) []agent.Update {
out = append(out, agent.Update{Key: key, Value: raw, Revision: c.rev}) out = append(out, agent.Update{Key: key, Value: raw, Revision: c.rev})
} }
for key := range c.handed { for key := range c.handed {
if _, module, _, _, err := delivery.ParseComposedKey(key); err == nil && module != "" {
if _, isHeld := held[module]; isHeld {
continue // held: what was handed of it stays, so the node keeps running it
}
}
if !now[key] { if !now[key] {
c.rev++ c.rev++
delete(c.handed, key) delete(c.handed, key)
@@ -483,56 +605,107 @@ func (c *Composer) compose(ctx context.Context) []agent.Update {
return out return out
} }
// refused tells the agent why the node cannot compose, or "" that it composes again, once per change. // heldModules is the modules of held, sorted.
func (c *Composer) refused(why string) { func heldModules(held map[string]string) []string {
out := make([]string, 0, len(held))
for m := range held {
out = append(out, m)
}
sort.Strings(out)
return out
}
// refused tells the agent why the node cannot compose (why), or which modules it holds (held), or both empty that it
// composes every module again, once per change.
func (c *Composer) refused(why string, held map[string]string) {
said := why
for _, m := range heldModules(held) {
said += "\x00" + m + "\x00" + held[m]
}
c.mu.Lock() c.mu.Lock()
same := c.refusal == why same := c.refusal == said
c.refusal = why c.refusal = said
c.held = heldModules(held)
if why != "" {
c.held = append(c.held, delivery.HaveNodeWord)
}
c.mu.Unlock() c.mu.Unlock()
if !same && c.env.Refused != nil { if !same && c.env.Refused != nil {
c.env.Refused(why) c.env.Refused(why, held)
} }
} }
// runningBox is the box the node already runs that opens to the same value as box, or box: what the agent runs, and // heldNow is what the composer's last pass held, sorted.
// what each wanted module's have says it runs sealed (Held). Read before c.mu is taken, since the agent's lock is taken func (c *Composer) heldNow() []string {
// to read what runs. c.mu.Lock()
func (c *Composer) runningBox(box string) string { defer c.mu.Unlock()
if c.env.Unseal == nil || (c.env.Running == nil && c.env.Held == nil) { return slices.Clone(c.held)
return box }
// keepRunning is comp with each sealed value the node already runs at the same slot kept, where it opens to the same
// value (delivery.KeepSeals, issue 1126, C1): of what runs (running, the agent's), else of what its module's have says
// it runs (Held). A have written before slots names no slot: its boxes are reused within their own module alone, at a
// slot no box is kept for, and only one opening to the slot's value.
func (c *Composer) keepRunning(comp delivery.Composed, running []delivery.Composed) delivery.Composed {
if c.env.Unseal == nil {
return comp
} }
value, err := c.env.Unseal(box) module := comp.Module
if err != nil { if module == "" {
return box module = agent.NodeModule
}
var held []string
if c.env.Running != nil {
for _, comp := range c.env.Running() {
held = append(held, agent.SealedIn(comp)...)
}
} }
kept := delivery.Seals{}
var legacy []string
if c.env.Held != nil { if c.env.Held != nil {
c.mu.Lock() slots, old, err := c.env.Held(module)
// The node's own key among them (agent.NodeModule): its core boxes, as its last run said them. c.note(err)
modules := make([]string, 0, len(c.wants)) for slot, box := range slots {
for m := range c.wants { kept[slot] = box
modules = append(modules, m)
} }
c.mu.Unlock() legacy = old
sort.Strings(modules) }
for _, m := range modules { for _, r := range running {
held = append(held, c.env.Held(m)...) if r.Module == comp.Module {
for slot, box := range delivery.SealsIn(r) {
kept[slot] = box
}
} }
} }
for _, h := range held { for slot, box := range delivery.SealsIn(comp) {
if h == box { if _, ok := kept[slot]; ok {
continue continue
} }
if v, err := c.env.Unseal(h); err == nil && bytes.Equal(v, value) { for _, h := range legacy {
return h if same, known := c.sameValue(box, h); known && same {
kept[slot] = h
break
}
} }
} }
return box return delivery.KeepSeals(comp, kept, c.sameValue)
}
// waiting notes what the composer waits on until the returned func is called, which restores what it waited on before.
func (c *Composer) waiting(what string) func() {
c.doingMu.Lock()
before := c.doing
c.doing = what
c.doingMu.Unlock()
return func() {
c.doingMu.Lock()
c.doing = before
c.doingMu.Unlock()
}
}
// waitingOn is what the composer waits on now, in words.
func (c *Composer) waitingOn() string {
c.doingMu.Lock()
defer c.doingMu.Unlock()
if c.doing == "" {
return "nothing it reads (composing)"
}
return c.doing
} }
func (c *Composer) sayOnce(line string) { func (c *Composer) sayOnce(line string) {
@@ -548,30 +721,6 @@ func (c *Composer) sayOnce(line string) {
} }
} }
// builtManifest is the digest of the manifest a module's build at a fingerprint was made from (the builder's build key):
// what a node composes a module from when its want names another build than the trunk's.
func (c *Composer) builtManifest(ctx context.Context, module, fp string) (string, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return "", err
}
raw, err := c.env.Get(ctx, delivery.BuilderBuildsBucket, key)
if err != nil {
return "", err
}
if raw == nil {
return "", fmt.Errorf("the builder holds no build %s", key)
}
b, err := delivery.DecodeBuilderBuild(raw)
if err != nil {
return "", err
}
if b.Manifest == "" {
return "", fmt.Errorf("the build %s names no manifest", key)
}
return b.Manifest, nil
}
// Policy is a module's upgrade policy as its manifest at the indexer's trunk derives it (catalogue.DerivedUpgrade), for // Policy is a module's upgrade policy as its manifest at the indexer's trunk derives it (catalogue.DerivedUpgrade), for
// the agent's judgement of a move (ADR 0547 §3); empty when the manifest is not known or cannot be read. The manifest is // the agent's judgement of a move (ADR 0547 §3); empty when the manifest is not known or cannot be read. The manifest is
// read within ctx, the agent's (review of L2, round 1, finding 5), so a stopping agent stops the read. // read within ctx, the agent's (review of L2, round 1, finding 5), so a stopping agent stops the read.
@@ -638,7 +787,8 @@ func (r reader) Manifest(ctx context.Context, digest string) ([]byte, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
value, err := r.c.env.Get(ctx, delivery.ManifestsBucket, key) defer r.c.waiting("the manifest " + digest + " (" + delivery.ManifestsBucket + ")")()
value, err := r.c.get(ctx, delivery.ManifestsBucket, key)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -655,6 +805,77 @@ func (r reader) Manifest(ctx context.Context, digest string) ([]byte, error) {
return []byte(m.JSON), nil return []byte(m.JSON), nil
} }
func (r reader) Build(ctx context.Context, module, fp string) (*delivery.BuilderBuild, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return nil, err
}
defer r.c.waiting("the build of " + module + " at " + fp + " (" + delivery.BuilderBuildsBucket + ")")()
raw, err := r.c.get(ctx, delivery.BuilderBuildsBucket, key)
if err != nil || raw == nil {
return nil, err
}
b, err := delivery.DecodeBuilderBuild(raw)
if err != nil {
return nil, fmt.Errorf("the builder's %s: %w", key, err)
}
return &b, nil
}
// archiveWithin bounds one read of a templates archive in the artifact store.
const archiveWithin = 30 * time.Second
// Archive reads a templates archive once per digest (an archive is the whole of its digest), checked against it.
func (r reader) Archive(ctx context.Context, url, digest string) ([]byte, error) {
r.c.mu.Lock()
body, ok := r.c.archives[digest]
r.c.mu.Unlock()
if ok {
return body, nil
}
if r.c.env.Fetch == nil {
return nil, fmt.Errorf("this node reads no archive (%s)", url)
}
// Bounded: a store that does not answer must not hold the composer or the node's answer (issue 1126).
defer r.c.waiting("the archive " + url)()
fetching, cancel := context.WithTimeout(ctx, archiveWithin)
defer cancel()
body, err := r.c.env.Fetch(fetching, url)
if r.c.note(err) != nil {
return nil, err
}
sum := sha256.Sum256(body)
if got := "sha256:" + hex.EncodeToString(sum[:]); got != digest {
return nil, fmt.Errorf("%s was recorded as %s and what arrived is %s", url, digest, got)
}
r.c.mu.Lock()
r.c.archives[digest] = body
r.c.mu.Unlock()
return body, nil
}
// Served reads, of another node's module, the keys named of mesh-config's resolved values, one key each: the node is
// granted those keys alone (buscompose Principal.ServedReads, issue 1126).
func (r reader) Served(ctx context.Context, node, module string, keys []string) (map[string]any, error) {
defer r.c.waiting("what " + module + " serves on " + node + " (" + ResolvedBucket + ")")()
out := map[string]any{}
for _, k := range keys {
raw, err := r.c.get(ctx, ResolvedBucket, node+"."+module+"."+k)
if err != nil {
return nil, err
}
value, laid := catalogue.LaidValue(raw)
if raw == nil || !laid {
continue
}
out[k] = value
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
func (r reader) Values(_ context.Context, node, module string) (map[string]any, error) { func (r reader) Values(_ context.Context, node, module string) (map[string]any, error) {
if node != r.c.env.Node { if node != r.c.env.Node {
return nil, nil // a node reads its own slice alone (ADR 0547 §2) return nil, nil // a node reads its own slice alone (ADR 0547 §2)
@@ -675,7 +896,10 @@ func (r reader) OwnSecrets(ctx context.Context, node string) ([]compose.OwnSecre
if node != r.c.env.Node || r.c.env.Own == nil { if node != r.c.env.Node || r.c.env.Own == nil {
return nil, nil return nil, nil
} }
done := r.c.waiting("the vault's own secrets")
raw, err := r.c.env.Own(ctx, map[string]any{}) raw, err := r.c.env.Own(ctx, map[string]any{})
r.c.note(err)
done()
if err != nil { if err != nil {
return nil, fmt.Errorf("the vault's own: %w", err) return nil, fmt.Errorf("the vault's own: %w", err)
} }
@@ -696,25 +920,16 @@ func (r reader) OwnSecrets(ctx context.Context, node string) ([]compose.OwnSecre
if a.Node != node { if a.Node != node {
return nil, fmt.Errorf("the vault answered %q's own secrets to %s", a.Node, node) return nil, fmt.Errorf("the vault answered %q's own secrets to %s", a.Node, node)
} }
ids := make([]string, len(a.Own))
fresh := make([]string, len(a.Own))
for i, o := range a.Own {
ids[i] = o.Module + "/" + o.Name
if !r.c.keeps(ids[i], o.Generation, a.Key) {
fresh[i] = r.c.runningBox(string(o.Sealed))
}
}
r.c.mu.Lock() r.c.mu.Lock()
defer r.c.mu.Unlock() defer r.c.mu.Unlock()
out := make([]compose.OwnSecretRow, 0, len(a.Own)) out := make([]compose.OwnSecretRow, 0, len(a.Own))
for i, o := range a.Own { for _, o := range a.Own {
id := ids[i] // The box first had is kept while its generation stands, so sealing it again moves no composition; the box the
// node runs at the same slot is kept after the composition (keepRunning).
id := o.Module + "/" + o.Name
kept, ok := r.c.sealedOwn[id] kept, ok := r.c.sealedOwn[id]
if !ok || kept.generation != o.Generation || kept.key != a.Key { if !ok || kept.generation != o.Generation || kept.key != a.Key {
kept = ownSealed{generation: o.Generation, key: a.Key, sealed: fresh[i]} kept = ownSealed{generation: o.Generation, key: a.Key, sealed: string(o.Sealed)}
if kept.sealed == "" {
kept.sealed = string(o.Sealed)
}
r.c.sealedOwn[id] = kept r.c.sealedOwn[id] = kept
} }
out = append(out, compose.OwnSecretRow{Node: node, Module: o.Module, Name: o.Name, Origin: o.Origin, out = append(out, compose.OwnSecretRow{Node: node, Module: o.Module, Name: o.Name, Origin: o.Origin,
@@ -738,7 +953,10 @@ func (r reader) Credentials(ctx context.Context, node string, asConsumer, asProv
for _, k := range asProvider { for _, k := range asProvider {
ask = append(ask, asked{Role: "provider", CredentialKey: k}) ask = append(ask, asked{Role: "provider", CredentialKey: k})
} }
done := r.c.waiting("the vault's credentials")
raw, err := r.c.env.Own(ctx, map[string]any{"credentials": ask}) raw, err := r.c.env.Own(ctx, map[string]any{"credentials": ask})
r.c.note(err)
done()
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("the vault's own: %w", err) return nil, nil, fmt.Errorf("the vault's own: %w", err)
} }
@@ -759,28 +977,17 @@ func (r reader) Credentials(ctx context.Context, node string, asConsumer, asProv
if a.Node != node { if a.Node != node {
return nil, nil, fmt.Errorf("the vault answered %q's credentials to %s", a.Node, node) return nil, nil, fmt.Errorf("the vault answered %q's credentials to %s", a.Node, node)
} }
ids := make([]string, len(a.Credentials))
fresh := make([]string, len(a.Credentials))
for i, c := range a.Credentials {
ids[i] = c.Role + "|" + c.Name + "|" + c.Provision + "|" + c.Consumer + "|" + c.ConsumerModule + "|" + c.Local
if !r.c.keeps(ids[i], c.Generation, a.Key) {
fresh[i] = r.c.runningBox(string(c.Sealed))
}
}
r.c.mu.Lock() r.c.mu.Lock()
defer r.c.mu.Unlock() defer r.c.mu.Unlock()
var creds []compose.Credential var creds []compose.Credential
var issued []compose.Issued var issued []compose.Issued
for i, c := range a.Credentials { for _, c := range a.Credentials {
// The value first had is kept while its generation stands, so sealing it again moves no composition; and the // The value first had is kept while its generation stands, so sealing it again moves no composition; the box the
// first had is the box the node runs when it opens to the same value (runningBox). // node runs at the same slot is kept after the composition (keepRunning).
id := ids[i] id := c.Role + "|" + c.Name + "|" + c.Provision + "|" + c.Consumer + "|" + c.ConsumerModule + "|" + c.Local
kept, ok := r.c.sealedOwn[id] kept, ok := r.c.sealedOwn[id]
if !ok || kept.generation != c.Generation || kept.key != a.Key { if !ok || kept.generation != c.Generation || kept.key != a.Key {
kept = ownSealed{generation: c.Generation, key: a.Key, sealed: fresh[i]} kept = ownSealed{generation: c.Generation, key: a.Key, sealed: string(c.Sealed)}
if kept.sealed == "" {
kept.sealed = string(c.Sealed)
}
r.c.sealedOwn[id] = kept r.c.sealedOwn[id] = kept
} }
k := c.CredentialKey k := c.CredentialKey
@@ -796,10 +1003,129 @@ func (r reader) Credentials(ctx context.Context, node string, asConsumer, asProv
return creds, issued, nil return creds, issued, nil
} }
// keeps says the composer holds id's sealed value at this generation and key already. // Placed reads whether the catalogue places node now, for the node agent choosing whose compositions it applies at
func (c *Composer) keeps(id string, generation int, key string) bool { // each start: true when its placement key holds a value, false only when a read succeeds and finds no key or a delete
c.mu.Lock() // marker. A read that fails chooses neither (review of the shadow, round 1: a timeout or a bus not ready at a restart
defer c.mu.Unlock() // flipped a self-composing node to the controller's compositions for a whole run): it is tried again after wait(try),
kept, ok := c.sealedOwn[id] // each distinct failure said once, until ctx ends, which is then its error.
return ok && kept.generation == generation && kept.key == key //
// switched says the node composed from its own placement key before (its have says it composes its own,
// delivery.ComposesOwn, and that is kept on the bus, so it survives a restart). Only a node that never switched takes a
// read that fails (a timeout, or a state its account may not read yet, as the seat's before its grant arrives) as "not
// placed": it stays on the controller's compositions, says why once, and switches on a positive read of its key (the
// watch endWhenPlaced keeps). A node that switched never goes back on a failed read, however long it lasts.
func Placed(ctx context.Context, get func(ctx context.Context, bucket, key string) ([]byte, error), node string,
wait func(try int) time.Duration, say func(string), switched bool) (bool, error) {
key, err := delivery.PlacementKey(node)
if err != nil {
return false, err
}
said := ""
for try := 1; ; try++ {
raw, err := get(ctx, delivery.PlacementBucket, key)
if errors.Is(err, nats.ErrBucketNotFound) {
return false, nil // the seat has written no placement state yet: nothing places it, read and known
}
if err == nil {
return len(raw) > 0, nil
}
if ctx.Err() != nil {
return false, ctx.Err()
}
if !switched {
say(fmt.Sprintf("this node's key in the seat's placement state cannot be read (%s): it has never composed its own, so it applies the controller's compositions and switches when its key is read: %v", key, err))
return false, nil
}
if why := err.Error(); why != said {
say(fmt.Sprintf("this node's key in the seat's placement state cannot be read (%s): the node agent waits, choosing neither its own compositions nor the controller's: %v", key, err))
said = why
}
select {
case <-ctx.Done():
return false, ctx.Err()
case <-time.After(wait(try)):
}
}
}
// ComposedItsOwn says whether the node's own have, kept on the bus, says it composed from its placement key at its last
// start. A have that cannot be read says yes: the safe side, a node waits and does not fall back.
func ComposedItsOwn(ctx context.Context, get func(ctx context.Context, bucket, key string) ([]byte, error), node string) bool {
key, err := delivery.HaveNodeKey(node)
if err != nil {
return true
}
raw, err := get(ctx, delivery.HaveBucket, key)
if err != nil {
return true
}
if len(raw) == 0 {
return false
}
n, err := delivery.DecodeHaveNode(raw)
return err != nil || n.Composes == delivery.ComposesOwn
}
// composeAgain are the waits before a pass that met a bus not answering is tried again.
var composeAgain = []time.Duration{time.Second, 2 * time.Second, 4 * time.Second}
// laterAgain are the waits, after a pass or an answer refused for a bus that did not answer past composeAgain, before
// it is made again; the last repeats while the bus does not answer (issue 1126, 2026-10-28: a failed read was kept as
// the answer until another input moved).
var laterAgain = []time.Duration{5 * time.Second, 15 * time.Second, 30 * time.Second, time.Minute, 2 * time.Minute, 5 * time.Minute}
// LaterWait is the wait before the try'th making again (from 0), of a pass, an answer or an ask the bus did not answer.
func LaterWait(try int) time.Duration { return laterAgain[min(try, len(laterAgain)-1)] }
// ErrBusMissed is a pass that met a read the bus did not answer, though the composition was made around it (a have of
// sealed values that could not be read): never handed as made.
var ErrBusMissed = errors.New("a read on the bus did not answer during the pass")
// missedIn is err, or ErrBusMissed when no error came but a read since missed failed on the bus.
func (c *Composer) missedIn(missed uint64, err error) error {
if err == nil && c.missed.Load() > missed {
return ErrBusMissed
}
return err
}
// BusFailed says err is a read the bus did not answer, for any reason but the run's own end: Transient, a read that
// timed out, or no link to the bus. Such a read is tried again after a bound, never kept as an answer; a key absent, a
// value refused or a verdict is an answer (issue 1126, 2026-10-28). Matched by value and by words, as Transient.
func BusFailed(err error) bool {
if err == nil || errors.Is(err, context.Canceled) {
return false
}
if Transient(err) || errors.Is(err, context.DeadlineExceeded) || errors.Is(err, nats.ErrTimeout) ||
errors.Is(err, nats.ErrConnectionClosed) || errors.Is(err, nats.ErrDisconnected) || errors.Is(err, nats.ErrNoServers) {
return true
}
words := err.Error()
for _, w := range []string{"timeout", "deadline exceeded", "no link to the bus", "did not answer", "connection closed"} {
if strings.Contains(words, w) {
return true
}
}
return false
}
// Transient says err is the bus not answering for a moment, as a reload of its accounts makes it for a few seconds:
// JetStream not enabled, or no responders; each fails at once, so a pass tried again costs only its wait. A read that
// timed out is not one: it already waited longer than such a pause, and is most often a read the bus refuses. Matched
// by value and by words, since the vault's and the reads' errors arrive wrapped in words (issue 1126).
func Transient(err error) bool {
if err == nil {
return false
}
if errors.Is(err, nats.ErrJetStreamNotEnabled) || errors.Is(err, nats.ErrJetStreamNotEnabledForAccount) || errors.Is(err, nats.ErrNoResponders) ||
errors.Is(err, ErrBusMissed) {
return true
}
words := err.Error()
for _, w := range []string{"jetstream not enabled", "no responders"} {
if strings.Contains(words, w) {
return true
}
}
return false
} }
+764 -34
View File
@@ -2,10 +2,16 @@ package selfcompose
import ( import (
"context" "context"
"crypto/sha256"
"encoding/hex"
"encoding/json" "encoding/json"
"fmt" "fmt"
"os"
"slices"
"strconv"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"testing" "testing"
"time" "time"
@@ -13,6 +19,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/compose" "git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus" "git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/agent" "github.com/novox/mesh-host/internal/agent"
) )
@@ -236,8 +243,158 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
t.Fatalf("handed %v, want %s and %s", got, lemursKey, nodeKey) t.Fatalf("handed %v, want %s and %s", got, lemursKey, nodeKey)
} }
// Issue 1126, the switch made safe: a bus not answering for a moment (a reload of its accounts) is tried again and
// refuses nothing; a pass whose run ended refuses nothing either.
setFor(t, &composeAgain, []time.Duration{time.Millisecond, time.Millisecond, time.Millisecond})
fresh := func(get func(context.Context, string, string) ([]byte, error)) (*Composer, *[]string) {
var refusals []string
e := b.env()
e.Get = get
e.Refused = func(why string, held map[string]string) {
for _, m := range heldModules(held) {
why += "held " + m + ": " + held[m]
}
refusals = append(refusals, why)
}
c2 := New(e)
c.mu.Lock()
for k, v := range c.wants {
c2.wants[k] = v
}
for k, v := range c.placements {
c2.placements[k] = v
}
for k, v := range c.records {
c2.records[k] = v
}
for k, v := range c.modules {
c2.modules[k] = v
}
c.mu.Unlock()
return c2, &refusals
}
for name, row := range map[string]struct {
failures int // reads failing before the bus answers
cancel bool
composes bool
}{
"the bus paused for one read": {failures: 1, composes: true},
"the bus paused past every try": {failures: 100},
"the run ended mid-pass": {failures: 100, cancel: true},
} {
failed := 0
pctx, pcancel := context.WithCancel(ctx)
c2, refusals := fresh(func(gctx context.Context, bucket, key string) ([]byte, error) {
if failed < row.failures {
failed++
if row.cancel {
pcancel()
return nil, fmt.Errorf("the vault's own could not be asked: %w", context.Canceled)
}
return nil, fmt.Errorf("the revisions of %s in %s: %w", key, bucket, nats.ErrJetStreamNotEnabled)
}
b.mu.Lock()
defer b.mu.Unlock()
return b.kv[bucket+"/"+key], nil
})
handed := c2.compose(pctx)
pcancel()
switch {
case row.composes && (len(handed) == 0 || len(*refusals) != 0):
t.Errorf("%s: handed %d, refusals %q; want composed and nothing refused", name, len(handed), *refusals)
case !row.composes && !row.cancel && (len(*refusals) != 1 || failed != 1+len(composeAgain)):
t.Errorf("%s: refusals %q after %d reads; want one refusal after %d tries", name, *refusals, failed, 1+len(composeAgain))
case row.cancel && len(*refusals) != 0:
t.Errorf("%s: a pass whose run ended refused %q", name, *refusals)
}
}
if !Transient(fmt.Errorf("the manifest of avahi: the revisions of x in mesh-delivery-indexer_manifests: nats: jetstream not enabled")) ||
!Transient(fmt.Errorf("x: %w", nats.ErrNoResponders)) || Transient(fmt.Errorf("nats says ${dir:mesh-state}, and nats declares no directory")) ||
Transient(fmt.Errorf("the revisions of it2.mailer.domain: %w", context.DeadlineExceeded)) {
t.Error("Transient misjudges a bus that does not answer, or a refusal")
}
// Issue 1126, the applying path (review of the seed): one module's own refusal holds that module alone. On novox,
// de-spiegel's unset route names (its manifest at the indexer's commit e25e1beb, captured) refused the whole node and
// marked all its modules not composed. Now lemurs and proxy are handed; de-spiegel is held alone (issue 1960: a hold
// never spreads to proxy, which composes without its routes, nor to its smtp provider): what was handed of de-spiegel
// stays handed, so the node keeps running it, and the agent is told it alone, in its config form's words.
spiegel, err := os.ReadFile("testdata/de-spiegel-e25e1beb.module.json")
if err != nil {
t.Fatal(err)
}
proxy := `{"module":"proxy","version":"1","capabilities":["systemd"],"provides":[{"name":"route","scope":"mesh"},{"name":"smtp","scope":"mesh"}],` +
`"resources":[{"id":"service","type":"service","unit":"proxy.service","state":"running","boot":"enabled"}]}`
c3, told := fresh(env.Get)
for m, raw := range map[string]string{"de-spiegel": string(spiegel), "proxy": proxy} {
dig := delivery.ManifestDigest([]byte(raw))
enc, _ := delivery.EncodeManifest(delivery.Manifest{Module: m, JSON: raw})
k, _ := delivery.ManifestsKey(dig)
b.mu.Lock()
b.kv[delivery.ManifestsBucket+"/"+k] = enc
b.mu.Unlock()
mfp := delivery.SourcePrefix2 + strings.Repeat(map[string]string{"de-spiegel": "d", "proxy": "e"}[m], 64)
c3.modules[m] = delivery.Module{FP2: mfp, Manifest: dig, Repository: "novox/" + m, Commit: "c"}
c3.wants[m] = delivery.Want{FP: mfp, Self: true}
}
p3 := c3.placements["n1"]
p3.Modules = append(append([]delivery.PlacedModule{}, p3.Modules...), delivery.PlacedModule{Module: "proxy"}, delivery.PlacedModule{Module: "de-spiegel"})
c3.placements["n1"] = p3
r3 := c3.records["n1"]
r3.Capabilities = []string{"systemd", "container-runtime"}
c3.records["n1"] = r3
spiegelKey, _ := delivery.ComposedKey("n1", "de-spiegel", c3.wants["de-spiegel"].FP)
proxyKey, _ := delivery.ComposedKey("n1", "proxy", c3.wants["proxy"].FP)
c3.handed[spiegelKey] = "sha256:" + strings.Repeat("5", 64) // what the node runs of it, handed before
handed3 := map[string]agent.Update{}
for _, u := range c3.compose(ctx) {
handed3[u.Key] = u
}
lemursKey3, _ := delivery.ComposedKey("n1", "lemurs", c3.wants["lemurs"].FP)
if u, ok := handed3[lemursKey3]; !ok || u.Deleted {
t.Errorf("lemurs, which composes, is not handed beside de-spiegel's refusal: %v (told %q)", handed3, *told)
}
// Issue 1960: a hold never spreads. proxy is handed, composed without held de-spiegel's routes, which do not compose.
if u, ok := handed3[proxyKey]; !ok || u.Deleted {
t.Errorf("proxy is not handed beside held de-spiegel: %v (told %q)", handed3, *told)
}
if u, ok := handed3[spiegelKey]; ok {
t.Errorf("held de-spiegel is handed %+v, want what was handed of it kept", u)
}
if len(*told) != 1 || !strings.HasPrefix((*told)[0], "held de-spiegel: ") || !strings.Contains((*told)[0], "${config:praktijk-name}") ||
!strings.Contains((*told)[0], "without what it contributes to route") || strings.Contains((*told)[0], "held proxy") ||
strings.Contains((*told)[0], "held lemurs") {
t.Errorf("the agent was told %q, want de-spiegel held alone, in its config form's words, without its routes", *told)
}
// A read the bus did not answer, timed out or not, is tried again later; the run's own end and a refusal are not.
if !BusFailed(fmt.Errorf("the revisions of it2.mailer.domain: %w", context.DeadlineExceeded)) || !BusFailed(fmt.Errorf("x: %w", nats.ErrNoResponders)) ||
!BusFailed(fmt.Errorf("the have of web: %w", nats.ErrTimeout)) || BusFailed(fmt.Errorf("x: %w", context.Canceled)) ||
BusFailed(fmt.Errorf("nats says ${dir:mesh-state}, and nats declares no directory")) || BusFailed(nil) {
t.Error("BusFailed misjudges a read the bus did not answer, the run's end, or a refusal")
}
// Issue 1126, C2: a config value is laid as the controller lays it (catalogue.LaidValue), from the mesh's, the node's
// or the module's own and never from a default or an unset one, both of this node's and of what another node serves:
// a default laid blanked Element's links on ace.
for _, row := range []struct {
source string
laid bool
}{{"mesh", true}, {"node", true}, {"module", true}, {"default", false}, {"unset", false}, {"", false}} {
probe := New(env)
value := []byte(`{"value":["x"],"source":"` + row.source + `"}`)
probe.takeValue(agent.Update{Key: "n1.lemurs.links", Value: value})
_, laid := probe.values["lemurs"]["links"]
b.mu.Lock()
b.kv[ResolvedBucket+"/n2.lemurs.links"] = value
b.mu.Unlock()
served, err := reader{c: probe}.Served(ctx, "n2", "lemurs", []string{"links"})
if laid != row.laid || err != nil || (served["links"] != nil) != row.laid {
t.Errorf("a value from %q: laid %v, served %v (%v), want laid %v", row.source, laid, served, err, row.laid)
}
}
// A config value of another module, and a value changing nothing composed, hand nothing. // A config value of another module, and a value changing nothing composed, hand nothing.
b.put(t, ResolvedBucket, "n1.other.k", []byte(`{"value":1}`)) b.put(t, ResolvedBucket, "n1.other.k", []byte(`{"value":1,"source":"node"}`))
quiet(t, out) quiet(t, out)
// A want removed takes its composition back. // A want removed takes its composition back.
@@ -297,7 +454,7 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
t.Fatalf("keeper's composition does not carry the sealed value: %s", handed) t.Fatalf("keeper's composition does not carry the sealed value: %s", handed)
} }
before := handed before := handed
b.put(t, ResolvedBucket, "n1.other.k", []byte(`{"value":2}`)) b.put(t, ResolvedBucket, "n1.other.k", []byte(`{"value":2,"source":"node"}`))
quiet(t, out) quiet(t, out)
b.mu.Lock() b.mu.Lock()
generation++ generation++
@@ -313,32 +470,52 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
// node runs comes from keeper's have (Held); the vault seals the value again, other bytes, and the box the have holds // node runs comes from keeper's have (Held); the vault seals the value again, other bytes, and the box the have holds
// is kept, since it opens to the same value: the composition does not move. The box from before the rotation opens // is kept, since it opens to the same value: the composition does not move. The box from before the rotation opens
// to the old value, so the vault's is taken. // to the old value, so the vault's is taken.
boxOf := func(raw string) string { // Issue 1126, C1: a box is kept only at the slot that holds it (the same field of the same resource of the same
// module). A box another module holds (a provider's grant of the value its consumer holds as a secret, on one node)
// or another resource of keeper holds is never taken, though it opens to the same value; a have written before slots
// (a list) is reused within its own module alone.
boxOf := func(raw string) (string, string) {
c, err := delivery.DecodeComposed([]byte(raw)) c, err := delivery.DecodeComposed([]byte(raw))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
sealed := agent.SealedIn(c) sealed := delivery.SealsIn(c)
if len(sealed) != 1 { if len(sealed) != 1 {
t.Fatalf("keeper's composition carries %v", sealed) t.Fatalf("keeper's composition carries %v", sealed)
} }
return sealed[0] for slot, box := range sealed {
return slot, box
}
return "", ""
} }
// And the node's own key (review of L2, round 3, finding 1): a core box its own composition carries is held there, and slot, rotatedBox := boxOf(string(rotated.Value))
// kept after a restart as a module's is, so the node's own composition does not move either. _, beforeBox := boxOf(before)
for _, row := range []struct { for _, row := range []struct {
name, held, in string name, in, held string
slots delivery.Seals
kept bool kept bool
}{{"the have holds the rotated value", boxOf(string(rotated.Value)), "keeper", true}, fails bool // the have's read the bus did not answer: no composition, made again later (2026-10-28)
{"the have holds the value before the rotation", boxOf(before), "keeper", false}, }{
{"the node's own key holds the rotated value", boxOf(string(rotated.Value)), agent.NodeModule, true}} { {"the have holds the rotated value at its slot", "keeper", rotatedBox, delivery.Seals{slot: rotatedBox}, true, false},
{"the have holds the value before the rotation", "keeper", beforeBox, delivery.Seals{slot: beforeBox}, false, false},
{"a have written before slots holds the rotated value", "keeper", rotatedBox, nil, true, false},
{"another module holds the same value", "lemurs", rotatedBox, delivery.Seals{"lemurs.grant-x#sealed": rotatedBox}, false, false},
{"another resource of keeper holds the same value", "keeper", rotatedBox, delivery.Seals{"keeper.other#sealed": rotatedBox}, false, false},
{"the bus does not answer for the have", "keeper", rotatedBox, nil, false, true},
} {
restarted := env restarted := env
restarted.Running = func() []delivery.Composed { return nil } restarted.Running = func() []delivery.Composed { return nil }
restarted.Held = func(module string) []string { restarted.Held = func(module string) (delivery.Seals, []string, error) {
if module == row.in { if module != row.in {
return []string{row.held} return nil, nil, nil
} }
return nil if row.fails {
return nil, nil, fmt.Errorf("the have of %s: %w", module, nats.ErrTimeout)
}
if row.slots == nil {
return nil, []string{row.held}, nil
}
return row.slots, nil, nil
} }
restarted.Unseal = func(box string) ([]byte, error) { restarted.Unseal = func(box string) ([]byte, error) {
b.mu.Lock() b.mu.Lock()
@@ -349,13 +526,102 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
return nil, fmt.Errorf("not sealed to n1: %s", box) return nil, fmt.Errorf("not sealed to n1: %s", box)
} }
again := New(restarted) again := New(restarted)
again.wants = map[string]delivery.Want{"keeper": {FP: fp, Self: true}, agent.NodeModule: {}} c.mu.Lock()
rows, err := reader{c: again}.OwnSecrets(ctx, "n1") for k, v := range c.wants {
if err != nil || len(rows) != 1 { again.wants[k] = v
t.Fatalf("%s: own secrets %+v, %v", row.name, rows, err)
} }
if kept := rows[0].Sealed == row.held; kept != row.kept { for k, v := range c.placements {
t.Errorf("%s: composed with %s (held %s), kept %v, want %v", row.name, rows[0].Sealed, row.held, kept, row.kept) again.placements[k] = v
}
for k, v := range c.records {
again.records[k] = v
}
for k, v := range c.modules {
again.modules[k] = v
}
c.mu.Unlock()
var composed string
for _, u := range again.compose(ctx) {
if u.Key == keeperKey {
composed = string(u.Value)
}
}
if row.fails {
if composed != "" || !again.owed.Load() {
t.Errorf("%s: composed %v and owes a pass again %v; want nothing handed and a pass owed", row.name, composed != "", again.owed.Load())
}
// Then the catalogue no longer places the node: it composes nothing of its own, and owes no pass either, so
// no timer is armed again (rev-seed-same-3, round 1).
again.mu.Lock()
clear(again.placements)
again.mu.Unlock()
if again.compose(ctx); again.owed.Load() {
t.Errorf("%s: an unplaced node still owes a pass", row.name)
}
continue
}
if composed == "" {
t.Fatalf("%s: keeper was not composed", row.name)
}
if _, box := boxOf(composed); (box == row.held) != row.kept {
t.Errorf("%s: composed with %s (held %s), kept %v", row.name, box, row.held, row.kept)
}
}
// And the node's own key (review of L2, round 3, finding 1): a core box its own composition carries is held there, and
// kept after a restart as a module's is, so the node's own composition does not move either.
ownComp := delivery.Composed{Node: "n1", Resources: []json.RawMessage{json.RawMessage(`{"id":"core-box","sealed":"{\"ask\":999}"}`)}}
b.mu.Lock()
sealedAt[`{"ask":999}`] = generation + 1
b.mu.Unlock()
coreEnv := env
coreEnv.Unseal = func(box string) ([]byte, error) {
b.mu.Lock()
defer b.mu.Unlock()
if g, ok := sealedAt[box]; ok {
return []byte(fmt.Sprintf("password-%d", g)), nil
}
return nil, fmt.Errorf("not sealed to n1: %s", box)
}
coreEnv.Held = func(module string) (delivery.Seals, []string, error) {
if module == agent.NodeModule {
return delivery.Seals{"core-box#sealed": rotatedBox}, nil, nil
}
return nil, nil, nil
}
if got := delivery.SealsIn(New(coreEnv).keepRunning(ownComp, nil))["core-box#sealed"]; got != rotatedBox {
t.Errorf("the node's own core box after a restart is %s, want the one its key holds %s", got, rotatedBox)
}
// Issue 1126, C1, the seed-7 probe: each entry of a file's secrets (catalogue's intoFile, the shape captured from it
// with the module a composition names) is a slot of its own: the box that runs at that entry is kept where it opens
// to the same value; one at another entry is not, though it opens to the same.
b.mu.Lock()
sealedAt["box-again"] = sealedAt[rotatedBox]
sealedAt["box-other"] = sealedAt[rotatedBox] + 100
b.mu.Unlock()
ini := func(db, key string) delivery.Composed {
return delivery.Composed{Node: "n1", Module: "grafana", Resources: []json.RawMessage{json.RawMessage(
`{"content":"[database]\npassword = ${secret:postgres-database}\n[security]\nsecret_key = ${secret:secret-key}\n",` +
`"id":"grafana.ini","mode":"0600","module":"grafana","path":"/etc/grafana/grafana.ini","secrets":{"postgres-database":` +
strconv.Quote(db) + `,"secret-key":` + strconv.Quote(key) + `},"type":"file"}`)}}
}
for _, row := range []struct {
name string
held delivery.Seals
want string
}{
{"the box that runs at its entry", delivery.Seals{"grafana.ini#secrets.postgres-database": rotatedBox}, rotatedBox},
{"a box at another entry", delivery.Seals{"grafana.ini#secrets.secret-key": rotatedBox}, "box-again"},
} {
secretsEnv := coreEnv
secretsEnv.Held = func(module string) (delivery.Seals, []string, error) {
if module == "grafana" {
return row.held, nil, nil
}
return nil, nil, nil
}
got := delivery.SealsIn(New(secretsEnv).keepRunning(ini("box-again", "box-other"), nil))
if got["grafana.ini#secrets.postgres-database"] != row.want || got["grafana.ini#secrets.secret-key"] != "box-other" {
t.Errorf("%s: the secrets kept are %v", row.name, got)
} }
} }
@@ -382,7 +648,7 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
return nil, fmt.Errorf("not sealed to n1: %s", box) return nil, fmt.Errorf("not sealed to n1: %s", box)
} }
var refusals []string var refusals []string
pairEnv.Refused = func(why string) { refusals = append(refusals, why) } pairEnv.Refused = func(why string, held map[string]string) { refusals = append(refusals, why) }
pc := New(pairEnv) pc := New(pairEnv)
consumer := []compose.CredentialKey{{Provision: "pg", Consumer: "n1", ConsumerModule: "web", Provider: "n2"}} consumer := []compose.CredentialKey{{Provision: "pg", Consumer: "n1", ConsumerModule: "web", Provider: "n2"}}
provider := []compose.CredentialKey{{Provision: "s3", Consumer: "n3", ConsumerModule: "files", Provider: "n1"}} provider := []compose.CredentialKey{{Provision: "s3", Consumer: "n3", ConsumerModule: "files", Provider: "n1"}}
@@ -390,14 +656,19 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
if err != nil || len(creds) != 1 || len(issued) != 1 { if err != nil || len(creds) != 1 || len(issued) != 1 {
t.Fatalf("credentials %+v issued %+v, %v", creds, issued, err) t.Fatalf("credentials %+v issued %+v, %v", creds, issued, err)
} }
if creds[0].State != compose.CredentialHeld || creds[0].Provider != "n2" || creds[0].Sealed != `{"run":1}` || if creds[0].State != compose.CredentialHeld || creds[0].Provider != "n2" || creds[0].Sealed != `{"c":1}` ||
issued[0].Consumer != "n3" || issued[0].ForProvider != `{"p":1}` { issued[0].Consumer != "n3" || issued[0].ForProvider != `{"p":1}` {
t.Errorf("credentials %+v issued %+v", creds, issued) t.Errorf("credentials %+v issued %+v", creds, issued)
} }
// What runs at the credential's slot is kept, after the composition (keepRunning, issue 1126, C1).
webComp := delivery.Composed{Node: "n1", Module: "web", Resources: []json.RawMessage{json.RawMessage(`{"id":"web.secret-pg","sealed":"{\"c\":1}"}`)}}
if got := delivery.SealsIn(pc.keepRunning(webComp, pairEnv.Running()))["web.secret-pg#sealed"]; got != `{"run":1}` {
t.Errorf("web's credential composed as %s, want the box that runs", got)
}
// A node that cannot compose tells the agent once, and once again when it composes (the agent's haves say it). // A node that cannot compose tells the agent once, and once again when it composes (the agent's haves say it).
pc.refused("a stand-in for web's password") pc.refused("a stand-in for web's password", nil)
pc.refused("a stand-in for web's password") pc.refused("a stand-in for web's password", nil)
pc.refused("") pc.refused("", nil)
if len(refusals) != 2 || refusals[0] == "" || refusals[1] != "" { if len(refusals) != 2 || refusals[0] == "" || refusals[1] != "" {
t.Errorf("the agent was told %q", refusals) t.Errorf("the agent was told %q", refusals)
} }
@@ -420,11 +691,15 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
readOnly := &fakeBus{kv: map[string][]byte{delivery.ManifestsBucket + "/" + key: man, delivery.ManifestsBucket + "/" + kkey: kman}} readOnly := &fakeBus{kv: map[string][]byte{delivery.ManifestsBucket + "/" + key: man, delivery.ManifestsBucket + "/" + kkey: kman}}
senv := readOnly.env() senv := readOnly.env()
senv.Watch = sb.watch senv.Watch = sb.watch
var ownFails atomic.Bool // the vault's own on a bus that does not answer (2026-10-28, ace's answer to seed-7)
senv.Own = func(context.Context, map[string]any) (json.RawMessage, error) { senv.Own = func(context.Context, map[string]any) (json.RawMessage, error) {
if ownFails.Load() {
return nil, fmt.Errorf("the vault's own could not be asked: %w", nats.ErrNoResponders)
}
return json.RawMessage(`{"node":"n1","key":"sk","own":[]}`), nil // nothing issued for keeper: it is not placed here return json.RawMessage(`{"node":"n1","key":"sk","own":[]}`), nil // nothing issued for keeper: it is not placed here
} }
// What n1 runs: lemurs and its own composition, as its composer composes them from the trunk. // What n1 runs: lemurs and its own composition, as its composer composes them from the trunk.
running, _, err := frombus.Compose(ctx, fixedReader{placements: map[string]delivery.Placement{"n1": trunk}, running, _, _, err := frombus.Compose(ctx, fixedReader{placements: map[string]delivery.Placement{"n1": trunk},
records: map[string]delivery.NodeRecord{"n1": {Node: "n1", TunnelKey: "k", SealingKey: "sk", Capabilities: []string{"systemd"}}}, records: map[string]delivery.NodeRecord{"n1": {Node: "n1", TunnelKey: "k", SealingKey: "sk", Capabilities: []string{"systemd"}}},
modules: mods, manifests: map[string][]byte{digest: []byte(lemurs), kdigest: []byte(keeper)}}, frombus.Mesh{}, frombus.Scope{Node: "n1"}, map[string]string{"lemurs": fp}) modules: mods, manifests: map[string][]byte{digest: []byte(lemurs), kdigest: []byte(keeper)}}, frombus.Mesh{}, frombus.Scope{Node: "n1"}, map[string]string{"lemurs": fp})
if err != nil { if err != nil {
@@ -437,6 +712,18 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
} }
return out return out
} }
// The controller's compositions at n1's wants, in n1's composition bucket: what a head deleting its placement
// returns it to (the seed's revert); here the ones it runs.
composedBucket, _ := delivery.ComposedBucketFor("n1")
for key, c := range running {
raw, err := delivery.EncodeComposed(c)
if err != nil {
t.Fatal(err)
}
readOnly.mu.Lock()
readOnly.kv[composedBucket+"/"+key] = raw
readOnly.mu.Unlock()
}
answered := make(chan map[string]delivery.CandidateAnswer, 8) answered := make(chan map[string]delivery.CandidateAnswer, 8)
sh := NewShadow(senv, func(_ context.Context, a map[string]delivery.CandidateAnswer) { answered <- a }) sh := NewShadow(senv, func(_ context.Context, a map[string]delivery.CandidateAnswer) { answered <- a })
sctx, scancel := context.WithCancel(ctx) sctx, scancel := context.WithCancel(ctx)
@@ -447,7 +734,7 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
go sh.Keep(sctx, func(int) time.Duration { return 5 * time.Millisecond }) go sh.Keep(sctx, func(int) time.Duration { return 5 * time.Millisecond })
time.Sleep(30 * time.Millisecond) time.Sleep(30 * time.Millisecond)
sb.missing(delivery.PlacementCandidatesBucket, false) sb.missing(delivery.PlacementCandidatesBucket, false)
sb.made(6) sb.made(7)
rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: "n1", TunnelKey: "k", SealingKey: "sk", Capabilities: []string{"systemd"}}}) rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: "n1", TunnelKey: "k", SealingKey: "sk", Capabilities: []string{"systemd"}}})
tp, _ := delivery.EncodePlacement(trunk) tp, _ := delivery.EncodePlacement(trunk)
sb.put(delivery.PlacementBucket, "placement.n1", tp, 1) sb.put(delivery.PlacementBucket, "placement.n1", tp, 1)
@@ -456,14 +743,28 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
raw, _ := delivery.EncodeModule(m) raw, _ := delivery.EncodeModule(m)
sb.put(delivery.ModulesBucket, name, raw, 1) sb.put(delivery.ModulesBucket, name, raw, 1)
} }
wantLemurs, _ := delivery.WantKey("n1", "lemurs")
var lemursDigest string
for _, c := range running {
if c.Module == "lemurs" {
lemursDigest, _ = delivery.ComposedDigestOf(c)
}
}
wantRaw, err := delivery.EncodeWant(delivery.Want{FP: fp, Composed: lemursDigest})
if err != nil {
t.Fatal(err)
}
sb.put(delivery.AssignerWantsBucket, wantLemurs, wantRaw, 1)
wantNode, _ := delivery.WantNodeKey("n1")
nodeRaw, _ := delivery.EncodeWant(delivery.Want{})
sb.put(delivery.AssignerWantsBucket, wantNode, nodeRaw, 1)
candidate := func(group string, rev uint64, modules ...string) { candidate := func(group string, rev uint64, modules ...string) {
p := trunk p := trunk
p.Modules = nil p.Modules = nil
for _, m := range modules { for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m}) p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
} }
raw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Group: group, raw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: group, Placement: &p})
Pull: delivery.Head{Repository: delivery.CatalogueRepository, Number: 1, Head: "h-" + group}, Placement: &p})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -474,6 +775,23 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
candidate("adds", 4, "lemurs", "keeper") candidate("adds", 4, "lemurs", "keeper")
candidate("drops", 5) candidate("drops", 5)
candidate("ghost", 6, "lemurs", "ghost") candidate("ghost", 6, "lemurs", "ghost")
// A head deleting n1's placement file (the seed's revert, mesh-catalog #526): n1 returns to the controller's
// compositions, answered as such, not as every module removed.
if raw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "deletes"}); err != nil {
t.Fatal(err)
} else {
k, _ := delivery.PlacementCandidateKey("deletes", "n1")
sb.put(delivery.PlacementCandidatesBucket, k, raw, 7)
}
// A proposal that changes another node's key and reaches n1's composition (ADR 0567 §3): n1's candidate holds no
// placement of its own change (Reached, nil), and n1 answers against its current placement, never as one whose key
// the proposal takes away.
if raw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "reach", Reached: true}); err != nil {
t.Fatal(err)
} else {
k, _ := delivery.PlacementCandidateKey("reach", "n1")
sb.put(delivery.PlacementCandidatesBucket, k, raw, 8)
}
sb.caught() sb.caught()
var answers map[string]delivery.CandidateAnswer var answers map[string]delivery.CandidateAnswer
waitFor := func(why string, ok func(map[string]delivery.CandidateAnswer) bool) { waitFor := func(why string, ok func(map[string]delivery.CandidateAnswer) bool) {
@@ -487,7 +805,7 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
} }
} }
} }
waitFor("every group answered", func(a map[string]delivery.CandidateAnswer) bool { return len(a) == 4 }) waitFor("every group answered", func(a map[string]delivery.CandidateAnswer) bool { return len(a) == 6 })
// Every watch and get the shadow makes is one the node's grant allows (issue 1126, 2026-10-27 09:21). // Every watch and get the shadow makes is one the node's grant allows (issue 1126, 2026-10-27 09:21).
sb.mu.Lock() sb.mu.Lock()
for _, w := range sb.watches { for _, w := range sb.watches {
@@ -505,7 +823,7 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
readOnly.mu.Unlock() readOnly.mu.Unlock()
for group, want := range map[string]func(delivery.CandidateAnswer) bool{ for group, want := range map[string]func(delivery.CandidateAnswer) bool{
"same": func(a delivery.CandidateAnswer) bool { "same": func(a delivery.CandidateAnswer) bool {
return a.Verdict == delivery.CandidateSame && a.Head == "h-same" && a.Revision == 3 && len(a.Same) == 2 return a.Verdict == delivery.CandidateSame && a.Proposal == "same" && a.Revision == 3 && len(a.Same) == 2
}, },
"adds": func(a delivery.CandidateAnswer) bool { "adds": func(a delivery.CandidateAnswer) bool {
return a.Verdict == delivery.CandidateDiffers && len(a.Added) == 1 && a.Added[0].Module == "keeper" && return a.Verdict == delivery.CandidateDiffers && len(a.Added) == 1 && a.Added[0].Module == "keeper" &&
@@ -517,6 +835,14 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
"ghost": func(a delivery.CandidateAnswer) bool { "ghost": func(a delivery.CandidateAnswer) bool {
return a.Verdict == delivery.CandidateRefused && a.Refused != "" return a.Verdict == delivery.CandidateRefused && a.Refused != ""
}, },
"reach": func(a delivery.CandidateAnswer) bool {
return a.Verdict == delivery.CandidateSame && a.Returns == "" && a.Revision == 8 && len(a.Removed) == 0 &&
fmt.Sprint(a.Same) == "[lemurs node]"
},
"deletes": func(a delivery.CandidateAnswer) bool {
return a.Verdict == delivery.CandidateSame && a.Returns == delivery.ReturnsToController && a.Revision == 7 &&
len(a.Removed) == 0 && fmt.Sprint(a.Same) == "[lemurs node]"
},
} { } {
if !want(answers[group]) { if !want(answers[group]) {
t.Errorf("%s: %+v", group, answers[group]) t.Errorf("%s: %+v", group, answers[group])
@@ -529,9 +855,100 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
t.Errorf("answers handed again for nothing changed: %+v", a) t.Errorf("answers handed again for nothing changed: %+v", a)
case <-time.After(100 * time.Millisecond): case <-time.After(100 * time.Millisecond):
} }
// Issue 1126, 2026-10-28 (shanks answered probe/1126-seed-7 at revision 84 every 13-17 s): a candidate revision is
// answered once. A module's have whose health or words alone move (each look rewrites them), and a placement, a
// module or a config value written again with the value it held, answer nothing again; a new revision answers its
// group alone, and what runs moving answers every group once.
answering := func() int {
readOnly.mu.Lock()
defer readOnly.mu.Unlock()
n := 0
for _, l := range readOnly.said {
if strings.HasPrefix(l, "answering placement candidate ") {
n++
}
}
return n
}
drain := func() {
for {
select {
case answers = <-answered:
case <-time.After(150 * time.Millisecond):
return
}
}
}
lemursHave := func(detail string, rev uint64) {
raw, err := delivery.EncodeHave(delivery.Have{FP: fp, Composed: lemursDigest, Phase: delivery.PhaseRunning,
Health: delivery.HealthFailed, Detail: detail})
if err != nil {
t.Fatal(err)
}
hk, _ := delivery.HaveKey("n1", "lemurs")
sb.put(delivery.HaveBucket, hk, raw, rev)
}
lemursHave("its tool lemurs_health: no link to the bus is open", 10)
drain()
settled := answering()
lemursHave("its tool lemurs_health: no answer within 10s", 11)
lemursHave("its tool lemurs_health: no answer within 20s", 12)
sb.put(delivery.PlacementBucket, "placement.n1", tp, 2)
lraw, _ := delivery.EncodeModule(mods["lemurs"])
sb.put(delivery.ModulesBucket, "lemurs", lraw, 2)
drain()
if got := answering(); got != settled {
t.Errorf("answered %d more time(s) with no candidate revision and no input of the node changed", got-settled)
}
candidate("same", 8, "lemurs")
waitFor("the new revision answered", func(a map[string]delivery.CandidateAnswer) bool { return a["same"].Revision == 8 })
drain()
if got := answering(); got != settled+1 {
t.Errorf("a new revision of one group answered %d group(s), want 1", got-settled)
}
// Issue 1126, 2026-10-28 (seed-9's probe): each answer says which input woke it, so a loop is told from inputs moving.
lastSaid := func(prefix string) string {
readOnly.mu.Lock()
defer readOnly.mu.Unlock()
for i := len(readOnly.said) - 1; i >= 0; i-- {
if strings.HasPrefix(readOnly.said[i], prefix) {
return readOnly.said[i]
}
}
return ""
}
if got := lastSaid("answering placement candidate same "); got != "answering placement candidate same (revision 8; woken by the candidate)" {
t.Errorf("a new revision's answer said %q", got)
}
settled = answering()
hk, _ := delivery.HaveKey("n1", "lemurs")
moved, _ := delivery.EncodeHave(delivery.Have{FP: fp, Composed: "sha256:" + strings.Repeat("e", 64), Phase: delivery.PhaseRunning})
sb.put(delivery.HaveBucket, hk, moved, 13)
drain()
if got := answering(); got != settled+len(answers) {
t.Errorf("what runs moved, and %d answer(s) were composed again for %d group(s)", got-settled, len(answers))
}
if got := lastSaid("answering placement candidate drops "); got != "answering placement candidate drops (revision 5; woken by haves)" {
t.Errorf("an answer woken by what runs moving said %q", got)
}
// A read the bus did not answer is no answer to keep: the group is answered again after a bound, though nothing moved.
setFor(t, &laterAgain, []time.Duration{20 * time.Millisecond})
ownFails.Store(true)
candidate("adds", 9, "lemurs", "keeper")
waitFor("the answer met the vault not answering", func(a map[string]delivery.CandidateAnswer) bool {
return a["adds"].Revision == 9 && strings.Contains(a["adds"].Refused, "no responders")
})
ownFails.Store(false)
waitFor("answered again once the bus answers, with nothing else moved", func(a map[string]delivery.CandidateAnswer) bool {
return a["adds"].Revision == 9 && a["adds"].Verdict == delivery.CandidateDiffers
})
drain()
if got := lastSaid("answering placement candidate adds "); got != "answering placement candidate adds (revision 9; woken by a retry after a read the bus did not answer)" {
t.Errorf("an answer composed again after the bus did not answer said %q", got)
}
k, _ := delivery.PlacementCandidateKey("same", "n1") k, _ := delivery.PlacementCandidateKey("same", "n1")
sb.del(delivery.PlacementCandidatesBucket, k) sb.del(delivery.PlacementCandidatesBucket, k)
waitFor("the deleted group's answer stays", func(a map[string]delivery.CandidateAnswer) bool { _, ok := a["same"]; return !ok && len(a) == 3 }) waitFor("the deleted group's answer stays", func(a map[string]delivery.CandidateAnswer) bool { _, ok := a["same"]; return !ok && len(a) == 5 })
// Review of L2b, round 2: a watch ends while the link stays; a group's key is deleted before the shadow watches // Review of L2b, round 2: a watch ends while the link stays; a group's key is deleted before the shadow watches
// again. The new run reads afresh, so that group is no longer answered. // again. The new run reads afresh, so that group is no longer answered.
@@ -543,9 +960,209 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
sb.missing(delivery.PlacementCandidatesBucket, false) sb.missing(delivery.PlacementCandidatesBucket, false)
waitFor("a group deleted while the shadow was down is still answered", func(a map[string]delivery.CandidateAnswer) bool { waitFor("a group deleted while the shadow was down is still answered", func(a map[string]delivery.CandidateAnswer) bool {
_, ok := a["drops"] _, ok := a["drops"]
return !ok && len(a) == 2 return !ok && len(a) == 4
}) })
// Whose compositions a node applies at a start (review of the shadow, round 1): only a read that succeeds chooses; a
// failing read is tried again, said once, and chooses neither, even when the link ends.
for name, row := range map[string]struct {
answers []error // per try; nil then the value
value []byte
end bool
placed bool
tries int
quiet bool // read and known: nothing to say
// switched: the node's have says it composed its own before (it never falls back on a failed read).
switched bool
}{
"switched: placed after two failures": {answers: []error{fmt.Errorf("nats: timeout"), fmt.Errorf("nats: timeout")}, value: []byte(`{}`), placed: true, tries: 3, switched: true},
"never switched: a timeout is not placed, said once": {answers: []error{fmt.Errorf("nats: timeout")}, placed: false, tries: 1},
"no key": {placed: false, tries: 1},
"no placement state yet": {answers: []error{fmt.Errorf("the bucket %s: %w", delivery.PlacementBucket, nats.ErrBucketNotFound)}, placed: false, tries: 1, quiet: true},
"switched: failing until the link ends": {answers: []error{fmt.Errorf("jetstream not ready"), fmt.Errorf("jetstream not ready"),
fmt.Errorf("jetstream not ready")}, end: true, tries: 3, switched: true},
} {
lctx, lcancel := context.WithCancel(ctx)
tries := 0
var said []string
get := func(context.Context, string, string) ([]byte, error) {
tries++
if tries <= len(row.answers) {
if row.end && tries == len(row.answers) {
lcancel()
}
return nil, row.answers[tries-1]
}
return row.value, nil
}
placed, err := Placed(lctx, get, "n1", func(int) time.Duration { return time.Millisecond }, func(s string) { said = append(said, s) }, row.switched)
lcancel()
if placed != row.placed || tries != row.tries || (err != nil) != row.end || (len(row.answers) > 0 && !row.quiet) != (len(said) == 1) {
t.Errorf("%s: placed %v, err %v after %d reads, said %q", name, placed, err, tries, said)
}
}
// Whether the node composed its own before is read from its own have: own says yes, the controller's or no have says
// no, and a have that cannot be read says yes (the safe side: a node waits and does not fall back).
for name, row := range map[string]struct {
raw []byte
err error
want bool
}{
"own": {raw: []byte(`{"v":1,"composes":"own"}`), want: true},
"the controller's": {raw: []byte(`{"v":1,"composes":"controller"}`), want: false},
"no have yet": {want: false},
"a have that cannot be read": {err: fmt.Errorf("nats: timeout"), want: true},
"a have that is no have": {raw: []byte(`{`), want: true},
} {
get := func(context.Context, string, string) ([]byte, error) { return row.raw, row.err }
if got := ComposedItsOwn(ctx, get, "n1"); got != row.want {
t.Errorf("composed its own, %s: %v, want %v", name, got, row.want)
}
}
// Issue 1126, the shadow shows its work: one input not read after the wait is said, naming it, in the journal and in
// a refused answer to each group known; once read, the group is answered for real.
setFor(t, &notReadyAfter, 30*time.Millisecond)
nb := newPatternBus()
nenv := readOnly.env()
nenv.Watch = nb.watch
nenv.Running = senv.Running
var nsaid []string
var nmu sync.Mutex
nenv.Say = func(l string) { nmu.Lock(); nsaid = append(nsaid, l); nmu.Unlock() }
nanswered := make(chan map[string]delivery.CandidateAnswer, 8)
nsh := NewShadow(nenv, func(_ context.Context, a map[string]delivery.CandidateAnswer) { nanswered <- a })
nctx, ncancel := context.WithCancel(ctx)
defer ncancel()
go func() { _ = nsh.Run(nctx) }()
nb.made(7)
nb.put(delivery.PlacementBucket, "placement.n1", tp, 1)
nb.put(delivery.HaveBucket, "n1.node", rec, 1)
for name, m := range mods {
raw, _ := delivery.EncodeModule(m)
nb.put(delivery.ModulesBucket, name, raw, 1)
}
sameRaw, _ := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "same", Placement: &trunk})
sameKey, _ := delivery.PlacementCandidateKey("same", "n1")
nb.put(delivery.PlacementCandidatesBucket, sameKey, sameRaw, 3)
// Issue 1126, T3: an older group (a closed pull request's candidate, rev 2) is answered after the newest.
oldRaw, _ := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: "older", Placement: &trunk})
oldKey, _ := delivery.PlacementCandidateKey("older", "n1")
nb.put(delivery.PlacementCandidatesBucket, oldKey, oldRaw, 2)
for _, bucket := range []string{delivery.PlacementCandidatesBucket, delivery.PlacementBucket, delivery.HaveBucket, delivery.ModulesBucket, ResolvedBucket} {
nb.send(bucket, agent.Update{Caught: true})
}
select {
case a := <-nanswered:
if g := a["same"]; g.Verdict != delivery.CandidateRefused || !strings.Contains(g.Refused, delivery.AssignerWantsBucket) || g.Proposal != "same" {
t.Errorf("a shadow waiting for its wants answered %+v", g)
}
case <-time.After(5 * time.Second):
t.Fatal("a shadow not ready after its wait answered nothing")
}
nmu.Lock()
if !strings.Contains(strings.Join(nsaid, "\n"), "this node answers no placement candidate yet: the node has not read "+delivery.AssignerWantsBucket) {
t.Errorf("a shadow not ready said %q", nsaid)
}
nmu.Unlock()
setFor(t, &slowAfter, time.Nanosecond) // every answer is slow: said, and noted in words that do not move with the time
nb.send(delivery.AssignerWantsBucket, agent.Update{Caught: true})
select {
case a := <-nanswered:
if g := a["same"]; g.Verdict != delivery.CandidateSame || !strings.Contains(g.Note, "this answer took over") {
t.Errorf("the shadow, once ready, answered %+v", g)
}
// The newest first, handed at once: the older group still holds the answer it had.
if g := a["older"]; g.Verdict != delivery.CandidateRefused {
t.Errorf("the older group was answered before the newest was handed: %+v", g)
}
if g := (<-nanswered)["older"]; g.Verdict != delivery.CandidateSame {
t.Errorf("the older group, answered next: %+v", g)
}
// Issue 1126, T2: an answer says when it starts and when it ends (and, every minute between, what it waits on).
nmu.Lock()
said := strings.Join(nsaid, "\n")
for _, line := range []string{"the answer to same took", "answering placement candidate same (revision 3; woken by its first answer since the shadow began)",
"answered placement candidate same in", "answering placement candidate older (revision 2;"} {
if !strings.Contains(said, line) {
t.Errorf("%q was not said: %q", line, nsaid)
}
}
if strings.Index(said, "answering placement candidate same") > strings.Index(said, "answering placement candidate older") {
t.Errorf("the older group was answered first: %q", nsaid)
}
nmu.Unlock()
done := nsh.c.waiting("the vault's own secrets")
if got := nsh.c.waitingOn(); got != "the vault's own secrets" {
t.Errorf("a running answer would say it waits on %q", got)
}
done()
case <-time.After(10 * time.Second):
t.Fatal("the shadow, once ready, answered nothing")
}
// Issue 1126, probe #532 on shanks: what runs is the composition the have names, never the copy of the declaration
// kept before the switch (no fp, no digest): a module composed at the digest its have says runs is the same.
{
probe := delivery.CandidateAnswer{Proposal: "pro", Revision: 68, Verdict: delivery.CandidateDiffers, Same: []string{"adwaita"},
Differs: []delivery.ModuleDiffers{
{Module: "xterm", Composed: "sha256:3c0a31df64283c0678b1cdbeff8ac030609cab65cc7e47faaf714da743d9340e", Why: "the fingerprint differs"},
{Module: "zsh", Composed: "sha256:6093f31056d39eaf604410b7289ec6b5fe602b19c8547283562fcfc078c4844f", Resource: "zsh.bundle-tools", Why: "the resource differs"},
{Module: "brother", Composed: "sha256:0702b13991288da72d230460624e949d3c1d7d833eac2ec2134073c82e5a61cf",
Running: "sha256:c11c14af2738e3fcdb04da4004bbe84728bdefaf8c986d76601ffbe502435871", Resource: "brother.needs-broker-device", Why: "the resource differs"}}}
got := asHavesSay(probe, map[string]string{
"xterm": "sha256:3c0a31df64283c0678b1cdbeff8ac030609cab65cc7e47faaf714da743d9340e",
"zsh": "sha256:6093f31056d39eaf604410b7289ec6b5fe602b19c8547283562fcfc078c4844f",
"brother": "sha256:c11c14af2738e3fcdb04da4004bbe84728bdefaf8c986d76601ffbe502435871"})
if fmt.Sprint(got.Same) != "[adwaita xterm zsh]" || len(got.Differs) != 1 || got.Differs[0].Module != "brother" || got.Verdict != delivery.CandidateDiffers {
t.Errorf("shanks' probe answered %+v", got)
}
if one := asHavesSay(delivery.CandidateAnswer{Proposal: "pro", Revision: 1, Verdict: delivery.CandidateDiffers, Differs: probe.Differs[:1]},
map[string]string{"xterm": probe.Differs[0].Composed}); one.Verdict != delivery.CandidateSame {
t.Errorf("a module the same as its have says is answered %+v", one)
}
}
// The way back, row by row (review of the shadow, round 1): the controller's compositions at n1's wants compared
// with what runs; one differing is answered so; one the controller holds none of refuses, without returning.
var runsList []delivery.Composed
for _, c := range running {
runsList = append(runsList, c)
}
lemursKey, _ = delivery.ComposedKey("n1", "lemurs", fp)
changed := running[lemursKey]
changed.Resources = []json.RawMessage{json.RawMessage(`{"id":"lemurs.service","type":"service","module":"lemurs","unit":"lemurs.service","state":"stopped","boot":"disabled"}`)}
changedRaw, _ := delivery.EncodeComposed(changed)
for name, row := range map[string]struct {
lemurs []byte
check func(delivery.CandidateAnswer) bool
}{
"the controller's differs": {changedRaw, func(a delivery.CandidateAnswer) bool {
return a.Verdict == delivery.CandidateDiffers && a.Returns == delivery.ReturnsToController &&
len(a.Differs) == 1 && a.Differs[0].Module == "lemurs" && fmt.Sprint(a.Same) == "[node]"
}},
"the controller holds none": {nil, func(a delivery.CandidateAnswer) bool {
return a.Verdict == delivery.CandidateRefused && a.Returns == "" && strings.Contains(a.Refused, "lemurs: the controller holds no composition")
}},
} {
get := func(_ context.Context, bucket, key string) ([]byte, error) {
if key == lemursKey {
return row.lemurs, nil
}
readOnly.mu.Lock()
defer readOnly.mu.Unlock()
return readOnly.kv[bucket+"/"+key], nil
}
a := returning(ctx, get, "n1", map[string]delivery.Want{"lemurs": {FP: fp}, agent.NodeModule: {}}, runsList, "pro", 1)
if !row.check(a) {
t.Errorf("%s: %+v", name, a)
}
if _, err := delivery.EncodeHaveNode(delivery.HaveNode{Candidates: map[string]delivery.CandidateAnswer{"pro": a}}); err != nil {
t.Errorf("%s: the answer is not one a have holds: %v", name, err)
}
}
// Issue 1126, the seed's review: a module n1 runs inside its own composition (nats, at a build of the old form) is // Issue 1126, the seed's review: a module n1 runs inside its own composition (nats, at a build of the old form) is
// composed there again, not at the trunk's build as one added; lemurs at the build it runs, keeper (added) at the // composed there again, not at the trunk's build as one added; lemurs at the build it runs, keeper (added) at the
// trunk's. // trunk's.
@@ -556,6 +1173,54 @@ func TestTheNodeComposesItsWantsFromTheBus(t *testing.T) {
if fmt.Sprint(gotFps) != fmt.Sprint(map[string]string{"lemurs": fp, "keeper": other}) { if fmt.Sprint(gotFps) != fmt.Sprint(map[string]string{"lemurs": fp, "keeper": other}) {
t.Errorf("shadow builds %v; want lemurs at what runs, keeper at the trunk's, nats at none (folded)", gotFps) t.Errorf("shadow builds %v; want lemurs at what runs, keeper at the trunk's, nats at none (folded)", gotFps)
} }
// Issue 1126, the seed's refusals: what a node composes from its builds and reads of a provider elsewhere. A build
// is the builder's key; a templates archive is fetched once per digest and refused when it is not that digest; another
// node's served key is read alone, as the node's grant allows it (buscompose Principal.ServedReads).
built, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: digest})
buildKey, _ := delivery.BuilderBuildKey("lemurs", fp)
b.mu.Lock()
b.kv[delivery.BuilderBuildsBucket+"/"+buildKey] = built
b.kv[ResolvedBucket+"/n2.mailer.domain"] = []byte(`{"value":"example.org","source":"module"}`)
b.reads = nil
b.mu.Unlock()
fetched := 0
archive := []byte("an archive")
sum := sha256.Sum256(archive)
env.Fetch = func(context.Context, string) ([]byte, error) { fetched++; return archive, nil }
r := reader{c: New(env)}
if got, err := r.Build(ctx, "lemurs", fp); err != nil || got == nil || got.Manifest != digest {
t.Errorf("the build of lemurs: %+v, %v", got, err)
}
if got, err := r.Build(ctx, "lemurs", other); err != nil || got != nil {
t.Errorf("no build at another fingerprint: %+v, %v", got, err)
}
for range 2 {
if got, err := r.Archive(ctx, "http://store/v2/x/files/blobs/d", "sha256:"+hex.EncodeToString(sum[:])); err != nil || string(got) != "an archive" {
t.Errorf("the archive: %q, %v", got, err)
}
}
if _, err := r.Archive(ctx, "http://store/v2/x/files/blobs/e", "sha256:"+strings.Repeat("0", 64)); err == nil || fetched != 2 {
t.Errorf("fetched %d times (want once per digest), and an archive of another digest: %v", fetched, err)
}
if got, err := r.Served(ctx, "n2", "mailer", []string{"domain"}); err != nil || fmt.Sprint(got) != "map[domain:example.org]" {
t.Errorf("what n2's mailer serves: %v, %v", got, err)
}
served, err := buscompose.PermissionsFor(buscompose.Principal{Kind: buscompose.KindNode, Node: "n1", ServedReads: []string{"n2.mailer.domain"}})
if err != nil {
t.Fatal(err)
}
b.mu.Lock()
for _, read := range b.reads {
ok := grantAllows(t, "n1", read[0], read[1])
if read[0] == ResolvedBucket {
ok = slices.Contains(served.Publish, "$JS.API.CONSUMER.CREATE.KV_"+ResolvedBucket+".*.$KV."+ResolvedBucket+"."+read[1])
}
if !ok {
t.Errorf("the composer reads %s of %s, which the node's grant does not allow", read[1], read[0])
}
}
b.mu.Unlock()
} }
// patternBus is a bus whose every watch gets the updates of its bucket matching its pattern, as the bus's own watches // patternBus is a bus whose every watch gets the updates of its bucket matching its pattern, as the bus's own watches
@@ -719,3 +1384,68 @@ func (r fixedReader) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow
func (r fixedReader) Credentials(context.Context, string, []compose.CredentialKey, []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) { func (r fixedReader) Credentials(context.Context, string, []compose.CredentialKey, []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
return nil, nil, nil return nil, nil, nil
} }
func (r fixedReader) Build(context.Context, string, string) (*delivery.BuilderBuild, error) {
return nil, nil
}
func (r fixedReader) Archive(_ context.Context, url, _ string) ([]byte, error) {
return nil, fmt.Errorf("no archive %s", url)
}
func (r fixedReader) Served(context.Context, string, string, []string) (map[string]any, error) {
return nil, nil
}
// TestAHaveSaysWhetherItsModuleRunsInsideTheNodesOwn feeds takeModuleHave a module's have and reads whether the node
// counts the module as run inside its own composition (issue 1126, review round 2): a failed have that names nothing
// it runs (NotComposed before the module ever ran) is not, so the next composition does not take the module from its
// manifest past its build.
func TestAHaveSaysWhetherItsModuleRunsInsideTheNodesOwn(t *testing.T) {
for _, row := range []struct {
name string
have delivery.Have
in bool
// heldToday says the shadow counts the module held today (issue 1126: a seed holding it too holds it as today).
heldToday bool
}{
{"failed, naming nothing it runs", delivery.Have{Phase: delivery.PhaseFailed, Health: delivery.HealthFailed,
Detail: "the node cannot compose it, and keeps running what it ran: no build"}, false, true},
{"failed at a build fingerprint", delivery.Have{FP: "build-0123abcd", Composed: "sha256:" + strings.Repeat("c", 64), Phase: delivery.PhaseFailed,
Health: delivery.HealthFailed, Detail: "the node cannot compose it, and keeps running what it ran: no build"}, true, true},
// The live have of novox's de-spiegel (2026-10-28): a build of the old form, named by `build`, and no `fp`.
{"running a build of the old form", delivery.Have{Build: "build-1790368817085312877", Phase: delivery.PhaseRunning, Health: delivery.HealthOK}, true, true},
{"running a build of the new form", delivery.Have{FP: delivery.SourcePrefix2 + strings.Repeat("a", 64), Composed: "sha256:" + strings.Repeat("c", 64),
Phase: delivery.PhaseRunning, Health: delivery.HealthOK}, false, false},
{"running with no fingerprint (mesh-wireguard)", delivery.Have{Phase: delivery.PhaseRunning, Health: delivery.HealthOK}, true, false},
{"held today", delivery.Have{FP: "build-0123abcd", Composed: "sha256:" + strings.Repeat("c", 64), Phase: delivery.PhaseHeld,
Health: delivery.HealthWaits, Detail: "held: a required config value is unset"}, true, true},
} {
t.Run(row.name, func(t *testing.T) {
c := New(Env{Node: "n1"})
key, err := delivery.HaveKey("n1", "web")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeHave(row.have)
if err != nil {
t.Fatal(err)
}
c.takeModuleHave(agent.Update{Key: key, Value: value})
if c.inOwn["web"] != row.in {
t.Errorf("web counted inside the node's own composition: %v, want %v", c.inOwn["web"], row.in)
}
s := &Shadow{c: c, ran: map[string]string{}}
s.takeRan(agent.Update{Key: key, Value: value})
if _, got := s.heldToday()["web"]; got != row.heldToday {
t.Errorf("web held today: %v, want %v", got, row.heldToday)
}
})
}
}
// setFor sets a package global (a wait the composer or the shadow reads) for one test, and puts back what it held when
// the test ends, so no later test runs on another test's waits.
func setFor[T any](t *testing.T, p *T, v T) {
t.Helper()
old := *p
*p = v
t.Cleanup(func() { *p = old })
}
+538 -75
View File
@@ -1,9 +1,12 @@
package selfcompose package selfcompose
import ( import (
"bytes"
"context" "context"
"fmt" "fmt"
"reflect"
"sort" "sort"
"strings"
"sync" "sync"
"time" "time"
@@ -13,8 +16,8 @@ import (
"github.com/novox/mesh-host/internal/agent" "github.com/novox/mesh-host/internal/agent"
) )
// Shadow is the node's answer to placement candidates (novox/hq ADR 0559 §4, issue 1126): for every group with a key in // Shadow is the node's answer to placement candidates (novox/hq ADR 0559 §4, issue 1126): for every proposal with a key in
// the indexer's state `placement-candidates`, it composes this node in shadow from the trunk's placement with the group's // the placement seat's state `candidates`, it composes this node in shadow from the trunk's placement with the proposal's
// keys laid over it and its own inputs (the sealed values and boxes it holds among them), compares the result with what // keys laid over it and its own inputs (the sealed values and boxes it holds among them), compares the result with what
// the node runs, and hands the answers to be written into the node's own have key, `<node>.node`, under `candidates`. // the node runs, and hands the answers to be written into the node's own have key, `<node>.node`, under `candidates`.
// It applies nothing and hands no composition: what runs is still composed from `placement` alone, by the Composer. // It applies nothing and hands no composition: what runs is still composed from `placement` alone, by the Composer.
@@ -22,15 +25,72 @@ import (
// **Event-driven, like the Composer.** It watches every candidate key, the trunk's placement, every node's record, the // **Event-driven, like the Composer.** It watches every candidate key, the trunk's placement, every node's record, the
// indexer's modules, this node's config values, and this node's module haves (what it runs moved); a burst of changes // indexer's modules, this node's config values, and this node's module haves (what it runs moved); a burst of changes
// answers once. It runs whether or not the node is placed yet: a node that applies the controller's compositions answers // answers once. It runs whether or not the node is placed yet: a node that applies the controller's compositions answers
// against what it runs from them. An answer is handed only when it changes, and a group's answer is dropped when its // against what it runs from them. An answer is handed only when it changes, and a proposal's answer is dropped when its
// keys go. // keys go.
type Shadow struct { type Shadow struct {
c *Composer c *Composer
answer func(ctx context.Context, answers map[string]delivery.CandidateAnswer) answer func(ctx context.Context, answers map[string]delivery.CandidateAnswer)
mu sync.Mutex mu sync.Mutex
candidates map[string]map[string]candidateKey // by group, by node candidates map[string]map[string]candidateKey // by proposal, by node
answers map[string]delivery.CandidateAnswer answers map[string]delivery.CandidateAnswer
// wants are the node's wants by module (agent.NodeModule for its own): what a proposal deleting its placement returns
// it to, the controller's compositions at them.
wants map[string]delivery.Want
// ran is, by module, the digest the module's have says runs: what runs, where the agent's list holds only the copy
// from the declaration kept before the switch (issue 1126, probe #532 on shanks).
ran map[string]string
// heldNow is, by module, how its have says the node keeps it today as it runs: "held", "failed", or "runs a build of
// the old form" (a `build` and no `fp`, which no registered build of the trunk names); "" when it is composed as any
// other. A candidate that cannot compose it, or holds it only behind one of these, keeps it as today
// (frombus.Scope.HeldToday, issue 1126).
heldNow map[string]string
// answered is, by proposal, the candidate revisions and the input changes (moved) each answer was composed at: a proposal
// is answered again only when either moved (issue 1126, 2026-10-28: shanks answered probe/1126-seed-7 at revision 84
// every 13-17 s, woken by its own modules' haves whose health words changed at every look).
answered map[string]answeredAt
// moved counts, by input (inputWords), the changes of each of the node's own inputs, and retry names the proposals
// answered again after a read the bus did not answer: what woke each answer is said with it (issue 1126,
// 2026-10-28, seed-9's probe: a safety check could not tell a loop from inputs moving).
moved [len(inputWords)]uint64
retry map[string]bool
// owed says an answer met a read the bus did not answer: it is no answer to keep, and its proposal is answered again
// after a bound (laterAgain), whether or not an input moves (issue 1126, 2026-10-28: ace kept "the vault's own: not
// yet ... no responders" in its answer to seed-7).
owed bool
}
// answeredAt is what one proposal's answer was composed at: its keys' revisions, by node, and the changes of each input.
type answeredAt struct {
revs string
moved [len(inputWords)]uint64
}
// inputWords name the shadow's inputs, in Run's order, as an answer says which of them woke it.
var inputWords = [...]string{"the candidate", "placement", "records", "modules", "values", "haves", "wants"}
// wokenBy says what moved between the answer a proposal stands at (had, when done) and the one about to be composed (at):
// its first answer, a retry, its candidate's revisions, or the inputs that changed, by their words.
func wokenBy(had answeredAt, done, retry bool, at answeredAt) string {
switch {
case retry:
return "a retry after a read the bus did not answer"
case !done:
return "its first answer since the shadow began"
}
var words []string
if had.revs != at.revs {
words = append(words, inputWords[0])
}
for i := 1; i < len(inputWords); i++ {
if had.moved[i] != at.moved[i] {
words = append(words, inputWords[i])
}
}
if len(words) == 0 {
return "nothing that moved"
}
return strings.Join(words, ", ")
} }
// candidateKey is one candidate key as last read: its value and revision. // candidateKey is one candidate key as last read: its value and revision.
@@ -42,7 +102,7 @@ type candidateKey struct {
// NewShadow is the shadow composer of one node; answer writes the node's answers into its own have key. // NewShadow is the shadow composer of one node; answer writes the node's answers into its own have key.
func NewShadow(env Env, answer func(ctx context.Context, answers map[string]delivery.CandidateAnswer)) *Shadow { func NewShadow(env Env, answer func(ctx context.Context, answers map[string]delivery.CandidateAnswer)) *Shadow {
return &Shadow{c: New(env), answer: answer, candidates: map[string]map[string]candidateKey{}, return &Shadow{c: New(env), answer: answer, candidates: map[string]map[string]candidateKey{},
answers: map[string]delivery.CandidateAnswer{}} answers: map[string]delivery.CandidateAnswer{}, wants: map[string]delivery.Want{}, ran: map[string]string{}}
} }
// Keep runs the shadow until ctx ends, beginning again after backoff(try) whenever it stops: an input it cannot watch // Keep runs the shadow until ctx ends, beginning again after backoff(try) whenever it stops: an input it cannot watch
@@ -84,10 +144,16 @@ func (s *Shadow) Run(ctx context.Context) error {
defer cancel() defer cancel()
s.mu.Lock() s.mu.Lock()
s.candidates = map[string]map[string]candidateKey{} s.candidates = map[string]map[string]candidateKey{}
s.wants = map[string]delivery.Want{}
s.ran = map[string]string{}
s.answered = map[string]answeredAt{}
s.moved = [len(inputWords)]uint64{}
s.retry = map[string]bool{}
s.c = New(s.c.env) s.c = New(s.c.env)
s.mu.Unlock() s.mu.Unlock()
c := s.c c := s.c
node := c.env.Node node := c.env.Node
// In inputWords' order.
ins := []input{ ins := []input{
{delivery.PlacementCandidatesBucket, ">", nil, nil}, {delivery.PlacementCandidatesBucket, ">", nil, nil},
{delivery.PlacementBucket, ">", (*Composer).takePlacement, nil}, {delivery.PlacementBucket, ">", (*Composer).takePlacement, nil},
@@ -96,6 +162,8 @@ func (s *Shadow) Run(ctx context.Context) error {
{ResolvedBucket, node + ".>", (*Composer).takeValue, nil}, {ResolvedBucket, node + ".>", (*Composer).takeValue, nil},
// What this node runs moved: an apply ends in its module's have. // What this node runs moved: an apply ends in its module's have.
{delivery.HaveBucket, node + ".>", nil, nil}, {delivery.HaveBucket, node + ".>", nil, nil},
// Its wants: a proposal deleting its placement returns it to the controller's compositions at them.
{delivery.AssignerWantsBucket, "want." + node + ".>", nil, nil},
} }
type tagged struct { type tagged struct {
in int in int
@@ -124,26 +192,51 @@ func (s *Shadow) Run(ctx context.Context) error {
} }
caught := make([]bool, len(ins)) caught := make([]bool, len(ins))
ready, dirty := false, false ready, dirty := false, false
// Not ready after a minute is said, naming each input not read yet, in the journal and in an answer to each proposal it
// knows: a shadow that waits is never silent (issue 1126: ace, g14 and shanks answered nothing to seed #531).
waiting := time.NewTimer(notReadyAfter)
defer waiting.Stop()
var again <-chan time.Time
tries := 0
for { for {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return nil return nil
case <-again:
again = nil
dirty = true
case <-waiting.C:
if !ready {
s.notReady(ctx, ins, caught)
}
continue
case t := <-merged: case t := <-merged:
if !t.ok { if !t.ok {
return fmt.Errorf("the watch of %s ended; the node answers no placement candidate until its agent starts again", ins[t.in].bucket) return fmt.Errorf("the watch of %s ended; the node answers no placement candidate until its agent starts again", ins[t.in].bucket)
} }
// Only a change of what an answer is composed from answers again: a key written again with the value it held,
// this node's own key (its answers), or a module's have whose health or words moved but not what it runs, is
// none (issue 1126, 2026-10-28: each look rewrote shanks' unhealthy modules' haves, and each woke a full answer).
changed := false
switch { switch {
case t.u.Caught: case t.u.Caught:
caught[t.in] = true caught[t.in] = true
case t.in == 0: case t.in == 0:
dirty = s.takeCandidate(t.u) || dirty changed = s.takeCandidate(t.u)
case t.in == 2: case t.in == 2:
dirty = s.takeRecord(t.u) || dirty changed = s.takeRecord(t.u)
case t.in == 5: case t.in == 5:
c.takeModuleHave(t.u) folded := c.takeModuleHave(t.u)
dirty = !isNodeKey(t.u.Key) || dirty changed = s.takeRan(t.u) || folded
case t.in == 6:
changed = s.takeWant(t.u)
default: default:
ins[t.in].take(c, t.u) changed = s.takeChanged(ins[t.in], t.u)
}
if changed {
if t.in != 0 {
s.moved[t.in]++ // the candidates' own changes are their revisions
}
dirty = true dirty = true
} }
if len(merged) > 0 { if len(merged) > 0 {
@@ -159,46 +252,110 @@ func (s *Shadow) Run(ctx context.Context) error {
if dirty { if dirty {
dirty = false dirty = false
s.answerAll(ctx) s.answerAll(ctx)
if s.owed {
wait := LaterWait(tries)
tries++
c.env.Say(fmt.Sprintf("this node answers again in %s the placement candidates whose answer met a bus that did not answer", wait))
again = time.After(wait)
} else {
tries, again = 0, nil
}
} }
} }
} }
func isNodeKey(key string) bool { // takeChanged takes u through in.take and says whether the entry it names changed: a placement, a module or a config
_, _, isNode, err := delivery.ParseHaveKey(key) // value written again with the value it held changes nothing composed.
return err == nil && isNode func (s *Shadow) takeChanged(in input, u agent.Update) bool {
before, had, known := s.entry(in.bucket, u.Key)
in.take(s.c, u)
if !known {
return true
}
after, has, _ := s.entry(in.bucket, u.Key)
return had != has || !reflect.DeepEqual(before, after)
}
// entry is what the composer holds of one key of bucket, whether it holds it, and whether the bucket is one it knows.
func (s *Shadow) entry(bucket, key string) (v any, has, known bool) {
c := s.c
c.mu.Lock()
defer c.mu.Unlock()
switch bucket {
case delivery.PlacementBucket:
node, err := delivery.ParsePlacementKey(key)
if err != nil {
return nil, false, true
}
p, ok := c.placements[node]
return p, ok, true
case delivery.ModulesBucket:
name, err := delivery.ParseModulesKey(key)
if err != nil {
return nil, false, true
}
m, ok := c.modules[name]
return m, ok, true
case ResolvedBucket:
rest, ok := strings.CutPrefix(key, c.env.Node+".")
if !ok {
return nil, false, true
}
module, k, ok := strings.Cut(rest, ".")
if !ok {
return nil, false, true
}
val, ok := c.values[module][k]
return val, ok, true
}
return nil, false, false
}
// revsOf is a proposal's keys' revisions, by node, as one word.
func revsOf(keys map[string]candidateKey) string {
nodes := make([]string, 0, len(keys))
for n := range keys {
nodes = append(nodes, n)
}
sort.Strings(nodes)
var b strings.Builder
for _, n := range nodes {
fmt.Fprintf(&b, "%s=%d;", n, keys[n].rev)
}
return b.String()
} }
// takeCandidate takes one candidate key, and says whether it changed anything. // takeCandidate takes one candidate key, and says whether it changed anything.
func (s *Shadow) takeCandidate(u agent.Update) bool { func (s *Shadow) takeCandidate(u agent.Update) bool {
group, node, err := delivery.ParsePlacementCandidateKey(u.Key) proposal, node, err := delivery.ParsePlacementCandidateKey(u.Key)
if err != nil { if err != nil {
return false return false
} }
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
if u.Deleted { if u.Deleted {
if _, ok := s.candidates[group][node]; !ok { if _, ok := s.candidates[proposal][node]; !ok {
return false return false
} }
delete(s.candidates[group], node) delete(s.candidates[proposal], node)
if len(s.candidates[group]) == 0 { if len(s.candidates[proposal]) == 0 {
delete(s.candidates, group) delete(s.candidates, proposal)
} }
return true return true
} }
c, err := delivery.DecodePlacementCandidate(u.Value) c, err := delivery.DecodePlacementCandidate(u.Value)
if err != nil || c.Group != group || (c.Placement != nil && c.Placement.Node != node) { if err != nil || c.Proposal != proposal || (c.Placement != nil && c.Placement.Node != node) {
// Not one: read as gone. // Not one: read as gone.
if _, ok := s.candidates[group][node]; ok { if _, ok := s.candidates[proposal][node]; ok {
delete(s.candidates[group], node) delete(s.candidates[proposal], node)
return true return true
} }
return false return false
} }
if s.candidates[group] == nil { if s.candidates[proposal] == nil {
s.candidates[group] = map[string]candidateKey{} s.candidates[proposal] = map[string]candidateKey{}
} }
s.candidates[group][node] = candidateKey{c: c, rev: u.Revision} s.candidates[proposal][node] = candidateKey{c: c, rev: u.Revision}
return true return true
} }
@@ -220,70 +377,199 @@ func (s *Shadow) takeRecord(u agent.Update) bool {
return had != has || (has && !delivery.SameNodeRecord(before, after)) return had != has || (has && !delivery.SameNodeRecord(before, after))
} }
// answerAll answers every group and hands the answers when any changed. // How long the shadow waits before it says it is not ready, and how long a proposal's answer runs before it is said slow.
var notReadyAfter, slowAfter = time.Minute, time.Minute
// notReady says which inputs the shadow has not read yet, and answers each proposal it knows refused for it.
func (s *Shadow) notReady(ctx context.Context, ins []input, caught []bool) {
var waits []string
for i, in := range ins {
if !caught[i] {
waits = append(waits, in.bucket+" ("+in.pattern+")")
}
}
why := fmt.Sprintf("the node has not read %s after %s; it answers once it has", strings.Join(waits, ", "), notReadyAfter)
s.c.env.Say("this node answers no placement candidate yet: " + why)
s.mu.Lock()
next := map[string]delivery.CandidateAnswer{}
for proposal, keys := range s.candidates {
own, ok := keys[s.c.env.Node]
if !ok {
continue // no key of this node's: nothing asks it to answer
}
next[proposal] = delivery.CandidateAnswer{Proposal: proposal, Revision: own.rev, Refused: why, Verdict: delivery.CandidateRefused}
}
s.answers = next
s.mu.Unlock()
if len(next) > 0 && s.answer != nil {
s.answer(ctx, next)
}
}
// answerAll answers every proposal, the newest candidate first, and hands the answers each time one changes (issue 1126,
// T3: ace answered a closed pull request's group for 22 minutes before the newer one, all handed at the end). Newest
// is the highest revision of the proposal's keys: a withdrawn proposal's candidate is the older and waits; the seat
// drops a closed one's keys, which ends its answer here (a node does not read the forge). Each answer says when it
// starts, every progressEvery while it runs what it waits on, and when it ends (T2: the node said nothing for 22
// minutes).
func (s *Shadow) answerAll(ctx context.Context) { func (s *Shadow) answerAll(ctx context.Context) {
c := s.c c := s.c
node := c.env.Node node := c.env.Node
s.mu.Lock() s.mu.Lock()
groups := make([]string, 0, len(s.candidates)) type listed struct {
for g := range s.candidates { proposal string
groups = append(groups, g) rev uint64
at answeredAt
} }
s.mu.Unlock() var proposals []listed
sort.Strings(groups) for g, keys := range s.candidates {
l := listed{proposal: g, at: answeredAt{revs: revsOf(keys), moved: s.moved}}
for _, k := range keys {
l.rev = max(l.rev, k.rev)
}
proposals = append(proposals, l)
}
// A proposal whose keys' revisions and the node's inputs are those its answer stands at is not answered again.
for g := range s.answered {
if _, ok := s.candidates[g]; !ok {
delete(s.answered, g)
}
}
// Each proposal's last answer stands until it is answered again; a proposal whose keys went is dropped.
next := map[string]delivery.CandidateAnswer{} next := map[string]delivery.CandidateAnswer{}
for _, group := range groups { for _, l := range proposals {
if a, ok := s.answerGroup(ctx, group); ok { if a, ok := s.answers[l.proposal]; ok {
next[group] = a next[l.proposal] = a
} }
} }
s.mu.Lock() s.owed = false
same := len(next) == len(s.answers)
for g, a := range next {
if old, ok := s.answers[g]; !ok || !sameAnswer(old, a) {
same = false
}
}
if !same {
s.answers = next
}
s.mu.Unlock() s.mu.Unlock()
if !same && s.answer != nil { sort.Slice(proposals, func(i, j int) bool {
if proposals[i].rev != proposals[j].rev {
return proposals[i].rev > proposals[j].rev
}
return proposals[i].proposal < proposals[j].proposal
})
hand := func() {
s.mu.Lock()
same := len(next) == len(s.answers)
for g, a := range next {
if old, ok := s.answers[g]; !ok || !sameAnswer(old, a) {
same = false
}
}
out := make(map[string]delivery.CandidateAnswer, len(next)) out := make(map[string]delivery.CandidateAnswer, len(next))
for g, a := range next { for g, a := range next {
out[g] = a out[g] = a
} }
s.answer(ctx, out) if !same {
c.env.Say(fmt.Sprintf("%s answered %d placement candidate group(s)", node, len(out))) s.answers = out
}
s.mu.Unlock()
if !same && s.answer != nil {
given := make(map[string]delivery.CandidateAnswer, len(out))
for g, a := range out {
given[g] = a
}
s.answer(ctx, given)
c.env.Say(fmt.Sprintf("%s answered %d placement candidate proposal(s)", node, len(given)))
}
} }
for _, l := range proposals {
if ctx.Err() != nil {
return
}
proposal := l.proposal
s.mu.Lock()
at, done := s.answered[proposal]
retry := s.retry[proposal]
s.mu.Unlock()
if done && at == l.at {
continue
}
began := time.Now()
c.env.Say(fmt.Sprintf("answering placement candidate %s (revision %d; woken by %s)", proposal, l.rev, wokenBy(at, done, retry, l.at)))
stop := make(chan struct{})
go func() {
tick := time.NewTicker(progressEvery)
defer tick.Stop()
for {
select {
case <-stop:
return
case <-tick.C:
c.env.Say(fmt.Sprintf("the answer to %s has run %s and waits on %s", proposal,
time.Since(began).Round(time.Second), c.waitingOn()))
}
}
}()
missed := c.missed.Load()
a, ok := s.answerProposal(ctx, proposal)
close(stop)
provisional := c.missed.Load() > missed && ctx.Err() == nil
took := time.Since(began)
if took > slowAfter {
c.env.Say(fmt.Sprintf("the answer to %s took %s", proposal, took.Round(time.Second)))
if ok {
// Words that do not move with the time it took, so an answer the same as before is not written again.
a.Note = strings.TrimPrefix(a.Note+"; ", "; ") + "this answer took over " + slowAfter.String() + " to compose"
}
}
if ok {
a = delivery.FitAnswer(a, delivery.MaxAnswerBytes)
next[proposal] = a
c.env.Say(fmt.Sprintf("answered placement candidate %s in %s: %s", proposal, took.Round(time.Millisecond), a.Verdict))
} else {
delete(next, proposal)
c.env.Say(fmt.Sprintf("placement candidate %s does not place this node: nothing to answer", proposal))
}
s.mu.Lock()
if provisional {
// What the bus did not answer is no answer: handed as it stands, and composed again after a bound.
s.owed = true
s.retry[proposal] = true
delete(s.answered, proposal)
c.env.Say(fmt.Sprintf("the answer to %s met a read on the bus that did not answer: it is answered again", proposal))
} else {
s.answered[proposal] = l.at
delete(s.retry, proposal)
}
s.mu.Unlock()
hand()
}
hand() // a proposal whose keys went, when no proposal is left to answer
} }
// answerGroup is the node's answer to one group, and false when the node has nothing to answer: neither the trunk nor // How often a running answer says what it waits on.
// the group places it. var progressEvery = time.Minute
func (s *Shadow) answerGroup(ctx context.Context, group string) (delivery.CandidateAnswer, bool) {
// answerProposal is the node's answer to one proposal, and false when the node has nothing to answer: neither the trunk nor
// the proposal places it.
func (s *Shadow) answerProposal(ctx context.Context, proposal string) (delivery.CandidateAnswer, bool) {
c := s.c c := s.c
node := c.env.Node node := c.env.Node
defer c.waiting("composing")()
s.mu.Lock() s.mu.Lock()
keys := s.candidates[group] keys := s.candidates[proposal]
given := make(map[string]*delivery.Placement, len(keys)) given := make(map[string]*delivery.Placement, len(keys))
var head string
for n, k := range keys { for n, k := range keys {
if k.c.Reached && k.c.Placement == nil {
continue // reached, with no key of its own: nothing laid over it
}
given[n] = k.c.Placement given[n] = k.c.Placement
head = k.c.Pull.Head
} }
// The revision answered is the node's own key's, 0 when the group has none for it (judged by the head alone). // The revision answered is the node's own candidate key's; a node with none is not asked.
own, mine := keys[node] own, mine := keys[node]
rev := own.rev rev := own.rev
s.mu.Unlock() s.mu.Unlock()
c.mu.Lock() c.mu.Lock()
_, placed := c.placements[node]
modules := make(map[string]delivery.Module, len(c.modules)) modules := make(map[string]delivery.Module, len(c.modules))
for k, v := range c.modules { for k, v := range c.modules {
modules[k] = v modules[k] = v
} }
c.mu.Unlock() c.mu.Unlock()
if !placed && !mine { if !mine {
return delivery.CandidateAnswer{}, false return delivery.CandidateAnswer{}, false
} }
@@ -291,19 +577,17 @@ func (s *Shadow) answerGroup(ctx context.Context, group string) (delivery.Candid
if c.env.Running != nil { if c.env.Running != nil {
running = c.env.Running() running = c.env.Running()
} }
if p, ok := given[node]; ok && p == nil { if p, ok := given[node]; ok && p == nil && !own.c.Reached {
// The head removes this node's placement file: everything it runs is removed. // The proposal deletes this node's placement key: the node then returns to the controller's compositions, as one the
a := delivery.CandidateAnswer{Head: head, Revision: rev} // catalogue does not place (endWhenUnplaced). Those at its wants are compared with what runs (issue 1126, the
for _, r := range running { // seed's revert, whose check read "would stop running node").
if r.Module == "" { s.mu.Lock()
a.Removed = append(a.Removed, delivery.HaveNodeWord) wants := make(map[string]delivery.Want, len(s.wants))
} else { for m, w := range s.wants {
a.Removed = append(a.Removed, r.Module) wants[m] = w
}
} }
sort.Strings(a.Removed) s.mu.Unlock()
a.Verdict = delivery.VerdictOf(a) return returning(ctx, c.get, node, wants, running, proposal, rev), true
return a, true
} }
placement := given[node] placement := given[node]
@@ -323,12 +607,7 @@ func (s *Shadow) answerGroup(ctx context.Context, group string) (delivery.Candid
if mod, ok := modules[module]; ok && mod.FP2 == fp { if mod, ok := modules[module]; ok && mod.FP2 == fp {
continue continue
} }
digest, err := c.builtManifest(ctx, module, fp) kept[module] = fp // composed from that build (frombus.Scope.KeptAt, issue 1126)
if err != nil {
return delivery.CandidateAnswer{Head: head, Revision: rev, Verdict: delivery.CandidateRefused,
Refused: fmt.Sprintf("%s runs at %s, whose manifest cannot be read: %v", module, fp, err)}, true
}
kept[module] = digest
} }
// The boxes it runs, by what each module's have and its own key say after a restart (Held), are reused for the // The boxes it runs, by what each module's have and its own key say after a restart (Held), are reused for the
// modules it runs: a value sealed again is no difference. // modules it runs: a value sealed again is no difference.
@@ -339,8 +618,93 @@ func (s *Shadow) answerGroup(ctx context.Context, group string) (delivery.Candid
c.mu.Lock() c.mu.Lock()
c.wants = held c.wants = held
c.mu.Unlock() c.mu.Unlock()
a := frombus.Answer(ctx, reader{c: c}, c.env.Mesh, frombus.Scope{Node: node, Kept: kept, Placements: given, InOwn: inOwn}, fps, running, head, rev) a := frombus.Answer(ctx, reader{c: c}, c.env.Mesh, frombus.Scope{Node: node, KeptAt: kept, Placements: given, InOwn: inOwn,
return a, true SameValue: c.sameValue, HeldToday: s.heldToday()}, fps, running, proposal, rev)
s.mu.Lock()
ran := make(map[string]string, len(s.ran))
for m, d := range s.ran {
ran[m] = d
}
s.mu.Unlock()
return asHavesSay(a, ran), true
}
// heldToday is what the node holds today: each module whose have's phase is held, and what its own composer holds, the
// node's own composition ("node") among them when it composes none of its own.
func (s *Shadow) heldToday() map[string]string {
out := map[string]string{}
s.mu.Lock()
for m, how := range s.heldNow {
if how != "" {
out[m] = how
}
}
s.mu.Unlock()
for _, m := range s.c.heldNow() {
if _, said := out[m]; !said {
out[m] = "held"
}
}
return out
}
// takeRan keeps the digest one of this node's module haves says runs, and says whether it changed: a have whose health
// or words alone moved runs the same.
func (s *Shadow) takeRan(u agent.Update) bool {
node, module, isNode, err := delivery.ParseHaveKey(u.Key)
if err != nil || isNode || node != s.c.env.Node {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
before, had := s.ran[module]
h, err := delivery.DecodeHave(u.Value)
if s.heldNow == nil {
s.heldNow = map[string]string{}
}
wasHeld := s.heldNow[module]
s.heldNow[module] = ""
if !u.Deleted && err == nil {
switch {
case h.Phase == delivery.PhaseHeld:
s.heldNow[module] = "held"
case h.Phase == delivery.PhaseFailed:
s.heldNow[module] = "failed"
case h.Phase != delivery.PhaseRemoved && h.FP == "" && h.Build != "":
// A build of the old form: its have names a `build` and no `fp`, which no registered build of the trunk
// names, so the controller keeps it as it runs (issue 1126: novox's de-spiegel at build-…); a seed keeps it so.
s.heldNow[module] = "runs a build of the old form"
}
}
holdMoved := wasHeld != s.heldNow[module] // a hold's start or end changes the answer too
if u.Deleted || err != nil || h.Composed == "" || h.Phase == delivery.PhaseRemoved {
delete(s.ran, module)
return had || holdMoved
}
s.ran[module] = h.Composed
return !had || before != h.Composed || holdMoved
}
// asHavesSay is a with each module that differs only from the copy the agent keeps of the declaration before the switch
// counted the same where the composed digest is the one its have says runs: what runs is the composition at that
// digest, never the kept copy (issue 1126, probe #532: shanks' xterm, zsh, zsh-autosuggestions and
// zsh-syntax-highlighting).
func asHavesSay(a delivery.CandidateAnswer, ran map[string]string) delivery.CandidateAnswer {
if a.Refused != "" || len(a.Differs) == 0 {
return a
}
kept := a.Differs[:0:0]
for _, d := range a.Differs {
if d.Composed != "" && ran[d.Module] == d.Composed {
a.Same = append(a.Same, d.Module)
continue
}
kept = append(kept, d)
}
a.Differs = kept
sort.Strings(a.Same)
a.Verdict = delivery.VerdictOf(a)
return a
} }
// shadowBuilds is the build each module placed is composed at in shadow: the one it runs, and for one the candidate // shadowBuilds is the build each module placed is composed at in shadow: the one it runs, and for one the candidate
@@ -374,3 +738,102 @@ func sameAnswer(a, b delivery.CandidateAnswer) bool {
y, errB := delivery.EncodeHaveNode(delivery.HaveNode{Candidates: map[string]delivery.CandidateAnswer{"g": b}}) y, errB := delivery.EncodeHaveNode(delivery.HaveNode{Candidates: map[string]delivery.CandidateAnswer{"g": b}})
return errA == nil && errB == nil && string(x) == string(y) return errA == nil && errB == nil && string(x) == string(y)
} }
// takeWant takes one of the node's wants, and says whether it changed.
func (s *Shadow) takeWant(u agent.Update) bool {
module := ""
var w delivery.Want
if _, err := delivery.ParseWantNodeKey(u.Key); err == nil {
module = agent.NodeModule
} else if node, m, err := delivery.ParseWantKey(u.Key); err == nil && node == s.c.env.Node {
module = m
if !u.Deleted {
if w, err = delivery.DecodeWant(u.Value); err != nil || w.Removed || w.FP == "" {
u.Deleted = true
}
}
} else {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
old, had := s.wants[module]
if u.Deleted {
delete(s.wants, module)
return had
}
s.wants[module] = w
return !had || old.FP != w.FP
}
// returning is the node's answer to a proposal deleting its placement: it returns to the controller's compositions, each
// read at the fingerprint its want names in the node's composition bucket and compared with what runs
// (delivery.CompareComposed). A want the controller holds no composition for, or one that cannot be read, refuses the
// answer, naming each: returning then would leave that module with nothing to apply.
func returning(ctx context.Context, get func(ctx context.Context, bucket, key string) ([]byte, error), node string,
wants map[string]delivery.Want, running []delivery.Composed, proposal string, rev uint64) delivery.CandidateAnswer {
a := delivery.CandidateAnswer{Proposal: proposal, Revision: rev}
bucket, err := delivery.ComposedBucketFor(node)
if err != nil {
a.Refused, a.Verdict = err.Error(), delivery.CandidateRefused
return a
}
modules := make([]string, 0, len(wants))
for m := range wants {
modules = append(modules, m)
}
sort.Strings(modules)
comps := map[string]delivery.Composed{}
var missing []string
for _, m := range modules {
var key string
if m == agent.NodeModule {
key, err = delivery.ComposedNodeKey(node)
} else {
key, err = delivery.ComposedKey(node, m, wants[m].FP)
}
if err != nil {
missing = append(missing, m+": "+err.Error())
continue
}
raw, err := get(ctx, bucket, key)
switch {
case err != nil:
missing = append(missing, fmt.Sprintf("%s: the controller's composition %s cannot be read: %v", m, key, err))
continue
case raw == nil:
missing = append(missing, fmt.Sprintf("%s: the controller holds no composition %s, which its want names", m, key))
continue
}
comp, err := delivery.DecodeComposed(raw)
if err != nil {
missing = append(missing, fmt.Sprintf("%s: the controller's composition %s: %v", m, key, err))
continue
}
comps[key] = comp
}
if len(missing) > 0 {
a.Refused = "returning to the controller's compositions, " + strings.Join(missing, "; ")
a.Verdict = delivery.CandidateRefused
return a
}
cmp := delivery.CompareComposed(comps, running)
a.Returns = delivery.ReturnsToController
a.Same, a.Differs, a.Added, a.Removed = cmp.Same, cmp.Differs, cmp.Added, cmp.Removed
a.Verdict = delivery.VerdictOf(a)
return a
}
// sameValue says whether two sealed values open, with this node's key, to the same value; unknown when either does not
// open (or the node holds no opener). Only the comparison leaves: never a value (issue 1126, the shadow shows its work).
func (c *Composer) sameValue(a, b string) (same, known bool) {
if c.env.Unseal == nil {
return false, false
}
x, errA := c.env.Unseal(a)
y, errB := c.env.Unseal(b)
if errA != nil || errB != nil {
return false, false
}
return bytes.Equal(x, y), true
}
@@ -0,0 +1,129 @@
{
"module": "de-spiegel",
"version": "1",
"slug": "spiegel",
"capabilities": [
"container-runtime"
],
"requires": [
"route",
"smtp"
],
"contributes": {
"route": {
"site": {
"label": "de-spiegel",
"port": 35621
},
"praktijk": {
"name": "${config:praktijk-name}",
"port": 35621
},
"praktijk-www": {
"name": "${config:praktijk-www-name}",
"redirect": "https://${config:praktijk-name}"
},
"psychologie": {
"name": "${config:psychologie-name}",
"port": 35621
},
"psychologie-www": {
"name": "${config:psychologie-www-name}",
"redirect": "https://${config:psychologie-name}"
}
},
"smtp": {
"account": "de-spiegel"
}
},
"binds": {
"route": "/var/lib/de-spiegel/route.json",
"smtp": "/var/lib/de-spiegel/smtp.json"
},
"config": {
"praktijk-name": {
"type": "string",
"description": "The public name the practice's site is served on; the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"praktijk-www-name": {
"type": "string",
"description": "The public name that redirects to the practice's site (its www form); the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"psychologie-name": {
"type": "string",
"description": "The public name the psychology practice's site is served on; the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"psychologie-www-name": {
"type": "string",
"description": "The public name that redirects to the psychology practice's site (its www form); the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
}
},
"listens": [
{
"port": 35621,
"protocol": "tcp",
"from": "mesh",
"why": "the de-spiegel site and its /contact endpoint over http; the practice's three public names are route grants, and the proxy reaches every one of them here"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"path": "/var/lib/de-spiegel",
"mode": "0700"
},
{
"id": "server-env",
"type": "file",
"path": "/var/lib/de-spiegel/server.env",
"mode": "0600",
"content": "SMTP_HOST=${bound:smtp:at}\nSMTP_PORT=${bound:smtp:port}\nSMTP_AUTH_USER=de-spiegel@${bound:smtp:domain}\nSMTP_AUTH_PASS=${secret:smtp}\n"
},
{
"id": "net",
"type": "network",
"name": "de-spiegel"
},
{
"id": "server",
"type": "container",
"name": "de-spiegel",
"artifact": "site",
"network": "de-spiegel",
"env-file": [
"/var/lib/de-spiegel/server.env"
],
"ports": [
"35621"
],
"secrets-in-environment": "the application's own code reads SMTP_AUTH_USER/PASS from the environment (server/index.js); converting is this repository's change"
}
],
"build": {
"on": [
{
"arg": "NODE_BASE",
"image": "node@sha256:48e4b67d85f87bd551df43704e24d252f56cc5f8e9718841aace50f19948f0f9"
}
],
"artifacts": [
{
"name": "site",
"kind": "image",
"from": "Dockerfile"
}
]
},
"secrets": {
"smtp": "/var/lib/de-spiegel/smtp.secret"
}
}
+73 -36
View File
@@ -187,38 +187,11 @@ func describeAge(when, now time.Time) string {
// so LoadState is what is read — and it is the thing an interface spanning systemd and OpenRC // so LoadState is what is read — and it is the thing an interface spanning systemd and OpenRC
// would have had to drop. // would have had to drop.
func (arch) ServiceState(ctx context.Context, run Runner, unit string) (string, error) { func (arch) ServiceState(ctx context.Context, run Runner, unit string) (string, error) {
out, asked := run(ctx, "systemctl", "show", u, err := showUnit(ctx, run, unit)
"--property=LoadState", "--property=ActiveState", "--property=Type", if err != nil {
"--property=RemainAfterExit", "--property=ExecMainStatus", "--", unit) return "", err
var load, active, kind, remains, exited string
for _, line := range strings.Split(out, "\n") {
key, value, found := strings.Cut(strings.TrimSpace(line), "=")
if !found {
continue
}
switch key {
case "LoadState":
load = value
case "ActiveState":
active = value
case "Type":
kind = value
case "RemainAfterExit":
remains = value
case "ExecMainStatus":
exited = value
}
} }
switch u.load {
switch load {
case "":
// With why, where it said why: an account's manager that could not be reached says so
// here, and nowhere else (novox/hq ADR 0177).
if asked != nil {
return "", fmt.Errorf("the service manager said nothing about %s: %w", unit, asked)
}
return "", fmt.Errorf("the service manager said nothing about %s", unit)
case "not-found": case "not-found":
return "", fmt.Errorf( return "", fmt.Errorf(
"%s does not exist on this machine. A declaration naming a unit that is not "+ "%s does not exist on this machine. A declaration naming a unit that is not "+
@@ -227,7 +200,7 @@ func (arch) ServiceState(ctx context.Context, run Runner, unit string) (string,
case "masked": case "masked":
return "", fmt.Errorf("%s is masked, so its state cannot be declared", unit) return "", fmt.Errorf("%s is masked, so its state cannot be declared", unit)
case "error", "bad-setting": case "error", "bad-setting":
return "", fmt.Errorf("%s is installed but its unit file cannot be loaded (%s)", unit, load) return "", fmt.Errorf("%s is installed but its unit file cannot be loaded (%s)", unit, u.load)
} }
// **A one-shot that finished is not stopped.** A unit whose whole job is to apply something // **A one-shot that finished is not stopped.** A unit whose whole job is to apply something
@@ -243,24 +216,88 @@ func (arch) ServiceState(ctx context.Context, run Runner, unit string) (string,
// wrong with. // wrong with.
// //
// So for that shape, what "running" means is "it ran, and it worked". // So for that shape, what "running" means is "it ran, and it worked".
if kind == "oneshot" && remains != "yes" && active == "inactive" { //
if exited == "0" || exited == "" { // It cannot be told from one that never ran: systemd says ExecMainStatus=0 and a start timestamp
// of 0 for a one-shot that never ran, and also for one that ran and was unloaded because nothing
// referenced it, then loaded again. So a one-shot declared running that must be seen to have run
// lingers (RemainAfterExit) or is enabled at boot; one declared stopped is judged at rest by the
// apply (ServiceAtRest, novox/hq issue 1126).
if u.oneShotAtRest() {
if u.exited == "0" || u.exited == "" {
return "running", nil return "running", nil
} }
return "stopped", nil return "stopped", nil
} }
switch active { switch u.active {
case "active", "activating", "reloading": case "active", "activating", "reloading":
return "running", nil return "running", nil
case "inactive", "failed", "deactivating": case "inactive", "failed", "deactivating":
return "stopped", nil return "stopped", nil
default: default:
return "", fmt.Errorf( return "", fmt.Errorf(
"the service manager reports %s as %q, which is neither running nor stopped", unit, active) "the service manager reports %s as %q, which is neither running nor stopped", unit, u.active)
} }
} }
// ServiceAtRest is whether unit is a one-shot that does not linger and has nothing running now
// (novox/hq issue 1126). Such a unit has no state in which a process is up between its runs, so a
// declaration that it be stopped is satisfied whatever its last run did: there is nothing to stop,
// and asking the manager to stop it changes nothing. A boot-time cleanup a package ships is declared
// this way — enabled at boot, never started by an apply.
func (arch) ServiceAtRest(ctx context.Context, run Runner, unit string) (bool, error) {
u, err := showUnit(ctx, run, unit)
if err != nil {
return false, err
}
return u.oneShotAtRest(), nil
}
// shownUnit is what `systemctl show` says of a unit, as far as its state is read.
type shownUnit struct {
load, active, kind, remains, exited string
}
// oneShotAtRest is a one-shot that does not linger and is not running now.
func (u shownUnit) oneShotAtRest() bool {
return u.kind == "oneshot" && u.remains != "yes" && u.active == "inactive"
}
// showUnit reads a unit's state properties. A manager that said nothing is an error, with why
// where it said why: an account's manager that could not be reached says so here, and nowhere
// else (novox/hq ADR 0177).
func showUnit(ctx context.Context, run Runner, unit string) (shownUnit, error) {
out, asked := run(ctx, "systemctl", "show",
"--property=LoadState", "--property=ActiveState", "--property=Type",
"--property=RemainAfterExit", "--property=ExecMainStatus", "--", unit)
var u shownUnit
for _, line := range strings.Split(out, "\n") {
key, value, found := strings.Cut(strings.TrimSpace(line), "=")
if !found {
continue
}
switch key {
case "LoadState":
u.load = value
case "ActiveState":
u.active = value
case "Type":
u.kind = value
case "RemainAfterExit":
u.remains = value
case "ExecMainStatus":
u.exited = value
}
}
if u.load == "" {
if asked != nil {
return u, fmt.Errorf("the service manager said nothing about %s: %w", unit, asked)
}
return u, fmt.Errorf("the service manager said nothing about %s", unit)
}
return u, nil
}
func (arch) SetServiceState(ctx context.Context, run Runner, unit, state string) error { func (arch) SetServiceState(ctx context.Context, run Runner, unit, state string) error {
verb := "start" verb := "start"
if state == "stopped" { if state == "stopped" {
+45 -41
View File
@@ -5,47 +5,51 @@ import (
"testing" "testing"
) )
// A one-shot unit that did its work and exited is satisfied, not stopped. // How a one-shot unit's state is read: what it did, not whether a process is up.
// //
// **This made the firewall permanently unsatisfiable.** Its unit loads a rule set and exits, so it // **A one-shot that did its work and exited is satisfied, not stopped.** Its unit loads a rule set
// is inactive the instant it succeeds — the host started it, it worked, the host read back // and exits, so it is inactive the instant it succeeds; reading that as stopped made the firewall
// "stopped" and reported the machine as not doing what it was told. On every apply, for ever, with // permanently unsatisfiable, on every apply, with the rules correctly in place the whole time.
// the rules correctly in place the whole time. //
func TestAOneShotThatFinishedIsRunning(t *testing.T) { // **One that failed is still stopped**, or the host would report success for work that did not
run := func(_ context.Context, _ string, _ ...string) (string, error) { // happen — the opposite mistake and the worse one.
return "LoadState=loaded\nActiveState=inactive\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=0\n", nil //
} // **At rest** (novox/hq issue 1126, the kernel's boot-time cleanup on ace) is whether a one-shot that does not linger has nothing running now: a declaration
state, err := arch{}.ServiceState(context.Background(), run, "nftables.service") // that it be stopped is then satisfied, whatever it last did, because there is nothing to stop.
if err != nil { func TestAOneShotIsReadByWhatItDid(t *testing.T) {
t.Fatalf("a one-shot that succeeded was an error: %v", err) for _, c := range []struct {
} name, show, want string
if state != "running" { atRest bool
t.Fatalf("a one-shot that did its work reads as %q, so it can never be satisfied", state) }{
} {"finished", "ActiveState=inactive\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=0\n",
} "running", true},
{"failed", "ActiveState=inactive\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=1\n",
// And one that failed is still stopped, or the host would report success for work that did not "stopped", true},
// happen — which is the opposite mistake and the worse one. {"running now", "ActiveState=activating\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=0\n",
func TestAOneShotThatFailedIsStopped(t *testing.T) { "running", false},
run := func(_ context.Context, _ string, _ ...string) (string, error) { {"lingers", "ActiveState=active\nType=oneshot\nRemainAfterExit=yes\nExecMainStatus=0\n",
return "LoadState=loaded\nActiveState=inactive\nType=oneshot\nRemainAfterExit=no\nExecMainStatus=1\n", nil "running", false},
} {"a daemon, stopped", "ActiveState=inactive\nType=simple\nRemainAfterExit=no\nExecMainStatus=0\n",
state, err := arch{}.ServiceState(context.Background(), run, "nftables.service") "stopped", false},
if err != nil { } {
t.Fatalf("unexpected error: %v", err) t.Run(c.name, func(t *testing.T) {
} run := func(_ context.Context, _ string, _ ...string) (string, error) {
if state != "stopped" { return "LoadState=loaded\n" + c.show, nil
t.Fatalf("a one-shot that failed reads as %q, so a broken firewall reports success", state) }
} state, err := arch{}.ServiceState(context.Background(), run, "thing.service")
} if err != nil {
t.Fatalf("read as an error: %v", err)
// A unit that lingers deliberately is unaffected: it says so, and its active state is the answer. }
func TestAOneShotThatRemainsIsJudgedByItsActiveState(t *testing.T) { if state != c.want {
run := func(_ context.Context, _ string, _ ...string) (string, error) { t.Errorf("reads as %q, want %q", state, c.want)
return "LoadState=loaded\nActiveState=active\nType=oneshot\nRemainAfterExit=yes\nExecMainStatus=0\n", nil }
} rest, err := arch{}.ServiceAtRest(context.Background(), run, "thing.service")
state, _ := arch{}.ServiceState(context.Background(), run, "thing.service") if err != nil {
if state != "running" { t.Fatalf("at rest read as an error: %v", err)
t.Fatalf("a lingering one-shot reads as %q", state) }
if rest != c.atRest {
t.Errorf("at rest is %v, want %v", rest, c.atRest)
}
})
} }
} }