Compare commits

...
Author SHA1 Message Date
jschoubben 575cb82fba A runtime that already dropped a carried module's tools serves them again, read from what runs and its build (issue 1126, review)
mesh/delivery-group group fix/1126-924-held-and-judged rejected: composed together: the checker's verdict at the members' heads (7e4e345e9969): mesh-build-go
mesh/delivery closed: the pull request closed unmerged
mesh/check checking 575cb82fba69 on 394fec8a63c1 | novox/mesh-sdk#130 in group fix/1126-924-held-and-judged
rev-held2: ToolsRunning took a carried module's bundles from the runtime's MESH_TOOL_MODULES alone, and shanks had
already applied the drop of systemd-resolved, so its runs stayed empty and the drop permanent. Now every bundle
archive that runs at its module's BundlePath is a running bundle; one the runtime loads nothing of takes its loads and
words from the module's build (its built bundles, else its build's artifacts, by catalogue.BundleLoads, the rule a
build's resolution keeps, now one function). The runtime process and each archive are matched by their owning module
too, not only by id and path. A row added: the runtime already dropped it.
2026-10-29 01:37:47 +01:00
jschoubben b6768dad25 A data flag that differs alone moves no data: the judgement reads the data items first, and says the flag loudly (issue 1126)
novox's mongodb and mssql asked a yes for moving their data on a toolchain move: the running side, read back from the
kept declaration, carries no data flag, and judge.go returned moves-data on the flag before it compared the data
directories and files, so a flag mismatch both raised a false ask and hid a real move. Now moves-data only when a
data item (a directory or file within the data places, or a process of a module with data) changes; a flag-only
difference is neither, with a WARNING in its words. One row changed and one added in TestJudgeMove.
2026-10-29 01:36:17 +01:00
jschoubben f50e004e8d A module composed as what runs of it serves its tools as they run, so the runtime no longer drops it (issue 1126)
After the take-over of placement on 2026-10-29, shanks's runtime stopped serving the folded systemd-resolved (the
node-resolver seat's verbs answer "nothing serves") and novox's the folded nats. Neither has a want: each runs inside
its node's own composition at a build of the old form (Scope.InOwn), and is composed as Carried of the indexer's
manifest. That manifest is the catalogue's, which no build resolved, so it has no bundles (catalogue/build.go: Bundles
are set only by a build's resolution), and the runtime's MESH_TOOL_MODULES and MESH_TOOL_ENV are made from bundles
(catalogue/runtime.go). The controller composed them from the resolved manifest its store kept.

The kind: a module composed as what runs of it (Stub, Carried: held, or folded at an old build) contributes to the
others from its manifest, not from what runs. For a held module with a new build it is the other way round: its new
bundles are served, which were never delivered.

- The quick fix: read the old-form build's resolved manifest for a folded module. One instance; a held module still
  hands its new bundles.
- The thorough fix, chosen: compose.ToolsRunning reads what the node's runtime serves as it runs (its process's
  MESH_TOOL_MODULES and MESH_TOOL_ENV, each bundle's archive), frombus gives it as Node.RunsTools, and every stubbed
  module of the machine's runtime (held or carried) contributes those bundles, whatever its manifest says.

Tests: two rows (folded, held) added to TestCompositionsAreTheDeclarationSplitByModule; no new test function.
2026-10-29 01:30:50 +01:00
mesh-tools 394fec8a63 Merge pull request 'A held module composes only as what runs of it, so its hold no longer spreads to the modules that use it (issue 1126)' (#128) from fix/1126-held-as-runs into main 2026-10-29 00:07:03 +00:00
jschoubben c2999a4e13 A held module is no dependence of its neighbours: its hold stops spreading (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-sdk@394fec8a63c1
mesh/delivery-group group fix/1126-held-as-runs merged: every member is merged
mesh/check pass c2999a4e13cc on d4c61ee8ffd0 | unit 3054 run, 234 kept, 36 skipped | integration 3075 run, 266 kept, 29 skipped | over bound: unit (sl…
novox's seed p-20261028t231141-ec84b4 refused 11 modules "what it composes of ..., held, is not what runs of them":
de-spiegel's hold spread through mailu to postgres, nftables and route-proxy, and from postgres to every consumer, and
gitea, held so, refused the node: "its credential of postgres-database is handed to the provisioner and the module is
not: nothing would fill it". Three rounds each moved the error. The kind: a held module still took part in the
composition as a dependence, so its hold spread.

- The quick fix: let heldAsToday also follow dependsOnHeld's words, and skip the handed refusal for gitea. It moves the
  error once more and leaves every node's applying path holding consumers.
- The thorough fix, chosen: a held module composes only as what runs of it. composeEach no longer compares the
  composition with each held module as its stub (dependsOnHeld removed): a consumer or collector composes against what
  runs of the held module like any other module, and a real difference is still a named difference in the answer, a
  module's own refusal still its refusal. A stub (compose.Stub, catalogue.Manifest.Stubbed) leaves the credentials and
  own secrets it would hand the provisioner out of its plan as they run, never refused for want of a plan. Only what a
  held module's new manifest says to others in their own files is still held behind it (dependents: the provider of
  what it contributes to, the filter where it listens), which heldAsToday already follows.

Tests: rows added to existing tests (a held provider with a binding consumer, which the old code held in the seed's
words; a held feeder of a collector; a held module with a handed credential and own secret). No new test function.
2026-10-29 00:49:00 +01:00
mesh-tools d4c61ee8ff Merge pull request 'The seed leaves a module as it runs today when it is held, failed or at an old-form build, and the modules held only behind it too (issue 1126)' (#127) from fix/1126-seed-oldbuild into main 2026-10-28 23:05:04 +00:00
jschoubben 76f7c4d223 A stub keeps its items and their templates, so a held module still contributes to other modules' collections; it drops only the templates for its own files (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-sdk@d4c61ee8ffd0
mesh/delivery-group group fix/1126-seed-oldbuild merged: every member is merged
mesh/check pass 76f7c4d2238f on d747c1128e42 | unit 3018 run, 270 kept, 36 skipped | integration 3039 run, 302 kept, 29 skipped | over bound: unit (sl…
2026-10-28 23:56:33 +01:00
jschoubben f609be7864 Keep as today what a node keeps today and what is held only behind it, saying how; a stub hands and collects nothing of its own (issue 1126) 2026-10-28 23:49:22 +01:00
mesh-tools d747c1128e Merge pull request 'L3: the vault mints bus accounts, the grants part and the node seat node-engine (issue 774, ADR 0561), rebased onto L2' (#126) from feat/774-vault-mints-sdk into main 2026-10-28 22:27:36 +00:00
mesh-tools f9afe32888 Merge pull request 'A node's answer holds as today what it holds today, and the seed carries its nodes' differences, named (issue 1126)' (#125) from fix/1126-seed-sdk into main 2026-10-28 21:59:48 +00:00
jschoubben 77b48a4f1a Name in the seed's brief the first differing field's value as it runs and as composed, so the operator sees what changes (issue 1126)
mesh/delivery-group group fix/1126-seed-held rejected: composed together: the checker's verdict at the members' heads (63905bd4b284): the composition
mesh/delivery merged: source.main moved to novox/mesh-sdk@f9afe328881f
mesh/check being carried to 77b48a4f1aff on f9afe328881f | from 53f26493697e
2026-10-28 22:26:05 +01:00
jschoubben e84a121b8a Let the seed carry the differences its nodes found, each named in its brief for the operator to approve knowingly; a refusal or silence still fails (issue 1126) 2026-10-28 22:24:57 +01:00
jschoubben 06b13ec1cc Show in a proposal's brief what stays held as each node holds it today (issue 1126)
mesh/delivery-group group fix/1126-seed-held rejected: composed together: the checker's verdict at the members' heads (2b8e3078c117): the composition
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: the composition: the manifest of mesh-delivery-assigner uses what mesh-controller at ba4d15657343d3116bd6657f85e53630a7dbabf2 (mesh…
2026-10-28 22:17:57 +01:00
jschoubben 0830fcb25e Hold that a refused module not held today keeps the answer refused (issue 1126) 2026-10-28 22:17:40 +01:00
jschoubben b03bf4c118 A node's answer holds as today what it holds today, so a seed that reproduces today is the same (issue 1126) 2026-10-28 22:13:37 +01:00
mesh-tools 53f2649369 Merge pull request 'A build puller parked at capacity 0 wakes when its settings file changes, and retries a failed setting follow (issue 734's kind)' (#122) from fix/park-wakes into main 2026-10-28 21:03:13 +00:00
jschoubben 8eafc0c5ce The placement reader names the seat mesh-placement's state, held by the assigner, as L2 moved it (issue 774, ADR 0567)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-delivery@f6df07605dd0 was judged on trunk 5b2e5bc5…
mesh/delivery merged: source.main moved to novox/mesh-sdk@d747c1128e42
mesh/check checking 8eafc0c5ce3c on d747c1128e42
2026-10-28 21:54:04 +01:00
jschoubben 50de0a5a71 The node seat's acts reads one act by its id (issue 774, ADR 0563 §9) 2026-10-28 21:54:04 +01:00
jschoubben 2323ba19db The node-engine's declared state acts: created by the grants part as a placed module's, its manifest placed on every node, and written by each node's engine under its own keys alone (issue 774, ADR 0563 §2, §3) 2026-10-28 21:54:04 +01:00
jschoubben 63607ba169 The vault's secret-ask is said to answer the callers its manifest names (issue 774, ADR 0563 §8) 2026-10-28 21:54:04 +01:00
jschoubben 82c54870ed A module's trusted list of callers in the SDK, so the vault's secret-ask answers the callers its manifest names as the runtime proved them (issue 774, ADR 0563 §8) 2026-10-28 21:54:04 +01:00
jschoubben 79231a9e6c A node's engine answers its own node's node-engine verbs, says the seat's events and asks the operator as node-engine-<node>, and no other account answers them (issue 774, ADR 0563 §1, §3, §6) 2026-10-28 21:54:04 +01:00
jschoubben 222d605f6f The node seat node-engine: held by every node's engine, its six verbs and three events, and no module may claim it (issue 774, ADR 0563 §1) 2026-10-28 21:54:04 +01:00
jschoubben 3a8997c58a Bundle the mesh's authorities by certificate, and check a node's certificate against every one a bundle holds (issue 774, ADR 0550 §2) 2026-10-28 21:54:04 +01:00
jschoubben d6db3f38e6 The vault signs: the contract of its state signers, its verb sign and its event signed, a node's grants to ask it and to read what it trusts, and the authority composed as a bundle (issue 774, ADR 0550 §2) 2026-10-28 21:54:04 +01:00
jschoubben 18a47ac3c7 Placed: a module in the first revision heard of a node's key stands since before it, so the seed's placement does not leave out what already ran (issue 774) 2026-10-28 21:54:04 +01:00
jschoubben 1386219d4d One reader of the catalogue's placement and the toolchain gate over it, in place of the three copies over the controller's assignments (issue 774, ADR 0549 §3, issue 1940); the assigner may read the lease (ADR 0549 §5) 2026-10-28 21:54:04 +01:00
jschoubben 10d4d2f845 Checker seat test: the try replaces both shell commands (issue 774) 2026-10-28 21:54:04 +01:00
jschoubben b3100c55a1 Checker seat: the try's verb also replaces docker run postgres, as the controller's try did (ADR 0547 §10, issue 774) 2026-10-28 21:54:04 +01:00
jschoubben 0062bdc4a6 buscompose: the assigner may read the controller's lease, to choose where grants are read from (issue 774) 2026-10-28 21:54:04 +01:00
jschoubben a1a688b19e delivery: the grants part's grant keys and the source the assigner reads (a1; issue 774)
Why: the grants part writes grants.<module>.<fp> in shadow beside the controller's; the assigner switches source only when the lease is handed away.
2026-10-28 21:54:04 +01:00
jschoubben 54b17164ef Checker seat test: fold into the closed-set seats test, no new test function (issue 774) 2026-10-28 21:54:04 +01:00
jschoubben 3ed9a0e342 Checker seat: the try's end by hand is a verb too (issue 774) 2026-10-28 21:54:04 +01:00
jschoubben 9b5657a3d6 Seat mesh-delivery-checker: the try and its read are verbs of the checker's own seat, replacing sh merge-check.sh (ADR 0547 §10, issue 774) 2026-10-28 21:54:04 +01:00
jschoubben 1d4637e46b Name the vault's verb issue-account on its seat, so its holder may serve it (issue 774, ADR 0561 §9) 2026-10-28 21:54:04 +01:00
jschoubben fefa8b88ea Say when the lease is handed away, let the vault and the grants part read it, and let the grants part create the consumers an account needs, so both turn on at the lease write alone (issue 774, ADR 0561 §3, §6) 2026-10-28 21:54:04 +01:00
jschoubben 8d9b3f779d Build a minted account's value as the controller does, byte for byte, so a runtime reads the vault's unchanged (issue 774, ADR 0561 §2) 2026-10-28 21:54:04 +01:00
jschoubben da506bb3b8 Say which bus accounts the grants part wants, in a state only the vault reads, and what the vault says when it mints one, so the vault can mint from the grants part's word (issue 774, ADR 0561 §2) 2026-10-28 21:54:04 +01:00
mesh-tools af08be36a2 Merge pull request 'A waiting ask hears its step's end even when its own watch goes deaf (four lost checks, 2026-10-28)' (#121) from fix/lost-check into main 2026-10-28 20:40:58 +00:00
mesh-tools 6b43afa2ab Merge pull request 'One predicate for a module the mesh provides itself (mesh-wireguard), used by the placement check (issue 1126)' (#120) from fix/1126-provided into main 2026-10-28 20:39:20 +00:00
jschoubben f91340fe82 buildrun: open a steps watch again that closed having heard nothing, up to 3 closes in a row
mesh/delivery merged: source.main moved to novox/mesh-sdk@af08be36a27a
mesh/delivery-group group fix/lost-check merged: every member is merged
mesh/check checking f91340fe82ef on 6b43afa2ab38 | novox/mesh-sdk#121 in group fix/lost-check
A watch that closed at once was left for good, so an ask whose own watch
went deaf was read only at a reconnect (review of fix/lost-check).
2026-10-28 20:35:02 +01:00
jschoubben 6aeb401095 buildrun: read an ask's step at every step end of its build module, so a deaf watch loses no end
mesh-catalog #553's verdict stayed running after its step passed on shanks
at 19:09:31Z on 2026-10-28: the watch of that one step key, opened at
18:46:05Z, heard neither the start nor the end and never closed. HeardEnds
stands between Asker and each ask's watch: nudged at each step end of the
same build module (HearEnds: one watch per steps state) and at a reconnect,
every waited ask reads its keys now; a watch that closes is opened again, at
most 3 times. The checker and the builder both wait through it.

Tests: a section of the asker's test (deaf, nudged by another module and by
its own, a watch that closes); counts unchanged.
2026-10-28 20:29:30 +01:00
jschoubben 5228a5782d Name the modules the mesh provides itself in one predicate, so the seed's check takes the private network as the composer and the grants part do (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-sdk@6b43afa2ab38
mesh/delivery-group group fix/1126-provided merged: every member is merged
mesh/check checking 5228a5782d5c on d7c790d2422c | novox/mesh-sdk#120 in group fix/1126-provided
2026-10-28 20:12:51 +01:00
jschoubben 9e5666b4d6 Settings watch: arm again when the kernel drops it (IN_IGNORED), said once; the test waits on the arm, not a sleep
mesh/delivery merged: source.main moved to novox/mesh-sdk@53f26493697e
mesh/delivery-group group fix/park-wakes merged: every member is merged
mesh/check pass 9e5666b4d6f5 on af08be36a27a | unit 5960 run, 332 kept, 18 skipped | integration 6012 run, 365 kept, 30 skipped | over bound: unit, in…
2026-10-28 20:08:21 +01:00
jschoubben c006aa8242 Build puller: wake on a settings file write, retry a failed config follow at the next event, say a capacity of 0 and why 2026-10-28 20:01:46 +01:00
mesh-tools d7c790d242 Merge pull request 'The seat mesh-placement's contract, and node grants for its states (issue 1126, delivery 1 of 2)' (#119) from feat/1126-grants-first into main 2026-10-28 18:21:33 +00:00
jschoubben 38dbe1d3a3 Let a node agent hear the creation of the seat's placement and candidates, so it binds them with no timer (ADR 0567 §9, issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-sdk@d7c790d2422c
mesh/delivery-group group feat/1126-grants-first merged: every member is merged
mesh/check pass 38dbe1d3a388 on 98f2f2406921 | unit 1909 run, 715 kept, 3 skipped | integration 1931 run, 813 kept, 3 skipped | over bound: unit (slow…
2026-10-28 18:47:13 +01:00
jschoubben a2dbcf670b Bound a proposal's answers and ask by pulse keys whose expiry the holder hears, and name a hand-back proposal, so no timer waits (ADR 0567 §4, §7, issue 1126) 2026-10-28 18:43:39 +01:00
jschoubben e987a8280b Let a module's config values be proposed before the lease write: only a layer's places and accesses wait (ADR 0567 §13, issue 1126) 2026-10-28 18:42:05 +01:00
jschoubben 45f5f76811 Keep the placement ask among the assigner's asks: its Approve's level, not its kind, needs the proof (ADR 0567 §5, issue 1126) 2026-10-28 18:30:00 +01:00
jschoubben 4b4a173d84 Let withdraw mark a proposal for its holder to end, since the holder's process alone says the seat's events (ADR 0567 §2, issue 1126) 2026-10-28 18:21:59 +01:00
jschoubben 0282229fd9 Name the placement ask and a proposal's again and warrant, so the holder asks through the teller as its other asks (ADR 0567 §5, §10, issue 1126) 2026-10-28 18:20:27 +01:00
jschoubben 5a2a7eebfd Hold that a check's node names reach the digest tool too, so a kept verdict misses when they change (ADR 0567 §12, issue 1126) 2026-10-28 18:17:08 +01:00
jschoubben 004bbc2ce2 Carry every node's name on a check ask to the names check, so no repository lists them (ADR 0567 §12, issue 1126) 2026-10-28 18:13:57 +01:00
jschoubben fed64292f0 Read build capacity as an integer and build hours as one window, default 0: no repository names a node (ADR 0567 §12, issue 1126) 2026-10-28 18:13:18 +01:00
jschoubben 7a82384e21 Grant a node's agent the seat's placement and candidates, so it composes from the seat's state (ADR 0567 §3, §9, issue 1126) 2026-10-28 18:12:09 +01:00
jschoubben 7c565f2aa5 Move Intended and JudgeAnswers into mesh-sdk for a proposal: a difference is shown, only a refusal or silence fails (ADR 0567 §4, issue 1126) 2026-10-28 18:11:18 +01:00
jschoubben edc6f269f3 Name the seat's states, candidates by proposal and the proposal itself, so readers and the holder share one contract (ADR 0567 §1-§6, issue 1126) 2026-10-28 18:09:38 +01:00
jschoubben 902e3c9e96 Refuse placement files on a forge message and delete placement-seed: no repository holds placement (ADR 0567 §11, issue 1126) 2026-10-28 18:09:38 +01:00
jschoubben c16e6213e1 Define the seat mesh-placement with its seven verbs, so placement has one contract whoever holds it (ADR 0567, issue 1126) 2026-10-28 18:05:16 +01:00
mesh-tools 98f2f24069 Merge pull request 'A node holds the one module it cannot compose and applies the rest, as its shadow already answers (group fix/1126-apply-culprit, issue 1126)' (#118) from fix/1126-apply-culprit into main 2026-10-28 16:34:46 +00:00
jschoubben 0de3006415 A module refused alone is held, composed as what runs of it, on the node that applies as in its answer: one rule for both paths (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-sdk@98f2f2406921
mesh/delivery-group group fix/1126-apply-culprit merged: every member is merged
mesh/check pass 0de30064156a on 7aedd28f01dc | unit 2667 run, 590 kept, 36 skipped | integration 2676 run, 633 kept, 29 skipped | over bound: unit (sl…
The applying path (frombus.Compose) composed with no culprit, so one module's
refusal came back as the whole node's: novox's apply would have refused all its
modules for de-spiegel's unset route names. composeEach is now the one rule
Compose and Answer both follow:
- a module whose stub changes the refusal is held (compose.Node.Held); in the
  applying path a stand-in for a module placed here holds that module; only a
  refusal of no single module refuses the node;
- a held module is composed from what runs of it (compose.Carried of its own
  composition in Running) and carried in the node's own wherever that runs it,
  folded or leaving the fold, so the node's own apply never removes it (issue
  1945 by a new path) and what the others read of it is what runs;
- none of a held module's own composition is in what is applied;
- a module whose composition differs when the held ones are composed as stubs
  is held too; the node's own differing refuses the node;
- since a held module is composed from its new manifest, what reads that
  manifest is held with it: the providers of what it contributes to (the
  proxy its routes) and, when it listens, the filter's holder.

Quick fix weighed: a culprit loop in Compose alone. Thorough, chosen: one
shared function, since the kind met twice (seed-8 shadow, now the apply), and
review rounds 1 and 2 found its leaks.
2026-10-28 17:22:35 +01:00
mesh-tools 7aedd28f01 Merge pull request 'A module's own refusal refuses that module, not the node; contributions say unset values in config words; a folded module owns its manifest's paths (issue 1126, group fix/1126-seed-8)' (#117) from fix/1126-seed-8 into main 2026-10-28 13:27:05 +00:00
jschoubben f387581b6b A module's own refusal refuses that module, not the node: a stub whose contributions still fail the same is tried bare (issue 1126, seed-8: novox refused the whole candidate for de-spiegel's unset route names)
mesh/delivery merged: source.main moved to novox/mesh-sdk@7aedd28f01dc
mesh/delivery-group group fix/1126-seed-8 merged: every member is merged
mesh/check pass f387581b6bfb on 3d021445f743 | unit 3018 run, 234 kept, 36 skipped | integration 3038 run, 266 kept, 29 skipped | over bound: unit (sl…
2026-10-28 14:07:59 +01:00
jschoubben eda82b1598 A folded module owns its manifest's paths, as the controller tells node-files them, not every path it runs at (issue 1126, seed-8: f1-circuits, snake and txt-game shifted node-tools' owned paths on ace) 2026-10-28 14:07:59 +01:00
jschoubben 06742d0448 A contribution's unset required value is refused in its module's own form's words, as a file's is (issue 1126, seed-8: de-spiegel in the config form was told ${setting:…} and settings set) 2026-10-28 14:07:59 +01:00
mesh-tools 3d021445f7 Merge pull request 'A build module may say its node is not eligible, so asks go to a node that can build (group fix/ace-tun)' (#116) from fix/ace-tun into main 2026-10-28 13:07:40 +00:00
jschoubben 1618837ea9 A build seat's puller takes nothing while its module finds its node cannot build, and says why
mesh/delivery merged: source.main moved to novox/mesh-sdk@3d021445f743
mesh/delivery-group group fix/ace-tun merged: every member is merged
mesh/check pass 1618837ea9fe on 6097e4f3e040 | carried from mesh-catalog 25f615456423 | unit 2293 run, 3984 kept, 16 skipped | integration 2331 run, 4…
A node whose module proves a need of the machine missing (ace's /dev/net/tun, 2026-10-28) pulled
image asks and failed each one; judged ineligible, its capacity is 0 with the reason, as build hours
are, so the asks go to a node that can build them (issue 1126 note).
2026-10-28 12:50:16 +01:00
mesh-tools 6097e4f3e0 Merge pull request 'A file's secrets are sealed values entry by entry, masked and named per secret (issue 1126; group fix/1126-seed-same-3)' (#115) from fix/1126-seed-same-3 into main 2026-10-28 11:40:35 +00:00
jschoubben 8520a4c099 Only a file's secrets are split entry by entry when masked and named: a vault box under vault-sealed is one masked word and one path, judged whole by what it opens to (issue 1126, C1; review of seed-same-3, round 1)
mesh/delivery merged: source.main moved to novox/mesh-sdk@6097e4f3e040
mesh/delivery-group group fix/1126-seed-same-3 merged: every member is merged
mesh/check pass 8520a4c09960 on 79eb58efe4b0 | carried from mesh-catalog 774fcdfa93ad | unit 2188 run, 1064 kept, 36 skipped | integration 2197 run, 1…
2026-10-28 12:22:37 +01:00
jschoubben 70de9429e5 A file's secrets are sealed values entry by entry: each a slot of its own, kept where it opens to the value that runs, masked and named per secret (issue 1126, C1; the seed-7 probe: about 45 files on ace and novox differed by their secrets' bytes alone) 2026-10-28 12:02:39 +01:00
mesh-tools 79eb58efe4 Merge pull request 'Compose as the controller does, so a node's shadow can answer same (issue 1126, group fix/1126-seed-same)' (#114) from fix/1126-seed-same into main 2026-10-28 09:46:24 +00:00
jschoubben 3028c7ec4e A drifted own secret is read from a resource id as a node runs it, its module's name first (only-office.needs-jwt), so the node agent finds what to hand the vault (issue 1126, cause C4; review round 2)
mesh/delivery merged: source.main moved to novox/mesh-sdk@79eb58efe4b0
mesh/delivery-group group fix/1126-seed-same merged: every member is merged
mesh/check failed: integration: github.com/novox/mesh-host/cmd/mesh-host.TestNatsANodeThePlacementNamesComposesAndAppliesItsOwn: the node did not comp…
delivery.OwnNeedName reads <module>.needs-<name> and the bare needs-<name>; OwnValuesDiffer and the node agent use it.
The fixtures carry the id shape captured from novox's shadow answer.
2026-10-28 10:00:29 +01:00
jschoubben 1062b33d32 A node's shadow answer counts the values that drifted from the vault's, and names the own secrets among them (issue 1126, cause C4)
delivery.OwnValuesDiffer, ValuesDiffer and WithValuesDiffer read the shown bodies masked as another value; a candidate
answer carries values_differ. The node agent hands the vault those own secrets' running values through move-in's
replace_running, so no seat verb and no bus grant changes here.
2026-10-28 09:49:03 +01:00
jschoubben 16b8d8a1ee Compose as the controller does, so a node's shadow can answer same (issue 1126, C1 C2 C3 C5 C6, T1)
Seals are reused and compared by slot (resource, field), never across resources;
one rule says which resolved config values are laid; a seat path that steps back
is owned by the seat it steps back for, whatever the seat order; a folded module
of an old-form build is composed from what runs of it; the operator export's file
is carried as it runs; a differing resource names its differing JSON paths, found
before its bodies are cut, and an answer is fitted to the node's key.
2026-10-28 09:36:36 +01:00
92 changed files with 6518 additions and 756 deletions
+5
View File
@@ -571,3 +571,8 @@ type ProofAnswer struct {
Accepted bool `json:"accepted"` Accepted bool `json:"accepted"`
Words string `json:"words"` Words string `json:"words"`
} }
// EngineAsker is the asker name a node's engine asks the operator under (novox/hq ADR 0563 §3, which changes ADR 0259
// §2 to admit a node's engine as an asker): `node-engine-<node>`, the last token of the by-caller subjects granted to that
// node's engine account alone, so no engine acts on another node's warrant.
func EngineAsker(node string) string { return "node-engine-" + node }
+3
View File
@@ -200,6 +200,9 @@ func (s AskSteps) level(ctx context.Context, job delivery.CheckJob, level string
if runner != "" { if runner != "" {
args["runner"] = runner args["runner"] = runner
} }
if len(job.NodeNames) > 0 {
args[delivery.NodeNamesArg] = job.NodeNames // the names check's (ADR 0567 §12)
}
return args return args
} }
test := func(args map[string]any) (bool, error) { test := func(args map[string]any) (bool, error) {
+9
View File
@@ -143,6 +143,15 @@ test {"level":"unit","packages":["x/pkg/a"],"workspace":"/w"}`
t.Fatalf("the unit level %+v", u) t.Fatalf("the unit level %+v", u)
} }
// ADR 0567 §12: a check's node names reach the digest and test tools, for the names check (a kept verdict misses when
// they change).
r.calls = nil
if _, err := RunAsk(context.Background(), r.steps(), ask(delivery.CheckJob{Levels: []string{delivery.LevelUnit}, NodeNames: []string{"anchor"},
Fetch: map[string]any{"repository": "novox/x", "commit": runCommit}})); err != nil || !strings.Contains(strings.Join(r.calls, "\n"), `digest {"level":"unit","node-names":["anchor"]`) ||
!strings.Contains(strings.Join(r.calls, "\n"), `test {"level":"unit","node-names":["anchor"]`) {
t.Fatalf("the node names are not handed to the digest and test tools: %v, %v", r.calls, err)
}
// Issue 890/AC2: a package declaring node-build-node is not tested on a node that lacks it but named unmet, and is // Issue 890/AC2: a package declaring node-build-node is not tested on a node that lacks it but named unmet, and is
// tested on a node that has it. // tested on a node that has it.
for _, has := range []bool{false, true} { for _, has := range []bool{false, true} {
+31 -37
View File
@@ -8,14 +8,15 @@ import (
"time" "time"
) )
// **A node may build only within its build hours** (the operator, 2026-10-26 21:18: "we should only use g14 as a builder // **A node may build only within its build hours** (the operator, 2026-10-26 21:18: the laptop builds only outside
// outside office hours", "from 18 o'clock until 6 in the morning"). A build module's `build-hours` setting names, per // office hours, "from 18 o'clock until 6 in the morning"). A build module's `build-hours` setting is the node's window
// node, the window in the node's local time in which it takes asks; outside it the capacity judged by is 0: steps running // in its local time, a config value set on each node's layer of mesh-config (ADR 0567 §12: no repository names a node);
// finish, no new one is taken. A node it does not name, or an empty setting, builds at any hour. The puller judges it at // outside it the capacity judged by is 0: steps running finish, no new one is taken. An empty value builds at any hour.
// every judgement and wakes on one timer to the window's next edge (a clock edge, not a poll of another part). // The puller judges it at every judgement and wakes on one timer to the window's next edge (a clock edge, not a poll of
// another part).
// BuildHoursSetting is the build modules' setting of the nodes' build windows: `<node>=HH:MM-HH:MM` pairs separated by // BuildHoursSetting is the build modules' setting of the node's build window: HH:MM-HH:MM in the node's local time; an
// commas, in the node's local time; an end before its start crosses midnight (g14=18:00-06:00). // end before its start crosses midnight (18:00-06:00).
const BuildHoursSetting = "build-hours" const BuildHoursSetting = "build-hours"
// BuildWindow is one node's build hours: minutes after local midnight, Start inclusive, End exclusive. // BuildWindow is one node's build hours: minutes after local midnight, Start inclusive, End exclusive.
@@ -51,35 +52,28 @@ func (w BuildWindow) NextEdge(t time.Time) time.Time {
return next return next
} }
// ParseBuildHours is node's window in a build-hours setting: nil when the setting is empty or does not name the node. // ParseBuildWindow reads a build-hours value, HH:MM-HH:MM: nil when it is empty.
func ParseBuildHours(setting, node string) (*BuildWindow, error) { func ParseBuildWindow(setting string) (*BuildWindow, error) {
var found *BuildWindow setting = strings.TrimSpace(setting)
for _, pair := range strings.Split(setting, ",") { if setting == "" {
pair = strings.TrimSpace(pair) return nil, nil
if pair == "" {
continue
}
name, span, ok := strings.Cut(pair, "=")
from, to, ok2 := strings.Cut(span, "-")
if !ok || !ok2 || strings.TrimSpace(name) == "" {
return nil, fmt.Errorf("the build hours %q: %q is not <node>=HH:MM-HH:MM", setting, pair)
}
start, err := minuteOf(from)
if err != nil {
return nil, fmt.Errorf("the build hours %q: %v", setting, err)
}
end, err := minuteOf(to)
if err != nil {
return nil, fmt.Errorf("the build hours %q: %v", setting, err)
}
if start == end {
return nil, fmt.Errorf("the build hours %q: %s opens and closes at the same minute", setting, pair)
}
if strings.TrimSpace(name) == node {
found = &BuildWindow{Start: start, End: end}
}
} }
return found, nil from, to, ok := strings.Cut(setting, "-")
if !ok {
return nil, fmt.Errorf("the build hours %q are not HH:MM-HH:MM", setting)
}
start, err := minuteOf(from)
if err != nil {
return nil, fmt.Errorf("the build hours %q: %v", setting, err)
}
end, err := minuteOf(to)
if err != nil {
return nil, fmt.Errorf("the build hours %q: %v", setting, err)
}
if start == end {
return nil, fmt.Errorf("the build hours %q open and close at the same minute", setting)
}
return &BuildWindow{Start: start, End: end}, nil
} }
// minuteOf reads HH:MM as minutes after midnight. // minuteOf reads HH:MM as minutes after midnight.
@@ -92,7 +86,7 @@ func minuteOf(s string) (int, error) {
return h*60 + m, nil return h*60 + m, nil
} }
// NodeBuildHours is node's build window, read from the module's settings file at path at every call, and why it is // NodeBuildHours is the node's build window, read from the module's settings file at path at every call, and why it is
// none when the value cannot be read. No path, no file value or an empty one is no window and no fault. A value that // none when the value cannot be read. No path, no file value or an empty one is no window and no fault. A value that
// cannot be read is no window too, so the node builds at any hour; the setting is a preference of hours, not a safety // cannot be read is no window too, so the node builds at any hour; the setting is a preference of hours, not a safety
// bound (a node that must never build has capacity 0). It is loud: the fault is said once per distinct fault through // bound (a node that must never build has capacity 0). It is loud: the fault is said once per distinct fault through
@@ -125,7 +119,7 @@ func NodeBuildHours(path, node string, logf func(string, ...any)) func() (*Build
if v, ok := s[BuildHoursSetting]; ok && json.Unmarshal(v, &setting) != nil { if v, ok := s[BuildHoursSetting]; ok && json.Unmarshal(v, &setting) != nil {
return fault(fmt.Sprintf("the setting %s %s is not a string", BuildHoursSetting, v)) return fault(fmt.Sprintf("the setting %s %s is not a string", BuildHoursSetting, v))
} }
w, err := ParseBuildHours(setting, node) w, err := ParseBuildWindow(setting)
if err != nil { if err != nil {
return fault(err.Error()) return fault(err.Error())
} }
+35 -35
View File
@@ -4,45 +4,25 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"os" "os"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
) )
// CapacitySetting is the build modules' setting of each node's build capacity (novox/hq ADR 0525 §5): `<node>=<asks at // CapacitySetting is the build modules' setting of the node's build capacity (novox/hq ADR 0525 §5, ADR 0567 §12): how
// once>` pairs separated by commas, as the builder's was (ADR 0485 §1). Each build module declares it with // many asks the node takes at once, an integer config value set on each node's layer of mesh-config. No repository names
// DefaultCapacities as its default, and the controller lays it into the module's settings file. // a node: each build module declares it with the default 0, which pulls nothing, and the controller (or mesh-config)
// lays the node's value into the module's settings file.
const CapacitySetting = "capacity" const CapacitySetting = "capacity"
// DefaultCapacities is the setting's default, the builder's before R4g: the control-node (novox) and the laptop (g14) // NodeCapacity is the node's build capacity, read from the module's settings file at path at every call, so a change
// build nothing. // takes effect at the next pull. No path, no file value, or a value that is not a whole number of at least 0 gives 0 and
const DefaultCapacities = "shanks=3,ace=2,g14=0,novox=0" // a value that cannot be read says why: pulling on a setting nobody can read could build on a node that must not.
func NodeCapacity(path, node string, logf func(string, ...any)) func() int {
// NodeCapacity is node's build capacity, read from the module's settings file at path at every call, so a change takes
// effect at the next pull; with no path it reads fallback. A node the setting does not name runs one ask at a time (as
// the builder's did). A file or a setting that cannot be read gives 0 and says why: pulling on a setting nobody can read
// could build on a node that must not.
func NodeCapacity(path, fallback, node string, logf func(string, ...any)) func() int {
say := func(format string, args ...any) { say := func(format string, args ...any) {
if logf != nil { if logf != nil {
logf(format, args...) logf(format, args...)
} }
} }
return func() int { return func() int {
setting := fallback n, err := ReadCapacity(path)
if path != "" {
raw, err := os.ReadFile(path)
if err != nil {
say("the build capacity cannot be read, so %s pulls nothing: %v", node, err)
return 0
}
var s map[string]any
if err := json.Unmarshal(raw, &s); err != nil {
say("the settings file %s is not JSON, so %s pulls nothing: %v", path, node, err)
return 0
}
setting, _ = s[CapacitySetting].(string)
}
n, err := capacityOf(setting, node)
if err != nil { if err != nil {
say("%v, so %s pulls nothing", err, node) say("%v, so %s pulls nothing", err, node)
return 0 return 0
@@ -51,11 +31,31 @@ func NodeCapacity(path, fallback, node string, logf func(string, ...any)) func()
} }
} }
// capacityOf is node's capacity in a setting's `<node>=<asks>` pairs: DefaultCapacity for a node it does not name. // ReadCapacity is the capacity the settings file at path holds, and why it cannot be read when it cannot (then 0).
func capacityOf(setting, node string) (int, error) { func ReadCapacity(path string) (int, error) {
caps, err := buildrun.ParseCapacity(setting) if path == "" {
if err != nil { return 0, nil
return 0, fmt.Errorf("the build capacity: %w", err)
} }
return (&buildrun.Places{Capacity: caps}).CapacityOf(node), nil raw, err := os.ReadFile(path)
if err != nil {
return 0, fmt.Errorf("the build capacity cannot be read: %w", err)
}
var s map[string]json.RawMessage
if err := json.Unmarshal(raw, &s); err != nil {
return 0, fmt.Errorf("the settings file %s is not JSON: %w", path, err)
}
return capacityOf(s[CapacitySetting])
}
// capacityOf is the capacity a config value holds: absent or null is 0; anything but a whole number of at least 0 is
// refused, naming it.
func capacityOf(value json.RawMessage) (int, error) {
if len(value) == 0 || string(value) == "null" {
return 0, nil
}
var n int
if err := json.Unmarshal(value, &n); err != nil || n < 0 {
return 0, fmt.Errorf("the build capacity %s is not a whole number of asks at once, 0 or more", value)
}
return n, nil
} }
+2 -2
View File
@@ -14,7 +14,7 @@ import (
) )
// **A build seat's capacity follows the CPU its node gives builds** (issue 1806's follow-up; issue 1890, 2026-10-26 // **A build seat's capacity follows the CPU its node gives builds** (issue 1806's follow-up; issue 1890, 2026-10-26
// 19:41: under the 50 % cap shanks' mesh-build-go kept capacity 3, three full mesh-catalog checks shared 12 cores, and // 19:41: under the 50 % cap the workstation's mesh-build-go kept capacity 3, three full mesh-catalog checks shared 12 cores, and
// each step ran past the 30-minute step bound with no test run). The configured capacity is an upper bound; the node's // each step ran past the 30-minute step bound with no test run). The configured capacity is an upper bound; the node's
// build slice bounds it again by the cores a step needs: floor(the slice's quota in cores / cores-per-step), one at // build slice bounds it again by the cores a step needs: floor(the slice's quota in cores / cores-per-step), one at
// least. The quota is read live from the cgroup at every judgement, so a changed cap is followed at the next take // least. The quota is read live from the cgroup at every judgement, so a changed cap is followed at the next take
@@ -27,7 +27,7 @@ import (
const CoresPerStepSetting = "cores-per-step" const CoresPerStepSetting = "cores-per-step"
// DefaultCoresPerStep are the build modules' defaults by language, from measured steps (2026-10-26): a full mesh-catalog // DefaultCoresPerStep are the build modules' defaults by language, from measured steps (2026-10-26): a full mesh-catalog
// check on mesh-build-go takes what it is given (three at once took 965 % of shanks' 1200 % slice, about 4 cores each, // check on mesh-build-go takes what it is given (three at once took 965 % of the workstation's 1200 % slice, about 4 cores each,
// and ran past the 30-minute step bound), so a Go step is given 6, half again what starved; a Node step (tsc, esbuild, a // and ran past the 30-minute step bound), so a Go step is given 6, half again what starved; a Node step (tsc, esbuild, a
// test runner: median 15 s) and an image build (buildah, its RUN lines mostly fetches) need 2. // test runner: median 15 s) and an image build (buildah, its RUN lines mostly fetches) need 2.
var DefaultCoresPerStep = map[string]float64{delivery.AskGo: 6, delivery.AskNode: 2, delivery.AskImage: 2} var DefaultCoresPerStep = map[string]float64{delivery.AskGo: 6, delivery.AskNode: 2, delivery.AskImage: 2}
+34 -33
View File
@@ -14,30 +14,20 @@ import (
// issue 1806's follow-up: the capacity judged by is the configured one bounded by the build slice's cores over the cores // issue 1806's follow-up: the capacity judged by is the configured one bounded by the build slice's cores over the cores
// a step needs (the three build nodes as they stand on 2026-10-26), and cores-per-step is read from the settings file. // a step needs (the three build nodes as they stand on 2026-10-26), and cores-per-step is read from the settings file.
func TestNodeCapacityAndLimits(t *testing.T) { func TestNodeCapacityAndLimits(t *testing.T) {
if got := NodeCapacity("", DefaultCapacities, "novox", nil)(); got != 0 { // ADR 0567 §12: the capacity is the node's integer, its default 0, and no default names a node.
t.Errorf("novox = %d by default, want 0", got) if got := NodeCapacity("", "laptop", nil)(); got != 0 {
} t.Errorf("no settings file = %d, want 0", got)
if got := NodeCapacity("", DefaultCapacities, "shanks", nil)(); got != 3 {
t.Errorf("shanks = %d by default, want 3", got)
} }
file := filepath.Join(t.TempDir(), "settings.json") file := filepath.Join(t.TempDir(), "settings.json")
os.WriteFile(file, []byte(`{"capacity":"shanks=1,ace=2"}`), 0o600) capacity := NodeCapacity(file, "anchor", nil)
capacity := NodeCapacity(file, DefaultCapacities, "shanks", nil) for value, want := range map[string]int{`{"capacity":1}`: 1, `{"capacity":4}`: 4, `{}`: 0, `{"capacity":null}`: 0,
if got := capacity(); got != 1 { `{"capacity":"anchor=3"}`: 0, `{"capacity":-1}`: 0, `{"capacity":1.5}`: 0} {
t.Errorf("shanks = %d from the file, want 1", got) os.WriteFile(file, []byte(value), 0o600)
if got := capacity(); got != want {
t.Errorf("%s = %d, want %d", value, got, want)
}
} }
os.WriteFile(file, []byte(`{"capacity":"shanks=4"}`), 0o600) if got := NodeCapacity(filepath.Join(t.TempDir(), "none"), "anchor", nil)(); got != 0 {
if got := capacity(); got != 4 {
t.Errorf("shanks = %d after the change, want 4", got)
}
if got := NodeCapacity(file, "", "pi", nil)(); got != 1 {
t.Errorf("a node not named = %d, want 1", got)
}
os.WriteFile(file, []byte(`{"capacity":"shanks=lots"}`), 0o600)
if got := capacity(); got != 0 {
t.Errorf("an unreadable setting = %d, want 0", got)
}
if got := NodeCapacity(filepath.Join(t.TempDir(), "none"), DefaultCapacities, "shanks", nil)(); got != 0 {
t.Errorf("a missing file = %d, want 0", got) t.Errorf("a missing file = %d, want 0", got)
} }
t.Setenv("MESH_BUILD_CPU_SHARE", "") t.Setenv("MESH_BUILD_CPU_SHARE", "")
@@ -74,7 +64,7 @@ func TestNodeCapacityAndLimits(t *testing.T) {
for _, c := range []struct { for _, c := range []struct {
file string file string
want float64 want float64
}{{`{"capacity":"shanks=2"}`, 6}, {`{"cores-per-step":4}`, 4}, {`{"cores-per-step":"3.5"}`, 3.5}, {`{"cores-per-step":0}`, 6}, {`{"cores-per-step":0}`, 6}} { }{{`{"capacity":2}`, 6}, {`{"cores-per-step":4}`, 4}, {`{"cores-per-step":"3.5"}`, 3.5}, {`{"cores-per-step":0}`, 6}, {`{"cores-per-step":0}`, 6}} {
os.WriteFile(file, []byte(c.file), 0o600) os.WriteFile(file, []byte(c.file), 0o600)
if got := cores(); got != c.want { if got := cores(); got != c.want {
t.Errorf("cores-per-step from %s = %g, want %g", c.file, got, c.want) t.Errorf("cores-per-step from %s = %g, want %g", c.file, got, c.want)
@@ -86,6 +76,8 @@ func TestNodeCapacityAndLimits(t *testing.T) {
// The operator, 2026-10-26 21:18: g14 builds only from 18:00 to 06:00, its local time. Outside its build hours the // The operator, 2026-10-26 21:18: g14 builds only from 18:00 to 06:00, its local time. Outside its build hours the
// capacity judged by is 0, said with the window and when it opens; a node not named, or a bad value, builds at any hour. // capacity judged by is 0, said with the window and when it opens; a node not named, or a bad value, builds at any hour.
// And a node its module proves cannot build (ace without /dev/net/tun, 2026-10-28) judges by 0 at every hour, saying
// why, so its asks go to a node that can.
cet := time.FixedZone("CET", 3600) cet := time.FixedZone("CET", 3600)
day := func(hh, mm int) time.Time { return time.Date(2026, 10, 26, hh, mm, 0, 0, cet) } day := func(hh, mm int) time.Time { return time.Date(2026, 10, 26, hh, mm, 0, 0, cet) }
was := now was := now
@@ -96,17 +88,21 @@ func TestNodeCapacityAndLimits(t *testing.T) {
effective int effective int
edge string // the window's next edge, HH:MM; "" for no window edge string // the window's next edge, HH:MM; "" for no window
bad bool bad bool
ineligible string // what the module's Eligible says; "" for eligible
}{ }{
{"g14 in the office", "g14=18:00-06:00", "g14", day(10, 0), 0, "18:00", false}, {"the laptop in the office", "18:00-06:00", "laptop", day(10, 0), 0, "18:00", false, ""},
{"g14 in the evening", "g14=18:00-06:00", "g14", day(18, 0), 2, "06:00", false}, {"the laptop in the evening", "18:00-06:00", "laptop", day(18, 0), 2, "06:00", false, ""},
{"g14 past midnight", "g14=18:00-06:00", "g14", day(5, 59), 2, "06:00", false}, {"the laptop past midnight", "18:00-06:00", "laptop", day(5, 59), 2, "06:00", false, ""},
{"g14 at the close", "g14=18:00-06:00", "g14", day(6, 0), 0, "18:00", false}, {"the laptop at the close", "18:00-06:00", "laptop", day(6, 0), 0, "18:00", false, ""},
{"a day window", "shanks=09:00-17:30,g14=18:00-06:00", "shanks", day(17, 29), 2, "17:30", false}, {"a day window", "09:00-17:30", "anchor", day(17, 29), 2, "17:30", false, ""},
{"a node not named", "g14=18:00-06:00", "ace", day(10, 0), 2, "", false}, {"no window", "", "anchor", day(10, 0), 2, "", false, ""},
{"a bad time", "g14=18:00-6", "g14", day(10, 0), 2, "", true}, {"a bad time", "18:00-6", "laptop", day(10, 0), 2, "", true, ""},
{"no span", "g14=18:00-18:00", "g14", day(10, 0), 2, "", true}, {"no span", "18:00-18:00", "laptop", day(10, 0), 2, "", true, ""},
{"the old form naming a node", "laptop=18:00-06:00", "laptop", day(10, 0), 2, "", true, ""},
{"a node without /dev/net/tun", "", "anchor", day(10, 0), 0, "", false, "/dev/net/tun cannot be opened: no such device"},
{"not eligible within its hours", "18:00-06:00", "laptop", day(19, 0), 0, "06:00", false, "/dev/net/tun cannot be opened: no such device"},
} { } {
w, err := ParseBuildHours(c.setting, c.node) w, err := ParseBuildWindow(c.setting)
if (err != nil) != c.bad { if (err != nil) != c.bad {
t.Errorf("%s: parse %v", c.name, err) t.Errorf("%s: parse %v", c.name, err)
} }
@@ -121,11 +117,16 @@ func TestNodeCapacityAndLimits(t *testing.T) {
file := filepath.Join(t.TempDir(), "settings.json") file := filepath.Join(t.TempDir(), "settings.json")
os.WriteFile(file, []byte(`{"build-hours":"`+c.setting+`"}`), 0o600) os.WriteFile(file, []byte(`{"build-hours":"`+c.setting+`"}`), 0o600)
now = func() time.Time { return c.at } now = func() time.Time { return c.at }
p := &Puller{Language: "go", Node: c.node, Capacity: func() int { return 2 }, BuildHours: NodeBuildHours(file, c.node, nil)} p := &Puller{Language: "go", Node: c.node, Capacity: func() int { return 2 }, BuildHours: NodeBuildHours(file, c.node, nil),
Eligible: func() string { return c.ineligible }}
got := p.CapacityDetail() got := p.CapacityDetail()
if got.Effective != c.effective || got.OutsideHours != (c.effective == 0) || (c.effective == 0 && !strings.Contains(got.Why, "outside its build hours 18:00-06:00 CET until 18:00")) { outside := c.effective == 0 && c.ineligible == ""
if got.Effective != c.effective || got.OutsideHours != outside || (outside && !strings.Contains(got.Why, "outside its build hours 18:00-06:00 CET until 18:00")) {
t.Errorf("%s: %+v, want capacity %d", c.name, got, c.effective) t.Errorf("%s: %+v, want capacity %d", c.name, got, c.effective)
} }
if got.Ineligible != c.ineligible || (c.ineligible != "" && !strings.HasPrefix(got.Why, "0, not eligible: "+c.ineligible+" (otherwise ")) {
t.Errorf("%s: not eligible is not said: %+v", c.name, got)
}
// A value that cannot be read builds at any hour, loudly: the record names the value and says why. // A value that cannot be read builds at any hour, loudly: the record names the value and says why.
if c.bad && (!strings.Contains(got.BuildHoursFault, c.setting) || !strings.Contains(got.Why, "at any hour, because its build hours cannot be read")) { if c.bad && (!strings.Contains(got.BuildHoursFault, c.setting) || !strings.Contains(got.Why, "at any hour, because its build hours cannot be read")) {
t.Errorf("%s: a bad value is not said in the record: %+v", c.name, got) t.Errorf("%s: a bad value is not said in the record: %+v", c.name, got)
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
// **A running ask says its phases** (issue 1890, 2026-10-26 19:09: a mesh-catalog try's step on shanks ran 30 minutes // **A running ask says its phases** (issue 1890, 2026-10-26 19:09: a mesh-catalog try's step on the workstation ran 30 minutes
// and its module's journal held no line naming its ask, so nobody could tell whether it fetched, linted or tested). The // and its module's journal held no line naming its ask, so nobody could tell whether it fetched, linted or tested). The
// puller hands every ask's run a phase reporter in its context; RunAsk and a module's own step bodies call Phase as they // puller hands every ask's run a phase reporter in its context; RunAsk and a module's own step bodies call Phase as they
// move (fetch and merge, a level's digest, lint, a level's tests), and each phase is logged with the ask's id and its // move (fetch and merge, a level's digest, lint, a level's tests), and each phase is logged with the ask's id and its
+101 -18
View File
@@ -86,11 +86,24 @@ type Puller struct {
// mesh-config resolves for `build-hours` takes its place once handed. Outside the window the capacity judged by is 0: // mesh-config resolves for `build-hours` takes its place once handed. Outside the window the capacity judged by is 0:
// steps running finish, no new one is taken; the puller wakes at the window's next edge. Nil builds at any hour. // steps running finish, no new one is taken; the puller wakes at the window's next edge. Nil builds at any hour.
BuildHours func() (*BuildWindow, string) BuildHours func() (*BuildWindow, string)
// Eligible says why this node cannot run the seat's asks at all now, and "" when it can: what the module proves of
// its machine and finds missing (mesh-build-image: /dev/net/tun cannot be opened, so buildah's network for a RUN line
// cannot be made; 2026-10-28 on ace). Read at every judgement, so it is cheap (a module answers its last look).
// While it says a reason the capacity judged by is 0, said with the reason: steps running finish, no new one is
// taken, and the asks go to a node that can run them instead of failing here. The module calls Wake when its answer
// changes. Nil is always eligible.
Eligible func() string
// Has says whether this node has a need an ask names (delivery.AskNeeds, issue 890). An ask needing what it lacks is // Has says whether this node has a need an ask names (delivery.AskNeeds, issue 890). An ask needing what it lacks is
// declined: its step ends refused as declined (delivery.StepDeclined), the message is dropped for its asker to put // declined: its step ends refused as declined (delivery.StepDeclined), the message is dropped for its asker to put
// again, and the worker it came from is not looked at again until the next pull's wait, so a node lacking the need // again, and the worker it came from is not looked at again until the next pull's wait, so a node lacking the need
// does not take the asker's next copy at once. Nil has nothing. // does not take the asker's next copy at once. Nil has nothing.
Has func(need string) bool Has func(need string) bool
// SettingsFile is the module's settings file. With it set the puller wakes whenever the file is written (a file event,
// WatchSettings), and reads Capacity from it itself, keeping why it cannot be read (CapacityNote); a parked puller at
// capacity 0 therefore wakes when the node-engine refills the file, and a failed config follow is tried again then.
SettingsFile string
// watching, when set, is called each time the settings file's watch is (re)armed: a test waits on it, not on a sleep.
watching func()
mu sync.Mutex mu sync.Mutex
inHand int inHand int
@@ -101,6 +114,14 @@ type Puller struct {
// watched is the capacity the config watch handed last, used in place of Capacity once hasWatched. // watched is the capacity the config watch handed last, used in place of Capacity once hasWatched.
watched int watched int
hasWatched bool hasWatched bool
// capFollow and hoursFollow are the state of the two config follows: followed once WatchConfig took them, trying
// while a start is in flight; followFault is the last failure said, so each failure is said once.
capFollowed, capTrying, hoursFollowed, hoursTrying bool
followFault string
// fileFault is why the settings file's capacity cannot be read, "" when it can.
fileFault string
// capWhy is why the config watch's last value could not be read, "" when it could.
capWhy string
// hours is the build window the config watch handed last, and hoursFault why its value cannot be read; used in place // hours is the build window the config watch handed last, and hoursFault why its value cannot be read; used in place
// of BuildHours once hasHours. // of BuildHours once hasHours.
hours *BuildWindow hours *BuildWindow
@@ -241,6 +262,21 @@ func (p *Puller) roomNow() (ok bool, capacity, inHand int) {
return !p.paused && p.inHand < capacity, capacity, p.inHand return !p.paused && p.inHand < capacity, capacity, p.inHand
} }
// CapacityNote says when the puller takes nothing for want of a capacity: why, and whether it is a fault (a setting that
// cannot be read or is refused, so a person must look) and not a choice (a capacity set to 0). "" when it pulls.
func (p *Puller) CapacityNote() (why string, fault bool) {
c := p.CapacityDetail()
if c.Effective > 0 {
return "", false
}
p.mu.Lock()
defer p.mu.Unlock()
if p.fileFault != "" && !p.hasWatched {
return "the " + p.Language + " build capacity of " + p.Node + " is 0 because it cannot be read: " + p.fileFault, true
}
return "the " + p.Language + " build capacity of " + p.Node + " is " + c.Why, p.hasWatched && p.capWhy != ""
}
// CapacityNow is the capacity the puller judges by (CapacityDetail's Effective). // CapacityNow is the capacity the puller judges by (CapacityDetail's Effective).
func (p *Puller) CapacityNow() int { func (p *Puller) CapacityNow() int {
return p.CapacityDetail().Effective return p.CapacityDetail().Effective
@@ -258,6 +294,15 @@ func (p *Puller) CapacityDetail() delivery.Capacity {
switch { switch {
case has: case has:
configured, from = watched, "mesh-config, as the config watch handed it" configured, from = watched, "mesh-config, as the config watch handed it"
case p.SettingsFile != "":
n, err := ReadCapacity(p.SettingsFile)
configured = n
p.mu.Lock()
p.fileFault = ""
if err != nil {
p.fileFault = err.Error()
}
p.mu.Unlock()
case p.Capacity != nil: case p.Capacity != nil:
configured = p.Capacity() configured = p.Capacity()
} }
@@ -291,6 +336,13 @@ func (p *Puller) CapacityDetail() delivery.Capacity {
c.Why = fmt.Sprintf("%s; within its build hours %s until %s", c.Why, c.BuildHours, w.NextEdge(at).Format("15:04")) c.Why = fmt.Sprintf("%s; within its build hours %s until %s", c.Why, c.BuildHours, w.NextEdge(at).Format("15:04"))
} }
} }
if p.Eligible != nil {
if why := p.Eligible(); why != "" {
c.Ineligible = why
c.Why = fmt.Sprintf("0, not eligible: %s (otherwise %s)", why, c.Why)
c.Effective = 0
}
}
p.sayCapacity(c) p.sayCapacity(c)
return c return c
} }
@@ -326,7 +378,8 @@ func (p *Puller) sayCapacity(c delivery.Capacity) {
p.mu.Lock() p.mu.Lock()
same := p.hasJudged && p.judged.Effective == c.Effective && p.judged.Configured == c.Configured && p.judged.ByCPU == c.ByCPU && same := p.hasJudged && p.judged.Effective == c.Effective && p.judged.Configured == c.Configured && p.judged.ByCPU == c.ByCPU &&
p.judged.CoresPerStep == c.CoresPerStep && p.judged.QuotaCores == c.QuotaCores && p.judged.OutsideHours == c.OutsideHours && p.judged.CoresPerStep == c.CoresPerStep && p.judged.QuotaCores == c.QuotaCores && p.judged.OutsideHours == c.OutsideHours &&
p.judged.BuildHours == c.BuildHours && p.judged.BuildHoursFault == c.BuildHoursFault p.judged.BuildHours == c.BuildHours && p.judged.BuildHoursFault == c.BuildHoursFault &&
p.judged.Ineligible == c.Ineligible
if !same { if !same {
p.judged, p.hasJudged, p.capWritten = c, true, false p.judged, p.hasJudged, p.capWritten = c, true, false
} }
@@ -371,8 +424,8 @@ func (p *Puller) Pull(ctx context.Context) error {
// node kept from building is held by its capacity (0), not by a pause. // node kept from building is held by its capacity (0), not by a pause.
p.logf("the pause of %s cannot be read: %v; pulling as not paused", p.Node, err) p.logf("the pause of %s cannot be read: %v; pulling as not paused", p.Node, err)
} }
go p.watchCapacity() // WatchConfigServed waits for Serve, which a module starts after its pull p.follow() // WatchConfigServed waits for Serve, which a module starts after its pull
go p.watchHours() go p.watchSettings(ctx)
wait := p.PullWait wait := p.PullWait
if wait <= 0 { if wait <= 0 {
wait = DefaultPullWait wait = DefaultPullWait
@@ -399,6 +452,7 @@ func (p *Puller) Pull(ctx context.Context) error {
case <-edge: case <-edge:
} }
stop() stop()
p.follow() // a follow that failed is tried again at the next event
_, capacity, inHand = p.roomNow() _, capacity, inHand = p.roomNow()
p.logf("the %s puller on %s wakes: %d of capacity %d in hand, paused %v", p.Language, p.Node, inHand, capacity, p.Paused()) p.logf("the %s puller on %s wakes: %d of capacity %d in hand, paused %v", p.Language, p.Node, inHand, capacity, p.Paused())
continue continue
@@ -417,6 +471,7 @@ func (p *Puller) Pull(ctx context.Context) error {
case <-ctx.Done(): case <-ctx.Done():
return ctx.Err() return ctx.Err()
case <-p.wake: case <-p.wake:
p.follow()
case <-time.After(wait): case <-time.After(wait):
} }
continue continue
@@ -595,7 +650,7 @@ func (p *Puller) run(ctx context.Context, msg *Pulled, ask delivery.StepAsk, key
} }
} }
// watchCapacity follows the module's config value `capacity` when WatchConfig is set (issue 1887, 2026-10-25: g14's // watchCapacity follows the module's config value `capacity` when WatchConfig is set (issue 1887, 2026-10-25: the laptop's
// puller started at capacity 0 sat parked with capacity 2 set, because nothing woke it to read its settings file // puller started at capacity 0 sat parked with capacity 2 set, because nothing woke it to read its settings file
// again). The value comes as mesh-config resolved it for this node, the event of the change itself: the runtime hands // again). The value comes as mesh-config resolved it for this node, the event of the change itself: the runtime hands
// the value now and then each change its machine's provisioner client says moved, so nothing polls, and the puller does // the value now and then each change its machine's provisioner client says moved, so nothing polls, and the puller does
@@ -605,11 +660,18 @@ func (p *Puller) watchCapacity() {
if p.WatchConfig == nil { if p.WatchConfig == nil {
return return
} }
p.mu.Lock()
if p.capFollowed || p.capTrying {
p.mu.Unlock()
return
}
p.capTrying = true
p.mu.Unlock()
err := p.WatchConfig(CapacitySetting, func(c stdio.ProvisionChange) error { err := p.WatchConfig(CapacitySetting, func(c stdio.ProvisionChange) error {
n, why := p.capacityFrom(c) n, why := p.capacityFrom(c)
p.mu.Lock() p.mu.Lock()
before, had := p.watched, p.hasWatched before, had := p.watched, p.hasWatched
p.watched, p.hasWatched = n, true p.watched, p.hasWatched, p.capWhy = n, true, why
p.mu.Unlock() p.mu.Unlock()
switch { switch {
case why != "": case why != "":
@@ -620,9 +682,31 @@ func (p *Puller) watchCapacity() {
p.Wake() p.Wake()
return nil return nil
}) })
p.followed(&p.capFollowed, &p.capTrying, err, "build capacity")
}
// follow starts each config follow that is not running: at the start, and at every wake after (a file event, a resume, a
// step's end), so a follow that failed is tried again by the next event and not given up for good.
func (p *Puller) follow() {
go p.watchCapacity()
go p.watchHours()
}
// followed records the end of one follow's start. A failure is said once per distinct error, with that it is tried again.
func (p *Puller) followed(done, trying *bool, err error, what string) {
p.mu.Lock()
*trying = false
*done = err == nil
said := err != nil && err.Error() == p.followFault
if err != nil { if err != nil {
p.logf("the %s build capacity of %s cannot be followed on the bus, so it is read from the settings file when the "+ p.followFault = err.Error()
"puller wakes: %v", p.Language, p.Node, err) } else {
p.followFault = ""
}
p.mu.Unlock()
if err != nil && !said {
p.logf("the %s %s of %s cannot be followed on the bus, so it is read from the settings file; the follow is tried "+
"again at the next event: %v", p.Language, what, p.Node, err)
} }
} }
@@ -633,6 +717,13 @@ func (p *Puller) watchHours() {
if p.WatchConfig == nil || p.BuildHours == nil { if p.WatchConfig == nil || p.BuildHours == nil {
return return
} }
p.mu.Lock()
if p.hoursFollowed || p.hoursTrying {
p.mu.Unlock()
return
}
p.hoursTrying = true
p.mu.Unlock()
err := p.WatchConfig(BuildHoursSetting, func(c stdio.ProvisionChange) error { err := p.WatchConfig(BuildHoursSetting, func(c stdio.ProvisionChange) error {
setting := "" setting := ""
var w *BuildWindow var w *BuildWindow
@@ -644,7 +735,7 @@ func (p *Puller) watchHours() {
why = fmt.Sprintf("the setting %s %s is not a string", BuildHoursSetting, c.Value) why = fmt.Sprintf("the setting %s %s is not a string", BuildHoursSetting, c.Value)
default: default:
var err error var err error
if w, err = ParseBuildHours(setting, p.Node); err != nil { if w, err = ParseBuildWindow(setting); err != nil {
why = err.Error() why = err.Error()
} }
} }
@@ -657,9 +748,7 @@ func (p *Puller) watchHours() {
p.Wake() p.Wake()
return nil return nil
}) })
if err != nil { p.followed(&p.hoursFollowed, &p.hoursTrying, err, "build hours")
p.logf("the %s build hours of %s cannot be followed on the bus, so they are read from the settings file: %v", p.Language, p.Node, err)
}
} }
// WatchConfigServed is stdio.WatchConfig once Serve has the runtime's channel (stdio.Served): a build module starts its // WatchConfigServed is stdio.WatchConfig once Serve has the runtime's channel (stdio.Served): a build module starts its
@@ -674,13 +763,7 @@ func (p *Puller) capacityFrom(c stdio.ProvisionChange) (int, string) {
if c.Refused != "" { if c.Refused != "" {
return 0, "its capacity is no longer answered: " + c.Refused return 0, "its capacity is no longer answered: " + c.Refused
} }
setting := "" n, err := capacityOf(c.Value)
if len(c.Value) > 0 && string(c.Value) != "null" {
if err := json.Unmarshal(c.Value, &setting); err != nil {
return 0, fmt.Sprintf("its capacity %s is not a string", c.Value)
}
}
n, err := capacityOf(setting, p.Node)
if err != nil { if err != nil {
return 0, err.Error() return 0, err.Error()
} }
+2 -2
View File
@@ -22,7 +22,7 @@ func (w *configWatch) watch(key string, handle func(stdio.ProvisionChange) error
w.asked, w.handle = key, handle w.asked, w.handle = key, handle
value := w.value value := w.value
w.mu.Unlock() w.mu.Unlock()
return handle(stdio.ProvisionChange{Kind: "config", Key: key, Value: json.RawMessage(`"` + value + `"`)}) return handle(stdio.ProvisionChange{Kind: "config", Key: key, Value: json.RawMessage(value)})
} }
func (w *configWatch) set(t *testing.T, value string) { func (w *configWatch) set(t *testing.T, value string) {
@@ -33,7 +33,7 @@ func (w *configWatch) set(t *testing.T, value string) {
if handle == nil { if handle == nil {
t.Fatal("the puller watches no config value") t.Fatal("the puller watches no config value")
} }
if err := handle(stdio.ProvisionChange{Kind: "config", Key: key, Value: json.RawMessage(`"` + value + `"`)}); err != nil { if err := handle(stdio.ProvisionChange{Kind: "config", Key: key, Value: json.RawMessage(value)}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
} }
+69 -2
View File
@@ -5,6 +5,8 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"os"
"path/filepath"
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
@@ -282,7 +284,7 @@ func TestPullAtCapacityZeroTakesNothing(t *testing.T) {
t.Run("config watch wakes", func(t *testing.T) { t.Run("config watch wakes", func(t *testing.T) {
src, st := NewFakeAsks(), newSteps() src, st := NewFakeAsks(), newSteps()
_ = src.PublishAsk(testAsk(delivery.AskDelivery)) _ = src.PublishAsk(testAsk(delivery.AskDelivery))
w := &configWatch{value: "shanks=3,g14=0"} w := &configWatch{value: "0"}
p := &Puller{Source: src, Language: delivery.AskGo, Node: "g14", Put: st.put, PullWait: 10 * time.Millisecond, p := &Puller{Source: src, Language: delivery.AskGo, Node: "g14", Put: st.put, PullWait: 10 * time.Millisecond,
Capacity: func() int { return 0 }, // the settings file, not refilled yet Capacity: func() int { return 0 }, // the settings file, not refilled yet
WatchConfig: w.watch, WatchConfig: w.watch,
@@ -298,9 +300,74 @@ func TestPullAtCapacityZeroTakesNothing(t *testing.T) {
if w.asked != CapacitySetting { if w.asked != CapacitySetting {
t.Fatalf("the puller watches %q, want %q", w.asked, CapacitySetting) t.Fatalf("the puller watches %q, want %q", w.asked, CapacitySetting)
} }
w.set(t, "shanks=3,g14=2") w.set(t, "2")
waitFor(t, "the ask taken once the capacity moved to 2", func() bool { a, _, _ := src.Fates(); return len(a) == 1 }) waitFor(t, "the ask taken once the capacity moved to 2", func() bool { a, _, _ := src.Fates(); return len(a) == 1 })
}) })
t.Run("settings refill wakes a parked puller and retries a failed follow", func(t *testing.T) {
// 2026-10-28: a replaced builder held the old "shanks=2,..." in its settings file, its config follow timed out, and
// the puller parked at 0 with nothing to wake it when the node-engine refilled the file.
src, st := NewFakeAsks(), newSteps()
_ = src.PublishAsk(testAsk(delivery.AskDelivery))
file := filepath.Join(t.TempDir(), "settings.json")
if err := os.WriteFile(file, []byte(`{"capacity":"shanks=2,ace=2"}`), 0o600); err != nil {
t.Fatal(err)
}
var mu sync.Mutex
attempts := 0
watch := func(key string, handle func(stdio.ProvisionChange) error) error {
mu.Lock()
defer mu.Unlock()
attempts++
if attempts == 1 {
return errors.New("nats: timeout")
}
return nil // followed; no value handed yet
}
armed := make(chan struct{}, 8)
var parked atomic.Int32
parks := func() int32 { return parked.Load() }
logf := func(format string, args ...any) {
if strings.Contains(format, "parks") {
parked.Add(1)
}
}
p := &Puller{Source: src, Language: delivery.AskGo, Node: "shanks", Put: st.put, PullWait: 10 * time.Millisecond,
SettingsFile: file, WatchConfig: watch, Logf: logf, watching: func() { armed <- struct{}{} },
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return nil, nil }}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go p.Pull(ctx)
waitFor(t, "the parked puller to say why", func() bool { why, fault := p.CapacityNote(); return fault && why != "" })
if src.Waiting(delivery.AskDelivery) != 1 {
t.Fatal("a node whose capacity cannot be read took an ask")
}
<-armed // the file watch is in place
tmp := file + ".new"
if err := os.WriteFile(tmp, []byte(`{"capacity":2}`), 0o600); err != nil {
t.Fatal(err)
}
if err := os.Rename(tmp, file); err != nil {
t.Fatal(err)
}
waitFor(t, "the ask taken once the file was refilled", func() bool { a, _, _ := src.Fates(); return len(a) == 1 })
// and the directory itself replaced (IN_IGNORED): the watch is armed again and a later refill still wakes the puller
dir := filepath.Dir(file)
before := parks()
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(dir, 0o700); err != nil {
t.Fatal(err)
}
<-armed // armed again
waitFor(t, "the puller parked with no settings file", func() bool { return parks() > before })
_ = src.PublishAsk(testAsk(delivery.AskDelivery))
if err := os.WriteFile(file, []byte(`{"capacity":2}`), 0o600); err != nil {
t.Fatal(err)
}
waitFor(t, "the second ask taken after the directory was replaced", func() bool { a, _, _ := src.Fates(); return len(a) == 2 })
waitFor(t, "the failed follow tried again", func() bool { mu.Lock(); defer mu.Unlock(); return attempts >= 2 })
})
for _, c := range []struct { for _, c := range []struct {
name string name string
watch func(string, func(stdio.ProvisionChange) error) error watch func(string, func(stdio.ProvisionChange) error) error
+1 -1
View File
@@ -149,7 +149,7 @@ func endRunning(scan Scan, key string, step delivery.Step, revision uint64, why
// asker takes the first end it reads, so an end written here would fail a check the redelivery would still run. Its // asker takes the first end it reads, so an end written here would fail a check the redelivery would still run. Its
// asker's watch takes puts only and keeps waiting for the redelivery's key, its bound the backstop; the stale running // asker's watch takes puts only and keeps waiting for the redelivery's key, its bound the backstop; the stale running
// key leaves the steps state, and with it the delivery view's and the checker's count of what runs (note 21: it // key leaves the steps state, and with it the delivery view's and the checker's count of what runs (note 21: it
// held one of shanks' two slots). // held one of a build node's two slots).
// - A step owned by this process is left: it may run now. A redelivered ask begun on this node is owned by it. // - A step owned by this process is left: it may run now. A redelivered ask begun on this node is owned by it.
// - A step with no owner (written by an older build) keeps the earlier rule: a hand-started step started before // - A step with no owner (written by an older build) keeps the earlier rule: a hand-started step started before
// `before` (the start of this process) is ended; a pulled ask is left to the stream's redelivery (isAsk) until its // `before` (the start of this process) is ended; a pulled ask is left to the stream's redelivery (isAsk) until its
+112
View File
@@ -0,0 +1,112 @@
//go:build linux
package buildstep
import (
"context"
"encoding/binary"
"os"
"path/filepath"
"syscall"
)
const settingsMask = syscall.IN_CLOSE_WRITE | syscall.IN_MOVED_TO | syscall.IN_CREATE
// watchSettings wakes the puller each time its settings file is written or replaced (inotify on the file's directory, so
// a rename over it counts): the event the node-engine's refill of the file is, which a parked puller at capacity 0
// otherwise never sees. No timer: it blocks on the kernel's event. When the kernel drops the watch (IN_IGNORED: the
// directory was replaced or removed) it is added again, said once; while the directory is missing the nearest existing
// ancestor is watched for the directory's return. A watch that cannot start is said once.
func (p *Puller) watchSettings(ctx context.Context) {
if p.SettingsFile == "" {
return
}
fd, err := syscall.InotifyInit1(syscall.IN_CLOEXEC | syscall.IN_NONBLOCK)
if err != nil {
p.logf("the settings file %s cannot be watched, so a refill of it wakes the puller only at another event: %v", p.SettingsFile, err)
return
}
f := os.NewFile(uintptr(fd), "inotify")
defer f.Close()
dir, base := filepath.Split(p.SettingsFile)
dir = filepath.Clean(dir)
// arm watches dir, or when it is missing its nearest existing ancestor; it returns the watch descriptor and whether it
// is the directory's own.
arm := func() (int, bool, error) {
wd, err := syscall.InotifyAddWatch(fd, dir, settingsMask)
if err == nil {
return wd, true, nil
}
for up := filepath.Dir(dir); ; up = filepath.Dir(up) {
wd, aerr := syscall.InotifyAddWatch(fd, up, settingsMask)
if aerr == nil {
// the directory may have come back between the two calls: look once more, now that the ancestor is watched
if own, oerr := syscall.InotifyAddWatch(fd, dir, settingsMask); oerr == nil {
return own, true, nil
}
return wd, false, nil
}
if up == filepath.Dir(up) {
return 0, false, err
}
}
}
wd, own, err := arm()
if err != nil {
p.logf("the settings file %s cannot be watched, so a refill of it wakes the puller only at another event: %v", p.SettingsFile, err)
return
}
if p.watching != nil {
p.watching()
}
go func() { <-ctx.Done(); f.Close() }()
buf := make([]byte, 4096)
for {
n, err := f.Read(buf)
if err != nil {
return
}
hit, rearm := false, false
for off := 0; off+16 <= n; {
ewd := int(int32(binary.LittleEndian.Uint32(buf[off:])))
mask := binary.LittleEndian.Uint32(buf[off+4:])
l := int(binary.LittleEndian.Uint32(buf[off+12:]))
name := buf[off+16 : off+16+l]
for i, c := range name {
if c == 0 {
name = name[:i]
break
}
}
switch {
case mask&syscall.IN_IGNORED != 0 && ewd == wd:
rearm = true
case ewd == wd && own && string(name) == base:
hit = true
case ewd == wd && !own:
rearm = true // something appeared where the directory was: look again
}
off += 16 + l
}
if rearm {
nwd, nown, err := arm()
if err != nil {
p.logf("the settings file %s is no longer watched, so a refill of it wakes the puller only at another event: %v", p.SettingsFile, err)
return
}
if own || nown { // said once per loss of the watch on the directory itself
if own {
p.logf("the watch of the settings file %s was dropped (its directory was replaced); watching again", p.SettingsFile)
}
}
wd, own = nwd, nown
if p.watching != nil {
p.watching()
}
hit = true // the file may have been refilled meanwhile
}
if hit {
p.Wake()
}
}
}
+8
View File
@@ -0,0 +1,8 @@
//go:build !linux
package buildstep
import "context"
// watchSettings watches nothing off Linux: the puller wakes at its other events.
func (p *Puller) watchSettings(ctx context.Context) {}
+244
View File
@@ -0,0 +1,244 @@
package buscompose
import (
"encoding/json"
"fmt"
"slices"
"strings"
"time"
)
// **The accounts the grants part wants, and the one shape a held account takes** (novox/hq ADR 0561 §1, §2; issue 774).
//
// The bus composer derives every user from placement, the manifests and the wants. For each account the mesh mints (a
// module's, a module's process's, a machine's provisioner client's), the grants part writes one key of its state
// `accounts`, keyed by the bus user, holding the account's fields that are not secret: what the controller's credential
// carries today beside the password. The vault watches that state, mints the user it holds no value for, and keeps the
// value as the module's own secret on that node, built by HeldCredential, which gives byte for byte the controller's
// shape, so a runtime reads it unchanged. The state is read by the vault alone (NamedReaders), as `bus-accounts` is read
// by the grants part alone.
//
// The node accounts, the controller's own, an enrolment user and a person are never wanted here: the vault mints none
// of them (ADR 0561 §4, §5).
// AccountsState is the grants part's state of wanted accounts, by bus user.
const AccountsState = "accounts"
// GrantsAccounts is that state as a reader names it (`<owner>.<name>`).
const GrantsAccounts = GrantsModule + "." + AccountsState
// AccountsBucket is the bucket the state lives in.
var AccountsBucket = BucketName(GrantsModule, AccountsState)
// AccountsSubject is the subject of one wanted account's key: `$KV.mesh-grants_accounts.<user>`.
func AccountsSubject(user string) string { return "$KV." + AccountsBucket + "." + user }
// AccountIssuedEvent is what the vault says when it minted an account: AccountIssued, never a value or a hash.
const AccountIssuedEvent = "account-issued"
// AccountIssuedSubject is where it says it, as the runtime addresses mesh-vault's events.
const AccountIssuedSubject = "mesh.mod." + VaultSeat + ".event." + AccountIssuedEvent
// ClaimedSeat is one seat a module claims, as its runtime needs it (the controller's seatClaimed): the name, the scope (a
// node-scoped seat's verb carries the machine) and the verbs the runtime serves for it.
type ClaimedSeat struct {
Seat string `json:"seat"`
Scope string `json:"scope"`
Serves []string `json:"serves,omitempty"`
}
// WantedAccount is one key of the grants part's state `accounts`: an account the mesh needs, without its password.
type WantedAccount struct {
User string `json:"user"`
// Kind is KindModule (KindNodeTools for the runtime, named as a module is), KindProcess or KindProvisioner.
Kind Kind `json:"kind"`
Node string `json:"node"`
Module string `json:"module"`
// Process is a KindProcess account's process id.
Process string `json:"process,omitempty"`
// URL is the bus's address as that node reaches it, `nats://<host>:<port>`; Fingerprint the bus certificate's.
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
// Claims are a module's claims with the verbs each seat promises: none for a process or a provisioner client.
Claims []ClaimedSeat `json:"claims,omitempty"`
// At is when the grants part wrote the key: not compared (Same).
At time.Time `json:"at"`
}
// mintedKinds are the kinds of account the vault mints (ADR 0561 §1).
var mintedKinds = []Kind{KindModule, KindNodeTools, KindProcess, KindProvisioner}
// principal is the account as the composer names it.
func (w WantedAccount) principal() Principal {
return Principal{Kind: w.Kind, Node: w.Node, Module: w.Module, Process: w.Process}
}
// Valid says why w is not an account the vault may mint, or nil: a kind it mints, every name a subject token, the user
// the one the composer derives from them, and an address to reach the bus at.
func (w WantedAccount) Valid() error {
if !slices.Contains(mintedKinds, w.Kind) {
return fmt.Errorf("%s is a %s account, which the vault never mints (ADR 0561 §4)", w.User, w.Kind)
}
for what, v := range map[string]string{"node": w.Node, "module": w.Module} {
if !safeSubject.MatchString(v) {
return fmt.Errorf("%s: the %s %q is not a name the mesh writes", w.User, what, v)
}
}
if w.Kind == KindProcess && !safeSubject.MatchString(w.Process) {
return fmt.Errorf("%s: the process %q is not a name the mesh writes", w.User, w.Process)
}
if w.Kind != KindProcess && w.Process != "" {
return fmt.Errorf("%s: only a process's account names a process", w.User)
}
if want := w.principal().Username(); w.User != want {
return fmt.Errorf("%s is not the user of a %s account of %s on %s (that is %s)", w.User, w.Kind, w.Module, w.Node, want)
}
if !strings.HasPrefix(w.URL, "nats://") || len(w.URL) == len("nats://") {
return fmt.Errorf("%s names no bus address (url, nats://<host>:<port>)", w.User)
}
if len(w.Claims) > 0 && w.Kind != KindModule && w.Kind != KindNodeTools {
return fmt.Errorf("%s: a %s account holds no seat, so it carries no claims", w.User, w.Kind)
}
return nil
}
// Same says whether two wanted accounts want the same account: every field but when it was written.
func (w WantedAccount) Same(o WantedAccount) bool {
a, b := w, o
a.At, b.At = time.Time{}, time.Time{}
ra, _ := json.Marshal(a)
rb, _ := json.Marshal(b)
return string(ra) == string(rb)
}
// OwnSecret is the module's own secret the account is held in: `broker`, `broker-<process id>` or `provisioner-broker`
// (the controller's names, ADR 0436 §2, ADR 0349 §11).
func (w WantedAccount) OwnSecret() string {
switch w.Kind {
case KindProcess:
return "broker-" + w.Process
case KindProvisioner:
return "provisioner-broker"
}
return "broker"
}
// ReadWantedAccount reads the key user of the state: refused when it is not a valid wanted account of that user.
func ReadWantedAccount(user string, raw []byte) (WantedAccount, error) {
var w WantedAccount
if err := json.Unmarshal(raw, &w); err != nil {
return WantedAccount{}, fmt.Errorf("%s %s is not a wanted account: %w", GrantsAccounts, user, err)
}
if w.User != user {
return WantedAccount{}, fmt.Errorf("%s %s names the user %q", GrantsAccounts, user, w.User)
}
return w, w.Valid()
}
// AccountIssued is the vault's account-issued: which account it minted, and its generation. Never a value or a hash.
type AccountIssued struct {
User string `json:"user"`
Kind Kind `json:"kind"`
Node string `json:"node"`
Module string `json:"module"`
Process string `json:"process,omitempty"`
Generation int `json:"generation"`
At time.Time `json:"at"`
}
// HeldCredential is the value the account's holder reads, exactly as the controller builds it today (issueWith,
// sealProcess and sealProvisionerClient): a module's `{url, fingerprint, node, module, user, password, claims}`, a
// process's without claims, a provisioner client's without module or claims; an empty fingerprint or claims left out.
func HeldCredential(w WantedAccount, password string) ([]byte, error) {
if err := w.Valid(); err != nil {
return nil, err
}
if password == "" {
return nil, fmt.Errorf("%s: an account is held with its password", w.User)
}
switch w.Kind {
case KindProcess:
return json.Marshal(struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
}{URL: w.URL, Fingerprint: w.Fingerprint, Node: w.Node, Module: w.Module, User: w.User, Password: password})
case KindProvisioner:
return json.Marshal(struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node"`
User string `json:"user"`
Password string `json:"password"`
}{URL: w.URL, Fingerprint: w.Fingerprint, Node: w.Node, User: w.User, Password: password})
}
return json.Marshal(struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
Claims []ClaimedSeat `json:"claims,omitempty"`
}{URL: w.URL, Fingerprint: w.Fingerprint, Node: w.Node, Module: w.Module, User: w.User, Password: password,
Claims: w.Claims})
}
// **One issuer at every moment: minting turns on at the lease write** (novox/hq ADR 0561 §3, §6). The controller's lease
// is the one key `holder` of its bucket (LeaseBucket). The operator's credential hands it away by writing a value that
// names its new holder in `handed-to`; while the key holds that, the vault mints and the grants part creates the durable
// consumers an account needs. A controller's value, a key deleted or expired, or one that cannot be read or parsed, turn
// both off: when in doubt nothing is minted, so the controller and the vault never both mint one account. No clock and
// no setting turns it.
// ControllerLease is the lease's bucket as a reader names it; the vault and the grants part alone read it
// (NamedReaders), the controller's announcement marking it readable by nobody.
const ControllerLease = ControllerSeat + ".lease"
// ControllerLeaseKey is the lease's one key.
const ControllerLeaseKey = leaseKey
// LeaseHolder is the lease key's value: a controller instance's (mesh-controller internal/lease Holder), or, once handed
// away, one naming its holder in HandedTo.
type LeaseHolder struct {
Instance string `json:"instance,omitempty"`
Host string `json:"host,omitempty"`
Epoch uint64 `json:"epoch,omitempty"`
Taken time.Time `json:"taken,omitzero"`
Renewed time.Time `json:"renewed,omitzero"`
// HandedTo is who holds the lease once the controller handed it away: never a controller instance.
HandedTo string `json:"handed-to,omitempty"`
// HandedBy is the credential that wrote it (the operator's), and Why its reason.
HandedBy string `json:"handed-by,omitempty"`
Why string `json:"why,omitempty"`
}
// LeaseHandedAway reads the lease key as a getter answered it (its value, whether it was there, the read's error) and
// says whether it is handed to a holder that is no controller, with why either way.
func LeaseHandedAway(raw []byte, found bool, readErr error) (bool, string) {
switch {
case readErr != nil:
return false, "the controller's lease cannot be read (" + readErr.Error() + "), so nothing is minted"
case !found || len(raw) == 0 || string(raw) == "null":
return false, "no lease is written (expired, deleted, or never taken): a controller may take it, so nothing is minted"
}
var h LeaseHolder
if err := json.Unmarshal(raw, &h); err != nil {
return false, "the controller's lease is not readable as a holder (" + err.Error() + "), so nothing is minted"
}
if strings.TrimSpace(h.HandedTo) == "" {
who := h.Instance
if who == "" {
who = "an instance that names itself not"
}
return false, "the controller holds the lease (" + who + "): it mints, so the vault mints nothing (ADR 0561 §3)"
}
if h.Instance != "" {
return false, "the lease names both a controller instance (" + h.Instance + ") and a holder it is handed to (" +
h.HandedTo + "), so nothing is minted"
}
return true, "the lease is handed to " + h.HandedTo + ": the vault mints every new account (ADR 0561 §1, §3)"
}
+10
View File
@@ -1,6 +1,7 @@
package buscompose package buscompose
import ( import (
"slices"
"sort" "sort"
"strings" "strings"
@@ -58,6 +59,10 @@ func ReadsAnnouncedBy(reader string, reads []string, announced map[string]announ
switch { switch {
case reader != "" && !ReaderMay(reader, r): case reader != "" && !ReaderMay(reader, r):
refused = append(refused, r) refused = append(refused, r)
case reader != "" && namedReader(reader, r):
// A state the composer names its readers for is theirs whatever its owner announced (ADR 0561 §3: the
// controller's lease, which its announcement marks readable by nobody).
granted = append(granted, r)
case reader != "" && ok && said && s.Keeps(name): case reader != "" && ok && said && s.Keeps(name):
if s.ReadableBy(name, reader) { if s.ReadableBy(name, reader) {
granted = append(granted, r) granted = append(granted, r)
@@ -95,3 +100,8 @@ func NotAnnounced(r Records) []string {
sort.Strings(out) sort.Strings(out)
return out return out
} }
// namedReader says the composer names module among the readers of read (NamedReaders).
func namedReader(module, read string) bool {
return slices.Contains(NamedReaders[strings.TrimSpace(read)], module)
}
+279 -2
View File
@@ -2,6 +2,7 @@ package buscompose
import ( import (
"encoding/json" "encoding/json"
"errors"
"os" "os"
"path/filepath" "path/filepath"
"reflect" "reflect"
@@ -9,6 +10,11 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
) )
// Defends issue 1377 and novox/hq ADR 0547 §5: this is the controller's bus composer copied, and it composes byte for // Defends issue 1377 and novox/hq ADR 0547 §5: this is the controller's bus composer copied, and it composes byte for
@@ -33,7 +39,7 @@ func beyondTheControllers(user string) map[string]bool {
if !ok { if !ok {
return out return out
} }
for _, s := range append(readGrants("node-engine_have", "*.node"), NodeComposeReads(node)...) { for _, s := range append(append(readGrants("node-engine_have", "*.node"), NodeComposeReads(node)...), NodeComposeAdvisories()...) {
out[s] = true out[s] = true
} }
return out return out
@@ -49,6 +55,26 @@ func mayBeyondTheControllers(user string) map[string]bool {
VaultSecretChanged: true} VaultSecretChanged: true}
} }
// engineBeyond is a node's engine holding its node seat and asking the operator in its own name (novox/hq ADR 0563 §1,
// §3), beyond the controller's on purpose: its own node's verbs, events and discovery, the bus's discovery asked, and
// the router's seat traffic under node-engine-<node>.
func engineBeyond(user, g string) bool {
node, ok := strings.CutPrefix(user, "node.")
if !ok {
return false
}
return strings.HasPrefix(g, "mesh.seat."+EngineSeat+".") && strings.Contains(g, "."+node) ||
strings.HasPrefix(g, "$SRV.") || strings.Contains(g, ".node-engine-"+node) ||
strings.Contains(g, "KV_"+EngineActsBucket) || strings.HasPrefix(g, "$KV."+EngineActsBucket+"."+node+".")
}
// createsAnAccountsConsumer is the grants part's create and read of a durable consumer an account needs, by its stream
// and name (novox/hq ADR 0561 §6): beyond the controller's on purpose, since the controller made them alone.
func createsAnAccountsConsumer(user, g string) bool {
return strings.HasSuffix(user, "."+GrantsModule) &&
(strings.HasPrefix(g, "$JS.API.CONSUMER.CREATE.") || strings.HasPrefix(g, "$JS.API.CONSUMER.INFO."))
}
// quotedToken is a quoted token of a composed list. // quotedToken is a quoted token of a composed list.
var quotedToken = regexp.MustCompile(`"[^"]*"`) var quotedToken = regexp.MustCompile(`"[^"]*"`)
@@ -100,7 +126,8 @@ func grantsAsTheControllers(t *testing.T, got, want string) {
} }
} }
for g := range now { for g := range now {
if !was[g] && !intended[g] && !mayBeyondTheControllers(user)[g] && !createdAdvisory.MatchString(`"`+g+`", `) { if !was[g] && !intended[g] && !mayBeyondTheControllers(user)[g] && !createdAdvisory.MatchString(`"`+g+`", `) &&
!createsAnAccountsConsumer(user, g) && !engineBeyond(user, g) {
t.Errorf("%s: the copy grants %s beyond the controller's, and no decision names it", user, g) t.Errorf("%s: the copy grants %s beyond the controller's, and no decision names it", user, g)
} }
} }
@@ -117,7 +144,23 @@ func grantsAsTheControllers(t *testing.T, got, want string) {
} }
} }
const SignersBucketName = "mesh-vault_signers"
func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) { func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
// ADR 0567 §3, §9: a node reads every key of the seat's placement and its candidates (it composes with the whole
// proposal laid over), and none of its open proposals; it writes neither.
reads := strings.Join(NodeComposeReads("laptop"), " ")
for bucket, granted := range map[string]bool{delivery.PlacementBucket: true, delivery.PlacementCandidatesBucket: true,
delivery.PlacementProposalsBucket: false} {
if strings.Contains(reads, "$JS.API.DIRECT.GET.KV_"+bucket+".>") != granted || strings.Contains(reads, "$KV."+bucket+".laptop") {
t.Errorf("a node is granted %s: %v, want %v", bucket, !granted, granted)
}
}
// It hears the creation of those two buckets, and of no other.
if got := NodeComposeAdvisories(); len(got) != 2 || got[0] != "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_mesh-delivery-assigner_placement" ||
got[1] != "$JS.EVENT.ADVISORY.STREAM.CREATED.KV_mesh-delivery-assigner_candidates" {
t.Errorf("a node hears the creation of %v", got)
}
raw, err := os.ReadFile(filepath.Join("testdata", "controller-user-lists.json")) raw, err := os.ReadFile(filepath.Join("testdata", "controller-user-lists.json"))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -205,6 +248,55 @@ func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
t.Error("a user of a seat not pulled by kind reads DEAD_LETTERS") t.Error("a user of a seat not pulled by kind reads DEAD_LETTERS")
} }
}) })
// ADR 0563 §1, §3, §6: each node's engine alone answers its own node's node-engine verbs and says its events, any
// runtime may call them, and it asks the operator under node-engine-<node> alone.
t.Run("a node's engine holds its node seat and asks in its own name", func(t *testing.T) {
router := Seat{Name: AsksSeat, Accepts: []string{"ask", "cancel"}, Emits: []string{"decided"},
ByCaller: []string{"ask", "cancel", "decided"}, Records: []string{"messenger_asks"}}
users, err := Users(Records{Nodes: []string{"one", "two"}, Assigned: map[string][]Declared{
"one": {{Module: "messenger", Holds: []Seat{router}, Invokes: []string{"seat:" + EngineSeat + ".root-free"}}},
"two": {{Module: RuntimeModule}, {Module: "shop", Invokes: []string{"*"}}}}})
if err != nil {
t.Fatal(err)
}
for _, u := range users {
perms, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
answers := MaySubscribe(perms, "mesh.seat."+EngineSeat+".tool.root-free.one") ||
MaySubscribe(perms, "mesh.seat."+EngineSeat+".call.root-free.one.messenger~one")
if engineOne := u.Kind == KindNode && u.Node == "one"; answers != engineOne {
t.Errorf("%s answers one's root-free: %v", u.Username(), answers)
}
switch {
case u.Kind == KindNode && u.Node == "one":
for subject, want := range map[string]bool{
"mesh.seat.operator-channel.accept.ask.node-engine-one": true,
"mesh.seat.operator-channel.accept.cancel.node-engine-one": true,
"mesh.seat.operator-channel.accept.ask.node-engine-two": false,
"$JS.API.DIRECT.GET.KV_messenger_asks.$KV.messenger_asks.node-engine-one.hand-over-1": true,
"$JS.API.DIRECT.GET.KV_messenger_asks.$KV.messenger_asks.node-engine-two.hand-over-1": false,
"mesh.seat." + EngineSeat + ".event.done.one": true,
"mesh.seat." + EngineSeat + ".event.done.two": false,
"$SRV.INFO": true,
} {
if got := MayPublish(perms, subject); got != want {
t.Errorf("node one may publish %s: %v, want %v", subject, got, want)
}
}
if !MaySubscribe(perms, "mesh.seat.operator-channel.event.decided.node-engine-one") ||
MaySubscribe(perms, "mesh.seat.operator-channel.event.decided.node-engine-two") ||
!MaySubscribe(perms, "$SRV.INFO."+EngineSeat) || MaySubscribe(perms, "mesh.seat."+EngineSeat+".tool.hand-over.two") {
t.Errorf("node one hears %v", perms.Subscribe)
}
case u.Kind == KindNodeTools || u.Module == "messenger":
if !MayPublish(perms, "mesh.seat."+EngineSeat+".tool.root-free.one") {
t.Errorf("%s cannot call one's root-free", u.Username())
}
}
}
})
// Issue 1126 (ADR 0550 §5): a node's engine asks the vault's `own` in its own name while the vault serves it, and no // Issue 1126 (ADR 0550 §5): a node's engine asks the vault's `own` in its own name while the vault serves it, and no
// other node's. // other node's.
t.Run("a node asks the vault its own secrets", func(t *testing.T) { t.Run("a node asks the vault its own secrets", func(t *testing.T) {
@@ -223,6 +315,21 @@ func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
if !slices.Contains(perms.Subscribe, VaultSecretChanged) { if !slices.Contains(perms.Subscribe, VaultSecretChanged) {
t.Errorf("node one does not hear the vault's secret-changed: %v", perms.Subscribe) t.Errorf("node one does not hear the vault's secret-changed: %v", perms.Subscribe)
} }
// L3 s6 (ADR 0550 §2): while the vault serves `sign`, a node asks it in its own name alone, and reads the vault's
// `signers` and the lease key, never writing either.
signs, _ := PermissionsFor(Principal{Kind: KindNode, Node: "one", AsksTheVaultToSign: true})
signersGet := "$JS.API.DIRECT.GET.KV_" + SignersBucketName + ".$KV." + SignersBucketName + ".key"
leaseGet := "$JS.API.DIRECT.GET.KV_" + LeaseBucket + ".$KV." + LeaseBucket + ".holder"
for subject, want := range map[string]bool{VaultSignCall(one): true, signersGet: true, leaseGet: true,
VaultSignCall(Principal{Kind: KindNode, Node: "two"}): false, "$KV." + SignersBucketName + ".key": false,
"$KV." + LeaseBucket + ".holder": false} {
if MayPublish(signs, subject) != want {
t.Errorf("a node asking sign may publish %s: %v, want %v", subject, !want, want)
}
}
if MayPublish(perms, VaultSignCall(one)) || MayPublish(perms, signersGet) {
t.Error("a node asks sign or reads signers while the vault does not serve sign")
}
// And what a module on another node serves from its settings, that key alone (mailu's smtp domain, issue 1126). // And what a module on another node serves from its settings, that key alone (mailu's smtp domain, issue 1126).
users, err := Users(Records{Nodes: []string{"one", "two"}, Assigned: map[string][]Declared{ users, err := Users(Records{Nodes: []string{"one", "two"}, Assigned: map[string][]Declared{
"two": {{Module: "mailer", Served: []string{"domain", "not.a.token"}}}}}) "two": {{Module: "mailer", Served: []string{"domain", "not.a.token"}}}}})
@@ -277,4 +384,174 @@ func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
} }
grantsAsTheControllers(t, got, string(want)) grantsAsTheControllers(t, got, string(want))
}) })
// 774/AC2 (novox/hq ADR 0561 §2): a wanted account round-trips through the grants part's state `accounts`, keyed by
// its user, at the subject `$KV.mesh-grants_accounts.<user>`, read by the vault alone; and the vault's account-issued
// names the account, never a value.
t.Run("a wanted account is kept by its user and read by the vault alone", func(t *testing.T) {
for _, w := range []WantedAccount{
{User: "ace.mesh-build-go", Kind: KindModule, Node: "ace", Module: "mesh-build-go", URL: "nats://novox.mesh:4222",
Fingerprint: "ab", Claims: []ClaimedSeat{{Seat: "node-build-go", Scope: "node", Serves: []string{"build"}}}},
{User: "novox.mesh-mail.send", Kind: KindProcess, Node: "novox", Module: "mesh-mail", Process: "send",
URL: "nats://novox.mesh:4222"},
{User: "provisioner.g14", Kind: KindProvisioner, Node: "g14", Module: "mesh-provisioner-client",
URL: "nats://novox.mesh:4222", Fingerprint: "ab"},
} {
if err := w.Valid(); err != nil {
t.Fatalf("%s: %v", w.User, err)
}
raw, err := json.Marshal(w)
if err != nil {
t.Fatal(err)
}
back, err := ReadWantedAccount(w.User, raw)
if err != nil || !back.Same(w) {
t.Errorf("%s does not round-trip: %+v, %v", w.User, back, err)
}
if got := AccountsSubject(w.User); got != "$KV.mesh-grants_accounts."+w.User {
t.Errorf("%s is kept at %s", w.User, got)
}
if _, err := ReadWantedAccount("someone.else", raw); err == nil {
t.Errorf("%s was read under another user's key", w.User)
}
}
for _, bad := range []WantedAccount{
{User: "ace.x", Kind: KindModule, Node: "ace", Module: "y", URL: "nats://a:1"},
{User: "ace.x.p", Kind: KindProcess, Node: "ace", Module: "x", URL: "nats://a:1"},
{User: "node.ace", Kind: KindNode, Node: "ace", URL: "nats://a:1"},
{User: "controller", Kind: KindController, URL: "nats://a:1"},
{User: "ace.x", Kind: KindModule, Node: "ace", Module: "x"},
} {
if bad.Valid() == nil {
t.Errorf("%+v is taken as a wanted account", bad)
}
}
if ReaderMay("mesh-delivery-assigner", GrantsAccounts) || !ReaderMay(VaultSeat, GrantsAccounts) {
t.Errorf("%s is read by %v alone, want the vault's", GrantsAccounts, NamedReaders[GrantsAccounts])
}
e := AccountIssued{User: "ace.x", Kind: KindModule, Node: "ace", Module: "x", Generation: 1}
raw, _ := json.Marshal(e)
if strings.Contains(string(raw), "password") || strings.Contains(string(raw), "hash") ||
AccountIssuedSubject != "mesh.mod.mesh-vault.event.account-issued" {
t.Errorf("account-issued says %s at %s", raw, AccountIssuedSubject)
}
})
// 774/AC3 (novox/hq ADR 0561 §2): the account the vault seals is, byte for byte, what the controller hands for each of
// its three kinds: testdata/controller-credentials.json was written by the controller's own code (its "how").
t.Run("a held account is the controller's shape byte for byte", func(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("testdata", "controller-credentials.json"))
if err != nil {
t.Fatal(err)
}
var captured struct {
Rows []struct{ Name, Held string }
}
if err := json.Unmarshal(raw, &captured); err != nil {
t.Fatal(err)
}
vault := ClaimedSeat{Seat: "mesh-vault", Scope: "mesh", Serves: []string{"get", "accept", "secret-ask", "issued",
"rotate", "move-in", "node-key", "own", "move-asks"}}
controller := ClaimedSeat{Seat: "mesh-controller", Scope: "mesh", Serves: strings.Split("tools,calls,status,nodes,"+
"node,placement-export,modules,seats,builds,plans,delivery-plan,delivery-order,delivery-check,deliver,"+
"delivery-stop,delivery-backups-again,delivery-walks,plan,assign,unassign,pin,unpin,push,rotate,give,issued,"+
"secret-move,secret-ask,issue,token,settings,command,queue,cancel,clear,replay,try,kill,pause,resume,hand-act,"+
"drill,warranted,hand-acts,durations,conditions,healers,doctor,upgrade,bus,retire,cleanup,judge-directories,"+
"root-free,dead-letters,data,build,artifacts,held,collect,move-placement,move-config,mirrors,images", ",")}
at := "nats://10.0.0.1:4222"
wanted := map[string]struct {
w WantedAccount
password string
}{
"module": {WantedAccount{User: "laptop.cap-bot", Kind: KindModule, Node: "laptop", Module: "cap-bot", URL: at,
Fingerprint: "ab:cd", Claims: []ClaimedSeat{vault, {Seat: "node-service-manager", Scope: "node",
Serves: []string{"status"}}, controller, {Seat: "no-such-seat", Scope: "node"}}}, "pw-module"},
"module-no-claims-no-fingerprint": {WantedAccount{User: "laptop.cap-plain", Kind: KindModule, Node: "laptop",
Module: "cap-plain", URL: at}, "pw-plain"},
"provisioner": {WantedAccount{User: "provisioner.laptop", Kind: KindProvisioner, Node: "laptop", Module: "nats",
URL: at, Fingerprint: "ab:cd"}, "pw-prov"},
"process": {WantedAccount{User: "laptop.mesh-mail.send", Kind: KindProcess, Node: "laptop", Module: "mesh-mail",
Process: "send", URL: at, Fingerprint: "ab:cd"}, "pw-proc"},
}
if len(captured.Rows) != len(wanted) {
t.Fatalf("%d rows captured, %d wanted", len(captured.Rows), len(wanted))
}
for _, row := range captured.Rows {
c, ok := wanted[row.Name]
if !ok {
t.Fatalf("no wanted account for the captured %s", row.Name)
}
got, err := HeldCredential(c.w, c.password)
if err != nil || string(got) != row.Held {
t.Errorf("%s:\n got %s (%v)\nwant %s", row.Name, got, err, row.Held)
}
}
if _, err := HeldCredential(wanted["module"].w, ""); err == nil {
t.Error("an account was held with no password")
}
})
// 774 subtasks 6 and 8 (novox/hq ADR 0561 §3, §6): minting and the grants part's consumers turn on only while the
// lease is handed to a holder that is no controller; the vault and the grants part alone read the lease, whatever the
// controller announced; a module's consumer is created as the controller's EnsureConsumer creates it.
t.Run("the lease turns minting on only when handed away; a consumer is created as the controller's", func(t *testing.T) {
for _, row := range []struct {
name string
raw string
found bool
err error
on bool
}{
{"a controller holds it", `{"instance":"novox/1234/2026-10-27T20:00:00Z","epoch":57}`, true, nil, false},
{"handed away", `{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`, true, nil, true},
{"expired or deleted", ``, false, nil, false},
{"unreadable", ``, false, errors.New("refused"), false},
{"not a holder", `[1]`, true, nil, false},
{"both a controller and a handed holder", `{"instance":"novox/1","handed-to":"x"}`, true, nil, false},
} {
if on, why := LeaseHandedAway([]byte(row.raw), row.found, row.err); on != row.on || why == "" {
t.Errorf("%s: on %v (%s), want %v", row.name, on, why, row.on)
}
// The assigner's grants source follows it: the grants part's bucket only for a handed-away lease (issue 774).
on, _ := LeaseHandedAway([]byte(row.raw), row.found, row.err)
if bucket, _ := delivery.GrantSource(on); (bucket == delivery.PartGrantsBucket) != row.on {
t.Errorf("%s: the grants source is %s", row.name, bucket)
}
}
announced := map[string]announce.States{ControllerSeat: ControllerAnnouncement("novox")}
for reader, want := range map[string]bool{VaultSeat: true, GrantsModule: true, "mesh-delivery-assigner": true, "mesh-delivery-view": false} {
granted, _ := ReadsAnnouncedBy(reader, []string{ControllerLease}, announced)
if (len(granted) == 1) != want {
t.Errorf("%s reads the lease: %v, want %v", reader, granted, want)
}
}
c, _ := ConsumerFor(Principal{Kind: KindModule, Node: "a", Module: "m", Consumes: []string{"x.happened"}})
cfg := ConsumerConfig(c)
if cfg.Durable != "a_m" || cfg.AckPolicy != jetstream.AckExplicitPolicy || cfg.AckWait != 30*time.Second ||
cfg.MaxDeliver != 5 || cfg.DeliverPolicy != jetstream.DeliverNewPolicy || len(cfg.FilterSubjects) != 2 ||
cfg.Description == "" {
t.Errorf("a module's consumer is created as %+v", cfg)
}
// The grants part is granted the create of each account's consumer by its stream and name, and no other.
users, err := Users(Records{Nodes: []string{"a"}, Assigned: map[string][]Declared{"a": {
{Module: GrantsModule}, {Module: "m", Consumes: []string{"x.happened"}}, {Module: "quiet"}}}})
if err != nil {
t.Fatal(err)
}
for _, u := range users {
if u.Module != GrantsModule || u.Kind != KindModule {
continue
}
perms, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
if !MayPublish(perms, "$JS.API.CONSUMER.CREATE.EVENTS.a_m") || MayPublish(perms, "$JS.API.CONSUMER.CREATE.EVENTS.a_quiet") ||
MayPublish(perms, "$JS.API.CONSUMER.DELETE.EVENTS.a_m") || MayPublish(perms, "$JS.API.CONSUMER.CREATE.EVENTS.a_m.mesh.mod.x.event.happened") {
t.Errorf("the grants part creates consumers as %v", u.CreatesConsumers)
}
}
q := ConsumerConfig(QueueConsumer(Principal{Kind: KindProcess, Node: "a", Module: "m", Process: "p"},
QueueOwner{Process: "p", Stream: "M_WORK", MaxDeliver: 3}))
if q.Durable != "a_m_p" || q.FilterSubject != "" || q.MaxDeliver != 3 || q.DeliverPolicy != jetstream.DeliverAllPolicy {
t.Errorf("a process's queue consumer is created as %+v", q)
}
})
} }
+27
View File
@@ -86,3 +86,30 @@ func ModuleStreamConfig(m ModuleStream) jetstream.StreamConfig {
} }
return cfg return cfg
} }
// ConsumerConfig is a pull consumer as the controller's EnsureConsumer creates it (mesh-controller internal/broker
// JetStream.EnsureConsumer, a consumer not there yet): its durable name, explicit acks, its ack wait, max deliver and max
// ack pending, its description, one filter or several, and from the stream's end when it is made FromNow. The grants part
// creates a module's and a process's consumers with it from the lease write (novox/hq ADR 0561 §6): a create alone,
// never an update of one the bus holds. A push consumer or a queue group is the controller's own and never made here.
func ConsumerConfig(c Consumer) jetstream.ConsumerConfig {
cfg := jetstream.ConsumerConfig{
Durable: c.Name,
AckPolicy: jetstream.AckExplicitPolicy,
AckWait: time.Duration(c.AckWaitSeconds) * time.Second,
MaxDeliver: c.MaxDeliver,
MaxAckPending: c.MaxAckPending,
Description: c.Why,
}
switch len(c.Filters) {
case 0:
case 1:
cfg.FilterSubject = c.Filters[0]
default:
cfg.FilterSubjects = c.Filters
}
if c.FromNow {
cfg.DeliverPolicy = jetstream.DeliverNewPolicy
}
return cfg
}
+11
View File
@@ -23,6 +23,8 @@ func TestTheGrantsPartCreatesOnlyDeclaredStreams(t *testing.T) {
{Module: GrantsModule, State: []Bucket{{Module: GrantsModule, Name: "assertions"}}, Reads: []string{VaultBusAccounts}}, {Module: GrantsModule, State: []Bucket{{Module: GrantsModule, Name: "assertions"}}, Reads: []string{VaultBusAccounts}},
{Module: RuntimeModule}, {Module: RuntimeModule},
}}, }},
// The node-engine's own manifest, placed on every node though no node is assigned it (novox/hq ADR 0563 §2, §3).
Engine: &Declared{Module: EngineModule, State: []Bucket{{Module: EngineModule, Name: EngineActsState}}},
} }
users, err := Users(records) users, err := Users(records)
if err != nil { if err != nil {
@@ -64,6 +66,15 @@ func TestTheGrantsPartCreatesOnlyDeclaredStreams(t *testing.T) {
{GrantsModule, "$JS.API.STREAM.CREATE.ASSIGNMENTS", false}, {GrantsModule, "$JS.API.STREAM.CREATE.ASSIGNMENTS", false},
{GrantsModule, "$JS.API.STREAM.INFO.EVENTS", false}, {GrantsModule, "$JS.API.STREAM.INFO.EVENTS", false},
{"mesh-delivery-builder", "$JS.API.STREAM.CREATE.KV_mesh-delivery-builder_steps", false}, {"mesh-delivery-builder", "$JS.API.STREAM.CREATE.KV_mesh-delivery-builder_steps", false},
// The node-engine's acts: created by the grants part, written by each node's engine under its own keys alone.
{GrantsModule, "$JS.API.STREAM.CREATE.KV_mesh-host_acts", true},
{"node.ace", "$JS.API.STREAM.CREATE.KV_mesh-host_acts", false},
{"node.ace", "$KV.mesh-host_acts.ace.act.hand-over-1", true},
{"node.ace", "$KV.mesh-host_acts.ace.open.hand-over.abc", true},
{"node.ace", "$KV.mesh-host_acts.novox.act.hand-over-1", false},
{"node.ace", "$JS.API.DIRECT.GET.KV_mesh-host_acts.$KV.mesh-host_acts.ace.ask.hand-over-1", true},
{"node.ace", "$JS.API.DIRECT.GET.KV_mesh-host_acts.$KV.mesh-host_acts.novox.ask.hand-over-1", false},
{"novox.node-tools", "$KV.mesh-host_acts.novox.act.hand-over-1", false},
// The sealed user list (ADR 0547 clause 7): the grants part writes its own bucket and no other. // The sealed user list (ADR 0547 clause 7): the grants part writes its own bucket and no other.
{GrantsModule, "$JS.API.STREAM.CREATE.OBJ_mesh-grants-user-list", true}, {GrantsModule, "$JS.API.STREAM.CREATE.OBJ_mesh-grants-user-list", true},
{GrantsModule, "$O.mesh-grants-user-list.C.abc", true}, {GrantsModule, "$O.mesh-grants-user-list.C.abc", true},
+24
View File
@@ -247,6 +247,30 @@ func ConsumersOf(users []Principal) []ModuleConsumer {
return out return out
} }
// AccountConsumers are the durable consumers an account needs (novox/hq ADR 0561 §6): of ConsumersOf, each module's on
// EVENTS and each process's on its own work queue, never one on a module stream it hears (those the controller makes with
// the stream's grant). The grants part creates them from the lease write, with these configurations.
func AccountConsumers(users []Principal) []ModuleConsumer {
queues := map[string]bool{}
for _, p := range users {
for _, q := range p.Queues {
queues[q.Stream] = true
}
for _, d := range p.Carries {
for _, q := range d.Queues {
queues[q.Stream] = true
}
}
}
var out []ModuleConsumer
for _, mc := range ConsumersOf(users) {
if mc.Consumer.Stream == consumerStream(Principal{Kind: KindModule}) || queues[mc.Consumer.Stream] {
out = append(out, mc)
}
}
return out
}
// HolderConsumerFor is the worker a seat's holders share on that seat's work queue. // HolderConsumerFor is the worker a seat's holders share on that seat's work queue.
// //
// **One worker for every holder, and each holder pulls one ask when it is idle** (novox/hq ADR // **One worker for every holder, and each holder pulls one ask when it is idle** (novox/hq ADR
+45
View File
@@ -0,0 +1,45 @@
package buscompose
import (
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// The node seat node-engine (novox/hq ADR 0563 §1): every node's engine holds it on its own bus account, and no module
// claims it. Its verbs and events are the catalogue's (catalogue.EngineVerbs), read here rather than copied.
// EngineSeat is the seat.
const EngineSeat = catalogue.EngineSeat
// EngineModule is the node-engine's manifest's module name, and EngineActsState its state the acts asked through the
// node seat and its own asks are kept in (ADR 0563 §2, §3: "the engine's own state"): the bucket mesh-host_acts, keys
// `<node>.act.<id>`, `<node>.open.<act>.<digest>` and `<node>.ask.<id>`, each written by that node's engine alone.
const (
EngineModule = "mesh-host"
EngineActsState = "acts"
)
// EngineActsBucket is that state's bucket.
var EngineActsBucket = BucketName(EngineModule, EngineActsState)
// engineGrants is what a node's engine is granted to hold its node seat and to ask the operator (ADR 0563 §1, §3): it
// answers its own node's verbs (the caller-named subjects are added beside them with every principal's), says the
// seat's events about its node, answers discovery for the seat, asks who answers what (the login shell's execute, ADR
// 0563 §6), and asks the operator under asks.EngineAsker(node) on the router's seat as its holder declares it, as any
// user of that seat is granted.
func engineGrants(p Principal) (pub, sub []string) {
for _, v := range catalogue.EngineVerbs() {
sub = append(sub, "mesh.seat."+EngineSeat+".tool."+v.Name+"."+p.Node)
}
for _, e := range catalogue.EngineEvents {
pub = append(pub, "mesh.seat."+EngineSeat+".event."+e+"."+p.Node)
}
sub = append(sub, announcing(EngineSeat)...)
pub = append(pub, discovering()...)
tp, ts := seatTrafficGrants(SeatTrafficOf(asks.EngineAsker(p.Node), nil, p.Uses, nil))
// And its acts and asks, under its own node's keys alone; never the bucket's creation, which is the grants part's.
if own, err := NodeKeysGrants(EngineActsBucket, p.Node); err == nil {
pub = append(pub, own...)
}
return append(pub, tp...), append(sub, ts...)
}
+52 -2
View File
@@ -217,6 +217,11 @@ type Principal struct {
// ADR 0547 §2, ADR 0550 §5, issue 1126): its modules' own secrets sealed to its key, for the compositions it makes // ADR 0547 §2, ADR 0550 §5, issue 1126): its modules' own secrets sealed to its key, for the compositions it makes
// itself. Set while the vault's holder serves it. // itself. Set while the vault's holder serves it.
AsksItsOwnSecrets bool AsksItsOwnSecrets bool
// AsksTheVaultToSign is a machine principal whose node-engine may ask the vault's `sign` in its own name, read the
// vault's public signing halves (`signers`) and the controller's lease key (novox/hq ADR 0550 §2): its own node's
// certificate, the keys it trusts, and the lease write that drops the controller's key. Set while the vault's holder
// serves `sign`.
AsksTheVaultToSign bool
// ServedReads are, for a machine principal, the keys of mesh-config's resolved values that other nodes' modules serve // ServedReads are, for a machine principal, the keys of mesh-config's resolved values that other nodes' modules serve
// from (`<node>.<module>.<key>`, catalogue.ServedSettings): what every consumer is told, and so what a node composing // from (`<node>.<module>.<key>`, catalogue.ServedSettings): what every consumer is told, and so what a node composing
// its own modules reads of a provider elsewhere (issue 1126), each key alone. Sorted, none twice. // its own modules reads of a provider elsewhere (issue 1126), each key alone. Sorted, none twice.
@@ -232,6 +237,10 @@ type Principal struct {
// by name. Created only, never updated or deleted: a new module state is on the bus when the grants part hears its // by name. Created only, never updated or deleted: a new module state is on the bus when the grants part hears its
// manifest, not at the controller's next start (issue 1790). // manifest, not at the controller's next start (issue 1790).
CreatesStreams []string CreatesStreams []string
// CreatesConsumers are the durable consumers the grants part's principal alone may create beside the controller
// (novox/hq ADR 0561 §6): each placed module's on EVENTS and each process's on its work queue (AccountConsumers), by
// stream and name. Created only, never updated or deleted.
CreatesConsumers []ModuleConsumer
// PasswordHash is the bcrypt hash the mesh minted. The plaintext is sealed to the principal // PasswordHash is the bcrypt hash the mesh minted. The plaintext is sealed to the principal
// and never appears here: this file is written to a node's disk and read by a server, and a // and never appears here: this file is written to a node's disk and read by a server, and a
@@ -356,6 +365,10 @@ var VaultNodeKey = SeatVerb{Seat: "mesh-vault", Verb: "node-key"}
// key (issue 1126, ADR 0550 §5); granted as Principal.AsksItsOwnSecrets says. // key (issue 1126, ADR 0550 §5); granted as Principal.AsksItsOwnSecrets says.
var VaultOwn = SeatVerb{Seat: "mesh-vault", Verb: "own"} var VaultOwn = SeatVerb{Seat: "mesh-vault", Verb: "own"}
// VaultSign is the vault's verb a machine's node-engine asks in its own name for its own node's certificate (novox/hq
// ADR 0550 §2); granted as Principal.AsksTheVaultToSign says.
var VaultSign = SeatVerb{Seat: "mesh-vault", Verb: delivery.SignVerb}
// VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant // VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant
// chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the // chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the
// controller's own `plans`, a machine's held builds sent by its own `push` and the builds held after a failed release // controller's own `plans`, a machine's held builds sent by its own `push` and the builds held after a failed release
@@ -898,8 +911,8 @@ func PermissionsFor(p Principal) (Permissions, error) {
// its own modules reads the other nodes' public records, which its firewall and resolver need (ADR 0547 §2). Read // its own modules reads the other nodes' public records, which its firewall and resolver need (ADR 0547 §2). Read
// only: each node still writes its own keys alone. // only: each node still writes its own keys alone.
pub = append(pub, readGrants(delivery.HaveBucket, "*."+delivery.HaveNodeWord)...) pub = append(pub, readGrants(delivery.HaveBucket, "*."+delivery.HaveNodeWord)...)
// And what it composes its own modules from (ADR 0547 §2, ADR 0549 §3): the indexer's placement, modules and // And what it composes its own modules from (ADR 0547 §2, ADR 0567 §9): the seat mesh-placement's placement and
// manifests, the builder's builds (the manifest of a build its want names), and mesh-config's resolved values of // candidates, the indexer's modules and manifests, the builder's builds (the manifest of a build its want names), and mesh-config's resolved values of
// its own node alone. Read only. // its own node alone. Read only.
pub = append(pub, NodeComposeReads(p.Node)...) pub = append(pub, NodeComposeReads(p.Node)...)
// And, of the other nodes' resolved values, the keys their modules serve from (issue 1126), each one key. // And, of the other nodes' resolved values, the keys their modules serve from (issue 1126), each one key.
@@ -931,6 +944,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And the backup holder's word that it backed the module up (ADR 0539 clause 16), whichever module holds // And the backup holder's word that it backed the module up (ADR 0539 clause 16), whichever module holds
// node-backup: one event, by its name. // node-backup: one event, by its name.
sub = append(sub, NodeAgentHears...) sub = append(sub, NodeAgentHears...)
// And the creation of the seat mesh-placement's two states it composes from (ADR 0567 §9), so a node agent started
// before they exist binds each when the bus says it was made, with no timer; no other bucket's.
sub = append(sub, NodeComposeAdvisories()...)
// And, while the vault serves it, the vault's move-in in its own name and nobody else's (issue 676): the // And, while the vault serves it, the vault's move-in in its own name and nobody else's (issue 676): the
// values sealed to this machine, which only it opens, handed in sealed to the vault's key for the call. // values sealed to this machine, which only it opens, handed in sealed to the vault's key for the call.
if p.MovesIntoTheVault { if p.MovesIntoTheVault {
@@ -949,6 +965,18 @@ func PermissionsFor(p Principal) (Permissions, error) {
// on it rather than at the next unrelated change (ADR 0547 §2). // on it rather than at the next unrelated change (ADR 0547 §2).
sub = append(sub, VaultSecretChanged) sub = append(sub, VaultSecretChanged)
} }
// And, while the vault serves `sign`, its own node's certificate in its own name, the vault's public signing
// halves, read and watched, and the controller's lease key, read and watched, which says when the controller's key
// is no longer trusted (ADR 0550 §2). Read only.
if p.AsksTheVaultToSign {
pub = append(pub, VaultSignCall(p))
pub = append(pub, readGrants(delivery.SignersBucket, "")...)
pub = append(pub, readGrants(LeaseBucket, leaseKey)...)
}
// And its node seat, node-engine, and its asks to the operator in its own name (novox/hq ADR 0563 §1, §3).
ep, es := engineGrants(p)
pub = append(pub, ep...)
sub = append(sub, es...)
// And, on the node holding mesh-broker, the grants part's sealed user list it follows from take-over (novox/hq ADR // And, on the node holding mesh-broker, the grants part's sealed user list it follows from take-over (novox/hq ADR
// 0547 clause 7): one key, and an object by its name. // 0547 clause 7): one key, and an object by its name.
if p.FollowsGrantsUserList { if p.FollowsGrantsUserList {
@@ -969,6 +997,8 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And creating each stream a manifest declares, by its own name (the operator, 2026-10-26 08:54; issue // And creating each stream a manifest declares, by its own name (the operator, 2026-10-26 08:54; issue
// 1790): the create and the read of that one stream, never an update, a delete or a stream not declared. // 1790): the create and the read of that one stream, never an update, a delete or a stream not declared.
pub = append(pub, CreateStreamGrants(p.CreatesStreams)...) pub = append(pub, CreateStreamGrants(p.CreatesStreams)...)
// And creating each durable consumer an account needs, by its stream and name (ADR 0561 §6).
pub = append(pub, CreateConsumerGrants(p.CreatesConsumers)...)
// And its own object bucket, where it writes the user list sealed to the bus node (ADR 0547 clause 7). // And its own object bucket, where it writes the user list sealed to the bus node (ADR 0547 clause 7).
pub = append(pub, GrantsUserListWrites()...) pub = append(pub, GrantsUserListWrites()...)
} }
@@ -1656,6 +1686,21 @@ func CreateStreamGrants(streams []string) []string {
return out return out
} }
// CreateConsumerGrants are the subjects that create each consumer by its stream and name, and read its info back: the
// create of the client library's CreateConsumer for a consumer with no filter or several, never one filter's form (a
// wildcard in a filter would widen the grant), an update or a delete.
func CreateConsumerGrants(consumers []ModuleConsumer) []string {
var out []string
for _, mc := range consumers {
c := mc.Consumer
if !safeSubject.MatchString(c.Stream) || !safeSubject.MatchString(c.Name) || len(c.Filters) == 1 {
continue
}
out = append(out, "$JS.API.CONSUMER.CREATE."+c.Stream+"."+c.Name, "$JS.API.CONSUMER.INFO."+c.Stream+"."+c.Name)
}
return out
}
// GrantsModule is the one module whose principal publishes the modules' memberships (novox/hq ADR 0547 §5): the grants // GrantsModule is the one module whose principal publishes the modules' memberships (novox/hq ADR 0547 §5): the grants
// part, which composes them from the manifests and placement. It defines no stream and is granted no other subject here. // part, which composes them from the manifests and placement. It defines no stream and is granted no other subject here.
const GrantsModule = "mesh-grants" const GrantsModule = "mesh-grants"
@@ -1824,6 +1869,11 @@ func VaultOwnCall(p Principal) string {
return "mesh.seat." + VaultOwn.Seat + "." + CallKind + "." + VaultOwn.Verb + "." + CallerToken(p.Username()) return "mesh.seat." + VaultOwn.Seat + "." + CallKind + "." + VaultOwn.Verb + "." + CallerToken(p.Username())
} }
// VaultSignCall is the subject a machine principal's node-engine asks the vault's `sign` on, in its own name.
func VaultSignCall(p Principal) string {
return "mesh.seat." + VaultSign.Seat + "." + CallKind + "." + VaultSign.Verb + "." + CallerToken(p.Username())
}
// VaultSecretChanged is the vault's event that a secret changed (a name and its generation, never a value), which a // VaultSecretChanged is the vault's event that a secret changed (a name and its generation, never a value), which a
// node composing its own hears to compose again (issue 1126, ADR 0547 §2). // node composing its own hears to compose again (issue 1126, ADR 0547 §2).
var VaultSecretChanged = "mesh.mod." + VaultOwn.Seat + ".event.secret-changed" var VaultSecretChanged = "mesh.mod." + VaultOwn.Seat + ".event.secret-changed"
+12 -6
View File
@@ -311,12 +311,12 @@ func RaiseBuckets(a BucketAsserter, buckets []Bucket) (undeclared []string, err
// ResolvedBucket is mesh-config's state of resolved values, keyed `<node>.<module>.<key>`. // ResolvedBucket is mesh-config's state of resolved values, keyed `<node>.<module>.<key>`.
const ResolvedBucket = "mesh-config_resolved" const ResolvedBucket = "mesh-config_resolved"
// NodeComposeReads is what a node's agent reads to compose its own modules (novox/hq ADR 0547 §2, ADR 0549 §3, issue // NodeComposeReads is what a node's agent reads to compose its own modules (novox/hq ADR 0547 §2, ADR 0567 §3, §9,
// 1126): every key of the indexer's placement, placement candidates, modules and manifests and of the builder's builds, // issue 1126): every key of the seat mesh-placement's placement and candidates (both the assigner's states), of the
// and of mesh-config's resolved values only its own node's, `<node>.>`. Every candidate key, not only `*.<node>`: a node // indexer's modules and manifests and of the builder's builds, and of mesh-config's resolved values only its own node's,
// answering its candidate composes with the whole group's placement (the seed places every node at once, and a binding // `<node>.>`. Every candidate key, not only `*.<node>`: a node answering its candidate composes with the whole
// crosses nodes), and nothing in a placement is secret. Reads only; the node is checked as every subject token is by the // proposal laid over (the seed places every node at once, and a binding crosses nodes), and nothing in a placement is
// caller's principal. // secret. Reads only; the node is checked as every subject token is by the caller's principal.
func NodeComposeReads(node string) []string { func NodeComposeReads(node string) []string {
var out []string var out []string
for _, bucket := range []string{delivery.PlacementBucket, delivery.PlacementCandidatesBucket, delivery.ModulesBucket, for _, bucket := range []string{delivery.PlacementBucket, delivery.PlacementCandidatesBucket, delivery.ModulesBucket,
@@ -326,6 +326,12 @@ func NodeComposeReads(node string) []string {
return append(out, readGrants(ResolvedBucket, node+".>")...) return append(out, readGrants(ResolvedBucket, node+".>")...)
} }
// NodeComposeAdvisories are the stream-created advisories of the seat mesh-placement's placement and candidates
// (ADR 0567 §9): what a node agent hears to bind each once it is made. Those two and no wider.
func NodeComposeAdvisories() []string {
return []string{StreamCreatedAdvisory(delivery.PlacementBucket), StreamCreatedAdvisory(delivery.PlacementCandidatesBucket)}
}
// ServedReadGrants is the read of each key of mesh-config's resolved values named, `<node>.<module>.<key>` // ServedReadGrants is the read of each key of mesh-config's resolved values named, `<node>.<module>.<key>`
// (Principal.ServedReads, issue 1126): a get and a watch of that key alone. Each part is checked as every subject token // (Principal.ServedReads, issue 1126): a get and a watch of that key alone. Each part is checked as every subject token
// is, so a key cannot widen the grant. // is, so a key cannot widen the grant.
+21
View File
@@ -0,0 +1,21 @@
{
"how": "Written by mesh-controller 0fbb5ef's own issueWith, sealProvisionerClient and heldProcessAccount (cmd/mesh-controller), run unchanged in a scratch integration test against a local store with the default seats seeded, on 2026-10-27 (issue 774): each row is the exact value the controller handed the vault. Never edit a row by hand: capture it again.",
"rows": [
{
"name": "module",
"held": "{\"url\":\"nats://10.0.0.1:4222\",\"fingerprint\":\"ab:cd\",\"node\":\"laptop\",\"module\":\"cap-bot\",\"user\":\"laptop.cap-bot\",\"password\":\"pw-module\",\"claims\":[{\"seat\":\"mesh-vault\",\"scope\":\"mesh\",\"serves\":[\"get\",\"accept\",\"secret-ask\",\"issued\",\"rotate\",\"move-in\",\"node-key\",\"own\",\"move-asks\"]},{\"seat\":\"node-service-manager\",\"scope\":\"node\",\"serves\":[\"status\"]},{\"seat\":\"mesh-controller\",\"scope\":\"mesh\",\"serves\":[\"tools\",\"calls\",\"status\",\"nodes\",\"node\",\"placement-export\",\"modules\",\"seats\",\"builds\",\"plans\",\"delivery-plan\",\"delivery-order\",\"delivery-check\",\"deliver\",\"delivery-stop\",\"delivery-backups-again\",\"delivery-walks\",\"plan\",\"assign\",\"unassign\",\"pin\",\"unpin\",\"push\",\"rotate\",\"give\",\"issued\",\"secret-move\",\"secret-ask\",\"issue\",\"token\",\"settings\",\"command\",\"queue\",\"cancel\",\"clear\",\"replay\",\"try\",\"kill\",\"pause\",\"resume\",\"hand-act\",\"drill\",\"warranted\",\"hand-acts\",\"durations\",\"conditions\",\"healers\",\"doctor\",\"upgrade\",\"bus\",\"retire\",\"cleanup\",\"judge-directories\",\"root-free\",\"dead-letters\",\"data\",\"build\",\"artifacts\",\"held\",\"collect\",\"move-placement\",\"move-config\",\"mirrors\",\"images\"]},{\"seat\":\"no-such-seat\",\"scope\":\"node\"}]}"
},
{
"name": "module-no-claims-no-fingerprint",
"held": "{\"url\":\"nats://10.0.0.1:4222\",\"node\":\"laptop\",\"module\":\"cap-plain\",\"user\":\"laptop.cap-plain\",\"password\":\"pw-plain\"}"
},
{
"name": "provisioner",
"held": "{\"url\":\"nats://10.0.0.1:4222\",\"fingerprint\":\"ab:cd\",\"node\":\"laptop\",\"user\":\"provisioner.laptop\",\"password\":\"pw-prov\"}"
},
{
"name": "process",
"held": "{\"url\":\"nats://10.0.0.1:4222\",\"fingerprint\":\"ab:cd\",\"node\":\"laptop\",\"module\":\"mesh-mail\",\"user\":\"laptop.mesh-mail.send\",\"password\":\"pw-proc\"}"
}
]
}
+6 -2
View File
@@ -77,8 +77,12 @@ func grantsOwnObjects(p Principal, pattern string) bool {
// readers too (announce.State.Readers); both are honoured. // readers too (announce.State.Readers); both are honoured.
const VaultBusAccounts = "mesh-vault.bus-accounts" const VaultBusAccounts = "mesh-vault.bus-accounts"
// NamedReaders are the states only the modules named may read, by `<owner>.<name>`. // NamedReaders are the states only the modules named may read, by `<owner>.<name>`: the grants part's wanted accounts
var NamedReaders = map[string][]string{VaultBusAccounts: {GrantsModule}} // are read by the vault alone (novox/hq ADR 0561 §2).
// The controller's lease is read by the parts that turn on at its write (ADR 0561 §3, §6), and by the assigner, which
// reads the catalogue's placement alone from then (ADR 0549 §5, issue 774).
var NamedReaders = map[string][]string{VaultBusAccounts: {GrantsModule}, GrantsAccounts: {VaultSeat},
ControllerLease: {VaultSeat, GrantsModule, delivery.AssignerModule}}
// ReaderMay says whether module may read the state read (`<owner>.<name>`) by the named readers: always, for a state // ReaderMay says whether module may read the state read (`<owner>.<name>`) by the named readers: always, for a state
// that names none. // that names none.
+21 -1
View File
@@ -134,6 +134,11 @@ type Records struct {
// account agents run as, judged unable to become root by its node-engine, and serves no login shell // account agents run as, judged unable to become root by its node-engine, and serves no login shell
// execute. A machine absent is not free: no record of a pass is no pass. // execute. A machine absent is not free: no record of a pass is no pass.
RootFree map[string]bool RootFree map[string]bool
// Engine is the node-engine's own declaration, from its manifest (EngineModule): placed on every node though no node is
// assigned it, since every node runs it (novox/hq ADR 0563 §2, §3). Its state, logs and streams are created by the
// grants part as a placed module's are; it adds no principal, since each node's engine speaks as the node's account.
// Nil where its manifest was not read.
Engine *Declared
// Announced is each module's last announcement of the states it keeps (novox/hq ADR 0495, issue 1790), as the // Announced is each module's last announcement of the states it keeps (novox/hq ADR 0495, issue 1790), as the
// controller keeps it; a module absent has announced nothing yet. Nil where none was read. // controller keeps it; a module absent has announced nothing yet. Nil where none was read.
Announced map[string]announce.States Announced map[string]announce.States
@@ -149,6 +154,7 @@ func Users(r Records) ([]Principal, error) {
AsksTheConfig: servedOf(r, VerbsTheControllerAsksTheConfig), FilesFor: sortedCopy(r.Nodes)}} AsksTheConfig: servedOf(r, VerbsTheControllerAsksTheConfig), FilesFor: sortedCopy(r.Nodes)}}
served := servedReadsOf(r) served := servedReadsOf(r)
grants := -1
for _, node := range sortedCopy(r.Nodes) { for _, node := range sortedCopy(r.Nodes) {
witness := false witness := false
var checks []string var checks []string
@@ -164,7 +170,7 @@ func Users(r Records) ([]Principal, error) {
} }
out = append(out, Principal{Kind: KindNode, Node: node, WitnessesController: witness, Checks: checks, out = append(out, Principal{Kind: KindNode, Node: node, WitnessesController: witness, Checks: checks,
MovesIntoTheVault: servesMoveIn(r), StatesItsKeyToTheVault: servesVerb(r, VaultNodeKey), MovesIntoTheVault: servesMoveIn(r), StatesItsKeyToTheVault: servesVerb(r, VaultNodeKey),
AsksItsOwnSecrets: servesVerb(r, VaultOwn), AsksItsOwnSecrets: servesVerb(r, VaultOwn), AsksTheVaultToSign: servesVerb(r, VaultSign), Uses: asksSeatOf(r),
FollowsGrantsUserList: bus, ServedReads: servedOn(served, node)}) FollowsGrantsUserList: bus, ServedReads: servedOn(served, node)})
// **And the machine's provisioner client, on an account of its own** (novox/hq ADR 0349 clause 11): one // **And the machine's provisioner client, on an account of its own** (novox/hq ADR 0349 clause 11): one
// per machine that runs it, delivered to the module running it. // per machine that runs it, delivered to the module running it.
@@ -225,6 +231,7 @@ func Users(r Records) ([]Principal, error) {
// 08:54): every placed module's buckets, logs and streams, each by its stream's name. // 08:54): every placed module's buckets, logs and streams, each by its stream's name.
if d.Module == GrantsModule { if d.Module == GrantsModule {
out[len(out)-1].CreatesStreams = declaredStreamsOf(r) out[len(out)-1].CreatesStreams = declaredStreamsOf(r)
grants = len(out) - 1
} }
// **The vault's holder asks the desks and the issuing providers itself** (issue 723), under the // **The vault's holder asks the desks and the issuing providers itself** (issue 723), under the
// account its own runtime runs on (ADR 0259 §8's runs-as): the one module granted it. // account its own runtime runs on (ADR 0259 §8's runs-as): the one module granted it.
@@ -258,6 +265,11 @@ func Users(r Records) ([]Principal, error) {
out = append(out, p) out = append(out, p)
} }
// The grants part creates the durable consumers the accounts need (novox/hq ADR 0561 §6), by stream and name.
if grants >= 0 {
out[grants].CreatesConsumers = AccountConsumers(out)
}
// Refused here rather than discovered by the server. Two users with one name is a file the // Refused here rather than discovered by the server. Two users with one name is a file the
// server reads as one of them, and which one depends on the order — so a module assigned to a // server reads as one of them, and which one depends on the order — so a module assigned to a
// node twice, or a person named after nothing, is a composition that must not be written. // node twice, or a person named after nothing, is a composition that must not be written.
@@ -444,7 +456,15 @@ func runnerServes(d Declared) bool {
// name. What the grants part may create, and nothing beside. // name. What the grants part may create, and nothing beside.
func declaredStreamsOf(r Records) []string { func declaredStreamsOf(r Records) []string {
seen := map[string]bool{} seen := map[string]bool{}
placed := [][]Declared{}
for _, mods := range r.Assigned { for _, mods := range r.Assigned {
placed = append(placed, mods)
}
// The node-engine's manifest is placed on every node (ADR 0563): its declared state is created as a placed module's.
if r.Engine != nil {
placed = append(placed, []Declared{*r.Engine})
}
for _, mods := range placed {
for _, d := range mods { for _, d := range mods {
for _, b := range d.State { for _, b := range d.State {
seen["KV_"+b.Bucket()] = true seen["KV_"+b.Bucket()] = true
+5
View File
@@ -245,6 +245,11 @@ var WritersTable = []WriterRow{
{State: "a machine's have", Writer: "the node-engine of the machine, under its own keys `<node>.>`", {State: "a machine's have", Writer: "the node-engine of the machine, under its own keys `<node>.>`",
KeptIn: "key-value " + delivery.HaveBucket + ", twenty revisions of each key", KeptIn: "key-value " + delivery.HaveBucket + ", twenty revisions of each key",
Others: "read through `reads` as node-engine.have", Subjects: kvOf(delivery.HaveBucket), Writes: ownNodeKeys}, Others: "read through `reads` as node-engine.have", Subjects: kvOf(delivery.HaveBucket), Writes: ownNodeKeys},
// The acts a node's engine was asked through its node seat, and its own asks (novox/hq ADR 0563 §2, §3): the
// node-engine's declared state, created by the grants part, written by each node's engine under its own keys.
{State: "a machine's hand acts", Writer: "the node-engine of the machine, under its own keys `<node>.>`, compare-and-set",
KeptIn: "key-value " + EngineActsBucket + ", the node-engine's declared state acts",
Others: "read through the node seat's verb acts", Subjects: kvOf(EngineActsBucket), Writes: ownNodeKeys},
// The contents of a declaration's files over 64 KiB (novox/hq ADR 0486 §2): the controller alone writes the bucket, each // The contents of a declaration's files over 64 KiB (novox/hq ADR 0486 §2): the controller alone writes the bucket, each
// object under the digest of its content; a node-engine reads one by its name, with no list and no write. // object under the digest of its content; a node-engine reads one by its name, with no list and no write.
{State: "a declaration's large files", Writer: "controller, each object under the sha256 digest of its content; the " + {State: "a declaration's large files", Writer: "controller, each object under the sha256 digest of its content; the " +
+15
View File
@@ -75,6 +75,21 @@ func TestACallerInTheArgumentsIsNeverTheCaller(t *testing.T) {
if got != "shanks.node-tools" { if got != "shanks.node-tools" {
t.Fatalf("with a caller in the arguments the tool read %q, want the runtime's", got) t.Fatalf("with a caller in the arguments the tool read %q, want the runtime's", got)
} }
// A module's trusted list of callers (issue 774, ADR 0563 §8) grants the callers it names as the runtime proved them:
// a module on any machine, <machine>.<module> exactly, person.<name> exactly; no wildcard, never one unproven.
callers, err := ParseCallers([]byte(`{"callers":["node-tools","ace.claude-code","person.jo","*"]}`))
if err != nil {
t.Fatal(err)
}
for caller, want := range map[string]bool{"shanks.node-tools": true, "ace.claude-code": true, "g14.claude-code": false,
"person.jo": true, "person.node-tools": false, Unproven: false, "": false, "novox.mesh-vault": false} {
if callers.Allows(caller) != want {
t.Errorf("%q allowed: %v, want %v", caller, !want, want)
}
}
if _, err := ParseCallers([]byte(`{"callers":["x"],"more":1}`)); err == nil {
t.Error("a list with a field it does not know was read")
}
} }
// A runtime that names no caller — one from before issue 365, or a call on the subject that names none — is // A runtime that names no caller — one from before issue 365, or a call on the subject that names none — is
+48
View File
@@ -0,0 +1,48 @@
package stdio
import (
"bytes"
"encoding/json"
"fmt"
"strings"
)
// Callers is a module's trusted list of who may call a verb of its (novox/hq ADR 0459 §2, ADR 0563 §8), as the mesh
// writes a list setting into a file the module reads: `{"callers": [...]}`. An entry is a module's name (that module on
// any machine), exactly <machine>.<module>, or exactly person.<name>. Each caller is the runtime's proof (CallerOf), never
// an argument; nobody beyond the list is granted, a wildcard is no entry, and an unproven caller never matches.
type Callers []string
// ParseCallers reads a callers file: the list under `callers`, and nothing else.
func ParseCallers(raw []byte) (Callers, error) {
var f struct {
Callers Callers `json:"callers"`
}
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&f); err != nil {
return nil, fmt.Errorf("the callers setting is not a list of callers: %v", err)
}
return f.Callers, nil
}
// Allows says whether caller, as the runtime proved it, is one the list names.
func (c Callers) Allows(caller string) bool {
if caller == "" || caller == Unproven {
return false
}
for _, entry := range c {
if entry == "" || strings.ContainsAny(entry, "*> ") {
continue
}
if entry == caller {
return true
}
if !strings.Contains(entry, ".") {
if _, module, ok := strings.Cut(caller, "."); ok && module == entry && !strings.HasPrefix(caller, "person.") {
return true
}
}
}
return false
}
+29 -17
View File
@@ -126,23 +126,7 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
continue continue
} }
made := by[a.Name] made := by[a.Name]
// What the runtime loads: what the artifact said, else every entrypoint of a module loads := BundleLoads(m, a, run)
// that declares tools, else nothing (the field's own rule; see Artifact.Loads).
//
// **Never, unasked, a bundle one of the module's own resources runs** (novox/hq issue 213).
// A process the host runs is the module's service, not its tools: the controller declares
// the verbs it answers as `tools` and serves them itself, and its bundle would otherwise
// have been launched a second time by the node's runtime, as an MCP child it is not.
loads := append([]string(nil), a.Loads...)
if a.Loads == nil && len(m.Tools) > 0 && !run[a.Name] {
loads = append([]string(nil), a.Entrypoints...)
// A bundle compiled to a binary has no entrypoints: the binary is what it is, and what
// the runtime starts to serve it (novox/hq ADR 0193). So a Go tools bundle is served
// as Go — the runtime execs it — exactly as a TypeScript one is through its launcher.
if bin := BinaryOf(a); bin != "" {
loads = []string{bin}
}
}
// **Kept, never routed** (ADR 0155): the builder publishes to the store at the address // **Kept, never routed** (ADR 0155): the builder publishes to the store at the address
// it reached it by, and a manifest carrying that address names an installation — // it reached it by, and a manifest carrying that address names an installation —
// registration refused node-tools for exactly this on 2026-10-02. The build record // registration refused node-tools for exactly this on 2026-10-02. The build record
@@ -715,3 +699,31 @@ func undeliveredBundles(m Manifest) []string {
} }
return problems return problems
} }
// BundleLoads is what the runtime loads of the bundle artifact a of m (run: the bundles one of m's own resources runs,
// runByAResource): what the artifact said, else every entrypoint of a module that declares tools, else nothing; a
// bundle compiled to a binary is served by its binary. The rule a build's resolution keeps on Bundle.Loads, and what a
// module composed as what runs of it reads again when the runtime already dropped it (issue 1126).
func BundleLoads(m Manifest, a Artifact, run map[string]bool) []string {
// What the runtime loads: what the artifact said, else every entrypoint of a module
// that declares tools, else nothing (the field's own rule; see Artifact.Loads).
//
// **Never, unasked, a bundle one of the module's own resources runs** (novox/hq issue 213).
// A process the host runs is the module's service, not its tools: the controller declares
// the verbs it answers as `tools` and serves them itself, and its bundle would otherwise
// have been launched a second time by the node's runtime, as an MCP child it is not.
loads := append([]string(nil), a.Loads...)
if a.Loads == nil && len(m.Tools) > 0 && !run[a.Name] {
loads = append([]string(nil), a.Entrypoints...)
// A bundle compiled to a binary has no entrypoints: the binary is what it is, and what
// the runtime starts to serve it (novox/hq ADR 0193). So a Go tools bundle is served
// as Go — the runtime execs it — exactly as a TypeScript one is through its launcher.
if bin := BinaryOf(a); bin != "" {
loads = []string{bin}
}
}
return loads
}
// RunByAResource is the bundles of m one of its own resources runs (its service, not its tools).
func RunByAResource(m Manifest) map[string]bool { return runByAResource(m) }
+37
View File
@@ -0,0 +1,37 @@
package catalogue
// The delivery's checker (novox/hq ADR 0547 §10, issue 774): one holder for the mesh, the module
// mesh-delivery-checker, which judges a pull request's head and a pushed head the same way. The try the
// controller forwarded to it is a verb of this seat, answered by the checker with the verdict it gives a pull
// request, so the try outlives the controller.
// CheckerSeat is the seat mesh-delivery-checker holds.
const CheckerSeat = "mesh-delivery-checker"
// checkerVerbs are the checker seat's verbs: ask a try, read one, end one by hand. Optional until the checker serves them
// everywhere.
func checkerVerbs() []Verb {
return []Verb{
{Name: "try", Optional: true, Description: "Try a pushed head before its pull request is opened: the checker " +
"judges it as it judges a pull request's head, merged onto its trunk's head, its unit and integration " +
"levels through the build modules, and owes the forge nothing. Answers the try's id at once; try-read " +
"reads where it stands. A try of the same head that waits or runs is the answer to a second ask; at most " +
"4 wait or run at once, mesh-wide.",
Input: schema(map[string]string{"repository": "<owner>/<name>",
"head": "the full commit to try, 40 or 64 hex digits",
"trunk": "the trunk commit to merge it onto, or the trunk's branch for its head now; the trunk's head when absent",
"tests": "a go test -run pattern: only those tests run, and nothing is kept",
"level": "unit or integration: only this level; both when absent"},
[]string{"repository", "head"}),
Replaces: []string{"sh merge-check.sh", "docker run postgres"}},
{Name: "try-read", Optional: true, Description: "Where a try stands, as stored: queued, running, or its end " +
"(pass, fail, or error when it could not run), with what was said of it, the head and the trunk commit " +
"it was merged onto, the time it took and what failed first. Read-only.",
Input: schema(map[string]string{"id": "the try's id, try-<digits>"}, []string{"id"})},
{Name: "try-end", Optional: true, Description: "End a try that waits or runs, as error, by hand: say why (kept in " +
"what it says). Its place among the tries is free at once and the next try of its head asks anew with a new " +
"id. A try that ended already is refused.",
Input: schema(map[string]string{"id": "the try's id, try-<digits>", "why": "why it is ended, in words"},
[]string{"id", "why"})},
}
}
+13
View File
@@ -445,6 +445,19 @@ func TestAnUnsetRequiredValueIsRefusedInTheConfigFormsWords(t *testing.T) {
t.Fatalf("not typed as a value nobody gave: %v", err) t.Fatalf("not typed as a value nobody gave: %v", err)
} }
// A contribution's value too (issue 1126, seed-8: de-spiegel's route names were refused with ${setting:…}).
withRoute := inConfigForm()
withRoute["contributes"] = map[string]any{"route": map[string]any{"site": map[string]any{"name": "${config:sender}", "port": 80}}}
routed, err := parseRaw(t, withRoute)
if err != nil {
t.Fatal(err)
}
_, err = Resolution{}.composed(routed, "route", routed.ContributesMany["route"]["site"], nil, "notifier contributing site to route")
if !errors.As(err, &unset) || unset.Module != "notifier" || !strings.Contains(err.Error(),
`notifier has a contribution site to route, in "name", that says ${config:sender}`) || strings.Contains(err.Error(), "${setting:") {
t.Fatalf("a contribution's unset value is not said in the config form's words: %v", err)
}
settingsForm := notifier() settingsForm := notifier()
settingsForm.Resources[0]["content"] = "from = ${setting:domain}\n" settingsForm.Resources[0]["content"] = "from = ${setting:domain}\n"
err = unsetInConfigWords(settingInto(settingsForm.Resources[0], nil, "notifier"), settingsForm) err = unsetInConfigWords(settingInto(settingsForm.Resources[0], nil, "notifier"), settingsForm)
+26 -1
View File
@@ -10,6 +10,7 @@ package catalogue
import ( import (
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"sort" "sort"
"strconv" "strconv"
@@ -138,6 +139,9 @@ type Rendering struct {
// Kept is every operator-sealed secret in the mesh, for a module that `keeps` them. Nil when // Kept is every operator-sealed secret in the mesh, for a module that `keeps` them. Nil when
// nothing on this node keeps them, or the mesh has no operator key. // nothing on this node keeps them, or the mesh has no operator key.
Kept *KeptExport Kept *KeptExport
// KeptRuns is, by module, the file of the export as the node runs it, for a composer that holds no export (a node
// composing its own: the export is the controller's store's, issue 1126, C6): carried as it runs when Kept is nil.
KeptRuns map[string]map[string]any
// OutwardLinks is the links this machine reported as facing outside it, which the filter is // OutwardLinks is the links this machine reported as facing outside it, which the filter is
// written around (novox/hq ADR 0140). Empty means the machine has not said, and the mesh // written around (novox/hq ADR 0140). Empty means the machine has not said, and the mesh
@@ -929,6 +933,14 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
} }
// Read by the module as its secrets' owner when it names one: the vault, as its own account (issue 723). // Read by the module as its secrets' owner when it names one: the vault, as its own account (issue 723).
first = append(first, ownedBy(m.SecretsOwner, file)) first = append(first, ownedBy(m.SecretsOwner, file))
} else if runs, ok := with.KeptRuns[m.Module]; m.Keeps != "" && ok {
file := make(map[string]any, len(runs))
for k, v := range runs {
file[k] = v
}
file["id"] = KeptID()
delete(file, "module")
first = append(first, file)
} }
if m.Computed != "" { if m.Computed != "" {
mine, part := generated[m.Module] mine, part := generated[m.Module]
@@ -1091,6 +1103,9 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
} }
for _, unsettled := range resources { for _, unsettled := range resources {
if pr, ok := unsettled[HandedProvisionField].(PlanProvision); ok { if pr, ok := unsettled[HandedProvisionField].(PlanProvision); ok {
if handing == nil && m.Stubbed {
continue // held: its credential file is what runs of it, left out (issue 1126)
}
if handing == nil { if handing == nil {
return nil, fmt.Errorf("%s: its credential of %s is handed to the provisioner and the module is "+ return nil, fmt.Errorf("%s: its credential of %s is handed to the provisioner and the module is "+
"not: nothing would fill it", m.Module, pr.Provision) "not: nothing would fill it", m.Module, pr.Provision)
@@ -1115,6 +1130,9 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
continue continue
} }
if unsettled[HandedSecretField] != nil { if unsettled[HandedSecretField] != nil {
if handing == nil && m.Stubbed {
continue // held: its own secret's file is what runs of it, left out (issue 1126)
}
if handing == nil { if handing == nil {
return nil, fmt.Errorf("%s: its own secret %v is handed to the provisioner and the module is "+ return nil, fmt.Errorf("%s: its own secret %v is handed to the provisioner and the module is "+
"not: nothing would fill it", m.Module, unsettled[HandedSecretField]) "not: nothing would fill it", m.Module, unsettled[HandedSecretField])
@@ -1821,7 +1839,14 @@ func (r Resolution) composed(m Manifest, to string, raw map[string]any, layers [
// module's files is no part of it (novox/hq 04-ISSUES/173). // module's files is no part of it (novox/hq 04-ISSUES/173).
values, err := overridden(raw, WithDefaults(m, layers), what) values, err := overridden(raw, WithDefaults(m, layers), what)
if err != nil { if err != nil {
return nil, fmt.Errorf("%s: %w", what, err) // A required value nothing sets is said in the words of the module's form, as a file's, a unit's and an owner's
// are (issue 1126, seed-8: de-spiegel in the config form was refused with ${setting:…} and `settings set`).
var unset *UnsetSettingError
if errors.As(err, &unset) && unset.Module == "" {
unset.Module = m.Module
unset.where = fmt.Sprintf("a contribution%s, in %q,", strings.TrimPrefix(what, m.Module+" contributing"), unset.field)
}
return nil, unsetInConfigWords(fmt.Errorf("%s: %w", what, err), m)
} }
reaches, err := Reaches(m, layers) reaches, err := Reaches(m, layers)
if err != nil { if err != nil {
+57
View File
@@ -0,0 +1,57 @@
package catalogue
// The node seat node-engine (novox/hq ADR 0563 §1): held on every node by that node's engine (mesh-host), which serves
// its verbs on its own bus account. No module claims it and no assignment holds it: what the mesh knows of its holder
// is the node's record and the engine's haves (ADR 0563 §1 changes ADR 0110 §6 and §7 for this one seat).
//
// Its verbs carry no signature (§7): the runtime proves the caller, which matters only for the record. The read-only
// ones change nothing or are bounded in the engine's own state (§2); hand-over and access-act perform nothing on the
// call, and act only on the operator's warrant to the engine's own ask (§3).
// EngineSeat is the node seat every node's engine holds.
const EngineSeat = "node-engine"
// EngineVerbs are the node-engine seat's verbs, in the order ADR 0563 §1 names them.
func EngineVerbs() []Verb {
return []Verb{
{Name: "hand-over", Description: "Hand a directory this machine uses as found to the mesh (novox/hq ADR 0563 §3, " +
"§5): the engine judges the path as it does every hand-over (absolute, as it states it, used as found, " +
"reached through no link) and refuses in words, or asks the operator on their channel at the level approve, " +
"binding the owner, group and mode it will set. Nothing changes on the call: only the operator's yes hands " +
"it over. One ask is open per path; acts shows where it stands.",
Input: schema(map[string]string{"path": "the directory's absolute path, as the engine states it",
"why": "why it is asked, in words the operator reads"}, []string{"path"})},
{Name: "access-act", Description: "Ask the operator for the take act on a library whose root holds data outside its " +
"group (novox/hq ADR 0328 §7, ADR 0563 §3, §4), on the library's home engine: the access as declared and its " +
"root as last declared, judged and refused in words or asked at the level approve. Only the operator's yes " +
"sets the library's group, group write and setgid under its root. One ask is open per access.",
Input: schema(map[string]string{"act": "the act: take", "access": "the access, library:<name>",
"why": "why it is asked, in words the operator reads"}, []string{"act", "access"})},
{Name: "judge-directories", Description: "Which of this machine's declared directories the engine would refuse to " +
"make or keep, one line each (novox/hq ADR 0563 §2). Read-only: changes no owner or mode.",
Input: schema(map[string]string{}, nil)},
{Name: "setuid-search", Description: "Throw away the engine's last search for setuid programs and start a full one " +
"(novox/hq ADR 0563 §2); until it completes root-free answers that the machine is not judged yet. Refused while " +
"one runs, and when one was started within the hour, naming when the next may start.",
Input: schema(map[string]string{"why": "why it is asked, in words"}, nil)},
{Name: "root-free", Description: "Whether this machine is root-free (novox/hq ADR 0259 §8, ADR 0563 §6): free only " +
"on the engine's own verdict, at most fifteen minutes old, that the account agents run as cannot become root " +
"without a person, and the login shell's execute not served here. Anything else is no pass, and says why. " +
"Answered by this machine's engine alone.",
Input: schema(map[string]string{}, nil)},
{Name: "acts", Description: "Every act this machine's engine was asked through this seat (novox/hq ADR 0563 §9): " +
"each hand-over, access act and setuid search, open, done or refused, with who asked, the ask and the " +
"operator's warrant. Read-only.",
Input: schema(map[string]string{"limit": "at most this many, newest", "id": "one act, by its id"}, nil)},
}
}
// EngineEvents are what the seat says of an act, each naming the act, the path or access, the caller as the runtime
// proved it and the ask (ADR 0563 §1).
var EngineEvents = []string{"asked", "done", "refused"}
// engineSeat is the seat's row in the closed set.
func engineSeat() Seat {
return Seat{Name: EngineSeat, Scope: ScopeNode, Serves: EngineVerbs(), Emits: append([]string(nil), EngineEvents...),
HeldByTheEngine: true, Decision: "novox/hq ADR 0563"}
}
+17 -1
View File
@@ -121,7 +121,7 @@ func (r Resolution) ownedPaths() map[string]string {
for _, s := range Seats() { for _, s := range Seats() {
for _, sp := range s.Paths { for _, sp := range s.Paths {
for _, m := range r.Modules { for _, m := range r.Modules {
if m.ClaimsSeat(s.Name) && declaresPath(m, path.Clean(sp.Path)) { if m.ClaimsSeat(s.Name) && declaresPath(m, path.Clean(sp.Path)) && !r.stepsBackHere(m, sp) {
owned[sp.Path] = m.Module + " (" + s.Name + ")" owned[sp.Path] = m.Module + " (" + s.Name + ")"
} }
} }
@@ -130,6 +130,22 @@ func (r Resolution) ownedPaths() map[string]string {
return owned return owned
} }
// stepsBackHere says m's seat path steps back on this machine (novox/hq ADR 0287): it names a seat whose holder here
// declares the path too, so that holder owns it, as stepsBack leaves m's own rendering of it out. Without it the owner
// told was the last seat in the order of the seat table, which the controller reads sorted by name and a node in the
// order compiled (issue 1126, C3: /etc/resolv.conf, node-uplink against node-resolver).
func (r Resolution) stepsBackHere(m Manifest, sp SeatPath) bool {
if sp.StepsBackFor == "" {
return false
}
for _, o := range r.Modules {
if o.Module != m.Module && holdsSeat(o, sp.StepsBackFor) && declaresPath(o, path.Clean(sp.Path)) {
return true
}
}
return false
}
// filesWords is what the holder of node-files is given beside its own words: the node's secret places and the // filesWords is what the holder of node-files is given beside its own words: the node's secret places and the
// paths modules own here. // paths modules own here.
func (r Resolution) filesWords(m Manifest, words map[string]string, with Rendering) (map[string]string, error) { func (r Resolution) filesWords(m Manifest, words map[string]string, with Rendering) (map[string]string, error) {
+34
View File
@@ -2,6 +2,7 @@ package catalogue
import ( import (
"encoding/json" "encoding/json"
"slices"
"sort" "sort"
"strings" "strings"
"testing" "testing"
@@ -86,6 +87,39 @@ func TestTheHolderOfNodeFilesIsToldSecretPlacesAndOwnedPaths(t *testing.T) {
if !strings.HasPrefix(owned["/etc/app/app.conf"], "app") { if !strings.HasPrefix(owned["/etc/app/app.conf"], "app") {
t.Errorf("a placed file is owned by %q, want app", owned["/etc/app/app.conf"]) t.Errorf("a placed file is owned by %q, want app", owned["/etc/app/app.conf"])
} }
// Issue 1126, C3: a seat path that steps back for a seat held here is that holder's, whatever the order of the seat
// table (the controller reads it sorted by name, a node in the order compiled).
resolver := Manifest{Module: "systemd-resolved", Version: "1", Claims: []Claim{{Name: ResolverSeat}},
Resources: []map[string]any{{"id": "stub", "type": "file", "path": ResolverFile}}}
uplink := Manifest{Module: "networkmanager", Version: "1", Claims: []Claim{{Name: "node-uplink"}},
Resources: []map[string]any{{"id": "resolv", "type": "file", "path": ResolverFile}}}
compiled := seats
t.Cleanup(func() { seats = compiled })
for _, order := range []string{"compiled", "reversed"} {
if order == "reversed" {
seats = append([]Seat(nil), compiled...)
slices.Reverse(seats)
}
both := Resolution{Node: "anchor", Modules: []Manifest{files, uplink, resolver}}
w, err := both.filesWords(files, nil, Rendering{})
if err != nil {
t.Fatal(err)
}
var o map[string]string
if err := json.Unmarshal([]byte(w[FilesOwnedPaths]), &o); err != nil || o[ResolverFile] != "systemd-resolved ("+ResolverSeat+")" {
t.Errorf("seat table %s: %s is owned by %q, want systemd-resolved (%s)", order, ResolverFile, o[ResolverFile], ResolverSeat)
}
}
seats = compiled
// Issue 1126, C5 and seed-8: a module carried as it runs (compose.Carried) owns its manifest's paths (OwnsPaths), as
// the controller tells them, and not the paths its running resources are at (a directory the mesh placed).
carried := Manifest{Module: "de-spiegel", Version: "1", OwnsPaths: []string{"/var/lib/de-spiegel/server.env"},
Resources: []map[string]any{{"id": "state", "type": "directory", "path": "/var/lib/de-spiegel"}}}
w, err := Resolution{Node: "anchor", Modules: []Manifest{files, carried}}.filesWords(files, nil, Rendering{})
if err != nil || !strings.Contains(w[FilesOwnedPaths], `"/var/lib/de-spiegel/server.env":"de-spiegel (its own tools)"`) ||
strings.Contains(w[FilesOwnedPaths], `"/var/lib/de-spiegel":`) {
t.Errorf("a carried module does not own its manifest's paths alone: %v %v", w[FilesOwnedPaths], err)
}
if w, _ := r.filesWords(app, nil, Rendering{}); w[FilesOwnedPaths] != "" { if w, _ := r.filesWords(app, nil, Rendering{}); w[FilesOwnedPaths] != "" {
t.Error("a module not holding node-files is told the owned paths") t.Error("a module not holding node-files is told the owned paths")
} }
+16
View File
@@ -185,6 +185,22 @@ func TestAHandedModulesCredentialIssuedForItIsNamedInItsPlanAndNoValueIsSealed(t
if !reflect.DeepEqual(c.Plans["bot"].Provisions, want) { if !reflect.DeepEqual(c.Plans["bot"].Provisions, want) {
t.Fatalf("the plan names %+v, want %+v", c.Plans["bot"].Provisions, want) t.Fatalf("the plan names %+v, want %+v", c.Plans["bot"].Provisions, want)
} }
// Issue 1126 (novox's seed: "gitea: its credential of postgres-database is handed to the provisioner and the module
// is not"): held, the module is composed as its stub, with no template and no item, so nothing is handed. Its
// credential and its own secret are what runs of it, left out, and the composition is not refused.
r := forgeReader(t, true, true)
for i := range r.Modules {
if r.Modules[i].Module == "bot" {
h := *r.Modules[i].Handed
h.Templates = nil
r.Modules[i].Handed, r.Modules[i].Stubbed, r.Modules[i].Build = &h, true, nil
r.Modules[i].Resources = slices.DeleteFunc(r.Modules[i].Resources, func(x map[string]any) bool { return x["type"] != "directory" })
}
}
c, got = composeWith(t, r, handingTo("store:5000"))
if strings.Contains(got, SecretID("forge-read")) || strings.Contains(got, NeedID("token")) || c.Plans["bot"].Provisions != nil {
t.Fatalf("a held module's handed credential or own secret is composed: %s", got)
}
} }
func TestACredentialIssuedForAHolderIsSealedAsBeforeWhereItsModuleIsNotHanded(t *testing.T) { func TestACredentialIssuedForAHolderIsSealedAsBeforeWhereItsModuleIsNotHanded(t *testing.T) {
+44
View File
@@ -0,0 +1,44 @@
package catalogue
import "encoding/json"
// The sources mesh-config says a resolved value came from, in mesh-config_resolved (novox/hq ADR 0348 §9): a key of
// it holds {"value": …, "source": …}.
const (
SourceMesh = "mesh"
SourceNode = "node"
SourceModule = "module"
SourceDefault = "default"
SourceUnset = "unset"
)
// LaidSource says whether a value resolved from source is laid over a module's files: mesh config, node config and the
// module's own value are; a manifest's default is not, since it is the composition's already and a taken key is laid
// only when a layer sets it, never from a default (TakesSettingsField); an unset value is not, since a required one is
// refused by the composition by name. **The one rule for every composer** (issue 1126, C2): the controller's config
// read and a node composing its own lay the same values, so the two compose the same files.
func LaidSource(source string) bool {
switch source {
case SourceMesh, SourceNode, SourceModule:
return true
}
return false
}
// LaidValue is one key of mesh-config_resolved read as a composer lays it: its value, and true when its source is laid
// (LaidSource) and it holds a value; false for a default, an unset value, a source not known and a value that cannot be
// read.
func LaidValue(raw []byte) (any, bool) {
var r struct {
Value json.RawMessage `json:"value"`
Source string `json:"source"`
}
if json.Unmarshal(raw, &r) != nil || !LaidSource(r.Source) || len(r.Value) == 0 {
return nil, false
}
var v any
if json.Unmarshal(r.Value, &v) != nil {
return nil, false
}
return v, true
}
+11
View File
@@ -864,6 +864,17 @@ type Manifest struct {
// that could only see its own ports would write a rule set that closed everything else. // that could only see its own ports would write a rule set that closed everything else.
Filtering *Filtering `json:"filtering,omitempty"` Filtering *Filtering `json:"filtering,omitempty"`
// OwnsPaths is, for a module composed from what runs of it (compose.Carried: folded into its node's own composition
// at a build of the old form, issue 1126, C5), the paths it owns: its manifest's (DeclaredPaths), as the controller
// tells the holder of node-files them, in place of its resources', which are what runs (a directory placed at a
// path its manifest leaves to the mesh, a binding). Nil for any other module. Never in a manifest's JSON.
OwnsPaths []string `json:"-"`
// Stubbed marks a module composed as its stub (compose.Stub: held on its node, issue 1126): none of its own
// composition is applied, so what it hands the provisioner (a credential the vault keeps for it, its own secrets)
// is left out of its plan as it runs, never refused for want of a plan. Never in a manifest's JSON.
Stubbed bool `json:"-"`
// Jails are the fail2ban jails this module declares for its own service (novox/hq to-be 31). // Jails are the fail2ban jails this module declares for its own service (novox/hq to-be 31).
// Written into whichever node runs the module, the same way `listens` become that node's rules. // Written into whichever node runs the module, the same way `listens` become that node's rules.
Jails []Jail `json:"jails,omitempty"` Jails []Jail `json:"jails,omitempty"`
+74
View File
@@ -0,0 +1,74 @@
package catalogue
// The seat of placement (novox/hq ADR 0567): which module runs on which node, with its host ports, bindings, the
// node's pins and holdings, the config placement and values, and what a person decides of the node, is state on the
// bus held by this seat, changed only by a proposal the nodes it touches answer and the operator approves. Held once
// for the mesh by the assigner (mesh-delivery-assigner), beside its wants. Its states are named after the holding
// module, as every module state is (delivery.PlacementBucket, PlacementProposalsBucket, PlacementCandidatesBucket);
// they are the seat's contract, so every reader takes their names from mesh-sdk, and a change of holder is a migration
// the new holder's record decides (ADR 0567 §1). Its events are the holder's too, carried on the holder's stream
// (delivery.PlacementStream): the names are in delivery (PlacementProposedEvent and the rest).
// PlacementSeat is the seat the assigner holds for placement.
const PlacementSeat = "mesh-placement"
// The seat's seven verbs (ADR 0567 §10).
const (
PlacementVerbPlacement = "placement"
PlacementVerbProposals = "proposals"
PlacementVerbPropose = "propose"
PlacementVerbWithdraw = "withdraw"
PlacementVerbAgain = "again"
PlacementVerbAdopt = "adopt"
PlacementVerbHandBack = "hand-back"
)
// placementVerbs are the seat's verbs: what reads all it keeps and does by hand all it does by itself (ADR 0567 §10).
// Optional while the holder that serves them is not yet on every node (design 33 §7): the seat is new in the same
// delivery as its holder.
func placementVerbs() []Verb {
changes := map[string]any{"type": "array", "minItems": 1, "description": "the changes, each an object with `op`: " +
"place (node, module, ports, bindings), take-off (node, module), pin (node, provision, provider {node, module}), " +
"unpin (node, provision), hold (node, module, claim, scope, kind, site), unhold (node, module, claim), decide " +
"(node, field, value: a decided field of the node, its address among them), config (node, module, mesh-wide, " +
"layer, mesh-wide-values, values)",
"items": map[string]any{"type": "object", "required": []string{"op", "node"}}}
return []Verb{
{Name: PlacementVerbPlacement, Optional: true, Description: "Read placement: one node's key, the nodes one " +
"module is placed on, or every key, each with its revision, and how full the state is against its cap. With " +
"history, one node's or one module's changes in order: a revision the state keeps whole, an older one as the " +
"proposal's changes from its placement-changed event.",
Input: schema(map[string]string{"node": "one node", "module": "one module: the nodes it is placed on",
"history": "\"true\": the changes in order, from the stream and the state's revisions"}, nil, "history")},
{Name: PlacementVerbProposals, Optional: true, Description: "Every open proposal of placement: its changes, " +
"the revision of each key it was laid over, its candidates, every node's answer, the differences computed and " +
"answered, its verdict and its ask. Narrowed to one by id.",
Input: schema(map[string]string{"id": "one proposal's id"}, nil)},
{Name: PlacementVerbPropose, Optional: true, Description: "Propose a change of placement, with why: laid over " +
"the current keys and refused at once, naming every finding, when the result cannot be placed (two modules " +
"on one host port, a provider not placed, a seat held once held twice, two nodes with one address, two " +
"hubs, a module the indexer does not hold, a node with no record). Otherwise the nodes it touches compose " +
"it in shadow and answer, and on a passing verdict the operator is asked to approve it. Who proposes is the " +
"caller, as the runtime proves it.",
Input: map[string]any{"type": "object", "additionalProperties": false, "required": []string{"why", "changes"},
"properties": map[string]any{
"why": map[string]any{"type": "string", "description": "why, shown to the operator with the proposal"},
"changes": changes,
}}},
{Name: PlacementVerbWithdraw, Optional: true, Description: "Withdraw an open proposal: its key and its " +
"candidates are deleted and placement-withdrawn carries it whole. With why.",
Input: schema(map[string]string{"id": "the proposal's id", "why": "why, kept in the event"}, []string{"id", "why"})},
{Name: PlacementVerbAgain, Optional: true, Description: "Judge an open proposal now with the answers that " +
"stand, and raise its ask again when its verdict passes and the operator's channel does not hold the ask. " +
"Nothing more: the recovery of a proposal whose judgement or ask was lost.",
Input: schema(map[string]string{"id": "the proposal's id"}, []string{"id"})},
{Name: PlacementVerbAdopt, Optional: true, Description: "Take placement over from the controller: its " +
"placement-export is proposed whole as the seed, every node's key, which every node must answer as the same. " +
"Its Approve is the take-over; at the warrant the export is read again and a changed one is proposed anew.",
Input: schema(map[string]string{"why": "why, shown to the operator"}, []string{"why"})},
{Name: PlacementVerbHandBack, Optional: true, Description: "The operator's alone, recorded as a hand act: " +
"delete every placement key, the history kept, so every node returns to the controller's compositions. " +
"Refused after the lease write.",
Input: schema(map[string]string{"why": "why, recorded with the hand act"}, []string{"why"})},
}
}
+14
View File
@@ -123,6 +123,14 @@ func namesWithoutSteppingBack(s Seat, p string) bool {
// as written would pass for a path nobody named. // as written would pass for a path nobody named.
func declaredPaths(m Manifest) map[string]bool { func declaredPaths(m Manifest) map[string]bool {
out := map[string]bool{} out := map[string]bool{}
if m.OwnsPaths != nil {
// Carried: what its manifest declares, never what runs of it (issue 1126, seed-8: f1-circuits, snake and txt-game
// on ace owned their /var/lib paths on the node and none on the controller).
for _, p := range m.OwnsPaths {
out[p] = true
}
return out
}
for _, r := range m.Resources { for _, r := range m.Resources {
if p, ok := r["path"].(string); ok && strings.HasPrefix(p, "/") { if p, ok := r["path"].(string); ok && strings.HasPrefix(p, "/") {
out[path.Clean(p)] = true out[path.Clean(p)] = true
@@ -136,6 +144,12 @@ func declaredPaths(m Manifest) map[string]bool {
return out return out
} }
// DeclaredPaths is every machine path a module's manifest declares it owns (declaredPaths), sorted and never nil: what
// a carried module owns (Manifest.OwnsPaths).
func DeclaredPaths(m Manifest) []string {
return append([]string{}, sortedKeys(declaredPaths(m))...)
}
// declaresPath says whether a module declares one path, as a resource or as a file the mesh renders. // declaresPath says whether a module declares one path, as a resource or as a file the mesh renders.
func declaresPath(m Manifest, p string) bool { return declaredPaths(m)[p] } func declaresPath(m Manifest, p string) bool { return declaredPaths(m)[p] }
+34 -2
View File
@@ -67,6 +67,10 @@ type Seat struct {
// names paths. Compiled, never stored, like Receives: adding or taking away a path is a decision, // names paths. Compiled, never stored, like Receives: adding or taking away a path is a decision,
// recorded, and the store's rows carry no column for it. // recorded, and the store's rows carry no column for it.
Paths []SeatPath Paths []SeatPath
// HeldByTheEngine says the seat is held on every node by that node's engine, which serves its verbs on its own bus
// account (novox/hq ADR 0563 §1): no module claims it and no assignment holds it. Compiled, never stored, like
// Receives.
HeldByTheEngine bool
// Decision is the record that made it a seat. // Decision is the record that made it a seat.
Decision string Decision string
} }
@@ -171,7 +175,8 @@ var defaultSeats = append([]Seat{
"origin": "issued for a bus account the controller minted for a core or moved module and hands " + "origin": "issued for a bus account the controller minted for a core or moved module and hands " +
"at once (ADR 0374, issue 787); left out for a value given at the terminal"}), nil)}, "at once (ADR 0374, issue 787); left out for a value given at the terminal"}), nil)},
{Name: "secret-ask", Optional: true, Description: "Ask the operator for a module's own secret at a desk, " + {Name: "secret-ask", Optional: true, Description: "Ask the operator for a module's own secret at a desk, " +
"from the controller alone: the ask is recorded in the vault's store and bounded before the prompt " + "from the controller or a caller the vault's manifest names, named as the bus proved it (novox/hq ADR " +
"0563 §8): the ask is recorded in the vault's store and bounded before the prompt " +
"opens, and what is typed is sealed to a key the vault made for the call and opens there alone.", "opens, and what is typed is sealed to a key the vault made for the call and opens there alone.",
Input: schema(vaultIntake(map[string]string{"node": "the machine the secret is for", Input: schema(vaultIntake(map[string]string{"node": "the machine the secret is for",
"module": "the module", "name": "the secret's name in the module", "module": "the module", "name": "the secret's name in the module",
@@ -224,6 +229,18 @@ var defaultSeats = append([]Seat{
Input: map[string]any{"type": "object", "properties": map[string]any{"asks": map[string]any{ Input: map[string]any{"type": "object", "properties": map[string]any{"asks": map[string]any{
"type": "array", "items": map[string]any{"type": "object"}, "type": "array", "items": map[string]any{"type": "object"},
"description": "the controller's rows: id, node, module, name, desk, asked_by, opened_at, ended_at, outcome"}}}}, "description": "the controller's rows: id, node, module, name, desk, asked_by, opened_at, ended_at, outcome"}}}},
{Name: "issue-account", Optional: true, Description: "Mint one bus account the grants part wants, now (novox/hq " +
"ADR 0561 §9): only while the controller's lease is handed away, never a node's account or the " +
"controller's, never anew for one the vault holds a value or an adopted hash for. Never the value.",
Input: schema(map[string]string{"user": "the bus user, e.g. ace.mesh-build-go"}, []string{"user"})},
{Name: "sign", Optional: true, Description: "Sign with the vault's key or authority (novox/hq ADR 0550 §2): " +
"kind node-certificate signs the certificate of the calling machine's internal name for the serving key " +
"its record states, built from bus state alone and asked by that machine's node-engine only; kind " +
"enrolment signs an enrolment reply under its own context, for a caller the adoption record names. " +
"Answers the certificate or the signature and the signer's fingerprint, and says signed; never a key.",
Input: schema(map[string]string{"kind": "node-certificate or enrolment",
"node": "the machine, for a node certificate: the caller's own",
"reply": "the enrolment reply's bytes in base64, for an enrolment"}, []string{"kind"})},
}}, }},
// The owner of every config value (novox/hq ADR 0348 §9, design 57 K3): its holder alone writes the buckets // The owner of every config value (novox/hq ADR 0348 §9, design 57 K3): its holder alone writes the buckets
// mesh-config_values, _resolved, _predefined and _proposals, and serves the one verb `config` (§1, §10). // mesh-config_values, _resolved, _predefined and _proposals, and serves the one verb `config` (§1, §10).
@@ -258,6 +275,11 @@ var defaultSeats = append([]Seat{
// machine, and a group of them, recorded and ordered by its holder, which asks the controller for each act. // machine, and a group of them, recorded and ordered by its holder, which asks the controller for each act.
// What it says it says as its module (`mesh-delivery.transition`), through the runtime that launches it. // What it says it says as its module (`mesh-delivery.transition`), through the runtime that launches it.
{Name: DeliverySeat, Scope: ScopeMesh, Serves: deliveryVerbs(), Decision: "novox/hq ADR 0239"}, {Name: DeliverySeat, Scope: ScopeMesh, Serves: deliveryVerbs(), Decision: "novox/hq ADR 0239"},
// Placement (novox/hq ADR 0567): which module runs on which node, state on the bus held by the assigner, changed only by
// a proposal the nodes answer and the operator approves. Its states and events are the holder module's.
{Name: PlacementSeat, Scope: ScopeMesh, Serves: placementVerbs(), Decision: "novox/hq ADR 0567"},
// The delivery's checker (novox/hq ADR 0547 §10, issue 774): the try is its verb, not the controller's.
{Name: CheckerSeat, Scope: ScopeMesh, Serves: checkerVerbs(), Decision: "novox/hq ADR 0547"},
// Deferred renames (novox/hq ADR 0121): these deliver a provision, so renaming them is a // Deferred renames (novox/hq ADR 0121): these deliver a provision, so renaming them is a
// delivering-seat migration with a mesh-wide cascade if a holder stops resolving mid-flight. // delivering-seat migration with a mesh-wide cascade if a holder stops resolving mid-flight.
// They keep their names until that migration is done deliberately, apart from the node-* pass. // They keep their names until that migration is done deliberately, apart from the node-* pass.
@@ -439,6 +461,8 @@ var defaultSeats = append([]Seat{
// A node's files, read and managed (novox/hq ADR 0394, files.go): beside node-reads until mesh-files is assigned // A node's files, read and managed (novox/hq ADR 0394, files.go): beside node-reads until mesh-files is assigned
// on every node, then node-reads goes (ADR 0394 §9). // on every node, then node-reads goes (ADR 0394 §9).
{Name: FilesSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0394", Serves: filesVerbs()}, {Name: FilesSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0394", Serves: filesVerbs()},
// The node's engine's own verbs and asks (novox/hq ADR 0563 §1): held by the engine on every node, by no module.
engineSeat(),
}, },
// The graphical session's roles (novox/hq ADR 0208), last because they are a workstation's. // The graphical session's roles (novox/hq ADR 0208), last because they are a workstation's.
graphicalSessionSeats()...) graphicalSessionSeats()...)
@@ -512,7 +536,8 @@ func UseSeats(s []Seat) {
row.Paths = d.Paths row.Paths = d.Paths
row.Replicated = d.Replicated row.Replicated = d.Replicated
row.PulledByKind = d.PulledByKind row.PulledByKind = d.PulledByKind
row.Yields = d.Yields // never stored (ADR 0553) row.Yields = d.Yields // never stored (ADR 0553)
row.HeldByTheEngine = d.HeldByTheEngine // never stored (ADR 0563)
row.Serves = optionalAsCompiled(row.Serves, d.Serves) row.Serves = optionalAsCompiled(row.Serves, d.Serves)
} }
merged = append(merged, row) merged = append(merged, row)
@@ -625,6 +650,13 @@ func claimProblems(m Manifest) []string {
} }
for _, c := range m.Claims { for _, c := range m.Claims {
if s, known := SeatNamed(c.Name); known && s.HeldByTheEngine {
// The node's engine holds it on every node, and no module may (novox/hq ADR 0563 §1): a module answering
// root-free or a hand-over would be an agent answering it.
problems = append(problems, fmt.Sprintf("%s claims %s, which every node's engine holds itself and no "+
"module may (novox/hq ADR 0563 §1)", m.Module, c.Name))
continue
}
if _, known := SeatNamed(c.Name); known { if _, known := SeatNamed(c.Name); known {
// **A seat's scope and what it delivers are not judged here** (novox/hq ADR 0122). // **A seat's scope and what it delivers are not judged here** (novox/hq ADR 0122).
// This function runs wherever a manifest is parsed, and one of those places is the // This function runs wherever a manifest is parsed, and one of those places is the
+46 -2
View File
@@ -1,6 +1,7 @@
package catalogue package catalogue
import ( import (
"reflect"
"regexp" "regexp"
"strings" "strings"
"testing" "testing"
@@ -43,6 +44,48 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
delivered[s.Delivers] = s.Name delivered[s.Delivers] = s.Name
} }
} }
// Forty-seven with mesh-placement (novox/hq ADR 0567), serving its seven verbs.
if s, ok := SeatNamed(PlacementSeat); !ok || s.Scope != ScopeMesh || !reflect.DeepEqual(VerbNames(s.Serves),
[]string{"placement", "proposals", "propose", "withdraw", "again", "adopt", "hand-back"}) {
t.Errorf("the placement seat is %v, %v", ok, s)
}
// The checker's seat serves the try, which replaces the shell's merge check (ADR 0547 §10, ADR 0245).
var checkerVerbs []string
for _, s := range Seats() {
if s.Name == CheckerSeat {
for _, v := range s.Serves {
checkerVerbs = append(checkerVerbs, v.Name)
if v.Name == "try" && strings.Join(v.Replaces, "|") != "sh merge-check.sh|docker run postgres" {
t.Errorf("try replaces %v", v.Replaces)
}
}
}
}
if strings.Join(checkerVerbs, ",") != "try,try-read,try-end" {
t.Errorf("the checker seat serves %v", checkerVerbs)
}
// The node seat node-engine is the node's engine's (novox/hq ADR 0563 §1): at node scope, delivering nothing, its six
// verbs and three events, held by no assignment, and a module's claim of it refused.
if engine, ok := SeatNamed(EngineSeat); !ok || engine.Scope != ScopeNode || engine.Delivers != "" || !engine.HeldByTheEngine {
t.Errorf("node-engine is %+v, not a node seat delivering nothing that the engine holds", engine)
} else {
var verbs []string
for _, v := range engine.Serves {
verbs = append(verbs, v.Name)
}
// acts reads one act by its id too (ADR 0563 §9: the seat's tools are its command line).
if props, _ := engine.Serves[len(engine.Serves)-1].Input["properties"].(map[string]any); props["id"] == nil {
t.Errorf("acts takes no id: %v", engine.Serves[len(engine.Serves)-1].Input)
}
if strings.Join(verbs, ",") != "hand-over,access-act,judge-directories,setuid-search,root-free,acts" ||
strings.Join(engine.Emits, ",") != "asked,done,refused" {
t.Errorf("node-engine serves %v and emits %v", verbs, engine.Emits)
}
}
if _, err := ParseManifest([]byte(`{"module":"thing","version":"1","claims":[{"name":"node-engine","scope":"node"}]}`)); err == nil ||
!strings.Contains(err.Error(), "node-engine") {
t.Errorf("a module's claim of node-engine was not refused: %v", err)
}
// Forty-six with the three build seats replacing node-build-agent and the retired mesh-build-machine (novox/hq // Forty-six with the three build seats replacing node-build-agent and the retired mesh-build-machine (novox/hq
// ADR 0525 §2, §10); forty-five with node-files (novox/hq ADR 0394); forty-four again once node-reads goes. // ADR 0525 §2, §10); forty-five with node-files (novox/hq ADR 0394); forty-four again once node-reads goes.
// Forty-four with node-provisioner (novox/hq ADR 0349 clause 3); forty-three with mesh-provisioner (novox/hq ADR 0349 clause 3); forty-two with mesh-config (novox/hq ADR 0348 §9); forty-one with node-reads (novox/hq ADR 0310); forty with node-nfs-server and node-mounts (novox/hq ADR 0263); thirty-eight with node-resolver (novox/hq ADR 0247); thirty-seven with mesh-delivery (novox/hq ADR 0239); thirty-six since node-resolver-config retired // Forty-four with node-provisioner (novox/hq ADR 0349 clause 3); forty-three with mesh-provisioner (novox/hq ADR 0349 clause 3); forty-two with mesh-config (novox/hq ADR 0348 §9); forty-one with node-reads (novox/hq ADR 0310); forty with node-nfs-server and node-mounts (novox/hq ADR 0263); thirty-eight with node-resolver (novox/hq ADR 0247); thirty-seven with mesh-delivery (novox/hq ADR 0239); thirty-six since node-resolver-config retired
@@ -56,8 +99,9 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
// node-container-runtime (ADR 0207); nineteen with node-environment and node-login-shell (ADR 0203, // node-container-runtime (ADR 0207); nineteen with node-environment and node-login-shell (ADR 0203,
// ADR 0204); seventeen with node-build-agent (ADR 0190). One fewer once the retired // ADR 0204); seventeen with node-build-agent (ADR 0190). One fewer once the retired
// mesh-build-machine row goes, when no registered manifest claims it. // mesh-build-machine row goes, when no registered manifest claims it.
if len(Seats()) != 46 { // Forty-eight with mesh-delivery-checker (novox/hq ADR 0547 §10, issue 774); forty-nine with node-engine (novox/hq ADR 0563 §1).
t.Errorf("the mesh defines %d seats rather than 46; the set is closed, so a change here is "+ if len(Seats()) != 49 {
t.Errorf("the mesh defines %d seats rather than 49; the set is closed, so a change here is "+
"a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames()) "a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames())
} }
} }
+2
View File
@@ -51,6 +51,8 @@ type UnsetSettingError struct {
// where says what asks for it — "a file", "a service whose unit", "a directory <id> whose owner" — and // where says what asks for it — "a file", "a service whose unit", "a directory <id> whose owner" — and
// setToday which of the module's keys a layer sets, for saying it again in the config form's words. // setToday which of the module's keys a layer sets, for saying it again in the config form's words.
where, setToday string where, setToday string
// field is the key of a contribution whose value asks for it, where the contribution's words are made by its caller.
field string
} }
func (e *UnsetSettingError) Error() string { return e.said } func (e *UnsetSettingError) Error() string { return e.said }
+11 -3
View File
@@ -5,6 +5,7 @@ import (
"crypto/x509" "crypto/x509"
"encoding/json" "encoding/json"
"encoding/pem" "encoding/pem"
"errors"
"fmt" "fmt"
"regexp" "regexp"
"sort" "sort"
@@ -129,7 +130,12 @@ func ApplySettings(resource map[string]any, layers []Layer) (map[string]any, err
// contract. Until issue 696, a layer replaced any served key it named; influxdb on ace was the one // contract. Until issue 696, a layer replaced any served key it named; influxdb on ace was the one
// layer relying on it, and serves `${setting:org}` and `${setting:bucket}` since. // layer relying on it, and serves `${setting:org}` and `${setting:bucket}` since.
func Settle(base map[string]any, layers []Layer) (map[string]any, error) { func Settle(base map[string]any, layers []Layer) (map[string]any, error) {
return filledFromSettings(deepCopy(base), layers, "what is served") out, err := filledFromSettings(deepCopy(base), layers, "what is served")
if err != nil {
// Said as it was, untyped: what is served names no module here to type it by.
return nil, errors.New(err.Error())
}
return out, nil
} }
// ServedSettings is every key a module's served values and its zone's name fill from its settings (`${setting:<key>}` // ServedSettings is every key a module's served values and its zone's name fill from its settings (`${setting:<key>}`
@@ -193,10 +199,12 @@ func filledFromSettings(merged map[string]any, layers []Layer, what string) (map
for _, asked := range settingsUsed(s) { for _, asked := range settingsUsed(s) {
v, set := settingValue(layers, asked) v, set := settingValue(layers, asked)
if !set { if !set {
return nil, fmt.Errorf( // Typed, its module and where left to the caller that knows them (Resolution.composed), so a module in
// the config form is refused in its own words (unsetInConfigWords; issue 1126, seed-8).
return nil, &UnsetSettingError{Setting: asked, field: key, setToday: orNoSettings(layers), said: fmt.Sprintf(
"%s says ${setting:%s} for %q, and nothing sets %q — an operator's value is the "+ "%s says ${setting:%s} for %q, and nothing sets %q — an operator's value is the "+
"assignment's, never the definition's (novox/hq ADR 0112)%s", "assignment's, never the definition's (novox/hq ADR 0112)%s",
what, asked, key, asked, orNoSettings(layers)) what, asked, key, asked, orNoSettings(layers))}
} }
s = strings.ReplaceAll(s, "${setting:"+asked+"}", plainly(v)) s = strings.ReplaceAll(s, "${setting:"+asked+"}", plainly(v))
} }
-139
View File
@@ -1,139 +0,0 @@
// placement-seed writes the catalogue's first placement files (novox/hq ADR 0549 §4, ADR 0555 §6, issue 1126) from the
// answer of the controller's one read-only verb `placement-export`: one `placement/<node>.json` per node, each the
// node's placement as the controller's own code read it from its own store, checked as the catalogue's check checks it
// (delivery.CheckPlacements) before anything is written. The pull request it makes merges only when its composition check
// composes every node equal to what it runs.
//
// placement-seed -catalogue <checkout of mesh-catalog> < export.json
// placement-seed -catalogue <dir> -export export.json
//
// export.json is the verb's answer as it prints it ({"<node>": <placement>, ...}, or the verb's wrapping with it under
// "answer"). It opens no network connection, bus or store; it writes only <dir>/placement/<node>.json, and refuses to
// write over a placement file already there. Exit codes: 0 written; 1 the export cannot be read, is refused by the
// check, or a file is there; 2 the flags are wrong.
package main
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func main() {
catalogue := flag.String("catalogue", "", "the checkout of the catalogue to write placement/<node>.json into")
export := flag.String("export", "", "the verb's answer; standard input when absent")
flag.Parse()
if *catalogue == "" || flag.NArg() > 0 {
fmt.Fprintln(os.Stderr, "usage: placement-seed -catalogue <dir> [-export <file>]")
os.Exit(2)
}
in := io.Reader(os.Stdin)
if *export != "" {
f, err := os.Open(*export)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defer f.Close()
in = f
}
written, err := Seed(in, *catalogue)
for _, path := range written {
fmt.Println(path)
}
if err != nil {
fmt.Fprintln(os.Stderr, "placement-seed:", err)
os.Exit(1)
}
}
// Seed reads the export and writes each node's placement file under root, answering the paths written. Nothing is
// written when the export does not read, the check refuses it, or a placement file is already there.
func Seed(export io.Reader, root string) ([]string, error) {
raw, err := io.ReadAll(export)
if err != nil {
return nil, err
}
placements, err := read(raw)
if err != nil {
return nil, err
}
all := func(string) bool { return true }
if refused := delivery.CheckPlacements(placements, all, all); len(refused) > 0 {
return nil, fmt.Errorf("the export is refused by the catalogue's check: %v", refused)
}
nodes := make([]string, 0, len(placements))
for node := range placements {
nodes = append(nodes, node)
}
sort.Strings(nodes)
files := map[string][]byte{}
for _, node := range nodes {
p := placements[node]
p.Revision = "" // only the indexer writes a revision, into the key
if _, err := delivery.EncodePlacement(p); err != nil {
return nil, fmt.Errorf("%s: %w", node, err)
}
p.V = delivery.Version
body, err := json.MarshalIndent(p, "", " ")
if err != nil {
return nil, err
}
path := filepath.Join(root, filepath.FromSlash(delivery.PlacementFile(node)))
if _, err := os.Stat(path); err == nil {
return nil, fmt.Errorf("%s is already there; the seed is written once (ADR 0549 §4)", path)
} else if !errors.Is(err, os.ErrNotExist) {
return nil, err
}
files[path] = append(body, '\n')
}
var written []string
for _, node := range nodes {
path := filepath.Join(root, filepath.FromSlash(delivery.PlacementFile(node)))
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return written, err
}
if err := os.WriteFile(path, files[path], 0o644); err != nil {
return written, err
}
written = append(written, path)
}
return written, nil
}
// read is the export as the verb prints it, or wrapped as a verb's answer (the controller's verbAnswer: its JSON under
// "answer").
func read(raw []byte) (map[string]delivery.Placement, error) {
var wrapped struct {
Answer json.RawMessage `json:"answer"`
}
if json.Unmarshal(raw, &wrapped) == nil && len(wrapped.Answer) > 0 {
raw = wrapped.Answer
}
var byNode map[string]json.RawMessage
if err := json.Unmarshal(raw, &byNode); err != nil {
return nil, fmt.Errorf("the export is not {<node>: <placement>}: %w", err)
}
if len(byNode) == 0 {
return nil, errors.New("the export names no node")
}
out := map[string]delivery.Placement{}
for node, body := range byNode {
p, err := delivery.DecodePlacement(body)
if err != nil {
return nil, fmt.Errorf("%s: %w", node, err)
}
if p.Node != node {
return nil, fmt.Errorf("the export's placement of %s names the node %q", node, p.Node)
}
out[node] = p
}
return out, nil
}
-61
View File
@@ -1,61 +0,0 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Issue 1126 (ADR 0549 §4, ADR 0555 §6): the seed writes one placement file per node of the export, as the catalogue's
// check and the indexer read it, from the verb's answer bare or wrapped; it writes nothing for an export the check
// refuses, and never over a file already there.
func TestTheSeedWritesEachNodesPlacementFileOnce(t *testing.T) {
export := `{"answer":{"ace":{"v":1,"node":"ace","revision":"x","decided":{"address":"10.77.0.2"},"modules":[{"module":"plex","ports":{"32400":32400}}]},` +
`"novox":{"v":1,"node":"novox","decided":{"hub":true,"address":"10.77.0.1"},"modules":[]}}}`
root := t.TempDir()
written, err := Seed(strings.NewReader(export), root)
if err != nil || len(written) != 2 {
t.Fatalf("written %v, %v", written, err)
}
raw, err := os.ReadFile(filepath.Join(root, "placement", "ace.json"))
if err != nil {
t.Fatal(err)
}
p, err := delivery.DecodePlacement(raw)
if err != nil || p.Node != "ace" || p.Revision != "" || p.HostPorts()["plex"][32400] != 32400 {
t.Errorf("ace's file reads %+v, %v", p, err)
}
if _, err := Seed(strings.NewReader(export), root); err == nil || !strings.Contains(err.Error(), "already there") {
t.Errorf("a second seed over the files: %v", err)
}
clash := `{"ace":{"v":1,"node":"ace","decided":{"address":"10.77.0.1"},"modules":[]},"novox":{"v":1,"node":"novox","decided":{"address":"10.77.0.1"},"modules":[]}}`
other := t.TempDir()
if _, err := Seed(strings.NewReader(clash), other); err == nil || !strings.Contains(err.Error(), "both have the address") {
t.Errorf("an export the check refuses: %v", err)
}
if entries, _ := os.ReadDir(filepath.Join(other, "placement")); len(entries) != 0 {
t.Errorf("a refused export wrote %v", entries)
}
// What the controller's placement-export prints, frombus.Export per node, is what the seed takes: a node with no
// module placed included.
in := compose.Input{Nodes: []compose.Node{{Name: "solo", Place: compose.Place{Hub: true, Address: "10.77.0.9"}}}}
exported, err := frombus.Export(in, "solo")
if err != nil {
t.Fatal(err)
}
answer, _ := json.Marshal(map[string]any{"answer": map[string]delivery.Placement{"solo": exported}})
if written, err := Seed(strings.NewReader(string(answer)), t.TempDir()); err != nil || len(written) != 1 {
t.Errorf("Export's answer %s was not seeded: %v, %v", answer, written, err)
}
for _, bad := range []string{`[]`, `{}`, `{"ace":{"v":1,"node":"novox","modules":[]}}`} {
if _, err := Seed(strings.NewReader(bad), t.TempDir()); err == nil {
t.Errorf("%s was seeded", bad)
}
}
}
+242 -6
View File
@@ -55,11 +55,19 @@ func CompositionsShadow(in Input, node string, fps map[string]string) (out map[s
return compositions(in, node, fps, Foreseeing, false) return compositions(in, node, fps, Foreseeing, false)
} }
// CompositionsIn is the node's compositions in mode with what they stand in for answered (stood, sorted, the bus's user
// list left out) rather than refused: CompositionsShadow in Foreseeing, and in Reading what CompositionsApplied refuses
// before it is refused, so a caller that applies refuses each stand-in to the module it is for alone (issue 1126).
func CompositionsIn(in Input, node string, fps map[string]string, mode Mode) (out map[string]delivery.Composed, skipped, stood []string, err error) {
return compositions(in, node, fps, mode, false)
}
func compositions(in Input, node string, fps map[string]string, mode Mode, applied bool) (out map[string]delivery.Composed, skipped, stood []string, err error) { func compositions(in Input, node string, fps map[string]string, mode Mode, applied bool) (out map[string]delivery.Composed, skipped, stood []string, err error) {
reading := in reading := in
reading.Nodes = append([]Node(nil), in.Nodes...) reading.Nodes = append([]Node(nil), in.Nodes...)
found := false found := false
inOwn := map[string]bool{} inOwn := map[string]bool{}
held := map[string]bool{}
for i := range reading.Nodes { for i := range reading.Nodes {
if reading.Nodes[i].Name == node { if reading.Nodes[i].Name == node {
reading.Nodes[i].ReadsModules = true reading.Nodes[i].ReadsModules = true
@@ -67,6 +75,9 @@ func compositions(in Input, node string, fps map[string]string, mode Mode, appli
for m, ok := range reading.Nodes[i].InOwn { for m, ok := range reading.Nodes[i].InOwn {
inOwn[m] = ok inOwn[m] = ok
} }
for m, ok := range reading.Nodes[i].Held {
held[m] = ok
}
} }
} }
// The private network's module is built by nobody: no fingerprint of the new form ever names it. // The private network's module is built by nobody: no fingerprint of the new form ever names it.
@@ -77,8 +88,9 @@ func compositions(in Input, node string, fps map[string]string, mode Mode, appli
if !found { if !found {
return nil, nil, nil, fmt.Errorf("compose: the mesh has no node %q", node) return nil, nil, nil, fmt.Errorf("compose: the mesh has no node %q", node)
} }
// A folded module built by a build of the old form (Node.RunsOwn) is left to the node: composed as a stub, so what // A folded module built by a build of the old form (Node.RunsOwn) is left to the node: composed from what runs of it
// it provides and requires still shapes the rest, and its resources are then the ones the node runs. // (Carried), so what it provides and requires still shapes the rest and what the others read of it (its published
// ports, the paths it owns) is what runs, and its resources are then the ones the node runs.
carried := map[string]bool{} carried := map[string]bool{}
var runsOwn *delivery.Composed var runsOwn *delivery.Composed
for i := range reading.Nodes { for i := range reading.Nodes {
@@ -87,15 +99,26 @@ func compositions(in Input, node string, fps map[string]string, mode Mode, appli
continue continue
} }
runsOwn = n.RunsOwn runsOwn = n.RunsOwn
candidates := map[string]bool{}
for m := range inOwn { for m := range inOwn {
if !folds(m) { candidates[m] = true
}
for m := range held {
candidates[m] = true
}
for m := range candidates {
// A held module is carried as it runs wherever the node's own runs it, folded or leaving the fold for a
// composition of its own (its want new, its names still unset): its stub is never what the node's own applies,
// and what the node's own applies never drops what runs of it (issue 1126, review of the apply; issue 1945).
heldHere := held[m] && RunsAnyOf(runsOwn, m)
if !folds(m) && !heldHere {
continue continue
} }
man, ok := n.Kept[m] man, ok := n.Kept[m]
if !ok { if !ok {
man, ok = reading.Catalogue[m] man, ok = reading.Catalogue[m]
} }
if !ok || !NamesBuild(man) { if !ok || (!NamesBuild(man) && !held[m]) {
continue continue
} }
carried[m] = true carried[m] = true
@@ -103,10 +126,26 @@ func compositions(in Input, node string, fps map[string]string, mode Mode, appli
for k, v := range n.Kept { for k, v := range n.Kept {
kept[k] = v kept[k] = v
} }
kept[m] = Stub(man) kept[m] = Carried(man, m, runsOwn)
n.Kept = kept n.Kept = kept
} }
} }
// A module composed as what runs of it contributes to the runtime what the runtime serves of it as it runs
// (Node.RunsTools); one of its own account is served by a runtime of its own, from its manifest as before.
for i := range reading.Nodes {
n := &reading.Nodes[i]
if n.Name != node || n.RunsTools == nil {
continue
}
kept := make(map[string]catalogue.Manifest, len(n.Kept))
for m, man := range n.Kept {
if man.Stubbed && man.RunsAs == "" {
man.Bundles = runningBundles(n.RunsTools[m], reading.Catalogue[m])
}
kept[m] = man
}
n.Kept = kept
}
isCarried := func(module, id string) bool { isCarried := func(module, id string) bool {
if module == "" { if module == "" {
module, _ = declaration.ModuleGuess(id) module, _ = declaration.ModuleGuess(id)
@@ -230,6 +269,10 @@ func compositions(in Input, node string, fps map[string]string, mode Mode, appli
out[own] = c out[own] = c
} }
for key, c := range out { for key, c := range out {
if held[c.Module] {
delete(out, key) // its stub's: the node keeps what it last applied of it (Node.Held)
continue
}
if m, ok := in.Catalogue[c.Module]; ok && c.Module != "" { if m, ok := in.Catalogue[c.Module]; ok && c.Module != "" {
c.Data = len(m.DataItems()) > 0 c.Data = len(m.DataItems()) > 0
out[key] = c out[key] = c
@@ -242,6 +285,130 @@ func compositions(in Input, node string, fps map[string]string, mode Mode, appli
return out, skipped, stood, nil return out, skipped, stood, nil
} }
// RunsAnyOf says runs (a composition as it runs) holds a resource of module, by its module or the module its id names.
func RunsAnyOf(runs *delivery.Composed, module string) bool {
if runs == nil {
return false
}
for _, raw := range runs.Resources {
var head struct {
ID string `json:"id"`
Module string `json:"module"`
}
if json.Unmarshal(raw, &head) != nil {
continue
}
owner := head.Module
if owner == "" {
owner, _ = declaration.ModuleGuess(head.ID)
}
if owner == module {
return true
}
}
return false
}
// ToolsRunning is, by module, the tools bundles that run on the node (Node.RunsTools): each bundle archive a module's
// resource unpacks at its BundlePath, with what the runtime's own process loads of it and the words it hands it
// (catalogue.RuntimeToolModules, catalogue.RuntimeToolEnv, the mesh's own words left out). A bundle the runtime loads
// nothing of has no Loads: the runtime dropped it, and the composition reads its loads again from the module's build
// (issue 1126, review: shanks had already applied the drop of systemd-resolved). Nil when running holds no runtime process.
func ToolsRunning(running []delivery.Composed) map[string][]catalogue.Bundle {
runtimeID := catalogue.RuntimeModule + "." + catalogue.RuntimeProcessID()
type resource struct {
ID string `json:"id"`
Module string `json:"module"`
Type string `json:"type"`
Path string `json:"path"`
Source string `json:"source"`
Digest string `json:"digest"`
Env map[string]string `json:"env"`
}
var env map[string]string
out := map[string][]catalogue.Bundle{}
for _, c := range running {
for _, raw := range c.Resources {
var r resource
if json.Unmarshal(raw, &r) != nil {
continue
}
owner := r.Module
if owner == "" {
owner, _ = declaration.ModuleGuess(r.ID)
}
switch {
case r.Type == "process" && r.ID == runtimeID && owner == catalogue.RuntimeModule:
env = r.Env
case r.Type == "archive" && owner != "" && strings.HasPrefix(r.ID, owner+"."+catalogue.BundleID("")):
name := strings.TrimPrefix(r.ID, owner+"."+catalogue.BundleID(""))
if r.Path == catalogue.BundlePath(owner, name) {
out[owner] = append(out[owner], catalogue.Bundle{Name: name, Source: catalogue.Recorded(r.Source), Digest: r.Digest})
}
}
}
}
if env == nil {
return nil
}
var words map[string]map[string]string
_ = json.Unmarshal([]byte(env[catalogue.RuntimeToolEnv]), &words)
for _, entry := range strings.Split(env[catalogue.RuntimeToolModules], ",") {
module, path, ok := strings.Cut(entry, "=")
if !ok || !strings.HasPrefix(path, catalogue.BundleRoot+"/"+module+"/") {
continue
}
name, load, ok := strings.Cut(strings.TrimPrefix(path, catalogue.BundleRoot+"/"+module+"/"), "/")
bundles := out[module]
i := slices.IndexFunc(bundles, func(b catalogue.Bundle) bool { return b.Name == name })
if !ok || i < 0 {
continue // no archive of it runs: nothing to load
}
var own map[string]string
for w, v := range words[module] {
if w == catalogue.FilesOwnedPaths || w == catalogue.ReadsSecretPlaces {
continue // the mesh's to say, composed again
}
if own == nil {
own = map[string]string{}
}
own[w] = v
}
bundles[i].Loads = append(bundles[i].Loads, load)
bundles[i].Env = own
}
for _, bundles := range out {
sort.Slice(bundles, func(i, j int) bool { return bundles[i].Name < bundles[j].Name })
}
return out
}
// runningBundles is the bundles of a module composed as what runs of it (Node.RunsTools): each that runs, with what the
// runtime loads of it and its words; one the runtime loads nothing of takes its loads and words from the module's build
// (man: its built bundles, else its build's artifacts, catalogue.BundleLoads), so a drop already applied is undone.
func runningBundles(runs []catalogue.Bundle, man catalogue.Manifest) []catalogue.Bundle {
var out []catalogue.Bundle
for _, b := range runs {
if len(b.Loads) == 0 {
for _, built := range man.Bundles {
if built.Name == b.Name {
b.Loads, b.Env, b.Launchers, b.Binary = built.Loads, built.Env, built.Launchers, built.Binary
}
}
if man.Build != nil {
run := catalogue.RunByAResource(man)
for _, a := range man.Build.Artifacts {
if a.Kind == catalogue.ArtifactBundle && a.Name == b.Name {
b.Loads, b.Env, b.Binary = catalogue.BundleLoads(man, a, run), a.Env, catalogue.BinaryOf(a)
}
}
}
}
out = append(out, b)
}
return out
}
// NamesBuild says a manifest names what a build makes: its build, or a resource naming an artifact. A node cannot // NamesBuild says a manifest names what a build makes: its build, or a resource naming an artifact. A node cannot
// compose such a module folded into its own composition: the build of the old form it runs is held by the controller // compose such a module folded into its own composition: the build of the old form it runs is held by the controller
// alone (issue 1126). // alone (issue 1126).
@@ -259,9 +426,25 @@ func NamesBuild(m catalogue.Manifest) bool {
// Stub is m with nothing to build and no resource but its directories: what it provides, requires, serves and grants // Stub is m with nothing to build and no resource but its directories: what it provides, requires, serves and grants
// still shape the rest of the composition, the places its own secrets, data and grants name (${dir:…}) still resolve, // still shape the rest of the composition, the places its own secrets, data and grants name (${dir:…}) still resolve,
// and nothing else of its own is composed (issue 1126: nats's ${dir:mesh-state} on novox). // and nothing else of its own is composed (issue 1126: nats's ${dir:mesh-state} on novox). A module refused in a
// node's answer is composed so; a folded one the node runs is Carried.
func Stub(m catalogue.Manifest) catalogue.Manifest { func Stub(m catalogue.Manifest) catalogue.Manifest {
// The paths it owns stay its manifest's, as the controller tells the holder of node-files them (issue 1126, seed-8).
if m.OwnsPaths == nil {
m.OwnsPaths = catalogue.DeclaredPaths(m)
}
m.Build = nil m.Build = nil
// What it hands the provisioner is left out as it runs, not refused for want of a plan (catalogue.Manifest.Stubbed).
m.Stubbed = true
// Nor is a template its build kept for its own files handed: a stub carries none of them (issue 1126: novox's node
// refused "postgres: its build kept a template for consumers, which it does not declare" with postgres held). Its
// items stay, and the templates they render through: a held module's contributions to other modules' collections
// (seat contributions, shell blocks, seat data) still reach their collectors.
if m.Handed != nil {
h := *m.Handed
h.Templates = nil
m.Handed = &h
}
var dirs []map[string]any var dirs []map[string]any
for _, r := range m.Resources { for _, r := range m.Resources {
if r["type"] == "directory" { if r["type"] == "directory" {
@@ -272,6 +455,59 @@ func Stub(m catalogue.Manifest) catalogue.Manifest {
return m return m
} }
// Carried is m, folded into its node's own composition at a build of the old form, as the node runs it (issue 1126,
// C5): nothing to build; its containers and directories those the node's own composition (runs) carries of it, their
// module's word taken off their ids, with the manifest's directories it does not run. So what other modules read of it
// is what runs: the machine ports its containers publish (the filter's rules, the routes reaching it: MachineSide). The
// paths it owns, which the holder of node-files is told, are its manifest's, as for its Stub: the controller composes
// them from the manifest, not from every path a resource of it runs at (issue 1126, seed-8: f1-circuits, snake and
// txt-game on ace, whose manifests leave their directories to the mesh). Its other resources are not taken, since the
// composition makes some of them again from its manifest (its secrets' files) and one id twice is refused; none of them
// is composed into what is applied either way (compositions leaves a carried module's out, and carries the ones that
// run). What it provides, requires, listens on and grants is its manifest's. A module whose resources runs holds none
// of is its Stub, as before.
func Carried(m catalogue.Manifest, module string, runs *delivery.Composed) catalogue.Manifest {
if runs == nil {
return Stub(m)
}
var taken []map[string]any
runsAny := false
ids := map[string]bool{}
for _, raw := range runs.Resources {
var r map[string]any
if json.Unmarshal(raw, &r) != nil {
continue
}
id, _ := r["id"].(string)
owner, _ := r["module"].(string)
if owner == "" {
owner, _ = declaration.ModuleGuess(id)
}
if owner != module {
continue
}
runsAny = true
if typ := r["type"]; typ != "container" && typ != "directory" {
continue
}
r["id"] = strings.TrimPrefix(id, module+".")
delete(r, "module")
ids[r["id"].(string)] = true
taken = append(taken, r)
}
if !runsAny {
return Stub(m)
}
stub := Stub(m)
for _, r := range stub.Resources {
if id, _ := r["id"].(string); !ids[id] {
taken = append(taken, r)
}
}
stub.Resources = taken
return stub
}
// carry puts into the node's resources (kept, in the declaration's order) the resources of the node's own composition as // carry puts into the node's resources (kept, in the declaration's order) the resources of the node's own composition as
// it runs (runs) that belong to a carried module, each after the resource it runs after, or first when that one is // it runs (runs) that belong to a carried module, each after the resource it runs after, or first when that one is
// not among them; a file among them is a folded path. // not among them; a file among them is a folded path.
+86
View File
@@ -8,6 +8,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
) )
func fpFor(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) } func fpFor(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
@@ -96,6 +97,91 @@ func TestCompositionsAreTheDeclarationSplitByModule(t *testing.T) {
if _, _, err := Compositions(in, "nowhere", fps, Foreseeing); err == nil { if _, _, err := Compositions(in, "nowhere", fps, Foreseeing); err == nil {
t.Errorf("a node the mesh does not have was composed") t.Errorf("a node the mesh does not have was composed")
} }
// Issue 1126, 2026-10-29: a module composed as what runs of it (folded at a build of the old form, or held) still
// contributes to the runtime what the runtime serves of it as it runs (Node.RunsTools): its manifest at the
// indexer's commit has no bundles (shanks's runtime lost the folded systemd-resolved, novox's the folded nats), and a
// held module's new build's bundles are not delivered.
dg := func(c byte) string { return "sha256:" + strings.Repeat(string(c), 64) }
runtime := catalogue.Manifest{Module: catalogue.RuntimeModule, Version: "1",
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/var/lib/mesh/node-tools/broker"}},
Bundles: []catalogue.Bundle{{Name: "runtime", Source: catalogue.ArtifactStoreScheme + "node-tools/runtime/blobs/" + dg('a'),
Digest: dg('a'), Language: "go", Binary: "node-tools"}}}
store := catalogue.Manifest{Module: "artifacts", Version: "1",
Serves: map[string]map[string]any{catalogue.ArtifactStoreProvision: {"port": 5100}}}
lamp := catalogue.Manifest{Module: "lamp", Version: "1", Tools: []string{"on"},
Resources: []map[string]any{{"id": "state", "type": "directory", "path": "/var/lib/lamp", "mode": "0700"}},
Build: &catalogue.Build{Artifacts: []catalogue.Artifact{{Name: "tools", Kind: catalogue.ArtifactBundle,
Language: "go", System: "arch", From: "cmd/lamp-tools", Env: map[string]string{"LAMP_CONF": "/var/lib/lamp/conf"}}}}}
newLamp, err := lamp.Resolve([]catalogue.Built{{Name: "tools", Kind: catalogue.ArtifactBundle,
Reference: catalogue.ArtifactStoreScheme + "lamp/tools/blobs/" + dg('c'), Digest: dg('c')}})
if err != nil {
t.Fatal(err)
}
served := "lamp=" + catalogue.BundlePath("lamp", "tools") + "/lamp-tools"
runtimeRuns := func(dropped bool) delivery.Composed {
env := `"MESH_TOOL_MODULES":"` + served + `","MESH_TOOL_ENV":"{\"lamp\":{\"LAMP_CONF\":\"/var/lib/lamp/conf\"}}"`
if dropped {
env = `"MESH_TOOL_MODULES":""` // the drop already applied (shanks, 2026-10-29)
}
return delivery.Composed{Node: "anchor", Module: catalogue.RuntimeModule, Resources: []json.RawMessage{json.RawMessage(
`{"id":"node-tools.runtime","module":"node-tools","type":"process","name":"node-tools","env":{` + env + `}}`)}}
}
lampRuns := `{"id":"lamp.bundle-tools","type":"archive","source":"` + catalogue.ArtifactStoreScheme + `lamp/tools/blobs/` + dg('b') +
`","digest":"` + dg('b') + `","path":"` + catalogue.BundlePath("lamp", "tools") + `"}`
for _, row := range []struct {
name string
set func(n *Node) delivery.Composed // and what runs of lamp
fps map[string]string
dropped bool
}{
{"folded at a build of the old form", func(n *Node) delivery.Composed {
n.InOwn = map[string]bool{"lamp": true}
n.RunsOwn = &delivery.Composed{Node: "anchor", Resources: []json.RawMessage{json.RawMessage(lampRuns)}}
return *n.RunsOwn
}, map[string]string{}, false},
{"folded, the runtime already dropped it", func(n *Node) delivery.Composed {
n.InOwn = map[string]bool{"lamp": true}
n.RunsOwn = &delivery.Composed{Node: "anchor", Resources: []json.RawMessage{json.RawMessage(lampRuns)}}
return *n.RunsOwn
}, map[string]string{}, true},
{"held, its new build not delivered", func(n *Node) delivery.Composed {
n.Held = map[string]bool{"lamp": true}
lr := delivery.Composed{Node: "anchor", Module: "lamp", Resources: []json.RawMessage{json.RawMessage(
strings.Replace(lampRuns, `"type"`, `"module":"lamp","type"`, 1))}}
n.Kept = map[string]catalogue.Manifest{"lamp": Carried(newLamp, "lamp", &lr)}
return lr
}, map[string]string{"lamp": fpFor('1')}, false},
} {
tools := aMesh(t)
tools.Catalogue[catalogue.RuntimeModule], tools.Catalogue["artifacts"], tools.Catalogue["lamp"] = runtime, store, lamp
anchor := nodeOf(t, &tools, "anchor")
anchor.Assigned = append(anchor.Assigned, "artifacts", catalogue.RuntimeModule, "lamp")
tools.BusMinted = []string{"anchor." + catalogue.RuntimeModule}
anchor.SealingKey = keyOf(t, "anchor")
anchor.RunsTools = ToolsRunning([]delivery.Composed{runtimeRuns(row.dropped), row.set(anchor)})
row.fps[overlay.Name], row.fps["artifacts"], row.fps[catalogue.RuntimeModule] = fpFor('a'), fpFor('5'), fpFor('e')
out, _, _, err := CompositionsIn(tools, "anchor", row.fps, Foreseeing)
if err != nil {
t.Fatalf("%s: %v", row.name, err)
}
var env map[string]any
for _, c := range out {
for _, raw := range c.Resources {
var r struct {
ID string `json:"id"`
Env map[string]any `json:"env"`
}
if json.Unmarshal(raw, &r) == nil && r.ID == "node-tools.runtime" {
env = r.Env
}
}
}
if env[catalogue.RuntimeToolModules] != served || env[catalogue.RuntimeToolEnv] != `{"lamp":{"LAMP_CONF":"/var/lib/lamp/conf"}}` {
t.Errorf("%s: the runtime serves %v with %v, want what runs of lamp, %s", row.name,
env[catalogue.RuntimeToolModules], env[catalogue.RuntimeToolEnv], served)
}
}
} }
// 1943/AC1: the upgrade policies and sources a composition's value may name are the catalogue's own words. // 1943/AC1: the upgrade policies and sources a composition's value may name are the catalogue's own words.
+295 -34
View File
@@ -13,18 +13,18 @@ import (
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
// Answer is a node's answer to its group's placement candidates (novox/hq ADR 0559 §4): s.Node composed in shadow with // Answer is a node's answer to its proposal's placement candidates (novox/hq ADR 0567 §3, ADR 0559 §4): s.Node composed in shadow with
// s.Placements laid over the trunk's placement (compose.CompositionsShadow, its own sealed values and boxes through r as // s.Placements laid over the trunk's placement (compose.CompositionsShadow, its own sealed values and boxes through r as
// for its own composition), compared with what it runs (delivery.CompareComposed). It applies nothing and writes // for its own composition), compared with what it runs (delivery.CompareComposed). It applies nothing and writes
// nothing; head and revision are the candidate's, said back in the answer. // nothing; the proposal and the revision are the candidate's, said back in the answer.
// //
// The node's own composition is compared only where the node runs one. fps gives the build each module is composed at: what the node runs, and for a module the candidate places that the // The node's own composition is compared only where the node runs one. fps gives the build each module is composed at: what the node runs, and for a module the candidate places that the
// node does not run, the build the trunk holds; a placed module with none is added, with no build of it known. What the // node does not run, the build the trunk holds; a placed module with none is added, with no build of it known. What the
// composition stands in for is judged: a value of a module the group adds (on this node or another: its credential, own // composition stands in for is judged: a value of a module the proposal adds (on this node or another: its credential, own
// secrets and boxes are made only once it is placed, ADR 0549 §5) is not yet issued, which is no refusal; any other // secrets and boxes are made only once it is placed, ADR 0549 §5) is not yet issued, which is no refusal; any other
// stand-in is a value missing for a module the node runs, a refusal naming it ("node" when it is the node's own). // stand-in is a value missing for a module the node runs, a refusal naming it ("node" when it is the node's own).
func Answer(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]string, running []delivery.Composed, head string, revision uint64) delivery.CandidateAnswer { func Answer(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]string, running []delivery.Composed, proposal string, revision uint64) delivery.CandidateAnswer {
a := delivery.CandidateAnswer{Head: head, Revision: revision} a := delivery.CandidateAnswer{Proposal: proposal, Revision: revision}
refuse := func(err error) delivery.CandidateAnswer { refuse := func(err error) delivery.CandidateAnswer {
a.Refused = err.Error() a.Refused = err.Error()
a.Verdict = delivery.CandidateRefused a.Verdict = delivery.CandidateRefused
@@ -70,21 +70,14 @@ func Answer(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]st
return refuse(fmt.Errorf("frombus: the credentials of %s: %w", s.Node, err)) return refuse(fmt.Errorf("frombus: the credentials of %s: %w", s.Node, err))
} }
} }
composed, skipped, stood, err := compose.CompositionsShadow(in, s.Node, fps) composed, skipped, stood, err := composeEach(in, s.Node, fps, false, s.Running, failed)
for err != nil { if err != nil {
module := culprit(in, s.Node, fps, failed, err) // Not one module's: the node's own (a value served elsewhere, the mesh's network), said once as the node's.
if module == "" { if len(failed) == 0 {
// Not one module's: the node's own (a value served elsewhere, the mesh's network), said once as the node's. return refuse(err)
if len(failed) == 0 {
return refuse(err)
}
failed[delivery.HaveNodeWord] = err.Error()
composed, skipped, stood = nil, nil, nil
break
} }
failed[module] = err.Error() failed[delivery.HaveNodeWord] = err.Error()
in = stubbed(in, s.Node, module) composed, skipped, stood = nil, nil, nil
composed, skipped, stood, err = compose.CompositionsShadow(in, s.Node, fps)
} }
// The node's own composition is judged only where the node runs one: one that runs none applies it only once a want // The node's own composition is judged only where the node runs one: one that runs none applies it only once a want
// names it (the node agent's own composer hands it on that want alone), so a candidate does not add it. // names it (the node agent's own composer hands it on that want alone), so a candidate does not add it.
@@ -104,6 +97,7 @@ func Answer(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]st
for _, m := range sortedKeys(failed) { for _, m := range sortedKeys(failed) {
a.Refusals = append(a.Refusals, delivery.ModuleRefused{Module: m, Why: failed[m]}) a.Refusals = append(a.Refusals, delivery.ModuleRefused{Module: m, Why: failed[m]})
} }
heldAsToday(&a, s.HeldToday)
a.Verdict = delivery.VerdictOf(a) a.Verdict = delivery.VerdictOf(a)
return a return a
} }
@@ -143,7 +137,7 @@ func Answer(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]st
cmp.Added = slices.DeleteFunc(cmp.Added, func(d delivery.ModuleAdded) bool { return !notFailed(d.Module) }) cmp.Added = slices.DeleteFunc(cmp.Added, func(d delivery.ModuleAdded) bool { return !notFailed(d.Module) })
skipped = slices.DeleteFunc(skipped, func(m string) bool { return !notFailed(m) }) skipped = slices.DeleteFunc(skipped, func(m string) bool { return !notFailed(m) })
// What the group adds, by node and module: placed by a candidate and not by the trunk on that node. // What the proposal adds, by node and module: placed by a candidate and not by the trunk on that node.
added := map[string]bool{} added := map[string]bool{}
for node, p := range s.Placements { for node, p := range s.Placements {
if p == nil { if p == nil {
@@ -208,10 +202,69 @@ func Answer(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]st
a.Refusals = append(a.Refusals, delivery.ModuleRefused{Module: m, Why: failed[m]}) a.Refusals = append(a.Refusals, delivery.ModuleRefused{Module: m, Why: failed[m]})
} }
sort.Slice(a.Refusals, func(i, j int) bool { return a.Refusals[i].Module < a.Refusals[j].Module }) sort.Slice(a.Refusals, func(i, j int) bool { return a.Refusals[i].Module < a.Refusals[j].Module })
heldAsToday(&a, s.HeldToday)
a.Verdict = delivery.VerdictOf(a) a.Verdict = delivery.VerdictOf(a)
return a return a
} }
// heldBehind is why a module is held with a held one it reads (composeEach's dependents).
func heldBehind(m string) string {
return "it reads what " + m + " contributes or listens on, and " + m + " is held"
}
// behind is the held module a refusal's words say it is held behind, or "".
func behind(why string) string {
rest, ok := strings.CutPrefix(why, "it reads what ")
if !ok {
return ""
}
m, _, ok := strings.Cut(rest, " contributes or listens on, and ")
if !ok || why != heldBehind(m) {
return ""
}
return m
}
// heldAsToday moves into Held each refusal of a module the node keeps today as it runs (Scope.HeldToday, with how:
// held, failed, running a build of the old form), and of each module held only behind one of those: the candidate keeps
// them as the node does, which is no change (issue 1126: a seed reproduces today, and today keeps them so). Each says
// how it stands today (ModuleRefused.Today).
func heldAsToday(a *delivery.CandidateAnswer, today map[string]string) {
if len(today) == 0 {
return
}
how := map[string]string{}
for m, w := range today {
how[m] = w
}
for moved := true; moved; {
moved = false
kept := a.Refusals[:0]
for _, r := range a.Refusals {
w, keep := how[r.Module]
if !keep {
if m := behind(r.Why); m != "" {
if _, behindKept := how[m]; behindKept {
w, keep = "held behind "+m, true
}
}
}
if keep {
r.Today = w
how[r.Module] = w
a.Held = append(a.Held, r)
moved = true
} else {
kept = append(kept, r)
}
}
a.Refusals = kept
}
if len(a.Refusals) == 0 {
a.Refusals = nil
}
}
// maxCulprits bounds the modules one refusal is tried on, each a whole composition. // maxCulprits bounds the modules one refusal is tried on, each a whole composition.
const maxCulprits = 8 const maxCulprits = 8
@@ -244,10 +297,177 @@ func standsInFor(x, node string) (string, string) {
return node, "" return node, ""
} }
// composeEach is the one rule, for the node that applies and for its answer to a candidate alike, by which a module's
// own refusal refuses that module and never the node (issue 1126: seed-8 fixed it in the answer alone, and the
// applying path still refused all of novox for de-spiegel's unset route names). The node is composed; a refusal is laid
// on the module whose stub (compose.Stub) changes it (culprit), that module is held (compose.Node.Held) and said in
// failed with the refusal's words, and the rest is composed again, until it composes or the refusal is no single
// module's, which is answered as the node's (err). Applied composes as the node applies (compose.Reading) and holds the
// module placed here each stand-in is for; one for no module placed here (the node's own, a value this node holds for a
// module elsewhere) refuses the node (compose.ErrStoodIn). Not applied, it composes in compose.Foreseeing and answers
// every stand-in for the caller to judge (Answer).
//
// **A held module is what runs of it** (review of the apply, round 1): composed from what running holds of it
// (compose.Carried of its own composition there; the node's own carries it wherever it runs there), so what the others
// read of it (the machine ports its containers publish, the routes reaching it) is what runs, and none of its own
// composition is in out: the node keeps what it last applied. **A held module is no dependence of its neighbours**
// (issue 1126, the spreading hold): a consumer of what it provides, or a collector of its items, composes against what
// runs of it like any other module, and is never held for how the held one would compose as its stub (that comparison
// held every postgres consumer behind de-spiegel's hold on novox, three rounds in a row). What it composes is compared
// with what runs as any module's is: a real difference is still named, and a refusal of its own still refuses it. Only
// what a held module's new manifest says to others, which may not be the build's that runs, is held with it
// (dependents): the providers of what it contributes to in their own files (the proxy its routes), and, when it
// listens, the filter; its items reach their collectors through its own plan, which it keeps as it runs.
func composeEach(in compose.Input, node string, fps map[string]string, applied bool, running []delivery.Composed, failed map[string]string) (out map[string]delivery.Composed, skipped, stood []string, err error) {
mode := compose.Foreseeing
if applied {
mode = compose.Reading
}
runs := map[string]*delivery.Composed{}
for i := range running {
if m := running[i].Module; m != "" {
runs[m] = &running[i]
}
}
src := map[string]catalogue.Manifest{} // each held module's manifest before it was held
bareOf := map[string]bool{}
var hold func(m, why string, bare bool)
hold = func(m, why string, bare bool) {
if _, done := failed[m]; !done {
failed[m] = why // a held module tried bare keeps the words it was first held for
}
first := false
if _, ok := src[m]; !ok {
man, ok := manifestOn(in, node, m)
if !ok {
return
}
src[m], first = man, true
}
bareOf[m] = bareOf[m] || bare
in = heldAs(in, node, m, src[m], bareOf[m], runs[m])
if !first {
return
}
// What the others read of its manifest (what it contributes, what it listens on) is its new manifest's, not
// necessarily the build's that runs: what reads it is held with it (review of the apply, round 2).
for _, r := range dependents(in, node, src[m], failed) {
hold(r, heldBehind(m), false)
}
}
for _, m := range sortedKeys(failed) {
hold(m, failed[m], false)
}
for {
out, skipped, stood, err = compose.CompositionsIn(in, node, fps, mode)
if err != nil {
module, bare := culprit(in, node, fps, mode, failed, err)
if module == "" {
return nil, nil, nil, err
}
hold(module, err.Error(), bare)
continue
}
again := false
if applied && len(stood) > 0 {
placed := map[string]bool{}
for _, m := range assignedOn(in, node) {
placed[m] = true
}
of := map[string][]string{}
var nodes []string
for _, x := range stood {
if n, m := standsInFor(x, node); n == node && placed[m] {
of[m] = append(of[m], x)
} else {
nodes = append(nodes, x)
}
}
for _, m := range sortedKeys(of) {
if _, done := failed[m]; !done {
hold(m, "stands in for "+strings.Join(of[m], ", "), false)
again = true
}
}
if !again && len(nodes) > 0 {
return nil, nil, nil, fmt.Errorf("%w on %s: %s", compose.ErrStoodIn, node, strings.Join(nodes, ", "))
}
}
if !again {
break
}
}
skipped = slices.DeleteFunc(skipped, func(m string) bool { _, held := failed[m]; return held })
return out, skipped, stood, nil
}
// dependents is, of the modules placed on node and not held, those that read what man says to them: each that provides
// what man contributes to (the proxy its routes), and, when man listens on a port, each that holds the filter (the
// openings).
func dependents(in compose.Input, node string, man catalogue.Manifest, failed map[string]string) []string {
to := map[string]bool{}
for p := range man.Contributes {
to[p] = true
}
for p := range man.ContributesMany {
to[p] = true
}
listens := len(man.Listens) > 0
if len(to) == 0 && !listens {
return nil
}
var out []string
for _, m := range assignedOn(in, node) {
if _, held := failed[m]; held || m == man.Module {
continue
}
pm, ok := manifestOn(in, node, m)
if !ok {
continue
}
if listens && pm.Filtering != nil {
out = append(out, m)
continue
}
for _, o := range pm.Provides {
if to[o.Name] {
out = append(out, m)
break
}
}
}
return out
}
// assignedOn is the modules placed on node, in placement order.
func assignedOn(in compose.Input, node string) []string {
for _, n := range in.Nodes {
if n.Name == node {
return n.Assigned
}
}
return nil
}
// manifestOn is the manifest node composes module from: its Kept, else the catalogue's.
func manifestOn(in compose.Input, node, module string) (catalogue.Manifest, bool) {
for _, n := range in.Nodes {
if n.Name == node {
if man, ok := n.Kept[module]; ok {
return man, true
}
}
}
man, ok := in.Catalogue[module]
return man, ok
}
// culprit is the module placed on node whose composition the error is of: the first, in placement order, whose stub // culprit is the module placed on node whose composition the error is of: the first, in placement order, whose stub
// (compose.Stub) makes the composition succeed or fail otherwise; empty when none does (the error is not one module's). // (compose.Stub) makes the composition in mode succeed or fail otherwise; empty when none does (the error is not one
// Modules already refused are passed over. // module's). Modules already refused are passed over. A module whose stub still fails the same is tried bare (its stub
func culprit(in compose.Input, node string, fps map[string]string, failed map[string]string, err error) string { // contributing nothing, bare true): a refusal of what it contributes is its own, which the stub keeps (issue 1126,
// seed-8: de-spiegel's route names on novox, an unset required value, refused the whole node).
func culprit(in compose.Input, node string, fps map[string]string, mode compose.Mode, failed map[string]string, err error) (string, bool) {
var assigned []string var assigned []string
for _, n := range in.Nodes { for _, n := range in.Nodes {
if n.Name == node { if n.Name == node {
@@ -263,37 +483,78 @@ func culprit(in compose.Input, node string, fps map[string]string, failed map[st
order = append(order, m) order = append(order, m)
} }
} }
changes := func(m string, bare bool) bool {
_, _, _, again := compose.CompositionsIn(stubbed(in, node, m, bare), node, fps, mode)
return again == nil || again.Error() != err.Error()
}
for _, m := range order { for _, m := range order {
if _, done := failed[m]; done { if _, done := failed[m]; done {
continue continue
} }
if _, _, _, again := compose.CompositionsShadow(stubbed(in, node, m), node, fps); again == nil || again.Error() != err.Error() { if changes(m, false) {
return m return m, false
} }
} }
return "" var kept map[string]catalogue.Manifest
for _, n := range in.Nodes {
if n.Name == node {
kept = n.Kept
}
}
for _, m := range order {
// One already held is tried bare too, once: held for its build, its stub may still refuse (its own secret
// never issued, issue 1126).
if _, done := failed[m]; done && isBare(kept[m]) {
continue
}
if changes(m, true) {
return m, true
}
}
return "", false
} }
// stubbed is in with module composed as a stub on node alone (compose.Stub, as node's Kept). // isBare says a stub contributes nothing and declares no own secret (stubbed, bare).
func stubbed(in compose.Input, node, module string) compose.Input { func isBare(m catalogue.Manifest) bool {
return len(m.Contributes) == 0 && len(m.ContributesMany) == 0 && len(m.OwnSecrets) == 0
}
// stubbed is in with module held on node as its stub alone (heldAs, nothing of it running), from the manifest node
// composes it from: a culprit's try.
func stubbed(in compose.Input, node, module string, bare bool) compose.Input {
man, ok := manifestOn(in, node, module)
if !ok {
return in
}
return heldAs(in, node, module, man, bare, nil)
}
// heldAs is in with module held on node (compose.Node.Held): composed from man as what runs of it (compose.Carried of
// runs; its stub, compose.Stub, when runs is nil or holds nothing of it), as node's Kept; bare, contributing nothing to
// what it requires and declaring no own secret: none of a held module's own composition is applied, so what it has not
// been issued (restic's repository on a node it never ran on) is no refusal of the rest.
func heldAs(in compose.Input, node, module string, man catalogue.Manifest, bare bool, runs *delivery.Composed) compose.Input {
out := in out := in
out.Nodes = append([]compose.Node(nil), in.Nodes...) out.Nodes = append([]compose.Node(nil), in.Nodes...)
for i := range out.Nodes { for i := range out.Nodes {
if out.Nodes[i].Name != node { if out.Nodes[i].Name != node {
continue continue
} }
man, ok := out.Nodes[i].Kept[module] if bare {
if !ok { man.Contributes, man.ContributesMany, man.OwnSecrets = nil, nil, nil
if man, ok = in.Catalogue[module]; !ok {
return in
}
} }
kept := make(map[string]catalogue.Manifest, len(out.Nodes[i].Kept)+1) kept := make(map[string]catalogue.Manifest, len(out.Nodes[i].Kept)+1)
for k, v := range out.Nodes[i].Kept { for k, v := range out.Nodes[i].Kept {
kept[k] = v kept[k] = v
} }
kept[module] = compose.Stub(man) kept[module] = compose.Carried(man, module, runs)
out.Nodes[i].Kept = kept out.Nodes[i].Kept = kept
held := make(map[string]bool, len(out.Nodes[i].Held)+1)
for k, v := range out.Nodes[i].Held {
held[k] = v
}
held[module] = true
out.Nodes[i].Held = held
} }
return out return out
} }
+385 -10
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"os"
"reflect" "reflect"
"slices" "slices"
"strings" "strings"
@@ -69,7 +70,7 @@ func answerNamesEveryRefusalAndLeavesAnOldBuildToTheNode(t *testing.T) {
// Every refusal: two builds taken in, and a module the composition refuses (its image never built), each named. // Every refusal: two builds taken in, and a module the composition refuses (its image never built), each named.
b := mk("lemurs", "web", "api", "spiegel") b := mk("lemurs", "web", "api", "spiegel")
a := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b)}, a := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b)},
map[string]string{"lemurs": fp("e"), "web": fp("1"), "api": fp("2"), "spiegel": fp("3")}, nil, "h1", 3) map[string]string{"lemurs": fp("e"), "web": fp("1"), "api": fp("2"), "spiegel": fp("3")}, nil, "p-h1", 3)
var refused []string var refused []string
for _, r := range a.Refusals { for _, r := range a.Refusals {
refused = append(refused, r.Module) refused = append(refused, r.Module)
@@ -90,6 +91,27 @@ func answerNamesEveryRefusalAndLeavesAnOldBuildToTheNode(t *testing.T) {
if lemursOK := slices.ContainsFunc(a.Added, func(m delivery.ModuleAdded) bool { return m.Module == "lemurs" }); !lemursOK { if lemursOK := slices.ContainsFunc(a.Added, func(m delivery.ModuleAdded) bool { return m.Module == "lemurs" }); !lemursOK {
t.Errorf("lemurs, which composes, is not answered beside the refusals: %+v", a) t.Errorf("lemurs, which composes, is not answered beside the refusals: %+v", a)
} }
// Issue 1126: a module the node holds today that the candidate holds too is held as today, no refusal: with every
// refused module held today the verdict is no refusal, and each is named under Held.
today := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), HeldToday: map[string]string{"api": "held", "spiegel": "held", "web": "held"}},
map[string]string{"lemurs": fp("e"), "web": fp("1"), "api": fp("2"), "spiegel": fp("3")}, nil, "p-h1", 3)
if today.Verdict == delivery.CandidateRefused || len(today.Refusals) != 0 || len(today.Held) != 3 || today.Held[0].Why == "" {
t.Errorf("held as today: %+v", today)
}
// A refused module the node does not hold today stays a refusal: holding only api today leaves spiegel and web refused.
part := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), HeldToday: map[string]string{"api": "held"}},
map[string]string{"lemurs": fp("e"), "web": fp("1"), "api": fp("2"), "spiegel": fp("3")}, nil, "p-h1", 3)
if part.Verdict != delivery.CandidateRefused || len(part.Refusals) != 2 || len(part.Held) != 1 || part.Held[0].Module != "api" {
t.Errorf("held as today in part: %+v", part)
}
// A module held only behind one kept today is kept as today too, saying behind which (novox's mailu behind de-spiegel).
if a := (delivery.CandidateAnswer{Refusals: []delivery.ModuleRefused{{Module: "de-spiegel", Why: "a value is unset"},
{Module: "mailu", Why: heldBehind("de-spiegel")}, {Module: "proxy", Why: heldBehind("mailu")}, {Module: "web", Why: "no build"}}}); true {
heldAsToday(&a, map[string]string{"de-spiegel": "runs a build of the old form"})
if len(a.Refusals) != 1 || a.Refusals[0].Module != "web" || len(a.Held) != 3 || a.Held[2].Today != "held behind mailu" {
t.Errorf("held behind a module kept today: %+v", a)
}
}
// The fold: spiegel runs inside the node's own composition at a build of the old form. Its resources are carried as // The fold: spiegel runs inside the node's own composition at a build of the old form. Its resources are carried as
// they run, each after the one it runs after: the node's own is the one it runs, and the node composes itself. // they run, each after the one it runs after: the node's own is the one it runs, and the node composes itself.
@@ -110,7 +132,7 @@ func answerNamesEveryRefusalAndLeavesAnOldBuildToTheNode(t *testing.T) {
} }
// A refusal naming no module placed here is no module's: no compose is spent stubbing each in turn (issue 1126: ace's // A refusal naming no module placed here is no module's: no compose is spent stubbing each in turn (issue 1126: ace's
// answer took 7m46s, one compose per module), though here a stub would compose. // answer took 7m46s, one compose per module), though here a stub would compose.
if m := culprit(in, "n1", fps, map[string]string{}, fmt.Errorf("nothing in this mesh provides \"artifact-store\"")); m != "" { if m, _ := culprit(in, "n1", fps, compose.Foreseeing, map[string]string{}, fmt.Errorf("nothing in this mesh provides \"artifact-store\"")); m != "" {
t.Errorf("a refusal naming no placed module was laid on %s", m) t.Errorf("a refusal naming no placed module was laid on %s", m)
} }
ownKey, _ := delivery.ComposedNodeKey("n1") ownKey, _ := delivery.ComposedNodeKey("n1")
@@ -128,7 +150,10 @@ func answerNamesEveryRefusalAndLeavesAnOldBuildToTheNode(t *testing.T) {
running = append(running, c) running = append(running, c)
} }
scope := Scope{Node: "n1", InOwn: []string{"spiegel"}, Running: running} scope := Scope{Node: "n1", InOwn: []string{"spiegel"}, Running: running}
got, _, err := Compose(ctx, b, Mesh{}, scope, fps) got, _, held, err := Compose(ctx, b, Mesh{}, scope, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil { if err != nil {
t.Fatalf("the node does not compose itself with spiegel folded at a build of the old form: %v", err) t.Fatalf("the node does not compose itself with spiegel folded at a build of the old form: %v", err)
} }
@@ -137,14 +162,14 @@ func answerNamesEveryRefusalAndLeavesAnOldBuildToTheNode(t *testing.T) {
if string(gotOwn) != string(wantOwn) { if string(gotOwn) != string(wantOwn) {
t.Errorf("the node's own composition is\n%s\nwant what runs\n%s", gotOwn, wantOwn) t.Errorf("the node's own composition is\n%s\nwant what runs\n%s", gotOwn, wantOwn)
} }
a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, running, "h1", 3) a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, running, "p-h1", 3)
if a.Verdict != delivery.CandidateSame { if a.Verdict != delivery.CandidateSame {
t.Errorf("the fold's answer: %+v, want the same", a) t.Errorf("the fold's answer: %+v, want the same", a)
} }
// (A) The agent also says spiegel runs by module, from the declaration kept before the switch (agent.Everything): a // (A) The agent also says spiegel runs by module, from the declaration kept before the switch (agent.Everything): a
// module the composition folds is judged inside the node's own, not as removed (issue 1126, seed #531's answer). // module the composition folds is judged inside the node's own, not as removed (issue 1126, seed #531's answer).
keptToo := append(append([]delivery.Composed{}, running...), delivery.Composed{Node: "n1", Module: "spiegel", Resources: carriedRes}) keptToo := append(append([]delivery.Composed{}, running...), delivery.Composed{Node: "n1", Module: "spiegel", Resources: carriedRes})
if a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, keptToo, "h1", 3); a.Verdict != delivery.CandidateSame || len(a.Removed) != 0 { if a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, keptToo, "p-h1", 3); a.Verdict != delivery.CandidateSame || len(a.Removed) != 0 {
t.Errorf("a folded module running from the kept declaration is answered %+v, want the same", a) t.Errorf("a folded module running from the kept declaration is answered %+v, want the same", a)
} }
// (B) The node's own composition carries its bus membership as it runs it (the controller's, from its store): composed // (B) The node's own composition carries its bus membership as it runs it (the controller's, from its store): composed
@@ -157,13 +182,363 @@ func answerNamesEveryRefusalAndLeavesAnOldBuildToTheNode(t *testing.T) {
} }
withMembership = append(withMembership, c) withMembership = append(withMembership, c)
} }
if a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, withMembership, "h1", 3); a.Verdict != delivery.CandidateSame { if a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"spiegel"}}, fps, withMembership, "p-h1", 3); a.Verdict != delivery.CandidateSame {
t.Errorf("the node's own with its bus membership is answered %+v, want the same", a) t.Errorf("the node's own with its bus membership is answered %+v, want the same", a)
} }
// Without what runs, the fold composes spiegel from its manifest, as before, and the composition refuses it. // Without what runs, the fold composes spiegel from its manifest, as before, and the composition refuses it.
if _, _, err := Compose(ctx, b, Mesh{}, Scope{Node: "n1", InOwn: []string{"spiegel"}}, fps); err == nil || if _, _, held, err := Compose(ctx, b, Mesh{}, Scope{Node: "n1", InOwn: []string{"spiegel"}}, fps); err != nil ||
!strings.Contains(err.Error(), "spiegel has not been built for this mesh") { !strings.Contains(held["spiegel"], "spiegel has not been built for this mesh") {
t.Errorf("composed spiegel without what runs: %v", err) t.Errorf("composed spiegel without what runs: %v %v, want spiegel held alone", err, held)
}
// (C) Issue 1126, C5: what the others read of a module carried as it runs is what runs, never its stub's: the filter
// opens the machine port its running container publishes (de-spiegel's 20005 on novox, not its software's 8080; a
// stub had no container, so snake's 80 on ace was opened beside the route proxy's). And C6: the operator export's file
// a module that keeps it runs is carried as it runs (mesh-vault.kept on novox), since the node holds no export.
listening := []byte(`{"module":"spiegel","version":"1","capabilities":["container-runtime"],` +
`"listens":[{"port":8080,"from":"mesh","why":"the site"}],"own-secrets":{"token":"${dir:state}/token"},` +
`"resources":[{"id":"state","type":"directory","path":"/var/lib/spiegel","mode":"0700"},` +
`{"id":"server","type":"container","name":"spiegel","artifact":"site","ports":["8080"]}],` +
`"build":{"artifacts":[{"name":"site","kind":"upstream","from":"` + image + `"}]}}`)
nft := []byte(`{"module":"nft","version":"1","capabilities":["systemd"],"filtering":{"into":"/etc/nftables.conf"}}`)
vault := []byte(`{"module":"vault","version":"1","capabilities":["systemd"],"keeps":"/var/lib/vault/root",` +
`"resources":[{"id":"service","type":"service","unit":"vault.service","state":"running","boot":"enabled"}]}`)
b = mk("lemurs", overlay.Name, "spiegel", "nft", "vault")
for m, raw := range map[string][]byte{"spiegel": listening, "nft": nft, "vault": vault} {
b.manifests[delivery.ManifestDigest(raw)] = raw
b.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
p = b.placements["n1"]
p.Decided.Endpoint = "n1.example:51820"
for i := range p.Modules {
if p.Modules[i].Module == "spiegel" {
p.Modules[i].Ports = map[string]int{"8080": 20005}
}
}
b.placements["n1"] = p
rec := b.records["n1"]
rec.OutwardLinks = []string{"eth0"}
b.records["n1"] = rec
fps = map[string]string{"lemurs": fp("e"), "nft": fp("f"), "vault": fp("9")}
if in, err = Input(ctx, b, Mesh{}, stub); err != nil {
t.Fatal(err)
}
in.Nodes[0].Kept = map[string]catalogue.Manifest{"spiegel": compose.Stub(in.Catalogue["spiegel"])}
if base, _, err = compose.Compositions(in, "n1", fps, compose.Reading); err != nil {
t.Fatal(err)
}
keptFile := json.RawMessage(`{"digest":"sha256:` + strings.Repeat("d", 64) + `","id":"vault.kept","mode":"0600","module":"vault","path":"/var/lib/vault/root/export.json","size":143604,"type":"file"}`)
running = nil
for _, c := range base {
switch c.Module {
case "":
c.Resources = append(append([]json.RawMessage{}, c.Resources...),
json.RawMessage(`{"id":"spiegel.state","type":"directory","path":"/var/lib/spiegel","mode":"0700"}`),
json.RawMessage(`{"id":"spiegel.server","type":"container","name":"spiegel","image":"registry.example/spiegel@sha256:`+strings.Repeat("c", 64)+`","ports":["10.77.0.1:20005:8080"]}`),
json.RawMessage(`{"id":"spiegel.server-env","type":"file","path":"/var/lib/spiegel/server.env","mode":"0600","content":"A=1\n"}`))
case "vault":
c.Resources = append(append([]json.RawMessage{}, c.Resources...), keptFile)
}
running = append(running, c)
}
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", InOwn: []string{"spiegel"}, Running: running}, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil {
t.Fatalf("the node does not compose itself with spiegel carried: %v", err)
}
var rules string
var keptComposed json.RawMessage
for _, c := range got {
for _, raw := range c.Resources {
var r struct {
ID string `json:"id"`
Content string `json:"content"`
}
_ = json.Unmarshal(raw, &r)
switch r.ID {
case "nft." + catalogue.FilteringID():
rules = r.Content
case "vault." + catalogue.KeptID():
keptComposed = raw
}
}
}
if !strings.Contains(rules, "# spiegel — the site") || !strings.Contains(rules, "dport 20005 accept") || strings.Contains(rules, "dport 8080") {
t.Errorf("the filter does not open the machine port spiegel runs at:\n%s", rules)
}
// The paths a carried module owns are its manifest's, as the controller composes them, never what runs of it (issue
// 1126, seed-8 #542: on ace, the folded f1-circuits, snake and txt-game, whose manifests leave their directories and
// files to ${dir:…}, owned their /var/lib paths on the node and none on the controller, shifting node-tools'
// MESH_OWNED_PATHS). f1-circuits' manifest at the indexer's commit (c0ea2c94) is captured; what runs of it is
// composed from that manifest by the controller's rule (the directory at /var/lib/f1-circuits, its file, its container).
for _, c := range running {
if c.Module == "" && !reflect.DeepEqual(compose.Carried(in.Catalogue["spiegel"], "spiegel", &c).OwnsPaths, []string{"/var/lib/spiegel"}) {
t.Errorf("spiegel carried owns %v, want its manifest's /var/lib/spiegel alone", compose.Carried(in.Catalogue["spiegel"], "spiegel", &c).OwnsPaths)
}
}
f1raw, err := os.ReadFile("testdata/f1-circuits-c0ea2c94.module.json")
if err != nil {
t.Fatal(err)
}
f1, err := catalogue.ParseManifest(f1raw)
if err != nil {
t.Fatal(err)
}
f1runs := delivery.Composed{Node: "ace", Resources: []json.RawMessage{
json.RawMessage(`{"id":"f1-circuits.state","module":"f1-circuits","type":"directory","path":"/var/lib/f1-circuits","mode":"0700"}`),
json.RawMessage(`{"id":"f1-circuits.cache","module":"f1-circuits","type":"directory","path":"/var/lib/f1-circuits/cache","mode":"0700"}`),
json.RawMessage(`{"id":"f1-circuits.site-env","module":"f1-circuits","type":"file","path":"/var/lib/f1-circuits/site.env","mode":"0644","content":"F1_CIRCUITS_BASE_URL=https://f1.example\n"}`),
json.RawMessage(`{"id":"f1-circuits.server","module":"f1-circuits","type":"container","name":"f1-circuits","image":"registry.example/f1@sha256:` + strings.Repeat("c", 64) + `"}`)}}
if got, want := catalogue.DeclaredPaths(compose.Carried(f1, "f1-circuits", &f1runs)), catalogue.DeclaredPaths(f1); !reflect.DeepEqual(got, want) {
t.Errorf("f1-circuits carried owns %v, want what the controller composes from its manifest, %v", got, want)
}
// Issue 1126: a held module hands no template its build kept for a file it no longer carries (novox's node:
// "postgres: its build kept a template for consumers"), but keeps its items and their templates, so what it
// contributes to other modules' collections still reaches them.
for _, row := range []struct {
name string
handed *catalogue.Handed
items []catalogue.Item
}{
{"templates and items", &catalogue.Handed{Templates: map[string]string{"consumers": "files/consumers.json"},
Items: map[string]string{"routes": "files/routes.tmpl"}}, []catalogue.Item{{Collection: "routes", Content: "x"}}},
{"items alone", nil, []catalogue.Item{{Collection: "routes", Content: "x"}}},
} {
kept := f1
kept.Handed, kept.Items = row.handed, row.items
st := compose.Stub(kept)
if st.Handed != nil && st.Handed.Templates != nil {
t.Errorf("%s: a stub hands templates %v", row.name, st.Handed.Templates)
}
if !reflect.DeepEqual(st.Items, row.items) {
t.Errorf("%s: a stub collects %+v, want its items %+v", row.name, st.Items, row.items)
}
if row.handed != nil && (st.Handed == nil || !reflect.DeepEqual(st.Handed.Items, row.handed.Items) || row.handed.Templates == nil) {
t.Errorf("%s: a stub hands item templates %+v, want %v and its manifest untouched", row.name, st.Handed, row.handed.Items)
}
}
var k1, k2 any
_ = json.Unmarshal(keptComposed, &k1)
_ = json.Unmarshal(keptFile, &k2)
if !reflect.DeepEqual(k1, k2) {
t.Errorf("the operator export is composed as %s, want the file that runs %s", keptComposed, keptFile)
}
// (D) Issue 1126, seed-8 (mesh-catalog #542): a module's own refusal refuses that module, never the node. novox runs
// de-spiegel folded, and its manifest at the indexer's commit (e25e1beb, captured) asks four required config values
// for its route names that nothing sets. novox answered the whole candidate refused ("de-spiegel contributing
// praktijk to route says ${setting:praktijk-name} ...": its stub still contributed the route). Only de-spiegel is
// refused, in the config form's words, and proxy, which receives its routes (what it would compose lacks them: review
// of the apply, round 1); the rest is judged.
spiegel, err := os.ReadFile("testdata/de-spiegel-e25e1beb.module.json")
if err != nil {
t.Fatal(err)
}
proxy := []byte(`{"module":"proxy","version":"1","capabilities":["systemd"],"provides":[{"name":"route","scope":"mesh"},{"name":"smtp","scope":"mesh"}],` +
`"resources":[{"id":"service","type":"service","unit":"proxy.service","state":"running","boot":"enabled"}]}`)
b = mk("lemurs", overlay.Name, "proxy", "de-spiegel")
for m, raw := range map[string][]byte{"de-spiegel": spiegel, "proxy": proxy} {
b.manifests[delivery.ManifestDigest(raw)] = raw
b.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
p = b.placements["n1"]
p.Decided.Endpoint = "n1.example:51820"
b.placements["n1"] = p
fps = map[string]string{"lemurs": fp("e"), "proxy": fp("f")}
a = Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given(b), InOwn: []string{"de-spiegel"}}, fps, nil, "p-h1", 3)
refused = nil
for _, r := range a.Refusals {
refused = append(refused, r.Module)
if r.Module == "de-spiegel" && (!strings.Contains(r.Why, "${config:praktijk-name}") || strings.Contains(r.Why, "${setting:")) {
t.Errorf("de-spiegel is refused in other words than its config form's: %s", r.Why)
}
}
if a.Refused != "" || !reflect.DeepEqual(refused, []string{"de-spiegel", "proxy"}) {
t.Errorf("a module's own refusal: %+v, want de-spiegel refused with proxy, which receives its routes, the node not", a)
}
if !slices.ContainsFunc(a.Added, func(m delivery.ModuleAdded) bool { return m.Module == "lemurs" }) {
t.Errorf("lemurs, which composes, is not judged beside de-spiegel's refusal: %+v", a)
}
// (E) Issue 1126, the applying path (review of the seed): the composition novox applies follows the same rule. It
// composed with no culprit, so de-spiegel's unset route names refused the whole node and every module it runs was
// marked not composed. Now de-spiegel is held, its refusal said in its config form's words, with proxy, which
// receives its routes; lemurs composes, and the node's own composition carries de-spiegel as it runs, so applying it
// removes nothing of it.
spiegelRuns := []json.RawMessage{
json.RawMessage(`{"id":"de-spiegel.state","type":"directory","path":"/var/lib/de-spiegel","mode":"0755"}`),
json.RawMessage(`{"id":"de-spiegel.server-env","type":"file","path":"/var/lib/de-spiegel/server.env","mode":"0600","content":"A=1\n"}`),
json.RawMessage(`{"id":"de-spiegel.server","type":"container","name":"de-spiegel","image":"registry.example/de-spiegel@sha256:` + strings.Repeat("c", 64) + `"}`)}
if in, err = Input(ctx, b, Mesh{}, stub); err != nil {
t.Fatal(err)
}
in.Nodes[0].Kept = map[string]catalogue.Manifest{"de-spiegel": compose.Stub(in.Catalogue["de-spiegel"])}
in.Nodes[0].Kept["de-spiegel"] = func(m catalogue.Manifest) catalogue.Manifest {
m.Contributes, m.ContributesMany = nil, nil
return m
}(in.Nodes[0].Kept["de-spiegel"])
if base, _, err = compose.Compositions(in, "n1", fps, compose.Reading); err != nil {
t.Fatal(err)
}
running = nil
for _, c := range base {
if c.Module == "" {
c.Resources = append(append([]json.RawMessage{}, c.Resources...), spiegelRuns...)
}
running = append(running, c)
}
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", InOwn: []string{"de-spiegel"}, Running: running}, fps)
if err != nil {
t.Fatalf("de-spiegel's own refusal refuses the node's whole composition: %v", err)
}
if len(held) != 2 || !strings.Contains(held["de-spiegel"], "${config:praktijk-name}") || !strings.Contains(held["proxy"], "de-spiegel") {
t.Errorf("held %v, want de-spiegel in its config form's words, and proxy for its routes", held)
}
lemursKey, _ := delivery.ComposedKey("n1", "lemurs", fps["lemurs"])
proxyKey, _ := delivery.ComposedKey("n1", "proxy", fps["proxy"])
if _, ok := got[lemursKey]; !ok {
t.Errorf("lemurs, which composes, is not composed beside de-spiegel's refusal: %v", sortedKeys(got))
}
if _, ok := got[proxyKey]; ok {
t.Errorf("proxy is composed without the routes of held de-spiegel: %v", sortedKeys(got))
}
carries := func(c delivery.Composed, what string) {
t.Helper()
ownRes, _ := json.Marshal(c.Resources)
for _, raw := range spiegelRuns {
var r struct{ ID string }
_ = json.Unmarshal(raw, &r)
if !strings.Contains(string(ownRes), `"id":"`+r.ID+`"`) {
t.Errorf("%s: the node's own composition drops %s, which de-spiegel runs: %s", what, r.ID, ownRes)
}
}
}
carries(got[ownKey], "folded")
// (F) Leaving the fold (review of the apply, round 1, finding 1): de-spiegel's want names a build of the new form,
// so it is no longer folded, while its names are still unset and the node's own still runs it. Held, it is carried
// in the node's own as it runs until a composition of its own is handed: else the node's own apply removes its
// container, files and units (issue 1945 by a new path).
fps["de-spiegel"] = fp("7")
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", Running: running}, fps)
if err != nil || !strings.Contains(held["de-spiegel"], "${config:praktijk-name}") {
t.Fatalf("de-spiegel leaving the fold: %v, held %v", err, held)
}
spiegelKey, _ := delivery.ComposedKey("n1", "de-spiegel", fps["de-spiegel"])
if _, ok := got[spiegelKey]; ok {
t.Errorf("held de-spiegel's own composition is handed")
}
carries(got[ownKey], "leaving the fold")
// (G) A held module that runs a composition of its own (review of the apply, round 1, finding 2): what the others
// compose of it is what runs, never its stub's. site listens on 8080, placed at the machine's 20005, and its own
// secret was never issued here: held, the filter must not open 8080 and drop the 20005 that runs. The filter, which
// reads what site's manifest listens on, is held with it (dependents).
site := []byte(`{"module":"site","version":"1","capabilities":["container-runtime"],` +
`"listens":[{"port":8080,"from":"mesh","why":"the site"}],"own-secrets":{"key":"${dir:state}/key"},` +
`"resources":[{"id":"state","type":"directory","path":"/var/lib/site","mode":"0700"},` +
`{"id":"server","type":"container","name":"site","image":"registry.example/site@sha256:` + strings.Repeat("c", 64) + `","ports":["8080"]}]}`)
b = mk("lemurs", overlay.Name, "site", "nft")
for m, raw := range map[string][]byte{"site": site, "nft": nft} {
b.manifests[delivery.ManifestDigest(raw)] = raw
b.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
p = b.placements["n1"]
p.Decided.Endpoint = "n1.example:51820"
for i := range p.Modules {
if p.Modules[i].Module == "site" {
p.Modules[i].Ports = map[string]int{"8080": 20005}
}
}
b.placements["n1"] = p
rec = b.records["n1"]
rec.OutwardLinks = []string{"eth0"}
b.records["n1"] = rec
fps = map[string]string{"lemurs": fp("e"), "nft": fp("f"), "site": fp("8")}
siteRuns := delivery.Composed{Node: "n1", Module: "site", FP: fps["site"], Resources: []json.RawMessage{
json.RawMessage(`{"id":"site.state","module":"site","type":"directory","path":"/var/lib/site","mode":"0700"}`),
json.RawMessage(`{"id":"site.server","module":"site","type":"container","name":"site","image":"registry.example/site@sha256:` + strings.Repeat("c", 64) + `","ports":["10.77.0.1:20005:8080"]}`)}}
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", Running: []delivery.Composed{siteRuns}}, fps)
if err != nil || held["site"] == "" || held["nft"] == "" {
t.Fatalf("site, its own secret never issued: %v, held %v; want site held, and the filter, which depends on it", err, held)
}
for key, c := range got {
for _, raw := range c.Resources {
if strings.Contains(string(raw), "dport 8080") || (c.Module == "nft" && !strings.Contains(string(raw), "dport 20005") && strings.Contains(string(raw), "# site")) {
t.Errorf("%s composes held site from its stub, not from what runs: %s", key, raw)
}
}
}
// (G2) Issue 1126, the spreading hold: a held module is no dependence of its neighbours. site, held, provides page, and
// app requires and binds it: app composes against what runs of site and is not held (novox's seed held every postgres
// consumer behind de-spiegel's hold, "what it composes of …, held, is not what runs of them").
provider := []byte(strings.Replace(string(site), `"listens":`, `"provides":[{"name":"page","scope":"node"}],"serves":{"page":{"port":8080}},"listens":`, 1))
app := []byte(`{"module":"app","version":"1","capabilities":["systemd"],"requires":["page"],"binds":{"page":"/etc/app/page.json"},` +
`"resources":[{"id":"service","type":"service","unit":"app.service","state":"running","boot":"enabled"}]}`)
b2 := mk("lemurs", overlay.Name, "site", "nft", "app")
for m, raw := range map[string][]byte{"site": provider, "nft": nft, "app": app} {
b2.manifests[delivery.ManifestDigest(raw)] = raw
b2.modules[m] = delivery.Module{Manifest: delivery.ManifestDigest(raw)}
}
p2 := b2.placements["n1"]
p2.Decided.Endpoint = "n1.example:51820"
for i := range p2.Modules {
if p2.Modules[i].Module == "site" {
p2.Modules[i].Ports = map[string]int{"8080": 20005}
}
}
b2.placements["n1"] = p2
b2.records["n1"] = rec
b2.held = true
fps["app"] = fp("9")
got, _, held, err = Compose(ctx, b2, Mesh{}, Scope{Node: "n1", Running: []delivery.Composed{siteRuns}}, fps)
appKey, _ := delivery.ComposedKey("n1", "app", fps["app"])
if _, composed := got[appKey]; err != nil || held["site"] == "" || held["app"] != "" || !composed {
t.Errorf("a consumer of held site: %v, held %v; want site held, and app composed against what runs of it", err, held)
}
if bound, _ := json.Marshal(got[appKey].Resources); !strings.Contains(string(bound), "20005") {
t.Errorf("app's binding does not reach held site where it runs (20005): %s", bound)
}
delete(fps, "app")
// A held feeder of a collector holds no collector: its items reach their collection through its own plan, which it
// keeps as it runs. Only a provider it contributes to in that provider's own files is held behind it (the proxy its
// routes), as is the filter where it listens.
withCollector := compose.Input{Catalogue: map[string]catalogue.Manifest{
"feeder": {Module: "feeder", Items: []catalogue.Item{{Collection: "routes", Content: "x"}}},
"col": {Module: "col", Collections: []catalogue.Collection{{Name: "routes"}}},
"proxy": {Module: "proxy", Provides: []catalogue.Offer{{Name: "route"}}}},
Nodes: []compose.Node{{Name: "n1", Assigned: []string{"feeder", "col", "proxy"}}}}
feeder := withCollector.Catalogue["feeder"]
if d := dependents(withCollector, "n1", feeder, map[string]string{"feeder": "held"}); len(d) != 0 {
t.Errorf("a held feeder of a collector holds %v, want none", d)
}
feeder.Contributes = map[string]map[string]any{"route": {"name": "x"}}
if d := dependents(withCollector, "n1", feeder, map[string]string{"feeder": "held"}); !reflect.DeepEqual(d, []string{"proxy"}) {
t.Errorf("a held module contributing routes holds %v, want the proxy", d)
}
// (H) Review of the apply, round 2: a held module is composed from its new manifest, so what the others read of its
// manifest (its listens, what it contributes) may not be what runs. site's new manifest moves it to 9090 while the
// build that runs listens on 8080: the filter, the holder of what site listens on, is held with it, and nothing
// applied opens 9090.
moved := []byte(strings.Replace(string(site), `"port":8080`, `"port":9090`, 1))
b.manifests[delivery.ManifestDigest(moved)] = moved
b.modules["site"] = delivery.Module{Manifest: delivery.ManifestDigest(moved)}
for i := range p.Modules {
p.Modules[i].Ports = nil
}
b.placements["n1"] = p
siteRuns.Resources[1] = json.RawMessage(`{"id":"site.server","module":"site","type":"container","name":"site","image":"registry.example/site@sha256:` + strings.Repeat("c", 64) + `","ports":["8080"]}`)
got, _, held, err = Compose(ctx, b, Mesh{}, Scope{Node: "n1", Running: []delivery.Composed{siteRuns}}, fps)
if err != nil || held["site"] == "" || held["nft"] == "" {
t.Errorf("site moved to 9090 in its new manifest: %v, held %v; want site held, and the filter with it", err, held)
}
for key, c := range got {
for _, raw := range c.Resources {
if strings.Contains(string(raw), "dport 9090") {
t.Errorf("%s opens 9090, which held site's new manifest names and nothing runs: %s", key, raw)
}
}
} }
} }
@@ -187,7 +562,7 @@ func otherNodeNotResolvedIsNamed(t *testing.T) {
p := p p := p
given[n] = &p given[n] = &p
} }
a := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given}, map[string]string{"cons": "src2:" + strings.Repeat("1", 64)}, nil, "h", 1) a := Answer(ctx, b, Mesh{}, Scope{Node: "n1", Placements: given}, map[string]string{"cons": "src2:" + strings.Repeat("1", 64)}, nil, "p-h1", 1)
if a.Verdict != delivery.CandidateRefused || !strings.Contains(a.Refused, "prov on n2 lacks container-runtime") { if a.Verdict != delivery.CandidateRefused || !strings.Contains(a.Refused, "prov on n2 lacks container-runtime") {
t.Errorf("a refusal for want of n2's provider = %+v", a) t.Errorf("a refusal for want of n2's provider = %+v", a)
} }
+6 -5
View File
@@ -10,9 +10,10 @@ import (
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
// Export is one node's placement file as the composition input the controller fills says it (novox/hq ADR 0555 §6, // Export is one node's placement as the composition input the controller fills says it (novox/hq ADR 0555 §6, ADR 0567
// issue 1126): the inverse of Input for what a person decides, so the controller's one read-only verb // §7): the inverse of Input for what a person decides, so the controller's one read-only verb `placement-export`
// `placement-export` answers, from its own store by its own code, exactly what the seed writes into the catalogue. // answers, from its own store by its own code, exactly what the seat's adopt proposes as the seed. Its answer must stay
// stable for the same store, since adopt checks its digest again at the warrant.
// What the machine observes (its record) is not in it. // What the machine observes (its record) is not in it.
func Export(in compose.Input, node string) (delivery.Placement, error) { func Export(in compose.Input, node string) (delivery.Placement, error) {
var n *compose.Node var n *compose.Node
@@ -24,8 +25,8 @@ func Export(in compose.Input, node string) (delivery.Placement, error) {
if n == nil { if n == nil {
return delivery.Placement{}, fmt.Errorf("frombus: the input has no node %q", node) return delivery.Placement{}, fmt.Errorf("frombus: the input has no node %q", node)
} }
// At the version the placement file is read at, with a list of modules even when none is placed: the export is the // At the version a placement key is read at, with a list of modules even when none is placed: the export is the
// file the seed writes (cmd/placement-seed), and the reader refuses a value of version 0 or without its modules. // seed's value, and the reader refuses a value of version 0 or without its modules.
p := delivery.Placement{V: delivery.Version, Node: node, Modules: []delivery.PlacedModule{}, Decided: delivery.NodeDecided{ p := delivery.Placement{V: delivery.Version, Node: node, Modules: []delivery.PlacedModule{}, Decided: delivery.NodeDecided{
Endpoint: n.Place.Endpoint, Site: n.Place.Site, Hub: n.Place.Hub, Address: n.Place.Address, Endpoint: n.Place.Endpoint, Site: n.Place.Site, Hub: n.Place.Hub, Address: n.Place.Address,
PublicDomain: n.PublicDomain, Account: n.Account, AccountHome: n.AccountHome, AgentAccount: n.AgentAccount, PublicDomain: n.PublicDomain, Account: n.Account, AccountHome: n.AccountHome, AgentAccount: n.AgentAccount,
+82 -25
View File
@@ -2,13 +2,13 @@
// and the grants part compose with one composer over the same inputs (novox/hq ADR 0547 §2, ADR 0549 §3; issue 1126). // and the grants part compose with one composer over the same inputs (novox/hq ADR 0547 §2, ADR 0549 §3; issue 1126).
// //
// **One composer, three callers.** compose.Compositions is the composer; Input is how each caller fills its // **One composer, three callers.** compose.Compositions is the composer; Input is how each caller fills its
// compose.Input from the bus: the node agent for its own node (ADR 0547 §2), the assigner's composition check for every // compose.Input from the bus: the node agent for its own node (ADR 0547 §2), the seat mesh-placement's holder judging a
// node (ADR 0539 §6's check, ADR 0547 §4), and the grants part (ADR 0547 §5). Each reads the bus with its own credential // proposal for every node (ADR 0567 §4), and the grants part (ADR 0547 §5). Each reads the bus with its own credential
// through a Reader, so this package holds no bus client, no store and no timer: what it is given is all it reads. // through a Reader, so this package holds no bus client, no store and no timer: what it is given is all it reads.
// //
// **What it reads, and from whom.** // **What it reads, and from whom.**
// - which modules run on each node, with their pins, holdings, host ports, bindings and settings placement: the // - which modules run on each node, with their pins, holdings, host ports, bindings and settings placement: the
// indexer's state `placement`, one key per node (ADR 0549 §2); // seat mesh-placement's state `placement`, one key per node (ADR 0567 §1);
// - what a person decided of each node (its endpoint, site, hub, address on the private network, public domain and // - what a person decided of each node (its endpoint, site, hub, address on the private network, public domain and
// accounts): the same placement key, Placement.Decided; // accounts): the same placement key, Placement.Decided;
// - what each node observes of itself (its tunnel key, capabilities, keys, certificate, links and networks): the // - what each node observes of itself (its tunnel key, capabilities, keys, certificate, links and networks): the
@@ -25,8 +25,8 @@
// **Own secrets and pair credentials** come from the vault sealed to the composing node's key (OwnSecrets, Credentials), // **Own secrets and pair credentials** come from the vault sealed to the composing node's key (OwnSecrets, Credentials),
// for that node alone: as a consumer and as a provider, exactly the keys compose.CredentialsNeeded names. **Not yet // for that node alone: as a consumer and as a provider, exactly the keys compose.CredentialsNeeded names. **Not yet
// read** (each a later subtask of issue 1126): licences (none is placed today) and core boxes. Without one // read** (each a later subtask of issue 1126): licences (none is placed today) and core boxes. Without one
// the composition stands in for it, and Compose refuses a composition that stands in for anything (compose.ErrStoodIn): // the composition stands in for it, and Compose holds the module it stands in for (or refuses the node, for a stand-in
// a node never applies a stand-in. No node is joining before ADR 0547's third milestone (ADR 0550 §6), so Joining is // of no module placed there; compose.ErrStoodIn): a node never applies a stand-in. No node is joining before ADR 0547's third milestone (ADR 0550 §6), so Joining is
// always empty. // always empty.
package frombus package frombus
@@ -83,6 +83,16 @@ type Reader interface {
Served(ctx context.Context, node, module string, keys []string) (map[string]any, error) Served(ctx context.Context, node, module string, keys []string) (map[string]any, error)
} }
// SignersReader is a Reader that also reads the vault's public authority from its state `signers` (delivery.SignersBucket,
// key delivery.SignersAuthorityKey; ADR 0550 §2). A Reader that has it composes the mesh's authority as a bundle, the
// controller's and the vault's; one that has not, or that answers none, composes the controller's alone, as before.
type SignersReader interface {
// VaultAuthority is the vault's authority certificate as PEM, checked (delivery.DecodeSignerAuthority); empty when the
// vault has published none. A read that fails is said by the reader and answered empty: a node that may not read the
// state yet composes as before rather than not at all.
VaultAuthority(ctx context.Context) string
}
// Mesh is what is true of the whole mesh and is no part's state on the bus yet: the private network's range and the // Mesh is what is true of the whole mesh and is no part's state on the bus yet: the private network's range and the
// internal names' suffix as genesis was told them, and the mesh's authority certificate (public). Empty fields take // internal names' suffix as genesis was told them, and the mesh's authority certificate (public). Empty fields take
// compose's defaults. // compose's defaults.
@@ -107,9 +117,9 @@ type Mesh struct {
// build of at the trunk's FP2 composes from the indexer's manifest, as before. Every leaves the catalogue as the indexer // build of at the trunk's FP2 composes from the indexer's manifest, as before. Every leaves the catalogue as the indexer
// read it: the check of every node judges placement, and reads no archive. // read it: the check of every node judges placement, and reads no archive.
// //
// Placements, when set, stands in for the indexer's placement of the nodes it names: a node's candidate placement // Placements, when set, stands in for the placement of the nodes it names: a proposal's candidates
// (delivery.PlacementCandidate), so a node composes in shadow with a catalogue pull request's placement and compares // (delivery.PlacementCandidate), so a node composes in shadow with a proposal laid over and compares the result with
// the result with what it runs (issue 1126); a nil value removes the node's placement. The rest is read as always. // what it runs (ADR 0567 §3); a nil value removes the node's placement. The rest is read as always.
// //
// InOwn is the modules the composing node runs inside its own composition (InOwn of what it runs), composed there again // InOwn is the modules the composing node runs inside its own composition (InOwn of what it runs), composed there again
// (compose.Node.InOwn) rather than cut out as modules no build is given for (issue 1126). // (compose.Node.InOwn) rather than cut out as modules no build is given for (issue 1126).
@@ -124,6 +134,10 @@ type Scope struct {
Placements map[string]*delivery.Placement Placements map[string]*delivery.Placement
InOwn []string InOwn []string
Running []delivery.Composed Running []delivery.Composed
// HeldToday is, in an answer, the modules the node keeps today as they run ("node" for its own composition), each with
// how ("held", "failed", "runs a build of the old form"): a candidate that cannot compose one, or holds one only
// behind one of them, keeps it as today, which is said under CandidateAnswer.Held and is no refusal (issue 1126).
HeldToday map[string]string
// SameValue, in an answer, says whether two sealed values open to the same value (the node opens them with its own // SameValue, in an answer, says whether two sealed values open to the same value (the node opens them with its own
// key), so a differing resource shows a value sealed again as such (delivery.CompareComposedWith). Nil shows them // key), so a differing resource shows a value sealed again as such (delivery.CompareComposedWith). Nil shows them
// as differing. // as differing.
@@ -219,6 +233,13 @@ func input(ctx context.Context, r Reader, mesh Mesh, s Scope, refuse func(module
} }
} }
} }
// And the vault's authority beside it, once the vault publishes one (ADR 0550 §2): a certificate either signed checks
// against the bundle. The controller's stays while a certificate it issued is in a node's record.
if sr, ok := r.(SignersReader); ok {
if vault := sr.VaultAuthority(ctx); vault != "" {
in.Authority = delivery.AuthorityBundle(in.Authority, vault)
}
}
parse := func(raw []byte) (catalogue.Manifest, error) { parse := func(raw []byte) (catalogue.Manifest, error) {
if mesh.Predefined != nil { if mesh.Predefined != nil {
return catalogue.ParseManifestWith(raw, mesh.Predefined) return catalogue.ParseManifestWith(raw, mesh.Predefined)
@@ -246,15 +267,17 @@ func input(ctx context.Context, r Reader, mesh Mesh, s Scope, refuse func(module
// The module the mesh provides itself (the private network) has no repository the indexer reads: its manifest comes // The module the mesh provides itself (the private network) has no repository the indexer reads: its manifest comes
// with mesh-sdk and always wins, as the controller's providedModules and the grants part have it. // with mesh-sdk and always wins, as the controller's providedModules and the grants part have it.
provided, err := json.Marshal(overlay.Manifest()) for _, name := range delivery.ProvidedModules() {
if err != nil { raw, err := json.Marshal(overlay.Manifest())
return compose.Input{}, err if err != nil {
return compose.Input{}, err
}
var private catalogue.Manifest
if err := json.Unmarshal(raw, &private); err != nil {
return compose.Input{}, fmt.Errorf("frombus: the manifest mesh-sdk provides for %s: %w", name, err)
}
in.Catalogue[name] = private
} }
var private catalogue.Manifest
if err := json.Unmarshal(provided, &private); err != nil {
return compose.Input{}, fmt.Errorf("frombus: the private network's manifest mesh-sdk provides: %w", err)
}
in.Catalogue[overlay.Name] = private
var missing []string var missing []string
for _, node := range nodesOf(placements, records) { for _, node := range nodesOf(placements, records) {
@@ -290,6 +313,8 @@ func input(ctx context.Context, r Reader, mesh Mesh, s Scope, refuse func(module
} }
} }
if node == s.Node { if node == s.Node {
// What its runtime serves as it runs: what a module composed as what runs of it contributes there (issue 1126).
n.RunsTools = compose.ToolsRunning(s.Running)
for _, module := range s.InOwn { for _, module := range s.InOwn {
if n.InOwn == nil { if n.InOwn == nil {
n.InOwn = map[string]bool{} n.InOwn = map[string]bool{}
@@ -297,6 +322,21 @@ func input(ctx context.Context, r Reader, mesh Mesh, s Scope, refuse func(module
n.InOwn[module] = true n.InOwn[module] = true
} }
for i := range s.Running { for i := range s.Running {
// The operator export as the vault runs it: the controller composes it from its store, which the node
// does not read; until the export is on the bus, the node carries the file it runs (issue 1126, C6:
// mesh-vault.kept on novox, "in what runs, and not composed").
if m := s.Running[i].Module; m != "" {
if kept := keptOf(s.Running[i]); kept != nil {
if in.KeptRuns == nil {
in.KeptRuns = map[string]map[string]map[string]any{}
}
if in.KeptRuns[s.Node] == nil {
in.KeptRuns[s.Node] = map[string]map[string]any{}
}
in.KeptRuns[s.Node][m] = kept
}
continue
}
if s.Running[i].Module == "" { if s.Running[i].Module == "" {
own := s.Running[i] own := s.Running[i]
n.RunsOwn = &own n.RunsOwn = &own
@@ -364,25 +404,31 @@ func input(ctx context.Context, r Reader, mesh Mesh, s Scope, refuse func(module
return in, nil return in, nil
} }
// Compose is the node's compositions over the bus (ADR 0547 §2): Input for s, then compose.CompositionsApplied at the // Compose is the node's compositions over the bus (ADR 0547 §2): input for s, then composed at the fingerprints fps
// fingerprints fps gives each module: composed in the Reading mode as the controller composes them, and refused when it // gives each module in the Reading mode, as the controller composes them, by the one rule its answer to a candidate
// stands in for anything the input does not carry (compose.ErrStoodIn), since the node applies what it composed. // follows too (composeEach, issue 1126): a module refused alone (its build unread or not taken in, its composition
func Compose(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]string) (map[string]delivery.Composed, []string, error) { // refused, a stand-in for a value of its own the input does not carry, compose.ErrStoodIn) is held, named in held with
in, err := Input(ctx, r, mesh, s) // why, and none of its own composition is in out, so the node keeps what it last applied of it and applies the rest; a
// node never applies a stand-in. Only a refusal that is no single module's (the node's own resources, a value served
// elsewhere, a stand-in for the node's own) refuses the node, err.
func Compose(ctx context.Context, r Reader, mesh Mesh, s Scope, fps map[string]string) (out map[string]delivery.Composed, skipped []string, held map[string]string, err error) {
held = map[string]string{}
in, err := input(ctx, r, mesh, s, func(module string, err error) { held[module] = err.Error() })
if err != nil { if err != nil {
return nil, nil, err return nil, nil, held, err
} }
// The pair credentials its composition reads, asked of the vault sealed to it (ADR 0550 §5). // The pair credentials its composition reads, asked of the vault sealed to it (ADR 0550 §5).
asConsumer, asProvider, err := compose.CredentialsNeeded(in, s.Node) asConsumer, asProvider, err := compose.CredentialsNeeded(in, s.Node)
if err != nil { if err != nil {
return nil, nil, err return nil, nil, held, err
} }
if len(asConsumer)+len(asProvider) > 0 { if len(asConsumer)+len(asProvider) > 0 {
if in.Credentials, in.Issued, err = r.Credentials(ctx, s.Node, asConsumer, asProvider); err != nil { if in.Credentials, in.Issued, err = r.Credentials(ctx, s.Node, asConsumer, asProvider); err != nil {
return nil, nil, fmt.Errorf("frombus: the credentials of %s: %w", s.Node, err) return nil, nil, held, fmt.Errorf("frombus: the credentials of %s: %w", s.Node, err)
} }
} }
return compose.CompositionsApplied(in, s.Node, fps) out, skipped, _, err = composeEach(in, s.Node, fps, true, s.Running, held)
return out, skipped, held, err
} }
// fromBuilds puts in the built manifest of each module placed on the composing node, at the trunk's FP2, and of each // fromBuilds puts in the built manifest of each module placed on the composing node, at the trunk's FP2, and of each
@@ -708,6 +754,17 @@ func storeSought(in compose.Input) string {
", none of it on the private network with a port" ", none of it on the private network with a port"
} }
// keptOf is the operator export's file (catalogue.KeptID) of a module's composition as it runs, or nil.
func keptOf(c delivery.Composed) map[string]any {
for _, raw := range c.Resources {
var r map[string]any
if json.Unmarshal(raw, &r) == nil && r["id"] == c.Module+"."+catalogue.KeptID() {
return r
}
}
return nil
}
// membershipOf is the sealed value of the bus membership resource (catalogue.BusMembershipID) of a node's own // membershipOf is the sealed value of the bus membership resource (catalogue.BusMembershipID) of a node's own
// composition, or "". // composition, or "".
func membershipOf(own delivery.Composed) string { func membershipOf(own delivery.Composed) string {
+74 -25
View File
@@ -25,14 +25,15 @@ import (
// bus is a Reader over values held in memory, counting whose config values were read. // bus is a Reader over values held in memory, counting whose config values were read.
type bus struct { type bus struct {
placements map[string]delivery.Placement vaultAuthority string
records map[string]delivery.NodeRecord placements map[string]delivery.Placement
modules map[string]delivery.Module records map[string]delivery.NodeRecord
manifests map[string][]byte modules map[string]delivery.Module
values map[string]map[string]any // by node.module manifests map[string][]byte
valuesRead []string values map[string]map[string]any // by node.module
own []compose.OwnSecretRow valuesRead []string
ownErr error own []compose.OwnSecretRow
ownErr error
// asked is every consumer credential asked; the fake refuses the whole ask, as the vault's own does, for a consuming // asked is every consumer credential asked; the fake refuses the whole ask, as the vault's own does, for a consuming
// module its placement does not put on the node. // module its placement does not put on the node.
asked []compose.CredentialKey asked []compose.CredentialKey
@@ -44,6 +45,8 @@ type bus struct {
served map[string]map[string]any served map[string]map[string]any
servedRead []string servedRead []string
buildsRead []string buildsRead []string
// held answers every credential asked as held, sealed to the consumer.
held bool
} }
func (b *bus) Build(_ context.Context, module, fp string) (*delivery.BuilderBuild, error) { func (b *bus) Build(_ context.Context, module, fp string) (*delivery.BuilderBuild, error) {
@@ -74,6 +77,7 @@ func (b *bus) Served(_ context.Context, node, module string, keys []string) (map
func (b *bus) Placements(context.Context) (map[string]delivery.Placement, error) { func (b *bus) Placements(context.Context) (map[string]delivery.Placement, error) {
return b.placements, nil return b.placements, nil
} }
func (b *bus) VaultAuthority(context.Context) string { return b.vaultAuthority }
func (b *bus) Records(context.Context) (map[string]delivery.NodeRecord, error) { return b.records, nil } func (b *bus) Records(context.Context) (map[string]delivery.NodeRecord, error) { return b.records, nil }
func (b *bus) Modules(context.Context) (map[string]delivery.Module, error) { return b.modules, nil } func (b *bus) Modules(context.Context) (map[string]delivery.Module, error) { return b.modules, nil }
func (b *bus) Manifest(_ context.Context, digest string) ([]byte, error) { func (b *bus) Manifest(_ context.Context, digest string) ([]byte, error) {
@@ -87,7 +91,7 @@ func (b *bus) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow, error
return b.own, b.ownErr return b.own, b.ownErr
} }
func (b *bus) Credentials(_ context.Context, node string, asConsumer, _ []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) { func (b *bus) Credentials(_ context.Context, node string, asConsumer, asProvider []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
b.asked = append(b.asked, asConsumer...) b.asked = append(b.asked, asConsumer...)
for _, k := range asConsumer { for _, k := range asConsumer {
placed := false placed := false
@@ -98,7 +102,17 @@ func (b *bus) Credentials(_ context.Context, node string, asConsumer, _ []compos
return nil, nil, fmt.Errorf("%s is not on %s: neither assigned there nor required by a module there", k.ConsumerModule, node) return nil, nil, fmt.Errorf("%s is not on %s: neither assigned there nor required by a module there", k.ConsumerModule, node)
} }
} }
return nil, nil, nil var out []compose.Credential
var issued []compose.Issued
for _, k := range append(append([]compose.CredentialKey{}, asConsumer...), asProvider...) {
if b.held {
out = append(out, compose.Credential{Provision: k.Provision, Consumer: k.Consumer, ConsumerModule: k.ConsumerModule,
Provider: k.Provider, Local: k.Local, State: compose.CredentialHeld, Sealed: `"box"`})
issued = append(issued, compose.Issued{Provision: k.Provision, Consumer: k.Consumer, ConsumerModule: k.ConsumerModule,
Provider: k.Provider, Local: k.Local, ForProvider: `"box"`})
}
}
return out, issued, nil
} }
func (b *bus) Values(_ context.Context, node, module string) (map[string]any, error) { func (b *bus) Values(_ context.Context, node, module string) (map[string]any, error) {
@@ -216,6 +230,14 @@ func TestInput(t *testing.T) {
wj, _ := json.MarshalIndent(want, "", " ") wj, _ := json.MarshalIndent(want, "", " ")
t.Fatalf("Input from the bus differs from the store's:\n got %s\nwant %s", gj, wj) t.Fatalf("Input from the bus differs from the store's:\n got %s\nwant %s", gj, wj)
} }
// L3 s8 (ADR 0550 §2): with the vault's authority published, the mesh's authority is the bundle of the controller's
// and the vault's; with none, the controller's alone, as above.
signing := mk()
signing.vaultAuthority = "the vault's certificate"
if bundled, err := Input(ctx, signing, Mesh{}, Scope{Node: "laptop"}); err != nil ||
bundled.Authority != "the mesh's certificate\nthe vault's certificate\n" {
t.Errorf("the authority with the vault's published: %q, %v", bundled.Authority, err)
}
if !reflect.DeepEqual(b.valuesRead, []string{"laptop." + overlay.Name, "laptop.xterm", "laptop.media"}) { if !reflect.DeepEqual(b.valuesRead, []string{"laptop." + overlay.Name, "laptop.xterm", "laptop.media"}) {
t.Errorf("laptop read config values %v; a node reads its own slice alone", b.valuesRead) t.Errorf("laptop read config values %v; a node reads its own slice alone", b.valuesRead)
} }
@@ -223,7 +245,10 @@ func TestInput(t *testing.T) {
// The same composer over both: the compositions from the bus are the store's, byte for byte. // The same composer over both: the compositions from the bus are the store's, byte for byte.
fps := map[string]string{overlay.Name: "src2:" + strings.Repeat("a", 64), "xterm": "src2:" + strings.Repeat("b", 64), fps := map[string]string{overlay.Name: "src2:" + strings.Repeat("a", 64), "xterm": "src2:" + strings.Repeat("b", 64),
"media": "src2:" + strings.Repeat("c", 64)} "media": "src2:" + strings.Repeat("c", 64)}
fromBus, _, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop"}, fps) fromBus, _, held, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop"}, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -243,7 +268,10 @@ func TestInput(t *testing.T) {
running = append(running, c) running = append(running, c)
} }
laptop := mk().placements["laptop"] laptop := mk().placements["laptop"]
asIs, _, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop", Placements: map[string]*delivery.Placement{"laptop": &laptop}}, fps) asIs, _, held, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop", Placements: map[string]*delivery.Placement{"laptop": &laptop}}, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -252,7 +280,10 @@ func TestInput(t *testing.T) {
} }
moved := mk().placements["anchor"] moved := mk().placements["anchor"]
moved.Decided.Endpoint = "anchor.example:51821" moved.Decided.Endpoint = "anchor.example:51821"
changed, _, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop", Placements: map[string]*delivery.Placement{"laptop": &laptop, "anchor": &moved}}, fps) changed, _, held, err := Compose(ctx, mk(), Mesh{}, Scope{Node: "laptop", Placements: map[string]*delivery.Placement{"laptop": &laptop, "anchor": &moved}}, fps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -289,7 +320,10 @@ func TestInput(t *testing.T) {
} }
oneFps := map[string]string{"keeper": "src2:" + strings.Repeat("d", 64), "lemurs": "src2:" + strings.Repeat("e", 64), oneFps := map[string]string{"keeper": "src2:" + strings.Repeat("d", 64), "lemurs": "src2:" + strings.Repeat("e", 64),
"objects": "src2:" + strings.Repeat("f", 64), "files": "src2:" + strings.Repeat("0", 64)} "objects": "src2:" + strings.Repeat("f", 64), "files": "src2:" + strings.Repeat("0", 64)}
runs, _, err := Compose(ctx, one("keeper", "lemurs"), Mesh{}, Scope{Node: "n1"}, oneFps) runs, _, held, err := Compose(ctx, one("keeper", "lemurs"), Mesh{}, Scope{Node: "n1"}, oneFps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -297,7 +331,10 @@ func TestInput(t *testing.T) {
for _, c := range runs { for _, c := range runs {
runsNow = append(runsNow, c) runsNow = append(runsNow, c)
} }
objectsRun, _, err := Compose(ctx, one("objects"), Mesh{}, Scope{Node: "n1"}, oneFps) objectsRun, _, held, err := Compose(ctx, one("objects"), Mesh{}, Scope{Node: "n1"}, oneFps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -386,7 +423,9 @@ func TestInput(t *testing.T) {
check func(delivery.CandidateAnswer) bool check func(delivery.CandidateAnswer) bool
}{ }{
"the same": {one("keeper", "lemurs"), placedAs("keeper", "lemurs"), runsNow, delivery.CandidateSame, "the same": {one("keeper", "lemurs"), placedAs("keeper", "lemurs"), runsNow, delivery.CandidateSame,
func(a delivery.CandidateAnswer) bool { return len(a.Same) == 3 && a.Head == "h1" && a.Revision == 7 }}, func(a delivery.CandidateAnswer) bool {
return len(a.Same) == 3 && a.Proposal == "p-h1" && a.Revision == 7
}},
"the node runs no own composition": {one("keeper", "lemurs"), placedAs("keeper", "lemurs"), withoutOwn(runsNow), delivery.CandidateSame, "the node runs no own composition": {one("keeper", "lemurs"), placedAs("keeper", "lemurs"), withoutOwn(runsNow), delivery.CandidateSame,
func(a delivery.CandidateAnswer) bool { return len(a.Same) == 2 && len(a.Added) == 0 }}, func(a delivery.CandidateAnswer) bool { return len(a.Same) == 2 && len(a.Added) == 0 }},
"one removed": {one("keeper", "lemurs"), placedAs("keeper"), runsNow, delivery.CandidateDiffers, "one removed": {one("keeper", "lemurs"), placedAs("keeper"), runsNow, delivery.CandidateDiffers,
@@ -427,7 +466,7 @@ func TestInput(t *testing.T) {
if name == "mesh-wireguard and an old-form module, folded" { if name == "mesh-wireguard and an old-form module, folded" {
fps = map[string]string{"keeper": oneFps["keeper"]} // the node's wants: no build of lemurs or the private network fps = map[string]string{"keeper": oneFps["keeper"]} // the node's wants: no build of lemurs or the private network
} }
a := Answer(ctx, c.b, Mesh{}, Scope{Node: "n1", Placements: c.given, InOwn: InOwn(c.running, *c.given["n1"])}, fps, c.running, "h1", 7) a := Answer(ctx, c.b, Mesh{}, Scope{Node: "n1", Placements: c.given, InOwn: InOwn(c.running, *c.given["n1"])}, fps, c.running, "p-h1", 7)
if a.Verdict != c.verdict || !c.check(a) { if a.Verdict != c.verdict || !c.check(a) {
t.Errorf("%s: %+v", name, a) t.Errorf("%s: %+v", name, a)
} }
@@ -579,11 +618,21 @@ func TestInput(t *testing.T) {
} }
return b return b
} }
if _, _, err := Compose(ctx, movedMesh(false), Mesh{}, Scope{Node: "n1"}, movedFps); err == nil || !strings.Contains(err.Error(), `restic needs a secret called "repository"`) { if out, _, held, err := Compose(ctx, movedMesh(false), Mesh{}, Scope{Node: "n1"}, movedFps); err != nil || !strings.Contains(held["restic"], `restic needs a secret called "repository"`) {
t.Errorf("restic composed from the indexer's manifest: %v, want its own secret refused", err) t.Errorf("restic composed from the indexer's manifest: %v %v, want its own secret refusing restic alone", err, held)
} else {
for key, c := range out {
if c.Module == "restic" {
// Its stub's composition, applied, would remove what restic runs: a held module keeps its last one.
t.Errorf("held restic is composed as %s, want none of its own composition handed", key)
}
}
} }
builtBus := movedMesh(true) builtBus := movedMesh(true)
fromBuild, _, err := Compose(ctx, builtBus, Mesh{}, Scope{Node: "n1"}, movedFps) fromBuild, _, held, err := Compose(ctx, builtBus, Mesh{}, Scope{Node: "n1"}, movedFps)
if err == nil && len(held) > 0 {
err = fmt.Errorf("modules held: %v", held)
}
if err != nil { if err != nil {
t.Fatalf("restic composed from its build: %v", err) t.Fatalf("restic composed from its build: %v", err)
} }
@@ -597,8 +646,8 @@ func TestInput(t *testing.T) {
// A done key an artifact of which names no reference is waited for, in the controller's words (ADR 0498 §8). // A done key an artifact of which names no reference is waited for, in the controller's words (ADR 0498 §8).
unreferenced := movedMesh(true) unreferenced := movedMesh(true)
unreferenced.builds["restic "+movedFps["restic"]].Artifacts = []delivery.BuiltArtifact{{Name: "tools", Kind: "bundle", Digest: digestOf("6")}, resticArtifacts[1]} unreferenced.builds["restic "+movedFps["restic"]].Artifacts = []delivery.BuiltArtifact{{Name: "tools", Kind: "bundle", Digest: digestOf("6")}, resticArtifacts[1]}
if _, _, err := Compose(ctx, unreferenced, Mesh{}, Scope{Node: "n1"}, movedFps); err == nil || !strings.Contains(err.Error(), "the build of restic is done and waits for the reference of tools") { if _, _, held, err := Compose(ctx, unreferenced, Mesh{}, Scope{Node: "n1"}, movedFps); err != nil || !strings.Contains(held["restic"], "the build of restic is done and waits for the reference of tools") {
t.Errorf("a done key with an unreferenced artifact: %v", err) t.Errorf("a done key with an unreferenced artifact: %v %v, want restic held alone", err, held)
} }
// A module the node runs inside its own composition (the fold) composes from the indexer's manifest though a done // A module the node runs inside its own composition (the fold) composes from the indexer's manifest though a done
// build of it is at the trunk's FP2: its build is not read. // build of it is at the trunk's FP2: its build is not read.
@@ -662,12 +711,12 @@ func TestInput(t *testing.T) {
} }
return b return b
} }
if _, _, err := Compose(ctx, withMailer(false), Mesh{}, Scope{Node: "laptop"}, fps); err == nil || !strings.Contains(err.Error(), `nothing sets "domain"`) { if _, _, _, err := Compose(ctx, withMailer(false), Mesh{}, Scope{Node: "laptop"}, fps); err == nil || !strings.Contains(err.Error(), `nothing sets "domain"`) {
t.Errorf("a provider's served setting nobody sets: %v", err) t.Errorf("a provider's served setting nobody sets: %v", err)
} }
served := withMailer(true) served := withMailer(true)
if _, _, err := Compose(ctx, served, Mesh{}, Scope{Node: "laptop"}, fps); err != nil { if _, _, held, err := Compose(ctx, served, Mesh{}, Scope{Node: "laptop"}, fps); err != nil || len(held) > 0 {
t.Errorf("laptop with what anchor's mailer serves: %v", err) t.Errorf("laptop with what anchor's mailer serves: %v %v", err, held)
} }
if !reflect.DeepEqual(served.servedRead, []string{"anchor.mailer:domain"}) || slices.Contains(served.valuesRead, "anchor.mailer") { if !reflect.DeepEqual(served.servedRead, []string{"anchor.mailer:domain"}) || slices.Contains(served.valuesRead, "anchor.mailer") {
t.Errorf("laptop read %v of other nodes (and %v of config values): the served keys of anchor's mailer alone", served.servedRead, served.valuesRead) t.Errorf("laptop read %v of other nodes (and %v of config values): the served keys of anchor's mailer alone", served.servedRead, served.valuesRead)
@@ -0,0 +1,129 @@
{
"module": "de-spiegel",
"version": "1",
"slug": "spiegel",
"capabilities": [
"container-runtime"
],
"requires": [
"route",
"smtp"
],
"contributes": {
"route": {
"site": {
"label": "de-spiegel",
"port": 35621
},
"praktijk": {
"name": "${config:praktijk-name}",
"port": 35621
},
"praktijk-www": {
"name": "${config:praktijk-www-name}",
"redirect": "https://${config:praktijk-name}"
},
"psychologie": {
"name": "${config:psychologie-name}",
"port": 35621
},
"psychologie-www": {
"name": "${config:psychologie-www-name}",
"redirect": "https://${config:psychologie-name}"
}
},
"smtp": {
"account": "de-spiegel"
}
},
"binds": {
"route": "/var/lib/de-spiegel/route.json",
"smtp": "/var/lib/de-spiegel/smtp.json"
},
"config": {
"praktijk-name": {
"type": "string",
"description": "The public name the practice's site is served on; the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"praktijk-www-name": {
"type": "string",
"description": "The public name that redirects to the practice's site (its www form); the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"psychologie-name": {
"type": "string",
"description": "The public name the psychology practice's site is served on; the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
},
"psychologie-www-name": {
"type": "string",
"description": "The public name that redirects to the psychology practice's site (its www form); the operator's, with no default (novox/hq ADR 0112).",
"trusted": true,
"required": true
}
},
"listens": [
{
"port": 35621,
"protocol": "tcp",
"from": "mesh",
"why": "the de-spiegel site and its /contact endpoint over http; the practice's three public names are route grants, and the proxy reaches every one of them here"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"path": "/var/lib/de-spiegel",
"mode": "0700"
},
{
"id": "server-env",
"type": "file",
"path": "/var/lib/de-spiegel/server.env",
"mode": "0600",
"content": "SMTP_HOST=${bound:smtp:at}\nSMTP_PORT=${bound:smtp:port}\nSMTP_AUTH_USER=de-spiegel@${bound:smtp:domain}\nSMTP_AUTH_PASS=${secret:smtp}\n"
},
{
"id": "net",
"type": "network",
"name": "de-spiegel"
},
{
"id": "server",
"type": "container",
"name": "de-spiegel",
"artifact": "site",
"network": "de-spiegel",
"env-file": [
"/var/lib/de-spiegel/server.env"
],
"ports": [
"35621"
],
"secrets-in-environment": "the application's own code reads SMTP_AUTH_USER/PASS from the environment (server/index.js); converting is this repository's change"
}
],
"build": {
"on": [
{
"arg": "NODE_BASE",
"image": "node@sha256:48e4b67d85f87bd551df43704e24d252f56cc5f8e9718841aace50f19948f0f9"
}
],
"artifacts": [
{
"name": "site",
"kind": "image",
"from": "Dockerfile"
}
]
},
"secrets": {
"smtp": "/var/lib/de-spiegel/smtp.secret"
}
}
@@ -0,0 +1,82 @@
{
"module": "f1-circuits",
"slug": "f1",
"version": "1",
"capabilities": [
"container-runtime"
],
"requires": [
"route"
],
"contributes": {
"route": {
"label": "f1-circuits",
"endpoint": "web"
}
},
"binds": {
"route": "${dir:state}/route.json"
},
"listens": [
{
"name": "web",
"port": 8000,
"protocol": "tcp",
"from": "mesh",
"why": "the track-layout CDN, the race posters and the Kometa metadata over http; its public name is a route grant and the proxy reaches it here. Kometa fetches the metadata and every poster by that public name"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "cache",
"type": "directory",
"mode": "0700"
},
{
"id": "site-env",
"type": "file",
"path": "${dir:state}/site.env",
"mode": "0644",
"content": "F1_CIRCUITS_BASE_URL=https://${bound:route:name}\n"
},
{
"id": "server",
"type": "container",
"name": "f1-circuits",
"artifact": "cdn",
"ports": [
"8000"
],
"env-file": [
"${dir:state}/site.env"
],
"volumes": [
"${dir:cache}:/app/cache"
],
"restart-on": [
"site-env"
]
}
],
"build": {
"on": [
{
"arg": "PYTHON_BASE",
"image": "python@sha256:6d43704baacd1bfbe7c295d7f13079d5d8104ed33568873133f8fc69980419df"
}
],
"artifacts": [
{
"name": "cdn",
"kind": "image",
"from": "service/Dockerfile"
}
]
}
}
+16
View File
@@ -68,6 +68,10 @@ type Input struct {
// OperatorExport is every operator-sealed secret in the mesh, for a module that keeps them (inv.OperatorExport): sealed // OperatorExport is every operator-sealed secret in the mesh, for a module that keeps them (inv.OperatorExport): sealed
// values the composition carries without opening. Nil when the mesh has no operator key. // values the composition carries without opening. Nil when the mesh has no operator key.
OperatorExport *catalogue.KeptExport OperatorExport *catalogue.KeptExport
// KeptRuns is, by node and module, the file of the export as that node runs it (its resource of a module's running
// composition): what a node composing its own carries where it holds no OperatorExport (catalogue.Rendering.KeptRuns,
// issue 1126, C6). Used only while OperatorExport is nil.
KeptRuns map[string]map[string]map[string]any
// Issued is the credentials made for the consumers of each provider (inv.SecretsFrom): one row for each pair credential // Issued is the credentials made for the consumers of each provider (inv.SecretsFrom): one row for each pair credential
// that exists, by the provider node it is for. The grants a provider must create are read from these, so a credential not // that exists, by the provider node it is for. The grants a provider must create are read from these, so a credential not
@@ -200,6 +204,18 @@ type Node struct {
// so it is left to the node: Compositions composes it as a stub (Stub) and carries its resources as they run, each // so it is left to the node: Compositions composes it as a stub (Stub) and carries its resources as they run, each
// after the resource it runs after (issue 1126). Nil composes every folded module from its manifest, as before. // after the resource it runs after (issue 1126). Nil composes every folded module from its manifest, as before.
RunsOwn *delivery.Composed RunsOwn *delivery.Composed
// Held is the modules refused alone on this node, each composed as what runs of it (Carried, in Kept; its Stub when
// nothing of it runs) so the rest is composed (novox/hq issue 1126): none of a held module's own composition is in what
// Compositions answers, so the node keeps what it last applied of it; one whose resources the node's own runs
// (RunsOwn), folded or leaving the fold, is carried there as it runs, whatever its manifest names, so the node's own
// never drops it.
Held map[string]bool
// RunsTools is, by module, the tools bundles the node's runtime serves of it as it runs (ToolsRunning of what the
// node runs); nil when not known. A module composed as what runs of it (its Stub or Carried: held, or folded at a
// build of the old form) contributes these to the runtime, never its manifest's: a manifest no build resolved has no
// bundles, and a held module's new build's bundles are not delivered (issue 1126, 2026-10-29: shanks's runtime lost
// the folded systemd-resolved, novox's the folded nats).
RunsTools map[string][]catalogue.Bundle
} }
// Place is where a node is, as its placement says (inv.Overlay). // Place is where a node is, as its placement says (inv.Overlay).
+372
View File
@@ -0,0 +1,372 @@
// Package judge is how the seat mesh-placement's holder judges a proposal (novox/hq ADR 0567 §3, §4): which nodes must
// answer it and what it changes on each (Intended), and the verdict of their answers (JudgeAnswers). Moved here from
// the assigner, where it judged a catalogue pull request's placement (ADR 0559 §5), with a proposal in place of a pull
// request's change. It holds no bus client and no timer: the holder gives it what it read.
package judge
import (
"encoding/json"
"fmt"
"reflect"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// AnswerBound is how long the holder waits for a node's answer after its candidate was written (ADR 0567 §4): the time to
// live of the node's pulse key (delivery.PulseAnswerKey), whose expiry names it silent, never passed. A variable so a test
// on the bus can shorten it.
var AnswerBound = 10 * time.Minute
// Intended is, by node, the modules whose composition the proposal changes, and the nodes that must answer it (§3).
// current is the composition input with the current placement and laidInput the same with the proposal's placements
// (laid, by node; nil removes the node's key) laid over it; nodes is every node to look at (every node with a record).
// A node's set holds the modules the proposal adds, removes or whose placed entry differs on that node, and those whose
// composition differs between the two inputs (delivery.CompareComposed), the node's own named "node". Must is every node
// whose key the proposal changes and every other whose set is not empty, sorted. A node that has no key now gets an
// empty set (§7: the seed is answered as the same). A node whose compositions cannot be made on either side keeps the
// modules its entries name, said in notes; such a node is in Must only when its key changes.
func Intended(current, laidInput *compose.Input, currentPlaced map[string]delivery.Placement, laid map[string]*delivery.Placement,
nodes []string, fps map[string]string) (intended map[string]map[string]bool, must []string, notes []string) {
intended = map[string]map[string]bool{}
inMust := map[string]bool{}
all := map[string]bool{}
for _, n := range nodes {
all[n] = true
}
for n := range laid {
all[n] = true
inMust[n] = true
}
names := make([]string, 0, len(all))
for n := range all {
names = append(names, n)
}
sort.Strings(names)
for _, node := range names {
before, placed := currentPlaced[node]
set := map[string]bool{}
after, changesKey := laid[node]
if changesKey && !placed {
intended[node] = set // a node new to placement: every answer must be the same
continue
}
if changesKey {
var afterModules []delivery.PlacedModule
if after != nil {
afterModules = after.Modules
}
entries := map[string]delivery.PlacedModule{}
for _, m := range before.Modules {
entries[m.Module] = m
}
seen := map[string]bool{}
for _, m := range afterModules {
seen[m.Module] = true
if e, ok := entries[m.Module]; !ok || !reflect.DeepEqual(e, m) {
set[m.Module] = true
}
}
for name := range entries {
if !seen[name] {
set[name] = true
}
}
}
if placed && current != nil && laidInput != nil {
was, _, _, errB := compose.CompositionsShadow(*current, node, fps)
now, _, _, errA := compose.CompositionsShadow(*laidInput, node, fps)
if errA == nil && errB == nil {
running := make([]delivery.Composed, 0, len(was))
for _, c := range was {
running = append(running, c)
}
cmp := delivery.CompareComposed(now, running)
for _, d := range cmp.Differs {
set[d.Module] = true
}
for _, d := range cmp.Added {
set[d.Module] = true
}
for _, m := range cmp.Removed {
set[m] = true
}
} else if changesKey {
notes = append(notes, fmt.Sprintf("what the proposal changes on %s is read from its placement alone: its "+
"compositions cannot be made (now: %v; proposed: %v)", node, errB, errA))
}
}
if len(set) > 0 {
inMust[node] = true
}
if inMust[node] {
intended[node] = set
}
}
for n := range inMust {
must = append(must, n)
}
sort.Strings(must)
return intended, must, notes
}
// JudgeAnswers is the verdict of a proposal's answers (§4). answers is by node its HaveNode.Candidates entry for the
// proposal (nil when it has none) and revisions by node the revision of its candidate key: an answer counts when it
// names the proposal and that revision. It fails on a refusal by a node that must answer, and on every node of Must with
// no answer that counts once its pulse key has ended (pulsing false: the bound's expiry, heard on the bus); while it
// stands it waits for them.
// A difference fails nothing: one Intended computed is listed under Intended, one it did not under Unintended, both
// shown in the ask's brief. The seed alone fails on any difference.
func JudgeAnswers(p delivery.PlacementProposal, intended map[string]map[string]bool, answers map[string]*delivery.CandidateAnswer,
revisions map[string]uint64, online map[string]bool, pulsing map[string]bool, now time.Time) delivery.PlacementVerdict {
v := delivery.PlacementVerdict{At: now.UTC(), Intended: map[string][]string{}, Unintended: map[string][]string{}, Carried: map[string][]string{},
Answers: map[string]delivery.CandidateAnswer{}}
for node, set := range intended {
for m := range set {
v.Intended[node] = append(v.Intended[node], m)
}
sort.Strings(v.Intended[node])
}
must := p.Must
if len(must) == 0 {
must = p.Nodes()
}
for _, node := range must {
a := answers[node]
if a == nil || a.Proposal != p.ID || a.Revision == 0 || a.Revision != revisions[node] {
v.Silent = append(v.Silent, node)
continue
}
v.Answers[node] = *a
if a.Refused != "" {
v.Fails = append(v.Fails, fmt.Sprintf("%s cannot compose the proposal: %s", node, a.Refused))
continue
}
for _, r := range a.Refusals {
v.Fails = append(v.Fails, fmt.Sprintf("%s refuses the proposal for %s: %s", node, r.Module, r.Why))
}
want := intended[node]
differs := func(module, words string) {
if p.Seed {
// The seed carries what its nodes found, each named in the brief for the operator who approves it.
v.Carried[node] = append(v.Carried[node], words)
return
}
if !want[module] {
v.Unintended[node] = append(v.Unintended[node], module)
}
}
for _, d := range a.Differs {
differs(d.Module, differenceWords(d))
}
for _, d := range a.Added {
differs(d.Module, "would run "+d.Module)
}
for _, m := range a.Removed {
differs(m, "would stop running "+m)
}
sort.Strings(v.Unintended[node])
}
if len(v.Intended) == 0 {
v.Intended = nil
}
if len(v.Unintended) == 0 {
v.Unintended = nil
}
if len(v.Carried) == 0 {
v.Carried = nil
}
if len(v.Answers) == 0 {
v.Answers = nil
}
var waiting, gone []string
for _, node := range v.Silent {
named := node
if online != nil && !online[node] {
named += " (offline)"
}
if pulsing[node] {
waiting = append(waiting, named)
} else {
gone = append(gone, named)
}
}
if len(waiting) > 0 {
v.Notes = append(v.Notes, fmt.Sprintf("waits for the answers of %s, each for %s from its candidate's write",
strings.Join(waiting, ", "), AnswerBound))
}
if len(gone) > 0 {
v.Fails = append(v.Fails, fmt.Sprintf("%s did not answer within %s of its candidate's write; judged again when it "+
"answers", strings.Join(gone, ", "), AnswerBound))
}
switch {
case len(v.Fails) > 0:
v.State = delivery.VerdictFail
case len(waiting) > 0:
v.State = delivery.VerdictWaits
default:
v.State = delivery.VerdictPass
}
return v
}
// Brief is the verdict as the ask's brief shows it, by node (§5): what the proposal changes and what each node answered
// beside it, marked, and what stays held as the node holds it today, with the proposal's why.
func Brief(p delivery.PlacementProposal, v delivery.PlacementVerdict) string {
var b strings.Builder
fmt.Fprintf(&b, "%s (proposed by %s)", p.Why, p.By)
nodes := map[string]bool{}
for n := range v.Intended {
nodes[n] = true
}
for n := range v.Unintended {
nodes[n] = true
}
for _, n := range p.Nodes() {
nodes[n] = true
}
for n, a := range v.Answers {
if len(a.Held) > 0 {
nodes[n] = true
}
}
for n := range v.Carried {
nodes[n] = true
}
names := make([]string, 0, len(nodes))
for n := range nodes {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
fmt.Fprintf(&b, "; %s:", n)
if m := v.Intended[n]; len(m) > 0 {
fmt.Fprintf(&b, " changes %s", strings.Join(m, ", "))
} else {
b.WriteString(" composes the same")
}
if m := v.Unintended[n]; len(m) > 0 {
fmt.Fprintf(&b, "; also differs, not computed: %s", strings.Join(m, ", "))
}
if m := v.Carried[n]; len(m) > 0 {
fmt.Fprintf(&b, "; the seed carries: %s", strings.Join(m, "; "))
}
if held := v.Answers[n].Held; len(held) > 0 {
var names []string
for _, h := range held {
today := h.Today
if today == "" {
today = "held"
}
names = append(names, h.Module+" ("+today+")")
}
fmt.Fprintf(&b, "; stays as today: %s", strings.Join(names, ", "))
}
}
return b.String()
}
// differenceWords names one module's difference: the module, the first field that differs in it with its value as it
// runs and as the seed composes it, each cut to about 60 characters ("(value cut)" where a body shown is cut before the
// field), and how many fields more; or the resource and why when no body is shown.
func differenceWords(d delivery.ModuleDiffers) string {
for _, b := range d.Bodies {
if len(b.Paths) == 0 {
continue
}
path := b.Paths[0]
words := fmt.Sprintf("%s: %s %s → %s", d.Module, fieldName(b.Running, b.Composed, path),
valueAt(b.Running, path), valueAt(b.Composed, path))
if n := len(b.Paths) - 1; n > 0 {
words += fmt.Sprintf(" (and %d more fields)", n)
}
return words
}
if d.Resource != "" {
return fmt.Sprintf("%s: %s (%s)", d.Module, d.Resource, d.Why)
}
return fmt.Sprintf("%s: %s", d.Module, d.Why)
}
// shownValue bounds a value named in a brief.
const shownValue = 60
// walk follows a differing path (delivery.DifferingPaths: keys joined by dots, a key may hold dots, and a JSON value
// written in a string is read as JSON) through a body, answering the keys it passed and the value at its end.
func walk(body, path string) (keys []string, value any, ok bool) {
var v any
if json.Unmarshal([]byte(body), &v) != nil {
return nil, nil, false
}
rest := path
for rest != "" {
if s, isString := v.(string); isString {
var inner any
if json.Unmarshal([]byte(s), &inner) != nil {
return nil, nil, false
}
v = inner
}
switch t := v.(type) {
case map[string]any:
best := ""
for k := range t {
if (rest == k || strings.HasPrefix(rest, k+".")) && len(k) > len(best) {
best = k
}
}
if best == "" {
return nil, nil, false
}
keys, v, rest = append(keys, best), t[best], strings.TrimPrefix(strings.TrimPrefix(rest, best), ".")
case []any:
seg, after, _ := strings.Cut(rest, ".")
n, err := strconv.Atoi(seg)
if err != nil || n < 0 || n >= len(t) {
return nil, nil, false
}
keys, v, rest = append(keys, seg), t[n], after
default:
return nil, nil, false
}
}
return keys, v, true
}
// valueAt is the value at path in a shown body, cut to shownValue, or why it is not shown.
func valueAt(body, path string) string {
if body == "" {
return "(none)"
}
_, v, ok := walk(body, path)
if !ok {
return "(value cut)"
}
s, isString := v.(string)
if !isString {
raw, _ := json.Marshal(v)
s = string(raw)
}
if r := []rune(s); len(r) > shownValue {
s = string(r[:shownValue-1]) + "…"
}
return s
}
// fieldName is the differing field as a person reads it: its parent's name and its own, `PARENT[key]`, from whichever
// body reaches it; the path's end when neither does.
func fieldName(running, composed, path string) string {
for _, body := range []string{running, composed} {
if keys, _, ok := walk(body, path); ok && len(keys) > 0 {
if len(keys) == 1 {
return keys[0]
}
return keys[len(keys)-2] + "[" + keys[len(keys)-1] + "]"
}
}
return path
}
+89
View File
@@ -0,0 +1,89 @@
package judge
import (
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0567 §3, §4 (rows moved from the assigner's check of a pull request's placement, ADR 0559 §5): what a proposal
// changes by node, and the verdict of the answers: differences shown and failing nothing, a refusal and a silent node
// failing, a later answer judging again, the seed failing on any difference.
func TestJudge(t *testing.T) {
current := map[string]delivery.Placement{"anchor": {Node: "anchor", Modules: []delivery.PlacedModule{{Module: "mail"}}}, "server": {Node: "server"}}
laid := map[string]*delivery.Placement{
"anchor": {Node: "anchor", Modules: []delivery.PlacedModule{{Module: "mail", Ports: map[string]int{"25": 2525}}, {Module: "web"}}},
"laptop": {Node: "laptop", Modules: []delivery.PlacedModule{{Module: "web"}}}}
got, must, _ := Intended(nil, nil, current, laid, []string{"anchor", "laptop", "server"}, nil)
if !got["anchor"]["mail"] || !got["anchor"]["web"] || len(got["laptop"]) != 0 || !slices.Equal(must, []string{"anchor", "laptop"}) {
t.Errorf("intended = %v, must %v: want anchor's mail and web, nothing on the laptop that has no key, server untouched", got, must)
}
at := time.Date(2026, 10, 27, 2, 0, 0, 0, time.UTC)
p := delivery.PlacementProposal{ID: "p-one", By: "op", Why: "move web", Written: at, Base: map[string]uint64{"anchor": 3, "laptop": 0},
Must: []string{"anchor", "laptop"}}
revisions := map[string]uint64{"anchor": 7, "laptop": 8}
intended := map[string]map[string]bool{"anchor": {"web": true}}
same := func(rev uint64) *delivery.CandidateAnswer {
return &delivery.CandidateAnswer{Proposal: "p-one", Revision: rev, Verdict: delivery.CandidateSame, Same: []string{"mail"}}
}
differs := func(rev uint64, module string) *delivery.CandidateAnswer {
return &delivery.CandidateAnswer{Proposal: "p-one", Revision: rev, Verdict: delivery.CandidateDiffers,
Differs: []delivery.ModuleDiffers{{Module: module, Why: "the port differs"}}}
}
for _, row := range []struct {
name string
seed bool
answers map[string]*delivery.CandidateAnswer
pulsing map[string]bool
state string
words string
marked string
}{
{"all the same", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
{"a module added where intended", false, map[string]*delivery.CandidateAnswer{"anchor": {Proposal: "p-one", Revision: 7, Verdict: delivery.CandidateDiffers,
Added: []delivery.ModuleAdded{{Module: "web", Unissued: []string{"its own secret"}}}}, "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
{"a difference not computed passes, marked", false, map[string]*delivery.CandidateAnswer{"anchor": differs(7, "mail"), "laptop": same(8)}, nil, delivery.VerdictPass, "", "mail"},
{"the seed carries a difference, named", true, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8,
Verdict: delivery.CandidateDiffers, Differs: []delivery.ModuleDiffers{{Module: "node-tools", Resource: "node-tools.runtime", Why: "the resource differs",
Bodies: []delivery.ResourceBodies{{ID: "node-tools.runtime", Paths: []string{"env.MESH_OWNED_PATHS./etc/resolv.conf", "env.MESH_TOOL_MODULES"},
Running: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/hosts\":\"hostname\",\"/etc/resolv.conf\":\"networkmanager (node-uplink)\"}"}}`,
Composed: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/hosts\":\"hostname\",\"/etc/resolv.conf\":\"systemd-resolved (node-resolver)\"}","MESH_TOOL_MODULES":"… (cut at 8192 bytes)`}}}}}}, nil, delivery.VerdictPass, "", ""},
{"a refusal", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8, Verdict: delivery.CandidateRefused,
Refusals: []delivery.ModuleRefused{{Module: "web", Why: "no own secret"}}}}, nil, delivery.VerdictFail, "laptop refuses", ""},
{"a node not answering, within its bound", false, map[string]*delivery.CandidateAnswer{"anchor": same(7)}, map[string]bool{"laptop": true}, delivery.VerdictWaits, "laptop (offline)", ""},
{"an answer of an earlier revision, past its bound", false, map[string]*delivery.CandidateAnswer{"anchor": same(6), "laptop": same(8)}, nil, delivery.VerdictFail, "anchor did not answer", ""},
{"a module held as today passes, shown in the brief", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8,
Verdict: delivery.CandidateSame, Held: []delivery.ModuleRefused{{Module: "spiegel", Why: "a required value is unset"}}}}, nil, delivery.VerdictPass, "", ""},
{"its later answer", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
} {
q := p
q.Seed = row.seed
v := JudgeAnswers(q, intended, row.answers, revisions, map[string]bool{"anchor": true}, row.pulsing, at)
all := strings.Join(append(append([]string{}, v.Fails...), v.Notes...), "; ")
if v.State != row.state || !strings.Contains(all, row.words) {
t.Errorf("%s: %s, %q", row.name, v.State, all)
}
if row.seed && row.state == delivery.VerdictPass && !strings.Contains(Brief(q, v), "the seed carries: node-tools: MESH_OWNED_PATHS[/etc/resolv.conf] networkmanager (node-uplink) → (value cut) (and 1 more fields)") {
t.Errorf("%s: the brief does not name it: %s", row.name, Brief(q, v))
}
if row.seed && row.state == delivery.VerdictPass {
whole := row.answers["laptop"].Differs[0]
whole.Bodies = []delivery.ResourceBodies{{ID: "r", Paths: []string{"env.MESH_OWNED_PATHS./etc/resolv.conf"},
Running: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/resolv.conf\":\"networkmanager (node-uplink)\"}"}}`,
Composed: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/resolv.conf\":\"systemd-resolved (node-resolver)\"}"}}`}}
if got := differenceWords(whole); got != "node-tools: MESH_OWNED_PATHS[/etc/resolv.conf] networkmanager (node-uplink) → systemd-resolved (node-resolver)" {
t.Errorf("the difference is named %q", got)
}
}
if strings.HasPrefix(row.name, "a module held as today") && !strings.Contains(Brief(q, v), "stays as today: spiegel (held)") {
t.Errorf("%s: the brief does not show it: %s", row.name, Brief(q, v))
}
if row.marked != "" && (!slices.Contains(v.Unintended["anchor"], row.marked) || !strings.Contains(Brief(q, v), "not computed: "+row.marked)) {
t.Errorf("%s: the difference is not marked: %+v, %s", row.name, v.Unintended, Brief(q, v))
}
}
}
+3
View File
@@ -128,6 +128,9 @@ func (c *composer) places(nodeName string, resolved Resolved) (Placed, error) {
continue continue
} }
r.Kept = c.in.OperatorExport r.Kept = c.in.OperatorExport
if r.Kept == nil {
r.KeptRuns = c.in.KeptRuns[node.Name]
}
break break
} }
+177
View File
@@ -0,0 +1,177 @@
package delivery
import (
"bytes"
"encoding/json"
"sort"
"strconv"
"strings"
)
// DifferingPaths is every JSON path at which two bodies of one resource differ, sorted (issue 1126, T1: the shadow
// shows its work): object keys and list places joined by dots (`content.url`, `links.0`), a string that holds a JSON
// object or list read as that JSON and walked on (a runtime's config carried as text), a key or place one side alone
// holds named as it is. A key a secret could be under (secretWords) is named and not walked into. A body that is not
// JSON differs at "" (the whole), and a resource one side alone holds has none. At most MaxPaths: the last then says how many more there are. A path names keys
// alone, never a value, so it holds no secret.
func DifferingPaths(composed, running json.RawMessage) []string {
if composed == nil || running == nil {
return nil // one side alone holds it: its why says so
}
var a, b any
okA := unmarshalNumbers(composed, &a) == nil
okB := unmarshalNumbers(running, &b) == nil
if !okA || !okB {
if bytes.Equal(composed, running) {
return nil
}
return []string{""}
}
var out []string
walkDiffering(a, b, "", &out)
sort.Strings(out)
if len(out) > MaxPaths {
more := len(out) - (MaxPaths - 1)
out = append(out[:MaxPaths-1], "… "+strconv.Itoa(more)+" more")
}
return out
}
func unmarshalNumbers(raw []byte, into *any) error {
d := json.NewDecoder(bytes.NewReader(raw))
d.UseNumber()
return d.Decode(into)
}
// asJSON is a string holding a JSON object or list read as that value; v itself otherwise.
func asJSON(v any) any {
s, ok := v.(string)
if !ok {
return v
}
t := strings.TrimSpace(s)
if t == "" || (t[0] != '{' && t[0] != '[') {
return v
}
var parsed any
if unmarshalNumbers([]byte(t), &parsed) != nil {
return v
}
return parsed
}
func join(at, key string) string {
if at == "" {
return key
}
return at + "." + key
}
func walkDiffering(a, b any, at string, out *[]string) {
a, b = asJSON(a), asJSON(b)
ma, isMA := a.(map[string]any)
mb, isMB := b.(map[string]any)
if isMA && isMB {
keys := map[string]bool{}
for k := range ma {
keys[k] = true
}
for k := range mb {
keys[k] = true
}
for k := range keys {
x, inA := ma[k]
y, inB := mb[k]
switch {
case !inA || !inB:
*out = append(*out, join(at, k))
case secretKey(k):
// A file's secrets (SecretsField) are named entry by entry, each entry never entered; any other
// object under a secret key, a vault box included, is one path.
sx, isSX := x.(map[string]any)
sy, isSY := y.(map[string]any)
if k != SecretsField || !isSX || !isSY {
if !sameAny(x, y) {
*out = append(*out, join(at, k))
}
continue
}
for name := range sx {
if v, in := sy[name]; !in || !sameAny(sx[name], v) {
*out = append(*out, join(join(at, k), name))
}
}
for name := range sy {
if _, in := sx[name]; !in {
*out = append(*out, join(join(at, k), name))
}
}
default:
walkDiffering(x, y, join(at, k), out)
}
}
return
}
la, isLA := a.([]any)
lb, isLB := b.([]any)
if isLA && isLB {
for i := range max(len(la), len(lb)) {
if i >= len(la) || i >= len(lb) {
*out = append(*out, join(at, strconv.Itoa(i)))
continue
}
walkDiffering(la[i], lb[i], join(at, strconv.Itoa(i)), out)
}
return
}
if !sameAny(a, b) {
*out = append(*out, at)
}
}
func sameAny(a, b any) bool {
x, errA := json.Marshal(a)
y, errB := json.Marshal(b)
return errA == nil && errB == nil && bytes.Equal(x, y)
}
// FitAnswer is a with its shown bodies cut until its JSON is within limit bytes (MaxAnswerBytes for the node's key):
// first the bodies' texts, from the last differing module back, keeping their paths; then the paths too. What is cut
// is said in the answer's note. The verdict and every list of modules are never cut.
func FitAnswer(a CandidateAnswer, limit int) CandidateAnswer {
size := func() int {
raw, err := json.Marshal(a)
if err != nil {
return 0
}
return len(raw)
}
if size() <= limit {
return a
}
a.Differs = append([]ModuleDiffers(nil), a.Differs...)
for i := range a.Differs {
a.Differs[i].Bodies = append([]ResourceBodies(nil), a.Differs[i].Bodies...)
}
cutTexts, cutPaths := false, false
for i := len(a.Differs) - 1; i >= 0 && size() > limit; i-- {
for j := range a.Differs[i].Bodies {
a.Differs[i].Bodies[j].Composed, a.Differs[i].Bodies[j].Running = "", ""
}
cutTexts = true
}
for i := len(a.Differs) - 1; i >= 0 && size() > limit; i-- {
for j := range a.Differs[i].Bodies {
a.Differs[i].Bodies[j].Paths = nil
}
cutPaths = true
}
why := "the bodies shown were cut to keep the answer within " + strconv.Itoa(limit>>10) + " KiB"
if cutPaths {
why += ", their paths too"
}
if cutTexts {
a.Note = strings.TrimPrefix(a.Note+"; ", "; ") + why
}
return a
}
+19
View File
@@ -298,6 +298,25 @@ func TestControllerGrant(t *testing.T) {
if _, err := EncodeControllerGrant(ControllerGrant{Accounts: []string{""}}); err == nil { if _, err := EncodeControllerGrant(ControllerGrant{Accounts: []string{""}}); err == nil {
t.Errorf("a grant naming an empty account was accepted") t.Errorf("a grant naming an empty account was accepted")
} }
// The grants part's keys are the controller's: the same key round-trips; the source is the part's bucket only for a
// handed-away lease (issue 774).
if pk, err := PartGrantKey("mesh-delivery-assigner", fp); err != nil || pk != key {
t.Errorf("PartGrantKey = %q, %v, want the controller's %q", pk, err, key)
}
if m, f, err := ParsePartGrantKey(key); err != nil || m != "mesh-delivery-assigner" || f != fp {
t.Errorf("ParsePartGrantKey = %q, %q, %v", m, f, err)
}
for _, c := range []struct {
away bool
bucket, read string
}{
{false, "mesh-controller_grants", "mesh-controller.grants"},
{true, "mesh-grants_grants", "mesh-grants.grants"},
} {
if b, r := GrantSource(c.away); b != c.bucket || r != c.read {
t.Errorf("GrantSource(%v) = %q, %q, want %q, %q", c.away, b, r, c.bucket, c.read)
}
}
g := ControllerGrant{Accounts: []string{"ace.mesh-build-go", "shanks.mesh-mail.reader"}} g := ControllerGrant{Accounts: []string{"ace.mesh-build-go", "shanks.mesh-mail.reader"}}
for _, c := range []struct { for _, c := range []struct {
g ControllerGrant g ControllerGrant
+48
View File
@@ -109,6 +109,31 @@ func TestAnAskerPutsOnceAndReadsTheStep(t *testing.T) {
if err != nil || !IsLost(step) { if err != nil || !IsLost(step) {
t.Fatalf("an ask heard given up after the hearing came back: %+v, %v", step, err) t.Fatalf("an ask heard given up after the hearing came back: %+v, %v", step, err)
} }
// The lost check of 2026-10-28: a step whose own watch went deaf is read at the next nudge (a step of its build module
// ended) and answered; a nudge of another build module reads nothing; a watch that closes is opened again.
d := &FakeAsks{}
heard := &HeardEnds{Inner: deafWatch{}, Read: d}
k = Asker{Publisher: d, Waiter: heard, Bound: time.Minute}
d.Write(delivery.Step{ID: "mvl0frv18d1980", Node: "shanks", State: delivery.StepPassed})
got := make(chan delivery.Step, 1)
go func() { s, _ := k.Wait(context.Background(), goAsk("mvl0frv18d1980")); got <- s }()
for heard.Waiting() == 0 {
time.Sleep(time.Millisecond)
}
heard.Nudge(NodeBuildModule)
select {
case s := <-got:
t.Fatalf("a nudge of another build module answered %+v", s)
case <-time.After(20 * time.Millisecond):
}
heard.Nudge(GoBuildModule)
if s := <-got; s.State != delivery.StepPassed || s.Node != "shanks" {
t.Fatalf("a step its watch never handed over: %+v", s)
}
closing := &HeardEnds{Inner: &closingWatch{inner: d}, Read: d}
if s, err := (Asker{Publisher: d, Waiter: closing}).Wait(context.Background(), goAsk("mvl0frv18d1980")); err != nil || s.State != delivery.StepPassed {
t.Fatalf("a watch that closed once: %+v, %v", s, err)
}
for _, row := range []struct { for _, row := range []struct {
h *Hearing h *Hearing
want string want string
@@ -165,3 +190,26 @@ func (f *refusingFollower) Follow(_ context.Context, _ string, _ time.Time, hear
func (f *refusingFollower) calls() int { f.mu.Lock(); defer f.mu.Unlock(); return f.n } func (f *refusingFollower) calls() int { f.mu.Lock(); defer f.mu.Unlock(); return f.n }
func (f *refusingFollower) heard(g GivenUp) { f.mu.Lock(); h := f.hand; f.mu.Unlock(); h(g) } func (f *refusingFollower) heard(g GivenUp) { f.mu.Lock(); h := f.hand; f.mu.Unlock(); h(g) }
// deafWatch is a watch of an ask's step that stays open and hears nothing.
type deafWatch struct{}
func (deafWatch) WatchAsk(context.Context, string, string) (<-chan delivery.Step, error) {
return make(chan delivery.Step), nil
}
// closingWatch closes its first watch at once, and hands the later ones to inner.
type closingWatch struct {
inner AskWaiter
closed bool
}
func (c *closingWatch) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
if !c.closed {
c.closed = true
ch := make(chan delivery.Step)
close(ch)
return ch, nil
}
return c.inner.WatchAsk(ctx, module, askID)
}
+236
View File
@@ -0,0 +1,236 @@
package buildrun
import (
"context"
"fmt"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A step's end is never only its own watch's to hear** (the lost check of 2026-10-28: mesh-catalog #553's verdict
// stayed running after its step mvsgthyjed5786 had passed on shanks at 19:09:31Z; the watch of that one key, opened at
// 18:46:05Z, heard neither the step's start nor its end and never closed, while a watch opened at 18:58:48Z heard its own
// step end). HeardEnds stands between the asker (Asker) and the watch of each ask's step key: when it is
// nudged, every ask it waits on reads its step keys now (a read, not the watch) and hands what it reads to its wait, so a
// step that ended is judged even when its own watch went deaf. A watch that closes under a wait is opened again, at most
// reopenTries times. It is nudged on an event (Nudge): any step of the same build module ending (one watch of each steps
// state, HearEnds), a reconnect to the bus, and whatever else the asker hears (the checker: checker_ask_again).
// It holds no timer. Every asker of the build seats waits through it (the checker and the builder, HearEnds).
// StepReader reads, now, every step a build module's steps state holds for an ask: its keys `*.<ask id>`.
type StepReader interface {
AskSteps(ctx context.Context, module, askID string) ([]delivery.Step, error)
}
// reopenTries is how often a wait opens its ask's watch again after it closed, before the wait ends failed.
const reopenTries = 3
// HeardEnds is an AskWaiter over Inner that also reads an ask's steps when nudged. Safe for concurrent use.
type HeardEnds struct {
Inner AskWaiter
Read StepReader
Logf func(format string, args ...any)
mu sync.Mutex
waiting map[string]*waitedAsk // by ask id
}
type waitedAsk struct {
module string
nudge chan struct{}
}
func (h *HeardEnds) logf(format string, args ...any) {
if h.Logf != nil {
h.Logf(format, args...)
}
}
// WatchAsk is AskWaiter's: what Inner's watch hands, and, at each nudge, what the ask's keys hold now.
func (h *HeardEnds) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
in, err := h.Inner.WatchAsk(ctx, module, askID)
if err != nil {
return nil, err
}
w := &waitedAsk{module: module, nudge: make(chan struct{}, 1)}
h.mu.Lock()
if h.waiting == nil {
h.waiting = map[string]*waitedAsk{}
}
h.waiting[askID] = w
h.mu.Unlock()
out := make(chan delivery.Step)
go func() {
defer close(out)
defer func() {
h.mu.Lock()
if h.waiting[askID] == w {
delete(h.waiting, askID)
}
h.mu.Unlock()
}()
send := func(s delivery.Step) bool {
select {
case out <- s:
return true
case <-ctx.Done():
return false
}
}
reopened := 0
for {
select {
case <-ctx.Done():
return
case s, ok := <-in:
if !ok {
if ctx.Err() != nil || reopened >= reopenTries {
h.logf("the watch of the ask %s's step on %s closed %d times; its wait ends", askID, module, reopened+1)
return
}
reopened++
if in, err = h.Inner.WatchAsk(ctx, module, askID); err != nil {
h.logf("the watch of the ask %s's step on %s closed and cannot be opened again, so its wait ends: %v", askID, module, err)
return
}
h.logf("the watch of the ask %s's step on %s closed; opened again (%d of %d)", askID, module, reopened, reopenTries)
continue
}
if !send(s) {
return
}
case <-w.nudge:
if h.Read == nil {
continue
}
steps, err := h.Read.AskSteps(ctx, module, askID)
if err != nil {
h.logf("the steps of the ask %s on %s cannot be read: %v", askID, module, err)
continue
}
for _, s := range steps {
if s.State != delivery.StepRunning {
h.logf("the ask %s's step on %s ended %s, read at a nudge: its watch had not handed it over", askID, s.Node, s.State)
}
if !send(s) {
return
}
}
}
}
}()
return out, nil
}
// Nudge has every ask waited on of module read its steps now; module "" nudges every ask.
func (h *HeardEnds) Nudge(module string) {
h.mu.Lock()
defer h.mu.Unlock()
for _, w := range h.waiting {
if module == "" || w.module == module {
select {
case w.nudge <- struct{}{}:
default: // a nudge is pending already
}
}
}
}
// Waiting is how many asks are waited on now.
func (h *HeardEnds) Waiting() int {
h.mu.Lock()
defer h.mu.Unlock()
return len(h.waiting)
}
// AskSteps is StepReader's: the keys `*.<ask id>` of the build module's steps state, read now.
func (s JetStreamAsks) AskSteps(ctx context.Context, module, askID string) ([]delivery.Step, error) {
pattern, err := delivery.AskStepWatch(askID)
if err != nil {
return nil, err
}
kv, err := s.JS.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
lister, err := kv.ListKeysFiltered(ctx, pattern)
if err != nil {
return nil, err
}
var out []delivery.Step
for key := range lister.Keys() {
e, err := kv.Get(ctx, key)
if err != nil {
continue // deleted between the list and the read
}
if step, err := delivery.DecodeStep(e.Value()); err == nil {
out = append(out, step)
}
}
return out, nil
}
// HearEnds is the HeardEnds every asker of the build seats waits through: over inner (the asks' own watches, or a
// LostAskWaiter's inner), reading with asks, and nudged at each step end of each of modules (one watch of each steps state,
// updates only; one that closes is opened again, and said and left after reopenTries closes in a row that heard nothing). The asker nudges it at a reconnect too (Nudge("")).
func HearEnds(ctx context.Context, asks JetStreamAsks, inner AskWaiter, modules []string, logf func(string, ...any)) *HeardEnds {
if logf == nil {
logf = func(string, ...any) {}
}
h := &HeardEnds{Inner: inner, Read: asks, Logf: logf}
for _, module := range modules {
kv, err := asks.JS.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
logf("the steps of %s cannot be read (%v): an ask whose own watch goes deaf is read only at a reconnect", module, err)
continue
}
go h.follow(ctx, kv, module)
}
return h
}
func (h *HeardEnds) follow(ctx context.Context, kv jetstream.KeyValue, module string) {
silent := 0 // closes in a row having heard nothing
for ctx.Err() == nil {
w, err := kv.WatchAll(ctx, jetstream.UpdatesOnly())
if err != nil {
h.logf("the steps of %s cannot be watched, so an ask whose own watch goes deaf is read only at a reconnect: %v", module, err)
return
}
heard := false
for e := range w.Updates() {
if e == nil || e.Operation() != jetstream.KeyValuePut {
continue
}
heard = true
if step, err := delivery.DecodeStep(e.Value()); err == nil && step.State != delivery.StepRunning {
h.Nudge(module)
}
}
_ = w.Stop()
if ctx.Err() != nil {
return
}
h.Nudge(module) // a step may have ended while the watch was closing
if heard {
silent = 0
} else if silent++; silent >= reopenTries {
h.logf("the watch of the steps of %s closed %d times in a row having heard nothing, so it is left: an ask whose own watch goes deaf is read only at a reconnect", module, silent)
return
}
h.logf("the watch of the steps of %s closed; opened again", module)
}
}
// AskSteps is StepReader's: the last step written for the ask, if any.
func (f *FakeAsks) AskSteps(_ context.Context, _ string, askID string) ([]delivery.Step, error) {
f.mu.Lock()
defer f.mu.Unlock()
if s := f.steps[askID]; len(s) > 0 {
return []delivery.Step{s[len(s)-1]}, nil
}
return nil, nil
}
-26
View File
@@ -4,38 +4,12 @@ import (
"context" "context"
"fmt" "fmt"
"sort" "sort"
"strconv"
"strings"
"sync" "sync"
) )
// DefaultCapacity is the build capacity of a node that has no setting: one build step at a time. // DefaultCapacity is the build capacity of a node that has no setting: one build step at a time.
const DefaultCapacity = 1 const DefaultCapacity = 1
// ParseCapacity reads a capacity setting: `node=steps` pairs separated by commas, such as
// `shanks=3,ace=2`. A node without a pair has DefaultCapacity. A pair with no node, a count that is not a whole number of
// at least 0, and a node named twice are refused. A count of 0 keeps the part off the node: it is never chosen.
func ParseCapacity(s string) (map[string]int, error) {
out := map[string]int{}
for _, pair := range strings.Split(s, ",") {
pair = strings.TrimSpace(pair)
if pair == "" {
continue
}
node, count, ok := strings.Cut(pair, "=")
node, count = strings.TrimSpace(node), strings.TrimSpace(count)
n, err := strconv.Atoi(count)
if !ok || node == "" || err != nil || n < 0 {
return nil, fmt.Errorf("the capacity %q is not <node>=<steps at once, 0 or more>", pair)
}
if _, dup := out[node]; dup {
return nil, fmt.Errorf("the capacity names %s twice", node)
}
out[node] = n
}
return out, nil
}
// Places counts the build steps a part runs on each node and holds a lane back while its node is at its // Places counts the build steps a part runs on each node and holds a lane back while its node is at its
// capacity (the operator's decision of 2026-10-21; ADR 0478 clause 3). A lane is the fetch and the builds of one build // capacity (the operator's decision of 2026-10-21; ADR 0478 clause 3). A lane is the fetch and the builds of one build
// module on one node, run one step at a time, so a lane holds one place while it runs. A node is chosen for a lane by // module on one node, run one step at a time, so a lane holds one place while it runs. A node is chosen for a lane by
+2 -24
View File
@@ -7,21 +7,6 @@ import (
"time" "time"
) )
func TestParseCapacityReadsNodeEqualsCountPairs(t *testing.T) {
c, err := ParseCapacity(" shanks=3, ace=2 ")
if err != nil || c["shanks"] != 3 || c["ace"] != 2 || len(c) != 2 {
t.Errorf("capacity = %v, %v", c, err)
}
if c, err := ParseCapacity(""); err != nil || len(c) != 0 {
t.Errorf("empty: %v, %v", c, err)
}
for _, bad := range []string{"shanks", "shanks=x", "=2", "a=1,a=2", "shanks=-1"} {
if _, err := ParseCapacity(bad); err == nil {
t.Errorf("%q accepted", bad)
}
}
}
func acquireAsync(p *Places, node string) chan func() { func acquireAsync(p *Places, node string) chan func() {
got := make(chan func(), 1) got := make(chan func(), 1)
go func() { go func() {
@@ -95,17 +80,10 @@ func TestAWaitingStepStopsWhenItsContextEnds(t *testing.T) { // round 3
// The operator's decision of 2026-10-21 13:40: the builder builds on shanks and ace only; a capacity of 0 keeps it off a node. // The operator's decision of 2026-10-21 13:40: the builder builds on shanks and ace only; a capacity of 0 keeps it off a node.
func TestACapacityOfZeroKeepsThePartOffTheNodeAndANodeNotNamedKeepsOne(t *testing.T) { // issue 1661 func TestACapacityOfZeroKeepsThePartOffTheNodeAndANodeNotNamedKeepsOne(t *testing.T) { // issue 1661
c, err := ParseCapacity("shanks=3,ace=2,g14=0,novox=0") p := &Places{Capacity: map[string]int{"shanks": 3, "ace": 2, "g14": 0, "novox": 0}}
if err != nil || c["g14"] != 0 || c["novox"] != 0 || c["shanks"] != 3 || len(c) != 4 {
t.Fatalf("capacity = %v, %v", c, err)
}
p := &Places{Capacity: c}
if p.CapacityOf("g14") != 0 || p.CapacityOf("novox") != 0 || p.CapacityOf("ace") != 2 || p.CapacityOf("other") != DefaultCapacity { if p.CapacityOf("g14") != 0 || p.CapacityOf("novox") != 0 || p.CapacityOf("ace") != 2 || p.CapacityOf("other") != DefaultCapacity {
t.Errorf("capacities = %d %d %d %d, want 0 0 2 and the default for a node not named", p.CapacityOf("g14"), p.CapacityOf("novox"), p.CapacityOf("ace"), p.CapacityOf("other")) t.Errorf("capacities = %d %d %d %d, want 0 0 2 and the default for a node not named", p.CapacityOf("g14"), p.CapacityOf("novox"), p.CapacityOf("ace"), p.CapacityOf("other"))
} }
if _, err := ParseCapacity("shanks=-1"); err == nil {
t.Error("a negative capacity accepted")
}
entries := tcs("go", "go1.27.1", "ace", "g14", "novox", "shanks", "other") entries := tcs("go", "go1.27.1", "ace", "g14", "novox", "shanks", "other")
for i := 0; i < 6; i++ { for i := 0; i < 6; i++ {
nodes, _, err := ChooseNodes("go", entries, p, nil, 2) nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
@@ -119,7 +97,7 @@ func TestACapacityOfZeroKeepsThePartOffTheNodeAndANodeNotNamedKeepsOne(t *testin
} }
} }
// Only nodes of capacity 0 report: nothing qualifies, and the build says why. // Only nodes of capacity 0 report: nothing qualifies, and the build says why.
_, _, err = ChooseNodes("go", tcs("go", "go1.27.1", "g14", "novox"), p, nil, 2) _, _, err := ChooseNodes("go", tcs("go", "go1.27.1", "g14", "novox"), p, nil, 2)
if err == nil || !strings.Contains(err.Error(), "capacity") { if err == nil || !strings.Contains(err.Error(), "capacity") {
t.Errorf("only capacity-0 nodes: error = %v, want it to name the capacity", err) t.Errorf("only capacity-0 nodes: error = %v, want it to name the capacity", err)
} }
+225
View File
@@ -0,0 +1,225 @@
package buildrun
import (
"context"
"errors"
"fmt"
"maps"
"slices"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module leaves a node: no part writes another module's state, and the module is gone. A report of a node the
// build module no longer runs on is therefore no builder's (issue 1940: mesh-build-node taken off ace on 2026-10-23 left
// ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// ToolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// ToolchainSet): an entry is passed on only while the placement (the seat mesh-placement's state `placement`, held by
// the assigner, delivery.PlacementBucket, read by delivery.Placed; ADR 0567, ADR 0549 §3) places its build module on
// its node, and only once it was written since that placement; it is passed on as deleted otherwise, said once, and
// passed on when its node writes it again. A report written before the placement is one an earlier run of the module
// left (issue 1940, 2026-10-25 20:56: mesh-build-node put on ace again brought ace.typescript v26.12.0 of 2026-10-23
// back into the agreement before the module ran there). Until the placement is settled, and while no node's placement
// is held (before the assigner's adopt writes the first one, or where the state cannot be read), every entry is passed
// on. Builders that run and report different versions still disagree: that refusal is the guard of novox/hq ADR 0465,
// and it stays.
//
// This replaces the copies of the gate the indexer, the builder and the checker each kept over the controller's
// assignments (issue 1940's subtask, issue 774: those go with the controller's lease, ADR 0549 §6).
type ToolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
placed *delivery.Placed
known bool // the placement is settled
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
// NewToolchainGate is a gate over the toolchains of modules, passing what stands on to forward and saying with logf what
// it leaves out.
func NewToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *ToolchainGate {
g := &ToolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, placed: delivery.NewPlaced(),
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *ToolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Placement takes one revision of the seat mesh-placement's placement in (at is when the bus wrote it) and answers whether what it
// passed on changed. A value that does not read is said, and places nothing on its node.
func (g *ToolchainGate) Placement(key string, value []byte, deleted bool, at time.Time) bool {
g.mu.Lock()
defer g.mu.Unlock()
was := g.placed.Seeded()
node, changed, err := g.placed.Put(key, value, deleted, at)
if err != nil {
g.logf("%s: %v (issue 1940)", delivery.PlacementBucket, err)
}
if !g.known {
return false
}
if was != g.placed.Seeded() {
// The first placement came, or the last went: every report's standing changed.
return g.all()
}
return g.again(node, changed)
}
// Settled says the placement's history is read: from now on only the reports of placed nodes pass, once a node's
// placement is held. It answers whether what it passed on changed.
func (g *ToolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
return g.all()
}
// all passes on every report again. The caller holds mu.
func (g *ToolchainGate) all() bool {
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// again passes on every report of node by a build module among modules. The caller holds mu.
func (g *ToolchainGate) again(node string, modules []string) bool {
changed := false
for _, module := range modules {
if !g.modules[module] {
continue
}
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is placed and wrote it since (or the placement is not
// known), deleted otherwise. The caller holds mu.
func (g *ToolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known && g.placed.Seeded() {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: the catalogue does not place the module on its
// node, or the report was written before it did. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *ToolchainGate) leftOut(module, node string, value []byte) string {
since, placed := g.placed.Since(node, module)
if !placed {
return fmt.Sprintf("the catalogue does not place %s on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s was placed on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// FollowPlacement reads the seat mesh-placement's placement with its history into gate, says it settled, and follows it; changed
// is told whenever what the gate passes on changed. Where the state cannot be read or watched (not there, or this part
// may not read it) it says so and the gate passes every report on: not fatal, since an indexer, builder or checker that
// stops for it stops every delivery (issue 1945). An error of the watch after that ends on the channel.
func FollowPlacement(ctx context.Context, js process.BucketOpener, gate *ToolchainGate, changed func(context.Context), logf func(string, ...any)) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, delivery.PlacementBucket)
if err != nil {
logf("the state %s cannot be read, so every node's toolchain report counts, a build node's the catalogue does not place too (issue 1940): %v", delivery.PlacementBucket, err)
return errs, nil
}
w, err := kv.WatchAll(ctx, jetstream.IncludeHistory())
if err != nil {
logf("the state %s cannot be watched, so every node's toolchain report counts, a build node's the catalogue does not place too (issue 1940): %v", delivery.PlacementBucket, err)
return errs, nil
}
put := func(e jetstream.KeyValueEntry) bool {
return gate.Placement(e.Key(), e.Value(), e.Operation() != jetstream.KeyValuePut, e.Created())
}
for settled := false; !settled; {
select {
case <-ctx.Done():
_ = w.Stop()
return errs, ctx.Err()
case e, ok := <-w.Updates():
if !ok {
return errs, errors.New("the watch of " + delivery.PlacementBucket + " ended before it settled")
}
if e == nil {
settled = true
continue
}
put(e)
}
}
if gate.Settled() {
changed(ctx)
}
go func() {
defer func() { _ = w.Stop() }()
for {
select {
case <-ctx.Done():
errs <- ctx.Err()
return
case e, ok := <-w.Updates():
if !ok {
errs <- errors.New("the watch of " + delivery.PlacementBucket + " ended")
return
}
if e != nil && put(e) {
changed(ctx)
}
}
}
}()
return errs, nil
}
+238
View File
@@ -0,0 +1,238 @@
package buildrun
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func gateReport(t *testing.T, node, version string, at time.Time) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
// placing is a value of the state `placement` placing modules on node.
func placing(t *testing.T, node string, modules ...string) (string, []byte) {
t.Helper()
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
key, err := delivery.PlacementKey(node)
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// Issue 1940, the gate over the catalogue's placement (issue 774, ADR 0549 §3): a build node's report counts only while
// the placement places its build module on its node, and only once written since; before the placement is settled, and
// while no node's placement is held, every report counts. The live cases: mesh-build-node taken off ace left
// ace.typescript at v26.12.0 against v26.12.1 on the three others; put on ace again, its report of 2026-10-23 06:29 came
// back before the module ran there.
func TestToolchainGate(t *testing.T) {
old := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
placed := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node placed on ace again
three := []string{"g14", "novox", "shanks"}
start := func(t *testing.T, seeded bool) (*ToolchainGate, *ToolchainSet, *[]string) {
set := &ToolchainSet{}
said := &[]string{}
gate := NewToolchainGate([]string{GoBuildModule, NodeBuildModule}, set.Put, func(f string, a ...any) { *said = append(*said, fmt.Sprintf(f, a...)) })
for _, node := range three {
if seeded {
key, value := placing(t, node, NodeBuildModule)
gate.Placement(key, value, false, old.Add(-time.Hour))
}
key, value := gateReport(t, node, "v26.12.1", placed.Add(time.Minute))
gate.Toolchain(NodeBuildModule, key, value, false)
}
return gate, set, said
}
t.Run("not placed on ace: its report is no builder's", func(t *testing.T) {
gate, set, said := start(t, true)
key, value := gateReport(t, "ace", "v26.12.0", old)
gate.Toolchain(NodeBuildModule, key, value, false)
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the placement is settled every report counts; got %q", none)
}
k, v := placing(t, "ace", GoBuildModule) // another build module on ace: no matter
gate.Placement(k, v, false, old)
if !gate.Settled() {
t.Fatal("settling left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the placed nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(*said) != 1 || !strings.Contains((*said)[0], "does not place mesh-build-node on ace") {
t.Fatalf("ace's report is said left out once; said %q", *said)
}
if gate.Toolchain(NodeBuildModule, key, value, false) {
t.Fatal("a report of a node not placed changed what was passed on")
}
// Placed on ace with a report written since: it counts, and the builders really disagree.
k, v = placing(t, "ace", GoBuildModule, NodeBuildModule)
gate.Placement(k, v, false, old.Add(-time.Minute))
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// Taken off ace again (its key deleted): its report goes again.
if !gate.Placement(k, nil, true, placed) {
t.Fatal("taking ace's placement away passed nothing on")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
// A report deleted from the bucket goes whether its node is placed or not.
key, _ = gateReport(t, "g14", "v26.12.1", placed)
if !gate.Toolchain(NodeBuildModule, key, nil, true) || len(set.Entries("typescript")) != 2 {
t.Fatalf("deleting g14's report: %d entries left, want novox and shanks", len(set.Entries("typescript")))
}
})
for _, c := range []struct {
name string
at time.Time // when ace wrote its report
counts bool
}{
{"a report left before the placement", old, false},
{"a report written since the placement", placed.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
gate, set, said := start(t, true)
k, v := placing(t, "ace") // ace's key, first heard placing nothing
gate.Placement(k, v, false, old.Add(-time.Hour))
gate.Settled()
k, v = placing(t, "ace", NodeBuildModule)
if gate.Placement(k, v, false, placed) {
t.Fatal("a placement with no report passed something on")
}
key, value := gateReport(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// ace's placement written again with another module beside: its standing does not move, nothing is said again.
k, v = placing(t, "ace", GoBuildModule, NodeBuildModule)
if gate.Placement(k, v, false, placed.Add(time.Minute)) {
t.Fatal("the same placement of mesh-build-node changed what was passed on")
}
if c.counts {
return
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's; got %q, %q", v, none)
}
if len(*said) != 1 || !strings.Contains((*said)[0], "before mesh-build-node was placed on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", *said)
}
key, value = gateReport(t, "ace", "v26.12.1", placed.Add(time.Minute))
if !gate.Toolchain(NodeBuildModule, key, value, false) || len(set.Entries("typescript")) != 4 {
t.Fatal("ace's new report was not passed on")
}
})
}
t.Run("no placement held: every report counts", func(t *testing.T) {
gate, set, _ := start(t, false)
key, value := gateReport(t, "ace", "v26.12.0", old)
gate.Toolchain(NodeBuildModule, key, value, false)
gate.Settled()
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("before the first placement file every report counts; got %d", n)
}
k, v := placing(t, "g14", NodeBuildModule)
if !gate.Placement(k, v, false, old.Add(-time.Hour)) || len(set.Entries("typescript")) != 1 {
t.Fatalf("the first placement leaves every node it does not place out; got %d", len(set.Entries("typescript")))
}
if !gate.Placement(k, nil, true, placed) || len(set.Entries("typescript")) != 4 {
t.Fatal("the last placement gone: every report counts again")
}
})
t.Run("the history: a module in the first revision heard of a key stands since before it", func(t *testing.T) {
p := delivery.NewPlaced()
k, v := placing(t, "ace", NodeBuildModule)
p.Put(k, v, false, old)
k2, v2 := placing(t, "g14")
p.Put(k2, v2, false, old)
k2, v2 = placing(t, "g14", NodeBuildModule)
if _, changed, _ := p.Put(k2, v2, false, placed); strings.Join(changed, ",") != NodeBuildModule {
t.Errorf("a module placed says changed %v", changed)
}
if _, _, err := p.Put("placement.novox", []byte("not json"), false, old); err == nil {
t.Error("a value that does not read was not said")
}
if s, ok := p.Since("ace", NodeBuildModule); !ok || !s.IsZero() {
t.Errorf("ace's module stands since %v, %v; want the zero time", s, ok)
}
if s, ok := p.Since("g14", NodeBuildModule); !ok || !s.Equal(placed) {
t.Errorf("g14's module stands since %v, %v; want the revision that placed it", s, ok)
}
if got := p.PlacedNodes(NodeBuildModule); strings.Join(got, ",") != "ace,g14" || len(p.ByNode()) != 2 || !p.Seeded() {
t.Errorf("PlacedNodes = %v, ByNode = %v", got, p.ByNode())
}
})
t.Run("a placement the bus refuses is said, and the part goes on", func(t *testing.T) {
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-delivery-assigner_placement")
gate, _, _ := start(t, false)
for _, bus := range []refusingBus{{err: refused}, {refuseOpen: true, err: refused}} {
if _, err := FollowPlacement(context.Background(), bus, gate, func(context.Context) {}, logger.Printf); err != nil {
t.Errorf("a refused placement stopped the start: %v", err)
}
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
})
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
+2 -4
View File
@@ -35,10 +35,8 @@ func TestToolchainSetKeepsEachNodesEntryByLanguageAndSaysWhenOneChanged(t *testi
if !s.Put(GoBuildModule, "ace.go", nil, true) || len(s.Entries("go")) != 1 { if !s.Put(GoBuildModule, "ace.go", nil, true) || len(s.Entries("go")) != 1 {
t.Error("a delete did not remove the node's entry") t.Error("a delete did not remove the node's entry")
} }
} // What is not a toolchain of the module's language is left out.
s = ToolchainSet{}
func TestToolchainSetLeavesOutWhatIsNotAToolchainOfTheModulesLanguage(t *testing.T) { // 1607/AC6
var s ToolchainSet
if s.Put(GoBuildModule, "novox.typescript", tcValue(t, "typescript", "x"), false) { if s.Put(GoBuildModule, "novox.typescript", tcValue(t, "typescript", "x"), false) {
t.Error("a typescript toolchain in the go build module's bucket counted") t.Error("a typescript toolchain in the go build module's bucket counted")
} }
+40 -4
View File
@@ -80,6 +80,9 @@ func CompareComposedWith(composed map[string]Composed, running []Composed, sameV
// compareOne is one module's comparison: true when the digests agree, else what differs first. // compareOne is one module's comparison: true when the digests agree, else what differs first.
func compareOne(module string, composed, running Composed, sameValue func(a, b string) (bool, bool)) (ModuleDiffers, bool) { func compareOne(module string, composed, running Composed, sameValue func(a, b string) (bool, bool)) (ModuleDiffers, bool) {
composed, running = withoutBusUsers(composed), withoutBusUsers(running) composed, running = withoutBusUsers(composed), withoutBusUsers(running)
// A sealed value is judged by the value it opens to, slot by slot (issue 1126, C1): a value sealed again at the same
// field of the same resource is the box that runs, never a difference; and a box is never matched across resources.
composed = KeepSeals(composed, SealsIn(running), sameValue)
dc, errC := ComposedDigestOf(composed) dc, errC := ComposedDigestOf(composed)
dr, errR := ComposedDigestOf(running) dr, errR := ComposedDigestOf(running)
if errC != nil { if errC != nil {
@@ -128,8 +131,11 @@ func compareOne(module string, composed, running Composed, sameValue func(a, b s
found = true found = true
} }
if len(bodies) < MaxBodies { if len(bodies) < MaxBodies {
// The paths are found on the whole bodies, before either is cut (issue 1126, T1: three resources over
// MaxBody could not be told apart).
c, r := MaskedPair(x, y, sameValue) c, r := MaskedPair(x, y, sameValue)
bodies = append(bodies, ResourceBodies{ID: id, Why: why, Composed: cut(c), Running: cut(r)}) bodies = append(bodies, ResourceBodies{ID: id, Why: why, Paths: DifferingPaths(x, y),
Composed: cut(c), Running: cut(r)})
} }
} }
if found { if found {
@@ -259,12 +265,12 @@ func maskBoth(a, b any, sameValue func(a, b string) (bool, bool)) (any, any) {
va, inA := ma[k] va, inA := ma[k]
vb, inB := mb[k] vb, inB := mb[k]
if secretKey(k) { if secretKey(k) {
word := maskWord(va, vb, inA && inB, sameValue) x, y := maskAll(k, va, vb, inA && inB, sameValue)
if inA { if inA {
outA[k] = word outA[k] = x
} }
if inB { if inB {
outB[k] = word outB[k] = y
} }
continue continue
} }
@@ -322,6 +328,36 @@ func maskBoth(a, b any, sameValue func(a, b string) (bool, bool)) (any, any) {
return a, b return a, b
} }
// maskAll is a value under a key a secret could be under, masked as one word (maskWord); a file's secrets
// (SecretsField) alone entry by entry, each its own sealed value: "another seal of the same value" or "another value"
// per secret, its name shown (issue 1126, C1). Any other object, a vault box included, is one word, judged whole.
func maskAll(key string, a, b any, both bool, sameValue func(a, b string) (bool, bool)) (any, any) {
ma, isA := a.(map[string]any)
mb, isB := b.(map[string]any)
if key != SecretsField || (!isA && !isB) {
w := maskWord(a, b, both, sameValue)
return w, w
}
outA, outB := map[string]any{}, map[string]any{}
for k, va := range ma {
vb, inB := mb[k]
outA[k] = maskWord(va, vb, isB && inB, sameValue)
}
for k, vb := range mb {
va, inA := ma[k]
outB[k] = maskWord(va, vb, isA && inA, sameValue)
}
// A map on one side and another value on the other: the other is one word.
var ra, rb any = outA, outB
if !isA {
ra = maskWord(a, b, both, sameValue)
}
if !isB {
rb = maskWord(a, b, both, sameValue)
}
return ra, rb
}
func maskWord(a, b any, both bool, sameValue func(a, b string) (bool, bool)) string { func maskWord(a, b any, both bool, sameValue func(a, b string) (bool, bool)) string {
if !both { if !both {
return "<masked>" return "<masked>"
+194
View File
@@ -0,0 +1,194 @@
package delivery
import (
"bytes"
"encoding/json"
"sort"
"strconv"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Seals is, by slot, each value one composition carries sealed to its node's key (issue 1126, C1): a file's `sealed`
// a vault box (catalogue.VaultSealedField) and each entry of a file's secrets (SecretsField), as their JSON text (a box is carried as its JSON, a sealed value as
// the string it is). A slot is one field of one resource of the composition's module (SealSlot): a box is reused only
// at the slot that holds it, never for another resource whose value opens to the same (a provider's grant and its
// consumer's secret on one node hold one value).
type Seals map[string]string
// SealedFields are the fields of a resource that carry a value sealed to the node's key.
var SealedFields = []string{"sealed", catalogue.VaultSealedField}
// SealSlot is the slot of one sealed field of one resource, by the resource's id (its place, `#<i>`, when it has none).
// The module is the composition's: a slot is never compared across compositions.
func SealSlot(resource, field string) string { return resource + "#" + field }
// SecretsField is the field of a file that carries, by name, each sealed value its content asks for
// (${secret:<name>}, catalogue's intoFile): each entry is a slot of its own (SecretSlotField), so a secret sealed again
// is judged by the value it opens to, entry by entry, like a file's `sealed` (issue 1126, C1; the seed-7 probe: about 45
// files on ace and novox differed by their secrets' bytes alone).
const SecretsField = "secrets"
// SecretSlotField is the field part of the slot of one entry of a file's secrets (SealSlot(id, SecretSlotField(name))).
func SecretSlotField(name string) string { return SecretsField + "." + name }
// sealAt is one sealed value of a resource: its slot's field, its place (a top-level field, or an entry of the
// secrets), and its raw JSON.
type sealAt struct {
field, entry string
raw json.RawMessage
}
// sealsAt is every sealed value of one resource's fields, in a fixed order: SealedFields, then the secrets by name.
func sealsAt(fields map[string]json.RawMessage) []sealAt {
var out []sealAt
for _, field := range SealedFields {
if f, has := fields[field]; has {
out = append(out, sealAt{field: field, raw: f})
}
}
var secrets map[string]json.RawMessage
if f, has := fields[SecretsField]; has && json.Unmarshal(f, &secrets) == nil {
names := make([]string, 0, len(secrets))
for name := range secrets {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
out = append(out, sealAt{field: SecretSlotField(name), entry: name, raw: secrets[name]})
}
}
return out
}
// SealsIn is every sealed value c carries, by slot; nil when it carries none.
func SealsIn(c Composed) Seals {
var out Seals
for i, raw := range c.Resources {
id, fields, ok := sealedFieldsOf(raw, i)
if !ok {
continue
}
for _, s := range sealsAt(fields) {
if text := sealedText(s.raw); text != "" {
if out == nil {
out = Seals{}
}
out[SealSlot(id, s.field)] = text
}
}
}
return out
}
// KeepSeals is c with each sealed value replaced by the one kept at the same slot, where same says the two open to the
// same value (known and same): a value the vault sealed again is then the box that runs, so the composition is the
// one that runs (ADR 0547 §3, issue 1126). A slot kept holds nothing for, or whose value differs or cannot be opened,
// keeps c's. Every other byte of a resource is kept as it is; c itself is not changed.
func KeepSeals(c Composed, kept Seals, same func(a, b string) (same, known bool)) Composed {
if len(kept) == 0 || same == nil {
return c
}
out := c
out.Resources = nil
for i, raw := range c.Resources {
id, fields, ok := sealedFieldsOf(raw, i)
if ok {
for _, s := range sealsAt(fields) {
ours := sealedText(s.raw)
theirs, held := kept[SealSlot(id, s.field)]
if !held || ours == "" || theirs == ours {
continue
}
if opens, known := same(ours, theirs); !known || !opens {
continue
}
value := json.RawMessage(theirs)
if s.raw[0] == '"' {
value, _ = json.Marshal(theirs)
}
if s.entry == "" {
if swapped, ok := withField(raw, s.field, value); ok {
raw = swapped
}
continue
}
// An entry of the secrets: swapped inside the secrets as they stand in raw, then the secrets in raw.
var now map[string]json.RawMessage
if json.Unmarshal(raw, &now) != nil {
continue
}
if secrets, ok := withField(now[SecretsField], s.entry, value); ok {
if swapped, ok := withField(raw, SecretsField, secrets); ok {
raw = swapped
}
}
}
}
out.Resources = append(out.Resources, raw)
}
return out
}
// sealedFieldsOf is a resource's id (`#<i>` when it has none) and its top-level fields, false when it is no object.
func sealedFieldsOf(raw json.RawMessage, i int) (string, map[string]json.RawMessage, bool) {
var fields map[string]json.RawMessage
if json.Unmarshal(raw, &fields) != nil {
return "", nil, false
}
var id string
if f, ok := fields["id"]; !ok || json.Unmarshal(f, &id) != nil || id == "" {
id = "#" + strconv.Itoa(i)
}
return id, fields, true
}
// sealedText is a sealed field as its text: a string as it is, a box as its JSON.
func sealedText(f json.RawMessage) string {
var text string
if json.Unmarshal(f, &text) == nil {
return text
}
var compact bytes.Buffer
if json.Compact(&compact, f) != nil {
return string(f)
}
return compact.String()
}
// withField is the object raw with its top-level field replaced by value, every other byte as it was (so a composition
// whose sealed values are those that run is the one that runs, byte for byte); false when raw is no object holding it.
func withField(raw json.RawMessage, field string, value json.RawMessage) (json.RawMessage, bool) {
d := json.NewDecoder(bytes.NewReader(raw))
if t, err := d.Token(); err != nil || t != json.Delim('{') {
return nil, false
}
for d.More() {
t, err := d.Token()
if err != nil {
return nil, false
}
key, _ := t.(string)
before := d.InputOffset()
var skip json.RawMessage
if err := d.Decode(&skip); err != nil {
return nil, false
}
if key != field {
continue
}
after := d.InputOffset()
// The span holds the colon, any space and the value: kept up to the colon, the value replaced.
span := raw[before:after]
colon := bytes.IndexByte(span, ':')
if colon < 0 {
return nil, false
}
out := make([]byte, 0, len(raw)-len(span)+colon+1+len(value))
out = append(out, raw[:int(before)+colon+1]...)
out = append(out, value...)
out = append(out, raw[after:]...)
return out, true
}
return nil, false
}
+87
View File
@@ -3,9 +3,12 @@ package delivery
import ( import (
"crypto/ed25519" "crypto/ed25519"
"crypto/rand" "crypto/rand"
"crypto/x509"
"encoding/json" "encoding/json"
"encoding/pem"
"strings" "strings"
"testing" "testing"
"time"
) )
func aSigningKey(t *testing.T) (ed25519.PublicKey, ed25519.PrivateKey) { func aSigningKey(t *testing.T) (ed25519.PublicKey, ed25519.PrivateKey) {
@@ -100,4 +103,88 @@ func TestACompositionIsSignedByAKeptKey(t *testing.T) {
if _, err := SignComposedBy(c, pub, func(m []byte) ([]byte, error) { return ed25519.Sign(other, m), nil }); err == nil { if _, err := SignComposedBy(c, pub, func(m []byte) ([]byte, error) { return ed25519.Sign(other, m), nil }); err == nil {
t.Errorf("a signature by another key than the one named was accepted") t.Errorf("a signature by another key than the one named was accepted")
} }
// L3 s1 (ADR 0550 §2): a message the vault or the controller signed under a kind's context verifies against the
// signers a node trusts and names which; an unknown key and another kind's context are refused.
vaultPub, vault := aSigningKey(t)
controllerPub, controller := aSigningKey(t)
trusted := []Signer{{Name: SignerController, Key: controllerPub}, {Name: SignerVault, Key: vaultPub}}
enrol, _ := SignContext(KindEnrolment)
certCtx, _ := SignContext(KindNodeCertificate)
msg := []byte("the reply")
for _, row := range []struct {
name string
key ed25519.PrivateKey
signed string
checked string
want string
}{
{"the vault's key verifies", vault, enrol, enrol, SignerVault},
{"the controller's key verifies", controller, enrol, enrol, SignerController},
{"an unknown key is refused", other, enrol, enrol, ""},
{"the wrong context is refused", vault, certCtx, enrol, ""},
} {
got, err := VerifyBy(trusted, row.checked, msg, SignMessage(row.key, row.signed, msg))
if row.want == "" && err == nil {
t.Errorf("%s: verified as %s", row.name, got.Name)
}
if row.want != "" && (err != nil || got.Name != row.want) {
t.Errorf("%s: %+v, %v", row.name, got.Name, err)
}
}
if _, err := SignContext("hand-over"); err == nil {
t.Error("a kind the vault does not sign has a context")
}
// The signers' key reads back only when its fingerprint is its key's, and what a node trusts names both holders.
k := NewSignerKey(vaultPub, time.Unix(0, 0))
raw, _ := json.Marshal(k)
if back, err := DecodeSignerKey(raw); err != nil || back.Fingerprint != SignerFingerprint(vaultPub) {
t.Errorf("the signers' key: %+v, %v", back, err)
}
k.Fingerprint = SignerFingerprint(controllerPub)
if raw, _ := json.Marshal(k); true {
if _, err := DecodeSignerKey(raw); err == nil {
t.Error("a signers' key naming another fingerprint was taken")
}
}
if got := Trusts(trusted, nil); len(got) != 2 || !TrustsKey(got, SignerVault, SignerFingerprint(vaultPub)) ||
!TrustsKey(got, SignerController, SignerFingerprint(controllerPub)) {
t.Errorf("trusts: %v", got)
}
// A certificate the vault's authority signs for a node's name and serving key checks against a bundle holding that
// authority, and not for another name, another key, or against the controller's alone; a bundle given again adds no
// certificate twice.
authority := func(pub ed25519.PublicKey, priv ed25519.PrivateKey) string {
tpl := AuthorityTemplate(pub, time.Now().Add(-time.Hour))
der, err := x509.CreateCertificate(nil, tpl, tpl, pub, priv)
if err != nil {
t.Fatal(err)
}
return string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))
}
vaultCA, controllerCA := authority(vaultPub, vault), authority(controllerPub, controller)
parent, _ := ParseCertificate(vaultCA)
serving, _ := aSigningKey(t)
der, err := x509.CreateCertificate(nil, NodeCertificateTemplate("ace.internal", serving, time.Now().Add(-time.Hour)), parent, serving, vault)
if err != nil {
t.Fatal(err)
}
leaf := string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))
bundle := AuthorityBundle(controllerCA, AuthorityBundle(controllerCA, vaultCA))
if n := strings.Count(bundle, "BEGIN CERTIFICATE"); n != 2 {
t.Errorf("the bundle holds %d certificates, want 2", n)
}
now := time.Now()
if err := CheckNodeCertificate(leaf, []string{bundle}, "ace.internal", serving, now); err != nil {
t.Errorf("the vault's certificate does not check: %v", err)
}
for name, err := range map[string]error{
"another name": CheckNodeCertificate(leaf, []string{vaultCA}, "g14.internal", serving, now),
"another key": CheckNodeCertificate(leaf, []string{vaultCA}, "ace.internal", controllerPub, now),
"the controller's alone": CheckNodeCertificate(leaf, []string{controllerCA}, "ace.internal", serving, now),
} {
if err == nil {
t.Errorf("%s: checked", name)
}
}
} }
+12 -4
View File
@@ -48,6 +48,7 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"maps"
"reflect" "reflect"
"regexp" "regexp"
"slices" "slices"
@@ -355,6 +356,10 @@ type Have struct {
// after a restart while it opens to the vault's value, so a value sealed again moves no composition. Openable by the // after a restart while it opens to the vault's value, so a value sealed again moves no composition. Openable by the
// node alone, as the vault hands them over the bus. Empty on every other have. // node alone, as the vault hands them over the bus. Empty on every other have.
Sealed []string `json:"sealed,omitempty"` Sealed []string `json:"sealed,omitempty"`
// SealedAt is Sealed by slot (Seals: a field of a resource, issue 1126, C1): the composer reuses a box only at the
// slot that holds it, so a provider and a consumer of one value on one node each keep their own. Written in Sealed's
// place by a node-engine that knows it; Sealed is read only from a have written before.
SealedAt Seals `json:"sealed-at,omitempty"`
} }
// UserListWritten is the bus's user list as the node-engine of the bus node wrote it to the bus module's file, applied // UserListWritten is the bus's user list as the node-engine of the bus node wrote it to the bus module's file, applied
@@ -406,11 +411,13 @@ type HaveNode struct {
// applied (Have.Sealed for the node's own, issue 1126): a value the vault seals again after a restart moves no // applied (Have.Sealed for the node's own, issue 1126): a value the vault seals again after a restart moves no
// composition, so a restart applies nothing again. // composition, so a restart applies nothing again.
Sealed []string `json:"sealed,omitempty"` Sealed []string `json:"sealed,omitempty"`
// Candidates is the node's answer to each placement candidate naming it, by group (PlacementCandidate): written by // SealedAt is Sealed by slot (Have.SealedAt).
SealedAt Seals `json:"sealed-at,omitempty"`
// Candidates is the node's answer to each placement candidate naming it, by proposal id (PlacementCandidate): written by
// its shadow composer whenever an answer changes, removed when the candidate key goes. The node applies nothing of a // its shadow composer whenever an answer changes, removed when the candidate key goes. The node applies nothing of a
// candidate. // candidate.
Candidates map[string]CandidateAnswer `json:"candidates,omitempty"` Candidates map[string]CandidateAnswer `json:"candidates,omitempty"`
// Composes is whose compositions the node applies: its own (ComposesOwn, the catalogue places it) or the // Composes is whose compositions the node applies: its own (ComposesOwn, placement places it) or the
// controller's (ComposesController); ComposesWhy says why in words. Written by the node agent at each start, so a // controller's (ComposesController); ComposesWhy says why in words. Written by the node agent at each start, so a
// node returning to the controller's compositions when its placement key is deleted says so (issue 1126). Empty // node returning to the controller's compositions when its placement key is deleted says so (issue 1126). Empty
// from an engine older than the field. // from an engine older than the field.
@@ -586,10 +593,11 @@ func DecodePulse(data []byte) (Pulse, error) {
func SameHave(a, b Have) bool { func SameHave(a, b Have) bool {
ua, ub := a.UserList, b.UserList ua, ub := a.UserList, b.UserList
a.UserList, b.UserList = nil, nil a.UserList, b.UserList = nil, nil
if !slices.Equal(a.Sealed, b.Sealed) { if !slices.Equal(a.Sealed, b.Sealed) || !maps.Equal(a.SealedAt, b.SealedAt) {
return false return false
} }
a.Sealed, b.Sealed = nil, nil a.Sealed, b.Sealed = nil, nil
a.SealedAt, b.SealedAt = nil, nil
if !reflect.DeepEqual(a, b) || (ua == nil) != (ub == nil) { if !reflect.DeepEqual(a, b) || (ua == nil) != (ub == nil) {
return false return false
} }
@@ -607,7 +615,7 @@ func SameHaveNode(a, b HaveNode) bool {
return a.V == b.V && a.Composed == b.Composed && sameJSON(a.Windows, b.Windows) && sameJSON(a.Busy, b.Busy) && return a.V == b.V && a.Composed == b.Composed && sameJSON(a.Windows, b.Windows) && sameJSON(a.Busy, b.Busy) &&
sameJSON(a.Strays, b.Strays) && sameJSON(a.Filters, b.Filters) && sameJSON(a.Profile, b.Profile) && sameJSON(a.Strays, b.Strays) && sameJSON(a.Filters, b.Filters) && sameJSON(a.Profile, b.Profile) &&
(a.Record == nil) == (b.Record == nil) && (a.Record == nil || SameNodeRecord(*a.Record, *b.Record)) && (a.Record == nil) == (b.Record == nil) && (a.Record == nil || SameNodeRecord(*a.Record, *b.Record)) &&
jsonEqual(a.Sealed, b.Sealed) && jsonEqual(a.Candidates, b.Candidates) && a.Composes == b.Composes && jsonEqual(a.Sealed, b.Sealed) && maps.Equal(a.SealedAt, b.SealedAt) && jsonEqual(a.Candidates, b.Candidates) && a.Composes == b.Composes &&
a.ComposesWhy == b.ComposesWhy a.ComposesWhy == b.ComposesWhy
} }
+7 -16
View File
@@ -9,10 +9,9 @@ const oddManifest = "{\n\t\"module\": \"a\",\n \"z\": 1, \"a\": [ \"x\",\"y\"
func mergedWithInputs() Forge { func mergedWithInputs() Forge {
return Forge{Kind: ForgeMerged, Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa", Merge: "cccc", Root: "t0", return Forge{Kind: ForgeMerged, Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa", Merge: "cccc", Root: "t0",
Trees: map[string]string{"a": "t1", "b": "t2"}, Trees: map[string]string{"a": "t1", "b": "t2"},
Manifests: map[string]string{"a": oddManifest}, Manifests: map[string]string{"a": oddManifest},
Locks: map[string]map[string]string{"a": {"go.sum": strings.Repeat("ab", 32)}}, Locks: map[string]map[string]string{"a": {"go.sum": strings.Repeat("ab", 32)}}}
Placements: map[string]string{"ace": `{"v":1,"node":"ace","modules":[]}`}, PlacementsRead: true}
} }
// 1442/AC1 // 1442/AC1
@@ -22,7 +21,7 @@ func TestForgeMergedCarriesManifestsAndLocks(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
for _, frag := range []string{`"manifests":{"a":`, `"locks":{"a":{"go.sum":`, `"placements":{"ace":`, `"placements-read":true`} { for _, frag := range []string{`"manifests":{"a":`, `"locks":{"a":{"go.sum":`} {
if !strings.Contains(string(data), frag) { if !strings.Contains(string(data), frag) {
t.Errorf("the encoding lacks %s: %s", frag, data) t.Errorf("the encoding lacks %s: %s", frag, data)
} }
@@ -48,9 +47,9 @@ func TestForgeMergedRefusesBadInputs(t *testing.T) {
"locks outside trees": {func(f *Forge) { f.Locks["yy"] = map[string]string{"go.sum": "h"} }, `"yy"`}, "locks outside trees": {func(f *Forge) { f.Locks["yy"] = map[string]string{"go.sum": "h"} }, `"yy"`},
"empty manifest": {func(f *Forge) { f.Manifests["b"] = "" }, `"b"`}, "empty manifest": {func(f *Forge) { f.Manifests["b"] = "" }, `"b"`},
"empty lock hash": {func(f *Forge) { f.Locks["a"]["go.sum"] = "" }, `"a"`}, "empty lock hash": {func(f *Forge) { f.Locks["a"]["go.sum"] = "" }, `"a"`},
"placements unsaid": {func(f *Forge) { f.PlacementsRead = false }, "placements-read"}, // No repository holds placement (ADR 0567 §11): a message carrying placement files is refused, on a merge or a pull.
"placement of no node": {func(f *Forge) { f.Placements["a.b"] = "{}" }, "names no node"}, "placement files": {func(f *Forge) { f.RetiredPlacements = []byte(`{"anchor":"{}"}`) }, "mesh-placement"},
"placements on a pull": {func(f *Forge) { f.Kind, f.Action, f.Merge, f.Root, f.Trees = ForgePull, PullOpened, "", "", nil }, "only a merged message"}, "placements read": {func(f *Forge) { f.RetiredRead = true }, "mesh-placement"},
} { } {
f := mergedWithInputs() f := mergedWithInputs()
tc.mut(&f) tc.mut(&f)
@@ -86,14 +85,6 @@ func TestForgePullCarriesManifestsAndLocks(t *testing.T) {
if _, err := EncodeForge(f); err == nil || !strings.Contains(err.Error(), `"c"`) { if _, err := EncodeForge(f); err == nil || !strings.Contains(err.Error(), `"c"`) {
t.Errorf("locks outside Dirs: %v", err) t.Errorf("locks outside Dirs: %v", err)
} }
// A pull message of the catalogue carries its head's placement files whole (placement candidates, issue 1126).
f = Forge{Kind: ForgePull, Repository: CatalogueRepository, Number: 2, Head: "h", Action: PullUpdated,
Placements: map[string]string{"ace": `{"v":1,"node":"ace","modules":[]}`}, PlacementsRead: true}
if data, err := EncodeForge(f); err != nil {
t.Errorf("a catalogue pull with placement files: %v", err)
} else if got, err := DecodeForge(data); err != nil || !SameForge(got, withV(f)) {
t.Errorf("a catalogue pull's placement files changed on the way: %+v, %v", got, err)
}
for name, f := range map[string]Forge{ for name, f := range map[string]Forge{
"refused manifests": {Kind: ForgeRefused, Repository: "a/b", Act: "x", Words: "no", Manifests: map[string]string{"a": "{}"}}, "refused manifests": {Kind: ForgeRefused, Repository: "a/b", Act: "x", Words: "no", Manifests: map[string]string{"a": "{}"}},
"refused locks": {Kind: ForgeRefused, Repository: "a/b", Act: "x", Words: "no", Locks: map[string]map[string]string{"a": {"go.sum": "h"}}}, "refused locks": {Kind: ForgeRefused, Repository: "a/b", Act: "x", Words: "no", Locks: map[string]map[string]string{"a": {"go.sum": "h"}}},
+36
View File
@@ -90,3 +90,39 @@ func (g ControllerGrant) NamesNode(node, module string) bool {
} }
return false return false
} }
// **The grants part's grants** (novox/hq ADR 0547 §4, §5, ADR 0561 §2, §6; issue 774). The grants part writes the same
// keys, with the same value, into its own state `grants` (the bucket PartGrantsBucket), beside the controller's: first in
// shadow, which nobody reads, so a tool can compare the two before anything switches. The assigner reads the part's keys
// only once the controller's lease is handed away (GrantSource), and the controller's until then.
const (
// PartGrantsState is the grants part's state of issued credentials by fingerprint, as its manifest names it.
PartGrantsState = "grants"
// PartGrantsBucket is that state's bucket, `<owner module>_<name>`.
PartGrantsBucket = "mesh-grants_" + PartGrantsState
// PartGrantsRead is that state as a reader names it in `reads`.
PartGrantsRead = "mesh-grants." + PartGrantsState
// PartGrantsHistory is the revisions the bucket keeps per key.
PartGrantsHistory = ControllerGrantsHistory
)
// PartGrantKey is the key, in PartGrantsBucket, of a module's credential issued for a fingerprint: the layout of the
// controller's key (ControllerGrantKey), so a key of one is the key of the other and a compare is by key.
func PartGrantKey(module, fingerprint string) (string, error) {
return ControllerGrantKey(module, fingerprint)
}
// ParsePartGrantKey is the inverse of PartGrantKey.
func ParsePartGrantKey(key string) (module, fingerprint string, err error) {
return ParseControllerGrantKey(key)
}
// GrantSource names the bucket the assigner reads grants from, and the `reads` name that grants it: the grants part's
// once the controller's lease is handed away (buscompose.LeaseHandedAway, which refuses every other value of the lease,
// an unreadable one included), the controller's in every other case.
func GrantSource(leaseHandedAway bool) (bucket, read string) {
if leaseHandedAway {
return PartGrantsBucket, PartGrantsRead
}
return ControllerGrantsBucket, "mesh-controller.grants"
}
+13 -4
View File
@@ -3,6 +3,7 @@ package delivery
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"fmt"
"path" "path"
"sort" "sort"
"strings" "strings"
@@ -84,9 +85,6 @@ func judged(from, to Composed) (Judgement, string) {
if ids := changedOf(before, after, func(kind string) bool { return kind == string(declaration.TypeContainer) }); len(ids) > 0 { if ids := changedOf(before, after, func(kind string) bool { return kind == string(declaration.TypeContainer) }); len(ids) > 0 {
return JudgedRecreates, "it recreates " + strings.Join(ids, ", ") return JudgedRecreates, "it recreates " + strings.Join(ids, ", ")
} }
if from.Data != to.Data {
return JudgedMovesData, "its data declaration changes"
}
places := dataPlaces(before, after) places := dataPlaces(before, after)
held := func(kind string) bool { held := func(kind string) bool {
return kind == string(declaration.TypeDirectory) || kind == string(declaration.TypeFile) return kind == string(declaration.TypeDirectory) || kind == string(declaration.TypeFile)
@@ -104,9 +102,20 @@ func judged(from, to Composed) (Judgement, string) {
if to.Data || from.Data { if to.Data || from.Data {
moved = append(moved, changedOf(before, after, func(kind string) bool { return kind == string(declaration.TypeProcess) })...) moved = append(moved, changedOf(before, after, func(kind string) bool { return kind == string(declaration.TypeProcess) })...)
} }
// A data flag that differs is no move of data by itself: data moves only when a data item does (issue 1126, 2026-10-29:
// a running composition read back from the kept declaration carries no flag, and mongodb's toolchain move asked a yes for
// moving its data). It is said loudly in the words, and never hides an item that moved.
if len(moved) > 0 { if len(moved) > 0 {
sort.Strings(moved) sort.Strings(moved)
return JudgedMovesData, "it changes where its data is kept: " + strings.Join(moved, ", ") words := "it changes where its data is kept: " + strings.Join(moved, ", ")
if from.Data != to.Data {
words += fmt.Sprintf("; its data flag differs too (running %v, composed %v)", from.Data, to.Data)
}
return JudgedMovesData, words
}
if from.Data != to.Data {
return JudgedNeither, fmt.Sprintf("it recreates no container and moves no data; WARNING: its data flag differs "+
"(running %v, composed %v) while no data item does, so one side's flag is wrong", from.Data, to.Data)
} }
return JudgedNeither, "it recreates no container and moves no data" return JudgedNeither, "it recreates no container and moves no data"
} }
+5 -1
View File
@@ -48,7 +48,8 @@ func TestJudgeMove(t *testing.T) {
{"a container gone", mail('a', true, container, spool), mail('b', true, spool), JudgedRecreates, false}, {"a container gone", mail('a', true, container, spool), mail('b', true, spool), JudgedRecreates, false},
{"its spool moved", mail('a', true, container, spool), mail('b', true, container, spoolMoved), JudgedMovesData, false}, {"its spool moved", mail('a', true, container, spool), mail('b', true, container, spoolMoved), JudgedMovesData, false},
{"a file in its spool", mail('a', true, container, spool, config), mail('b', true, container, spool, config2), JudgedMovesData, false}, {"a file in its spool", mail('a', true, container, spool, config), mail('b', true, container, spool, config2), JudgedMovesData, false},
{"its data declaration", mail('a', false, container, spool), mail('b', true, container, spool), JudgedMovesData, false}, {"its data flag alone", mail('a', false, container, spool), mail('b', true, container, spool), JudgedNeither, false},
{"its data flag and its spool", mail('a', false, container, spool), mail('b', true, container, spoolMoved), JudgedMovesData, false},
{"a process of a module with data", mail('a', true, spool, process), mail('b', true, spool, process2), JudgedMovesData, false}, {"a process of a module with data", mail('a', true, spool, process), mail('b', true, spool, process2), JudgedMovesData, false},
{"a process of a module without", mail('a', false, process), mail('b', false, process2), JudgedNeither, false}, {"a process of a module without", mail('a', false, process), mail('b', false, process2), JudgedNeither, false},
{"its tools only", mail('a', true, container, spool, tools), mail('b', true, container, spool, tools2), JudgedNeither, false}, {"its tools only", mail('a', true, container, spool, tools), mail('b', true, container, spool, tools2), JudgedNeither, false},
@@ -61,6 +62,9 @@ func TestJudgeMove(t *testing.T) {
if got.Judgement != c.want || got.UserListOnly != c.userList || got.Words == "" { if got.Judgement != c.want || got.UserListOnly != c.userList || got.Words == "" {
t.Errorf("%s: %+v, want %s (user list only %v)", c.name, got, c.want, c.userList) t.Errorf("%s: %+v, want %s (user list only %v)", c.name, got, c.want, c.userList)
} }
if strings.Contains(c.name, "data flag alone") && !strings.Contains(got.Words, "WARNING: its data flag differs") {
t.Errorf("%s: the flag's difference is not said: %q", c.name, got.Words)
}
} }
if got := JudgeMove(nil, Composed{Node: "ace"}); got.Judgement != JudgedNeither { if got := JudgeMove(nil, Composed{Node: "ace"}); got.Judgement != JudgedNeither {
t.Errorf("the node's own composition: %+v", got) t.Errorf("the node's own composition: %+v", got)
+4
View File
@@ -55,6 +55,10 @@ type NodeRecord struct {
// Authority is the mesh's own certificate as this machine holds it (public): the one its declaration last wrote where // Authority is the mesh's own certificate as this machine holds it (public): the one its declaration last wrote where
// a module asked for it. A node composing its own composes a certificate with it (compose.Input.Authority). // a module asked for it. A node composing its own composes a certificate with it (compose.Input.Authority).
Authority string `json:"authority,omitempty"` Authority string `json:"authority,omitempty"`
// Trusts is what this machine trusts a mesh signature from now (Trusts in signers.go, ADR 0550 §2): the controller's
// key until the lease is handed away, and the vault's key and authority once it reads them from `signers`. The gate
// before the lease write reads every node's from the bus.
Trusts []string `json:"trusts,omitempty"`
} }
func (r NodeRecord) check() error { func (r NodeRecord) check() error {
+1 -1
View File
@@ -18,7 +18,7 @@ var secretNamed = regexp.MustCompile(`(?i)(secret|passw|token|api[-_]?key|privat
func TestNodeRecord(t *testing.T) { func TestNodeRecord(t *testing.T) {
hub := NodeRecord{Node: "novox", TunnelKey: "k", Capabilities: []string{"docker"}, SealingKey: "age1x", ServingKey: "pub", hub := NodeRecord{Node: "novox", TunnelKey: "k", Capabilities: []string{"docker"}, SealingKey: "age1x", ServingKey: "pub",
Certificate: "cert", OutwardLinks: []string{"eth0"}, Networks: []catalogue.AttachedNetwork{{Link: "eth0", Network: "192.0.2.0/24"}}, Certificate: "cert", OutwardLinks: []string{"eth0"}, Networks: []catalogue.AttachedNetwork{{Link: "eth0", Network: "192.0.2.0/24"}},
NetworksSaid: true, Contract: 3} NetworksSaid: true, Contract: 3, Trusts: []string{"controller key sha256:aa", "vault key sha256:bb"}}
raw, err := EncodeNodeRecord(hub) raw, err := EncodeNodeRecord(hub)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
+95
View File
@@ -0,0 +1,95 @@
package delivery
// The own secrets a node's shadow answer finds the vault holds another value for than the node runs (novox/hq issue 1126,
// cause C4): the vault's `move-in` never replaces a value it holds, so a value the controller made anew later (a rotation
// at start, a re-issue) runs while the vault keeps the first one it was handed. The node opens both seals with its own
// key and its answer's shown bodies mask the value as MaskedAnotherValue; this reads which own secrets those are, for the
// node agent (which hands the vault what runs) and for the vault's `own_drift` (which lists them mesh-wide), so both read
// the answer one way.
import (
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// MaskedAnotherValue is the word a shown body masks a value with when the node opened both sides with its key and they
// are two values (MaskedPair, maskWord).
const MaskedAnotherValue = "<masked: another value>"
// OwnValueDiffers is one own secret of a module whose value composed (the vault's) is another than the one that runs.
type OwnValueDiffers struct {
Module string `json:"module"`
Name string `json:"name"`
}
// ownNeedPrefix starts the id of the file a module's own secret is placed in (catalogue.NeedID).
var ownNeedPrefix = catalogue.NeedID("")
// OwnNeedName is the name of module's own secret whose file a resource id names, and whether it names one. On a node a
// resource's id carries its module's name (`only-office.needs-jwt`, as a live shadow answer shows it); the bare form a
// declaration composes (`needs-jwt`, catalogue.NeedID) is read too. An id of another module's is not this module's.
func OwnNeedName(module, id string) (string, bool) {
if module != "" {
id = strings.TrimPrefix(id, module+".")
}
name, own := strings.CutPrefix(id, ownNeedPrefix)
return name, own && name != "" && !strings.Contains(name, ".")
}
// OwnValuesDiffer is every own secret a's shown bodies say holds another value composed than running: a body of a
// module's own secret's file (OwnNeedName: `<module>.needs-<name>` on a node) masked MaskedAnotherValue on either side. Sorted by module and name, each
// once. A body the answer did not show (beyond MaxBodies) is not read.
func OwnValuesDiffer(a CandidateAnswer) []OwnValueDiffers {
seen := map[OwnValueDiffers]bool{}
var out []OwnValueDiffers
for _, d := range a.Differs {
for _, b := range d.Bodies {
name, own := OwnNeedName(d.Module, b.ID)
if !own || !anotherValue(b) {
continue
}
o := OwnValueDiffers{Module: d.Module, Name: name}
if !seen[o] {
seen[o] = true
out = append(out, o)
}
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Module != out[j].Module {
return out[i].Module < out[j].Module
}
return out[i].Name < out[j].Name
})
return out
}
// ValuesDiffer is how many shown bodies of a hold a value masked MaskedAnotherValue, own secrets and every other
// resource alike: the count a node's answer carries as CandidateAnswer.ValuesDiffer.
func ValuesDiffer(a CandidateAnswer) int {
n := 0
for _, d := range a.Differs {
for _, b := range d.Bodies {
if anotherValue(b) {
n++
}
}
}
return n
}
// WithValuesDiffer is answers with each one's ValuesDiffer counted.
func WithValuesDiffer(answers map[string]CandidateAnswer) map[string]CandidateAnswer {
out := make(map[string]CandidateAnswer, len(answers))
for g, a := range answers {
a.ValuesDiffer = ValuesDiffer(a)
out[g] = a
}
return out
}
func anotherValue(b ResourceBodies) bool {
return strings.Contains(b.Composed, MaskedAnotherValue) || strings.Contains(b.Running, MaskedAnotherValue)
}
+126
View File
@@ -0,0 +1,126 @@
package delivery
import (
"fmt"
"maps"
"slices"
"sync"
"time"
)
// Placed is where the placement (the seat mesh-placement's state `placement`, held by the assigner; ADR 0567, ADR 0549 §3) places each module, and since when:
// the one reader of that state for every part that asks "does module stand on node" (issue 774, design 47's third
// milestone), in place of each part reading the controller's assignments (`mesh-controller_assignments`, which goes
// with the controller's lease, ADR 0549 §6).
//
// Its input is the state's revisions in the order the bus wrote them, as a watch with the history (jetstream's
// IncludeHistory) hands them: Put takes each. A module stands since the revision of its node's key that first named it
// in an unbroken run. A module in the first revision heard of a key stands since the zero time: that revision is the
// key's first (the seed of ADR 0549 §4 placed what already ran there) or the oldest the bus still keeps
// (PlacementHistory), and either way the module may have run there before it. A key's value that does not read places
// nothing on its node, and Put says why.
//
// Safe for use from a watch and from a reader at once.
type Placed struct {
mu sync.Mutex
since map[string]map[string]time.Time // node -> module -> since when it stands
heard map[string]bool // the nodes whose key was heard at all
}
// NewPlaced is a Placed that holds no placement yet.
func NewPlaced() *Placed {
return &Placed{since: map[string]map[string]time.Time{}, heard: map[string]bool{}}
}
// Put takes one revision of a key of the state `placement` (its value, whether it is a delete, and when the bus wrote
// it). It answers the node the key is of and the modules whose standing on it changed (placed or removed), in name
// order, and an error for a key or a value that does not read (the node then places nothing).
func (p *Placed) Put(key string, value []byte, deleted bool, at time.Time) (node string, changed []string, err error) {
node, err = ParsePlacementKey(key)
if err != nil {
return "", nil, err
}
now := map[string]bool{}
if !deleted {
pl, derr := DecodePlacement(value)
switch {
case derr != nil:
err = fmt.Errorf("the placement of %s does not read, so it places nothing there: %w", node, derr)
case pl.Node != node:
err = fmt.Errorf("the placement under %s names the node %q, so it places nothing there", key, pl.Node)
default:
for _, m := range pl.Assigned() {
now[m] = true
}
}
}
p.mu.Lock()
defer p.mu.Unlock()
if !p.heard[node] {
p.heard[node] = true
at = time.Time{}
}
was := p.since[node]
next := map[string]time.Time{}
for m := range now {
if s, ok := was[m]; ok {
next[m] = s
} else {
next[m] = at
changed = append(changed, m)
}
}
for m := range was {
if !now[m] {
changed = append(changed, m)
}
}
if len(next) == 0 {
delete(p.since, node)
} else {
p.since[node] = next
}
slices.Sort(changed)
return node, changed, err
}
// Seeded says some node's placement is held: before the assigner's adopt writes the first placement nothing is placed,
// and a reader keeps what it did before (ADR 0549 §5).
func (p *Placed) Seeded() bool {
p.mu.Lock()
defer p.mu.Unlock()
return len(p.since) > 0
}
// Since says whether module stands on node, and since when (the zero time: since before the revisions kept).
func (p *Placed) Since(node, module string) (time.Time, bool) {
p.mu.Lock()
defer p.mu.Unlock()
s, ok := p.since[node][module]
return s, ok
}
// PlacedNodes is every node module is placed on, in name order.
func (p *Placed) PlacedNodes(module string) []string {
p.mu.Lock()
defer p.mu.Unlock()
var out []string
for node, mods := range p.since {
if _, ok := mods[module]; ok {
out = append(out, node)
}
}
slices.Sort(out)
return out
}
// ByNode is, by node, the modules placed on it in name order.
func (p *Placed) ByNode() map[string][]string {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string][]string, len(p.since))
for node, mods := range p.since {
out[node] = slices.Sorted(maps.Keys(mods))
}
return out
}
+50 -54
View File
@@ -3,7 +3,6 @@ package delivery
import ( import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"path"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
@@ -11,29 +10,38 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
) )
// Placement of modules on nodes lives in the catalogue (novox/hq ADR 0549): one file per node, `placement/<node>.json` // Placement of modules on nodes is the state of the seat mesh-placement (novox/hq ADR 0567), held once for the mesh by
// in the catalogue repository, changed only by a pull request. At each trunk merge of the catalogue the indexer reads // the assigner: one key `placement.<node>` per node in its state `placement`, written only on the operator's Approve of a
// the files and writes each node's key `placement.<node>` in its state `placement`, only when it differs, and says // proposal the nodes it touches answered (§2-§5), by compare-and-set at the revision the proposal was laid over. No
// `placement-changed` naming the node, the revision and the modules added, removed or changed (ADR 0549 §2). The node // repository holds it. The node agents, the grants part, mesh-config and the status view read it (§9); nothing in it is
// agents, the assigner, the grants part and mesh-config read it (§3); nothing in it is secret. // secret. Its states are named after the holding module, as every module state is, and these constants are the seat's
// contract: every reader takes the names from here (§1).
const ( const (
// PlacementState is the indexer's state of placements, and PlacementBucket its bucket. // PlacementState is the seat's state of placements, and PlacementBucket its bucket. PlacementRead is the name a
// reader's manifest declares in its reads.
PlacementState = "placement" PlacementState = "placement"
PlacementBucket = IndexerModule + "_" + PlacementState PlacementBucket = AssignerModule + "_" + PlacementState
// PlacementHistory is the revisions the placement bucket keeps per key. PlacementRead = AssignerModule + "." + PlacementState
PlacementHistory = 10 // PlacementHistory is the revisions the placement bucket keeps per key: the bus's most (§6).
// PlacementDir is the catalogue's directory of placement files, one `<node>.json` per node. PlacementHistory = 64
PlacementDir = "placement"
// PlacementChangedEvent is what the indexer says when a node's placement key changed. // PlacementStream is the holder's stream of the seat's events, kept PlacementStreamDays (§6, within the mesh's bound
PlacementChangedEvent = "placement-changed" // on a module's stream, catalogue.StreamMostDays), and declared as the holder's data so the mesh's backups keep it.
PlacementStream = "placement"
PlacementStreamDays = 30
// The seat's events, said by its holder as its own (§2, §4, §5, §6), each carrying a PlacementEvent.
PlacementProposedEvent = "placement-proposed"
PlacementWithdrawnEvent = "placement-withdrawn"
PlacementJudgedEvent = "placement-judged"
PlacementChangedEvent = "placement-changed"
placementWord = "placement" placementWord = "placement"
// CatalogueRepository is the repository whose placement files are the mesh's placement (ADR 0549 §1): the forge bridge
// reads them at each of its merges, and no other repository's.
CatalogueRepository = "novox/mesh-catalog"
) )
// PlacementEvents are the seat's four events, in the order a proposal says them.
var PlacementEvents = []string{PlacementProposedEvent, PlacementJudgedEvent, PlacementChangedEvent, PlacementWithdrawnEvent}
// PlacementKey is the key, in PlacementBucket, of one node's placement: `placement.<node>`. // PlacementKey is the key, in PlacementBucket, of one node's placement: `placement.<node>`.
func PlacementKey(node string) (string, error) { func PlacementKey(node string) (string, error) {
if err := checkName("node", node); err != nil { if err := checkName("node", node); err != nil {
@@ -51,20 +59,6 @@ func ParsePlacementKey(key string) (node string, err error) {
return node, nil return node, nil
} }
// PlacementFile is the path, in the catalogue repository, of one node's placement file.
func PlacementFile(node string) string { return path.Join(PlacementDir, node+".json") }
// PlacementFileNode is the node a catalogue path is the placement file of, and whether it is one: `placement/<node>.json`
// directly under PlacementDir.
func PlacementFileNode(p string) (string, bool) {
dir, file := path.Split(p)
node, ok := strings.CutSuffix(file, ".json")
if path.Clean(dir) != PlacementDir || !ok || checkName("node", node) != nil {
return "", false
}
return node, true
}
// Provider is the module on a node that answers a requirement or holds a seat: a pin's or a binding's provider. // Provider is the module on a node that answers a requirement or holds a seat: a pin's or a binding's provider.
type Provider struct { type Provider struct {
Node string `json:"node"` Node string `json:"node"`
@@ -109,8 +103,8 @@ type Holding struct {
Kind string `json:"kind,omitempty"` Kind string `json:"kind,omitempty"`
} }
// Placement is one node's placement: the file the catalogue keeps and, with Revision, the value of its key. Modules are // Placement is one node's placement, the value of its key. Modules are in the order the composition reads the node's
// in the order the file names them, which is the order the composition reads the node's modules in. Pins is, by // modules in. Revision is unused since placement left the catalogue (ADR 0567): the key's own revision is the bus's. Pins is, by
// provision, the provider chosen for the node where more than one could answer. // provision, the provider chosen for the node where more than one could answer.
type Placement struct { type Placement struct {
V int `json:"v"` V int `json:"v"`
@@ -122,13 +116,13 @@ type Placement struct {
Holds []Holding `json:"holds,omitempty"` Holds []Holding `json:"holds,omitempty"`
} }
// NodeDecided is what a person decides of the node, kept in its placement file (issue 1126's K2: what a person decides // NodeDecided is what a person decides of the node, kept in its placement key (issue 1126's K2: what a person decides
// lives in the catalogue, what the machine observes lives in its record, NodeRecord): where it is dialled (endpoint) and // lives in placement, what the machine observes lives in its record, NodeRecord): where it is dialled (endpoint) and
// its site, whether it is the private network's hub, its address on the private network, the domain its routed names // its site, whether it is the private network's hub, its address on the private network, the domain its routed names
// are composed under, its operator and agent logins by name and their homes, whether it is adopted rather than // are composed under, its operator and agent logins by name and their homes, whether it is adopted rather than
// converged, the modules taken on an adopted node, the tunnel found on it at adoption and, on the hub, the names given // converged, the modules taken on an adopted node, the tunnel found on it at adoption and, on the hub, the names given
// to that tunnel's peers. The address is a decision written once, at enrolment, by a // to that tunnel's peers. The address is a decision written once, at enrolment, by an
// catalogue pull request: nothing allocates it at run time, and CheckPlacements refuses two nodes with one address. // approved proposal (ADR 0567 §2): nothing allocates it at run time, and CheckPlacements refuses two nodes with one address.
// No secret and no key material: accounts are names, never credentials. // No secret and no key material: accounts are names, never credentials.
type NodeDecided struct { type NodeDecided struct {
Endpoint string `json:"endpoint,omitempty"` Endpoint string `json:"endpoint,omitempty"`
@@ -211,8 +205,8 @@ func DecodePlacement(data []byte) (Placement, error) {
return p, nil return p, nil
} }
// SamePlacement says whether two placements are the same value, so the indexer writes only when they differ. The // SamePlacement says whether two placements are the same value, so the holder writes only when they differ. The
// revision is not compared: a merge that changes no placement writes nothing (ADR 0549 §2). // revision is not compared.
func SamePlacement(a, b Placement) bool { func SamePlacement(a, b Placement) bool {
a.Revision, b.Revision = "", "" a.Revision, b.Revision = "", ""
ja, errA := EncodePlacement(a) ja, errA := EncodePlacement(a)
@@ -220,21 +214,23 @@ func SamePlacement(a, b Placement) bool {
return errA == nil && errB == nil && string(ja) == string(jb) return errA == nil && errB == nil && string(ja) == string(jb)
} }
// PlacementChange is what `placement-changed` says: the node, the catalogue revision, and its modules added, removed // PlacementChange is what `placement-changed` says of one node (ADR 0567 §6): the node, the revision of its key before
// or changed against the placement before. // (0: it had none) and after (0: the key was deleted), and its modules added, removed or changed against the placement
// before.
type PlacementChange struct { type PlacementChange struct {
Node string `json:"node"` Node string `json:"node"`
Revision string `json:"revision"` Before uint64 `json:"before"`
Added []string `json:"added,omitempty"` After uint64 `json:"after"`
Removed []string `json:"removed,omitempty"` Added []string `json:"added,omitempty"`
Changed []string `json:"changed,omitempty"` Removed []string `json:"removed,omitempty"`
Changed []string `json:"changed,omitempty"`
} }
// ChangeOf is what changed from before to after on one node; before is the zero Placement for a node new to placement. // ChangeOf is what changed from before to after on one node; before is the zero Placement for a node new to placement.
// A change of the node's pins or holdings counts as a change of every module placed on it both times, since the // A change of the node's pins or holdings counts as a change of every module placed on it both times, since the
// composition of each may read them. // composition of each may read them.
func ChangeOf(before, after Placement) PlacementChange { func ChangeOf(before, after Placement) PlacementChange {
c := PlacementChange{Node: after.Node, Revision: after.Revision} c := PlacementChange{Node: after.Node}
if c.Node == "" { if c.Node == "" {
c.Node = before.Node c.Node = before.Node
} }
@@ -368,8 +364,8 @@ func (p Placement) Held() []catalogue.Held {
return out return out
} }
// CheckPlacements is the catalogue's own check of every placement file (ADR 0549 §1), over the placements by node, the // CheckPlacements is the holder's check of a proposal laid over every placement (ADR 0567 §2), over the placements by
// modules the catalogue holds and the nodes enrolled. It returns every refusal, in node order, never only the first: a // node, the modules the indexer holds and the nodes with a record. It returns every refusal, in node order, never only the first: a
// placement naming a node not enrolled or a module the catalogue does not hold; two modules of one node on the same host // placement naming a node not enrolled or a module the catalogue does not hold; two modules of one node on the same host
// port; a pin, binding or holding naming a provider that is not placed where it says; a seat of scope mesh held twice, unless the seat is replicated (catalogue.Seat's Replicated, ADR 0223), which // port; a pin, binding or holding naming a provider that is not placed where it says; a seat of scope mesh held twice, unless the seat is replicated (catalogue.Seat's Replicated, ADR 0223), which
// is held once per node; // is held once per node;
@@ -401,7 +397,7 @@ func CheckPlacements(placements map[string]Placement, held func(module string) b
for _, node := range nodes { for _, node := range nodes {
p := placements[node] p := placements[node]
if p.Node != node { if p.Node != node {
refusals = append(refusals, fmt.Sprintf("the placement file of %s names the node %q", node, p.Node)) refusals = append(refusals, fmt.Sprintf("the placement of %s names the node %q", node, p.Node))
} }
if a := strings.TrimSpace(p.Decided.Address); a != "" { if a := strings.TrimSpace(p.Decided.Address); a != "" {
if other, twice := addresses[a]; twice { if other, twice := addresses[a]; twice {
@@ -413,12 +409,12 @@ func CheckPlacements(placements map[string]Placement, held func(module string) b
hubs = append(hubs, node) hubs = append(hubs, node)
} }
if !enrolled(node) { if !enrolled(node) {
refusals = append(refusals, fmt.Sprintf("%s is not an enrolled node", node)) refusals = append(refusals, fmt.Sprintf("%s is a node with no record", node))
} }
hostPorts := map[int]string{} hostPorts := map[int]string{}
for _, m := range p.Modules { for _, m := range p.Modules {
if !held(m.Module) { if !held(m.Module) && !ProvidedByMesh(m.Module) { // the mesh provides the private network itself
refusals = append(refusals, fmt.Sprintf("%s: the catalogue holds no module %s", node, m.Module)) refusals = append(refusals, fmt.Sprintf("%s: the indexer holds no module %s", node, m.Module))
} }
for _, port := range sortedKeys(m.Ports) { for _, port := range sortedKeys(m.Ports) {
at := m.Ports[port] at := m.Ports[port]
+70 -61
View File
@@ -3,73 +3,66 @@ package delivery
import ( import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"slices"
"strings" "strings"
) )
// Placement candidates (issue 1126; ADR 0549 §4: a placement change passes the composition check before any node sees // Placement candidates (novox/hq ADR 0567 §3, ADR 0559 §4): for each node a proposal must hear from, the seat's holder
// it). A catalogue pull request that changes a placement file carries every placement file at its head on its pull // writes one key `<proposal>.<node>` in its state `candidates`: the node's placement with the proposal laid over it for a
// message (Forge.Placements). The indexer writes, for each node whose file at the head differs from the trunk's, the // node whose key it changes, or the node's current key for one whose composition it changes only through another node
// key `<group>.<node>` of its state `placement-candidates`, and deletes it when the pull request merges or closes. It is // (Reached: a binding, a firewall or a resolver entry reaching it). It deletes them when the proposal ends. A state of its
// a state of its own, since the readers of `placement` refuse any key that is not `placement.<node>`. // own, since the readers of `placement` refuse any key that is not `placement.<node>`.
// //
// The node it names composes in shadow with the candidate (its own sealed values, the boxes it already runs) and // The node watches `*.<node>`, composes in shadow with every candidate of its proposal laid over the current placement
// compares the result with what it runs; it applies nothing. Its answer rides in its own have key, `<node>.node` // (its own sealed values, the boxes it already runs) and compares the result with what it runs; it applies nothing. Its
// (HaveNode.Candidates, by group), which that node alone writes; the assigner's check of the group reads every placed // answer rides in its own have key, `<node>.node` (HaveNode.Candidates, by proposal), which that node alone writes; the
// node's answer at the head and fails on any difference, refusal or silence. Event-driven both ways: the candidate key's // holder reads it and judges (ADR 0567 §4). Event-driven both ways: the candidate key's write is the node's event, the
// write is the node's event, the `<node>.node` change the check's. // `<node>.node` change the holder's.
const ( const (
// PlacementCandidatesState is the indexer's state of placement candidates, and PlacementCandidatesBucket its bucket. // PlacementCandidatesState is the seat's state of placement candidates, PlacementCandidatesBucket its bucket and
PlacementCandidatesState = "placement-candidates" // PlacementCandidatesRead the name a reader's manifest declares.
PlacementCandidatesBucket = IndexerModule + "_" + PlacementCandidatesState PlacementCandidatesState = "candidates"
PlacementCandidatesBucket = AssignerModule + "_" + PlacementCandidatesState
PlacementCandidatesRead = AssignerModule + "." + PlacementCandidatesState
// PlacementCandidatesHistory is the revisions the bucket keeps per key. // PlacementCandidatesHistory is the revisions the bucket keeps per key.
PlacementCandidatesHistory = 5 PlacementCandidatesHistory = 5
) )
// PlacementCandidateKey is the key, in PlacementCandidatesBucket, of one node's candidate placement in one group: // PlacementCandidateKey is the key, in PlacementCandidatesBucket, of one node's candidate in one proposal:
// `<group>.<node>`, the group escaped as CandidatesKey escapes it, so a node watches `*.<node>` and a group's keys are // `<proposal>.<node>`, so a node watches `*.<node>` and a proposal's keys are `<proposal>.*`.
// `<group>.*`. func PlacementCandidateKey(proposal, node string) (string, error) {
func PlacementCandidateKey(group, node string) (string, error) { if err := CheckProposalID(proposal); err != nil {
g, err := groupKey(group)
if err != nil {
return "", err return "", err
} }
if err := checkName("node", node); err != nil { if err := checkName("node", node); err != nil {
return "", err return "", err
} }
return g + "." + node, nil return proposal + "." + node, nil
} }
// ParsePlacementCandidateKey is the inverse of PlacementCandidateKey; it refuses any other key, naming it. // ParsePlacementCandidateKey is the inverse of PlacementCandidateKey; it refuses any other key, naming it.
func ParsePlacementCandidateKey(key string) (group, node string, err error) { func ParsePlacementCandidateKey(key string) (proposal, node string, err error) {
g, node, ok := strings.Cut(key, ".") proposal, node, ok := strings.Cut(key, ".")
if !ok || checkName("node", node) != nil { if !ok || checkName("node", node) != nil || CheckProposalID(proposal) != nil {
return "", "", fmt.Errorf("delivery: %q is not a placement candidate key, <group>.<node>", key) return "", "", fmt.Errorf("delivery: %q is not a placement candidate key, <proposal>.<node>", key)
} }
if group, err = parseGroupKey(g); err != nil { return proposal, node, nil
return "", "", fmt.Errorf("delivery: %q is not a placement candidate key: %w", key, err)
}
return group, node, nil
} }
// PlacementCandidate is the value of a candidate key: the group, the catalogue pull request and the head its placement // PlacementCandidate is the value of a candidate key: the proposal it is of, and the node's placement with the proposal
// files were read at, and the node's placement at that head; nil Placement says the head removes the node's file. // laid over it; nil Placement says the proposal takes the node's key away. Reached says the proposal changes not this
// node's key but its composition, through another node's: Placement is then its current key (nil when it has none).
type PlacementCandidate struct { type PlacementCandidate struct {
V int `json:"v"` V int `json:"v"`
Group string `json:"group"` Proposal string `json:"proposal"`
Pull Head `json:"pull"` Reached bool `json:"reached,omitempty"`
Placement *Placement `json:"placement,omitempty"` Placement *Placement `json:"placement,omitempty"`
} }
func (c PlacementCandidate) check() error { func (c PlacementCandidate) check() error {
if c.Group == "" { if err := CheckProposalID(c.Proposal); err != nil {
return fmt.Errorf("delivery: a placement candidate names no group")
}
if err := checkRepository(c.Pull.Repository); err != nil {
return err return err
} }
if c.Pull.Number <= 0 || c.Pull.Head == "" {
return fmt.Errorf("delivery: a placement candidate needs its pull request's number and head")
}
if c.Placement != nil { if c.Placement != nil {
return c.Placement.check() return c.Placement.check()
} }
@@ -90,8 +83,8 @@ func EncodePlacementCandidate(c PlacementCandidate) ([]byte, error) {
return json.Marshal(c) return json.Marshal(c)
} }
// DecodePlacementCandidate reads a candidate; it refuses an unknown version, no group, a pull request without number or // DecodePlacementCandidate reads a candidate; it refuses an unknown version, a proposal id that is not one, and a
// head, and a placement DecodePlacement refuses. // placement DecodePlacement refuses.
func DecodePlacementCandidate(data []byte) (PlacementCandidate, error) { func DecodePlacementCandidate(data []byte) (PlacementCandidate, error) {
var c PlacementCandidate var c PlacementCandidate
if err := decode(data, &c); err != nil { if err := decode(data, &c); err != nil {
@@ -115,14 +108,13 @@ const (
CandidateRefused = "refused" CandidateRefused = "refused"
) )
// CandidateAnswer is a node's answer to its group's candidate keys, in HaveNode.Candidates (ADR 0559 §4): the head // CandidateAnswer is a node's answer to its proposal's candidate keys, in HaveNode.Candidates (ADR 0567 §3): the
// and the revision of the node's own key in the group it answered (0 when the group has no key of its own for it: its // proposal and the revision of the node's own candidate key it answered, its verdict, the modules composed the same, those that differ with both digests, those
// file is unchanged and it answers as placed at the head or on the trunk, judged by Head alone), its verdict, the modules composed the same, those that differ with both digests, those
// the candidate adds (naming the values not yet issued for each, which is no refusal: a module's credential and boxes // the candidate adds (naming the values not yet issued for each, which is no refusal: a module's credential and boxes
// are made only once it is placed, ADR 0549 §5) or removes, and each refusal. The node's own composition is named // are made only once it is placed, ADR 0549 §5) or removes, and each refusal. The node's own composition is named
// "node" (HaveNodeWord). // "node" (HaveNodeWord).
type CandidateAnswer struct { type CandidateAnswer struct {
Head string `json:"head"` Proposal string `json:"proposal"`
Revision uint64 `json:"revision"` Revision uint64 `json:"revision"`
Verdict string `json:"verdict"` Verdict string `json:"verdict"`
Same []string `json:"same,omitempty"` Same []string `json:"same,omitempty"`
@@ -130,21 +122,27 @@ type CandidateAnswer struct {
Added []ModuleAdded `json:"added,omitempty"` Added []ModuleAdded `json:"added,omitempty"`
Removed []string `json:"removed,omitempty"` Removed []string `json:"removed,omitempty"`
Refusals []ModuleRefused `json:"refusals,omitempty"` Refusals []ModuleRefused `json:"refusals,omitempty"`
// Held is each module the candidate cannot compose that the node holds today too, with why: held as today, which
// changes nothing and counts for no verdict (issue 1126: a seed reproduces today).
Held []ModuleRefused `json:"held,omitempty"`
// Refused is why the node could not compose with the candidate at all; Refusals is then empty. // Refused is why the node could not compose with the candidate at all; Refusals is then empty.
Refused string `json:"refused,omitempty"` Refused string `json:"refused,omitempty"`
// Returns is, for a head that deletes the node's placement file, what the node does then: it returns to the // Returns is, for a proposal that takes the node's placement key away, what the node does then: it returns to the
// controller's compositions (ReturnsToController), as a node the catalogue does not place composes none of its own // controller's compositions (ReturnsToController), as a node with no placement key composes none of its own. Same, Differs, Added and Removed then compare the controller's compositions at
// (issue 1126: the seed's revert). Same, Differs, Added and Removed then compare the controller's compositions at
// the node's wants with what it runs, and the verdict is theirs; a want the controller holds no composition for is // the node's wants with what it runs, and the verdict is theirs; a want the controller holds no composition for is
// refused (Refused, without Returns). // refused (Refused, without Returns).
Returns string `json:"returns,omitempty"` Returns string `json:"returns,omitempty"`
// Note is what the node says of how it answered, in words, beside the verdict: a pass that ran over a minute, inputs // Note is what the node says of how it answered, in words, beside the verdict: a pass that ran over a minute, inputs
// it waited for (issue 1126: the shadow shows its work). It changes no verdict. // it waited for (issue 1126: the shadow shows its work). It changes no verdict.
Note string `json:"note,omitempty"` Note string `json:"note,omitempty"`
// ValuesDiffer is how many shown bodies hold a value the node opened on both sides to two values (ValuesDiffer,
// MaskedAnotherValue): the vault's value and the one that runs drifted apart (issue 1126, cause C4). It changes no
// verdict.
ValuesDiffer int `json:"values_differ,omitempty"`
} }
// ReturnsToController is what a node answers a head that deletes its placement file. // ReturnsToController is what a node answers a proposal that takes its placement key away.
const ReturnsToController = "the head deletes this node's placement: it returns to the controller's compositions" const ReturnsToController = "the proposal takes this node's placement away: it returns to the controller's compositions"
// ModuleDiffers is one module the node runs and the candidate still places whose composition is not the one it runs: // ModuleDiffers is one module the node runs and the candidate still places whose composition is not the one it runs:
// both digests (Composed with the candidate, Running what runs), the first resource that differs by its id, or empty // both digests (Composed with the candidate, Running what runs), the first resource that differs by its id, or empty
@@ -163,16 +161,24 @@ type ModuleDiffers struct {
// ResourceBodies is one differing resource of a module: its id, why, and its body composed and as it runs, masked // ResourceBodies is one differing resource of a module: its id, why, and its body composed and as it runs, masked
// (MaskedPair); a side that does not hold it is empty. // (MaskedPair); a side that does not hold it is empty.
type ResourceBodies struct { type ResourceBodies struct {
ID string `json:"id"` ID string `json:"id"`
Why string `json:"why"` Why string `json:"why"`
Composed string `json:"composed,omitempty"` // Paths is every JSON path at which the two bodies differ (DifferingPaths: `sealed`, `content.url`, `links.0`), found
Running string `json:"running,omitempty"` // on the whole bodies before they are cut, a JSON value written in a string read as JSON; a path names keys alone,
// never a value. At most MaxPaths, the last then saying how many more.
Paths []string `json:"paths,omitempty"`
Composed string `json:"composed,omitempty"`
Running string `json:"running,omitempty"`
} }
// The bounds of a module's shown bodies. // The bounds of a module's shown bodies, and of one answer as the node's own key holds it: an answer over
// MaxAnswerBytes is fitted (FitAnswer). The node's key (HaveBucket) takes a value of about 1 MiB
// (buscompose.ChainValueBytes) and holds every proposal's answer beside the node's facts, so one answer keeps to a sixth.
const ( const (
MaxBodies = 20 MaxBodies = 20
MaxBody = 8192 MaxBody = 8192
MaxPaths = 64
MaxAnswerBytes = 160 << 10
) )
// ModuleAdded is one module the candidate places on the node and the node does not run: its composition's digest when // ModuleAdded is one module the candidate places on the node and the node does not run: its composition's digest when
@@ -184,10 +190,13 @@ type ModuleAdded struct {
Unissued []string `json:"unissued,omitempty"` Unissued []string `json:"unissued,omitempty"`
} }
// ModuleRefused is a value missing for a module the node runs, in the words the composition stood in for it with. // ModuleRefused is a value missing for a module the node runs, in the words the composition stood in for it with. Today,
// in CandidateAnswer.Held, is how the node keeps it today (held, failed, runs a build of the old form, held behind
// another), which the candidate keeps.
type ModuleRefused struct { type ModuleRefused struct {
Module string `json:"module"` Module string `json:"module"`
Why string `json:"why"` Why string `json:"why"`
Today string `json:"today,omitempty"`
} }
// VerdictOf is the verdict of an answer's lists: refused on any refusal, differs on any difference, addition or // VerdictOf is the verdict of an answer's lists: refused on any refusal, differs on any difference, addition or
@@ -203,8 +212,8 @@ func VerdictOf(a CandidateAnswer) string {
} }
func (a CandidateAnswer) check() error { func (a CandidateAnswer) check() error {
if a.Head == "" { if CheckProposalID(a.Proposal) != nil || a.Revision == 0 {
return fmt.Errorf("delivery: a candidate answer names no head") return fmt.Errorf("delivery: a candidate answer names no proposal or no revision of its candidate")
} }
switch a.Verdict { switch a.Verdict {
case CandidateSame, CandidateDiffers, CandidateRefused: case CandidateSame, CandidateDiffers, CandidateRefused:
@@ -231,7 +240,7 @@ func (a CandidateAnswer) check() error {
return fmt.Errorf("delivery: a module the candidate adds is not named") return fmt.Errorf("delivery: a module the candidate adds is not named")
} }
} }
for _, d := range a.Refusals { for _, d := range append(slices.Clone(a.Refusals), a.Held...) {
if d.Module == "" || d.Why == "" { if d.Module == "" || d.Why == "" {
return fmt.Errorf("delivery: a candidate refusal names no module or no why") return fmt.Errorf("delivery: a candidate refusal names no module or no why")
} }
+517
View File
@@ -0,0 +1,517 @@
package delivery
import (
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"regexp"
"sort"
"strings"
"time"
)
// A placement changes only by a proposal (novox/hq ADR 0567 §2): `mesh-placement.propose` takes why and a list of
// changes; the holder lays them over the current keys, names the revision of each key it laid them over, and either
// refuses at once with every finding or writes the proposal under its id in its state `proposals` and a candidate for
// each node it must hear from (PlacementCandidate). The proposal's key leaves the state when it is applied, withdrawn,
// declined or expired; its whole content stays in the event that ends it (PlacementEvent), so no proposal is kept in
// the state once it has ended.
const (
// PlacementProposalsState is the seat's state of open proposals, PlacementProposalsBucket its bucket and
// PlacementProposalsRead the name a reader's manifest declares; a key is the proposal's id.
PlacementProposalsState = "proposals"
PlacementProposalsBucket = AssignerModule + "_" + PlacementProposalsState
PlacementProposalsRead = AssignerModule + "." + PlacementProposalsState
// PlacementProposalsHistory is the revisions the bucket keeps per key.
PlacementProposalsHistory = 5
// PlacementPulsesState is the holder's state of bounds (§4): a key per node a proposal waits on, written with a time
// to live of the answer bound, and a key per raised ask, with its ask's; the server's delete marker when one ends is
// the event that fails a silent node or ends an unanswered ask, so the holder keeps no timer. Its bucket allows a
// time to live per key (catalogue.StateDeclaration.KeyTTL).
PlacementPulsesState = "pulses"
PlacementPulsesBucket = AssignerModule + "_" + PlacementPulsesState
)
// PulseAnswerKey is the key of the bound on one node's answer to a proposal: `answer.<proposal>.<node>`.
func PulseAnswerKey(proposal, node string) (string, error) {
if err := CheckProposalID(proposal); err != nil {
return "", err
}
if err := checkName("node", node); err != nil {
return "", err
}
return "answer." + proposal + "." + node, nil
}
// PulseAskKey is the key of the bound on a proposal's raised ask: `ask.<proposal>`.
func PulseAskKey(proposal string) (string, error) {
if err := CheckProposalID(proposal); err != nil {
return "", err
}
return "ask." + proposal, nil
}
// The operations a proposal's change names (ADR 0567 §2).
const (
OpPlace = "place" // place a module on a node, with its host ports and bindings
OpTakeOff = "take-off" // take a module off a node
OpPin = "pin" // pin the provider of a provision on a node
OpUnpin = "unpin" // remove a pin
OpHold = "hold" // a module placed on the node holds a seat
OpUnhold = "unhold" // it holds it no more
OpDecide = "decide" // set a decided field of the node (NodeDecided by its JSON name), its address among them
OpConfig = "config" // set a module's config placement or values on the node
OpSeed = "seed" // the node's whole placement: the take-over (adopt, §7), never given by a caller
)
// The states of an open proposal.
const (
// ProposalAnswering: its candidates are written and the nodes it must hear from have not all answered.
ProposalAnswering = "answering"
// ProposalFailed: its verdict failed; it stays open, for its proposer to read and withdraw, or for a node's later
// answer to judge it again.
ProposalFailed = "failed"
// ProposalAsked: its verdict passed and the operator is asked to approve it.
ProposalAsked = "asked"
)
// VerdictWaits is a proposal's verdict while a node it must hear from has not answered within the bound; it passes
// (VerdictPass) or fails (VerdictFail) as a check's verdict does (ADR 0567 §4).
const VerdictWaits = "waits"
// PlacementProposalChange is one change of a proposal. Node is always named; the rest is the operation's.
type PlacementProposalChange struct {
Op string `json:"op"`
Node string `json:"node"`
Module string `json:"module,omitempty"`
Ports map[string]int `json:"ports,omitempty"`
Bindings map[string]Provider `json:"bindings,omitempty"`
Provision string `json:"provision,omitempty"`
Provider *Provider `json:"provider,omitempty"`
// Claim, Scope, Kind and Site are a holding's (OpHold, OpUnhold).
Claim string `json:"claim,omitempty"`
Scope string `json:"scope,omitempty"`
Kind string `json:"kind,omitempty"`
Site string `json:"site,omitempty"`
// Field and Value are a decided field's (OpDecide): its JSON name in NodeDecided and its value.
Field string `json:"field,omitempty"`
Value json.RawMessage `json:"value,omitempty"`
// MeshWide, Layer, MeshValues and Values are a module's config placement and values (OpConfig), as PlacedModule
// carries them; each given replaces what the module has.
MeshWide map[string]any `json:"mesh-wide,omitempty"`
Layer map[string]any `json:"layer,omitempty"`
MeshValues map[string]any `json:"mesh-wide-values,omitempty"`
Values map[string]any `json:"values,omitempty"`
// Whole is the node's whole placement (OpSeed).
Whole *Placement `json:"whole,omitempty"`
}
// BeforeLease is why the change cannot be proposed before the lease write, or "" (ADR 0567 §8): until then the holder
// keeps the controller's assignments in step through its verbs assign, unassign, pin and unpin, so only those are
// proposable — a module placed or taken off with no host port or binding, a pin set or removed, a module's config values
// and the seed. A host port, a binding, a holding, a decided field and a config layer's places or accesses wait for the
// lease write, which the refusal names.
func (c PlacementProposalChange) BeforeLease() string {
const lease = "which the controller's verbs cannot take from a caller, so it waits for the lease write (ADR 0567 §8)"
switch c.Op {
case OpPlace:
if len(c.Ports)+len(c.Bindings) > 0 {
return fmt.Sprintf("the place of %s on %s names host ports or bindings, %s", c.Module, c.Node, lease)
}
case OpTakeOff, OpPin, OpUnpin, OpSeed:
case OpConfig:
// A module's config values are the state's alone (§13); its layers' places and accesses are the controller's
// until the lease write.
if c.MeshWide != nil || c.Layer != nil {
return fmt.Sprintf("the config of %s on %s sets a layer's places or accesses, %s", c.Module, c.Node, lease)
}
default:
return fmt.Sprintf("a change %s of %s is no assignment or pin, %s", c.Op, c.Node, lease)
}
return ""
}
func (c PlacementProposalChange) check() error {
if err := checkName("node", c.Node); err != nil {
return err
}
needs := func(what string, ok bool) error {
if !ok {
return fmt.Errorf("delivery: a %s change of %s names no %s", c.Op, c.Node, what)
}
return nil
}
switch c.Op {
case OpPlace, OpTakeOff, OpConfig:
return needs("module", checkName("module", c.Module) == nil)
case OpPin:
return needs("provision or provider", c.Provision != "" && c.Provider != nil && c.Provider.Node != "" && c.Provider.Module != "")
case OpUnpin:
return needs("provision", c.Provision != "")
case OpHold, OpUnhold:
return needs("module or claim", checkName("module", c.Module) == nil && c.Claim != "")
case OpDecide:
return needs("field or value", c.Field != "" && len(c.Value) > 0)
case OpSeed:
if c.Whole == nil || c.Whole.Node != c.Node {
return fmt.Errorf("delivery: the seed of %s names no placement of that node", c.Node)
}
return c.Whole.check()
}
return fmt.Errorf("delivery: %q is no change of placement: place, take-off, pin, unpin, hold, unhold, decide or config", c.Op)
}
// LayChanges is the placements of the nodes the changes name with the changes laid over current, in the order given;
// current is every node's placement now (absent: the node has no key). It returns, by node, the laid placement, and
// every finding: a change that does not fit its node (a module taken off that is not placed, configured or holding
// there; a field NodeDecided does not have) or a placement DecodePlacement refuses. What CheckPlacements refuses
// across nodes is the caller's to ask of the result laid over current.
func LayChanges(current map[string]Placement, changes []PlacementProposalChange) (map[string]Placement, []string) {
laid := map[string]Placement{}
var findings []string
of := func(node string) Placement {
if p, ok := laid[node]; ok {
return p
}
p, ok := current[node]
if !ok {
return Placement{V: Version, Node: node, Modules: []PlacedModule{}}
}
return clonePlacement(p)
}
for i, c := range changes {
if err := c.check(); err != nil {
findings = append(findings, fmt.Sprintf("change %d: %v", i+1, err))
continue
}
p := of(c.Node)
at := -1
for j, m := range p.Modules {
if m.Module == c.Module {
at = j
}
}
notPlaced := func() {
findings = append(findings, fmt.Sprintf("change %d: %s is not placed on %s", i+1, c.Module, c.Node))
}
switch c.Op {
case OpSeed:
p = clonePlacement(*c.Whole)
case OpPlace:
m := PlacedModule{Module: c.Module, Ports: c.Ports, Bindings: c.Bindings}
if at >= 0 {
m.MeshWide, m.HasMesh, m.Layer, m.HasLayer = p.Modules[at].MeshWide, p.Modules[at].HasMesh, p.Modules[at].Layer, p.Modules[at].HasLayer
m.MeshValues, m.Values = p.Modules[at].MeshValues, p.Modules[at].Values
p.Modules[at] = m
} else {
p.Modules = append(p.Modules, m)
}
case OpTakeOff:
if at < 0 {
notPlaced()
continue
}
p.Modules = append(p.Modules[:at:at], p.Modules[at+1:]...)
var holds []Holding
for _, h := range p.Holds {
if h.Module != c.Module {
holds = append(holds, h)
}
}
p.Holds = holds
case OpConfig:
if at < 0 {
notPlaced()
continue
}
m := &p.Modules[at]
if c.MeshWide != nil {
m.MeshWide, m.HasMesh = c.MeshWide, true
}
if c.Layer != nil {
m.Layer, m.HasLayer = c.Layer, true
}
if c.MeshValues != nil {
m.MeshValues = c.MeshValues
}
if c.Values != nil {
m.Values = c.Values
}
case OpPin:
if p.Pins == nil {
p.Pins = map[string]Provider{}
}
p.Pins[c.Provision] = *c.Provider
case OpUnpin:
delete(p.Pins, c.Provision)
if len(p.Pins) == 0 {
p.Pins = nil
}
case OpHold:
if at < 0 {
notPlaced()
continue
}
p.Holds = append(p.Holds, Holding{Claim: c.Claim, Scope: c.Scope, Module: c.Module, Site: c.Site, Kind: c.Kind})
case OpUnhold:
kept := p.Holds[:0:0]
for _, h := range p.Holds {
if h.Module != c.Module || h.Claim != c.Claim {
kept = append(kept, h)
}
}
if len(kept) == len(p.Holds) {
findings = append(findings, fmt.Sprintf("change %d: %s holds no %s on %s", i+1, c.Module, c.Claim, c.Node))
continue
}
p.Holds = kept
case OpDecide:
d, err := decide(p.Decided, c.Field, c.Value)
if err != nil {
findings = append(findings, fmt.Sprintf("change %d: %v", i+1, err))
continue
}
p.Decided = d
}
laid[c.Node] = p
}
for _, node := range sortedKeys(laid) {
if err := laid[node].check(); err != nil {
findings = append(findings, err.Error())
}
}
return laid, findings
}
// decide is d with the field named by its JSON name set to value, refused when NodeDecided has no such field or the
// value does not fit it.
func decide(d NodeDecided, field string, value json.RawMessage) (NodeDecided, error) {
raw, _ := json.Marshal(d)
var fields map[string]json.RawMessage
_ = json.Unmarshal(raw, &fields)
if fields == nil {
fields = map[string]json.RawMessage{}
}
if !decidedFields[field] {
return d, fmt.Errorf("a node has no decided field %q", field)
}
fields[field] = value
raw, _ = json.Marshal(fields)
var out NodeDecided
dec := json.NewDecoder(strings.NewReader(string(raw)))
dec.DisallowUnknownFields()
if err := dec.Decode(&out); err != nil {
return d, fmt.Errorf("the decided field %q does not take %s: %v", field, value, err)
}
return out, nil
}
// decidedFields are NodeDecided's fields by their JSON names.
var decidedFields = func() map[string]bool {
raw, _ := json.Marshal(NodeDecided{Endpoint: "x", Site: "x", Hub: true, Address: "x", PublicDomain: "x", Account: "x",
AccountHome: "x", AgentAccount: "x", AgentHome: "x", Adopted: true, Taken: []string{"x"}, Tunnel: &RecordTunnel{},
PeerNames: map[string]string{"x": "x"}})
var m map[string]any
_ = json.Unmarshal(raw, &m)
out := map[string]bool{}
for k := range m {
out[k] = true
}
return out
}()
func clonePlacement(p Placement) Placement {
raw, _ := json.Marshal(p)
var out Placement
_ = json.Unmarshal(raw, &out)
if out.Modules == nil {
out.Modules = []PlacedModule{}
}
return out
}
var proposalIDOK = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{2,62}$`)
// CheckProposalID refuses an id that is no proposal's: lower-case letters, digits and dashes, 3 to 63 of them, so it is
// one key token.
func CheckProposalID(id string) error {
if !proposalIDOK.MatchString(id) {
return fmt.Errorf("delivery: %q is no proposal id (lower-case letters, digits and dashes, one key token)", id)
}
return nil
}
// NewProposalID is a fresh proposal id: its time in UTC and six random hex digits, `p-20261028t183000-1a2b3c`.
func NewProposalID(at time.Time) string {
var b [3]byte
_, _ = rand.Read(b[:])
return "p-" + strings.ToLower(at.UTC().Format("20060102T150405")) + "-" + hex.EncodeToString(b[:])
}
// PlacementVerdict is the holder's judgement of a proposal's answers (ADR 0567 §4). It fails only on a refusal by a
// node that must answer or on its silence past the bound (and, for the seed alone, on any difference); every difference
// is shown by node: Intended is what the holder computed the proposal changes, Unintended what a node answered beside
// it, marked as such in the brief.
type PlacementVerdict struct {
State string `json:"state"`
At time.Time `json:"at"`
Fails []string `json:"fails,omitempty"`
Silent []string `json:"silent,omitempty"`
Until time.Time `json:"until,omitempty"`
Intended map[string][]string `json:"intended,omitempty"`
Unintended map[string][]string `json:"unintended,omitempty"`
// Carried is, for the seed, by node each difference a node found that the seed carries, named for the operator who
// approves it knowingly ("<module>: <resource> at <path>"): the seed fails on no difference, only on a refusal or
// silence (issue 1126).
Carried map[string][]string `json:"carried,omitempty"`
Answers map[string]CandidateAnswer `json:"answers,omitempty"`
Notes []string `json:"notes,omitempty"`
}
// PlacementProposal is an open proposal, the value of its key in PlacementProposalsBucket.
type PlacementProposal struct {
V int `json:"v"`
ID string `json:"id"`
By string `json:"by"`
Why string `json:"why"`
// At is when it was proposed, Written when its newest candidate was written (the answers' bound runs from it).
At time.Time `json:"at"`
Written time.Time `json:"written,omitempty"`
// Changes are as proposed; Seed says they are the take-over (adopt, §7), ExportDigest the digest of the export it
// proposed (ExportDigest), checked again at the warrant.
Changes []PlacementProposalChange `json:"changes"`
Seed bool `json:"seed,omitempty"`
ExportDigest string `json:"export-digest,omitempty"`
// Base is, by node whose key it changes, the revision of the key it was laid over (0: the node had none).
Base map[string]uint64 `json:"base"`
// Must is every node it must hear from (§3): each whose key it changes and each whose composition it changes
// through another's, sorted; set by the holder once it composed, before then the nodes of Base.
Must []string `json:"must,omitempty"`
// HandBack says it is the operator's hand-back (§7): every key of Base deleted on its warrant, with no change and no
// node's answer.
HandBack bool `json:"hand-back,omitempty"`
// State is ProposalAnswering, ProposalFailed or ProposalAsked; Verdict is the latest judgement, Ask the authorising
// ask raised on a passing one.
State string `json:"state"`
Verdict *PlacementVerdict `json:"verdict,omitempty"`
Ask string `json:"ask,omitempty"`
// Words is what happened last, in words (laid again over moved keys, asked again).
Words string `json:"words,omitempty"`
// Again is when `again` asked the holder to judge it now and raise its ask again (§10); the holder does once per
// value.
Again time.Time `json:"again,omitempty"`
// Withdrawn is who withdrew it and why (`withdraw`, §2): the holder then deletes it and its candidates and says
// placement-withdrawn, which only the holder's process can say on its stream.
Withdrawn string `json:"withdrawn,omitempty"`
}
// Nodes is every node whose key the proposal changes, sorted.
func (p PlacementProposal) Nodes() []string { return sortedKeys(p.Base) }
func (p PlacementProposal) check() error {
if err := CheckProposalID(p.ID); err != nil {
return err
}
if strings.TrimSpace(p.By) == "" || strings.TrimSpace(p.Why) == "" {
return fmt.Errorf("delivery: the proposal %s names no proposer or no why", p.ID)
}
if len(p.Base) == 0 || (len(p.Changes) == 0) != p.HandBack || (p.HandBack && p.Seed) {
return fmt.Errorf("delivery: the proposal %s changes nothing, or is a hand-back with changes", p.ID)
}
for i, c := range p.Changes {
if err := c.check(); err != nil {
return fmt.Errorf("delivery: the proposal %s, change %d: %w", p.ID, i+1, err)
}
if (c.Op == OpSeed) != p.Seed {
return fmt.Errorf("delivery: the proposal %s mixes the seed with other changes", p.ID)
}
if _, ok := p.Base[c.Node]; !ok {
return fmt.Errorf("delivery: the proposal %s changes %s and names no base revision of its key", p.ID, c.Node)
}
}
switch p.State {
case ProposalAnswering, ProposalFailed:
case ProposalAsked:
if p.Ask == "" || p.Verdict == nil || p.Verdict.State != VerdictPass {
return fmt.Errorf("delivery: the proposal %s is asked with no ask or no passing verdict", p.ID)
}
default:
return fmt.Errorf("delivery: the proposal %s is in no state %q", p.ID, p.State)
}
if p.Seed && p.ExportDigest == "" {
return fmt.Errorf("delivery: the seed %s names no export digest", p.ID)
}
return nil
}
// EncodePlacementProposal is p as JSON at Version; it refuses what DecodePlacementProposal refuses.
func EncodePlacementProposal(p PlacementProposal) ([]byte, error) {
if err := p.check(); err != nil {
return nil, err
}
p.V = Version
return json.Marshal(p)
}
// DecodePlacementProposal reads a proposal; it refuses an unknown version, an id that is none, no proposer or why, no
// change, a change it does not take or of a node with no base revision, the seed mixed with other changes or without its
// export digest, and a state it does not know.
func DecodePlacementProposal(data []byte) (PlacementProposal, error) {
var p PlacementProposal
if err := decode(data, &p); err != nil {
return PlacementProposal{}, err
}
if err := p.check(); err != nil {
return PlacementProposal{}, err
}
return p, nil
}
// ExportDigest is the digest of an export's answer, every node's placement, as the seed proposes it: the same
// placements give the same digest, whatever order they came in.
func ExportDigest(placements map[string]Placement) string {
h := sha256.New()
for _, node := range sortedKeys(placements) {
raw, _ := EncodePlacement(placements[node])
fmt.Fprintf(h, "%s\n%s\n", node, raw)
}
return "sha256:" + hex.EncodeToString(h.Sum(nil))
}
// ChangesDigest is the digest of a proposal's changes, which its authorising ask is bound to (§5).
func ChangesDigest(p PlacementProposal) string {
raw, _ := json.Marshal(struct {
ID string `json:"id"`
Changes []PlacementProposalChange `json:"changes"`
Base map[string]uint64 `json:"base"`
}{p.ID, p.Changes, p.Base})
sum := sha256.Sum256(raw)
return "sha256:" + hex.EncodeToString(sum[:])
}
// PlacementWarrant is who approved a proposal, through which channel and when (§5, §6): the warrant's id as the teller
// wrote it (the ask's id and when), and its words, which name the person and the channel that proved them
// (asks.Warrant.Says).
type PlacementWarrant struct {
Warrant string `json:"warrant"`
Words string `json:"words"`
At time.Time `json:"at"`
}
// PlacementEvent is what each of the seat's events carries (§6): the proposal whole (its id, by, why, changes, base
// revisions and verdict with every node's answer); for placement-changed, by node the revisions before and after and
// the modules added, removed or changed, and the warrant; for placement-withdrawn, why it ended.
type PlacementEvent struct {
Proposal PlacementProposal `json:"proposal"`
Nodes []PlacementChange `json:"nodes,omitempty"`
Warrant *PlacementWarrant `json:"warrant,omitempty"`
Ended string `json:"ended,omitempty"`
}
// SortChanges orders a placement-changed event's nodes by node.
func SortChanges(c []PlacementChange) {
sort.Slice(c, func(i, j int) bool { return c[i].Node < c[j].Node })
}
+294 -55
View File
@@ -1,11 +1,15 @@
package delivery package delivery
import ( import (
"bytes"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"reflect" "reflect"
"slices"
"strings" "strings"
"testing" "testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
) )
@@ -29,15 +33,18 @@ func TestPlacement(t *testing.T) {
if _, err := PlacementKey("a.b"); err == nil { if _, err := PlacementKey("a.b"); err == nil {
t.Error("PlacementKey took a dotted node") t.Error("PlacementKey took a dotted node")
} }
if PlacementBucket != "mesh-delivery-indexer_placement" || PlacementFile("ace") != "placement/ace.json" { // ADR 0567 §1: the seat's three states are named after the holding module, and every reader takes the names here.
t.Fatalf("bucket %q, file %q", PlacementBucket, PlacementFile("ace")) for got, want := range map[string]string{PlacementBucket: "mesh-delivery-assigner_placement",
} PlacementProposalsBucket: "mesh-delivery-assigner_proposals", PlacementCandidatesBucket: "mesh-delivery-assigner_candidates",
for p, want := range map[string]string{"placement/ace.json": "ace", "./placement/ace.json": "ace", PlacementRead: "mesh-delivery-assigner.placement", PlacementCandidatesRead: "mesh-delivery-assigner.candidates",
"placement/x/ace.json": "", "modules/ace.json": "", "placement/ace.yaml": ""} { PlacementProposalsRead: "mesh-delivery-assigner.proposals"} {
if n, ok := PlacementFileNode(p); n != want || ok != (want != "") { if got != want {
t.Errorf("PlacementFileNode(%q) = %q, %v", p, n, ok) t.Errorf("a bucket is named %q, want %q", got, want)
} }
} }
if PlacementHistory != 64 || PlacementStreamDays != 30 {
t.Errorf("history %d, stream %d days", PlacementHistory, PlacementStreamDays)
}
}) })
g14 := Placement{Node: "g14", Revision: "abc", g14 := Placement{Node: "g14", Revision: "abc",
@@ -126,10 +133,9 @@ func TestPlacement(t *testing.T) {
t.Run("what a change says", func(t *testing.T) { t.Run("what a change says", func(t *testing.T) {
after := g14 after := g14
after.Revision = "def"
after.Modules = []PlacedModule{{Module: "app", Ports: map[string]int{"8080": 18081}}, {Module: "new"}} after.Modules = []PlacedModule{{Module: "app", Ports: map[string]int{"8080": 18081}}, {Module: "new"}}
got := ChangeOf(g14, after) got := ChangeOf(g14, after)
want := PlacementChange{Node: "g14", Revision: "def", Added: []string{"new"}, Removed: []string{"zsh"}, Changed: []string{"app"}} want := PlacementChange{Node: "g14", Added: []string{"new"}, Removed: []string{"zsh"}, Changed: []string{"app"}}
if !reflect.DeepEqual(got, want) { if !reflect.DeepEqual(got, want) {
t.Errorf("ChangeOf = %+v, want %+v", got, want) t.Errorf("ChangeOf = %+v, want %+v", got, want)
} }
@@ -149,6 +155,16 @@ func TestPlacement(t *testing.T) {
if r := CheckPlacements(map[string]Placement{"g14": g14, "novox": novox}, held, enrolled); len(r) != 0 { if r := CheckPlacements(map[string]Placement{"g14": g14, "novox": novox}, held, enrolled); len(r) != 0 {
t.Fatalf("a sound placement is refused: %v", r) t.Fatalf("a sound placement is refused: %v", r)
} }
// Issue 1126: the private network is the mesh's own module, held by no repository the indexer reads, and placed on
// every node by the seed; the check takes it whatever the indexer holds.
wired := Placement{Node: "novox", Modules: []PlacedModule{{Module: "mesh-wireguard"}}}
if r := CheckPlacements(map[string]Placement{"novox": wired}, func(string) bool { return false }, enrolled); len(r) != 0 ||
!ProvidedByMesh("mesh-wireguard") || ProvidedByMesh("postgres") {
t.Errorf("the private network placed with no repository holding it is refused: %v", r)
}
if m, _, err := ProvidedManifest("mesh-wireguard"); err != nil || m.Module != "mesh-wireguard" {
t.Errorf("the provided manifest = %+v, %v", m, err)
}
// A replicated mesh seat (ADR 0223) is held on two nodes; a seat that is not replicated is refused there. // A replicated mesh seat (ADR 0223) is held on two nodes; a seat that is not replicated is refused there.
for seat, refused := range map[string]bool{"mesh-dns-resolver": false, "mesh-broker": true} { for seat, refused := range map[string]bool{"mesh-dns-resolver": false, "mesh-broker": true} {
a := Placement{Node: "ace", Modules: []PlacedModule{{Module: "dns"}}, Holds: []Holding{{Claim: seat, Scope: "mesh", Module: "dns"}}} a := Placement{Node: "ace", Modules: []PlacedModule{{Module: "dns"}}, Holds: []Holding{{Claim: seat, Scope: "mesh", Module: "dns"}}}
@@ -168,7 +184,7 @@ func TestPlacement(t *testing.T) {
twice.Decided.Address = "10.77.0.2" twice.Decided.Address = "10.77.0.2"
r := CheckPlacements(map[string]Placement{"g14": clash, "novox": novox, "stranger": twice}, held, enrolled) r := CheckPlacements(map[string]Placement{"g14": clash, "novox": novox, "stranger": twice}, held, enrolled)
for _, want := range []string{"holds no module ghost", "ghost and app are both published on host port 18080", for _, want := range []string{"holds no module ghost", "ghost and app are both published on host port 18080",
"bound to ace/postgres", "pin of shell names g14/bash", "stranger is not an enrolled node", "bound to ace/postgres", "pin of shell names g14/bash", "stranger is a node with no record",
"mesh-store, held once for the mesh, is held on novox and stranger", "mesh-store, held once for the mesh, is held on novox and stranger",
"g14 and stranger both have the address 10.77.0.2", "novox, stranger are each placed as it"} { "g14 and stranger both have the address 10.77.0.2", "novox, stranger are each placed as it"} {
found := false found := false
@@ -181,58 +197,56 @@ func TestPlacement(t *testing.T) {
} }
}) })
// Issue 1126: a catalogue pull request's placement, one candidate key per node, and each node's answer in its have. // ADR 0567 §3: a proposal's candidates, one key `<proposal>.<node>` per node it must hear from, and each node's answer
// in its have.
t.Run("candidates", func(t *testing.T) { t.Run("candidates", func(t *testing.T) {
key, err := PlacementCandidateKey("feat/1.2", "g14") const id = "p-20261028t183000-1a2b3c"
if err != nil || key != "feat/1=2e2.g14" { key, err := PlacementCandidateKey(id, "g14")
if err != nil || key != id+".g14" {
t.Fatalf("PlacementCandidateKey = %q, %v", key, err) t.Fatalf("PlacementCandidateKey = %q, %v", key, err)
} }
if g, n, err := ParsePlacementCandidateKey(key); err != nil || g != "feat/1.2" || n != "g14" { if p, n, err := ParsePlacementCandidateKey(key); err != nil || p != id || n != "g14" {
t.Fatalf("ParsePlacementCandidateKey = %q, %q, %v", g, n, err) t.Fatalf("ParsePlacementCandidateKey = %q, %q, %v", p, n, err)
} }
for _, bad := range []string{"g14", ".g14", "feat.g.14", "feat.", "fe=t.g14"} { for _, bad := range []string{"g14", ".g14", "p-x.g.14", "p-x.", "P-X.g14", "feat/1.g14"} {
if _, _, err := ParsePlacementCandidateKey(bad); err == nil { if _, _, err := ParsePlacementCandidateKey(bad); err == nil {
t.Errorf("ParsePlacementCandidateKey(%q) took it", bad) t.Errorf("ParsePlacementCandidateKey(%q) took it", bad)
} }
} }
if PlacementCandidatesBucket != "mesh-delivery-indexer_placement-candidates" {
t.Errorf("bucket %q", PlacementCandidatesBucket)
}
pull := Head{Repository: CatalogueRepository, Number: 7, Head: "h1"}
for name, c := range map[string]PlacementCandidate{ for name, c := range map[string]PlacementCandidate{
"placed": {Group: "feat/x", Pull: pull, Placement: &g14}, "placed": {Proposal: id, Placement: &g14},
"removed": {Group: "feat/x", Pull: pull}, "removed": {Proposal: id},
"reached": {Proposal: id, Reached: true, Placement: &g14},
} { } {
raw, err := EncodePlacementCandidate(c) raw, err := EncodePlacementCandidate(c)
if err != nil { if err != nil {
t.Fatalf("%s: %v", name, err) t.Fatalf("%s: %v", name, err)
} }
back, err := DecodePlacementCandidate(raw) back, err := DecodePlacementCandidate(raw)
if err != nil || back.Group != c.Group || back.Pull != pull || (back.Placement == nil) != (c.Placement == nil) || if err != nil || back.Proposal != id || back.Reached != c.Reached || (back.Placement == nil) != (c.Placement == nil) ||
(c.Placement != nil && !SamePlacement(*back.Placement, g14)) { (c.Placement != nil && !SamePlacement(*back.Placement, g14)) {
t.Errorf("%s: round trip %+v, %v", name, back, err) t.Errorf("%s: round trip %+v, %v", name, back, err)
} }
} }
for name, c := range map[string]PlacementCandidate{ for name, c := range map[string]PlacementCandidate{
"no group": {Pull: pull}, "no proposal": {},
"no head": {Group: "g", Pull: Head{Repository: CatalogueRepository, Number: 7}}, "a group": {Proposal: "feat/x"},
"bad place": {Group: "g", Pull: pull, Placement: &Placement{Node: "a.b"}}, "bad place": {Proposal: id, Placement: &Placement{Node: "a.b"}},
"no repo form": {Group: "g", Pull: Head{Repository: "mesh-catalog", Number: 7, Head: "h"}},
} { } {
if _, err := EncodePlacementCandidate(c); err == nil { if _, err := EncodePlacementCandidate(c); err == nil {
t.Errorf("%s: taken", name) t.Errorf("%s: taken", name)
} }
} }
answers := map[string]CandidateAnswer{ answers := map[string]CandidateAnswer{
"same": {Head: "h1", Revision: 3, Verdict: CandidateSame, Same: []string{"app", "node"}}, "same": {Proposal: id, Revision: 3, Verdict: CandidateSame, Same: []string{"app", "node"}},
"differs": {Head: "h1", Revision: 3, Verdict: CandidateDiffers, Differs: []ModuleDiffers{{Module: "app", Resource: "file:/etc/app", Why: "the resource differs"}}}, "differs": {Proposal: id, Revision: 3, Verdict: CandidateDiffers, Differs: []ModuleDiffers{{Module: "app", Resource: "file:/etc/app", Why: "the resource differs"}}},
"refused": {Head: "h1", Revision: 3, Verdict: CandidateRefused, Refused: "unbound postgres"}, "refused": {Proposal: id, Revision: 3, Verdict: CandidateRefused, Refused: "unbound postgres"},
"added": {Head: "h1", Verdict: CandidateDiffers, Same: []string{"node"}, Added: []ModuleAdded{{Module: "app", Unissued: []string{"app/token"}}}, Removed: []string{"old"}}, "added": {Proposal: id, Revision: 1, Verdict: CandidateDiffers, Same: []string{"node"}, Added: []ModuleAdded{{Module: "app", Unissued: []string{"app/token"}}}, Removed: []string{"old"}},
"a value": {Head: "h1", Verdict: CandidateRefused, Same: []string{"node"}, Refusals: []ModuleRefused{{Module: "db", Why: "stands in for db/password"}}}, "a value": {Proposal: id, Revision: 1, Verdict: CandidateRefused, Same: []string{"node"}, Refusals: []ModuleRefused{{Module: "db", Why: "stands in for db/password"}}},
// A head that deletes the node's placement file: the node returns to the controller's compositions (issue 1126, // A head that deletes the node's placement file: the node returns to the controller's compositions (issue 1126,
// the seed's revert, mesh-catalog #526). // the seed's revert, mesh-catalog #526).
"returns": {Head: "h1", Verdict: CandidateSame, Returns: ReturnsToController, Same: []string{"app", "node"}}, "returns": {Proposal: id, Revision: 1, Verdict: CandidateSame, Returns: ReturnsToController, Same: []string{"app", "node"}},
"returns, the controller's differs": {Head: "h1", Verdict: CandidateDiffers, Returns: ReturnsToController, Same: []string{"node"}, "returns, the controller's differs": {Proposal: id, Revision: 1, Verdict: CandidateDiffers, Returns: ReturnsToController, Same: []string{"node"},
Differs: []ModuleDiffers{{Module: "app", Resource: "file:/etc/app", Why: "the resource differs"}}, Removed: []string{"old"}}, Differs: []ModuleDiffers{{Module: "app", Resource: "file:/etc/app", Why: "the resource differs"}}, Removed: []string{"old"}},
} }
n := HaveNode{Composed: "sha256:" + strings.Repeat("a", 64), Sealed: []string{"box"}, Candidates: answers, n := HaveNode{Composed: "sha256:" + strings.Repeat("a", 64), Sealed: []string{"box"}, Candidates: answers,
@@ -288,16 +302,30 @@ func TestPlacement(t *testing.T) {
sealedFile := func(id, sealed, mode string) json.RawMessage { sealedFile := func(id, sealed, mode string) json.RawMessage {
return json.RawMessage(`{"id":"` + id + `","type":"file","module":"nats","path":"/etc/` + id + `","mode":"` + mode + `","sealed":"` + sealed + `"}`) return json.RawMessage(`{"id":"` + id + `","type":"file","module":"nats","path":"/etc/` + id + `","mode":"` + mode + `","sealed":"` + sealed + `"}`)
} }
shown := CompareComposedWith(map[string]Composed{"k": comp(sealedFile("a", "box-1", "0600"), sealedFile("b", "box-2", "0600"), file("c", "x"))}, // Issue 1126, C1: a sealed value is judged by the value it opens to at its own slot: b sealed again (box-2 and
[]Composed{comp(sealedFile("a", "box-1", "0644"), sealedFile("b", "box-3", "0600"), file("c", "y"))}, // box-3 open to one value) is no difference; d holds at its slot a box of another value, though that value is
func(a, b string) (same, known bool) { return a == "box-2" && b == "box-3", true }) // the one b holds, and differs at its path `sealed`.
if len(shown.Differs) != 1 || len(shown.Differs[0].Bodies) != 3 { sameBox := func(a, b string) (same, known bool) {
t.Fatalf("every differing resource is not shown: %+v", shown.Differs) return (a == "box-2" && b == "box-3") || a == b, true
} }
for _, row := range []struct{ id, has, hasNot string }{ shown := CompareComposedWith(map[string]Composed{"k": comp(sealedFile("a", "box-1", "0600"), sealedFile("b", "box-2", "0600"),
{"a", `"mode":"0600"`, "box-1"}, file("c", "x"), sealedFile("d", "box-2", "0600"))},
{"b", "another seal of the same value", "box-2"}, []Composed{comp(sealedFile("a", "box-1", "0644"), sealedFile("b", "box-3", "0600"), file("c", "y"), sealedFile("d", "box-4", "0600"))},
{"c", `"content":"x"`, ""}, sameBox)
if len(shown.Differs) != 1 || len(shown.Differs[0].Bodies) != 3 {
t.Fatalf("every differing resource is not shown, or b sealed again is shown: %+v", shown.Differs)
}
if c := CompareComposedWith(map[string]Composed{"k": comp(sealedFile("b", "box-2", "0600"))},
[]Composed{comp(sealedFile("b", "box-3", "0600"))}, sameBox); len(c.Same) != 1 {
t.Errorf("a value sealed again at its own slot is not the same: %+v", c)
}
for _, row := range []struct {
id, has, hasNot string
paths []string
}{
{"a", `"mode":"0600"`, "box-1", []string{"mode"}},
{"c", `"content":"x"`, "", []string{"content"}},
{"d", "<masked: another value>", "box-2", []string{"sealed"}},
} { } {
var body ResourceBodies var body ResourceBodies
for _, b := range shown.Differs[0].Bodies { for _, b := range shown.Differs[0].Bodies {
@@ -306,16 +334,118 @@ func TestPlacement(t *testing.T) {
} }
} }
both := body.Composed + body.Running both := body.Composed + body.Running
if !strings.Contains(both, row.has) || (row.hasNot != "" && strings.Contains(both, row.hasNot)) { if !strings.Contains(both, row.has) || (row.hasNot != "" && strings.Contains(both, row.hasNot)) ||
t.Errorf("%s shown as %+v", row.id, body) !slices.Equal(body.Paths, row.paths) {
t.Errorf("%s shown as %+v, want paths %v", row.id, body, row.paths)
} }
} }
// Issue 1126, C1, the seed-7 probe: a file naming secrets carries each sealed value in its `secrets`, and each entry
// is a slot of its own. The shape is intoFile's output, captured from catalogue for a file naming two secrets
// (2026-10-28), with the module a composition names each resource of; only the boxes are replaced.
withSecrets := func(db, key string) json.RawMessage {
return json.RawMessage(`{"content":"[database]\npassword = ${secret:postgres-database}\n[security]\nsecret_key = ${secret:secret-key}\n",` +
`"id":"grafana.ini","mode":"0600","module":"nats","path":"/etc/grafana/grafana.ini","secrets":{"postgres-database":"` + db +
`","secret-key":"` + key + `"},"type":"file"}`)
}
if c := CompareComposedWith(map[string]Composed{"k": comp(withSecrets("box-2", "box-1"))},
[]Composed{comp(withSecrets("box-3", "box-1"))}, sameBox); len(c.Same) != 1 {
t.Errorf("a secret sealed again at its own entry is not the same: %+v", c)
}
if kept := KeepSeals(comp(withSecrets("box-2", "box-1")), SealsIn(comp(withSecrets("box-3", "box-1"))), sameBox); !bytes.Equal(kept.Resources[0], withSecrets("box-3", "box-1")) {
t.Errorf("the box that runs is not kept byte for byte: %s", kept.Resources[0])
}
if s := SealsIn(comp(withSecrets("box-2", "box-1"))); s[SealSlot("grafana.ini", SecretSlotField("postgres-database"))] != "box-2" || len(s) != 2 {
t.Errorf("the secrets' slots are %v", s)
}
// Another value at one entry differs there alone, each entry masked by what it opens to.
// box-2 and box-4 open to one value, so the box that runs is kept there; box-5 and box-6 are two values.
mixed := CompareComposedWith(map[string]Composed{"k": comp(withSecrets("box-2", "box-5"))},
[]Composed{comp(withSecrets("box-4", "box-6"))}, func(a, b string) (bool, bool) {
return (a == "box-2" && b == "box-4") || a == b, true
})
if len(mixed.Differs) != 1 || len(mixed.Differs[0].Bodies) != 1 {
t.Fatalf("a secret of another value is not shown: %+v", mixed)
}
if body := mixed.Differs[0].Bodies[0]; !slices.Equal(body.Paths, []string{"secrets.secret-key"}) ||
!strings.Contains(body.Composed, `"secret-key":"<masked: another value>"`) || strings.Contains(body.Composed+body.Running, "box-") ||
!strings.Contains(body.Running, `"postgres-database":"<masked: the same on both sides>"`) {
t.Errorf("a file's secrets shown as %+v", body)
}
if c, _ := MaskedPair(withSecrets("box-2", "box-5"), withSecrets("box-4", "box-6"), func(a, b string) (bool, bool) {
return a == "box-2", true
}); !strings.Contains(c, `"postgres-database":"<masked: another seal of the same value>"`) || !strings.Contains(c, `"secret-key":"<masked: another value>"`) {
t.Errorf("each secret is not masked by what it opens to: %s", c)
}
// Only a file's secrets are split by entry: a vault box (an object under `vault-sealed`) is one masked word and one
// path, judged whole by what it opens to (rev-seed-same-3, round 1).
box := func(ct string) json.RawMessage {
return json.RawMessage(`{"id":"core-box","type":"file","module":"nats","path":"/etc/core","mode":"0600","vault-sealed":{"v":1,"eph":"e-` +
ct + `","ct":"` + ct + `"}}`)
}
vault := CompareComposedWith(map[string]Composed{"k": comp(box("c1"))}, []Composed{comp(box("c2"))},
func(a, b string) (bool, bool) {
return false, strings.Contains(a, `"ct":"c1"`) && strings.Contains(b, `"ct":"c2"`)
})
if len(vault.Differs) != 1 || len(vault.Differs[0].Bodies) != 1 {
t.Fatalf("a box of another value is not shown: %+v", vault)
}
if body := vault.Differs[0].Bodies[0]; !slices.Equal(body.Paths, []string{"vault-sealed"}) ||
!strings.Contains(body.Composed, `"vault-sealed":"<masked: another value>"`) || strings.Contains(body.Composed+body.Running, "c1") {
t.Errorf("a vault box shown as %+v", body)
}
// Issue 1126, T1: the paths are found on the whole bodies, before they are cut; JSON in a string is walked into, a
// key a secret could be under is named and never shown.
long := strings.Repeat("x", MaxBody)
conf := func(link, password string) json.RawMessage {
inner, _ := json.Marshal(map[string]any{"pad": long, "links": []string{"a", link}, "password": password})
outer, _ := json.Marshal(map[string]any{"id": "conf", "type": "file", "module": "nats", "path": "/etc/conf",
"mode": "0600", "content": string(inner)})
return outer
}
far := CompareComposedWith(map[string]Composed{"k": comp(conf("b", "hunter2"))}, []Composed{comp(conf("c", "hunter3"))}, nil)
if len(far.Differs) != 1 || len(far.Differs[0].Bodies) != 1 ||
!slices.Equal(far.Differs[0].Bodies[0].Paths, []string{"content.links.1", "content.password"}) ||
!strings.Contains(far.Differs[0].Bodies[0].Composed, "cut at") || strings.Contains(fmt.Sprint(far), "hunter") {
t.Errorf("a difference past the cut: %+v", far.Differs)
}
// And an answer too large for the node's key is fitted: the texts go first, the verdict and the paths stay.
big := CandidateAnswer{Proposal: "p-x1", Revision: 1, Verdict: CandidateDiffers, Differs: far.Differs}
if fitted := FitAnswer(big, 4<<10); len(fitted.Differs[0].Bodies[0].Composed) != 0 || len(fitted.Differs[0].Bodies[0].Paths) != 2 ||
fitted.Verdict != CandidateDiffers || !strings.Contains(fitted.Note, "cut") || big.Differs[0].Bodies[0].Composed == "" {
t.Errorf("fitted as %+v", fitted)
}
if c, r := MaskedPair(json.RawMessage(`{"env":{"DB_PASSWORD":"hunter2","PORT":"80"},"list":[{"token":"t1"}]}`), if c, r := MaskedPair(json.RawMessage(`{"env":{"DB_PASSWORD":"hunter2","PORT":"80"},"list":[{"token":"t1"}]}`),
json.RawMessage(`{"env":{"DB_PASSWORD":"hunter3","PORT":"81"},"list":[{"token":"t1"}]}`), nil); strings.Contains(c+r, "hunter") || json.RawMessage(`{"env":{"DB_PASSWORD":"hunter3","PORT":"81"},"list":[{"token":"t1"}]}`), nil); strings.Contains(c+r, "hunter") ||
strings.Contains(c+r, `"t1"`) || !strings.Contains(r, `"PORT":"81"`) || !strings.Contains(c, "<masked: differs>") || strings.Contains(c+r, `"t1"`) || !strings.Contains(r, `"PORT":"81"`) || !strings.Contains(c, "<masked: differs>") ||
!strings.Contains(c, "<masked: the same on both sides>") { !strings.Contains(c, "<masked: the same on both sides>") {
t.Errorf("masked as %s and %s", c, r) t.Errorf("masked as %s and %s", c, r)
} }
// Issue 1126, cause C4: the own secrets whose two sides open to two values are read from the shown bodies, and
// every body so masked is counted, own or not.
// The ids are as a node runs them, its module's name first (captured from novox's shadow answer, 2026-10-28:
// `only-office.needs-jwt`); another module's need, named here, is not nats's.
drift := CompareComposedWith(map[string]Composed{"k": comp(sealedFile("nats.needs-jwt", "box-1", "0600"),
sealedFile("nats.needs-salt", "box-4", "0600"), sealedFile("nats.conf", "box-6", "0600"),
sealedFile("other.needs-key", "box-8", "0600"))},
[]Composed{comp(sealedFile("nats.needs-jwt", "box-2", "0600"), sealedFile("nats.needs-salt", "box-5", "0600"),
sealedFile("nats.conf", "box-7", "0600"), sealedFile("other.needs-key", "box-9", "0600"))},
func(a, b string) (same, known bool) { return a == "box-4", true })
got := CandidateAnswer{Proposal: "p-x1", Revision: 1, Differs: drift.Differs}
if own := OwnValuesDiffer(got); len(own) != 1 || own[0] != (OwnValueDiffers{Module: "nats", Name: "jwt"}) {
t.Errorf("the own secrets with another value are %+v, want nats jwt alone", own)
}
if n := ValuesDiffer(got); n != 3 {
t.Errorf("%d bodies hold another value, want 3 (jwt, conf and other's key)", n)
}
for id, want := range map[string]string{"nats.needs-jwt": "jwt", "needs-jwt": "jwt", "other.needs-jwt": "",
"nats.conf": "", "nats.needs-": ""} {
if name, own := OwnNeedName("nats", id); name != want && (own || want != "") {
t.Errorf("%s names nats's own secret %q (%v), want %q", id, name, own, want)
}
}
if w := WithValuesDiffer(map[string]CandidateAnswer{"g": got}); w["g"].ValuesDiffer != 3 {
t.Errorf("the answer carries values_differ %d", w["g"].ValuesDiffer)
}
if c := CompareComposed(nil, []Composed{{Node: "novox", Resources: []json.RawMessage{}}}); len(c.Removed) != 1 || c.Removed[0] != "node" { if c := CompareComposed(nil, []Composed{{Node: "novox", Resources: []json.RawMessage{}}}); len(c.Removed) != 1 || c.Removed[0] != "node" {
t.Errorf("the node's own composition run and not composed is removed: %+v", c) t.Errorf("the node's own composition run and not composed is removed: %+v", c)
} }
@@ -324,16 +454,16 @@ func TestPlacement(t *testing.T) {
} }
for name, a := range map[string]CandidateAnswer{ for name, a := range map[string]CandidateAnswer{
"no head": {Verdict: CandidateSame}, "no head": {Verdict: CandidateSame},
"same with a diff": {Head: "h", Verdict: CandidateSame, Differs: answers["differs"].Differs}, "same with a diff": {Proposal: "p-x1", Revision: 1, Verdict: CandidateSame, Differs: answers["differs"].Differs},
"differs with none": {Head: "h", Verdict: CandidateDiffers}, "differs with none": {Proposal: "p-x1", Revision: 1, Verdict: CandidateDiffers},
"refused no reason": {Head: "h", Verdict: CandidateRefused}, "refused no reason": {Proposal: "p-x1", Revision: 1, Verdict: CandidateRefused},
"unknown verdict": {Head: "h", Verdict: "maybe"}, "unknown verdict": {Proposal: "p-x1", Revision: 1, Verdict: "maybe"},
"diff with no module": {Head: "h", Verdict: CandidateDiffers, Differs: []ModuleDiffers{{Why: "x"}}}, "diff with no module": {Proposal: "p-x1", Revision: 1, Verdict: CandidateDiffers, Differs: []ModuleDiffers{{Why: "x"}}},
"same with an added": {Head: "h", Verdict: CandidateSame, Added: []ModuleAdded{{Module: "a"}}}, "same with an added": {Proposal: "p-x1", Revision: 1, Verdict: CandidateSame, Added: []ModuleAdded{{Module: "a"}}},
"differs refusing": {Head: "h", Verdict: CandidateDiffers, Removed: []string{"a"}, Refusals: []ModuleRefused{{Module: "b", Why: "x"}}}, "differs refusing": {Proposal: "p-x1", Revision: 1, Verdict: CandidateDiffers, Removed: []string{"a"}, Refusals: []ModuleRefused{{Module: "b", Why: "x"}}},
"refused and modules": {Head: "h", Verdict: CandidateRefused, Refused: "x", Same: []string{"a"}}, "refused and modules": {Proposal: "p-x1", Revision: 1, Verdict: CandidateRefused, Refused: "x", Same: []string{"a"}},
"returns refusing": {Head: "h", Verdict: CandidateRefused, Returns: ReturnsToController, Refusals: []ModuleRefused{{Module: "b", Why: "x"}}}, "returns refusing": {Proposal: "p-x1", Revision: 1, Verdict: CandidateRefused, Returns: ReturnsToController, Refusals: []ModuleRefused{{Module: "b", Why: "x"}}},
"returns refused": {Head: "h", Verdict: CandidateRefused, Returns: ReturnsToController, Refused: "x"}, "returns refused": {Proposal: "p-x1", Revision: 1, Verdict: CandidateRefused, Returns: ReturnsToController, Refused: "x"},
} { } {
if _, err := EncodeHaveNode(HaveNode{Candidates: map[string]CandidateAnswer{"g": a}}); err == nil { if _, err := EncodeHaveNode(HaveNode{Candidates: map[string]CandidateAnswer{"g": a}}); err == nil {
t.Errorf("%s: taken", name) t.Errorf("%s: taken", name)
@@ -341,4 +471,113 @@ func TestPlacement(t *testing.T) {
} }
}) })
// ADR 0567 §2, §8: a proposal's changes laid over the current keys, what waits for the lease write, and the proposal's
// encoding and refusals.
t.Run("proposal", func(t *testing.T) {
current := map[string]Placement{"g14": g14, "novox": novox}
laid, findings := LayChanges(current, []PlacementProposalChange{
{Op: OpPlace, Node: "g14", Module: "new"},
{Op: OpTakeOff, Node: "g14", Module: "zsh"},
{Op: OpUnpin, Node: "g14", Provision: "shell"},
{Op: OpDecide, Node: "novox", Field: "site", Value: json.RawMessage(`"attic"`)},
})
if len(findings) != 0 || len(laid) != 2 || !reflect.DeepEqual(laid["g14"].Assigned(), []string{"app", "new"}) ||
len(laid["g14"].Holds) != 0 || laid["g14"].Pins != nil || laid["novox"].Decided.Site != "attic" || !laid["novox"].Decided.Hub {
t.Fatalf("laid %+v, %v", laid, findings)
}
if !reflect.DeepEqual(current["g14"].Assigned(), []string{"zsh", "app"}) {
t.Error("laying changed the current placement")
}
if fresh, f := LayChanges(current, []PlacementProposalChange{{Op: OpPlace, Node: "pi", Module: "zsh"}}); len(f) != 0 || fresh["pi"].Node != "pi" {
t.Errorf("a node with no key: %+v, %v", fresh, f)
}
for name, c := range map[string]struct {
change PlacementProposalChange
want string
}{
"not placed": {PlacementProposalChange{Op: OpTakeOff, Node: "g14", Module: "nope"}, "not placed"},
"no such field": {PlacementProposalChange{Op: OpDecide, Node: "g14", Field: "colour", Value: json.RawMessage(`"x"`)}, "no decided field"},
"wrong type": {PlacementProposalChange{Op: OpDecide, Node: "g14", Field: "hub", Value: json.RawMessage(`"x"`)}, "does not take"},
"unknown op": {PlacementProposalChange{Op: "move", Node: "g14"}, "no change of placement"},
"pin of nobody": {PlacementProposalChange{Op: OpPin, Node: "g14", Provision: "x"}, "provider"},
"not held": {PlacementProposalChange{Op: OpUnhold, Node: "g14", Module: "app", Claim: "x"}, "holds no"},
"bad port": {PlacementProposalChange{Op: OpPlace, Node: "g14", Module: "x", Ports: map[string]int{"a": 1}}, "port"},
"seed of another": {PlacementProposalChange{Op: OpSeed, Node: "g14", Whole: &novox}, "seed"},
} {
if _, f := LayChanges(current, []PlacementProposalChange{c.change}); len(f) != 1 || !strings.Contains(f[0], c.want) {
t.Errorf("%s: %v, want a finding naming %q", name, f, c.want)
}
}
for change, waits := range map[*PlacementProposalChange]bool{
{Op: OpPlace, Node: "g14", Module: "x"}: false, {Op: OpTakeOff, Node: "g14", Module: "x"}: false,
{Op: OpPin, Node: "g14"}: false, {Op: OpUnpin, Node: "g14"}: false, {Op: OpSeed, Node: "g14"}: false,
{Op: OpPlace, Node: "g14", Module: "x", Ports: map[string]int{"80": 80}}: true,
{Op: OpPlace, Node: "g14", Module: "x", Bindings: map[string]Provider{"p": {}}}: true,
{Op: OpHold, Node: "g14"}: true, {Op: OpDecide, Node: "g14"}: true,
{Op: OpConfig, Node: "g14", Values: map[string]any{"puid": 1}}: false, {Op: OpConfig, Node: "g14", MeshValues: map[string]any{"a": 1}}: false,
{Op: OpConfig, Node: "g14", Layer: map[string]any{"places": 1}}: true, {Op: OpConfig, Node: "g14", MeshWide: map[string]any{"accesses": 1}}: true,
} {
if got := change.BeforeLease(); (got != "") != waits || (waits && !strings.Contains(got, "lease write")) {
t.Errorf("%+v before the lease write: %q", *change, got)
}
}
const id = "p-20261028t183000-1a2b3c"
if got := NewProposalID(time.Date(2026, 10, 28, 18, 30, 0, 0, time.UTC)); !strings.HasPrefix(got, "p-20261028t183000-") || CheckProposalID(got) != nil {
t.Errorf("NewProposalID = %q", got)
}
p := PlacementProposal{ID: id, By: "operator@anchor", Why: "move the app", At: time.Unix(1, 0).UTC(),
Changes: []PlacementProposalChange{{Op: OpTakeOff, Node: "g14", Module: "zsh"}}, Base: map[string]uint64{"g14": 4},
State: ProposalAnswering}
raw, err := EncodePlacementProposal(p)
if err != nil {
t.Fatal(err)
}
if back, err := DecodePlacementProposal(raw); err != nil || back.ID != id || back.Base["g14"] != 4 || !reflect.DeepEqual(back.Nodes(), []string{"g14"}) {
t.Fatalf("round trip %+v, %v", back, err)
}
seed := p
seed.Seed, seed.ExportDigest = true, ExportDigest(current)
seed.Changes = []PlacementProposalChange{{Op: OpSeed, Node: "g14", Whole: &g14}}
if _, err := EncodePlacementProposal(seed); err != nil {
t.Errorf("the seed: %v", err)
}
if ExportDigest(current) != ExportDigest(map[string]Placement{"novox": novox, "g14": g14}) || ExportDigest(current) == ExportDigest(nil) {
t.Error("the export digest depends on order or not on the placements")
}
back := p
back.HandBack, back.Changes = true, nil
if _, err := EncodePlacementProposal(back); err != nil {
t.Errorf("a hand-back: %v", err)
}
if k, err := PulseAnswerKey(id, "g14"); err != nil || k != "answer."+id+".g14" {
t.Errorf("PulseAnswerKey = %q, %v", k, err)
}
if k, err := PulseAskKey(id); err != nil || k != "ask."+id || PlacementPulsesBucket != "mesh-delivery-assigner_pulses" {
t.Errorf("PulseAskKey = %q, %v", k, err)
}
if ChangesDigest(p) == ChangesDigest(seed) {
t.Error("two proposals' changes digest alike")
}
for name, mut := range map[string]func(*PlacementProposal){
"no id": func(p *PlacementProposal) { p.ID = "" },
"no proposer": func(p *PlacementProposal) { p.By = "" },
"no why": func(p *PlacementProposal) { p.Why = " " },
"no change": func(p *PlacementProposal) { p.Changes = nil },
"no base": func(p *PlacementProposal) { p.Base = map[string]uint64{"novox": 1} },
"no state": func(p *PlacementProposal) { p.State = "" },
"asked, no ask": func(p *PlacementProposal) { p.State = ProposalAsked },
"seed mixed": func(p *PlacementProposal) { p.Seed, p.ExportDigest = true, "d" },
"bad change": func(p *PlacementProposal) { p.Changes[0].Op = "swap" },
"seed, no digest": func(p *PlacementProposal) { *p = seed; p.ExportDigest = "" },
"hand-back with changes": func(p *PlacementProposal) { p.HandBack = true },
} {
q := p
q.Changes = append([]PlacementProposalChange(nil), p.Changes...)
mut(&q)
if _, err := EncodePlacementProposal(q); err == nil {
t.Errorf("%s: taken", name)
}
}
})
} }
+37
View File
@@ -0,0 +1,37 @@
package delivery
import (
"encoding/json"
"fmt"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/overlay"
)
// The modules the mesh provides itself (issue 1126): no repository holds them, so the indexer reads none; their
// manifests come with mesh-sdk, built in code, and always win over any other. Today the private network alone
// (overlay.Name), built by nobody and run inside each node's own composition. One predicate for every part that judges
// or composes a placement: CheckPlacements, the seat's holder, the composer's input (frombus) and the grants part.
// ProvidedModules is every module the mesh provides itself, sorted.
func ProvidedModules() []string { return []string{overlay.Name} }
// ProvidedByMesh says whether the mesh provides module itself, so a placement may name it with no repository holding it.
func ProvidedByMesh(module string) bool { return module == overlay.Name }
// ProvidedManifest is a provided module's manifest, as its readers parse it, and its raw JSON; an error for a module the
// mesh does not provide or a manifest its own reader refuses.
func ProvidedManifest(module string) (catalogue.Manifest, []byte, error) {
if !ProvidedByMesh(module) {
return catalogue.Manifest{}, nil, fmt.Errorf("delivery: the mesh provides no module %s", module)
}
raw, err := json.Marshal(overlay.Manifest())
if err != nil {
return catalogue.Manifest{}, nil, err
}
m, err := catalogue.ParseManifestForInputs(raw)
if err != nil {
return catalogue.Manifest{}, raw, fmt.Errorf("delivery: the manifest mesh-sdk provides for %s is refused by its own reader: %w", module, err)
}
return m, raw, nil
}
+350
View File
@@ -0,0 +1,350 @@
package delivery
import (
"crypto/ed25519"
"crypto/sha256"
"crypto/x509"
"crypto/x509/pkix"
"encoding/base64"
"encoding/hex"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"math/big"
"slices"
"strings"
"time"
)
// **The vault signs** (novox/hq ADR 0550 §2; design 47's L3 item d). The vault makes an ed25519 signing key and an x509
// authority once, keeps both as its own secrets with their operator copies, and publishes their public halves in its
// state `signers`, which only the vault writes and every node principal and composing part reads. Its verb `sign` signs a
// node's certificate for that node's own node-engine, from bus state alone (the name from the node's placement, the
// serving key from its record), and says `signed` naming the caller, the kind, the node and the digest, never the key.
//
// A node trusts the vault's key through the bus grants alone (the lead's design limit, 2026-10-28): only the vault may
// write `signers`, the same trust the wants and the placement rest on. Until the controller's lease is handed away the
// node trusts the controller's key beside it; from then on the vault's alone (LeaseHandedAway). What a node trusts now
// it states in its record, NodeRecord.Trusts, so the gate before the lease write reads it from the bus.
const (
// VaultModule is the module holding the seat mesh-vault, whose state `signers` is.
VaultModule = "mesh-vault"
// SignersState is the vault's state of its public signing halves, and SignersBucket its bucket.
SignersState = "signers"
SignersBucket = VaultModule + "_" + SignersState
// SignersRead is the state as a manifest's `reads` names it.
SignersRead = VaultModule + "." + SignersState
// SignersKeyKey holds the signing key's public half (SignerKey); SignersAuthorityKey the authority (SignerAuthority).
SignersKeyKey = "key"
SignersAuthorityKey = "authority"
// SignVerb is the vault's verb that signs.
SignVerb = "sign"
// SignedEvent is what the vault says when it signed (Signed), on SignedSubject.
SignedEvent = "signed"
SignedSubject = "mesh.mod." + VaultModule + ".event." + SignedEvent
// KindNodeCertificate is a node's certificate for its internal name, asked by that node's node-engine alone.
KindNodeCertificate = "node-certificate"
// KindEnrolment is an enrolment reply, signed under its own context; no caller is named until adoption's record
// (ADR 0561 §5).
KindEnrolment = "enrolment"
// KindAuthority is the vault's own authority certificate, which the vault signs for itself.
KindAuthority = "authority"
// SignerController and SignerVault name the two signers a node may trust.
SignerController = "controller"
SignerVault = "vault"
// CertificateLife is how long a certificate the vault signs for a node lasts: the controller's ten years
// (identity.forever), renewed by asking again, never by a clock running out unseen.
CertificateLife = 10 * 365 * 24 * time.Hour
)
// SignKinds are the kinds `sign` knows.
var SignKinds = []string{KindNodeCertificate, KindEnrolment, KindAuthority}
// SignContext is the line a message of kind is signed after, so a signature made for one kind never verifies as
// another's (as the composition's and the asks' contexts are kept apart).
func SignContext(kind string) (string, error) {
if !slices.Contains(SignKinds, kind) {
return "", fmt.Errorf("delivery: %q is not a kind the vault signs (%s)", kind, strings.Join(SignKinds, ", "))
}
return "novox " + kind + " v1\n", nil
}
// SignerKey is the key `key` of `signers`: the vault's signing key's public half in standard base64, its fingerprint,
// and when it was made.
type SignerKey struct {
Key string `json:"key"`
Fingerprint string `json:"fingerprint"`
Made time.Time `json:"made"`
}
// SignerAuthority is the key `authority` of `signers`: the vault's authority certificate as PEM, its fingerprint, and
// when it was made.
type SignerAuthority struct {
Certificate string `json:"certificate"`
Fingerprint string `json:"fingerprint"`
Made time.Time `json:"made"`
}
// SignerFingerprint is an ed25519 public key's fingerprint, `sha256:<hex>` of its 32 bytes.
func SignerFingerprint(key ed25519.PublicKey) string {
sum := sha256.Sum256(key)
return "sha256:" + hex.EncodeToString(sum[:])
}
// AuthorityFingerprint is an authority certificate's fingerprint, `sha256:<hex>` of its DER; refused when certificate is
// not one PEM certificate.
func AuthorityFingerprint(certificate string) (string, error) {
c, err := ParseCertificate(certificate)
if err != nil {
return "", err
}
sum := sha256.Sum256(c.Raw)
return "sha256:" + hex.EncodeToString(sum[:]), nil
}
// ParseCertificate is the first certificate of a PEM text.
func ParseCertificate(text string) (*x509.Certificate, error) {
block, _ := pem.Decode([]byte(strings.TrimSpace(text)))
if block == nil || block.Type != "CERTIFICATE" {
return nil, errors.New("delivery: not a PEM certificate")
}
return x509.ParseCertificate(block.Bytes)
}
// NewSignerKey is the value of `key` for pub.
func NewSignerKey(pub ed25519.PublicKey, made time.Time) SignerKey {
return SignerKey{Key: base64.StdEncoding.EncodeToString(pub), Fingerprint: SignerFingerprint(pub), Made: made.UTC()}
}
// PublicKey is k's key, checked: an ed25519 public key whose fingerprint is the one k names.
func (k SignerKey) PublicKey() (ed25519.PublicKey, error) {
raw, err := base64.StdEncoding.DecodeString(k.Key)
if err != nil || len(raw) != ed25519.PublicKeySize {
return nil, errors.New("delivery: the signers' key is not an ed25519 public key in base64")
}
if fp := SignerFingerprint(raw); fp != k.Fingerprint {
return nil, fmt.Errorf("delivery: the signers' key has the fingerprint %s, and names %s", fp, k.Fingerprint)
}
return ed25519.PublicKey(raw), nil
}
// DecodeSignerKey reads the key `key` of `signers`, refused when it does not check.
func DecodeSignerKey(raw []byte) (SignerKey, error) {
var k SignerKey
if err := json.Unmarshal(raw, &k); err != nil {
return SignerKey{}, fmt.Errorf("delivery: %s %s: %w", SignersRead, SignersKeyKey, err)
}
if _, err := k.PublicKey(); err != nil {
return SignerKey{}, err
}
return k, nil
}
// DecodeSignerAuthority reads the key `authority` of `signers`: one authority certificate (a CA that signs leaves
// alone) whose fingerprint is the one named.
func DecodeSignerAuthority(raw []byte) (SignerAuthority, error) {
var a SignerAuthority
if err := json.Unmarshal(raw, &a); err != nil {
return SignerAuthority{}, fmt.Errorf("delivery: %s %s: %w", SignersRead, SignersAuthorityKey, err)
}
c, err := ParseCertificate(a.Certificate)
if err != nil {
return SignerAuthority{}, err
}
if !c.IsCA {
return SignerAuthority{}, errors.New("delivery: the signers' authority is not an authority certificate")
}
if fp, _ := AuthorityFingerprint(a.Certificate); fp != a.Fingerprint {
return SignerAuthority{}, fmt.Errorf("delivery: the signers' authority has the fingerprint %s, and names %s", fp, a.Fingerprint)
}
return a, nil
}
// Signer is one key a node trusts, by who holds it (SignerController, SignerVault, or the operator's).
type Signer struct {
Name string
Key ed25519.PublicKey
}
// Fingerprint is the signer's key's fingerprint.
func (s Signer) Fingerprint() string { return SignerFingerprint(s.Key) }
// VerifyBy is the trusted signer whose key verifies sig over context and msg: refused when no trusted key does, saying
// so, and when a key is not an ed25519 key. A signature made under another context never verifies.
func VerifyBy(trusted []Signer, context string, msg, sig []byte) (Signer, error) {
if len(sig) != ed25519.SignatureSize {
return Signer{}, errors.New("delivery: not an ed25519 signature")
}
message := append([]byte(context), msg...)
for _, s := range trusted {
if len(s.Key) != ed25519.PublicKeySize {
continue
}
if ed25519.Verify(s.Key, message, sig) {
return s, nil
}
}
names := make([]string, 0, len(trusted))
for _, s := range trusted {
names = append(names, s.Name)
}
return Signer{}, fmt.Errorf("delivery: the signature verifies against no trusted key (%s)", strings.Join(names, ", "))
}
// SignMessage is sig over context and msg by key: what the vault's `sign` does for a kind whose bytes are given.
func SignMessage(key ed25519.PrivateKey, context string, msg []byte) []byte {
return ed25519.Sign(key, append([]byte(context), msg...))
}
// Signed is the vault's `signed`: who asked, the kind, the node, and the sha256 digest of what was signed. Never a key.
type Signed struct {
Caller string `json:"caller"`
Kind string `json:"kind"`
Node string `json:"node,omitempty"`
Digest string `json:"digest"`
Signer string `json:"signer"`
At time.Time `json:"at"`
}
// SignAnswer is what `sign` answers: for a node certificate the certificate as PEM and the authority that signed it,
// for an enrolment the signature in base64; always the signer's fingerprint and the digest. Never a key's private half.
type SignAnswer struct {
Kind string `json:"kind"`
Node string `json:"node,omitempty"`
Certificate string `json:"certificate,omitempty"`
Authority string `json:"authority,omitempty"`
Signature string `json:"signature,omitempty"`
Signer string `json:"signer"`
Digest string `json:"digest"`
}
// DigestOf is the digest a `signed` names: `sha256:<hex>` of the bytes signed.
func DigestOf(b []byte) string {
sum := sha256.Sum256(b)
return "sha256:" + hex.EncodeToString(sum[:])
}
// NodeCertificateTemplate is the certificate the vault signs for node's internal name, as the controller's authority
// shaped it (identity.Certify): the name in the subject and the subject alternative names, server and client use, the
// serial derived from the name and the key so the same ask gives the same certificate, and its life from notBefore.
func NodeCertificateTemplate(name string, servingKey ed25519.PublicKey, notBefore time.Time) *x509.Certificate {
sum := sha256.Sum256(append([]byte(name+"\n"), servingKey...))
return &x509.Certificate{
SerialNumber: new(big.Int).SetBytes(sum[:16]),
Subject: pkix.Name{CommonName: name},
DNSNames: []string{name},
NotBefore: notBefore.UTC().Truncate(time.Second),
NotAfter: notBefore.UTC().Truncate(time.Second).Add(CertificateLife),
KeyUsage: x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth},
}
}
// AuthorityTemplate is the vault's authority certificate for its key, made once: an authority that signs leaves and no
// other authority, as the controller's.
func AuthorityTemplate(key ed25519.PublicKey, made time.Time) *x509.Certificate {
sum := sha256.Sum256(append([]byte("authority\n"), key...))
return &x509.Certificate{
SerialNumber: new(big.Int).SetBytes(sum[:16]),
Subject: pkix.Name{CommonName: "the mesh's vault"},
NotBefore: made.UTC().Truncate(time.Second),
NotAfter: made.UTC().Truncate(time.Second).Add(2 * CertificateLife),
IsCA: true,
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign,
BasicConstraintsValid: true,
MaxPathLen: 0,
MaxPathLenZero: true,
}
}
// CheckNodeCertificate says why certificate is not a certificate of name for servingKey by one of the authorities, or
// nil: a node uses it to tell whether its record holds a certificate from an authority it trusts, and another node to
// check one the vault signed (ADR 0550 §2's live check).
func CheckNodeCertificate(certificate string, authorities []string, name string, servingKey ed25519.PublicKey, at time.Time) error {
c, err := ParseCertificate(certificate)
if err != nil {
return err
}
pool := x509.NewCertPool()
for _, a := range authorities {
pool.AppendCertsFromPEM([]byte(a)) // a bundle is every certificate it holds
}
if _, err := c.Verify(x509.VerifyOptions{Roots: pool, DNSName: name, CurrentTime: at,
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}}); err != nil {
return fmt.Errorf("delivery: the certificate of %s is not one a trusted authority signed for it: %w", name, err)
}
held, ok := c.PublicKey.(ed25519.PublicKey)
if !ok || !held.Equal(servingKey) {
return fmt.Errorf("delivery: the certificate of %s binds another key than the node serves with", name)
}
return nil
}
// AuthorityBundle is the authorities given as one PEM text, each certificate once, in the order given, blanks left out:
// what a module asking for the mesh's authority is given once the vault signs beside the controller (ADR 0550 §2). An
// authority given is split into its certificates, so a bundle given again (the one a node kept from its last
// composition) adds nothing twice; a text that holds no PEM block is taken whole.
func AuthorityBundle(authorities ...string) string {
var out []string
for _, a := range authorities {
for _, piece := range pemPieces(a) {
if !slices.Contains(out, piece) {
out = append(out, piece)
}
}
}
if len(out) == 0 {
return ""
}
return strings.Join(out, "\n") + "\n"
}
// pemPieces is each PEM block of text re-encoded, or the trimmed text whole when it holds none.
func pemPieces(text string) []string {
text = strings.TrimSpace(text)
if text == "" {
return nil
}
var out []string
rest := []byte(text)
for {
block, more := pem.Decode(rest)
if block == nil {
break
}
out = append(out, strings.TrimSpace(string(pem.EncodeToMemory(block))))
rest = more
}
if len(out) == 0 {
return []string{text}
}
return out
}
// Trusts is what a node states it trusts now (NodeRecord.Trusts): `<who> key <fingerprint>` for each signer and
// `<who> authority <fingerprint>` for each authority certificate, by who holds it, each once, sorted. The gate before the
// lease write reads `vault key <fingerprint>` in every node's record.
func Trusts(signers []Signer, authorities map[string]string) []string {
var out []string
for _, s := range signers {
if len(s.Key) == ed25519.PublicKeySize {
out = append(out, s.Name+" key "+s.Fingerprint())
}
}
for who, a := range authorities {
if fp, err := AuthorityFingerprint(a); err == nil {
out = append(out, who+" authority "+fp)
}
}
slices.Sort(out)
return slices.Compact(out)
}
// TrustsKey says whether trusts names the key with fingerprint fp held by who.
func TrustsKey(trusts []string, who, fp string) bool { return slices.Contains(trusts, who+" key "+fp) }
+8 -19
View File
@@ -537,13 +537,11 @@ type Forge struct {
Act string `json:"act,omitempty"` Act string `json:"act,omitempty"`
Words string `json:"words,omitempty"` Words string `json:"words,omitempty"`
Open []OpenPull `json:"open,omitempty"` Open []OpenPull `json:"open,omitempty"`
// Placements are, on a merged message of the catalogue repository, every placement file of the merge commit, the // RetiredPlacements is the placement files a bridge before ADR 0567 carried (ADR 0549 §2, ADR 0559 §2): placement is
// bytes exact, by node (PlacementFile; novox/hq ADR 0549 §2), and PlacementsRead says they were read, so a catalogue // the seat mesh-placement's state now and no repository holds it (ADR 0567 §11), so a message carrying them is
// whose last placement file was removed is told from a message that read none (older writers, another repository). // refused, loudly, rather than read with them dropped.
// On a pull message of the catalogue (CatalogueRepository) they are every placement file at the pull request's head, RetiredPlacements json.RawMessage `json:"placements,omitempty"`
// read whole or not at all, which the indexer writes as placement candidates (PlacementCandidate). RetiredRead bool `json:"placements-read,omitempty"`
Placements map[string]string `json:"placements,omitempty"`
PlacementsRead bool `json:"placements-read,omitempty"`
} }
func (f Forge) check() error { func (f Forge) check() error {
@@ -556,18 +554,9 @@ func (f Forge) check() error {
if len(f.Open) > 0 { if len(f.Open) > 0 {
return fmt.Errorf("delivery: only an open message names open pull requests") return fmt.Errorf("delivery: only an open message names open pull requests")
} }
// A pull message of the catalogue carries its head's placement files, so its change of placement is composed and if len(f.RetiredPlacements) > 0 || f.RetiredRead {
// answered by the nodes it places before it merges (placement candidates, issue 1126; ADR 0549 §4). return fmt.Errorf("delivery: a forge message carries placement files, which no repository holds since ADR 0567: " +
if (len(f.Placements) > 0 || f.PlacementsRead) && f.Kind != ForgeMerged && (f.Kind != ForgePull || f.Repository != CatalogueRepository) { "placement is the seat mesh-placement's state")
return fmt.Errorf("delivery: only a merged message, or a pull message of the catalogue, carries placement files")
}
if len(f.Placements) > 0 && !f.PlacementsRead {
return fmt.Errorf("delivery: a message with placement files must say it read them (placements-read)")
}
for node := range f.Placements {
if err := checkName("node", node); err != nil {
return fmt.Errorf("delivery: a placement file names no node: %w", err)
}
} }
if err := checkRepository(f.Repository); err != nil { if err := checkRepository(f.Repository); err != nil {
return err return err
+22
View File
@@ -340,8 +340,20 @@ type CheckJob struct {
// Packages, when set, are the only packages each level tests (issue 890, ADR 0552 §4): a placed ask names the // Packages, when set, are the only packages each level tests (issue 890, ADR 0552 §4): a placed ask names the
// packages the first ask left unmet, and the others are neither tested nor taken kept, so nothing is judged twice. // packages the first ask left unmet, and the others are neither tested nor taken kept, so nothing is judged twice.
Packages []string `json:"packages,omitempty"` Packages []string `json:"packages,omitempty"`
// NodeNames are the names of every node with a record (novox/hq ADR 0567 §12), read by the checker from each
// `<node>.node` of node-engine's have: the names check of ADR 0507 §9 looks for them in the tree, so no repository
// has to list them. The build step hands them to its test tool as the argument NodeNamesArg, and the build module
// gives them to the tests in NodeNamesEnv.
NodeNames []string `json:"node-names,omitempty"`
} }
// NodeNamesArg is the test tool's argument that carries a check's node names, and NodeNamesEnv the environment
// variable a build module sets for the tests it runs from it, the names separated by commas.
const (
NodeNamesArg = "node-names"
NodeNamesEnv = "MESH_NODE_NAMES"
)
func (a StepAsk) check() error { func (a StepAsk) check() error {
if err := checkAskID(a.ID); err != nil { if err := checkAskID(a.ID); err != nil {
return err return err
@@ -447,6 +459,13 @@ func (c CheckJob) check(id string) error {
return fmt.Errorf("delivery: the check ask %s names a package %q that is no import path", id, p) return fmt.Errorf("delivery: the check ask %s names a package %q that is no import path", id, p)
} }
} }
seen := map[string]bool{}
for _, n := range c.NodeNames {
if err := checkName("node", n); err != nil || strings.Contains(n, ",") || seen[n] {
return fmt.Errorf("delivery: the check ask %s names the node %q, which is no node name or named twice", id, n)
}
seen[n] = true
}
return checkFetchArgs(id, c.Fetch) return checkFetchArgs(id, c.Fetch)
} }
@@ -730,6 +749,9 @@ type Capacity struct {
OutsideHours bool `json:"outside-hours,omitempty"` OutsideHours bool `json:"outside-hours,omitempty"`
// BuildHoursFault is why the node's build-hours value cannot be read, naming the value: it then builds at any hour. // BuildHoursFault is why the node's build-hours value cannot be read, naming the value: it then builds at any hour.
BuildHoursFault string `json:"build-hours-fault,omitempty"` BuildHoursFault string `json:"build-hours-fault,omitempty"`
// Ineligible is why the module found its node cannot run its asks (buildstep.Puller.Eligible), absent when it can:
// Effective is then 0 at every hour.
Ineligible string `json:"ineligible,omitempty"`
// Why is the capacity in words: what bounds it. // Why is the capacity in words: what bounds it.
Why string `json:"why"` Why string `json:"why"`
At time.Time `json:"at,omitzero"` At time.Time `json:"at,omitzero"`
+10 -1
View File
@@ -24,7 +24,8 @@ func checkAsk() StepAsk {
return StepAsk{ID: "mh3k2x9qa1b2c4", Kind: AskCheck, Language: AskNode, Asker: "mesh-delivery-checker", return StepAsk{ID: "mh3k2x9qa1b2c4", Kind: AskCheck, Language: AskNode, Asker: "mesh-delivery-checker",
At: time.Date(2026, 10, 23, 13, 0, 0, 0, time.UTC), At: time.Date(2026, 10, 23, 13, 0, 0, 0, time.UTC),
Check: &CheckJob{Fetch: map[string]any{"repository": "novox/mesh-catalog", "commit": askCommit, "ref": "refs/pull/3/head", "base": askCommit}, Check: &CheckJob{Fetch: map[string]any{"repository": "novox/mesh-catalog", "commit": askCommit, "ref": "refs/pull/3/head", "base": askCommit},
Levels: []string{LevelUnit, LevelIntegration}, Kept: true, Runner: "workspace", RunnerWhen: "modules/mesh-build-go/", Run: ""}} Levels: []string{LevelUnit, LevelIntegration}, Kept: true, Runner: "workspace", RunnerWhen: "modules/mesh-build-go/", Run: "",
NodeNames: []string{"anchor", "laptop"}}}
} }
// 1887/A1a: an ask encodes and decodes with every field. // 1887/A1a: an ask encodes and decodes with every field.
@@ -65,6 +66,14 @@ func TestAskRefusesUnknownKindAndLanguage(t *testing.T) {
t.Errorf("%s: encoded", name) t.Errorf("%s: encoded", name)
} }
} }
// ADR 0567 §12: a check's node names are each a node name, once.
for _, names := range [][]string{{"anchor", "anchor"}, {"a.b"}, {""}, {"a,b"}} {
c := checkAsk()
c.Check.NodeNames = names
if _, err := EncodeStepAsk(c); err == nil {
t.Errorf("the node names %q: encoded", names)
}
}
// A copy alone fetches nothing and needs no path. // A copy alone fetches nothing and needs no path.
a := buildAsk() a := buildAsk()
a.Build.Fetch, a.Build.Path, a.Build.Artifacts = nil, "", []AskArtifact{{Artifact: "base", From: "registry/x@sha256:00"}} a.Build.Fetch, a.Build.Path, a.Build.Artifacts = nil, "", []AskArtifact{{Artifact: "base", From: "registry/x@sha256:00"}}
+23 -4
View File
@@ -38,14 +38,19 @@ const (
// AskKindRecord is the kind of the assigner's ask about a move a person must allow, for a module of policy record: // AskKindRecord is the kind of the assigner's ask about a move a person must allow, for a module of policy record:
// a composition judged to recreate a container or move data, or one that cannot be judged (ADR 0539 clause 14). // a composition judged to recreate a container or move data, or one that cannot be judged (ADR 0539 clause 14).
AskKindRecord = "record" AskKindRecord = "record"
// AskKindPlacement is the kind of the seat mesh-placement's authorising ask about a proposal (novox/hq ADR 0567 §5):
// `placement.<proposal id>`. The kind needs no proof, as the bus's put does not: its yes is an Approve, which only a
// channel proving who answered takes and the view never does (ViewAnswerFits).
AskKindPlacement = "placement"
viewWord = "view" viewWord = "view"
) )
// AskKey is an ask's key read whole: its Kind, and the parts its kind has: a merge's Group and heads Hash, a hold's // AskKey is an ask's key read whole: its Kind, and the parts its kind has: a merge's Group and heads Hash, a hold's
// Module and fingerprint FP, a restore's and a backup's Node, Module and FP, a record's Node, Module and the digest // Module and fingerprint FP, a restore's and a backup's Node, Module and FP, a record's Node, Module and the digest
// Composed of the composition it would allow. // Composed of the composition it would allow, a placement's Proposal.
type AskKey struct { type AskKey struct {
Proposal string
Kind string Kind string
Group string Group string
Hash string Hash string
@@ -61,11 +66,11 @@ func AskNeedsProof(kind string) (bool, error) {
switch kind { switch kind {
case AskKindMerge: case AskKindMerge:
return true, nil return true, nil
case AskKindHold, AskKindRestore, AskKindBackup, AskKindRecord: case AskKindHold, AskKindRestore, AskKindBackup, AskKindRecord, AskKindPlacement:
return false, nil return false, nil
} }
return false, fmt.Errorf("delivery: %q is not a kind of ask, %s, %s, %s, %s or %s", kind, AskKindMerge, AskKindHold, return false, fmt.Errorf("delivery: %q is not a kind of ask, %s, %s, %s, %s, %s or %s", kind, AskKindMerge, AskKindHold,
AskKindRestore, AskKindBackup, AskKindRecord) AskKindRestore, AskKindBackup, AskKindRecord, AskKindPlacement)
} }
// HoldAskKey is the key, in AssignerAsksBucket, of the ask about a module's held fingerprint: `hold.<module>.<hex>`. // HoldAskKey is the key, in AssignerAsksBucket, of the ask about a module's held fingerprint: `hold.<module>.<hex>`.
@@ -100,6 +105,15 @@ func BackupAskKey(node, module, fingerprint string) (string, error) {
return AskKindBackup + "." + key, nil return AskKindBackup + "." + key, nil
} }
// PlacementAskKey is the key, in AssignerAsksBucket, of the authorising ask about a proposal of placement:
// `placement.<proposal id>` (ADR 0567 §5).
func PlacementAskKey(proposal string) (string, error) {
if err := CheckProposalID(proposal); err != nil {
return "", err
}
return AskKindPlacement + "." + proposal, nil
}
// RecordAskKey is the key, in AssignerAsksBucket, of the ask about a move of a module on a node to the composition // RecordAskKey is the key, in AssignerAsksBucket, of the ask about a move of a module on a node to the composition
// whose digest it names: `record.<node>.<module>.<digest hex>`, so a new revision at the same fingerprint is asked // whose digest it names: `record.<node>.<module>.<digest hex>`, so a new revision at the same fingerprint is asked
// again (ADR 0539 clause 14). // again (ADR 0539 clause 14).
@@ -154,6 +168,11 @@ func ParseAnyAskKey(key string) (AskKey, error) {
return refused(fmt.Errorf("a backup ask is backup.<node>.<module>.<fingerprint hex>")) return refused(fmt.Errorf("a backup ask is backup.<node>.<module>.<fingerprint hex>"))
} }
return AskKey{Kind: kind, Node: node, Module: module, FP: fp}, nil return AskKey{Kind: kind, Node: node, Module: module, FP: fp}, nil
case AskKindPlacement:
if err := CheckProposalID(rest); err != nil {
return refused(err)
}
return AskKey{Kind: kind, Proposal: rest}, nil
case AskKindRecord: case AskKindRecord:
parts := strings.Split(rest, ".") parts := strings.Split(rest, ".")
if len(parts) != 3 { if len(parts) != 3 {
+11 -3
View File
@@ -33,9 +33,17 @@ func TestParseAnyAskKey(t *testing.T) {
if err != nil || restore != "restore.ace.mesh-mail."+strings.Repeat("d", 64) { if err != nil || restore != "restore.ace.mesh-mail."+strings.Repeat("d", 64) {
t.Fatalf("RestoreAskKey = %q, %v", restore, err) t.Fatalf("RestoreAskKey = %q, %v", restore, err)
} }
placement, err := PlacementAskKey("p-one")
if err != nil || placement != "placement.p-one" {
t.Fatalf("PlacementAskKey = %q, %v", placement, err)
}
if proof, err := AskNeedsProof(AskKindPlacement); proof || err != nil {
t.Error("the assigner cannot keep a placement ask: its kind needs proof (its yes's level does)")
}
for key, want := range map[string]AskKey{ for key, want := range map[string]AskKey{
hold: {Kind: AskKindHold, Module: "mesh-mail", FP: fp}, hold: {Kind: AskKindHold, Module: "mesh-mail", FP: fp},
restore: {Kind: AskKindRestore, Node: "ace", Module: "mesh-mail", FP: fp}, restore: {Kind: AskKindRestore, Node: "ace", Module: "mesh-mail", FP: fp},
placement: {Kind: AskKindPlacement, Proposal: "p-one"},
} { } {
if got, err := ParseAnyAskKey(key); err != nil || got != want { if got, err := ParseAnyAskKey(key); err != nil || got != want {
t.Errorf("ParseAnyAskKey(%q) = %+v, %v", key, got, err) t.Errorf("ParseAnyAskKey(%q) = %+v, %v", key, got, err)
@@ -44,7 +52,7 @@ func TestParseAnyAskKey(t *testing.T) {
if got, err := ParseAnyAskKey(merge); err != nil || got.Kind != AskKindMerge || got.Group != "feat/r4g" || len(got.Hash) != 12 { if got, err := ParseAnyAskKey(merge); err != nil || got.Kind != AskKindMerge || got.Group != "feat/r4g" || len(got.Hash) != 12 {
t.Errorf("ParseAnyAskKey(%q) = %+v, %v", merge, got, err) t.Errorf("ParseAnyAskKey(%q) = %+v, %v", merge, got, err)
} }
for _, bad := range []string{"want.ace.mesh-mail", "hold.mesh-mail", "restore.mesh-mail." + strings.Repeat("d", 64), "approve.x.y", "restore.a.b.c.d", ""} { for _, bad := range []string{"want.ace.mesh-mail", "hold.mesh-mail", "restore.mesh-mail." + strings.Repeat("d", 64), "approve.x.y", "restore.a.b.c.d", "placement.P", ""} {
if _, err := ParseAnyAskKey(bad); err == nil || !strings.Contains(err.Error(), "\""+bad+"\"") { if _, err := ParseAnyAskKey(bad); err == nil || !strings.Contains(err.Error(), "\""+bad+"\"") {
t.Errorf("ParseAnyAskKey(%q) = %v, want a refusal naming it", bad, err) t.Errorf("ParseAnyAskKey(%q) = %v, want a refusal naming it", bad, err)
} }